Skip to main content

Media types and metadata

Every media type follows the same pattern: the scanner registers entities from the path, *_FOUND events trigger enrichment from an external provider, images flow through IMAGE_FOUND. Metadata providers live in worker/.../events/; their HTTP clients are centralized in worker/.../http/MetadataRestClients, and every external base URL is a property with the real service as default — the chart's e2e serves them all from one WireMock pod and fails on dead-lettered events, so a hardcoded URL breaks CI.

Languages​

app.ister.languages / ISTER_LANGUAGES (ISO-639-1 tags, default en,nl, exposed as LanguageProperties) is the single app-wide language list. Movie/show/episode handlers fetch TMDB details once per configured tag, producing one MetadataEntity row per language per item; the language is stored as ISO-639-3 even though the fetch uses ISO-639-1. The first tag is the primary/fallback language. The same list drives the search schema (chapter 6). Adding a language requires a re-scan plus a reindex.

Movies, shows, episodes (TMDB)​

MOVIE_FOUND / SHOW_FOUND / EPISODE_FOUND handlers fetch TMDB details per language, save MetadataEntity rows, and download posters/backdrops (emitted as IMAGE_FOUND on the cache directory).

The details response is mined for more than title/overview. Per language the metadata row also carries the localized genre list (comma-separated names — this is what lights up the GENRE filter and the genre_<tag> search fields for video) and the tagline. Language-independent facts are columns on the entity itself, applied once on the first successful language: tmdbId, imdbId, voteAverage/voteCount (null while there are zero votes), runtime (movies/episodes, minutes), status, homepage, originCountry, studios, collection id/name (movies, from belongs_to_collection) and networks (shows). After the language loop, TmdbExtrasService makes a handful of extra endpoint calls per item — release dates/content ratings (certification for app.ister.worker.tmdb.certification-country, default US, falling back to US and then any), videos (one YouTube trailer key, preferring official trailers), keywords, and TV external ids (imdb). Each extras call is individually fault-tolerant: a failure logs and leaves the field null instead of dead-lettering the event (the chart e2e stubs TMDB with WireMock).

Credits come along in the same pass: movie credits, show aggregate credits, and episode credits (cast + guest stars) become PersonEntity + CreditEntity rows, written directly to the database. A CreditEntity links a person to exactly one of movie/show/episode. A PersonEntity is shared between actors and music artists; TMDB cast members are deduplicated against existing persons on exact name + birth year. The GraphQL Credit type exposes the back-references (movie/show/episode, batch-resolved in CreditController), so a filmography is queryable via personById { credits { movie/show/episode } }.

Show.related(limit) (ShowController, ShowRepository.findRelatedShowIds) turns that same enrichment into a "more like this" list, without any extra provider call: it scores the other shows of the same library on shared TMDB keywords (3.0 each, capped at five so one long keyword list cannot dominate), shared genres (1.5 each) and shared cast members (1.0 each, capped at three). An overlapping network, origin country or a release year within five years adds half a point each, but only ranks shows that already share content — a common broadcast year is not a relation by itself. Shows that share nothing are left out, so a show without TMDB enrichment yields an empty list instead of arbitrary neighbours, and the list gets better with every refreshMetadata.

Keywords, networks and origin countries are the comma-space joined strings on show_entity; genres live per language on the metadata rows, so the query pairs the two shows' metadata rows on language (the genre names themselves are translated) and takes the best-matching language. The candidates are by definition in the library the caller was already granted access to through showById, so the field needs no access check of its own. It is deliberately not a @BatchMapping: only the show detail page asks for it, one show at a time.

Music (MusicBrainz)​

Artist directories become PersonEntity rows (PERSON_FOUND), albums AlbumEntity (ALBUM_FOUND), tracks via AUDIO_FILE_FOUND (ffprobe + ID3 tags + embedded cover). Album identity comes from the path, never from tags. The track's artist, however, comes from the artist tag (falling back to the path artist): on compilation albums the per-track performer only exists there, while the album keeps the path-derived album artist. The worker's HandleAlbumFound queries MusicBrainz and downloads the release-group cover; the disk-side HandlePersonFound/HandleAlbumFound look for artist.nfo/album.nfo. Artists get a birthYear (MusicBrainz life-span, or the folder name) precisely so the TMDB actor dedup above can match them.

Every track also carries credits (TrackCreditEntity): the primary artist — the same person as TrackEntity.personEntity — plus any featured guests, one row per person. That join table is what makes an artist's compilation and guest appearances reachable from their page (chapter 7); the album keeps a single album artist.

Person bios and portraits (Wikipedia/Wikidata)​

WikipediaService (worker) enriches persons with multilingual biographies and portraits: Wikidata resolves the entity and its image/sitelinks, the Wikipedia summary endpoint (a URL template property) supplies per-language extracts. The same service backs comic-series descriptions.

Books (LibraryType.BOOK)​

Directory grammar is author-first: Author/Book.epub and Author/Book/NNN_Chapter.mp3. All formats of one (normalized) book name converge on a single BookEntity (author = PersonEntity); formats are attachments — epubs link via MediaFileEntity.bookEntity, audiobook mp3s via ChapterEntity (streamed over the same audio-only HLS path as tracks). HandleAudioFileFound branches on library type to create chapters instead of tracks.

  • Media overlays (EPUB 3 read-aloud) are flagged on MediaFileEntity.mediaOverlays, detected exclusively from the epub contents (SMIL entries in the OPF manifest, parsed by disk/.../epub/EpubParser) — never from the filename.
  • BOOK_FOUND triggers Open Library enrichment (description, and a cover only when none exists yet); Wikidata adds series membership (series name + position); NFO data is deduplicated against provider data so re-scans do not double descriptions.
  • Series (BookSeriesService, core) come from three sources with fixed precedence: epub series metadata (calibre / EPUB 3 belongs-to-collection) is authoritative and rewrites the link on every scan; a path-prefix heuristic fills series-less books when ≥2 books of the author share the prefix before a -/: separator; and Wikidata series discovery (WikidataBookSeriesService.discoverSeries, run from BOOK_FOUND) links a series-less book into one of the author's existing series via its P179 (part of series) statement — it never creates a series, and requires a P50 (author) label match so the same-titled film or game can never link. The position/original-year enrichment for books that already have a series (findBookInSeries) applies the same P50 check: the series label alone cannot reject the film, because a film series may carry exactly the same label as the book series ("Harry Potter"). Label matching accepts Wikidata's mul (default-for-all-languages) label next to the configured languages: bots consolidate labels that are identical across languages into it, deleting the per-language ones. Discovery covers what the other two can't see: titles without a separator ("Harry Potter en de steen der wijzen") and audiobook-only books without epub metadata. When epub metadata creates a series, BOOK_FOUND re-fires once for the author's series-less books, so discovery converges within one scan regardless of scan order. The analyze backfill also re-dispatches BOOK_FOUND for every series-less book whose author has a series — a rescan skips already-known files, so without this a book whose discovery ran too early (or failed against Wikidata at the time) would stay out of its series forever.
  • Epubs are read lazily by the client through GET /epub/{mediaFileId}/resource/{entry} (chapter 7); reading position is a WatchStatusEntity with readingLocation (epubcfi) + readingProgress.

Comics (LibraryType.COMIC, migration V23)​

Comics are series-first, the opposite of the book grammar: {root}/{Series Name (start year)}/Volume 27.cbz (also Vol 3 - Subtitle.pdf, Issue 8.epub, and tolerated wild patterns; cover.jpg in the series directory is series artwork). The (YYYY) suffix is the series start year, not an author year — comics have no author in the path; parsing lives in ComicPathObject + ComicFileNameParser, anything nested deeper than the series directory is ignored.

ComicScanner creates the SeriesEntity and the volume (a BookEntity without author) from the path and attaches the file as a MediaFileEntity; all formats of one volume (same basename) converge on one volume row. Content reading is asynchronous: HandleComicFileFound (disk) reads cbz via CbzParser and PDF via PDFBox (PdfParser) — page count onto the MediaFileEntity, embedded ComicInfo.xml (cbz) becomes volume metadata and can refine the filename-derived series position and title, and the cover (first cbz page, or PDF page 1 rendered) is extracted to the cache. Epub volumes reuse the full EPUB_FILE_FOUND pipeline. COMIC_SERIES_FOUND (worker, HandleComicSeriesFound) adds per-language series descriptions and a thumbnail from Wikipedia/Wikidata — local artwork always wins and is never overwritten. Pages are served to the reader by ComicResourceController (/comic/{mediaFileId}/manifest, /page/{index}, /file).

Podcasts (LibraryType.PODCAST)​

The first feed-based library type: there is no library directory. subscribePodcast(feedUrl) or the hourly PodcastRefreshScheduler (guarded by lastRefreshedAt so multiple nodes do not sweep twice) sends PODCAST_REFRESH_REQUESTED (global queue). The worker's RssFeedParser fetches the feed with a conditional GET (ETag/Last-Modified), caps at 500 items, syncs channel metadata + cover, and creates PodcastEpisodeEntity rows deduplicated on guid.

The newest N episodes (app.ister.worker.podcast.auto-download-count, default 3) get PODCAST_EPISODE_DOWNLOAD_REQUESTED on the cache-directory queue of the refreshing node; the disk handler downloads the enclosure (following redirects) to {cache}/podcasts/ and emits AUDIO_FILE_FOUND, after which playback is identical to tracks. Older episodes download on demand via the downloadPodcastEpisode mutation. Retention: the daily cache cleanup deletes downloads older than podcast-retention-days (default 30) unless someone is mid-episode — the episode row survives and can re-download. Directory search uses the free iTunes Search API (ItunesSearchService, api module).

NFO files​

HandleNfoFileFound (disk) parses XML NFO files into metadata: title, description, release date for film/TV, biography for artists, review for albums, book metadata for books. PERSON_FOUND/ALBUM_FOUND on the disk side proactively look for artist.nfo/album.nfo next to the media. NFO-sourced and provider-sourced metadata are deduplicated so neither overwrites richer data from the other.

Last updated on by Gerben Droogers