Transcoding

Streaming is HLS, produced by FFmpeg (via Jaffree) in the transcoder module. HlsService + HlsTranscodeService coordinate playlist building and the FFmpeg processes. See the transcode-flow diagram; three triggers feed the same queue: interactive playback, the periodic pre-transcode task, and the play-queue prefetch.

Playlists up-front, segments lazy

A GET .../master.m3u8 cache miss sends TRANSCODE_REQUESTED; HandleTranscodeRequested → generateAllPlaylists writes the master and per-stream .m3u8 files to tmpDir/{mediaFileId}/, while the HTTP thread polls until they exist. Segments are only produced when asked for: the first .ts request for a quality level sends TRANSCODE_PASS_REQUESTED, and segment requests poll the cache directory until the pass has written (and closed) that segment.

One continuous pass per quality

Each quality level is one continuous FFmpeg pass over the whole file, not one process per segment, using -f segment -segment_times so the encoder never resets PTS — that is what prevents A/V drift. The trade-off: passes encode sequentially from t=0, so a forward seek waits for the encoder to catch up to the requested segment.

This includes the copy (direct-play) qualities: video is stream-copied and cut on the same grid the playlists advertise (see the next section for where that grid stops), and copy audio keeps any MPEG-TS-native codec (AAC, MP3, AC-3, E-AC-3, DTS) untouched, falling back to AAC only for codecs MPEG-TS cannot carry. Copy audio used to be advertised as a single whole-file segment generated on demand; on long files that blocked the first request for minutes and starved the client's audio stream while video segments raced ahead. Video passes also set omit_video_pes_length=0: without explicit PES lengths the final PES packet of every segment is unbounded, and a client reading segments back to back flags it corrupt on each boundary — a decode hiccup every few seconds.

The grid stops where the stream stops

The cut grid comes from the video keyframes, but each stream ends somewhere else. The segment muxer cuts at the first keyframe at or after each requested time, so a cut with nothing left to cut at produces no file — and a playlist that advertised it was promising something no request could ever be answered with. Two shapes occur in practice: audio commonly ends before the container does (ordinary in mkv), and the last cut of a video copy can land on the very last keyframe, where whether it still splits depends on timestamp rounding and the cut tolerance.

Trimming such a boundary costs nothing. The grid feeds the FFmpeg pass as well as the playlist, so the segment is not orphaned — the previous one runs to the end of the stream instead, and a packet count over a trimmed grid shows every packet of the source still present.

SegmentGrid is therefore built per stream and per role: HlsTranscodeService.gridFor measures where that stream ends — the video packet scan reports both the last keyframe (for a copy) and the end of the last packet (for a re-encode), and audio gets one extra ffprobe that reads only the final 30 seconds through -read_intervals. Boundaries that would leave less than a quarter of a second are dropped, and the last #EXTINF follows the measured end rather than the container duration. A probe that cannot measure falls back to the container duration and trims nothing, so a failing probe can never shorten a playlist. Remote inputs are never probed this way: they are read over /mediaFile/{id}/download, which serves no byte ranges, so a seeking probe would degrade into streaming the whole file across the network.

The playlist and the pass both go through gridFor with the same arguments, so what is advertised and what is produced cannot drift apart. As a net for the cases the measurement cannot reach, the segments a finished pass produced are counted before its done marker is written and the playlist is cut back to them; that also repairs cache directories written before this existed. A segment a completed pass never wrote answers 404, not 503 — it is not coming back, and "try again" made clients retry for hours.

The visible consequence: for such a file the playlist can end a fraction of a second, and for audio a second or so, before the container duration says. No media is lost — the final segment carries it — only the advertised duration is now the honest one.

Concurrency

transcodeExecutor is a fixed pool sized by app.ister.transcoder.hls.max-concurrent-passes (default 4), additionally bounded by the concurrentFileSlots semaphore (max-concurrent-files, default 2). A pass holds a thread for the entire file duration. Pre-transcoding competes for the same pool, so it is easy to starve interactive playback — which is why background work is throttled and preemptible (below).

The request side: waiting without holding anything

A playlist or segment request that arrives before the transcoder has produced the file parks in a poll loop (SharedTmpReadThrough) for up to master-playlist-timeout-ms / segment-timeout-ms. Two things keep that cheap:

• No database connection during the wait. HlsService reads its entities in short read-only transactions and hands plain values to the loops, and the HLS paths (plus the node-to-node /mediaFile/{id}/download streams) are excluded from open-in-view in OpenInViewConfig. With the default open-in-view wiring the request's Hibernate session, and with it a pooled JDBC connection, lived until the response was written, so every waiting player cost one of the pool's connections. HlsWaitConnectionIntegrationTest samples the Hikari pool during such a wait and fails if a connection is held.
• No platform thread either. spring.threads.virtual.enabled is on (core.yaml, fixed when the native image is built because it drives bean conditions): Tomcat, the RabbitMQ listeners and the schedulers run on virtual threads, so a parked request is a parked continuation rather than one of Tomcat's 200 threads. Java 25 does not pin a virtual thread inside synchronized, so the monitors the transcoder holds around ffmpeg runs need no special treatment. One thing virtual threads must not change: with them enabled Spring Boot would run every GraphQL resolver on the application task executor, off the request thread and therefore without its open-in-view session. GraphQlControllerConfig (api) defines the controller configurer without that executor, so resolvers keep running on the request thread.

Pre-transcoding and background priority

PRE_TRANSCODE_RECENTLY_WATCHED (per disk, every 15 minutes) reads the continue-watching entries (chapter 5) — exactly the items users will play next, plus the episode after, so autoplay never stalls — and sends TRANSCODE_REQUESTED with preTranscode=true. Files without analyzed streams are first sent back through MEDIA_FILE_FOUND.

Pre-transcode passes are narrowed by PassFilter from the settings of the users who pulled the file in: only audio streams in a preferred language (user_settings.preferred_audio_languages, falling back to app.ister.languages) and only video variants up to max_video_height. It never produces the 64k audio bitrate — HlsPlaylistBuilder folds that group into 192k, so no master playlist ever references it. Interactive playback uses PassFilter.none(): it must be able to serve any track a player asks for, and starts passes lazily anyway. A finished background pass pulls in the next pending pass of the same file; otherwise a budget-dropped pass would wait for the next pre-transcode cycle.

Background passes (background=true) run only on spare capacity (max-background-files, max-background-passes) and are preempted — FFmpeg stopped, the event discarded; the scheduler/prefetch will resend — as soon as interactive playback needs a slot or thread. Background FFmpeg additionally runs under OS niceness (app.ister.server.background-nice, default 10, 0 = off; the same BackgroundFfmpeg wrapper serves the analysis handlers in disk) via a wrapper script generated at startup, falling back to normal priority when nice is missing. A successful pass writes a done_<segmentPrefix> marker; only that marker (not the mere presence of segments) lets a later pre-transcode skip the pass.

Crop detection and transcoding

File analysis detects baked-in black bars and stores the crop rectangle on the video's MediaFileStreamEntity (chapter 2). The transcoder deliberately does not apply it: no FFmpeg pass adds a crop filter, and streams are transcoded with the bars intact. Cropping is the client's job — the player reads the rectangle from the GraphQL crop fields, scales it to the decoded dimensions of whatever it is playing (direct play or any HLS variant, which may be downscaled) and applies it as mpv video-crop.

That split avoids three server-side problems at once: the COPY variant (-c:v copy) can never be filtered, so a server-side crop would make ABR switches between COPY and a transcoded variant jump in framing; a server-side crop without a matching client contract would be applied twice; and the advertised RESOLUTION values in the master playlist would have to be recomputed per variant. The known limitation of the client-side design is the web player, which has no mpv and shows the bars — the only place a server-side crop would ever add value.

Bitmap subtitles

Blu-ray (PGS) and DVD (VobSub) subtitles are pictures, not text. They are never burned in and never OCR'd: the server hands the pictures themselves to the player, which draws them in an overlay above the video. Switching or changing a subtitle therefore never touches a transcoded segment, and what the viewer sees is exactly what the disc shipped.

HlsBitmapSubtitleService produces, per bitmap stream, a cue index plus a few sprite sheets in the file's transcode tmp dir:

• bsub_{streamId}.json — {version, width, height, sheets[], cues[]}; a cue is {s, e, x, y, w, h, sheet, sx, sy, forced}: start/end in milliseconds, position and size on the width×height subtitle canvas, and the sprite's corner inside sheet number sheet.
• bsub_{streamId}_{NN}.png — RGBA sheets, shelf-packed in display order (height capped at 2048 px), so a player only ever needs the sheet around the playhead.
• bsub_{streamId}.gen — generation marker (BITMAP_GENERATION); bump it when the parsers or the format change and stale artifacts are regenerated on the next request.

Generation is lazy and cheap. FFmpeg decodes nothing: one run copies the packets of all bitmap streams of the file out (-c:s copy, PGS as .sup, VobSub as .mks → mkvextract → .idx/.sub), and the pure-Java PgsParser/VobSubParser in transcoder/.../bitmapsub/ do the rest — a few seconds for a 40-minute Blu-ray episode, dominated by reading the container once. PngEncoder writes the sheets by hand (Deflater + CRC32): ImageIO's PNG writer has no JNI hints in the native image, and nothing here needs AWT. Generation is kicked off in the background (a virtual thread, not a transcode slot) when the playlists of a file are pre-generated, and the GET /hls/{mediaFileId}/bsub_… endpoints generate on a miss, under a per-file lock. With a shared tmp store the artifacts are published there and read through like segments. Because they live in the tmp dir they age out with the rest of the transcode cache — no database row, no cache-directory file.

Cue times are raw: they sit on the same zero-based timeline as the player's position, since the packet copy rebases to the container start just like the transcode passes do. The SUBTITLE_OFFSET_MS shift of the WebVTT/SRT renditions compensates an MPEG-TS muxer detail and does not apply here. dvb_subtitle has no parser and is not served.

Retention

Two separate sweeps clean the transcode cache, steered by different properties:

• HlsTranscodeService.cleanupOldFiles runs every 15 minutes. It removes a cache dir only when every file in it has been untouched for app.ister.transcoder.hls.cache-retention-hours (default 2) and the dir's keep_until deadline (the highest keepUntilEpochMillis ever received) has passed. Play-queue prefetch sends now + app.ister.server.prefetch.keep-hours (default 24 h); the periodic pre-transcode sends +30 min and refreshes it every 15 minutes as long as the entry qualifies. One exemption: a dir holding nothing but .m3u8 playlists is kept permanently — those are the scan-time playlists for music, a few KB that make first play instant. The exemption is deliberately that narrow: a dir with extracted subtitles or other leftovers but no .ts still ages out.
• TmpTranscodeCleanupScheduler runs daily (cron app.ister.server.cache-cleanup.cron) and handles what the first sweep cannot see: orphans — dirs whose media file no longer exists in the database — are removed unconditionally, and idle dirs with no active FFmpeg pass are removed after app.ister.server.cache-cleanup.min-age (default 24 h). It ignores keep_until, and obeys the shared app.ister.server.cache-cleanup.dry-run flag (default true — it only logs until switched off).

Multi-node

Transcode queues are directory-scoped (TranscoderQueueNamingConfig over DirectoryQueueNames, chapter 1), so a transcode runs on the node that owns the source file — or on a helper node that lists the directory under app.ister.helper.disks with the TRANSCODE job, which consumes the same queues. When the node that transcodes is not the one serving the playback, a watcher thread uploads each stable segment to the owner via POST /transcode/upload/{id}/{fileName} (FileController). When the source itself is remote, MediaFileInputResolver feeds FFmpeg the owner's tokenized /mediaFile/{id}/download URL instead of a local path; that endpoint serves byte ranges, so FFmpeg's seeks and the transcoder's stream-end probes stay cheap. An EXTERNAL_SUBTITLE stream's path is owner-local too (cache directory or a sidecar .srt), so a remote transcoder fetches it once through /mediaFileStream/{id}/download into the file's transcode cache dir.

An S3 directory has no owner: any attached node may consume the transcode queues, and the node the client talks to is not necessarily the one running the pass. TranscodeRequestedData and TranscodePassRequestedData therefore carry requestingNodeUrl; a transcoding node pushes to that node when it is another one (SegmentUploadCoordinator.uploadTarget, the class that owns the push targets and the segment watcher), which reuses the helper-to-owner segment push.

With app.ister.server.tmp-s3-connection set, the push target becomes the cluster-shared TmpStore (S3TmpStore, tmp/{mediaFileId}/…) instead: the watcher publishes stable segments, playlists and — last, after the pass — the done marker. The read side is a read-through: every HlsService lookup checks the local tmp dir first and then pulls the missing playlist, marker or segment from the store into it (SharedTmpReadThrough.syncFromShared), so the encoder-side invariants (size stability, done markers, keep_until, the local sweeps) are untouched and HlsController keeps serving real files. Waiting for a segment produced elsewhere polls the store instead of the local pass (SharedTmpReadThrough.waitForSegment); that class holds every request-side wait loop of HLS playback. The tmp cleanup also sweeps the store, one node per run under an advisory lock; re-analysis (deleteHlsCache) drops the published directory too.