Self-Hosted Video Transcoding: Jellyfin, Handbrake Server, and FFmpeg Pipelines

Running a media server is one of the most popular self-hosting projects — but nothing kills the experience faster than watching your server CPU spike to 100% whenever a family member tries to play a video on an incompatible device. That's transcoding: converting video on the fly from one format, resolution, or codec to another that the client device can handle.

Photo by Wesson Wang on Unsplash

Understanding how transcoding works, and how to set it up properly, is the difference between a media server that runs smoothly and one that stutters, crashes, or overheats your hardware.

What Is Video Transcoding?

When a device requests a video file, it might not support the codec (H.265/HEVC, AV1) or the resolution (4K on a device that can only handle 1080p). Rather than failing, a media server can transcode the video in real time — decode the original and re-encode it into a format the device understands, streaming the result as it goes.

This is computationally expensive. Transcoding a single 1080p H.265 stream can consume 100% of a modern CPU core. Multiple streams simultaneously, or 4K with HDR tone mapping, can overwhelm even powerful servers.

The three approaches to handling this:

1. Remuxing / Direct Play: The client supports the codec as-is; no transcoding needed. Always the goal.
2. Hardware transcoding: Use your GPU's dedicated video encode/decode engines (Intel QuickSync, NVIDIA NVENC, AMD VCE). Fast, low CPU usage.
3. Software transcoding: CPU-only FFmpeg. Works everywhere but is much slower and more power-hungry.

Jellyfin Hardware Transcoding

Jellyfin is the most popular open-source media server and handles transcoding natively. The key is enabling hardware acceleration in the admin dashboard.

Intel QuickSync (Most Common for Homelabs)

Intel's integrated GPU in 8th-gen and newer CPUs includes a dedicated video engine that supports H.264, H.265, AV1 (12th gen+), and VP9. It's the most power-efficient option for homelab use.

In Docker:

services:
  jellyfin:
    image: jellyfin/jellyfin
    devices:
      - /dev/dri:/dev/dri  # expose GPU devices
    group_add:
      - "44"  # video group GID (check on your system)

In Jellyfin admin → Playback → Transcoding:

• Set hardware acceleration to Intel QuickSync Video
• Enable Allow encoding in HEVC format
• Enable Enable VPP Tone Mapping for HDR → SDR conversion

Verify it's working: start a stream that should transcode, then check Dashboard → Active Streams. Look for "(hw)" next to the codec name.

NVIDIA NVENC

NVIDIA GPUs include NVENC (encode) and NVDEC (decode) engines. You'll need to install the NVIDIA Container Toolkit and pass the GPU to the container:

services:
  jellyfin:
    image: jellyfin/jellyfin
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all

NVENC is fast and supports 4K, but NVIDIA consumer cards are limited to 3 simultaneous NVENC sessions on most driver versions. Check your card's specifications.

AMD GPU (VAAPI)

AMD GPUs use VAAPI on Linux. Similar to Intel, expose /dev/dri and select VAAPI in Jellyfin's settings.

Automated Transcoding with Handbrake + Watch Folders

If you want to pre-transcode your library into device-compatible formats (rather than transcode on the fly), the Handbrake Docker container with watch folders is a clean solution.

services:
  handbrake:
    image: jlesage/handbrake
    ports:
      - "5800:5800"  # Web UI
    volumes:
      - /mnt/media/incoming:/watch  # drop files here to auto-convert
      - /mnt/media/converted:/output
    environment:
      - AUTOMATED_CONVERSION_PRESET=H.265 MKV 1080p30
      - AUTOMATED_CONVERSION_OUTPUT_DIR=/output
      - AUTOMATED_CONVERSION_FORMAT=mkv

Drop a video file into /watch and Handbrake automatically picks it up, converts it using your preset, and saves the result to /output. Combine with a download manager (qBittorrent + Sonarr/Radarr) using a "convert then move" pipeline for a fully automated library management workflow.

Custom FFmpeg Pipelines

For advanced control, FFmpeg directly gives you the most flexibility. Common use cases:

Batch convert a directory to H.265 with Intel QuickSync:

for f in /media/incoming/*.mkv; do
  ffmpeg -hwaccel qsv -hwaccel_device /dev/dri/renderD128 \
    -i "$f" \
    -c:v hevc_qsv -preset medium -global_quality 23 \
    -c:a copy \
    "/media/converted/$(basename "$f")"
done

Extract and preserve subtitles:

ffmpeg -i input.mkv -c:v hevc_qsv -c:a aac -c:s copy output.mkv

Create optimized web versions (HLS streaming):

ffmpeg -i input.mp4 \
  -c:v h264_qsv -c:a aac \
  -hls_time 6 -hls_playlist_type vod \
  -hls_segment_filename 'segment_%03d.ts' \
  output.m3u8

Choosing the Right Codec for Your Library

For a homelab media library in 2026, H.265 MKV is the practical sweet spot — excellent compression, supported by all modern Jellyfin clients, and hardware-encodable on virtually any Intel NUC or home server built in the last 6 years.

Monitoring Transcoding Load

Once transcoding is running, monitor it:

Check GPU utilization (Intel):

intel_gpu_top  # install intel-gpu-tools

Check active Jellyfin streams: Navigate to Dashboard → Active Streams. Each stream shows whether it's direct playing, remuxing, or transcoding (with hw or sw indication).

Grafana dashboard: The Jellyfin plugin for Grafana can pull metrics from the Jellyfin API to show streams, transcoding load, and library statistics over time.

Getting transcoding right transforms your media server from a frustrating source of buffering and compatibility errors into something that just works for everyone in the house, regardless of what device they're using. Start with hardware acceleration in Jellyfin — it's the highest-leverage change you can make with the least configuration effort.