[Bug] AMD VAAPI: green cast / no red when encoding software-decoded sources (XviD) — missing nv12

Plex Media Server
, ,
1

Server Version#: 1.43.4.10903-e5521bd8c (official plexinc/pms-docker:plexpass image)
Player Version#: n/a — server-side output bug, reproduced identically on Roku, Plex Web and Plex for Windows

Environment

  • TrueNAS SCALE 25.10.4, kernel 6.12.91, /dev/dri + /dev/kfd passed to the container
  • AMD Ryzen 7 5700G (Cezanne / Vega iGPU), /dev/dri/renderD128
  • VAAPI driver: Plex’s own bundle mesa-va-radeonsi/rsv-a71f0abe5d8bf5cda93eec5b-linux-x86_64 (Mesa Gallium 23.0.2), downloaded automatically by PMS when HW transcoding is enabled
  • Plex Pass; “Use hardware acceleration when available” = on; “Use hardware-accelerated video encoding” = on

Symptom

Any source the GPU cannot decode (XviD / MPEG-4 Part 2 ASP in AVI in my case — presumably also WMV, MPEG-1, VP8, …) is transcoded with a strong green cast and no red channel at all. H.264 / HEVC sources transcode with correct colours.

Log

[Transcode] Codecs: testing mpeg4 (decoder) with hwdevice vaapi
[Transcode/…/TPU] hardware transcoding: final decoder: vaapi, final encoder: vaapi
[mpeg4 @ 0x…] No support for codec mpeg4 profile 15.
[mpeg4 @ 0x…] Failed setup for format vaapi: hwaccel initialisation returned error.

Generated transcoder command line (trimmed):

-codec:0 mpeg4 -hwaccel:0 vaapi -hwaccel_output_format:0 vaapi -hwaccel_device:0 vaapi
-filter_complex "[0:0]hwupload[0];[0]scale_vaapi=w=640:h=480:format=nv12[1];[1]hwupload[2]"
-codec:0 h264_vaapi

Root cause — reproduced outside PMS with the bundled Plex Transcoder and Plex’s own driver

  1. VAAPI decode of MPEG-4 ASP fails to initialise and ffmpeg falls back to the software decoder (yuv420p output). That part is fine and expected.
  2. The generated filter graph then runs hwupload directly on the yuv420p software frames. With the bundled radeonsi driver that upload produces garbled chroma (green cast, red missing).
  3. Adding a pixel-format conversion before the upload fixes it completely.

Three frames from the same file at the same timestamp, using Plex Transcoder with LIBVA_DRIVERS_PATH pointing at PMS’s Cache/va-dri-linux-x86_64:

Pipeline Result
A: software only (-frames:v 1 out.png) correct colours
B: exactly PMS’s graph: [0:0]hwupload[0];[0]scale_vaapi=w=640:h=480:format=nv12[1] → h264_vaapi green cast, no red
C: same as B but [0:0]format=nv12,hwupload[0];… correct colours, identical to A

Commands for B and C:

"Plex Transcoder" -init_hw_device vaapi=va:/dev/dri/renderD128 -filter_hw_device va -ss 120 -i in.avi -frames:v 24 \
  -filter_complex "[0:0]hwupload[0];[0]scale_vaapi=w=640:h=480:format=nv12[1]" -map "[1]" -c:v h264_vaapi B.mp4

"Plex Transcoder" -init_hw_device vaapi=va:/dev/dri/renderD128 -filter_hw_device va -ss 120 -i in.avi -frames:v 24 \
  -filter_complex "[0:0]format=nv12,hwupload[0];[0]scale_vaapi=w=640:h=480:format=nv12[1]" -map "[1]" -c:v h264_vaapi C.mp4

Suggested fix

When the decoder falls back to software but the encoder stays on VAAPI, insert format=nv12 (or nv12|p010 as appropriate) before hwupload in the generated filter graph — the standard ffmpeg recipe for VAAPI-encoding software frames. Alternatively, treat the hwaccel init failure as a reason to also fall back to software encoding for that session, which is what the docs promise (“automatic fallback from hardware to software”).

Workaround for other users

Turn off “Use hardware-accelerated video encoding” and leave “Use hardware acceleration when available” on. GPU decode of H.264/HEVC still works, software frames are never uploaded, colours are correct.

Related

Same symptom reported in July on an AMD 780M (also TrueNAS): Possible transcoding bug? — it was answered with “AMD is unsupported” and no root cause. Note that PMS itself downloads and installs the mesa-va-radeonsi driver bundle when HW transcoding is enabled on AMD, so this is a code path Plex ships.