Plex QSV degradation/artifacting issue

Plex Media Server
,
1

Plex injects bitrate-capping FFmpeg arguments (-b:0, -maxrate:0, -bufsize:0) into every transcode session, even when “Limit remote stream bitrate” is set to “No limit”. On VAAPI hardware encoders, this likely forces the encoder into CVBR mode, which may be the cause of visible quality degradation and blocking artifacts I was seeing, though I can’t say for certain this is the root cause. What I can say is that after intercepting these arguments and switching to CQP mode, the degradation stopped.

There is no Plex setting to disable bitrate capping for hardware-accelerated transcodes, so this report documents the issue, the workaround I’m using, and a request for a proper fix.

This is related to this older forum thread, which is now closed.

Environment

  • CPU: 12th Gen Intel(R) Core™ i9-12900K
  • iGPU: Intel® UHD Graphics 770
  • Container: linuxserver/plex (issue also present on the official Plex container; LSIO was used for final testing since it gave better results overall)
  • Hardware encoder: VAAPI (h264_vaapi)
  • Transcoder: Plex Media Server (Linux)

Possible Root Cause

Plex passes explicit bitrate constraints to FFmpeg regardless of the stream bitrate setting. My guess is that when VAAPI receives these constraints, it operates in Constrained Variable Bitrate (CVBR) mode, and VAAPI’s CVBR handling causes the encoder to drop quality more aggressively than expected, but I haven’t been able to confirm this definitively.

What I did confirm is that stripping those bitrate args and switching to Constant Quantization Parameter (CQP) mode stops the degradation. Whether the bitrate caps are the actual cause or just a contributing factor, I’m not sure.

Workaround

On linuxserver/plex, container init scripts placed in /custom-cont-init.d/ run on every container start. The workaround replaces the Plex Transcoder binary with a shell wrapper that:

  1. Strips -b:0, -maxrate:0, and -bufsize:0 from the FFmpeg argument list
  2. Injects -qp:0 20 -quality:0 4 after the h264_vaapi codec argument
  3. Forwards all remaining arguments to the original transcoder binary

Plex is unaware of the substitution and logs its original command unchanged. The same wrapper approach should work on the official Plex container, though the init mechanism would differ.

Make and write ./custom-cont-init.d/01-transcoder-wrapper.sh next to compose:

#!/bin/bash
TRANSCODER='/usr/lib/plexmediaserver/Plex Transcoder'
[ ! -f "${TRANSCODER}.orig" ] && mv "${TRANSCODER}" "${TRANSCODER}.orig"
cat > /tmp/plex_wrapper << 'WRAPPER'
#!/bin/bash
ARGS=("$@")
NEW_ARGS=()
i=0
while [[ $i -lt ${#ARGS[@]} ]]; do
  arg="${ARGS[$i]}"
  if [[ "$arg" == "-b:0" ]] || [[ "$arg" == "-maxrate:0" ]] || [[ "$arg" == "-bufsize:0" ]]; then
    ((i+=2))
    continue
  fi
  if [[ "$arg" == "h264_vaapi" ]]; then
    NEW_ARGS+=("h264_vaapi" "-qp:0" "20" "-quality:0" "4")
    ((i++))
    continue
  fi
  NEW_ARGS+=("$arg")
  ((i++))
done
exec "/usr/lib/plexmediaserver/Plex Transcoder.orig" "${NEW_ARGS[@]}"
WRAPPER
chmod +x /tmp/plex_wrapper
ln -sf /tmp/plex_wrapper "${TRANSCODER}"

And then mount it in compose:

    volumes:
      - ./custom-cont-init.d:/custom-cont-init.d

We can verify our change with:

# Confirm wrapper is active and CQP args are present, bitrate args are absent
docker exec plex bash -c "ps auxww | grep 'Transcoder.orig' | grep -v grep"

It would be useful to have a native option to disable bitrate capping for hardware-accelerated transcodes, or to expose CQP as an alternative encode strategy. Even if the bitrate caps aren’t the direct cause of the issue, having more control over how Plex handles hardware encoder parameters would make it easier to diagnose and work around problems like this without binary interception hacks.

2 Likes