[Solved] HW Transcoding issues with nvidia quadro, on docker and debian 11 bullseye

[dovys_plex]

Server Version#: v1.24.1.4931-1a38e63c6
Player Version#: 7.21.1 iOS

Hi folks, recently got the Plex Pass and an NVIDIA Quadro card to enable hardware transcoding so I can watch stuff on the go, but it’s not working as expected. Any pointers on how can I further debug this would be greatly appreciated!

❯ lspci | grep NVIDIA
01:00.0 VGA compatible controller: NVIDIA Corporation GP107GL [Quadro P620] (rev a1)

I’m running plex in docker with the following settings:

services:
  plex:
    image: ghcr.io/linuxserver/plex:amd64-latest
    network_mode: host
    deploy:
      resources:
        reservations:
          devices:
          - capabilities: [gpu]
    runtime: nvidia # probably superfluous
    environment:
      - PUID=1002
      - PGID=998
      - VERSION=docker
      - UMASK=022
      - NVIDIA_VISIBLE_DEVICES=all
    volumes:
      - "/etc/cluster/plex:/config"
      - "/var/lib/plex/transcode:/transcode"
      - "/store/media/series:/series:ro"
      - "/store/media/movies:/movies:ro"

Got the following NVIDIA drivers on the host

vidia-docker2/buster,now 2.6.0-1 all [installed]
nvidia-driver-bin/unstable,now 470.57.02-2 amd64 [installed,automatic]
nvidia-driver-libs/unstable,now 470.57.02-2 amd64 [installed,automatic]
nvidia-driver/unstable,now 470.57.02-2 amd64 [installed]

I’m also running debian11. I had to edit /etc/nvidia-container-runtime/config.toml ldconfig from

ldconfig = "@/sbin/ldconfig" 

to

ldconfig = "/sbin/ldconfig" 

to make it work with debian11 and followed this guide to install nvidia container toolkit - https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker
Note: had to fake distribution to debian10 for the installer to work. Seems fine.

I can run nvidia-smi in the plex container:

❯ docker exec -it cluster_plex_1 nvidia-smi
Sun Aug 29 18:37:37 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: N/A      |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro P620         On   | 00000000:01:00.0 Off |                  N/A |
| 34%   46C    P8    N/A /  N/A |      1MiB /  1998MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

However when I try to transcode I get this

Aug 29, 2021 18:39:03.826 [0x7ffac5f09b38] DEBUG - [Transcode/FD4267D9-D53A-46A6-98E9-97AAB014C339] TPU: hardware transcoding: enabled, but no hardware decode accelerator found
Aug 29, 2021 18:39:03.826 [0x7ffac5f09b38] DEBUG - [Transcode/FD4267D9-D53A-46A6-98E9-97AAB014C339] TPU: hardware transcoding: final decoder: , final encoder:

And my CPU shoots up immediately

Restarted the whole system, deleted and recreated the containers, etc. I’ve tried h264 and hevc video formats.

Any pointers what else can I debug?

Plex Media Server.log (1.5 MB)

[dovys_plex]

Oh so I just got it working, not really sure exactly how. I’ve uninstalled nvidia-driver, then nvidia-installer-cleanup, removed unused apt packages, upgraded libc6 and installed nvidia-driver and nvidia-smi again. Fully expected this to not make a difference, but as I’ve tested if I haven’t made it worse it actually started working. Survived a few restarts too.

My gut feel says it was something to do with a mini GPU I had before Quadro and replacing GPUs requires reinstalling drivers? Or just too many nvidia utilities installed from earlier. Uninstalling nvidia-installer-cleanup actually uninstalled a ton of packages.

.

[KJKingJ]

I was running in to something similar, nvidia-smi would work fine inside the container but Plex claimed that no hardware decode(r) was found. Further up the logs however I saw another hint;

Nov 06, 2021 18:06:43.758 [0x7f5124553b38] ERROR - [Transcode] [FFMPEG] - Cannot load libnvidia-encode.so.1
Nov 06, 2021 18:06:43.758 [0x7f5124553b38] ERROR - [Transcode] [FFMPEG] - The minimum required Nvidia driver for nvenc is 418.30 or newer
Nov 06, 2021 18:06:43.811 [0x7f5124553b38] WARN - [Transcode] Codecs: avcodec_open2 returned -1 for encoder 'h264_nvenc'

Although I had the nvidia runtime installed all OK, and the GPU was exposed to the container, it still needs the appropriate libraries to to HW encoding/decoding. These get mounted in by the runtime if they exist on the host.

After installing libnvidia-decode and libnvidia-encode on the host, at the same version as the driver, Plex was able to utilise the GPU for HW decode and encode successfully. Perhaps your cleanup and re-install included installing the encoding/decode libraries and got it working again?