HEVC transcoding stopped working (H264 works fine)

Plex Media Server
1

Server Version#: 1.43.0.10389
Player Version#: Tested with app.plex.tv

My setup: Plex running on proxmox privileged LXC. nvidia P5000 passed through.

HEVC transcoding stopped working on my server with a P5000 GPU. Uncertain when this started to happen, but my best guess is it was along side upgrading to Proxmox 9.1, which was a frustrating experience due to the compatibility of nvidia drivers and 6.17 kernel. I’m pinned to 6.14.

I used Claude to run some extensive tests – may be helpful or potentially way off course. I’ve posted my findings along with the actual server logs.

Any help would be appreciated.

claude_summary.txt (10.1 KB)

Plex Media Server Logs_2026-01-02_10-48-25.zip (3.7 MB)

2

I’ve seen other similar user reports. Tried pinning container back to stable release rather than beta:

image: plexinc/pms-docker:1.42.2.10156-f737b826c

edit seems like your ubuntu ver and kernel don’t match, you’ve gone forward past 6.8 for 22.04… could make for a headache https://ubuntu.com/kernel/lifecycle might be unrelated.

3 
  NVIDIA Driver Installation (Host):
  $ dpkg -l | grep nvidia
  ii  libnvcuvid1:amd64     550.163.01-2    amd64    NVIDIA CUDA Video Decoder runtime library
  ii  libcuda1:amd64        550.163.01-2    amd64    NVIDIA CUDA Driver Library
  ii  nvidia-driver:amd64   550.163.01-2    amd64    NVIDIA metapackage
  ...

What about libnvidia-encode and libnvidia-decode ??
nvidia-smi will show the GPU but no transcoding without the decode/encode.

4

@ChuckPa I believe that’s because Debian (my host machine for Proxmox) uses different packages than Ubuntu:

Approach Debian (Host) Library Name Ubuntu Package Name
Decode libnvcuvid.so.1 libnvidia-decode-XXX
Encode libnvidia-encode.so.1 libnvidia-encode-XXX

I’m bind mounting my drivers to the ubuntu lxc. Here’s a test to show them loading on the plex lxc:

Test Command:
sudo -u plex python3 -c "
import ctypes
import os
print(f’Running as: {os.getuid()} ({os.popen(“whoami”).read().strip()})')

try:
ctypes.CDLL(‘libnvcuvid.so.1’)
print(‘✓ libnvcuvid.so.1 (DECODE): LOADED’)
except Exception as e:
print(f’✗ libnvcuvid.so.1 (DECODE): FAILED - {e}')

try:
ctypes.CDLL(‘libnvidia-encode.so.1’)
print(‘✓ libnvidia-encode.so.1 (ENCODE): LOADED’)
except Exception as e:
print(f’✗ libnvidia-encode.so.1 (ENCODE): FAILED - {e}')
"

Output:
Running as: 999 (plex)
✓ libnvcuvid.so.1 (DECODE): LOADED
✓ libnvidia-encode.so.1 (ENCODE): LOADED

ldconfig shows both libraries in the linker cache:
$ ldconfig -p | grep -E ‘nvcuvid|nvidia-encode’
libnvidia-encode.so.1 => /usr/lib/x86_64-linux-gnu/nvidia/current/libnvidia-encode.so.1
libnvcuvid.so.1 => /usr/lib/x86_64-linux-gnu/nvidia/current/libnvcuvid.so.1

5

Thanks for noting the kernel support being topped out at 6.8 for 22.04, I wasn’t aware.

If the consensus is that this could be the issue, I’m happy to give upgrading a try.

6

It was just extra pondering.

I’m more hung up on your suspicion that your proxmox upgrade broke it is likely a redherring… vs the PMS package update from stable to beta.
I suspect your pms package broke it. Youre on a beta release… come on… first thing I’d always do with an issue under beta is revert to public stable.

7

@Menel – Ok! I’ve been treating the beta for plex as stable for nearly 10 years :slight_smile:

Results were the same after downgrading. I also tested using stable and disabling HEVC encode.

:record_button: Test Results: Stable (1.42.2.10156)

NVIDIA Library Test

$ sudo -u plex python3 -c “import ctypes; ctypes.CDLL(‘libnvcuvid.so.1’); print(‘DECODE: OK’); ctypes.CDLL(‘libnvidia-encode.so.1’); print(‘ENCODE: OK’)”
DECODE: OK
ENCODE: OK

HEVC Playback Test (Ocean’s Eleven 4K HEVC)

TPU: hardware transcoding: enabled, but no hardware decode accelerator found
TPU: hardware transcoding: final decoder: , final encoder:
Result: Software transcoding (same as PlexPass)

With “Use hardware-accelerated video encoding” DISABLED

Changed setting in Plex Settings > Transcoder > unchecked “Use hardware-accelerated video encoding”

Result:
Video=(id=273541 decision=copy width=3840 height=2160)
FFmpeg command showed: -codec:0 copy

Video was direct streamed (copied) instead of transcoded - bypassing the decode issue entirely.

FFmpeg Build Config (Both versions identical)

–external-decoder=h264
(No --external-decoder=hevc present)

8

@ChuckPa / @Menel - did a fresh install/migration to 24.04.3. So far getting the same results.

Welcome to Ubuntu 24.04.3 LTS (GNU/Linux 6.14.11-4-pve x86_64)

9

Sorry, it was just a best first guess, Chuck is the expert

10

Did you use TTECK helper scripts to create the VM?

IIRC, there is something about needing to blacklist the card so Proxmox’s drivers don’t get control of it. Does that apply here again in this newer release?

On regular Linux (Ubuntu/Debian/Redhat), you install all the drivers on the native host. Physical passthrough of the GPU is done via /dev/dri which then gets handled in the host’s kernel space and not the LXC

EDIT: When you install the PMS DEB package, I check for the presence of the device / inodes and make the appropriate configuration changes to PMS.
If the report (at end of install) can’t find the GPU then it’s not being passed in correctly.

11

After some extensive testing, I discovered the issue.

Root Cause:
NVIDIA libraries were bind-mounted to /usr/lib/x86_64-linux-gnu/nvidia/current/ but Plex’s FFmpeg looks for them directly in /usr/lib/x86_64-linux-gnu/.

Solution:
Create symlinks from the standard library path to the nvidia subdirectory:

cd /usr/lib/x86_64-linux-gnu
for lib in nvidia/current/*.so.1; do
sudo ln -sf “$lib” $(basename “$lib”)
done

12

@mdlamoureux

Interesting

On Ubuntu 22.04, and Ubuntu Server 24.04, installing the Nvidia drivers puts them in x86_64_linux_gnu

[chuck@lizum x86_64-linux-gnu.2001]$ ll | grep nvid
lrwxrwxrwx   1 root root        26 Oct 23 13:55 libEGL_nvidia.so.0 -> libEGL_nvidia.so.580.95.05
-rw-r--r--   1 root root   1296640 Oct 23 13:55 libEGL_nvidia.so.580.95.05
lrwxrwxrwx   1 root root        32 Oct 23 13:55 libGLESv1_CM_nvidia.so.1 -> libGLESv1_CM_nvidia.so.580.95.05
-rw-r--r--   1 root root     80336 Oct 23 13:55 libGLESv1_CM_nvidia.so.580.95.05
lrwxrwxrwx   1 root root        29 Oct 23 13:55 libGLESv2_nvidia.so.2 -> libGLESv2_nvidia.so.580.95.05
-rw-r--r--   1 root root    129496 Oct 23 13:55 libGLESv2_nvidia.so.580.95.05
lrwxrwxrwx   1 root root        26 Oct 23 13:55 libGLX_nvidia.so.0 -> libGLX_nvidia.so.580.95.05
-rw-r--r--   1 root root   1211968 Oct 23 13:55 libGLX_nvidia.so.580.95.05
lrwxrwxrwx   1 root root        24 Oct 23 13:55 libnvidia-allocator.so -> libnvidia-allocator.so.1
lrwxrwxrwx   1 root root        32 Oct 23 13:55 libnvidia-allocator.so.1 -> libnvidia-allocator.so.580.95.05
-rw-r--r--   1 root root    164936 Oct 23 13:55 libnvidia-allocator.so.580.95.05
-rw-r--r--   1 root root    716008 Oct 23 13:55 libnvidia-api.so.1
lrwxrwxrwx   1 root root        18 Oct 23 13:55 libnvidia-cfg.so -> libnvidia-cfg.so.1
lrwxrwxrwx   1 root root        26 Oct 23 13:55 libnvidia-cfg.so.1 -> libnvidia-cfg.so.580.95.05
-rw-r--r--   1 root root    395352 Oct 23 13:55 libnvidia-cfg.so.580.95.05
-rw-r--r--   1 root root  35068240 Oct 23 13:55 libnvidia-eglcore.so.580.95.05
lrwxrwxrwx   1 root root        26 May 13  2025 libnvidia-egl-gbm.so.1 -> libnvidia-egl-gbm.so.1.1.2
-rw-r--r--   1 root root     22840 Oct 23 13:55 libnvidia-egl-gbm.so.1.1.2
lrwxrwxrwx   1 root root        31 Feb  9  2025 libnvidia-egl-wayland.so.1 -> libnvidia-egl-wayland.so.1.1.17
-rw-r--r--   1 root root     77560 Feb  9  2025 libnvidia-egl-wayland.so.1.1.17
lrwxrwxrwx   1 root root        26 Aug  7 19:10 libnvidia-egl-xcb.so.1 -> libnvidia-egl-xcb.so.1.0.3
-rw-r--r--   1 root root     72592 Oct 23 13:55 libnvidia-egl-xcb.so.1.0.3
lrwxrwxrwx   1 root root        27 Aug  7 19:10 libnvidia-egl-xlib.so.1 -> libnvidia-egl-xlib.so.1.0.3
-rw-r--r--   1 root root     72640 Oct 23 13:55 libnvidia-egl-xlib.so.1.0.3
lrwxrwxrwx   1 root root        21 Oct 23 13:55 libnvidia-encode.so -> libnvidia-encode.so.1
lrwxrwxrwx   1 root root        29 Oct 23 13:55 libnvidia-encode.so.1 -> libnvidia-encode.so.580.95.05
-rw-r--r--   1 root root    285464 Oct 23 13:55 libnvidia-encode.so.580.95.05

Thanks for the info