Hardware Transcoding stopped Working (GENTOO LINUX) - SOLVED

[SleepwalkerX]

Server Version#: Version 1.23.2.4600
Player Version#: Version 4.57.4
Platform: Linux (Kernel 5.10.27-gentoo / x86_64 Intel(R) Xeon(R) E-2124 CPU @ 3.30GHz) - No VM
GPU: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB]

Hey Plex Forum,

I’m not sure if this is a bug or if I fail to understand what’s happening here. I’ve done my part and searched the forum but couldn’t identify a related issue.

Here’s what’s going on:

I’m on Plex Pass and use HW Transcoding to support multiple simultaneous streams (wife, kids’ ipads, etc.). This all worked like a charm for years… until I upgraded to the current version that is released for Plex Pass customers. I believe it worked fine until the 1.22 release.
Switching back to the non-Plex Pass release 1.22 it works fine and as expected; I hope you understand that I want to live on the bleeding edge, so downgrading is not a permanent solution for me ;-).

Judging by the logs, it appears to be an issue related to Plex’ internal transcoding engine not being able to load the CUDA extensions:

[Transcode] Codecs: hardware transcoding: testing API vaapi
[Transcode] [FFMPEG] - libva: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null)
[Transcode] [FFMPEG] - Failed to initialise VAAPI connection: -1 (unknown libva error).
[Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: I/O error
[Transcode] Codecs: testing h264_nvenc (encoder)
[Transcode] Codecs: hardware transcoding: testing API nvenc
[Transcode] [FFMPEG] - Cannot load libcuda.so.1
[Transcode] [FFMPEG] - Could not dynamically load CUDA
[Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: Unknown error occurred
[Transcode] Codecs: testing h264 (decoder) with hwdevice vaapi
[Transcode] Codecs: hardware transcoding: testing API vaapi
[Transcode] [FFMPEG] - libva: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null)
[Transcode] [FFMPEG] - Failed to initialise VAAPI connection: -1 (unknown libva error).

Any help is sincerely appreciated as my CPU bogs down with more than one 4K stream.

Thank you so very much in advance for your help!

[ChuckPa]

I am researching an issue which might have the same root as this.

I’ve added your info to the existing report to Enginering.

[SleepwalkerX]

Thanks, I appreciate it!

[SleepwalkerX]

Hey there,
is there an update on this topic? In the meantime, I have upgrade to PMS 1.23.2.4656 and this issue still prevails.

Thanks!

[ChuckPa]

What I had previously found turned out not to be related as I first thought

What I did find was the biggest issue is the Nvidia drivers themselves.

Folks have had to back down (roll back) to drivers a few versions older (450-ish) and things magically work.

[SleepwalkerX]

Thanks for the swift reply, @ChuckPa.

In the meantime, I have tried downgrading to the following versions of Nvidia-Drivers:

450.80.02
455.45.01
460.73.01

It’s still the same issue as mentioned above.

Frankly, I believe it is Plex-related since the stable version (outside of Plex Pass) works like a charm.

[SleepwalkerX]

Small correction: It used to work on PMS 1.22. Now that the public version is bumped to 1.23, it is also affected. Some change from 1.22 to 1.23 has caused this issue.

[ChuckPa]

Here’s the kicker —

If it were the changes made for 1.23.x, everyone would be screaming too.

1.23.x represents a compiler change. Everything would break. The other 11th Gen CPUs in use wouldn’t be working. There are user who have success with it.

Here is the thread where a user has it working and has documented exactly how to do it.

Working Rocket Lake GPU with HDR the easy way both docker and bare metal!

Server Version#:1.23.2.4656 I’ve been trying to get GPU support with HDR working for a couple days now and after trying multiple ways wanted to post the easy way if your running ubuntu since i didn’t find anything clearly posted in the forums. Install 21.04, that’s it. I’m a LST person in general but 21.04 running 5.11.0-18-generic (out of the box kernel) with the intel compute runtime packages installed has GPU offload with HDR working. [Capture] The full steps of what i did: Installed 21…

That 11th Gen CPU is “bleeding edge” stuff.

[SleepwalkerX]

That is indeed strange. I’ll try downgrading to 1.22 to reproduce my issue. What do you make of this error message “Cannot load libcuda.so.1”?

[ChuckPa]

The libcuda error tells me it’s looking for the Nvidia drivers.

As the trancoder starts, it looks at the available hardware.

1. Check if QSV is present (/sys/modules and /dev/dri)
2. Check if Nvidia is present (/sys/modules)
3. If finding QSV, load the Plex-provided QSV libraries
4. If finding Nvidia, load the cuda libraries
5. Unless otherwise directed by the HardwareDevicePath preference,
6. – Open /dev/dri/renderD128 & card0 to see what the capabilities are
7. Determine actions based on that communication.

The most common cause of the error you’re seeing is:

1. A Nvidia device is being seen.
2. There are no Nvidia drivers loaded to communicate with it

Do you have an Nvidia card or Nvidia display chip on the motherboard?

[SleepwalkerX]

These paths do exist on my system:

$ ~ # ls /dev/dri/
by-path card0 renderD128
$ ~ # ls /dev/dri/by-path/
pci-0000:01:00.0-card pci-0000:01:00.0-render

Where would I set the HardwareDevicePath preference?

I use an external GTX 1060 6GB as PCIe card.

[SleepwalkerX]

I am also able to trigger CUDA from ffmpeg directly, for instance by running:

ffmpeg -hwaccel cuda -i 01.mp4 test.mkv

I see the transcoding process with nvidia-smi:

Thu Jun 10 18:36:39 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 465.27       Driver Version: 465.27       CUDA Version: 11.3     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
|  0%   37C    P2    28W / 156W |    199MiB /  6077MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      5123      C   ffmpeg                            197MiB |
+-----------------------------------------------------------------------------+

(Apologies for pasting the whole output here)

I wonder why Plex is not able to use the same extensions.

[SleepwalkerX]

I added

HardwareDevicePath=“/dev/dri/renderD128”

to my Preferences.xml but still no luck.

[ChuckPa]

@SleepwalkerX

When you look at /dev/dri,

Do you see renderD128 and renderD129 ?

If you see both AND have an Intel CPU AND an Nvidia card,

1. Intel ASIC (QSV) is renderD128
2. Nvidia card is renderD129

It works this way because the kernel initializes the CPU first.
It finds all that hardware and adds it

It them searches the PCI bus where it finds the Nvidia
and adds it next

[SleepwalkerX]

No, I only see card0 and renderD128… Does this mean that my Nvidia card is not recognized?

[ChuckPa]

Open the terminal window
cd /sys/module

look for the nvidia directory and if there then start looking at where the links go.

They should all resolve to the PCI slot where the card is.

if that’s intact then the disconnect is with the Nvidia drivers installed.

[SleepwalkerX]

I hope you trust me when I say that I am not kidding now:

I have pulled an old .deb file of plex-media-server-1.22.3.4523, installed it without changing ANYTHING else. Transcoding works perfectly now on this version, despite my Preferences.xml pointing at renderD128… I’m really puzzled now.

Have a look:

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      9675      C   ...diaserver/Plex Transcoder      206MiB |
+-----------------------------------------------------------------------------+

[ChuckPa]

Please grab all the logs ZIP file before they disappear and attach.

Which specific GPU card and drivers version ?

I am totally befuddled.

[SleepwalkerX]

I’m completely befuddled myself. Please find my logs enclosed.

I disabled debug logging and do not see any CUDA-specific entries. However, with debug view on, I can see that libcuda is loaded successfully:

[Transcode] [FFMPEG] - Loaded lib: libcuda.so.1

[ChuckPa]

Please leave DEBUG logging ON!

We use / need that to track what’s happening.

Isolated Error messages don’t mean anything without knowing what it’s trying to do.