Plex HW transcode randomly stops - Codecs: hardware transcoding: opening hw device failed - probably

Plex Media Server
1

Server Version#: 1.32.3.7089
Player Version#: NA

I have noticed periodically that Plex will stop communicating with my GPU and go back to transcoding via CPU. Everything else works fine with the system GPU related and once I do a simple restart of the docker container (not the entire docker daemon) it works fine again for a while. I noticed these logs:

May 26, 2023 23:17:39.611 [139812131887928] DEBUG - [Req#1ad/Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: I/O error
May 26, 2023 23:17:39.355 [139812131887928] DEBUG - [Req#1ad/Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: I/O error
May 26, 2023 23:17:39.045 [139812131887928] DEBUG - [Req#1ad/Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: I/O error
May 26, 2023 23:17:38.825 [139812131887928] DEBUG - [Req#1ad/Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: I/O error
May 26, 2023 23:17:38.582 [139812131887928] DEBUG - [Req#1ad/Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: I/O error
May 26, 2023 23:17:38.387 [139812131887928] DEBUG - [Req#1ad/Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: I/O error
May 26, 2023 23:17:38.235 [139812131887928] DEBUG - [Req#1ad/Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: I/O error
May 26, 2023 23:10:32.393 [140187008752440] DEBUG - [Req#64955/Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: I/O error
May 26, 2023 23:10:32.132 [140187008752440] DEBUG - [Req#64955/Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: I/O error
May 26, 2023 23:10:31.883 [140187008752440] DEBUG - [Req#64955/Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: I/O error
May 26, 2023 23:10:31.632 [140187008752440] DEBUG - [Req#64955/Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: I/O error
May 26, 2023 23:10:31.425 [140187008752440] DEBUG - [Req#64955/Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: I/O error
May 26, 2023 23:10:31.177 [140187008752440] DEBUG - [Req#64955/Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: I/O error
May 26, 2023 23:08:41.689 [140187008752440] DEBUG - [Req#64168/Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: I/O error
May 26, 2023 23:08:41.426 [140187008752440] DEBUG - [Req#64168/Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: I/O error
May 26, 2023 23:08:41.226 [140187008752440] DEBUG - [Req#64168/Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: I/O error
May 26, 2023 23:08:41.037 [140187008752440] DEBUG - [Req#64168/Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: I/O error
May 26, 2023 23:08:40.829 [140187008752440] DEBUG - [Req#64168/Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: I/O error
May 26, 2023 23:08:40.602 [140187008752440] DEBUG - [Req#64168/Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: I/O error
May 26, 2023 22:30:26.979 [140187261872952] DEBUG - [Req#5de03/Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: I/O error

Plex Media Server Logs_2023-05-26_23-21-32.zip (4.2 MB)

2

@ChuckPa any ideas why this would keep falling out?

3
  1. You have the wildest network config. Is this a commercial rig or do you have a bazillion VMs ? :rofl:
May 26, 2023 23:17:34.018 [139812121340728] DEBUG - [HCl#8] HTTP requesting GET https://plex.tv/api/v2/user?includeSubscriptions=1&includeProviders=1
May 26, 2023 23:17:34.020 [139812229565072] DEBUG - Detected primary interface: 192.168.1.1
May 26, 2023 23:17:34.020 [139812229565072] DEBUG - Network interfaces:
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 1 lo (127.0.0.1) (00-00-00-00-00-00) (loopback: 1)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 3 enp38s0 (192.168.1.1) (2C-F0-5D-2A-36-2D) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 7 br-d645b544cbd1 (172.25.0.1) (02-42-DA-B9-DE-8A) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 8 br-e30664bbd418 (192.168.96.1) (02-42-71-4D-83-F8) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 9 br-5a6e2ef7c8d0 (192.168.128.1) (02-42-19-14-B9-AF) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 10 br-7c64d49923c8 (192.168.208.1) (02-42-1D-73-75-06) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 12 br-e5645eeeb268 (192.168.16.1) (02-42-E8-61-AF-CF) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 13 br-123f6063e941 (192.168.144.1) (02-42-11-E3-B9-BA) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 14 br-8b5f9a574d4c (172.23.0.1) (02-42-F8-C1-4E-92) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 15 br-mailcow (172.16.1.1) (02-42-78-F9-D2-D7) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 16 br-2cb135a7717c (192.168.48.1) (02-42-CF-6D-CC-96) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 17 br-9fef102d8fd9 (172.28.0.1) (02-42-44-A7-74-AD) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 18 br-01317c006d04 (172.31.0.1) (02-42-B0-88-31-74) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 19 br-2094d38029c0 (192.168.160.1) (02-42-71-54-16-A4) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 20 br-84fbbef8bbb1 (172.27.0.1) (02-42-1A-EE-42-2F) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 21 br-8d0ae67a4a86 (172.21.0.1) (02-42-10-E0-DD-83) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 22 br-cd45655d3773 (192.168.32.1) (02-42-87-58-E0-D6) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 23 br-227b825277e5 (172.22.0.1) (02-42-29-8F-E7-34) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 24 br-8393a5c5413e (172.29.0.1) (02-42-D6-F1-CA-D9) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 25 br-eb5f1fad554e (172.26.0.1) (02-42-03-59-A7-77) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 26 br-ce6a6e3dbf83 (192.168.112.1) (02-42-C9-2A-57-F7) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 27 br-5654b8205ba6 (192.168.64.1) (02-42-CD-59-57-C4) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 28 br-c9758ad02a23 (172.24.0.1) (02-42-13-D9-58-52) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 29 br-6c4984d78b61 (172.19.0.1) (02-42-A0-A4-F3-E8) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 30 br-a97a379e1a03 (172.20.0.1) (02-42-D9-7A-D2-F5) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 31 br-e8432c913cc3 (192.168.176.1) (02-42-E6-33-D1-50) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 32 br-0b6c060fc225 (192.168.192.1) (02-42-C5-32-E3-70) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 33 br-410ba794d573 (172.18.0.1) (02-42-D8-3A-68-4C) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 34 br-4e89077f1f1d (192.168.80.1) (02-42-32-3C-54-C1) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 35 br-a8f47009de04 (172.30.0.1) (02-42-34-0C-15-A0) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 1 lo (::1) (00-00-00-00-00-00) (loopback: 1)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 3 enp38s0 (2600:1700:2f70:a30f:81ec:cdbd:e828:d11f) (2C-F0-5D-2A-36-2D) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 3 enp38s0 (2600:1700:2f70:a30f:c94e:ba9e:86d3:cda) (2C-F0-5D-2A-36-2D) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG -  * 15 br-mailcow (fd4d:6169:6c63:6f77::1) (02-42-78-F9-D2-D7) (loopback: 0)
May 26, 2023 23:17:34.020 [139812229565072] DEBUG - Creating NetworkServices singleton.
May 26, 2023 23:17:34.020 [139812229565072] DEBUG - NetworkServices: Initializing...

  1. You have a proxy - Anything is fair game with those. If it burps, youā€™re hosed and connection breaks then falls back to SW

  2. This bothers me:

Request: [192.168.64.2:51476 (WAN)]

Itā€™s an RFC-1918 (Private Network) address which is WAN? (See massive network spaghetti above :wink: )

Iā€™ll be in health-recovery mode next 12 weeks. Today was 5th day home.
I tried to work on my own code and couldnā€™t.

Perhaps next week after Iā€™ve had more time to rest?

Stupid checks: Does your nvidia allow more than 2 simultaneous transcodes ??

My bet is the network / bonding / bridging / aliasing on how this is all mapped together. Until I know more, best I can give is dumb looks and :man_shrugging: Sorry

4

Iā€™ll be in health-recovery mode next 12 weeks. Today was 5th day home.
I tried to work on my own code and couldnā€™t.

Sorry to hear that, wishing you a speedy recovery!

To answer your questions:

  1. I have a bunch of dockers running so that probably explains all the weird network configs
  2. Sorry Iā€™m not totally following how that would cause it, if the connection drops from the outside Plex usually picks right back up, I can usually restart my plexmediaserver docker instance without loosing anyone. Restarting the Plex docker is usually the solve for this, my proxy doesnā€™t need restarted
  3. enp38s0 (192.168.1.1) is my NIC, the rest are just dockers on the network.

Stupid checks: Does your nvidia allow more than 2 simultaneous transcodes ??: From my testing, yes. But I see this even when its just 1 transcode ongoing.

Thanks again!
Trever

5

@ChuckPa - I played around with this more to try and eliminate the proxy and it happens even when Iā€™m locally connected without going through my reverse proxy. Iā€™ve changed a bunch of docker settings without avail.

FFMPEG continues to log a

Jun 09, 2023 23:18:07.075 [140432751758136] DEBUG - [Req#1b4/Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: I/O error

Error and I cant figure out why- sometimes if I restart the docker it works, sometimes it doesnt. I am passing through /dev/dri/ fully and I have also tried using the docker functions but ether way eventually Plex starts failing out. And on top of that, it doesnt fallback to CPU, it just fails out the steam entirely.

6

Did you put enough water in the :teapot: ? lol (I couldnā€™t resist. sorry)

iā€™ll need to see that sometime (when you can catch it).

1 Like
7

I actually index all of my Plex logs to a Splunk index- so I have ~2.3B events (I am a data hoarder for no particular reason) that I am happy to give you access to if that would help. I can reproduce the issue pretty often- just not sure what I am looking for.

8

Plex Media Server Logs_2023-07-03_00-51-51.zip (3.5 MB)

But here is a new log file- I tried to playback from my phone. My GPU is fine I can see it in the docker but it just spins on my phone.

9

However, I can see this:

Sun Jul  2 17:53:22 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.105.01   Driver Version: 515.105.01   CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:2E:00.0 Off |                  N/A |
| 78%   45C    P2    38W / 170W |   5479MiB / 12288MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A   3216761      C   ...aserver/Plex Media Server     5488MiB |
+-----------------------------------------------------------------------------+

And according to the dashboard, no one is transcoding. (Except me trying, but failing to buffer)

10

One more example: Right after a fresh bounce of the docker container.

Plex Media Server Logs_2023-07-03_00-57-23.zip (3.3 MB)

11

I donā€™t see what the source bitrate or codec is but youā€™re burning the subtitles into the stream

Jul 03, 2023 00:43:00.247 [139648243284792] DEBUG - [Req#5ff38/Transcode] MDE: Cannot direct stream audio stream due to profile or setting limitations
Jul 03, 2023 00:43:00.247 [139648243284792] DEBUG - [Req#5ff38/Transcode] MDE: analyzing media item 253043
Jul 03, 2023 00:43:00.247 [139648243284792] DEBUG - [Req#5ff38/Transcode] MDE: Dune: Part One (2021): Direct Play is disabled
Jul 03, 2023 00:43:00.247 [139648243284792] DEBUG - [Req#5ff38/Transcode] MDE: Dune: Part One (2021): media must be transcoded in order to use the hls protocol
Jul 03, 2023 00:43:00.247 [139648243284792] DEBUG - [Req#5ff38/Transcode] MDE: Dune: Part One (2021): no direct play video profile exists for http/mpegts/hevc
Jul 03, 2023 00:43:00.248 [139648243284792] DEBUG - [Req#5ff38/Transcode] MDE: Dune: Part One (2021): no direct play video profile exists for http/mpegts/hevc/ac3
Jul 03, 2023 00:43:00.248 [139648243284792] DEBUG - [Req#5ff38/Transcode] MDE: Dune: Part One (2021): no direct play video profile exists for http/mpegts/hevc
Jul 03, 2023 00:43:00.248 [139648243284792] DEBUG - [Req#5ff38/Transcode] MDE: Dune: Part One (2021): no direct play video profile exists for http/mpegts/hevc/ac3
Jul 03, 2023 00:43:00.248 [139648243284792] DEBUG - [Req#5ff38/Transcode] Dune: Part One - audio.channels limitation applies: 8 > 6
Jul 03, 2023 00:43:00.248 [139648243284792] DEBUG - [Req#5ff38/Transcode] MDE: Dune: Part One (2021): selected subtitle cannot be converted to a compatible format, burning into video stream
Jul 03, 2023 00:43:00.248 [139648243284792] DEBUG - [Req#5ff38/Transcode] MDE: Dune: Part One (2021): avoiding video remux due to burned subtitle stream
Jul 03, 2023 00:43:00.248 [139648243284792] DEBUG - [Req#5ff38/Transcode] MDE: Dune: Part One (2021): no remuxable profile found, so video stream will be transcoded
Jul 03, 2023 00:43:00.248 [139648243284792] DEBUG - [Req#5ff38/Transcode] Codecs: testing h264_nvenc (encoder)
Jul 03, 2023 00:43:00.248 [139648243284792] DEBUG - [Req#5ff38/Transcode] Codecs: hardware transcoding: testing API nvenc
Jul 03, 2023 00:43:00.248 [139648243284792] DEBUG - [Req#5ff38/Transcode] [FFMPEG] - Loaded lib: libcuda.so.1
Jul 03, 2023 00:43:00.248 [139648243284792] DEBUG - [Req#5ff38/Transcode] [FFMPEG] - Loaded sym: cuInit
Jul 03, 2023 00:43:00.248 [139648243284792] DEBUG - [Req#5ff38/Transcode] [FFMPEG] - Loaded sym: cuDeviceGetCount

Subtitle burning of PGS, VOBSUB, DVDRIP (image formats are hard for everyone)

Subtitle burning is SINGLE CORE / STREAM. (It canā€™t be done any other way. One core to write the subtitles image into each frame as the frame comes out of the HW decoding)

I have an i7-7700 CPU. You and I have about the same per-thread performance.

https://www.cpubenchmark.net/cpu.php?cpu=AMD+Ryzen+9+3900&id=3575

You can do a lot at the same time but no one thing is blazing fast.

30823

Single Thread Rating: 2612
Samples: 292*

Identical problem with my E5-2690 v4 + Nvidia.

https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E5-2690+v4+%40+2.60GHz

19958

Single Thread Rating: 2064
Samples: 123*

ā€“ Or am I missing the root issue here?

12

Sorry- this should be using my NVIDIA GPU for all of that shouldnā€™t it? I have a 3060 in here and what I am seeing constantly is an error with CUDA despite everything else working fine with it.

For the sake of ruling out any Docker issues, I just installed it bare metal. I appear to have the exact same issues.

If I turn off HW accel, works without issues.

Fom my Mac (using the Plex App)

image
image957Ɨ523 17.6 KB

13

Let me try to repro here.

I will grab something with subtitles and see what happens.

14

Thank you!

I am going to run this on bare metal for a bit to see if it was just a hiccup earlier. If it works flawlessly without running it in a docker it would seem that there is something up with the docker interface with Plex.

15

Iā€™m trying to break it and canā€™t.

Difference for me is SRT subtitles (which are sent as a 3rd stream)

Screenshot from 2023-07-02 22-33-19

16

Hmph- ok. Thanks for giving it a go. I am going to rule out the Docker component of this. I like having everything in containers. If thats the issue and it runs beautifully over the next few weeks I can report back. Maybe there is something funky about the way the container is set up (I am using the official Plex container from plexinc fwiw)

Do you run yours bare metal?

17

I run it bare metal.

For me, there is nothing to gain by using the namespace & runtime environment wrapping of the container. With bare metal -

  • I get to use the same OS libraries the host uses for everything else.
  • My container uses ā€œhostā€ networking so I bypass the networking layer
  • The Nvidia layer is common anyway ā€“ less the pass through of the /dev/dri into the containerā€¦ Now Iā€™ve got it directly.

If I were to run it in some kind of container, Iā€™d spin up a proper LXC (about 200 MB) where you can put big applications easily. The only thing bigger than that is a full VM .

Docker Plex came about, and is used for, those situations where there is no native app. (thatā€™s what Docker is all about)

We have a native app. The PMS metadata is mapped to the file system (relocated with a systemd override.conf file). Thereā€™s no functional difference

FWIW: All my media processing tools (which are largely web browser-based) run in their own LXC. It makes backup trivial because of how LXCs work. Those apps think they are installed ā€˜nativeā€™. (thatā€™s the beauty of virtualizing the OS)

18

Thanks! I have never used LXC but I will keep that in mind. I run as much as I can in dockers, like my email and such I host, but this might be more trouble than its worth.

Iā€™ll monitor and report back if there is an issue with GPU and Plex in Docker. Appreciate all the help!

19

So it doesnt seem as bad, but I still have the issue of

Jul 07, 2023 18:41:03.960 [139998464449336] DEBUG - [Req#1d3a5/Transcode] Codecs: hardware transcoding: opening hw device failed - probably not supported by this system, error: Generic error in an external library

Plex Media Server Logs_2023-07-07_18-49-18.zip (4.9 MB)

If I restart the Plex service, it starts to work again. (This is running bare metal now)

20

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.