Server Version#: 1.32.5.7349
Player Version#: WebOS Latest
In the last few weeks, I’ve been getting transcoder crashes while streaming locally. I’ve updated drivers, I’ve downgraded Plex. I’ve checked storage etc.
I honestly am out of ideas.
I’ve had several in the last hour (current time is 8:45PM Central 8/1/2023)
Could someone take a look at the logs and see if they can find any reason for this?
Plex Media Server Logs_2023-08-02_01-45-18.zip (3.4 MB)
It is also worth mentioning that it doesn’t occur every time. I’ve tried swapping media source as well.
Those of us who use Ubuntu and Nvidia have been plagued with Nvidia driver problems for the past few weeks.
If you got caught up in it, it’s been a sleigh ride with nothing making sense.
Nvidia-drivers 535.86.05 (In the Ubuntu repo) finally gets it sorted out.
Recommendation is to
Go into PMS settings for the transcoder and select the Nvidia then SAVE the changes.
Damn I appreciate you. I thought I was losing my mind.
Hmm actually that didn’t work 
@ChuckPa do we have any indication this is strictly a driver issue? Kernel + driver?
I ask, because you mentioned the Ubuntu / Nvidia combo.
Would it be worthwhile porting my install to another distro (It’s pretty minimal effort if needed)
Would a driver rollback help?
Please detail what you have… (roll up the sleeves)
nvidia-smi works correctly and shows the card.Yes, this is strictly a Ubuntu / Nvidia driver issue.
There are several of us battling it.
Personally I have been battling it on my server and with our lab server.
It’s a Royal P.I.T.A.
The 525 drivers were fine.
The initial releases of the 535 drivers broke everything CUDA related (including Windows systems)
Supplemental:
The QA AV1 machine (RocketLake + Nvidia RTX 3060) has the 525 drivers installed.
My personal machine has the working 535 drivers with a P2200 .
( Card + driver version does matter while Nvidia sorts it out )
chuck@plexqa-av1:~$ dpkg -l | grep nvidia
ii libnvidia-cfg1-525:amd64 525.125.06-0ubuntu0.23.04.1 amd64 NVIDIA binary OpenGL/GLX configuration library
ii libnvidia-common-525 525.125.06-0ubuntu0.23.04.1 all Shared files used by the NVIDIA libraries
ii libnvidia-compute-525:amd64 525.125.06-0ubuntu0.23.04.1 amd64 NVIDIA libcompute package
ii libnvidia-decode-525:amd64 525.125.06-0ubuntu0.23.04.1 amd64 NVIDIA Video Decoding runtime libraries
ii libnvidia-egl-wayland1:amd64 1:1.1.10-1 amd64 Wayland EGL External Platform library -- shared library
ii libnvidia-encode-525:amd64 525.125.06-0ubuntu0.23.04.1 amd64 NVENC Video Encoding runtime library
ii libnvidia-extra-525:amd64 525.125.06-0ubuntu0.23.04.1 amd64 Extra libraries for the NVIDIA driver
ii libnvidia-fbc1-525:amd64 525.125.06-0ubuntu0.23.04.1 amd64 NVIDIA OpenGL-based Framebuffer Capture runtime library
ii libnvidia-gl-525:amd64 525.125.06-0ubuntu0.23.04.1 amd64 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii nvidia-compute-utils-525 525.125.06-0ubuntu0.23.04.1 amd64 NVIDIA compute utilities
ii nvidia-dkms-525 525.125.06-0ubuntu0.23.04.1 amd64 NVIDIA DKMS package
ii nvidia-driver-525 525.125.06-0ubuntu0.23.04.1 amd64 NVIDIA driver metapackage
ii nvidia-kernel-common-525 525.125.06-0ubuntu0.23.04.1 amd64 Shared files used with the kernel module
ii nvidia-kernel-source-525 525.125.06-0ubuntu0.23.04.1 amd64 NVIDIA kernel source package
ii nvidia-prime 0.8.17.1 all Tools to enable NVIDIA's Prime
ii nvidia-settings 510.47.03-0ubuntu1 amd64 Tool for configuring the NVIDIA graphics driver
ii nvidia-utils-525 525.125.06-0ubuntu0.23.04.1 amd64 NVIDIA driver support binaries
ii screen-resolution-extra 0.18.3 all Extension for the nvidia-settings control panel
ii xserver-xorg-video-nvidia-525 525.125.06-0ubuntu0.23.04.1 amd64 NVIDIA binary Xorg driver
chuck@plexqa-av1:~$
logout
Connection to av1 closed.
[chuck@lizum ~.2000]$ gog dpkg -l | grep nvidia
ii gpustat 0.6.0-1 all pretty nvidia device monitor
ii libnvidia-cfg1-535-server:amd64 535.54.03-0ubuntu0.22.04.1 amd64 NVIDIA binary OpenGL/GLX configuration library
ii libnvidia-common-535-server 535.54.03-0ubuntu0.22.04.1 all Shared files used by the NVIDIA libraries
ii libnvidia-compute-535-server:amd64 535.54.03-0ubuntu0.22.04.1 amd64 NVIDIA libcompute package
ii libnvidia-decode-535-server:amd64 535.54.03-0ubuntu0.22.04.1 amd64 NVIDIA Video Decoding runtime libraries
ii libnvidia-encode-535-server:amd64 535.54.03-0ubuntu0.22.04.1 amd64 NVENC Video Encoding runtime library
ii libnvidia-extra-535-server:amd64 535.54.03-0ubuntu0.22.04.1 amd64 Extra libraries for the NVIDIA Server Driver
ii libnvidia-fbc1-535-server:amd64 535.54.03-0ubuntu0.22.04.1 amd64 NVIDIA OpenGL-based Framebuffer Capture runtime library
ii libnvidia-gl-535-server:amd64 535.54.03-0ubuntu0.22.04.1 amd64 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii libnvidia-ml-dev:amd64 11.5.50~11.5.1-1ubuntu1 amd64 NVIDIA Management Library (NVML) development files
ii nvidia-compute-utils-535-server 535.54.03-0ubuntu0.22.04.1 amd64 NVIDIA compute utilities
ii nvidia-cuda-dev:amd64 11.5.1-1ubuntu1 amd64 NVIDIA CUDA development files
ii nvidia-cuda-gdb 11.5.114~11.5.1-1ubuntu1 amd64 NVIDIA CUDA Debugger (GDB)
ii nvidia-cuda-toolkit 11.5.1-1ubuntu1 amd64 NVIDIA CUDA development toolkit
ii nvidia-cuda-toolkit-doc 11.5.1-1ubuntu1 all NVIDIA CUDA and OpenCL documentation
ii nvidia-dkms-535-server 535.54.03-0ubuntu0.22.04.1 amd64 NVIDIA DKMS package
ii nvidia-driver-535-server 535.54.03-0ubuntu0.22.04.1 amd64 NVIDIA Server Driver metapackage
ii nvidia-firmware-535-server-535.54.03 535.54.03-0ubuntu0.22.04.1 amd64 Firmware files used by the kernel module
ii nvidia-kernel-common-535-server 535.54.03-0ubuntu0.22.04.1 amd64 Shared files used with the kernel module
ii nvidia-kernel-source-535-server 535.54.03-0ubuntu0.22.04.1 amd64 NVIDIA kernel source package
ii nvidia-opencl-dev:amd64 11.5.1-1ubuntu1 amd64 NVIDIA OpenCL development files
ii nvidia-prime 0.8.17.1 all Tools to enable NVIDIA's Prime
ii nvidia-profiler 11.5.114~11.5.1-1ubuntu1 amd64 NVIDIA Profiler for CUDA and OpenCL
ii nvidia-utils-535-server 535.54.03-0ubuntu0.22.04.1 amd64 NVIDIA Server Driver support binaries
ii nvidia-visual-profiler 11.5.114~11.5.1-1ubuntu1 amd64 NVIDIA Visual Profiler for CUDA and OpenCL
ii xserver-xorg-video-nvidia-535-server 535.54.03-0ubuntu0.22.04.1 amd64 NVIDIA binary Xorg driver
[chuck@lizum ~.2001]$
Ahhhh you are using the server version, I didn’t think that would be the applicable instance. Let me try that. Otherwise, I’ll just roll it back to 525 and call it a day
Still fails (purged, rebooted, installed, dkms / initramfs ran, reboot, reinstall plex)
Rolling back to 525, will report back
Nope. 525.125.06 is still doing it
is the Plex user (either plex:plex or the UID you run as) a member of the group which owns /dev/dri ?
This is not rocket science… taps foot … lol
Last login: Wed Aug 2 15:33:27 2023 from 10.2.15.100
chuck@plexqa-av1:~$ groups plex
plex : plex video render
chuck@plexqa-av1:~$ ls -la /dev/dri
total 0
drwxr-xr-x 3 root root 140 Jul 29 16:43 .
drwxr-xr-x 19 root root 4780 Jul 30 03:11 ..
drwxr-xr-x 2 root root 120 Jul 29 16:43 by-path
crw-rw---- 1 root render 226, 0 Jul 29 16:43 card0
crw-rw---- 1 root render 226, 1 Jul 29 16:43 card1
crw-rw---- 1 root render 226, 128 Jul 29 16:43 renderD128
crw-rw---- 1 root render 226, 129 Jul 29 16:43 renderD129
chuck@plexqa-av1:~$
plex : plex video
root@gf:~# ls -la /dev/dri
total 0
drwxr-xr-x 3 root root 100 Aug 2 17:02 .
drwxr-xr-x 19 root root 4420 Aug 2 17:08 …
drwxr-xr-x 2 root root 80 Aug 2 17:05 by-path
crw-rw---- 1 root video 226, 0 Aug 2 17:05 card0
crw-rw---- 1 root video 226, 128 Aug 2 17:05 renderD128
lol you would think? A couple decades in IT but here I am stumped
groups plex returns video as one of the members?
When you start Plex (DEBUG loggging)
At the top of the log, it will query the PCIe bus.
Does it show the Nividia ?
Aug 02, 2023 19:09:08.757 [140521999346320] INFO - Running migrations. (EPG 0)
Aug 02, 2023 19:09:08.757 [140521999346320] DEBUG - Captured session 0.
Aug 02, 2023 19:09:08.757 [140521999346320] DEBUG - Captured session 1.
Aug 02, 2023 19:09:08.759 [140521959430968] DEBUG - [GPU] Got device: TU106 [GeForce RTX 2070], nvidia@unknown, default true, best true, ID 10de:1f02:1043:8796@0000:09:00.0, DevID [10de:1f02:1043:8796], flags 0xe8
I believe this is what you are looking for?
So Plex does see it.
Now go down to the MDE: section and look what the final decision was.
Then look at the TPU: line to see what the transcoder decided.
(Just above TPU: you will see it testing the GPU capabilities)
Please show me the entire FFMPEG invocation command line
Aug 02, 2023 19:09:16.717 [140521872440120] DEBUG - [Req#43/Transcode] MDE: Selected protocol hls; container: mpegts
Aug 02, 2023 19:09:16.717 [140521872440120] DEBUG - [Req#43/Transcode] MDE: analyzing media item 154709
Aug 02, 2023 19:09:16.717 [140521872440120] DEBUG - [Req#43/Transcode] MDE: Bambi (1942): Direct Play is disabled
Aug 02, 2023 19:09:16.717 [140521872440120] DEBUG - [Req#43/Transcode] MDE: Bambi (1942): media must be transcoded in order to use the hls protocol
Aug 02, 2023 19:09:16.717 [140521872440120] DEBUG - [Req#43/Transcode] Bambi - video.width limitation applies: 1454 > 1280
Aug 02, 2023 19:09:16.717 [140521872440120] DEBUG - [Req#43/Transcode] Bambi - audio.channels limitation applies: 6 > 2
Aug 02, 2023 19:09:16.717 [140521872440120] DEBUG - [Req#43/Transcode] Bambi - video.width limitation applies: 1454 > 1280
Aug 02, 2023 19:09:16.717 [140521872440120] DEBUG - [Req#43/Transcode] MDE: Bambi (1942): no remuxable profile found, so video stream will be transcoded
Aug 02, 2023 19:09:16.717 [140521872440120] DEBUG - [Req#43/Transcode] Codecs: testing h264_nvenc (encoder)
Aug 02, 2023 19:09:16.717 [140521872440120] DEBUG - [Req#43/Transcode] Codecs: hardware transcoding: testing API nvenc for device 'pci:0000:09:00.0' (TU106 [GeForce RTX 2070])
MDE
MODERATOR EDIT: Legibility
Aug 02, 2023 19:09:18.276 [140521872440120] DEBUG - [Req#25/Transcode] TranscodeUniversalRequest: using augmented profile Roku-7.x
Aug 02, 2023 19:09:18.276 [140521872440120] DEBUG - [Req#25/Transcode] Downloading document http://127.0.0.1:32400/library/metadata/96656?checkFiles=1&includeBandwidths=1&offset=4010&X-Plex-Client-Capabilities=videoDecoders%3dh264%7bprofile%3ahigh%26resolution%3a1080%26level%3d41%7d%3baudioDecoders%3daac%7bchannels%3a2%7d%2cac3%7bchannels%3a6%7d%2ceac3%7bchannels%3a8%7d&X-Plex-Incomplete-Segments=1&X-Plex-Session-Identifier=3289d58981c53c9399762cd1dbeb5f33
Aug 02, 2023 19:09:18.276 [140521872440120] DEBUG - [Req#25/Transcode/Req#4e] Auth: authenticated user 2619325 as McGergs
Aug 02, 2023 19:09:18.281 [140521872440120] DEBUG - [Req#25/Transcode/Req#4e] We're going to try to auto-select an audio stream for account 2619325.
Aug 02, 2023 19:09:18.281 [140521872440120] DEBUG - [Req#25/Transcode/Req#4e] Selecting best audio stream for part ID 162928 (language: )
Aug 02, 2023 19:09:18.281 [140521872440120] DEBUG - [Req#25/Transcode/Req#4e] Audio Stream: 505193, Subtitle Stream: -1
Aug 02, 2023 19:09:18.282 [140521872440120] DEBUG - [Req#25/Transcode] Found session GUID of 62bc1f5a-e6b7-4a2a-a930-0611f9896b8e-1 in session start.
Aug 02, 2023 19:09:18.282 [140521872440120] DEBUG - [Req#25/Transcode] Using session GUID 62bc1f5a-e6b7-4a2a-a930-0611f9896b8e-1 for new transcode session.
Aug 02, 2023 19:09:18.282 [140521872440120] DEBUG - [Req#25/Transcode] Cleaning directory for session 62bc1f5a-e6b7-4a2a-a930-0611f9896b8e-1 ()
Aug 02, 2023 19:09:18.282 [140521989897016] DEBUG - Completed: [208.73.94.124:36750] 200 GET /video/:/transcode/universal/start.m3u8?protocol=hls&mediaIndex=0&waitForSegments=1&videoQuality=100&session=62bc1f5a-e6b7-4a2a-a930-0611f9896b8e-1&maxVideoBitrate=4000&mediaBufferSize=50000&offset=4010&partIndex=0&videoResolution=1280x720&directPlay=0&path=%2Flibrary%2Fmetadata%2F96656&directStream=1&skipSubtitles=1 (14 live) #25 TLS GZIP 6ms 481 bytes (pipelined: 5)
Aug 02, 2023 19:09:18.308 [140521992006456] DEBUG - Auth: authenticated user 2619325 as McGergs
Aug 02, 2023 19:09:18.308 [140521880116024] DEBUG - Request: [208.73.94.124:36750 (WAN)] GET /video/:/transcode/universal/session/62bc1f5a-e6b7-4a2a-a930-0611f9896b8e-1/base/index.m3u8?protocol=hls&mediaIndex=0&waitForSegments=1&videoQuality=100&session=62bc1f5a-e6b7-4a2a-a930-0611f9896b8e-1&maxVideoBitrate=4000&mediaBufferSize=50000&offset=4010&partIndex=0&videoResolution=1280x720&directPlay=0&path=%2Flibrary%2Fmetadata%2F96656&directStream=1&skipSubtitles=1 (14 live) #50 TLS GZIP Signed-in Token (McGergs)
Aug 02, 2023 19:09:18.308 [140521880116024] DEBUG - [Req#50/Transcode/62bc1f5a-e6b7-4a2a-a930-0611f9896b8e-1] Starting a transcode session 62bc1f5a-e6b7-4a2a-a930-0611f9896b8e-1 at offset -1.0 (state=3)
Aug 02, 2023 19:09:18.309 [140521880116024] DEBUG - [Req#50/Transcode/62bc1f5a-e6b7-4a2a-a930-0611f9896b8e-1] TPU: hardware transcoding: using hardware decode accelerator nvdec
Aug 02, 2023 19:09:18.309 [140521880116024] DEBUG - [Req#50/Transcode/62bc1f5a-e6b7-4a2a-a930-0611f9896b8e-1] TPU: hardware transcoding: zero-copy support present
Aug 02, 2023 19:09:18.309 [140521880116024] DEBUG - [Req#50/Transcode/62bc1f5a-e6b7-4a2a-a930-0611f9896b8e-1] TPU: hardware transcoding: using zero-copy transcoding
Aug 02, 2023 19:09:18.309 [140521880116024] DEBUG - [Req#50/Transcode/62bc1f5a-e6b7-4a2a-a930-0611f9896b8e-1] [Universal] Using local file path instead of URL: /media/Data/PlexMedia/Kids Movies/Bambi (1942) [1080p] [BluRay] [5.1] [YTS.MX]/Bambi.1942.1080p.BluRay.x264.AAC5.1-[YTS.MX].mp4
Aug 02, 2023 19:09:18.309 [140521880116024] DEBUG - [Req#50/Transcode/62bc1f5a-e6b7-4a2a-a930-0611f9896b8e-1] TPU: hardware transcoding: final decoder: nvdec, final encoder: nvenc
Aug 02, 2023 19:09:18.309 [140521880116024] DEBUG - [Req#50/Transcode/62bc1f5a-e6b7-4a2a-a930-0611f9896b8e-1/JobRunner] Job running: CUDA_CACHE_PATH="/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Cache/Shaders/CUDA" FFMPEG_EXTERNAL_LIBS='/var/lib/plexmediaserver/Library/Application\ Support/Plex\ Media\ Server/Codecs/8217c1c-4578-linux-x86_64/' X_PLEX_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx "/usr/lib/plexmediaserver/Plex Transcoder" "-codec:#0x01" h264 "-hwaccel:#0x01" nvdec "-hwaccel_fallback_threshold:#0x01" 10 "-threads:#0x01" 1 "-hwaccel_output_format:#0x01" cuda "-hwaccel_device:#0x01" cuda "-codec:#0x02" aac -ss 4010 -analyzeduration 20000000 -probesize 20000000 -i "/media/Data/PlexMedia/Kids Movies/Bambi (1942) [1080p] [BluRay] [5.1] [YTS.MX]/Bambi.1942.1080p.BluRay.x264.AAC5.1-[YTS.MX].mp4" -filter_complex "[0:#0x01]hwupload[0];[0]scale_cuda=w=968:h=720:format=nv12[1]" -map "[1]" -codec:0 h264_nvenc -b:0 2307k -maxrate:0 3076k -bufsize:0 6152k -forced-idr:0 1 -r:0 23.975999999999999 -force_key_frames:0 "expr:gte(t,n_forced*5)" -filter_complex "[0:#0x02] aresample=async=1:ochl='5.1':rematrix_maxval=0.000000dB:osr=48000[2]" -map "[2]" -codec:1 ac3 -b:1 537k -segment_format mpegts -f ssegment -individual_header_trailer 0 -segment_time 5 -segment_start_number 802 -segment_copyts 1 -segment_time_delta 0.0625 -segment_list "http://127.0.0.1:32400/video/:/transcode/session/62bc1f5a-e6b7-4a2a-a930-0611f9896b8e-1/29424ffa-2e78-4c16-b832-4c2d2388bbb3/manifest?X-Plex-Http-Pipeline=infinite" -segment_list_type csv -segment_list_size 5 -segment_list_separate_stream_times 1 -segment_list_unfinished 1 -max_delay 5000000 -avoid_negative_ts disabled -map_metadata -1 -map_chapters -1 "media-%05d.ts" -start_at_zero -copyts -init_hw_device cuda=cuda:pci:0000:09:00.0 -filter_hw_device cuda -y -nostats -loglevel quiet -loglevel_plex error -progressurl http://127.0.0.1:32400/video/:/transcode/session/62bc1f5a-e6b7-4a2a-a930-0611f9896b8e-1/29424ffa-2e78-4c16-b832-4c2d2388bbb3/progress
Aug 02, 2023 19:09:18.309 [140521880116024] DEBUG - [Req#50/Transcode/62bc1f5a-e6b7-4a2a-a930-0611f9896b8e-1/JobRunner] In directory: "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Cache/Transcode/Sessions/plex-transcode-62bc1f5a-e6b7-4a2a-a930-0611f9896b8e-1-29424ffa-2e78-4c16-b832-4c2d2388bbb3"
TPU and FFMPEG
MODERATOR EDIT: Legibility
That is showing me it’s engaging and using the GPU.
Now where are you (not) seeing it failing to use the GPU?
The log ZIP file might be appropriate?
I’ll be able to see why it’s crashing during transcode
Plex Media Server.log (1.8 MB)
Figured I’d save you the time, line 571 is where the latest replication started.
