sorry for not understanding (very under the weather)
Please show me the different command lines (works vs fails) for the same videos?
@pozunmv is this the command used by 1.31.1.6617 ??as it is using nvdec (which as per our ffmpeg tests worked) they didn’t switch to hwaccel cuda ? <<<-hwaccel:0 nvdec >>>
plex 640864 23.5 1.2 5275764 397668 ? Sl 08:48 0:00 /usr/lib/plexmediaserver/Plex Transcoder -codec:0 hevc -hwaccel:0 nvdec -hwaccel_fallback_threshold:0 10 -threads:0 1 -extra_hw_frames:0 8 -hwaccel_output_format:0 cuda -hwaccel_device:0 cuda -analyzeduration 20000000 -probesize 20000000 -i /srv/ftp/movies/test/jellyfish-30-mbps-hd-hevc.mkv -filter_complex [0:0]hwupload[0];[0]scale_cuda=w=1920:h=1080:format=nv12[1] -map [1] -codec:0 h264_nvenc -b:0 20000k -preset:0 hq -forced-idr:0 1 -r:0 29.969999999999999 -force_key_frames:0 expr:gte(t,n_forced*3) -f dash -seg_duration 3 -dash_segment_type mp4 -init_seg_name init-stream$RepresentationID$.m4s -media_seg_name chunk-stream$RepresentationID$-$Number%05d$.m4s -window_size 5 -delete_removed false -skip_to_segment 1 -time_delta 0.0625 -manifest_name http://127.0.0.1:32400/video/:/transcode/session/6vt223fzg6e5vviqes0eoi8q/c897846c-d569-46fd-9fe2-799d8a621ced/manifest?X-Plex-Http-Pipeline=infinite -avoid_negative_ts disabled -map_metadata -1 -map_chapters -1 dash -start_at_zero -copyts -vsync cfr -init_hw_device cuda=cuda: -filter_hw_device cuda -y -nostats -loglevel quiet -loglevel_plex error -progressurl http://127.0.0.1:32400/video/:/transcode/session/6vt223fzg6e5vviqes0eoi8q/c897846c-d569-46fd-9fe2-799d8a621ced/progress
Sat Feb 11 08:48:53 2023
@plex_famverhaegen.be the issue is with the hwaccel_output_format being cuda, or at least when I can reliably force the issue outside of plex with ffmpeg. What’s really interesting is that the transcoding fails when going to 1080 from 1080, however stepping it down to output 720 it works fine with how plex is doing it, which seems to indicate a problem either in ffmpeg or nvidia. However given the various drivers I’ve tried I would say it’s more likely an issue in ffmpeg with the options plex is running and with how they are compiling it.
@pozunmv agree it is ffmpeg
using ffmpeg directly i can confirm that ffmpeg -hwaccel nvdec -hwaccel_output_format cuda -i input.mkv -c:v h264_nvenc -preset slow output.mp4 is not working
Will see if i can the full option list from the previous working PMS.
@ChuckPa
Forget this “” hwaccel:0 nvdec""
I had a detailed look at the transcode parameters between 1.29.2.6364 and 1.31.1.6617 (share by @pozunmv) and besides the -extra_hw_frames:0 8 can’t spot a significant difference.
From the tests performed; we know that there’s an issue with ffmpeg (with recent versions) when using -hwaccel_output_format:0 cuda with hvec 8 bit encoded files.
Was the version of the ffmpeg libraries linked by the transcoder updated after 1.29.2.6364 … ? and by this plex picked up a ffmpeg bug.
I confirm the testing done by @pozunmv changing -hwaccel_output_format cuda to -hwaccel_output_format auto looks to be the change required to resolve the current issue with ffmpeg and 8 bit hvec
I don’t know if any libraries were bumped when AV1 decoding was added.
From where I sit, I can see (like you)
- Transcoder was OK
- Nvidia driver version major update.
- AV1 codec added
- bit reduction transcoding fails but resolution change succeeds
- Selected GPUs fail with Nvidia drivers CUDA 12.0
Known
-
Nvidia limits the number of transcodes on GeForce cards
-
Nvidia doesn’t limit the number of transcodes on Quadro Cards (P2xxx)
-
Nvidia GeForce card drivers must be “patched” to allow multiple streams (more than a typical gamer would need)
Challenge:
- Did Nvidia, with CUDA version 12.0 drivers, make a change such that the existing ‘patch’ process breaks/disables decoding in such a watch that patching is no longer valid and the driver ultimately fails?
If you step back from this:
- Run PMS 1.29.2.6364 on CUDA 12.0 drivers – What happens? – Failure (?)
- Do ALL Quadra cards work with any PMS and any Nvidia driver (up to limit of the card) ?
@ChuckPa Plex fails when utilizing CUDA 11.x or CUDA 12… you cannot blame it on the CUDA version as the same behavior is seen with both 11 and 12 versions.
The NVIDIA patch allows more than 2 concurrent streams to be enabled on the card at the same time. The current patch method didnt have a patch for the most recent nvidia drivers I just installed. So I am bone standard nvidia drivers and its failing when its the only transcode, but will successfully transcode other media and that same media when stepping down from 1080p to 720p. This should rule out the issue with NVIDIA drivers being the cause of it.
Thank you.
I put up a hypothesis and was hoping it would be immediately definitive either way.
My process is to keep eliminating variables until only one remains.
@ChuckPa
Agree with @pozunmv …
It is not the CUDA version nor nvidia drivers … i tried all go them down to 470
I’m still running Pre CUDA 12 at the moment. I can try the CUDA 12 with 1.29 but that’s another test scenario that doesn’t add to the 8 bit hvec issue. Yes there might be other issue with CUDA 12; that 's why when this all started i went down to pre 12, to exclude CUDA 12
I don’t patch my driver card as the 3 slots(not 2) are more than enough for normal use @home
This seems to affect all nvidia users that enable hardware transcoding with different hardware.
The only change that resolves the issue is downgrade to 1.29.2.6364 ; looking at the transcoding settings nothing has changed. Earlier this month I found an ffmpeg issue that was not followed up as the team classified it as a muxing fault (while my files are coded with ffmpeg --cuvid HW option-- ).
All this makes me conclude the library were bumped and now Plex has imported this outstanding ffpmeg bug.
The good news is that hwaccel_output_format auto doesn’t seem to have the issue. Can you ask the team to explore this instead of the extra_hw_frames option.
I’m trying to put this in a spreadsheet.
Probably separated by:
- PMS version
- One group of QSV vs PMS version, Another group of Nvidia vs PMS version.
- HEVC input
- H264 input
- (other codecs?)
- 2160p resolution → Same dropping to 1080/720
- 1080p resolution → Same dropping to 720
- 2160p → 2160p bit rate reducton
- 1080p → 1080p bit rate reduction
- 720p → 720p bit rate reduction
I think PMS version can easily be columns with Yes/No answers
when Features/Capabilities are the rows.
Anyone have an idea how to layout and group the rows?
(we should be inclusive of Nvidia and QSV in order to present a clear status)
this is a big task unless someone has a better idea?
@ChuckPa
Not sure what matrix you are trying to build ?
1/ YES : sure and the linked ffmpeg lib version! we know already that it works upto 1.29.2.6364 by all the test that have been done
2/ NO? : I haven’t seen any intel HW issues in this post ?
3/ YES : but you need A/8 bit and B/10 bit >> when i reconvert troubling content to 10 bits all goes fine again[used handbrake]
5/ NO?: do we have any more issues reported here
6/ NO?: seems all fine
7/ NO?: that looks to work also
8-10 / ?? : not tested / not using
What problem are we trying to pin point here ? the 8 bit hvec issue or any transcoder problem that could occur …?
========edit========
Maybe instead of creating this impossible matrix 
Can we have version that exposes hwaccel_output_format and hwaccel in prefences file?
This will allow us to test and give much needed feedback
@ChuckPa Full of enthusiasm & optimism, I have now replicated what I could based on your unbreakable setup. Took one of my trusty old Intel servers from ~2014. Installed Ubuntu 20.04.5 LTS on a SSD. Located a Nvidia P2000 card and installed it. Carefully used the exact 525.60.13 nvidia drivers from the Ubuntu repository. Installed PMS 1.31.0.6654, and started testing in my favorite player, Chrome.
Imagine my surprise when transcoding breaks on absolutely anything I could throw at it. 100% failure rate when changing from direct play to any quality on any file whatsoever. Incredibly slow starting, spinning & hanging for a long time, or forever in some cases. On the occasional instance where playback actually started (only possible with transcoding on by default). It would also fail in 100% cases when I tried switching to any other quality.
I then decided to test the same on my Intel workstation. Same GPU, drivers, Ubuntu, PMS version etc etc. Results: Impossible to make it fail. I can literally change the quality back and forth forever on any file I tested previously, and it works 100% of the time on all of them.
I tested another workstation and two other servers, and exact same behavior for all. Works on the workstations, not on the servers.
I tried a BUNCH of different versions of Ubuntu, drivers, PMS and the results were always the same no matter what I did. Going back to PMS 1.29.2.6364 does not work any better regarding my particular issue, although it is obviously solving another issue people are also running into.
I think it is equally important to note also. I tested changing Ubuntu versions, Nvidia versions, PMS versions many many times on the working setups. And they also keep on working, no matter what I do. While the servers stay broken no matter what I do.
To summarize:
OS: Ubuntu 20.04.5 LTS (Also tested 18.04, 22.04, 22.10 & others)
GPU: Nvidia Quadro P2000
Drivers: 525.60.13 (Also tested 470, 515, & others)
PMS: 1.31.0.6654 (Also tested 1.29.2.6364, 1.30.2.6563 & others)
Workstation 1:
CPU: 12th Gen Intel(R) Core™ i9-12900KS
Motherboard: MSI Z690-A WIFI
Memory: 64GB DDR5 (5200 MT/s)
Result: Works
Workstation 2:
CPU: 11th Gen Intel(R) Core™ i9-11900K
Motherboard: Gigabyte Z590 Vision G
Memory: 128GB DDR4 (3200 MT/s)
Result: Works
Server 1:
CPU: Intel® Xeon® Processor E5-2660 v3 (Intel Xeon Processor E52660 v3 25M Cache 2.60 GHz Product Specifications)
Motherboard: Intel® Server Board S2600WTT (Intel Server Board S2600WTT Product Specifications)
Memory: 128GB DDR4 ECC (2133 MT/s)
Result: Fails
Server 2:
CPU: Intel® Xeon® Processor E5-2690 v2 (Intel Xeon Processor E52690 v2 25M Cache 3.00 GHz Product Specifications)
Motherboard: Intel® Server Board S2600GZ Family (Intel® Server Board S2600GZ)
Memory: 256GB DDR3 ECC (1066 MT/s)
Result: Fails
Server 3:
CPU: Intel® Xeon® Processor E5-2630 v4 (Intel® Xeon® Processor E5-2630 v4)
Motherboard: Gigabyte MD90-FS0-XX (MD90-FS0 (rev. 1.1) | Server Motherboard - GIGABYTE Global)
Memory: 128GB DDR4 ECC (2133 MT/s)
Result: Fails
In hindsight, I realize now that exactly ALL my friends who are having issues with nvidia transcoding, have the same thing in common, they are running on server class equipment. Even recent one from 2020+. And those who don’t have issues are running on regular desktop workstations.
I have also tested transcoding with JF with all 3 servers. It works fine, no difference compared to the workstations.
@ChuckPa any thoughts ?
@Ossalingur
Does the problem appear on the server class equipment with the 515 Drivers?
Wild guess but it could be a problem with the PCI-e bus/transfers. Can you reproduce with standard ffmpeg? if you can try hwclass cuda instead of hwclass nvdec (both with hwaccel_output_format auto).
Yes, the issue persists on all driver versions between 470 - 525 I could find in both Ubuntu & Debian repositories .
I am really curious if there is actually anyone, who can confirm that they have a server class motherboard, any Intel Xeon, any Nvidia card, any PMS version. And transcoding is working properly ?
By properly, I mean that you can switch between different quality in Chrome, and playback starts again. If playback starts again, try seeking forward a few minutes.
I have not tested anything with the standalone ffmpeg, I am pretty sure that the real underlying issue that I am dealing with, is happening before Plex Transcoder is actually started, maybe during whatever Plex Media Server is doing when it’s looping over testing VAAPI/NVENC/NVDEC 6 times.
Because when the Plex Transcoder actually manages to start properly, both playback & seeking back and forth nonstop works just fine. And the Plex Transcoder process is being restarted during each seek since there is a new PID each time.
@pozunmv
Did some further research on the different hwacell and some say that hwaccel cuda and nvdec are the same [an alias] next to hwaccel cuvid
Unclear in the blog; is not really explaining the difference between cuvid and cuda either.
And the ffmpeg wiki >> FFmpeg offers both NVDEC and CUVID hwaccel , but uses cuda in the examples (except the last nvdec)
http://ffmpeg.org/pipermail/ffmpeg-devel/2018-November/235929.html
http://ffmpeg.org/pipermail/ffmpeg-devel/2018-November/235930.html
@ChuckPa
Would like to chime in on this thread that I too had this issue. I posted about this on /r/plex but will repaste here for you guys.
Hi all, I am on Unraid using an Nvidia GPU to transcode. HW transcoding comes on ONLY at bitrates less than 8 Mbps. I tested by pulling up a movie, did play original quality. I then stepped through the convert options (20 Mbps, 12 Mbps, 10 Mbps, 8 Mbps, 4 Mbps, 3 Mbps) and waiting 2 minutes on each one. HW tag only started appearing on Tautulli at 4 Mbps and less; on higher bitrates it was still transcoding but no HW tag and according to my unraid dashboard the CPU started getting hit hard.
The strange thing is that at the higher bitrates (when the HW tag is missing) the job shows up on nvidia-smi command, but the power state is P8 (sleep). The power state only leaves P8 at the 4 Mbps or less threshold mentioned above. Any thoughts?
Plex job recognized by GPU but in sleep state
Plex job recognized by GPU but in sleep state
No HW tag at 8 Mbps; starts working at 4 Mbps and lower
Let me know if you need any more information. For further context, I am on Nvidia driver version525.89.02.
When I downgraded to Plex Media Server version 1.30.0.6486 the issue was fixed.
1 Like
If you’re going to “Me too” here, PLEASE remember to include the DEBUG server logs which capture this.
Seeing screenshots tells us nothing about what happened.
We need to understand HOW it happened.
@ChuckPa
Apologies, figured the extra detail about quality cutoff and GPU power state was useful. Here are today’s logs.
Plex Media Server Logs_2023-02-13_21-51-58.zip (6.6 MB)
You’re burning in subtitles.
PMS is using the GPU but the CPU is doing the bulk of the work to burn in the subtitles into each frame after encoding
EBUG - Request: [192.168.1.193:37164 (Subnet)] GET /video/:/transcode/universal/session/dcf5334784e7f1fa-com-plexapp-android/base/index.m3u8 (18 live) #eb5 TLS GZIP Signed-in
Feb 13, 2023 18:19:56.218 [0x1467e00fab38] DEBUG - [Req#eb5/Transcode/dcf5334784e7f1fa-com-plexapp-android] Starting a transcode session dcf5334784e7f1fa-com-plexapp-android at offset -1.0 (state=3)
Feb 13, 2023 18:19:56.219 [0x1467e00fab38] DEBUG - [Req#eb5/Transcode/dcf5334784e7f1fa-com-plexapp-android] TPU: hardware transcoding: using hardware decode accelerator nvdec
Feb 13, 2023 18:19:56.219 [0x1467e00fab38] DEBUG - [Req#eb5/Transcode/dcf5334784e7f1fa-com-plexapp-android] TPU: hardware transcoding: zero-copy support present
Feb 13, 2023 18:19:56.219 [0x1467e00fab38] DEBUG - [Req#eb5/Transcode/dcf5334784e7f1fa-com-plexapp-android] TPU: hardware transcoding: not using zero-copy because subtitle burning is required
Feb 13, 2023 18:19:56.220 [0x1467e00fab38] DEBUG - [Req#eb5/Transcode/dcf5334784e7f1fa-com-plexapp-android] [Universal] Using local file path instead of URL: /data/Movies/Brave (2012)/Brave (2012) - Remux-1080p.mkv
Feb 13, 2023 18:19:56.221 [0x1467e00fab38] DEBUG - [Req#eb5/Transcode/dcf5334784e7f1fa-com-plexapp-android] TPU: hardware transcoding: final decoder: nvdec, final encoder: nvenc
It’s working for you. The issue here is related to the content you’re trying to play.
Remove/Replace the Image-based subtitles (PGS,VOBSUB, DVDRIP) and/or turn off subtitle burning unless needed.
@ChuckPa Did you see my post about the difference with server & consumer class equipment ?
I can have Plex & transcodes on Nvidia working perfectly on any consumer equipment. Impossible to break transcodes. Move the SSD & GPU from the consumer grade equipment into any server grade equipment. And in the server, it will have incredibly inconsistent behaviour. The most obvious difference is when trying to change the quality in Chrome. It will work 100% of the time in consumer equipment, and fails 100% of the time in server equipment.
While for example transcoding & changing quality in Nvidia shield, works 100% on both.
I have a garage full of old servers. Mostly stuff I got for free over the years, because it was going to be thrown away. So far I have not found a single server where PMS & NVIDIA transcodes work normal. The same problem happens on all of them. It’s a night & day difference, and really obvious now.
This happens so far the exact same way, with ANY combination of Debian/Ubuntu, Nvidia drivers & PMS that I have tried.
Is this a known issue ? Do you want to see the logs from any of the 5 setups I documented ? Which ones ? During startup, playback or attempted transcodes ?
I can also do clean install of any version of Ubuntu on any server, and give you SSH access. You will be able to reproduce it, no matter what versions of Nvidia & PMS you install.
