We’ll just got an email that my fix will no longer work as plex is dropping support for the older versions that use the old encoder. This sucks because I have 1300 movies and no idea which ones cause the issue…until I do. Which means the majority of viewers will have to hit me up as they try to watch something. GAH. Googling finds dozens of threads with this same issue. Can we get some sort of confirmation of the issue and if it’ll be addressed? I can add more logs or samples.
I could open a new topic but this is the same issue.
Here is a sample of a file that causes the green tint.
IF it is something like number of reference frames then fine, I’ll figure out a way to scan thousands of files and flag the ones with Ref frames too high…this feels different though…
I know the thought was a bad encoded video…but the linked came from HDBits which is renowned for encodes…so I doubt it’s quality of encode.
still using docker and docker-compose…old transcoder worked great…new transcoder displays the above “green” tint on some videos.
here is my docker compose file.
It’s kinda messy now as I’ve tried fiddling with what’s exposed to the docker image trying to resolve this issue…as of now, HW encoding IS working on the p2000, showing up in the nvidia-smi from the host as diaserver/Plex Transcoder.
Can I get some sort of response on this? it’s been a year and there are plenty of other posts out there with this issue…
TL;DR This issue could be completely fixed in Plex with an ffmpeg update or by pulling in this patch.
There is a bug in ffmpeg when using a high number of threads that affects some h264 bitstreams being decoded in hardware. It’s not clear to me what specifically causes it, but my current examples are all video files with a high ref frame count (16). One very bad, not good workaround on the current Plex build is to set -threads to something low like 1 which avoids this entirely.
Anyway, the result of the CUDA decode failure is that the video ends up falling back to being decoded on CPU. When the CPU decodes it, the pixel format that it outputs to the filter chain is now yuv420p, not NV12 as expected. Then the filter chain tries to run, fails (unclear why), and hands off yuv420p frames to nvenc which actually (mistakenly) thinks they are cuda/nv12 frames. Then it incorrectly processes them as nv12, which implies a U/V swap, and now you’ve got crazy green colors! Also, I’m pretty sure the scale never occurs and you get a bitstream of one size inside a container with another size as a side effect. Great.
That important bit where nvenc mistakenly processed the inbound frames as cuda, has been fixed in this patch in 2020. So, although there’s still some kind of issue where decode will fail over to CPU on certain h264 bitstreams (but only with high thread counts), the CUDA filter chain and NVENC should still work fine in those conditions – as soon as Plex pulls in that patch, or updates ffmpeg. See you in five years. The onerous mitigation for most users is to re-encode the “faulty” files, if you can ever identify them all.
Thanks for the INFO! this is at least directionally correct to resolve the issue and I truly appreciate you taking the time to respond.
So, for my clarity, when you say threads, are you referring to single threaded in the CPU or CUDA? where can you specify threads in Plex? or were you referring to Plex having to hard code the single thread into he transcode options it uses in the ffmpeg string…assuming the latter.
I’m wondering why going single threaded would resolved the issue. I mean I could force the single threaded…maybe by being wonky in linux…basically create a new Plex user that has a thread limit of 1 as I don’t think you can limit a single process and it’s more global in threads.
again, thank you!
With this information, is there anything you can do to get engineering to look into this bug? This appears to be a known issue and the “not sure what causes it” is worrisome because we don’t know what to fix or avoid…REF frames, Sample Rate, number of Subtitles…
thanks!
When I say running with -threads 1, I’m talking about hand-running Plex Transcoder (for testing) which is what Plex uses under the hood. It’s a custom build of ffmpeg. Limiting threads like that is going to severely affect performance and that’s why I said it’s a terrible idea. Also to do what you’re describing is a lot more work than just updating ffmpeg which fixes this bug entirely.
It sucks but until Plex pulls in that patch or updates ffmpeg entirely, you’re pretty much stuck re-encoding those files.
You may be able to build your own version of Plex Transcoder with the necessary updates, but Plex has always been hostile about following the ffmpeg licensing and only provides the source code with their own patches already baked in, as far as I know. It’s doable, but a pain.
It’s a gross workaround that I do not recommend for anyone, ever, but you could probably write a wrapper for Plex Transcoder that adds -threads 8 or similar to get “a decent amount of threads but not too many” to work around these issues for now. The number of threads that triggers the bug may vary by individual file though.
If you want a quick and dirty script to find affected movies, this worked for me. I had it generate 3 second test clips to visually confirm the movies were broken, but in my experience everything this matched was affected so you could remove the clip part.
#!/bin/bash
errors=$(FFMPEG_EXTERNAL_LIBS="/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Codecs/7c50c14-3596-linux-x86_64/" '/usr/lib/plexmediaserver/Plex Transcoder' -codec:0 h264 -hwaccel:0 nvdec -hwaccel_fallback_threshold:0 10 -hwaccel_output_format:0 cuda -hwaccel_device:0 cuda -i "$1" -filter_complex '[0:0]hwupload[0];[0]scale_cuda=w=480:h=262:format=nv12[1]' -map [1] -codec:0 h264_nvenc -b:0 416k -maxrate:0 555k -bufsize:0 1110k -an -sn -map_metadata -1 -map_chapters -1 -init_hw_device cuda=cuda: -filter_hw_device cuda -to 00:00:03 -f null - 2>&1 | grep "Parsed_scale_cuda" | grep "invalid argument" | wc -l)
if (($errors > 0)); then
echo "$1: Likely a green boi"
# generate test clip
clip_fn=$(basename -s .mkv "$1")_clip.mkv
echo "Clip FN $clip_fn"
FFMPEG_EXTERNAL_LIBS="/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Codecs/7c50c14-3596-linux-x86_64/" '/usr/lib/plexmediaserver/Plex Transcoder' -codec:0 h264 -hwaccel:0 nvdec -hwaccel_fallback_threshold:0 10 -hwaccel_output_format:0 cuda -hwaccel_device:0 cuda -ss 00:05:00 -to 00:05:10 -i "$1" -filter_complex '[0:0]hwupload[0];[0]scale_cuda=w=480:h=262:format=nv12[1]' -map [1] -codec:0 h264_nvenc -b:0 416k -maxrate:0 555k -bufsize:0 1110k -an -sn -map_metadata -1 -map_chapters -1 -init_hw_device cuda=cuda: -filter_hw_device cuda -v quiet -y "$clip_fn"
fi
Then you can run it like this:
nice ionice find /path/to/Movies -exec ./find_green_movies.sh '{}' \; | tee -a log.txt
This will make clips in the current directory with names of all affected files, and output some info to log.txt with the names.
In my case all affected movies had between 10-16 ref frames, but that doesn’t mean a whole ton. It’s probably multiple variables at play here.
Once again, all of these work just fine if you manually set -threads to a low number or if you just use an updated version of ffmpeg rather than Plex’s version. However, even on current ffmpeg, the hardware decode does fail on these files and falls back to CPU decode unless you manually set -threads; this should be fixed in ffmpeg. It’s the broken scaling issue that is the higher priority here since the rare movie with this issue can just be decoded on CPU instead.
1 Like
@honkdazzle You should get 10 internet points for this! Worked like a charm…found 165 of 1300+ Movies…I didn’t do the clip as I randomly ran through the list and they were all green.
Once the 165 were identified I wrote a script to copy them to a new DIR and automatically re-econde them…think it’s all solved for now…prolly cron this out to make sure Radarr doesn’t add new ones…and if it does just fix it straight away.
thanks again! Hopefully Plex pulls the fix down from ffmpeg at some point.
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.
