Color corruption on resize when HW Transcoding HEVC content to h264, + 4K subs - Synology DS918+ NAS

Plex Media Server NAS & Devices
1

I’m using a Synology DS918+ NAS (Intel Celeron J3455 with QuickSync)
I’ve a number of videos in HEVC format - and am using the hardware transcode functionality to convert these to h264 as required.
If the video is resized as part of the transcode, I experience color corruption in the form vertical stripes - the intensity seems correct, but the red and blue coloring seems to seperate. Each stripe looks to be about 16 pixels - and is most noticable in red areas.
This only occurs if the video is resized - eg 1080p->720p. A straight 1080p->1080p transcode does not have this problem.
It also only seems to occur when transcoding h265 to h264, I’ve not seen it in h264->h264 transcodes with resizes.

I’ve also a problem with 4K videos and subtitles.
Without subtitles - the NAS will happily transcode HEVC 4K to 1080p. With subtitles, although the CPU still reads only 10% (again - this is with hardware transcoding) - I get an error warning of a slow server, and the video will not play in real time.

Other than these issues - really impressed with Plex, I hope they can be corrected!

1 Like
2

Forgot to mention - currently using Version 1.11.1.4730 (64bit) - but this issue has been present in all versions since I started using Plex a month ago.

3

I experience the exact same issue on my Synology DS218+ (Intel Celeron J3355, Apollo Lake iGPU).
Vertical bars are appearing, and is very annoying and degrading the global quality.
Here is an illustration of the problem :

The source file is an 1080p HEVC 10 bit movie. This occurs for example when I select 720p, 4 Mbps or any lower quality. If I select 1080p 8 Mbps, there is absolutely no problem, the image is clear even though it gets transcoded.

This issue is particularly annoying to me since I only have a 5 Mbps upload speed, so basically all my videos get transcoded to 720p 4 Mbps when I try to watch remotely.

I hope this gets fixed soon, but this issue is definitely still present with latest 1.13.0.5003 PMS (x64) version for Synology.

4

See here : https://forums.plex.tv/discussion/306889/color-corruption-when-resizing-an-hevc-hardware-transcode-on-synology-ds918#latest

5

Sounds like your hardware is not capable of handling the 10-bit color in the source. Do you see the same with 8-bit HEVC video?

6

I had a similar issue with HEVC 10-bit content showing up discolored when played on my Synology device.

7

The problem doesn’t occur with 8bit video, but the hardware does support hardware decode of 10bit HEVC, (though admitedly only 8bit, not 10bit hardware encode of HEVC). Since Plex currently only encodes to 8bit h264 anyway, this isn’t a limitation of the hardware. As mentioned, there is no problem unless a resize is performed. The corruption appears to be from the resize getting the memory layout of the YUV format wrong, smearing the U and V components out of place.

It’s worth mentioning that Emby media server manages to resize and transcode the same files on the same hardware without this corruption. Plex and Emby both use FFMPEG under the hood from what I gather, so the problem is almost certainly with the wrong colour format being specified to the resize filter.

Also interesting is that forcing Plex to bake in subtitles also fixes the problem (though you now have subtitles baked in… not really a fix…), presumably because the colour space is correctly converted before the subtitle filter is applied (I guess this is done in RGB???)

8

I decided to get to the bottom of this as it’s been irritating me for months.

From the logs - FFMPEG, (or ‘Plex Transcoder’) is being kicked off with the following command:

'/volume1/@appstore/Plex Media Server/Plex Transcoder' 
'-codec:0' 'hevc' 
'-hwaccel:0' 'vaapi' 
'-hwaccel_fallback_threshold:0' '10' 
'-hwaccel_output_format:0' 'vaapi' 
'-codec:1' 'aac' 
'-ss' '1720' 
'-i' '/10BIT-HEVC-VIDEO.mkv' 
'-filter_complex' '
    [0:0]hwupload[0];
    [0]scale_vaapi=w=1278:h=538:format=nv12[1];
    [1]hwupload[2]' 
'-filter_complex' '
    [0:1] aresample=async=1:ocl='\''stereo'\'':osr=48000[3]' 
'-map' '[2]' 
'-codec:0' 'h264_vaapi' 
'-b:0' '2717k' 
'-maxrate:0' '3623k' 
'-bufsize:0' '7246k' 
'-r:0' '23.975999999999999' 
'-force_key_frames:0' 'expr:gte(t,1720+n_forced*5)' 
'-map' '[3]' 
'-codec:1' 'aac' 
'-b:1' '148k' 
'-f' 'dash' 
'-min_seg_duration' '5000000' 
'-skip_to_segment' '345' 
'-time_delta' '0.0625' 
'-manifest_name' 'http://127.0.0.1:32400/video/:/transcode/session/1o4wm49ikl2is56e87zvlwq5/4c54f2cc-9edb-46a6-9107-898a944a77e9/manifest' 
'-avoid_negative_ts' 'disabled' 
'-map_metadata' '-1' 
'-map_chapters' '-1' 
'dash' 
'-start_at_zero' 
'-copyts' 
'-y' 
'-vaapi_device' '/dev/dri/renderD128' 
'-nostats' 
'-loglevel' 'quiet' 
'-loglevel_plex' 'error' 
'-progressurl' 'http://127.0.0.1:32400/video/:/transcode/session/1o4wm49ikl2is56e87zvlwq5/4c54f2cc-9edb-46a6-9107-898a944a77e9/progress'

The interesting bit is probably:

'-filter_complex' '
    [0:0]hwupload[0];
    [0]scale_vaapi=w=1278:h=538:format=nv12[1];
    [1]hwupload[2]'

This is basically taking the frame - making sure it’s in GPU memory, using a hardware scale to resize and convert the format to nv12, and then making sure this frame is in GPU memory.

To fix it - I guess either remove the :format=nv12 (as the buffer is staying in hardware for the encode - I’d have thought it can remain a vaapi surface?)

'-filter_complex' '
    [0:0]hwupload[0];
    [0]scale_vaapi=w=1278:h=538[1];
    [1]hwupload[2]'

Or convert to nv12 in a seperate step, as the scale_vaapi seems to be messing it up:

'-filter_complex' '
    [0:0]hwupload[0];
    [0]scale_vaapi=w=1278:h=538[1];
    [1]format=nv12[2]'

I’ve tried running the command directly on my DS918+ as the plex user, but for some reason it whines about being unable to connect to the vaapi device:

[AVHWDeviceContext @ 0x22c9d80] Failed to initialise VAAPI connection: -1 (unknown libva error).
Device creation failed: -5.
Failed to set value '/dev/dri/renderD128' for option 'vaapi_device': Input/output error

If one of the developers reads this and has an idea what’s wrong, I’d love to know.

9

Please post a sample file that reproduces the issue.

10

Sure - all the Jellyfish Bitrate Test Files encoded with the Main10 profile at http://jell.yfish.us/ have this problem, it’s more visible with reds - (18 seconds in). I’ve taken and attached screen grabs from http://jell.yfish.us/media/jellyfish-3-mbps-hd-hevc-10bit.mkv

Screen grab 18 seconds in from web client, (transcoded to 4Mb h264 with resize to 720p):

Screen grab 18 seconds in from web client, (transcoded to h264 with no resize):

11

Thanks for the sample and screenshots. I’m not reproducing this here, but I’m on a Kaby Lake processor, so it’s possible that this issue is processor-generation-specific. Could you test a few things for me:

  • Update to PMS version 1.13.0
  • Post the CPU model name from /proc/cpuinfo
  • Run a transcode with these simplified args (and the same env vars as a regular one):
[/path/to/]Plex\ Transcoder -hwaccel vaapi -ss 17 -i jellyfish-3-mbps-hd-hevc-10bit.mkv '-vaapi_device' '/dev/dri/renderD128' '-filter_complex' '[0:0]hwupload,scale_vaapi=w=1278:h=538:format=nv12,hwupload[0]' -map '[0]' -t 3 '-codec:0' 'h264_vaapi'  '-b:0' '2717k' jellyfish-out2.mp4

And see if the resulting file has the chroma issue.

This looks to me as if either the scaler and encoder are miscommunicating regarding the plane format of the video, or the scaler is converting to the wrong format. The formats involved are NV12 (where the U and V components are interleaved in a single plane) and YUV420P (where the U and V components are in separate planes).

12

No problem - thanks for taking a look. The CPU is from the Apollo Lake series I believe, cpu info gives:

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 92
model name      : Intel(R) Celeron(R) CPU J3455 @ 1.50GHz
stepping        : 9
microcode       : 0x2c
cpu MHz         : 1501.000
cache size      : 1024 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 21
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx est tm2 ssse3 sdbg cx16 xtpr pdcm sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave rdrand lahf_lm 3dnowprefetch ida arat pln pts dtherm intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust smep erms mpx rdseed smap clflushopt sha_ni xsaveopt xsavec xgetbv1
bugs            :
bogomips        : 2995.03
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

(x4…)

Unfortunately I’m having trouble kicking off the transcoder manually - and keep getting the following error:

[AVHWDeviceContext @ 0x336fc80] Failed to initialise VAAPI connection: -1 (unknown libva error).
Device creation failed: -5.
Failed to set value '/dev/dri/renderD128' for option 'vaapi_device': Input/output error

I’m guessing there’s an .so file not being found somewhere - I’ll dig into it and see.

[EDIT- Forgot to mention, this is using 1.13.0.5003]

I agree this is likely a miscommunication of surface formats somewhere in the pipeline. I’m trying to get my head round what the problem is though, because 10bit HEVC comes out in YUV420p10le - which is a plane of 2byte 10bit values for intensity, then a 1/4 resolution 2 byte 10bit U plane and a 1/4 resolution 2 byte 10bit V plane. YUV420p 8bit video only uses 1 byte for each component - so a mismatch would explain what looks to be happening with the UV, but the Y (intensity) - is correct (that is, the image is not being stretched weirdly over two frames). Odd.

13

Oh, sorry! You’ll want to set LIBVA_DRIVERS_PATH=[/path/to/PMS/install]/dri.

14

Aha - yes, that got the transcode working - thanks :smile:

The output had the same colour problem though.

Changing scale_vaapi=w=1278:h=538:format=nv12 argument to just scale_vaapi=w=1278:h=538 fixed the colour problem.

Any chance this change could be made in a new release? I guess in the meantime it might be possible to create a proxy script to make this change on the fly.

15

Sorry, the conversion to NV12 is deliberate (on some hardware, it’s required); I’m hoping we can work out what’s causing the format mismatch within ffmpeg (or the VA driver) and get it resolved there. I’ll see if anybody here has an Apollo Lake system we can investigate further on, and keep you posted if I need any more information. Thanks for the help!

16

No problem at all - glad the information’s been useful, and hope you manage to get to the bottom of it.

Until the issue can be fixed properly, I’ve slightly hacked my install of Plex to work around the issue on my Synology DS918+.

In case anyone else with this problem is interested, I’ve renamed Plex Transcoder in the Plex installation directory (on my box this was /volume1/@appstore/Plex Media Server) to Plex_Transcoder - and saved the following script in its place (and made it executable).

#!/bin/sh
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
declare -a new_args
forbidden_text=":format=nv12"

for param in "$@"; do
  new_args+=("${param//$forbidden_text/}")
done
"${DIR}/Plex_Transcoder" "${new_args[@]}"

Plex now hardware transcodes with resize from 10bit HEVC without corrupting the colour - and the world is good :slight_smile:

17

Excellent work guys ! I was fearing this was a hardware fault, but happy to see you’re putting the fingers on a solution.

Looking forward to a definitive PMS update to fix it all :smile:

I will try your workaround @jaB00M , thanks a lot for the trick

EDIT : confirmed, your script is working great, no more color corruption on HEVC 10 bit resized transcodes ! :blush:

18

This turns out to be a bug in the Intel driver when running on Apollo Lake hardware. You can follow the issue thread here; we’ll update the driver in PMS once it’s fixed: https://github.com/intel/intel-vaapi-driver/issues/398

20

Hello,

Has this been resolved? I can still see this problem manifest in these scenarios.

Tnx for the update.

1 Like
21

Nope. I still apply @jaB00M’s fix after every update.

1 Like