Transcoding 4K streams on linux

[wjw]

Server Version#: 1.29.0.6244
Player Version#: 1.29.0.6244

All the discussions around this seem quite old. I’ve been running plex in a VM on an Optiplex 7050 running a Core i7-7700 (QC/8MB/8T/ 3.6GHz/65W) for about 4 years. Overtime I’ve had to make changes for it to keep up, like using a ramdisk for transcoding and moving the entire VM to SSD.

Now my question since 4K came along it can’t keep up with transcoding. I’ve rebuilt it so that its native on the same hardware but its still not keeping up.

I’ve checked out the logs and the speed will be fine for a little while (between 1.1 and 1.3) then it will drop back to 0.0 for 500ms or so. This is causing the playback to pause. I’ve increased the number of seconds in the transcode buffer but i doesnt seem to have helped.

The dashboard shows the CPU load running at about 63%, top showsing between 500-550%

load average: 5.45, 6.07, 5.76

I dont have hardware accelaration.

I’m looking for a second opinion before I start looking at new hardware. TIA!

[ChuckPa]

First, the obvious.

1. You don’t need new hardware.
2. I have a QNAP TVS-1282-i7-7700 and it’s flawless with everything running the native package.
3. The i7-7700 has all the hardware acceleration capability we need for 2160p UHD HDR → SDR tone mapping.

Perhaps you can make certain DEBUG logging is enabled and capture how it doesn’t keep up / fails and also include a screenshot of the activity page (upper right corner arrow)

[wjw]

Thats good news

With the log file, probably best to start from the bottom up.

Plex-Media-Server.log (2.5 MB)

[edit] Replying to myself, maybe I need to install the Intel GPU drivers, rather than using the built-in Ubuntu ones?

[wjw]

ok so, hardware accelaration isnt working. I’ve been googling and installed “intel-media-va-driver”.

intel_gpu_top shows no activity across all engines…

[ChuckPa]

@wjw

The log you provided shows it working.

Sep 29, 2022 20:41:12.240 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] Starting a transcode session bbs74kjyw4gih2h29bchfpi1 at offset -1.0 (state=3)
Sep 29, 2022 20:41:12.241 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] TPU: hardware transcoding: using hardware decode accelerator vaapi
Sep 29, 2022 20:41:12.241 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] TPU: hardware transcoding: zero-copy support present
Sep 29, 2022 20:41:12.241 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] TPU: hardware transcoding: using zero-copy transcoding
Sep 29, 2022 20:41:12.241 [0x7efbfc5cfb00] INFO - [Req#1bf/Transcode] CodecManager: starting EAE at "/tmp/pms-822f37f9-dc30-4b43-b7e4-3e0ddfd841e6/EasyAudioEncoder"
Sep 29, 2022 20:41:12.241 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode/JobRunner] Job running: "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Codecs/EasyAudioEncoder-1785-linux-x86_64/EasyAudioEncoder/EasyAudioEncoder"
Sep 29, 2022 20:41:12.241 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode/JobRunner] Jobs: Starting child process with pid 4465
Sep 29, 2022 20:41:12.241 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [Universal] Using local file path instead of URL: /media/nas12-media2/new/Luck.2022.HDR.2160p.WEB.h265-KOGi/Luck.2022.HDR.2160p.WEB.h265-KOGi.mkv
Sep 29, 2022 20:41:12.241 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] Codecs: hardware transcoding: testing API vaapi
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x41524742 -> bgra.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x42475241 -> argb.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x41424752 -> rgba.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x52474241 -> abgr.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x58524742 -> bgr0.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x42475258 -> 0rgb.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x58424752 -> rgb0.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x52474258 -> 0bgr.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x30335241 -> unknown.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x30334241 -> unknown.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x30335258 -> x2rgb10le.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x30334258 -> unknown.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x36314752 -> unknown.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x50424752 -> unknown.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x50524742 -> unknown.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x56555941 -> unknown.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x30303859 -> gray.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x3231564e -> nv12.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x3132564e -> unknown.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x32595559 -> yuyv422.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x59565955 -> uyvy422.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x32315659 -> yuv420p.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x30323449 -> yuv420p.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x50313134 -> yuv411p.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x48323234 -> yuv422p.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x56323234 -> yuv440p.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x50343434 -> yuv444p.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x33434d49 -> unknown.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Format 0x30313050 -> p010le.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Created surface 0.
Sep 29, 2022 20:41:12.242 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] [FFMPEG] - Direct mapping possible.
Sep 29, 2022 20:41:12.243 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] TPU: hardware transcoding: final decoder: vaapi, final encoder: vaapi
Sep 29, 2022 20:41:12.243 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode/JobRunner] Job running: EAE_ROOT=/tmp/pms-822f37f9-dc30-4b43-b7e4-3e0ddfd841e6/EasyAudioEncoder EnableExtendedVaFormats=1 FFMPEG_EXTERNAL_LIBS='/var/lib/plexmediaserver/Library/Application\ Support/Plex\ Media\ Server/Codecs/c4efa15-4369-linux-x86_64/' LIBVA_DRIVERS_PATH=/usr/lib/plexmediaserver/lib/dri NEOReadDebugKeys=1 X_PLEX_TOKEN=xxxxxxxxxxxxxxxxxxxx "/usr/lib/plexmediaserver/Plex Transcoder" -codec:0 hevc -hwaccel:0 vaapi -hwaccel_fallback_threshold:0 10 -hwaccel_output_format:0 vaapi -hwaccel_device:0 vaapi -codec:1 eac3_eae -eae_prefix:1 bbs74kjyw4gih2h29bchfpi1_ -analyzeduration 20000000 -probesize 20000000 -i /media/nas12-media2/new/Luck.2022.HDR.2160p.WEB.h265-KOGi/Luck.2022.HDR.2160p.WEB.h265-KOGi.mkv -filter_complex "[0:0]hwupload[0];[0]scale_vaapi=w=2368:h=1280:format=p010[1];[1]hwmap=derive_device=opencl[2];[2]tonemap_opencl=tonemap=mobius:format=nv12:m=bt709:p=bt709:r=tv[3];[3]hwmap=derive_device=vaapi:reverse=1[4];[4]hwupload[5]" -map "[5]" -metadata:s:0 language=eng -codec:0 h264_vaapi -b:0 20000k -r:0 23.975999999999999 -force_key_frames:0 "expr:gte(t,n_forced*3)" -filter_complex "[0:1] aresample=async=1:ocl='stereo':rematrix_maxval=0.000000dB:osr=48000[6]" -map "[6]" -metadata:s:1 language=eng -codec:1 aac -b:1 256k -f dash -seg_duration 3 -dash_segment_type mp4 -init_seg_name 'init-stream$RepresentationID$.m4s' -media_seg_name 'chunk-stream$RepresentationID$-$Number%05d$.m4s' -window_size 5 -delete_removed false -skip_to_segment 1 -time_delta 0.0625 -manifest_name "http://127.0.0.1:32400/video/:/transcode/session/bbs74kjyw4gih2h29bchfpi1/d7bf8245-75ef-49fa-a440-b8379c9c2f14/manifest?X-Plex-Http-Pipeline=infinite" -avoid_negative_ts disabled -map_metadata -1 -map_chapters -1 dash -start_at_zero -copyts -vsync cfr -init_hw_device vaapi=vaapi: -filter_hw_device vaapi -y -nostats -loglevel quiet -loglevel_plex error -progressurl http://127.0.0.1:32400/video/:/transcode/session/bbs74kjyw4gih2h29bchfpi1/d7bf8245-75ef-49fa-a440-b8379c9c2f14/progress
Sep 29, 2022 20:41:12.243 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode/JobRunner] Jobs: Starting child process with pid 4466

Sep 29, 2022 20:41:12.243 [0x7efbfc5cfb00] DEBUG - [Req#1bf/Transcode] TPU: hardware transcoding: final decoder: vaapi, final encoder: vaapi

Just to double check, Confirm the “Use hardware acceleration when available” option is set in Settings - Server - Transcoder - Show Advanced .

All indications in your logs say it is working.

[wjw]

@ChuckPa

Just to double check, Confirm the “Use hardware acceleration when available” option is set in Settings - Server - Transcoder - Show Advanced .

yes, thats is ticked.

[wjw]

Testing the speed to the NAS with a file copy, looks ok

I tried copying a 4k movie to the local SSD just in case and that had the same problem.

[ChuckPa]

I think I’m missing something here.

1. We can see where PMS is transcoding and tonemapping in hardware
2. You’re showing a 60% CPU load.

Here is an i7-7700 (default clock rate)

Transcoding Gemini Man (59.94 FPS 87 Mbps average)

You have an i7-7700T which is clocking unlocked.

Absolute silly question – Did the clock get turned down?

[wjw]

no, but your screenshot shows “Transcode (hw)”, mine shows “Transcode” that was what suggested to me that harware transcoding wasnt working, after reading this:

https://support.plex.tv/articles/115002178853-using-hardware-accelerated-streaming/

Intresting bandwidth graph too, yours is steady then spikes, mine completely drops off

[edit] Checked the bios, looks good, its an i7-7700T. I changed the Video from Auto to force the Intel HD graphics.

Still the same result

Made some more changes, Disabled Intel SpeedStep and made sure TurboBoost was enabled in the Bios and OS.

[ChuckPa]

Please confirm for me the contents of /dev/dri

PMS is telling me VAAPI so it’s there.

[~] # ls -la /dev/dri
total 0
drwxr-xr-x  2 admin administrators       80 2022-09-29 06:27 ./
drwxr-xr-x 17 admin administrators    21580 2022-09-29 18:46 ../
crw-------  1 admin administrators 226,   0 2022-09-29 06:27 card0
crw-------  1 admin administrators 226, 128 2022-09-29 06:27 renderD128
[~] # 

With PMS completely idle / stopped, are you still showing

load average: 5.45, 6.07, 5.76

and top of 500-600% on something ?
If so, what is it? What is it showing?

[wjw]

root@plex:/home/billw# ls -l /dev/dri
total 0
drwxr-xr-x 2 root root 80 Sep 29 23:26 by-path
crw-rw---- 1 root render 226, 0 Sep 29 23:26 card0
crw-rw---- 1 root render 226, 128 Sep 29 23:20 renderD128

If it wasn’t for plex updating metadata it would be idling

This is what top shows when playing:

[ChuckPa]

Ok… when you see 500% when playing, that’s everything being done in software.
When I test, I confirm the loads and that’s what I see.

Confirm this please.

[chuck@lizum ~.2001]$ groups plex
plex : plex video render
[chuck@lizum ~.2002]$ 

Also, cat /proc/cpuinfo | grep 'model name' | uniq

[wjw]

root@plex:/etc/default# groups plex
plex : plex video render

model name : Intel(R) Core™ i7-7700T CPU @ 2.90GHz

[wjw]

Replying to myself after quite a bit more reading and some more testing:

It doesnt seem to want to transcode the 4K codec in hardware

Working on the 1080 codec:

At last it proves that Plex can use the hardware and that side of it is working.

The CPU should support 4K HEVC 10:

[ChuckPa]

The cpu does support 2160p. I am started to get concerned about the motherboard.

I am now grasping at straws

1. Stop plex
2. Open a terminal window
3. cd '/var/lib/plemediaserver/Library/Application Support/Plex Media Server/Codecs'
4. Remove everything in there EXCEPT the .device_id file
5. Start Plex again
6. Retry your transcodes

What will happen: As each codec is needed again, it will download fresh. This will add a few extra seconds to the first-start time.

[wjw]

sigh ok done:

root@plex:/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Codecs# ls .*
.device-id

Same again

just found this…

My PLEX setup - Media players

which suggests it should work…

There is a two point (1.21 vs 1.19) bios revision available so I’m updating it.

[wjw]

Update done, no change, is it worth me trying the beta?

[ChuckPa]

Go for it. I’m starting to get ready to hand you a to put under the server box.

[wjw]

@ChuckPa

you a

PlexMediaServer install: PlexMediaServer-1.29.1.6260-420892357 - Installation starting.
PlexMediaServer install:
PlexMediaServer install: Now installing based on:
PlexMediaServer install: Installation Type: Update
PlexMediaServer install: Process Control: systemd
PlexMediaServer install: Plex User: plex
PlexMediaServer install: Plex Group: plex
PlexMediaServer install: Video Group: render
PlexMediaServer install: Metadata Dir: /var/lib/plexmediaserver/Library/Application Support
PlexMediaServer install: Temp Directory: /media/ramdisk (set in Preferences.xml)
PlexMediaServer install: Lang Encoding: en_US.UTF-8
PlexMediaServer install: Processor: Intel(R) Core™ i7-7700T CPU @ 2.90GHz
PlexMediaServer install: Intel i915 Hardware: Found
PlexMediaServer install: Nvidia GPU card: Not Found
PlexMediaServer install:
PlexMediaServer install: Completing final configuration.
PlexMediaServer install: Starting Plex Media Server.
PlexMediaServer install: PlexMediaServer-1.29.1.6260-420892357 - Installation successful. Errors: 0, Warnings: 0

[wjw]

That fixed it…