High CPU usage causing buffering when transcoding compared to ffmpeg

[csantiago78]

Server Version#: 1.25.0.5246 (docker linuxserver/plex:latest image)
Player Version#: 4.66.1 (web)

System’s Hardware specs:

CPU: 8x Intel(R) Core™ i7-9700 CPU @ 3.00GHz (8th Gen Core 8-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S])
Chipset: Intel Z390
GPU: UHD Graphics 630 (Desktop 9 Series) Intel Corporation
M2 SSD (for transcoding cache): XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive - ADATA Technology Co., Ltd.
RAM: Dual Channel 2x8gb DDR4 modules (2666) (16gb total)

I have both Emby and Plex docker images on a headless Ubuntu server. I have noticed that whenever I am trying to use plex and the player needs transcoding it stutters.

After some investigation and attempting to tweak the transcoding options, I noticed that the CPU usage was off the roof. Granted that it is transcoding a 4k HDR source, but it used to work with no issues. I have always supported both Plex and Emby projects (Plex Pass (lifetime) and Emby Premiere subscriber for years now). So I started testing the same file on both systems and the results show that Plex’s Transcoder is using at least twice the CPU that Emby’s ffmpeg is.

I share my Plex library with my family and I would love to be able to keep doing so without upgrading my hardware.

Here’s the screenshots comparing the same movie being played:
Plex:

ffmpeg:

Also attaching the verbose logs. If you need me to I could attach the transcoding config settings for both. Please don’t hesitate to ask for further details/files.

Thanks!!

Best,

Cristian
Plex Media Server Logs_2021-11-25_16-52-28.zip (3.2 MB)

[Sillyvainplex]

Hi Csantiago,

You have a nice setup. I also have the same gig as you sharing it with families and friends. I have over 20 plex servers running together so I don’t run the same issue has you. it distribute users to multiple servers so the CPU doesn’t go at 100%. I used to have 4K content in the past and this is why I don’t anymore. Its too much problems and if you want to be able to share your 4K content you need a internet upload of at least 300mb + Also it all depends of the devices your user has (Roku, Firestick. Chromecast, Nvidia Shield… etc. You can have one user streaming a content with 4K without any issues and then more users gets on your server and then the CPU is at 100%. What I should suggest is to get 2 servers. One for 4K content and that you know your users have a decent internet speed and a good streaming devices the other one for just 1080p or 720p or SD… When I re-encode a Blu-ray with handbrake I make it so when a user streams from me its direct stream for Video and Audio and the CPU is flatline at 0% Its the only way to go…

[csantiago78]

Thanks for your suggestion Sillyvainplex!

But I don’t actually want to spend more money in a new server, I was just pointing out that there’s a lot of room for improvement performance wise, as if the other project was able to make it work, then it is possible.

I am not actually looking for a solution where I have to change my hardware specs, I like plex’s interface better and there is a larger support amongst smart tvs and players for Plex than for Emby, but if I have to have my family use the alternative, so be it.

If you look at the numbers, for a huge bitrate 4k UHD Blu-Ray rip (REMUX)(~65mbps), ffmpeg’s process can do around 75fps transcoding using less than 65% of the CPU power. So it could literally pull at least 3(probably 4) simultaneous transcoding streams of such a huge bitrate without any stuttering.

Anyway, hopefully the Plex team can take a look at this and check if they can find a way to improve the perf.

Cheers!

[ChuckPa]

I cannot reproduce. I do see the OpenCL running (resulting in the wait time)

Here are 4 movies, playing simultaneously with HDR → SDR tonemapping to Chrome and audio conversion.

I start to run out of QSV with the 6th transcode.

There is a bigger load (411% total) on my desktop NUC than on the NUC server which runs it (both are NUC8-i7-HVK

[csantiago78]

Thanks for trying to reproduce this ChuckPa!

This is the actual quality of the movie Direct Playing to my nVidia Shield Pro Android TV works with no issues at all:

This is playing here locally on a Mac on a Firefox Browser. (that and the PS5’s app is where I am seeing the issues when transcoding)

Files

Delete Files
/movies/A Hard Days Night (1964)/A.Hard.Days.Night.1964.UHD.BluRay.2160p.DTS-HD.MA.5.1.SDR.HEVC.HYBRID.REMUX-FraMeSToR.mkv

Media

Duration 1:27:38
Bitrate 65900 kbps
Width 3840
Height 2160
Aspect Ratio 1.78
Video Resolution 4K
Container MKV
Video Frame Rate 24p
Audio Profile ma
Video Profile main 10

Part

Duration 1:27:38
File A.Hard.Days.Night.1964.UHD.BluRay.2160p.DTS-HD.MA.5.1.SDR.HEVC.HYBRID.REMUX-FraMeSToR.mkv
Size 40.34 GB
Audio Profile ma
Container MKV
Video Profile main 10

Codec HEVC
Bitrate 64172 kbps
Bit Depth 10
Chroma Location left
Chroma Subsampling 4:2:0
Coded Height 2160
Coded Width 3840
Color Primaries bt709
Color Range tv
Color Space bt709
Color Trc bt709
Frame Rate 23.976 fps
Height 2160
Level 5.1
Profile main 10
Ref Frames 1
Width 3840
Display Title 4K (HEVC Main 10)
Extended Display Title 4K (HEVC Main 10)

Codec DCA
Channels 6
Bitrate 1536 kbps
Language English
Language Tag en
Audio Channel Layout 5.1(side)
Bit Depth 24
Profile ma
Sampling Rate 48000 Hz
Title DTS-HD MA 5.1
Display Title English (DTS-HD MA 5.1)
Extended Display Title DTS-HD MA 5.1 (English)

Codec FLAC
Channels 2
Bitrate 1304 kbps
Language English
Language Tag en
Audio Channel Layout stereo
Bit Depth 24
Sampling Rate 48000 Hz
Title FLAC 2.0
Display Title English (FLAC Stereo)
Extended Display Title FLAC 2.0 (English)

Codec FLAC
Channels 1
Bitrate 695 kbps
Language English
Language Tag en
Audio Channel Layout mono
Bit Depth 24
Sampling Rate 48000 Hz
Title FLAC 1.0
Display Title English (FLAC Mono)
Extended Display Title FLAC 1.0 (English)

Codec AC3
Channels 1
Bitrate 192 kbps
Language English
Language Tag en
Audio Channel Layout mono
Sampling Rate 48000 Hz
Title Commentary by cast and crew
Display Title English (AC3 Mono)
Extended Display Title Commentary by cast and crew (English AC3 Mono)

Codec PGS
Bitrate 68 kbps
Language English
Language Tag en
Title English (SDH)
Display Title English (PGS)
Extended Display Title English (SDH) (PGS)

Codec PGS
Bitrate 28 kbps
Language German
Language Tag de
Title German
Display Title German (PGS)
Extended Display Title German (German PGS)

Codec PGS
Bitrate 37 kbps
Language Japanese
Language Tag ja
Title Japanese
Display Title Japanese (PGS)
Extended Display Title Japanese (Japanese PGS)

Codec PGS
Bitrate 53 kbps
Language Japanese
Language Tag ja
Title Japanese (Commentary)
Display Title Japanese (PGS)
Extended Display Title Japanese (Commentary) (Japanese PGS)

Codec SRT
Language English
Language Tag en
Format SRT
Display Title English (SRT External)
Extended Display Title English (SRT External)

Codec SRT
Language Spanish
Language Tag es
Format SRT
Display Title Spanish (SRT External)
Extended Display Title Spanish (SRT External)

If you need me to I could upload a couple of these high bitrate files, I have 200mbps fiber so I could have a couple of them uploaded in a relatively short time if you can provide an sftp or some other sort of repo.

Also notice that the CPU is an i7 9700, so it is not bleeding edge, but the integrated 630 gpu should have plenty of resources to pull this off pretty well.

Please let me know your thoughts. Thanks!

[ChuckPa]

Look at your Plex/Web playback setting !!!

720p - 6 Mbps ? (deliberately converting HEVC HDR → 1/9th the original quality ? )

Let me spin up here and make a comparative test.

[csantiago78]

Sorry those were the automatically adjust quality that kicked in.

I have made an interesting test:

If I force the same quality on the shield it doesn’t drop frames and only uses like a much smaller fraction of the resources:

[csantiago78]

Even if I use the alternate streaming protocol on the web player’s settings I get that:

[csantiago78]

Same on the PS5:

[ChuckPa]

Which shield pro app version are you using? 8.19.4 ? (yes, that’s back a few versions)

[ChuckPa]

When I force the transcoder to enage:

it will run tone mapping because it’s flagged HEVC HDR → H.264 (SDR)

You do have Intel Compute Runtime installed ?

As for movie video quality - It is what it is. It’s a Super8 master which was digitized up.

I remember watching Super 8 movies and this is exactly how they looked when they were new LOL

[ChuckPa]

[csantiago78]

I don’t mind the tone mapping(as you say it adds to the original experience lol) or the Super8 quality.
I am running the docker container of the PMS with the following params(including the intel hw accell /dev/dri):

docker run -d --name plex -h ‘plexms’ --device=/dev/dri:/dev/dri -p 32400:32400 -p 1901:1900/udp -p 3005:3005 -p 5354:5353/udp -p 8324:8324 -p 32410:32410/udp -p 32412:32412/udp -p 32413:32413/udp -p 32414:32414/udp -p 32469:32469 -e TZ=“America/Argentina/Buenos_Aires” -e VERSION=latest -e “PUID=0” -e “PGID=0” -e PLEX_CLAIM=“claim-TVxSbBLbsK-SsJ8a4N3L” -v /mnt/cont_config/configs/plex:/config -v /mnt/cont_config/certs:/config/certs -v /mnt/transcodecache:/transcode -v /mnt/storage/media/tv:/tv -v /mnt/storage/media/movies:/movies -v /mnt/storage/videos:/videos --restart unless-stopped linuxserver/plex:latest

But what I meant is that on the shield app(v8.26.2.29389) it works fine when forcing the transcode, no perf issues.

But on the other hand on the PS5’s app (1.003.000) and on the Web Player (4.66.1 running on firefox) the performance issues hit pretty hard.

[ChuckPa]

This is interesting.

I have Plex/web 4.54.5 bundled (I don’t run current server)

now, if I run 4.70.0 ( app.plex.tv ) I get better

My system , because it’s an i7-8xxx is using Beignet and OpenCL.

Your machine should be using Intel Compute Runtime and OpenCL.

May I see the DEBUG logs from starting A Hard Day’s Night? (play for 30 seconds then stop then download logs)

Either way, I’m not getting the multiple streams you show — which is pointing directly at software decode / software tonemapping.

[csantiago78]

Here’s what I have from the player’s logs:

[
  {
    "type": "info",
    "host": "csantiago78.me:22880",
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:95.0) Gecko/20100101 Firefox/95.0",
    "browser": {
      "platformVersion": "10.15",
      "id": "firefox",
      "name": "Firefox",
      "version": "95.0",
      "gecko": true,
      "platform": "OSX",
      "platformID": "osx",
      "platformModifierKey": "command"
    },
    "version": "4.66.1",
    "username": "csantiago78",
    "cloudUrl": [
      {
        "scheme": "https",
        "address": "plex.tv",
        "uri": "https://plex.tv",
        "testState": "untested",
        "isBundled": false,
        "isFallback": false,
        "relay": false,
        "isUntested": true,
        "isPending": false,
        "isConnected": false,
        "isUnauthorized": false,
        "isUnavailable": false,
        "isFailed": false,
        "isAborted": false,
        "isLoopback": false,
        "isPrivate": false,
        "isHttps": true,
        "isSecure": true,
        "isPlexDirect": false,
        "sources": [
          {
            "id": "internal"
          }
        ]
      }
    ]
  }
]

And attached are the server logs after I enabled the debug logs on the player:

Let me know if I missed anything.
Plex Media Server Logs_2021-11-27_00-53-11.zip (3.9 MB)

[ChuckPa]

Thank you.

I see why the difference between your server and mine.

You’re burning subtitles.

Nov 26, 2021 22:03:47.290 [0x7f6fb0c07b38] DEBUG - [Transcode] [FFMPEG] - Slice 0 param buffer (264 bytes) is 0x1.
Nov 26, 2021 22:03:47.290 [0x7f6fb0c07b38] DEBUG - [Transcode] [FFMPEG] - Slice 0 data buffer (47 bytes) is 0x2.
Nov 26, 2021 22:03:47.290 [0x7f6fb0c07b38] DEBUG - [Transcode] [FFMPEG] - Decode to surface 0x13.
Nov 26, 2021 22:03:47.290 [0x7f6fb0c07b38] DEBUG - [Transcode] Codecs: 10-bit HEVC test succeeded
Nov 26, 2021 22:03:47.291 [0x7f6fb0c07b38] DEBUG - [Transcode] Scaled up video bitrate to 288774Kbps based on 4.500000x fudge factor.
Nov 26, 2021 22:03:47.291 [0x7f6fb0c07b38] DEBUG - [Transcode] Scaled maximum bitrate for resolution reduction to 101427Kbps.
Nov 26, 2021 22:03:47.291 [0x7f6fb0c07b38] DEBUG - [Transcode] MDE: Selected protocol dash; container: mp4
Nov 26, 2021 22:03:47.291 [0x7f6fb0c07b38] DEBUG - [Transcode] MDE: analyzing media item 50
Nov 26, 2021 22:03:47.291 [0x7f6fb0c07b38] DEBUG - [Transcode] MDE: A Hard Day's Night (1964): Direct Play is disabled
Nov 26, 2021 22:03:47.291 [0x7f6fb0c07b38] DEBUG - [Transcode] MDE: A Hard Day's Night (1964): media must be transcoded in order to use the dash protocol
Nov 26, 2021 22:03:47.291 [0x7f6fb0c07b38] DEBUG - [Transcode] MDE: A Hard Day's Night (1964): selected subtitle cannot be converted to a compatible format, burning into video stream
Nov 26, 2021 22:03:47.291 [0x7f6fb0c07b38] DEBUG - [Transcode] MDE: A Hard Day's Night (1964): avoiding video remux due to burned subtitle stream
Nov 26, 2021 22:03:47.291 [0x7f6fb0c07b38] DEBUG - [Transcode] MDE: A Hard Day's Night (1964): no remuxable profile found, so video stream will be transcoded
Nov 26, 2021 22:03:47.291 [0x7f6fb0c07b38] DEBUG - [Transcode] MDE: Cannot direct stream video stream due to profile or setting limitations
Nov 26, 2021 22:03:47.291 [0x7f6fb0c07b38] DEBUG - [Transcode] Codecs: testing hevc (decoder) with hwdevice vaapi
Nov 26, 2021 22:03:47.291 [0x7f6fb0c07b38] DEBUG - [Transcode] Codecs: hardware transcoding: testing API vaapi
Nov 26, 2021 22:03:47.291 [0x7f6fb0c07b38] DEBUG - [Transcode] [FFMPEG] - Format 0x41524742 -> bgra.

Nov 26, 2021 22:03:47.490 [0x7f6fb1bccb38] DEBUG - [Transcode] [FFMPEG] - Direct mapping possible.
Nov 26, 2021 22:03:47.491 [0x7f6fb1bccb38] DEBUG - [Transcode] TPU: hardware transcoding: final decoder: vaapi, final encoder: vaapi
Nov 26, 2021 22:03:47.491 [0x7f6fb1bccb38] DEBUG - [Transcode/JobRunner] Job running: FFMPEG_EXTERNAL_LIBS='/config/Library/Application\ Support/Plex\ Media\ Server/Codecs/ea7e8c9-4118-linux-x86_64/' LIBVA_DRIVERS_PATH='/usr/lib/plexmediaserver/lib/dri' X_PLEX_TOKEN='xxxxxxxxxxxxxxxxxxxx' '/usr/lib/plexmediaserver/Plex Transcoder' '-codec:0' 'hevc' '-hwaccel:0' 'vaapi' '-hwaccel_fallback_threshold:0' '10' '-hwaccel_device:0' 'vaapi' '-codec:1' 'dca' '-analyzeduration' '20000000' '-probesize' '20000000' '-i' '/movies/A Hard Days Night (1964)/A.Hard.Days.Night.1964.UHD.BluRay.2160p.DTS-HD.MA.5.1.SDR.HEVC.HYBRID.REMUX-FraMeSToR.mkv' '-filter_complex' '[0:5]scale=3840:2160[0];[0:0][0]overlay[1];[1]scale=w=2276:h=1280:force_divisible_by=4[2];[2]format=pix_fmts=nv12[3];[3]hwupload[4]' '-map' '[4]' '-codec:0' 'h264_vaapi' '-b:0' '76070k' '-maxrate:0' '101427k' '-bufsize:0' '202854k' '-r:0' '23.975999999999999' '-force_key_frames:0' 'expr:gte(t,n_forced*3)' '-filter_complex' '[0:1] aresample=async=1:ocl='\''stereo'\'':rematrix_maxval=0.000000dB:osr=48000[5]' '-map' '[5]' '-metadata:s:1' 'language=eng' '-codec:1' 'aac' '-b:1' '256k' '-f' 'dash' '-seg_duration' '3' '-dash_segment_type' 'mp4' '-init_seg_name' 'init-stream$RepresentationID$.m4s' '-media_seg_name' 'chunk-stream$RepresentationID$-$Number%05d$.m4s' '-window_size' '5' '-delete_removed' 'false' '-skip_to_segment' '1' '-time_delta' '0.0625' '-manifest_name' 'http://127.0.0.1:32400/video/:/transcode/session/h7hgzy7p9h6ov52rg2zooovm/e1e97b3c-49d7-4ba2-b91d-761104ddc798/manifest?X-Plex-Http-Pipeline=infinite' '-avoid_negative_ts' 'disabled' '-map_metadata' '-1' '-map_chapters' '-1' 'dash' '-start_at_zero' '-copyts' '-vsync' 'cfr' '-init_hw_device' 'vaapi=vaapi:' '-filter_hw_device' 'vaapi' '-y' '-nostats' '-loglevel' 'quiet' '-loglevel_plex' 'error' '-progressurl' 'http://127.0.0.1:32400/video/:/transcode/session/h7hgzy7p9h6ov52rg2zooovm/e1e97b3c-49d7-4ba2-b91d-761104ddc798/progress'
Nov 26, 2021 22:03:47.491 [0x7f6fb1bccb38] DEBUG - [Transcode/JobRunner] Jobs: Starting child process with pid 84514

At this bitrate, Decode to raw → burn subtitles → tonemap → Encode is going to eat up a great deal of CPU to put the subtitles in.

I went and grabbed the same file you have. Because there are no subtitles involved, I don’t use a fraction of the CPU you do.

I believe the answer here is “Media Curation”. You’re using PGS subtitles. ANY image based subtitle will cause this. ( &codec=hdmv_pgs_subtitle& ) (PGS, VOBSUB, or DVDRIP – all image based)

If you use text based subtitles, the subtitles can be sent as a 3rd stream. This allows the QSV hardware to transcode from the HEVC source → H.264 target in one hardware operation without CPU involvement.

I recommend using SRT subtitles wherever possible. They are the most commonly accepted and most easily converted to other text formats.

[csantiago78]

Wow, amazing job! Worked like a charm! Thanks!

The subs were even already downloaded as I have Sonarr and Radarr for this. It would be a good feature to make SRT subs default even if the container has a PGS sub particularly when transcoding is needed. Also another thing for subs, I know PMS has it’s DB and particular location for the subs, but it would be great to have the option to store the subs at the same folder than the media file. I believe this way it would be more efficient and easier to troubleshoot any subtitles’s issues.

Once again thank you very much for the thorough investigation!

Cheers!