100 Percent System Load on Transcodes with Subtitles Causing Crashes

[laharah]

Server Version#: 1.18.5.2260
Player Version#: 4.20.1

When a transcode starts that has subtitle burn-in and transcode, the system load starts to spike and climb until it consumes all CPU resources. These netdata screenshots are from a Debian Buster server running on a Ryzen 9 3900X, so 100% cpu usage on 12 cores:

Here you can see someone starts a playback at about 10:55:

You can already see system usage climbing.

Here about 15 min later:

And finally what it’s like now:

Here are the logs, though they don’t seem to show anything out of the ordinary:
Plex Media Server Logs_2020-01-26_11-43-18.zip (1.2 MB)

I first saw this happening about 8-9 months ago, on my old server. It was Debian stretch and a little underpowered. A plex update started causing this problem, so I had to roll back to plexmediaserver_1.15.1.710, which I know for sure did not have this bug. I would periodically update and see if it got fixed, but after a few days someone would play something that caused the system to crash, so I’d roll back again. I ended up running 1.15.1.710 for nearly 6 months until I upgraded my server, so I know for sure it wasn’t present in that build.

After I built my new server I didn’t see the problem for a few months, so I thought it was an issue with my older server. Now I can see that it’s definitely an issue with PMS.

Also, since I last looked, someone else has posted about the same exact issue:

High System load on Subtitile transcode

Server Version#: 1.15.4.919 Player Version#: Web Player 3.83.1 I am running Plex in a docker container on an unraid System with Hardware Acceleration enabled. None of the Video files cause any load on my System and I’ve been noticing some strange load which increased over time. On netdata, this load is shown as System load [image] After identifiying when this happens, I noticed that first of all, at this time there is no transcoding in place [image] but what I noticed is the transcoding …

so at least I know it’s not just a weird config with my servers. It has a bit more information, however it doesn’t seem like the post went anywhere

Update:

It seems that my new rig has enough resources to at least avoid a crash. At an arbitrary point in the plaback (about 55% of the way through Zombie Land: Double Tap if it matters), whatever transcoding thread is eating the system resources releases them:

I’ve tested and can see that if I restart plex, and have my friend re-start playback of the same film, the system use starts climbing again.

Also, here is a screen cap with some more information about one of the particular transcodes that causes the problem:

[ChuckPa]

As confirmation of the ASS subtitles, please play to a web browser or set the playback on the player to “Always burn”.

There is a problem with converting to/from ASS subtitles.

[laharah]

I did as you asked:

When set to always burn everything looked okay:

[ChuckPa]

Without a matching scale, that seems more like it.

There is the initial surge as the transcoder fills all the buffers.
Once the buffers are filled and streaming begins, there is the first ‘pulse’ where it and the player become more in-sync with other. After that, normal playback with much smaller increases in activity.

To me, that confirms the ASS subtitle problem.
Do you concur?

[laharah]

From what I can tell, it seems to alleviate the issue. However it certainly seems to be treating the symptom rather than the cause. I can’t really enforce that users turn on always burn. Particularly people on xboxes/chromecasts etc.

[ChuckPa]

I won’t try to deflect there’s something going on but, I do also ask, given the newness of the Ryzen 9 3900x, if everything is indeed “up to snuff” with it?

Kernel spin-locks waiting for resources will drive a CPU mad.
Support for the 3900x is new.

Yes, “Academically”, might there still be a few kinks to work out?

I assert this because of how Intel chips behave in this scenario.
The Ryzen is indeed behaving differently. Motherboard firmware?

[laharah]

From what I can tell, yes. I’m on the latest motherboard firmware and linux kernel 4.19.0-6.

I haven’t seen any other high system usage like this, and since it’s almost the exact same behavior I had on my last build which was an intel i3 it seems to point to an issue with transcoding. I’ll keep monitoring and note down any suspiciously high system usage though.

[ChuckPa]

That’s an old kernel. Most distros, of which I believe Debian 10 qualifies, are 5.0+
I’m using 5.3 on Fedora 30

[laharah]

I’ll do a kernel update then. I haven’t done a major version kernel upgrade non-fresh machine before, so I’ll have to do some snapshots and double check everything I have installed will work on the new kernel.

[ChuckPa]

To help me fully understand,

What actually crashes? Plex? (hard crash or non-responsive?) or the entire machine?

[laharah]

Library browsing becomes unresponsive once system use gets to 100, and the system slows to a crawl. When at high usage, plexmediaserver service can’t be stopped. At least it hanged for more than 10 min before I killed the process. My older system would actually hard crash, new one just crawls. Though It’s likely the old system was crashing from heat.

update: I’ve upgraded to kernel 5.4.0 I’ll see if it has any affect and post an update when I can.

[ChuckPa]

Thanks. This is good information to gather. I am interesting in learning what I can about the Ryzen systems.

[DasMarx]

The original issue with SRT-ASS subtitles has been fixed with 1.17.0.1709.
See the following link for more discussions:

Plex Media Server process uses 100% cpu (SRT->ASS Conversion)

Hi, Server Version#: 1.15.3.876 (docker/plexpass) Player Version#: 3.95.1 I’m seeing 100% CPU usage of the Plex Media Server process only when something is playing. It’s using hw transcoding when needed and it works well, eg 4k -> 1080. CPU is Intel(R) Core(TM) i7-7567U CPU @ 3.50GHz with QuickSync available. This is with one stream in direct play (I usually can go to 4 to 5 transcoded streams in parallel without any problems): [image] As you can see, the transcoder isn’t doing much but …

I have not seen that particular issue anymore after installing 1.17.0.1709 and any version afterwards until 1.18.5.2260 on my system with chromecast/xbox users.

Nevertheless, I do see some cycles on a user level in Netdata too, but not as bad as the initial bug mentioned in the forum post above. and also not on a system level.

As of now with the same version that you have, on my system only 1 core is under load when doing SRT-ASS conversion, even if the movie buffer is full.

The screenshot below shows a movie playback via browser with SRT-ASS subtitles.

IOWAIT happens when the disk spins up. Until 13:33, I’ve let the playback pause at the beginning. Keep in mind that at this time no playback happens.
I’ve switched to No subtitles afterwards and the base load went down to nothing again.

[ChuckPa]

You refer to 1.17.0.1709 and current together.

Is there a definitive and quantitative difference between the two when transcoding?