Transcoder background tasks being throttled

Server Version#: 1.19.5.2889-52a517a93 (and older)
FreeNAS. FreeBSD version manually installed in a jail.

TLDR: I think Plex Transcoder is being unnecessarily throttled by PMS when processing background media tasks. I modified the Transcoder binary to test this theory, with great results.

Generating video preview thumbnail / media index / index-sd.bif files has been surprisingly slow for me.

Today I got frustrated and started looking. I noticed that while being slow, it’s also hardly doing any I/O or asking for any CPU, and that the Transcoder process is spending a lot of time sleeping.

I looked at how Plex Transcoder is called. I ran ffmpeg with the same arguments, and extracting the jpg files was hundreds (!!!) of times faster.

It didn’t make sense for ffmpeg to be faster, so I set the environment variables and called Plex Transcoder directly.

LD_LIBRARY_PATH=/PATH/TO/PLEX/LIB
FFMPEG_EXTERNAL_LIBS=/PATH/TO/DATA/CODECS/
X_PLEX_TOKEN=TOKENHERE

Calling Plex Transcoder in the same way PMS does is very slow. This is how it extracts jpg images for index-sd.bif preview thumbnail generation:

$ ./Plex\ Transcoder -skip_frame:v nokey -i input.mkv -threads 0 -q 3 \
      -filter_complex "[0:V:0] fps=fps=0.500000:round=up,scale=w=320:h=240:force_original_aspect_ratio=decrease [out]" \
      -map "[out]" img-%06d.jpg \
      -progressurl http://127.0.0.1:32400/video/:/transcode/session/bif/SESSION-ID/progress
// (output trimmed)
frame=   64 fps=0.4 q=3.0 Lsize=N/A time=00:02:08.00 bitrate=N/A speed=0.79x

0.4 FPS! Barf! I cancelled that job. That’s how bad it has been from inside Plex, too.

(Note: I removed -nostats and -loglevel quiet so I could see what was going on. They don’t seem to affect the speed.)

If I remove -progressurl it becomes fast:

$ ./Plex\ Transcoder -skip_frame:v nokey -i input.mkv -threads 0 -q 3 \
      -filter_complex "[0:V:0] fps=fps=0.500000:round=up,scale=w=320:h=240:force_original_aspect_ratio=decrease [out]" \
      -map "[out]" img-%06d.jpg
// (output trimmed)
frame=  687 fps= 55 q=3.0 Lsize=N/A time=00:22:54.00 bitrate=N/A speed= 111x

It’s truly +100x faster.

Without -progressurl, or if I don’t set a good X_PLEX_TOKEN, or if I give a bad URL, then it is fast.

When I provide -progressurl, and a good X_PLEX_TOKEN, and a valid URL, then it becomes dramatically slower.

At first I thought the Transcoder must be doing something wrong. It sends status updates to PMS every 500ms, and PMS responds very quickly. The Transcoder is sleeping when it should be working, so I blamed it. :sleeping:

Then I looked at the responses from PMS to the Transcoder, and realized the Transcoder is actually following instructions. PMS is sending this:

<MediaContainer size="0" canThrottle="1">

That canThrottle there is very interesting. I think it’s the smoking gun. Is the Transcoder obeying this instruction to throttle?

To test, I hacked up a copy of the Transcoder binary, changing every instance of “Throttle” to “Thrapple”, so it wouldn’t parse canThrottle any more.

perl -pi -e 's/Throttle/Thrapple/g' "Plex Transcoder"

That worked great manually, so I dropped it into the Plex directory and restarted Plex. It works great there, too.

Video preview thumbnail generation is super fast now. Moments vs. hours. I’ve generated more previews in the last hour than I did all last week.

I also tested with several Sync tasks. I previously ignored this feature because of how slow conversions were. They also work much more quickly with the modified Transcoder binary.

My theory is that Plex Media Server is (incorrectly) sending canThrottle="1" to the Plex Transcoder for these “background” media processing tasks. If I understand correctly, throttling the transcoder makes sense for foreground/attended/interactive/immediate transcoded playback, not background tasks.

I’ve made sure that all of my clients are configured to Direct Play, so I don’t think it will hurt anything if I leave it this way.

I’m very curious to find out if this is a Plex issue or something with my config.

This is a very detailed report :smiley: thanks for doing most of the work :slight_smile:

I wonder does this happen if you restart the server and make sure no one is streaming/syncing etc?

We do set canThrottle to 1 but it seems to be as part of Automatically adjust quality only (not 100% sure need to confirm with devs)

To follow up, all of my Thumbnail/Preview/Index generation finished last night. Intro Detection did too. (Is it Chromaprint? I bet it’s Chromaprint.)

Automatically adjust quality is a client playback setting, isn’t it? Other than bandwidth, does that use information from the server?

I restarted the server a few times while doing this. I’m confident these tasks were slow when there was no active playback. I’m not sure if they’re slow while playback is active. It would be funny if they were faster when the server was busier.

On a thread about slow Intro Detection, one suggestion was to start/pause/resume/stop an Optimization job, to “tickle” the engine. That sounded like superstition and didn’t help me. Maybe there’s something to it - a different way to avoid throttling? I can confirm that canThrottle also affects the Transcoder’s role in performing Intro Detection.

I’m hoping a dev comes along and says “Oh! That should be set to 0! I’ll fix it!” or “Hey dummy, you aren’t using a blue patch cable, are you? That’s your problem.” or “The server is doing the right thing, but the Transcoder is being silly about it.”

If they don’t, I’ll compare behavior with a default install or on a different platform.

Yeah its a client setting but I was just saying “this is the only place I seem to see it explicitly set to 1”

We might have other cases, but we do need a PMS dev to confirm, this can be set just in general to background task (or processing on media added) so we avoid making actually playback transcodes slower. But I can see how in some cases having the ability to run those faster if we can (even if based on a Pref) can help.

I should be able to have more detailing info Monday/early in the week, its a bit harder on weekends :smiley:

BTW I though I had asked but clearly not, can you post the PMS logs while this is throttled? they might indicate why.

This is directly related to the Disable video stream transcoding setting. It appears to have the side effect of throttling all Plex Transcoder workers, including those doing optimization, thumbnail generation, and intro detection.

With Disable video stream transcoding unchecked, when a media optimization job is started, this is immediately logged:

Jun 08, 2020 11:11:22.546 [0x813f29f00] DEBUG - Started session successfully: d63319ce-e599-4c68-9168-7b31773244f0
Jun 08, 2020 11:11:22.546 [0x80c92c000] DEBUG - Completed: [127.0.0.1:46499] 204 PUT /video/:/transcode/session/d63319ce-e599-4c68-9168-7b31773244f0/de014a7c-6ff9-428f-ac72-d8c92cbfd96e/progress?duration=3614.320000 (7 live) 0ms 203 bytes (pipelined: 9) (range: bytes=0-) 
Jun 08, 2020 11:11:23.050 [0x813ca8300] DEBUG - Streaming Resource: Unthrottling session 0x8140483d8:d63319ce-e599-4c68-9168-7b31773244f0 which is using static transcoder slot.  Used slots is now 1

The job progresses normally, and canThrottle=1 is never sent. It appears PMS proactively disables (“unthrottles”) the backpressure mechanism for those non-streaming optimization jobs.

With Disable video stream transcoding checked, when a media optimization job is started, this is logged:

Jun 08, 2020 11:04:26.248 [0x81406dd00] DEBUG - Started session successfully: 2e995488-5c5c-48c1-a054-1b385b692e1a
Jun 08, 2020 11:04:26.248 [0x80bd5ac00] DEBUG - Completed: [127.0.0.1:64138] 204 PUT /video/:/transcode/session/2e995488-5c5c-48c1-a054-1b385b692e1a/468f94ea-abe2-438f-bf62-5c905e0076ac/progress?duration=3614.320000 (8 live) 0ms 203 bytes (pipelined: 9) (range: bytes=0-) 
Jun 08, 2020 11:04:26.752 [0x81406d300] DEBUG - Session 2e995488-5c5c-48c1-a054-1b385b692e1a (0) is throttling
Jun 08, 2020 11:04:26.753 [0x81406d300] DEBUG - [Transcoder] Throttle - Going into sloth mode.

And it goes sloooow. Sloth-like. :sloth: But it does make progress, and it does eventually complete. I never guessed it was broken, I just didn’t use it because it was so slow.


The logging for video preview thumbnail (index-sd.bif) generation jobs doesn’t give as much info.

With Disable video stream transcoding unchecked, PMS doesn’t log that same “Unthrottling … static” message. PMS doesn’t send canThrottle="1" to the Transcoder. It’s not “fast” on my system, but it’s fast-as-can-be.

With Disable video stream transcoding checked, PMS doesn’t log anything to indicate that it is going to be slow. PMS does send canThrottle="1" to the Transcoder. The Transcoder doesn’t log that it is throttling. It sits, using barely any CPU. It’s slooow. :turtle:


Doctor, it hurts when I disable transcoding!

I enable Disable video stream transcoding because though my CPU is limited, I have plenty of bandwidth and capable clients. I only want to disable video transcoding for real-time playback. I’m happy to share why, if that discussion is important.

I don’t want to slow down optimization jobs, video preview thumbnail generation, or intro detection. I also want a pony, peace on earth, and goodwill toward man.

I’m guessing that throttling should only apply to realtime transcoding (and remuxing) sessions, and that throttling background job types when Disable video stream transcoding is enabled is an unintended side effect.


:unicorn: :rainbow: :robot:

1 Like

Per comments in another thread, a fix may be on the way.

I started to experiment with using Plex to Optimize some of my media.

Disable video stream transcoding also silently changes optimization job settings. They can still be initiated and are processed. Audio is transcoded if necessary. Video is copied and remuxed, but not transcoded.

In all fairness, that means the checkbox does what it says. :smile::confused::cry: I recognize that this is basically my issue. I’d love for that checkbox to behave differently - to only disable video transcoding during playback. And to have my cake. :cake:

I was confused because when Disable video stream transcoding is enabled, the Optimize... option is still available in the Web UI, and it still allows size and bitrate selection.

Same for download jobs - video quality settings are silently ignored. The jobs are accepted, audio is transcoded. Video is remuxed but not transcoded. Files are downloaded.

“Wow, 200Kbps video looks remarkably good. I don’t think it lost any quality at all. Wow.”

“…”

“Wait a minute …”

HAH, wait, no. It’s weirder than that. For optimization and download jobs, the impact of Disable video stream transcoding depends on the original video codec.

When the source video is h264, Disable video stream transcoding causes the video transcoding settings to be silently ignored. Regardless of source size and bitrate, target settings are ignored and the video is only remuxed.

When the source video is any other codec (hevc), video transcoding settings, target dimensions and bitrate, are obeyed. Disable video stream transcoding doesn’t change anything, and the video is transcoded.

1.19.4.2935, optimization/preview/intro jobs aren’t permanently throttled any more. Nice.

I’ll split my comments about optimizer job settings to another thread. I found other optimizer job settings that don’t work at all, too.

Mentioning this here because I promised to post it.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.