Explain this image [TRANSODER/SYNC BUG] - FIXED in 0.9.17.3

General Discussions
21

@MikeG6.5 said:
@MovieFan.Plex’s point is easy to understand when you figure out how transcoding works.

First off, there’s a switch in the settings that determines how far ahead the transcoder will try to stay in front of the video that is currently streaming. This is found at Settings > Transcoder > Transcoder default throttle buffer.

It’s used to tell Plex how many seconds in advance you want Plex to transcode, in relation to the current playback spot. (sort of.) What really happens is the client requests the next “block” and the server transcodes the next block or two + the time in this setting to stay ahead of your playing.

While you are watching ANYTHING that requires transcoding, until the stream is done, the transcoder is “tied” up or in use. At least this has been my experiences with it and the OM feature. The only time the transcoder isn’t considered in use, is when you Direct Play something while you are streaming. Then the OM feature can make as many files as it can, until you start transcoding a different video again.

There isn’t really an easy way to get around all of this, if your media is stored in file containers or codecs that will require transcoding on some or all of your clients. Short of doing a conversion outside of Plex itself, as long as you have a stream open which requires transcoding, then the OM versions can’t be made, whether the transcoder is actually running on the media or not. (At least that has been my personal observations on my own media.)

Conversion scripts, apps like HandBrake or similar can be used to do conversions as well as the built in OM feature, and are likely to produce a higher quality conversion. They are going to require more efforts on your part to make sure it all happens. But the quality differences are worth the efforts to get things set up IMO.

I personally use @cayars conversion scripts on any media I have prior to adding it to a Plex library. The reason is pretty simple, actually… The media starts its life in my library already in the best format available to Direct Play. Now the only times my server transcodes is when the stored bitrate is higher than the client is set to, the levels are not 4.0 or 4.1, or I have something set up wrong on a client app.

Also keep in mind that even though PlexPy is using the API for getting most of its data from the server, it’s not always showing something exactly as it is on the server. It does a pretty good job, but it’s not 100%. Until more calls for the API are added, the best PlexPy can do on some things is an educated guess. And it’s been my experience that the transcode bar on the playing stream is one of those “educated guess” situations. Until PlexPy is built into PMS and the Web App itself, or until more calls are made in the API, this is the best you can expect from a 3rd party tool.

Don’t get me wrong! This tool is GREAT for the historical information it provides ALONE! But it’s not going to give you the information that Plex itself could give, if the Web App had the interface to allow this type of reporting.

If all of this is still “unacceptable” well, I’m sorry. There is always a difference between expectations and reality. And the reality is, this is what we have. What we all want is something built by the Plex Team, using Plex’s own tools built into the Plex Web App itself. Until we get that, we have to face it, we gets what we gets.

Ok so even if I accept the premise that the transcoder is “tied up” as you say, you’re telling me that the intended behavior is for a single transcode that is using less than 1% of my CPU power, should cause all background transcodes to stop in their tracks? What’s the point of having the “Prioritize streaming transcodes” setting (which I have disabled) if it’s going to stop all background transcodes anyway during from a single streaming transcode?

22

The background process can kick in if there is enough time after the transcoder goes idle. That’s why it could work after the initial transcoding work since there is a large buffer and plenty of time until the transcoder is needed again. But during normal use, there is no large buffer and the transcoder comes on and off. The off time just might not be long enough.

Also, I don’t know what priority for background tasks are, but it could be there is something else going on too, like BIF creation. Hopefully your log will explain what is going on. When you recreate, make sure the log has the start of the transcoding processes and to at least some point where the transcoder is going on and off after the initial burst.

23

@MovieFan.Plex said:
The background process can kick in if there is enough time after the transcoder goes idle. That’s why it could work after the initial transcoding work since there is a large buffer and plenty of time until the transcoder is needed again. But during normal use, there is no large buffer and the transcoder comes on and off. The off time just might not be long enough.

Also, I don’t know what priority for background tasks are, but it could be there is something else going on too, like BIF creation. Hopefully your log will explain what is going on. When you recreate, make sure the log has the start of the transcoding processes and to at least some point where the transcoder is going on and off after the initial burst.

Will do. Should be able to upload it in an hour or so.

24

Ok just played a file in its entirety (46 min.). Here is the PMS log as well as logs of my CPU usage ever 2 seconds for the duration of the play. CPU_Usage1 shows all 16 cores where as CPU_Usage2 shows just the overall CPU usage as well as the average at the end.

If you’ll notice, I had an average CPU usage of 0.78% and for the last 20 minutes there was literally no transcoding happening at all. Meanwhile, this is apparently enough to tell Plex to stop all transcodes in the background.

25

@IamSpartacus said:

@MikeG6.5 said:
@MovieFan.Plex’s point is easy to understand when you figure out how transcoding works.

First off, there’s a switch in the settings that determines how far ahead the transcoder will try to stay in front of the video that is currently streaming. This is found at Settings > Transcoder > Transcoder default throttle buffer.

It’s used to tell Plex how many seconds in advance you want Plex to transcode, in relation to the current playback spot. (sort of.) What really happens is the client requests the next “block” and the server transcodes the next block or two + the time in this setting to stay ahead of your playing.

While you are watching ANYTHING that requires transcoding, until the stream is done, the transcoder is “tied” up or in use. At least this has been my experiences with it and the OM feature. The only time the transcoder isn’t considered in use, is when you Direct Play something while you are streaming. Then the OM feature can make as many files as it can, until you start transcoding a different video again.

There isn’t really an easy way to get around all of this, if your media is stored in file containers or codecs that will require transcoding on some or all of your clients. Short of doing a conversion outside of Plex itself, as long as you have a stream open which requires transcoding, then the OM versions can’t be made, whether the transcoder is actually running on the media or not. (At least that has been my personal observations on my own media.)

Conversion scripts, apps like HandBrake or similar can be used to do conversions as well as the built in OM feature, and are likely to produce a higher quality conversion. They are going to require more efforts on your part to make sure it all happens. But the quality differences are worth the efforts to get things set up IMO.

I personally use @cayars conversion scripts on any media I have prior to adding it to a Plex library. The reason is pretty simple, actually… The media starts its life in my library already in the best format available to Direct Play. Now the only times my server transcodes is when the stored bitrate is higher than the client is set to, the levels are not 4.0 or 4.1, or I have something set up wrong on a client app.

Also keep in mind that even though PlexPy is using the API for getting most of its data from the server, it’s not always showing something exactly as it is on the server. It does a pretty good job, but it’s not 100%. Until more calls for the API are added, the best PlexPy can do on some things is an educated guess. And it’s been my experience that the transcode bar on the playing stream is one of those “educated guess” situations. Until PlexPy is built into PMS and the Web App itself, or until more calls are made in the API, this is the best you can expect from a 3rd party tool.

Don’t get me wrong! This tool is GREAT for the historical information it provides ALONE! But it’s not going to give you the information that Plex itself could give, if the Web App had the interface to allow this type of reporting.

If all of this is still “unacceptable” well, I’m sorry. There is always a difference between expectations and reality. And the reality is, this is what we have. What we all want is something built by the Plex Team, using Plex’s own tools built into the Plex Web App itself. Until we get that, we have to face it, we gets what we gets.

Ok so even if I accept the premise that the transcoder is “tied up” as you say, you’re telling me that the intended behavior is for a single transcode that is using less than 1% of my CPU power, should cause all background transcodes to stop in their tracks? What’s the point of having the “Prioritize streaming transcodes” setting (which I have disabled) if it’s going to stop all background transcodes anyway during from a single streaming transcode?

I was explaining the switch and my own personal observations for the very few times I ever have media transcoding, and how it interacts with other transcoder processes.

I guess if you are only using 1% of the entire CPU during a transcode, you must have a pretty healthy CPU in your machine. Any transcode sessions I ever have going are in the 300% or higher range on a 4 core system. I use 2-5% during a Direct Play session, so for the CPU in your machine, that’s not even going to bump up on a trace for a Direct Play stream.

I would be very interested in knowing what CPU you have in your PMS machine given those kinds of usages.

26

I was explaining the switch and my own personal observations for the very few times I ever have media transcoding, and how it interacts with other transcoder processes.

I guess if you are only using 1% of the entire CPU during a transcode, you must have a pretty healthy CPU in your machine. Any transcode sessions I ever have going are in the 300% or higher range on a 4 core system. I use 2-5% during a Direct Play session, so for the CPU in your machine, that’s not even going to bump up on a trace for a Direct Play stream.

I would be very interested in knowing what CPU you have in your PMS machine given those kinds of usages.

I do have a very good CPU but not THAT good. My point is that even though Plex CLAIMS it’s transcoding, it’s not. My CPU usage being at less than 1% during this time proves that. Yes, when an ACTUAL transcode is taking place, my CPU usage will get get somewhere in the 500-600% range on a 8-core/16-thread system. But what’s happening here is no actual transcoding, yet Plex is reporting it as such.

I just spent the some time watching VERY closely at some live transcodes on one screen and the list of file conversions on another. While PlexPy may not be perfect, it’s really helpful in this sense. What I’m seeing is the following:

  1. During a transcoded stream, file conversions pause during the few seconds Plex is transcoding the live stream (every 30-60 seconds or so). This is expected and makes case.

  2. When the transcoded stream is throttled, file conversions resume PROVIDED the entire live stream has not been 100% transcoded (as demonstrated by the stream on the left in the below pic).

  3. If the live stream has been transcoded 100% (as demonstrated by the stream on the right in below pic), file conversions will pause for the remainder of the stream. Even though it appears the file is 100% transcoded, Plex seems to treat it like it is transcoding afterall. And as explained, while Plex shows a transcoding speed of 3.7 (again refer to pic), no actual transcoding is taking place (CPU is idle).

So, in conclusion, for whatever reason once the file reaches a transcoded state of 100% (grey bar gets all the way to the end), Plex will treat that file as if it’s being transcoded for the remainder of the stream even though no actual transcoding is taking place. This, in turn, pauses all background transcodes.

27

@IamSpartacus said:
If you’ll notice, I had an average CPU usage of 0.78% and for the last 20 minutes there was literally no transcoding happening at all. Meanwhile, this is apparently enough to tell Plex to stop all transcodes in the background.

@“MovieFan.Plex” is far better at examining the logs so I will leave that to him, but for comparison I have an 8 core CPU and while I am watching transcoded content the background transcoding for a Cloud Sync job continues to run

28

I’ll check your logs but yeah that does sound like a bug where the end of a transcode is not technically idle so the background processes don’t pick back up. I’ll get back to you after I look over the logs.

29

Any more info I can provide on this to help? I’m seeing it across different file types (at least AVI and MKV’s that I noticed) and it happens during direct streams with audio transcoding or straight transcodes.

30

The file is being transcoded into a matroska (mkv) format, which is not quite the same as HLS, so you don’t see the typical segments and wait processes, but background processes should still kick in.

19:00:40 - Transcoder starts
19:01:32 - Transocder idles completing 34.5 percent (impressive)
19:01:37 - Trancoder spins pack up
19:01:38 - Transcoder spins back down, now at 35.5 precent
19:01:40 - The analyzer for generating BIF files kicks in, found nothing to do and stops
19:01:58 - Trancoder spins pack up
19:02:00 - Transcoder spins back down, now at 37.0 precent
19:02:18 - Trancoder spins pack up
19:02:20 - Transcoder spins back down, now at 38.1 precent

It basically keeps doing this every 20-30 seconds to stay just ahead of playback.
Finishing the transcode at 19:27:04 and playback just a little over 50%.
Playback finishes around 19:46:43.

So all of that looks normal. I do not see any background processes kick in. Although I can’t tell if there were suppose to be any. Your log only starts at the beginning of playback so I can’t see if there were things going on that had to be stopped first or not. Your log ends just shortly after you finished playback so I can’t tell if these processes came on after either. You say that when the transcoder hits 100%, then the background stuff does kick in, but I don’t see anything in your log. Do you still have the before and after portions of the log, about 5-10 worth should do? Did you possibly pick a time when all the background stuff was done? At least for this test you did.

31

Thanks for getting back to me @MovieFan.Plex.

If you’re looking at the last log I posted, no I did not have any background processes queued up for that one. I wanted to keep the log as clean as possible in case there was something that stood out with regard to the transcoding of the live stream itself. That’s also why I kept the log as concise as I could with regard to that one stream. The log I posted here had background transcodes in the queue though.

When the transcoder hits 100%, background processes DO NOT kick back in, that’s the main problem I’m trying to address.

Would you like me to recreate the same log but with a few items queued up for conversion so you can compare?

32

Let me check the other log first.

33

Ok. I’m not totally positive, but in that log it showed that the system was monitor your drives for updates. When your transcoder stopped, PMS started generating thumbnails for a bunch of stuff. I don’t know if it’s because it found new stuff, updating old stuff, or some other reason, but it is working. It’s not the transcoder but PMS was busy doing something. I don’t know the priority of what should be done when or if they can be done in parallel, but your PMS was not idle.

If you want to try another test, wait until PMS is idle, start a sync or optimize conversion process, and then hit play and we can see what happens.

34

Ok, here is a log file for you. Let me give you a little summary of what I did and what I witnessed to possibly help you sift through the log.

1. Started optimizing Law & Order episodes (have a queue of a few seasons worth)
2. Resumed playing Veep S05E01 with about 5 minutes remaining. Appeared to take about 30 seconds to transcode the remainder of the video. This of course, paused the optimized conversion immediately upon the start of the video.
3. Once Veep S05E01 ends, it takes a good 1-3 minutes before the optimized conversion resumes.
4. I let the conversion go for 30 seconds or so and then start playing Law & Order S15E09 and skip close to the end. This of course pauses the optimized conversion again.
5. Even though the transcoder reaches 100% very quickly, the optimized conversion remains paused until the stream ends. Once it ends, the file conversion resumes immediately.

*A few things of note here.

First off, the behavior after the streams end was different for the two files. With the first file, conversion doesn’t resume until full minutes after the stream ends. With the second file, conversion resumes immediately after the stream ends.

The other thing to note is that when I say “optimized version pauses”, I’m really meaning it slows to an absolute crawl. During a normal conversion I’ll usually see ‘Converting (16x)’ for example. But when in this “paused” state, there is no number next to ‘Converting’ and it appears to be converting at a rate of about 1% every 5 minutes or so. I figured this could be important because the log may show the file is converting, meanwhile it’s just sitting there chugging along at a solid 1% clip every 5 minutes (solid 8.33 hour ETA!!)

35

This is still a major issue for me. I had 2 users syncing yesterday starting at about 4-5pm. 20-25 items total between the two and based on how long each item takes to convert when Plex is otherwise idle it should have taken a few hours.

Unfortunately since I had live streams being played all night, those items didn’t finish syncing until this morning (roughly 12 hours later). I watched this happening throughout the process and again the culprit were these transcodes for low resolution streams that would finish transcoding the whole video in 1-3 minutes and then hang in a perpetual transcoding state until the stream ended.

I’m happy to do more testing if needed because this is really affecting the way I use my server.

36

Ok, just got some info from the devs. The background transcoding processes are set to idle so that live transcoding can take precedence. The background processes do not come back out of idling until playback is finished. This may change in the future, but for now, it is working as expected.

37

So what does this setting do if it doesn’t prohibit that behaviour?

kxz3thfr6aca.png847×114 4.39 KB

38

@Christophorus said:
So what does this setting do if it doesn’t prohibit that behaviour?

847×114 4.39 KB

My question exactly!!

@anon18523487 said:
Ok, just got some info from the devs. The background transcoding processes are set to idle so that live transcoding can take precedence. The background processes do not come back out of idling until playback is finished. This may change in the future, but for now, it is working as expected.

Unfortunately @anon18523487 this statement does not reflect what I see. Based on what you’re saying, all background transcoding would be paused during any live stream that has to be transcoded. Yet I have many videos that will transcode for a period to get ahead, throttle, and then background transcoding will result DURING those streams. However I’m also seeing a significant amount of files being transcoding to 100% rapidly (within 2 minutes) and then report to be in a perpetual transcoding state even though they are not transcoding. This is what pauses all background transcodes and annoys me to no end.

So I’m sorry to say but something is off here because what you’re described is the “intended” behavior is not what I’m seeing all the time. If every time a file was transcoded it pauses background transcodes, I wouldn’t have even brought this topic to light because I’d assume it was intended. But since many times my background transcodes work just fine during live transcodes I have a point of comparison.

And to @Christophorus ’ point, what would be the point of having a setting to prioritize streaming transcodes if Plex’ software is doing this anyway even if unchecked?

39

Unfortunately @anon18523487 this statement does not reflect what I see. Based on what you’re saying, all background transcoding would be paused during any live stream that has to be transcoded. Yet I have many videos that will transcode for a period to get ahead, throttle, and then background transcoding will result DURING those streams.
Sorry, I must have mis-understood. I thought the background processes always stopped for you.
However I’m also seeing a significant amount of files being transcoding to 100% rapidly (within 2 minutes) and then report to be in a perpetual transcoding state even though they are not transcoding. This is what pauses all background transcodes and annoys me to no end.
This part is the intended behavior if that setting is enabled. If it is not enabled, the background processes should not go idle like this. I did not realize you had that setting off.
And to @Christophorus ’ point, what would be the point of having a setting to prioritize streaming transcodes if Plex’ software is doing this anyway even if unchecked?
It shouldn’t. Can you provide a screenshot of that entire settings page?

40

@anon18523487 said:

Unfortunately @anon18523487 this statement does not reflect what I see. Based on what you’re saying, all background transcoding would be paused during any live stream that has to be transcoded. Yet I have many videos that will transcode for a period to get ahead, throttle, and then background transcoding will result DURING those streams.
Sorry, I must have mis-understood. I thought the background processes always stopped for you.
However I’m also seeing a significant amount of files being transcoding to 100% rapidly (within 2 minutes) and then report to be in a perpetual transcoding state even though they are not transcoding. This is what pauses all background transcodes and annoys me to no end.
This part is the intended behavior if that setting is enabled. If it is not enabled, the background processes should not go idle like this. I did not realize you had that setting off.
And to @Christophorus ’ point, what would be the point of having a setting to prioritize streaming transcodes if Plex’ software is doing this anyway even if unchecked?
It shouldn’t. Can you provide a screenshot of that entire settings page?

Yes, for the entire length of this discussion that setting has been unchecked which is why this whole thing is so frustrating. I’d have no qualms about this behavior if I had that setting checked off. I’ve actually never had it checked as far as I’ve known about it (basically since it was implemented). Here is a screen shot of the settings page and I also confirmed the following settings in my Preferences.xml.

BackgroundQueueIdlePaused="0" BackgroundTranscodeThrottle="0" TranscoderQuality="0" TranscoderH264BackgroundPreset="veryfast"