Inconsistent Plex Network Traffic?

chromecast

#1

Can someone explain why Plex's network traffic (playing from remote server at a datacenter) is so inconsistent and unsmooth as compared to a 1080p stream from Netflix? The spikes on Netflix are from when the video is loading. Plex is always spiking. Is the transcoder sending only buffer chunks at a time?

 

t4p7.jpg


#2

Is it causing a problem?


#3

I see the same find of fluctuation on my VPS when someone watches something.

It doesn't seem to bother anything that I can see

bandwidth.jpg


#4

I cant completely say it is or isnt causing problems. If the packets get spread out too far due to this, yes it could cause the buffering/stuttering. Sometimes that does happen... but also if it is spiking, perhaps instead of normalizing the data transfer, the spikes could be causing a bandwidth issue (really isnt in my case since I have a very large pipe with this server, but others sharing from their homes for example could have issues with this). 

Either way, it shouldnt be spikey and it should be more smooth and normalized.


#5

First off you have to understand just how different streaming from your own Server is when compared to Netflix.

Most importantly is that Netflix probably invested millions in transcoder hardware that blows away anything your likely to have in your house. 

Second Netflix knows their stream is going over an internet with latency issues and therefore have maximized and normalized their streams for better use of and consistency of stream. The internet is much slower than your local network and therefore they must be very wary of spikes and valleys that would not work well over an internet. 

Third is that Netflix isn't transcoding at all, Everything you watch has already been transcoded into a variety of formats which it decides which is the best version to send you when you start the movie. 

The Plex transcoder is working on the fly in the same way a relay race or bucket brigade works. You need a file in one format, source is saved in another, the PlexTranscoder merely translates the incoming data and then hands it off. Interruptions to the read of source, the write of buffering on top of OS maintenance and Wireless duplexing all throw valleys that require peaks to make up for....

And it will be much more apparent on a graph because your most probably trying to send a much higher bandwidth locally than Netflix is trying to send you over the net.


#6

I am well aware of everything you posted and hypothesized. Explaining that while Plex started out as a small home solution where these spikes are fine, its grown to allow sharing outside a home network as well as hosting on remote servers. This means that an inconsistent stream should be looked into as it could be an early sign of something needing to be optimized or planned for prior to it becoming a larger issue.


A comparison between Netflix and Plex remote is just showing the ideal vs current. I know how transcoding works so perhaps a new utility running alongside Plex transcoder that can act as an intermidiary when hosting external that can normalize the streams could eliminate some buffering and stuttering issues down the line.


#7

I am well aware of everything you posted and hypothesized. Explaining that while Plex started out as a small home solution where these spikes are fine, its grown to allow sharing outside a home network as well as hosting on remote servers. This means that an inconsistent stream should be looked into as it could be an early sign of something needing to be optimized or planned for prior to it becoming a larger issue.

A comparison between Netflix and Plex remote is just showing the ideal vs current. I know how transcoding works so perhaps a new utility running alongside Plex transcoder that can act as an intermidiary when hosting external that can normalize the streams could eliminate some buffering and stuttering issues down the line.

What I'm trying to get across here is go buy an industrial transcoding server (Xeon, Octacore) and Business class networking gear and plex will get you the same result with the exception being that Plex will STILL be transcoding on the fly compared to Netflix which doesn't have to wait for a conversion process to complete because it was completed months ago, cached and sent whenever required in a simple file send operation.

If you want to wait for the Plex Transcoder to finish transcoding the entire file before it send (the way Netflix does) then you might get results closer to what they get.


#8

Please read and comprehend prior to replying. I understand you either do or think that you know hardware and networking as well as the process for companies you don't have intimate knowledge of but step outside of the box for now.


The network does not need to be inconsistent and should not now that the solution has grown outside of small home networking constraints. As I stated, it could cause problems down the line and there may be a way to solve it using another library or utility/application. If the transcoder is truly dumping just chunks of data rather than writing them into a live file(appending) where that file is being streamed, then we are missing some performance and stability areas.


Analogy time!


You are to give me $100 in quarters, 4 quarters per second. Would you rather grab 4 quarters and hand them to.me each second and run the risk of a bad handoff? Or would you rather put them as fast as you can into a jar that I then grab from as required? Essentially allowing you to get much further into filling the jar so that if you stumble, there is enough buffer that it won't impede my retrieval?


And yes, I have multiple enterprise hardware in a data center on a gigabit port with a full gigabit connection with more than enough resources that a single transcode @1080p barely makes 2% of resource use, all on an OS that has the most minimal overhead possible. All of which is being adopted and used more frequently. So if a stutter happens simply because of a spikey transfer, it should raise the flag now rather than later.


#9

Please read and comprehend prior to replying. I understand you either do or think that you know hardware and networking as well as the process for companies you don't have intimate knowledge of but step outside of the box for now.

The network does not need to be inconsistent and should not now that the solution has grown outside of small home networking constraints. As I stated, it could cause problems down the line and there may be a way to solve it using another library or utility/application. If the transcoder is truly dumping just chunks of data rather than writing them into a live file(appending) where that file is being streamed, then we are missing some performance and stability areas.

Analogy time!

You are to give me $100 in quarters, 4 quarters per second. Would you rather grab 4 quarters and hand them to.me each second and run the risk of a bad handoff? Or would you rather put them as fast as you can into a jar that I then grab from as required? Essentially allowing you to get much further into filling the jar so that if you stumble, there is enough buffer that it won't impede my retrieval?

And yes, I have multiple enterprise hardware in a data center on a gigabit port with a full gigabit connection with more than enough resources that a single transcode @1080p barely makes 2% of resource use, all on an OS that has the most minimal overhead possible. All of which is being adopted and used more frequently. So if a stutter happens simply because of a spikey transfer, it should raise the flag now rather than later.

BAD analogy...

You want $100 in quarters...One guy has $100 all in Quarters ready to had to you on a timely 1 second basis...

Plex However only has a $100 bill and must break it into Quarters before it can hand you anything...

And also needs to find all these Quarters and may not have enough on hand even the stream of Quarters isn't consistent!

NETFLIX IS AL PRE-TRANSCODED!

PLEX is not!

Now you may think you understand the difference but apparently you don't!

Not only do they have the Hardware that can create streams much more efficiently than the technology you have in your house...

They did it days and hours before you asked for the file which is why they can consistently send ALREADY CONVERTED content consistently while Plex can not because it is still busy converting it!

Netflix is just a file server! What you get from them isn't being transcoded it was done weeks ago and therefore can be consistently sent!


#10

How hard is it for you to read?!?! Have I ONCE said I didn't think that Netflix has all types file formats? You seem to be stuck on that. I weep for our youth if you are an example of the inability to comprehend text and concepts.


Again.. Maybe this time I should simply draw it out for you in crayon.


But alas, I'll just explain again and ignore your future posts as you can't look past your own nose at a problem.


Hypothetical process:

The plex server will transcode source into an intermediary file on the fly (using the correct container and codec support based on the client hardware) and then Plex streams that file to the client rather than bits or chunks (if that is truly what it is doing as that was the original point of this post).


Now, what does this accomplish?

The source files remain unchanged and thusly can be used as direct play for clients that can do so. It also alleviates the need to pretranscode to multiple formats/containers etc so that each client type is accounted for. Allowing a fully dynamic server with enough buffered content that the streams are smooth and optimized with only a slight increase on the initial play/start/seek time.


However, back to the original question.


Is Plex sending the transcoded segments of the video in chunks to fill the clients buffer?


#11

Can someone explain why Plex's network traffic (playing from remote server at a datacenter) is so inconsistent and unsmooth as compared to a 1080p stream from Netflix? The spikes on Netflix are from when the video is loading. Plex is always spiking. Is the transcoder sending only buffer chunks at a time?t4p7.jpg

So back to my origional question..

Is it causing a problem? 

You see this is one of thoes "It's only a problem if it's a problem" If you are not having ANY other issues, then it's not a problem.


#12

Is Plex sending the transcoded segments of the video in chunks to fill the clients buffer?

Answer: no it is sending it as soon as the Transcoder is finished created them t the buffer, any delay in the transcoding means the buffer is not full enough to send data and you get the notification that it is bufferring ....

Since Netflix did it 6 Months ago it doesn't have to wait and therefore sends a more consistent stream...


#13

So back to my origional question..

Is it causing a problem? 

You see this is one of thoes "It's only a problem if it's a problem" If you are not having ANY other issues, then it's not a problem.

Well yes, tracking down a stuttering/buffering problem and this is just one area of concern. The hardware is overly capable and the network is as well. The only paths that I can follow to find the issue is in the inconsistency of the network transmissions. Simply stating there isnt a problem, does not make it true.

Answer: no it is sending it as soon as the Transcoder is finished created them t the buffer, any delay in the transcoding means the buffer is not full enough to send data and you get the notification that it is bufferring ....

Since Netflix did it 6 Months ago it doesn't have to wait and therefore sends a more consistent stream...

Originally i was again irritated by your lack of comprehension, but I realized you may have been on to something without knowing it yourself. As the transcoder truly is capable of transcoding much faster than required to be "live" (* speed => 15 with 3 1080p streams running concurrently), then perhaps it was another area of the same utility. As the transcoder does truly have a "sleep" or "throttle" period, increasing it means that either it transcodes and then sends a larger "buffer" or it has more to send at once. Looks like I have more to investigate.

Good day.


#14

Any ideas on settings for the transcoder to normalize the transfer? Make it more of a fluid transfer rather than spikey?

Thinking specifically to the "segmented transcoder timeout" and the "transcode default throttle buffer". 

Would increasing the throttle buffer to a quite large amount allow it to write to the temp file for a longer amount of time, or would it transcode that amount and then append the temp file with the large chunk rather than appending live? What about the transcode segments... larger number so that there is a larger buffer in the temp file before the transcoder gets taken out of sloth mode?


#15

Well yes, tracking down a stuttering/buffering problem and this is just one area of concern. The hardware is overly capable and the network is as well. The only paths that I can follow to find the issue is in the inconsistency of the network transmissions. Simply stating there isnt a problem, does not make it true.

Originally i was again irritated by your lack of comprehension, but I realized you may have been on to something without knowing it yourself. As the transcoder truly is capable of transcoding much faster than required to be "live" (* speed => 15 with 3 1080p streams running concurrently), then perhaps it was another area of the same utility. As the transcoder does truly have a "sleep" or "throttle" period, increasing it means that either it transcodes and then sends a larger "buffer" or it has more to send at once. Looks like I have more to investigate.

Good day.

Your expecting the Transcoder to go back when it gets far enough ahead and then normalize a stream in a file it sent to someone that may or may not have already been played by that device?

You need to come to grips with the fact that Plex is LIVE and Netflix is Memorex!

Netflix transcodes the file ahead of time and can do it in 4 or 5 passes to normalize the stream long before anyone has access to it.

If you want the same experience then you need to bypass the Plex Transcoder entirely, Pre-Transcode your source using the same methodology NetFlix does and do all this normalization before you even entertain the notion of playing it!

Then the file will direct play and since it is merely sending a file not doing any computational work it will stream quite evenly and consistently.

Plex Transcoder is a LIVE service, Netflix is not!

Netflix can process that file your looking at 10 times before you ever are able to request it and on top of that have multiple servers spreading out the sending of data compared to your one!

Your asking why Apples can't taste like oranges because to you they are both fruit that grow on trees!

So they should be the same...They are VERY VERY different!


#16

@coreyzw said:
Any ideas on settings for the transcoder to normalize the transfer? Make it more of a fluid transfer rather than spikey?

Thinking specifically to the "segmented transcoder timeout" and the "transcode default throttle buffer". 

Would increasing the throttle buffer to a quite large amount allow it to write to the temp file for a longer amount of time, or would it transcode that amount and then append the temp file with the large chunk rather than appending live? What about the transcode segments... larger number so that there is a larger buffer in the temp file before the transcoder gets taken out of sloth mode?

Hi Corey, sorry to bring back an old thread but I was wondering if you ever made any headway on this?

I understand the issue you brought up (unlike, it seems, other people in this thread) and I have noticed the same thing with my server. Like you, I am running an enterprise grade dedicated server at a datacenter with plenty of bandwidth.

I have methodically tracked down some buffering problems that exist only on a handful of remote clients and I'm thoroughly convinced the bursting data transmission issue you brought up is the cause.

It seems to be related to the quality of their routed connection to the data center. I came to this conclusion because I have noticed that running a traceroute or ping to my PMS server from these affected client locations that they have more latency than I would like but more importantly there is inconsistency between the initial packet latency and subsequent data transfer. I am not well versed in networking, but it seems like getting the data initially flowing between my PMS server to these clients is difficult but once the initial connection has been made then the data flows "freely." I have replicated this behavior from the client location to other services as well like Amazon Video and Netflix, it seems to be a characteristic of their internet connection as it is not particular to just my server.

If I monitor the bandwidth usage on the server side while transcoding to these clients, I see the burst of data transfer that you experienced. My idea was perhaps due to my initial latency problem, these bursts cause the client to exhaust the data stream and the playback stuttering occurs-- if the data came through normalized as you said, would I still have the buffering problem?

I found that the throughput of the PMS to client connection is plenty high enough-- if I transfer a large file from my PMS server to the same client through another means (like sftp), I will see a fluid transfer on both server bandwidth monitoring tools and client-side, which maintains and far exceeds the bandwidth that is needed to transfer the transcoded file via plex. Monitoring the client-side connection to Netflix demonstrates the same thing, their internet connection is plenty "fast" enough to stream.

Just to be sure, I checked CPU usage, disk i/o, etc. while a transcoded plex transfer was occurring and the data transfer spikes are not caused by lack of server hardware capabilities-- the stream is barely using 2% cpu, memory usage is low, and disk i/o is very reasonable.

Anyways, if you made any headway or have any ideas I would greatly appreciate it. I have ran up against the same misunderstandings that you did in this thread-- people don't seem to understand my/our issue.

I thought of re-encoding my media so it could be direct streamed to these affected clients, but I really don't have the space or want to go through the trouble when it seems that the usual suspects (lack of bandwidth/throughput or cpu) are not causing their buffering problem-- but rather plex's method of segmented transcoding / data transfer to clients combined with poor "network consistency" from their location to internet services.

Thanks for your help and bringing this issue/oddity to light,
Tim


#17

Having the same issues right now. I’m never using more than 12% of my Skylake Xeon CPU no matter how many steams I’m transcoding and I have 32GB of ECC RAM. I’m getting the same spikes when streaming remotely, and they cause a slight pause in the video every time they happen. This is definitely a PLEX thing, so I hope they fix it.


#18

Just read this thread as I was about to post the same issue. I just got a dedicated server from hetzner with 60tb hdd and while I have transcoding speed always above 5 with multiple streams I always have the similar pauses and buffering.

Did anyone figure out how to solve this issue?. Maybe changing the buffer might help?


#19

I'm having this problem as well.. I am streaming from another country.. To my plex server I have a fairly good connection.. About 6 megabits but since plex does spikes instead of consistent streaming it will cause it to buffer every time it hits a spike. If I could just have a small buffer or something it would be fine.. but even with 6 megabits to my server the best I can do is 320kbits a second and even that sometimes lags. Netflix, Amazon video, and even transcoding the file down to a smaller size say 1-2 megabits a second and vlcing it does fine. Any chance plex will be making these transfers more smooth?


#20

Going to Necro this thread!

Same issue from delimiter. Hardware is not even breaking a sweat and the network is WAY more than capable in the aspect of throughput. Ive noticed this happens the worst when doing a direct stream of any size of file. It can be as small as 2mbps and it has problems streaming it. Now, if I transcode a different, larger file from ~30mbps to 4mpbs, it has almost no problem at all.

This is what I don’t understand.