distribute transcoding/cpu intense tasks to other devices

Feature Suggestions
21

well, helper, server, client or even assistant, it doesn't matter what you call it right?

my point is,

1. it doesn't have to be a ''server/NAS'', or full blown PC to offload transcoding, other devices, like openelec nuc, or even powerful arm tv boxes could also contribute

2. there has to be some sort of selection, like every 'helper/server/client' or whatever reports it's own computational power and (you made a good point) cpu idle status, as well as LAN connection status. 

Well I was sort of suggesting with the phrase "Distributed computing" that all available computers might be able to contribute transcoding at the same time thereby reducing the total power required by any one unit.

So if there were 5 computers or NUCs on the network, no one unit would have to transcode the entire file by itself.

Server would send data to multiple available machines, and then reflect the transcoded data onto the client.

But it is probably a bit overkill for what we are talking about.

22

That is probably not possible without a rewrite of parts of Plex.

Right now the way the transcode engine works is that it takes a small part of what the user is watching and hands it off to the transcoder to process. It also tries to stay X seconds a head of the user (config settings). It doesn’t spread out what needs to get processed or start multiple threads for each video. It does however start a different transcode process for each /client/video being transcoded.

It’s these separate videos processes that we can pick off and run on another computer. As an example if you have a total of 4 computers including the server available to transcode but there are only three clients watching something that needs the transcode engine then at most 3 of the 4 computers will be doing something. There is no work for the 4th computer. In reality however this would probably get handled by only two computers. Even two slower computers can probably handle 3 total streams. It just depends on how often each transcoder is busy or paused.

Carlo

23

Yes the practicality coding wise of doing 1 stream split between 2 clients isn’t practical.


You can’t really do 60% from 1 transcoder and 40% from another.


So this rules out any clients that can’t transcode a file. And tvs are usually turned off when not in use (at least in my household)


Home pcs though are usually always on, 3 windows/apple pcs with kids etc isn’t that abnormal to have in a typical plex household. Plus most plex users are typically pretty tech savvy and so have more hi powered clients/servers


Sent from my SM-N910F using Tapatalk

24

Yes but in the end when you get right down to it what the current system is doing is creating transcoded packets that get sent in a stream...

What machine creates that packet should not matter to the server or end user, only that the packet has been completed and returned on time.

For a 2 Hour Movie using multiple CPUs could in theory finish transcoding the entire movie in under 2 hours leaving the CPUs free to take on other transcoding tasks for other clients.

But I agree it is making something that can be done very simply more complex and perhaps more complex than it has to be...

But if the idea is increase efficiency distributed computing is the path of the future. Why do you think we all have multicore CPUs in our PCs these days....Pretty much the same concept.

25

I would just love it if some coding wizard took the current "Plex New Transcoder", and re wrote it as a TC "Manager" to both perform onboard transcoding with the additional ability to hand out chunks of a TC job for whatever processing power is available on the LAN.

I could just imagine it running like "Folding@home". Then your clients could have a checker that looks at capability/ availability of themselves, and request work from the manager as apropos. Since that guy most definitely isn't me, any volunteers?

26

we

That is probably not possible without a rewrite of parts of Plex.

Right now the way the transcode engine works is that it takes a small part of what the user is watching and hands it off to the transcoder to process. It also tries to stay X seconds a head of the user (config settings). It doesn't spread out what needs to get processed or start multiple threads for each video. It does however start a different transcode process for each /client/video being transcoded.

It's these separate videos processes that we can pick off and run on another computer. As an example if you have a total of 4 computers including the server available to transcode but there are only three clients watching something that needs the transcode engine then at most 3 of the 4 computers will be doing something. There is no work for the 4th computer. In reality however this would probably get handled by only two computers. Even two slower computers can probably handle 3 total streams. It just depends on how often each transcoder is busy or paused.

Carlo

well, i think it doesn't matter how many 'movies' you are watching at the same time.   as i observed (correct me if I'm wrong) plex doesn't TC the entire movie at once. instead, it cuts movies into  many smaller 'clips'.    so as far as plex concerned, you are not watching 1 single movie, you are watch 100 different clips. it doesn't matter whether those 100 clips are 'related' to each other, all it matters is that the server finds the file (101010....) for that clip and hands it to the transcoder. 

27

Yes the practicality coding wise of doing 1 stream split between 2 clients isn't practical.

You can't really do 60% from 1 transcoder and 40% from another.

So this rules out any clients that can't transcode a file. And tvs are usually turned off when not in use (at least in my household)

Home pcs though are usually always on, 3 windows/apple pcs with kids etc isn't that abnormal to have in a typical plex household. Plus most plex users are typically pretty tech savvy and so have more hi powered clients/servers

Sent from my SM-N910F using Tapatalk

1. why you can't split 1 videos for 2 transcoders to decode?  it's just many smaller clips.  (maybe i misunderstand you?)

2. well, I'm not talking about tv's. tv's are usually equipped with low-power cpus that are not capable enough anyway. what I'm saying is tv boxes, like apple tv (though i know it's not possible to do anything to it), or other tv boxes.   those devices are usually left on, or at least standby right?   for some of them you don't even have a real power-off option, the best you can do is standby. but pc's, they do have power off. 

anyway, pc or box, doesn't matter, the point is distributing.

28

we
 
 
well, i think it doesn't matter how many 'movies' you are watching at the same time.   as i observed (correct me if I'm wrong) plex doesn't TC the entire movie at once. instead, it cuts movies into  many smaller 'clips'.    so as far as plex concerned, you are not watching 1 single movie, you are watch 100 different clips. it doesn't matter whether those 100 clips are 'related' to each other, all it matters is that the server finds the file (101010....) for that clip and hands it to the transcoder.

More or less yes, you understand it. But what my util is doing is intercepting the PlexTranscoderNew.exe and replacing itself with a "director" that knows how to funnel the requests to a 2nd or 3rd computer. The part you can't overlook however is that without a fundamental change to the Plex server itself we have constraints we have to work with. In this case, Plex thinks it's talking to the transcoder and is handing off the one small piece to get transcoded. Until this piece is done the Plex server will not send another piece to get worked on.

So each "client" transcoder has to be able to transcode at least at a speed of 1.x or you can run into problems.
 

1. why you can't split 1 videos for 2 transcoders to decode?  it's just many smaller clips.  (maybe i misunderstand you?)
 
2. well, I'm not talking about tv's. tv's are usually equipped with low-power cpus that are not capable enough anyway. what I'm saying is tv boxes, like apple tv (though i know it's not possible to do anything to it), or other tv boxes.   those devices are usually left on, or at least standby right?   for some of them you don't even have a real power-off option, the best you can do is standby. but pc's, they do have power off. 
anyway, pc or box, doesn't matter, the point is distributing.


1. This would need a change in the server. At present it assumes a speed of at least 1.x to 1 or it will cause stuttering.
2. My util is only going to focus on real computers that can run ffmpeg or PlexNewTranscoder on them at better than 1.x speed.

While I'm sure distributed encoding could be done numerous ways, I'm only going to focus on Windows machines to start with then Linux computers. My goal isn't to make use of low power CPUs but to lead the way in building out big Plex server environments where you can stream transcoded material to 20 clients for example. With 3 i7s computers for example you could handle a huge "personal" streaming load (not commercial) as well as build BIF/Index files, sync files etc at the same time while reserving CPU cycles on the main Plex server.

That's what I envision anyway. The replacement of having to try and run 2 or 3 different Plex servers to handle the load...

Carlo

PS Now if you were talking in general and not what I'm currently working on then just ignore this. :)
29

More or less yes, you understand it. But what my util is doing is intercepting the PlexTranscoderNew.exe and replacing itself with a "director" that knows how to funnel the requests to a 2nd or 3rd computer. The part you can't overlook however is that without a fundamental change to the Plex server itself we have constraints we have to work with. In this case, Plex thinks it's talking to the transcoder and is handing off the one small piece to get transcoded. Until this piece is done the Plex server will not send another piece to get worked on.

So each "client" transcoder has to be able to transcode at least at a speed of 1.x or you can run into problems.
 

1. This would need a change in the server. At present it assumes a speed of at least 1.x to 1 or it will cause stuttering.
2. My util is only going to focus on real computers that can run ffmpeg or PlexNewTranscoder on them at better than 1.x speed.

While I'm sure distributed encoding could be done numerous ways, I'm only going to focus on Windows machines to start with then Linux computers. My goal isn't to make use of low power CPUs but to lead the way in building out big Plex server environments where you can stream transcoded material to 20 clients for example. With 3 i7s computers for example you could handle a huge "personal" streaming load (not commercial) as well as build BIF/Index files, sync files etc at the same time while reserving CPU cycles on the main Plex server.

That's what I envision anyway. The replacement of having to try and run 2 or 3 different Plex servers to handle the load...

Carlo

PS Now if you were talking in general and not what I'm currently working on then just ignore this. :)

yeah i get, and agree with, what you are saying (mostly), since we are just 'outsiders', there is many restrictions/limits without a big change inside plex itself.   we you are doing is actually a very early stage of this, just proving that distribution of transcoding is possible. the rest i believe has to be done by someone inside plex.

like you said, if you replace the transcoder.exe with 'distributor.exe', then the listing server will not be doing transcoding. well, if we have help from plex, this should be resolved very easily. it only needs plex to change it process logic. like on the listing server, we have at least a "distributor", and optionally a transcoder, so when the transcode starts, server passes the file to distributor first, and then, distributor sends clips back to transcoder on the very same computer, plus other transcoders is there is any.

what i don't quite understand is why at least 1x speed is need? is it just the problem we have at this stage, or is it a logical problem?

from my understanding, let's say i have one server with 0.6x capability, and plex splits clips into 1 minute segments. now before playing, i need to wait 100 seconds for the first minutes to be transcoded and then send back to me. and after 1 minute's playback, i will wait an extra 40 second for the second minute's clip to play.

but if i have two 0.6x transcoders, plex will further split this 1 minute clip into two 30 seconds clip. and then, i still wait 100 seconds, and start the first 30 second's playback. indeed, but after this first buffering, i will not need to buffer any more. because for the second 30-s clip, the second server is taking 60-s to transcode (which is more than enough). and for every 30 seconds afterwards, the transcoder has 60 seconds to transcode it. so, no more wait.

30

Maybe not the place to ask this, but will you be releasing source to your tool? That way if people want to play with it a bit more on lower power cpu's they can.

31

what i don't quite understand is why at least 1x speed is need? is it just the problem we have at this stage, or is it a logical problem?
from my understanding, let's say i have one server with 0.6x capability, and plex splits clips into 1 minute segments. now before playing, i need to wait 100 seconds for the first minutes to be transcoded and then send back to me. and after 1 minute's playback, i will wait an extra 40 second for the second minute's clip to play.
but if i have two 0.6x transcoders, plex will further split this 1 minute clip into two 30 seconds clip. and then, i still wait 100 seconds, and start the first 30 second's playback. indeed, but after this first buffering, i will not need to buffer any more. because for the second 30-s clip, the second server is taking 60-s to transcode (which is more than enough). and for every 30 seconds afterwards, the transcoder has 60 seconds to transcode it. so, no more wait.

Well in the early stages of thinking my thoughts are this. If we are basically re-routing a transcode to another machine Plex will not know about this. It will think it's being done on the server itself. It will follow all the "normal" checks and whatnot and will expect the transcode to work at least at 1.x speed. It won't issue the next set of transcodes until the first are complete. So even ifyou have 2 or 3 "machines" available that can process at say 0.75 speed they will not get used if there is say only one transcode in process.

In the real world if you have a NUC or two that could be used that are close but not over 1.x speed they may still be able to get used if the "server" can pick up the slack in between the NUC jobs. Any way you cut it the transcoding has to stay ahead of where the client is or there will be buffering/stuttering issues.
 

Maybe not the place to ask this, but will you be releasing source to your tool? That way if people want to play with it a bit more on lower power cpu's they can.


Haven't thought that far ahead to be honest. Thinking out loud: If it turns out to be very complex and time consuming I could reserve it to "selling" it at say $10 to $20 per client with a license key and of course not release source code.

However, I really doubt that will be the case but reserve the right to do that. In most likely terms I'll keep it closed source for a release or two then most likely release it open source so it can be expanded and other people can chip in dev wise.

If you look at the link in my sig you'll see I try and share info with others and to be honest don't see this being any different. So my expectation would be open source at this point in time.

Wishy-washy answer, but hopefully answers the question at this point in time the best I can based on what I know now.

Carlo
32

So I guess the real issue (if I read Cayars correctly) is that Plex will not send the second part to the transcoder until such time as it received the first completed transcode.

But it will start sending the first part to the client before it is completed...

And there is no feasible way to have two PCs working on the same part.

At least not without a re-write of Plex Base.