Distributed Transcoding/Scaling

[jreinhoud]

I would like to add, that not only load ballancing for transcoding would be nice, but also for upload bandwith. I want to setup an extra server on a different location with mirrored movie database (functions also as backup of my movies and if 1 server is offline plex still would be working). If plex could make a decision which client is using which server based on available resources and available bandwith that would be great.

That way if no transcoding is required you can get 8 - 10 streams with 2 servers depending on upload speed and video quality of course.

[someCynic]

Wow, 6 years and this thread is still active.

Now, it may not fit everyone’s needs, but I have built something for myself that accomplishes part of what has been described in this thread, and may be of use to someone or give some ideas.

In my case I decided to shoot myself in the foot, and instead of buying proper standalone hardware, I’m running most of my apps at home on a Raspberry Pi 3 cluster of half a dozen nodes, using Docker Swarm. And that includes running PMS on it as well (I know, I know, It was mainly to see how far I could push the hardware and experiment with it).

So in order to be able to use multiple nodes for transcoding, I’ve basically built 3 things:

1. A custom Docker image based on the official linuxserver ARM image, in which I rename plex’s own transcoder and put a shim instead to call a NodeJS app, which ultimately communicates to an Orchestrator over websockets.

2. An Orchestrator (node application running in its own container which receives all transcoding requests from PMS and forwards it to one of the active Workers available over websockets.

3. Worker docker image based on the PMS image as well, with another custom NodeJS app running on it, which receives requests from the Orchestrator and kicks off the transcoding and reports progress back. Workers can come online or go offline and the Orchestrator manages their registrations and availability. These Workers run as replicated services managed by Swarm.

The advantage of all this, is that upgrading Plex when a new version comes out is basically just rebuilding the docker images to get the update. I’m not making any changes to the actual Transcoder or writing my own so everything just works whenever a new version is released and settings change.

Now, for all this to work, I actually have the Plex’s Application Data and transcoding folders running off shared storage over NFS (mounted on all nodes) and the Media Libraries are all in a NAS (also mounted in each node). So all nodes are able to mount the same docker volumes and access the same data in the same locations.

I know it’s overkill to build something like this when I could just buy one single powerful server and call it a day, but it was fun to build, and see if it was actually possible to offload work without breaking PMS.

The code for all this is in GitHub if someone wants to experiment:

pabloromeo/clusterplex

And the Docker images for ARM are here:
https://hub.docker.com/repository/docker/pabloromeo/clusterplex/tags

Unfortunately I haven’t written documentation on it or examples of how to configure it, but if anyone finds it interesting and with a similar setup I can try to find the time and at least include my current swarm stack configuration settings.

One thing I wish Plex would support, is actually processing more than one item at a time from the queue for Optimized Versions/Conversions. If it supported that, the cluster could optimize things in parallel quite efficiently instead of just handling one at a time.
I created a Feature Request for this, so if you can, please vote : Optimize / Transcode content in parallel

[Maxou44]

There is also UnicornTranscoder to make Plex Great Again

[Release] UnicornTranscoder, create a Plex Transcoding Cluster

Hello everyone, Today we are releasing the UnicornTranscoder, it’s a project composed of 3 software to create a fully scalable Plex transcoding cluster. The project is written in NodeJS. The three components are: UnicornLoadBalancer: It is installed in front of Plex Media Server, it will catch the request and forward it to a UnicornTranscoder UnicornTranscoder: It is the transcoder software, it will get the transcoding parameters from Plex and start the Job. UnicornFFMEG: It’s a binary to…

[CatKing]

This would be totally sick. I could use one of my old mining rigs to offload transcoding to.

[flecom]

@someCynic

Wow, 6 years and this thread is still active.

ya no kidding, I’m glad to see people ran with the idea

[nydave69]

@flecom

ya no kidding, I’m glad to see people ran with the idea

Sad thing is if you go through the Feature Suggestion list you will find a lot of them that are that old with no Plex activity or response.

[holefx]

Wow. ive been chasing this idea around the internet for a few days and here i land have my ovote for this feature! This would ease my pain with upgrading servers. i can balance it out over the active gamer rigs in my house!

[holefx]

is there a way for newbs to make this work?

[frankpanduh]

Howdy, it would be rad if you could use multiple Plex server instances on different machines to pull the resources of other machines to speed up transcode’s and analysis.

Similar to this:
https://github.com/UnicornTranscoder

Why some NAS devices are really slow, but if you have a cluster of pi’s or other more powerful devices on your LAN network it would be really rad if you could pull more power for tasks by distributing the workloads to other machines. Like even making a worker node companion software would be nice so you don’t need as much headroom to break up the tasks.

Just allow setting available worker tasks priority and limits based on device.

Example:
-Main Older workstation with Networked drives. (Main rig)

• Raspberry 4 (worker over lan for cpu tasks)
• Windows Gaming PC (GPU and CPU tasks)
• NAS unit (CPU + GPU tasks)
• Mac Mini (CPU + GPU tasks)
• Remote Server connections with Public SSL keys only (passwordless) and TLS only for workers on cloud servers for offloading on vps severs.
• TPU offloading (usb TPU testing like google coral)

Mostly think it would give people with older hardware more options to speed up setups and also give the ability to process a lot more data faster for people with more powerful hardware on the same network or distributed network.

Bonus points:
Allow users to use Plex version of FFmpeg or test local upstream versions of FFmpeg.
Why people can compile it with cpu features they need or just testing with faster updates between releases.

[desi76]

@frankpanduh

Main Older workstation with Networked drives. (Main rig)

• Raspberry 4 (worker over lan for cpu tasks)
• Windows Gaming PC (GPU and CPU tasks)
• NAS unit (CPU + GPU tasks)
• Mac Mini (CPU + GPU tasks)
• Remote Server connections with Public SSL keys only (passwordless) and TLS only for workers on cloud servers for offloading on vps severs.
• TPU offloading (usb TPU testing like google coral)

Another benefit of a distributed transcoding and scaling cluster is the ability to downscale the cluster. This would allow the more robust PCs to power down when not needed, with the low-power SBC serving as the frontend for the cluster and only calling or waking the more robust servers when necessary.

An example would be laying in bed listening to a podcast. You don’t need your high-end power-hungry server to stream a podcast.

[seanpv]

The main problem with this is you’re essentially needing to send key frames to all the nodes, decode, send them back and then reencode on one system anyways. The fact that’s it’s compressed video doesn’t lend itself to this kind of distributed system. Not to say you couldn’t do it, but it would not be efficient and it would need to be overly complex to manage and setup. It would be cheaper just to build a better server.