Distributed transcoding for NAS/low power servers

Feature Suggestions
1

Let me start this off by saying this could be a pie in the sky suggestion. I’m under no illusion that geek culture will go to great lengths to get something working, for minimal return. This may be that, but hear me out as it really is for a great number of Plex users with all skill levels of nerdyness.

The Problem: Proliferation of data necessitated proliferation of storage (vice versa?). The NAS appliance has become extremely popular in the past few years, and they’re great at what they were designed to do. Many NAS manufactures have taken it a step further, such as synology and created a fully working, viable and open operating system interface. We can load ‘apps’ and plugins to add functionality. Plex for NAS is one of these. Fantastic! But the first thing many less savvy users find out after loading up their expensive, but sorely under-powered appliance is, the thing is slow and gets bogged down easily. The processor power in any consumer NAS is not great for Plex. Streaming, sure, transcoding no.

The Solution: Distributed Computing
The entire premise of having a NAS dictates you probably have other computers in the local network. These other computers usually are real full fledged CPUs that sit idle for 90% of their life. What if a NAS/low power based Plex server could be directed to use the compute power of known local machines to transcode? This model could massively scale in the right environment. I think it would require some server side tweaks to be able to disconnect the transcode component and point it at an installed client component living on a single or multiple computers.

The Environment:

The Plex NAS server would have a new settings page to manage clients, and the network transcoding. I can think of several settings here that would be needed.

The plex ‘Client’ would be installed. (Probably could be rolled into any of the Plex players and have an on/off toggle.)
The client would touch the server and say I’m ready and have cpu cycles to burn. There could also be a benchmark component so the server could do intelligent distribution if there were more than one client connected. The benchmark would give a rating to the PC so it knew not to throw a 1080p transcode at it if it couldn’t handle it.

The client would receive instructions to encode \NAS\Media\Movies\X.mp4 file in y format. Then stream directly (or through the server??) to whatever player is asking for it. Essentially then the NAS would only be handling the file serving and streaming!
The end consumer would never know the difference. And the NAS wouldn’t have to struggle. And mom, grandma, dad wouldn’t be frustrated that this stupid thing that their son/daughter sold them ‘doesn’t work right’

It would also save tons of money in building a dedicated Plex server. I myself did this, building a Plex + Nas from a high end machine. It works great, but it was costly to put a high-end CPU in there, and still I have a ton of CPU cycles just sitting around on my network. With the distributed model, I could have just built a NAS from parts laying round and then the distributed Plex would use the various high-powered computers I’ve got to do the work.

Have it built by next week? Go! well ok, maybe not. But this would really be a killer feature. Not too difficult to get going either for a normal person. Some good guides/walk-through in the setup process would go a long way in making this a viable solution for many Plex + Nas/ARM/Low power devices. Perhaps a long term feature goal?

My inspiration for this: This morning I was able to get Plex up and running on my Router! sweet! but it can’t do much but serve up streams.

2

If you have your media set up to support your user’s connections (bitrate?) and eliminate transcoding as much as possible, you can run PMS on almost any NAS. I did for almost a year on a small ATOM CPU with under 300 passmarks. By optimizing the media into something that was Direct Playable from the start.

Transcoding is a point of diminishing return. The more media you have that needs transcoding, the more chances you have that more sessions are going to require it at the same time. If things are Direct Playing instead, then it doesn’t matter one bit.

This idea, which is already requested in one or two other requests here, even though not specific to a NAS platform, relies on the idea that you don’t need to worry about how you store the media or what codecs are used to store it. You are relying on converting it for whatever device requests it. You aren’t actively involved in determining how your media plays on your client devices. You are forcing that device to have to work harder because you perceive an advantage using the file container or codecs you are using.

In other words, you are ensuring that your media will need to be transcoded. You haven’t done the job to make sure your device can handle the task you are asking it to do.

Don’t get me wrong. The idea of distributed transcoding is one I fully support. But so is the Plex admin doing whatever he can to ensure transcoding requirements are minimized.

Here are some existing Feature Requests Add your vote by adding a “Like” to the first post: https://forums.plex.tv/discussion/165608/distribute-transcoding-cpu-intense-tasks-to-other-devices
https://forums.plex.tv/discussion/121215/distributed-transcoding-scaling/p1
https://forums.plex.tv/discussion/178320/plex-remote-transcoder-a-distributed-transcoding-backend-for-plex/p1

3

huh… none of my searches found those. but I only looked here.

Your argument… i just don’t know. It hinges on a few assumptions. The first and most glaring one I see is that it boils down to this: “Transcode your media first, then put it on Plex”. Where are you getting the CPU cycles for that? Essentially it would be a process of manual distributed computing. I’m just suggesting to automate it. In my estimation this totally negates the beauty of what Plex is actually designed to do. Sure, be real, don’t fill up your library with uncompressed .avi files or some craptastic video container dejour. But the real world scenario is that you have your media in the highest bitrate possible and let the device/bandwidth determine the stream.

By your reckoning, we should size the media to fit the server and recode for all the different formats you may want to direct stream to. Seems like a waste of time, cpu cycles and space. What happens when you get a new Plex server that can actually handle the transcoding… now you’ve got a bunch of copies of your media that really isn’t useful anymore now that you can do it on the fly.

I think Plex’s logic is solid. One file that can be converted to any device format. With modern codecs media is becoming more stream-able anyhow. We have this great software that can do it very well. I was just trying to solve for the compute power using existing hardware. I know its not a new concept, but it would be that killer feature that really sets Plex apart further.

In the how-to post for this suggestions sub-forum there are explicit instructions to make this fit the most amount of plex users. No niche advanced, or personal requests. This is the definition of fitting the most people/users. Make it simple so that $200 2 disk NAS can actually do what you thought it would. Just enable transcoding on your desktop computer. I don’t need the feature but I think many, if not most casual Plex users only use it as an iTunes streaming replacement. Plex is so much more, and this kind of feature would turn a casual user into a paying Plex Pass Subscriber.

Oh… and thanks for taking the time to respond and think about it. I just think your solution is a work-around and not something that the mom/pop crowd will bother doing or learning. You’re an advanced user, you know how to transcode your media. Most don’t that is the main reason for Plex. It just works.

4

The Plex Team themselves, through designing the Optimize Media feature are already tacitly telling us that transcoding isn’t always the answer that so many believe it to be. A 10Mbps 1080p stream takes roughly 2K passmarks to transcode. Up that to a 45Mbps or higher 4K example and the requirements are drastically increased. (For every 10Mbps, it’s going to take another 2K passmarks.) So that 45Mbps is going to take 9K passmarks to transcode. And a 60Mbps is going to require 12K passmarks. For a single stream.

So when do you reach a point of no return with transcodes? At about 22K passmarks. Because that’s the highest rated CPU made to date. And the CPU itself runs a whooping $3700+ US. That’s not the CPU and board, but just the CPU… You still need to buy a board to handle it, memory to run it with, PS, etc. and you are still going to be limited by using strictly transcoding.

And that same device is only going to be able to handle 2 to 3 4K streams at a time via transcode. But it could likely handle 100 or 200 streams at a time if the media is already in the Direct Playable file and codecs.

As far as where do I get the CPU cycles for the conversions? I get it from the NAS itself. Either through the conversions scripts @cayars has developed and that I modified to fit my device or through Optimize Media feature built into Plex itself. That lowly NAS now does those conversions when it’s not doing anything else CPU intensive.

I’m a computer guy, so I’m not the typical Mom/Pop set up, I know. But I have helped quite a number of the mom/pop people set up a solution for doing automatic transcoding through the scripts that @cayars has a link to in his signature. I took his scripts and set them up to run on the three NASes I have and they work well. Slowly on the two smaller devices, but they still do the job. It took almost 3 hours to convert an MKV to MP4 using the scripts on a less than 300 passmark NAS. For a 2 hour + movie… The point is, though, it still did the job!

With OM, it’s not hard to use those dead cycle times on your NAS to have Plex itself do the conversions, putting the “new” media into a different folder, and using that as the library you share out to others off you local network. You can set your NAS to optimize the media for you, and it will slowly grind through it until the entire library is done. It took me almost 2 weeks to do my entire library the first time I tested it on my current NAS. And that was about 1700 movies with an i3-4330 with 5K passmarks.

If you start with the idea of maximizing your media to Direct Play, BEFORE you have 2000+ movies you can make a process flow that works with a bit of practice that even my 71 yo mother is capable of following.

If this idea comes about, people that are running on a way under-powered NAS now that can’t transcode are going to complain that they have to leave their desktop on in order to watch their media. It’s a no win situation, no matter how you look at it. OM I’m sure came about in answer to parts of the issues people with smaller NASes are having. (Although nothing officially from the Team about why they gave us this when we didn’t really specifically ask for it.)

Relying on transcode means you are relying on something that is going to ultimately fail you at some point. Either fail by not being able to do it because of bitrate requirements or because the client app streams the media faster than the PMS can convert. Or by asking it to transcode too many streams at one time. On slower NASes, you are asking it to send the data to a machine that converts, then send the new data back, so the NAS can send it out to the requesting client device. Some of these smaller NASes might not be able to handle that much data that fast. My first one certainly couldn’t… .

I’m not likely to change your mind. I’m not really worried about it, either. When you start to realize that transcoding has limits, regardless of the device you have to run PMS on, you will likely come to the same conclusion many of the larger operators have already come to. Converting the media before Plex gets it is the only real viable long term solution.

BTW, there is a way to use the OM feature to remotely transcode for your NAS. If you want more details look at this test I ran: https://forums.plex.tv/discussion/comment/1136699/#Comment_1136699 It works beautifully, and you only need to have the PC or Mac running PMS only when you add new media. I’ve tested this pretty well, so using the i7 to transcode for the i3 worked very quickly and made a new library for remote streaming from my original (converted) files.

5

@landcameras, you too are making assumptions.

That it’s possible to transcode all file types at any given time.

That real-time transcodes are going to be as good quality wise as something pre-transcoded.

That people who run Plex servers are dumb and not up for learning how to do things better/more proper.

That distributing the load on different computers is going to solve this problem. In fact with many blu ray rips that use VC-1 (quite common) or h.265 or 4K content you can’t currently get a computer fast enough to be able to transcode these files in real time with high quality.

So the way to solve many of these problems is as @MikeG6.5 pointed out and to transcode (if needed)/remux your media into a specific format outside of Plex before you add the media to Plex. If you do this the quality will always be better than if you had to transcode in real-time.

Carlo

6

Plus there are already many distributed (nevermind cloud/SaaS) encoding solutions for the pre-transcoding…not sure it’s worth investing heavily in something that already has ample options available by simply planning (slightly) ahead.

7

oookk guys. you got me, transcoding will fail, its not good enough, ppl are dumb (?), hardware encoding will never encode a BR disc so whats the point, mom should encode with the cloud.

I’m actually kind of sorry I suggested anything. I didn’t come here to argue about encoding or whatever the point is now.
I saw a fairly straightforward way to decentralize what plex already does, minimize investment and make the product better. That is my only real point, but I see that i’ve touched a sensitive topic.
I’ll step out now.
thanks for taking the time.

8

I come at this problem of needing to transcode from a different angle;
My issues were multiple remote transcode session and available upload bandwidth.

In my home, any TV in the house uses a plex client/hardware that can DIRECT PLAY anything I have in my library (combination of Intel NUC’s and RPI2s). For the occasional viewing using a Smartphone, laptop, or Chromebook the server has enough CPU to handle that when it occurs.

For any family I share with remotely, which based on my bandwidth, would require transcoding 90% of the time I have took the time to upload my content using Cloud Sync and the family that streams remotely only sees the Cloud Sync server. This way any piece of media is transcoded once, during the upload process and my server is not taxed with multiple transcode sessions at once. The only thing remote users lose is multi-channel sound, but in my situation, none of them have surround sound setups anyway so the 720p 4Mbps cloud sync file I share with them is perfect.

9

@landcameras said:
oookk guys. you got me, transcoding will fail, its not good enough, ppl are dumb (?), hardware encoding will never encode a BR disc so whats the point, mom should encode with the cloud.

I’m actually kind of sorry I suggested anything. I didn’t come here to argue about encoding or whatever the point is now.
I saw a fairly straightforward way to decentralize what plex already does, minimize investment and make the product better. That is my only real point, but I see that i’ve touched a sensitive topic.
I’ll step out now.
thanks for taking the time.

Plex can already do what you are asking…
Simply install Plex on as many PCs as you have, Map the network drives from your NAS to each and you will have multiple servers to choose from for transcoding when needed. All it will take is switching to the free server in the app.

There is no reason to duplicate the files for each server, simply have each server access the same central NAS library of movies.

Just because your NAS can run Plex does not mean i is the best way to use plex.
Just as a Screwdriver can hammer in a nail in a pinch it still isn’t better than using a hammer!
The use of NAS’ are great provided you come to grips with the fact they are not really all that powerful to do things like transcode and serve multiple clients effectively.

Serving files requires few CPU cycles and it can serve many files to many transcoders if you just forget about the media server installed as a marketing ploy to get you to buy that NAS.

And If you don’t have the computers to do this then you also don’t have the machines to distribute transcoding either.

And if that’s not an option for you the only thing left is to re-encode your entire library into MKV with Direct playable codecs for video and audio that is compatible with all your target devices.
Usually a video using H.264 and at least one AAC audio codec track will do that,

10

@landcameras I apologize for the “dumb” comment as that wasn’t what I really meant.
What I was getting at is that new Plex admins are usually “un-informed” and will learn as they go. They just “assume” they can put little effort into running a good system and can just put any file type (because it’s supported) and expect their Plex system to work like Netflix or similar. As we learn and grow we become more informed and find out things like real-time encodes are as good quality wise as pre-transcoded versions.

We find out that 1 of the 3 codecs used for blu rays (VC-1) doesn’t support more than one core during transcoding so you can’t fully make use of your CPU. We find out that 4K video is much harder on the CPU than 1080p or 720p resolutions for transcoding. We find out that those 30+Mbit rips are a lot harder to transcode down than a 8 to 12Mbit rip/transcode.

We find out that transcoding a file that started as a HVEC is brutal to the CPU.

It’s a learning process. What many of us have found is that transcoding your media a head of time to something in the 8 to 12Mbit rate (crf 18) will get you super clean transcodes that are much friendlier and easier to use/stream and are undistinguishable from the originals. If you then need to transcode on the fly to a cell phone or to sync you can start with this much smaller file which is much easier on the CPU.

Or another great way is what @hthighway described which can provide streamable files to friends/family without using your own bandwidth (besides the initial upload).

Now with all that said, I DO think your idea is a good one. If you skim through the thread in my SIG you will see I even started working on this and already did it for Emby. The problem is that the quality still isn’t as good so I more or less stopped working on this and focused on the pre-transcode side of things which give better quality and allows me to run on moderate hardware using only one computer.

There is also another thread you might want to read: https://forums.plex.tv/discussion/178320/plex-remote-transcoder-a-distributed-transcoding-backend-for-plex/p1

Carlo

11

duplicate feature request

closing

Please respect the rules for this subforum

  • Search first if the request has already been posted before. The goal is to collect the votes for a particular feature in one thread - not to scatter them around.
  • Please only one feature per post