For normal stuff like reading files (playing an mkv) its exactly the same speed. The random access that plex scan uses is not as fast
Late to the party, but I would have guessed DT only because I was lurking on the emby forums when you were discussing it. If you are going to want someone to help test the Plex variation count me in!
Looks like Plex soon will have a new employee? :lol:
Sounds very intresting, might be usefull for me in a few weeks/months
I may go back and look at it again in the future when he does add this. I'm always open to testing and trying new stuff (actually enjoy it). But for right now I'm happy with DrivePool and SnapRaid combo.
I actually use Drivepool + Flexraid snapshot for over 3yrs now and Flex has saved my data on more then one occasion. best investment i have made
Distributed Encoding is the correct answer.
Like I said, I have this working in a prototype mode for Emby which directly uses ffmpeg. Plex uses a modified version of ffmpeg with a different name but should roughly work the same.
In a nutshell it will work similar to Emby and this is the "setup" on that platform at present: To use this presently you would just rename ffmpeg.exe to ffmpegOriginal.exe (or similar) and then drop in a replacement ffmpeg.exe I've created. The drop in replacement is basically just a "proxy" that takes the command line options sent to it and hands them off to ffmpeg.exe running on another computer (or the server). It watches stderr, stdin, stdout and shuffles these messages to and from the actual ffmpeg and the drop in Proxy. Plex should not even be aware it's not "talking" to the real transcoder.
As long as the 2nd, 3rd computer have the same drive mapping as the server to the transcode directory and the media itself everything seems to work. You'll of course also have to copy the Plex transcoder over to these additional machines from the server.
All of the above presently work (for Emby). Should just need to change names to use the Plex Transcode EXE and it should roughly work (might need a tweak here or there). What I need to do to go from prototype to real working edition is to figure out how I want to monitor and figure out which computer to run the transcode on. This could be as simple as monitoring the CPU use on each computer and using the lowest.
Where it will get interesting is to be able to wake up computers on the LAN via WOL functionality to dynamically bring online additional transcoders when needed.
For Plex this should work for real-time streaming, cloud sync, device sync, Bif/index file creation or any other time the transcoder is called.
I like this approach to scaling out more so then trying to run multiple Plex servers as there is only one DB, one set of meta-data and no need to "sync" data among the servers.
What do you guys think?
Carlo
PS at present it's only "proto-typed" for Windows machines but i don't see why I couldn't do this for Linux and other platforms also. Could be useful for those running Plex on a NAS as they could wake a computer on the LAN to do the transcoding.
This is the best news i have heard in a long time :)...I only use Emby Locally right now and have started with plex for remote access for my users and offline sync. I love Emby but like you, i find that the main focus is still too much on the server and not enough on client apps(which seem to always be playing catch up to the server) and there sync option is not mature enough for my daily usages. But the major thing was the transcoding was pining my CPU. Im still testing out Plex (only been 2 weeks)but it looks promising. If you were to bring a feature like this to the table for both Plex and Emby, you will be the savior of many
Surely you kid ;)Looks like Plex soon will have a new employee? :lol:
Sounds very intresting, might be usefull for me in a few weeks/months
Curious, since you are using Flexraid for the snapshot how come you use Drivepool instead of the pooling in Flexraid? But yea I totally agree having a parity(s) drive you SYNC at will can have many benefits of typical RAID with parity striping. The former can allow you to undelete files while the latter is of no help at all in this situation.I actually use Drivepool + Flexraid snapshot for over 3yrs now and Flex has saved my data on more then one occasion. best investment i have made
I don't know about you guys but I destroy much more data the failed drives ever have. Nothing like that feeling when you know you just deleted the wrong directory (and everything under it) when you are in a hurry trying to get something done quickly!
This is the best news i have heard in a long time :)...I only use Emby Locally right now and have started with plex for remote access for my users and offline sync. I love Emby but like you, i find that the main focus is still too much on the server and not enough on client apps(which seem to always be playing catch up to the server) and there sync option is not mature enough for my daily usages. But the major thing was the transcoding was pining my CPU. Im still testing out Plex (only been 2 weeks)but it looks promising. If you were to bring a feature like this to the table for both Plex and Emby, you will be the savior of many
For a period of time I was gung-ho on Emby/MB3 as it had a few features I wanted Plex to have. But overall it's still missing more than Plex for daily use and it's no where near as "smooth" as Plex is. Not to mention it doesn't have nearly as many usable clients. I do use it however for PVR integration and for casting to DLNA devices which is a shortcoming of Plex IMHO.
I actually first started with trying to bring GPU transcoding to the table but ffmpeg just isn't there yet for either QuickSync or NVENC so I started playing with the distributed method. It works well for Emby but is "hard coded" specifically for my system and has no intelligence for knowing which computer to run the new transcode on. I stopped playing with because Emby was having lots of issues with hung transcodes and not stopping/pausing transcodes when the user abandoned a session. I figured I'd let the devs clean that stuff up first.
I then sort of lost interest and put my focus back on Plex. The grass isn't always greener on the other side. :)
Probably in about a week or two I'll hit this project up and put a good amount of time in it. I had planned on doing it right away but "work" dropped a project on me (right after I had posted about it) that a team of devs had been working on for a couple of months and can't get it done. Gave me some incentive $ to get it done by EOM so...
It will only be on the back burner for a bit. In the mean time we can talk about how you guys would like it to work. IE how it should schedule the jobs. Obviously there will pretty much always be two (Plex server and at least one other computer) or there is no reason to use this. So let me know or give me ideas how you envision it working. I know what I'm thinking but I'd like to get feedback and ideas from you guys.
Carlo
Personally I'm thinking either one or two or an array of NUC-like devices. They'll sit there and sleep and when either a master NUC or something else running Plex needs a transcode it'll wake one of them up and start a transcode. After X amount of time it'll go back to sleep. This would not only allow me to make something that uses far less electricity, but it'll scale out better than Plex otherwise does now. That and a handful of NUCs could potentially be cheaper than a bleeding edge i-something CPU which would also be advantageous from a cost perspective. I can flesh this out a bit more but off the top of my head that's a rough draft way of me seeing this working.
Personally I'm thinking either one or two or an array of NUC-like devices.
Is 10 too many? :)
http://www.tranquilpcshop.co.uk/ubuntu-orange-box/
Sent from my HTC One_M8 using Tapatalk
Not at all! Granted I'm thinking smaller scale than that.
I hate flexraid for pooling.. It is way less effective compared to drivepool and also forces you to use cruise control instead of expert mode.Surely you kid ;)
Curious, since you are using Flexraid for the snapshot how come you use Drivepool instead of the pooling in Flexraid? But yea I totally agree having a parity(s) drive you SYNC at will can have many benefits of typical RAID with parity striping. The former can allow you to undelete files while the latter is of no help at all in this situation.
I don't know about you guys but I destroy much more data the failed drives ever have. Nothing like that feeling when you know you just deleted the wrong directory (and everything under it) when you are in a hurry trying to get something done quickly!
For a period of time I was gung-ho on Emby/MB3 as it had a few features I wanted Plex to have. But overall it's still missing more than Plex for daily use and it's no where near as "smooth" as Plex is. Not to mention it doesn't have nearly as many usable clients. I do use it however for PVR integration and for casting to DLNA devices which is a shortcoming of Plex IMHO.
I actually first started with trying to bring GPU transcoding to the table but ffmpeg just isn't there yet for either QuickSync or NVENC so I started playing with the distributed method. It works well for Emby but is "hard coded" specifically for my system and has no intelligence for knowing which computer to run the new transcode on. I stopped playing with because Emby was having lots of issues with hung transcodes and not stopping/pausing transcodes when the user abandoned a session. I figured I'd let the devs clean that stuff up first.
I then sort of lost interest and put my focus back on Plex. The grass isn't always greener on the other side. :)
Probably in about a week or two I'll hit this project up and put a good amount of time in it. I had planned on doing it right away but "work" dropped a project on me (right after I had posted about it) that a team of devs had been working on for a couple of months and can't get it done. Gave me some incentive $ to get it done by EOM so...
It will only be on the back burner for a bit. In the mean time we can talk about how you guys would like it to work. IE how it should schedule the jobs. Obviously there will pretty much always be two (Plex server and at least one other computer) or there is no reason to use this. So let me know or give me ideas how you envision it working. I know what I'm thinking but I'd like to get feedback and ideas from you guys.
Carlo
Do you use unc path with your setup and what do you use as scanning method?
I find emby as smooth as plex playback wise but you are right that the client choice is limited because most clients are not really stable. The only real stable and fast client is emby for Kodi, emby for Mce and Roku. The web client is good also but that is because lots of work was put into the server. Right now my only beef with plex is the lack of capabilities to scan library when new content is added (not entire library.. Only the show with new content) from unc or mapped drive.
I was thinking the opposite.Personally I'm thinking either one or two or an array of NUC-like devices. They'll sit there and sleep and when either a master NUC or something else running Plex needs a transcode it'll wake one of them up and start a transcode. After X amount of time it'll go back to sleep. This would not only allow me to make something that uses far less electricity, but it'll scale out better than Plex otherwise does now. That and a handful of NUCs could potentially be cheaper than a bleeding edge i-something CPU which would also be advantageous from a cost perspective. I can flesh this out a bit more but off the top of my head that's a rough draft way of me seeing this working.
Surely this could be done. I wonder how well they would hold up to transcoding as their main purpose in life? I would think these NUCs would need to have good cooling.
Is 10 too many? :)
http://www.tranquilpcshop.co.uk/ubuntu-orange-box/
Sent from my HTC One_M8 using Tapatalk
Almost reminds me of blade servers. Problem with this approach is you won't have enough LAN throughput.
I hate flexraid for pooling.. It is way less effective compared to drivepool and also forces you to use cruise control instead of expert mode.
Do you use unc path with your setup and what do you use as scanning method?
I use drive letters so Plex will pickup the last modified date (doesn't work with UNC).
What you do you mean by "scanning method?
Periodic, auto update or auto update+partial update. Ok so I need mapped drive to get it to work. Right now it doesn't trigger at all when I add content content to tv folder or if it does, it scans all the Tv libraryI use drive letters so Plex will pickup the last modified date (doesn't work with UNC).
What you do you mean by "scanning method?
I had switch my tv shows to a mapped drive and all i was getting is full scans. Then I did a optimization of DB and restarted and no more scan at all when I add contentPeriodic, auto update or auto update+partial update. Ok so I need mapped drive to get it to work. Right now it doesn't trigger at all when I add content content to tv folder or if it does, it scans all the Tv library
On my system it does pickup that I added new content but it doesn’t scan just the libs that were updated. It scans the whole library which it shouldn’t.
To be honest I really don’t care much for my system. I have it set to re-scan libs every 6 hours which is pretty much the sweet spot for me. I have it set not to scan music however as I’ll do that manually.
Right now I still have a lot going on behind the scenes. I’m uploading to ADC (23TB and still going) and I’m presently adding TV Shows to my new pool.
I’ve got Indexing running rebuilding BIF/Index files and I’m converting some files I thought were h.264 and web optimized but weren’t (not optimized or mpeg4).
I figured this out with another “add-on” utillity I’ll release in a bit that works hand-in-hand with the auto converter published earlier.
Carlo
On my system it does pickup that I added new content but it doesn't scan just the libs that were updated. It scans the whole library which it shouldn't.
To be honest I really don't care much for my system. I have it set to re-scan libs every 6 hours which is pretty much the sweet spot for me. I have it set not to scan music however as I'll do that manually.
Right now I still have a lot going on behind the scenes. I'm uploading to ADC (23TB and still going) and I'm presently adding TV Shows to my new pool.
I've got Indexing running rebuilding BIF/Index files and I'm converting some files I thought were h.264 and web optimized but weren't (not optimized or mpeg4).
I figured this out with another "add-on" utillity I'll release in a bit that works hand-in-hand with the auto converter published earlier.
Carlo
Well at least im not the only one that has the scan problem...for the conversion, is there any quality lost and is audio still 5.1?
I was thinking the opposite.
How so?
I was thinking it could be cool to have a NUC running Plex (storage on NAS or other server) and have it wake up power hungry i7/XEON computers when needed to handle the “overload” on transcoding. Assumes you can run Plex on the NUC.
The only “obstacle” I see for using NUCs as transcode “agents/clients” is that they really aren’t designed to handle high CPU use for long periods of time and may not like the higher temps and it could shorten their life.
Although from the “distributed transcodeing” standpoint the utils could care less what hardware they are running on.
@cayars
How does your conversion script handles sub and audio that are tagged as unknown instead of english?
I was thinking it could be cool to have a NUC running Plex (storage on NAS or other server) and have it wake up power hungry i7/XEON computers when needed to handle the "overload" on transcoding. Assumes you can run Plex on the NUC.
The only "obstacle" I see for using NUCs as transcode "agents/clients" is that they really aren't designed to handle high CPU use for long periods of time and may not like the higher temps and it could shorten their life.
Although from the "distributed transcodeing" standpoint the utils could care less what hardware they are running on.
Fair enough. I also don't have nearly the users you do and my system has more downtime so I'm looking at it from an energy usage and cost savings perspective even when the system is on. I haven't yet built a dedicated Plex box, my year of Plex has been upgrading my in-home network and building a NAS. If you actually build a viable way of doing distributed transcoding it'll likely influence my Plex server build. Looking forward to playing with what you come up with.
I do think I could move my Plex install to my unRaid box and then have it wake my desktop up if/when you add WOL functionality. That alone would be hugely beneficial. So there's probably a whole pool of use cases for remote transcoding besides the whole distributed part.
