Cayars - Setup walk through and some tips and tricks

General Discussions
41

Here's another useful thing to do:

Install the Unofficial Appstore channel.  Information is located here https://forums.plex.tv/topic/25523-unsupported-as-in-totally-unofficial-appstore/

Once installed you can use it to install other useful Apps and Channels.

The Channel I'd suggest installing right off the bat is FindUnmatched.

Right after installation you will need to go through the very minimal setup of the channel.

But once installed you can now use this App/Channel to walk through different libraries you have setup.  It will identify any media you have on disk that is not showing up in a library.

This is probably the easiest way to identify problematic files or bad naming of media files.

Another really cool App that can be installed via the Unofficial Appstore is Plex2CSV which is also by the same person as FindUnmatched (dane22).  This little app will allow you to create CSV files of each of your libraries.  This is a great way to track offline the media you have.

Carlo

42

My use of PVR might be different than yours.  I did post what my objectives were so you can see where I'm coming from in my "integration" needs.  For the most part my PVR recordings are throw away.  I record a couple of local channels for NEWS in case anything is going on I might want to catch up on.  Typically these shows don't get watched much.  

 

I record a lot of sports shows and typically allow them to get deleted after a while.  I do have an extensive NFL library but these are "condensed" games I capture directly from the NFL NOW package I subscribe to.

It looks like our use is similar for PVR  I use it for news so that I can breeze through it later.  My problem is at the PVR - Time Warner has ALL content protected.  I've been using WMC to record, but I'm unable to strip out commercials and convert to a plex-friendly video format...  Major bummer.

I've had no issues with drives falling off.  The wait when they have to be awaken... the only irritant. 

The limitations of the music management totally suck!!!!  I would love for them to work with mediamonkey and create some type of hook to access what I get from them (the playlists, search and generation of playlist on-the-fly).  To utilize the playlist capabilities they recently added are just too many hoops to jump through at this point, since I don't use itunes...

43

Yea, you're kinda screwed as far as PVR use goes if they have it locked down.  Here on Verizon FIOS they only lock premium channels like HBO, etc

The only work-around is to use a Media Center Extender or "cloned" VMs.  I have Media Center running in a VM that I cloned.  So now I can take a few TV shows/movies recorded on the locked channels with me on my laptop.  I just fire up my Cloned VM and it works fine.  My daughter who is away at a University has a few shows I record for her.  She uses a cloned VM also and I have SYNCing setup with a special directory her stuff gets dropped into so she can watch it. That of course isn't going to cut it for integration into Plex.

Depending on how much you want unlocked PVR use you might want to see if there is another local cable provider or think about switching to satellite just for the TV.  Of course for news you might be able to pickup an antenna and record over the air broadcasts if they are available in your area.  Personally I'd complain to them if the same channels are available over the air!

I hate Plex's Music management or lack of...  By far the weakest link in Plex.  I have 8,686 albums according to Madsonic (Subsonic fork) but only 4,936 according to Plex.  Plex doesn't see 1/3 of my music.

If you haven't checked out Madsonic I'd give it a look.  You should be able to run both Madsonic and Plex on the same box.  Just point Madsonic at your music library. It supports many more formats and is built for music.  It will probably due everything you want Plex to do for music.

Madsonic is really great for music.  They of course have apps for IOS, Android, Rokus, etc as well as a web interface.

44

It's been a while since my last post.  Been up to a bunch. :)

So today's topic is file formats, transcoding and remuxing

In the past I've been using a combination of Handbrake and XMedia Recode to either transcode or remux all my video files to MP4.

In particular the setup that I go for is this:
MP4 container using h.264 video
Remove all non English audio tracks
Remove all subtitles from file
Add AAC 2 channel 256kbps audio as first track (for compatibility with all devices) if needed
Copy all remaining English audio tracks.
Setup MP4 for "fast start" or "web optimized" whichever you wish to call it.

If the file already contains an h.264 video then the file is remuxed but if the video is not already h.264 then the file is transcoded using Handbrake with a HIGH PROFILE and web optimize.

I've been wanting to automate this into a more streamlined universal process where I can just put files in a directory and have a set of programs completely handle this without any manual intervention.  I have this running on one computer at the moment and should no longer need to use either Handbrake or Xmedia software.  I started playing with FFMPEG and looked for any scripts people had.  Got some ideas from some and then went nuts.

The scripts can create the file as defined above and will either remux or transcode automatically depending on the the input file.  I have FFMPEG doing the equivalent of a High Profile encode when transcoding is needed. It can of course delete or move/archive the source files after processing.

I'm going to play around tonight and see if I can add the ability to EASILY create a cluster of computers that can do distributed encoding (one complete media file per machine) and work off a directory queue.  Once I have this working I'll go back and add the ability to create a 2nd MP4 file of 720p 3mbit with only AAC 256 kbit 2 channel audio like I talked about previously.

With the two files the Plex servers can almost always direct stream without having to transcode on the fly.

Carlo

45

Really appreciate your post! I started with Sage back in 2004 and recognize you :). I am now starting to transition to Plex (and bought a lifetime Pass), but am very disapointed to discover the issues with full disc folder rips. As you know, disc rips are basically the core of SageTV for local/personal video from the beginning. We were even using DVDprofiler for meta back when before Stuckless saved us. :)

Really, this is my main problem to overcome with over 36TBs of bluray and dvd folder rips alone, re-coding all of these is simply not something I am going to be able to do. I am curious how you tackled this problem? Or didi you just recode everything? I use Sage's external app plugin and Total Media Theater right now on the front end. I was planning on using Plexbmc and Kodi to overcome this shortsighted limitation, but it looks like the plexbmc plugin is in a non-working state right now. So, for now, I am going with with using Plex's front end.

I was also interested in your post about Media Center Master. Can you elaborate more on how you use this? Would this overcome the Plex server limitation with disc rips by subsituting a kind of alternate scanning system?

Thanks and nice to see a familiar username :).

46

I enjoyed SageTV a lot back in the day.  They had a good scripting language and you could do some amazing things as was obvious do to my total customized setup I released for that platform.

I "bit the bullet" and converted all DVD and Blu Ray discs.  Google a program called MakeMKV.  With this program you can point it to a raw disk (DVD/Blu Ray), to an ISO image or to the video folders inf file.  What it will do is create a series of MKV files from the "disc".  The largest file is almost always the movie and the smaller ones are trailers and stuff from the disc.  These MKV files are UNCOMPRESSED and contain all video,audio,subtitles from the disc itself.  These MKV files are usable 99%+ of the time by Plex.

Grab a copy of FileBot which does renaming for both movies and TV Shows and you almost guarantee Plex will find all the media perfectly.

There are several tools you can use to take these MKV files and compress them into a unified format such as mp4. Handbrake and Xmedia's are pretty popular.  As previously mentioned a day or two ago I'll be releasing in about a week a set of scripts that will do all this for you and will be very customizable via INI files.

What I'd suggest is setting up two distinct directory/folder structures.  Use one to hold the MKV rips and a different to hold the compressed files such as:

\RawRips

\Movies

You then just setup your Movie Library to watch two different folders so it will not matter in which directory the file exists.

MakeMKV is pretty quick creating the MKV files as it's not really doing any processing.  It basically works as fast as your read/write speed of your hard drives.  Doing any type of "compression" on the other hand takes hours per disc.  So by using the combination of both techniques you get the best of both worlds.

Carlo

PS one other alternative is to "hook up" with someone who has already done much of this work who can allow you to d/l these compressed files since you already own them. Then just process the disc you own that you can't get another way.  :)

47

I appreciate the advice! Yeah, getting the movies from a "friend" would seem like a much better option than mass transcoding all these discs.

48

I just posted this info in another thread and thought it might be useful so I duplicated it here. This was in response to a thread about creating dynamic libraries from meta data:

I too would love to see this come to light.

I do something MANUALLY right now in a "hackish" kiind of way.  I wanted to create different Libs for season/holidays (Christmas, New Years, Val Day, Memorial Day, etc) with relevant movies.  What I resorted to is this directory structure:
F:\SeasonHoliday\Christmas
F:\SeasonHoliday\Easter
F:\SeasonHoliday\Halloween
etc

I then symlink directories with a windows command like the following:
mklink /J "F:\SeasonHoliday\Christmas\A Disney Christmas Gift (1983)"  "F:\Movies\A\A Disney Christmas Gift (1983)"
(I keep all symlink commands in a batch file for archive purposes in case I ever need to rebuild)

In this case it creates a directory called "A Disney Christmas Gift (1983)" under Christmas and points it to the real location on my system (last parameter).

I then just create a real movie library ie Christmas pointed to F:\SeasonHoliday\Christmas directory.

This works great EXCEPT that I don't want all the holiday libraries showing all the time.  Maybe only 2 at a time (previous/next holiday).  This is easy enough to do for shared users as you can just check or uncheck the library for that user BUT as the admin you ALWAYS see ALL LIBRARIES which sucks.  You can work around this problem sort of by creating a new normal home user for yourself and only share the active holidays at the time OR

Setup a 2nd server.  I did the 2nd.  I moved all my music and music videos to this server and have all the season/holiday libs setup here.  Now in most of the main clients I use I only select Server1 so I don't see them.  I would have just setup and used a 2ndary home user but this conflicts with my use of the xBox as I use Skype on this often and need to be logged in as my "real user".  Since Plex doesn't allow quick user change on the xBox but instead uses the logged in user I'm screwed with that approach. The old 2 steps forward, 1 step back routine. :(

But anyway I just wanted to share a work around that can be done now to get "some" of this functionality today.

Carlo

49

Just found this thread. Thanks for all the good info Carlo, especially the tip about multiple versions of a movie at different bitrates. I had no idea Plex was smart enough to understand that multiple versions are the same movie and would use them appropriately. I'm going to tinker with this, as my main complaint with Plex is the very long startup time for transcoding a stream at less than max quality.

I'll also second use of ZFS for those not using Windows for storage. It is well worth the setup time for the considerable integrity and performance benefits. It also allows use of a SSD as a cache, much like Storage Spaces. (L2ARC) Plex seems to work just fine reading media from a ZFS volume also, (At least for me) in contrast to those reporting issues with ReFS.

50

This thread has given me some Ideas to look into and insight into a few different setups.

Thanks for the Wonderful posts all.

Joe

51

Carlo, your information is wonderful. thanks.

Thought I would add my 2 cents:

ReFS does not support symlinks (Microsoft plans to add symlink feature in a future version). As such, you cannot place the Plex database/Metadata on it successfully.  This is the cause of metadata not showing up for new movies/episodes.  ReFS is good for the actual media files though.

My Plex Server is a Hyper-V virtual running Server 2012 R2. The OS drive (where the database/MetaData is located) is a vhdx file formatted as NTFS on a dedicated SSD formated as NTFS.

I created a 48TB virtual disk on a parity thin-provisioned storage pool (across 7 drives). On this virtual disk I created a 32TB partition formatted as ReFS (32TB chosen to optimize allocation unit size). This is where my video media is stored.  I also created an 8TB (allocation size of 4096B) NTFS partition where I have my music media.

I handle backup of my media by using external drives (WD MyBooks) where I have copies of every media file I add in the same folder layout as on my media drive.

I have no issues with metadata, buffering, or responsiveness.

52

I should probably update this thread to my new current setup which has changed. I'm now using a combination of Plex and MB3.

sGarver, the main reason non of us use ReFS is that it's slow and lacking useful features.  ReSF is also more CPU unfriendly. On my spanned drives I can easily get 300-400+MB read rates from NTFS but only about 80-100MB via ReFS at the host level.  I tried this on Windows 2012 R2 as well as the next release candidate of windows server 10.  ReFS is more handy with smaller files that change often like office docs and such and will have less going for it in a setup used for large media files or especially vhdx files..  With Plex the media is generally written once and not touched and with the speed differences make NTFS a much better solution at least for now.

Couple question for you:

Sounds like you are over provisioning your storage pool. Any reason you are doing this?  How many drives are parity out of you 7 drives?  What size drives are you running in your storage pool?

Curious, why are you running Plex in a VM as opposed to running on the native host where transcoding will perform better?

I too use WD MyBooks for backup.  Doesn't get much easier than that and they are fast!

Carlo

53

I should probably update this thread to my new current setup which has changed. I'm now using a combination of Plex and MB3.

sGarver, the main reason non of us use ReFS is that it's slow and lacking useful features.  ReSF is also more CPU unfriendly. On my spanned drives I can easily get 300-400+MB read rates from NTFS but only about 80-100MB via ReFS at the host level.  I tried this on Windows 2012 R2 as well as the next release candidate of windows server 10.  ReFS is more handy with smaller files that change often like office docs and such and will have less going for it in a setup used for large media files or especially vhdx files..  With Plex the media is generally written once and not touched and with the speed differences make NTFS a much better solution at least for now.

Couple question for you:

Sounds like you are over provisioning your storage pool. Any reason you are doing this?  How many drives are parity out of you 7 drives?  What size drives are you running in your storage pool?

Curious, why are you running Plex in a VM as opposed to running on the native host where transcoding will perform better?

I too use WD MyBooks for backup.  Doesn't get much easier than that and they are fast!

Carlo

Carlo,

I considered and use ReFS for the media store because of the built-in data scrubbing, something that NTFS does lack. My concern being data integrity and media failure.  And I SO agree with you, that ReFS is NOT where you would place a vhdx file, the data scrubbing wreaks performance havoc with it.

I over provisioned as a way to minimize growth pains. The current pool consist of five 3T drives and one 6T drive (started out with four 2T drives).  As I bump up against the actual HD capacity, I can add/swap a drive without having to take down my system and rebuild the "array" (been there, done that, don't like it). As I can afford them/need them, new drives are added. I use single parity and my column count is 4 (recall original was 4 drives).

Why virtual???... well... lack of $$$... as some of my older gear started to fail, I had to consolidate.  So now in one tower server chassis, there exist the physical 2012R2 MinShell Hyper-V server, a 2012R2 Server Core Domain Controller, a 2012R2 Full GUI PLEX server, a Win7Pro used for torrenting (used to be an physical computer that the MB got really unstable), and a Win8.1Pro used for Calibre.  :)

I am able to handle three internal users and 1 remote user without overloading the CPU.

54

I got ya.

You originally said you had a 7 drives striped but just said you have 6 drives?  Typo on one of them?

Yes you can add drives but not the way you would think.  Storage Spaces only allows 8 drives per pool but you can have basically unlimited pools.  So without causing yourself grief you'll have to update the current drives to bigger ones or split your data out a bit more with a couple of drive letters.    Edit: Not clear as to what I meant and covered in message #56 below.

So right now the system is treating your hardware as if you have 6 3TB drives since it can't do anything with the extra 3 TB on the last drive.  You need a multiple of the span count to be able to use it.  So if you are spanning across 4 drives you need 3 more 6TB to be able to use them in fault tolerant format.  So MS lets you mix and match drives but unless they are similar size you aren't the full benefit from them storage space wise.

I don't know if you are aware of this or not but you can create a new provisioned drive consisting of a 6 drive parity (since you have 6 drives).  Then you move the data from your 4 striped drive to the 6 striped drive and end up with faster overall read and write rates since the data is split across 6 drives instead of 4.  However, since you are using mixed size drives it's not a good idea for you to do that.  But keep it in mind if you ever end up with the same size drives.

I myself would not use ReFS for media files either so you don't really gain any benefit from it on such large files and it's so much slower. You basically write them once to the array and that's it.  Most of the benefit of ReFS is for fixing writing mistakes/errors and things like that.  I personally wouldn't use it for anything right now as there are better solutions that actually work correctly without killing the CPU.  ReFS is very hard on the CPU in windows.

You're going to find out as you grow your system that running Plex in a VM isn't going to scale as well as you think.  You will be better able to server more simultaneous streams if you move the Plex stuff to the host where the transcoder can get full use of what's left of the CPU instead of it being managed by the VM.

So take this as just a recommendation and nothing more but if you need to support more clients consider moving Plex to the host instead of running it in the VM.  Consider creating the largest spanned drive you can to get the best performance you can from your drives and consider ditching ReFS and going to the tried and true NTFS.  You'll EASILY double if not triple the number of clients you can handle.  With that said, if what you have now works leave it alone but just keep these things in mind if you do a new server build or need to expand.

Carlo

55

Carlo.

I appreciate the advice and if I can ever allocate funds to a bigger upload pipe out of here, I will definitely rebuild. Having Plex in a VM was more of a necessity than an performance decision (although the Hyper-V backup method is comforting; simple, containerized recovery).

7 was the typo, quick count from memory.

Storage Spaces only allows 8 drives per pool

When I read this, you left me scratching my head.  <_<  According to MS:  "Up to 240 physical disks in a non-clustered Windows Server 2012 R2 storage pool (4 x 60 disk JBOD)"  Ref: http://social.technet.microsoft.com/wiki/contents/articles/11382.storage-spaces-frequently-asked-questions-faq.aspx#What_are_the_recommended_configuration_limits

Also, http://www.miru.ch/why-column-size-does-matter-with-storage-spaces  shows testing at some pretty big configurations - 64 disks / 32 colums.

56

Yes, I wasn't clear enough on that.  You can build truly large clusters or even single server system with Storage Spaces.  I gave the impression is was limited to 8 drives but that's not what I meant.

http://social.technet.microsoft.com/wiki/contents/articles/11382.storage-spaces-frequently-asked-questions-faq.aspx

Scroll down a bit more than half way to the chart. In the Resiliency type column go down to Single Parity and then look at the far right column "Maximum Column Count".

You'll see for single parity the maximum column count is 8. <-- That is what I was getting at. For dual parity you can go up to 17 columns.

So even with single parity I could have 9, 10, 11 or 12 drives.  In case it's not clear here's an example with 12 drives using 8 columns for a series of writes.  The number below would show how each drive is used.

1,2,3,4,5,6,7,8 (first write to disk)

2,3,4,5,6,7,8,9 (second write to disk)

3,4,5,6,7,8,9,10 (third write to disk)

4,5,6,7,8,9,10,11 (forth write to disk)

5,6,7,8,9,10,11,12 (fifth write to disk)

1,2,3,4,5,6,7,8 (sixth write to disk)
2,3,4,5,6,7,8,9
3,4,5,6,7,8,9,10
4,5,6,7,8,9,10,11
5,6,7,8,9,10,11,12

So it alternates the uses of the disks but in the case of single parity will only use 8 at any one time. All drives will get equally used. So as you can see, storage spaces can and will use all 12 drives but each read or write will only use 8 drives at a time since I'm using single parity which supports a maximum of 8 columns.  This could have been 20 or 30 disks.  It would just follow the same type of pattern only using any 8 at one time.

You could do the same exercise as above with 17 drives using dual parity.

You can do things like setup 16 disks in dual parity mode then mirror this for a total of 32 drives in use.  It's really flexible. In configurations like this one even with NTFS you can get almost all the features of ZFS because storage spaces can use the parity information along with the 2nd set of data to autofix almost anything wrong with the data including true bitrot in real-time without the typical overhead of ZFS or even ReFS.

So Storage Spaces is almost endlessly scale-able for our use and why I use it.  I hope that explained a bit better what I was getting at without confusing things further concerning 8 columns. :)

It really is very flexible in setup and use of hardware. You can mix and match SAS, SATA and USB (USB3 for performance). You can mix and match different sizes of discs also.

Carlo

57

Excellent topic you have started here Carlo, some really informative guides and insites into what works. I have followed this from the start and benefited from the custon libraries the most so cudos for taking the time to explain.


Just to quote you from afew posts back,

“I should probably update this thread to my new current setup which has changed. I’m now using a combination of Plex and MB3”


What are you using MB3 for compared to plex. I still think plex is superior but I do like what they are doing over at MB3 and I was abouts to do the same, mainly to get my content onto the kodi devices alot easier. As for MB3 trancoding di you have any input on this as from researching I believe it trancodes files plex plays natively.

58

Is there a way to have a person pick the quality...

for example.

Cars

Cars 1080p

Cars 720p

Cars 420p

Right now I am at 5 users connected at the same time! but my CPU can't handle it. My Network is at 75/75 with verizon Fios 

59

We are in the same boat.  We both have good upload bandwidth and can do FULL direct streams without regard to bandwidth.  I'm 300/300. However we are both CPU bound.  I can't do cloud sync or client sync because of this as there isn't enough CPU to go around for these features plus my normal streaming.

With most clients you don't have to manually pick the file as the client is smart enough to do it automatically.  I had thought this was a server function but recently found out the client does this. The client device gets the list of available resolution/bitrates/meta for all files and it chooses which one to play. So if you have multiple resolutions of the same file and assume different bitrates for these files then when a client goes to play the movie it will choose the first one it can direct play at the highest res/bitrate.

I try and create 720p @ 3mbit as the secondary file.  Using Handbrake you can create good looking files for that res/bitrate combination.  It's small compared to the 1080p and will almost always direct stream if created as an MP4 with only AAC audio.

I'm typically in the 6 to 12 range in the evening streaming client wise and see my CPU get pegged once in a while but so far no complaints so I get by.  I only keep about 200 of my most recent movies in 720p.  Typically I've found most people play the recently added stuff the most so this takes a large part of the strain off transcoding.  If someone goes for an older movie it can get transcoded as I still have enough CPU to handle it.

This has worked pretty well for me unless a few people go on a TV Show binge where they start at season 1 going through the whole list of seasons.  I've got one person doing this with Law and Order SVU and another watching CSI Vegas and both are using the Now TV box which I believe only does 720p.

I'm getting low on storage space and due another upgrade.  In this next update I'll probably allow for 5 to 10 TB just for 720/3mb files so I can have more of these files on hand and further reduce CPU usage.

One obvious thing you can do is try and communicate with those using your system to see if you can get them setup so they direct stream (assuming they have adequate bandwidth).  This is obviously the best thing you can do as it instantly takes the load off of your CPU.

Let me know how you make out and progress doing this.  There are only a couple of us doing this that I know of so it would be good to learn from each other.

Carlo

60

wow! 36TB is crazy i am at 10TB 

I think I will upgrade to a i7 in a few months to help.

Max amount of people that are logged in at the same

time are 5-7 and mostly its direct stream. Only problem

is when the Roku3 or a PS4 hits my server it transcodes

the movies.