Cayars - Setup walk through and some tips and tricks

General Discussions
61

Hi Carlo,

it's very exciting to read along this thread. Thank you for sharing!

Handbrake and I aren't good friends :unsure: - too many settings I don't understand and weird erratic results (in consequence of my weird interpretation of the settings, I guess). It sounds as you are encoding with one profile that works good with most items and makes a good quality and compatibility for Plex.

Would you mind to share this profile as a file so that I could try to import it and give Handbrake another chance? 

Did I get you right you figured out a script for batch converting?

Greetings

62

Well I have a set of Handbrake settings that work great for me.  That doesn't mean it will be great for someone else. :)  

However, I haven't use HB in a couple of months unless I'm converting something like flash which is rare.  I'm using ffmpeg to do all conversions and remuxing in a set of python scripts.  I just move any files I want to work on into a directory, kick off the script and then when done have my resulting files in an output directory I can further process.

My general process it to only work on MKV, MP4 or AVI files.  Anything other than these I'll do by hand.

But with the above three file types I'll:

Pull subtitles if they are in English (user configurable) and create an SRT file

Remux or transcode to MP4 with h.264 video.

I discard all audio tracks not in English (user configurable) from my final output

I create a first track of AAC stereo if it doesn't exist

The transcode of video uses the equivalent 264 settings of Handbrake at High Profile, 20 CQ, slow preset which could be changed in the python scripts themselves.  These are configurable since I didn't really want to adjust them as I just wanted something that works across the board well and gives good results.  

Just a note, but I don't automatically add files to my libraries as I like to quickly test each file to make sure I've got english sound, good picture quality etc.  I typically don't use the SRT files and discard them.  However If I've got  high ranked IMDB foreign film that I can't get in English I'll strip all audio but use the SRT file.

Carlo

63

This is a great thread Carlo, thanks for taking th time to detail your set up!

I have been looking for other ways to automate my set up, as what I'm doing currently has its (minor) limitations.  Here's a brief overview of what I'm currently doing and what my goals are.  

  1. I have custom RSS feeds configured on a few private sites, where I add the titles I want in the quality and source I'd like.  
  2. I then set up uT to grab everything in those feeds, which are saved to my Storage Spaces 'RAID#1 Seeding' (single 6tb red in "Simple").  
  3. Once the dl completes I have uT configured to run Filebot's AMC script.  This script automagically determines if it's a TV or Movie and handles all of the renaming and relocating, and is completely automated.
    1. If it detects a Movie it is renamed to 'Title (year).ext' and copied to my RAID#2 Movies (4 x 4TB reds in Parity)
    2. For TV its renamed and relocated as T:\TV\Title\Season #\Title S00E00.ext and its copied to my RAID#3 TV (4 x 6TB in Parity)
  4. I have Plex watch the Movie and TV folders.
  5. At this point I have two copies of all my media.  Hardlinks don't work in SS, and symlinks don't work in uT, so I use copy.  I then have to manually point uT to the files in RAID#2 & #3 and then delete the files in #1 so Im able to perma-seed.  There is a way to relocate all the TV shows in bulk by using TVRename which modifys uT's resume.dat but I still have to manually initiate it.  There is no way for me to do movies in bulk, that is all individually which gets old and annoying.
  6. I use PlexWatchWeb to send tweets when ever somebody starts watching something in this format "#user is watching: Plex Web (Chrome) currently 32 minutes in with 14 minutes left. Audio:transcode Video:transcode" That way any of my friends and family can monitor how many users are currently streaming from me incase of spotty service.

Like you, I have a 120GB SSD as my c drive that was filling up with over 60GB of plex app data.  As a temporary solution I created a junction and moved the actual app data to RAID#4 Data (2 x 1TB & 2 x 4TB in Mirror).  My biggest hang up is with trying to perma-seed all my renamed/relocated files automatically.  Is there a way to automate that with MCM?  With the glaring exception to the lack of PVR, do you see ways I can streamline or automate a better experience for my Plex streamers?  I also run into the CPU limitations when I've got 5+ simultaneous remote streams. My bandwidth doesn't help either I'm only able to get 80/5, 5up is like being on dialup.

The one functionality I would love to implement would be for the media to be renamed/relocated/updated in uT to seed,  AND make a copy  and put it in a 720p 3mbps mp4 aac container in a rolling 2TB partition without any user interaction.  Chromes lack of support for mkv is killing my processor.

Advice:

I absolutely love Storage Spaces for this application, but I have found that even though it lets you mix and match sizes you really shouldnt as space is wasted. I originally had one giant 16HDD (54TB) Storage Space in parity and that was a bad idea.  First of all the electricity and wear-and-tear of 16hdd's spinning all day every day is just wasteful.  Secondly the heat produced by all those drives caused lots of headaches (until I got my 4U server case with proper fans).  And Third Storage Spaces is meant to grow, so i ran into major bugs when trying to split 1 raid into 4 and had to buy 6tb drives to offload over 25TB of data in order to completely delete the original SS, which as you can imagine took weeks.

I am definitly looking into MCM but for those looking for a simple, and free, alternative Filebot's AMC script is amazing http://www.filebot.net/forums/viewtopic.php?t=215 

64

Not that I know anything about torrents as my stuff is ripped from media I have. ;)  Let me ask a few questions to make sure we are on the same page.

I assume you are using private sites and need to seed to keep up ratios, is that correct?

Out of curiosity have you played with newsgroups at all?  Since with newsgroups there really isn't a concept of "seeding" and that would get rid of part of your problem.

What do you mean when you say hardlinks don't work in Storage Spaces?  Are you using NTFS or ReFS?  I'm a bit confused because you do mention using junctions which are one type of hardlink.

I'm wondering if you are making the process harder than it needs to be.  In uT you can setup Download Folders and Completed Folders.  So once it completely downloads it gets moved to a completed folder.  You could setup scripting to monitor this "completed" folder and process anything new there (done downloading) without touching the media.  This way you get you new stuff and can process a copy of it and at the same time can continue to seed it.  Would something simple like this not work?

Hmm, concerning storage spaces.  It depends on how you setup things up. Yes, you can waste space in SS by mixing and matching drives but this "wasted" space can again be used in the future as you add additional drives.  By this I mean if you currently have qty 4 4TB drives and qty 2 6TB drives in an array with a column count of 6 then it will only be able to use 4TB of each drive.  You'll have 2 drives with 2TB "free" and not able to be used. If you added 2 additional 2 TB drives you could then use these drives by creating a new array with a column count of 4.

Now lets take a different configuration.  Lets suppose you start with 4 drives of 4TB with a column count of 4.  You then add a single 2TB drive to the mix which still only has a column striping of 4.  SS will alternate writing to each drive as I showed in an earlier post.  The 2TB drive will fill up most likely first but then since you are using a column count of 4 it can still write to the other 4 4TB drives and not waste any space.  You could add in a 1TB drive and it most likely will get filled up first, but again you won't waste any space.

If you had 4 drives of 4TB and 2 drives of 6TB as long as you keep the column count to 4 you won't waste space.  There are background tasks that can run that will relocate data to make better use of striping.

Where you get screwed on space is having a setup similar to 4 drives of 4TB and 4 drives of 6TB with anything over a column count of 4. So it really comes down to how you setup the spaces and if you intend on continuing to add drives or not (and how many you add at one time).  But anyway you cut it, SS is far more flexible than the typical (regardless of price) hardware RAID boxes and is also more flexible than many lowend SANs.

65

I assume you are using private sites and need to seed to keep up ratios, is that correct?

No some are actually ratioless, my thoughts are just that since I will have the media sitting there on my raids why not.  But you are right in that this is probably unnessicary.

I have not played with newsgroups ever, as private sites have satisfied my needs.  I think that now though in order to get to the next level that is something I have to start considering.

What do you mean when you say hardlinks don’t work in Storage Spaces

Sorry that wasn't too clear, hardlinks (keeplink) of individual files do not work, but folder junctions do.   Hard links cannot link to a file that is on a different partition, volume or drive.

In uT you can setup Download Folders and Completed Folders.  So once it completely downloads it gets moved to a completed folder.  You could setup scripting to monitor this “completed” folder and process anything new there (done downloading) without touching the media.

I'm already doing this. The problems is I have two copies of the same media.  Ideally I would like for the dl to get processed (renamed/relocated) and have the original file deleted and have uT pointed to the new files.  It sounds like MCM might be able to do this because it has a uT integration feature, but I just am not clear as to the specifics (and I'd have to pay to find out)

If you had 4 drives of 4TB and 2 drives of 6TB as long as you keep the column count to 4 you won’t waste space.  There are background tasks that can run that will relocate data to make better use of striping.

I'm running Windows 8.1 and it doesnt look like I have the option to  set column counts.  What background tasks are you referring to?  I have been looking for exactly that type of utility, because all my SS are unbalanced.

66

OK got you on the hardlinks.  Just curious but could you not use junctions there also?  I've found I can pretty much use junctions for all my needs when spanning volumes or drives but you may need to think about how you use them a bit more. If you are working at the directory level and not file level they can work essentially the same as hardlinks.

Windows workstation vs server work the same for Storage Spaces in setup so you have the option to set the column count.  Maybe you just never saw it and went with the default settings?  Did you not get the chance to pick how many drives to use for each parity array?

Do a search on Storage Space Powershell and you'll find some nifty scripts you can setup via task scheduler to run on occasion that can reorganize data. I honestly only use SS on Windows 2012 R2 Server but they should work across the board with Windows 8.1 also.  I know some of them work with Windows 10 Preview because I've played with them on this operating system.

67

Storage Spaces on windows 8.1 is severely limited.  There are no advanced options, I was just looking at youtube and drooling over  all the fancy options Server 2012 has.  I'm going to get a 1TB SSD and just instal Server 2012 R2 DCE and keep Plex locally instead of the junction to the raid.  I actually ran into some bugs in storage spaces where the storage pool wasn't releasing the deleted data's size on disk.  The only solution was to run powershell and call on Diskshadow, but windows 8 doesn't come with diskshadow and Microsoft techsupport said that I am unable to run it.

68

No you should have the same options I'm pretty sure.  You might just need to use Powershell on workstation for config. :)

Regardless, I think you'll find Server 2012 R2 much easier to use for sure with Storage Spaces!.  It also has far better notifications built in if it finds any issues.

69

Thanks to this thread, I feel I finally have a good way to get my 300+ educational courses into plex! I just wanted to share another tool that I've found helpful, the program Advance Renamer ver3.65. It has numerous file renaming options like sequential numbering (useful for S01EXX) and it can read video title and genre metadata. This has given me an easy way to change filenames for 20-30 files at a time and get them picked up properly by the extended personal media scanner linked here in the first post. I still have to do some work getting posters/backgrounds, but it's coming together and looking pretty sweet. 

70

Nice, glad you found it helpful! My “Learning” Library is now over 1100 videos and this method continues to work well for me.

I’ve never like “Learning” as the lib name but for some reason couldn’t think of anything better.
“Educational” sounds so much better! (and so obvious) doh! So thanks for that!

71

I’m summing up the rest of this post with "Storage Spaces Have to Go!"

As most of you know, I had setup my storage using Windows Storage Spaces in Server 2012 R2. My initial setup used qty 8 4TB drives. I then added an additional 4TB then a 6TB drive. This used a single parity drive and was setup using 8 column striping. Initial testing seemed good. Fault tolerance was good (could pull drive and rebuild).

However, for a couple of reasons I’ve decided to move away from storage spaces:
1 - Due to the fact I used 8 column striping, 8 of my present 10 drives are always involved in any read/write operation which in itself isn’t bad. However, as additional drives were added and I started noticing a slowdown in speed. Most likely this is due to data being on the outside of some disks and inside of others. The overall effect is that the slowest drive dictates your speed.
2 - I originally used 8 columns because the thought was that only 1/8th of the total space would go to parity. I have no idea why but the storage space is using 1/3 for parity. If I were to put 2 TB of actual data in the pool it would tell me 2TB of info but 3TB used!
3 - I tried to add an additional parity drive for a total of two and it would not let me do this. I would have to rebuild it from scratch which means I’d have to move all data off the storage space first! This one hurt a lot.
4 - If for some reason I were to loose 2 drives I would loose everything (since only a single parity drive).

Mainly due to those 4 reasons I’ve decided to move away from storage spaces. I’ve come up with a list of features that I want in my new storage arrangement:
A - I want resiliency and want to be able to add additional parity drives at a later date without a complete rebuild.
B - I want to be able to add additional data drives as needed without having to rebuild the entire array.
C - I DO NOT want the drives “striped” in the conventional sense. This way if resiliency doesn’t work and a drive fails I only loose data on that one drive and not the entire array.
D - I do want all drives “pooled” under on drive letter
E - I want to be able to pull any drive from the array, stick it in a different computer and be able to read it.

I have been looking at numerous different solutions that fit most of what I’d like to have moving forward. Transparent RAID (www.flexraid.com/download-try-buy/) seems like it will fit the bill rather nicely at a modest price.

My thoughts are this:
Theoretically using TRAID I should be able to completely fill the first drive, then 2nd and so on. In this way once a drive is filled I can upload an image of the disk to a cloud provider. The disk will be “static” for the most part (unless I overwrite a file with a new version). I’m currently testing Amazon Cloud Drive and will soon test with MS OneDrive. I can currently upload close to 1TB of content to Amazon per day.

So I’m thinking with dual parity using TRAID and keeping an image of each disk in the cloud (2 services) I could forgo local backups. It would take a 3 drive loss (1 data, 2 parity) for me to loose any info locally. Even so, a loss of one data drive would not bring down the whole array. I could swap the bad drive and download the image (disk contents) from one of the cloud providers in 4 to 6 days (4TB or 6TB disk) at most.

So with 2 parity disks (for now) and 2 copies of my data in the cloud I’d feel rather well protected.

At present I have 3 additional servers online with 3 to 4 SATA drives in each that I’m coping data to from my current storage spaces. I’ve got all these drives mapped on my Plex server and have plex updating it’s database with new file locations every half hour. I’ve been doing this for a few days. I’ve got less than 4TB to go so at some point today I’ll be break down my storage array and will be able to setup TRAID for testing and implementation.

Carlo

72

I’ve got 4 Servers and 3 Notebooks stuffed with every drive I could find around the house including my old backup drives. I’ve been able to move all data off storage spaces to these networked drives. Plex stayed up the whole time I was moving the data off.

I had to use a spreadsheet to keep track of everything. My TV section is using 6 shared drives and my movie section alone is using 8 shares. Right now I still have 2TB free on an external USB drive that I’m copying redundant data to. In the last 5 days I’ve uploaded 10.5 TB of data to Amazon Cloud Drive. I still have the storage space up and running with the 8 drives.

1 drive is parity so I’m going to pull that drive and let the array fall back into downgraded status. I’ll them copy an additional 3.64TB of extra redundant data from the array. Between my local “extra/backup” copies and what I’ve uploaded to Amazon I have all but 4 or 5TB in redundant fashion. As soon as I fill the “parity” drive up I’ll break the storage array down and get 7 more drives to play with.

I’ll need to use two of them to get the 4 or 5TB I don’t have a dupe of right away so I’ll feel better. Then I’ll need to use 2 more so I can transfer one of the WD Red 6TB drives to so I can use this for the Transparent Raid Parity drive. That will leave me a couple of drive free to start the RAID setup with.

I’ll then be able to play a bit and test fail-over and make sure it works exactly how I think it should. Better to play while I have 2 copies of my media on hand!

Carlo

73

How much you paying for Amazon cloud? I can’t figure out how you’ve got 10TB+


The tariff says £320 for 1TB!?


Or is it a loophole of Amazon Prime???

74

$59 a year for unlimted with a free 3 month free trial.
https://www.amazon.com/clouddrive/home
(I do have Amazon Prime but don’t think this has anything to do with it)

I’ve uploaded about 13TB now in a week playing around.
Some caveats I’ve found. The desktop app works well for uploading but not so good for downloading. It’s an all or nothing download from the desktop client. OR you use the web browser to download. But using the web browser you hit a MAJOR snag. It won’t allow you to download more than 1000 files at a time AND your download can’t exceed 5GB. Yep 5GB. So you can happilly upload blu-Ray rips or large MP4/MKV files but you won’t be able to get the files back easily!

HOWEVER, “I has the solution”. NetDrive 2.5.0 Beta8 http://www.netdrive.net/download.htmlwill let you mount the Amazon Cloud Drive as a drive letter. It’s still in beta and you can get IO errors on occasion when downloading. It’s not as fast for me to upload so I still use Amazon’s desktop to “seed” my cloud drive. NetDrive is free to try or you have to pay for it UNLESS you only want to use one cloud drive then it’s free to use. So for me right now it’s free as I’m only using it for this one provider.

Another software a bit more polished but a bit different is SyncBackPro www.2brightsparks.com/syncback/sbpro7-intro.html which is designed for backup, mirror or sync (one or two way). I haven’t experienced any issues at all with it but it’s not quite as fast.

I’ve got all my different libs uploads except for movies (working on them now). Using NetDrive I’ll be able to take everything I’ve uploaded and move them into master directories like:
Disk1
Disk2
Disk3
etc

So since I’m switching to FlexRaid each drive is available as part of the RAID or as an individual drive letter (basically the disk). So I could access all 20 disks individually or pull a disk and the others keep working.

Obviously the contents of my first RAID disk will go under Disk1, Contents of drive8 under Disk8, etc. Since I’m completely filling each disk before adding a new one to the array I can easily control the layout. These disks once filled will pretty much be “static”. I’ll on occasion overwrite a media file with an updated higher res version but that’s about it. Using SyncBackupPro I’ll be able to SYNC my local drive to the amazon directory with the same name (ie Disk5) using my local drive as the master. So it will delete anything I’ve deleted or update anything that’s changed locally.

Using this setup of Transparent Raid with 2 parity disks and an “image” uploaded to Amazon Cloud Drive (and soon to MS OneDrive) I’ll feel comfortable in not having “local” backups anymore. I could loose 2 drives locally without worry but if a 3rd drive crashed I’d loose only the contents of that disk but the rest of the RAID would continue to function. Then I’d just need to replace the disk and download 4 to 6 TB (less then a week for me) to the replacement drive.

This to me seems like it’s going to be an ideal setup.

Just an update: I got as much off the Storage Space as I could leaving 3.2TB not redundant on my local network. I then broke down the storage space, got 8 additional drives back and immediately backed up that 3.2TB of info so I have dupes of everything for now in case something goes wrong. I’m now shuffling files from my 6TB drives to 4TBs so I can use the 6TBs for parity. I’m also pre-loading my first Drive with files.

I’ve kept Plex up to date with all the different drive letters and keep having it scan my files so it can update locations of my media. My FTP server on the other hand is in “left field” and of no use at the moment since none of the directories match the settings. I’ll take care of that once the new RAID array is back up.

Some time late tonight or early tomorrow I’ll have 2 disks ready with pre-loaded data and a free parity disk so I’ll be testing the setup more thoroughly.

So far, so good!

Carlo

75

so is there a way of using the unlimited cloud as a "drive" on plex and stream from it?

kinda like a chromecast, you control the stream you point it to, but once its connected to the stream its direct from the source aka cloud?

would mean people with naff upload speed could theoretically [once they'd got it all up there] be able to serve loads of people from the cloud uplink.

ahhh, but the PMS is what does the transcoding where required.

its an interesting idea.

hell, for the cost of the cloud space, I'd send an i7 NUC to them and tell them to plug it in! :D

76

let me know how transparent raid goes. i like snapraid but the manual snapshots are its only real weakness as far as data protection

77

so is there a way of using the unlimited cloud as a "drive" on plex and stream from it?

Maybe, I was able to do this with Bitcasa and think it should be possible with ACD also, but I'd think we may need to wait until NetDrive gets a wee bit more stable. I'm still seeing a bit to many IO timeouts (recovers fine).

kinda like a chromecast, you control the stream you point it to, but once its connected to the stream its direct from the source aka cloud?
 
would mean people with naff upload speed could theoretically [once they'd got it all up there] be able to serve loads of people from the cloud uplink.

Nope, It would just become a drive letter on your system like any internal, network or USB drive. Everything would still need to go through your Plex Server. However, if down the road Plex adds ACD as a true cloud provider then this could become possible.

ahhh, but the PMS is what does the transcoding where required.
 
its an interesting idea.
 
hell, for the cost of the cloud space, I'd send an i7 NUC to them and tell them to plug it in! :D

 You could get a VM machine and setup Netdrive on it and run everything in the cloud that way I suppose.
 

let me know how transparent raid goes. i like snapraid but the manual snapshots are its only real weakness as far as data protection


Yea, not a fan of snapraid either. FlexRaid also has a product similar to that called Raid-F.

I'm still shuffling data around my network. I'm doing multiple things at once:
1 - Moving data off 2 new SSDs so I can use one for operating system and 1TB SSD for Plex MetaData.
2 - Moving stuff from one of the 6TB drives so I can remove the file system so it can become a parity drive.
3 - Already filled first data drive and working on a 2nd data drive at present (pre-populating them)
4 - Continuing to upload data to Amazon Cloud Disk (ACD) and have 13.6TB uploaded so far.

I'll be away for 2 days starting in the morning (picking my daughter up from university) so I'm trying to get the hardware setup the best I can so I can remote in and keep things moving along properly.

Once I get 2 & 3 done I'll be trying a 3 disk test just to get the hang of it. I'll definitely keep notes of any "gotchas" or "goodies" I come across and will post them here.

Carlo
78

Right before a 6 hour road trip I try and “push” getting the raid setup. (shouldn’t have done that)

I get TRaid installed and try go to set it up with 4 external WD USB3 4TB drive and a 6TB WD Red internal for parity.
TRAID does not like WD My Book drives as they don’t pass the 512 byte direct access test.

SO I open one of them up and pull the drive from the enclosure to install in the computer. I was going to install two of these in the computer to play with (remotely).
Trying to be lazy I’m pushing wires a bit to much to make room to slide it in and “snap”. On one existing drive I snapped the SATA connector off.

Wanna guess which drive it was? Yep the boot disk. :frowning:

Lucky for me most of the “pins” were sort off ok but all the plastic was gone so I couldn’t get a good connection. After an hour of jury soddering and jury rigging, I was able to shim a sata cable with tooth picks to fit snug enough to get the system to boot. Dropped some crazy glue on it to hold for now. Needless to say I immediately started coping as much off the disk as I can.

I’ve got a 1 TB Samsung 850 Pro SSD I was going to install in it but hadn’t gotten around to it yet. Looks like this will be the first thing I do tomorrow even when I get back home.

I’m also going to order one of these http://www.startech.com/HDD/Enclosures/8-bay-removable-hard-drive-enclosure~S358BU33ERM
I’ll then pull 8 of the drives out of the My Book enclosures to mount in this. I’ll connect the box via eSATA (reminds me I need to order a eSATA with multiplier).

79

Eeek...  good luck.

The SSD will be a great improvement, I'm sure.

I've had four different Startech external enclosures, and all have proven themselves reliable, for what it's worth.

80

I’ve been meaning to upgrade my Plex server to a couple of SSDs just haven’t gotten around to it.
This just pushed me over the edge quicker. At least I already had the SSDs on hand.

I’ve got 2 SSDs plus a spindled drive in my main Notebook and an SSD in my other couple of notebooks and wouldn’t want to have a “daily driver” without them anymore.

Not so much as big a deal for a typical server but with plex’s metadata it will probably come in handy.

Good to hear on the Startech enclosures. They seem to get pretty good reviews. I looked around at both going with a big case (ie 20 bay) or keeping the standard case and just adding a couple of 8 to 12 bay enclosures. These seemed like the best bang for the buck with the features I wanted. HDD cooling will probably be better this way as most typical cases get “hot spots” in the case unless you use an ungodly amount of fans which get noisy.