Cayars - Setup walk through and some tips and tricks

General Discussions
581

@bhav111 said:
Hi,
So I got upto page #19 with all this awesome info and reading material but I think at that point it was way too much info. For now, I just have a simple Q for all. I’m building a new system. It should have 8x3TB drives. I’ll have all the data backed up to external drives as well. I wanted to use some form of RAID. Options are Storage Space, on board raid card (it’s fairly well known and stable MB company - ASRock - so getting a replacement board with same RAID card shouldn’t be an issue OR get some other software based RAID like FlexRAID or something. Please provide suggestions. Sorry, it’s a short post after reading 19 pages… half the brain keeps saying PLEX PLEX PLEX…

Since you mentioned Storage Spaces I’ll take it you are running on windows. I went down the path of using Storage Spaces for a while but moved away from it and never looked back. I myself would suggest you take a look at two pieces of software:
1 Stablebit Drivepool $29.95
2 SnapRAID free/donation based

You use DrivePool to setup your drives and to create a “pooled” drive letter that gives you easy access to all media using one Windows drive letter.

SnapRAID (better if named SnapPARITY) is used to create parity/checksums across your different drives.

Both of these pieces of software are extremely flexible in the sense that you can add and remove drives from your pool without having to “rebuild” everything from the ground up and you tend to have to do with many true RAID products. Drivepool allows you to combine drives of any type together as a pool. Doesn’t matter if you have 512GB drives or 10TB drives or any combination. SnapRAID allows you to start with 1 parity drive and add additional parity drives (up to 6) as your library grows.

The beauty of DrivePool over RAID is that you can loose a single drive and only loose the info on that spacific drive. So if you had 10 data drives and 1 crashed you lost only 10% of your media and not all of it with programs that “stripe” data. Now combine this with SnapRAID which creates parity across the drives and you can rebuild the data that was on that failed drive.

With 1 parity drive you could loose any one DATA drive and still be able to “fix” your data without loss. If you had 3 parity drives you could loose up to 3 drives and still not loose any data, etc

There is a bit more work involved in using DrivePool/SnapRAID then say a normal RAID setup but it’s also more powerful also. With SnapRAID you need to setup a schedule or manually run a “sync” process which builds/modifies the parity info it has on hand. It’s not done automatically whenever you add or change files. This is both a PLUS and a MINUS. To me it’s a PLUS.

Let’s say you accidentally delete a complete directory with hundreds of media files in it. With hardware based RAID or Storage Spaces you are in trouble because the operating system/hardware has just modified the parity it has on hand as the files are deleted. No easy way to get your files back reliable. However, with SnapRAID as long as you haven’t ran a sync job since your accidental delete you can have SnapRAID recreate the files since it still has all the info and parity information at hand to re-create your files!!!

So SnapRAID used properly can also act as a “backup” and you can restore data from it. This has saved my butt a couple of times. :slight_smile:

Both of these programs work just fine with external USB3 drives. When I first started using this combination I had 12 external 4GB drives. Both flexraid and storage spaces would have an occasional hiccup with the external drives which is what pushed me to find a different solution.

I still have about a dozen external drives attached via USB3 to my Plex server and have no problems. Drivepool has duplication built in to store your “prized media” or multiple drives. I use this for my “personal family videos” that I could never easily replace.

I’ve only touched the surface on these programs but have talked about them a lot in this thread.

Carlo

582

Thank you for the response. So from what I understand, Use Windows 10 if option available, Install and setup Stablebit DrivePool then install and configure SnapRAID. I already have 6 drives at home but remaining two will arrive in a week or two. Adding more drives once all has been set up shouldn’t be an issue correct?

583

Just wanted to share…Final buildout…
Intel i7-4790K
ASRock Z97 Extreme4
16GB RAM
NZXT S220 Case
8x 3TB for storage (4 and 4 WD RED and HGST NAS)
500GB Samsung EVO SSD for Boot
Windows 10
Now the fun times to put it together with kids running around at home :slight_smile:

584

HI Carlo, I was curious if you were still working on distributed trans-coding for windows? I know they have a Linux version going, but a Windows version would still be great.

Thanks for all you do, you’ve helped me out quite a bit with my install.

585

bhav, also see Cayars post here…

586

@Pope Viper said:
HI Carlo, I was curious if you were still working on distributed trans-coding for windows? I know they have a Linux version going, but a Windows version would still be great.

Thanks for all you do, you’ve helped me out quite a bit with my install.

Nope, haven’t even given it a thought. In reality if you use the scripts i provide to get your media in the correct format (IMHO) then you don’t really need to transcode. Hence no need to distribute the transcoding. :slight_smile:

587

That’s what I’m contemplating now. I’ve just gotten done with my second pass with Filebot, and I’ve run a test conversion with one of your scripts (Wow, that ffmpeg eats up the cpu), so I’m looking get a secondary box going to handle conversion duties. My big concern is that I have a various mixtures of blue rays, DVDs, etc, with various audio formats and video types, and I want to make sure that I keep the “best” available. The problem is I need to determine what that “best” is.

I’d hate to convert all my files, only to discover I reduced the quality in order to get consistency, if that makes sense.

588

Makes total sense and I think most of us went down this path.
I’ll suggest what I’ll tell everyone. Take a subset of your movies of different qualities and test them with the scripts. I think you’ll find due to the settings it may take a while to convert but the end result is 99+% the same quality wise.

I tried to walk a fine line in the “default” settings for speed vs quality If you know anything about ffmpeg you’ll see I “error” on the side of quality over speed within reason.

Obviously any videos already in 264 format will get a remux vs a transcode so there will be no quality degradation on the video for these videos. Also the audio tracks you keep will not be touched, but the scripts may create a new first AAC audio track (while keeping your other tracks).

Carlo

589

So, my new plex box is ready. Now to transfer everything from old HTPC/NAS setup to the new server. Are there any issues that I should watch out for? Has anyone done that?

590

carlos any updates on the multi pc transcoding?

591

@PXANETWORK said:
carlos any updates on the multi pc transcoding?

Read 4 posts up.

592

how do i use these scripts and where do i get them? im curious now and a total newb lol.

593

Explanation on use and links provided in #145 MP4 Auto Convert Scripts (remux and transcode)…

594

thank you so much it runs perfect.

595

on snapraid what if i have two pools and have two seperate arrays how would i set that up like let say i want 5 drives to be array 1 with 1 parity drive and 5 drives to be array 2 with 1 parity drive also do all drives have to be the same size?

596

Personally I wouldn’t break up the parity “pools” like you describe. In your proposed setup each array only has 1 parity drive.

What I would do is setup the two pools in DrivePool BUT combine them all for SNAPRAID use so that you end up with 2 parity drives protecting all your HDDs. This way you could still loose any 2 drives before the loss of data could occur.

BTW, the drives do NOT have to be local to snapraid. For example my parity drives are stored on a combination of NAS drives and other file servers. As long as you can “map” the drive, then snapraid can use it.

Carlo

597

im not sure if its been asked before but how can i use the MP4 Auto Convert Scripts in ubuntu. I am just starting with ubuntu therefore don’t really know how it would work. I am in the process of moving my entire collection of 24tb to a dedicated server so it might be a good time to convert as well.

598

If you look at my signature I have a link to the steps I needed to do in order to convert @cayars scripts over for the Linux on my NAS. (It’s a version of BusyBox which is modified from Ubuntu I hear.) You should be able to do the same conversions I did and make them work.

599

thanks MikeG6.5 . can you guys advice me on how i should transfer my data. it’s approx 24tb on my home server and I need to move it to hetzner server. I was thinking of first converting the media then uploading to amazon cloud drive and download those files to the hetzner server. It’s a long process but i’m not sure If there’s any other way to do it. My upload speed at home is 20mbps.

600

Yeah, sorry, but I would advise you to either buy or build a system that you personally have physical control of, and not rely on any cloud or services for hosting PMS on. I personally don’t have any trust in any company, after reading some of the EULA’s that are out there, that my data will remain just my data, and that it’s not going to get farmed out, deleted or modified out of my control.

20Mbps is going to take a fair amount of time to transfer 24TB. Several days at least, I would suspect. And possibly much longer if there are any problems. Converting it even with @cayars scripts is likely to take as long or longer than the actual uploads would take…