@cayars said:
@jjross asked how I use SnapRaid and to put together a tutorial: https://forums.plex.tv/discussion/comment/1113441#Comment_1113441This fits in nicely with this thread so here we go:
from www.snapraid.it homepage
SnapRAID is a backup program for disk arrays. It stores parity information of your data and it recovers from up to six disk failures.SnapRAID is mainly targeted for a home media center, with a lot of big files that rarely change.
Beside the ability to recover from disk failures, other features of SnapRAID are:
All your data is hashed to ensure data integrity and to avoid silent corruption.
If the failed disks are too many to allow a recovery, you lose the data only on the failed disks. All the data in the other disks is safe.
If you accidentally delete some files in a disk, you can recover them.
You can start with already filled disks.
The disks can have different sizes.
You can add disks at any time.
It doesnāt lock-in your data. You can stop using SnapRAID at any time without the need to reformat or move data.
To access a file, a single disk needs to spin, saving power and producing less noise.
SnapRAID is different from hardware RAID, unRAID, FlexRAID, ZFS, Storage Spaces in one IMPORTANT way.
All of the other programs/solutions just mentioned use real-time parity creation. SnapRAID only creates/modifies parity when you tell it to.How is this an important distinction and is a PRO or CON???
Most people think of parity as a way to rebuild an array when you have a hardware failure. While this is true, itās not the only reason to use parity. Parity information can be used for other reasons as well. So quickly I will tell you I have my system setup to āSYNC parityā at 4am. Iām usually done for the day way before this so this time of day works for me.
So in a nutshell, everyday at 4am my system will do a quick check to see what files are new or modified in my āarrayā and will recalculate the parity. I can also manually kickoff a parity sync anytime I want with a simple SNAPRAID SYNC command.
Now that you have the ābackground infoā on my sync schedule Iāll explain a PRO of SnapRAID. So lets assume I do a big āuser errorā and delete all my movies that start with the letter āfā from one of my drives. I then copy other data to this drive over writing what was once my āfā movies. With other programs/hardware the operating system is quite happy to delete these files and since the parity is getting written in real-time all parity is up to date and the parity knows nothing of the now missing files.
Now in this example where I just wacked one of my libraries I can get it back without having to resort to āundeleteā utils which probably wouldnāt work since I just overwrote them anyway. In this case I have the parity from my last 4am run. So I just move the ānewā data back off the disk to free up enough room to restore my files. I issue the proper SnapRAID command line and POOF my files are back. In this case SnapRAID is able to compare the parity and to use the other drives to figure out whatās missing. As long as I donāt remove data from the other drives I can ārestoreā my files pretty easily. In the event I also changed data on other drives that could effect the parity calculations SnapRAID will tell me and bypass those files. It will always recover as much as possible and not just stop.
So that is just one simple example of a PRO. Another is the ability to check for bitrot.
The biggest CON of SnapRAID is that itās not real-time and any new additions of data you add between SYNCs is possibly at a loss in the event of a hard drive crash.
So when I add lots of new data I typically kick off a manual run instead of waiting until 4am. As easy as āsnapraid syncā from the command line.
It is usually quite fast to sync data unless you change/move data around on your drives which is not ideal with snapraid.
SnapRAID is perfect IMHO for media files that donāt change often. I wouldnāt use it for repositories that are used in business where files are constantly changing. But for media libraries where we typically keep adding data without changing what is already there itās close to perfect. BTW, you can change your files but this just causes the sync to take longer.
With the above said I also use StableBitās DrivePool (StableBit - The home of StableBit CloudDrive, StableBit DrivePool and the StableBit Scanner). I think most people are familar with this but just in case your not familar with it. DrivePool allows you to take a JBOD (just a bunch of disks) and combine them together into one large drive. DrivePool can go beyond this and can duplicate data on different drives. You can have multiple copies of files spread out on different disks. This would require you to have 2x or 3x the amount of storage to have one or two backups of your data. Obviously this provides protection that comes at the cost of double or tripple your storage costs!
The great thing about DrivePool is that you can remove a drive (or have a crash) and your POOL survives. Only the data on the missing drive is removed from the pool. You can remove a drive, put it in another computer and access (read/write) the drive. There is no propriatary format on the drive. The data is not striped for example.
So in the event of a hard drive crash, only the data on the crashed drive is missing from your pool. If you have 5 data drives and loose 1 drive you still have 80% of media online.
So at this point we know DrivePool is pretty great for use with Windows Plex systems. You can add additional space super easiely and never hit obstacles like with other NAS or hardware RAID. You donāt need to add drives in groups nor worry about striping.
NOW THE PROBLEM WITH DRIVEPOOL
Drivepoolās solution to ābackupā is to duplicate your data over multiple drives. This requires you to have double the storage. This isnāt to hard when you only have 2 or 3 disks of original storage but becomes more of a problem as your library grows. What if you have 8 drives at present, do you want to purchase another 8 drives to store these dupes? I know I donāt!ENTER SNAPRAID WITH DRIVEPOOL
Letās assume we have 5 original disk in our pool. Instead of adding another 5 drives in order to duplicate our data the ādrivepool wayā we can instead add ONE new drive for SNAPRAIDās use to store parity on. SnapRAID once setup looks at those 5 drives and creates the parity for the 5 drives but stores them only on the parity drive. You could technically now loose a drive in your DRIVEPOOL, replace the drive and use SNAPRAID to recover your data.As we all know you can sometimes loose multiple drives at the same time. What if you were using the Duplication feature of DrivePool and lost two drives? If these two drives happen to both have the original file and the duplicated file you just lost that file!
So with our 5 original drives and 5 āextraā drives (10 total drives) for DrivePool you could loose two drives and lose data. With SnapRAID you can have multiple Parity drives. With SnapRaid we could have our 5 original drives plus 2 parity drives (7 total drives) and we could now sustain a loss of any two drives without total loss of our data. For these same 10 drives in total we could have 5 data drives and 5 parity drives and it would take a loss of 6 drives for us to loose our data.
So as you can see as your collection of media grows you can further protect it by adding a new parity drive. As an example if you had 20 data drives you could protect them pretty well with the use of 3 parity drives. You would have to loose 4 drives in total for data loss. You could technically have 25+ data drives protected by 1 parity drive but I would not consider anything less than 2 parity drives.
Once you get the hang of how to setup and use the combination of DrivePool and SnapRaid you can get a bit more elaborate. For example you can use SELECTIVE duplication in DrivePool at the directory level. On my system I have directories such as MOVIES, NFL, TV SHOWS, EDUCATION, etc that match my Plex Libraries. None of these directories are duplicated. I then have a directory called DOCUMENTS where I typically store Word, Excel, PDF, etc. These are small files compared to my media files. This DOCUMENTS directory is setup to duplicate to 2 other drives by DrivePool.
This DOCUMENTS directory is really the only directory that will change during the day for me where Iād want to be able to quickly recover/access data in the event of a hard drive crash. Since this directory is only a few GB in size it replicates quickly and I always have the data available even in the event of a couple of drive crashes.
NOW ON TO THE SETUP
The first thing you want to do is TURN OFF all balancing options in DrivePool. You do not want DrivePool moving your files around automatically to try and keep the drives balanced as this will cause you grief with SnapRAID. I personally self balance my drives. I try to attain 250GB of free space on each drive. This allows me to add a lot of media to each drive during the day without DrivePool shifting things and making my parity go out of whack.On my system I will typically do something like this:
Drive 1: Movies# to Movies\E
Drive 2: Movies\F to Movies\G
Drive 3: Movies\H to Movies\K
For some letters like T that use mulitple drives and just keep them ābalancedā aphabetically.Any time I self balance my drives to free each drive back to 250 I rerun a sync command which gets my parity all nice again.
So back to the SnapRAID setup. So once you have turned off all the self balancing options in DrivePool you are ready to install and configure SnapRAID. This is simple. Just install it.
Then you need to edit the snapraid.conf file. Here is an example from my system shopped down to 10 data drives and 2 parity drives for easier reading:
Content Drives are: E, G, M, N, O, P, Q, R, S & T
Parity Drives are Y & Z (these are both network NAS drives)So in the config file you setup what drives are parity and what drives are content. SnapRaid will write a single file to the root of this drive (in my config). I use a name that makes sense to me. My parity drives are numbered starting at Z working backwards. My content drives in this example start at E and go upward. For content drives I use the drive letter in the file name. At any time I can look at the root directory of any drive and can tell what drive letter it is this way. No guessing in the event of a problem.
SnapRaid only requires one content list to be stored but you can use multiple drives. I write the content to all drives. Just in case. Nothing like writing it to 2 drives and having both of these drives fail.
content E:\snapraid.E.content
content G:\snapraid.G.content
content M:\snapraid.M.contentNext you setup the drives/directories you wish to apply parity to and give it a name. I start at ād1ā and work my way up. You will also note I do not create parity from the root of the drive. Instead I set the ādiskā to start with the GUID of the DrivePool disk/poolpart (PoolPart.aaed7454-f99c-4644-8463-cb727acc4eac). I do this because if you loose a drive and replace it DrivePool will create a new GUID. I then edit the config file to use the new GUID and then do a restore/recover op. This way as the files are being restored they are immediately available to DrivePool since they get restored to the new GUID being used.
disk d1 E:\PoolPart.aaed7454-f99c-4644-8463-cb727acc4eac
disk d2 G:\PoolPart.1abeb4f3-90e0-4895-82df-50df19529469
disk d3 M:\PoolPart.9cae94bc-8f75-4bf0-8f38-0ff2175a1d8dNext up is the exclusions you want SnapRAID to ignore. Iām using the defaults except for:
exclude \CloudPart.e238a496-4114-4699-9782-d229dafa06ec\I set an autosave duration so parity is save every so often instead of waiting till the end:
autosave 5000Everything else is defaults. Hereās the complete conf file contents:
Note had to add ā.txtā to file name for it to upload here.Carlo
Gotta be honest, you can be a Dell H700 RAID controller used for $89 and just build a RAID5 or RAID6. Or you could go the software route with UNRAID and virtualize your Plex box.
Manually balancing drives? Absurd amount of effort.
