Cayars - Setup walk through and some tips and tricks

General Discussions
421

This is fine for the forum.

I have 5 members of my HOME. This is my father, daughter, son, x-wife and myself. Everyone else is a friend.

Each member of my HOME has a plex.tv account so they can use my system remotely. Except for my Dad all members of my HOME are actually remote. My father lives in the same house as I.

So basically you only want members in your HOME who are family or whom you trust. The only reason I use HOME is for the quick user change on clients like the Roku.

Carlo

PS When you go into the USERS tab in server management there are two choices on the left (HOME, FRIENDS). You want to make sure you choose the correct tab before sending out an invite. Just about a month ago while talking to @MikeG6.5 on the phone testing something I accidentally sent him a Home invite instead of a friend invite. He was quick to catch my mistake (so it happens).

So pay special attention to this when adding friends.

422

Thanks for that explanation, @cayars
What feature(s) (or faults) of HOME are undesirable for friends?

423

@cayars said:

@don.alcombright said:
@cayars , what is your best recommendation for generating a list of file locations (for files needed to be be converted)?

My original thought was the get the list of file paths of these files and then run it via a batch command 1 at a time so I could do lets say 50 a day or so and actually monitor the outcome easily, aka if audio messes up or i lose something i didn’t mean too etc etc. But I can’t seem to find a way to get the full list of files exported to a list, or better yet a full list of files that should be converted to a list.

something like this so I can monitor it in the bat file (and comment out the ones I want to do that day:

c:\python27\python manual.py -a -i “\hades\movies\3 Days to Kill (2014)” >> D:\Convert\log.txt
c:\python27\python manual.py -a -i “\hades\movies\8 Mile (2002)” > D:\Convert\log.txt
c:\python27\python manual.py -a -i “\hades\movies\8MM (1999)” > D:\Convert\log.txt

OK. first and for most I want you to shut down Plex and copy the plex database out to a new location to play with as a test. Re-Start plex again. This is just for safety and always a good idea to do when testing anything new.

Download and install SQLite3 binaries onto your plex server from here: SQLite Download Page
You will want SQLite3 and all the files below in the same directory.

What’s sharing from here on out is for Windows since it uses batch files but would be super easy to change for Linux. The important part is to get the idea how to do it.

create a new text file called transcode.txt and put the following in it:
.open ‘C:\PlexData\Plex Media Server\Plug-in Support\Databases\com.plexapp.plugins.library.db’
.output Transcode.bat
SELECT ‘c:\python27\Python C:\convert\manual.py -a -i "’ || file || ‘"’ FROM media_parts join media_items
on media_parts.media_item_id=media_items.id
where container=‘mp4’ and optimized_for_streaming <> 1
order by file;

In the above file change the first line to point to your copy of the database. After you are sure all is working correctly you can change this to point back to your real version.

Now all you have to do is run:
sqlite3 < transcode.txt

You can put that in a batch file if you want.

When you run that it fires up SQLite and uses the contents of the text file to tell it what database to use. It tells it to pipe the output out to transcode.bat (so you can run it when done). It also contains the SQL code used to run against the database.

You can adjust as needed (or I’ll help) to match your system and needs. The output as it stands will generate something like the following in the batch file:
c:\python27\Python C:\convert\manual.py -a -i “F:\TV Shows\Ongoing\Top Shot (2010)\Season 0\Top Shot - S00E01 - Additional Footage.MP4”
c:\python27\Python C:\convert\manual.py -a -i “F:\TV Shows\Ongoing\Top Shot (2010)\Season 0\Top Shot - S00E02 - Contestant Bios.MP4”
c:\python27\Python C:\convert\manual.py -a -i “F:\TV Shows\Ongoing\Top Shot (2010)\Season 0\Top Shot - S00E03 - Elimination Interviews.MP4”

Hope this helps!

Carlo

PS this one particular one above looks ONLY for MP4 files that aren’t “web optimized” (proper moov atom at start of file).

This would be the first iteration you would want to run. After this you can adjust it to look at things like ref frames, 10 bit color, no AAC as first track, or other types other than MP4.

Of course once you take care or your existing MP4s you can just run the normal conversion scripts against your library to take care of AVIs, MPG, MKV, etc.

I’ll help with the different iterations you will need but want to start you off with an EASY SQL to get your “feet wet”.

Awesome and thank you. I am in the process of step 1, it found about 15 movies only. Once those are done I will hit you up :), would you prefer this thread or PM?

424

@kieranc said:
Thanks for that explanation, @cayars
What feature(s) (or faults) of HOME are undesirable for friends?

Home has a user limit (15?), but the main difference visible is that Home users can have the PIN setup, friends can not. I switched some family members of mine that dont live with me (parents) to Home users. To step back a bit the reason I wanted to was because the Fire TV Stick has a TERRIBLE UI for navigating plex servers. I got sick of the monthly call of how do I find your server :). With a “home” user they are part of my server, just remotely.

425

@don.alcombright said:
Awesome and thank you. I am in the process of step 1, it found about 15 movies only. Once those are done I will hit you up :), would you prefer this thread or PM?

Once you process these15 movies do the same thing for a message that follows dealing with 10 bit videos. It’s in a message about 10 back or so but the key for this is the following transcode.txt file:

Create a text file call transcode.txt and put the following inside the text file:
.open ‘C:\PlexData\Plex Media Server\Plug-in Support\Databases\com.plexapp.plugins.library.db’
.output Transcode.bat
select ‘C:\Convert\ffmpeg -i "’ || file || ‘" -c:v libx264 -c:v libx264 -preset slow -crf 18 -acodec copy -map 0 -profile:v high -level 4.0 “C:\convert\done' ||
replace(file, rtrim(file, replace(file, '', ‘’ ) ), ‘’) || '”’
FROM media_parts join media_items
on media_parts.media_item_id=media_items.id
where media_parts.extra_data like ‘%AVideoProfile=high%%2010%’
order by file
limit 100;

That will search your database for any videos that are encoded with 10 bit video which also causes transcodes for many clients. Adjust the contents and paths above as needed.

This one will be good to run from time to time as the conversion scripts will just remux any video already in h.264 format.

I’ll probably make some changes to the python scripts to transcode these types of files instead of doing a remux. Just need to dig in. :slight_smile:

426

@kieranc said:
Thanks for that explanation, @cayars
What feature(s) (or faults) of HOME are undesirable for friends?

Read this:
https://support.plex.tv/hc/en-us/articles/203960236-Consequences-of-Being-in-a-Plex-Home

427

@hthighway and @don.alcombright 

I hear you, and all these reasons seem like reasons why you’d want friends inside your HOME
 seems more secure, more control over their accounts, etc.

428

On to the full library conversion, as the 10 bit script found none.

429

@don.alcombright well that certainly makes it easier. You will still want to run that from time to time until I put out a new version of the scripts that will handle this much better up front.

Carlo

430

Thanks for the Agent and Scanner reccomendation. I used it as a step by step guide and both appear in the selection.

Sadly no media is beeing indexed by Plex. Full log: http://pastebin.com/XcDk4AzH

What’s going on? It finds all the MP4 but doesnt add them?

EDIT: I’m an idiot, should read your tutorial properly


431

LOL, it happens to the best of us. :slight_smile:

432

@cayars, quick one for you. How many “conversions” can I run from my PC at once. As you know I am converting my entire library (10,000 or so titles in all including movies/tv), I have a 4 core, 16gb PC with Plex and the files hosted on a different machine, so the converter is remotely accessing the files. I have 3 running right now, but I wanted to make sure that was ok/not ok, could effect the quality etc etc.

433

Not a problem to do that. Actually the VERY BEST THING you can do is to instruct ffmpeg to only use 1 thread at a time with the “-thread 1” option.

By doing this you will get better compression as all routines use the same thread. Since you have 4 cores you could then run 4 separate conversions at the same time each with the 1 thread option.

HOWEVER, what I would not do is run it with the default “-thread 0” option which tell it to use all core while running multiple conversions which will give you degraded compression.

Granted this isn’t a big deal and we’re not talking more than 3% difference, but if you plan on running multiple and are going to do this to your whole library you might as well do it “right”. I can tell you how to do it or I can provide you with an update to a file that will do this for you if interested. Just PM me.

Carlo

434

@cayars
Is the post on the first page about agents still up to date so you can follow it

435

@goktytan
Yes, still works the same way and I’m still using that same setup.

436

@jjross asked how I use SnapRaid and to put together a tutorial: https://forums.plex.tv/discussion/comment/1113441#Comment_1113441

This fits in nicely with this thread so here we go:
from www.snapraid.it homepage
SnapRAID is a backup program for disk arrays. It stores parity information of your data and it recovers from up to six disk failures.

SnapRAID is mainly targeted for a home media center, with a lot of big files that rarely change.

Beside the ability to recover from disk failures, other features of SnapRAID are:

All your data is hashed to ensure data integrity and to avoid silent corruption.
If the failed disks are too many to allow a recovery, you lose the data only on the failed disks. All the data in the other disks is safe.
If you accidentally delete some files in a disk, you can recover them.
You can start with already filled disks.
The disks can have different sizes.
You can add disks at any time.
It doesn’t lock-in your data. You can stop using SnapRAID at any time without the need to reformat or move data.
To access a file, a single disk needs to spin, saving power and producing less noise.

SnapRAID is different from hardware RAID, unRAID, FlexRAID, ZFS, Storage Spaces in one IMPORTANT way.
All of the other programs/solutions just mentioned use real-time parity creation. SnapRAID only creates/modifies parity when you tell it to.

How is this an important distinction and is a PRO or CON???

Most people think of parity as a way to rebuild an array when you have a hardware failure. While this is true, it’s not the only reason to use parity. Parity information can be used for other reasons as well. So quickly I will tell you I have my system setup to “SYNC parity” at 4am. I’m usually done for the day way before this so this time of day works for me.

So in a nutshell, everyday at 4am my system will do a quick check to see what files are new or modified in my “array” and will recalculate the parity. I can also manually kickoff a parity sync anytime I want with a simple SNAPRAID SYNC command.

Now that you have the “background info” on my sync schedule I’ll explain a PRO of SnapRAID. So lets assume I do a big “user error” and delete all my movies that start with the letter “f” from one of my drives. I then copy other data to this drive over writing what was once my “f” movies. With other programs/hardware the operating system is quite happy to delete these files and since the parity is getting written in real-time all parity is up to date and the parity knows nothing of the now missing files.

Now in this example where I just wacked one of my libraries I can get it back without having to resort to “undelete” utils which probably wouldn’t work since I just overwrote them anyway. In this case I have the parity from my last 4am run. So I just move the “new” data back off the disk to free up enough room to restore my files. I issue the proper SnapRAID command line and POOF my files are back. In this case SnapRAID is able to compare the parity and to use the other drives to figure out what’s missing. As long as I don’t remove data from the other drives I can “restore” my files pretty easily. In the event I also changed data on other drives that could effect the parity calculations SnapRAID will tell me and bypass those files. It will always recover as much as possible and not just stop.

So that is just one simple example of a PRO. Another is the ability to check for bitrot.

The biggest CON of SnapRAID is that it’s not real-time and any new additions of data you add between SYNCs is possibly at a loss in the event of a hard drive crash.
So when I add lots of new data I typically kick off a manual run instead of waiting until 4am. As easy as “snapraid sync” from the command line.
It is usually quite fast to sync data unless you change/move data around on your drives which is not ideal with snapraid.

SnapRAID is perfect IMHO for media files that don’t change often. I wouldn’t use it for repositories that are used in business where files are constantly changing. But for media libraries where we typically keep adding data without changing what is already there it’s close to perfect. BTW, you can change your files but this just causes the sync to take longer.

With the above said I also use StableBit’s DrivePool (StableBit - The home of StableBit CloudDrive, StableBit DrivePool and the StableBit Scanner). I think most people are familar with this but just in case your not familar with it. DrivePool allows you to take a JBOD (just a bunch of disks) and combine them together into one large drive. DrivePool can go beyond this and can duplicate data on different drives. You can have multiple copies of files spread out on different disks. This would require you to have 2x or 3x the amount of storage to have one or two backups of your data. Obviously this provides protection that comes at the cost of double or tripple your storage costs!

The great thing about DrivePool is that you can remove a drive (or have a crash) and your POOL survives. Only the data on the missing drive is removed from the pool. You can remove a drive, put it in another computer and access (read/write) the drive. There is no propriatary format on the drive. The data is not striped for example.

So in the event of a hard drive crash, only the data on the crashed drive is missing from your pool. If you have 5 data drives and loose 1 drive you still have 80% of media online.

So at this point we know DrivePool is pretty great for use with Windows Plex systems. You can add additional space super easiely and never hit obstacles like with other NAS or hardware RAID. You don’t need to add drives in groups nor worry about striping.

NOW THE PROBLEM WITH DRIVEPOOL
Drivepool’s solution to “backup” is to duplicate your data over multiple drives. This requires you to have double the storage. This isn’t to hard when you only have 2 or 3 disks of original storage but becomes more of a problem as your library grows. What if you have 8 drives at present, do you want to purchase another 8 drives to store these dupes? I know I don’t!

ENTER SNAPRAID WITH DRIVEPOOL
Let’s assume we have 5 original disk in our pool. Instead of adding another 5 drives in order to duplicate our data the “drivepool way” we can instead add ONE new drive for SNAPRAID’s use to store parity on. SnapRAID once setup looks at those 5 drives and creates the parity for the 5 drives but stores them only on the parity drive. You could technically now loose a drive in your DRIVEPOOL, replace the drive and use SNAPRAID to recover your data.

As we all know you can sometimes loose multiple drives at the same time. What if you were using the Duplication feature of DrivePool and lost two drives? If these two drives happen to both have the original file and the duplicated file you just lost that file!

So with our 5 original drives and 5 “extra” drives (10 total drives) for DrivePool you could loose two drives and lose data. With SnapRAID you can have multiple Parity drives. With SnapRaid we could have our 5 original drives plus 2 parity drives (7 total drives) and we could now sustain a loss of any two drives without total loss of our data. For these same 10 drives in total we could have 5 data drives and 5 parity drives and it would take a loss of 6 drives for us to loose our data.

So as you can see as your collection of media grows you can further protect it by adding a new parity drive. As an example if you had 20 data drives you could protect them pretty well with the use of 3 parity drives. You would have to loose 4 drives in total for data loss. You could technically have 25+ data drives protected by 1 parity drive but I would not consider anything less than 2 parity drives.

Once you get the hang of how to setup and use the combination of DrivePool and SnapRaid you can get a bit more elaborate. For example you can use SELECTIVE duplication in DrivePool at the directory level. On my system I have directories such as MOVIES, NFL, TV SHOWS, EDUCATION, etc that match my Plex Libraries. None of these directories are duplicated. I then have a directory called DOCUMENTS where I typically store Word, Excel, PDF, etc. These are small files compared to my media files. This DOCUMENTS directory is setup to duplicate to 2 other drives by DrivePool.

This DOCUMENTS directory is really the only directory that will change during the day for me where I’d want to be able to quickly recover/access data in the event of a hard drive crash. Since this directory is only a few GB in size it replicates quickly and I always have the data available even in the event of a couple of drive crashes.

NOW ON TO THE SETUP
The first thing you want to do is TURN OFF all balancing options in DrivePool. You do not want DrivePool moving your files around automatically to try and keep the drives balanced as this will cause you grief with SnapRAID. I personally self balance my drives. I try to attain 250GB of free space on each drive. This allows me to add a lot of media to each drive during the day without DrivePool shifting things and making my parity go out of whack.

On my system I will typically do something like this:
Drive 1: Movies# to Movies\E
Drive 2: Movies\F to Movies\G
Drive 3: Movies\H to Movies\K
For some letters like T that use mulitple drives and just keep them “balanced” aphabetically.

Any time I self balance my drives to free each drive back to 250 I rerun a sync command which gets my parity all nice again. :slight_smile:

So back to the SnapRAID setup. So once you have turned off all the self balancing options in DrivePool you are ready to install and configure SnapRAID. This is simple. Just install it. :slight_smile: Then you need to edit the snapraid.conf file. Here is an example from my system shopped down to 10 data drives and 2 parity drives for easier reading:

Content Drives are: E, G, M, N, O, P, Q, R, S & T
Parity Drives are Y & Z (these are both network NAS drives)

So in the config file you setup what drives are parity and what drives are content. SnapRaid will write a single file to the root of this drive (in my config). I use a name that makes sense to me. My parity drives are numbered starting at Z working backwards. My content drives in this example start at E and go upward. For content drives I use the drive letter in the file name. At any time I can look at the root directory of any drive and can tell what drive letter it is this way. No guessing in the event of a problem.

SnapRaid only requires one content list to be stored but you can use multiple drives. I write the content to all drives. Just in case. Nothing like writing it to 2 drives and having both of these drives fail. :frowning:
content E:\snapraid.E.content
content G:\snapraid.G.content
content M:\snapraid.M.content

Next you setup the drives/directories you wish to apply parity to and give it a name. I start at “d1” and work my way up. You will also note I do not create parity from the root of the drive. Instead I set the “disk” to start with the GUID of the DrivePool disk/poolpart (PoolPart.aaed7454-f99c-4644-8463-cb727acc4eac). I do this because if you loose a drive and replace it DrivePool will create a new GUID. I then edit the config file to use the new GUID and then do a restore/recover op. This way as the files are being restored they are immediately available to DrivePool since they get restored to the new GUID being used.
disk d1 E:\PoolPart.aaed7454-f99c-4644-8463-cb727acc4eac
disk d2 G:\PoolPart.1abeb4f3-90e0-4895-82df-50df19529469
disk d3 M:\PoolPart.9cae94bc-8f75-4bf0-8f38-0ff2175a1d8d

Next up is the exclusions you want SnapRAID to ignore. I’m using the defaults except for:
exclude \CloudPart.e238a496-4114-4699-9782-d229dafa06ec\

I set an autosave duration so parity is save every so often instead of waiting till the end:
autosave 5000

Everything else is defaults. Here’s the complete conf file contents:
Note had to add “.txt” to file name for it to upload here.

Carlo

2 Likes
437

Excellent tutorial! This almost needs it’s own thread :slight_smile:

438

@vanstinator said:
Excellent tutorial! This almost needs it’s own thread :slight_smile:

Thanks :slight_smile:

439

I do believe it might be time for a re-write of the conversion scripts.
If I do this I’ll start from scratch to keep things nice and tidy.

I’m not sure exactly how I want to approach this as there are some conflicting situations developing in Plex land.
On the one hand I’ve always strived to remux a file instead of converting/transcoding it when the video is already in h.264 format thus not loosing any quality.

However this can cause problems if:
1 - bitrate is to high (say higher than 12Mb or 20Mb)
2 - uses 10 bit video
3 - uses a profile level higher than 4.0
4 - HLS development mentioned below:
Another new development I just became aware of last night concerns the way Plex does HLS if the file contains h.264 encoding. It basically just remuxes to TS/m3u8 without proper checks for keyframe alignment. This can cause stuttering with many devices (Apple, Nvidia, Xbox, etc). More info is contained here if you want to read it: https://forums.plex.tv/discussion/comment/1113596#Comment_1113596

I’m almost thinking it would be better at this point to totally re-encode all videos to a certain known format such as:
20Mb bitrate limit (still targeting specific quality)
8 bit video
profile 4.0
deinterlaced (if was interlaced to begin with)
Keyframes aligned properly for proper frames/buffers if remuxed by Plex to HLS (seem to be doing this much more often).

What do you guys think?
Carlo

440

@cayars said:

@vanstinator said:
Excellent tutorial! This almost needs it’s own thread :slight_smile:

Thanks :slight_smile:

I agree.

I was hoping to better understand how to recover the data and how you are checking that things are working properly.

How do you get notified if there is bitrot?