Cayars - Setup walk through and some tips and tricks

when I open powershell and copy/paste, I get this message:

The term ‘pip’ is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spellin
g of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:4

  • pip <<<< install requests
    • CategoryInfo : ObjectNotFound: (pip:String) [], CommandNotFoundException
    • FullyQualifiedErrorId : CommandNotFoundException

Did you install Python?

You can also:

Most distributions of Python come with pip preinstalled. If pip is missing, it can be installed through the system package manager or by invoking cURL, a client-side data transfer tool:

curl https://bootstrap.pypa.io/get-pip.py | python

Sounds like you didn’t install python for all users or didn’t select to have it add python to the path environment.

Try installing Python again.

@shpankey said:

@cayars said:

You’re already maxed out at 8 drives for the “pool” so this new drive won’t be part of any striping. That’s just the beginning of the problems you’ll experience with RAID or Storage Spaces in general.

That’s not really true, at least in my case. I can add to my RAID pool at any time. I have this RAID box…

http://www.datoptic.com/ec/thunderbolt2-20gb-twelve-6gb-bay-hardware-raid5-6-quiet-tower.html

…and buying another and plugging it in will expand my original.

Scale-able up to 256 HDDs, …supports daisy-chain to multiple T1216-S6 chassis’ for a 256 SAS/SATA drives can be connected. These SAS chassis daisy chain together via miniSAS cables and full support hot to expand the RAID from MASTER chassis. It also supports Array roaming - Move RAID to different system without rebuild and Supports Multiple RAID Array volumes, if you wish.

The speed of this thing is off the charts. My current one is 96TB in RAID 6 (so 80 useable, of that 15 left :dizzy: ). But I plan on adding another soon, so I will have two of these daisey chained for a total of 24 drives w/ 160TB useable in a RAID 6 setup. Did I mention the speed? I am tapping out Thunderbolt 2.

They now have 16 bay Thunderbolt 3 options, and those tap out Thunderbolt 3 (40G).

I was talking specifics for that hardware. Yours is different and uses the ARECA RAID engine which allows you to dynamically resize the array (last one built only). They are decent boxes for the money especially if you are running osx.

You do need to be careful with them as certain “features” can be finicky and known to have corruption problems. You definitely want to use it with a UPS. Your box is expandable to 128 drives if my memory serves me correct and not 256 unless this has changed recently.

Don’t let the speed quoted fool you because it depends on the drives used and of course the onboard memory really boosts benchmark speeds to something “fake”. Don’t get me wrong, it’s no slouch but non of these boxes can delivery data faster than the drives themselves unless the data is cached.

As an example of speed they claim 20GB compared to say a 6GB interface. But what if you have an 8 port card each with 6GB and each port connected to it’s own drive. Then you have 48GB of throughput. Get a 12 port 12GB SAS card and well… Then you can have single or multiple SSD drives for reading/writing to that sit between the computer and drives as well as GBs of RAM for caching…

You can build out your LUNs any number of ways and get outstanding performance these days.

But anyway as much as I love RAID I won’t use it for video as it can cause more problems than it’s worth. RAID is great for small groups of disks to gain redundancy and speed for things like databases, file servers, etc but come with to many “cons” for video storage of the nature Plex uses IMHO.

I use “left over” drives from my servers so I always have a mix of brands or types with different sizes.

@buckeyesfan said:
when I open powershell and copy/paste, I get this message:

The term ‘pip’ is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spellin
g of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:4

  • pip <<<< install requests
    • CategoryInfo : ObjectNotFound: (pip:String) , CommandNotFoundException
    • FullyQualifiedErrorId : CommandNotFoundException

python -m pip install in windows put python -m in front

Woops - wrong thread

@cayars i got this error and had it in the prior built , but everything still works

Please install QTFastStart via PIP, relocate_moov will be disabled without this module.
Invalid subtitle codec, defaulting to ‘[‘srt’]’.
64bit Python.

is there anyother way to install QTfastStart , because it doesn’t work via pip

@cayars said:

@shpankey said:

@cayars said:

You’re already maxed out at 8 drives for the “pool” so this new drive won’t be part of any striping. That’s just the beginning of the problems you’ll experience with RAID or Storage Spaces in general.

I was talking specifics for that hardware. Yours is different and uses the ARECA RAID engine which allows you to dynamically resize the array (last one built only). They are decent boxes for the money especially if you are running osx.

You do need to be careful with them as certain “features” can be finicky and known to have corruption problems. You definitely want to use it with a UPS. Your box is expandable to 128 drives if my memory serves me correct and not 256 unless this has changed recently.

Don’t let the speed quoted fool you because it depends on the drives used and of course the onboard memory really boosts benchmark speeds to something “fake”. Don’t get me wrong, it’s no slouch but non of these boxes can delivery data faster than the drives themselves unless the data is cached.

As an example of speed they claim 20GB compared to say a 6GB interface. But what if you have an 8 port card each with 6GB and each port connected to it’s own drive. Then you have 48GB of throughput. Get a 12 port 12GB SAS card and well… Then you can have single or multiple SSD drives for reading/writing to that sit between the computer and drives as well as GBs of RAM for caching…

You can build out your LUNs any number of ways and get outstanding performance these days.

But anyway as much as I love RAID I won’t use it for video as it can cause more problems than it’s worth. RAID is great for small groups of disks to gain redundancy and speed for things like databases, file servers, etc but come with to many “cons” for video storage of the nature Plex uses IMHO.

I use “left over” drives from my servers so I always have a mix of brands or types with different sizes.

Yeah, I’m aware… I just read your comment as a general term, so I was just responding to it, since you said: “That’s just the beginning of the problems you’ll experience with RAID or Storage Spaces in general.”

Anywhoo… I am using it on OSX btw, so yeah, definitely works well. Btw, the speed I am talking about is from my own tests, which pretty much match their tests as well. They are in no way being misleading and any test they are quoting I assure you w/ the same setup (box, drives) will get the same results. AJA does allow you to disable file system cache btw, which they do in their tests. I’ve ran this setup for over a year without a single issue, or corruption of any kind or any stability issues; I really couldn’t be any happier. Dat Optic is one of the best companies I’ve ever dealt w/ for support and I highly recommend them to anyone. They will work w/ you on a setup for any situation and help you configure it.

Not trying to sell anyone on anything here, do whatever suits you. Just sharing what I have and how well it’s worked for me. But not being able to expand your storage is not always an issue for RAID, that’s all I was saying.

got it to run, but saying no module named guessit?

quick question, can this be used to convert to h265 instead?

now getting error from rebulk.introspector import introspect
import error no module named rebulk.introspector???

btw, thought I should add this. This probably goes without saying for most of you, but in order for it to actually work for me, I had to do a few things for QSV. After some research I had to enable the Intel GPU in the motherboard options (duh). Then, once in Windows (using creators update myself, so may vary), install the Intel GPU video drivers.

@TwistedEndz said:
@cayars i got this error and had it in the prior built , but everything still works

Please install QTFastStart via PIP, relocate_moov will be disabled without this module.
Invalid subtitle codec, defaulting to ‘[‘srt’]’.
64bit Python.

is there anyother way to install QTfastStart , because it doesn’t work via pip

That’s a problem. QTFastStart is what reorganizes the MOOV item to make the files “web optimized” for streaming. Any files you processed should be reprocessed again after you/we fix this. The files are fine, just not optimized. Running them through it again after fixed is all that’s needed and should be quick.

Let’s get some info. What version of windows are you running and what version of python?

What do you get when you type “pip install qtfaststart” from the shell prompt? Did you have any other problems with the PIP installs?

Carlo

@shpankey said:
Yeah, I’m aware… I just read your comment as a general term, so I was just responding to it, since you said: “That’s just the beginning of the problems you’ll experience with RAID or Storage Spaces in general.”

Anywhoo… I am using it on OSX btw, so yeah, definitely works well. Btw, the speed I am talking about is from my own tests, which pretty much match their tests as well. They are in no way being misleading and any test they are quoting I assure you w/ the same setup (box, drives) will get the same results. AJA does allow you to disable file system cache btw, which they do in their tests. I’ve ran this setup for over a year without a single issue, or corruption of any kind or any stability issues; I really couldn’t be any happier. Dat Optic is one of the best companies I’ve ever dealt w/ for support and I highly recommend them to anyone. They will work w/ you on a setup for any situation and help you configure it.

Not trying to sell anyone on anything here, do whatever suits you. Just sharing what I have and how well it’s worked for me. But not being able to expand your storage is not always an issue for RAID, that’s all I was saying.

@shpankey, hey let me say first I apologize for the previous post. I was in a hurry and reading it now seems rather harsh and wasn’t my intention. It’s a fine box and if I were an osx guy I’d have no problem with owning one!

My original comments were really geared toward Storage Spaces which are built into Windows. I extensively tested and used this technology for a while (logged in this thread) and was just speaking of my own history with it and the downsides to how IT STRIPES and how it expands/contracts.

Let me share my philosophy about Plex storage so it’s a bit more clear where I come from with some of my comments. I design and work in data centers so I know most of the technologies out there and used most. So I’ve got a lot of experience with different hardware. I feel that “business” and “consumer” needs can be quite different.

As an example I’d have no problem telling a business they need to order 16 or 32 LIKE drives a pop (plus spares) but wouldn’t want to give this advise to a consumer for Plex type video storage. Speed isn’t a key metric I worry about compared to flexibility. I like having the ability to mix RED, GREEN & BLACK drives. Being able to use 2TB, 4TB, 6TB, 8TB, 10TB & 12TB drives together. Being able to use any drive on hand regardless of manufacture. I like knowing that if I have a drive or two fail and have an imperfect backup that at worst I only loose the data on those specific drives and not the whole array.

I don’t worry about speed so much because most people don’t have GB upload nor anything close to that so the speed of their drives is really irrelevant. A single drive can handle 10 streams. With more drives chances are the media is spread out over those drives. Cheap drives using SATA/ESATA at 6TB is plenty for this purpose.

So as you can probably read into it already for Plex purposes there is no need for RAID if you take other precautions like using SnapRAID which isn’t RAID at all but parity of your existing drives. You can have up to 8 parity drives with this open source freeware product. So with 2 or 3 parity disks you can cover really large storage systems made up of dozens of drives.

SnapRAID is a poor choice for “business” type files that change often but for “static” files is great. With our libraries we typically write data (files) and just keep adding but don’t modify them so they don’t change. SnapRAID is great for this.

With the software I usually recommend for Windows if you want a speed boost you can add an SSD or two which will get used for caching (reads and writes).

So for me it’s about LOW ENTRY COST that you really can’t outgrow regardless of system size and offers maximum flexibility (SAS, SATA, eSATA, USB2, USB3, UASP, ect) and allows a mix of technologies so you can adapt and change over time while still using all your older stuff.

So anyway, that is a fine box and I apologize if the post came off like I was attacking it or you as that was not my intent at all!

Carlo

No worries, I didn’t take it that way at all, was just trying to add a bit to it. Your logic and advice is definitely spot on (as always) and couldn’t agree more, it definitely is the way to go for anyone running mainly a Plex server. It’s safer, easier, cheaper, more flexible and more easily expandable on every level. Thanks for the extended explanation, I always learn something else when reading your posts. :smile:

I really need to get a SnapRAID setup going. I’ve read your previous setup articles as well and it really is a perfect fit for what we’re doing.

One thing though, I do plan on moving my RAID box over from my Mac to my PC tomorrow, once the Thunderbolt 3 to Thunderbolt 2 adapter arrives (using a Gigabyte mb w/ T3, but the box is T2). Mainly b/c it has a Kaby Lake i7 7700k and the iMac is just a middle of the road mid-2012 variety (old i5). I hope I don’t have any issues. It’s been literally flawless since I got it, but your comment about ‘especially on osx’ is worrying me a bit (implying maybe not so great on windows?). Anyhow, I guess we’ll see what happens. But I suppose that’s just in general anyhow, my Mac is an absolute rock solid system and never even needs a reboot… like, ever (unless upgrading the os), whereas Windows… not so much. Usually fiddling w/ something on it.

@cayars thanks , looking at this now
python V2.7 windows 10

PS C:\python27> pip install QTFastStart
pip : The term ‘pip’ is not recognized as the name of a cmdlet, function, script file, or operable program. Check the
spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:1

  • pip install QTFastStart
  •   + CategoryInfo          : ObjectNotFound: (pip:String) [], CommandNotFoundException
      + FullyQualifiedErrorId : CommandNotFoundException
    
    

REASON is windows 10 … i had to do this first
[System.Environment]::SetEnvironmentVariable(“PATH”, $Env:Path + “;C:\Python27”, “Machine”)

then i could do the commands , and here it says PS C:\Python27> pip install qtfaststart
Requirement already satisfied: qtfaststart in c:\python27\lib\site-packages

this is what i get now

Relocating MOOV atom to start of file.

C:\Convert>TIMEOUT /T 60

Waiting for 0 seconds, press a key to continue …

C:\Convert>goto start

C:\Convert>c:\python27\python manual.py -a -i C:\convert\process
Manual processor started.
c:\python27\python.exe
Invalid subtitle codec, defaulting to ‘[‘srt’]’.
64bit Python.

Web Optimized Yes <-------------

thanks it’s fixed

Hey Cayars, another quick question… are you sure QuickSync is working correctly? I ran into a couple of posts about needing to add h264_qsv in the .ini file. btw, when I did this, it DEFINITELY worked, but the output file was pretty bad. It took a 6G file and created a 600meg file in it’s place, that was blocky and not very watchable. It definitely worked though, CPU % hovers around 13%.

If, for example, I don’t have that option listed in the .ini (even w/ the qsv = true) and I change the h264-max-level = 4.0, my CPU % is 100% on all 4 cores of the i7 770k CPU. Here, this will explain what I mean better hopefully…

With these options:

  • video-codec = h264_qsv,h264,x264
  • h264-max-level = [4.0, 4.1, doesn’t matter]
  • use-qsv-decoder-with-encoder = True
    …then low % CPU use. It goes fast too. But file is small (600meg from 6G) and looks awful.

With these options:

  • video-codec = h264,x264
  • h264-max-level = 4.0
  • use-qsv-decoder-with-encoder = True
    …then MAX % CPU use.

With these options:

  • video-codec = h264,x264
  • h264-max-level = 4.1
  • use-qsv-decoder-with-encoder = True
    …then low % CPU use most of the time, but I’m not sure if it’s b/c the file is already 4.1 and so it’s just changing containers (?) and not re-encoding is why (that’s my guess is what’s happening)

Thoughts?

@TwistedEndz

Is the Python path added to your environmental variables?

I would also do this from just a normal regular command prompt. I’ve had issues w/ powershell and elevated command prompts on Win 10. Seems quirky sometimes.

Found this post, so it seems it does need added…

mdhiggins commented on Mar 4, 2016
Just pushed an update to add the h264_qsv codec as an option.
Update your autoProcess.ini settings to the follow:
video-codec = h264qsv, h264, x264

Give it a try and let me know if its working

Going to do some testing w/ your and his w/ that enabled. Curiously, I had used: h264_qsv before and it worked, he says to use h264qsv instead. He probably catches both options, I’m sure.