Cayars - Setup walk through and some tips and tricks

I think my biggest DOH moment was an upgrade from 1gb to 10gb network card in a server for a brand new company I was working for and I didn't know their environment.

Ordered like 6 cards.  5 for use and a backup card.

Got the "10gb" card installed and noted the new switch said 100mb for one server.

Pulled card, looked at it, reference number=check.

Switched cables, etc.

Went through 3 or 4 "levels" of support. Installed new drivers, etc.  You know the drill.

Switched cards a few times and same results.

Moved up the chain of support to finally find no one could help. Plenty of people remoted in to the machine and did the usual.

Company sent a tech agent out to our location. Found nothing.

Repeated and ended up with about 5 field tech agents coming to our data center.

None found problem.

A few weeks later a "lacky" in the office asked me how come I have two 10gb cards on my desk.  I said nothing but went and checked.

Yep 2 10gb cards when I should have only had one.  Lights flashed, bells went off, Pulled said card from server and replaced it with card from desk and boom 10gb.

Ended up finding out the new 10gb cards and the 100gb cards differed by one (last) digit which was cut off in the admin/setup console. Cards looks so much alike (for 10+ years difference) that no one realized anytime we "switched cards" it was with another older 100mb card. :(

About 10 different guys actually handled this card and no one figured it out except the "lacky". (still think it was dumb luck) <--what I tell myself anyway!

I won't mention the name of the company except to say that it begins with "C" and is involved in the networking world.

That's probably my biggest DOH moment and it took over a month to figure out.  That's the problem with upgrading multiple things at one time (switch, lan card, etc).

Carlo

Ok... this time I was the lacky...

Certain company couldn't get their plotter to work.  It was working fine a couple days prior, but all of a sudden the workstation wouldn't communicate with the plotter.  Other techs Reinstalled software, reinstalled drivers, replaced cables, tested plotter on other workstations (worked fine)... spent the better part of two days trying to figure out why this device just stopped working on this workstation.

Now this was back the day of very high priced CAD software and each application or module had their own copy protection scheme.  Many included hardware "dongles" that had to be seen attached to serial or parallel ports in order for that copy of the software to run on that workstation.  This particular system had no less than four dongles, two 9-25 pin adapters, a gender mender, and a 25 pin breakout box, all screwed together or zip-tied on to each other to form one, 9-1/2 inch long "mass" that hung off the port... between the workstation and the plotter...  yup.

The light came on when someone said... "It was working fine before we moved it in here from that office down the hall."

I pulled that mass of dongles off, flipped it end-for-end, plugged it back in... and we lived happily ever after.  10 minutes.

Here's a good story for you.

Back when I was young and doing anything for a $ I took a job for Xerox who used to do onsite tech support.

Being the new guy on the block I got "assigned" a case that many others failed to troubleshoot.

This was  back in the DOS world.  It was a legal firm and the problem was one particular computer would "type" gibberish in Word Perfect multiple times a  day.

I got there and figured out the problem in like 5 minutes.  But it took me 30 or more minutes to figure out how to explain the problem.

Seems that a near sited secretary with bad glasses and big bust who leans over her desk to read the print/contract to type in has issues.

Yep, every time she would lean over the desk her bosom would lay/touch the keyboard and "type gibberish" AND she never knew it.

The hardest part was trying to figure out how to write the report. I'll never forget the title:

"Big busted secretary would get assistance from her ample bosom typing contracts without her express approval":

I was an instant hit and the guys loved me!

Carlo

Give me the model number of your Seagate(s) and I'll see if I have one laying around to play with.  I'm not a fan of Seagate but have a small pile of them I could check.

Hard Disk Sentinel reports the two drives that constantly power down as  ST3000DM001,  They are 3TB Seagate Barracuda drives.

I've loaded SeaTools but the darn program hangs when it's out looking at my USB drives.

Hard Disk Sentinel (the trial version) sees them but doesn't appear to let me modify the settings.

Can you pull them from the USB enclosure and hook one up to SATA and try it that way?

Can you pull them from the USB enclosure and hook one up to SATA and try it that way?


I've suggested that in his other thread this morning :D

Great minds and all that :)

+1 for the Norco, Carlo.

We messed around with USB first, single drives, then enclosures - first USB (unstable) then eSATA.

eSATA was much trouble for us, because the most (e)SATA-ports aren't  capable of port-multipling/multilane. So are the most SATA-cards. Or only up to 4 devices (and we had 5-bay-enclosures). The only cards with enough port-multiplier we found are those with the old Silicon Image 3132-chip (for a short time everything was stable, yeah!) )and this one became unreliable with upgrading from WHS to Windows 8 (last driver update 2009, damn, and the onboard Windows-8-driver cased bluescreens each time the enclosure was accessed a bit heavier ie Plex scanning a directory).

In addition the enclosures are often coming with an own RAID-card and their own energy-management = interactions with the OS and other drivers = trouble. Maybe it works fine when using the RAID to create a single volume and it keeps you from messing with the port-multiplier-thing, but we don't like RAID and the fact to be pinned to the RAID-card that created the RAID. We are using DrivePool for years now, just for the reason that we can keep control of the data on each drive including to still access the data after a crash or on another machine. We never dared to take the step to one of the software RAID-solutions out there.

I don't know how you want to connect your server to the enclosure you want to buy - as far as I understand you would need some SAS-cards in your server for one or two of the mentioned as well or some eSATA-cards that offers multilane ... but I only took a short look on the links so maybe I'm wrong.

After 3-4 years drama and many loss of data we last year build a server in the 24-bay-norco case (Inter Tech here in Germany), with IBM M1015 (cheap from ebay), crossflashed to LSI. We like the drives spinning down (and have no trouble that it takes too long, maybe because DrivePool is standing reachable between Plex and the drives, don't know) so we added a line in the LSI-driver to enable the spin-down. Everything is fine now. Very stable. No dataloss and finally we are living much nore peaceful sind the eSATA-days.

As you now are using DrivePool, too, you maybe like to check out Stablebits DriveScanner ... it's very cool and the discount is very fair if you already bought a licence for anothe SatbleBit-product.
(The Hard Disk Sentinel doesn't like so many drives ... it tends to crash when we use it. I try it once a year again and again on several machines - never worked stable for us when all drives are attached.)

:)

Yep, this is the issue I'm facing.

On the plus side I have 5 sata ports on the motherboard and have an addin card that support 8 additional sata ports and does support port-multiplier so I'm not worried about that.

I'm staying away from any type of hardware based RAID or propriety controller cards.  My goal is that I can pull any individual drive and move it to another windows machine and be able to access the data or use off-the-shelf utility software on it.

Yes I have Stablebits DriveScanner installed also and it's a great piece of software.

@higeko I'm really glad you posted this.  I you probably read already I crossed off this case.  I had my heart set on using an external enclosure.  Not sure why but after reading your post I'm dismissing my previous thoughts and going back to the drawing board on my thought process.

So I'm starting out fresh and this case is what I'm going to get UNLESS I can find a reason not to.  The more I think about it and get feedback from people like yourself the more this just makes sense!

If you wouldn't mind.  Can you share more info about your current use of this case.  IE what motherboard, CPU, powersupply, etc you have in it and how many drives you are presently using.

Do you have any idea what the internal temps are inside the case?

With your current build.  How noisy is it?  I'm not overly concerned but don't what a "blower".

Carlo

How'd your testing with tRaid or flexRaid or whatever they call themselves these days go? I was initially looking at them last year when building out my NAS but eventually settled on unRaid due to slightly more relaxed licensing terms in regards to moving my NAS to new hardware if need be. I do wish I had an additional parity disk but at the moment I'm running few enough drives it really isn't an issue.

I actually just flat out stopped with tRaid/FlexRaid testing for a couple of reasons.

Was slower access than native windows or through Drivepool.

The tech support forum/support is almost non-existent.  The author is a one-man show and a bit arrogant even for my taste.  Tons of posts never gets answered, etc...

DrivePool + SnapRaid does everything I need so I'm done looking and settled.  One of the cool things I like about SnapRaid is that the parity drive can be a network drive or NAS so it doesn't take up "pool space".

Carlo

This is going to be a bit technical and needs refinement which I'll get to.  I literally just whipped this up and thought I'd share before I finish automating it.

The premise is to give "virtual library" support to Plex in a way we can do much of it manually.

Here is how it works.  In Plex we can edit the metadata and add tags.  One of the tags is "SHARING" and is also used (you guessed it) when you share with other people.  So for example if you only wanted to share things with a tag of "CARLO" then you add this as a tag and then select this tag "CARLO" when setting up your share.  Pretty straightforward for normal plex use.

Now what I've been able to do is make use of this "SHARING" tag.  So if I wanted to create a library called Recommended I would edit the movie meta-data via the web browser and and tag it VL-Recommended.  The "VL-" part is the preface for "Virtual Lib".  Technically, I'm not using it yet but will in the next release after a bit more testing.  added

So I could setup a Kids library and flag media with the sharing tag of "VL-Kids", etc

Now what I need to do is create a real folder on a drive called "Recommended" and lets say it's "C:\VirtualLib\Recommended".  I could setup "C:\VirtualLib\Kids", etc  I also need to create a real library in Plex called Recommended and point it appropriately just like any library.

Now the trick is that we will make use of sqlite3.exe on windows to run a small file with commands.  Mine is called "Recommended.txt" VirtualLib.txt and contains:

.open 'e:\com.plexapp.plugins.library.db'
.output VirtualLib.bat
SELECT 'mklink /J "C:\VirtualLib\' || replace(tag,'VL-','') ||'\' || replace(replace(file, rtrim(file, replace(file, '\', '' ) ), ''), '.' || replace(file, rtrim(file, replace(file, '.', '' ) ), ''),'') || '" "' || replace(file,replace(file, rtrim(file, replace(file, '\', '' ) ), ''),'')|| '"' FROM media_parts
join media_items on media_parts.media_item_id=media_items.id
join taggings on media_items.metadata_item_id=taggings.metadata_item_id
join tags on taggings.tag_id=tags.id
where tag like 'VL-%' order by tag, file;

The first line will issue an open statement and you put in the path of your database.  In this example I'm using a copy of my database located in the root of e:\.  I'd suggest shutting down Plex and making a copy of your db for test purposes until you have everything working correctly.  This is purely "read only" and doesn't modify the database so there should not be a problem.  After being satisfied change this to your actual database but keep a backup on hand just as a precaution (always good idea).

The second line sends the output of the run to recommended.bat

The rest is the SQL we want to run.

From the command prompt you would switch to the directory that has sqlite3.exe and recommended.bat in it and issue the following:

sqlite3 < VirtualLib.txt

On my system with a few moves flagged I get the following generated:

mklink /J "C:\VirtualLib\Recommended\127 Hours (2010)" "F:\Movies\#\127 Hours (2010)\"
mklink /J "C:\VirtualLib\Recommended\13 Going On 30 (2004)" "F:\Movies\#\13 Going on 30 (2004)\"
mklink /J "C:\VirtualLib\Recommended\17 Again (2009)" "F:\Movies\#\17 Again (2009)\"
mklink /J "C:\VirtualLib\Recommended\The 40-Year-Old Virgin (2005)" "F:\Movies\T\The 40-Year-Old Virgin (2005)\"
mklink /J "C:\VirtualLib\Recommended\Air America (1990)" "F:\Movies\A\Air America (1990)\"
mklink /J "C:\VirtualLib\Recommended\Air Force One (1997)" "F:\Movies\A\Air Force One (1997)\"

So if you are familiar with Windows junction points this should make sense what I did.  If not then the basic principal is that each of these lines creates a "link" to the real directory.  Plex will know the difference nor care.

BTW, the above Batch file needs to get executed/run to do it's magic.

That's basically all these is to it.  The part in the above file "C:\VirtualLib" is hard coded in the sql which you can easily edit to match your system.  You'll also see "VL-Recommended" VL-% in the sql which would NOT need to get changed to the tag you are using.

So the script looks for any tags you create that start with "VL-" and strips off the VL- part leaving the DIRECTORY NAME that gets inserted behind the hard coded "C:\VirtualLib".  So whenever you want to use a new VL-TAG value you want to create a new folder under C:\VirtualLib to match the tag value and also create a new library in Plex pointed at that location.

So this is a basic release that is mostly manual but it allows you to start using it now.  I plan on making it smart and parsing the tags. DONE

So for example it will look for any tags that start with "VL-" and know these are Virtual Library tags.  It will then parse the tag throwing out the "VL-" keeping the rest as the library name.  Then the only change that would be needed to the sql would be to set the virtual drive path.  IE "C:\VirtualLib\"  With the parsing just mentioned this could become "C:\VirtualLib\Recommended" or "C:\VirtualLib\Kids", etc  The last part is from the tag.

Right now this doesn't automatically remove the junctions if you remove the tags but you can remove the directory "Recommended" and re-create it which removes all junctions just prior to running the created batch file.  Of course all this could easily be setup to run as a task.

On my live system it takes only a couple of seconds to create 500 junctions.  I have this set to run everything from the task scheduler once per hour which is good enough for me.

I have one job that runs 23 times a day and only does additions.  One job a day that removes the directories (Recommended, Christmas, etc) then recreates them again which effectively removes any media no longer tagged, followed by running the batch file to recreate all junctions.

Let me know what you guys think,

Carlo

PS I didn't want to automate it yet because I need some eyes looking at it and the results you get.  Just because it parses the libraries for me doesn't mean it will for you so VIEW the created batch file to make sure everything looks correct.  Once we get some good responses and any kinks worked out (if any) we'll take it to the next step.

PPS While the above is geared for windows platforms it can very easily be changed to work on any platform that supports symlinks or junctions.

The premise is to give "virtual library" support to Plex in a way we can do much of it manually.

So in essence, it's like sharing a playlists with other users (if that was possible)?

Well actually what it does is create junctions in a folder so Windows/Plex thinks there are real folders in it.  Then by creating a real plex library pointed at the junction spot it loads those into the library which can be used as normal libraries or shared as such.

Nothing super exciting or new about it EXCEPT that it pulls the info from TAGS you modify directly in Plex.

The kind of cool thing about it is you can view your normal movies folder and apply filters.  Once you get your return set you can mass update a tag on those movies selected.

So it's probably as close to a virtual library we can get at this point with a bit of manual setup until Plex adds this functionality to the product.

Make sense?

Well actually what it does is create junctions in a folder so Windows/Plex thinks there are real folders in it.  Then by creating a real plex library pointed at the junction spot it loads those into the library which can be used as normal libraries or shared as such.

Nothing super exciting or new about it EXCEPT that it pulls the info from TAGS you modify directly in Plex.

The kind of cool thing about it is you can view your normal movies folder and apply filters.  Once you get your return set you can mass update a tag on those movies selected.

So it's probably as close to a virtual library we can get at this point with a bit of manual setup until Plex adds this functionality to the product.

Make sense?

I understood what you were doing with the junctions and the new libraries, just trying to relate it to plex functionality (or lack there of)

1) Basically you don't have to keep a master text file with all your junctions/symlinks in it.

2) If you move files/directors in plex it will follow the next time you run it.

3) Once setup running on a schedule and you have the Libraries setup you just edit tags as if the functionality was built into Plex.

I plan on using it for Recommened (like top 100 or 500 movies)

Create a "Seasonal" Library and tag movies for seasons or holidays (Christmas, New Years, Memorial Day, Labor Day) and have each of these tags create junctions to "C:\VirtualLib\Seasonal so I can have one library that I can switch out super quick different times of the year without changing SHARE privs for every user.

ETC

If anyone is interested I've created a few other SQL scripts I've started using to "fix" some things.

One thing that has bugged me is that if I move a file out of a library and then add it back in it gets a new "added" date. The scanner records the "file date" properly but doesn't use it for the "added date"  With some scripting you can "adjust" the database to have your added date equal to the file date.  I created this so that if I were to blow away plex and reload libraries I could run this and in a few minutes have it look normal shows file added dates going back years and not everything with today's date.

If you have your media in MP4 but have some files that aren't "Web Optimized" I've got a script to find the file names of these files.  The SQL can be modified as I did above to include a command in front of the file name and a batch file you just run that "optimizes" the files as a one off run.

I've got a script that counts the distinct VEVO links being used for music in case you want to know how many Music Videos you have.

I'm also going to work on a util to move BIF files from the metadata folders to the same folder with the files and update the database.  Then a 2nd util could be created to reload these BIF files if you ever lost your plex setup and had to resetup your libraries.  This also comes in handy for people who want to run the metadata files on a small SSD but can't handle the size with BIF files.

Oh I also have the "universal converted to MP4" which will remux or transcode files to MP4 and create an AAC audio track if not present.  I use this to save transcoding in real time.

Let me know if any of this is stuff you guys would want.

Carlo

3) I'm not a fan of the way Plex handles music.  It also only picks up 2/3 of my music collection.

Is this still the case with your setup or has it improved after the recent update? Just curious on your experience with it so far. Plex Music Goes to Eleven!

@cayars - I like how you use SQLite to generate the bat file.  I never thought of that.  Mind if I steal that for my scripting tool?

Is this still the case with your setup or has it improved after the recent update? Just curious on your experience with it so far. Plex Music Goes to Eleven!

Yes it did a lot.  I have the same set of music in Madsonic, Emby and Plex to compare and the track counts are roughly the same.  They each quote different artist counts however.  Not really sure who is right on that one.

I might have an issue that only affects OGG files but before I say more than that I need to investigate but won't have the chance for a couple of day (away).

@cayars - I like how you use SQLite to generate the bat file.  I never thought of that.  Mind if I steal that for my scripting tool?

Absolutely, you never need to ask if I put something up here.

btw the parsing of the filename took me a while as sqlite doesn't support the same built in string functions as SQL Server or MySQL which would have made that easier and tidier.

I wouldn't doubt if there is a cleaner way (without adding functions) but I'm not a sqlite guru as I don't use it other than when dealing with Plex. (me like "real" databases)

PS which scripting tool?

Changed/Edited the message above and added support to parse all VL-TAGS. You can easily see what was modified.  HOWEVER only new sql code is shown.