yes I think. get on the sickrage fork that is actively developed
It sounds like you already know what to do. :)
There are a bunch of guides available for this that you can use if you have any questions.
Since we are talking tips and tricks here.... not directly related but am curious what people are using encoding wise these days?
I want something that is as close as possible to the original quality but not 30+ gigs per movie. What I have settled on for now is mkv with 14000 mbit h.264 video and 1.5 mbit DTS audio in a mkv container (strip out all the other audio/subtitle options). I personally want plex to handle the audio transcoding if necessary vs rip extra audio tracks for my non 5.1 systems.
Also starting to think about how to handle atmos because it seems like that is going to be the next big movie audio format and combined with h.265 video... will depend on how fast players start to support those codecs.
I use h.264/AAC M4Vs through handbrake set for constant quality instead of constant bitrate...it can vary WILDLY, though based on the movie. Michael Bay blu rays will be double to triple the bitrate of a more talking-head rom-com...but that's the point of CQ encoding.
As for Atmos, it's a waste for consumer media. So few blu-rays even bother with 7.1, you can't expect hardly anyone to bother with 16.4 or whatever nonsense it is. NOBODY will have that setup at home.
Also, for TV/PVR i find the MediaPortal channel does an admirable job with the notable exception being a lack of a real EPG (a plex channel UI issue, not a mediaportal issue)
I use h.264/AAC M4Vs through handbrake set for constant quality instead of constant bitrate...it can vary WILDLY, though based on the movie. Michael Bay blu rays will be double to triple the bitrate of a more talking-head rom-com...but that's the point of CQ encoding.
As for Atmos, it's a waste for consumer media. So few blu-rays even bother with 7.1, you can't expect hardly anyone to bother with 16.4 or whatever nonsense it is. NOBODY will have that setup at home.
The varying wildly is what gets me.... the movies I would have expected to have a higher bitrate didn't and certain talking head movies were larger. I know its supposed to be consistent but it wasn't in my tests but maybe I was using too low or high of an integer... what integer do you use?
That is the whole point of atmos.... its one audio stream for all channels and backwards compatible. IMO its very likely everything become atmos in the next year or two. Already a few discs out there now.
I use 20. And for reference, Transformers Dark of the Moon wound up at 11mbps, where as Rounders wound up at 5mbps...both from BluRay.
That said, color/contrast seems to play very heavily into it (which makes sense based on how the blocking in H.264 and most codecs works)...Harry Potter all compresses really well (I have all 8 on BluRay and they range from 3-6 mbps), but most of the movies have a dark brown/blue palette.
As for Atmos being backwards compatible, that is intriguing...does it work on, say, stereo or 5.1 without downmixing on hardware? or do you still need to either transcode it or downmix it?
Um I already have built diy speakers foran atmos setup. 11.1.4 or 13.1.4 when 2nd gen receivers are available and cheaper. Im pretty avsforum hardcore though. Still need to build the ceiling speakers but no hurryAs for Atmos, it's a waste for consumer media. So few blu-rays even bother with 7.1, you can't expect hardly anyone to bother with 16.4 or whatever nonsense it is. NOBODY will have that setup at home.
Object based audio will be the new standard as it does away with needing multiple channel mixes. Atmos and Dts's system can upmix any audio not of their format or take all the sound objects and place them in realtime using how ever many speakers you have. 2, 5.1, etc doesnt matter. all the way up to some large speaker number. Atmos processing computes what sound to play on what best combination of speaker to try to get the sound object positioned as close to possible in the 3d space you have vs where it is placed in the mix. Neat stuff
Next will be a storage spaces with no parity, mirror, parity with only HDD tested with both NTFS and ReFS to see what IOPS I can get
I'll play with both "thin" and "thick" provisioning of the above.
Then I'll play with storage spaces using "tiered" SSD and HDD combined for meta-data use using both NTFS and ReFS with no parity, mirrow, parity and I'll also check IOPS again in these configurations
For this set of tests only "thick" provisioning can be used of course. No "thin" provisioning when using "tiered" storage.
I recently spent quite a few weeks working on replacing my storage system specifically for plex storage. One of the ideas I tested with was running storage spaces as iSCSI targets to ESXi with 4x 1GB interfaces. I also tested running Plex on the server directly as well using ReFS and NTFS. This was all on server grade hardware from HP. I did not have an SSD tier on it at the time, but was something I would have tested if the base performance was up to expectations. Unfortunately, I wasn't very impressed with the performance in parity mode. In Mirror more I read that its phenomenal. Unfortunately I didn't want to give up so much space and money for that level of redundancy.
Regardless of my past testing and my own conclusions. I wanted to point out that I discovered Plex has an incompatibility with ReFS. It can read the files great, but it cannot detect new files automatically as they are placed. Plex must be using one of the NTFS functions that were not carried over to ReFS. There are a few posts about this on the forums. Much to my dismay. But if you are content with the periodic scans, then it should work just fine. I had far too many files for the periodic scan to be efficient.
As for Atmos being backwards compatible, that is intriguing...does it work on, say, stereo or 5.1 without downmixing on hardware? or do you still need to either transcode it or downmix it?
As it was explained about atmos is a object based system and not channel based so your receiver reports to atmos how many speakers you have and where and atmos determines how to best play that back. If it does not see the height channels atmos will just play the "3d" sound in the normal 2.0, 5.1 or 7.1 mix you have now. It is just not discrete channels like it used to be so its far more flexible.
Here are the movies that are already atmos ready, already a pretty decent mix. http://www.dolby.com/us/en/experience/dolby-atmos/movies.html
What will be interesting is what happens with dts's version because they are in danger of being left behind.... all you hear anybody talking about is atmos right now.
Time for a new post concerning storage spaces.
As previously mentioned I run Plex on an old generation i7 running Windows Server 2012 R2. So I have storage spaces available to me.
Storage spaces is also available in Windows 8/8.1 and works just as well.
I wanted to go through an exercise of how you could build out a striped array with minimal new hardware while keeping your system running the whole time except for brief reboots. My current "server" is old and doesn't support USB 3 on the motherboard. I have a cheap PCI "rocket" board card that has qty 2 USB 3 ports. I have each port run into a 4 Port Mini USB HUB for a total of 8 USB3 ports.
I tested this with 6 existing WD 4TB drives I had. While at Walmart I noticed they had this USB3 WD 4TB drive on sale for $130 but only had one drive left so I purchased it. Later in the day at Best Buy I checked and they had the same sale so I picked up 1 additional drive. So this allowed me to start with 8TB of new space for a total of $260.
Going forward I'll refer to the two new drives as drive 1 and 2 and the existing drives with data I already had as drives 3 - 8)
Step 1: I connected and added the 2 new drives to the system and configured them as an 8TB storage pool (no parity).
Step 2: I copied the contents of drive 3 over to this storage pool.
Step 3: with drive 3 empty I removed the volume and added this drive to the storage pool for a total of 12 TB.
At this point I have 3 drives in the array which is the minimal amount of space needed to create a striped array.
Step 4: I created a 3 column striped array and copied the contents of the previous move to this now striped array.
Step 5: I copied the contents of drive 4 into this striped 3 column array
Step 6: With drive 4 now empty I added this to the Storage Array
Step 7: I created a 4 column striped volume on this new array. So this array now has a 3 column and 4 column volume on it.
Step 8: I copied the contents of the 3 column volume to the 4 column array
Step 9: I copied the contents of drive 5 to the 4 column array
etc, etc...
So basically I would copy the existing drive into the largest X column array I had, then add the bare drive to the array and start over again.
I did this until I ended up with an 8 drive striped array.
So what this means is that 7 drives hold actual data and 1 drive holds parity data. I have "full use" of 7/8ths of my total storage.
When creating each volume I selected "thin provisioning" which simple means the total size of the volume IS NOT allocated up front but is allowed to grow as needed.
To be honest I didn't do exactly as the steps above indicate. At some points along the way I had 4, 5 & 6 column volumes all active at the same time, then 5,6,7 columns, then 6,7,8 then 7,8 until finally just 8 columns!. I just kept coping data into the array and would immediately use the new "empty" drive as soon as possible. I'd copy the lowest X column array data over to the largest X column array during the moves.
In the end, all drive in the 8 column array have equal use and equal free space. It took 8 days total to accomplish this set of tasks. Along the way at 6 columns I pulled a drive from the array and inserted a new drive and had the array "fix itself" to make sure everything worked as I thought.
The pros and cons of this arrangement:
With the USB 3 drives I now get over 300mb read from the 8 column PARITY array but about 80mb sustained write to the 8 column parity array. Since I only write my data one time I like this as I'll get faster reads then using single mounted drives at around 100mb.
However since I'm using thin provisioning I also created a 2 TB volume using basic striping (no parity) and copied the plex meta database and transcode directories over to this drive. In 8 column basic striping I get around 300 mb read and 250mb writes. PERFECT for the meta info directories. I back these paths up daily to another drive.
Previously on Xbox, Roku and web clients the menus had lag to them. Now they are super quick and responsive as if the meta directories were stored on SSD drives.
Storage spaces only allows a max of 8 drives per array using simple 8 column parity (more if you mirror your arrays).
Speaking of SSD drives. So maybe next week I'll add a 1 TB drive as a separate storage tier. In storage spaces you can combine tiers of storage and use faster drives as a cache or set them up to hold the most recently used data. You can also setup part of an SSD tier to use as a buffer or "write cache" for the HDD tier. Only downside is that when using multiple tiers you can not use thin provisioning but only thick provisioning.
NOW THE IRONIC PART:
I have 2 different RAID controllers and have high speed 6GB SATA controllers.
I setup similar storage tiers and hardware based arrays using WD 6TB drives.
The USB 3 EXTERNAL drive arrays are faster!!!
This caught me off guard as I wasn't expecting it. But when you think about it most PCI controller cards only have 2 lanes of throughput at up to 6GB each. Put 8 drives on 2 lanes and the bandwidth is reduced. On the other hand even with 2 USB3 hubs inline the 8 USB 3 drives weren't able to saturate the 5 GB (4.8) lane each had. :)
The bottom 8 drives are used in this array.
My internal WD Red 6TB drives are now used to backup the USB drives. :)
I'll be changing things up shortly but for a week or so will run the system this way. Maybe longer. I will be moving the meta drive directory off the striped array onto a mirror of SSD drives. But for now I want to see how well the 8 external USB drives can function all by themselves on a busy Plex server.
I'm in play mode over the holidays as I'm working from home and have "extra" free time. I'll cover how I "redirected" folders via symlinks in an upcoming post which might be useful for windows users. Linux users are probably already family with them.
Thanks for posting this! Most of it goes beyond my needs, but post #6, regarding Agents, fixed the cast member issues on almost all my 150 movies.
Thanks again, and a Happy New Year!
Is the storage spaces on WS 2012 basically just RAID 5? or can you go for more Parity? Can you mix hard drives sizes for the pool? This sounds pretty cool.
Edit: Nvm, YouTube ftw.
https://www.youtube.com/watch?v=O30oMkaO7fk
Very nice posts OP. Informative, well structured and is truly what makes this forum great. Highly skilled people helping each other out without looking down on them.
For us running things in Linux I can add that there are several sweet options for the automation part re downloading tv-shows automatically (and also to a degree movies*). I guess, If you've taken the plunge into the Linux-world tihs might be old news and easypeasy (pun intended) stuff but.. I've set it up as such that rtorrent/rutorrent takes the rss feed and matches it against regexp statements, downloads and upon completion it automatically calls for Filebot to unrar, rename and move it to my TV-folder all neatly structured in correct folders. Plex then does its magic once every half hour which ofc be done automatically too in case of changes to folder, but I've shared my folders via NFS so in my case it doesn't work. Voilá, every morning my TV section is fully updated without me lifting a finger :)
(*I know that there are better torrent rss readers such as Flexget that has plugins like IMDB for making movies automatically download, but hey.. haven't gotten that far, yet)
Personally I'd go ZFS for storage.
As an overview of my smallish setup -
I have about 60TB on 3 continents for my Plex boxes (20TB in Bangkok, 20TB+ in Shanghai, and 20TB+ in Cape Town).
I usually split up TV and Movies onto separate boxes region.
Using HP N54L nas's with 5 x 4tb's in ZFS RAIDZ1 + OS drive on SSD (mostly as I've had multiple drive failures on the OS drive and have gone SSD due to price - its cheaper now than HDD in China for a small SSD). I have 90% of the setup scripted in ansible, so its pretty fast to add boxes.
For remote access I usually install Hamachi and setup a network (invaluable for remote access!) for all the NAS's.
Only really issues I have are related to power outages - especially in Cape Town. Plex does indexing on the SSD's.
I probably have to setup a proper re-encoding/transcoding box for 4k / h.265 files as Plex is still a bit flaky with those and tries to transcode when it shouldn't on occasion. I only have 4k tv's in Cape Town and Shanghai so its not a huge deal yet). Mostly the low end HPN54L's are fine streaming HD stuff otherwise.
Only issues I have are the slower continents timing out on page loads.
China is also a bit of a pain in the ascii as a lot of stuff is blocked, so Sickbeard/rage etc get to be a bit complicated.
Torrents work really well in China, so I usually go with SickRage, NNTP works semi ok in Cape Town, as its throttled to hell and back.
I keep movies on separate download boxen as its too painful running different ports for torrent clients for Couch Potato, and Couch Potato really doesn't play nicely with others - it likes to rename files it shouldn't. Having it completely separate works better I find.
Have a seedbox setup for syncing in France (my brother is playing around with that), but I think the internet will eventually improve in South Africa to the point of being usable. Right now its the only continent I can't stream from.
Everything is mostly automated - files rename correctly, only housekeeping I need to do is check the download directories every now and then and cleanout, and fix misnamed stuff - usually happens where tvdb and I disagree - eg American Dad or WSOP..
Most continents I have a "smart tv" so Plex can be built in.
I'd like the Android app to be better - some things are really annoying, like not being able to navigate to next/previous episode from within a given episode. The web app allows you to do that. God knows why its not an option, but I'm not the Android developer. Little niggles like that aside, it mostly works well as a media library player and indexer, especially with a small-medium sized library like mine.
shanghailoz, nice post.
First off I have to ask. Why multiple servers running in Thailand, China and South Africa?
Have you tried using something like Bittorent Sync to sync up your servers or for your brother to use?
Are your machines co-located or virtual machines running in a data center?
Not that it matters but I would agree ZFS is a strong contender to use for storage if you have the knowledge and the hardware for it. I thought about going that route myself but instead tried to stay on the Windows side of things. So far I'm really happy with the way Storage Spaces is working but I'd be just as happy with ZFS except for it's high resource use.
Just curious but how many unique movies and tv show episodes to you have between your servers?
Carlo
I travel between locations for work, so I like to have access to stuff. A lot of stuff is replicated - eg TV shows. Movies I generally don't bother - I let things do their thing.
China the streaming stuff has come a long way along though, I *almost* don't need Plex in China.
Youku etc all have decent apps with popular English language stuff, plus there are a bunch of HD streaming channels with movies - eg there is a Jackie Chan movie channel, Steven Chou movie channel (which usually have english subs).
Haven't tried BT Sync - given the file sizes I don't think its really feasible for China -> SA due to the ***heavy*** throttling of torrents in .za. If I do need to sync stuff I do rsync though., rsync over ssh is acceptable, although takes a while. Shanghai -> Bangkok would be ok though, but I can stream from Shanghai without too many issues so don't bother with that. I was in a beach cottage in Koh Chang earlier this year streaming stuff without issues from China.
ZFS isn't resource heavy - needs ram, but otherwise fairly low key. My HP NAS's are low end 1.5 -1.8ghz AMD boxes. Usually stick 8G of ram in them though. Plex usually is the cpu guzzler vs anything else.
The NAS's are in my apt's. Cape Town is 4M/512kb ADSL, Shanghai 100M/20M Fibre, Bangkok 50M/ (?? up) cable
Not sure how many unique movie and tv shows - how do I check?
Going to give flexget and filebot a try, as CouchPotato really is crud, thanks for the heads up on that Peter_W
Cayars,
Wow, I thought my data was sizeable.... I was happy to see that I'm not the only person running with a pile of external USB drives! I'm going to have to work through this thread - thanks for mentioning it in another thread!
Movies: 1240
TV programs: 8039
Music Library: 142,000 songs (didn't look at my album count...)
I'm running an 8core cpu and it snoozes through most of what I do - I only have one household I'm sharing with so I rarely see more than 3 streams between our two households.
I've been looking for a way to deal with PVR w/o much luck and I see there's some discussion of that in this thread also!
shanghailoz,
You can get the counts easily right from the web interface. For TV shows you need to go into that library then switch to EPISODE mode and you can get the counts.
Ahh, have 3 different apartments makes sense why you would have 3 different setups for sure.
grandpacarp,
I have internal drives also but after playing with storage spaces I noticed that the striped USB3 drives performed faster then the striped internal drives. Some people have issues with USB 3 drives loosing connections but I've never experienced this at all. So for me at present it's working quite well.
My use of PVR might be different than yours. I did post what my objectives were so you can see where I'm coming from in my "integration" needs. For the most part my PVR recordings are throw away. I record a couple of local channels for NEWS in case anything is going on I might want to catch up on. Typically these shows don't get watched much.
I record a lot of sports shows and typically allow them to get deleted after a while. I do have an extensive NFL library but these are "condensed" games I capture directly from the NFL NOW package I subscribe to.
I also have a bunch of shows that different family members watch that I record for them. I have all these shows setup in my TV SHOWS library also but don't add the PVR records. I have automated software that grabs the shows via torrents for me. But sometimes different family members want to watch a show right after it aired and I won't have it in my TV Shows library for a day typically.
The other reason I like this PVR setup is that I can watch live stuff from my cell phone from any connected WIFI or 4G and the software mentioned in that post will take care of transcoding it for me in real-time similar to how plex transcoder works.
Quick check of the Shanghai one shows I have 598 Shows / 19035 Episodes on that specific server.
That does conflict with Sickbeard, which claims 25083 episodes.
Only about 6TB odd.
Cape Town should be larger sizewise. Most of the size is movies typically.
