Optimize Folder and Library Structure for Very Large Plex Collections

Plex Media Server Desktops & Laptops
,
1

Server Version#: Version 1.19.1.2645
Player Version#: 1.6.5.1097-3bb9dc68
Operating System: macOS High Sierra 10.13.6
Media storage: 2 x Synology DS1817 (128 TB raw) formatted NTSF, connected iSCSI via 10 Gbe

THE QUESTIONS ARE… (Please no basic answers, I looking for ADVANCED user experience)

How does the folder size impact server performance and remote user experience?
Does having less, but much larger Libraries have a significant performance impact?
Does one or the other setup improve performance locally and for remote clients?
Are there setups that favor a more efficient database and metadata performance in very large collections?
Does NTFS file system perform better or worse than others? I have a macOS system but have storage volumes formatted as NTFS and run Paragon NTFS for Mac.

Any additional guidance regarding the topic of performance and management of very large Plex server databases will be appreciated.

BACKGROUND INFORMATION:
I have used Plex Server since May 2013 as a Plex Pass user. I have about 10,000 movies and for the past 3 years my typical movie is much larger. They range from 6GB to 60GB each. Most are remuxed with Atmos, DTS-X, TrueHD audio for my local Home Theater use and remotely the server transcodes as expected.

As I quickly built my collection of movies I got the impression that having over 500 or more movie subfolders in one folder would slowdown the directory listing. This lead me to create a series of nested and non-nested folders, each containing less than 500 movie subfolders each.

I got the impression back then, that having one big Library was inefficient because the scanning took very long and the display of the library artwork was slower. Then Atmos and 4K became common place and having separate listings (libraries) gave me quick access to enjoy these types of movies at will. All this lead me to create more and more libraries. I have 15 libraries now.

But over the years I improved my server and it now has 16 GB memory, SSD storage, multiple 10 Gbe iSCSI connection to NAS devices, etc. Although it is only a Mac mini (2012) quad core, it run great other that transcoding H.265.

It has become difficult for me to tell if folder structure size and library size have a big effect on the server’s efficiency anymore. You would think “if it ain’t broke, don’t fix it” but it is “broke” in other ways.

I have multiple “collections” of the same type as collections are “per library”. On remote client devices, searching is more complicated as it searches the current library only.

2

follow these, or don’t.

https://support.plex.tv/articles/categories/your-media/naming-your-movie-media-files/

https://support.plex.tv/articles/categories/your-media/naming-and-organizing-tv-shows/

https://support.plex.tv/articles/categories/your-media/naming-and-organizing-music-media/

3 Likes
3

In addition, specifically when organizing your movies, follow the guidance here for “Movies in Their Own Folder” (as opposed to “Stand-Alone Movie Files”):

It can help reduce scan times when new media is added, particularly when you’ve got your library configured to automatically detect changes.

2 Likes
4

Thanks for the comment. All media is in their own nested folder and the library types are appropriate to the type of media they contain.

My question is the large amount of folders and having many libraries vs. the advantage of a more condensed setup where searching and collections are more integrated as having multple copied of a film is common etc.

Thanks again for the comment

5

Thanks for your reply, but I guess my post is not clear enough. I’m not a rookie. The links are Plex Server 101 material. I’ve been doing this since 2013.

I’m seeking expert advice and guidance for and advanced user. Thank you very much, must be my long post gets my message lost

Thanks for the good intention.

6

heh, ok.

the file naming does not change no matter how large or how advanced you are.

what may change is putting content on multiple volumes to split up disk io load.

this may also keep large numbers of sub directories down.

since plex libraries can have multiple paths, you can organize the lower levels however you want.

so if you want to break up into genre folder or other means, you can, as long as the top level naming remains plex standard.

example~ movie library

/data/media/movies-current
/data/media2/movies-archive
/data/media2/movies2
/data/media3/movies-cartoons
/data2/media/movies

also, you may want to move off ntfs and paragon.
and get off of ISCSI, yuck.

move your media to a linux based storage server, use any of the top 3 file systems, ext4/zfs/btrfs.

the easiest way to increase your plex performance is to put the plex data on the fastest SSD system you can afford.

since plex uses sqlite for its database, and stores all of the metadata (posters/etc) within a huge mess of thousands of files/subfolders, you want the fastest IOPS and lowest latency access you can get.

for reference, my plexdata is on a dedicated 1tb SSD, my database is ~1.4 gig.

the entire plexdata folder is over 600 gig (with preview/thumbs) and growing, and media content exceeding 100tb.

when I first responded, you asked a basic question, so you got a basic answer.

in summary, when you start to get to ultra big, your options start going into big $$$.

start compartmentalizing, split up the load.

1 or more dedicated storage servers, storage only.
a small but powerful dedicated plex server, with fast SSD, a modern 7000 series intel cpu with igpu.
don’t virtualize plex, run bare metal. debian or ubuntu is best imo.
mount your storage to the plex server via NFS (not NTFS or iscsi)
any other apps, run on something else.

also, if you want mega server.

go big. FreeBSD Enterprise 1 PB Storage

7

Thanks for the reply. I’m sorry if I came off a bit frustrated in my response to your first post. I had been looking for a long while over days for anything other than the basics and it is hard to find any post at the level I need to guide me make a major decision like this.

I read NTFS was better with large files and SMB3 was better than AFP or NFS. But I realize this is all from the perspective of a macOS based server. I am open to switch to Windows, but going to Unix based OS is a whole other ball game! I am open-minded though if the benefits are clear and measurable.

So please everyone that has real experience, particularly if you have “personally experienced” more than one of these operating systems with Plex Server please share.

I need more info on NFS and Unix

  1. Is NFS just another “file sharing” protocol or on Unix does it act as an “attached device” like iSCSI does?
  2. If I go with Unix can I connect to the Synology iSCSI targets as is, or MUST I reformat and start over?
  3. What advantages does Unix have in my use scenario (Plex, ripping, transcoding)
  4. I see another limitation, I have a Lifetime DVDFab license. Do they do Unix?

For me MOVING the data is “almost impossible” as I don’t have enough devices I think to hold my data in the transfer process. So if I can do iSCSI connection to NTFS files I would entertain a discussion of the merits of Plex on Unix vs. Plex on Windows.

Coincidentally, I just ordered $2,300 in parts to build a new server, so this is good timing.

The basic setup is as follows:

  1. Asrock Tahiti M X399 Mother Board
  2. AMD Threadripper 1920X CPU (12 core/24 threads)
  3. nvidia Quadro P2200 GPU
  4. 32 GB of 3200 Mhz RAM
  5. Samsung 860 Pro SSD (for OS etc.)
  6. Intel Optane 905p 480GB SSD (for Plex database/metadata)
  7. Intel dual 10 Gbe NIC
  8. Thermaltake Core V21 case
  9. Thermaltake 850W Gold PSU
  10. Noctura CPU cooler (for Threadripper)
  11. Noctura 140 mm fan

I have always been 100% Intel. But reading up on Threadripper and a Quadro GPU combo I fell in love (LUST ACTUALLY). I believe this is way overkill for a Plex Server as it will do 30+ transcodes. My upload bandwidth would never support that. But it makes a great ripping/transcoding workstation. I could even consider setting up H.264 to H.265 transcoding automated workflows to save storage space etc. The possibilities are endless for the excess capacity.

I will appreciate any help from people that have REAL experience with these system, not just dropping an unsupported opinion.

Thanks

8

NTFS is windows file system. Trying to run NTFS on a mac, using paragon to translate the ntfs into something the mac understands is adding multiple layers of slowness to access your media.

isci is another layer of abstraction adding slowness, it is a file sitting on your synology, so that it looks like an attached hard drive.

this is not the way to go, especially for scalability.

you have put yourself between a rock and a hard place, until you have the additional space to duplicate your data onto something NOT ISCSI.

Like your synology native file system and network sharing.

You need to change your way of thinking, you do not need to store media on iscsi that looks like a fake hard drive.

You need to store your media directly on the nas’s, then mount those nas shares to your mac, or whatever you use for the plex server, via either NFS or SMB3/CIFS, or even AFP for the mac.

  1. NFS is a network fileshare protocol, just like smb3/samba/cifs or AFS. NFS is the native network share protocol for linux, and is generally the highest performance way of local network file sharing.

  2. the synology is already linux, you should really start learning everything about it that you can. You could even run plex directly on the NAS itself.

  3. it doesn’t really have anything to do with your ripping, you would still use your mac or whatever desktop you normally use.

  4. no idea about dvdfab, probably not, you just keep using as you currently do, you simply store your files on the NAS fileshare.

You really need to learn all you can about your NAS’s capabilities.

you don’t need a threadripper or quadro, you just need a 7000 series or newer intel cpu with a 600 series integrated gpu.

Sorry to be negative, but basically, it sounds like you have spent a bunch of money unnecessarily, and your whole existing architecture is causing you pains and will continue as long as you continue to do the same thing.

2 Likes
9

Thanks for your feedback. I know I threw a lot of money at this in an impulse, but most of us do that, be it with a overpriced projector and sound system, others with fancy vehicles…

But I say that in part because my system works great as it is. It only struggles with h.265 because of the aging integrated GPU. So it is hard to justify the cost, but that is my pocketbook.

I was hoping to hear more about the merits of Unix systems for Plex as an option I should consider maybe on my new build. This would be very helpful from people that have had both servers running for some time and would please share their thoughts.

Running Plex on a NAS is not applicable at all for me. I’m talking about a powerful server in place for years running 24/7 and looking to move to the next level.

Regarding “Paragon NTFS for Mac ver. 15” it is extremely efficient. This is not the freeware of years ago. I’ve tested it to read and write 95% as fast as a native Windows machine in my environments. I don’t know if you have personally experienced these technologies lately. They are much better than years ago. It’s not a virtualization or translation. It simply installs the file system to the Mac as another native file systems available. Just not made by Apple.

I respectfully disagree with you on the iSCSI issue in general. It works flawlessly and is very fast. I just posted a topic on my experience with iSCSI for Plex. Give it a read.

I had a few failed attempts years ago. But I learned a lot and this system is up running 24/7 serving transcodes to users from East to West coast, the Caribbean, and Europe. The mistake rookies make is following default setups and the load of unnecessary features that slows things down.

10

[quote=“The4Rum, post:9, topic:576955”]
It only struggles with h.265 because of the aging integrated GPU.
[/quote]Why are you reducing quality for your HomeTheater by transcoding HEVC for local use in your house, DirectPlay. Then you need no more hw…

Compartmentalize. Keep ripping/encoding box separate from plex server. Loading up all the cores transcoding for hours/days will by nature effect cpu time for PMS.

1:
Internal home theater use, DirecPlay: Can do as little as RaspberryPI.
Internal+remote people transcoding: Intel 7xxx iGPU.

2:
Ripping/encoding machine: threadripper
+what has already been said about dumping NTFS and iSCSI.

1 Like
11

Thanks Menel for the good advice. I will take it into account next time!

I can tell you’re a friend trying to keep me out of trouble and on the “straight and narrow road”. I accept I over did it with the Threadripper + Quadro P2200. But it was a “moment of weakness!!!” I had wasted so much time spec’ing out this “dream machine”. Weeks picking out every detail. And then… BANG! I whipped out the card and the rest is history…

Now the parts are arriving and I know the NUC i7 7th generation I have could have done it, but it’s just dual-core. Man, even my old Mac mini (2012) is a Quad-core + Hyper-threading. So the NUC seemed a bit weak. I thought the most elegant option was a new Mac mini (2018) spec’d out with 6-cores and 10 Gbe, but Mac has limitations Windows does not. And it cost as much as the “Dream Machine” anyway.

So it’s done… I’ll enjoy the bragging rights for years to come. And believe me, the dream machine will knock out Plex Server transcodes in it sleep. So I’m not worried about performance. My sever is not really that busy most of the day. I currently toddle my Torrent seeding on a schedule to accommodate peak Plex traffic times, so I can do the same with ripping and encoding tasks.

I know in principal you are correct, but I don’t agree with the blanket statements about iSCSI and NTFS for Mac. I have really optimized most things to near perfection. Also my 2012 Quad core i7 Mac mini must have enough horsepower in my setup that it as run smoothly for years.

About the transcoding I totally agree, but I may have said it wrong. I NEVER transcode locally. I discourage and some time prohibit my remote users from playing the 4K versions and especially any HDR content as that is crazy.

Locally I have all the equipment needed to direct play from server to an nvidia SHIELD dedicated as a Plex Player only set to bypass everything and bitstream it to a Marantz AV-7702 Mkii Pre-Pro that decodes the DTS-HD MA, TrueHD, Atmos, orDTS-X audio channels and feeds twelve channels of separate high-end amplification, speakers and sub-woofers. A JVC D-ILA Projector for 4K HDR handles the video without any transcoding ever.

12

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.