Database Limitations

I've been using Plex for quite some time now.  My Movie and TV show collections are getting sizable.  The only slow-down I've noticed is when starting Plex on my 2 Roku 3's.  It takes some time before the icons actually load - roughly a minute.  Otherwise, performance is just fine.

 

Its impressive what plex handles!  I have nearly 900 movies and over 5000 tv shows spread over 9TB of disk space on three External USB drives plugged into a Manhattan USB hub.  This is the coolest USB hub I've ever seen!!!  I have 9 external drives running through this hub.    

 

Are there any similar experiences out there?  How's performance for those with larger databases?  Does the plex team have an estimate on what kind of limitation there is on database/records size?

 

-Chris

You will probably get a more definitive answer to your exact question by Chris C or others who has such large library (I have about half and don't experience such issues) - but I was just curious about the rest of the hardware. Not too sure how it would affect but surely things like CPU, USB-standard (usb2 or usb3), network in general (wireless/routed Gbit) and other hardware also has an impact rather than just was the type of SQL that's running behind the scenes. Might also be good to know if you have experienced same stall with other apps to make sure that the problem isn't contained to Roku's.

Which version of PMS are you running?

Starting in 0.9.11 I think, they made the load times noticeably faster. 

My collection is as big if not bigger, but I have never noticed my roku take that long to load, even prior to 0.9.11.

Maybe post some logs?

It may be as simple as waiting for your drives to spin up when they have been idle a while.  Also, your actual Plex installation directory may be on your slowest drive.  Some time back, I put my Plex directory on an SSD to reduce lag when browsing.  I have around 2000 movies and 12000 episodes at this point.

Database size shouldn't be the issue here. Artwork for items is not part of the actual "database" itself (by which I mean the actual SQLite3 database used for your content). The earlier comments have some good suggestions about things you can check.

If you haven't done so recently, I'd also recommend performing a little [library maintenance](https://support.plex.tv/hc/en-us/articles/200392106-Library-Actions) by:

1. "Empty Trash" on the server

2. "Clean Bundles" on the server

3. Do an "Optimize" on the database

1615 movies and 5198 TV episodes and Plex is lightning fast.  All my media is on external USB 3 drives.  No more than 2 drive share the same bus (very important!!!).  My metadata (and database) is on a internal raid 5 array. 

This setup works perfectly.  While it was very fast before, it is even faster with the .11 upgrade. 

1615 movies and 5198 TV episodes and Plex is lightning fast.  All my media is on external USB 3 drives.  No more than 2 drive share the same bus (very important!!!).  My metadata (and database) is on a internal raid 5 array. 

This setup works perfectly.  While it was very fast before, it is even faster with the .11 upgrade. 

The main program is on an internal drive.  But if it has to spin up three external drives before sending/displaying the initial menu/icons, then that actually explains it.  I typically have about a 10 second delay when I start a video sitting on an idle external drive for that very reason.  I'll have to go back and look if I have the main database on a regular drive or on my SSD.  It's been way too long since I've dealt with it.  I know the main program sits on the SSD, but I'll bet you the database is on a regular internal drive.   But once the icons start loading, everything is smooth... 

Responses to other questions asked:  I'm running the latest version .9.9.10.458.....  I keep the database clean and optimized.  I'm running it with an 8 core amd with more ram than I will probably ever use.  The manhattan usb hub is connected via usb 3.  Only 2 of my 3 primary drives are plugged into a 3.0 port, but I don't have any issues when it comes to supporting 4 simultaneous streams - more typical is 2 or 3 in the evenings.  The server snores for the most part - although it can get a little busy (60-80%) if I have 3 streams transcoding. (usually not a good time to be ripping a DVD :)  But I must say, I've never seen any stuttering locally and it rarely hiccups streaming outside my network (and that isn't related to the my server when it happens).

 

Disk speed is probably the greatest hardware factor in the Plex database since it is just SQLLite



Sent from Tapatalk

Responses to other questions asked:  I'm running the latest version .9.9.10.458.....  

FYI - 0.9.9.12.504-3 is the latest version.  0.9.9.11 is the one with the speedup.  You are one release before this. 

FYI - 0.9.9.12.504-3 is the latest version.  0.9.9.11 is the one with the speedup.  You are one release before this. 

Oopsie.  I thought I had just gotten an update a few days ago....  LOL  I'll take care of that this week.  Thanks!

USB is most like the issue as probably said before. I would stay away from a USB setup if at all possible for any server type setup. Remove the drives from their enclosures and install them internally via SATA if you can.

USB is most like the issue as probably said before. I would stay away from a USB setup if at all possible for any server type setup. Remove the drives from their enclosures and install them internally via SATA if you can.

rotfl this is exactly what I do, I found the HDD's cheaper as externals than internals, while I may lose the warranty part, I use all seagates, and have not had any fail until well after the warranty expired anyways.

 2500+ movies, and 10,000+ TV shows and no lag except for spin up times occasionally which is no more than 15 seconds.

Can you guys please clarify something that doesn’t make sense for me ?

Some of you speak of having 5,000+ - 10,000+ TV Shows, which to me means a potential 100,000 - 200,000 episodes, assuming an average of at least 20 episodes per show. But I have a hunch you might really mean ‘Show Episodes’ rather than ‘Shows’. Personally I don’t see how it would be possible/practical to accumulate 10,000+ ‘Shows’, though I know that I have several times that amount of ‘Show Episodes’.

As for myself, I currently have 927 Shows in my main “TV Shows” section, with no proper count of the total number of episodes.
(They’re distributed over a dozen different HDDs mounted in various computers in my LAN.)

Best regards: dlanor

Can you guys please clarify something that doesn't make sense for me ?

Some of you speak of having 5,000+ - 10,000+ TV Shows, which to me means a potential 100,000 - 200,000 episodes, assuming an average of at least 20 episodes per show. But I have a hunch you might really mean 'Show Episodes' rather than 'Shows'. Personally I don't see how it would be possible/practical to accumulate 10,000+ 'Shows', though I know that I have several times that amount of 'Show Episodes'.

As for myself, I currently have 927 Shows in my main "TV Shows" section, with no proper count of the total number of episodes.
(They're distributed over a dozen different HDDs mounted in various computers in my LAN.)

Best regards: dlanor

I use "shows" as synonymous with "episodes".  I'm sure that most do. 

I wanted to update everyone on where I'm at with the load-time issue of metadata to my roku's.  I had decided to attempt moving my plex data-files from one of the internal drives to my C: drive (an SSD).  Before doing this I had to turn off and then delete all the media index files.  That was a very time-consuming process. 

Here's how I'll describe the action at my roku's:    BAM!!!!  Icons are up.   BAM!!!!  Metedata is there.

END OF STORY.  After seeing the difference, I must say that I wasn't really surprised at the new responsiveness - more by the sluggishness of the internal drive I had the data on previously.

800+ movies and 293 shows with 30220 total episodes no real issues, icons take a couple seconds to load as I scroll along but my server is off-site so it's within acceptable limits I think

USB is most like the issue as probably said before. I would stay away from a USB setup if at all possible for any server type setup. Remove the drives from their enclosures and install them internally via SATA if you can.

The problem I was having was actually the result of a slower internal drive.  Moving the plex database(s) to the SSD fixed it.  The only time the external drive response times are an issue is when that initial "spin-up" happens. But you know what it is, so you wait an extra few seconds prior to the video beginning. My movies/shows/music sit across 8 external drives.  (6 under plex - 3 live/3 backup).  I'd spend the money for a better performing - and "proper" server farm if it was called for.  For me, I'm unable to talk myself into something "better" at this point since I'm getting what I want at a great price and completely satisfactory performance.

I'd argue that you can't beat a "cheap guys server farm":  One Manhattan USB box with all the cheap external drives you need.   The system easily supports the two tv's on the local network plus the three TV's at the remote home.  I've observed 4 streams running at once and haven't had any issues.  And I've been running this puppy for a little while now.

 2500+ movies, and 10,000+ TV shows and no lag except for spin up times occasionally which is no more than 15 seconds.

I just did the math on that, assuming that your movies are all HD.  How much storage do you have for everything - and where/what are you backing up to?  What kind of box do you have that supports the storage internally?  

I am looking for database limitations with regards to large collections. I currently have approx 5200 movies and 22000 episodes. I have 5 raspberry pi’s and 1 openelec with plex addon. Because I have children I limit which folders each device can view based on the logins I have created for each device. I have lag on every one of my devices when opening any folder. All devices are on the same physical network, full gigabyte switching on Cat6 cabling throughout the house to limit wireless latency, and I am still seeing database lag.

Server : Dell PowerEdge R310, 16gig ram, dual 4 core Xeon Processors, Mint Linux
Storage: 30TB, 8 core, 16 gig ram, FreeNAS operating system

Players: 5 Raspberry Pi 3’s running Rasplex, 1 Lenovo M710 Tiny running Openelec with Plex Addon

Is there something I can do to decrease the lag?

Put the Plex data folder onto an SSD drive.

Second thing to consider:
Observing the recommendation to put every movie into its own subfolder,
Particularly if there are local assets in use.

Or disable Local Media Assets, which then disables external subtitles, local posters, local extras etc.