I’ve noted that no matter how much memory or processor I give Plex, it still slows down considerably with a large media collection. I’d like to see a concentration on using the resources I’m giving it. I have a large library of over 4000 movies, 20000 TV episodes and 40K+ musical tracks. It seems to me that SQLite is the main culprit, as it’s not made for large datasets.
For example, I can give the box 20 gigs of memory (my servers have a TB) and it hardly touches it. Therefore, there’s lots of opportunity to improve performance by actually using more of the resources. I’ve also noted that giving more cache to the database, for example, 4GB, actually slows operations. I have it at 512MB now and get better performance than when I set it higher.
Therefore, I’d like to see more concentration on multiple threads and possibly the option to run a read database, such as MySQL or Postgres. It would really help for those of us who have large personal collections.
Servers Like Emby and Jellyfin will use the resources I give them. Plex, however, seems to concentrate on not using resources. I understand this decision as a default, yet it would be nice to have features to accommodate advanced users and larger libraries.
Drop your music and I bet performance increases dramatically. Plex has never be great with large music libraries ime. I may have missed it, but what type of media is the plex app library on? SSD? HD? NAS?
It’s on an all-enterprise flash NVME array that pushes 3–4000 MBPS and over 300K IOPS on ZFS. The box itself is a 96-core system with a terabyte of RAM. It’s virtualized with KVM and is assigned 16 CPU’s and 16 gigabytes of RAM, of which almost nothing is used. I also have a 2 GTX 3090’s in there, of which it does use one and plays fine. The interface is what lags, which is why I think there’s room for improvement. I really love Plexamp, so I’m going to stay with the music and hope that there’s some improvement for those who have power to spare. It seems they’ve concentrated on optimizing for low-end NAS boxes, which is great for those users, I am not one of them.
Music is probably killing you. Also, I would not assign so many vCPUs to the box, if your VM is not busy, you are wasting more resources managing vcpus that you are gaining in functionality. The reason plex concentrates on “low-end” hardware is because for every one of a power user like yourself, there are thousands of folks with NAS devices. And most users who are using in a single-family home and not sharing out to large numbers of people have no need for the hardware you are using. It just isn’t their target audience.
I don’t think that SQLite is the problem. A database of 7 GB size (both files) is running fast as hell in a Docker Container (Linuxserver) on a Xeon class CPU (W1290P) with 128 GB RAM. OS is Unraid. Storage is a Samsung 970 EVO Plus NVMe single device pool (XFS).
IMHO there must be something wrong with your system or its configuration.
Sorry, I didn’t see that. It’s definitely a database contention issue, I get errors all the time about slow database reads, yet the disk is screaming fast. I can copy about a second, so it’s not IO contention that I can see. What I’m really confused about is this. It’s doing quite a bit right now (maintenance period), and this is what I see.
It just won’t push my gear, and I don’t understand why. It should be pounding the CPU, yet it seems to just kind of laze along.
As a test, I’m going to try Jellyfin with a Maria backend and see how that works. Maybe it’ll give me some insight; however, most of the DB should be in the ARC, as I have 500GB active at the moment, so perhaps it just doesn’t show up here. However, I’d expect to see some load and no slowdown.
I also do my transcoding in /dev/shm, so it goes directly to RAM. That seems fast enough; playback seems fine, yet the interface is too slow. I’ll also check my proxy; perhaps that’s part of the issue. I’m not convinced, though, as when I’m on my VPN, it’s just as slow and direct to the machine. I’m on a 10 gig enterprise-grade fiber connection all by myself, so it’s also not much bandwidth. Any other ideas would be appreciated.
It was right at the end of a maintenance cycle, yet it still never got close to capping out the assigned CPU’s. I imported a lot of music as of late, so this may have something to do with it. It’s a bummer, as I really like Plexamp, yet if music is what’s dragging me down, I suppose I can use an alternative system. I’ve got 16000 tracks, not a ton compared with other people I’ve seen discussing their collections, yet it is doing sonic analysis on 3000 tracks that were recently added, and this doesn’t seem to offload in any manner.
Do you have 40k+ or 16k? I thought you said earlier 40k+. Either way, in the past large music libraries were a problem for Plex. I am an old member, so this might be a thing of the past. But it was often reported that large music libraries were a performance issue for plex.
My music library is much larger than OP’s, my i5-10400 Plex server is perhaps a teeny tiny bit less powerful than the system described, and I have never noticed poor GUI performance. I’m not sure it is a fair generalization that large music libraries cause problems.
It would be easy to test though, OP just needs to spin up another Plex instance and add everything but the music.
I’m afraid we may need both @ChuckPa. I think OPs request for external DB support could still be a valid feature. The rest of our posts trying to better understand why OP is having so many performance issues is a separate but related thread, imo.
I have moved your post here. At present, it makes more sense to diagnose what’s happening.
From what you describe and based on the info already presented,
The number of items in the DB
The ZFS-based host
The very first thing I will suggest is externally optimizing the DB and tuning the DB for the ZFS filesystem.
OF IMPORTANCE HERE:
Being a ZFS filesystem, DBRepair will set the SQLite record_size when optimizing.
This takes better advantage of the underlying ZFS dataset record size.
SQLITE max record size is 65536
The ReleaseNotes above show how to use this capability.
BEFORE attempting to set SQLite record size, I first recommend a normal optimization (“auto”)
– QNAP QuTS systems (also ZFS) work quite well using the default SQLite record size. They have demonstrated needing the larger record size (65536) ONLY in extreme cases far above the quantity shown here.
My tool is here:
It is external to PMS.
It does run INSIDE docker containers (docker exec -it container_name bash)
It has automatic configuration detection of almost all environments
( I have a beta version which allows fully manual configuration to circumvent the problems associated with Kubernetes and other containers which do not support health-check suspension )
PLEASE take the time to read the documentation
ALSO, The tool does not “hang”. If it’s slow then let it take the time.
Lastly, There is a weird / known issue from Plex SQLite in PMS 1.40.2. If invoked too quickly after shutting down PMS, it will “kill” the database check process. It’s a bug which I’m working with Engineering to resolve.
[chuck@lizum Databases.2012]$ sqlite3 com.plexapp.plugins.library.db
SQLite version 3.37.2 2022-01-06 13:25:41
Enter ".help" for usage hints.
sqlite> select count(*) from metadata_items;
536877
sqlite> select count(*) from media_items;
630819
sqlite>
[chuck@lizum Databases.2013]$ ls -la
total 19114164
drwxr-xr-x 2 chuck chuck 4096 May 4 17:17 ./
drwxr-xr-x 7 chuck chuck 96 Nov 7 2022 ../
-rw-r--r-- 1 chuck chuck 83711 Sep 19 2023 blobs.sql
-rw-r--r-- 1 chuck chuck 1077248 May 4 17:17 com.plexapp.plugins.library.blobs.db
-rw-r--r-- 1 chuck chuck 1060864 Oct 17 2023 com.plexapp.plugins.library.blobs.db-2023-10-17
-rw-r--r-- 1 chuck chuck 1064960 Feb 2 02:02 com.plexapp.plugins.library.blobs.db-2024-02-02
-rw-r--r-- 1 chuck chuck 1064960 Feb 5 02:02 com.plexapp.plugins.library.blobs.db-2024-02-05
-rw-r--r-- 1 chuck chuck 1064960 Feb 8 02:02 com.plexapp.plugins.library.blobs.db-2024-02-08
-rw-r--r-- 1 chuck chuck 32768 May 4 17:17 com.plexapp.plugins.library.blobs.db-shm
-rw-r--r-- 1 chuck chuck 0 May 4 17:17 com.plexapp.plugins.library.blobs.db-wal
-rw-r--r-- 1 chuck chuck 1948942336 May 4 17:17 com.plexapp.plugins.library.db
-rw-r--r-- 1 chuck chuck 1747972096 Oct 17 2023 com.plexapp.plugins.library.db-2023-10-17
-rw-r--r-- 1 chuck chuck 1951059968 Feb 2 02:02 com.plexapp.plugins.library.db-2024-02-02
-rw-r--r-- 1 chuck chuck 1991020544 Feb 5 02:02 com.plexapp.plugins.library.db-2024-02-05
-rw-r--r-- 1 chuck chuck 2033364992 Feb 8 02:02 com.plexapp.plugins.library.db-2024-02-08
-rw-r--r-- 1 chuck chuck 32768 May 4 17:18 com.plexapp.plugins.library.db-shm
-rw-r--r-- 1 chuck chuck 309032 May 4 17:18 com.plexapp.plugins.library.db-wal
[chuck@lizum Databases.2014]$
Runtime to fully optimize (rebuild and reindex) the databases
[chuck@lizum PlexDBRepair.2004]$ sudo ./DBRepair.sh stop auto start
Plex Media Server Database Repair Utility (Ubuntu 22.04.4 LTS)
Version v1.06.00
[2024-05-04 17.12.37] Stopping PMS.
[2024-05-04 17.12.39] Stopped PMS.
[2024-05-04 17.12.39] Automatic Check,Repair,Index started.
[2024-05-04 17.12.39]
[2024-05-04 17.12.39] Checking the PMS databases
[2024-05-04 17.14.14] Check complete. PMS main database is OK.
[2024-05-04 17.14.14] Check complete. PMS blobs database is OK.
[2024-05-04 17.14.14]
[2024-05-04 17.14.14] Exporting current databases using timestamp: 2024-05-04_17.12.39
[2024-05-04 17.14.14] Exporting Main DB
[2024-05-04 17.14.48] Exporting Blobs DB
[2024-05-04 17.14.52] Successfully exported the main and blobs databases. Proceeding to import into new databases.
[2024-05-04 17.14.52] Importing Main DB.
[2024-05-04 17.16.34] Importing Blobs DB.
[2024-05-04 17.16.34] Successfully imported databases.
[2024-05-04 17.16.34] Verifying databases integrity after importing.
[2024-05-04 17.17.09] Verification complete. PMS main database is OK.
[2024-05-04 17.17.09] Verification complete. PMS blobs database is OK.
[2024-05-04 17.17.09] Saving current databases with '-BACKUP-2024-05-04_17.12.39'
[2024-05-04 17.17.09] Making repaired databases active
[2024-05-04 17.17.09] Repair complete. Please check your library settings and contents for completeness.
[2024-05-04 17.17.09] Recommend: Scan Files and Refresh all metadata for each library section.
[2024-05-04 17.17.09]
[2024-05-04 17.17.09] Backing up of databases
[2024-05-04 17.17.09] Backup current databases with '-BACKUP-2024-05-04_17.17.09' timestamp.
[2024-05-04 17.17.10] Reindexing main database
[2024-05-04 17.17.48] Reindexing main database successful.
[2024-05-04 17.17.48] Reindexing blobs database
[2024-05-04 17.17.48] Reindexing blobs database successful.
[2024-05-04 17.17.48] Reindex complete.
[2024-05-04 17.17.48] Automatic Check, Repair/optimize, & Index successful.
[2024-05-04 17.17.48] Starting PMS.
[2024-05-04 17.17.48] Started PMS
[chuck@lizum PlexDBRepair.2005]$
I tried this yesterday, yet in the official docker beta subscriber image it told me it wasn’t a valid installation environment. I put a comment on the git.
