Server Version#: 1.18.3.2156
Player Version#: 1.4.1.940-574c2fa7
I have a somewhat large collection, about 25TB with about 4000 movies, 30,000 TV episodes and 125,000 MP3s. I back up the database folders nightly to a huge ZIP file (that is about 50GB at this point) but I’m wondering what I can do to be more proactive to either shrink that size or ensure things are in a healthy state in the event I was to lose a drive (OS drive is mirrored and all media is backed up on a second 30TB array). So while I can reimport/restore everything I really don’t look forward to how long that would take.
Given I’ve been building this collection for over a decade I’d be rather upset if I lost it or had to re-import/organize everything. I see there are some scheduled tasks setup but am curious if with a collection this large if I should do more.
I’ve found that the best preventative measure is to organize your files so that they require the least amount of manual intervention when ingesting them into Plex.
Which means:
follow the required naming schemes closely
always add the (year) in file and folder names. For TV shows as well!
store custom posters beside the media files and don’t use the ‘upload’ mechanism in the web app
if you have “unmatchable” media, use mp4 files and embed as much metadata as possible directly into the files
keep an eye on the health of your primary database file. (The “notification” feature on mobile clients makes this easy)
don’t disable all of the scheduled tasks, which are designed to keep your data folder clean
Uh, thanks - maybe I wasn’t clear that I’ve been using this for almost a decade and with a collection as large as mine it is highly, highly organized already (in terms of the media). So I’m more looking for something that would be unique to a very large library. So thank for replying but that’s not really what I’m looking for as that’s all pretty obvious.
I use THIS Powershell script to handle my library backups. It can use 7zip which speeds things up considerably compared to built-in Windows Zip. It also keeps the sizes relatively quite small.
My library folder is approx 125GB (not sure exactly how much of that is the scrobbling thumbnails), and my Plex backups using the method above are approx 16GB total. It takes about an hour for it to process. Note that I’m using the default setting which is to exclude those thumbnails from the backups. The size would be much higher if they’re included. If I ever restore, Plex can just regenerate those at night for a couple months.
I’ve restored using the same script twice now. (1 SSD death, 1 moment of stupidity) and it works flawlessly. I’ve expanded it a bit personally to also handle all of my “Plex adjacent” items like Tautulli, content acquisition programs, and my eBook manager, too. I can provide details if you’d like
Sounds like you’re already doing more than you have to.
Plex makes a backup of the database every three days and if your Data is being held on a NON-OS drive - safe from ground zero should your OS drive be hit by an asteroid - you’re pretty safe without all the zip backups - until that drive blows up the same time the OS drive explodes.
Frankly I’d feel pretty good about your ‘method’ - it’s enough over-the-top to rank right up there and be featured in Paranoid Monthly.
I’m old enough (cough/wheeze) to know everything on my computer is expendable - 'cause there’s no way I’d go to all the trouble you do to make sure I wouldn’t have to rebuild my Plexiverse after a catastrophe (4 time loser here - 2 without my data on a NON-OS drive and 2 with - I like the latter much better).
all you really need to backup is the main database file, and perhaps the preferences.xml (which contains the unique server id to your server accoutn).
You don’t really need to backup any of the metadata or other folders, since plex can simply re-download all that if you ever have to start from scratch with only a database backup.
as far as preventive maintenance, really has already mentioned. Make sure the scheduled tasks stuff is enabled in the pms server settings.
for performance improvements, the best thing you can do is get a large SSD (ie 1tb+) and put the plex data folder on it. assuming it is not already on SSD.
this will drastically improve your library response time, since the sqlite database is ultimately limited by disk IO.
art load times will increase also, since the metadata folder contains way too much tiny folders/files that can be slower to access on spinning disk.
do note, you can change the location where plex automatically backs up the database.
I would suggest ensuring that location is on a separate disk or array than PMS and your media.
with various cloud products, you can also add that backup path to a cloud sync, so that the plex backups are automatically sync’d to a cloud account.
as your database grows there really isn’t much more you can do other than optimize the database more frequently.
I’m doing it twice a day now using crontab
#!/bin/sh
# Get the contents of the Preferences file, keep only what we need, push to a temp, then use it in the curl command
cat "/opt/plex/Library/Application Support/Plex Media Server/Preferences.xml" | \
sed -e 's;^.* PlexOnlineToken=";;' | sed -e 's;".*$;;' | tail -1 > /tmp/plex_token.tmp
curl --request PUT http://127.0.0.1:32400/library/optimize\?async=1\&X-Plex-Token=`cat /tmp/plex_token.tmp`
rm -f /tmp/plex_token.tmp
It is my belief, that the actual optimization does less to improve performance, than the result, which is to write a new database file that is hopefully less fragmented on the filesystem.
except perhaps on systems with older/low power cpus, disk access will always be the limitation on database performance.
io latency (how fast a disk can respond to any particular read or write) is the critical issue for a file based database.
when a sqlite database optimization occurs, it essentially exports the database to a temp file, then writes a new database file.
at the file system level (be linux/ext4/btrfs/zfs/etc or windows/ntfs/etc) the OS/filesystem will normally attempt to write the new file to the least fragmented blocks.
with an SSD, optimization/fragmentation becomes less relevant, although there can still be a performance penalty on SSD with a highly fragmented db file.
My suggestion is based on actual usage -
the daily optimization (or export/import) helps, in my use case.
I see far fewer WARN - Waited XXXX for a busy database, as well less PMS becoming unavailable when I optimize more often
