My plex server is constantly crashing and won't stay up

My server which is around 5 years old is consistently crashing suddenly, I’ve tried restarting but still can’t get the server to stay up for a long period of time (It crashes after about 10-15minutes)

I’m unsure what’s causing it, as there seems to be no immediate issues that I can see. However, in my docker container I can see the following errors almost all the time;

decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
decoder information: 249
Sqlite3: Sleeping for 200ms to retry busy DB.
Sqlite3: Sleeping for 200ms to retry busy DB.
Sqlite3: Sleeping for 200ms to retry busy DB.
Sqlite3: Sleeping for 200ms to retry busy DB.
Sqlite3: Sleeping for 200ms to retry busy DB.
Sqlite3: Sleeping for 200ms to retry busy DB.
Sqlite3: Sleeping for 200ms to retry busy DB.
Sqlite3: Sleeping for 200ms to retry busy DB.
Sqlite3: Sleeping for 200ms to retry busy DB.
Sqlite3: Sleeping for 200ms to retry busy DB.
Sqlite3: Sleeping for 200ms to retry busy DB.

I’ve tried doing a repair a corrupt database using the steps on the Plex website, with no improvement, the checks came back fine.

What would I need to provide, for someone to help me nail down this issue?

A set of debug level server logs capturing the crash.

After restarting Plex wait ~2 minutes before pulling the log files. This lets Plex fully start and log the startup sequence.

Retrieve server logs via Settings → Troubleshooting, and upload the ZIP file to the thread.

If you cannot access the logs via Plex Web, then manually navigate to the Plex Data Folder, tar/gzip/etc the Logs folder and attach to the thread.

Logs.zip (4.0 MB)

Oct 01, 2022 22:00:57.927 [0x7f255d55cb38] DEBUG - Performing a scan with 'Plex TV Series' (language: en-US virtual: 0).
Oct 01, 2022 22:00:57.927 [0x7f255d55cb38] DEBUG -   * Scanning /var/lib/plexmediaserver/Library/secret/Anime
Oct 01, 2022 22:00:57.928 [0x7f255d55cb38] DEBUG - Scanner: Processing directory /var/lib/plexmediaserver/Library/secret/Anime (parent: no)

Thanks for the log files.

Move all media out of the Plex Data Folder. It is reserved for use by Plex Media Server. Placing media there will cause Plex Media Server instability.

I realize that the Plex Data Folder is listed as /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/, not /var/lib/plexmediaserver/Library/, but it is best to avoid placing any media anywhere in /var/lib/plexmediaserver.

After you move the media and re-point the libraries:

  1. Re-scan the libraries
  2. Empty Trash
  3. Clean Bundles ← Also in Settings → Troubleshooting
  4. Optimize the database ← Also in Settings → Troubleshooting

Then restart Plex Media Server.

dance

I did a database check and I’m now getting this issue?

Recommend either:

  1. Dump the database, using Plex SQLite, to a .sql file and reimporting into fresh

-or-

  1. Remove the SHM and WAL file then COPY the most recent backup to take its place.

Unfortunately the 2nd option isn’t an option as all my backups are also corrupt. How would I perform option one?

Option 1:

As root in the container… but Plex MUST Be stopped:


# get into databases directory
cd "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Databases"

# Export `Library.db`
"/usr/lib/plexmediaserver/Plex SQLite"  com.plexapp.plugins.library.db .dump  >library.sql

# Remove the WAL and SHM if they exist
rm -f com.plexapp.plugins.library.db-wal
rm -f com.plexapp.plugins.library.db-shm

# Ensure all good records return to DB
sed -i -e 's/ROLLBACK;/COMMIT;/' library.sql

# Input good records
"/usr/lib/plexmediaserver/Plex SQLite"  com.plexapp.plugins.library.db < library.sql

# change permissions  (to match the UID:GID of the container
chown UID:GID com.plexapp.plugins.library.db

image

Getting all these error on the “input good records” part.

Wow. That’s a trashed database. The ID numbers are all compromised.

Under normal circumstances, each item gets a unique ID.

Somehow your database trashed all those tables.

Unfortunately all you can do is start over.

Is your database stored on a local device or over the network?

Network mounts do not provide absolute file locks.
Your DB got trashed because as the PMS tasks went to use it, they had no knowledge other tasks were writing to it. The result is a garbled mess.

The ONLY way you can put a database over the network is on an iSCSI LUN.
This works because you get a block device which the local kernel has complete control over. You format the block device. This gives you the file system for the DB.
The kernel file locks are obeyed.

I’m having an issue again with Plex consistently crashing multiple times a day and I have to manually restart the docker container. Here are the logs for the latest crashes, could you shed any light on this?

Its really frustrating now.

Plex Media Server Logs_2022-10-10_00-58-46.zip (3.4 MB)

I think the DB is damaged.

Not much but enough to make it crash. I’ve seen this before but can’t remember the details.

Oct 10, 2022 00:56:52.543 [0x7f8f07069b38] DEBUG - Auth: authenticated user 30049228 as Beth
Oct 10, 2022 00:56:52.630 [0x7f8f07069b38] DEBUG - Auth: authenticated user 30049228 as Beth
Oct 10, 2022 00:56:52.661 [0x7f8f0726cb38] DEBUG - Auth: authenticated user 30049228 as Beth
Oct 10, 2022 00:56:52.725 [0x7f8f07069b38] DEBUG - Auth: authenticated user 30049228 as Beth
Oct 10, 2022 00:56:52.879 [0x7f8f0726cb38] DEBUG - Auth: authenticated user 6767799 as war_khan
Oct 10, 2022 00:56:55.850 [0x7f8f0726cb38] DEBUG - Auth: authenticated user 6767799 as war_khan
Oct 10, 2022 00:56:55.965 [0x7f8f07069b38] DEBUG - Auth: authenticated user 6767799 as war_khan
Oct 10, 2022 00:56:56.000 [0x7f8f0726cb38] DEBUG - Auth: authenticated user 6767799 as war_khan
Oct 10, 2022 00:56:56.376 [0x7f8f0726cb38] DEBUG - Auth: authenticated user 6767799 as war_khan
Oct 10, 2022 00:56:58.744 [0x7f8f0726cb38] DEBUG - Auth: authenticated user 6767799 as war_khan
Oct 10, 2022 00:56:59.286 [0x7f8f0726cb38] DEBUG - Auth: authenticated user 6767799 as war_khan
Oct 10, 2022 00:56:59.327 [0x7f8f0726cb38] DEBUG - Auth: authenticated user 6767799 as war_khan
Oct 10, 2022 00:56:59.355 [0x7f8f0726cb38] DEBUG - Auth: authenticated user 6767799 as war_khan
Oct 10, 2022 00:56:59.602 [0x7f8f0726cb38] DEBUG - Auth: authenticated user 6767799 as war_khan
Oct 10, 2022 00:57:02.674 [0x7f8f07069b38] DEBUG - Auth: authenticated user 6767799 as war_khan

How good are you at using Plex SQLite to export the database to SQL then reimporting back into a clean/fresh .db ?

I have a tool I’ve written which should resolve this. It’s worked to solve other similar problems.

I’m pretty sure/confident I’d be able to perform the task, where do I obtain the tool?

Thanks.

Is this the same as what your tool achieves? I’ve done this numerous times but my database still ends up corrupt again, how is this happening?

@JL94x4

How do you use Docker on Windows?
If Docker & Linux, where’s the container? Is it local to the container or over a network mount?

Linux’s file locking isn’t compatible with windows.
Network SQLite databases in PMS won’t work either – no locking.

I’ll give you the tool but I am wondering how you keep the DB from corrupting (which WILL happen quickly when the file locking isn’t working)

Our database for Plex is local to the computer that PMS is running on, its just running via a docker container. The only thing that’s mounted to the computer that isn’t local is the rclone mount for my media.

I’m not sure what you mean by file locking, I’ve used the Windows version of Plex to fix my corrupt database as I’ve shown in the above screenshot before reuploading to my Linux server and starting the docker container.

Also, is there something in the logs that suggests file locking isn’t working on my setup?

There’s nothing in your logs which shows locking isn’t working.

The repetitive nature of this corruption tells us the DB is being compromised by something other than/outside PMS.

I don’t know if you have other plug-ins or programs which interact with PMS but that doesn’t matter.

Seeing a UNIQUE CONSTRAINT fail tells me there are records being inserted into the DB which do not belong there (whether it be the existing or those being inserted)

I don’t like the .recover directive followed by .read. That’s dangerous because it’s simply copying out the good records and, as your console session shows, immediately reading it back into the same DB. There is going to be duplication.

The better method:

  1. Export the entire database to SQL source.
  2. Delete the DB, WAL and SHM
  3. Create a new DB by reading from the SQL file

Now we know there is no duplication.
We also know we’ve captured all the records which we can safely get.
Scan Files & Refresh Metadata will fill in the rest if needed.

My tool handles all the database manipulation.
It also verifies the DB integrity at each step along the way.

You end up with a usable, fully intact, DB which might be missing those previously damaged records.

I’m going to speak to Docker in the context of “Docker & HW tone mapping” context.

  1. PMS 1.29.1 gave us built-in hardware tone mapping on ALL Intel Linux platforms.
  2. Docker is no longer needed if that’s all you wanted it for.

PERSONALLY, :slight_smile:

Some folks think Docker is easier. I disagree because I see more problems like you have here with Docker than I do with native platform installations.

I like simplicity. The more complex it is, the easier it is to break.
That’s why I firmly believe: PMS, native on the host without abstraction by a container is the best method.

People will yell at me (and I’ve been yelled at a lot) because they abstract everything and I generally disagree with that philosophy.

If that works best in the grand scheme for them then ok but it’s not the only solution and definitely not for everyone.

From the PMS level, working on the DB, inside a container – is a PITA.

  1. You can’t stop PMS because it automatically restarts
  2. You can’t fix the DB with PMS running.
  3. If PMS were stopped and you still had a command line, you could let my tool do the work.
  4. Catch 22.

We can fix your DB but, after all this, we don’t know the root cause of the corruption.
Without knowing the cause, we can’t prevent it from happening again.
It will occur again.

You can copy the steps out of it and adapt them to your environment.

My whole point is and has been – is Docker still necessary for PMS?

Please check your PM, I’ve sent you the tool.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.