Recovering from failed Blobs DB

Skrie · October 3, 2024, 1:15am

Hello! I recently made a post about an issue I’m having and it was suggested I ask you here. It sounds similar to what rdarhart above is experiencing regarding PMS and blobs being out of sync even tho sqlite reads as ok.

I have my plex server running in a docker container on UnRaid and after i got back over the weekend the server had gone down (as many friends were quick to tell me) and on start-up was experiencing issues.

I went into my logs and found I was getting a number of DB errors, so i went through the process of using ChuckPa’s script for the repair tool guided by this helpful video. After it completed I re-scanned my libraries and manually refreshed Metadata for each as well, which did cause everything that had downloaded over the weekend to show up and be viewable. Here’s me thinking it’s sorted.

Afterwards i noticed though that i was unable to search my media for anything as no results were appearing. I can still navigate my libraries and select things to view. I have gone through all the search filtering settings and have each library set to be included in the home screen and global search. Later that night i also noticed that new files were once again not being found by plex and the console/log showed there were “database disk image is malformed” errors.

Oct 02, 2024 14:03:03.970 [22727615204152] ERROR - SQLITE3:0x80000001, 267, statement aborts at 184: [update metadata_items set updated_at=null,guid=?,hash=? where id=?] database disk image is malformed
Oct 02, 2024 14:03:03.970 [22727615204152] ERROR - Exception inside transaction (inside=1) (/home/runner/actions-runner/_work/plex-media-server/plex-media-server/Library/MetadataItem.cpp:3877): sqlite3_statement_backend::loadOne: database disk image is malformed
Oct 02, 2024 14:03:03.971 [22727615204152] ERROR - Exception assimilating media item in Buffy the Vampire Slayer/Season 07: sqlite3_statement_backend::loadOne: database disk image is malformed

Going through the repair tool steps again however gave me positive results with each check passing (Below)

2024-10-01 12.56.58 - ============================================================
2024-10-01 12.56.58 - Session start: Host is Docker
2024-10-01 12.57.13 - Stop    - PASS
2024-10-01 12.57.33 - Auto    - START
2024-10-01 12.57.58 - Check   - Check com.plexapp.plugins.library.db - FAIL (DBRepair.log com.plexapp.dlna.db com.plexapp.plugins.library.blobs.db com.plexapp.plugins.library.blobs.db-2023-09-14 com.plexapp.plugins.library.blobs.db-2023-09-17 com.plexapp.plugins.library.blobs.db-2023-09-20 com.plexapp.plugins.library.blobs.db-2023-09-23 com.plexapp.plugins.library.db com.plexapp.plugins.library.db-2023-07-22-tmp com.plexapp.plugins.library.db-2023-09-14 com.plexapp.plugins.library.db-2023-09-17 com.plexapp.plugins.library.db-2023-09-20 com.plexapp.plugins.library.db-2023-09-23 dbtmp in database main DBRepair.log com.plexapp.dlna.db com.plexapp.plugins.library.blobs.db com.plexapp.plugins.library.blobs.db-2023-09-14 com.plexapp.plugins.library.blobs.db-2023-09-17 com.plexapp.plugins.library.blobs.db-2023-09-20 com.plexapp.plugins.library.blobs.db-2023-09-23 com.plexapp.plugins.library.db com.plexapp.plugins.library.db-2023-07-22-tmp com.plexapp.plugins.library.db-2023-09-14 com.plexapp.plugins.library.db-2023-09-17 com.plexapp.plugins.library.db-2023-09-20 com.plexapp.plugins.library.db-2023-09-23 dbtmp On tree page 835358 cell 97: 2nd reference to page 852646)
2024-10-01 13.00.32 - Check   - Check com.plexapp.plugins.library.blobs.db - PASS
2024-10-01 13.00.32 - Check   - FAIL
2024-10-01 13.15.52 - Repair  - Export databases - PASS
2024-10-01 13.18.44 - Repair  - Cannot import main database from './dbtmp/library.plexapp.sql-2024-10-01_12.57.33' - FAIL (1)
2024-10-01 13.18.44 - Repair  - FAIL
2024-10-01 13.18.44 - Auto    - FAIL
2024-10-01 13.25.13 - Ignoring database errors.
2024-10-01 13.25.36 - Auto    - START
2024-10-01 13.26.17 - Check   - Check com.plexapp.plugins.library.db - FAIL (DBRepair.log com.plexapp.dlna.db com.plexapp.plugins.library.blobs.db com.plexapp.plugins.library.blobs.db-2023-09-14 com.plexapp.plugins.library.blobs.db-2023-09-17 com.plexapp.plugins.library.blobs.db-2023-09-20 com.plexapp.plugins.library.blobs.db-2023-09-23 com.plexapp.plugins.library.db com.plexapp.plugins.library.db-2023-07-22-tmp com.plexapp.plugins.library.db-2023-09-14 com.plexapp.plugins.library.db-2023-09-17 com.plexapp.plugins.library.db-2023-09-20 com.plexapp.plugins.library.db-2023-09-23 dbtmp in database main DBRepair.log com.plexapp.dlna.db com.plexapp.plugins.library.blobs.db com.plexapp.plugins.library.blobs.db-2023-09-14 com.plexapp.plugins.library.blobs.db-2023-09-17 com.plexapp.plugins.library.blobs.db-2023-09-20 com.plexapp.plugins.library.blobs.db-2023-09-23 com.plexapp.plugins.library.db com.plexapp.plugins.library.db-2023-07-22-tmp com.plexapp.plugins.library.db-2023-09-14 com.plexapp.plugins.library.db-2023-09-17 com.plexapp.plugins.library.db-2023-09-20 com.plexapp.plugins.library.db-2023-09-23 dbtmp On tree page 835358 cell 97: 2nd reference to page 852646)
2024-10-01 13.28.57 - Check   - Check com.plexapp.plugins.library.blobs.db - PASS
2024-10-01 13.28.57 - Check   - FAIL
2024-10-01 13.44.09 - Repair  - Export databases - PASS
2024-10-01 13.51.43 - Repair  - Import - PASS
2024-10-01 13.53.01 - Repair  - Verify main database - PASS (Size: 833MB/797MB).
2024-10-01 13.53.33 - Repair  - Verify blobs database - PASS (Size: 1945MB/1965MB).
2024-10-01 13.53.33 - Making repaired databases active
2024-10-01 13.53.33 - Repair  - Move files - PASS
2024-10-01 13.53.33 - Repair  - PASS
2024-10-01 13.53.33 - Repair  - PASS
2024-10-01 13.53.36 - Reindex - MakeBackup com.plexapp.plugins.library.db - PASS
2024-10-01 13.53.44 - Reindex - MakeBackup com.plexapp.plugins.library.blobs.db - PASS
2024-10-01 13.53.44 - Reindex - MakeBackup - PASS
2024-10-01 13.55.19 - Reindex - Reindex: com.plexapp.plugins.library.db - PASS
2024-10-01 13.56.08 - Reindex - Reindex: com.plexapp.plugins.library.blobs.db - PASS
2024-10-01 13.56.08 - Reindex - PASS
2024-10-01 13.56.08 - Reindex - PASS
2024-10-01 13.56.08 - Auto    - COMPLETED
2024-10-01 13.56.35 - Start   - PASS
2024-10-01 13.56.45 - Exit    - Delete temp files.
2024-10-01 13.56.49 - Session end. Tue Oct  1 13:56:49 PDT 2024
2024-10-01 13.56.49 - ============================================================
2024-10-01 14.15.37 - ============================================================
2024-10-01 14.15.37 - Session start: Host is Docker
2024-10-01 14.15.46 - Stop    - PASS
2024-10-01 14.16.02 - Auto    - START
2024-10-01 14.17.10 - Check   - Check com.plexapp.plugins.library.db - PASS
2024-10-01 14.17.49 - Check   - Check com.plexapp.plugins.library.blobs.db - PASS
2024-10-01 14.17.49 - Check   - PASS
2024-10-01 14.31.36 - Repair  - Export databases - PASS
2024-10-01 14.39.13 - Repair  - Import - PASS
2024-10-01 14.40.29 - Repair  - Verify main database - PASS (Size: 800MB/798MB).
2024-10-01 14.41.04 - Repair  - Verify blobs database - PASS (Size: 1965MB/1965MB).
2024-10-01 14.41.04 - Making repaired databases active
2024-10-01 14.41.04 - Repair  - Move files - PASS
2024-10-01 14.41.04 - Repair  - PASS
2024-10-01 14.41.04 - Repair  - PASS
2024-10-01 14.41.08 - Reindex - MakeBackup com.plexapp.plugins.library.db - PASS
2024-10-01 14.41.16 - Reindex - MakeBackup com.plexapp.plugins.library.blobs.db - PASS
2024-10-01 14.41.16 - Reindex - MakeBackup - PASS
2024-10-01 14.42.54 - Reindex - Reindex: com.plexapp.plugins.library.db - PASS
2024-10-01 14.43.43 - Reindex - Reindex: com.plexapp.plugins.library.blobs.db - PASS
2024-10-01 14.43.43 - Reindex - PASS
2024-10-01 14.43.43 - Reindex - PASS
2024-10-01 14.43.43 - Auto    - COMPLETED
2024-10-01 14.56.47 - Exit    - Delete temp files.
2024-10-01 14.56.48 - Session end. Tue Oct  1 14:56:48 PDT 2024
2024-10-01 14.56.48 - ============================================================
2024-10-02 05.41.07 - ============================================================
2024-10-02 05.41.07 - Session start: Host is Docker
2024-10-02 05.42.19 - Check   - FAIL - PMS runnning
2024-10-02 05.42.26 - Stop    - PASS
2024-10-02 05.43.43 - Check   - Check com.plexapp.plugins.library.db - PASS
2024-10-02 05.45.03 - Check   - Check com.plexapp.plugins.library.blobs.db - PASS
2024-10-02 05.45.03 - Check   - PASS
2024-10-02 05.46.13 - Ignoring database errors.
2024-10-02 05.46.18 - Honoring database errors.
2024-10-02 05.47.04 - Start   - PASS

On top of all that apparently I had an issues a year ago which is when my 3-day back-ups stopped as the last set is from 17+20+23 September 2023 (Which i did copy to another folder so they don’t get overridden this week) but a year is a LOT of changes to rollback to (and I am unsure if these ones are damaged either).

I’m sorry if this is the wrong place to ask about it OR if the formatting of this post doesn’t copy over well.

ChuckPa · October 3, 2024, 5:25am

@Skrie

This sounds like a case where you’re best to pick the MAIN DB you want.
From there, we’ll REPLACE the existing main with it.
We’ll delete the Blobs DB.

The Blobs DB will regenerate. It takes a few steps but is fairly quick (it’s drawing on info taken from the main.

Let me know your thoughts and I’ll give you instructions

Skrie · October 3, 2024, 4:04pm

When you say to pick the MAIN DB i’d want, I have my current one but the 3 back-ups are from a year ago and the Oct 2nd one i have was saved after the issue presented itself. Is using the existing MAIN not ideal? (Is that where it is malformed?) I am interested in clearing out the blobs DB and letting that regenerate if that won’t massively impact the server as a whole as far as history/settings etc is concerned, but at this point am accepting that there will be some loss.

ChuckPa · October 3, 2024, 4:31pm

@Skrie

If your existing main DB is the one you want
then this is what we’ll do.

Stop PMS
Delete all the blobs DB files
Delete any WAL or SHM DB files. (these are transactions which aren’t completed)
Start PMS
Scan all files (every library section)
Refresh all metadata (every library section)
Make sure your “Scheduled Tasks” is set to analyze media during maintenance AND has several hours (this will help blobs rebuild quicker)

Over the next few days, monitor the size of the blobs DB.
If you have a lot of music, it will grow quickly as PMS processes the music again.

Intro and Credits detection might also run.

All you need to do is be patient with it as it catches up again.

Skrie · October 3, 2024, 7:07pm

I backed up my existing DB folder and then went about following the steps above and when running the library scans im still getting errors and nothing new from the last few days has shown up either. When the scans finished i started the metadata refresh for my movie section first and was getting “database disk image is malformed” still. Hmmm.

Failed to commit transaction (/home/runner/actions-runner/_work/plex-media-server/plex-media-server/Library/MetadataItem.cpp:1088) (tries=1): Cannot commit transaction. database disk image is malformed

Will i need to wait for the scheduled tasks/maintenance to run tonight or is there something still wrong with the MainDB that requires attention. Otherwise i can continue on with the refresh.

Edit: This is what i’m seeing during a scan after refreshing the metadata for each library

ChuckPa · October 3, 2024, 9:36pm

I expect it to complain during this phase.

The main DB is looking for entries in the Blobs DB and it’s complaining loudly that it’s not present.

Go make a ls -la of the databases directory and confirm it’s creating a new blobs DB.

ALSO, confirm file permissions are correct (a common typo issue)

Skrie · October 3, 2024, 10:17pm

I believe this is what you meant? I am VERY inexperienced in this area. The New blobs DB is there though it appears to be missing a Write permission in the last trio whereas the MainDB has it.

drwxr-xr-x 1 plex users       500 Oct  3 14:56 .
drwxr-xr-x 1 plex users       100 Aug 26  2019 ..
-rw-r--r-- 1 root root       6426 Oct  2 05:47 DBRepair.log
-rw-r--r-- 1 plex users     69632 Aug 26  2019 com.plexapp.dlna.db
-rw-rw-r-- 1 plex users    364544 Oct  3 14:56 com.plexapp.plugins.library.blobs.db
-rw-rw-r-- 1 plex users     32768 Oct  3 14:56 com.plexapp.plugins.library.blobs.db-shm
-rw-rw-r-- 1 plex users         0 Oct  3 14:56 com.plexapp.plugins.library.blobs.db-wal
-rw-rw-rw- 1 plex users 852017152 Oct  3 14:56 com.plexapp.plugins.library.db
-rw-rw-rw- 1 plex users     32768 Oct  3 15:05 com.plexapp.plugins.library.db-shm
-rw-rw-rw- 1 plex users   1442032 Oct  3 15:05 com.plexapp.plugins.library.db-wal
drwxr-xr-x 1 root root          0 Oct  2 05:41 dbtmp

shark2k · October 3, 2024, 10:57pm

Chuck, I may be wrong about this (though a quick Google search seems to confirm), but can’t the malformed error seen in the one screenshot also because by not having enough disk space? Might it be a good idea to have Skrie check the available disk space where the DB is stored just in case? You are better at the Linux commands so I will leave you to providing that if you think it is necessary.

If not helpful, please ignore me.

-Shark2k

ChuckPa · October 3, 2024, 11:13pm

If there were insufficient space on the device, it would say so. “No space left on device”.

Notice, blobs.db is rebuilding. PMS still has records pointing to the old blobs DB records. This is where the errors are coming from (it’s trying to reach to record “XXX”, getting EOF, and it’s reporting “malformed”.

Skrie · October 3, 2024, 11:18pm

That actually aligns with what likely started the error, my downloader grabbed too many files and filled up the cache drive which is also where the docker containers run. Apparently i didn’t have limits configured properly and it “downloaded itself to death” but now I believe settings are correct as it hasn’t happened in a while, plus i’m soon to get a new 2tb ssd that is just a cache for new download intake.

ChuckPa · October 3, 2024, 11:18pm

@Skrie

you can do, while it’s running, sudo chmod 644 *.db*

Default permissions in the Databases directory are 755 for the directory and 644 for all files. The files are owned by user plex and plex is the one PMS runs as so they get 7xx and 6xx permissions – perfect.

FWIW:

Your “downloader” should NOT PUT ANYTHING in /var/lib/plexmediaserver.
This area is for PMS use ONLY.

ChuckPa · October 3, 2024, 11:23pm

@Skrie @shark2k

I moved us here to our own thread.

We’re now in the “recovery” stage

Skrie · October 3, 2024, 11:26pm

It doesn’t, i may have worded that poorly. It’s just that that directory is on a drive pool that is also the cache for new files before moved to the disk array every 8hrs. And if that is still an unwise decision (as that drive filing up can break the docker container(s)) then luckily i am soon planning to just slap another SSD in there to be the dedicated cache drive and then the docker containers and whatnot will be all that’s on the original cache drive pool.

Good call on moving the thread, didn’t want to be flooding the development topic with my troubleshooting issues.

ChuckPa · October 3, 2024, 11:45pm

The basic rules:

You can have your media anywhere you want – EXCEPT inside /usr/lib/plexmediaserver, /var/snap/plexmediaserver, or /var/lib/plexmediaserver
PMS’ internal data files (/var/lib/plexmediaserver, /var/snap/plexmediaserver, or /usr/lib/plexmediaserver) may NOT be somewhere on the network.
– PMS requires file locking to ensure the database doesn’t get corrupted.
(Multiple tasks read/write the DB at the same time. Locks prevent them from damaging each others’ data)
If you need more space than /var/lib provides, I usually suggest /home/plex and use an “override.conf” to tell PMS where its data files are

Skrie · October 5, 2024, 12:32am

Alright so a minor update of what’s happening is that the blobs DB appears to be rebuilding, we’re at about 19mb now. I have the maintenance window set to be 1am-9am for an 8 hour window, is that a decent enough time or should that be longer during this process of recovery?

One thing that is still (or rather not) happening is that new media isn’t being picked up by Plex. Radarr and Sonarr are still churning movies/episodes etc into their respective folders but they’re not being added to the server. Is this to be expected while the blobs rebuild is in progress or are there some additional actions i should be taking?

Also search turns up 0 results but i can select “More ways to watch” and then it shows as available under my server so there’s at least a workaround in the meantime.

ChuckPa · October 5, 2024, 7:31am

I would like to see the logs. Not picking up media isn’t usually related to anything with the DB’s unless the DB’s are calling out major corruption.

Make certain DEBUG logging is enabled.
Restart Plex
Keep PMS fully IDLE during this data gathering step
Wait 5 minutes, fully idle
Download the logs ZIP file
Upload it here.
Let everyone have access again

Skrie · October 5, 2024, 1:31pm

Plex Media Server Logs_2024-10-05_06-30-20.zip (4.2 MB)
Let me know if this didn’t capture anything interesting. A scan didn’t happen between the restart and now.

nibbles · October 6, 2024, 6:39pm

The logs show error sqlite database malformed. If it were my server I’d stop non-essential services while working on it. Good luck!

Skrie · October 8, 2024, 5:19pm

Should I be running the DBRepair too again to see if anything needs that? Or am i just in the rebuild/waiting phase before anything new will be added.

ChuckPa · October 9, 2024, 1:21am

I would stop it and run DBRepair one more time.

If you can’t get resolution, we’re going to need a rebuild and then pull in watch history after the rebuild from the old main DB

I rebuilt my server overnight , from scratch. This should not be taking so long

Topic		Replies	Views
Plex Media Server keeps crashing Plex Media Server server-linux	26	225	February 24, 2024
PMS db corrupted - running in BINHEX docker on Unraid Metadata & Adding Files server-unraid	64	376	September 16, 2023
Metadata & Search Issues Metadata & Adding Files server-unraid , library-management	18	126	July 2, 2025
All Libraries Suddenly Gone Plex Pass Betas server-linux	10	121	July 5, 2023
Long Time Plex User in Need of Help NAS & Devices server-linux , server-docker	14	121	September 18, 2025

Recovering from failed Blobs DB

Related topics