Home screen is blank AGAIN -- Getting sick of this

[tomt92]

Server Version#: 1.18.1.1973
Web Version#: 4.8.3
OS: Ubuntu 16.04.6 LTS - (Not upgraded yet because of ZFS)

Plex Media Server.log (14.3 KB)

I have no idea why but Plex’s Database will randomly become corrupt and items on the home screen will all disappear, but they are still available on their individual library pages. I have the DB backing up every 3 days but the instructions here (https://support.plex.tv/articles/202485658-restore-a-database-backed-up-via-scheduled-tasks/) don’t correspond to my server instance. The only db-shm and db-wal are com.plexapp.dlna. inthe backup folder the only files that are recent are com.plexapp.plugins.library.blobs and the last com.plexapp.plugins.library was 5 months ago… Im at my wits end, my system has over 34,000 tv episodes, 4,000 movies and who knows how much music. I really don’t want to wipe and reinitialize the entire DB since it losses all my watch states…

[ChuckPa]

can you please show me a directory listing (ls -la) of what is in your Databases directory?

[tomt92]

Sure…

total 4450288
drwxrwxr-x 2 plex   plex       45056 Nov  7 10:25 .
drwxrwxr-x 7 thomas thomas      4096 Feb 13  2019 ..
-rwxrwxr-x 1 plex   plex     9417728 Jul 29  2018 com.plexapp.dlna.db
-rwxrwxr-x 1 plex   plex           0 Jan 10  2014 com.plexapp.dlna.db-journal
-rwxrwxr-x 1 plex   plex       98304 Jul 29  2018 com.plexapp.dlna.db-shm
-rwxrwxr-x 1 plex   plex     9638488 Jul 29  2018 com.plexapp.dlna.db-wal
-rwxrwxr-x 1 plex   plex    23546880 Nov  7 02:05 com.plexapp.plugins.library.blobs.db
-rwxrwxr-x 1 plex   plex    25204736 Apr  2  2019 com.plexapp.plugins.library.blobs.db-2019-04-02-tmp
-rwxrwxr-x 1 plex   plex    25204736 Apr  5  2019 com.plexapp.plugins.library.blobs.db-2019-04-05-tmp
-rwxrwxr-x 1 plex   plex    25207808 Apr  8  2019 com.plexapp.plugins.library.blobs.db-2019-04-08-tmp
-rwxrwxr-x 1 plex   plex    25210880 Apr 11  2019 com.plexapp.plugins.library.blobs.db-2019-04-11-tmp
-rwxrwxr-x 1 plex   plex    25210880 Apr 14  2019 com.plexapp.plugins.library.blobs.db-2019-04-14-tmp
-rwxrwxr-x 1 plex   plex    25198592 Apr 17  2019 com.plexapp.plugins.library.blobs.db-2019-04-17-tmp
-rwxrwxr-x 1 plex   plex    25198592 Apr 20  2019 com.plexapp.plugins.library.blobs.db-2019-04-20-tmp
-rw-r--r-- 1 plex   plex    25314304 Sep  8 02:00 com.plexapp.plugins.library.blobs.db-2019-09-08-tmp
-rwxrwxr-x 1 plex   plex       32768 Nov  7 10:25 com.plexapp.plugins.library.blobs.db-shm
-rwxrwxr-x 1 plex   plex           0 Nov  7 10:25 com.plexapp.plugins.library.blobs.db-wal
-rwxrwxr-x 1 plex   plex   624855040 Nov  7 10:25 com.plexapp.plugins.library.db
-rwxrwxr-x 1 plex   plex    72714240 Nov 24  2014 com.plexapp.plugins.library.db-2014-11-24-tmp
-rwxrwxr-x 1 plex   plex        1024 Nov 24  2014 com.plexapp.plugins.library.db-2014-11-24-tmp-journal
-rwxrwxr-x 1 plex   plex   595941376 Apr  2  2019 com.plexapp.plugins.library.db-2019-04-02-tmp
-rwxrwxr-x 1 plex   plex   594251776 Apr  5  2019 com.plexapp.plugins.library.db-2019-04-05-tmp
-rwxrwxr-x 1 plex   plex   599023616 Apr  8  2019 com.plexapp.plugins.library.db-2019-04-08-tmp
-rwxrwxr-x 1 plex   plex   602158080 Apr 11  2019 com.plexapp.plugins.library.db-2019-04-11-tmp
-rwxrwxr-x 1 plex   plex   599085056 Apr 14  2019 com.plexapp.plugins.library.db-2019-04-14-tmp
-rwxrwxr-x 1 plex   plex   601030656 Apr 17  2019 com.plexapp.plugins.library.db-2019-04-17-tmp
-rwxrwxr-x 1 plex   plex       32768 Nov  7 10:25 com.plexapp.plugins.library.db-shm
-rwxrwxr-x 1 plex   plex      405608 Nov  7 10:25 com.plexapp.plugins.library.db-wal
-rw-r--r-- 1 plex   plex    21879808 Nov  7 07:28 tv.plex.providers.epg.cloud-e0982379-dca2-4b78-aa04-fcf64c038c3c.db
-rw-r--r-- 1 plex   plex       32768 Nov  7 10:25 tv.plex.providers.epg.cloud-e0982379-dca2-4b78-aa04-fcf64c038c3c.db-shm
-rw-r--r-- 1 plex   plex     1077376 Nov  7 07:28 tv.plex.providers.epg.cloud-e0982379-dca2-4b78-aa04-fcf64c038c3c.db-wal

Just as a reference this is the backup directory:

total 7467842
drwxrwxr-x  2 thomas plex          40 Nov  7 09:49 .
drwxr-xr-x 48 thomas plex        4096 Aug 11 08:29 ..
-rwxrwxr-x  1 thomas plex     9417728 Jul 29  2018 com.plexapp.dlna.db
-rwxrwxr-x  1 thomas plex           0 Jan 10  2014 com.plexapp.dlna.db-journal
-rwxrwxr-x  1 thomas plex       98304 Jul 29  2018 com.plexapp.dlna.db-shm
-rwxrwxr-x  1 thomas plex     9638488 Jul 29  2018 com.plexapp.dlna.db-wal
-rwxrwxr-x  1 thomas plex    25198592 Apr 14  2019 com.plexapp.plugins.library.blobs.db
-rwxrwxr-x  1 thomas plex    25204736 Apr  2  2019 com.plexapp.plugins.library.blobs.db-2019-04-02-tmp
-rwxrwxr-x  1 thomas plex    25204736 Apr  5  2019 com.plexapp.plugins.library.blobs.db-2019-04-05-tmp
-rwxrwxr-x  1 thomas plex    25207808 Apr  8  2019 com.plexapp.plugins.library.blobs.db-2019-04-08-tmp
-rwxrwxr-x  1 thomas plex    25210880 Apr 11  2019 com.plexapp.plugins.library.blobs.db-2019-04-11-tmp
-rwxrwxr-x  1 thomas plex    25210880 Apr 14  2019 com.plexapp.plugins.library.blobs.db-2019-04-14-tmp
-rwxrwxr-x  1 thomas plex    25198592 Apr 17  2019 com.plexapp.plugins.library.blobs.db-2019-04-17-tmp
-rwxrwxr-x  1 thomas plex    25198592 Apr 20  2019 com.plexapp.plugins.library.blobs.db-2019-04-20-tmp
-rw-r--r--  1 plex   plex    25339904 Oct 29 02:02 com.plexapp.plugins.library.blobs.db-2019-10-29
-rw-r--r--  1 plex   plex    23546880 Nov  1 02:02 com.plexapp.plugins.library.blobs.db-2019-11-01
-rw-r--r--  1 plex   plex    23546880 Nov  4 02:04 com.plexapp.plugins.library.blobs.db-2019-11-04
-rw-r--r--  1 plex   plex    23546880 Nov  7 02:04 com.plexapp.plugins.library.blobs.db-2019-11-07
-rw-r--r--  1 thomas thomas     32768 Nov  7 09:49 com.plexapp.plugins.library.blobs.db-2019-11-07-shm
-rw-r--r--  1 thomas thomas         0 Nov  7 09:49 com.plexapp.plugins.library.blobs.db-2019-11-07-wal
-rwxrwxr-x  1 thomas plex       32768 Apr 28  2019 com.plexapp.plugins.library.blobs.db-shm
-rwxrwxr-x  1 thomas plex           0 Apr 28  2019 com.plexapp.plugins.library.blobs.db-wal
-rwxrwxr-x  1 thomas plex   601953280 Apr 28  2019 com.plexapp.plugins.library.db
-rwxrwxr-x  1 thomas plex    72714240 Nov 24  2014 com.plexapp.plugins.library.db-2014-11-24-tmp
-rwxrwxr-x  1 thomas plex        1024 Nov 24  2014 com.plexapp.plugins.library.db-2014-11-24-tmp-journal
-rwxrwxr-x  1 thomas plex   595941376 Apr  2  2019 com.plexapp.plugins.library.db-2019-04-02-tmp
-rwxrwxr-x  1 thomas plex   594251776 Apr  5  2019 com.plexapp.plugins.library.db-2019-04-05-tmp
-rwxrwxr-x  1 thomas plex   599023616 Apr  8  2019 com.plexapp.plugins.library.db-2019-04-08-tmp
-rwxrwxr-x  1 thomas plex   602158080 Apr 11  2019 com.plexapp.plugins.library.db-2019-04-11-tmp
-rwxrwxr-x  1 thomas plex   599085056 Apr 14  2019 com.plexapp.plugins.library.db-2019-04-14-tmp
-rwxrwxr-x  1 thomas plex   601030656 Apr 17  2019 com.plexapp.plugins.library.db-2019-04-17-tmp
-rw-r--r--  1 thomas plex   595641344 May  5  2019 com.plexapp.plugins.library.db-2019-05-05
-rw-r--r--  1 thomas plex   600122368 May  8  2019 com.plexapp.plugins.library.db-2019-05-08
-rw-r--r--  1 thomas plex   597159936 May 11 02:00 com.plexapp.plugins.library.db-2019-05-11
-rw-r--r--  1 thomas plex   599988224 May 14 02:02 com.plexapp.plugins.library.db-2019-05-14
-rwxrwxr-x  1 thomas plex     4685824 Apr 28  2019 com.plexapp.plugins.library.db-shm
-rwxrwxr-x  1 thomas plex   612030984 Apr 28  2019 com.plexapp.plugins.library.db-wal
-rwxrwxr-x  1 thomas plex    27502592 Apr 28  2019 tv.plex.providers.epg.onconnect-e0982379-dca2-4b78-aa04-fcf64c038c3c.db
-rwxrwxr-x  1 thomas plex       32768 Apr 28  2019 tv.plex.providers.epg.onconnect-e0982379-dca2-4b78-aa04-fcf64c038c3c.db-shm
-rwxrwxr-x  1 thomas plex    24219312 Apr 28  2019 tv.plex.providers.epg.onconnect-e0982379-dca2-4b78-aa04-fcf64c038c3c.db-wal

Im not sure why ( i did do a full library scan) but there is now 2 more db-shm and db-wal files one for com.plexapp.plugins.library and com.plexapp.plugins.library.blobs.

[DJ-BrianC]

The tmp files can be removed.

[tomt92]

Thanks, cleared those all out but no change…

[DJ-BrianC]

Yeah I didn’t expect any change. Just pointing out based on mine there weren’t needed. Ypu tried making a backup and repairing them?

[OttoKerner]

Restoring a backup of the database is not a ‘repair’. Particularly because the damage can already be growing in the backup file as well.

The fact that it is leaving the -tmp files in there points to a damaged database, IMHO.
Please try https://support.plex.tv/articles/201100678-repair-a-corrupt-database/

[tomt92]

Im trying to restore from the backup blob and library files i have now… So far the server is not starting.

[OttoKerner]

The blob files are quite irrelevant for this matter.
Concentrate on the library.

[tomt92]

will do…

[tomt92]

so i did a restore of library.db that is dated back in May… That worked. I have a copy of the original Database directory contents stored else where that i’ll try to run a repair on to see if that will work… For the time being i need to head to work so i wont be able to look at this until 2 or 3…

Thanks for the help, i may be back.

[ChuckPa]

I would like to ask where the files are stored.

Are they machine local or on a network share?

I ask this because if the DB is corrupting so frequently then file locking isn’t working correctly. The most common occurrences are with network-mounted shares not employing local_lock=posix in the mount options.

[tomt92]

Hi ChuckPA . … The data files are on another drive (DataBig) in the server. There is a link from the application directory to ../DataBig/Plex/Library/.... … Its a dedicated Plex data drive as well as a backup drive for a few things about 2T in size and 10% is used. Its posable that the drive is starting to get bad sectors ( i can run a scan then)…

Side note – Plex is not on the main system drive because at the time of build i only had a 128GB SSD drive and it was hitting the upper usage limits, so i moved the data files to a separate drive.

I did move the current DB to my mac, ran a repair and uplaoded it to the server and that did fix the issue. As i mentioned i’ll do a drive check on the data drive as well as look into writing a cron job that does a DB backup for me. My guess is if there is an issue with the DB when Plex tries to do a backup it fails silently. Im also questioning why if they repair is a data export and reimport why isnt the backup just a DB dump to SQL?.. (i may do this as my own backup script)

Thanks everyone for the help on this

[TeknoJunky]

@tomt92

-rwxrwxr-x 1 thomas plex

the backup directory should have plex:plex ownership.

do not put your plex data or backup in your HOME folder.

also, I can’t tell from your post if your ‘databig’ drive is LOCAL or if it is REMOTE.

Remote drives do not work well with databases, so if that is the case, then that is why your db keeps getting corrupt.

If databig is a local drive, you might want to run a S.M.A.R.T. scan on it to make sure its not failing sectors or other errors.

[ChuckPa]

Symlinks are a PITA and unreliable for PMS.

If you want the metadata directory moved from the default location:

1. Create a systemd override.conf file (preferred and fully supported method)
2. mount --bind method (for init-based systems)

Moving PMS 'Library'

Moving Plex’s metadata (Library) directory on systemd based systems This information is intended for those systems where cat /proc/1/comm returns systemd. If your system returns init, please see the next post in this thread. While you may change /lib/systemd/system/plexmediaserver.service, it is under the control of the package manager and subject to replacement without warning at any time during an update. To provide stability, systemd allows creation of an override.conf file to customiz…

Regarding the failing issue, Symlinks / remote mounts without posix locking are the two reasons it fails unless out of space. I’ve never been able to make it fail when mounted / defined correctly. It seems only by ‘cheating’ , does it get in trouble.

[tomt92]

Nothing is in home; ssd with limited space. DataBig (/mnt/thomas/DataBig) is a 2TB drive on the same machine, so local. Unfortunately the drive does not have any SMART capabilities that i can locate… I can try and changing the BU directories ownership to plex:plex and see if that helps in the backup process… I have a feeling it’s something else as the backups will run for a while and then just stop, also .blob seems to be backed up without any issue. The backup directory is on my ZFS array but that shouldn’t really affect anything.

[tomt92]

Thanks ChuckPA, i’ll look into this… Im off tomorrow and even-though i can do this from work id prefer to be in front of the machine.