Plex Database sqlite file has more than doubled in 10 days

Server Version#: Version 1.19.5.3112
Player Version#: PlexWeb Version 4.34.3

First, I want to know if I should be worried.

[ Databases]$ pwd
/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases
[ Databases]$ ls -l
total 8215848
-rw-rw-r--. 1 plex plex     991232 Aug 21 23:47 com.plexapp.dlna.db
-rw-rw-r--. 1 plex plex      32768 Aug 21 23:47 com.plexapp.dlna.db-shm
-rw-rw-r--. 1 plex plex    1268992 Aug 21 23:47 com.plexapp.dlna.db-wal
-rw-rw-r--. 1 plex plex     334848 Aug 21 23:41 com.plexapp.plugins.library.blobs.db
-rw-r--r--. 1 plex plex     334848 Aug 12 03:04 com.plexapp.plugins.library.blobs.db-2020-08-12
-rw-r--r--. 1 plex plex     334848 Aug 15 03:05 com.plexapp.plugins.library.blobs.db-2020-08-15
-rw-r--r--. 1 plex plex     334848 Aug 18 03:07 com.plexapp.plugins.library.blobs.db-2020-08-18
-rw-r--r--. 1 plex plex     334848 Aug 21 03:10 com.plexapp.plugins.library.blobs.db-2020-08-21
-rw-rw-r--. 1 plex plex      32768 Aug 21 23:47 com.plexapp.plugins.library.blobs.db-shm
-rw-rw-r--. 1 plex plex       2128 Aug 21 23:47 com.plexapp.plugins.library.blobs.db-wal
-rw-rw-r--. 1 plex plex 1796244480 Aug 22 03:09 com.plexapp.plugins.library.db
-rw-r--r--. 1 plex plex  714599424 Aug 12 03:04 com.plexapp.plugins.library.db-2020-08-12
-rw-r--r--. 1 plex plex  939709440 Aug 15 03:05 com.plexapp.plugins.library.db-2020-08-15
-rw-r--r--. 1 plex plex 1403302912 Aug 18 03:07 com.plexapp.plugins.library.db-2020-08-18
-rw-r--r--. 1 plex plex 1702551552 Aug 21 03:10 com.plexapp.plugins.library.db-2020-08-21
-rw-rw-r--. 1 plex plex   14057472 Aug 22 03:29 com.plexapp.plugins.library.db-shm
-rw-rw-r--. 1 plex plex 1838534728 Aug 22 03:29 com.plexapp.plugins.library.db-wal

I have 107 movies, about 400 TV episodes, 484 ā€˜other’ videos and about 50000 pictures

In the last 9 days, I added three movies and 20 other videos.

As you can see, 10 days ago, library.db was 714 M, now it is 1.8T? What could possibly have happened?

I decided to turn off backups though without them I guess I wouldn’t know it had grown so much. Is there a way to make it small again?

It is much larger than my media folder even, which seems odd with all those photos thumbnails in the media folder:

Plex Media Server]$ du -d 1
530060  ./Cache
9440    ./Codecs
204     ./Crash Reports
0       ./Diagnostics
74264   ./Logs
2815212 ./Media
377556  ./Metadata
8262264 ./Plug-in Support
35628   ./Plug-ins
12104636        .

Thanks for any insight and help you can offer!

Could I restore the 714 MB one and just rescan my files since they changed so little in the last 10 days? To at least get back to the smaller DB size?

Are you using the 3rd-party Plex API for Python?

I doubt that I am. I do everything through the webpage at port 32400. I have some 3rd party plugins, ā€œSub-Zeroā€ and ā€œWebtoolsā€ but they’ve been installed for months, I haven’t tweaked them to my knowledge. I haven’t used the Webtools since I rebuilt my server and restored the DB a few months ago.

The DLNA server is on, but I haven’t used it either except a couple tests. Plan to use it more when/if I add a music library. Have a couple Rokus, iPad, and Sony Smart TV that access Plex regularly through the app. Don’t do a lot of viewing of the photo library, mostly movies and TV watching.

I can only assume that the pictures are responsible for this.

Is there a way to check and make sure? Do you need logs or want me to look at log files? I know without specifics, this would be hard to track down. So I can do what’s needed there, just let me know.

Did you recently add more pictures?

Have you tried the database repair procedure?

If all else fails, try the above mentioned fallback to one of the backups.

If you have access to a SQLite editor, analyse the tables and try and name the one that is exceedingly large. This might provide a clue.

I ran sqlite3_analyzer. There is one table that is MUCH bigger than the rest, if I am reading this right:

*** Page counts for all tables with their indices *****************************

STATISTICS_MEDIA.................................. 1549733     93.2% 
METADATA_ITEMS.................................... 47767        2.9% 
MEDIA_PARTS....................................... 25429        1.5% 
TAGGINGS.......................................... 14972        0.90% 
MEDIA_ITEMS....................................... 11339        0.68% 
MEDIA_STREAMS..................................... 3224         0.19% 

What is the Statistics Media table? Is there a way to clean it out?

Sorry, I should have put a little more in here from the report. I pasted in the first few lines from the report of the ā€œbig stuffā€.

*** Page counts for all tables and indices separately *************************

STATISTICS_MEDIA.................................. 593307      35.7% 
INDEX_STATISTICS_MEDIA_ON_ACCOUNT_ID_AND_TIMESPAN_AND_AT 500120      30.1% 
INDEX_STATISTICS_MEDIA_ON_AT...................... 456306      27.4% 
METADATA_ITEMS.................................... 28503        1.7% 
MEDIA_PARTS....................................... 15461        0.93% 
TAGGINGS.......................................... 9876         0.59% 
MEDIA_ITEMS....................................... 7365         0.44% 
INDEX_MEDIA_PARTS_ON_FILE......................... 5030         0.30% 



*** All tables ****************************************************************

Percentage of total database......................  39.9%    
Number of entries................................. 16619201  
Bytes of storage consumed......................... 679804928 
Bytes of payload.................................. 529822166   77.9% 
Bytes of metadata................................. 125376747   18.4% 




*** All indices ***************************************************************

Percentage of total database......................  60.1%    
Number of entries................................. 34696084  
Bytes of storage consumed......................... 1022745600
Bytes of payload.................................. 886420022   86.7% 
Bytes of metadata................................. 116073731   11.3% 





*** Table STATISTICS_MEDIA and all its indices ********************************

Percentage of total database......................  93.2%    
Number of entries................................. 48121050  
Bytes of storage consumed......................... 1586926592
Bytes of payload.................................. 1325863632  83.5% 
Bytes of metadata................................. 228469921   14.4% 


*** Table STATISTICS_MEDIA w/o any indices ************************************

Percentage of total database......................  35.7%    
Number of entries................................. 16040350  
Bytes of storage consumed......................... 607546368 
Bytes of payload.................................. 473547885   77.9% 
Bytes of metadata................................. 120750717   19.9% 
B-tree depth...................................... 4         


*** Indices of table STATISTICS_MEDIA *****************************************

Percentage of total database......................  57.5%    
Number of entries................................. 32080700  
Bytes of storage consumed......................... 979380224 
Bytes of payload.................................. 852315747   87.0% 
Bytes of metadata................................. 107719204   11.0% 
Average payload per entry......................... 26.57     


*** Index INDEX_STATISTICS_MEDIA_ON_ACCOUNT_ID_AND_TIMESPAN_AND_AT of table STATISTICS_MEDIA 

Percentage of total database......................  30.1%    
Number of entries................................. 16040350  
Bytes of storage consumed......................... 512122880 
Bytes of payload.................................. 443669168   86.6% 
Bytes of metadata................................. 54122486    10.6% 


*** Index INDEX_STATISTICS_MEDIA_ON_AT of table STATISTICS_MEDIA **************

Percentage of total database......................  27.4%    
Number of entries................................. 16040350  
Bytes of storage consumed......................... 467257344 
Bytes of payload.................................. 408646579   87.5% 
Bytes of metadata................................. 53596718    11.5% 

I am asking about this. It might take a while before I get feedback, since it’s the weekend now.

Okay. Thanks. The statistics_media table content is very interesting. The database is over 2 years old that I have, and there’s just an explosion of entries as of late. There is a date with the entry, making it pretty easy to see when the bulk of the entries were added to the table. Of course not knowing how the table works I’m not certain of anything. It is odd that first date is exactly Jan 1st…

First entry :

row   id   account_id    device_id   timespan  at  metadata_type  count  duration
1  1	1	8	0	2018-01-01 00:00:00	12	5	1626
<1 year>
797  3576	1	8	0	2019-01-01 00:00:00	4	198	605589
<1 year>
1739   7204	1	28	0	2020-01-01 00:00:00	1	1	2452
<Here's where it gets weird.  There are about 7.4 million more rows with this date>
7435288    16053485		8	0	2020-01-01 00:00:00	1	0	0
7435289    7223	1	34	2	2020-01-06 00:00:00	4	21	53293

Then several million more rows, mostly of dates 2020-07-01 and 2020-08-01, but several million of dates in the last 10 days of 2020-08-10 thru 08-21.

I think it is important to note:

I JUST upgraded to the server version in my first post. So technically most of this happened while this version was installed.:

plexmediaserver-1.19.4.2935-79e214ead.x86_64

I have been running 1.19.4.2935 since July 15th, 2020 until just a few hours ago.

If you are willing to send me a copy of your database (please ZIP it), I can relay this to a developer, who has agreed to take a look at it.

Please download the database using the web app. If you want to copy it directly out of the plex data folder, make sure that the Plex server daemon is stopped before you do it. Otherwise the database copy will be damaged.

Here’s the logs. Working on the database…

Plex Media Server Logs_2020-08-22_05-48-47.zip (4.5 MB)

The database zipped is just over 200 MB. I’ll see if it will attach… Nope. Upload just aborts.

I put the zip file here:
[moderator edit: link removed]

Let me know if you have trouble getting it.

Thanks, please leave the file up there for a few days.
I will edit the post to remove the link though.

Thanks! I’ll leave the file up as long as you need it. Wish the developer good luck.

I’ve got disk space so no rush, and the server is still snappy enough even with the big DB. So I’ll go on standby. If I notice that upgrading to the latest version has slowed the DB growth, I’ll note that here.

You might wanna try the database repair procedure. It will throw away all index tables and Plex server will recreate them on first startup afterwards. Maybe this will make some difference.

I experienced this issue several months ago. I noticed it because database optimization on my system went from completing in 4 - 5 seconds to several minutes. This prompted me to check my database size and it had ballooned by an order of magnitude. I used sqldiff to compare the current DB with a normal-sized backup and found that the table statistics_media was being written to constantly, even with no server usage.

Examining the database further, I found that these ā€œrogueā€ additions were being made with a device_id not present in other entries in the database (or at least not many entries) and the account_id was null. As I had a recent, working database backup, I decided to whack all of the entries with a null account_id; after doing so, my database size reverted to its normal size.

I also signed out all of my known clients and cleaned up my authorized devices in settings. The issue has not recurred since doing so. I’d been playing around various clients (Plex desktop app, Plex Media Player, Plex Amp) around that time, so it may have been something I did with one of those which caused the rogue writes.

I’m not necessarily suggesting you modify your database, particularly if you don’t have recent, working backup. However, it did help in my case.

2 Likes

I’d really like to be able to disable statistics collection in general. I’d rather not write to the database every [minute] just because any client is open, anywhere.

Or similarly, it would be nice to move the statistics_* tables out of the primary db. They grow constantly over time and receive disproportionately more writes than any other table.

If you don’t care about the statistics, I don’t think there’s any danger in delete from statistics_media or delete from statistic_bandwidth.

Edit: there are LOTS of reasons you might care about statistics. See Otto’s comment below.

(Obviously make backups, etc. And do it AFTER you share with Plex so they can identify what’s going on.)

(If I’m wrong I’m sure somebody will say NO THAT’S WRONG, DON’T DELETE THAT DATA.)