Database size growth when using plexapi

Plex Media Server
1

Server Version#: 1.19.2.2737

Running on Debian:

Linux gemini 4.19.0-8-amd64 #1 SMP Debian 4.19.98-1+deb10u1 (2020-04-27) x86_64 GNU/Linux

I had recently started using plexapi to mark some shows watched automatically and tie into deleting trash as I use some cloud storage for my media which is all scripted together and working well.

I had noticed that my navigation was going slowly and dug into a bit more and noticed my database has had some massive growth at some point and was hitting near 2.5GB in size:

216M	com.plexapp.plugins.library.db
2.5G	com.plexapp.plugins.library.db.original
448K	com.plexapp.plugins.library.db-shm
219M	com.plexapp.plugins.library.db-wal

I had read through these posts:

and went through and deleted the old metaviews which seemed to be 5 million entires and the source of the space consumption when I sqldiff’ed the original vs the new:

metadata_item_views: 0 changes, 0 inserts, 5253810 deletes, 10 unchanged

With my DB back down to 200MB which seems reasonable again, things are working as expected again.

I was using 3.3.0 of the API and recently upgraded to 3.4.0.

Is that a bug in the API or something not being cleaned up on the Plex Side?

2

sounds like someone has been swimming in the deep end of the pool with the lights off :wink:

What I’m seeing above, albeit only a snippet, tells me there have been a lot of metadata changes & deletions.

The butler, by default, will run during scheduled maintenance and vacuum (compact) as it does backups of the DB.

This implies to me that the butler isn’t running for the DB / PMS or computer are shutdown when maintenance should run.

The “API” isn’t concerning here (aside from the number not looking like the right version).

I’m concerned what’s deleting so much.

What is generating the system activity? “Refresh all metadata” or ???
We need to understand the source of the activity and shine some lights on that first

3

I have all my scheduled tasks on and they run nightly as the server is on 24x7.

That’s a bad explanation by me. I used the linked post to delete all the metadata views to reclaim the 2.1GB of DB size back. I was comparing the large DB to the reclaimed DB as the 200MB was more of what I expected to see on size.

Here is the normal growth over 6 days from size:

felix@gemini:~$ ls -al /data/rsnapshot/daily.*/localhost/var/lib/plexmediaserver/Library/"Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db"
-rw-r--r-- 1 plex felix 2634797056 May  4 09:19 '/data/rsnapshot/daily.0/localhost/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db'
-rw-r--r-- 1 plex felix 2572644352 May  3 02:06 '/data/rsnapshot/daily.1/localhost/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db'
-rw-r--r-- 1 plex felix 2515505152 May  2 01:41 '/data/rsnapshot/daily.2/localhost/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db'
-rw-r--r-- 1 plex felix 2450825216 May  1 01:38 '/data/rsnapshot/daily.3/localhost/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db'
-rw-r--r-- 1 plex felix 2406170624 Apr 30 01:47 '/data/rsnapshot/daily.4/localhost/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db'
-rw-r--r-- 1 plex felix 2340978688 Apr 29 01:33 '/data/rsnapshot/daily.5/localhost/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db'
-rw-r--r-- 1 plex felix 2274791424 May  4 09:19 '/data/rsnapshot/daily.6/localhost/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db'

Here is the sqldiff from 6 days ago to the backup from last night as an example:

metadata_item_views: 0 changes, 664647 inserts, 46 deletes, 4589173 unchanged

I have a starting DB now and I compare a day’s worth with all my two plexapi scripts off.

4

how many people are using your server?
Just how much content is being watched?

What’s shown here is one HECK of a lot of activity.

5

Ijust figured something out.

Is this what you’re talking about?

Overview
Unofficial Python bindings for the Plex API. Our goal is to match all capabilities of the official Plex Web Client. A few of the many features we currently support are:
6

Yes, apologies if that was also unclear.

It’s a small setup as that hasn’t changed minus some slightly increased views with the whole lockdown, but nothing that large as it’s a few family members and some close friends.

The change I made was automating a few things via that link you have above, which seems to be the culprit.

7

Whatever that “PlexAPI” is doing, it’s making a mess.
If that’s the foundation of your automation then the foundation isn’t going to work.

I don’t know what they’re doing. They’re not official product which explains why I’ve not encountered it until now.

What are you trying to automate?

8

I was trying to consolidate a few things.

  • I move files from local to cloud storage and empty trash is off by default. I was using curl to delete libraries and I had moved function over to PlexAPI
  • I have something that marks shows watched that other folks might have added that I don’t so they don’t show up

I can move the first back to curl as I know that works without much of an issue. I have to find a different solution for the second.

I’m surprised other folks using that have not noticed it before as I can’t imagine I’m running something that crazy. I did log a post with them as well.

Empty Trash:
https://python-plexapi.readthedocs.io/en/latest/modules/library.html#plexapi.library.Library.emptyTrash

and Mark Watched:
https://python-plexapi.readthedocs.io/en/latest/modules/video.html?highlight=markwatched#plexapi.video.Video.markWatched

Nothing too fancy. I’m guessing my frequency is what was doing it and my method was probably it.

9

I wish you the best of luck with this but feel, from this point forward, it’s better evaluated and resolved through their github.

10

Does this mean you are deleting and recreating whole libraries in Plex on a regular basis?
If you do, I’d say this is a major factor in causing slowness. If you can change that to deleting and adding media items instead, it might improve responsiveness greatly.

11

Apparently Monday morning and written English are not working for me as that’s just a bad way for me to explain it.

I’m not deleting libraries as the use case is a mergerfs combo of a local mount and cloud mount.

At times, media gets upgraded so it’s deleted from the cloud mount and added to the local. This is all transparent to Plex so I use the API to empty the trash as I do not want to automatically empty the trash as with a disconnect or something goofy, that would delete all the cloud items since they are not present.

12

Just one last one @ChuckPa as I wanted to make sure I got the problem issue in case someone else gets it.

I did a test which marks about 20 shows and that generated 2k views for me:

felix@gemini:/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases$ sqldiff --summary com.plexapp.plugins.library.db.fixed com.plexapp.plugins.library.db | grep meta | grep view
metadata_item_views: 0 changes, 1965 inserts, 0 deletes, 531 unchanged

If do the math out, I’m getting roughly 700k per week, which matches up my 6 day growth pretty close. At some point if I was following you, the butler processes would eventually clean those up after ‘x’ days and my volume of generation was more the issue than a maintenance task not firing off.

I can go back and check the old DBs I have to see when the last records were in there as I have the last 7 days so I wanted to validate this so I can address it from my side.

13

If Plex is operating normally (not being manipulated by an external application), the normal growth observed from ingesting & updating metadata records is handled by the butler in its vacuuming (what it’s designed for).

14

Yep, makes sense. I was basically making the script click on 25 shows every 15 minutes and mark them watched so that is a bit outside of what a normal human would do.

I setup a few monitors to make the size does not grow as I am fully confident on the root cause now.

15

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.