Suggested SQLite3 DB Optimizations

If you are doing this on a broader scope than personal use, such as providing a packaged solution for others as you mention, you might want to receive written permission prior to doing so.

The other obvious problem is that if Plex makes code changes in the binary it could cause these solutions to cease to work, or worse cause corruption to a database, at any time. These tweaks incur a certain maintenance cost and risk.

Noted. The overlay isn’t in reapology or accessible from eselect, so it’s not that noticed.

This is a violation of Plex’s licensing .

License to modify was not granted.

If you want a tool for the database (which is currently Linux-based hosts only), I have one.

Playing with cache size is a bandaid for another problem. The problem should be fixed instead of applying more bandaids.

The tool I have corrected several problems on a machine with only 1GB of RAM.
If PMS runs fine there, all of the above is moot.

I’m sure we would all appreciate reviewing your script. Thank you.

1 Like

Chuck, can you poke somebody about modifying the default database page size?

It’s currently 1024 which is SO OLD. Like dawn-of-SQLite old, back when block devices had such small blocks, dinosaurs roamed the earth, and you and I were learning about computers. Even wildly-conservative SQLite has defaulted to 4096 forever, and they target literal embedded systems.

I’ve been using bigger values forever, but 4096 (at least) is a no-brainer.

(The template database file that ships with Plex is 1024. That’s also the right place to modify it on a brand new system. Before first run, the other files don’t exist.)

I’d be happy to grab some links and resources and even some generic benchmarks if that would be helpful for a discussion.

2 Likes

@Volts

  1. I ABSOLUTELY will poke folks about updating the page size.

  2. We need a very well-defined and objective demonstration which includes PMS operating in both large and small (1 GB total system RAM) environments).

  3. We further need to show how the different tables and indexes react to those changes.

With that in hand, I seriously doubt they will flinch at making changes.

2 Likes

Here is my tool.

  1. Stop PMS
  2. Run as root
  3. If you want the most expedient, I recommend actions:
  • Check
  • Repair
  • Reindex
  1. If you want to examine the individual facets,
  • Check
    ls -la the database sizes before starting
  • Vacuum
    – Now ls -la of the databases again
  • Reindex
    – One last check

Feel free to check the DB sizes between each step.

The only conflict possible is if PMS is running and you attempt to use the tool AFTER PMS starts.

This tool only checks for PMS at first startup so be careful.

DBRepair.tar (40 KB)

I typically see problems with PMS when:

  1. There is small, but non-fatal, database damage
  2. The WAL & SHM files do not fold back into the main DB at PMS exit (which it ALWAYS should)

NOTE:

I’ve added an extra couple checks based on feedback received.
Please let me know if you get errors before getting to the main menu.

15-Oct-2022

Restructured Docker test.

1 Like

Thanks! I’ll gather some references.

And I understand that you target relatively small systems. It’s funny how these days 1 GB is considered small-ish.

(The fact the template database’s page_size is 1024 is probably a really neat indication that the SAME file has been used by Plex for a very, very long time. That’s kinda cool history.)

1 Like

ABSOLUTELY!

  1. My first machine had 4K RAM
  2. The first multi-user machine I used (and help build) had 128KB RAM.

DA*&@#$&@#%$ PROGRAMMERS! BLOAT BLOAT BLOAT!!!

:rofl:

3 Likes

Then i’ll not use this script, but I reaally hope that Plex will give more space for cache, because obviously this has become ridiculsly little in 2022 :slight_smile:

@Tangs

I would like to suggest:

  1. Use my tool (script)
  2. REPAIR the DB (which does an export/import
  3. REINDEX
  4. Now start PMS and let me know how much quicker it is

(You speak of cache size , yes that’s one issue, locality of reference in a flat-file database is the bigger issue here. Export/Import makes the tables contiguous again )

If you have a ‘larger’ DB, I’m 100% certain the internal links are splattered all over the place. This action fixes that.

1 Like

For my education, how is import/export different from performing a vacuum? I was under the impression that VACCUM rebuilt the database and aligns table data to be contiguous.

From what I understand, you’re correct for default SQLite.

Engineering took SQLite3 source and added extra modules to it to create “Plex SQLite”

In all I’ve tested, VACUUM never does as well as export/import.
(My largest test was a 340K TV episode database)

1 Like

Thank you for the expalanation!

Hey @ChuckPa,

I realize there are always risks associated with doing stuff like your script (big or small).
Just curious, what are the potential risks with running your script?

Also, for Docker installs, would the script be run on the host where the data is persisted too since you say to stop PMS?

I’m interested in trying to run this but want to make sure I do it properly for my system. I quickly looked through the script and even though bash scripting isn’t my forte, I am able to understand enough programming/logic to figure out what is going on.

Thanks!
-Shark2k

I make backups of the database at each step along the way.

Each backup copy is placed in a subdirectory named dbtmp

Each backup uses a time stamp so you can see the chronology.

The Logfile (you can display while still in the tool as well) will let you see the steps you took

The Logfile persists after tool exit.

When you exit, you have the option of purging the intermediate backup databases you’ve accumulated along the way.

If you keep them , you can recover to any point in time you wish without data loss.

I put in all the precautions because I know how tired I get and want my *@#$@# covered :slight_smile:

1 Like

ALL:

Based on feedback, I added an extra precautionary test in the startup.

If permissions aren’t adequate, it’ll report so and exit.

This should prevent it from spitting errors when trying to initialize.
Please let me know if any other annoyances are detected.

Presumption is:

  1. UID/GID of the invoking user matches the PMS user
    -or-
  2. UID/GID of the invoking user is ‘root’ (0:0)

Please remember, this tool exists solely to assist and save all the typing which reduces the risk of error which really can destroy databases.

1 Like

Thanks for reassurance on running the script.

Regarding your last post, is there a new link to download the latest version with the extra precautionary test you added or is the same link earlier up in the thread updated?

Thanks,
-Shark2k

No new link. I updated the post.

1 Like