DBRepair development

Here’s what it’s looking like.

 
 
      Plex Media Server Database Repair Utility (Ubuntu 20.04.6 LTS)
                       Version v1.02.99
 

Select

  1 - 'stop'      - Stop PMS.
  2 - 'automatic' - Check, Repair/Optimize, and Reindex Database in one step.
  3 - 'check'     - Perform integrity check of database.
  4 - 'vacuum'    - Remove empty space from database without optimizing.
  5 - 'repair'    - Repair/Optimize databases.
  6 - 'reindex'   - Rebuild database database indexes.
  7 - 'start'     - Start PMS

  8 - 'import'    - Import watch history from another database independent of Plex. (risky).
  9 - 'replace'   - Replace current databases with newest usable backup copy (interactive).
 10 - 'show'      - Show logfile.
 11 - 'status'    - Report status of PMS (run-state and databases).
 12 - 'undo'      - Undo last successful command.

 21 - 'remove'    - Remove old jpg files from PhotoTranscoder cache.
 42 - 'ignore'    - Ignore duplicate/constraint errors.

 88 - 'update'    - Check for updates.
 99 - 'quit'      - Quit immediately.  Keep all temporary files.
      'exit'      - Exit with cleanup options.

Enter command # -or- command name (4 char min) : 

Will this work?

1 Like

In the realm of terminology the obvious choice is “Flush.” The less humorous choice is “Clean Cache” or “Garbage Collection.”

Regarding your implementation description -

  • Why not allow “age” to be passed to the utility (e.g. “1w” or “7d”, or something similar) rather than restrict to the 30 days that PMS may, or may not, actually enforce?
  • Offer an option to hard-link the duplicates?
  • Offer an option to take into consideration total utilization of the storage device (e.g, “Never allow PhotoTranscoder directory to consume more than X% of device storage.” and remove older files, regardless of age, until the total cache size is reduced below X%)?

And if you have access to the PMS source itself, some options that might be worthwhile in creating and managing the cache:

  • “Global” or “Combined Cache” vs “Per-Client cache”
  • JPEG image quality (set quality threshold on JPEG compression for the cache assets)

Also, your utility calls out specifically removing JPG files. I see in my PhotoTranscoder cache files with the extensions “jpg” “jpeg” and “ppm”. All three of these seem like good removal candidates.

@dzm06

I am trying to figure out how to implement " Settings / Preferences "
When I have figured out a way to do that (without making the script HUGE), I’ll add alot.

Yes, I’ll get JPG, and JPEG. I need to ask engineering about PPM files as I know those are in the DB.

While your other requests / recommendations are valid, this tool is a shell script.
There’s a limit to what I can do with /bin/sh. This tool was created to run in all environments PMS runs on. Not all NAS platforms offer /bin/bash.

Its primary focus is the DB and to do what PMS can’t while PMS is running.
There are limits to what I can do/publish in non-compiled code (specific APIs, etc)

Yep. Was just looking at the script and contemplating patches to submit back. The various flavors of bash, sh, ash, zsh etc offered in all these various environments will make it a challenge to make a single utility that will run everywhere. Especially once Windows (even Windows + LinuxShell that MSFT has been pushing for a while) is taken into account.

I’m always open to hear about new things which are of benefit to all the Linux platforms.

The nicest thing to have at this point would be a Windows equivalent.

I don’t know powershell . I’m largely Windows illiterate
(WSL doesn’t even have everything I need at this point. The old CIG_Win package comes the closest)

Of the options provided (purge, remove, clear, clean, delete) my vote is for “purge”. I would use “purge” when referring to stuff that can be re-created when prompted by an end-user action with a minor impact on key resources (time, storage).

This got me thinking about the drama of Doctor Who (Classic) “Castrovalva” (Season 19, Episode 2). The Master (The Doctor’s cackling foe) set the TARDIS on a course traveling back to the biggest explosion in history “Event One – The Big Bang”. Australia’s pioneering space and time traveler and, actually more capable space/time machine pilot than you might think, Tegan Jovanka, successfully jettisoned 1/4 of the TARDIS to avoid destruction. Disaster averted! TV series saved!

So, how about using the term jettison?

P.S. I’ve been using PlexDBRepair for about six months now. I use it on a regular basis (about twice monthly) to check and optimize several instances of Plex Media Server (2x Docker containers on 2x Unraid servers – total 4x instances of Plex Media Server). I love PlexDBRepair. Love your work!

  • PMS#1 /PhotoTranscoder 35G
  • PMS#2 /PhotoTranscoder 7G
  • PMS#3 /PhotoTranscoder 67G
  • PMS#4 /PhotoTranscoder 17G

Just found some .png files in there too after browsing with the iOS Plex client.

“Jettison” is grand. Other options (though with no Doctor Who connection):

Evict
Expunge
Defenestrate
Recycle
Tidy
Exsanguinate

:slight_smile:

@SamVen

Get me thinking SciFi and who knows what term will end up being used :rofl:

I could use the term “nuke” :exploding_head:

OOOOOOOO :smiling_imp: I like the sounds of Exsanguinate.
The abbrev would be exsa … a pain to type.

Castrate?
Neuter?
:thinking:

This is getting morbid

How about ‘prune’ ?

Prune old files from the Transcode Cache directory?

Pruning implies being selective which we are.

(This is the most annoying part of development; finding a keyword which makes sense to everyone)

The best list now stands at

  • Prune
  • Purge
  • Remove

“Remove” is the easiest to type when interactive (nearest to homerow and index fingers)

As currently implemented in your script you’re basically forcing the Cache garbage collection that some users report isn’t happening. Thus the most technically accurate description in the tool should be something like “Force PhotoTranscode Cache Garbage Collection”. Since your menu system allows for the option of typing the first several letters, perhaps it could be reworked as “PhotoTranscode Cache - Force Garbage Collection,” thus creating the (probably) unique option “phot” if a user goes that route.

Yes, that makes sense.

What really needs to happen is changing mtime to atime.

We will remove file(s) not accessed within the last N (30) days.

  • If the image file (whatever it is) hasn’t been accessed in 30 days, the likely need of carrying it seems low.
  • This will prevent frequently accessed files, created more than 30 days ago, from being deleted and then recreated again. (Wear & Tear on a SSD (?) )

The descriptive text will state this.
The keyword abbreviation of ‘photo’ is well targetted as it is clear what’s being operated on.

When I am able to save “settings”, I will be able to customize these limits

How about “exfoliate?”

Exfoliate is a good term too.

CLEARLY English is FAR too verbose! :roll_eyes:

If we stick to the functionality, I think we’re better off.

The other terms are DB-centric but their meaning is clear.
This new function, while not DB-centric, must also have a clear meaning keyword.

We are selectively removing files from the photo transcode cache.

In this context, ‘prune’ is the better term.

to cut off or cut back parts of for better shape or more fruitful growth . prune the branches. intransitive verb. : to cut away what is unwanted or superfluous.

with emphasis on: to cut away what is unwanted or superfluous.

When the function completes, we’ll have only those image files which have been accessed within the last N (30) days.

I need attend family now. I’ll contemplate this a while longer before sharing a forum preview for review.

As always, Thank you all for your input!

1 Like

The problem with switching from mtime to atime is that many modern distros now recommend mounting with -noatime (at least if you are using an SSD). Using that as your filter criteria may not work the way you’d hope. I’m unsure of the outcome, but I’d suspect that no files will ever match because atime will be missing altogether.

Precisely what I am going to test before the forum preview.

I only started writing this last night after seeing a few of the horrors in the forum.

“purge” - “Purge files older than 30 days from PhotoTranscode cache”

The term Purge gets my vote.

@ChuckPa Would you consider having two options ?

  1. “Purge files older than 30 days from PhotoTranscode cache”
  2. “Purge all files from PhotoTranscode cache”

While I like the term purge as well.. I have ‘remove’ coded in the menu

BUT it responds to both ‘purge’ or ‘remove’ :wink:

Try it, you’ll see I reference both for now so we can gauge which looks best.

ALSO

I have the other part of this release ready for you all to kick the tires.

  1. ZFS users will like this
  2. Initial support for customizing DBRepair is a reality

Database ‘page_size’ may now be set. (DBREPAIR_PAGESIZE)

  1. Environment variable DBREPAIR_PAGESIZE allows setting the DB’s internal page size.
    – Linux ext4 & xfs users need not bother with this as the filesystem already matches the DB page size (4096)
    – ZFS users can now increase the page size to match their ZFS datasets and recover some of the performance lost to HDDs

  2. Must be a multiple of 1024
    – May not exceed 65536 (SQLite limit)
    – Rounds to the nearest 1024 multiple if you can’t do the math! :stuck_out_tongue: lol

  3. Can be undone by unsetting the environment variable and rerunning “automatic”

Cache age maximum may now be specified (DBREPAIR_CACHEAGE)

  1. Environment variable DBREPAIR_CACHEAGE (export DBREPAIR_CACHEAGE=x)
  2. A simple positive integer (x) measured in days

SEE THE README for details.

PlexDBRepair-EnvVars.tar (100 KB)

My supply of :adhesive_bandage: is limited.
:slight_smile:

2 Likes

Given what I now provide with DBREPAIR_CACHEAGE, if you set a value=1, that would delete anything more than 1 day old.

Will this suffice ?

1 Like