Plex Stuck in Maintenance Mode after update to PlexMediaServer-1.30.2.6563 on QNAP TS-1277

[vmsman]

Server Version#: 1.30.2.6563
Player Version#: N/A
After starting PMS on my QNAP TS-1277 following an upgrade, the local web page reports:

This XML file does not appear to have any style information associated with it. The document tree is shown below.
<Response code="503" title="Maintenance" status="Plex Media Server is currently running startup maintenance tasks."/>

The startup log is as follows:

[/share/CACHEDEV1_DATA/.qpkg/PlexMediaServer/Library/Plex Media Server/Logs] # cat 'Plex Media Server.log'
Jan 11, 2023 15:59:12.595 [0x7f6789048b38] INFO - Plex Media Server v1.30.2.6563-3d4dc0cce - QNAP TS-1277 x86_64 - build: linux-x86_64 qnap - GMT -06:00
Jan 11, 2023 15:59:12.595 [0x7f6789048b38] INFO - Linux version: QTS 5.0.1.2248, language: en-US
Jan 11, 2023 15:59:12.595 [0x7f6789048b38] INFO - Processor: 16-core AMD Ryzen 7 PRO 1700 Eight-Core Processor      
Jan 11, 2023 15:59:12.595 [0x7f6789048b38] INFO - Compiler is - Clang 11.0.1 (https://plex.tv 9b997da8e5b47bdb4a9425b3a3b290be393b4b1f)
Jan 11, 2023 15:59:12.595 [0x7f6789048b38] INFO - ./Plex Media Server
Jan 11, 2023 15:59:12.595 [0x7f67892d4aa0] DEBUG - BPQ: [Idle] -> [Starting]
Jan 11, 2023 15:59:12.595 [0x7f67892d4aa0] DEBUG - FeatureManager: Using cached data for features list
Jan 11, 2023 15:59:12.606 [0x7f67892d4aa0] DEBUG - [CERT] Subject name is /CN=*.ae6e1c50ccd0421597cd46c9b27920d7.plex.direct
Jan 11, 2023 15:59:12.606 [0x7f67892d4aa0] DEBUG - [CERT] Installed certificate with fingerprint 28:69:17:dc:61:5b:20:b3:f4:48:4e:71:9f:ae:c8:f2:62:ed:cc:39.
Jan 11, 2023 15:59:12.606 [0x7f67892d4aa0] DEBUG - [CERT/OCSP] Stapling requests will be made to 'http://r3.o.lencr.org/'.
Jan 11, 2023 15:59:12.607 [0x7f67892d4aa0] INFO - [CERT/OCSP] Successfully retrieved response from cache.
Jan 11, 2023 15:59:12.607 [0x7f67892d4aa0] DEBUG - HttpServer: Listening on IPv6 as well as IPv4.
Jan 11, 2023 15:59:12.607 [0x7f67892d4aa0] ERROR - HttpServer: Error binding acceptor: Address in use
Jan 11, 2023 15:59:12.607 [0x7f67892d4aa0] ERROR - HttpServer: Error opening acceptor on IPv6, falling back to IPv4: Address in use
Jan 11, 2023 15:59:12.607 [0x7f67892d4aa0] ERROR - HttpServer: Error binding acceptor: Address in use
Jan 11, 2023 15:59:12.628 [0x7f67892d4aa0] DEBUG - [JobRunner] Job running: /share/CACHEDEV1_DATA/.qpkg/PlexMediaServer/CrashUploader "--directory=/share/CACHEDEV1_DATA/.qpkg/PlexMediaServer/Library/Plex Media Server/Crash Reports/1.30.2.6563-3d4dc0cce" --version=1.30.2.6563-3d4dc0cce --platform=Linux "--platformVersion=QTS 5.0.1.2248" --serverUuid=xxxxxxxxxxxxxxxxxxxxxxx --userId=xxx@gmail.com --sentryUrl=https://o17675.ingest.sentry.io/api/1233455/minidump/ --sentryKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx --vendor=QNAP --model=x86_64 --device=TS-1277
Jan 11, 2023 15:59:12.628 [0x7f67892d4aa0] DEBUG - [JobRunner] Jobs: Starting child process with pid 12008

I have been unable to find out how to get it out of maintenance mode.
I have no GUI Web interface per the 503 message on the local IP address.

The database checks out fine.

'/share/CACHEDEV1_DATA/.qpkg/PlexMediaServer/Plex SQLite' com.plexapp.plugins.library.db  

sqlite> PRAGMA integrity_check;
ok
sqlite> .exit

Any ideas?

[ChuckPa]

Give it time. It’s a BIG schema update.

Each time you restart it , you risk corrupting it.

[vmsman]

Gee in seven years of running PMS, this is the first time I have seen this. I have restarted maybe 10 times. My media is around 15,000 TV episodes, 2500 movies, 3000 home videos, 9000 music tracks, and maybe 20,000 personal pictures in around 15TB at this point. I am rebooting the NAS because since the database appeared fine, I didn’t know what else to do. I will let it sit. Any ideas how long that this schema rebuild will take?

[ChuckPa]

How big is the DB?

I’ve had some users tell me they needed hours.

I’m surprised this is the first you’ve seen it.

Engineering has been updating the schema and showing “503 Maintenance” records for a few releases now.

Given the mount of media you have, might it be time to fully tune up the DB ?
(I have a tool which does the job if you’re interested – at the shell command line)

[vmsman]

@ChuckPa So, my sizes are listed above. I ended up rebooting the QNAP TS-1277 and the PMS came up and is online. Not sure what happened, but all my hours of examining the database and reading log files yielded nothing. Heck, the QNAP is running Linux and not Windows… Anyway, thanks so much for your suggestions and a reboot brought everything online. To make it current, I just initiated a library scan and the new files popped right in. The obvious question is, if the web interface does not come up as in my case, is there a CLI command set to determine what exactly is going on with PMS? I host a YouTune channel @ScottiBYTE and if there are some tools, that might make a good future topic if you could provide insight.

[ChuckPa]

I see the quantity of media you have.

My question was targeted to asking how large the actual files are.

GitHub - ChuckPa/PlexDBRepair: Database repair utility for Plex Media Server...

Just a bunch of shell code to help folks out.
So far, it’s been really helpful.

It’s fixes things as well as does some serious performance cleaning / speed ups

[vmsman]

[ChuckPa]

They aren’t too big but that WAL file is really very unacceptable.

WAL contains all the pending database transactions.
It should never be bigger than the main DB

I would restart PMS one more time and verify the WAL gets folded back into the main DB as it should be.

[vmsman]

@ChuckPa I did another restart on the Plex. It came up and looks good. Here are the sizes now. Does this look better?

[ChuckPa]

That’s much better. Thank you.

[vmsman]

Thank you sire. You were most helpful. Have a great day.