Server not accessible - Error 503

Plex Media Server
3

The 503 errors are consequence of an earlier issue caused by some deadlock within Plex Media Server leading to loads of requests getting queued up and eventually reaching thousands and exhausting the queue which would result in 503 errors being returned.

This is the first indication I have of new deadlocks - we have had some months ago

When the 503 errors arise the Plex Media Server does not crash or close - so this forum topic is not representative of the problem and I will move your post to a new forum topic.

Deadlocks can only be investigated from process dumps coupled with debug server logs and list of connections obtained before forcing the process dump. See details below:

Diagnostics for a deadlock / lockout and subsequent Error 503s

  1. Prepare yourself by finding the server security token string as it will be needed for the connections list request. To find the token string, see this support article Finding an authentication token / X-Plex-Token | Plex Support

  2. Ensure the server debug logging is enabled in advanced server settings.

  3. It may be necessary to increase the number of log files from the default. See Adavnced Server Setting “LogNumFiles” in this support article. Suggest setting it to 10. Advanced, Hidden Server Settings | Plex Support

  4. Restart the server to get fresh log files created - and wait for time when the issue arises

  5. When the problem occurs (and please wait for sufficient time before capturing diagnostics or restarting eg 10 minutes of requests to the server not getting through) , get the output from a specific browser request on the server. In a browser, go to this url

http://localhost:32400/connections?X-Plex-Token=xxxxxxxx

where you put the server token string instead of the xxxxxxx. Copy the displayed output to a text file and save

  1. Next is to force a process dump for the Plex Media Server.exe process. We can do this with Windows Task Manager. Find the Plex Media Server.exe process in Task Manager, select it and right click on it and choose Create Dump

  2. Check that the dmp file has been created

  3. Restart the server and capture the zipped server logs

  4. Zip the dmp file and upload to dropbox or google drive etc

  5. send me by private message link to the zipped dmp files and send the saved /connections output text files and the zipped server logs

Variation to steps for Linux / NAS platforms

  1. If unable to get the /connections request directly on the server using localhost, give the local IP of the server

http://IP-OF-SERVER:32400/connections?X-Plex-Token=xxxxxxxx

As example: http://192.168.0.82:32400/connections?X-Plex-Token=xxxxxxxx if the server IP is 192.168.0.82

  1. To force a dump on Linux - you need to list the process identifiers using ps command in a command line session. Then get a process dump through the command line in Linux
    kill -SEGV <pid>
    where <pid> is the PID number for the plexmediaserver process

  2. The Plex Media Server will create a minidump and attempt to upload the crash report. Look for a copy of the minidump in the directory where environment variable $TMPDIR refers to for the plex media server process. The dump file would have a filename that is a string for random characters with dash separators = example:
    b2bc29d9-1c2b-4d13-6de103a6-23dd90b8

The rest of the steps would be the same as above