Server crashes every few days

Server Version#: 1.28.0.5999
Player Version#: Version 1.49.1.3146-73559c78

I have recently moved my plex server installation from one server to another. The installation on the old server was running without any downtime unless I installed an update or rebooted the machine, howerver on the new system the server crashes about once per week.

These are the last few lines from the log when the crash happens:

Aug 04, 2022 22:57:00.395 [0x7f5963037b38] WARN - [Req#170a1] QueryParser: Invalid field 'contentDirectoryID' found, ignoring.
Aug 04, 2022 22:57:00.415 [0x7f596234bb38] WARN - [Req#170a5] QueryParser: Invalid field 'sectionID' found, ignoring.
Aug 04, 2022 22:57:00.415 [0x7f596234bb38] WARN - [Req#170a5] QueryParser: Invalid field 'contentDirectoryID' found, ignoring.
Aug 04, 2022 22:57:00.546 [0x7f5962391b38] ERROR - [Req#170aa] Unknown metadata type: folder
Aug 04, 2022 22:57:00.546 [0x7f5962391b38] WARN - [Req#170aa] QueryParser: Invalid field 'contentDirectoryID' found, ignoring.
Aug 04, 2022 22:57:00.651 [0x7f59628f5b38] ERROR - [Req#170b1] Unknown metadata type: folder
Aug 04, 2022 22:57:00.652 [0x7f59628f5b38] WARN - [Req#170b1] QueryParser: Invalid field 'contentDirectoryID' found, ignoring.
Aug 04, 2022 22:59:36.787 [0x7f59627b4b38] WARN - NAT: PMP, got an error: Version not supported.
Aug 04, 2022 23:59:37.246 [0x7f5962bc6b38] WARN - NAT: PMP, got an error: Version not supported.
Aug 05, 2022 00:07:00.833 [0x7f5965330b38] ERROR - [EventSourceClient/pubsub/139.162.158.105:443] Retrying in 15 seconds.
Aug 05, 2022 00:07:12.780 [0x7f5966e72b38] WARN - JobManager: Could not find job for handle 3698322
Aug 05, 2022 00:07:12.786 [0x7f5966e72b38] WARN - JobManager: Could not find job for handle 3698324
Aug 05, 2022 00:07:12.790 [0x7f5966e72b38] WARN - JobManager: Could not find job for handle 3698326
Aug 05, 2022 00:07:12.801 [0x7f5966e72b38] WARN - JobManager: Could not find job for handle 3698230
Aug 05, 2022 00:07:12.850 [0x7f596b855380] INFO - Killing process: Plex EAE Service (pid: 3714457)

After that systemd tries to restart the service, but fails. When I login to that machine I can just start the service without any issue. Can someone tell me what the entries in the log mean and what I can do to identify this issue?

Same thing here since months for Unraid, PMS dockers after version 1.24.5.5173 are all plagued by the same bug : server crash after several hours/days…

I’ve tried on two differents servers from Unraid version 6.8.3 to 6.10.3 (configs were B450, Ryzen 2700x and Nvidia Quadro P2200, 32Gb ram And X470 / Ryzen 3900 / Quadro P2200 / 64Gb ram) and with every PMS versions after 1.24.5.5173 I’ve experienced random crashs, sometime crashing the entire Unraid Server.

Perhaps hardware transcoding or something else is brocken with Ryzen CPUs or Quadro cards?

Now i must stick to version 1.24.5.5173 if i want a stable and working version of Plex…

It’s a bit shameful that they still dare call this “updates” when it’s broken since months…

Hi

You have debug logging disabled

Please enable Debug Logging - see Reporting issues with Plex Media Server | Plex Support

and restart the server to get new set of log files created

Then after a crash, restart the server and immediately download the server logs and attach the zip file here

Thanks

Plex Media Server Logs_2022-10-03_06-35-27.zip (664.5 KB)

I have attached the log. Sorry, took a while. Error had appeared a few times since my original post.

@Frostelikus

Debug logging still not enabled. See the document sa2000 linked.

Next try, sorry.
Plex Media Server Logs_2022-10-12_12-39-29.zip (4.0 MB)

I’m running into the same issue. I recently moved from an intel Mac Mini server to an M1-based Mac Studio server. I’m getting a random crash every day or so. I can’t seem to pinpoint what’s going on from the logs, as there’s not a lot of consistency in the errors. My crashdumps are also always empty, so can’t see anything there either. I had no downtime on the server prior to this transition a few weeks ago. I’ve also tried clearing out plugin-caches, etc. to see if it was a cache issue from the old machine.

I had debugging enabled already and included my log dump after a restart.

Plex Media Server Logs_2022-10-12_22-05-19.zip (4.6 MB)

Thanks for the logs

I do not see a crash. What shows in the logs is that the process is being killed.

The logs show this happening at similar time two days in succession

Oct 11, 2022 00:40:12.679 [0x7f26d7477b00] DEBUG - Shutting down with signal 15 (Terminated)

and the last one

Oct 12, 2022 00:38:58.862 [0x7f7a6910eb00] DEBUG - Shutting down with signal 15 (Terminated)

So if it was not you shutting it down then some script may be doing it
The server was operational up to the point of the SIGTERM signal

Oct 12, 2022 00:37:25.705 [0x7f7a619b4b00] DEBUG - Request: [127.0.0.1:39728 (Loopback)] GET / (4 live) #24822 Signed-in Token (Frostelikus)
Oct 12, 2022 00:37:25.706 [0x7f7a68eccb00] DEBUG - Completed: [127.0.0.1:39728] 200 GET / (4 live) 1ms 8299 bytes (pipelined: 1)
Oct 12, 2022 00:37:56.858 [0x7f7a619b4b00] DEBUG - Request: [127.0.0.1:39730 (Loopback)] GET / (4 live) #24825 Signed-in Token (Frostelikus)
Oct 12, 2022 00:37:56.860 [0x7f7a68eccb00] DEBUG - Completed: [127.0.0.1:39730] 200 GET / (4 live) 1ms 8299 bytes (pipelined: 1)
Oct 12, 2022 00:38:28.009 [0x7f7a619b4b00] DEBUG - Request: [127.0.0.1:39732 (Loopback)] GET / (4 live) #24828 Signed-in Token (Frostelikus)
Oct 12, 2022 00:38:28.010 [0x7f7a68cc9b00] DEBUG - Completed: [127.0.0.1:39732] 200 GET / (4 live) 1ms 8299 bytes (pipelined: 1)

These look like the periodic healthcheck requests I normally see for a docker environment

Is this a hosted web server? I do see references to fritz boxes

If it is a hosted system, please ask the providers as to what may be killing the process and why

I see you are using IPV6 for remote connections - appears to be working

There is no indication of a crash other than the Plex Media Server process disappearing at Oct 12, 2022 12:38:36
Was that the time of the problem ?

Are there any crash logs captured by Mac OS for around that time?
Mac OS captured diagnostics would be in

~/Library/Logs/DiagnosticReports/

This would be
/Users/server/Library/Logs/DiagnosticReports for your machine

Thanks for taking a look! In the logs shared, it had stopped at 12:38. There’s nothing I’m aware of in Mac OS that would cause this, besides a Plex crash. This is basically a fresh OS install with just Plex and a few other apps running.

I’ve actually been checking the diagnostic logs as well and haven’t seen anything yet around the time Plex is crashing. I originally thought this might be related to the IMDB scanner, as it seems to throw some error, but that doesn’t always happen right when the Plex process stops. I’ll keep watching for future crashes and share any logs that might be helpful here.

I actually just attempted to check the settings on the server and now I’m getting Server Settings are Unavailable. on the Plex server settings. I’m assuming this is a pre-symptom of the server eventually crashing. Nothing in the logs stand out to me and I have attached the main server log here.

Plex Media Server.log.zip (595.1 KB)

could be - but the clues to the problem were in the other logs
For example in the last crash at 12:38 - it was the Plex Tuner Service.1.log that had the clue

Oct 12, 2022 12:38:36.625 [0x100a28580] DEBUG - Parent process no longer present; exiting

So to get the whole picture, should always include zip of all log files

@Robert_Alexander Could you look at the Mac OS system log and reports there for any related log entries around these times:

12:38 on Oct 12
20:29 on Oct 13

To view your Mac system logs, launch the Console app. You can launch it with Spotlight search by pressing Command+Space, typing “Console,” and then pressing Enter. You’ll also find it at Finder > Applications > Utilities > Console.

There are buttons for System Reports and User Reports which you could check out. Also there is a button for system.log

It is not running as a docker container. I have installed the apt package from the official repository.

No, it is a small physical server running at home fritz box is the DSL router I use.

I do not have a publicly accessible IPv4 address, but only a v6 address. The v6 address changes from time to time, so I have a script running to check if the address has been changed. If it has changed, it stops the Plex server, updates the customConnections part of the plex config, and then restarts the server. I guess the restart is not working fine. However, the config update seems to be working as all I have to do is start plex to fix the problem? Any idea what may cause this behavior?

Here is the script for reference:

#!/bin/bash
# Retrieve host IPv6 address and update Plex custom access url accordingly
# based on workaround posted by Pikey18 on the Plex subreddit: https://www.reddit.com/r/PleX/comments/b82opu/plex_remote_access_over_ipv6/

# IPv6 address expansion (https://stackoverflow.com/a/50208987)
# helper to convert hex to dec (portable version)
hex2dec(){
    [ "$1" != "" ] && printf "%d" "$(( 0x$1 ))"
}

# expand an ipv6 address
expand_ipv6() {
    ip=$1

    # prepend 0 if we start with :
    echo $ip | grep -qs "^:" && ip="0${ip}"

    # expand ::
    if echo $ip | grep -qs "::"; then
        colons=$(echo $ip | sed 's/[^:]//g')
        missing=$(echo ":::::::::" | sed "s/$colons//")
        expanded=$(echo $missing | sed 's/:/:0/g')
        ip=$(echo $ip | sed "s/::/$expanded/")
    fi

    blocks=$(echo $ip | grep -o "[0-9a-f]\+")
    set $blocks

    printf "%04x:%04x:%04x:%04x:%04x:%04x:%04x:%04x\n" \
        $(hex2dec $1) \
        $(hex2dec $2) \
        $(hex2dec $3) \
        $(hex2dec $4) \
        $(hex2dec $5) \
        $(hex2dec $6) \
        $(hex2dec $7) \
        $(hex2dec $8)
}

# Get IPv6 address of given interface (command adapted from: https://superuser.com/a/1057290)
IPv6=`/sbin/ip -6 addr show dev "$1" scope global | grep inet6 | awk -F '[ \t]+|/' '{print $3}'`
if [ -n "$IPv6" ]; then
	for ip in $IPv6
	do
		echo "Got IPv6 address: $ip"
		if [[ $ip == 2a* ]]
		then
        		echo "Use this IP to update the config"
        		# Format IPv6 for Plex (replace : with -)
        		PlexFormatIPv6=`expand_ipv6 "$ip" | sed -e "s/:/-/g"`
        		if ! grep -q "customConnections\=\"https\:\/\/$PlexFormatIPv6" "$2" ; then
                		echo "Current IPv6 does not match config, updating config"
                                systemctl stop plexmediaserver # systemd Linux distributions (Ubuntu, Debian, ...)
                                sleep 10
                		# Replace old IPv6 with new one
		               sed -i -e "s/customConnections\=\"https\:\/\/[a-fA-F0-9\-]*/customConnections\=\"https:\/\/$PlexFormatIPv6/" "$2"
                		# Restart Plex service, uncomment line for your server's os or add your own
                		systemctl start plexmediaserver # systemd Linux distributions (Ubuntu, Debian, ...)
                		# synoservice --restart pkgctl-Plex\ Media\ Server # Synology DiskStations (details: https://tech.setepontos.com/2018/03/25/control-synology-dsm-services-via-terminal-ssh/)
        		else
                		echo "Current IPv6 matches config, exiting"
		        fi
		else
			echo "Skipping IP"
		fi
	done
	exit 0
else
	echo "No IPv6 address found, exiting"
	exit 1
fi

So that was probably what happened at Oct 11, 2022 00:40:12.679 and Oct 12, 2022 00:38:58.862

Sorry I have not looked at your script - but does it log when it does the shutdown

Looks like it

Oct 11 00:40:12 blackbox systemd[1]: Starting Plex IP updater...
Oct 11 00:40:12 blackbox bash[2642870]: Got IPv6 address: 2a03:....
Oct 11 00:40:12 blackbox bash[2642870]: Use this IP to update the config
Oct 11 00:40:12 blackbox bash[2642870]: Current IPv6 does not match config, updating config
Oct 12 00:38:58 blackbox systemd[1]: Starting Plex IP updater...
Oct 12 00:38:58 blackbox bash[2799900]: Got IPv6 address: 2a03:...
Oct 12 00:38:58 blackbox bash[2799900]: Use this IP to update the config
Oct 12 00:38:58 blackbox bash[2799900]: Current IPv6 does not match config, updating config

Any idea why calling systemctl start plexmediaserver shortly after the new IP fails in the script, but it works fine a few hours later? Should plex start successfully if network connection is not fully functional when it is started?

The shutdown is not immediate - all child processes are asked to shutdown and then the database is shutdown

Oct 11, 2022 00:40:12.679 [0x7f26d7477b00] DEBUG - Shutting down with signal 15 (Terminated)
. . .
Oct 11, 2022 00:40:12.900 [0x7f26db42a1c0] DEBUG - Database: Shutting down.

and

Oct 12, 2022 00:38:58.862 [0x7f7a6910eb00] DEBUG - Shutting down with signal 15 (Terminated)
. . .
Oct 12, 2022 00:38:59.103 [0x7f7a6d0c11c0] DEBUG - Database: Shutting down.

and I am not sure how much time elapsed after logging “Database: Shutting” before the process exits

I would suggest having a delay after the shutdown - eg 5-10 second delay
You could also add a check with ps to see that the plex media server process has disappeared and wait in a loop with suspends until the process disappears

Then update the config

and after that completes, restart the service

There was already a 10 second delay between the stop command and the update of the config and restart, but I have just increased it to 2 minutes. I’ll check if that will fix the problem. Is /usr/lib/plexmediaserver/Plex Media Server the main binary on ubuntu that I should wait for before updating the config?

I’ve been digging into this a bit more and don’t see much in any system logs/reports whenever there is a crash. Earlier this week there was a crash at 14:06 on 10/18. I attached the logs here again in case anything sticks out, but I don’t see anything. It’s pretty odd given that the computer is a fresh install only running Plex and Tautulli. Maybe I’ll kill Tautulli and see if that’s related somehow.
Plex Media Server Logs_2022-10-22_14-59-43.zip (4.4 MB)

Could you look into $TMPDIR for copies of crash reports

In terminal session, find the path for $TMPDIR eg by typing
echo $TMPDIR
or open it directly by
open $TMPDIR

and can you look for files with these names

D99D7C09-76B0-4B91-A9C4-3A9C5EA9FC7E
CC18AF3C-F5CD-4D09-BFAD-6F98971440CE
CBD7C3B3-2ED3-4B8A-8A6B-37BE395DD48B
CBD7C3B3-2ED3-4B8A-8A6B-37BE395DD48B
88E9AFF9-16FE-4562-81EF-573BB3CE44B8
95051411-FDC2-436F-9EB8-2211091091DC

Please zip what you find and attach here

Thanks