[Ubuntu 16.04 LTS] Plex crashing often, randomly becomes indirect connection that never loads?

server-linux

#1

Hey! Thanks for checking my question out.

So basically my Plex Media Server keeps hanging pretty consistently every few hours. I've got logs from the crashes, and it usually seems to be something to do with Sync. Weirdly rather then the server just going offline, plex.tv reports an indirect connection - except it never loads.

If anyone can help or tell me what's going wrong, I'd be really grateful. The server runs on Ubuntu 16.04 LTS.

Restarting the service also takes a minute to react. I rebuilt the database (very big database, over 65,000 items) recently on another machine and the server ran fine without crashes for like, 3 weeks on end. That leads me to believe the database isn't corrupt or anything, so I'm totally stumped as to why it's hanging/crashing so often :(


#2

I see you’re running 1.11.2. 1.11.2 is beta.

Do you have this same issue with 1.10.1 ?

Also, when you upload logs, would you please use Settings - Server - Help - Download Logs and submit that ZIP ?


#3

@ChuckPA said:
I see you’re running 1.11.2. 1.11.2 is beta.

Do you have this same issue with 1.10.1 ?

Also, when you upload logs, would you please use Settings - Server - Help - Download Logs and submit that ZIP ?

I had pretty consistent crashes since around December. It’s why I rebuilt the database, just in case. So I guess the answer is probably yes, but I can’t say for definite that it crashed in the same way. That indirect connection thing is seemingly pretty new, before the server would just drop off and say it was offline on plex.tv (albeit it was still hanging).

Here’s my logs as requested, sorry about uploading in the wrong format. I’ve restarted the server since so the original log file I uploaded is now Plex Media Server.2.log instead.


#4

Crashed again, caught it with debug logging on too.
Crash is logged at the end of Plex Media Server.2.log

Crashed after ~20 hours of uptime since last crash.


#5

Your database is locked.

locked
Feb 11, 2018 23:04:51.408 [0x7ff3bb7ff700] ERROR - Failed to begin transaction (../Library/MetadataCollection.cpp:169) (tries=8): Cannot begin transaction. database is locked
Feb 11, 2018 23:04:52.066 [0x7ff3bb7ff700] ERROR - Failed to begin transaction (../Library/MetadataCollection.cpp:169) (tries=9): Cannot begin transaction. database is locked
Feb 11, 2018 23:04:52.382 [0x7ff3bb7ff700] ERROR - Failed to begin transaction (../Library/MetadataCollection.cpp:169) (tries=10): Cannot begin transaction. database is locked

And badly fragmented.

Feb 11, 2018 23:06:15.213 [0x7ff3d27ff700] WARN - Held transaction for too long (../Sync/SyncItemGenerator.cpp:142): 0.390000 seconds
Feb 11, 2018 23:06:26.265 [0x7ff3dafff700] WARN - SLOW QUERY: It took 270.000000 ms to retrieve 50 items.

I can’t see the reason for it but would like you to

  1. Manually Optimize Database (under the ellipsis in Libraries)
  2. If that fails to resolve it, I will need to get additional help.

#6

@ChuckPA said:
Your database is locked.
And badly fragmented.

I can’t see the reason for it but would like you to

  1. Manually Optimize Database (under the ellipsis in Libraries)
  2. If that fails to resolve it, I will need to get additional help.

Done! I’ve actually already optimised the database manually quite a few times and still had these crashes, but I’m gonna follow your steps exactly because I’m at the point of abandoning Plex because it’s that bad :frowning: Really grateful for your support though! I’ll get back to you with the logs of the next crash :slight_smile:


#7

It has once again crashed :frowning: I don’t think I caught it in the logs (only noticed an hour later, have a script to auto restart the server because it’s happening so often and I’m that sick of it). I’ve included the logs, albeit like aforementioned I don’t think the crash is caught in them.

I’m assuming it crashed like usual, this is after I’ve manually optimised the database. So, what do we do now @ChuckPA ? I don’t want to give up on Plex :confused:


#8

You’re right, it didn’t capture the crash.

Also, beta 1.11.2 has a rather ‘chatty’ log activity. it it probably the reason your crash isn’t in the logs.
We received the next update from Engineering and are testing it now. From what we are told, a lot of your errors and the chatty entry have been taken care of.

Until that time, I’d like to ask you to add LogNumFiles="10" to your Preferences.xml (PMS stopped of course). This increases retention from the default 5 to 10. (adds 20 MB on disk and about 4MB when ZIP’d for upload).

It will give you and us a bigger window to look at.

If you want to make it bigger, that’s fine. I wouldn’t go above 20. 10 usually works best, 15 in tough cases.

If you don’t mind my asking, what versions of PMS do you have saved? 1.10.0 ? 1.9.7?


#9

@ChuckPA
Until that time, I’d like to ask you to add
LogNumFiles=“10” to your Preferences.xml (PMS
stopped of course).

I’ve increased the LogNumFile to 13 and added it to my Preferences.xml - hopefully this’ll give me enough overhead to catch the crash in the logs with the auto restart script.

I actually had to develop a script that sends API requests to the server and gives it like 4 minutes to respond and if it doesn’t, it’ll send a restart request to the service. The service basically just hangs and doesn’t notify the system that it’s crashed, so this is the only way to catch the crash.

If you don’t mind my asking, what versions of PMS do you have saved? 1.10.0 ? 1.9.7?

I’ve ran basically every build and beta build since like, September, but I currently only have access to
plexmediaserver_1.11.0.4633-70f9c31ed_amd64
plexmediaserver_1.11.0.4666-fc63598ba_amd64
plexmediaserver_1.11.1.4760-58bcb2957_amd64

When I rebuilt the database (about halfway through January, maybe later) the server was running

plexmediaserver_1.11.0.4666-fc63598ba_amd64

Would updating to the absolute latest beta version solve any of the aforementioned crashes? Is that what the new build from the engineers is? Or is an internal only build you discussed


#10

Are you using curl to query it with a timeout? We’ve found curl to be great in ‘pinging’ the local server. It’s also the perfect tool to issue commands when the browser doesn’t respond because we’re performing a POST operation instead of a GET.


#11

@ChuckPA said:
Are you using curl to query it with a timeout? We’ve found curl to be great in ‘pinging’ the local server. It’s also the perfect tool to issue commands when the browser doesn’t respond because we’re performing a POST operation instead of a GET.

I am using curl :slight_smile: though I am actually using it with a GET as opposed to a POST (Though I do see how a POST might be better!) to the PMS API endpoint entitled /status/sessions and determining the HTTP response code returned, if I get a response back (or no response) with any other status code then 200 I log to a file the time, send the restart service command, then time how long it took the Plex service to restart (because this ends up taking nearly a minute, seems bad).

I seemingly cannot post the script but if you’re interested in it, I’ll put it on github :slight_smile:

Hopefully we can just solve the actual crash though so I can get rid of the need for a cron running every 5 minutes lol.


#12

I’m using the Engineering update which is going through QA now. Looking good on a lot of fronts. I’ve been too busy to drive it hard (unfortunately) due to “those pesky forum users” LOL


#13

you can include it.

use 3 grave characters on a line before it and after it. It will look like:

#!/bin/sh

echo  Hello World.

#14
#!/bin/bash

scripts_dir="/var/lib/plexmediaserver/scripts"
PIDFILE="${scripts_dir}/PlexUpCheck.pid"
LOGFILE="${scripts_dir}/services.log"
PLEX_TOKEN=""
PLEX_VERSION="1.11.2.4772-3e88ad3ba"
PLEX_PID=$(cat "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/plexmediaserver.pid")

if [ -f $PIDFILE ]
  then
     PID=$(cat $PIDFILE)
     ps -p "$PID" > /dev/null 2>&1
     if [ $? -eq 0 ]
         then
             echo "Process already running"
             exit 1
         else
             ## Process not found assume not running
             echo $$ > $PIDFILE
             if [ $? -ne 0 ]
                 then
                     echo "Could not create PID file"
                     exit 1
                 fi
         fi
  else
      echo $$ > $PIDFILE
      if [ $? -ne 0 ]
          then
              echo "Could not create PID file" 
              exit 1
          fi
fi

http_status=$(curl -LIX GET 127.0.0.1:32400/status/sessions -o /dev/null -w '%{http_code}
' -s -H "X-Plex-Token:${PLEX_TOKEN}" --max-time 90)
if  ! [[ $http_status = 200 ]]; then
    printf "$(date "+%d.%m.%Y %T") [Plex] ERROR: 	 Plex crash. Restarting. 
" | tee -a $LOGFILE
    kill -SEGV "$PLEX_PID"
    #zip -jqr "${scripts_dir}/Checks/Logs/Plex Media Server Crash Dump_$(date "+%Y-%m-%d_%H-%M-%S").zip" /var/lib/plexmediaserver/Library/Application\ Support/Plex\ Media\ Server/Crash\ Reports/$PLEX_VERSION
    printf "$(date "+%d.%m.%Y %T") [Plex] INFO: 	 Plex restarted. 
" | tee -a $LOGFILE
    sleep 15s
    wget -qO "${scripts_dir}/Checks/Logs/Plex Media Server Logs_$(date "+%Y-%m-%d_%H-%M-%S").zip" "https://127.0.0.1:32400/diagnostics/logs?X-Plex-Token=${PLEX_TOKEN}" --no-check-certificate
fi

rm $PIDFILE

Script to automatically reboot Plex Media Server on hang (run using cron every 5 minutes). Automatically downloads server logs and stores them in a folder named Logs, perfect for uploading to forums for tech support.

Still on the lookout for another crash to capture for you guys :slight_smile:

Edit 2: Added server log collection to script.
Edit 3: Fixed kill -SEGV being pushed file rather then contents of file.


#15

Whilst I wait for a crash, for your entertainment, here’s every crash I’ve had in the last month. Seems like they happen every day or two, then I’ll get a good period of like, 6 days, then it goes back to crashing. It’s pretty impressive it’s this unstable. I personally thought that rebuilding the database would fix it. Did it in the nicest way possible, adding a small selection, optimise, add more, optimise over like 2 weeks - yet it’s still crashing like normal.

Whilst obviously the log here provides no evidence of the problem, atleast it’ll give you an idea of how extended it is. I waited a month of hoping it’d go away till I wrote the script in my above post (what generated the attached log). I hope the engineers can take a look or someone can, or that the new internally testing patch can fix it. Don’t wanna abandon Plex. :frowning:


#16

thanks but that contains nothing actionable on my end.


#17

Script to automatically reboot Plex Media Server on hang (run using cron every 5 minutes). Automatically downloads server logs and stores them in a folder named Logs, perfect for uploading to forums for tech support.

Still on the lookout for another crash to capture for you guys :slight_smile:

Edit 2: Added server log collection to script.
Nice script. I did the same thing with monit. I was helping debug a hang on my server and found that you have to do a kill -SEGV pid in order to create the dump file that plex can use to debug when the server hangs and does not crash and restart on its own. You also have to make sure the crash reporter is turned on in the settings of the server.

p.s I had the same frustration and it was a bug with the trancoder hanging. It was just fixed in 1.11.1.4760-58bcb2957


#18

@WatchTowerPlex said:
Nice script. I did the same thing with monit. I was helping debug a hang on my server and found that you have to do a kill -SEGV pid in order to create the dump file that plex can use to debug when the server hangs and does not crash and restart on its own. You also have to make sure the crash reporter is turned on in the settings of the server.

p.s I had the same frustration and it was a bug with the trancoder hanging. It was just fixed in 1.11.1.4760-58bcb2957

I’ve changed it to now use kill -SEGV PID, reading the PID from the PID file in the PMS application folder. Where will the data be dumped if this is running from a crontab calling the script?

Also sadly I’m still experiencing issues, but it sounds like the database is the issue, not the transcoder - as mine wasn’t fixed by any recent updates. Hopefully it’ll be sorted soon though!


#19

It will be dumped in the crash folder. From my experience it will be empty if the server successfully uploads the crash after you restart it. You will see it upload the crash in the logs after you restart. You may want to copy it out before you start the service.

Crash folders:
https://support.plex.tv/articles/201455336-crash-logs-plex-media-server/


#20

Would like to +1 this thread, same issue to a tee. I’m traveling now but will upload my logs when I can. Thanks for the script, atleast it helps as a workaround.

I also didn’t have this issue on kernel 4.4 but am on 4.13 and this is happening, not sure if that helps at all.