Random Crashing > Centos 7

Hey,

I have attached the logs but my Plex server keeps randomly freezing/crashing and I have to manually restart the service.

I am using Centos 7.

I can’t seem to figure out why from looking in the logs but I am sure what ever it is, is staring me in the face and I can’t see it.

Plex Media Server Logs_2018-11-12_17-11-44.zip (6.1 MB)

Thanks for the logs. Reviewing them does seem to indicate a known deadlock problem engineering is working on.

I’m not 100% certain this is the same issue so will ask the team member specializing in these to take a look.

1 Like

It just locked up again after about an hour and half after the last restart.

Should I try upgrading to Version 1.14.0.5465 and see if it still occurs?

Thanks!

It does appear to have deadlocked around 17:07 - but the load on the server was running high for quite a while and there may have been some hung requests from before that time

Deadlocks need specific diagnostics - getting list of connections and process dump as well as logs. I will give the details below

There has not been a recent deadlock fix released - however i think it is important to be running on the latest version available and i would recommend updating to 1.14.0.5465

One element of the diagnostics will be to force a crash report. I am concerned that your Plex Crash Uploader logs show some uncompleted crash reporting eg

Nov 10, 2018 11:09:22.414 [0x7ff93f90a780] DEBUG - HTTP requesting POST https://crashreport.plexapp.com

with no log entry to show that it completed the request to upload. So suggest testing out producing a process dump by doing a kill -SEGV <pid> where <pid> is the Process ID of the Plex Media Server process. That should crash out Plex Media Server and a crash report should be generated which may get uploaded on next restart. So what would be needed here is to test this out and after the restart check the Plex Crash Uploader.log and Plex Crash Uploader.1.log and see if the upload was successful

Diagnostics for Deadlocks on NAS/Linux

  • Ensure debug logging remains enabled. (Settings / Server / General / Show Advanced)
  • Ensure crash reporting is enabled. (Settings / Server / General / Show Advanced)

When you get the lockout (and please wait at least 10 minutes trying to access the server, before deciding there is a deadlock), do the following

in a browser get this request to this server
http://<local IP of PMS>:32400/connections?X-Plex-Token=xxxxxxxxxxxxxxxxx
Substitute the server token for the xxxxxxxxxxxxxxxxxxx
See this article for how to find the token
https://support.plex.tv/articles/204059436-finding-an-authentication-token-x-plex-token/
(This assumes your server remains on IP address 10.10.10.11)

Copy the response to a text file and send it to me by private message referencing this forum topic

Then get a process dump through the command line in Linux
kill -SEGV <pid>
where <pid> is the PID number for the plexmediaserver process

The crash report should get uploaded when you restart the server

Wait for few minutes and then get the logs zip and attach here. The crash uploader logs would give the information I need to locate the dump. The other logs would be needed for the investigation of the deadlock

[I just noticed your local IP is a WAN IP and there is no local private network range IP - so is this web hosted server? Remember never to disable remote access as that would stop access to the server)]

Yes the server is hosted in a Datacenter I can use ssh ip r -L 8888:localhost:32400 to access anything if needed.

Just to confirm you wish for me to do kill -SEGV PID when the server Deadlocks next time? Or now?

i want you to do it now as test run to see that it all works - because your crash uploader logs show attempts to upload but no completion of uploads. You could also test out the /connections request with the token - so that when the deadlock occurs - getting what is needed is known to work

[root@pepe TVShows]# kill -SEGV 24949
[root@pepe TVShows]# service plexmediaserver status
● plexmediaserver.service - Plex Media Server for Linux
Loaded: loaded (/usr/lib/systemd/system/plexmediaserver.service; enabled; vendor preset: disabled)
Active: deactivating (stop-sigterm) (Result: signal) since Mon 2018-11-12 19:07:11 EST; 7s ago
Process: 24949 ExecStart=/bin/sh -c LD_LIBRARY_PATH=/usr/lib/plexmediaserver “/usr/lib/plexmediaserver/Plex Media Server” (code=killed, signal=SEGV)
Process: 24945 ExecStartPre=/bin/sh -c /usr/bin/test -d “${PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR}” || /bin/mkdir -p “${PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR}” (code=exited, status=0/SUCCESS)
Main PID: 24949 (code=killed, signal=SEGV)
CGroup: /system.slice/plexmediaserver.service

I ran that and it looks like it stopped at:

Nov 12, 2018 19:07:11.927 [0x7f2c7a72a780] DEBUG - HTTP requesting POST https://crashreport.plexapp.com

I will try the /connections request here in a little bit I gotta run to a late night meeting.

That is not good. Any blocks / firewall that may be stopping this ?

There should be other log entries like

DEBUG - HTTP 200 response from POST https://crashreport.plexapp.com
and
INFO - Successfully reported xxxxxxxxxxxxxxxxxxxxx

if it persists, could you repeat and try and capture packets in wireshark to see what happens

I actually just sent you a PM as you replied to the thread. I don’t have anything which would prevent it from sending. But I will look into wireshark.

The deadlocks you had could be the caused by the bug that has just been fixed in beta 1.14.0.,5468

So suggest you update to that version and let me know if the deadlocks / hangs stop

See Release Note Plex Media Server

  • (Transcoder) Resolved a rare deadlock (#9332)

Updated. Lets see how this goes. :slight_smile:

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.