Ubuntu 18.04.2 LTS - Plex Server Crashes - Double free or corruption (!prev)

Plex Media Server Desktops & Laptops
104

I just tried it on the plex process and it does generate a core dump when I issue kill -SIGSEGV to the plex server PID, both when running under systemd and when I invoke the process in a shell. I think the issue is that when we get the doublefree the process isn’t fully dumping to generate the core dump but remains hung. This is why systemd isn’t restarting the server because the main process never stops running.

I can run the process from a shell and see if it dumps properly at the next crash. Is there any benefit to sending the segfault signal and manually generating the dump?

105

No. The thread stack frames needed are the ones for the double free failure

106

I’ve been getting this issue in the Docker container. It’ll just crash and never reboot the container.

107

Mine is still crashing too. The latest update doesn’t seem to have made any improvement, I’ve had two crashes in the last 24 hours.

I haven’t had any luck getting the system to generate the core dumps the dev team is requesting. Regardless of whether I run plex via systemd or in a shell, when plex ‘crashes’ the process never terminates and generates the core dump. I have to manually kill the process a) for the dump to generate and b) for systemd to restart the server.

This issue would be much less annoying if the server would restart on its own, @sa2000 have you heard anything from the dev team about this? Is there a way for us to make the plex media server process terminate when the server crashes so that it will be restarted automatically? Would it be possible to roll back to the 1.14 branch?

108

I have been having these crashes on 1.15.2 and now on 1.15.3. I’m running CentOS 7.6.

Captured a core dump. Will PM that over along with server/plex logs.

109

Thank you very much. Core dump appears to have got triggered at the right time. I have passed it on to the development team

110

sa2000, I don’t know if it helps at all, but I rolled back to the latest release of 1.14 and have been up over 72 hours now without a crash so I think it is safe to say the issue was introduced in the 1.15 branch

111

sa2000, any update from the devs since baconator was able to capture a core dump?

112

Is everyone still having issues with Version 1.15.3.876?

Since rolling back to Version 1.14.1.5488 I have had zero crashes and don’t want to update back to 1.15 until this issue is resolved.

Server Crashes 503 Service Unavailable
113

I just crashed again on 1.15.3.876

114

Same here using docker on unraid.

115

I am also having these sporadic crashes. It seems to occur between watching episodes of a television show, though I do not know the root cause, this is just merely a symptom. I was playing an episode, got to the end, Plex automatically went to play the next, and that’s when I noticed it taking a while to buffer. I inspected the service and I see the same crash error as the original reporter.

Apr 08 22:37:20 anvil.localdomain sh[2499]: double free or corruption (!prev)
Apr 08 22:37:20 anvil.localdomain sh[2499]: ****** PLEX MEDIA SERVER CRASHED, CRASH REPORT WRITTEN: /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Crash Reports/1.15.3.876-ad6e39743/PLEX MEDIA SERVER/551b520a-4c20-f6ce-0d201c8e-33440253.dmp

I am using the RPM package from Plex’s RPM repository with Fedora 29 x86_64.

$ rpm -q -i plexmediaserver
Name        : plexmediaserver
Version     : 1.15.3.876
Release     : ad6e39743
Architecture: x86_64
Install Date: Fri 05 Apr 2019 03:41:02 PM CDT
Group       : Converted/video
Size        : 236118350
License     : Plex Inc.
Signature   : RSA/SHA1, Thu 04 Apr 2019 02:33:56 AM CDT, Key ID 97203c7b3adca79d
Source RPM  : plexmediaserver-1.15.3.876-ad6e39743.src.rpm
Build Date  : Thu 04 Apr 2019 02:33:18 AM CDT
Build Host  : build-slave-linux03.corp.sjc.plex.bz
Relocations : (not relocatable)
Summary     : Plex organizes all of your personal media so you can easily access and enjoy it.
Description :
Plex organizes all of your personal media so you can easily access and enjoy it.

This is the 3rd or 4th time it’s crashed within the last two weeks. There were new releases recently that I believe are the cause. This wasn’t happening a few versions ago (about a month ago).

I believe this bug is something new in plexmediaserver version 1.15.1.791-8bec0f76c. The last stable release I was running on was 1.14.1.5488-cc260c476.

It’s really a bummer because the systemd service doesn’t auto-restart the process when it crashes, the service reports everything is running; the process id stays running when it crashes, instead of exiting on failure, which is why the systemd watchdog thinks everything is fine when it’s not.

$ sudo systemctl status plexmediaserver -n 2
● plexmediaserver.service - Plex Media Server
   Loaded: loaded (/usr/lib/systemd/system/plexmediaserver.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2019-04-06 01:51:27 CDT; 2 days ago
  Process: 2497 ExecStartPre=/bin/sh -c /usr/bin/test -d "${PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR}" || /bin/mkdir -p "${PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR}" (code=exited, status=0/SUCCESS)
 Main PID: 2499 (Plex Media Serv)
    Tasks: 132 (limit: 4915)
   Memory: 6.1G
   CGroup: /system.slice/plexmediaserver.service
           ├─  461 Plex EAE Service
           ├─ 2499 /usr/lib/plexmediaserver/Plex Media Server
           ├─ 2539 Plex Plug-in [com.plexapp.system] /usr/lib/plexmediaserver/Resources/Plug-ins-ad6e39743/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py --server-version 1.15.3.876-ad6e39743 /usr/lib/plexmediaserver/Resources/Plug-ins-ad6e39743/System.bundle
           ├─ 2601 /usr/lib/plexmediaserver/Plex Tuner Service /usr/lib/plexmediaserver/Resources/Tuner/Private /usr/lib/plexmediaserver/Resources/Tuner/Shared 1.15.3.876-ad6e39743 32600 /waitmutex
           └─18680 /usr/lib/plexmediaserver/Plex Transcoder -codec:0 h264 -codec:1 ac3 -analyzeduration 20000000 -probesize 20000000 -i /srv/Movies/Glass.2019.1080p.WEB-DL.DD5.1.H264-CMRG/Glass.2019.1080p.WEB-DL.DD5.1.H264-CMRG.mkv -filter_complex [0:0]scale=w=718:h=300[0];[0]format=pix_fmts=yuv420p|nv1>

Apr 08 22:37:20 anvil.localdomain sh[2499]: double free or corruption (!prev)
Apr 08 22:37:20 anvil.localdomain sh[2499]: ****** PLEX MEDIA SERVER CRASHED, CRASH REPORT WRITTEN: /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Crash Reports/1.15.3.876-ad6e39743/PLEX MEDIA SERVER/551b520a-4c20-f6ce-0d201c8e-33440253.dmp
lines 1-16/16 (END)

When the crash occurs, the only resolution is to SIGKILL (sudo kill -9) all PIDs listed under the systemd service, then let systemd restart the services.

I am also on the following Linux kernel and uptime:

$ uname -r ; uptime
5.0.3-200.fc29.x86_64
 22:48:57 up 11 days,  4:51,  3 users,  load average: 3.10, 3.25, 3.44

The specifications of the host can be found at pcpartpicker. The two 5TB HDDs are attached to the motherboard joined in LVM striped pair and formatted with ext4.

I’ve attached the crash report and log directories from my server.

Logs-2019-04-08.7z (4.3 MB)

Repeated crashing "double free or corruption (out)"
116

Thanks to everyone for all the diagnostics and for the core dump provided by @baconator_admin from a Centos PMS server

The development team have identified an issue that does a double free but we are not 100% sure that this is the exact issue. I expect to have a development Plex Media Server build available this Friday which i would like to offer to users that had these crashes.

So please let me have the exact binary / package file used for your install and I will make that available. If the problem persists then will look for a new set of logs and and core dump from @baconator_admin

Thank you

1 Like
117

Good to hear. I’ve been away for a while, but I haven’t been seeing crashes for quite some time. Not sure if I’ve just been lucky or if something is different on my system now. I’ve been running the latest release of PMS.

I checked my /var/crash directory and found a core dump file created on Apr 1st. I’m not sure if it was due to “double free or corruption” error, but I could provide the core dump if you think it would be useful (877MB - created 2019-04-01 23:02). Let me know @sa2000

I’m willing to test out a dev build also if you’d like. I’m running ‘Ubuntu 64-bit’ (amd64.deb) build.

Thanks,
Kevin

118

Happy to hear the issue has potentially been identified. Hoping for a quick resolution and looking forward to Friday testing. Count me in!

I’m using Fedora 29 with package plexmediaserver-1.15.3.876-ad6e39743.x86_64 from the official Plex repository. I’m happy to install a specific rpm outside of that repository for testing.

Thank you.

119

Lets hold fire until the development build goes out and we should get dumps from that if the problem does not go away

120

Thanks for the update! I’ll give the beta package a try if you can provide an x86_64 RPM.

I’ve had the crash 7 times since updating to the 1.15.x branch. Mar 15, 16, 22, 24, 31 and twice on Apr 7. All of them are days of peak activity on the server (Fri/Sat/Sun). I went through Tautulli logs to see if I could identify a specific platform, e.g. Roku, that was in the middle of playback during all 7 of the crash events. The data, unfortunately, was inconclusive.

121

[root@streaming ~]# uname -r
4.18.19-100.fc27.x86_64
[root@streaming ~]# rpm -qa | grep plex
plexmediaserver-1.15.3.858-fbfb913f7.x86_64

Seeing the same error as people are reporting in here - dump didn’t get created, though.

122

Good to hear. I’m using the docker image.

123

Glad it has been identified. I would be willing to test the development build.

I am running CentOS 7