That sounds like a Ubuntu thing? I’m running CentOS 7, using the default Plex systemd service file, and mine has always automatically restarted Plex after these ‘double free’ crashes. I didn’t have to do anything to get it to do that.
I’m running Fedora 29 with a newer SystemD than CentOS 7 ships with and the service doesn’t auto-restart for me. SystemD never gets the signal that the PID has crashed because the PID stays running after it segfaults, instead of exiting with a SIGSEGV signal. Any exit signal would allow the SystemD watchdog to restart the service automatically, but because it stays running, SystemD takes no action after the double free crash. The PID needs to exit after it crashes, plain and simple, and it’s not doing that.
Not sure what to tell ya. Restarts fine for me. Here is what it looks like in syslog:
May 15 18:38:29 plex sh: ****** PLEX MEDIA SERVER CRASHED, CRASH REPORT WRITTEN: /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Crash Repo
rts/1.15.5.994-4610c6e8d/PLEX MEDIA SERVER/77a5231e-0d2e-2800-791aed79-55956299.dmp
May 15 18:38:29 plex abrt-hook-ccpp: Process 52924 (Plex Media Server) of user 774200019 killed by SIGABRT - dumping core
May 15 18:38:50 plex abrt-hook-ccpp: Failed to create core_backtrace: waitpid failed: No child processes
May 15 18:38:51 plex abrt-server: Package ‘plexmediaserver’ isn’t signed with proper key
May 15 18:38:51 plex abrt-server: ‘post-create’ on ‘/var/spool/abrt/ccpp-2019-05-15-18:38:29-52924’ exited with 1
May 15 18:38:51 plex abrt-server: Deleting problem directory ‘/var/spool/abrt/ccpp-2019-05-15-18:38:29-52924’
May 15 18:38:51 plex systemd: plexmediaserver.service: main process exited, code=killed, status=6/ABRT
May 15 18:40:01 plex systemd: Created slice User Slice of root.
May 15 18:40:01 plex systemd: Started Session 8627 of user root.
May 15 18:40:02 plex systemd: Removed slice User Slice of root.
May 15 18:40:21 plex systemd: plexmediaserver.service stop-sigterm timed out. Killing.
May 15 18:40:21 plex systemd: Unit plexmediaserver.service entered failed state.
May 15 18:40:21 plex systemd: plexmediaserver.service failed.
May 15 18:40:26 plex systemd: plexmediaserver.service holdoff time over, scheduling restart.
May 15 18:40:26 plex systemd: Stopped Plex Media Server.
May 15 18:40:26 plex systemd: Starting Plex Media Server…
May 15 18:40:26 plex systemd: Started Plex Media Server.
May 15 18:38:29 plex abrt-hook-ccpp: Process 52924 (Plex Media Server) of user 774200019 killed by SIGABRT - dumping core
This part never happens on my server.
The rest does occur; if I manually tell the service to restart, the timeout on SIGTERM forcing a SIGKILL which then allows a full restart of the service occurs, but only after manual intervention.
$ sudo sysctl -a | grep core_pattern
kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e
$ systemctl status abrtd.service abrt-ccpp.service abrt-oops.service
Unit abrtd.service could not be found.
Unit abrt-ccpp.service could not be found.
Unit abrt-oops.service could not be found.
Can’t say I have enough experience with Fedora 29 to tell you that abrt should or should not be there. It’s possible that Fedora has something newer/different for handling crashes. Could also be that you are rocking a minimum install that didn’t include it? All the abrt stuff came out-of-the box in CentOS 7 for me.
Yes, this was a minimal/server install, not a full DVD install. None of those abrt packages are installed.
Checking with some Fedora resources, it appears systemd-coredump replaced abrt back in Fedora 24, however to revert it’s as simple as installing it and using abrt-journal-core.service alongside abrt. More info: https://fedoraproject.org/wiki/Changes/coredumpctl
My coredumpctl is using defaults. To date, I’ve only had one Plex server crash actually produce a coredump. I am not certain where to configure the frequency that it does, since there’s currently none (I uploaded and then deleted the only coredump I had which was from last month), so it should always be producing them because I haven’t reached quota, from how I’ve read the manual.
Either way, none of this information seems to be the root cause; the PID never exits. Perhaps that is also why a coredump is never generated.
Edit: Here’s my coredump.conf settings, for posterity. Other locations mentioned by the manual do not exist/are not in use; this is the only coredump.conf I have on my system.
$ cat /etc/systemd/coredump.conf
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See coredump.conf(5) for details.
[Coredump]
#Storage=external
#Compress=yes
#ProcessSizeMax=2G
#ExternalSizeMax=2G
#JournalSizeMax=767M
#MaxUse=
#KeepFree=
There are no coredumps:
$ sudo ls -al /var/lib/systemd/coredump/
total 0
drwxr-xr-x. 2 root root 6 Apr 13 02:29 .
drwxr-xr-x. 5 root root 70 Feb 20 10:51 ..
The last coredump was Wed 2019-04-10 03:38:19 CDT (1 months 6 days ago) according to sudo coredumpctl info, which matches with that directory’s last modified date on the filesystem and the time I uploaded then deleted the coredump 3 days later.
I’ve been following this thread since the start and I just want to toss in that I’ve been having the same error in my logs on Unraid since the beginning like many here. A couple months with no fix is rough.
double free or corruption (!prev)
****** PLEX MEDIA SERVER CRASHED, CRASH REPORT WRITTEN
I really wish a developer could elaborate on what they’ve been able to uncover so far regarding the issue. Everyone who’s posted here seem to have different scenarios that led to the sporadic crash. A rhyme or reason to how the crash occurs would maybe prove beneficial to server owners, so they could avoid the crash altogether or at least mitigate it. At current rate, it seems it will happen around anywhere from once a day to every two weeks, at seemingly random. For me specifically, it’s occurred only when a user ends a stream, any other information related to that would be useful; is it my transcoding settings? a setting somewhere I could change that would mitigate it in some way? Any information here would be great.
All we’ve got right now is that “they’re working on it” and that “they have sufficient data at this time” (paraphrasing). That’s been the answer for several weeks. Has no progress been made after countless users submitted their error reports?
I scrolled up this thread for several posts in the past and all of us are Plex Pass members. I don’t feel like we’re getting what we signed up for.
I have been experiencing this issue for quite some time. I would simply like to roll back to 1.14 so that I can finally just use my Plex until they can figure this bug out. Does anybody have the mirror to download the old 1.14 for Ubuntu 64bit? I am having a hard time finding this online. Any help is much appreciated!
Another silent crash this morning. The worst part is that Tautulli (Plexpy) does not recognize this as Plex being down so I don’t even know it has happened until a while after it crashes.
@Samwiseg0 when your Plex docker shows “double free or corruption (!prev)” in the log, your Tautulli docker detects that Plex is actually down? Just so I know we are on the same page. If so then yes I will pop over to see about getting mine fixed so I can at least detect these things.
EDIT: I should mention that it detects when Plex is down if I shut it down, or in the past when it crashed for other reasons etc. It just doesn’t seem to handle detecting the “double free or corruption (!prev)” crash.
What I am saying is it should. I am not running the docker container but that has nothing to do with how Tautulli detects when it is down. If it works in other instances then for some reason your plex server is holding the websocket connection open. It is possible that certain tweaks could be made to detect that it is down on the Tautulli side if that is something you want to look into. Further discussion can be had on the Tautulli discord.
Just wanted to reiterate that version 1.14.1.5488 has been rock solid. Roll back to that and you’ll have 0 issues while you’re waiting for this to be fixed.
