Plexmediaserver not starting after Ubuntu crash

donald_christensenlighting_com · April 27, 2021, 4:01am

Server Version#: 1.22.3.4392
Player Version#: NA
Linux Version: Ubuntu 18.04.5 LTS

I had a hard server crash today resulting in plexmediaserver.service not shutting down correctly. After the machine rebooted, systemctl is not able to start plexmediaserver. I have been running this server for months without issue.

This is what the ‘systemctl status plexmediaserver’ reports:


plexmediaserver.service - Plex Media Server
   Loaded: loaded (/lib/systemd/system/plexmediaserver.service; enabled; vendor preset: enabled)
   Active: failed (Result: core-dump) since Mon 2021-04-26 23:39:11 EDT; 8s ago
  Process: 10689 ExecStart=/bin/sh -c  export PLEX_MEDIA_SERVER_INFO_VENDOR="$(grep ^NAME= /etc/os-release | awk -F= "{print \$2}" | tr -d \" )";  export PLEX_MEDIA_SERVER_INFO_DEVICE="PC";  export PLEX_MEDIA_SERVER_INFO_MODEL="$(uname -m)";  export PLEX_MEDIA_SERVER_INF
  Process: 10681 ExecStartPre=/bin/sh -c /usr/bin/test -d "${PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR}" || /bin/mkdir -p "${PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR}" (code=exited, status=0/SUCCESS)
 Main PID: 10689 (code=dumped, signal=SEGV)

Apr 26 23:39:11 server1301 systemd[1]: plexmediaserver.service: Service hold-off time over, scheduling restart.
Apr 26 23:39:11 server1301 systemd[1]: plexmediaserver.service: Scheduled restart job, restart counter is at 3.
Apr 26 23:39:11 server1301 systemd[1]: Stopped Plex Media Server.
Apr 26 23:39:11 server1301 systemd[1]: plexmediaserver.service: Start request repeated too quickly.
Apr 26 23:39:11 server1301 systemd[1]: plexmediaserver.service: Failed with result 'core-dump'.
Apr 26 23:39:11 server1301 systemd[1]: Failed to start Plex Media Server.

And this is what ‘journalctl’ reports:

Apr 26 23:39:06 server1301 systemd[1]: Started Plex Media Server.

-- Subject: Unit plexmediaserver.service has finished start-up

-- Defined-By: systemd

-- Support: http://www.ubuntu.com/support

--

-- Unit plexmediaserver.service has finished starting up.

--

-- The start-up result is RESULT.

Apr 26 23:39:06 server1301 kernel: Plex Media Serv[10689]: segfault at 19f2001 ip 00007fcc5d196438 sp 00007ffcb39c4ca0 error 4 in libc-2.27.so[7fcc5d0f6000+1e7000]

Apr 26 23:39:06 server1301 systemd[1]: plexmediaserver.service: Main process exited, code=dumped, status=11/SEGV

Apr 26 23:39:06 server1301 systemd[1]: plexmediaserver.service: Failed with result 'core-dump'.

Apr 26 23:39:11 server1301 systemd[1]: plexmediaserver.service: Service hold-off time over, scheduling restart.

Apr 26 23:39:11 server1301 systemd[1]: plexmediaserver.service: Scheduled restart job, restart counter is at 3.

-- Subject: Automatic restarting of a unit has been scheduled

-- Defined-By: systemd

-- Support: http://www.ubuntu.com/support

--

-- Automatic restarting of the unit plexmediaserver.service has been scheduled, as the result for

-- the configured Restart= setting for the unit.

Apr 26 23:39:11 server1301 systemd[1]: Stopped Plex Media Server.

-- Subject: Unit plexmediaserver.service has finished shutting down

-- Defined-By: systemd

-- Support: http://www.ubuntu.com/support

--

-- Unit plexmediaserver.service has finished shutting down.

Apr 26 23:39:11 server1301 systemd[1]: plexmediaserver.service: Start request repeated too quickly.

Apr 26 23:39:11 server1301 systemd[1]: plexmediaserver.service: Failed with result 'core-dump'.

Apr 26 23:39:11 server1301 systemd[1]: Failed to start Plex Media Server.

-- Subject: Unit plexmediaserver.service has failed

-- Defined-By: systemd

-- Support: http://www.ubuntu.com/support

--

-- Unit plexmediaserver.service has failed.

--

-- The result is RESULT.

Here is a list of things I have tried:

Using all backup databases
Upgrading Plex from 1.21.1.3876 to 1.22.3.4392 (to see if it would fix any issues, no luck)
I made sure ‘glibc’ was up to date by running ‘apt get glibc-source’ due to the libc-2.27.so error

Quick edit: There are no ‘crash reports’ associated with this.

Any help, questions or suggestions would be greatly appreciated. TIA

ChuckPa · April 27, 2021, 4:29am

Grab a tar.gz of the “Logs” directory and attach here please.

Let’s see if there’re any clues in it.

donald_christensenlighting_com · April 27, 2021, 4:42am

Log files as requested, thanks. logs.tar.gz (4.2 MB)

ChuckPa · April 27, 2021, 5:03am

@donald_christensenlighting_com

The last thing in your logs is this:

Apr 26, 2021 20:56:45.834 [0x7f2625dc3700] DEBUG - EventSource: Resolving 82.94.168.18 port 443
Apr 26, 2021 20:56:45.834 [0x7f2625dc3700] DEBUG - EventSource: Resolved 82.94.168.18 to 82.94.168.18
Apr 26, 2021 20:56:49.309 [0x7f26255c2700] DEBUG - Auth: authenticated user 27477161 as ***********************
Apr 26, 2021 20:56:50.938 [0x7f253e7fc700] WARN - Waited one whole second for a busy database.
Apr 26, 2021 20:56:50.937 [0x7f253effd700] WARN - Waited one whole second for a busy database.

A little bit before this, it lost connectivity with Plex.tv:

Apr 26, 2021 20:56:30.491 [0x7f260d5e0700] DEBUG - Signalling job ID 48444 with 9
Apr 26, 2021 20:56:30.834 [0x7f2625dc3700] ERROR - EventSource: Retrying in 15 seconds.
Apr 26, 2021 20:56:32.615 [0x7f260f5e4700] DEBUG - Client [8b90328f92fe0f55fa367aa335b3af35] reporting timeline state playing, progress of 4292000/5422474ms for guid=, playbackTime=4292364ms ratingKey=9312 url=, key=/library/metadata/9312, containerKey=, metadataId=9312, source=

Something was making your Xeon wait (not normal).

Did you run out of memory?
Did the root partition fill up?

If the root partition is full, PMS won’t start.

What is the current state of the machine, memory (free -m), and disk space (df -m) ?

donald_christensenlighting_com · April 27, 2021, 5:20am

20:56:32.615 is before the machine crashed. I’m still not sure why that happened, though all other services are working normally after reboot.

I currently have 10GB of ‘Free’ RAM, and 29GB available.
All disk partitions are under 29% including root partition.
Before the server crash, SSH had crashed. I logged into Webadmin, and was able to view stats - nothing crazy was in top, and lots of memory was free. Then nothing was accessible so I did a hard reboot on the machine (remote server FYI).

Thanks.

P.s. would you please remove the “Rick****@gmail.com” email address from your post, I do not want it indexing. Thanks!

ChuckPa · April 27, 2021, 6:00am

Let’s try this?

Settings - Server - General - Show Advanced
Turn off Crash Reporting

Now, when it does crash, instead of uploading the DMP file when it restarts, it will remain local for us to grab from “Crash Reports/version/PLEX MEDIA SERVER”

When you do get that, please put it in a ZIP so it will attach.

I’ll pull up the debugger and see where it burped

donald_christensenlighting_com · April 27, 2021, 6:10am

Hmm, perhaps I’m missing something. How can I change the settings if I cannot start plexmediaserver?

Thanks.

ChuckPa · April 27, 2021, 6:30am

I’m sorry (hour is late here).

We override the setting in Preferences.xml.

Stop Plex
as root, open Preferences.xml in your favorite linux editor
Before the closing /> on the last line,
add sendCrashReports="0"
It will look like this when done
SomePref="value" sendCrashReports="0" />
Save and exit
Make a backup copy just in case you fat-fingered something else
Now attempt start.

donald_christensenlighting_com · April 27, 2021, 6:55am

Hi, I made a backup of Preferences.xml and added the crash report setting as you described. I started Plex, it did not start, and it did not create any new folders in the ‘Crash Reports’ directory. I’m not sure how it would create a crash report if it does not even start?

Where do we go from here?

In any case - it’s late here also, and a glass of tequila is staring at me. I’ll catch you on the flip side. Thanks for your help so far.

donald_christensenlighting_com · April 27, 2021, 4:41pm

Hi, a quick update.

When using the Plex 'Repair a Corrupt Database" method, I get the following error:

Error: no such index: index_title_sort_naturalsort

When running the “PRAGMA integrity_check” I get this error:

Error: no such collation sequence: icu_root

This leads me to believe that my databases and backup databases my be currupt.

Then I found articles stating that I should use Plex to correctly repair databases:

"/usr/lib/plexmediaserver/Plex Media Server" --sqlite com.plexapp.plugins.library.db "PRAGMA integrity_check"

When I run the above I get:
Segmentation fault (core dumped)

Interestingly…when I check journalctl after the above, I get the following:
kernel: Plex Media Serv[23397]: segfault at 19f2001 ip 00007efc07890438 sp 00007ffd6663a0c0 error 4 in libc-2.27.so[7efc077f0000+1e7000]

This is the same error the server has when trying to start plexmediaserver.service (refer to first post with journalctl log for the same error)

This makes me think libc-2.27.so may be the issue?

ChuckPa · April 27, 2021, 5:09pm

We cannot use native sqlite anymore to fix the database. PMS has several extensions which it relies on.

Trying to integrate the extras into a disto sqlite3 is a pain. Engineering gave us a better way.

/usr/lib/plexmediaserver/Plex Media Server --sqlite

You will see the familiar

Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite>

when it opens.

Let’s first look at this then go after the glibc issue. That’s going to take some time because it shouldn’t be happening at all (PMS only needs 2.14.1)

donald_christensenlighting_com · April 27, 2021, 5:19pm

Right - I understand that.

When I run:
"/usr/lib/plexmediaserver/Plex Media Server" --sqlite

It crashes with the error:
Segmentation fault (core dumped)

And the following in journalctl:

kernel: Plex Media Serv[23397]: segfault at 19f2001 ip 00007efc07890438 sp 00007ffd6663a0c0 error 4 in libc-2.27.so[7efc077f0000+1e7000]

Interestingly the same libc-2.27.so error I get when trying to start plexmediaserver.service.

Thanks.

ChuckPa · April 27, 2021, 6:14pm

do you have gdb installed ?

I am asking because when invoking the --sqlite option, none of the PMS runtime system is invoked. All this does is:

Parse the command line options
Recognize --sqlite
Call the embedded shell

I’m starting to suspect a bad PMS lib directory (/usr/lib/plexmediaserver/lib).
If so, gdb will tell us a lot plus the remedy is as simple as uninstall, restart, and re-install.

Invoke gdb with

cd /usr/lib/plexmediaserver
gdb ./Plex\ Media\ Server 
run --sqlite
<when it breaks>
where

Show me that call back

donald_christensenlighting_com · April 27, 2021, 6:22pm

Copy, here is the call back:

(gdb) run --sqlite

Starting program: /usr/lib/plexmediaserver/Plex Media Server --sqlite

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.

__GI___memmem (haystack=<optimized out>, hs_len=<optimized out>, needle=0x7ffff7ff2411, ne_len=10) at memmem.c:101

101 memmem.c: No such file or directory.
(gdb) where
#0  __GI___memmem (haystack=<optimized out>, hs_len=<optimized out>, needle=0x7ffff7ff2411, ne_len=10) at memmem.c:101
#1  0x00007ffff7feeaef in ?? ()
#2  0x0000000000000000 in ?? ()
(gdb)

ChuckPa · April 27, 2021, 6:49pm

Oh man, you have a busted up installation.

That first stack frame is the start of PMS.
I recognize the offset.
It’s trying to identify the command line arguments and blowing up in the system libc.

What else runs on this host?
Are there any other programs with problems?

donald_christensenlighting_com · April 27, 2021, 6:56pm

Hmm - though plex was running fine without any crashes for 8 months.

About a month ago I added Sonarr, and it has also been working well.

The server also runs:

Apache2
Sonarr (I’ve disabled this for now)
Tautulli
Openproject
Seafile Server
Minio Server
Deluge
Webadmin

All the above are running perfectly.

Should I try reinstalling Plex?

Thanks

ChuckPa · April 27, 2021, 6:58pm

Yes, please do reinstall PMS.

If this is a ‘lightly damaged’ runtime library (not sure what constitutes ‘lightly’) then I can see how it might load without triggering a corruption flag but wow - quite a strange one. As if parts of the code in the lib itself was scribbled on.

donald_christensenlighting_com · April 27, 2021, 7:34pm

Re-installation worked! Plex fired back up as if nothing had happened. I did install the latest build form yesterday, 1.23.0.4438. Thanks for all your help!
Cheers,
Donald

system · July 26, 2021, 7:35pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Plex Server not starting Plex Media Server server-linux	29	251	July 29, 2021
Plex Media Server 1.30.1.6497 keeps crashing and restarting Plex Pass Betas server-linux	17	409	March 30, 2023
Ubuntu 18.04.2 LTS - Plex Server Crashes - Double free or corruption (!prev) Desktops & Laptops server-linux	264	7526	June 15, 2019
Plex Media Server keeps crashing Plex Media Server server-synology	51	1757	August 4, 2021
Plex crashes after only ~30 seconds Desktops & Laptops server-mac	26	349	November 19, 2020

Plexmediaserver not starting after Ubuntu crash

Related topics