Plex Server running on Linux keeps crashing

I have an instance of Plex Media Server running on a Linux Server and I keep getting random crashes with no indication as to the cause. I am running Linux 18.04 LTS (GNU/Linux 4.15.0-30-generic) and Plex Media Server version 1.13.5.5291. The crashes appear to be random, sometimes the server will be up for over 2 weeks while running 3 streams with no problem, and sometimes I will have one stream going with no transcoding, and less than 30% load on the CPU and the machine will freeze and I have to do a manual restart to get it going again. I am by no means an experienced user, but I have looked through several logs with some more experienced people and have not found any indication as to what might be causing the crashes. The only thing that is constant is that each time the entire machine freezes (cannot type at the local terminal) and the entire machine must be reset by hand. Any help is greatly appreciated, I am just not experienced enough to figure it out on my own, and I can’t seem to find similar problems after some basic googling. I will be happy to provide logs and crash reports. Thanks in advance!!

I’ve seen this happen when 1) Ram is toast at some point and/or 2) run out of space on OS partition as by default plex dumps transcodes to /tmp

Are you hovering close to 100% on /tmp mount point?
Try running memtest over night

1 Like

Running a memtest is a good idea, I will try that when I get off work. As for the running out of space option I don’t think that is the case, as my server OS is running on a 250 GB SSD. However I haven’t monitored the utilization of the /tmp mount point so I’m not 100% certain that isn’t the issue. I’ll run the memtest overnight tonight, though, and report back with the results tomorrow. Thank you!

1 Like

Oops, I lied… I changed the default location to /tmp on my box so it’s likely something else for you
https://support.plex.tv/articles/200250347-transcoder/

By default, /tmp is a ramfs created when linux boots. It is allocated 50% of installed memory.

Installed memory size will determine the maximum that can be transcoded at any one time. Remember, the transcoder is allowed to run X seconds ahead of real-time demand as buffer. If it doesn’t have enough room there’s a problem.

Transcoder temp should be on physical media (which it is by default).

Debug log files gathered as quickly as possible after re-creating the transcode failure or PMS restart are needed.

My availability is limited these next few days (engineering time) but will do my best to keep abreast and help here.

Ok so I let the memtest run last night and it didn’t encounter any errors. It made 6 passes over 13ish hours and didn’t have any signs of a problem, so I let it run and will check back in when I get off work today. Chuck you said if it does’t have enough room there might be a problem? Like I said I’m very new to the transcoding and linux server management game so I’m not completely sure I understand what you mean, but are you saying you think it might be running out of space? I have 8 GB of DDR3 ram installed so I could try doubling up on the ram? I will gather some debug logs after work today and attempt to recreate the problem so you have something relevant to look at. Thanks again for all the help!

1 Like

Nick,

If you have an 8GB RAM system. /tmp will be 4GB in size. period.
You do not need ā€œmemory speedā€ for the transcoder. HDD / SSD speed is more than sufficient.
Leave the transcoder temp at it’s default location (which is inside /var/lib/plexmediaserver/Library/Application Support/Plex Media Server).

Chasing memory is a red herring when host itself locks up. Bad physical memory will manifest as a crash or, worse, a kernel panic. You will most definitely know there is a physical issue.

To diagnose any issue like this needs log files.

Just like in Unversity… (ā€œpics or it didn’t happenā€), it applies here :slight_smile: ā€œLogs or it didn’t happenā€ because I can’t see what did happen. (and who doesn’t like seeing what really happened ? hahaha)

1 Like

Just to be clear… /tmp isn’t normal… I replied afterwards stating that’s just me so ignore that :slight_smile:
I run a bunch of slow 5400 greens and was running out of space so just moved to /tmp

No problem. :slight_smile:

Unless you’re transcoding Gigabits/second, the 100+ MB/sec you’ll get from a slow 5400 is more than enough to keep up :smiley:

Sorry for the delayed response, I haven’t been able to get the server to crash again, except it just did a few minutes ago, I think when I turned hardware acceleration on the problem stopped for a while. I will attach the zip file of the logs I downloaded from the server. Now again I really don’t have any knowledge as to how to look at this stuff, but to me, it seems like the problem occurs at the end of ā€˜Plex Media Server.2.log’. When looking at the logs there is a 14 minute gap from the end of log 2 to the beginning of log 1 where the server was frozen and this is when it stopped. Likewise at the beginning of log 2 it looks like some startup info is present which would make sense given that I reset the server. If there isn’t much to go on just let me know what I can do to get better results for when it crashes. Thanks again!!! Plex Media Server Logs_2018-08-12_22-50-28.zip (1.4 MB)

Nick,
Thanks for the logs but they’re totally useless.
All the [VERBOSE] statements filled the buffer and the needed details are gone off the back end.
Can you please turn verbose logging off and leave it off?

Will do. I have shut off verbose logging and will respond back with more logs when the server goes down again.

As it turns out the server crashed not long after I disabled verbose so I have some fresh data to look at. Hopefully this will provide some insight. Again if you look at the end of ā€˜Plex Media Server.1.log’ that is around when the server crashed. I noticed the crash around 0:41 but I think it usually goes down a minute or two before playback stops. Plex Media Server Logs_2018-08-13_00-46-24.zip (2.0 MB)

Nick,
Does the server hardware kernel panic? Is this in a VM? I need more info because the last thing written is a block of null bytes. This usually onlt happens when there is a page of zero-filled memory, partially useed, that the kernel is attempting to flush to disk so the filesystem doesn’t get corrupted.

Aug 13, 2018 00:39:18.202 [0x7f5a74fff700] DEBUG - Content-Length of /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Cache/Transcode/Sessions/plex-transcode-us7ardcw2ivw6vmqj8erdvlb-08560f14-1d3f-4699-be20-a7938dc9e640/init-stream1.m4s,/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Cache/Transcode/Sessions/plex-transcode-us7ardcw2ivw6vmqj8erdvlb-08560f14-1d3f-4699-be20-a7938dc9e640/chunk-stream1-03776.m4s is 24381.
Aug 13, 2018 00:39:18.203 [0x7f5a92bff700] DEBUG - Completed: [10.0.0.116:49302] 200 GET /video/:/transcode/universal/dash/us7ardcw2ivw6vmqj8erdvlb/1/3775.m4s (10 live) TLS GZIP 1ms 24381 bytes (pipelined: 20)
\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00

Chuck,
There is no evidence of Kernel panic. I had someone who is more experienced look through the logs with me and he said the same thing, he thought it was weird that there was no reported case of kernel panic. And no the PMS is running directly on the host OS, Ubuntu 18.04.

Then chalk it up to a bad version of PMS. Downgrade to the older and stay there until resolved by engineering.

Ok will do. Is there a ā€œstableā€ version of PMS you would recommend switching to?

Also, if I continue to get crashes is there anything I can do to help in the process of fixing it?

1.12.3 -> 1.13.2 are the most stable imho. 1.13.3 is marginally safe.

Mind you, each of those version had their own difficulties so be advised

Just noticed all the nulls in the log mentioned here Plex Server running on Linux keeps crashing

the last time this was looked into, it was thought to arise when the directories are mounted via NFS which has file locking issues that would lead to such symptoms as well as possible database corruptions.

Not sure if NFS is used here but if it is, then it could be the cause. Even with local_lock=posix in /etc/fstab NULLs were still getting appended to log files as well as corruptions

1 Like