Server Keeps Crashing

server-windows

#1

Version 1.11.0.4666

I am doing some rearranging of my titles. Last night I altered some libraries, adding "Collections" etc.

This morning I woke up to my Plex server crashed. Weird. Restarted it.

Today I have been doing some things like add movies to collections and re-group some movies with multiple versions etc. I'm doing this via the desktop app on another PC on my LAN.

In the process of doing this the server has crashed at least 3 times in an hour, causing me to have to restart. Something's not right. If I look in the local log folder on my server box (Win10-64) it shows 5 crash reports successfully sent to Plex. This is the contents of the latest one:

Dec 25, 2017 17:36:01.465 [9472] INFO - Crash Uploader - Platform: windows-i386 (6.2 (Build 9200)) ServerUUID: cf214e02b2b9aa7bd345d88d1a317064af75547f UserId: XXXXXXXXXXXXXXXXX Version: 1.11.0.4633-70f9c31ed
Dec 25, 2017 17:36:01.329 [3952] DEBUG - HTTP requesting POST https://crashreport.plexapp.com
Dec 25, 2017 17:36:03.138 [3952] DEBUG - HTTP 200 response from POST https://crashreport.plexapp.com
Dec 25, 2017 17:36:03.139 [3952] INFO - Successfully reported f2d15848-5ed9-46f8-bb1e-3dfd727c6b54

Anybody else?


#2

If Plex truly crashed (no longer in the task manager), then grab the crash log and the server logs and post them here. Since you don't have crash logs right now (have to get them before you restart Plex server), just attach your server logs to your next post so we can take a look.


#3

Having same issues,- Version 1.10.1.4602
Win 7, server runs no problem until accessed by a client, (Roku 3, XBox1, Tablet) runs for about 3~7 minutes. I noticed some high CPU usage (8 cores, on and off), then it crashes, causing a whole server reboot. got much worse with new update installed. I can be replicated over and over
regardless of what I'm watching
Attaching Plex log
We love PLEX, and are suffering with it down


#4

@kegobeer-plex said:
If Plex truly crashed (no longer in the task manager), then grab the crash log and the server logs and post them here. Since you don't have crash logs right now (have to get them before you restart Plex server), just attach your server logs to your next post so we can take a look.

OK, Plex did truly crash multiple times. No longer in the systray. Had to restart Plex server and all would be well for a while.

I found this article which told me how to get the crash logs because I didn't know:

https://support.plex.tv/hc/en-us/articles/201455336-Crash-Logs-Plex-Media-Server

...and this of course was the case:

"Remember, these crash logs (or "crash dumps" in Windows terminology) will typically be removed the next time Plex Media Server is launched, so please retrieve them beforehand."

There were two crash folders dated today, but they were empty.

So I have the server logs. Does this help?


#5

@KaraokeAmerica said:

@kegobeer-plex said:
If Plex truly crashed (no longer in the task manager), then grab the crash log and the server logs and post them here. Since you don't have crash logs right now (have to get them before you restart Plex server), just attach your server logs to your next post so we can take a look.

OK, Plex did truly crash multiple times. No longer in the systray. Had to restart Plex server and all would be well for a while.

I found this article which told me how to get the crash logs because I didn't know:

https://support.plex.tv/hc/en-us/articles/201455336-Crash-Logs-Plex-Media-Server

...and this of course was the case:

"Remember, these crash logs (or "crash dumps" in Windows terminology) will typically be removed the next time Plex Media Server is launched, so please retrieve them beforehand."

There were two crash folders dated today, but they were empty.

So I have the server logs. Does this help?

I found the crash report on the crash reporting system - I will add the logs to it and flag it to the development team


#6

@sa2000 said:

I found the crash report on the crash reporting system - I will add the logs to it and flag it to the development team

Thanks! It's been up and running now for 24 hours without issue, but I haven't been doing a lot with it like I was before. I'd just never had this happen before the latest PMS update. Could be coincidence, but.....


#7

@KaraokeAmerica said:

@sa2000 said:

I found the crash report on the crash reporting system - I will add the logs to it and flag it to the development team

Thanks! It's been up and running now for 24 hours without issue, but I haven't been doing a lot with it like I was before. I'd just never had this happen before the latest PMS update. Could be coincidence, but.....

The crash is to do with scanning media and collections - it is new


#8

@sa2000 said:

@KaraokeAmerica said:

@sa2000 said:

I found the crash report on the crash reporting system - I will add the logs to it and flag it to the development team

Thanks! It's been up and running now for 24 hours without issue, but I haven't been doing a lot with it like I was before. I'd just never had this happen before the latest PMS update. Could be coincidence, but.....

The crash is to do with scanning media and collections - it is new

That is definitely when it happens. It started when I combined two libraries by adding an existing folder to a different library and deleted the now unused library.

It's working right now, but I have a title that I added yesterday that it doesn't even see....at all. Usually when you add new content, even if it can't identify it it will just add an item to the library with the name of the file and a generic snapshot. in this case It just doesn't see it at all.


#9

@KaraokeAmerica said:

@sa2000 said:

@KaraokeAmerica said:

@sa2000 said:

I found the crash report on the crash reporting system - I will add the logs to it and flag it to the development team

Thanks! It's been up and running now for 24 hours without issue, but I haven't been doing a lot with it like I was before. I'd just never had this happen before the latest PMS update. Could be coincidence, but.....

The crash is to do with scanning media and collections - it is new

That is definitely when it happens. It started when I combined two libraries by adding an existing folder to a different library and deleted the now unused library.

It's working right now, but I have a title that I added yesterday that it doesn't even see....at all. Usually when you add new content, even if it can't identify it it will just add an item to the library with the name of the file and a generic snapshot. in this case It just doesn't see it at all.

Appears to be database lock that is remaining for long period and after a number of retries, we crash.
The evidence has been referred to the development team

Could you see if optimizing the database helps
See this support article - Library Actopm Optimize Database
https://support.plex.tv/hc/en-us/articles/200392106


#10

@sa2000 said:

Appears to be database lock that is remaining for long period and after a number of retries, we crash.
The evidence has been referred to the development team

Could you see if optimizing the database helps
See this support article - Library Actopm Optimize Database
https://support.plex.tv/hc/en-us/articles/200392106

I optimized the database and cleaned all the bundles this morning. Of course it hasn't crashed in a couple days, but better safe than sorry.


#11

@KaraokeAmerica said:

@sa2000 said:

Appears to be database lock that is remaining for long period and after a number of retries, we crash.
The evidence has been referred to the development team

Could you see if optimizing the database helps
See this support article - Library Actopm Optimize Database
https://support.plex.tv/hc/en-us/articles/200392106

I optimized the database and cleaned all the bundles this morning. Of course it hasn't crashed in a couple days, but better safe than sorry.

There will be a fix to handle the exception so we do not crash if the database is locked out for a few seconds

In the meantime, do run frequent optimizations so that database actions do not take long periods - thus leading to the exhaustion of retries and the crash


#12

Version 1.11.1 of Plex Media Server has now been released as beta. It should include a fix for the crashes that arise when the database is inaccessible for a period of time.

See Release Notice
http://forums.plex.tv/discussion/comment/1602026/#Comment_1602026

  • A crash could occur on Windows after a library scan (#8029)

#13

Well my plex keeps crashing..... Here is my log file.


#14

@FIGHT said:
Well my plex keeps crashing..... Here is my log file.

The logs do show crashes at 20:08 on the 17th April and before that at 14:25 on the 16th April
Unfortunately I cannot find any crash report on the systems that they get uploaded to. This could be because the dumps may be corrupt - eg this may arise if the crash is due to Heap Memory Corruption

Could you have a look at the Windows Event Log checking the Application log - so command eventvwr.msc and then select Windows Logs and then select Application
Then scroll through until you get close to 20:08 on 17 April and 14:25 on 16 April
See what Application Exception events are logged for the Plex Media Server crashes

We could try to see if windows can secure the dump - to do that see the registry edit outlined on this support article under heading Windows Crash Dumps Handled by Windows
Please add this registry entry and reboot the system
With this change when you get a crash restart the server and get the logs zip and also look in %LocalAppData%\CrashDumps folder for any Plex Media Server dmp files
You would need to zip to upload with the logs zip

I notice that you have Per Session Temp Directory in use - this became the default on the Windows Servers platform recently and also for any process started through RDP. I do not know if this has a bearing on the crashes - however I do know that are issues when Per Session Temp Directory is used. The clue to this was here

Apr 17, 2018 21:45:21.199 [13076] DEBUG - Content-Length of C:\Users\ADMINI~1\AppData\Local\Temp\2\logs.zipffd919cf-73a7-4449-b9a1-50dc3da2c34d is 7408338.

where you can see the Temp\ directory is actually Temp\2\ and not Temp\
There are articles available on how to disable this eg
https://www.windows-security.org/13618e03a81e77cd4fb79ccdf7974840/do-not-use-temporary-folders-per-session
Or if Plex Media Server.exe was launched in a RDP session - you could avoid doing that


#15

Hi sa2000, :slight_smile:

I just checked the logs and there nothing here

I have attached a screenshoot here.

My server is a windows 2012 and I am only running RDP to it, since it´s running on my exsi. :slight_smile:


#16

@FIGHT said:
Hi sa2000, :slight_smile:

I just checked the logs and there nothing here

I have attached a screenshoot here.

My server is a windows 2012 and I am only running RDP to it, since it´s running on my exsi. :slight_smile:

Strange there is no Application Exception logged

Could you try the registry setting to get windows to manage the dumps and see what happens on next crash - if that works, the dump would be in %LocalAppData%\CrashDumps and you can zip and upload together with zip of the logs


#17

Crap my server crashed again, has been running ok for the last many days.....


#18

@FIGHT said:
Crap my server crashed again, has been running ok for the last many days.....

The Plex Media Server.exe process ran out of memory. Could be a memory leak or insufficient memory available to the process.
The Plex Media Server process was started 7th May at 22:55. The logs which go back to 10:45 am 13th May showed the first memory allocation failure at 12:45

May 13, 2018 12:45:07.269 [8748] ERROR - Format [JPEG] - Insufficient memory (case 4)

This was followed by many allocation failures

May 13, 2018 12:45:07.372 [9700] ERROR - Format [JPEG] - Insufficient memory (case 4)
May 13, 2018 12:45:07.397 [13032] ERROR - Format [JPEG] - DIB allocation failed, maybe caused by an invalid image size or by a lack of memory
May 13, 2018 12:45:07.479 [2644] ERROR - Format [JPEG] - DIB allocation failed, maybe caused by an invalid image size or by a lack of memory
May 13, 2018 12:45:07.610 [2644] ERROR - Format [JPEG] - DIB allocation failed, maybe caused by an invalid image size or by a lack of memory
May 13, 2018 12:45:09.228 [13120] ERROR - Format [JPEG] - DIB allocation failed, maybe caused by an invalid image size or by a lack of memory

until eventual crash at 13:13

With memory leaks and getting to a point where the process reaches the maximum available memory, one would need to use diagnostic tools such SysInternals VMMap and SysInternals Process Monitor - following similar investigations as was done in these two forum threads

http://forums.plex.tv/discussion/319786/plex-media-server-crashing-memory-allocation-failures/p1
and
http://forums.plex.tv/discussion/310320/pms-crashing-version-1-11-3-4803-windows-7/p1

I hope you will be ok with going down this investigation route to try and see if there is a memory leak and where it might be

VMMap allows you to see in a timeline chart if memory usage is climbing up - the sign of a memory leak. And when memory usage remains climbing and is going over 1Gb (max possible is 2Gb) then one can run Process Monitor for 5 minutes at a time and capture events - a couple of such captures would probably be sufficient if captured during periods when VMMap timeline shows climbing memory use.

Have a look at the timeline charts pasted in the 2 forum threads.

In the last case, VMware video drivers were suspected as leading to a threads leak (too many threads 0 each using up 1.25Mb ram) - this was occurring when Plex Media Server was testing out codecs. The problem happened to go away after upgrading a client app !

The previous case was a bug in Plex Media Server which was fixed

These procmon captures together with saving the VMMap environment and the Plex Media Server logs zip, it should be possible then to narrow the problem down

SysInternal VMMap can be downloaded from here https://docs.microsoft.com/en-us/sysinternals/downloads/vmmap
SysInternals Process Monitor can be downloaded from here https://docs.microsoft.com/en-us/sysinternals/downloads/procmon

ProcMon captures would be very large and capturing of events should be limited to 5 minutes at a time. The saved capture PML files do zip very well. These should be uploaded to a private area eg dropbox and link sent to me by private message. Do not filter any events for procmon. I can do the filtering when investigating and looking at the captures

VMMap mmp saved files are not very large - and you can save the VMMap environment to mmp file once you noticed te memory used by the Plex Media Server.exe process has reached over 1Gb and you have captured a couple of process monitor captures during the climb

Best to have all evidence together and you can send me link

Let me know if you need any clarifications

Thanks


#19

My PMS app keeps crashing, but I've isolated it to what action is causing the crash, but not sure why or how to fix it.

I'll go to my music library and start an audio track (any track), it'll start playing the track fine. When it finishes and goes to the next track, the PMS app crashes. It does not do this with video file libraries. It only crashes with the audio file libraries.

I can also cause it to crash by jumping forward and backward in a track, or even by skipping and selecting other tracks. Sometimes, pausing & restarting, or closing the track out will also cause PMS to crash.

I'm trying to grab crash reports, but even when I do not re-launch PMS and go the various areas where PMS crash reports are supposed to be located, the following are empty:

%LocalAppData%\Plex Media Server\Crash Reports
Program Files (x86)\Plex\Plex Media Server\Crash Reports

I did find reports in the following location, but unsure if these are the correct ones (file attached):

%LocalAppData%\CrashDumps

My temporary work-around is running a program called Restart on Crash, which anytime my PMS crashes or hangs, it automatically restarts PMS. I do keep the log files for this application, as well, since this was the main reason I was able to isolate what specific activity was causing the problem. I've used this application many times in the past for other server setups and never figured I'd need to run it on my PMS since it has always run so stable, until now.

Any help would be greatly appreciated.


#20

@Doramius said:
My PMS app keeps crashing, but I've isolated it to what action is causing the crash, but not sure why or how to fix it.

I'll go to my music library and start an audio track (any track), it'll start playing the track fine. When it finishes and goes to the next track, the PMS app crashes. It does not do this with video file libraries. It only crashes with the audio file libraries.

I can also cause it to crash by jumping forward and backward in a track, or even by skipping and selecting other tracks. Sometimes, pausing & restarting, or closing the track out will also cause PMS to crash.

I'm trying to grab crash reports, but even when I do not re-launch PMS and go the various areas where PMS crash reports are supposed to be located, the following are empty:

%LocalAppData%\Plex Media Server\Crash Reports
Program Files (x86)\Plex\Plex Media Server\Crash Reports

I did find reports in the following location, but unsure if these are the correct ones (file attached):

%LocalAppData%\CrashDumps

My temporary work-around is running a program called Restart on Crash, which anytime my PMS crashes or hangs, it automatically restarts PMS. I do keep the log files for this application, as well, since this was the main reason I was able to isolate what specific activity was causing the problem. I've used this application many times in the past for other server setups and never figured I'd need to run it on my PMS since it has always run so stable, until now.

Any help would be greatly appreciated.

The crashes appear to be memory related. I am surprised the CrashUploader.exe is also crashing

Would like to see the server logs for period leading up to each of the crashes - with debug logging enabled

If you still have the logs covering the time of the last crash dumps, please zip and upload
See https://support.plex.tv/articles/201643703-reporting-issues-with-plex-media-server/
https://support.plex.tv/articles/200250417-plex-media-server-log-files/

You mentioned looking here Program Files (x86)\Plex\Plex Media Server\Crash Reports
A Crash Reports folder should not be there unless you have amended the Local App Data Path for Plex Media Server

Could you set debug logging on the server
Restart the server
and watch memory usage on the PC - see how much free memory there is and how much is used by the Plex Media Server.exe process
Start to play music tracks and check memory usage
When it crashes do not auto restart.
Check Windows Task Manager to see what plex processes remain running
Copy out the Plex Media Server Logs folder and zip the copy (do not zip in situ)
And locate any dmp files in the CrashDumps folder and Local App Data Path Plex Media Server\Crash Reports folders
and attach the zips and info on memory usage