Server crashing after latest update to 1.15.0 - Logs included

The problem is that we need a way to reproduce these crashes. So the magic would be identifying steps that would crash the server. Whilst I am being swamped with logs here - still the more examples the better in identifying patterns

What is also important in absence of any crash reports is to explore the use of windows registry settings to get windows to produce a dmp file - this has yet to be achieved - https://docs.microsoft.com/en-gb/windows/desktop/wer/collecting-user-mode-dumps

And since these crashes are not generating crash reports, for each crash would need to see export of the windows application event log for event id 1000 and event id 1001 together with logs

Could you see if any of these files exist ? if they do, please zip and let me see them

C:\ProgramData\Microsoft\Windows\WER\Temp\WER5CAE.tmp.mdmp

C:\ProgramData\Microsoft\Windows\WER\Temp\WER5F11.tmp.WERInternalMetadata.xml

C:\ProgramData\Microsoft\Windows\WER\Temp\WER5F50.tmp.xml

C:\ProgramData\Microsoft\Windows\WER\Temp\WER5F4E.tmp.csv

C:\ProgramData\Microsoft\Windows\WER\Temp\WER5F7E.tmp.txt

C:\Users\Rebel\AppData\Local\Temp\WER6137.tmp.appcompat.txt

C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_Plex Media Serve_896f15af20b985dac7c91dbb96885c146ea2626b_9900dce8_cab_51fc61b0\memory.hdmp

C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_Plex Media Serve_896f15af20b985dac7c91dbb96885c146ea2626b_9900dce8_cab_51fc61b0

The event log shows the crash to be at around 00:19 am on 27th
Appears to be a xc0000409

P1: Plex Media Server.exe
P2: 1.15.1.710
P3: 5c703559
P4: ucrtbase.dll
P5: 10.0.17763.1
P6: c3ae1887
P7: 000a5d0b
P8: c0000409
P9: 00000007
P10: 

and do you have the registry setting for getting windows to create dump files for app crashes ?

The dmp files would get placed in %LocalAppData%\CrashDumps

I haven’t run the SFC command; will try to remember that in future (also, to include the Locale Metadata folder in the Event Viewer uploads.)

I’m not sure taking BearcubII out of the equation is the best idea, but I understand your thinking here. It does make me wonder, though, where that leaves users with big setups – there are some Plex installations out there that make mine look thoroughly anemic (the various Plex subs on Reddit verge on the terrifying) and have me mulling the idea of throwing together a proper server rack. Mind you, I think many of these big rigs are intended mainly for film/TV, and multi-user – aside from me, I have one external user, and I don’t think he ever accesses much.

Looks like Plex crashed on Brainbuntu very soon after Scheduled Tasks started running. BearcubII on the other hand stayed up, not that the Android app would talk to it (No Internet, it said. I’ll take that to the Mobile App mob.)

Plex Media Server Logs_2019-03-01_09-47-22 (Brainbuntu).zip (5.8 MB)

Plex (Brainbuntu) Crash Event Viewer Log 03-01-2019 2.07AM.zip (509.4 KB)

I’ll see about setting up the User Mode Dump later. I’m about to go take the day off, considering that the massive physical efforts yesterday have left me feeling horrendous today.

Thanks, so exactly the track I thought was the likely problem. I’ll reprocess the album (and it may be reripped when I get to it, this time in FLAC) and see if that does anything to fix the issue.

For users that are getting many Database Busy errors/warnings in the log prior to crashing with exception code 0xc0000409 STATUS_STACK_BUFFER_OVERRUN, I have a diagnostic build to send out - build 1.15.1.725. If you don’t hear from me and you do see these warnings in the log before crashing, let me know, example

Feb 27, 2019 17:00:27.221 [4924] WARN - Waited one whole second for a busy database.
Feb 27, 2019 17:00:27.785 [15544] WARN - Waited one whole second for a busy database.
Feb 27, 2019 17:00:28.263 [14616] WARN - Waited one whole second for a busy database.
Feb 27, 2019 17:00:28.601 [8876] WARN - Waited one whole second for a busy database.
Feb 27, 2019 17:00:28.949 [6348] WARN - Waited one whole second for a busy database.

When I move Plex to the new machine (with an SSD, very likely) I might consider just moving some libraries onto Brain. Or, indeed, onto the Mouse Orchestra (the studio computer.) Having the audiobooks libraries there would probably help, although the wireless connection on Mouse is sluggish – not a problem with fairly low-res audiobooks, though, especially as I’d most often run them in situ.

Diagnostic build 1.15.1.725 sent to @Wang.Chung.Plex @foxmen and @wyldemusick

1 Like

Got it, thanks. Will set it all up later…too fuzzy right now to hack the registry.

i am also having crashes happen during the night about 1-3 minutes after starting scheduled tasks with the same 409 exception related to ucrtbase.dll. as seen above. happy to test out a new build.

Dont have logs handy right ow but i was having ā€œslow queries: waited XXX ms to retrieve XX items before crashā€

Done. Please follow the instructions in my message concerning logging (need both debug and verbose) and increasing number of log files

I still need to get to hacking the registry, but it may wait for a bit – to my amazement, neither server crashed overnight.

I’ve popped a few more pimples on BearcubII, as well – the current intensive response to issues is revealing more bad media (fortunately correctable) and improving the state of the libraries there. That will be helpful when the new box is in place and I rebuild from scratch (I’m not bothering porting Plex over, just starting clean.)

BearcubII stayed up, Brainbuntu crashed at 03:26 AZ time, nearly an hour and a half after scheduled tasks started.

Plex Brainbuntu 03 03 2019 03 26 am Event Viewer Log.zip (13.0 KB)

Plex Media Server Logs_2019-03-03_08-54-39.zip (4.9 MB)

Thanks. I will send the link in PM.
I’m on the latest beta channel Plex version, Windows update updated the ucrtbase.dll too.
The situation is the same, if I add this specific album to library the Plex server crashes. I tested now with a completly new library.
After I restart the Plex I deleted immediately the album to aviod further crashes.
It has nothing to do with the file itself, same happens if I try with flac or mp3.

Thanks!

Brainbuntu Plex Crash at 2:04am AZ time, four minutes after scheduled tasks are set to start. Did the usual pre-breakfast (for me, not the cats) and herewith, the logs.

Plex Media (8.8 MB)
Plex (Brainbuntu) Event Viewer 03 04 2019 02 04am AZ.zip (8.1 KB)

We have dumps from @Wang.Chung.Plex that were manually taken after the 0xc0000409 crashes

I would like to know from other users if after crashing with the 0xc0000409 stack overrun exception, if the Plex Media Server.exe process remains to show up in windows task manager

If it does remain, then please go through the process of forcing a process dump of what would be a hung crashed process and provide that dmp with zip of the logs and export of the application event log for event IDs 1000 and 1001

See https://support.plex.tv/articles/201373203-gather-a-process-dump-or-sample-process/

Thanks - although your crashes have different exception codes (0x40000015) - would also like to know if the crashing process does actually disappear from Windows Task Manager - if it hangs as well - then the manual dump of the process would be useful

@wyldemusick - i see you sent me some dmp files - brilliant. Will process these with the logs

In almost all instances the PMS process vanishes from Task Manager, as does Plex Media Scanner. Usually I’m left with only a litter of Plex Script hosts that need to be killed manually.

No crashes overnight in either server, although BearcubII was definitely processing scheduled tasks after being restarted following a full crash (same exception code as noted above.) Brainbuntu (computer) had been rebooted just before 2am, which meant Plex didn’t come back online until after the scheduled tasks start time.

I’m amused about those dmp files – accidentally got it right. I still can’t parse Microsoft’s instructions!

1 Like

The crashes were during scheduled tasks generation of video preview thumbnails. Could you disable this to see if there are any other causes for the crashes

So in Server Settings / Library settings. Set Generate Video Preview Thumbnails to Never

And as before, for any new crashes, look for the dmp files in %LocalAppData$\CrashDumps and let me have them with server logs and export of the 1000,1001 events from the Application Event Log

Thank you

As always.

Last night BearcubII went down; Brainbuntu stayed up. Very confusing.

I’m in the process of moving Audiobook and Radio libraries to the Mouse Orchestra, and the plan there is to leave that iteration of PMS shut down until whenever needed.