Server crashing after latest update to 1.15.0 - Logs included

[mcrommert]

Another crash at 12:30
Plex Media Server Logs_2019-02-26_09-18-37.zip (3.5 MB)

[sa2000]

@captdeemo

update… had it crash where i could not access plex on the server for 8 minute (2016-2024) 2/25 .
i’ve uploaded the plex log… i changed my username and media paths that is it.

nothing in system logs during that time… both the crash reports folders had nothing written

Have been looking at your log. This does not look like a crash.

With regards to earlier crashes on 22nd Februrary and after that overnight, were there any diagnostic reports captured by MacOS and written to

~/Library/Logs/CrashReporter/
~/Library/Logs/DiagnosticReports/

To distinguish between hang / failure to respond and a crash, look at Actiivity Monitor. If the Plex Media Server process has disappeared then it was a crash. If it is still running then it is not a crash

Going back to the log that you attached for the 25th February. The log covers period 19:44 to 20:40. You mention a crash between 20:16 and 20:24. There was no crash. The server was still running, You had 3 connections to the server.
You had two Apple TVs connected to the server. One of them appears to stopped talking to the server after 20:13 and resumed at 20:24 - this was on local IP 192.168.1.23. The other Apple TV (on 192.168.1.24) continued to playback media throughout this period. You also had an active connection to the server through 192.168.1.78

There is no indication from the server end as to why the Apple TV on 192.168.1.23 stopped sending requests to the server. This would need looking at the Plex for Apple TV log at the time to see what happened (together with server logs). See https://support.plex.tv/articles/212639598-apple-tv-logs/

Was there any other devices you attempted to connect to the server from within the local network?

[sa2000]

@mcrommert

’m getting a crash every night/early morning - saturday and sunday it was during scheduled tasks…was later today…hope the logs help

@mcrommert

Another crash at 12:30

Please could you open Windows Event Viewer by running eventvwr.exe and select Windows Logs → Application Log
Filter the log for Event IDs 1000 and 1001 (just enter 1000,1001 in the field) and then export the filtered view to evtx file and zip and attach here or send it to me by private message. Please include info on date/time of first or last filtered entry so i can calibrate the times when viewing it

Thank you

[wyldemusick]

Another day, another wipeouuuuuutttttt. 1.15.1.710 went down during the morning while running a full library scan – skipping an entire library in the process, not sure why.

Up to the point that I started the full scan, I’d finally gotten fed-up with the issues trying to redo Classical I, so it got deleted again, and the database optimized. Classical is now what used to be Classical III, and the Classical I and II directories were renamed, moved into “Classical” on E, and that moved into E:\Music on Bearcub II. A new library was then created in Plex, and named Classical 2. That’s currently populating.

Plex EV Log 02-26-2019 Bearcub II.zip (87.5 KB)

Plex Media Server Logs_2019-02-26_09-25-09.zip (5.9 MB)

Oh, and the transcoder is still thrashing at around 50%. Why is is doing that? Can you see anything in the logs that suggests there’s bad media?

[sa2000]

@wyldemusick

And, yes, I’ve gone through deleting the Gracenote cache files…twice so far. The last time I just moved them elsewhere. I’m right now doing a database optimization (due to consolidating the libraries, and having a crash gave me room to get that started. When it’s finished, I’ll shut down and do it a third time.)

Thanks for sending me the Gracenote cache databases. There may be a 20mb size limit on gracenote queries database

@wyldemusick

As to the crash, that happened while I was initially running the database optimization. I’ll go get etvx files, but here’s the logs and a text version from the event viewer (it’s a little harder from Bearcub, as it’s awkwardly set up.)

The crash on Bearcuill at 21:43:03 on 24th october appears to have been due to the database being busy and locked out during the optimize

Have you always had a 3Gb database with a million music tracks (well, 924,094 tracks) ?
Does 1.14.1.5488 handle the 3Gb database better?

[sa2000]

@bb1200x

Have you tested with standard or premium music library?
Today I did some tests again. Created a completely new folder on PC and new library in Plex, added this single album into it only. The standard music library always worked fine, the premium library always crashed the plex server with ucrtbase.dll, 0xc0000409, during library refresh and/or metadata refresh. Between the tests I always emptied trash, removed bundle, optimized database.

I did my tests with it as a new Premium Music Library and no crash.
If you have any more logs i can see if the crash is after similar gracenote search

[wyldemusick]

No full-on crash of PMS, but the Plex Media Scanner crashed at some point while doing a full scan. Not sure if then restarted or what the hell it did, but there were two iterations in Task Manager.

Plex Media Server Logs_2019-02-27_02-24-59.zip (5.1 MB)

[mcrommert]

Apologies…i meant to include that last time…Uptime Robot tells me my server went down (was uncommunicative) at 12:26 am last night…i’m including the plex logs again i downloaded plus the filtered event viewer
eventviewer.rar (93.1 KB)
Plex Media Server Logs_2019-02-27_08-06-21.zip (3.5 MB)

[wyldemusick]

Full on Plex crash at 3.45am or so (I haven’t had my morning cup of tea, so I’m a tad blurry.) I’d deleted the test “Music” library after restoring the remaining files to their original location, and the system had been busy optimizing the database when I went to bed.

BearcubII EV 02-27-2019.zip (58.2 KB)

Plex Media Server Logs_2019-02-27_08-56-20.zip (6.4 MB)

Fell down went boom at around 6pm.

BearcubII EV Log 02-17-2019 6.03pm.zip (59.2 KB)

Plex Media Server Logs_2019-02-27_18-07-21.zip (6.6 MB)

There was a Scanner crash after that, but the Scanner seems to have restarted…nope, spoke too soon, it’s Jim, dead.

Scanner Crash Bearcub EV.zip (918 Bytes)

Plex Media Server Logs_2019-02-27_20-44-52.zip (5.6 MB)

[wyldemusick]

Another Scanner crash, but it came back up again. I’m not sure precisely where it happened, certainly after Music: Electronic.

I’m also seeing weirdness in the library listings – if I sort by “Recently Added” a whole bunch of items seemingly vanish from the listing - everything in Classical/2; the album count remains steady, however. If I sort by artist, I can find some of the missing albums, but they come up incomplete. Sort by album title, though…everything seems to be there, and can be found by following the alpha sort.

Is this something a database optimization might help? Or is there a bug hiding in the code?

Edit: BearcubII crashed at 12:16am.

Plex Media Server Logs_2019-02-28_01-31-36.zip (5.8 MB)

[mcrommert]

same crash this morning - let me know if you need another set of logs

[wyldemusick]

Double whammy last night, both servers down together while running scheduled tasks from the looks of it.

Plex Brainbuntu 02-28-2019 EV.zip (1.7 KB)

Plex Media Server Logs_2019-02-28_07-56-45 Brain.zip (6.2 MB)

Plex BearcubII 02-28-2019 EV.zip (2.1 KB)

Plex Media Server Logs_2019-02-28_08-05-27 Bearcub.zip (5.8 MB)

[sa2000]

Times of crashes please and also let me know the date/time of the first entry in each of the events export

I see you launched Bearcubll at Feb 28, 2019 08:00:18 but it was running at 07:57 so that is not the crash time. Event log suggests a crash may be at 00:17 am (shows for me as 07:17 because i am at GMT time) - but oldest log is Feb 28, 2019 01:38:07 - so after the crash at 00:17 - if that was the time

Brainbuntu crashed i believe at 02:43 am

The crashes are different from the others in that they are not stack overrun 0xc0000409 but

Faulting module name: ucrtbase.DLL, version: 10.0.14393.2630, time stamp: 0x5bbec6e6
Exception code: 0x40000015
Fault offset: 0x000891fa

Do you have registry settings in windows to enable capture of user application dumps ?
see https://docs.microsoft.com/en-gb/windows/desktop/wer/collecting-user-mode-dumps

[wyldemusick]

I’ll see what I can do. There’s at least one crash event in the Brainbuntu Event Viewer capture that’s another app entirely, plus this Windows installation is kind of crashy itself (Explorer regularly dies on me) which seems to be fairly common stuff.

I’m on Arizona time here, which is offset from GMT by, what, -7?

I do notice on BearcubII that the Transcoder is no longer thrashing mightily non-stop.

It’s generally hard to know exact crash times, even with Event Viewer. It goes down when I’m asleep or away from the system, so I generally don’t see the crash for a while.

[sa2000]

@bb1200x

Today I did some tests again. Created a completely new folder on PC and new library in Plex, added this single album into it only. The standard music library always worked fine, the premium library always crashed the plex server with ucrtbase.dll, 0xc0000409, during library refresh and/or metadata refresh. Between the tests I always emptied trash, removed bundle, optimized database

Is the crash still reproducible with the music album you identified. May be i can try with your database - could you download the database and upload on dropbox etc and send me link by private message

[wyldemusick]

BearcubII - Plex Media Scanner crashed 1:44:09PM, Feb 28th 2019
BearcubII - Plex Media Server crashed 1:48:13PM, Feb 28th 2019

Both of these at Arizona, USA time.

Activity at the time of the crash: doing a scan of the media libraries. I’ve no idea how far it got – the Surfpunkgoth-etc library had the spinner next to it, but that usually seems to mean little.

Event Viewer log: Plex BearcubII 02-28-2019 EV last 01 48 13PM.zip (34.1 KB)

Plex Logs: Bearcub II Plex Media Server Logs_2019-03-01_01-19-56.zip (6.5 MB)

[sa2000]

Feedback on scanning hang whilst processing Benedictine Monks Of Silos

The PlexTranscoder.exe ended up in a loop whilst processing track 11
11 - Benedictine Monks Of Santo Domingo De Silos - Respice, Domine.mp3

This is not a new 1.15.x problem and is also present in earlier versions. It is suspected to be an ffmpeg issue when probing the file. This has been referred to the development team

[mcrommert]

Do you need more info for my crashes that are still continuing nightly or do you have what you need for the dev team?

[sa2000]

@wyldemusick

this Windows installation is kind of crashy itself

Have you gone through running this in elevated command line window when all programs shutdown?
sfc /scannow

[sa2000]

@wyldemusick

Activity at the time of the crash: doing a scan of the media libraries. I’ve no idea how far it got – the Surfpunkgoth-etc library had the spinner next to it, but that usually seems to mean little

Whilst we are trying to get to the cause of the crashes and database busy errors on 1.15.x, i would like to suggest we take out your server that is running with a 3Gb database and close to a million music tracks out of the investigation as this is not normal user database/setup and may have its own issues. So please let us just include your other server in the investigations which i believe has a reasonable size database

Do let me know if you have any non standard setup eg symlinlking app data to elsewhere