Plex Media Server Stops After Upgrade To Either 1.15.0.647 or 1.15.0.659

Plex Media Server Desktops & Laptops
21

Just would like to say that the only evidence of database corruption was in the logs provided here in the first post Plex Media Server Stops After Upgrade To Either 1.15.0.647 or 1.15.0.659 and the evidence there showed the corruption existed when running 1.14.1.5488 as well

Feb 07, 2019 14:19:36.573 [2216] INFO - Plex Media Server v1.14.1.5488-cc260c476 - Microsoft PC x64 - build: windows-i386 english - GMT -08:00
Feb 07, 2019 14:19:36.574 [2216] INFO - Windows version: 6.1 (Build 7601), language en-US
Feb 07, 2019 14:19:36.574 [2216] INFO - 4 3399 MHz processor(s): Architecture=0, Level=6, Revision=10759 Processor Identifier=Intel64 Family 6 Model 42 Stepping 7, GenuineIntel

Feb 07, 2019 14:19:36.646 [11732] INFO - SQLITE3:0x41e02417, 283, recovered 22 frames from WAL file D:\documents\Plex Media Server\Plug-in Support\Databases\com.plexapp.plugins.library.db-wal
Feb 07, 2019 14:19:36.809 [11732] INFO - SQLITE3:0x41e02417, 283, recovered 2 frames from WAL file D:\documents\Plex Media Server\Plug-in Support\Databases\com.plexapp.plugins.library.blobs.db-wal

Feb 07, 2019 14:19:42.871 [6756] INFO - SQLITE3:0x41e02417, 283, recovered 456 frames from WAL file D:\documents\Plex Media Server\Plug-in Support\Databases\tv.plex.providers.epg.onconnect-945a10e4-1f31-4765-b066-9e63d43a94f8.db-wal
Feb 07, 2019 14:19:42.872 [6756] ERROR - SQLITE3:0x41e02417, 11, database corruption at line 112037 of [fc49f556e4]
Feb 07, 2019 14:19:42.872 [6756] ERROR - SQLITE3:0x41e02417, 11, database corruption at line 112037 of [fc49f556e4]
Feb 07, 2019 14:19:42.873 [6756] ERROR - SQLITE3:0x41e02417, 11, database corruption at line 59437 of [fc49f556e4]
Feb 07, 2019 14:19:42.873 [6756] ERROR - SQLITE3:0x41e02417, 11, database corruption at line 59479 of [fc49f556e4]

Point to note is that pre-existing database corruptions would allow the server to run - perhaps crippled a bit with some functionality not working - but when upgrading versions of Plex Media Server, these pre-existing corruptions become a show stopper because the upgrade may include a database schema change and that to complete needs to access every part of the database and pre-existing corruptions would stop this and result in server failure

No logs have been provided here after the faulty database was replaced with a new one. So new failures and failures other users have may be different and they will not resolved or investigated if no debug logs are provided with debug logging enabled beforehand and the logs captured after the restart following the crash

See
https://support.plex.tv/articles/201643703-reporting-issues-with-plex-media-server/
https://support.plex.tv/articles/200250417-plex-media-server-log-files/

22

@sa2000: Thank you for pointing out some errors I made in first reporting this issue. I had not enabled debug logging, didn’t gather logs immediately after the updated PMS failed, and didn’t repeat the test after repairing the db.

To correct this, I have repeated my reproduction of the problem, following all the steps on the “Reporting issues with Plex Media Server” page and gathering the debug logs. I have then repaired my database and repeated reproduction of the problem again, gathering a new set of debug logs. See the end of this post for the logs.

I should note that due to the nature of the problem (PMS won’t stay running after performing the update), I was unable to use the “Download Logs” button under Settings > Server > Troubleshooting in the Plex Web App to gather the logs immediately after the problem ocurred. Instead, I gathered the logs manually, following the steps on the “Plex Media Server Log Files” page.

I notice that the PMS log continues to list the same four database corruption errors even after I’ve repaired my database following the instruction on https://support.plex.tv/articles/201100678-repair-a-corrupt-database/.

At this point, I don’t know what else I can do to resolve the database corruption errors PMS is reporting. As I said earlier, I am loath to start with a fresh database and lose all of the custom metadata I have entered into my library over the years. I have found that I can use the Export Tools plug-in ([REL] ExportTools) to at least export my metadata, but I have not found a way to programmatically re-import that metadata and it would take countless hours to manually re-enter it (or even copy/paste it).

The resulting debug logs are too large for me to upload here, so I am attaching only the PMS server logs:

Plex Media Server Log Piror To DB Repair.log (37.8 KB)

Plex Media Server Log After DB Repair.log (40.9 KB)

I have placed the complete debug logs in a public OneDrive folder here:

https://1drv.ms/f/s!An_wAcNHcCxJg6trNHlMTQ8w2Katqg

23

There is no longer any database corruption reported for the main database

The only corruption i can see and with debug logging enabled, it is clear, is the EPG database
D:\documents\Plex Media Server\Plug-in Support\Databases\tv.plex.providers.epg.onconnect-945a10e4-1f31-4765-b066-9e63d43a94f8.db

If you delete all db files for this (.db, .db-wal, .db-shm) it should get recreated with a fresh EPG on restart

There is evidence of potentially some slow responses from plex.tv but each log shows you are not really giving it a chance to establish that - for example 1.15.0.659 was launched at Feb 10, 2019 14:44:33 and I think you captured the logs 8 seconds later at Feb 10, 2019 14:44:41 - please confirm when the logs were captured.

If there is a hang i would like to know how long it hung for - requests should timeout after few minutes and there is no evidence of any timeouts

If we establish there is a hang then will need to get process dumps to go with the logs and also, before that, the output from http://127.0.0.1:32400/connections?X-Plex-Token=xxxxxxxxxxxx taken and the time noted. You would need to add the server token here - see https://support.plex.tv/articles/204059436-finding-an-authentication-token-x-plex-token/

Are you running any proxy software ? can you check Internet Options / Connections / LAN Settings / Proxy server

24

Further to my earlier response, i can now see that the logs were captured at least 2 minutes after what appears to have got hung up at 14:44

I can now see a request made by one of the PlexScriptHost.exe processes to Plex Media Server at 14:46:37 which did not get seen by the server and timedout

2019-02-10 14:46:37,631 (32c8) :  DEBUG (networking:166) - Requesting 'http://127.0.0.1:32400/servers'

2019-02-10 14:46:57,631 (32c8) :  CRITICAL (core:574) - Exception in thread named 'refresh_servers' (most recent call last):
timeout: timed out

What windows version / release is this? There were issues with winsock with Windows Server 2008 and Windows Server 2012 which were fixed in the R2 releases

When the hang occurs, does the Plex Media Server.exe show up in windows task manager?

25

Could we try one more time - leaving each session for 15 minutes

Start with 1.15.0.659 or 1.14.1.5488 and run for 15 minutes and capture logs
(if getting logs manually exclude the /installer logs directory

then upgrade or downgrade as appropriate and run for 15 minutes and capture logs

If there is a hang then get the following
http://127.0.0.1:32400/connections?X-Plex-Token=xxxxxxxxx output and copy to text file and note its time
Then get process dump - see windows section here https://support.plex.tv/articles/201373203-gather-a-process-dump-or-sample-process/
Then get the logs

26

@sa2000: Thanks much for taking the time to look through my logs and also for your detailed responses.

I was glad to hear that the main database was not the one that PMS was finding corrupt. As you suggested, I deleted the EPG database files and let PMS create fresh ones.

I don’t think your concerns about the cut off logs was due to prematurely capturing them. After upgrading, PMS was quitting entirely, not hanging. PMS was therefore probably not generating log entries after that.

Nevertheless, I set out to try a new capture of the logs as you requested. To my surprise, this time after upgrading to PMS 1.15.0.659, the server did not quit at all. It is still running as I write this some 30 minutes later. The problem seems to be resolved. Since the only change I made was to delete and build a fresh EPG database, I am thinking that the corrupt EPG database might have been the source of the problem from the beginning.

When submitting issues here in the future, I will certainly be more careful about submitting logs with debug logging enabled. If I had done that in the first place I could have perhaps avoided going down the “main database is corrupted” path. Ah well, at least I learned a lot about Plex’s databases.

EDIT: Here is the PMS server log that was generated once PMS started again after the update:

Plex Media Server.log (334.3 KB)

27

I believe that is what it was. The dmp file you provided for Feb 7 crash at 14:33 was for a fatal exit in soci_sqlite3.dll Normally EPG db issues would not bring PMS down. I will mention this to the development team

Thanks for the detailed feedback

28

Hi

Had the same issue on Windows 10 Home. Tried following:

  • Roll-back to Version 1.14.x didnt’do the trick.
  • Deleted the database and regenerated a new one from PMS. Didn’t Work.
  • Ran a Window’s Update, and now it work’s.
  • Ran PMS upgrade to 1.15, and still it work’s.

So conclusion must be that it where the Windows upgrade that did the trick for me. But I cant rule out that the rebuild of the database, has done some part of it.

Best regards
Brian

29

I’m having the same issues as well since the same 2 patches mentioned in the Topic.

I tried a couple of things that I’ve read others have mentioned, but to no effect.

TRIED:
Disabled hardware acceleration under transcoding
Checked my Windows Task Scheduler - nothing scheduled
Checked for Windows Updates (none)
Checked my drive spaces - plenty
Checked SSL Connections only setting - nope

Looking in the Tautulli logs, I saw that it started happening around 1AM, so I checked my server settings and saw that the Plex Scheduled Tasks were scheduled to begin at 1AM. To test my theory, I changed the Scheduled Tasks from 1AM to 4AM and the next morning, that’s when the Tautulli logs show the server started having problems due to being shut down
so it must be something happening during maintenance that’s causing it, at least for me.

Tautulli Error Msg:
2019-02-13 04:08:39 WARNING Tautulli Pmsconnect :: Unable to parse XML for get_server_response: ‘NoneType’ object has no attribute ‘getElementsByTagName’.
2019-02-13 04:08:39 WARNING Failed to access uri endpoint /myplex/account. Is your server maybe accepting SSL connections only? HTTPConnectionPool(host=‘192.168.1.8’, port=32400): Max retries exceeded with url: /myplex/account (Caused by ReadTimeoutError(“HTTPConnectionPool(host=‘192.168.1.8’, port=32400): Read timed out. (read timeout=15)”,))
2019-02-13 04:07:39 WARNING Failed to access uri endpoint /myplex/refreshReachability. Is your server maybe accepting SSL connections only? HTTPConnectionPool(host=‘192.168.1.8’, port=32400): Max retries exceeded with url: /myplex/refreshReachability (Caused by ReadTimeoutError(“HTTPConnectionPool(host=‘192.168.1.8’, port=32400): Read timed out. (read timeout=15)”,))

I grabbed my server log files before I restarted this morning and here are the only 3 files in that directory that are timestamped at 4:06 AM.

Plex DLNA Server.log (1.3 KB)
Plex Media Server.log (3.6 MB)
Plex Tuner Service.log (2.7 KB)

Any help would be appreciated. I can run more tests if needed - just give me detailed instructions.

Thanks,
Zer0

30

I’m having basically the same issue but on Ubuntu:

I have reverted to 1.14 for the time being. Hoping we can figure out a way to repair whatever is wrong with the database so I can continue upgrading.

32

One of your log files Plex Media Server Log After DB Repair.log is for a different server and different account and is one i already looked at in different forum post.

could you explain?

33

Sorry - that was someone else’s log file I accidentally clicked on while researching this issue. Nevermind that one. :slight_smile:

34

Could i have the previous log files as well. Would be good if they go back to when the server was launched. But any previous logs would help understand the issue better

35

always best to zip the logs using the web interface for download logs

36

This is all a miscommunication on my part. The one file I added today (Plex Media Server Log After DB Repair.log) was from someone else’s log file I downloaded to look at for research purposes.

Mine were from 2/13. I’ll removed the wrong file from my post.

My apologies for the confusion, but please take a look at the other 3 logs in my post to see if it helps with my nightly shutdown during maintenance.

37

I am going to delete my initial reply

38

The crucial question is was Plex Media Server.exe showing in Windows Task Manager or had the process crashed out after 04:06 am

Can you check the Windows Application Event Log using eventvwr.exe

39

I have seen a case which looks similar. I suspect that you will find an exception entry in the event log for around 4:06 am on 13 Feb.

I had a theory for the area we were crashing in.
Could you tell me what you have set for this server setting
Library / Generate Video Preview Thumbnails

40

Yes, at 2/13/2019 4:06 AM there is an Application Error in the Event Log - Event ID 1000 - Task Category (100)

Faulting application name: Plex Media Server.exe, version: 1.15.0.659, time stamp: 0x5c5c2814
Faulting module name: ucrtbase.dll, version: 10.0.17763.1, time stamp: 0xc3ae1887
Exception code: 0xc0000409
Fault offset: 0x000a5d0b
Faulting process id: 0x387b0
Faulting application start time: 0x01d4c2d1d12bcd77
Faulting application path: D:\Plex Media Server\Plex Media Server.exe
Faulting module path: C:\WINDOWS\System32\ucrtbase.dll
Report Id: 25f35980-4f19-4fbb-b94f-853e2dafc447
Faulting package full name:
Faulting package-relative application ID:

Under Details it says:

  • System

    • Provider
      [ Name] Application Error
    • EventID 1000
      [ Qualifiers] 0
      Level 2
      Task 100
      Keywords 0x80000000000000
    • TimeCreated
      [ SystemTime] 2019-02-13T09:06:06.716767200Z
      EventRecordID 1762
      Channel Application
      Computer HOMEPC
      Security

  • EventData
    Plex Media Server.exe
    1.15.0.659
    5c5c2814
    ucrtbase.dll
    10.0.17763.1
    c3ae1887
    c0000409
    000a5d0b
    387b0
    01d4c2d1d12bcd77
    D:\Plex Media Server\Plex Media Server.exe
    C:\WINDOWS\System32\ucrtbase.dll
    25f35980-4f19-4fbb-b94f-853e2dafc447

This is also an Information for Windows Error Reporting above this that says:

Fault bucket 1758766396513382936, type 5
Event Name: BEX
Response: Not available
Cab Id: 0

Problem signature:
P1: Plex Media Server.exe
P2: 1.15.0.659
P3: 5c5c2814
P4: ucrtbase.dll
P5: 10.0.17763.1
P6: c3ae1887
P7: 000a5d0b
P8: c0000409
P9: 00000007
P10:

Attached files:
\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER19F3.tmp.dmp
\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER1DDC.tmp.WERInternalMetadata.xml
\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER1DFC.tmp.xml
\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER1DFA.tmp.csv
\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER1E2A.tmp.txt

These files may be available here:
\?\C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_Plex Media Serve_349f73a67d629cae5c0095ded57f227aa726b_73f1b63c_241120d8

Analysis symbol:
Rechecking for solution: 0
Report Id: 25f35980-4f19-4fbb-b94f-853e2dafc447
Report Status: 268435456
Hashed bucket: 822ca44c80f9500f286864b82312f218
Cab Guid: 0

Under Details, is says:

  • System

    • Provider
      [ Name] Windows Error Reporting
    • EventID 1001
      [ Qualifiers] 0
      Level 4
      Task 0
      Keywords 0x80000000000000
    • TimeCreated
      [ SystemTime] 2019-02-13T09:06:09.468987700Z
      EventRecordID 1763
      Channel Application
      Computer HOMEPC
      Security

  • EventData
    1758766396513382936
    5
    BEX
    Not available
    0
    Plex Media Server.exe
    1.15.0.659
    5c5c2814
    ucrtbase.dll
    10.0.17763.1
    c3ae1887
    000a5d0b
    c0000409
    00000007

    \?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER19F3.tmp.dmp \?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER1DDC.tmp.WERInternalMetadata.xml \?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER1DFC.tmp.xml \?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER1DFA.tmp.csv \?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER1E2A.tmp.txt
    \?\C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_Plex Media Serve_349f73a67d629cae5c0095ded57f227aa726b_73f1b63c_241120d8

    0
    25f35980-4f19-4fbb-b94f-853e2dafc447
    268435456
    822ca44c80f9500f286864b82312f218
    0

41

“as a scheduled task”