Myriad of problems in past month: Mac restarting, plex crashing, database corruption...

server-mac

#1

I've been using Plex since the beginning, but the past month has me pulling out my gray hair.

It started last month when my mini appeared to be randomly restarting several times a day. After purchasing a new mac and migrating Plex, the problem remained. It also remaining after a fresh Plex install (after media loaded). I finally traced (what I thought was) the problem down to a corrupt movie file that a remote user kept trying to play. When this file was played (including played locally on the server Mac), it caused the entire system to restart. I deleted the file and thought all was well.

Now I'm getting similar problems but not traceable to any particular file. My mini still restarts or turns off completely randomly throughout the day (same after PMU and PRAM resetting and repairing startup drive in safe mode). And/or Plex home screen won't load, but the TV/Movies libraries are still accessible. Typically, at some point during the scanning of the files, the system will crash. Or, it may just corrupt the database, and I have to repair it to get plex to load again. Etracheck finds nothing wrong with the Mac. And this is AFTER completely deleting all traces of Plex and re-setting up my libraries.

Yesterday morning, I woke up to plex media server being closed, but Mac still on. I tried opening and it just quits immediately. After repairing the database, PMS stays open, but the web interface isn't reachable ("You must be lost"). So, I restored from the one database backup I had created at 2AM, now it's open again. The TV library works, but the Movies library is blank (it should have hundreds of movies inside). I removed and re-added a few sources from the Movies library and that seems to have got it back online.

Last night more and more problems (plex crashing/restarting entire computer, or just powering off completely). It would restart or turn off my Mac about 1 minute after starting PMS. I left mini running last night with PMS closed, and the computer has been fine. This morning, trying to get PMS back up and it just quits immediately after opening. I'm restoring from database backup I made 2 days ago (which is also corrupt, but repair works).

So, back up for right now, but I know it will become corrupt and cause Mac to restart or turn off again soon. I'm really at a loss now and have no idea why PMS is having so many problems.

Any/all ideas/suggestions welcome.

https://dl.dropboxusercontent.com/u/317958/Plex%20Media%20Server%20Logs_2017-07-12_10-30-33.zip
https://dl.dropboxusercontent.com/u/317958/Plex%20Media%20Server%20Databases_2017-07-12_10-30-47.zip

PMS Version 1.7.5.4035
Mac 10.12.5 / Mac Mini Late 2014


#2

very strange that problems continued after switching machines

Is the disc drive where the Plex Media Server database is on an internal HDD ?
'''
/Users/cometa/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db
'''
The latest corruption probably happened after Jul 11, 2017 23:30:52 because a scanner log stopped having log entries then

Is there a Mac OSX event log that you could browse to see what was happening in terms of system / application logs between 23:30 and 10:25 in the morning when the corruption was stopping PMS launching ?

Are you using any symlinks to have the files elsewhere?
Or running some tools that access the database?

Do you have any logs of previous occurrences ?


#3

I am running the same versions without issue.

I would suggest a clean install of PMS, remove everything. As you stated it has been working well for a long time. Key word "long". I no its a huge leap of faith.

If not, your logs are all found in Console in Utilities. It may well be a compatibility issue with an app that runs in the back ground. Also I would recommend to try Etrecheck a free app that runs heaps of tailored tests on your computer. It may well be some simple.

https://www.etrecheck.com/


#4

You said that etrecheck didn't find anything wrong, have you also used malwarebytes just in case?
If I'm not wrong Plex is self contained, you can try to run the Mac in safe mode. Restart while keeping shift down, it will take a little longer to reach the Finder and will load only the bare minimum extensions to run with network. That way you may find a conflicting extension, login item or similar. If this works check the etrecheck report on all startup items, deamons, etc.
Have you an older backup from before the corruption? if not as Gdr56 said you may need a fresh install of the libraries.

P.D. I asume you already checked the SMART state of your disc and its ok


#5

@macgen77 said:
You said that etrecheck didn't find anything wrong, have you also used malwarebytes just in case?
If I'm not wrong Plex is self contained, you can try to run the Mac in safe mode. Restart while keeping shift down, it will take a little longer to reach the Finder and will load only the bare minimum extensions to run with network. That way you may find a conflicting extension, login item or similar. If this works check the etrecheck report on all startup items, deamons, etc.
Have you an older backup from before the corruption? if not as Gdr56 said you may need a fresh install of the libraries.

P.D. I asume you already checked the SMART state of your disc and its ok

Just to clarify, I'm not suggesting a removal of your Titles Library. Just a clean install of PMS only.


#6

Hi Guys,

Thanks for your responses. Sorry for the late reply, I’m not getting notifications for some reason. Anyhow, after starting over from scratch a couple times, consolidating and upgrading my hard drive enclosures to thunderbolt, and checking/repairing all drives including System drive via Safe Mode startup, things seem to be working for the most part now. *This is without any remote users yet.

The only remaining issue now is another strange one. A small portion of my Library (only in “Movies”) I’m getting blank thumbnails and “Loading…” names in a chunk of the letter “J” section. This is after checking/repairing hard drives (only one had minor corruption), optimizing database, cleaning bundles, emptying trash, and manually backing up and restoring database corruption in Terminal (even though Terminal said database was “OK” to begin with), and re-scanning metadata.

Even though this is a minor issue, I can’t help but think some kind of lingering corruption is present. This problem happens on all devices…

**Plex/Web Grid
**

**…and List view
**

**iPhone (when scroller over “J”)
**

AppleTV
Says: “Nothing to see here” when scroller over “J”


#7

I guess no one has seen this issue before. :(


#8

@bcometa said:
. . . checking/repairing all drives including System drive via Safe Mode startup,

What "repair" action took place?

@bcometa said:
This is after checking/repairing hard drives (only one had minor corruption),

What was the "minor corruption" ?

@bcometa said:
manually backing up and restoring database corruption in Terminal (even though Terminal said database was "OK" to begin with),

Data corruptions do not always give rise to structural corruptions that would show up in a database integrity check. Corruptions could affect values and as a result would not show in any database integrity check

There is a possibility that you continued with a corrupt database after switching systems.

Nothing conclusive but is a possibility


#9

"What "repair" action took place?"

Starting up in safe mode, which runs fsck on the drive.

"What was the "minor corruption" ?"

Using DiskWarrior, just minor volume header damage or something similar, nothing that had any effect.

"There is a possibility that you continued with a corrupt database after switching systems."

I've reverted back to the original system. But, started Plex installation from scratch (including all libraries and database). Is there a solution to repair "invisible" database corruption without starting fresh again?


#10

@bcometa said:
I've reverted back to the original system. But, started Plex installation from scratch (including all libraries and database). Is there a solution to repair "invisible" database corruption without starting fresh again?

You could explore use of tools to export data out and have a look at it to see if you spot what is wrong
eg using tools such as http://forums.plex.tv/discussion/254248/rel-exporttools/p1
There is also http://forums.plex.tv/discussion/120292/moviefans-database-tool-for-plex/p1


#11

Ok, moviefans is PC only, and I'm on Macs. But, I tried exporttools and exported Movies Library. It fails/crashes and log shows it stops scanning once hitting the "J" section of Movies Library (as expected, I suppose). But, I still can't figure out why or how to fix (and I tried deleting several "J" movies around/right after the spot where it stopped). Hum.


#12

@bcometa said:
Ok, moviefans is PC only, and I'm on Macs. But, I tried exporttools and exported Movies Library. It fails/crashes and log shows it stops scanning once hitting the "J" section of Movies Library (as expected, I suppose). But, I still can't figure out why or how to fix (and I tried deleting several "J" movies around/right after the spot where it stopped). Hum.

This would be like having a hard disk crash and then only option you have is to take it to data recovery specialist to see what can be salvaged and the outcome is unknown

I do not think it would be straightforward to establish what is wrong with the database and there is always going to be the unknown unknowns !

If there has been a disk problem and corruption ensued then the scale of corruption would not be known and my personal advice is start afresh with a new empty database on a good disk and re scan all the media. You will lose the watched status and may be the export tools can help you identify what has been watched

Hopefully the problem won;t come back after that


#13

Hehe, just so happens that I am a data recovery specialist (my business is: 300ddr.com)!

The only problem is: I already tried what you're suggesting and this is where I ended up. Also, as far as I can tell, none of my drives have any problems (nor does my Mac). I guess I'll run Memtest for a while to confirm RAM is good.

My Mac is restarting more often again (when I mess with library, scan/cancel scan, etc), so I think things are getting worse again and restarting fresh may be the only option.

Thanks for your help sa2000.


#14

@bcometa said:
My Mac is restarting more often again (when I mess with library, scan/cancel scan, etc), so I think things are getting worse again and restarting fresh may be the only option.

Just checked your servers crash reports (for Plex-BC) and I can only see Plex Media Scanner crashes which I suspect are probably on media analysis in scheduled tasks - a problem that is expected ito be fixed in an ffmpeg change coming along with the next release (after 1.7.6)

The ffmpeg issue causes memory corruptions - i am not sure if it can affect anything beyond the scanner or transcoder processes


#15

makes one think - since you were mentioning memory


#16

Yes, it always seems to crash during a scan.


#17

It is only the scanner process that fails - PMS should continue to run


#18

Well... when it crashes, most often the entire computer restarts.


#19

That is interesting as in theory it ought to be only the failing process - but may be there is no protection and memory corruptions go beyond the process


#20

At the risk of jinxing myself, I believe I have found the problem: one of my 20 drives had a few bad sectors. I've cloned its files to a new drive, started fresh Plex database, and at least so far, no restarts/crashes.