Thanks - will be looking into the logs and process dump. Any reason why the kill -SEGV 24594 was done at 13.37.25 and after a shutdown attempt at 13.37.02 ?
Removed the connections snapshot as there were tokens in there. But have secured a copy. Converting pdf content to text to analysie
Because the plexmediaserver won’t shut down when it gets into this state and requires a force kill to end the process. Everything else will shutdown just fine, but this will just sit forever. When it does not get into this state, then the process will stop and start without any issues.
@trashlyn said:
Because the plexmediaserver won’t shut down when it gets into this state and requires a force kill to end the process. Everything else will shutdown just fine, but this will just sit forever. When it does not get into this state, then the process will stop and start without any issues.
i thought you were going to try to get a process dump for the initial problem. There may be still residual info after the hung initial shutdown
What is the disk configuration. The current process when the dump was forced with -SEGV was database related
Is it possible there is a disk issue where the database is held? Is it local ? NFS mounted ? RAID ?
The disk array is 8 disks and are not running a raid config. plex runs in virtual system mounts the library on nfsv4 mount, I also tried nfsv3 mount with same issues direct attach.
Is the database issue that is showing based upon a lock and can’t access or just hanging?
I am not sure if it is an NFS locking issue - but i am aware that there has been mention of issues with it
Looking at the /connections output in conjunction -
This is the oldest request coming in at 13:20:12.304
* Sep 12, 2017 13:20:12.304 [0x7fa2efbf2700] DEBUG - Request: [174.194.3.84:3274 (WAN)] GET /:/timeline?containerKey=%2FplayQueues%2F1709&duration=257489&guid=com.plexapp.agents.plexmusic%3A%2F%2Fgracenote%2Ftrack%2F172436043-9882A7A178D3326323916B943EE00B29%2F172436044-B44EA1A038CFD990FBEE92BD5E860954%3Flang%3Den&key=%2Flibrary%2Fmetadata%2F10790&machineIdentifier=8934515556def854cfcb4f9549f7bb93dd0fa863&playQueueItemID=7055&ratingKey=10790&state=playing&time=69437&token=xxxxxxxxxxxxxxxxxxxx (16 live) TLS GZIP Signed-in Token (trashlyn)
* Sep 12, 2017 13:20:12.308 [0x7fa2efbf2700] DEBUG - Client [b1821ad41323a021-com-plexapp-android] reporting timeline state playing, progress of 69437/257489ms for guid=com.plexapp.agents.plexmusic://gracenote/track/172436043-9882A7A178D3326323916B943EE00B29/172436044-B44EA1A038CFD990FBEE92BD5E860954?lang=en, ratingKey=10790 url=, key=/library/metadata/10790, containerKey=/playQueues/1709, metadataId=10790
* Sep 12, 2017 13:21:12.702 [0x7fa2efbf2700] DEBUG - Play progress on 10790 'Disloyal Order of Water Buffaloes' - got played 69437 ms by account 1!
* Sep 12, 2017 13:23:12.943 [0x7fa2efbf2700] WARN - Took too long (0.290000 seconds) to start a transaction on ../Library/MetadataItemSetting.cpp:48
* Sep 12, 2017 13:23:12.943 [0x7fa2efbf2700] WARN - Transaction that was running was started on thread 0x7fa3043ff700 at ../Library/MetadataItemSetting.cpp:48
and it is one that hung after this
and another one close
* Sep 12, 2017 13:20:12.593 [0x7fa3043ff700] DEBUG - Request: [174.194.3.84:3272 (WAN)] GET /:/timeline?containerKey=%2FplayQueues%2F1709&duration=257489&guid=com.plexapp.agents.plexmusic%3A%2F%2Fgracenote%2Ftrack%2F172436043-9882A7A178D3326323916B943EE00B29%2F172436044-B44EA1A038CFD990FBEE92BD5E860954%3Flang%3Den&key=%2Flibrary%2Fmetadata%2F10790&machineIdentifier=8934515556def854cfcb4f9549f7bb93dd0fa863&playQueueItemID=7055&ratingKey=10790&state=playing&time=69437&token=xxxxxxxxxxxxxxxxxxxx (17 live) TLS GZIP Signed-in Token (trashlyn)
* Sep 12, 2017 13:20:12.597 [0x7fa3043ff700] DEBUG - Client [b1821ad41323a021-com-plexapp-android] reporting timeline state playing, progress of 69437/257489ms for guid=com.plexapp.agents.plexmusic://gracenote/track/172436043-9882A7A178D3326323916B943EE00B29/172436044-B44EA1A038CFD990FBEE92BD5E860954?lang=en, ratingKey=10790 url=, key=/library/metadata/10790, containerKey=/playQueues/1709, metadataId=10790
* Sep 12, 2017 13:20:12.674 [0x7fa3043ff700] DEBUG - Play progress on 10790 'Disloyal Order of Water Buffaloes' - got played 69437 ms by account 1!
* Sep 12, 2017 13:23:12.943 [0x7fa2efbf2700] WARN - Transaction that was running was started on thread 0x7fa3043ff700 at ../Library/MetadataItemSetting.cpp:48
Need to see if the dump has any left clues for these threads from before you instigated the shutdown. Will have to wait for availability of development to go through the various stacks in the dump.
I will leave it for @chuckPA to discuss NFS and whether it is wise to have the database on an NFS mount event with NFS locking enabled.
Interesting that the /connections request you made seemed to have allowed 4 stuck requests to complete just after
I have attached logs file of the most recent of when everything froze - the connections file had up to 81, but I was looking at it on my phone at the time and couldn’t capture it. However, I let this one sit for close to 30 minutes since I was travelling and couldn’t restart, it looks like it restarted itself as it came back online about 30 minutes later and started a new long file. I have attached those below.
I really would like to eliminate nfs from this and see if the hangs still arise with the database being on local drive. Don;t know how easy that would be for you to try.
cc @ChuckPA
IF using NFS v4 and local_locking=posix with a fully compliant NFSv4 server, this won’t be an issue.
There can be up to 8 milliseconds delay. While this shouldn’t be an issue, there is no harm in creating a directory under /home and using the /etc/systemd/system/plexmediaserver.service.d/override.conf specification to relocate the library there temporarily.
Doing this will allow the OS to maintain full kernel-level locks in libpthread as it is best done.
I have mounted this as a local drive, instead of going through NFSv4 - I will let this run and see how this works.
