Server Randomly Crashes / Freezes up

[Treverr]

Server Version#: Version 1.19.4.2935
Player Version#: NA

Randomly I have noticed that my server will stop responding too all requests. I am unable to load via the website, clients, etc. If I restart it works fine for an undetermined amount of time before the issue repeats itself.

Plex Media Server Logs_2020-07-15_20-48-54.zip (4.4 MB)

[ChuckPa]

You’ve got quite a few addresses there.

Have you set one adapter as the preferred? (PMS doesn’t like multiple but will use them if it still has the primary)

Jul 15, 2020 20:48:10.320 [0x7fa8be9a1f40] DEBUG - Waiting for server condition to be signaled.
Jul 15, 2020 20:48:10.320 [0x7fa86cff9700] DEBUG - Job running: '/usr/lib/plexmediaserver/Plex Tuner Service' '/usr/lib/plexmediaserver/Resources/Tuner/Private' '/usr/lib/plexmediaserver/Resources/Tuner/Shared' '1.19.4.2935-79e214ead' '32600' '/waitmutex'
Jul 15, 2020 20:48:10.320 [0x7fa837fff700] DEBUG - HttpServer: Set up a thread pool with 2 threads.
Jul 15, 2020 20:48:10.320 [0x7fa837fff700] DEBUG - HTTP requesting GET https://plex.tv/api/v2/user/privacy?X-Plex-Token=xxxxxxxxxxxxxxxxxxxx
Jul 15, 2020 20:48:10.320 [0x7fa86cff9700] DEBUG - Jobs: Starting child process with pid 15978
Jul 15, 2020 20:48:10.320 [0x7fa8a7fff700] DEBUG - Using X-Forwarded-For: 2600:1702:3fd0:1170:a91c:2542:4aa2:b9ba as remote address
Jul 15, 2020 20:48:10.320 [0x7fa8a7fff700] DEBUG - Auth: authenticated user 15874443 as jja0538@gmail.com
Jul 15, 2020 20:48:10.320 [0x7fa86d7fa700] DEBUG - Network change for advertiser.
Jul 15, 2020 20:48:10.320 [0x7fa86d7fa700] DEBUG - NetworkService: Setting up multicast listener on 0.0.0.0:32410
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - Network change for advertiser.
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Setting up multicast listener on 0.0.0.0:32412
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Network change for browser (polled=0), closing 0 browse sockets.
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Setting up multicast listener on 0.0.0.0:32413
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Browsing on interface 127.0.0.1 on broadcast address 127.255.255.255 (index: 0)
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Browsing on interface 10.0.1.1 on broadcast address 10.0.1.255 (index: 1)
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Browsing on interface 172.23.0.1 on broadcast address 172.23.255.255 (index: 2)
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Browsing on interface 172.18.0.1 on broadcast address 172.18.255.255 (index: 3)
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Browsing on interface 172.20.0.1 on broadcast address 172.20.255.255 (index: 4)
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Browsing on interface 172.19.0.1 on broadcast address 172.19.255.255 (index: 5)
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Browsing on interface 172.22.1.1 on broadcast address 172.22.1.255 (index: 6)
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Browsing on interface 172.21.0.1 on broadcast address 172.21.255.255 (index: 7)
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Network change for browser (polled=1), closing 0 browse sockets.
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Browsing on interface 127.0.0.1 on broadcast address 127.255.255.255 (index: 0)
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Browsing on interface 10.0.1.1 on broadcast address 10.0.1.255 (index: 1)
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Browsing on interface 172.23.0.1 on broadcast address 172.23.255.255 (index: 2)
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Browsing on interface 172.18.0.1 on broadcast address 172.18.255.255 (index: 3)
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Browsing on interface 172.20.0.1 on broadcast address 172.20.255.255 (index: 4)
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Browsing on interface 172.19.0.1 on broadcast address 172.19.255.255 (index: 5)
Jul 15, 2020 20:48:10.321 [0x7fa8377fe700] DEBUG - Request: [172.21.0.2:54970 (WAN)] GET / (4 live) GZIP Signed-in Token (jja0538@gmail.com)
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Browsing on interface 172.22.1.1 on broadcast address 172.22.1.255 (index: 6)
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Browsing on interface 172.21.0.1 on broadcast address 172.21.255.255 (index: 7)
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Network change for browser (polled=0), closing 0 browse sockets.
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Setting up multicast listener on 0.0.0.0:1901
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Browsing on interface 10.0.1.1 on broadcast address 239.255.255.250 (index: 0)
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Browsing on interface 172.23.0.1 on broadcast address 239.255.255.250 (index: 1)
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Browsing on interface 172.18.0.1 on broadcast address 239.255.255.250 (index: 2)
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Browsing on interface 172.20.0.1 on broadcast address 239.255.255.250 (index: 3)
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Browsing on interface 172.19.0.1 on broadcast address 239.255.255.250 (index: 4)
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Browsing on interface 172.22.1.1 on broadcast address 239.255.255.250 (index: 5)
Jul 15, 2020 20:48:10.321 [0x7fa86d7fa700] DEBUG - NetworkService: Browsing on interface 172.21.0.1 on broadcast address 239.255.255.250 (index: 6)
Jul 15, 2020 20:48:10.321 [0x7fa8a7fff700] DEBUG - Using X-Forwarded-For: 2600:1700:2f70:a30f:7426:74f:5f5:1664 as remote address
Jul 15, 2020 20:48:10.321 [0x7fa8a7fff700] DEBUG - Auth: authenticated user 1 as Treverr
Jul 15, 2020 20:48:10.321 [0x7fa8377fe700] DEBUG - Translator: Found locale lt
Jul 15, 2020 20:48:10.321 [0x7fa8377fe700] DEBUG - Translator: Found locale af

It does find 10.0.1.1 as the default adapter for the host/VM

[Treverr]

I had it set to any, but I just set it to the eth for 10.0.1.1, the rest of them are dockers. Hopefully that helps!

Was that the only thing you saw that looked odd?

[ChuckPa]

Yes, that was the only thing I saw which stood out at me.
It’s not common to see so many network adapters.

[sa2000]

This was a deadlock where we ended up with all request handling being blocked

Last completed request was at 20:45:24

Jul 15, 2020 20:45:24.572 [0x7ff448ff5700] DEBUG - Request: [10.0.1.179:56360 (Allowed Network (Subnet))] GET /activities (175 live) TLS GZIP Signed-in Token (Treverr)
Jul 15, 2020 20:45:24.572 [0x7ff8aaffd700] DEBUG - Completed: [10.0.1.179:56360] 200 GET /activities (175 live) TLS GZIP 0ms 517 bytes (pipelined: 2)

After that all request got queued up after passing authentication

Deadlocks can only be investigated through process dumps coupled with the debug logs

This is one of several forum posts outlining how to get diagnostics for a deadlock

Random Crashing > Centos 7

It does appear to have deadlocked around 17:07 - but the load on the server was running high for quite a while and there may have been some hung requests from before that time Deadlocks need specific diagnostics - getting list of connections and process dump as well as logs. I will give the details below There has not been a recent deadlock fix released - however i think it is important to be running on the latest version available and i would recommend updating to 1.14.0.5465 One element o…