Connections Deadlocking Server Troubleshooting

Plex Media Server
,
1

Server Version#: Version 1.40.0.7998
Player Version#: Version 4.123.2

Greetings! I’ve been running into an issue with my Plex server the past few weeks with what appears to be an overflow of stale connections to the server ends up bring the server to a halt/deadlocking.

This issue has been described before on a few threads I found but I have had no luck implementing a solution to this after many many hours troubleshooting.

The issue initially had occurred roughly 7 Months ago and seemed to “fix” itself but now is back with a vengeance.

To summarize I have found that at times the server responsiveness becomes sluggish/difficult to navigate/have artwork load. When this occurs I can also see that in the Dashboard the server will report Nothing is currently being played even if actively streaming.

From a playback side playback will start to become plagued with buffering issues/messages of connection is not fastest enough. When this occurs I have found navigating to http://10.0.1.98:32400/connections?X-Plex-Token=xxxxxxxxxxxx I can see the server being overwhelmed with what appears to be stale connections.

Typically when it’s the worst can see upwards off 150+ connections.

I have attempted all number of steps to remedy, changing Transcoding locations, increase DB Cache size, running multiple repairs of the DB with the awesome ChuckPA script but the problem always seems to return.

As a test I have stood up a 2nd Plex server to compare the results and have noticed one oddity.

On my fresh server, no running streams - 3 Connections
On my server with issues, no running streams - 16 Connections

Most of these 16 Connections seem to be related to GET /:/websockets/notifications

I have attempted to disable Push Notifications in the “General” Settings as well as disable Webhooks in “Network”

I’m really at a loss of what to try next, waiting for the server to flood it’s connection to acquire a fresh set of Debug logs to post as well.

Edit: A few other troubleshooting steps I’ve taken + additional details.

System: Unraid 6.12.6
Docker: hotio/plex
Reverse Proxy Setup through NPM.

Troubleshooting

  • Increase fs.inotify.max_user_watches
  • Set Unraid IP address to Static
  • Confirmed Preferred network interface in Network settings to be br0
  • Tested copying library.db to a new docker instance and attempting playback - Don’t see the GET /:/websockets/notifications listed. (No Sharing setup on this server)
  • Disable server support for IPV6
  • Tested with my NPM docker container disabled.
  • Optimize Database
  • Power off and leave plex container off for 4+ minutes
  • Changing Transcode temporary Directory - Both with /tmp and a HDD location.
  • User I see present in the GET /:/websockets/notifications if I remove their access it goes away but soon as I grant access back the connection is back.

Thanks!

2

Not a full deadlock but an example of what I am seeing:

This is what I’m seeing currently on http://host:32400/connections?X-Plex-Token=token with one active WAN Stream.

Attached are the Debug logs of the server while this stream is currently running
Plex Media Server Logs_2024-03-13_17-38-18.zip (3.1 MB)

Refreshing the connections page shows typically a minimum of 17 connections with varying degrees of the Pending Request - Paused as well as GET /:/websockets/notifications

I’ve also spun up a 2nd server instance - no access granted and basic setup.
Starting one stream on a LAN connection see the following on the connections page.

3

Fortunately have not seen the complete deadlock the past night. I do still see however a number of /:/websockets/notifications connections even when no active streams. Wondering if

A) What is the /:/websockets/notifications actually mean and pertain to?
B) Should I be concerned even or is this normal behavior to see those lingering?

4

Attaching logs of lockup tonight/just after the lockup. Connections were up to 286 and couldn’t access web interface so restarted container and downloaded logs, hopefully that’s not going to impact them.
I also have a txt output of 286 connections but would rather not post here so can direct message if helpful. Thanks
Plex Media Server Logs_2024-03-15_20-23-11.zip (5.1 MB)

Following the restart connections were back down to roughly 30, could access webui, 15 minutes later connections back over 180, webui locked up and dashboard showing “Nothing is currently being played”

Typically the dashboard stops working once it has exceeded ~70 connections.

5

Another 345 Connections and Deadlocked this afternoon. I’ve grabbed a copy of the output of the Connections site as well as uploading another copy of the logs following the lockup.
Plex Media Server Logs_2024-03-16_14-35-07.zip (6.1 MB)
Unsure about sensitive information being present in the connections logs but can certainly provide as well!

Required a restart to get to the troubleshoot download log so hopefully again doesn’t impact the integrity of the logs.

Edit - In addition I’ve tried during off all other docker containers running on the system. Thought here was able something like Tautulli was interacting and causing an issue but no change if off/running.

Thanks!

6

The issue appears to be arising because some requests are taking too long to complete. I have recently identified a problem with one of our DB queries relating to Push Notifications. This is being fixed in 1.40.2

Could you try with Push Notifications disabled in Server Settings and see if the problem goes away

This is an example of a request taking 382 seconds to complete instead of few milliseconds

Mar 16, 2024 14:34:40.613 [22427019443000] DEBUG - Completed: [xxx.xx.xx.xx:39324] 200 GET /:/timeline?ratingKey=495827&key=%2Flibrary%2Fmetadata%2F495827&playbackTime=1097794&playQueueItemID=1075173&state=playing&hasMDE=1&context=source%3Acontent.collections&row=0&col=0&time=1096000&duration=7987000 (401 live) #2465fb GZIP 382764ms 892 bytes (pipelined: 2)
7

Thanks sa2000 for the follow up, greatly appreciated! I’ll test further with Push notifications turned off and follow up. Thanks again,.

8

Traveling last week so have not been able to keep as close an eye on Connections but last week after the suggested Disabling Push Notifications, but so far things do seem to be very stable so far.

I still see a number of /:/websockets/notifications from inactive clients but the number doesn’t seem to be growing/impacting playback. Had a number of active streams running and never did see the Dashboard going blank etc that was typical when connections would grow out of control.

Hopefully optimistic right now and will be keeping a closer eye on things this week and report back if the issues continues to behave itself or come back. Thanks again!

9

We have released the fix for the slow GET /:/timeline requests when Push Notifications is enabled - Plex Media Server Beta 1.40.2.8273

  • (Push Notifications) Used expensive DB query during playback progress notifications (PM-1166)

You could re-enable the Push Notifications after updating to the beta.

10

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.