"Server settings are unavailable" - bug found and fixed

Plex Media Server Desktops & Laptops
1

Server Version#: v1.19.3.2843-e3c1f7bcd
Player Version#: Plex Web Player
Operating System: Windows 10 Pro version 1903 build 18362.836

Like many others, I have been suffering from the “Server settings are unavailable” problem for several months. On top of that, The Plex Media Server and its child processes run at about 70% CPU constantly.

I decided to get to the bottom of things today. I searched the forums for similar problems and tried all the recommended solutions. Nothing helped.

I also completely uninstalled Plex, renamed my “AppData\Local\Plex Media Server” directory, and performed a clean install using a newly downloaded installer. The problems persisted even with this completely clean installation and with Windows Firewall turned off. If I took the extra step of renaming the “HKCU\Software\Plex, Inc.\Plex Media Server” key, I would reach a page stating that “A problem has been detected with a core component of Plex Media Server.”

Looking through my logs, I noticed many occurrences of the error “Timeout or error reading status line from plug-in pipe [com.plexapp.system], we’re killing it.” (My logs are attached. I had to create them manually since I can’t get to the relevant server settings page.)

Logs.zip (2.8 MB)

To debug, I decided to run the system plugin manually from the command line, using the command:

“C:\Program Files (x86)\Plex\Plex Media Server\PlexScriptHost.exe” “C:\Program Files (x86)\Plex\Plex Media Server\Resources\Plug-ins-219a9974e\Framework.bundle\Contents\Resources\Versions\2\Python\bootstrap.py” “C:\Program Files (x86)\Plex\Plex Media Server\Resources\Plug-ins-219a9974e\System.bundle”

No errors were displayed, but the process immediately started using 17% of CPU. I pressed Ctrl-C and got a stack trace, which I am attaching to this post. I noticed that, no matter how long I waited before pressing Ctrl-C, I always got this same stack trace.

plex-stack-trace.txt (3.0 KB)

My next step was to look at the runtimekit.py file and find the line that was executing. The line in question appears within a “while True” loop. The developer who wrote the code must be expecting the loop to break with an exception; but on my computer it appears to run in an infinite loop.

Of course, the system plugin is not the only one that suffers this fate. Several other plugins also get launched, hit this infinite loop, and pull 17% CPU each. When they don’t respond Plex kills and restarts them, every few minutes.

Looking around, it appears that this method is just part of a scan to see whether I have Flash or Silverlight installed. Since I don’t, I decided I could avoid the entire problem by adding a “return” statement on the first line of the _add_win_mimetypes method.

After saving that change, everything is working properly for me again. Server settings are now available, Plex isn’t using any noticeable CPU, and all is right with the world.

3 Likes
2

From a quick search of the forum, it seems like this bug might be affecting a number of people. If the Plex Developers are monitoring this forum, please feel free to reach out to me. I’m a software developer myself and I’ll be glad to help with any troubleshooting/debugging.

4

Server being unavailable may be due to different reasons and what you got may not be related to what other users may have encountered.

The issue in your case is that Plex Media Server process spawned a process for the com.plexapp.system and it then send a command to it through a pipe and it got no answer back

May 25, 2020 13:55:50.263 [19420] DEBUG - Plugin::createProcess: Creating plugin process: "C:\ProgramData\Plex\Plex Media Server\PlexScriptHost.exe" "C:\ProgramData\Plex\Plex Media Server\Resources\Plug-ins-e3c1f7bcd\Framework.bundle\Contents\Resources\Versions\2\Python\bootstrap.py" "C:\ProgramData\Plex\Plex Media Server\Resources\Plug-ins-e3c1f7bcd\System.bundle"
May 25, 2020 13:55:50.285 [19420] DEBUG - Spawned plug-in com.plexapp.system with PID 19436

May 25, 2020 13:55:50.285 [19420] DEBUG - [com.plexapp.system] Sending command: GET /:/prefixes

This is the issue that needs to be looked into

I do see that you have a non standard installation where you decided to install Plex Media Server binaries into C:\ProgramData rather than the windows default of C:\Program Files (x86)

These directories are special and specific windows permissions and attributes and are used for different purpose

It is possible that putting binaries in C:\ProgramData was a factor to the issues that you encountered.

Secondly I do not think it is appropriate to post in the forum edits to the Plex software - so I suggest you remove your post that advises making edits to the python module you identified.

Please uninstall Plex Media Server and make sure the C:\ProgramData\Plex area is empty and the Plex directory removed and then install the current version using the default path of “C:\Program Files (x86)” and if you still have a problem - please post fresh logs again

5

Thank you very much for your response. I have removed my post containing the edit that fixed the bug. I agree that regular users shouldn’t be editing python scripts. I hope the Plex Dev Team can fix the problem for everyone.

Server settings being unavailable may be due to different reasons and what you got may not be related to what other users may have encountered.

I agree. The forum search that seems most instructive is for people reporting constant high CPU usage (16 or 17% nonstop) from PlexScriptHost.exe, even when Plex is completely idle.

It is possible that putting binaries in C:\ProgramData was a factor to the issues that you encountered.

I’m happy to confirm that this was not the problem. I spent two full days troubleshooting, and I tried C:\ProgramData as one of my experiments to see if the problem was related to Windows UAC. But changing the installation directory had no effect at all. Here is a set of logs from a clean installation to C:\Program Files (x86), as you’ve requested. It exhibited the exact same behavior:

Logs.zip (4.4 MB)

The issue in your case is that Plex Media Server process spawned a process for the com.plexapp.system and it then send a command to it through a pipe and it got no answer back. This is the issue that needs to be looked into

Yes, that’s correct. Fortunately, I was able to identify why Plex Media Server got no answer back from the subprocess. Take a look at the stack trace I included in my original post: the runtimekit.py file is getting stuck in an infinite loop on line 389, in the _get_registry_subkeys method, as it calls _winreg.EnumKey.

I can’t guess why that call to _winreg.EnumKey would run in an infinite loop on my computer and not others. (Perhaps there’s some unusual data pattern in my Windows registry?) But I’ll be happy to work with a Plex Developer to troubleshoot.

6

Will have a look - would be best to add logging of each time going round to see what key is being appended

It should come out of the loop once the list of keys is exhausted. may be it is a long list and eventually does come out but goes back into it later

Again logging start of loop (when i = 0) would help

Anything logged in windows event logs about any registry corruption?

7

would be best to add logging of each time going round to see what key is being appended

That’s a great idea. As requested, I added some debugging code to the _get_registry_subkeys method. For clarity, here’s what my debug-enhanced method looks like:

  def _get_registry_subkeys(self, key, subkey):
    try:
      import _winreg
    except ImportError:
      return None, []

    handle = None
    keys = []
    try:
      flags = _winreg.KEY_ENUMERATE_SUB_KEYS | _winreg.KEY_WOW64_32KEY
      self._core.log.debug('Calling OpenKey(%s, %s, 0, %d)', key, subkey, flags)
      handle = _winreg.OpenKey(key, subkey, 0, flags)
      i = 0
      while i < 10000:
        onekey = _winreg.EnumKey(handle, i)
        self._core.log.debug('EnumKey returned %s', onekey)
        keys.append(onekey)
        i += 1
    except WindowsError:
      pass
    return handle, keys

In addition to the debug statements, you’ll notice that I changed the while loop from while True: to while i < 10000:. (This is very important - I’ll explain why in a moment.) With those debug statements in place, I started Plex Media Server, waited a few minutes, and stopped it again. Here are the log files that were generated by the com.plexapp.system plugin:

System Plugin Logs.zip (123.0 KB)

As you can see, it seems to be finding an enormous number of subkeys under the @java.com/JavaPlugin,version=12.0.1.0\MimeTypes key. I wondered to myself, just how many subkeys are there?

So I changed the loop back to while True: and checked the logs every few seconds. I was able to catch it counting up to at least application/x-java-applet;version=1.1792839. But at that point, the main Plex process gave up, killed the subprocess, and restarted it; so the counting began all over again.

I opened up regedit, and sure enough: all of those keys exist. (In fact, they go up to version=1.2203922.) I can’t guess why the Java 12 installer created so many keys - that installer might have a bug of its own. (But of course, you don’t want Plex to fail for anyone that’s ever installed Java 12.)

Based on these observations, one simple fix might be to change your loop to something like while i < 1000:. That would make Plex immune to strange registry patterns like this one without any loss of Plex functionality.

8

Thanks for all your work on this - The logs cover reading these registry keys 4 times

Extracting from the log entries for key sets that are about 10,000 lines apart - we get these

33 2020-06-02 12:26:22,970 (58f8) :  DEBUG (runtimekit:387) - Calling OpenKey(<PyHKEY:0x00000374>, @java.com/JavaPlugin,version=12.0.1.0\MimeTypes, 0, 520)

10034 2020-06-02 12:26:24,430 (58f8) :  DEBUG (runtimekit:387) - Calling OpenKey(<PyHKEY:0x00000374>, @Microsoft.com/NpCtrl,version=1.0\MimeTypes, 0, 520)

10173 2020-06-02 12:26:25,622 (3bdc) :  DEBUG (runtimekit:387) - Calling OpenKey(<PyHKEY:0x000002D4>, @java.com/JavaPlugin,version=12.0.1.0\MimeTypes, 0, 520)

20247 2020-06-02 12:26:27,944 (3bdc) :  DEBUG (runtimekit:387) - Calling OpenKey(<PyHKEY:0x000002D4>, @Microsoft.com/NpCtrl,version=1.0\MimeTypes, 0, 520)

21560 2020-06-02 12:27:30,500 (c30) :  DEBUG (runtimekit:387) - Calling OpenKey(<PyHKEY:0x00000D5C>, @java.com/JavaPlugin,version=12.0.1.0\MimeTypes, 0, 520)

31623 2020-06-02 12:27:32,880 (c30) :  DEBUG (runtimekit:387) - Calling OpenKey(<PyHKEY:0x00000D5C>, @Microsoft.com/NpCtrl,version=1.0\MimeTypes, 0, 520)

31687 2020-06-02 12:27:33,102 (4ff0) :  DEBUG (runtimekit:387) - Calling OpenKey(<PyHKEY:0x000007A0>, @java.com/JavaPlugin,version=12.0.1.0\MimeTypes, 0, 520)

41693 2020-06-02 12:27:34,940 (4ff0) :  DEBUG (runtimekit:387) - Calling OpenKey(<PyHKEY:0x000007A0>, @Microsoft.com/NpCtrl,version=1.0\MimeTypes, 0, 520)

All being for @java.com/JavaPlugin,version=12.0.1.0\MimeTypes

Out timeout is quite generous - giving the process 3 minutes to respond
We processed about 41,000 keys/subkeys in 70 seconds
So over 105,000 keys !

I will discuss your suggestion with the development team

(This is the first time I have come across this reason for server settings unavailable end result)

9

Thank you very much for passing this along to the development team!

Yes, in the logs I attached, the keys came in sets of 10,000. That is only because I changed the loop to while i < 10000 to capture these logs. And all plugins are affected, not just the system plugin. To illustrate that, here’s the full set of logs from that same startup run:
Logs.zip (947.6 KB)

In total, my registry has 2.2 million subkeys underneath the @java.com/JavaPlugin,version=12.0.1.0\MimeTypes key. With the original Plex code of while True, each plugin manages to process 1.8 million of those subkeys in the 3 minutes before it gets killed and relaunched. (I have not sent any logs from this “while true” scenario, because the logfile rotation quickly discards the crucial first part of the log, where the problematic “JavaPlugin,version=12.0.1.0” parent key is named.)

So with while true, all the plugins get stuck, and Plex kills and restarts them every 3 minutes. The tight while true loop causes each PlexScriptHost.exe subprocess to use 17% of my CPU. In a fresh install, I end up with four of these subprocesses running, so my CPU sits at 70%, even though Plex is completely idle. I can leave Plex running for hours and it will stay at a high CPU forever.

The forum includes a number of unresolved posts from people seeing high CPU while Plex is idle. This problem might explain some of those posts.

I did some searching and found that these runaway registry entries have been created by other versions of Java too. Here’s a bug report from someone encountering it in 2013 with Java 7. The Java devs closed the bug as unreproducable, but my registry is clear evidence that it was still occurring in JavaPlugin version 12 (which was part of Java 9, released in 2017). Bottom line is, there could be a lot of people out there with these weird Java registry keys; so this could possibly explain high CPU and “settings unavailable” for a number of your users.

10

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.