After switching to Plex Pass PMS is constantly utilizing one core (version 1.14.1.5488)

Plex does handle/speak IPv6 quite well, actually. There’s an option to turn it on or off in the Network pane in PMS. It works okay and I’ve tested it extensively on the networks I have access to. There are a few bugs, such as it identifying all IPv6 traffic as “remote” but I’ve turned it off on your earlier advice.

As you can see, Plex isn’t bound to any IPv6 on my server (remote addresses obfuscated):

admin@diskstation:~$ netstat -an | grep 32400
tcp        0      0 0.0.0.0:32400           0.0.0.0:*               LISTEN     
tcp        0      0 192.168.1.10:54571      xxx.xxx.xxx.xxx:32400   ESTABLISHED
tcp        0      0 192.168.1.10:32400      xxx.xxx.xxx.xxx:45434   ESTABLISHED
tcp        0      0 192.168.1.10:41653      xxx.xxx.xxx.xxx:32400    ESTABLISHED
tcp        0      0 192.168.1.10:55864      xxx.xxx.xxx.xxx:32400     ESTABLISHED
tcp        0      0 192.168.1.10:38068      xxx.xxx.xxx.xxx:32400   ESTABLISHED

In the past you told me IPv6 support in PMS wasn’t finished or supported. I didn’t gain much having it running beyond the rosy feeling of having all my services IPv6-enabled and not being one of Those People (looking at you, Domoticz), stuck in the past insisting IPv6 is pointless and will never work :grin:

As my PMS instance has IPv6 turned off, as you suggested, I don’t see how that could be at fault. Also, there are no IPv6 only devices on my LAN, it’s all dual-stacked.

You’d be hard pressed to find a LAN these days that doesn’t have IPv6 traffic on it somewhere. Even without a radvd present on the network and a delegated prefix from your ISP, modern OSes like Windows 10 and macOS, Playstations and Xboxes, will give themselves link-local addresses and chatter away to each other.

You’re missing the point.

  1. Plex.tv doesn’t speak IPv6 but is commonly accessible on IPv6 ISP’s who employ NAT64 service. I have confirmed again with our operations team.
  2. While I do not dispute routers are capable, ISP’s are in control of those configurations.
  3. I do not dispute all devices are IPv6-capable.

The focal point here is to determine why PMS is locking the single core. The only known datapoint here is miniupnpd which exists in routers (IGD).

Therefore, all other considerations and challenges aside, rephrased:

Given PMS speaks IPv4 to Plex.tv for all services, IPv6 metadata does not yet exist:

Does miniupnpd behave / respond differently on IPv6 and IPv4 LANs ?

If so, what is the difference? Only the Address format changes? What else if anything?

I did some more digging today in the code and experimentation:

Regarding the miniupnpd and other devices which may or may not be involved.

The following was discovered when testing a Ubitquity:

HTTP/1.1 200 OK
Cache-Control: max-age = 60
Ext:
Location: http://192.168.1.5:8080/upnp
SERVER: UPnP/1.0 UniFi/5.10.20
ST: upnp:rootdevice
USN: uuid:3C86D905-39E4-4FDC-963C-04E19F2791BB::upnp:rootdevice

while http://192.168.1.5:8080/upnp URL returns 404’s.

This is the type of non-Spec compliance I need to identify as part of this process.

You will also see “Schema” errors on these IPs, when it’s returning 404, in the PMS logs.

Sorry, you weren’t very clear. You said “Plex doesn’t yet handle IPv6”; I assumed you were referring to the interaction between my server, my clients and my router, over my LAN and the Internet as that’s what we’ve been discussing so far. Plex.tv has not been a factor at all until now. (I know, never make assumptions! :smirk:)

Please be more specific about which part of the larger infrastructure you’re referring to rather than using the generic “Plex”. It’s not easy for us to debug efficiently when it’s not obvious if you’re referring to a client, server or the backend cloud database services.

I’m in full control of my IPv6 as it’s not provided by my ISP and I’m not using NAT64.

I agree, however you queried whether IPv6 might be at fault, so I explored that.

In that case you should remove the AAAA records for the plex.tv servers. There’s no point having those servers resolvable if they’re not going to answer on those addresses and potentially cause problems.

Also, what metadata are we talking about?

It’s functionally identical, but responds and queries using IPv6 in addition to IPv4. Behaving differently at layer 4, dependent on what protocol is in use at layer 2, is very bad practice.

I still don’t understand why the presence of IPv6 on my LAN would have any impact on PMS since it’s turned off in the PMS network settings and not bound to any IPv6 addresses. SSDP does not present any data about IPv6 clients over IPv4, and vice versa. PMS is not connected to an IPv6 network in my instance.

Nice one…looks like we’re back on the track of PMS not gracefully handling errors, then :+1:

Let me have a look at what my router is offering via it’s advertise URL

Okay, so my router is advertising “http://192.168.1.1:5000/rootDesc.xml” via IPv4 SSDP (and “http://[xxxx:xxx:xxxx::1]:5000/rootDesc.xml” via IPv6 (address obfuscated)).

If I use curl (tested from a selection of hosts on my network) to retrieve the URL I get:

root@OpenWrt:~# curl http://192.168.1.1:5000/rootDesc.xml

curl: (7) Failed to connect to 192.168.1.1 port 5000: Connection refused

I have miniupnpd running in secure_mode on my router, that is: it will only accept requests to create port forwards to the host that is requesting it. However, running in secure mode also seems to prevent retrieval of the root device’s upnp schema in this instance :thinking:

I then temporarily disabled secure_mode (DO NOT ATTEMPT THIS AT HOME!) and tried again. This time miniupnpd returned the schema to me with no errors.

Following this I re-enabled upnp, restarted miniupnpd and, sadly, saw the Plex media server process go straight back up to 100% CPU.

This is obviously not exactly the same failure state as you’re seeing, but I think it shows that, even when the root device schema is available, PMS still gets into the loop.

edit
The only other device doing SSDP NOTIFYs with a URL for a schema on the network is a Hue Bridge. I’ve disconnected that from the network, restarted miniupnpd with UPnP enabled and observed high CPU use in the plex media server process still

Gary,
Would it be fair to summarize:

  1. PMS performs SSDP (L2 broadcast) to discover.
  2. Accepts all replies
  3. For each device:
    a. HTTP query for schema
    b. Receive reply
    c. Parse
    d. If error - disable (ignore) device
    e. Else - process as appropriate

Are you asking if that’s the way I believe it should work, or how it does work?

If it’s the former, I suppose it makes sense, although I don’t understand why PMS is using SSDP at all when it’s configured to use a manual port forward for Remote Access. What else does PMS need to discover on the LAN? As I understand it, clients connect to plex.tv over the Internet and are informed of your servers, or else you manually configure them, so what is PMS looking for besides a router? If I knew what PMS was using SSDP for I’d have a better idea.

If it’s the latter, I don’t know, it’s not my code :man_shrugging:t2:

I am asking if you feel that’s how it should work.

It doesn’t work that way currently. If we’re going to make it official, I want to get it so (mostly) everyone agrees it’s working in a sane and predictable way.

Okay, that makes sense, thanks.

Notes added in bold below following further information from sa2000

I’m tagging @sa2000 into this thread as I think what comes below may be relevant to the other work I’ve been doing with him.

Firstly, SSDP doesn’t broadcast, it uses a site-local multicast group, one specifically reserved for UPnP/SSDP: 239.255.255.250

I’d add a little more logic at step 1 or 3:

  1. the M-SEARCH packets PMS is sending out every 10 seconds (it’s definitely PMS, btw, more on that below) are asking for “ST: upnp:rootdevice”, which should cause all UPnP capable devices on the LAN to respond, not the specific kind it could be looking for. Asking only for the devices it’s interested in would be more efficient and potentially lower the risk of receiving malformed responses that cause undesirable behaviour
  2. If we’re not going to be selective at step 1, we have to accept all replies
  3. If we’re accepting all replies, let’s be selective at this point instead and only query the schema for devices we’re interested in, rather than everything that responds (so, for example, let’s query a TV but not a Hue Bridge, if that’s what PMS is searching for)

FYI, what comes below was all carried out with UPnP disabled on my router:

This brings me on to my other point, I still don’t know what PMS is doing with SSDP and what it’s looking for, I feel like I’m trying to reverse engineer it, here. Is it purely for finding an IGD (if you set it to automatically port-forward with UPnP for remote access) or is it doing other stuff like looking for TVs and speakers? FWIW, I’ve set my PMS to “Manually specify public port” and have “Enable local network discovery (GDM)” unticked but PMS still sends out M-SEARCH packets and gets responses. This occurs every 10 seconds, which seems little excessive to me.
This is expected behaviour as PMS uses GDM and SSDP seperately

The option for “GDM” is in the PMS Network settings page, and this appears to be a proprietary protocol that Plex has developed to search for “other servers and players”. There’s no documentation available for this although this provides some information. This may not be 100% factual: although it states that GDM multicasts on 239.255.255.255 I can’t see any traffic on this group with GDM enabled but there’s plenty on 239.255.255.250.
GDM doesn’t multicast at all but broadcasts every 5 seconds on whatever your LAN broadcast address is (192.168.1.255 in my case), confirmed with Wireshark.

GDM seems to be a modified version of SSDP, for all intents and purposes. Why SSDP wasn’t good enough for Plex, I don’t know. Why is Plex doing SSDP and GDM? I don’t know. I have a sneaky suspicion that Plex isn’t exactly doing SSDP at all, and that the traffic I’m seeing in my Wireshark sessions is GDM identified as SSDP as it’s so similar.
Above paragraph is entirely incorrect following sa2000’s information

To test this I stopped PMS entirely on my server and did a tcpdump: the “M-SEARCH” SSDP packets stopped being sent out from the server and the “HTTP/1.1 200 OK” responses from devices on my LAN stopped coming back. There were still periodic NOTIFYs from other devices on the LAN, but overall SSDP traffic was much lower with PMS stopped entirely.

I think:

  • a lot of the of the SSDP traffic we’re seeing in our packet dumps is actually Plex’s own “GDM” traffic nope, definitely not the case
  • turning off “Enable local network discovery (GDM)” doesn’t stop it, as I still see M-SEARCH packets this is expected behaviour as it’s confirmed that PMS is doing SSDP and GDM separately, however, GDM does not stop sending out packets on 32412 and 32414 if GDM is disabled in the PMS web interface (confirmed with Wireshark)
  • GDM is not only communicating with other Plex clients and servers, it’s eliciting responses from any UPnP enabled devices on the LAN as it’s using a standard multicast group address not the case, confirmed separate broadcast UDP traffic on ports 32414 and 32412

Is some GDM process in PMS getting upset when it tries to parse a response from miniupnpd with UPnP enabled? Remember, miniupnpd is still running, I’ve just turned off the UPnP functionality and it’s doing NAT-PMP only.
Nope, it’s not GDM, it’s SSDP

So, with GDM off and manual port forwarding specified, why is PMS still searching for UPnP devices and parsing their responses? Why, specifically, is it parsing a response from an IGD (miniupnpd) and getting into a loop over it?
See sa2000’s response below for more info on what PMS is doing over SSDP

Thanks for the detail.

I believe the SSDP multicasts help detect DVRs on the network as well Plex players for casting to - in addition to finding the IGD which is used for detecting the Public IP as well as port mapping. The Public IP is used to check for Double NAT and log that.

We do appear to send out a multicast M-SEARCH through each active network interface and that may double up the traffic - I have raised an issue on this as well as looking into the frequency of searches

For GDM we use broadcasts to destination UDP ports 32412 and 32414 which is done every 5 seconds

1 Like

Thanks for the informative reply @sa2000!

I think we may have found a clue there. This high CPU behaviour first manifested in release 1.14.1.5488. Looking at the release notes we can see:

Version 1.14.1.5488
NEW

  • New (Web) Updated to 3.77.4
  • (DVR) New DVR scheduler

FIXES:

  • (DLNA) DLNA clients would not report timeline states for Now Playing (#9347)
  • (DVR) Don’t return an error message when shutting down a live TV session in a normal way (#9277)
  • (DVR) Fixed bug that prevented scheduling of recordings with HD only setting. (#9226)
  • (DVR) Properly parse episode number from dd_progid field in XMLTV data (#9419)
  • (DVR) Revert timeout changes, should help with timeouts during tune startup (#9473,
  • (HTTP) Enabling IPv6 support could prevent IPv4 connections from being accepted on some FreeBSD systems (#9405)
  • (Radio) Only expose Discovery Radio if there is a linked TIDAL account (#9428)
  • (Relay) The relay service may not function when server support for IPv6 is enabled on some systems (#9399)
  • (SSDP) Better checking for invalid IP addresses in location field of SSDP XML
  • (Streaming Brain) More exact accounting of bandwidth buckets to reduce bandwidth going unused (#9273)
  • (Subtitles) Some subtitles could not be downloaded by shared users (#9323)
  • (Subtitles) Subtitles aquired by the OpenSubtitle agent could cause playback failures on Nvidia SHIELD
  • Improve robustness when routers fail to respond to port-mapping refresh requests in time (#9107)
  • The server could crash when using network interface aliases on macOS (#9344; rdar://46083980)

So PMS uses SSDP to search for DVRs on the LAN and the “New DVR scheduler” functionality was first introduced in this release.

edit: added fixes to release notes, I’m pretty sure the “better checking for invalid IP addresses” bit will be at fault here

3 Likes

This behaviour is still present in 1.15.4.993

Yep, getting this on QNAP. Unfortunately can’t find any old builds archived for my specific QNAP version.

I suspect that as well.

The current plan is to wait for an update to the cppnetlib software that is within Plex Media Server.

I have flagged the change you mentioned as the possible suspect fix - will let you know when i have an update on this

2 Likes

Good to hear, @sa2000. The cpp-netlib project released version 0.13.0 (stable) in January 2019. Looking at PMS on my Mac it appears to have the libcppnetlib-uri.dylib library included (as opposed to the full-blown cpp-netlib library) in its Frameworks folder.

Reading the docs for the URI library, it appears to be what is responsible for parsing URIs (duh :slight_smile: ) and is likely what is being used to process the SSDP responses from devices on the network. Unfortunately, for some reason, the library included in the MacOS PMS distribution doesn’t have a valid version number set for it (at least that I can see by querying it with otool; it returns a “current version” of 0.0.0, as do many of the libraries bundled with the PMS app).

However, I note that PMS 1.14.1.5488 (the first version we observed this behaviour in) was released to beta on December the 17th 2018 and the latest version of the URI library (1.1.0) was on November 24th, 2018. Could it be the new version of this library that caused the issue (assuming Plex is keeping up to date with the releases of the open source code it relies on)? Nope, don’t think so…read on…

/me Googles for a while…

Ah, found another thread where someone else came to a similar conclusion: Plex consuming a full core at idle, libcppnetlib-uri.so (boost) implicated

So, tinkering with my Synology box, I thought I’d drop the libcppnetlib-uri.so file from the 5470 build into my current production system (1.15.4.993) and see if it resolved the problem. Strangely, I can’t find the libcppnetlib-uri.so file in the installation package for earlier versions.

edit I just rolled back to 5470 on my Synology and it doesn’t have libcppnetlib-uri.so installed at all, so I’m guessing the use of that library was new (or changed) as of 5488. I checked older versions of the macOS distribution and these, also, do not include libcppnetlib-uri.dylib in versions prior to 1.14.1.5488

Some libraries are old and changes are not made unless something stops working. Of course - security related changes do get picked up and included

1 Like

Thanks for the investigation. I will pass the info on

1 Like

Good approach, imho: if it ain’t broke, don’t fix it :+1:

However (sorry for my multiple edits above) it seems to be the fundamental implementation of the library, rather than a specific version, that seems to have caused the problem.

Presentation of the Plex Media Server high CPU behaviour, with minupnd operating in UPnP on the LAN, seems to correlate directly with the change in PMS to utilise the cpp-netlib URI library, likely as part of the “(SSDP) Better checking for invalid IP addresses in location field of SSDP XML” fix in 5488.

This behaviour is still present in 1.15.4.994