Timout While Requesting Identity URL using UPNP

Plex Media Server Remote Access
1

Server Version#: 1.18.7.2457
Player Version#: N/A

I’ve noticed that I have the following ERROR in my logs every hour when Plex is re-requesting a port be opened for remote access via UPNP (IP Address replaced with X’s):

Mar 22, 2020 14:45:59.014 [0x7fd28f5ca700] DEBUG - HTTP requesting GET https://XX-XXX-XX-XXX.73e167975696474d89b06d28a5c66b8e.plex.direct:28548/identity
Mar 22, 2020 14:45:59.026 [0x7fd28d541700] DEBUG - NAT: UPnP, mapped port 28548 to 192.168.1.51:32400.
Mar 22, 2020 14:46:02.015 [0x7fd28f5ca700] ERROR - Error issuing curl_easy_perform(handle): 28
Mar 22, 2020 14:46:02.015 [0x7fd28f5ca700] DEBUG - HTTP simulating 408 after curl timeout

In this case, the above domain “XX-XXX-XX-XXX.73e167975696474d89b06d28a5c66b8e.plex.direct” resolves to the public IP address of my gateway which makes sense. It gives Plex a convenient way to reference and secure remote traffic using a wildcard certificate.

I took a while to look into this as I’ve been trying to spend some time tracking down errors in my server logs. It turns out that this is because the linuxigd upnp daemon only opens up the remote port on the internet interface, so there is no way that this curl will ever succeed from my Synology server on the LAN.

This can be seen by looking at iptables:

iptables -nvL -t nat                                               
Chain PREROUTING (policy ACCEPT 32985 packets, 3704K bytes)
 pkts bytes target     prot opt in     out     source               destination         
   13   796 DNAT       tcp  --  netext *       0.0.0.0/0            0.0.0.0/0            tcp dpt:28548 to:192.168.1.51:32400

Above “netext” is my internet facing interface. This means that it will only destination NAT the Plex port (28548) if the traffic ingresses through the internet facing interface.

For reference the traffic looks like this:

Synology Server (192.X) -> Gateway (192.X internal interface) -> Gateway (199.X:28548 external interface) -> blocked by iptables (as the traffic originates from the internal interface and not the external interface).

While I haven’t specifically tested other upnp clients (like miniupnp) I would expect them to work similarly which means that this check likely doesn’t succeed often when it needs to use upnp to establish remote connectivity.

So my main question is: It’s an error in the log, but my server not being able to hit the /identity path doesn’t seem to have any bad effects. Is this true? Can I safely ignore this error? Or are there side-effects to it not being able to hit the /identity endpoint?

Additionally, would Plex be open to changing how this mechanism works? For instance, I think that it would be better in almost all cases for this check to be carried out by a server on the internet (i.e. a Plex owned server) that could give better results. The way that it is currently, it will always have to hairpin on the gateway in order for this check to work correctly, which I would expect to fail in most cases? If it doesn’t provide substantial functionality to the Plex Server could we just disable the check altogether? Or at the very least reduce it’s verbosity level to info or debug?

Finally, I am able to see all of this information about why this wouldn’t work because I run my own Linux gateway and have terminal access to it. While most residential routers run the exact same setups, most users don’t have this much access to their gateway and would be unable to understand why they get errors in their logs.

2

Any thoughts on this @ChuckPa? I know it was a long post, but my guess is that this affects a large segment of your users. It would be good to know if this is an actual error and if not (which seems likely as my setup seems to be remotely accessible anyway) changing it to an INFO or DEBUG would be an easy win.

3

This doesn’t impact a large number of users.

It does impact those who like to play with iptables :wink:
or have less-than-optimal IGD UPNP services running on the edge device.

I would need to see what the debug traffic above it is because most of the timeout errors (408) don’t trigger until greater than 4 seconds

I am using Pfsense as my edge device with its native UPNP service.
All 3 of my devices can remote connect without issue.

There is never any issue with /identity (this is a problem which we should resolve for you)

4

Ha! I guess it’s fair to say that I like to play with iptables :stuck_out_tongue:

After your message I played around with miniupnpd as well. However, I can say that both of these, being the two most prominent open-source UPnP solutions that I know of, by default only DNAT traffic when it is ingressing through the internet interface.

Given that a lot of consumer routers run either miniupnpd or linuxigd I’m not sure how this isn’t a more common problem for you. However, I’ll take your word for it as you’re a lot more active in these forums than I am. :wink:

For the benefit of anyone else that happens to find this and had the same problem I had (and I suppose who also runs their own Linux gateway), I was able to solve this for myself like so:

In the end, I ended up just staying with miniupnpd after testing it for this post to see if it fixed the error. However, by default, miniupnpd creates it’s own NAT chain and then jumps to it from the nat PREROUTING chain ONLY when the input interface is your internet interface. I fixed this by providing my own chain to miniupnpd with something like the following:

iptables -t nat -N upnp
iptables -t nat -A PREROUTING -j upnp

Additionally, you have to add the following to your miniupnpd configuration file:

upnp_nat_chain=upnp

This allows miniupnpd to DNAT traffic both on the external interface (which it would have done anyways) as well as on the LAN interface to Plex when Plex sends it’s UPnP request. It might be worth noting that linuxigd doesn’t give you the ability to define your own NAT table, so this solution would only be possible with miniupnpd.

Now we are DNAT’ing both traffic from the Internet and the LAN which is good, however, as it stands now we will still run into a problem because the source IP is still on our LAN which means that the traffic flow looks something like the following:

                                                ROUTER
                                        199.0.0.1 / 192.168.1.1
                                            ^           |
                                            |           |
LAN CLIENT ----------- STEP 1 --------------|           |----- STEP 2 ------>Plex
192.168.1.50                                                            192.168.1.51
    ^                                                                         |
    |---------------------------- STEP 3 -------------------------------------|

Since both clients are on the LAN, while the request goes through the router, the response from plex back to our client comes directly from Plex. This breaks the TCP handshake and causes the connection to not work properly (i.e. your client sends a request to 199.0.0.1, but receives a response from 192.168.1.51 and says this isn’t what I was expecting, and throws it away).

In order to fix this we have to implement a hairpin NAT on our router. Adding something like the following to iptables fixes this:

iptables -t mangle -I PREROUTING -i netint1 -s 192.168.1.0/24 -j MARK --set-mark 0x400
iptables -t nat -I POSTROUTING -o netint1 -d 192.168.1.0/24 -s 192.168.1.0/24 -m mark --mark 0x400 -j SNAT --to 192.168.1.1

This makes it so that before the traffic leaves the router out of 192.168.1.1 it changes the source IP on the packet to it’s own address 192.168.1.1 so that when Plex responds, it responds to the router which then SNAT's the traffic back to the client. Now when the client sends a request to 199.0.0.1 it also receives the response from 199.0.0.1 which is what it is expecting and it accepts it and the TCP handshake is complete.

@ChuckPa, if you’ve never had this problem, I can only assume that somehow pfsense is doing something like this for you implicitly which is possible as pfsense is a nice product. But after looking at my own consumer router’s firmware it’s difficult for me to assume that most of them are doing this for people.

5

A friendly poke ? :rofl:

We don’t run Linux gateways. We run BSD gateways on old hardware and load Community Pfsense on it :rofl:

As for not having encountered it before, even my ISP-provided ActionTec T3200 never had the problem. I am using DSL and not cable.

In a nutshell, as I’ve always understood, SNAT/DNAT is intrinsic to the overall functionality of the edge device, falling under the umbrella term “NAT”, isn’t it?

6

Reading your description a bit further:

If both clients are on the LAN side, and both on the same subnet, the packets , should be switched at the router and never actually hairpin / go into the router functions.
The subnet mask should have prevented the packets from ever being considered as ‘further routable’

7

That would be true, EXCEPT that Plex specifically tries to route to the gateway’s IP address that’s what the whole XX-XXX-XX-XXX.73e167975696474d89b06d28a5c66b8e.plex.direct does… It resolves to your external address.

At that point, even though the packet would be better routed via the Plex server’s internal address, Plex insists on routing to your public address which makes it have to hairpin on the internet gateway since the destination on the packet isn’t in the LAN subnet.

That’s why my original point was that it would be better to trigger something on the Plex owned infrastructure that is already on the internet to do this test. Especially since the test is biased as it isn’t traversing the path that an internet client would traverse anyway.

8

No. it’s a LAN DNS overlay on the RFC-1918 side.
PMS has its own internal DNS server which works with Plex.tv to keep track of the LAN IPs and map them into your plex.direct domain.

9

Ahh… I think that may be the issue then. After you mentioned DNS I did some searching and it looks like Plex relies on DNS rebinding to change the plex.direct DNS name to resolve locally when issued from your network. Is this correct? If so then when I do an nslookup of my plex.direct domain the return should be the LAN IP of my Plex host not the external address?

Assuming so, then my problem is likely to be caused by the fact that I use a local version of bind for resolving and caching inside my network. It forwards to opendns for most queries, but it’s probably disrupting the rebind somehow. Do you know of another post or anything where someone posted how to allow DNS rebinding with bind DNS? I know you guys have examples for dnsmasq and pfsense already.

10

That is correct. Plex rebinds plex.direct on the LAN. The resultant addresses are local LAN addresses.

I’m not the best at dnsmasq. I know on pfsense I only need declare the private network

server:
private-domain: "plex.direct"
11

Thanks for continuing to troubleshoot this with me @ChuckPa! So I looked through my DNS logs and it looks like there are two of the plex.direct queries that get used. There is one that represents the internal address (192-168-1-51.hash.plex.direct) and one that represents the external address (199-0-0-1.hash.plex.direct if you can give me a place to send it privately, I can send you the actual address).

For the internal one, I have verified that it is able to return the private address all the way down even through my bind DNS server. For the external one, it seems to ALWAYS return the external address. I’ve tried querying Google’s DNS directly (8.8.8.8), OpenDNS’s, and 1.1.1.1.

Is the external DNS address supposed to return a private IP address at some point? Because that’s the one that Plex is trying to use when it is looking up the /identity url. It’s trying to get 199-0-0-1.hash.plex.direct/identity which then returns my public IP address which it then tries to resolve from the LAN and then runs into problems when I don’t have hairpin NAT’ing enabled on my router to handle this specific case.