Unknown Bug with Remote Access Apps (Roku, Android,etc) - Architectural Guidance with Proxy/Cloudflare

[bengalih]

Server Version#: 4.12.3
Player Version#: various

I’m really hoping I can get some dev/staff input on this particular info. I realize my specific configuration is non-standard, and possibly not supported (albeit there is a decent size community which uses this configuration). While this may not be a bug, there is definitely an anomaly in how things are working architecturally, and I would really like for someone from the Plex team to at least explain to me how these apps are functioning, so I can understand why I have these issues.

My understanding is that somehow the mobile apps always communicate with some central Plex servers initially, which perform some sort of testing to determine if the user’s own Plex server(s) on their account are reachable. It then returns this information to the mobile client which will then attempt a direct connection to the user’s Plex server. Based on my observations I believe there are some checks that are causing Plex to report back that a server is not reachable when indeed it is. Beyond this, I can’t explain exactly what is happening. Additionally, if I am wrong about the above statements, I also have no idea why I am having these issues:

I am using Plex via NGINX reverse proxy. There are many guides on how to set this up online and many of them use a configuration similar to this. Part of the configuration details DISABLING the remote access option in Plex, which should in part force all traffic only through NGINX. In my case I am forwarding all incoming 443 traffic to plex.mydomain.com to my NGINX server which directs it upstream to my server over port 32400. My NGINX server is secured with a 3rd party SSL certificate issued by Let’s Encrypt.

With this configuration I can access my Plex server remotely in the following ways:
https://app.plex.tv - Will show a secure connection via the Plex GmbH DigiCert certificate.
https://plex.mydomain.com - Will show a secure connection via my Let’s Encrypt cert.

In both cases my server shows with the green “Remote” lock next to it via the web-app.

However, I am unable to access my server using the Android or Roku apps which show my server as offline/unreachable.

Initially, I thought this was possibly due to my ISP blocking port 80, so I am unable to allow any http connections through at all. I was attempting to setup some type of redirect from http://plex.mydomain.com to https://plex.mydomain.com in case these mobile apps were attempting a connection over 80 first. In trying to solve this problem I decided to migrate my DNS over to Cloudflare for 2 reasons:

• I was told that they had an easy way to blanket redirect all http requests
• There are quite a few guides our there for using this configuration with Cloudflare. Most of those seem to have to do with peering advantages, but since many had reported successful configurations with NGINX/Plex/Cloudflare, I decided to give it a final try.

What I have found is that if I configure my DNS entry for plex.mydomain.com in Cloudflare as a standard (e.g. non-proxied) Cloudflare entry, then DNS queries for plex.mydomain.com will return my NGINX WAN address. Presumably this means anything attempting to communicate with my internal Plex server will first try to hit my NGINX server directly. This configuration has the same results described above, namely the Android/Roku clients fail to connect. This was also the same results I saw prior to my DNS being switched to Cloudflare (makes sense since I am using CF here as a standard/gray DNS entry).

If however I set my DNS entry to proxied via Cloudflare (using Full SSL), then DNS queries for plex.mydomain.com return a Cloudflare IP (104.16.0.0-104.31.255.255). Presumably this means anything attempting communication with my internal Plex server will first hit Cloudflare, which will then make the connection to my NGINX server and then proxy this back to the client. In this configuration my remote clients all work as expected!

I have not setup any redirection rules in Cloudflare, so indeed attempt to access my site at http://plex.mydomain.com will always fail when accessed through a browser. Cloudflare will deliver a 522 “Connection Timed Out” error showing the timeout on my host (NGINX server).
Because of this I don’t believe my initial assumption of my ISP blocking port 80 has anything to do with this.

When viewing the access logs on my NGINX server I never see any traffic coming in when using the non-proxied/direct connection.
When using the proxied connection, clearly my logs show tons of events, as I am obviously communicating with my server since it works!

This would seem to indicate that whatever process is responsible for determining the availability of my server has an issue when checking with my NGINX server directly, but works when Cloudflare initially answers the connection and then forwards it to my same NGINX server.
I have only two theories, neither of which seems very good:

1. The port 80 thing: Cloudflare at least returns an actual webpage to show the 522 issue over port 80. Perhaps Plex requires this somehow. Seems unlikely, also because it takes quite a while to return this 522 over a browser and my mobile clients seem to connect quickly.

2. A certificate issue: When connecting via Cloudflare proxy, the Cloudflare shared certificate is used. When connecting directly my Let’s Encrypt certificate is used. I don’t have a problem with my LE certificate over the web-app. The certificate is a totally valid 3rd party cert that should be trusted by all, but this is one of the only other differences I can perceive.

I see no reason why I should have to proxy my connection through Cloudflare for my configuration to work, but yet I do - and I’m hoping to find a reason why.

[bengalih]

@OttoKerner - sorry for calling on you. I’ve seen some input from you on other posts and you seem to have a good understanding of this. Would appreciate your time if you are available.

thanks.

[OttoKerner]

Nope, sorry. No experience with proxies at all. The default remote access of Plex is working fine for me.

[bengalih]

I realize that the NGINX proxy experience is not an actual supported method, but are you aware of any detailed info that describes the underlying connection process the Android/Roku clients take to determine how to connect to a server and then, what specifically they connect to?

[OttoKerner]

There is not, I’m afraid. Some users reported their experience with nginx and cloudflare here in the forums., though. Using the search might yield some interesting threads.

[bengalih]

Ok thanks. I have read dozens of posts here and been searching for weeks across here, reddit, and elsewhere on Plex/NGINX/Cloudflare/etc and have turned up nothing actionable after spending well over 20 hours on this.

More info for anyone who may be able to assist:

I’ve attempted to audit the Android mobile logs and the best root issue I can observe is that when things fail I get:

01-10 20:48:36.817  i: [conn] Device MYSERVER response time is Infinity ms
01-10 20:48:36.817  i: [conn] Connection FAILURE MYSERVER ~ Connection: https://plex.mydomain.com:443 token used: true types: [myplex] state: Unreachable

And when things work I get:

01-10 20:52:05.988  i: [conn] Device MYSERVER response time is 536.9013 ms
01-10 20:52:05.988  i: [conn] Connection SUCCESS MYSERVER ~ localConn: false conn: https://plex.mydomain.com:443
01-10 20:52:05.988  i: [conn] (MYSERVER) We found the first connection.

As far as the rest of the log, and the connections attempted it all seems fairly identical. The only difference is that seemingly the call to https://plex.mydomain.com:443 results in success if hitting Clouflare’s proxy first. Presumably CF has no issue hitting my Plex server behind my NGINX and returning that to the client, but for some reason the client has an issue when my DNS points directly to my NGINX (not proxied through Cloudflare).

I’ve examined logs on all components and either I’m not seeing anything out of the ordinary (or I don’t know what it is I’m looking for), or there just aren’t any logs to view (in the case of NGINX not even being communicated with on direct connection as explained above).

It is very difficult to try packet captures as these are only devices on remote networks. Even trying to test my local Roku or Android for a good connection is difficult because of not being able to run reliable packet capture tools on both (and I can’t get promiscuous captures).

I can certainly see what Cloudflare is communicating by looking at the logs on NGINX and a packet capture on my Plex server when these requests come in proxied, but it doesn’t really assist because I see absolutely none of the same on an unproxied connection.

[bengalih]

So I’ve solved my issues!

To recap - I initially was unable to establish a direct connection to my server through NGINX using Roku/Android app. This is detailed here. My first solution (detailed in that thread) was to move my DNS over to Cloudflare and use their “proxy” DNS which appeared to properly work. I was however left with a mystery as to why I was required to use their proxy option instead of just hitting my NGINX server directly.

I figured out how to do “debug” level logging on NGINX as well as how to get a local tcpdump on my android phone while attempting access to my IP directly. This provided some good evidence that this was some type of certificate issue:

From tcpdump:

377	9.746316	100.XX.XX.XX	70.XX.XX.XX	TLSv1.2	75	Alert (Level: Fatal, Description: Certificate Unknown)

From nginx log:

2020/01/14 05:32:27 [debug] 15340#0: *1403 SSL certificate status callback
2020/01/14 05:32:27 [debug] 15340#0: *1403 SSL ALPN supported by client: http/1.1
2020/01/14 05:32:27 [debug] 15340#0: *1403 SSL ALPN selected: http/1.1
2020/01/14 05:32:27 [debug] 15340#0: *1403 SSL_do_handshake: -1
2020/01/14 05:32:27 [debug] 15340#0: *1403 SSL_get_error: 2
2020/01/14 05:32:27 [debug] 15340#0: *1403 reusable connection: 0
2020/01/14 05:32:27 [debug] 15340#0: *1403 SSL handshake handler: 0
2020/01/14 05:32:27 [debug] 15340#0: *1403 SSL_do_handshake: -1
2020/01/14 05:32:27 [debug] 15340#0: *1403 SSL_get_error: 1
**2020/01/14 05:32:27 [info] 15340#0: *1403 SSL_do_handshake() failed (SSL: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:SSL alert number 46) while SSL handshaking, client: 172.58.99.136, server: 0.0.0.0:443**
2020/01/14 05:32:27 [debug] 15340#0: *1403 close http connection: 3
2020/01/14 05:32:27 [debug] 15340#0: *1403 event timer del: 3: 211227619
2020/01/14 05:32:27 [debug] 15340#0: *1403 reusable connection: 0
2020/01/14 05:32:27 [debug] 15340#0: *1403 free: 001BC360, unused: 8
2020/01/14 05:32:28 [debug] 15341#0: *1407 SSL certificate status callback
2020/01/14 05:32:28 [debug] 15341#0: *1407 SSL ALPN supported by client: http/1.1
2020/01/14 05:32:28 [debug] 15341#0: *1407 SSL ALPN selected: http/1.1
2020/01/14 05:32:28 [debug] 15341#0: *1407 SSL_do_handshake: -1
2020/01/14 05:32:28 [debug] 15341#0: *1407 SSL_get_error: 2
2020/01/14 05:32:28 [debug] 15341#0: *1407 reusable connection: 0

At this point I was pretty clear it was a certificate error though I didn’t exactly understand how it could be. After all when accessing the same URL via my browser I had a fully validated certificate chain. At first I thought that the Android client was somehow hard-coded with certain root CAs and was using them instead of honoring the system certificates. This was not the case, although the Android client does seem more sensitive to some SSL configuration parameters than any browser.

The answer comes down to the description of the ssl_certificate option in the NGINX documentation:

“Specifies a file with the certificate in the PEM format for the given virtual server. If intermediate certificates should be specified in addition to a primary certificate, they should be specified in the same file in the following order: the primary certificate comes first, then the intermediate certificates. A secret key in the PEM format may be placed in the same file.”

My ssl_certificate pem file was simply my server certificate as generated by Let’s Encrypt. I did have the LE intermediate certificate specified in the ssl_trusted_certificate directive, but apparently that was not sufficient. I created a new .pem file by concatenating the two together and used that new certificate as my ssl_certificate.
After reloading NGINX, my issues were solved!

I’m surprised that I was unable to track down the answer from other people’s configuration instructions. Not a single configuration guide I read (and I read dozens) mentioned configuring the certificate in this way. That being said, quite a few people were either:

• Not using the android/roku clients in their configurations and if they were they probably didn’t disable Plex remote access and were actually being relayed through plex direct.
• Using Cloudflare proxy (most common), in which case they would be presenting the CF cert to the client which apparently is configured properly so as to not cause this issue.

I really hope this helps someone else down the road!