PMS Security Hardening with Outbound Proxy

[Skeels_78]

Server Version#: 4.147.1
Player Version#: NA (direct web interface from PMS)

---

The Issue:

Attempting to scan one of my music libraries yields the following, on-screen, error message:

An internet connection is required to scan library files.
Please check this server's connection and try again.

This issue doesn’t occur when scanning my other music library. The only difference I can think is the failing one has new content that hasn’t been scanned yet.

Once I added new content to the other music library that wasn’t having this issue, it started having this same issue.

Background:

I’m about certain I know what the initial cause of this issue is.

I’ve recently started isolating my servers from the internet - they cannot talk outside their respective VLAN/subnet (unless a fw rule is added to allow for specific purposes). For internet access they must use the Squid proxy I had stood up. I’m filtering external domains/endpoints on the proxy to make sure only required resources are accessible. This is just another step in building a zero-trust environment.

I have added the following environment variables to my PMS systemd service file (as an override):

Environment="no_proxy=localhost,127.0.0.1,.int.mydomain.com"
Environment="https_proxy=http://forward-proxy.int.mydomain.com:3128"
Environment="http_proxy=http://forward-proxy.int.mydomain.com:3128"

Adding my proxy config like this works for the most part. PMS is using it to connect back to Plex’s servers as well as other services that are used to populate metadata, etc. I can see this in the proxy access logs.

I do not see any outbound requests in the proxy access logs, from PMS, when telling the server to run a library scan.

Rectification Attempts:

So far I have, mutually exclusively, tried the following:

• Restart the PMS service
• Reboot the entire server
• Remove the proxy env vars, restart, add them back, restart again
• Restoring outbound access, restarting PMS (with the proxy config still in place). This does solve the issue; I can then successfully scan the music library.

Additional Comments:

To me, this acts like whatever system is responsible for scanning libraries isn’t properly utilizing the proxy config env variables, or isn’t built to use a proxy at all.

I’ve enabled debug logging, tried scanning a few times and exported the logs. I’d like to share the logs, but don’t want username and all content if in open. How can I provide my logs securely, to Plex members only (not non-user accessible)?

Is this necessarily a bug or is this because of a lack of forward proxy support?

[dbirch]

Plex is designed to be simple server/app setup. I do not know if plex posts a full list of externally required resources. Anytime you are putting something between Plex and either the users or the servers plex talks to, you are probably going to break something. In this case Plex is trying to reach out to cloud based metadata providers for matching, and providing details on the new media. Something in your hyper secure environment is blocking the required traffic.

The concept of running Plex in an enterprise level/Environment isnt something that has been actively supported. On that scale you have copyright issues, and start to get more attention from copyright holders as a method of distribution. Plex ran into something similar with the hosting provider Hetzner and as a result the entire hosting service has been IP banned from plex.

[ChuckPa]

PMS does not support proxy outbound for its built-in capabilities.

PMS itself must

1. Directly connect to Plex.tv for authentication ( No Man-in-the-Middle actors )
2. Make authenticated metadata requests and receive that metadata (which is licensed)
3. Make authenticated codec requests and download those codecs (which are licensed)

As stated by DBirch, Enterprise level serving is not supported and will result in a ToS violation & termination of service. PMS has always been an “Always Connected” architecture.

If you wish it to be secure:

1. Do not enable Remote Access or Relay
2. Do not port-forward from the outside into your server.

My server is sufficiently secure that I could give you my WAN IP ; Port number and I guarantee you will not be able to reach it

The only outbound traffic I have is for Auth, Codecs, and Metadata.

[Skeels_78]

Thanks for the response @ChuckPa /@dbirch !

I think there are some misconceptions about what and why I’m trying to accomplish this.

This setup is meant to allow me, as a Plex user, to better secure my own internal infrastructure. In the event my PMS is compromised the bad actor would not be able to make outbound requests to unrelated endpoints (). The server will still be able to contact necessary resources via the internal proxy. It’s not about requests coming into 32400. It’s about preventing unauthorized outbound requests.

Using a forward proxy shouldn’t be a TOS violation. It would be similar to me building a VPN tunnel and routing all my PMS traffic over it to somewhere else than my home gateway.

Difference with the proxy is that I as the admin get more granular control over what my server is talking out to.

I think as more and more people start to home lab and get deeper into hosting, this will be more and more important. Lots of new and emerging threats, might as well add another layer of protection.

Since this isn’t supported today, is there a way for me to add a feature request?

Thank you again for your input. I’d really like to talk back and forth about the value of this functionality!

[ChuckPa]

If you want to secure your internal LAN, I will share what I did.

1. My PfSense firewall is the main router

2. It controls the LAN subnets
   – 192.168.0.x – Main LAN
   – 192.168.2.x – VLAN 2000
   – 192.168.3.x – VLAN 3000

3. My switch has multiple VLANs defined , different machines are isolated by PVID and VLAN tagging.

4. I control inter-VLAN traffic by writing firewall rules.
   This is how I allow (either by Alias or IP ( showing VLAN 2000 )
   – Allow my workstation to access the test PMS on VLAN 2000
   

   
   
   – Block everything else
   
   

5. The definition of the virtual interface in the PfSense for VLAN 2000
   

   

With this configuration, unless I specifically allow crossing of one VLAN to another, it’s blocked at the router (PfSense). The switch guarantees the physical isolation from the device to the PfSense.

[Skeels_78]

@ChuckPa thanks!

Yes, I already isolate my networks this way. I’m very explicit about where traffic is allowed to traverse. Below are my baseline rules for each interface. I also use Floating rules to deny outbound to known “bad” IP space (using pfBlockerNG).

My PMS is isolated internally because of this firewalling.

---

What I’d like to do now is similar - just like you said, unless I specifically allow PMS (or whatever server I’m working on) to access a resource on the internet, it can’t.

The way I see it, there are two (there are more ways than this, these are just the ones we’re talking about now) simpler, but not equal, solutions:

1. Setup specific firewall rules in pf to only allow PMS to talk outbound to the IP space it requires. Using an Alias that contains a list of DNS names PMS needs to talk to in the rule would do the trick. pf would resolve the DNS names and use the IPs from those names to filter. This definitely reduces outbound access, which is great, but not as strict.
2. Use a forward proxy that allows for filtering on a SNI level (at minimum. More specific filtering is possible, like by URL path). This way, when the PMS service or the underlying Linux system or any other process (maybe malware) on the machine makes an outbound request, the proxy will use the domain name being requested to make the decision on whether to allow the request.

The issue with option 1 is that pf will only filter by IP. This is an issue with shared hosting and CDNs. If any of the services my server needs to access is on a shared host, restricting by IP means my server can still access any content on that server/endpoint, regardless of the SNI.

Right now, if you lookup plex.tv one of the IPs it resolves to is 35.172.142.61. (At least for me, it could be geo based)

A reverse lookup on 35.172.142.61 resolves to ec2-35-172-142-61.compute-1.amazonaws.com.

# dig plex.tv +short
35.172.142.61
44.210.41.33
34.238.225.186

# dig -x 35.172.142.61 +short
ec2-35-172-142-61.compute-1.amazonaws.com.

So for example, let’s say my PMS server is compromised for some reason. Let’s also say that there’s another site on that same server - randomsoftware.dev’s git repository (obviously also resolves to 35.172.142.61)

If randomsoftware.dev’s repo becomes compromised or is malicious by design, my server would still be able to access that content since I’m only filtering by IP and not DNS name. My infected server can now grab content from the .dev site. Maybe that’s the malware’s CNC host.

In the same situation as above but with a proxy, the proxy would see a request for plex.tv and allow it, but a request for randomsoftware.dev would be denied, because the SNI/DNS name is being used to filter. So even though the two resources share an IP, the proxy can still filter the requests and deny the ones that I haven’t explicitly allowed.

A proxy also provides much more detailed logging for access auditing. I use Graylog, so I could have it send me a notification when it sees a log from the proxy that contains DENIED so I can investigate.

---

Also, from an earlier comment, I wouldn’t necessarily need a list of resources PMS needs to access provided by Plex. I can just watch my proxy logs, see what’s being denied then add it to the whitelist if I deem it necessary (does the server blow up if it can’t access it? Yes? Ok, whitelist it).

---

So all-in-all, I agree to a point, yes, filtering outbound by IP (resolved from DNS names being requested) will significantly reduce my server’s attack surface, but not as much as a proxy could.

A note about PMS:

As well-built as PMS is, no software is perfect and un-hackable. All software is vulnerable. It’s just a matter of timing and circumstance for when someone will find that one nasty bug. This is why Zero-Trust is a thing - there’s no way we can know for sure what will happen, so tighten things up as best as possible now to reduce blast radius later.

Happy New Year by the way!!