possible major security issue wrong network adapter AGAIN

General Discussions
1

well another server update and yet another dickdance to fix issues

I noticed that while my server was green for internet access, the identified networks was WRONG

I have my plex server running in esxi and the plex server vm has 2 network adapters

adapter 1 is my standard network 192.xxx for internet etc and the other is a virtual network on 10.55 for internal to the server NFS data sharing and has NO routing, connectiviity to ANYTHING outside the esxi host…

yet plex was showing that it was bound to the internal network and had external access which was confirmed working…

HOW THE HELL is plex bridging from a private non routed subnet to the internet…

it was tenacious and would not let that interface ens160 go… en33 is the standard interface and the one it should use…

I had to do a dickdance that was not used in a while…

I had to down the ens160…

log out of plex on plex.tv

log out of plex on the host server

log into plex.tv and delete my server device entry

then log back into everything…

after a couple cycles of this … with ens160 down… it would finally grap ens33 and get a standard address

dont know how long this is going to hold or if it is even going to survive a reboot…

this is unacceptable from many standpoints

I view not being able to control and force plex to use a specific network configuration to be both an annoyance and a major security issue…

it should never be able to bridge my networks without my express permission which I obviously didnt give it

@elan
@StSimm1

You guys want to weigh in on this…

2

Plex most certainly did not bridge your networks. This is an ESXi issue and not Plex. What you should have done right then and there is test the networking in the VM to verify it was actually working the way you think it is. You could have tried trace routing out to the internet over the network that Plex choose to use and you would have seen it did in fact have external access (and thus was valid for Plex to use).

When ever you have multiple NICs Plex is going to use the first one it can communicate on and get a response back that matches it’s needs. If you have things manually setup to use the “other” NIC, then it’s got roughly a 50/50 shot of working the way you want.

You really don’t want your VMs setup this way for PMS.

3

@cayars

no its a piex issue for sure … there is no way esxi is telling plex to attach to a network that doesnt even have internet access. esxi does not control what programs do inside the virtual machine.

there is no ping or any other tables in esxi or my unbuntu vm that show any connectivity from ens160 (the virtual subnet) yet plex is using it AND getting outside access though it. the only place that could bridge it would be inside the plex/ubuntu vm and since I have set no such routing tables to bridge the two adapters… plex is obviously doing it

I cant ping nor contact the router from the subnet that plex is using so it is being done internally to plex through some custom tunnel/bridging…

the networking in the VM is solid, tested, and setup properly… this is not the first time that plex server has grabbed the wrong interface and it was fixed for a while but the last upgrade to 1.7.3 has brought it back.

there are other threads mostly windows users that have also complained the server is grabbing the wrong interface as well as those with spit horizon setups with vpns that are pissed they cant get plex to bind to the correct adapter/subnet…

lets see what the devs think… if they ever answer…

4

Plex doesn’t modify your routing table. If your inside adapter somehow has “magical” access to the outside world this is a VM or ESXi issue. Plex does ZERO routing.

Plex will test the connections of each adapter and will use the first one (I believe) that can connect to the outside world.

How about posting a route print of the VM?

5 
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         router.asus.com 0.0.0.0         UG    0      0        0 ens33
10.55.0.0       *               255.255.255.0   U     0      0        0 ens160
192.168.0.0     *               255.255.255.0   U     0      0        0 ens33
6

Looks like PMS needs to stop choosing ens160 as the advertised private and stop tracking the wrong private address with Plex PAM–but it is not bridging. Traffic is being forwarded by ens33. PMS doesn’t have control over the network stack. This incorrect Private IP selection by PMS will come into play in a multi VLAN/subnet environment. Clients in a different local subnet then the server, will get the advertisement and try to get to 10.55.0.0/24 and can’t. They will end up going to the public IP to get to the server.

7

@dragonmel Question for you? When the wrong private IP is being advertised, which IP are your local clients connecting? Private 192.168.0.x/24 or Public IP?

8

@“Ach!lles”

they have to be connecting over 192.168 in the home as they dont have access to 10.55 on the esxi server

as for connecting from external the home… again… the router cant see 10.55 so again… even though plex is saying its connected via 10.55… there is no f-ing way unless its tunneling it through 192.168 as that is the only routed subnet that makes it to the router or cable modem…

9

@dragonmel Hold on, a bit of misunderstanding here. I think I may have confused you with my last question.

When the wrong private IP is being advertised, which IP are your local clients connecting? Private 192.168.0.x/24 or Public IP
I asked you that question because I wanted to know if the false advertisement of 10.55.0.X was throwing your local clients for a loop. We both know there would not be a way for them to access that isolated subnet. I think the local clients on the 192.168.0.0/24 subnet would still find your server via local discovery.

However hypothetically, if you had another subnet (for example: 192.168.1.0/24) in your environment–local discovery would not help the clients of that subnet find your PMS server in the 192.168.0.0/24 subnet. It would have to rely on Plex’s PAM to advertise the private and public IP addresses. In that scenario, when the advertised 10.55.0.X private IP address could not be reached, it would have to failover to the public IP and the traffic would U-turn via NAT Loopback. That would be the pitfall of this bug from my analysis.

Besides the fact that PMS is reporting the incorrect private IP, does it actually break internal or external streaming? I am not discounting your concern here. Myself and a few other Ninjas just need all the details to report this to the dev accurately.

10

Also if you truly believe that the packets are bridging across from the ens160 interface over to the ens33 interface…please provide a tcpdump for us to analyze. That will be the source of truth.

11

@“Ach!lles”

What don’t you understand

My home network is 192.168

All of my clients are on that network

10.55 lives inside one isolated esxi host as a non routed virtual subnet for vm to vm storage SMB shares.

The plex vm has 2 adapters. One for 192.168 and the other for esxi nfs/SMB data on 10.55

The plex server settings page was showing the plex server bound and routed from 10.55 out to my wan ip… and showing green for external connections wan side and my clients worked.

Now how the hell can plex be talking wan side on an adapter that has no routing to it?

Ssh into my router and I can ping all machines and clients on 192.168 put won’t ping a goddamn thing on 10.55 as it shouldn’t since its not routed

Now if plex server thinks its on 10.55 and the clients are on 192.168 that causes issues with steaming brain logic

If plex server really is bridging. That is a major f-ing issue and it can just like a vpn can create a software tunnel. Plex keeps pretty close to the vest on how it does server discovery, NAT translation and the rest.

I don’t have any way to packet sniff further nor does plex pay me to do such

If someone does indeed find in these cases that plex is somehow tunneling into or through subnets that might become a legal issue

12

@dragonmel said:
@“Ach!lles”

What don’t you understand
I am a CCIE working for a Fortune 10 with 25+ years experience. I fully understand networking much more complicated than this. I also am well versed in the wild world of software defined networking pioneered by VMware. I understand fully your thought process on this and where you are going with it.

My home network is 192.168
Already understood.
All of my clients are on that network
Thanks for clarifying.
10.55 lives inside one isolated esxi host as a non routed virtual subnet for vm to vm storage SMB shares.
Understood as this is standard best practice.
The plex vm has 2 adapters. One for 192.168 and the other for esxi nfs/SMB data on 10.55
Understood. Interface ens160 has no gateway set. Correct?
The plex server settings page was showing the plex server bound and routed from 10.55 out to my wan ip… and showing green for external connections wan side and my clients worked.
There is another perspective here with what is going on. This could be a bug showing the IP it has collected for PLEX PAM advertisement and not the interface with an actual default gateway–which is wrong in your scenario. This is the point I have been trying to make with you.
Now how the hell can plex be talking wan side on an adapter that has no routing to it?
It may or may not be. Only a tcpdump will reveal that here.
Ssh into my router and I can ping all machines and clients on 192.168 put won’t ping a goddamn thing on 10.55 as it shouldn’t since its not routed
And it should not as expected.
Now if plex server thinks its on 10.55 and the clients are on 192.168 that causes issues with steaming brain logic
Agreed, if 192.168.0.0/24 is not bound to PMS, it will treat 192.168.0.0/24 as remote. Streaming brain, if configured, will bitrate limit everything on the 192.168.0.0/24 connecting as remote unless 192.168.0.0/24 is added to LAN networks. Are your local clients being subjected to the remote stream bitrate limiting when this occurs?
If plex server really is bridging. That is a major f-ing issue and it can just like a vpn can create a software tunnel. Plex keeps pretty close to the vest on how it does server discovery, NAT translation and the rest.
I don’t believe this to be the case at all but rather PMS showing you what it is going to advertise to Plex PAM
If it is creating a tunnel between the two interfaces, you would see them in your routing table.
I don’t have any way to packet sniff further nor does plex pay me to do such
tcpdump is built into Ubuntu. I do not get paid by Plex either and here I am on a Saturday night volunteering my free time to help you instead of playing with my 18mo baby girl.

If someone does indeed find in these cases that plex is somehow tunneling into or through subnets that might become a legal issue
I will reproduce your environment and get to the bottom of this issue. Again a tcpdump will be the best way to prove your diagnosis.

13

@dragonmel

lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.10
Release:	16.10
Codename:	yakkety

uname -a
Linux 4.8.0-56-generic #61-Ubuntu SMP Wed Jun 14 08:15:42 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

sudo apt show plexmediaserver
Package: plexmediaserver
Version: 1.7.3.3937-70f781325
Status: install ok installed
Priority: extra
Section: video
Maintainer: Plex Inc <support@plex.tv>
Installed-Size: 232 MB
Homepage: https://plex.tv
Download-Size: unknown
APT-Manual-Installed: yes
APT-Sources: /var/lib/dpkg/status
Description: Plex organizes all of your personal media so you can easily access and enjoy it.

route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.1.0.1        0.0.0.0         UG    100    0        0 eno1
10.1.0.0        0.0.0.0         255.255.0.0     U     100    0        0 eno1
10.55.0.0       0.0.0.0         255.255.255.0   U     100    0        0 enx1491823be721

I have done the following:

1. Rebooted the host multiple times
2. Reinstalled PMS v1.7.3.3937 multiples times
3. Issued plexmediaserver start/stop multiple times
4. Issued plexmediaserver restart multiple times
5. Enable / Disable Remote Access multiple times

I can not reproduce your issue where PMS will choose the interface with the 10.55.0.x IP address on my Linux Ubuntu host.

14

@“Ach!lles”

look… to be sure I appreciate your help… I use to post an vol more time here back at the beginning when plex was open source and pretty much a single man operation. it was a collaberative env and people listened …

now… I pay for this software, there developlement direction is where they want it to go and they generally dont listen… like this issue is not new… there are others that have had the issue of plex selecting the wrong adapter… some windows users and just about everyone running split horizon with VPN and not wanting plex to route through the VPN and have been asking for the ability to bind plex to a specific adapter in settings… never implemented.

I had a rash of this happening about a year ago I think… and once you do the dance as posted it usually fixes it for awhile… but when plex grabs the wrong adapter, reboots etc never fix it … and I think it may be stored in the device profile on plex’s servers because the only way I have been able to fix it is by deleting it and logging everything back in without the adapter being up.

you clearly have far more expertise here… I am not a networking engineer… not even close… didnt even stay at a holiday in express last night…

however…

if plex server settings report the wrong adapter showing routing from the internet wan address through the wrong subnet … whether for GDM whatever… its at best a BUG… since if it were really on that subnet even with GDM my clients should not be able to see it or connect… but they do… also plex has a tunnel mode called indirect… sort of explained here https://support.plex.tv/hc/en-us/articles/216766168-Accessing-a-Server-through-Relay

now I have software on a workstation that blocks all outgoing network traffic unless it asks for permission… yet have several pieces of software that defeat it… and can only be blocked in the host files…

you as a network engineer must know that for every firewall there are ways around them…

anyway… one thing I noticed with your TEST setup is that your 10.55 network doesnt have an adapter… so is it even up?

anyway my fix has held for 2 reboots and 24 hrs now…

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    0      0        0 ens33
10.55.0.0       0.0.0.0         255.255.255.0   U     0      0        0 ens160
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 ens33

perhaps setup a working vm with 2 adapters and force plex to grab the data network adapter by downing the 10.1 en01 then up it and see if plex shows a green external connection through 10.55 as it was doing for me a couple days ago…

as far as streaming brain setup… I dont have anything filled in so I dont know if streaming brain is active or not… but external connections to my clients were ok… I also dont have any special network or subnet exclusions for security set… the only network options I use is specify port since I turn off unpnp on the router…

15

the other thing that I have noticed are read from another plex dev is that plex grabs the first adapter?

first in what list…

for example in your routing table your internet interface is above your data network and my routing table they are reversed…

I didnt edit the routing table this it the way it set itself up… dont know if that makes a difference in plex behavior

16

The 10.55.0.0 is a USB ethernet adapter and both are connected. The switch showed the mac address of both NICs in their respective VLANs. In my testing, I did try to change the subnet to 10.0.200.0/24 to make it appear above the 10.55.0.0/24. This made no difference.

I am aware of the Relay service through TCP port 32400. Thats why I asked for a tcpdump. It would reveal your suspicions if they are indeed occuring.

I would edit your metrics from 0 to anything higher for ens160 as thats typical best practice for the storage iface. Although that really should not matter since ens160 does not a default gateway assigned.

17

@“Ach!lles”

well at this point I cant do a dump since I have corrected it… if it happens again I will perhaps try and capture more info before I fix it.

still I dont like the idea that plex can grab whatever it wants whenever it wants it without granular control and plex is less than forthcomming on how exactly they tunnel the firewalls for remote access. let alone the folks that pull their hair out with VPNs and trying to get plex on the correct adapter.

now as for the metric suggestion … can you get into that in greater detail.

I basically have an all-in-one esxi host where I boot napp-it(omnios) vm first to provide ZFS backed NFS shares for esxi to run on… those NFS shares use the 10.55 network

I have read multiple conflicting articles on best setup pracices and tuning… some say leave it as mtu 1500 others say 9000 … others say since its virtual and not real tcp it doesnt matter… etc…

so what does metric do for me and how do I set it?

Thanks

18 
sudo apt-get update
sudo apt-get install ifmetric

Once installed use it to change metric value for interface, refer this http://manpages.ubuntu.com/manpages/vivid/man8/ifmetric.8.html

19

@“Ach!lles”

I will look into it… thanks

20

@“Ach!lles”

well its back… plex server without a reboot decided at somepoint last night to re-grab the wrong interface…

clients can connect from wan side as tested on my iphone with wifi turned off…

if you want a tcp dump you will need to walk me through what you want in ubuntu… or I can dump my asus router as it runs linux too I believe and I can telnet into it


net usage on the two adapters seemed to follow common sense… ens160 was getting the file from smb to transcode from storage and bursted with the tcoder and ens33 was pushing the 3mbs that the client was requesting over wan

so why does plex say its using the 10.55 adapter… and how to confirm what the app is REALLY doing