Lost a whole load of movies / TV shows this morning

This is where I fall on my sword.

I wasn’t looking at the right stuff.

I am VERY sorry to have confused.

I was able to recreate and capture. I was also chatting with Engineering.
I gave them a 50-deep logfile set. Hopefully this will give them what they need to see.

1 Like

[sigh] It’s reverted to the “lost movies” state, and the old poster-frames are back…

Current is RHS, Old page from a few hours ago is LHS.

Ok, here’s a clue for engineering.

I did a library rescan and was going to monitor it in Settings -> Console, and happened to see that the Settings -> ‘Remote Access’ line had a red warning icon next to it. I went to that tab, and it said the server was not accessible outside my network, even though the IP address seemed to be fine (it’s a public IP)

So, on a different machine, I ssh’d into my work’s VPN, and from my work machine I could ssh back into lyonesse (the plex media server machine). Odd.

So back on the plex ‘remote access’ tab, I clicked ‘Retry’ next to where it says ‘manually specify public port’ (note, this itself is not checked on), and the page changed to say ‘Fully accessible outside your network’. The ‘Retry’ turned into ‘Apply’ once it was successful.

On a hunch, I then refreshed the main plex page, and suddenly all my videos, tv shows and custom posters are restored…

Note that at no time did I change any firewall settings or IP address settings on the plex machine. It appears to be plex that had lost the internet connectivity, not the machine itself.

LOGS please I think you had a connection drop because Remote Access and Metadata aren’t even closely related.

Here you go:

Plex Media Server Logs_2019-08-01_19-36-05.zip (2.8 MB)

And just as another data point. I now have that red icon on remote access again, and I also have the missing movies/tv-shows/posters again.

I can still ssh into my plex server from work…

Must have been a coincidence, because this time the refresh doesn’t work when I do the ‘Retry’ thing successfully.

A couple question questions:

  1. The primary IP is directly connected to a routable IP ? Looks like it from here.
  2. eth0:0 is an alias on the adapter to give you a LAN presence?
  3. Any particular reason for not running a second segment and making that legit LAN ? It is making local discovery act flaky and probably driving . It’s seeing public IPs where it’s expecting WAN IPs and then turning around and calling them a private address (LAN).
NAT: UPnP, found device <http://6X.XXX.XXX.XXX:32469/DeviceDescription.xml> with private address <6X.XXX.XXX.XXX>

You did have an outage or at least a major confusion. We know that IP is a public routable WAN IP. it’s not private by any definition.

Jul 29, 2019 15:27:57.347 [0x7f0cccd6c700] DEBUG - MyPlex: We appear to have regained Internet connectivity.

As I look through the logs, While you do have direct public accessibility, PMS is having a hard time figuring out which to use.

How difficult would it be to split the alias off to be a true LAN segment on its own cable?

Huh.

The machine is using NetworkManager, there shouldn’t be an eth0 let alone an eth0.0

As far as I am aware, there ought to be 3 networks on that machine on two ethernet cards. The 10.1.x.x network is a 10-gigabit link to the house etc. And for legacy reasons there’s also a 192.68.1.x network on there too.

The 1-gigabit link is a fiber connection to the outside world, and uses the 63.x.x.x ip, which is public and ought to be firewall-free at the router, it depends on the firewall on the plex server itself for protection. There ought to be no other ip alias on that public address for pretty much the reason you mention - originally the 192.168.1.x network was on there, and AT&T’s (useless) router was getting confused because it doesn’t seem to like multi-homed machines, it interferes with the auto-machine detection it uses.

So, I just checked, and I have two ethernet interfaces (discounting the virtual bridge and loopback):

[simon@lyonesse ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp43s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:d8:61:76:42:92 brd ff:ff:ff:ff:ff:ff
    inet 63.x.x.x/28 brd 63.y.y.y scope global noprefixroute enp43s0
       valid_lft forever preferred_lft forever
    inet6 2600:1700:5450:46a0:c22c:2ec3:bb9c:5653/64 scope global noprefixroute dynamic 
       valid_lft 3567sec preferred_lft 3567sec
    inet6 fe80::c6a3:a5bd:e485:30aa/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: enp36s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:d8:61:76:42:93 brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.2/16 brd 10.1.255.255 scope global noprefixroute enp36s0
       valid_lft forever preferred_lft forever
    inet 192.168.1.180/24 brd 192.168.1.255 scope global noprefixroute enp36s0
       valid_lft forever preferred_lft forever
    inet6 2600:1700:5450:46a0:1ee2:14b3:8d8d:68e3/64 scope global noprefixroute dynamic 
       valid_lft 3567sec preferred_lft 3567sec
    inet6 fe80::6ef9:d92:4b1b:fbf0/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:e3:95:e0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:e3:95:e0 brd ff:ff:ff:ff:ff:ff

So you ought to be seeing enp43s0: (public IP) and enp36s0: (private ip’s). I also intentionally set no default route on the internal network, so the default route for all traffic with a destination that can’t be immediately satisfied is via the public ip gateway

[simon@lyonesse]$ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         63.x.x.x        0.0.0.0         UG        0 0          0 enp43s0
10.1.0.0        0.0.0.0         255.255.0.0     U         0 0          0 enp36s0
63.y.y.y        0.0.0.0         255.255.255.240 U         0 0          0 enp43s0
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 enp36s0
192.168.122.0   0.0.0.0         255.255.255.0   U         0 0          0 virbr0

The machine that used to be the video server was not running NetworkManager, and that one did have eth0 and eth0:0 - is it at all possible that the config is confused ? If you recall, I did a “swoopy” transfer of the database from that machine to this.

For what it’s worth, according to my firewall log, the last time the WAN went down was:

2019-08-01T07:13:46-07:00	network unavailable, WAN down

and it was only down for 2 seconds. Before that it’s been up as far as the log goes back ( a month or more). Plex has lost its connection a lot more than that.

In an effort to help plex out, I just configured the LAN networks on the Settings → Network tab, but it didn’t make any difference to the problem, which still shows the version with the missing shows :worried:

One final thought. I just restarted the server…

[root@lyonesse]# systemctl restart plexmediaserver

but it made no difference. Then I clicked to uncheck ‘Treat WAN IP as LAN Bandwidth’ on the Settings → Network tab, and refreshing the ‘Movies’ page got all my shows back again. Maybe another coincidence.

Please examine the logs yourself. I agree with you in that something is wrong. I recognize the enp notation ( I have it here on Fedora) .

  1. Open Plex Media Server.5.log
  2. Beginning at line 72 (I have obfuscated the data)
Jul 29, 2019 15:27:44.418 [0x7f0ce1758780] DEBUG - Detected primary interface: 6X.XXX.XXX.XXX
Jul 29, 2019 15:27:44.418 [0x7f0ce1758780] DEBUG - Network interfaces:
Jul 29, 2019 15:27:44.418 [0x7f0ce1758780] DEBUG -  * 1 lo (127.0.0.1) (loopback: 1)
Jul 29, 2019 15:27:44.418 [0x7f0ce1758780] DEBUG -  * 2 eth0 (6x.xxx.xxx.xxx.) (loopback: 0)
Jul 29, 2019 15:27:44.418 [0x7f0ce1758780] DEBUG -  * 2 eth0:0 (192.168.1.91) (loopback: 0)
Jul 29, 2019 15:27:44.418 [0x7f0ce1758780] DEBUG -  * 3 eth1 (10.1.1.1) (loopback: 0)
Jul 29, 2019 15:27:44.418 [0x7f0ce1758780] DEBUG -  * 6 virbr0 (192.168.122.1) (loopback: 0)
Jul 29, 2019 15:27:44.418 [0x7f0ce1758780] DEBUG -  * 1 lo (::1) (loopback: 1)
Jul 29, 2019 15:27:44.418 [0x7f0ce1758780] DEBUG -  * 2 eth0 (2600:XXXXXXXXX) (loopback: 0)
Jul 29, 2019 15:27:44.418 [0x7f0ce1758780] DEBUG -  * 2 eth0 (fe80::xxxxxxxxxxxx%eth0) (loopback: 0)
Jul 29, 2019 15:27:44.418 [0x7f0ce1758780] DEBUG - Creating NetworkServices singleton.
Jul 29, 2019 15:27:44.418 [0x7f0ce1758780] DEBUG - NetworkServices: Initializing...
  1. Please confirm you are seeing what I show here?

I’m seeing that in the logs, yes, but those logs are from a few days ago (.5 is the oldest log, right?)

It’s identifying its local ethernet address as 192.168.1.91 and this is Xanadu, not Lyonesse. This log is from before the plexmediaserver service was moved from Xanadu to Lyonesse, I guess the logs are written somewhere under the /var/lib/plexmediaserver directory ? Because that’s what was tarred up and transferred over.

Strangely, though, doing a grep for “Detected Primary interface” only brings up lines in “Plex Media Server.5.log” (and all referring to Xanadu), not in any of the other log files. Maybe I didn’t have a sufficient level of debugging switched on, on the Lyonesse server.

[aside] There’s some interesting lines in those log files “We computed 1 alternative realities in 0 ms.” … Um, well done! [grin].

It would be extremely helpful in the future if the logs are completely removed when cloning / moving from one to another.

When you have time, would you mind stopping, erasing all logs, and beginning again with a fresh set? It would greatly improve my sanity because I can’t tell PMS was moved to a different host :wink:

Will do - presumably just everything in “/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Logs” ?

I did mention the transfer thing above - I even linked to the post, but I guess you’d already started going through the logs at that point :slight_smile:

[edit: Done]

Now seeing:

Aug 02, 2019 09:08:14.041 [0x7efbf7fc7780] DEBUG - Network interfaces:
Aug 02, 2019 09:08:14.041 [0x7efbf7fc7780] DEBUG -  * 1 lo (127.0.0.1) (loopback: 1)
Aug 02, 2019 09:08:14.041 [0x7efbf7fc7780] DEBUG -  * 2 enp43s0 (63.x.x.x) (loopback: 0)
Aug 02, 2019 09:08:14.041 [0x7efbf7fc7780] DEBUG -  * 3 enp36s0 (10.1.1.2) (loopback: 0)
Aug 02, 2019 09:08:14.041 [0x7efbf7fc7780] DEBUG -  * 3 enp36s0 (192.168.1.180) (loopback: 0)
Aug 02, 2019 09:08:14.041 [0x7efbf7fc7780] DEBUG -  * 4 virbr0 (192.168.122.1) (loopback: 0)
Aug 02, 2019 09:08:14.041 [0x7efbf7fc7780] DEBUG -  * 1 lo (::1) (loopback: 1)
Aug 02, 2019 09:08:14.041 [0x7efbf7fc7780] DEBUG -  * 2 enp43s0 (2600:1700:5450:46a0:c22c:2ec3:bb9c:5653) (loopback: 0)
Aug 02, 2019 09:08:14.041 [0x7efbf7fc7780] DEBUG -  * 2 enp43s0 (fe80::c6a3:a5bd:e485:30aa%enp43s0) (loopback: 0)
Aug 02, 2019 09:08:14.041 [0x7efbf7fc7780] DEBUG -  * 3 enp36s0 (2600:1700:5450:46a0:1ee2:14b3:8d8d:68e3) (loopback: 0)
Aug 02, 2019 09:08:14.041 [0x7efbf7fc7780] DEBUG -  * 3 enp36s0 (fe80::6ef9:d92:4b1b:fbf0%enp36s0) (loopback: 0)

Yes, everything under Logs

I probably had started going through them.

Those names look a whole lot better and now make sense.

Now the logs will reflect truth

So for what it’s worth, I still can’t stay connected directly (according to plex) for any significant period of time. The “Remote Access” has the “Not available outside your network” red icon set.

It doesn’t seem to be impacting playback though. Running from work, the dashboard is telling me that if I start a stream (“Burn After Reading”, blu-ray capture) at 1080p, it’s using about 25 Mbit/sec of bandwidth (peaking at almost 200 at the start!) and about 5% or so CPU.

Also, I’m not missing any movies or anything atm.

Logs attached, in case they’re of any use. And yes, this time they’re the right ones :slightly_smiling_face:

Plex Media Server Logs_2019-08-02_10-06-22.zip (286.7 KB)

Here’s a thought. It should NOT matter but might resolve this.

  1. PMS isn’t the smartest about which IP address to listen to / respond on.
  2. It can’t hard bind() to any one port.
  3. All it can do is favor one over the other.

Settings -> Network - Show Advanced -> Preferred adapter

The default route should be taking care of staying connected to Plex.tv
If it’s becoming confused (PMS usually deploys on machines with one network adapter), help it by picking a preferred.

I don’t know which will work better for you or if none will help but that setting will have an impact, one way or another.

Ok, thanks - I actually had that set to the enp43s0 network interface, but I’ll try switching it to ‘Any’.

Hmm - actually it doesn’t seem to help too much. I click on “retry” to get the Fully Accessible message, then a few seconds later it’s back to “not accessible”.

I have tried going to whatismyip.host from Lyonesse, to make sure I wasn’t double-NAT’d, and they think my ip address is the 63.x.x.x one, so it’s not that.

The firewall rules for plex on the server are:

[root@lyonesse ~]# cat /etc/firewalld/services/plexmediaserver.xml 
<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>plexmediaserver</short>
  <description>Ports required by plexmediaserver.</description>
  <port protocol="tcp" port="32400"></port>
  <port protocol="udp" port="1900"></port>
  <port protocol="tcp" port="3005"></port>
  <port protocol="udp" port="5353"></port>
  <port protocol="tcp" port="8324"></port>
  <port protocol="udp" port="32410"></port>
  <port protocol="udp" port="32412"></port>
  <port protocol="udp" port="32413"></port>
  <port protocol="udp" port="32414"></port>
  <port protocol="tcp" port="32469"></port>
</service>

… which I got from somewhere on the internet. To me it looks as though it ought to be ok, but I’m willing to be educated :slight_smile:

As I said, it doesn’t seem to be a problem for actual use - I’ve just run 3 streams from home to work, and they all seem to be playing back fine…

I’ll add the UDP 32400 port, thanks :slight_smile:

Not sure if this will help, but, I noticed you don’t have UDP set for 32400, I opened up 32400 for TCP and UDP, because of this:

UDP is a very lightweight protocol defined in RFC 768. The primary uses for UDP include service advertisements, such as routing protocol updates and server availability, one-to-many multicast applications, and streaming applications, such as voice and video, where a lost datagram is far less important than an out-of-order datagram.

So I’m hesitant to suggest this, because there’s been “fixes” before that were just pure coincidences, but I have managed to seemingly “fix” my issues (two days without it happening again) by removing all the local storage that the plex-web interface uses. I use Safari, so it was a matter of:

Enable the Develop menu in Preferences
Develop -> Clear all caches
Develop -> Show Javascript Console
Console -> Local Storage tab -> select each and press the delete key
Close and relaunch Safari application (maybe not necessary)
Click on the “http://[server]:32400/web” menu-bar link
Enjoy.

What is really weird is that it was clearly using some older copy of the database, because the “friendly name” (I changed it to ‘Media Central’ rather than ‘Video Central’ when I added music in) would revert, along with the custom posters, and of course all the movies/tv-shows I’d recorded since that snapshot at time X.

What was even weirder was that it continued to use the DVR recording schedule set up when the interface was working - so the DVR must have been using a different version of the database (!)

This “solution” is predicated on there being some sort of key in the local storage that plex uses to initialise its state - like a node in the tree that describes the plex server content. Start with a different node, get a different “reality” in the web-interface. We’ll see.

Anyway, here’s to hoping the problem doesn’t come back. If it starts happening again, I’ll post again, but as of right now it seems to be working.

Me again :frowning:

The above (previous post) very often works, but I did have it not work on me one time last night. It did work once I’d restarted the plex service on the linux box.

I am curious if engineering has had any luck figuring out what’s causing this, because it’s still ongoing, albeit I can usually fix it now. There are days when the DVR is constantly recording from TV, so it’s still an issue because I obviously don’t want to restart halfway through a recording…

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.