All Home items suddently unreachable. Not storage

Plex Media Server Desktops & Laptops
1

Server Version#: 1.18.2.2029
Player Version#:

Running on : Ubuntu 18.04.3 LTS

Media is on a Synology NAS connected via NFS

Last night Plex was working fine. Started to watch something when going to bed and got an out of disk space error on my Roku. Jumped on the server and found that something had been throwing a ton of messages to syslog and had filled up /. I assumed something crashed and it was just constantly restarting so I rm’d the syslog log and restarted the server.

Now I wish I had actually dug in to syslog first. The web server comes up. I can connect on port 32400 but everything such as Continue Watching, On Deck, Movies, TV, etc shows Unreachable. The main Plex Media Server log looks fine. The only thing I can find is that there is no DLNA process running. Not sure if that’s normal as I’ve never looked. The DLNA server log shows:

Nov 15, 2019 00:15:23.191 [0x7efde3874700] INFO - Plex DLNA Server v1.18.0.1944-f2cae8d6b - Ubuntu PC x86_64 - build: linux-x86_64 - GMT -05:00
Nov 15, 2019 00:15:23.191 [0x7efde3874700] INFO - Linux version: 18.04.3 LTS (Bionic Beaver), language: en-US
Nov 15, 2019 00:15:23.191 [0x7efde3874700] INFO - Processor Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz
Nov 15, 2019 00:15:23.191 [0x7efdebc33740] DEBUG - Unique device name = 6feab8d7-2496-4e30-eed1-1956e29d8ee4
Nov 15, 2019 00:15:23.191 [0x7efdebc33740] DEBUG - DLNA server port: 32469
Nov 15, 2019 00:15:23.191 [0x7efdebc33740] DEBUG - PMS: http://127.0.0.1:32400/
Nov 15, 2019 00:15:23.191 [0x7efdebc33740] DEBUG - MyPlex: https://plex.tv/
Nov 15, 2019 00:15:23.201 [0x7efdebc33740] DEBUG - Reading system DLNA client profiles
Nov 15, 2019 00:15:23.209 [0x7efdebc33740] DEBUG - Read 22 DLNA client profiles
Nov 15, 2019 00:15:23.210 [0x7efdebc33740] DEBUG - DLNA device discovery enabled every 60 seconds
Nov 15, 2019 00:15:23.210 [0x7efdebc33740] DEBUG - Initialization complete
Nov 15, 2019 00:21:11.784 [0x7efdebc33740] DEBUG - Stopping UPNP server

Other processes that are running:

plex      4515  0.0  0.0   4624   832 ?        Ss   11:17   0:00 /bin/sh -c  PLEX_MEDIA_SERVER_INFO_VENDOR="$(grep ^NAME= /etc/os-release | awk -F= "{print \$2}" | tr -d \" )"  PLEX_MEDIA_SERVER_INFO_DEVICE="PC"  PLEX_MEDIA_SERVER_INFO_MODEL="$(uname -m)"  PLEX_MEDIA_SERVER_INFO_PLATFORM_VERSION="$(grep ^VERSION= /etc/os-release | awk -F= "{print \$2}" | tr -d \" )"  LD_LIBRARY_PATH=/usr/lib/plexmediaserver/lib  "/usr/lib/plexmediaserver/Plex Media Server"
plex      4529  0.2  0.4 1823916 68776 ?       Sl   11:17   0:12 /usr/lib/plexmediaserver/Plex Media Server
plex      4564  0.2  0.3 1868276 50200 ?       SNl  11:17   0:11 Plex Plug-in [com.plexapp.system] /usr/lib/plexmediaserver/Resources/Plug-ins-36236cc4c/Framework.bundle/Contents/Resources/Versions/2/Python/bootstrap.py --server-version 1.18.2.2029-36236cc4c /usr/lib/plex ediaserver/Resources/Plug-ins-36236cc4c/System.bundle
plex      4606  0.0  0.0 457028 14404 ?        Sl   11:17   0:01 /usr/lib/plexmediaserver/Plex Tuner Service /usr/lib/plexmediaserver/Resources/Tuner/Private /usr/lib/plexmediaserver/Resources/Tuner/Shared 1.18.2.2029-36236cc4c 32600 /waitmutex

I uninstalled and reinstalled the Plex packages with no help. Any ideas?
2

What does df -m show ? This will tell us which filesystem was filled up.

I suspect you have filled /var (default PMS location) because your library has grown over time (tends to happen naturally) and now we need to move it.

3

My /var is just off of / and isn’t out of space once I deleted the syslog files. Here is my output:

jason@Shockwave:/var$ df -m
Filesystem                    1M-blocks     Used Available Use% Mounted on
udev                               7924        0      7924   0% /dev
tmpfs                              1590        2      1588   1% /run
/dev/mapper/ubuntu--vg-root      185285    45269    130537  26% /
tmpfs                              7949        1      7949   1% /dev/shm
tmpfs                                 5        1         5   1% /run/lock
tmpfs                              7949        0      7949   0% /sys/fs/cgroup
/dev/loop0                           90       90         0 100% /snap/core/8039
/dev/loop2                            4        4         0 100% /snap/gnome-system-monitor/107
/dev/loop3                           98       98         0 100% /snap/docker/384
/dev/loop1                           43       43         0 100% /snap/gtk-common-themes/1313
/dev/loop4                           55       55         0 100% /snap/core18/1223
/dev/loop6                           90       90         0 100% /snap/core/7917
/dev/loop5                            5        5         0 100% /snap/gnome-calculator/536
/dev/loop7                           15       15         0 100% /snap/gnome-characters/359
/dev/loop9                            1        1         0 100% /snap/gnome-logs/81
/dev/loop8                          141      141         0 100% /snap/gnome-3-26-1604/98
/dev/loop11                          45       45         0 100% /snap/gtk-common-themes/1353
/dev/loop12                           1        1         0 100% /snap/gnome-logs/73
/dev/loop13                           1        1         0 100% /snap/indicator-sensors/194
/dev/loop14                          60       60         0 100% /snap/disk-space-saver/7
/dev/loop15                         157      157         0 100% /snap/gnome-3-28-1804/91
/dev/loop10                           9        9         0 100% /snap/canonical-livepatch/88
/dev/loop16                          60       60         0 100% /snap/disk-space-saver/6
/dev/loop18                          31       31         0 100% /snap/sonarr/184
/dev/loop17                           9        9         0 100% /snap/canonical-livepatch/84
/dev/sda1                           511        7       505   2% /boot/efi
/dev/loop19                           1        1         0 100% /snap/indicator-sensors/231
/dev/loop21                          53       53         0 100% /snap/minikube/4
/dev/loop22                         123      123         0 100% /snap/docker/418
/dev/loop20                          55       55         0 100% /snap/core18/1265
/dev/loop24                           4        4         0 100% /snap/gnome-system-monitor/111
/dev/loop23                          15       15         0 100% /snap/gnome-characters/367
/dev/loop26                          31       31         0 100% /snap/sonarr/179
/dev/loop25                           5        5         0 100% /snap/gnome-calculator/544
/dev/loop27                         157      157         0 100% /snap/gnome-3-28-1804/110
/dev/loop28                         141      141         0 100% /snap/gnome-3-26-1604/97
192.168.200.71:/volume1/Share  10974928  9305506   1669423  85% /mnt/Share
192.168.200.77:/volume1/Media  43926763 28678754  15248009  66% /mnt/media
192.168.200.72:/volume1/BT     22675521 21783594    891812  97% /mnt/BT
tmpfs                              1590        1      1590   1% /run/user/1000
/dev/sdc1                         14647       23     14625   1% /media/jason/USB
4

thanks.

With that out of the way, where is the Transcoder temporary directory pointed?
Is it in /tmp which is a tmpfs (memory-based) filesystem?

5

First, thank you very much for the help. With the text messages I’ve gotten from friends and family you’d think this was a Sev 1 Prod outage…

My transcode directory should be /mnt/transcode but I can’t confirm that since I can’t see settings in the UI. Can I see those in a config file somewhere?

And it’s in /mnt/transcode as I had planned to dedicate a SSD to it that would be mounted there but haven’t done it. So it’s just a folder off / for now and therefore isn’t full.

6

In the UI, Settings - Server - Transcoder - SHOW ADVANCED – it will let you see all the options.

user plex runs separate from your UID. have you verified UID/GID plex:plex can write to the subdirectories ? (inclusive of the requisite pass-through of the top-most dir)

7

Same problem here, found out syslog was over 150GB, docker is making this happen, needed to delete syslog and purge docker out so it would not start again.

So now my server is running, but PMS cant connect to anything, that result gives no option to go in Setting - Server because it appears to be offline, but I can log into him and browse him, but all libraries are offline.

I tried to reboot computer, restart service, downgrade pms, pretty sure this is not a access right issue.

Seem just docker royally fudged us up

more on the docker issue: https://askubuntu.com/questions/1188908/what-does-this-docker-log-entry-mean

8

Your problem and mine sound exactly the same except I’m not running with Docker. My install is just a straight install in Ubuntu…no containers. But I have the same symptoms you do. I can’t see any settings as the web server is running…but PMS can’t talk to anything.

9

I´m not running PMS with docker. I was using docker for https proxy, docker was only running one container, we have same problem, running same OS and both did, I think, docker installed on our system

10

Here is a screenshot of my Home and Settings. Notice I have no options for Server settings. This is me connected directly to the web server on the Plex system by direct IP.

[Album] Imgur

Permissions are fine. plex:plex has full access. Transcode dir has 777 permissions so anyone can write there no matter what. This system has been running like this as-is for 8 months or so.

11

Oh. Yeah. I do have it installed on there but haven’t used it. Had planned to for some things like Sonarr but never bothered… Fantastic. So haven’t found a fix?

12

The “unreachable” is Plex/Web telling us it can’t talk to the server.

The lack of posters is another issue.

I need the Server logs and info.

Settings - Server - Troubleshooting - Download Logs.

14

Since I can’t get to the Settings - Server stuff can I just zip up everything in /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Logs and put it on a DropBox or something for you?

15

same here, I´m unable to go in Setting - Server, I can of course zip logs and send to you via dropbox/wetransfer

17

logs from me

18

ZIP up (or tar.gz) the Logs directory and upload here please.

Pushing it to another site is extremely awkward.

When done, either you or I can delete them.
I will also open a PM as required.

19

logs.zip (483.7 KB)

hehe had no idea that was possible, here it is

20

Is this server running without authentication?

I see no communication with Plex.tv

I do see a VERY unstable physical network link

Nov 15, 2019 20:59:02.314 [0x7f66597fa700] DEBUG - Final path: "/usr/lib/plexmediaserver/Resources/Plug-ins-36236cc4c/WebClient.bundle/Contents/Resources/common/img/backgrounds/preset-dark2.6d2c8336496f761f38de8558a2b9709b.png"
Nov 15, 2019 20:59:02.314 [0x7f66597fa700] DEBUG - Content-Length of /usr/lib/plexmediaserver/Resources/Plug-ins-36236cc4c/WebClient.bundle/Contents/Resources/common/img/backgrounds/preset-dark2.6d2c8336496f761f38de8558a2b9709b.png is 113817 (of total: 113817).
Nov 15, 2019 20:59:02.320 [0x7f668641b700] DEBUG - Completed: [157.97.6.16:53483] 200 GET /web/common/img/backgrounds/preset-dark2.6d2c8336496f761f38de8558a2b9709b.png (7 live) GZIP 10ms 113817 bytes (pipelined: 1)
Nov 15, 2019 21:00:06.348 [0x7f6685419700] DEBUG - Statistics: Flushing 2 expired bandwidth entries, 0 expired media entries.
Nov 15, 2019 21:05:50.511 [0x7f6675bd9700] DEBUG - NetworkInterface: received Netlink message len=1328, type=RTM_NEWLINK, flags=0x0
Nov 15, 2019 21:05:50.511 [0x7f6675bd9700] DEBUG - NetworkInterface: Netlink information message family=0, type=1, index=4, flags=0x1002, change=0xffffffff
Nov 15, 2019 21:05:50.511 [0x7f6675bd9700] DEBUG - Network change.
Nov 15, 2019 21:05:50.511 [0x7f6675bd9700] DEBUG - NetworkInterface: Notified of network changed (force=0)
Nov 15, 2019 21:05:50.511 [0x7f6675bd9700] DEBUG - Network change notification but nothing changed.
Nov 15, 2019 21:05:50.511 [0x7f6675bd9700] DEBUG - NetworkInterface: received Netlink message len=1328, type=RTM_NEWLINK, flags=0x0
Nov 15, 2019 21:05:50.511 [0x7f6675bd9700] DEBUG - NetworkInterface: Netlink information message family=0, type=1, index=5, flags=0x1002, change=0xffffffff
Nov 15, 2019 21:05:50.511 [0x7f6675bd9700] DEBUG - Network change.
Nov 15, 2019 21:05:50.511 [0x7f6675bd9700] DEBUG - NetworkInterface: Notified of network changed (force=0)
Nov 15, 2019 21:05:50.511 [0x7f6675bd9700] DEBUG - Network change notification but nothing changed.
Nov 15, 2019 21:05:50.533 [0x7f6675bd9700] DEBUG - NetworkInterface: received Netlink message len=1604, type=RTM_NEWLINK, flags=0x0
Nov 15, 2019 21:05:50.533 [0x7f6675bd9700] DEBUG - NetworkInterface: Netlink information message family=0, type=1, index=5, flags=0x1002, change=0x0
Nov 15, 2019 21:05:50.533 [0x7f6675bd9700] DEBUG - Network change.
Nov 15, 2019 21:05:50.533 [0x7f6675bd9700] DEBUG - NetworkInterface: Notified of network changed (force=0)
Nov 15, 2019 21:05:50.533 [0x7f6675bd9700] DEBUG - Network change notification but nothing changed.
Nov 15, 2019 21:05:50.533 [0x7f6675bd9700] DEBUG - NetworkInterface: received Netlink message len=1604, type=RTM_NEWLINK, flags=0x0
Nov 15, 2019 21:05:50.533 [0x7f6675bd9700] DEBUG - NetworkInterface: Netlink information message family=0, type=1, index=5, flags=0x1002, change=0x100
Nov 15, 2019 21:05:50.533 [0x7f6675bd9700] DEBUG - Network change.
Nov 15, 2019 21:05:50.533 [0x7f6675bd9700] DEBUG - NetworkInterface: Notified of network changed (force=0)
Nov 15, 2019 21:05:50.533 [0x7f6675bd9700] DEBUG - Network change notification but nothing changed.
Nov 15, 2019 21:05:50.533 [0x7f6675bd9700] DEBUG - NetworkInterface: received Netlink message len=1716, type=RTM_NEWLINK, flags=0x0
Nov 15, 2019 21:05:50.533 [0x7f6675bd9700] DEBUG - NetworkInterface: Netlink information message family=0, type=1, index=3, flags=0x1003, change=0x0
Nov 15, 2019 21:05:50.533 [0x7f6675bd9700] DEBUG - Network change.
Nov 15, 2019 21:05:50.533 [0x7f6675bd9700] DEBUG - NetworkInterface: Notified of network changed (force=0)
Nov 15, 2019 21:05:50.533 [0x7f6675bd9700] DEBUG - Network change notification but nothing changed.
Nov 15, 2019 21:05:50.533 [0x7f6675bd9700] DEBUG - NetworkInterface: received Netlink message len=360, type=RTM_NEWLINK, flags=0x0
Nov 15, 2019 21:05:50.533 [0x7f6675bd9700] DEBUG - NetworkInterface: Netlink information message family=7, type=1, index=5, flags=0x1002, change=0x0
Nov 15, 2019 21:05:50.533 [0x7f6675bd9700] DEBUG - Network change.
Nov 15, 2019 21:05:50.533 [0x7f6675bd9700] DEBUG - NetworkInterface: Notified of network changed (force=0)
Nov 15, 2019 21:05:50.533 [0x7f6675bd9700] DEBUG - Network change notification but nothing changed.
Nov 15, 2019 21:05:50.534 [0x7f6675bd9700] DEBUG - NetworkInterface: received Netlink message len=1604, type=RTM_NEWLINK, flags=0x0
Nov 15, 2019 21:05:50.534 [0x7f6675bd9700] DEBUG - NetworkInterface: Netlink information message family=0, type=1, index=5, flags=0x1003, change=0x1
Nov 15, 2019 21:05:50.534 [0x7f6675bd9700] DEBUG - Network change.
Nov 15, 2019 21:05:50.534 [0x7f6675bd9700] DEBUG - NetworkInterface: Notified of network changed (force=0)
Nov 15, 2019 21:05:50.534 [0x7f6675bd9700] DEBUG - Network change notification but nothing changed.
Nov 15, 2019 21:05:50.534 [0x7f6675bd9700] DEBUG - NetworkInterface: received Netlink message len=360, type=RTM_NEWLINK, flags=0x0
Nov 15, 2019 21:05:50.534 [0x7f6675bd9700] DEBUG - NetworkInterface: Netlink information message family=7, type=1, index=5, flags=0x1003, change=0x0
Nov 15, 2019 21:05:50.534 [0x7f6675bd9700] DEBUG - Network change.
Nov 15, 2019 21:05:50.534 [0x7f6675bd9700] DEBUG - NetworkInterface: Notified of network changed (force=0)
Nov 15, 2019 21:05:50.534 [0x7f6675bd9700] DEBUG - Network change notification but nothing changed.
Nov 15, 2019 21:05:51.254 [0x7f6675bd9700] DEBUG - NetworkInterface: received Netlink message len=1604, type=RTM_NEWLINK, flags=0x0
Nov 15, 2019 21:05:51.254 [0x7f6675bd9700] DEBUG - NetworkInterface: Netlink information message family=0, type=1, index=5, flags=0x1002, change=0x41
Nov 15, 2019 21:05:51.254 [0x7f6675bd9700] DEBUG - Network change.
Nov 15, 2019 21:05:51.254 [0x7f6675bd9700] DEBUG - NetworkInterface: Notified of network changed (force=0)
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - Network change notification but nothing changed.
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - NetworkInterface: received Netlink message len=360, type=RTM_NEWLINK, flags=0x0
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - NetworkInterface: Netlink information message family=7, type=1, index=5, flags=0x1002, change=0x0
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - Network change.
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - NetworkInterface: Notified of network changed (force=0)
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - Network change notification but nothing changed.
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - NetworkInterface: received Netlink message len=360, type=RTM_NEWLINK, flags=0x0
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - NetworkInterface: Netlink information message family=7, type=1, index=5, flags=0x1002, change=0x0
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - Network change.
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - NetworkInterface: Notified of network changed (force=0)
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - Network change notification but nothing changed.
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - NetworkInterface: received Netlink message len=576, type=RTM_DELLINK, flags=0x0
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - NetworkInterface: Netlink information message family=0, type=1, index=4, flags=0x1002, change=0xffffffff
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - NetworkInterface: received Netlink message len=360, type=RTM_NEWLINK, flags=0x0
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - NetworkInterface: Netlink information message family=7, type=1, index=5, flags=0x1002, change=0x0
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - Network change.
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - NetworkInterface: Notified of network changed (force=0)
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - Network change notification but nothing changed.
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - NetworkInterface: received Netlink message len=104, type=RTM_DELLINK, flags=0x0
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - NetworkInterface: Netlink information message family=7, type=1, index=5, flags=0x1002, change=0x0
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - NetworkInterface: received Netlink message len=1716, type=RTM_NEWLINK, flags=0x0
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - NetworkInterface: Netlink information message family=0, type=1, index=3, flags=0x1003, change=0x0
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - Network change.
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - NetworkInterface: Notified of network changed (force=0)
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - Network change notification but nothing changed.
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - NetworkInterface: received Netlink message len=576, type=RTM_DELLINK, flags=0x0
Nov 15, 2019 21:05:51.287 [0x7f6675bd9700] DEBUG - NetworkInterface: Netlink information message family=0, type=1, index=5, flags=0x1002, change=0xffffffff
Nov 15, 2019 21:08:46.814 [0x7f6686c1c700] DEBUG - WebSocket: client initiated close
Nov 15, 2019 21:08:46.814 [0x7f6686c1c700] DEBUG - NotificationStream: Removing because of close
Nov 15, 2019 21:08:46.903 [0x7f6686c1c700] DEBUG - handleStreamRead code 2: End of file
Nov 15, 2019 21:08:46.903 [0x7f6686c1c700] DEBUG - Completed after connection close: [10.100.100.2:38616] -3 GET /
21

It was with authentication, cant see if it is active now. Can I check on it via ssh?

22

plex_logs.zip (355.2 KB)

Had no idea I could drop logs here either. Attached. Thanks yet again. Mine should be running authentication. I use it on the road via plex.tv all the time.