Server Version#: Noticed during 1.14.1.5488
Player Version#: N/A
From what I’ve noticed it is crashing / locking my server during a scan. I have the interval set to every 15 minutes and after about a day it locks up. By that I mean I am unable to SSH into the server and have to hard restart it with the physical power button. The crashing seemed to happen more sooner everytime after. At first it would go for a week and then steadily starting crashing almost daily. I don’t have any data before this server version. I moved my Windows 10 data to Ubuntu and this was the version when that migration happened. Something from the move may have caused this. My database got corrupt during one of these crashes and I had to go to a back up. I’m afraid my current database is corrupt again because of the crashes. Attached are logs from 1.14.1.5488. I have also updated to 1.15.0.647 and 1.15.0.659 with seemingly the same issue happening. I just turned verbose logging off so when I get another crash with the current server version I can upload more logs.
This shows it streaming normally, without the transcoder, and then the system locking up.
The NULL characters at the end are because the kernel locked up mid-write to the disk.
Feb 17, 2019 17:24:24.267 [0x7f2e58ffd700] DEBUG - Request: [127.0.0.1:41150 (Loopback)] GET /:/metadata/updateProgressMessage?message=Scanning%20Family%20Guy%2FSeason%2013 (30 live) GZIP Signed-in Token (freakytoad1)
Feb 17, 2019 17:24:24.267 [0x7f2f00c5a700] DEBUG - Completed: [127.0.0.1:41150] 200 GET /:/metadata/updateProgressMessage?message=Scanning%20Family%20Guy%2FSeason%2013 (30 live) GZIP 1ms 166 bytes
Feb 17, 2019 17:24:24.300 [0x7f2e6d7fa700] DEBUG - Request: [127.0.0.1:41152 (Loopback)] GET /:/metadata/updateProgressMessage?message=Scanning%20Family%20Guy%2FSeason%2012 (31 live) GZIP Signed-in Token (freakytoad1)
Feb 17, 2019 17:24:24.301 [0x7f2f00c5a700] DEBUG - Completed: [127.0.0.1:41152] 200 GET /:/metadata/updateProgressMessage?message=Scanning%20Family%20Guy%2FSeason%2012 (29 live) GZIP 1ms 166 bytes
Feb 17, 2019 17:24:24.786 [0x7f2f0145b700] DEBUG - Auth: authenticated user 10434655 as whitecastle200
Feb 17, 2019 17:24:24.786 [0x7f2e6cff9700] DEBUG - Request: [168.245.154.244:16680 (WAN)] GET /:/eventsource/notifications (29 live) TLS Signed-in Token (whitecastle200)
Feb 17, 2019 17:24:29.458 [0x7f2f00c5a700] DEBUG - handleStreamWrite code 32: Broken pipe
Feb 17, 2019 17:24:29.458 [0x7f2f00c5a700] DEBUG - NotificationStream: Removing because of error
Feb 17, 2019 17:24:34.245 [0x7f2f00c5a700] DEBUG - Completed: [10.0.0.14:61302] -2 GET /player/proxy/poll?deviceClass=pc&protocolVersion=1&protocolCapabilities=timeline%2Cplayback%2Cnavigation%2Cmirror%2Cplayqueues&timeout=1 (17 live) GZIP 20001ms 10 bytes (pipelined: 6)
Feb 17, 2019 17:24:42.670 [0x7f2efb7fe700] DEBUG - [CompanionProxy] player mxbey2agzzefb8ejhwhjeh41 was last refreshed 10 seconds ago
Feb 17, 2019 17:24:43.046 [0x7f2f0145b700] DEBUG - EventSource: Failure in IdleTimeout (0 - Success).
Feb 17, 2019 17:24:43.046 [0x7f2f0145b700] DEBUG - MyPlex: We appear to have lost Internet connectivity, resetting device URL cache.
Feb 17, 2019 17:24:43.046 [0x7f2f0145b700] ERROR - EventSource: Retrying in 15 seconds.
Feb 17, 2019 17:24:46.130 [0x7f2e6dffb700] DEBUG - NetworkServiceBrowser: SSDP departed after not being seen for 21.945803 seconds: 10.0.0.1 (FreeBSD router)
Feb 17, 2019 17:24:46.130 [0x7f2e6dffb700] DEBUG - NetworkServiceBrowser: SSDP departed after not being seen for 21.945833 seconds: 10.0.0.1 (WANDevice)
Feb 17, 2019 17:24:46.130 [0x7f2e6dffb700] DEBUG - NetworkServiceBrowser: SSDP departed after not being seen for 21.945845 seconds: 10.0.0.1 (WANConnectionDevice)
Feb 17, 2019 17:24:46.130 [0x7f2e6dffb700] DEBUG - NetworkServiceBrowser: SSDP departed after not being seen for 29.213143 seconds: 10.0.0.23 (SHIELD)
Feb 17, 2019 17:24:46.130 [0x7f2e6dffb700] DEBUG - NetworkServiceBrowser: SSDP departed after not being seen for 21.950641 seconds: 10.0.0.24 (RX-A780 94117C)
Feb 17, 2019 17:24:46.130 [0x7f2e6dffb700] DEBUG - NetworkServiceBrowser: SSDP departed after not being seen for 21.702541 seconds: 10.0.0.25 ([LG] webOS TV OLED65B7A)
Feb 17, 2019 17:24:46.130 [0x7f2e6dffb700] DEBUG - NetworkServiceBrowser: SSDP departed after not being seen for 21.595654 seconds: 10.0.0.44 (HDHomeRun DMS 104E0C00)
Feb 17, 2019 17:24:46.137 [0x7f2efbfff700] DEBUG - DVR:Device: Discovering and refreshing devices with identifier tv.plex.grabbers.hdhomerun
Feb 17, 2019 17:24:46.138 [0x7f2efbfff700] DEBUG - DVR:Grabber: HDHomerun discovered 0 compatible devices.
Feb 17, 2019 17:24:46.138 [0x7f2efbfff700] DEBUG - DVR:Device: Testing grabber HDHomerun device device://tv.plex.grabbers.hdhomerun/104E0C00 at http://10.0.0.44:80
Feb 17, 2019 17:24:46.138 [0x7f2efbfff700] DEBUG - DVR:Device: Device device://tv.plex.grabbers.hdhomerun/104E0C00 was already known, refreshing database info
Feb 17, 2019 17:24:46.139 [0x7f2efbfff700] DEBUG - HTTP requesting GET http://10.0.0.44:80/discover.json
Feb 17, 2019 17:24:51.140 [0x7f2efbfff700] ERROR - Error issuing curl_easy_perform(handle): 28
Feb 17, 2019 17:24:51.140 [0x7f2efbfff700] DEBUG - HTTP simulating 408 after curl timeout
Feb 17, 2019 17:24:51.148 [0x7f2efbfff700] ERROR - DVR:Device: Error refreshing existing device device://tv.plex.grabbers.hdhomerun/104E0C00, marking as dead.
Feb 17, 2019 17:24:52.670 [0x7f2edffff700] DEBUG - [CompanionProxy] player mxbey2agzzefb8ejhwhjeh41 was last refreshed 20 seconds ago
\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00
There wasn’t a crash log generated so I can’t provide a dump file.
I have media attached via NFS (from Unraid) but the Plex install and metadata resides on the same SSD that Ubuntu is on. I should add that sometimes I do notice that the plex scanner will get stuck scanning a library and has to be canceled for it to move on. Saw this today before the crash.
How would I verify that my ethernet driver is bad / where would I get a good driver?
Here are the results of running ethtool on that ethernet interface. Looks like LRO is already off?
If it truly is my ethernet card would putting in a different card help?
plex@hades:~$ ethtool -k eno1
Features for eno1:
Cannot get device udp-fragmentation-offload settings: Operation not permitted
rx-checksumming: on
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off
rx-all: off
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
Sorry email reply didn’t work. This is what I said:
This was a new server build as of 2 months ago. Plenty of airflow. I’m seeing that my current ethernet driver (e1000e) is 3.2.6-k. Looks like version this is a few years old. I’ll try to update it later tonight and see if that helps anything.
This is a bare metal server ( if I’m using that term correct ) so no virtualization going on. Ubuntu 18.04 is directly installed onto an SSD. The ethernet port is onboard my motherboard which is a Gigabyte H370-HD3. The port is an Intel I219-V. Not sure if this answers your question.
When I saw it last, the Ubuntu ethernet driver had a bug.
From what you’ve shown me here, knowing it’s a bare metal installation and not the result of ESXi emulation misalignmet, The fault can only be in Ubuntu itself.
Feb 19, 2019 06:08:55.251 [0x7fdafce60700] DEBUG - Auth: authenticated user 10434655 as whitecastle200
Feb 19, 2019 06:08:55.252 [0x7fda85ffb700] DEBUG - Request: [168.245.154.244:49220 (WAN)] GET /:/eventsource/notifications (13 live) TLS Signed-in Token (whitecastle200)
Feb 19, 2019 06:08:56.860 [0x7fda68ff9700] DEBUG - Statistics: Flushing 8 expired bandwidth entries, 0 expired media entries.
\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00
It appears 18.04.2 has regressed and that bug has returned.
