PMS not steable, freeize once a day Synology DS1817+

Server Version#: PlexMediaServer-1.18.5.2309-f5213a238-x86_64
Player Version#: latest in any platform

CPU and RAM is too high (not in transcoding) and plex web/ plex ios app/ plex android app show cannot connect the server.
Latest two server version are bad.

Plex Media Server Logs_2020-02-02_06-10-07.zip (4.0 MB)

uploaded the log

Before blaming the Plex server, I am concerned you are simply overloading the NAS itself.

Look at all the VM adapters and docker containers defined.

Feb 02, 2020 05:56:47.205 [0x7f42f25c7740] DEBUG - Network interfaces:
Feb 02, 2020 05:56:47.205 [0x7f42f25c7740] DEBUG -  * 1 lo (127.0.0.1) (loopback: 1)
Feb 02, 2020 05:56:47.205 [0x7f42f25c7740] DEBUG -  * 16 ovs_eth0 (192.168.1.125) (loopback: 0)
Feb 02, 2020 05:56:47.205 [0x7f42f25c7740] DEBUG -  * 17 ovs_eth1 (192.168.1.126) (loopback: 0)
Feb 02, 2020 05:56:47.205 [0x7f42f25c7740] DEBUG -  * 18 ovs_eth2 (192.168.1.127) (loopback: 0)
Feb 02, 2020 05:56:47.205 [0x7f42f25c7740] DEBUG -  * 19 ovs_eth3 (192.168.1.128) (loopback: 0)
Feb 02, 2020 05:56:47.205 [0x7f42f25c7740] DEBUG -  * 20 ovs_eth4 (10.0.0.125) (loopback: 0)
Feb 02, 2020 05:56:47.205 [0x7f42f25c7740] DEBUG -  * 21 ovs_eth5 (169.254.68.80) (loopback: 0)
Feb 02, 2020 05:56:47.205 [0x7f42f25c7740] DEBUG -  * 23 docker0 (172.17.0.1) (loopback: 0)
Feb 02, 2020 05:56:47.205 [0x7f42f25c7740] DEBUG -  * 1 lo (::1) (loopback: 1)
Feb 02, 2020 05:56:47.205 [0x7f42f25c7740] DEBUG -  * 16 ovs_eth0 (fd48:4e7d:aa84:0:211:32ff:fe84:2a60) (loopback: 0)
Feb 02, 2020 05:56:47.205 [0x7f42f25c7740] DEBUG -  * 16 ovs_eth0 (fe80::211:32ff:fe84:2a60%ovs_eth0) (loopback: 0)
Feb 02, 2020 05:56:47.205 [0x7f42f25c7740] DEBUG -  * 17 ovs_eth1 (fd48:4e7d:aa84::cfc) (loopback: 0)
Feb 02, 2020 05:56:47.205 [0x7f42f25c7740] DEBUG -  * 17 ovs_eth1 (fd48:4e7d:aa84:0:211:32ff:fe84:2a61) (loopback: 0)
Feb 02, 2020 05:56:47.206 [0x7f42f25c7740] DEBUG -  * 17 ovs_eth1 (fe80::211:32ff:fe84:2a61%ovs_eth1) (loopback: 0)
Feb 02, 2020 05:56:47.206 [0x7f42f25c7740] DEBUG -  * 18 ovs_eth2 (fd48:4e7d:aa84:0:211:32ff:fe84:2a62) (loopback: 0)
Feb 02, 2020 05:56:47.206 [0x7f42f25c7740] DEBUG -  * 18 ovs_eth2 (fe80::211:32ff:fe84:2a62%ovs_eth2) (loopback: 0)
Feb 02, 2020 05:56:47.206 [0x7f42f25c7740] DEBUG -  * 19 ovs_eth3 (fd48:4e7d:aa84:0:211:32ff:fe84:2a63) (loopback: 0)
Feb 02, 2020 05:56:47.206 [0x7f42f25c7740] DEBUG -  * 19 ovs_eth3 (fe80::211:32ff:fe84:2a63%ovs_eth3) (loopback: 0)
Feb 02, 2020 05:56:47.206 [0x7f42f25c7740] DEBUG -  * 20 ovs_eth4 (fe80::a236:9fff:fe28:3680%ovs_eth4) (loopback: 0)
Feb 02, 2020 05:56:47.206 [0x7f42f25c7740] DEBUG -  * 21 ovs_eth5 (fe80::a236:9fff:fe28:3682%ovs_eth5) (loopback: 0)
Feb 02, 2020 05:56:47.206 [0x7f42f25c7740] DEBUG -  * 23 docker0 (fe80::42:40ff:fe07:3f55%docker0) (loopback: 0)
Feb 02, 2020 05:56:47.206 [0x7f42f25c7740] DEBUG -  * 27 docker701aa8a (fe80::b8:10ff:fe75:8cef%docker701aa8a) (loopback: 0)
Feb 02, 2020 05:56:47.206 [0x7f42f25c7740] DEBUG -  * 29 dockerf4a43a2 (fe80::f8aa:7ff:fe4c:8e6d%dockerf4a43a2) (loopback: 0)
Feb 02, 2020 05:56:47.206 [0x7f42f25c7740] DEBUG -  * 33 dockera890375 (fe80::f455:d2ff:feda:e4e2%dockera890375) (loopback: 0)
Feb 02, 2020 05:56:47.206 [0x7f42f25c7740] DEBUG -  * 42 tap0211322db38b (fe80::c8ce:b9ff:fec2:7e7e%tap0211322db38b) (loopback: 0)
Feb 02, 2020 05:56:47.206 [0x7f42f25c7740] DEBUG -  * 43 tap021132252efc (fe80::58af:8dff:fec1:2370%tap021132252efc) (loopback: 0)
Feb 02, 2020 05:56:47.206 [0x7f42f25c7740] DEBUG -  * 60 dockere4dce07 (fe80::44c3:20ff:fe9c:78da%dockere4dce07) (loopback: 0)
Feb 02, 2020 05:56:47.206 [0x7f42f25c7740] DEBUG - NetworkService: Got notification of changed network (first change: 0)

How much memory is in the system?
What are those tasks doing?

It’s all running on a C2538

Feb 02, 2020 05:56:13.683 [0x7f42e7f23700] INFO - Plex Media Server v1.18.5.2309-f5213a238 - Synology DS1817+ x86_64 - build: linux-x86_64 synology - GMT 08:00
Feb 02, 2020 05:56:13.684 [0x7f42e7f23700] INFO - Linux version: DSM 6.2.2.24922-4, language: en-US
Feb 02, 2020 05:56:13.684 [0x7f42e7f23700] INFO - Processor         Intel(R) Atom(TM) CPU  C2538  @ 2.40GHz

The C2738 has a mere 2700 passmarks.

Dear

How much memory is in the system?
8G

What are those tasks doing?
Which tasks ? I checked the PMS app use 25% CPU and 1G+ RAM when the issue appears

Dear

I think the error log happens in “Plex Media Server.2.log” around Feb 02, 2020 03:29:47.
The time around “Feb 02, 2020 05:56:47” is after I restarted the server, the log is normal.

error log, millions lines of :

Feb 02, 2020 03:29:47.356 [0x7f8b0aa3a700] ERROR - Couldn’t accept new TLS connection: Too many open files
Feb 02, 2020 03:29:47.357 [0x7f8b0ad28700] ERROR - Couldn’t accept new TLS connection: Too many open files
Feb 02, 2020 03:29:47.358 [0x7f8b0ad28700] ERROR - Couldn’t accept new TLS connection: Too many open files
Feb 02, 2020 03:29:47.358 [0x7f8b0ad28700] ERROR - Couldn’t accept new TLS connection: Too many open files
Feb 02, 2020 03:29:47.358 [0x7f8b0ad28700] ERROR - Couldn’t accept new TLS connection: Too many open files
Feb 02, 2020 03:29:47.358 [0x7f8b0ad28700] ERROR - Couldn’t accept new TLS connection: Too many open files
Feb 02, 2020 03:29:47.358 [0x7f8b0ad28700] ERROR - Couldn’t accept new TLS connection: Too many open files
Feb 02, 2020 03:29:47.358 [0x7f8b0ad28700] ERROR - Couldn’t accept new TLS connection: Too many open files
Feb 02, 2020 03:29:47.358 [0x7f8b0ad28700] ERROR - Couldn’t accept new TLS connection: Too many open files
Feb 02, 2020 03:29:47.358 [0x7f8b0ad28700] ERROR - Couldn’t accept new TLS connection: Too many open files
Feb 02, 2020 03:29:47.358 [0x7f8b0ad28700] ERROR - Couldn’t accept new TLS connection: Too many open files

What do you mean by this?

May I have the entire DEBUG log files?

If you’ve turned DEBUG logging , please turn it back on. I need DEBUG on but VERBOSE off to see what it’s doing.

Then, after restart, and and it is exhibiting the problem again, capture the logs and attach the ZIP file

Normally, the PMS process (when no Transcoder task) only cost 5%-10% CPU usage and 100MB-200MB RAM uages. But when the PMS freezing, the PMS process will cost 25%CPU (100% usage for one core of C2538) uages and 1GB RAM, that will lead to my issue. It seems an unexpected infinitiy loop ?

Thanks. That helps me a lot. Now I know what to look for when you are able to forward the next set of logs

Do you have the DLNA server active? (Settings - Server - DLNA)

If you have it enabled (it should be disabled by default), and you don’t use DLNA, turn it off to see if that makes any difference

It was disabled.

FYI

the same ERROR log ? The same issue ?

Without knowing what PMS is doing, It could be and maybe it isn’t.

Running out of TLS (secure) connections means the OS itself is depleted.

When open connections are closed, there is a period of time before that TLS slot is released to other applications use.

This is why I asked about the VM and Containers.
It’s also vital I have DEBUG logging so I can see what PMS is trying to do.

  • Is PMS getting behind in handling requests?
  • Between other running tasks and PMS, does the total number of required connections exceed the system’s default limit?

Can you capture the logs I need please?

Dear

It freeze again today. I have to resume the service process manually.


a5623ddf1c4f6bfb30bb87f0b8e8024
0b9563d233187e9561a1b3c564d76c8

Plex Media Server Logs_2020-02-04_19-12-54.zip (4.4 MB)

How many users are connected at once?

By my count from your logs, there are 28.

 Auth: authenticated user 1 as cos003
 Auth: authenticated user 1 as cos003
 Auth: authenticated user 1 as cos003
 Auth: authenticated user 1 as cos003
 Auth: authenticated user 1 as cos003
 Auth: authenticated user 1 as cos003
[chuck@lizum user.363]$ grep 'authenticated user' * | awk -F- '{print $2}' | sort | uniq | wc -l
28
[chuck@lizum user.364]$ 

Also,

  1. Plex Media Server.1.log is missing. Did the machine crash?
  2. How much memory is installed in the unit?

There’s many authenticated users sharing my libery. But, there always no more than 5 active streams at once.
I also set remote streams to “1”, so they cannot make many streams at once. I also set terminate the paused session in 5 mins. So it can’t be to many streams at once unless there’s BUG.

BTW, I found a bug days ago I think maybe useful for you to investigate: last time, I found a user can make more than 1 streams at once, and it did not quit others streams. That time, the user made 8 streams, I was wonding how could he do that. I think may be it is the root cause.

Plex Media Server.1.log is missing. Did the machine crash?
I just kill the freezing PMS in Synology’s DMS GUI. I also wondering where the Plex Media Server.1.log going. They Synology’s DMS is OK,and PMS just freezing, not crashed.

How much memory is installed in the unit?
You mean the Nas ? 8GM RAM. It is far more than enough for the PMS I think.

image

If you’ve found a case where a user can have multiple open streams then you probably have found a bug.

If we can recreate it in a controlled manner, I will get it to Engineering right away to fix.

There are some steps we can take to confirm what’s happening.

First, we help the TCP/IP networking stack to recover more quickly than the default.

Are you familiar and comfortable with working at the Linux command line in the Syno itself (SSH login) ?

SSH is good for me. Any instructions ?

Yes,

Please check /etc/sysctl.conf
I use these two settings in addition to what’s already there.

admin@moesern:~$ cat /etc/sysctl.conf
net.core.somaxconn = 65535
net.ipv4.tcp_tw_reuse  = 1
admin@moesern:~$ 

if you need edit it,

sudo -su root
vi /etc/sysctl.conf
sysctl -p

After making this change, restart PMS.
This change will tell us if connections aren’t able to recycle fast enough.

done.

Is there SSH way to restart PMS ?

image

Yes,

“/var/packages/Plex Media Server/scripts/start-stop-status” stop

Repeat with “start”.