Ok. So My Memory has spiked again:
Here are the files you requested.
Plex Media Server Logs_2019-04-27_13-27-41.zip (6.4 MB)
memory-watcher.log (390.8 KB)
Let me know if you need anything else.
Thanks for the logs.
This wasn’t a spike per se. It was a steady build up from
Fri Apr 26 09:08:40 CDT 2019
total used free shared buff/cache available
Mem: 15913 2648 11884 255 1380 12738
Swap: 2047 0 2047
to
Sat Apr 27 11:58:50 CDT 2019
total used free shared buff/cache available
Mem: 15913 12595 277 237 3039 2751
Swap: 2047 0 2047
Sat Apr 27 11:59:50 CDT 2019
total used free shared buff/cache available
Mem: 15913 12561 1172 237 2178 2785
Swap: 2047 0 2047
This is where it’s just about reached the end of the buffers it can dump.
The last line in the file:
Sat Apr 27 13:26:50 CDT 2019
total used free shared buff/cache available
Mem: 15913 12511 542 237 2858 2835
Swap: 2047 0 2047
Shows it’s close to swapping. When free gets back down to 160-ish again, it will swap.
I would like you to temporarily increase the number of log files retained by the system.
- Stop Plex
- As root, add LogNumFiles=“20” in Prefernces.xml at the end, just before the closing
/> - Start PMS
Now, your logs will have 20 rollovers instead of 5. This will hopefully give us enough of a window.
Please reset your memory-watcher log and begin again.
Thanks for the diagnostics.
I thought we were dealing here with memory issue when there is no DVR. The DVR issue which relates to the periodic incremental update to the guide is known already and the development team are already looking into and reproduced the issue
What i would like to see is any evidence of memory issue when there is no DVR
The diagnostics show the following memory increase pattern
Notable deltas
Date Time mem delta
26/04/2019 09:15:40 +1108
26/04/2019 09:25:40 -741
26/04/2019 13:02:42 +826
26/04/2019 17:02:43 +1012
26/04/2019 21:02:45 +1011
26/04/2019 21:05:45 +1065
26/04/2019 22:22:45 -514
27/04/2019 00:01:46 +1106
27/04/2019 00:07:46 -510
27/04/2019 00:10:46 -635
27/04/2019 00:11:46 +526
27/04/2019 00:21:46 -676
27/04/2019 01:05:46 +986
27/04/2019 03:25:47 +1012
27/04/2019 07:25:48 +997
27/04/2019 11:25:50 +839
I just realized that the script @ChuckPa provided does not actually show the memory used by the Plex Media Server process but is looking at the whole system’s memory usage. Whilst this is good to see, it does not show what the Plex Media Server process is using and cannot therefore be used to prove what Plex Media Server is doing wrong
I would prefer if the script looks at the memory used by the Plex Media Server process as outlined here Memory Full
cc @ChuckPa
Please use this script as requested by sa2000. It provides the output in a format better suited to his analysis.
You can copy/paste this over my previous script in the file and rename the output filename if you wish for your sanity sake.
#!/bin/sh
while sleep 30s
do
now=$(date +"%D %T")
mem=$(ps -p $1 -o drs=)
echo $now "," $mem "\n" >> mem_snapshots.log
done
I copied the code and ran as you described. But now I’m getting an error.
error: process ID list syntax error
Usage:
ps [options]
Try 'ps --help <simple|list|output|threads|misc|all>'
or 'ps --help <s|l|o|t|m|a>'
for additional help text.
For more details see ps(1).
I checked the log and it is writing to the log. So not sure what this means.
I looked right at it and missed. My apologies.
Here is modified to specifically look for and report “Plex Media Server”
#!/bin/sh
while sleep 30s
do
now=$(date +"%D %T")
mem=$(ps -p $(pidof "Plex Media Server") -o drs=)
echo $now "," $mem >> mem_snapshots.log
done
The output file will look like this (Spreadsheet format)
chuck@ubuntu1904:~$ !c
cat *.log
04/27/19 22:26:37 , 2612400
04/27/19 22:27:53 , 2612400
04/27/19 22:28:23 , 2612400
04/27/19 22:28:53 , 2612400
04/27/19 22:29:23 , 2612400
04/27/19 22:29:53 , 2612400
04/27/19 22:30:23 , 2612400
chuck@ubuntu1904:~$
and you can plot it yourself
Date - X axis
Mem - Y axis
Let’s see if that gives the info SA is looking for.
Got it! Thanks!
Seems to be working now.
Plex Media Server Logs_2019-04-29_12-43-50.zip (6.5 MB)
snapshots.log (130.1 KB)
thank you - getting late today. Will try and look tomorrow
Thank you
I thought you were going to increase the number of logs as requested here Memory Full
We have logs covering about 2.5 hours from Apr 29, 2019 10:12 to 12:43
The DVR EPG incremental Refreshes happen every 4 hours
Looking at the memory usage data within the 2.5 hour period covered by the logs, we have
with the significant increase coming in at 10:30:13 (happened between 10:29:43 and 10:30:13)
There was DVR activity and XMLTV related activity but it is not conclusive
We could increase the frequency of the memory capture to be every 10 seconds instead of 30 seconds in the shell script - changing the 30s to 10s
#!/bin/sh
while sleep 10s
do
now=$(date +"%D %T")
mem=$(ps -p $(pidof "Plex Media Server") -o drs=)
echo $now "," $mem >> mem_snapshots.log
done
and also editing Preferences.xml to add the LogNumFiles field with value of 20 as suggested by @ChuckPa adding
LogNumFiles="20"
Please edit the Preferences.xml only when PMS is shutdown
This shows the major memory usage changes over 36 hour period
and last set of changes - 2 big changes we do not have logs for
I must have missed the LogNumFiles part.
I have updated the Preferences.xml to include LogNumFiles=“20” and updated the memory-watcher from 30s to 10s.
I will restart and post my updated numbers.
Thanks
The double quote needs to be the same as the others in the file. For some reason wrong character appears here
Needs to be "20" and not “20” - the latter looks like what it would like in MS Word
I double checked. Looks correct in the file.
LogNumFiles="20"/>
Not sure why it came out that way on here.
Update!
Plex Media (14.8 MB)
snapshots.log (258.4 KB)
Thanks for the logs and memory snapshots for the Plex Media Server process
So server was launched just before May 01, 2019 16:53:32 with 10-second memory snapshots for the Plex Media Server process from 16:55:33 up to May 2nd 18:49:17
During the snapshots period memory use for the process went up from 3,654,564 Kb (about 3.4Gb) to 14,049,716 Kb (about 13.3 Gb)
We have logs covering period May 2nd 07:34:54 am to 18:46:31. During the period of time covered by the logs, the memory snapshots shows increase from 10,617,896 Kb (about 10.1 Gb) to 14,049,716 Kb (about 13.3 Gb) - about 3Gb increase
DVR Guide refreshes happen by default every 4 hours plus one refresh for scheduled tasks. The 4-hour refresh is adjusted after the scheduled task is run - so the pattern would be every 4 hours until scheduled task runs and then every 4 hours from that time.
The period covered by the logs show 4 Guide Refreshes taking place and memory usage increased with 3 out of these 4 refreshes. So Guide Refresh activity is still suspect - but we would need to understand why one of the refreshes did not. There was a manual guide refresh and also a DVR setup which would recreate the guide - so we do not have just the 4-hourly incremental updates to the guide here
These charts show the increases - and as mentioned 3 of the increases correspond to DVR Guide and channel setup activities
This shows the times and what is implicated
The issue with suspected memory leaks relating to DVR Guide activities is already known and being looked into. The only anomaly we have here was the 17:31 Guide incremental refresh which did not appear to increase memory
Whilst the issue remains unresolved and under investigation by the development team, please schedule restarts of the server every 24-48 hours
For the periods before the time covered by the logs, there were 3 memory increases that clearly relate to the Guide refreshes. They appear at 4-hour intervals which must have been for the guide auto incremental refreshes
Thanks for all the diagnostics
since this does not seem to be happening to everyone with a dvr, perhaps it might be good to narrow which specific guide/area is being downloaded.
it would seem likely there is malformed xml that is causing the memory spikes.
could be something along those lines. It has been seen to be more severe with XMLTV.
@mshoward82 could you make your xmltv file available please and i will include it with the evidence that i am adding to the existing issue that the development team are working on
I’ve been monitoring this thread, as I am having similar issues. Since my system has only 4 GB of RAM, it occurs with greater frequency.
LIVINGROOM:~/# uname -a
Linux LIVINGROOM 4.19.23 #1 SMP Thu Feb 21 00:02:50 EST 2019 x86_64 GNU/Linux
LIVINGROOM:~/# lspci
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
00:02.0 VGA compatible controller: Intel Corporation HD Graphics 530 (rev 06)
00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller (rev 31)
00:16.0 Communication controller: Intel Corporation Sunrise Point-H CSME HECI (rev 31)
00:17.0 SATA controller: Intel Corporation Sunrise Point-H SATA controller [AHCI mode] (rev 31)
00:1c.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port (rev f1)
00:1f.0 ISA bridge: Intel Corporation Sunrise Point-H LPC Controller (rev 31)
00:1f.2 Memory controller: Intel Corporation Sunrise Point-H PMC (rev 31)
00:1f.3 Audio device: Intel Corporation Sunrise Point-H HD Audio (rev 31)
00:1f.4 SMBus: Intel Corporation Sunrise Point-H SMBus (rev 31)
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
I tried to edit the Preferences.xml file to keep 20 logs files, but unfortunately that setting gets over-written when the service restarts. Still, there should still be enough log files present to track from service restart to memory exhaustion.
Please find my memory snapshots and PMS debug logs attached.
20190505_mem_snapshots.log (225.9 KB)
20190507_mem_snapshots.log (306.2 KB)
Plex Media Server Logs_2019-05-05_21-09-54.zip (2.9 MB)
Plex Media Server Logs_2019-05-07_07-58-44.zip (3.6 MB)
If you edit Preferences.xml when PMS is shut down, and the syntax is correct, it will persist.
LogNumFiles="20"
Needs to be "20" and not “20” - the latter looks like what it would like in MS Word
