Memory Utilization issue introduced in 1.15 is still present in 1.16

[dugiehowsa]

Server Version#: 1.16.0.1226
Player Version#: N/A
Linux OS: Debian / Linux 4.19.36 #1 SMP Sat May 4 12:07:18 EDT 2019 x86_64 GNU/Linux

Hardware specs:

00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
00:02.0 VGA compatible controller: Intel Corporation HD Graphics 530 (rev 06)
00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller (rev 31)
00:16.0 Communication controller: Intel Corporation Sunrise Point-H CSME HECI (rev 31)
00:17.0 SATA controller: Intel Corporation Sunrise Point-H SATA controller [AHCI mode] (rev 31)
00:1c.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port (rev f1)
00:1f.0 ISA bridge: Intel Corporation Sunrise Point-H LPC Controller (rev 31)
00:1f.2 Memory controller: Intel Corporation Sunrise Point-H PMC (rev 31)
00:1f.3 Audio device: Intel Corporation Sunrise Point-H HD Audio (rev 31)
00:1f.4 SMBus: Intel Corporation Sunrise Point-H SMBus (rev 31)
01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)

I have a Media PC with 4 GB of RAM. I started to notice a Memory Utilization issue in 1.15 where the PMS would use up all the system’s available memory. I recently upgrade to version 1.16, and it looks like the issue still exists. Previous analysis of the logs seemed to indicate an issue with the EPG process. As a workaround, I have been downgrading to 1.14.1.5488 where I have not observed the problem.

One thing I have noticed is that in 1.16, it looks like the server crashed @ 06/26/19 19:26:51, as the PID updated and memory utilization returned to normal. New feature?

I ran the following script to produce the memory utilization logs.

I also added the LogNumFiles=“20” entry to my Preferences.xml file.

Memory Utilization Logs and Plex Logs Attached to ticket.
mem_snapshots.zip (40.3 KB) Plex Media Server Logs_2019-06-27_06-59-35.zip (6.8 MB)

[dugiehowsa]

Unplanned power outage took care of this morning’s needed restart.

Memory Utilization Logs and Plex Logs Attached to ticket.
mem_snapshots_2019_06_29.zip (34.3 KB) Plex Media Server Logs_2019-06-29_07-30-54.zip (7.9 MB)

[dugiehowsa]

Memory utilization got so high last night that neither the local console nor the remote console (via SSH) would respond. I had to reset with the physical power button.

I’ll be upgrading to 1.16.1.1291 this morning and continuing to monitor.

Memory Utilization Logs and Plex Logs Attached to ticket.mem_snapshots_2019_07_02.zip (52.2 KB) Plex Media Server Logs_2019-07-02_07-01-23.zip (8.5 MB)

[dugiehowsa]

This is discouraging. After upgrading to 1.16.1.1291 yesterday, it looks like all the system’s free memory got used up in about 24 hours.

Memory Utilization Logs and Plex Logs Attached to ticket.
Plex Media Server Logs_2019-07-03_08-28-55.zip (7.7 MB) mem_snapshots_2019_07_03.zip (20.7 KB)

[ChuckPa]

I believe you need to update to 1.16.2.1297 for the memory correction.
From what I’ve seen in the forum, there is substantial, if not complete, progress.

[dugiehowsa]

@ChuckPa

1.16.2.1297

Thanks for the update. I upgraded to 1.16.2.1297 earlier today and will provide another update after it runs for a few days.

[dugiehowsa]

After upgrading to 1.16.2.1297, sadly the issue is still present, and I do not notice any improvement in memory performance.

Memory Utilization Logs and Plex Logs Attached to ticket.
mem_snapshots_2019-07-08.zip (26.3 KB) Plex Media Server Logs_2019-07-08_19-06-36.zip (7.9 MB)

[dugiehowsa]

Had to setup a cron job to restart the PMS service every 12 hours.

Memory Utilization Logs and Plex Logs Attached to ticket.
mem_snapshots_2019-07-16.zip (119.6 KB) Plex Media Server Logs_2019-07-16_07-29-20.zip (5.3 MB)

[dugiehowsa]

Was on vacation, so I had a cron job restarting the service at 12 PM (noon) every day while I was away. This automatic service restart allowed the PMS service to run without any issues while I was away.

Upon my return, I removed the cron job, and I upgraded to Version 1.16.3.1402 on July 28, and began to gather statistics again.

The service required a manual restart on the evening of July 30, as the system was all out of memory.

It appears that the service crashed and restarted on the morning of July 30 and the evening of July 31.

The service required a manual restart on the morning of August 2, as the system was out of memory again.

Memory Utilization Logs and Plex Logs Attached to ticket.
mem_snapshots_2019-08-02.zip (97.8 KB)Plex Media Server Logs_2019-08-02_10-41-58.zip (8.1 MB)

[dugiehowsa]

This time around, it lasted almost 3 days before the system ran out of memory.

Memory Utilization Logs and Plex Logs Attached to ticket.
mem_snapshots_2019-08-05.zip (68.0 KB) Plex Media Server Logs_2019-08-05_18-21-30.zip (9.3 MB)

[dugiehowsa]

Modified the script to record Resident memory instead of Virtual memory.

Just before the service is restarted, the script fails occasionally to output data to the file, probably because the server is out of resources.

Let system run until 99% of system memory was utilized. I almost was not able to restart the service from the server command line due to lack of system resources (typing at the server console was slow and delayed).

Memory Utilization Logs and Plex Logs Attached to ticket.
Plex Media Server Logs_2019-08-09_15-26-47.zip (10.0 MB) mem_snapshots_2019-08-09.zip (73.4 KB)

[dugiehowsa]

It appears that the PMS service crashed and restarted on its own twice over the past 4 days. This took care of the need to manually restart the service in order to free up memory.

Memory Utilization Logs and Plex Logs Attached to ticket.
mem_snapshots_plex_2019-08-13.zip (72.6 KB) Plex Media Server Logs_2019-08-13_10-05-52.zip (10.5 MB)