Server Version#: 1.19.5.3112
Player Version#: Web 4.39.1
System Information:
CPU: Dual Xeon X5670
RAM: 36 GB DDR3 ECC
OS: FreeNAS 11.3 Stable
When performing scanning, PMS quickly eats through all available memory and swap before crashing. This frequently causes FreeNAS to crash as well, although sometimes it manages to terminate the scanner process. I’ve attached the logs after the most recent crash.
Same issue here, I had to roll back to 1.19.4.2935 what also rolled back the new version of the movie database. So can’t tell for sure right now this is related to the new version or to the upgraded database and new scanner.
That’s interesting.
On an underpowered FreeNAS box with 4GB of RAM, I see Plex Media Server 1.20 using about 270M-300M. It goes up when scanning and goes back down after sitting idle for a few minutes. Each Plex Script Host uses 75-90M while it’s active.
I was using the preview with the new scanner for a while, and had duplicated libraries, both old and new. Now I only have the new libraries & scanner.
I don’t have a Music library configured at the moment, if that’s a difference. I do have Movies, TV, Photos.
Do you have the DLNA server enabled? There have been other “runs out of memory” posts related to that.
I don’t have photos, only Movies, TV, and Music. I saw those posts too, DLNA is not on.
I will try rolling back, but I think it was happening on that version as well.
I notice you have Verbose logging enabled. Unless one of the Plex folks asks for Verbose logging, you probably want to turn it off. It can cause so much extra noise in the logs that it makes them less useful - even causing useful stuff to be lost because it gets rotated out too quickly.
(Keep Debug on. Just turn Verbose off.)
The rest of your log files seem to have a good variety of “stuff” in them.
I notice that the Deep Analysis logs start with this, always the same item id.
Aug 01, 2020 02:24:24.904 [0x809708900] INFO - /usr/local/share/plexmediaserver/Plex Media Scanner --analyze-loudness --force --item 15360 --log-file-suffix Deep Analysis
Aug 01, 2020 02:17:38.633 [0x80f53f900] ERROR - Analysis: Format error 'Cannot allocate memory'.
And then it just repeats that last “Cannot allocate memory” error forever, filling up the logs quite quickly. It fills them up so quickly that they’ve rolled over and don’t go as far back as the other logs.
I don’t know if that implies a problem with item 15360, or if it’s a symptom itself.
Does this show you which item it is?
https://192.168.1.170:32400/library/metadata/15360
I’m not guessing this is the core issue, but something’s wrong with this file:
Aug 01, 2020 02:16:34.915 [0x809618000] ERROR - Exception analyzing media file '/mnt/iTunes/James Taylor/Greatest Hits/01 Something In The Way She Moves 2.m4a' (Could not parse /mnt/iTunes/James Taylor/Greatest Hits/01 Something In The Way She Moves 2.m4a (error=-1094995529): Invalid data found when processing input)
Aug 01, 2020 02:16:34.915 [0x809618000] ERROR - Failed to successfully analyze part 22766.
Item 16360 ended up being just another song in the same library:
<MediaContainer size=“1” allowSync=“1” identifier=“com.plexapp.plugins.library” librarySectionID=“5” librarySectionTitle=“iTunes” mediaTagPrefix=“/system/bundle/media/flags/” mediaTagVersion=“1595237601”>
<Directory ratingKey=“15360” key=“/library/metadata/15360/children” parentRatingKey=“13931” guid=“local://15360” parentGuid=“plex://artist/5d07bbfc403c6402904a5ec9” type=“album” title=“3ala Kaifak (1) [2002]” parentKey=“/library/metadata/13931” librarySectionTitle=“iTunes” librarySectionID=“5” librarySectionKey=“/library/sections/5” parentTitle=“Various Artists” summary=“” index=“1” thumb=“/library/metadata/13931/thumb/1596184849” art=“/library/metadata/13931/art/1596184849” parentThumb=“/library/metadata/13931/thumb/1596184849” leafCount=“14” viewedLeafCount=“0” addedAt=“1136146864”>
<Genre id=“8367” filter=“genre=8367” tag=“Unknown”/>
</Directory>
</MediaContainer>
I think it just happened to be on that file when it ran out of memory. I also checked the second file listed, it played just fine through Plex, and all the correct metadata displays.
I think there are some other folks around who are more likely to be able to help you - and this might need the Plex folks. I was hoping for an “easy” solution. 
Have you considered taking 32GB of RAM out? After all, 4GB ought to be enough for anybody.
Obviously that’s a joke, but it makes me think of something. Are you able to trigger this on demand? Have you been able to actually see Plex using more memory when it happens? I wonder if it’s actually Plex causing the problem, or if it’s just the victim/symptom.
htop is a decent way to watch.
Another possibility is that it isn’t Plex, but the ZFS ARC is eating it all. It’s supposed to release memory when needed but it doesn’t always give back to the community.
On FreeNAS systems with more than 4GB (you know, most of them except mine …), I’ve usually had to tune the ZFS arc_max, because the default of all-but-1GB could hurt other processes.
You could run this and see if it addresses your problem; if you like the change you would add it as a tunable in the Freenas GUI.
The first to see what it’s currently set to, the second to set it to 28GB, if you like. This doesn’t reserve 28GB for the ARC, it sets a lower limit than you (probably) currently have.
28 (bytes)*1024 (kilo)*1024 (mega)*1024 (giga) = 30064771072
# sysctl -a vfs.zfs.arc_max
# sysctl vfs.zfs.arc_max=30064771072
In my case it happened every night during the Plex maintenance window, so I was greeted every morning with a non-working Plex or worst, a non-working nas that needed a power cycle.
I wasn’t able to reproduce it manually with scans or metadata refreshes from the gui.
So my ZFS ARC idea is obviously just a hunch, and the best thing would be to get real data / logs / evidence of exactly which processes are using the memory, the errors they’re encountering, etc. I’m still hoping a Plex guru/employee/admin/wizard will pop up and say something more coherent.
When Plex is doing maintenance & analysis, that’s a pretty heavy use of the filesystem. Plex is asking for files and data as fast as it can, ZFS is fetching them and caching them as quick as it can … maybe it’s just a perfect storm. “Stop hitting yourself”, as it were.
Putting a slightly lower limit on the ZFS ARC is something that’s easy to test, anyway.
It’s also been during the maintenance window for me. I’ll try reducing the ARC limit and seeing if there’s a crash this week. (Although the crashing is inconsistent anyways, so it may be hard to tell)
That explanation makes sense. I occasionally run memory and file IO intensive programs in other jails and have not run into issues like this, so I had just assumed that ZFS was smarter about shrinking the ARC when resources were low. But these things do happen, so we will see.
Thanks for your help so far.
I tried reducing the ARC size as suggested (I did 24 GB) and there was another crash this moring. I was looking in the FreeNAS logs, and I saw that ARC usage spiked right as the scan started, which makes sense, and then shrank as the scan continued all the way down to 2GB when the system crashed.
I guess I’ll try downgrading Plex versions now.
I did the following upgrade today after confirming that the rollback fixed my issues:
plexmediaserver-plexpass upgraded: 1.19.4.2935 -> 1.20.0.3133
This time however, I didn’t upgrade a library with the new ‘upgrade matching’ feature. Will report back if I wake up to a happy Plex or an angry box.
I added an ~80GB music library to see if I’d have the same issues as others.
While adding/scanning/matching, my Plex Media Server process went up and down, like it usually does. Now it’s sitting at about 300M, like usual.
The [com.plex.system] plug-in crept up to 470M and it’s sitting there. A couple of the others crept up quite a bit too.
Obviously that isn’t very much for the year 2020, but on my toy 4GB system it starts to be significant. That’s a lot resident in those python processes.
Will see if it survives an overnight maintenance.
Edit: Mine survived the night.
During the maintenance tonight the swap was once again completely eaten up, the system managed to stay online by killing the offending process.
swap_pager_getswapspace(32): failed
pid 91558 (Plex Media Server), jid 5, uid 972, was killed: out of swap space
I am rolling back to 1.19.4.2935 as there is not much left for me to test now.
Does Plex has a public bug tracking system in place or a way -other than the forum- to bring this issue to the developer’s attention?
I just noticed a similar thread about OOM issues during maintenance.
Thanks @Volts for looking into this as well. I guess I’ll upgrade once more and disable “Generate intro video markers”… Will report back.
Hey that’s me!
Hi @styno, I’m glad to learn I’m not the only person suffering from this issue. As more users discover this problem I’m hoping Plex will devote some man power to finding a proper resolution.
As far as I know there is no where to report issues with Plex besides the forum here. As the saying goes the squeeky wheel gets the grease so if more users bring up this issue it is more likely to be addressed.
I’m having roughly the same issue. Every other day during the maintenance window my resource usage goes off the charts. I’m running Plex on macOS. It’s not containerized, so after about 30-40 minutes the computer eventually panic and reboots.
@mikec_pt is this something you would be able to help troubleshoot, or could you help wrangle up people who could?
