After music scan completes, RAM usage spikes and NAS becomes unresponsive

Plex Media Server NAS & Devices
1

Server Version#: 1.18.2.2029 on a DS418play running DSM 6.2.2-24922 Update 2

I’ve been running Plex on this Synology for a little under two years now without issue. This problem started after I upgraded Plex from 1.16.5 -> 1.18.2. I’ve tried a few versions of 1.18.2 and am currently running the version listed above – all of the 1.18.2 versions have had this problem.

Whenever I initiate a scan of my music library, the scan takes a very long time – I haven’t timed it out exactly but it’s somewhere in the range of 5-10 hours. A scan of the same library used to take a few minutes.

The scan does seem to eventually complete (the “activity” icon in the web player stops spinning) but after that, Plex is clearly still doing some work (I can hear the Synology working away) and RAM usage then grows to 90% while Plex performance degrades, and then soon after that the Synology becomes almost entirely unresponsive. A full reboot of the Synology fixes the problem, and Plex comes back up just fine.

This last time I left Plex running for a few days without initiating a scan, and all was fine during that time – Plex was working perfectly and RAM usage on the NAS was typically in the 10-20% range. As soon as I did another scan, the problem came back. So as far as I can tell it is definitely related to the library scan.

The problem is confined to my music library – my movie and TV libraries scan without issue, and complete in a minute or so.

I searched around in the forums, the closest issue I found is Excessive Memory Usage; Plex Server 1.18.1.1973 but DLNA has been disabled on my Plex for a long time, so that doesn’t seem to be it.

I’m attaching a download of my debug logs roughly 15-20 minutes after I started a scan. I didn’t add any new music this time – it’s just a rescan where nothing should have changed. At the point I did the log capture the scan appears as though it should have finished – the text displayed when I hover over the Activity icon has stopped changing indicating which music it’s scanning – but I can hear the Synology working, the Activity icon still has a spinning border, and if I click on the music library menu it still says “cancel scan”. (If I wait too long to try to grab logs, the Plex/NAS become unresponsive and I’m not able to collect the logs.)

Any ideas on what Plex is doing, what’s changed, and how I can prevent it from doing that on every single scan?

Thanks for your help!

Plex Media Server Logs_2019-11-20_22-36-41.zip (5.0 MB)

1 Like
2

In case it’s helpful, here’s another log dump ~35 minutes after I started the scan.

Surprisingly this time the scan has actually finished (Activity icon doesn’t show anything and the context menu for the music library includes “Scan Library Files”). Perhaps it finished faster because I didn’t add anything new this time.

However the Synology is still audibly working, RAM usage is ~80%, and Plex performance is starting to degrade – took a very long time to download the logs.

Plex Media Server Logs_2019-11-20_22-52-55.zip (5.3 MB)

3

Overnight I let Plex keep doing whatever it was doing and made sure no clients were trying to play back any media.

This morning, things seem to be mostly back to normal. RAM usage is still a bit high (35%) but Plex appears to be functioning normally.

So it seems that eventually Plex finishes whatever it’s working on. But I would strongly prefer if Plex/my Synology weren’t unavailable for 12 hours every time I want to scan my music library. :slight_smile:

4

As far as i know the new PLEX music scanner uses some of the audio matching tech that used to only be available in the Plex pass music folks. So instead of just looking at the metadata, it plays some of the tracks to listen to it in order to more accurately match it to the album. This takes time. Good news it it seems to be MUCH better than it was when they first rolled this out the first time. I have over 250,000 songs in my library. When the scanner hit some albums in the past and it couldn’t figure out what the album was, it would just hang. Now it not only doesn’ hang, but it got the album right. (specific album in question was a Glenn Miller collected works box set with over 100 tracks). Does it take time? Heck yes.
I tossed up a demo server on a 8 core 16 thread pc and aimed it at my music library. That was several days ago and it is still running. It stopped a few times and I had to kick it off again but it has taken at least three or four solid days of scanning. The good news is the results are pretty good so far. MUCH better than before. And you can always cancel the scan if you need to scan a movie or TV library to add stuff to the server and restart it later.

5

Thanks for the reply!

That’s starting to make some sense – I do see a lot of activity in “Plex Media Scanner Matcher.log” about fingerprinting music. And the activity in the main “Plex Media Server.log” is mostly about those matcher subprocesses finishing.

But this is a pretty old library, and IIUC the matching has been available for Plex Pass users previously, so is it expected that it will take this long on every single scan? Even if no new music has been added?

It might also be worth mentioning that I have many albums that cannot possibly be matched – lots of live recordings (Grateful Dead/Phish/etc.). So could it be that it’s trying to match those albums on every scan?

Is it possible to disable the new matching to determine if that’s the culprit?

6

New set of logs from a new scan this morning. I disabled the lyricfind agent as it was causing a lot of log noise and I don’t use that feature anyway – didn’t have any effect on scan length but logs are easier to read now.

And FWIW most of the “matching” messages in the logs do look to be related to those live shows. But that’s probably most of my library so I can’t be totally sure if it’s only looking at those or not. :slight_smile:Plex Media Server Logs_2019-11-21_10-32-37.zip (6.1 MB)

7

I have a similar issue. It seams like the problem started, when Plex switched the music agent to Musicbrainz.org. If the album/track isn’t found in MB (musicbrainz), it looks like (from the logs), that Plex is doing string matching per track title (levenshtein distance I guess). This results in excessive CPU and memory consumption.

I have 98 albums which can’t be matched. So adding a single album/track, results in PMS running 80-97% CPU for +20 minutes (for the 98 unmatched also - everytime). In the end, I decided to disable “Include music libraries in automatic updates” (under Settings > Library). Which means, I can add multiple albums/tracks and update manually.

8

i know they try to make it unobtrusive, but still this new music scanner is a huge pain.

even if you disable ALL 3rd party metadata, it still scans forever then keep reprocessing files.

its worse on huge libraries mine can take ~24 hours (200k+ tracks) and I have a relatively meaty server.

You can avoid a lot of pain (but not all) with the following server settings.

server > settings > library

  • enable scan automatically
  • enable run partial scan
  • disable include music
  • disable scan periodically (it still will)
  • set scan interval > daily (daily is the max, there is no “NONE”)
9

I’d like to offer a different point of view.

  1. Periodic Scan - OFF (uncheck the box = OFF )
  2. Enable Partial Scan
  3. Curate the music into proper structure (Garbage in = Garbage out)
  4. Don’t expect to throw it all at Plex at once and be instantly available.
    a. Plex fingerprints the first track and last track
    b. If sufficient match confidence if obtained from Naming and Fingerprinting, Album is matched
    c. If not matched, Fingerprint all tracks and try again (this is most lose time due to less-than-optimal curation).

I can create the entire library, from scratch, on a little DS1815+ in 2 hours for just over 800 album sets (which I ripped and curated myself with Picard)

2776 directories, which includes Artist grouping and multi-disc directories.
32680 tracks

[chuck@lizum /vie.202]$ find ./music ./oldmusic -type d -print | wc -l
2776
[chuck@lizum /vie.203]$ find ./music ./oldmusic -type f -print | wc -l
32680
[chuck@lizum /vie.204]$

Plex settings

Screenshot%20from%202019-11-21%2013-56-07
Screenshot%20from%202019-11-21%2013-56-07759×659 134 KB

Picard renaming rules (Plex-compliant)

$if2(%albumartist%,%artist%)/
$if2(%albumartist%,%artist%) - %album%/
$if($gt(%totaldiscs%,1),Disc %discnumber%/,)
$num(%tracknumber%,2) - %title%

Anyone serious about their music will have it curated and partitioned.
It is recommended to add in smaller blocks
There is no limit to the number of ‘blocks’ (sub directories) you can add to a Library section. Use this to your advantage. Add one block, let it finish then EDIT and add the next.

10

I take that back, it seems that disabling the lyricfind agent did help quite a bit – scans seem to take 20-30 minutes now, and CPU/RAM is high that whole time but Plex does stay responsive. Still too long IMO, but way better!

My experience seems to match @mm98’s now – same CPU/RAM/scan time numbers.

FWIW I long ago disabled automatic and periodic library scans, I only trigger scans manually. I think this was recommend in the Synology guide or maybe I just had a bad experience with it. And I don’t mind doing manual scans really.

11

Chuck, the main points of contention are;

  • plex compliant > ridiculous just read the tags, you don’t need to rely on any filename or path if/when the tags exist.
  • my library worked fine BEFORE this new system, my paths/filenames have not changed, it is the new system that is still broken, when compared to the previous (premium or regular).
12

Tekno,
I just work here :smiley:
This is how I’ve always curated my music. For me, nothing’s changed.

Perhaps, since music curation is extremely subjective and there are multiple techniques, by following the documented technique. I don’t even think about tags. They’ve always been a bag of hurt for me.

13

Hi @ChuckPa and thanks for the help!

I’m not sure what the standard is but I feel like I’ve organized and curated my music files quite thoroughly – prior to Plex I had to with so many live shows in my collection, if I ever hoped to find anything.

I think the main sticking point with me (if our theory is true) has to do with the regular attempts to re-match all of my unlikely-to-ever-be-matched albums.

14

I am just the opposite … TAGS are >>>>>> everything else.

it is a POOR assumption that path/filename means anything when there are tags present.

believe, me I fully understand tagging and organization, I have a music library going back to what some claim is the first mp3 ever (toms diner - susan vega).

I’ve used foobar, mediamonkey, picard, and others.

I have my library organized MY way, not PLEX WAY and it was FINE BEFORE.

15

I think the biggest problem with music, unlike video, is there is No Standard

Can you give me some examples of what you’re trying to match which won’t ?

I will see what I can do here by experimentation.

16

makes :popcorn: :wink:

There is an argument floating around that Track numbers aren’t required.
Track numbers have always been part of the spec at Plex.

I won’t enter into the debate as to which is proper but all my CDs have track numbers printed on them. I wonder why? :thinking:

It does seem that Engineering decided to adhere to that spec.
Whether they deviate from that requirement remains to be seen and discussed.
Should there be enough reason to remove the requirement, I’m sure they will; but for now, it’s there.

Sorry :man_shrugging:

17

The main challenge for me is, that there is no way to exclude/skip the unmatched albums. Which means, everytime a change is made in the library - all unmatched albums/tracks are rescanned (up to 3-5 times on a single file change - more info in this thread).

All my albums are structured correctly on disk and contain valid/full ID3 tags.

Albums by single artist
\{artist-first-letter}\{album}\{artist} - {album} ({year})\{track-number} - {title}.ext

Various Artists (Compilations)
\V\Various Artists\VA - {album} ({year})\{track-number} - {artist} - {title}.ext

Would it maybe be an idea, to limit scanning unmatched albums (if not found by first scan) .. to the maintenance schedule? This way, the Plex proxy won’t get hammered all the time (for most users) .. and it would be a much better experience for the users, which has unmatched albums. You would still be able to manually match.

Most of my unmatched albums is by Various Artists (compilations) and local artists/albums (Denmark)

Cancel scanning doesn’t stop the job:

Screenshot_1
Screenshot_1871×574 198 KB

18

When I start and stop the scanner I see lots of “skipped directory - no changes found” sort of stuff so I assume it only scans the new stuff. But that being said, if you have thousands of albums, it takes time even if you skip the unchanged stuff.

19

May I ask an arguably lame question?

How can one possibly listen to “thousands of albums” ?
Granted, it’s not for me to judge but I have music running constantly and I think I repeat once every 4 years.

I ask only because i’m trying to understand what is the norm. Am I in the minority because I only have 800 albums ripped ?

I am asking because I’m looking at this as the computer guy. How much CPU is needed to get X amount of work done; What’s a reasonable about of work per unit time; etc.

20

the same could be asked about movies and tv shows, but that doesn’t stop anyone from collecting thousands of those either.

as for myself, music is usable just about anywhere, as opposed to movies/tv which, at a minimum needs a screen, and some paying attention to.