Iv’e been using plex for a while now, and recently got a Plex Pass and wanted to generate a Premium library. I previously had a standard one, which seemed to work well, but I wanted to update! I erased the old one and started from scratch
My system is a Mac Mini OSX 10.11.4, running as a server, PMS v. 0.9.16.4. The files are located on an NAS. It is a big library, about 900GB, 140k songs, 12K albums. all organized per itunes.
It started building the library fine, but would stop after about 100-200 albums had been scanned. It kept processing those and adding metadata etc, but it would stop scanning any additional files. I could stop the update and restart it but no new material would be scanned. I have to quit the PMS and restart it and re-start the library update. It then scans all the previously gathered files, and usually will scan a few hundred more new ones before stopping again. I can watch the logs in real time and see what it’s doing. I don’t see anything in particular that is stopping the process. There’s not one song or character that I can see that jams it up. It goes further every time, but now after restarting it multiple times over a few days, its only at 1400 albums, it takes 15 minutes reviewing before adding anything else. At this rate it will take a year to complete 12000 albums! FYI I am using the “caffeine” app to keep the computer from sleeping while processing.
Im attaching the latest logs after it stalled…
Any help would be appreciated!
I have about 550GB (800 ripped cds and vinyl rips). I can complete a build of a normal music library overnight. A premium build requires > 24 hours because of all the extra material (lyrics, etc) it’s getting. When I examine the core database (.db) it grows from 100 MB to 140MB for because of that singular change. I have a lot of TV episodes and why the DB is so large to begin with.
Right, I understand it takes a bit to gather all the data for the newly added tracks. I can see that happening in the logs. but after a short time, everything stops, It finishes adding all the metadata, thumbs etc for the few albums it added, but no new tracks are scanned and the process comes to a standstill. the update library circle keeps spinning, but there is no activity on the logs, and the mac activity monitor shows no network or disk i/o, and cpu usage is minimal. I let it run overnight on multiple occasions and nothing new is added after about the first 20 minutes… It’s strictly the music folder it chokes on. I have about 30 movies and tv shows at this point, which refresh in a minute or so.
What is your periodic update interval in settings? I have seen, on larger libraries, where one update gets cancelled because a new update was started.
right now there is no schedule. I have “run partial scan when changes are detected” selected, but no update auto or periodically. I suppose I should try unchecking that, see what happens.
Turn that off please. If it detects a change (meaning something is writing in there) it will restart.
Let’s stabilize the environment first.
I changed the setting, and restarted the computer to try again. I started with 1532 albums in the library I have turned off itunes on the other computer that shares that library so nothing will change mid-scan. it scanned all the previously entered items for 13 minutes, started creating new items for 10 minutes, and then processed newly added files for an additional 11 minutes. ending up with 1581 albums (adding 49 albums total) It then stopped, with the circle still spinning, but no activity in the logs or activity monitor… uggh…
Logs please. If you have hit a heretofore unknown limit, that needs to be addressed
For reference purposes only, how many individual files do you have?
If you can discern, by either count or percentage, FLAC, ALAC, MP3, etc for me that would also help
my total music files are right around 140K, of which 138K are MP3s. around 1700 M4As, 350 M4Ps, a couple other random ones. This last run didn’t add any new music, and generated 3 server logs and 2 scanner logs. I will upload the last ones generated. see if anything is revealed… I appreciate you taking the time to help! this is getting pretty frustrating!
First, to give you some ‘bragging rights’ 
To find the exact count uses ‘find’ and ‘wc’ (word count to count the lines output). It will give you the total
If you have both .mp3 and .MP3, use this
find /Volumes/Multimedia -name ( *.mp3 -o *.MP3 ) -print | wc -l
else
find /Volumes/Multimedia -name *.mp3 -print | wc -l
(you have to escape the characters the shell would otherwise try to interpret)
While you do that, I’ll go digging
Looking over what’s happening and discussing it with others, the only thing that makes sense is you have hit a limit which nobody knows exists.
That said, the question comes in how to recreate what you have. The only way I know is to ask you for a list of the directories and list of the files (in PM of course).
There are two ways, One asks a fair amount of effort of you, the other considerably less.
Option 1: divide & conquer
Make a holding area on the same filesystem as your media so it’s a ‘rename’ and not a ‘copy/move’
Move half your library to that holding area.
Create the premium library and see if it completes
If it does, complete, move a bit back until it breaks again.
Find where it stops and move some back. Make sure you can create fully from scratch.
When that completes, Collect the totals
See where this is going? a value even close to the actual limit is extremely helpful as it gives a sense of scope and we can look the other factors involved with ‘X’ number of files.
Option 2: Recreate your exact library in name only and let us do Option 1 but we watch what happens internally as it does it (those guys have ALL the neat tools haha)
I can help you create those lists easily. You can zip them up and PM them
From those lists, your exact directory structure will be created and a small appropriate audio file (to satisfy the scanner) will be placed; all with your names exactly as is.
Your thoughts?
Thanks Chuck, I appreciate your time and efforts. I’ve been busy and didn’t get to try again for a while. In the meantime I created a normal music library again, just to see what would happen. It worked fine, no need to restart or anything. I still can’t get the premium to go more than a few hundred new albums at a time.
In answer to your question, I would probably prefer “Option 2” and generate the list in name only Since this music folder is shared with other computers and is also the main itunes repository. If I start moving folders I’m afraid I will create more problems elsewhere.
If you would be so kind as to help me generate the list you speak of, I can get that to you and we can see what happens…
Just a thought here:
Whenever I see a premium library scan not finish, it has to do with too many files in a folder.
Let me explain.
Plex Premium loves albums. It was apparently developed and tested by ‘album listeners’.
So that is what works best.
If your files are organised in a folder structure
Artist / Album / Tracks
it works alright.
If you however have several folders by e.g. music style and have 100s or even thousands of files within a single folder, then it will very likely break down during the folder scan.
everything is in artist/album folders. they have the typical number of songs per album, mostly under 20 or so. even the compilations. Nothing in the hundreds. I’ve been scanning the logs looking for anything that seems to lock it up, these are the latest…
edit looking in the server log- there are X-Plex container-start numbers and size at the end of the file denoting the total number of albums after the last scan (4667). I don’t know what the debug lines mean around it, but maybe there is something there?..
Update still no luck with the initial Premium Music library on my home computer, same issues as stated above… eventually I see timouts on different ports in the log, and it shuts down the processes.
I did however create a Premium Music Library on my laptop at work, which has pretty much the exact same files, only the files are on a USB drive, rather than on an NAS… This one is cranking through it. It’s not complete, but its at 6K+ albums now, and still going. The problem seems to be related to the file location. I don’t know what settings in plex or on the NAS or maybe the router I should look at to figure out how to clear the jam…
@smorg007 said:
The problem seems to be related to the file location. I don’t know what settings in plex or on the NAS or maybe the router I should look at to figure out how to clear the jam…
Could it be as simple as directory/file permissions on the NAS? If it were the router, you would not get ANY access I would think
it still mystifies me why it would start, then stop. If it were simply file permissions, none of the files would scan. Since I’ve been working with it, it will go anywhere from 10 to 500 additional albums scanned before it stops again on its own. I need to stop and restart PMS and update the library to get it to add any more of the files from the folder. It still needs to re-scan previously added material before it adds any new. this takes about 15-20 minutes now. The standard music library added everything on the first run,same files.
I tried turning off all the agents I could to see if it would go any further or faster. It did seem to scan more files initially after turning them off, but now it’s back to just a few before it shuts down again. I suspect it’s waiting on information from somewhere and not getting it… I see a lot of this kind of stuff in the server logs when the scanning stops…
“We didn’t receive any data from ::ffff:127.0.0.1:55382 in time, dropping connection.”
