Could you confirm that this is normal behaviour? When scanning folders I see in the logs that if a folder has a new file in it, it will access each file in the folder to look for changes. Jun 29, 2020 15:04:02.623 [16344] DEBUG - Directory had 389 files, database had 388 files, can't skip.
If it’s a local or small folder, this is instant. If it’s a remote and a large one, it will take some time to download a piece of each and ever file because of the one new file. Existing files will show this in the log:
Jun 29, 2020 15:04:03.327 [16344] DEBUG - Looking for path match for [C:\TV\003 The Night Before Christmas [1941].avi]
Jun 29, 2020 15:04:03.327 [16344] DEBUG - Path matched, we're reusing media item 1287144
Where new files will be added and scanned fully. Is this normal? Is there any way of changing this behaviour so that it doesn’t need to check every single file in a large remote folder which can take hours vs the milliseconds if it only looked at the file names and added the one new file.
I would and do in my personal media but this is a read-only drive that I’m not the maintainer. Just trying to do the best I can with the limitations but not sure if this is normal or my config isn’t right.
Well, you’re not doing anything wrong particularly - that’s just how Plex works.
Movies in folders that don’t change are excluded from scans. Movies in with other movies when a new one is added - everybody gets frisked - “up against the wall you guys and empty your pockets”.
Damn, I was hoping something was wrong in my setup. It’s annoying that 300 files in a folder need to be partially downloaded from my remote and checked if 1 new one is added. Is this normal on both Windows and Linux systems?
I only recently did the big job myself - after turning 28 libraries into ‘One Each’ for Shows and Movies.
When I got all the movies, not in folders in the same library - my world was destroyed. It took Filebot and me a week or two to get the job done, but now - all is well.
So, yea… that movie in a folder thing does exactly what it says.
If my folder wasn’t read only I could re-organize it. Unfortunately I can’t. Can anyone else confirm that this is normal on Linux too? I could switch my server over if it’s the only option.
Yeah I am using rclone and it’s cache to mount GDrive. It helps but does not eliminate the problem. For example in one folder of 389 files, if it detects a change in number of files, even though all files have cached chunks, attributes and directory structure, it still is connecting to the remote to retrieve pieces of half to a third of files and path matching them to existing files (see below). However, since when you watch videos these larger files will fill the cache and when it hits the max-size, all of the smaller files will be deleted since they’re older. So the cache doesn’t become very helpful in practice.
Jun 29, 2020 15:04:02.272 [16344] DEBUG - Scanner: Processing directory C:\TV\Cartoon Shows\Tom and Jerry Cartoons Complete Collection (1940-2007) [DVDRip] (parent: yes)
Jun 29, 2020 15:04:02.623 [16344] DEBUG - Directory had 389 files, database had 388 files, can't skip.
Jun 29, 2020 15:04:03.327 [16344] DEBUG - Looking for path match for [C:\TV\Cartoon Shows\Tom and Jerry Cartoons Complete Collection (1940-2007) [DVDRip]\003 The Night Before Christmas [1941].avi]
Jun 29, 2020 15:04:03.327 [16344] DEBUG - Path matched, we're reusing media item 1287144
....and so on for all 389 files in the folder. It does this whenever any folder has a change
Ahhhhhhhhhhh barf. I would have hoped/guessed for more sophisticated caching, keeping different buckets for file contents vs. file metadata.
When the scanner identifies that the path matches, does it actually read from the file, or just stat it? If it doesn’t read, I wonder if there are rclone settings (or something else) that would be more effective.
I’ve been starting to think about implementing something similar. Interesting that you’re already doing this.
The best method to ensure a fast library scan is to be compliant with the naming guide.
This means Plex only needs to scan the folder once. But if you have a folder full of unmatched and unmatchable items, it will scan it again and again.
Stop trying to force Plex into matching your incompliant files and folders.
I know that can be a problem, but for example with that folder above, all but one file matched successfully. If I filter out that one file (since I can’t delete) then the folder scans fast, but if I add a new item that it will scan and then successfully match, it still goes through and accesses all the other previously matched files in that folder in the process. That’s the behaviour I want to confirm is normal or abnormal. Those files haven’t changed in any way and are all matched, it’s just one new one was added.
… and how many times shall we tell you that’s how Plex works?
…and how many times shall we tell you to fix your stuff - or deal with the way Plex works?
Yes, we know - been there - fixed it - got over it - not that long ago, actually.
That’s not how you name and structure Tom and Jerry, BTW.
Not even close.
A TV Show Library Folder\
...Tom and Jerry\
......Season 1940\
...........Tom and Jerry - S1940E01.mp4
...........Tom and Jerry - S1940E02.mp4
...........Tom and Jerry - S1940E03.mp4
I know you said you believe that’s how it works but I was looking for confirmation as you also said “Yea, pretty much, but only as far as I know.” Over at the Rclone forums, they’ve said that it does not work this way and my Plex install has something wrong with it.
I know the folder structure is wrong. I can’t fix it, it’s read-only. All of my local files are properly named, with proper folder structure and .nfo files, I can’t do that here. All I wanted is some clarity that this is normal and nothing can be done about it since I’ve been told by 2 senior members at the rclone forums it’s not, and by you here you think it might be. I appreciate the help in any case.
Tom and Jerry is just the example I’ve been working with but there are others that do this too. Even with the naming of these files not being correct, Plex did manage to pick them all up and grab correct metadata so I was told that it reading the files again when there is something new is not normal behavior.
A’ight… if you’re doing something OUTSIDE what is considered Compliant and Plex misbehaves - try something Compliant.
My Dad:
“if at first you don’t succeed - follow the instructions.”
(he never had to assemble an IKEA bookshelf)
(that would have been fun, actually)
(I wish he was here to tackle that Kitchen Cabinet Thing downstairs - the instructions appear to be Off World - and there’s no pictures)
@OttoKerner Awesome, thanks for that suggestion. I didn’t even consider that as an option for fixing the naming. I’ll play around with that and a few other things I’m looking at testing out and report back here so others can see a solution for future reference.
In small tests it seems Plex server on Linux doesn’t have this issue at all and it functions normally with the mount, so it might be isolated to the Windows server. Switching back to Linux (just went Windows 2 weeks ago) might be an option too. I’m adding more content to my test instance to confirm this.
