Library scanning on mount points from win10 changing between acceptable and very long scanning time

server-linux

#1

I've recently set up a debian machine to run plex on, which I will be getting the library files from a network share on my old windows 10 plex server.

I've shared the folders from windows 10 and mounted them with cifs on debian, but for some of the mount points, plex takes forever to scan the content, and I am also getting some errors saying unable to load files.

How should I troubleshoot this, being a fairly new linux user?


#2

Make certain user plex has permission to read everything on the Windows side of the share. If it gets some but not others, you have mixed permissions over there somewhere.

As for speed issues, anything from how fast the windows server is responding to requests , the network itself, the internet connection on the Linux machine… You need break down the problem and solve one piece at a time starting with the permissions. Next proceed by checking a recursive directory listing (easiest to do) at the command line.


#3

all mount points have been mounted with chmod 755, and chown’ed to my local user on debian.
I “fixed” the issue last night by restarting the plex service several times, but I assume this is not running optimally when it behaves like this.


#4

No. it’s not optimal. File system mounts and scanning should be stable.

Please invoke a scan.
As it starts displaying the issues you see,

Settings - Server - Help - Download Logs

Attach the ZIP so I may see it


#5

It does not display issues, it is just abnormally slow.
I added a folder that has ~5900 files. 16 hours later, 3000 files were added (without metadata)


#6

I suspect your database is VERY fragmented. This happens when adding large amounts of media at one time.

  1. Cancel the scan
  2. Manually optimize the database (Hover over Libraries -> click the ellipsis -> Optimize Database)
  3. Resume scan

#7

I have now tried optimizing the database before rescanning, with these results:
1st hour: 328 items added
2nd hour: 238 more items added

I tried cancelling the scan at this point and re-optimizing the database + restarting the plexmediaserver service, only to see this result the next hour: 162 items.

To compare, the previous library I added, mounted the same way indexed 18000 files with metadata in around 10-12 hours


#8

added: I also tried adding the same folder to emby, which had the entire library scanned and added in around 2,5 hours.


#9

Emby uses a different database engine. It will behave differently.


#10

Still though, 18000 files in 10-12 hours on the first library, and now ~4100 files in 24 hours doesn’t add up.
Since optimizing the database didn’t do the trick, what would be the next step in trying to resolve this issue?

The fact that emby was able to index it within the timeframe it did makes me think it is a plex problem rather than a network issue.


#11

18,000 files at once is definitely not a network issue. Until you told me how many files you were adding in one shot, I didn’t consider it as most problems are network related (PMS over wifi, etc).

Because folks are loading larger and larger libraries now, should Plex change the DB engine? Possibly.


#12

They were not presented all at once, but in smaller bulks as this one (one shared partition at a time).

Since I am building a new library on a new server (switching to linux to hopefully be able to take advantage of distributed transcoding with both computers in the long run, after I’ve restructued the file system with new drives in the new box) I am willing to start from scratch with this library if you see any possibility or locating and fixing the problem.

There’s a few things plex should change in my opinion, but the other bugs I’ve come across has been addressed in other topics (although without reply and solution in most cases), for now the issue I’m most keen on resolving is the slow library scanning, as a couple of houndred files per hour seems off compared to when I am adding items to the already built library on the host machine).


#13

As the maintenance step: In Settings - Server - Scheduled Tasks , is Optimize database every week enabled?
Since no log files have been presented here to verify what is happening, everything stated is therefore assumption until there is evidence.

I do not know what is implied with “distributed transcoding” but the Plex architecture scales no differently on Linux than it does Windows.

I recognize there are outstanding issues. Engineering hours to correct those issues are finite.

Counting all the media in my library, of movies, music, and dvr recordings, yield 49,119 files. All are named by automation to be 100% Plex-recommended naming format and structure. Should I reset the library to zero and begin again, within 8 hours (overnight), all files will have been indexed and properly entered into the database, with artwork retrieved. All this over a 24 Mbps (G.bond ADSL2+) internet service. Upon database rebuild completion, I then query the database for unmatched items. If there are any, that’s a problem. I know this to be true because as part of my QA duties, I frequently build “from scratch” to verify there are no regressions.

How large is your library ?


#14

Logs attached
mounpoint that is added slowly is /mnt/plex/TV-NOR, from the latest activity


#15

distributed transcoding: https://forums.plex.tv/discussion/178320/plex-remote-transcoder-a-distributed-transcoding-backend-for-plex


#16

I will speak to the development team when they are available about your name matching issue.

This is a language issue:

Apr 17, 2018 09:12:00.355 [0x7f0ca67fe700] DEBUG - Migrating metadata settings from local://33261 -> com.plexapp.agents.thetvdb://322078/2/30?lang=no

local:// -> a specific agent means it had the file and took time to resolve it. There is work being done in the resolver.


#17

Seems it is using a long time every time it scans that particular library, when the host server scans it rather quickly and adds the few new items that has been added to the folders.
The only difference between the two computers is that one run in windows 10 and the other one on debian, and that the library is mounted as network drives on the debian box.

It seems to be trying to match all the items every time it scans the library