Ubuntu 22.04 - Slow scanning of libraries

Plex Media Server
1

Server Version#: Version 1.31.2.6757
Player Version#: Version 4.102.1

I’m experiencing performance regression of the plex scanner. For me it started approx 1 week ago. Below are some timings from my three libraries …

Movies:

  • Mar 07, 2023 09:05:36.593 [0x7fdc5cbedb38] DEBUG - Activity: updated activity de81327c-36b3-4362-b22f-6401cf4cb17c - completed 0.0% - Scanning Movies
  • Mar 07, 2023 09:06:36.291 [0x7fdc5cbedb38] DEBUG - Activity: updated activity de81327c-36b3-4362-b22f-6401cf4cb17c - completed 100.0% - Scanning Movies

Music:

  • Mar 07, 2023 09:06:36.298 [0x7fdc5cbedb38] DEBUG - Activity: updated activity 189a7054-907d-4cee-b304-cd5ef804c5b0 - completed 0.0% - Scanning Music
  • Mar 07, 2023 09:13:55.491 [0x7fdc5cbedb38] DEBUG - Activity: updated activity 189a7054-907d-4cee-b304-cd5ef804c5b0 - completed 100.0% - Scanning Music

TV Shows:

  • Mar 07, 2023 09:13:55.903 [0x7fdc5cbedb38] DEBUG - Activity: updated activity dc8f286d-87c2-4a88-9844-6147a1b258f5 - completed 0.0% - Scanning TV Shows
  • Mar 07, 2023 09:16:43.384 [0x7fdc5cbedb38] DEBUG - Activity: updated activity dc8f286d-87c2-4a88-9844-6147a1b258f5 - completed 100.0% - Scanning TV Shows

For context:

  1. I have 3662 Movies, 32035 TV Episodes and 81357 tracks.
  2. Media is on multiple NAS devices which are SMB mounted to my physical server. Settings Scan my library automatically, Run a partial scan when changes are detected and Scan my library periodically are all off.
  3. I always initiate a scan manually after adding media. Nothing else outside of my plex server has access to the media shared.

Up until this started happening it would take between 60-120 seconds to scan my three libraries with no media added or changed. Now its taking 10+ minutes but I’ve seen it take more than 15.

My media is well structured following the plex guidelines. I even setup a brand new test server, scanned all my media from scratch (disabling all the scheduled heavy tasks) and it made no difference. I’m on linux (ubuntu 22.04).

BTW… what I’m experiencing is exactly the same as this thread Every folder in TV library being scanned individually = VERY slow scanning which closed unresolved.

2

Some additional investigation:

If I umount all my NAS points and then mount them again, while leaving plex running and then immediately do a scan, the scanner runs perfectly and back to its normal speed levels. Evidence below:

Movies:

  • Mar 07, 2023 11:47:02.074 [0x7f2a88e7db38] DEBUG - Activity: updated activity b660c108-db2a-42ba-acc2-d7c82ba8ce15 - completed 0.0% - Scanning Movies
  • Mar 07, 2023 11:47:13.131 [0x7f2a88e7db38] DEBUG - Activity: updated activity b660c108-db2a-42ba-acc2-d7c82ba8ce15 - completed 100.0% - Scanning Movies

Music:

  • Mar 07, 2023 11:47:13.139 [0x7f2a88e7db38] DEBUG - Activity: updated activity a315ba61-f033-4dfa-90b5-313351e497bd - completed 0.0% - Scanning Music
  • Mar 07, 2023 11:48:35.485 [0x7f2a88e7db38] DEBUG - Activity: updated activity a315ba61-f033-4dfa-90b5-313351e497bd - completed 100.0% - Scanning Music

TV Shows:

  • Mar 07, 2023 11:48:35.872 [0x7f2a88e7db38] DEBUG - Activity: updated activity 29226177-ff77-460c-b3d0-a6af1a546bb1 - completed 0.0% - Scanning TV Shows
  • Mar 07, 2023 11:49:00.148 [0x7f2a88e7db38] DEBUG - Activity: updated activity 29226177-ff77-460c-b3d0-a6af1a546bb1 - completed 100.0% - Scanning TV Shows

That’s a total scan time of 120 second. Again, all I done was un-mount and re-mount the NFS points, nothing else.

Looking back at my system updates I now see a kernel update to version 5.19.0-35 was done around the same time I started seeing this behaviour so its possible this is a factor. May also be just a coincidence. Also, doing another scan again (after this one) results in the slow scanner performance again.

Anyhow, anyone else experienced this on linux based systems ? @ChuckPa (sorry, hate pinging folks) but would you have any perspective on this.

3

I don’t mind you pinging me but why are all your issues weird ones? :wink:

I’m going to need the DEBUG logs - can’t pick two excepts.

Also need a ls -la of the DB dir.

I smell a rat here

4

I’m curious, if you perform the above and then run something like the following twice (referencing your media directories) do you see any speed difference between the first and second run?

find ./tv -type f -exec ls {} \;

This runs ls on every file in the specified directory and its subdirectories. It doesn’t really “do” anything useful, but it does traverse the filesystem via the network share. Not sure if it will show anything useful.

5

@ChuckPa I certainly only seem to find the hard stuff and this one has me scratching my head!

Done the following this morning…

  • stopped plex, scrubbed logs (so you can have clean unpolluted logs), umounted & remounted my NAS mounts and started plex again
  • ran a full scan of my three libraries - plex zipped through the scan as it would always (90 seconds) - timestamp from logs as reference:
Mar 08, 2023 08:09:08.981 [0x7f5661d2ab38] DEBUG - Activity: updated activity 2fd11e99-73c0-42f2-a0d9-e115f4876e53 - completed 0.0% - Scanning Movies
Mar 08, 2023 08:10:32.995 [0x7f5661d2ab38] DEBUG - Activity: updated activity c6106c3d-1a30-48fe-b5f0-3d7dd5c3fd77 - completed 100.0% - Scanning TV Shows
  • waited 30 seconds and then ran a second full scan of my name three libraries - plex then unfortunately slow slogged through the scan (17 minutes)
Mar 08, 2023 08:11:14.139 [0x7f565e017b38] DEBUG - Activity: updated activity 4068be63-4c9d-40d3-91a5-1d81ac32599e - completed 0.0% - Scanning Movies
Mar 08, 2023 08:28:43.301 [0x7f565e017b38] DEBUG - Activity: updated activity 33ffeb68-3435-4788-93a9-b5335ba9bd3e - completed 100.0% - Scanning TV Shows

Full logs attached below along with ls -la of the DB directory below:

xxxx@dokuro:/data/Personal/Scripts$ ls -la "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/"
total 497496
drwxr-xr-x 2 plex plex      4096 Mar  8 08:23 .
drwxr-xr-x 7 plex plex      4096 Mar  4 10:51 ..
-rw-r--r-- 1 plex plex   1040384 Mar  6 10:39 com.plexapp.plugins.library.blobs.db
-rw-r--r-- 1 plex plex     32768 Mar  8 08:08 com.plexapp.plugins.library.blobs.db-shm
-rw-r--r-- 1 plex plex         0 Mar  8 08:08 com.plexapp.plugins.library.blobs.db-wal
-rw-r--r-- 1 plex plex 506609664 Mar  8 08:08 com.plexapp.plugins.library.db
-rw-r--r-- 1 plex plex     32768 Mar  8 08:15 com.plexapp.plugins.library.db-shm
-rw-r--r-- 1 plex plex   1705712 Mar  8 08:15 com.plexapp.plugins.library.db-wal
xxxx@dokuro:/data/Personal/Scripts$

Plex Media Server Logs_2023-03-08_08-28-58.zip (833.7 KB)

@pshanew yeah, this is a good data point and its something I tested myself yesterday as I was thinking it could be some network bottleneck but alas does not seem to. Basically 5 second directory find and 48 seconds files find.

time find /data/MediaServer/{4a-Music/Music\ Collection,14a-Movies,14b-Movies,8a-TV,8b-TV,8c-TV,8d-TV} -type d -exec ls {} +
real	0m7.389s
user	0m0.592s
sys	    0m0.746s

time find /data/MediaServer/{4a-Music/Music\ Collection,14a-Movies,14b-Movies,8a-TV,8b-TV,8c-TV,8d-TV} -type f -exec ls {} +
real	0m48.181s
user	0m2.020s
sys	    0m3.471s

Ran this a few times and times before and after above tests and results are always +/- 5% of this.

1 Like
6

Could you try ls -l instead in the find command?

1.31.0.6654 did have the permissions changes for library updates and maybe that caused a regression for the regular scans too?

7

Having chronic issues here as well.

Ubuntu, nfs mounts, and it took days to scan my TVSeries tree. Used to take only a few hrs, tops from an empty DB.

total 406912
drwxr-xr-x 2 plex plex 4096 Mar 12 03:25 ./
drwxr-xr-x 7 plex plex 4096 Mar 11 21:27 …/
-rw-rw-r-- 1 plex plex 4972544 Mar 13 14:38 com.plexapp.plugins.library.blobs.db
-rw-r–r-- 1 plex plex 701440 Mar 12 03:24 com.plexapp.plugins.library.blobs.db-2023-03-12
-rw-rw-r-- 1 plex plex 32768 Mar 13 16:11 com.plexapp.plugins.library.blobs.db-shm
-rw-rw-r-- 1 plex plex 1195800 Mar 13 16:11 com.plexapp.plugins.library.blobs.db-wal
-rw-rw-r-- 1 plex plex 178078720 Mar 13 16:14 com.plexapp.plugins.library.db
-rw-r–r-- 1 plex plex 50765824 Mar 12 03:24 com.plexapp.plugins.library.db-2023-03-12
-rw-rw-r-- 1 plex plex 1376256 Mar 13 16:17 com.plexapp.plugins.library.db-shm
-rw-rw-r-- 1 plex plex 179526624 Mar 13 16:17 com.plexapp.plugins.library.db-wal

Version 4.102.1

Ubuntu 20.04

A rescan of a completed Media tree runs fast…not slow at all, seconds.

8

In the last 24hrs, I have been able to index -1- music track every -14- seconds.

Everything is on NVME, on a lot of CPU, 64G of ram.

118956 total music entries that I can “ls -lRat” out in 14 seconds.

9

Unfortunately real life has not given me much time to troubleshoot. One thing I’ve noticed is that adding new content is as fast as normal, at least for me. It’s scanned and matched as normal, blistering fast.

However when it’s processing unchanged folders it’s extremely slow. Whatever the scanner is doing has a regression in it or it’s being impacted somehow.

10

@ChuckPa as an FYI.

For me at least the issue is now resolved and I’m back to normal speedy scan speeds.

The problem was caused by a regression in kernel 5.19.0-35 which is now fixed in 5.19.0-38. Looking at the change log from UbuntuUpdates - Package "linux-modules-5.19.0-38-generic" (kinetic 22.10) it appears there was nfs deadlocks & cache timing issues.

Anyhow, issue resolved so marking as such.

3 Likes
11

Thank you !

That explains why I couldn’t reproduce it on my workstation or in my VMs.

1 Like
12

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.