Partial Scan not working

Johnsen1408 · December 23, 2018, 8:59am

With partial scan activated plex still scans all 4000 directories (15 minutes) to pick up the one that changed.

Plex Media Scanner v1.14.1.5488-cc260c476 - ubuntu PC x86_64 - build: linux-ubuntu-x86_64 - GMT 01:00
Linux version: 4.15.0-42-generic (#45-Ubuntu SMP Thu Nov 15 19:32:57 UTC 2018), language: en-US

Plex Media Scanner.log (4.6 MB)

Johnsen1408 · December 23, 2018, 9:08am

Looking thru the logs; this looks a little strange to me:

Dec 23, 2018 09:06:24.445 [0x7f2a02d89840] DEBUG -   * Scanning /media/Movies
Dec 23, 2018 09:06:24.453 [0x7f2a02d89840] DEBUG -     * Scanning directory /media/Movies (parent: no)
Dec 23, 2018 09:06:24.454 [0x7f2a02d89840] VERBOSE -     * Noting that '' was updated at 2018-12-23 08:48:04
Dec 23, 2018 09:06:24.454 [0x7f2a02d89840] VERBOSE -     * Taking directory '' (4) out of the map (3928 left)

The top folder is marked as parent: no and subsequent two entries with empty ‘’

ChuckPa · December 23, 2018, 9:43am

The problem is the /media directory.

Gnome/Nautilus takes exclusive ownership of everything under /media.

All the content is left because can only see the top most level.

Make a manual mount and mount the disk elsewhere and it will work fine.

This occurs all the time with non-FSTAB mounted USB drives

The procedure is here.

Johnsen1408 · December 23, 2018, 10:07am

I’m running the server version so no Gnome & Nautilus.

plexmediaserver is running as user aj:
130 aj@Plex:~$ cat /etc/systemd/system/plexmediaserver.service.d/override.conf
[Service]
User=aj
Group=aj

And aj is set as owner:

aj@Plex:~$ ls -al /media
total 8
drwxr-xr-x 5 root root 4096 Nov 4 18:23 .
drwxr-xr-x 22 root root 4096 Dec 8 09:08 …
drwxr-xr-x 2 aj aj 0 Dec 23 08:48 Movies
drwxr-xr-x 2 aj aj 0 Nov 4 18:22 RecyclingBin
drwxr-xr-x 2 aj aj 0 Dec 23 08:15 ‘TV Shows’

wperdue · December 23, 2018, 5:05pm

I think I’m having a similar problem. Running Version 1.14.1.5488. I have just over 7500 movies and 440 television series. When I move over a show or movie from my download directory, it can take anywhere from 30 minutes to several hours to scan and show the new content. I have partial scanning enabled, and everything is in a “movie” or “television” directory on a series of 8TB WD hard drives.

However, if I exit Plex in the taskbar then restart it, it picks up everything in under a minute. What can I change that will make this happen without having to constantly stop and start Plex? Thanks.

ChuckPa · December 24, 2018, 2:34am

If you run a find /dir1 /dir2 /dir3 -type d -print | wc -l (put in the real Top Level directory names for dirs 1-3) it will tell you how many directories are in use.

If the value is greater than 8192, your /etc/sysctl.conf needs increasing.

Your logs will confirm if iNotify has “Permission denied” errors.

Johnsen1408 · December 24, 2018, 6:41am

It is well below:

aj@Plex:~$ find /media -type d -print | wc -l
4118

ChuckPa · December 24, 2018, 6:59am

Were there any “Permission Denied” errors in your logs?
Are any of the directories over a network mount? (Notification doesn’t exist across networks. Windows emulates it by polling (rereading) directories frequently.)

Johnsen1408 · December 24, 2018, 7:18am

No error in the logs.

Yes, the directory are a SMB mount (Synology NAS). So I don’t expect the auto scan on changes to work. However the partial scan should.

ChuckPa · December 24, 2018, 7:27am

No. it won’t. ‘Automatic’ is based on the Linux iNotify kernel service.
INotify only monitors locally mounted physical filesystems.
As I stated above, Windows emulates this by polling (reading all the directories) frequently.

Johnsen1408 · December 24, 2018, 7:48am

So what you are saying is that the "Run a partial scan when changes are detected" setting is a sub-setting to the "Scan my library automatically" above it? If this is the case, I have misunderstood it. I was expecting this to run a partial scan during normal scans as well.

ChuckPa · December 24, 2018, 7:53am

Correct.

PMS can either scan the entire library if anything changes or only those diretor(ies) which did change.

Example: /movies

/movies/A (year) - exists
/movies/B (year) - is added

Partial scan scans only the conents of /movies/B (year) and below it.
Since none of the other date/time stamps have changed, there is no need to start running deeper into the directories.

Without Partial Scan, all of /movies would be rescanned.

Johnsen1408 · December 24, 2018, 8:15am

Okay. Too bad it cannot do the same without iNotify/Auto. You still have the folder modification times on a SMB share. This would have saved a lot of time scanning.

Nevertheless, plex is still extremely slow to do a full scan (15 minutes).
vs. the returned stats from the file system (17 seconds):

aj@Plex:~$ echo $(date +"%T") ; stat /media/Movies/* > /dev/null ; stat /media/Movies/*/* > /dev/null  ; echo $(date +"%T")
08:54:52
08:55:09

RazorBlades · December 24, 2018, 4:02pm

The database has to be created by running a full scan, then the intervals will work afterwards. Change your setting of Library scan interval to " Every 2 hours" . Then change your “Weeks to consider for On Deck and Continue Watching” from 52 to 2.

Johnsen1408 · December 24, 2018, 9:24pm

I am sorry but this answer does not make any sense to me. The database is obviously previously full scanned, and the issue has nothing to do with the intervals you mentioned. My issue is that plex is not partial scanning, and the full scan is taking considerably longer 50-60 times longer than the file system (I would expect some overhead checking against the database and etc. but not this much).

Johnsen1408 · December 24, 2018, 9:28pm

For anyone interested I made my own scanning scripts which for me takes 1 second instead of Plexs 15 minutes+ (4000 movies/100 TV Shows) if no files are changed. Feel free to tweak and use yourself. Strange that Plex cannot do this natively, checking modified dates on folders are only one line in the script, and the rest is just gravy.

#!/bin/bash

# User variables
RUNEVERYMIN=5 #Restart every x minutes
MOVIESECTION=2
MOVIEFOLDER="/media/Movies"
MOVIEDEPTH=1 #Folderdepth to scan "Movie (1900)/Movie.mp4"=1 "Action/Movie (1900)/Movie.mp4"=2 etc.
TVSECTION=1
TVFOLDER="/media/TV Shows"
TVDEPTH=1 #Folderdepth to scan "Serie (1900)/s01e01.mp4"=1 "Serie (1900)/Season 1/s01e01.mp4"=2 etc.
REFRESHMETA="YES" #Refresh metadata at the same time as scanning? YES/NO
ANALYZE="DEEP" #Analyze videofile? NO/NORMAL/DEEP

# Plex variables
export PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR="/var/lib/plexmediaserver/Library/Application Support"
export PLEX_MEDIA_SERVER_HOME="/usr/lib/plexmediaserver"
export PLEX_MEDIA_SERVER_MAX_PLUGIN_PROCS="6"
export PLEX_MEDIA_SERVER_TMPDIR="/tmp"
export LD_LIBRARY_PATH="/usr/lib/plexmediaserver"
DATABASE="/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db"
IFS=$'\n'
PLEXSCANNER="/usr/lib/plexmediaserver/Plex Media Scanner"

# First scan?
if [ ! -f /tmp/lastscan ]; then
	touch -t 190001010000  /tmp/lastscan
fi

# Loop forever
while true
do
	
	# Start
	echo -e "\033[0;32m$(date '+%Y-%m-%d %H:%M:%S') Started"
	
	# Save time
	touch /tmp/thisscan
	
	# MOVIES
	find $MOVIEFOLDER -mindepth $MOVIEDEPTH -maxdepth $MOVIEDEPTH -type d -newer /tmp/lastscan -print0 | 
	while IFS= read -r -d '' FOLDER; do
		
		# Scan
		if [[ $REFRESHMETA == "YES" ]]; then
			echo -e "\033[1;33m$(date '+%Y-%m-%d %H:%M:%S') '$FOLDER': Scanning & Refreshing"
			echo -ne "\033[0;37m"
			$PLEXSCANNER --scan --refresh --section $MOVIESECTION --directory "$FOLDER"
		else
			echo -e "\033[1;33m$(date '+%Y-%m-%d %H:%M:%S') '$FOLDER': Scanning"
			echo -ne "\033[0;37m"
			$PLEXSCANNER --scan --section $MOVIESECTION --directory "$FOLDER"
		fi
		sleep 1
		
		# Deep Analyze
		if [[ $ANALYZE == "DEEP" ]]; then
			SQL="SELECT COUNT(mi.metadata_item_id) FROM metadata_items md join media_items mi on mi.metadata_item_id = md.id join media_parts mp on mp.media_item_id = mi.id WHERE mp.extra_data not like '%deepAnalysisVersion=2%' and md.metadata_type in (1, 4, 12) and mp.file LIKE '${FOLDER//\'/\'\'}%'"
			QUERYTOTAL=(`sqlite3 "$DATABASE" "$SQL"`)
			if [[ $QUERYTOTAL > 0 ]]; then
				echo -e "\033[0;35m$(date '+%Y-%m-%d %H:%M:%S') '$FOLDER': Items needing deep analyzing: $QUERYTOTAL"
				QUERYCOUNTER=1
				SQL="SELECT mi.metadata_item_id FROM metadata_items md join media_items mi on mi.metadata_item_id = md.id join media_parts mp on mp.media_item_id = mi.id WHERE mp.extra_data not like '%deepAnalysisVersion=2%' and md.metadata_type in (1, 4, 12) and mp.file LIKE '${FOLDER//\'/\'\'}%'"
				QUERYITEMS=(`sqlite3 "$DATABASE" "$SQL"`)
				for ITEM in "${QUERYITEMS[@]}"; do
					echo -e "\033[0;35m$(date '+%Y-%m-%d %H:%M:%S') '$FOLDER': Deep analyzing $QUERYCOUNTER/$QUERYTOTAL: $ITEM"
					echo -ne "\033[0;37m"
					$PLEXSCANNER --analyze-deeply --item $ITEM
					((QUERYCOUNTER++))
					sleep 1
				done
			fi
			
		# Analyze
		elif [[ $ANALYZE == "NORMAL" ]]; then
			SQL="SELECT COUNT(mi.metadata_item_id) FROM media_parts mp LEFT JOIN media_items mi on mp.media_item_id=mi.id WHERE mi.bitrate is null and mp.file LIKE '${FOLDER//\'/\'\'}%'"
			QUERYTOTAL=(`sqlite3 "$DATABASE" "$SQL"`)
			if [[ $QUERYTOTAL > 0 ]]; then
				echo -e "\033[0;35m$(date '+%Y-%m-%d %H:%M:%S') '$FOLDER': Items needing analyzing: $QUERYTOTAL"
				QUERYCOUNTER=1
				SQL="SELECT mi.metadata_item_id FROM media_parts mp LEFT JOIN media_items mi on mp.media_item_id=mi.id WHERE mi.bitrate is null and mp.file LIKE '${FOLDER//\'/\'\'}%'"
				QUERYITEMS=(`sqlite3 "$DATABASE" "$SQL"`)
				for ITEM in "${QUERYITEMS[@]}"; do
					echo -e "\033[0;35m$(date '+%Y-%m-%d %H:%M:%S') '$FOLDER': Analyzing $QUERYCOUNTER/$QUERYTOTAL: $ITEM"
					echo -ne "\033[0;37m"
					$PLEXSCANNER --analyze --item $ITEM
					((QUERYCOUNTER++))
					sleep 1
				done
			fi
		fi
	done
	
	# Find newer TV
	find $TVFOLDER -mindepth $TVDEPTH -maxdepth $TVDEPTH -type d -newer /tmp/lastscan -print0 | 
	while IFS= read -r -d '' FOLDER; do
		
		# Scan
		if [[ $REFRESHMETA == "YES" ]]; then
			echo -e "\033[1;33m$(date '+%Y-%m-%d %H:%M:%S') '$FOLDER': Scanning & Refreshing"
			echo -ne "\033[0;37m"
			$PLEXSCANNER --scan --refresh --section $TVSECTION --directory "'$FOLDER'"
		else
			echo -e "\033[1;33m$(date '+%Y-%m-%d %H:%M:%S') '$FOLDER': Scanning"
			echo -ne "\033[0;37m"
			$PLEXSCANNER --scan --section $TVSECTION --directory "$FOLDER"
		fi
		sleep 1
		
		# Deep Analyze
		if [[ $ANALYZE == "DEEP" ]]; then
			SQL="SELECT COUNT(mi.metadata_item_id) FROM metadata_items md join media_items mi on mi.metadata_item_id = md.id join media_parts mp on mp.media_item_id = mi.id WHERE mp.extra_data not like '%deepAnalysisVersion=2%' and md.metadata_type in (1, 4, 12) and mp.file LIKE '${FOLDER//\'/\'\'}%'"
			QUERYTOTAL=(`sqlite3 "$DATABASE" "$SQL"`)
			if [[ $QUERYTOTAL > 0 ]]; then
				echo -e "\033[0;35m$(date '+%Y-%m-%d %H:%M:%S') '$FOLDER': Items needing deep analyzing: $QUERYTOTAL"
				QUERYCOUNTER=1
				SQL="SELECT mi.metadata_item_id FROM metadata_items md join media_items mi on mi.metadata_item_id = md.id join media_parts mp on mp.media_item_id = mi.id WHERE mp.extra_data not like '%deepAnalysisVersion=2%' and md.metadata_type in (1, 4, 12) and mp.file LIKE '${FOLDER//\'/\'\'}%'"
				QUERYITEMS=(`sqlite3 "$DATABASE" "$SQL"`)
				for ITEM in "${QUERYITEMS[@]}"; do
					echo -e "\033[0;35m$(date '+%Y-%m-%d %H:%M:%S') '$FOLDER': Deep analyzing $QUERYCOUNTER/$QUERYTOTAL: $ITEM"
					echo -ne "\033[0;37m"
					$PLEXSCANNER --analyze-deeply --item $ITEM
					((QUERYCOUNTER++))
					sleep 1
				done
			fi
			
		# Analyze
		elif [[ $ANALYZE == "NORMAL" ]]; then
			SQL="SELECT COUNT(mi.metadata_item_id) FROM media_parts mp LEFT JOIN media_items mi on mp.media_item_id=mi.id WHERE mi.bitrate is null and mp.file LIKE '${FOLDER//\'/\'\'}%'"
			QUERYTOTAL=(`sqlite3 "$DATABASE" "$SQL"`)
			if [[ $QUERYTOTAL > 0 ]]; then
				echo -e "\033[0;35m$(date '+%Y-%m-%d %H:%M:%S') '$FOLDER': Items needing analyzing: $QUERYTOTAL"
				QUERYCOUNTER=1
				SQL="SELECT mi.metadata_item_id FROM media_parts mp LEFT JOIN media_items mi on mp.media_item_id=mi.id WHERE mi.bitrate is null and mp.file LIKE '${FOLDER//\'/\'\'}%'"
				QUERYITEMS=(`sqlite3 "$DATABASE" "$SQL"`)
				for ITEM in "${QUERYITEMS[@]}"; do
					echo -e "\033[0;35m$(date '+%Y-%m-%d %H:%M:%S') '$FOLDER': Analyzing $QUERYCOUNTER/$QUERYTOTAL: $ITEM"
					echo -ne "\033[0;37m"
					$PLEXSCANNER --analyze --item $ITEM
					((QUERYCOUNTER++))
					sleep 1
				done
			fi
		fi
	done
	
	# Set last scan time
	mv -f /tmp/thisscan /tmp/lastscan
	
	# Finished
	echo -e "\033[0;31m$(date '+%Y-%m-%d %H:%M:%S') Finished"
	
	# Pause
	secs=$(($RUNEVERYMIN * 60))
	while [ $secs -gt 0 ]; do
		echo -ne "\033[1;30mSleeping $secs seconds\033[0K\r"
		sleep 1
		: $((secs--))
	done
done

RazorBlades · December 25, 2018, 2:07pm

I have to count 8915 files my library has to scan through when I add new or delete, from normal day to day recording. I don’t have any problems with my system with the above settings you are having. And I convert all my TS files into MP4. So my system is constantly updating with no issues.

Sorry just thought I’d offer my settings that seem to work. I’ve used PLEX since the year it;s been released.

system · March 25, 2019, 2:13pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Partial scan not work Metadata & Adding Files server-linux	23	279	August 4, 2020
Scan issue on Plex Media Server Ubuntu Desktops & Laptops server-linux	30	1647	June 22, 2019
PMS scanner question Dev/API Corner scanner-agent-dev	10	127	December 20, 2019
command line for plex media scanner Dev/API Corner other-dev	32	1142	December 20, 2019
Plex partial scanning is extremely slow to start on large libraries Desktops & Laptops server-windows	37	1022	October 2, 2019

Partial Scan not working

Related topics