Scanning Agent (again not working) status page

General Discussions
21

@ChuckPA said:
As for the ‘extras’, I have a secret script for that haha :slight_smile: (When ready for prime time, I’ll share)

scanning and emptying the trash isn’t enough to clear it out completely. You have to ‘Clean Bundles’. Only when the bundle files are gone does PMS remove the record from the database. Until that point, it’s progressively removing it from the different tables.

  1. Update — removes it from the section table and moves to Trash
  2. Empty Trash - does just that…
  3. Clean Bundles - removes the unused “Media Item” entries which still exist. These include the posters, etc and anything else PMS knows including the previous file’s fingerprint from the DB. ← finger prints are important… It’s part of duplicate detection.

cmd line perhaps?

22

It’s an internal curl call … The default is 10 seconds. Most of the calls go direct to the public databases and is subject to their response time.
If you’re seeing “Error 28” in the agent logs (PMS Plug-in Logs) and/or PMS itself, those are from those servers.

PMS does movie matching based on Name / Title (if given in MP4) else it does purely by Filename and Year released (IMDB published)

If you’re adding/moving a lot of data or your DB is just that big (implied ‘slow’), a “SLOW QUERY” will cause the agents themselves to start timing out because they communicate with PMS using curl on the local socket.

What’s your upload/download speed? I presume you’re not maxing it out in any way when this is happening?

I think I need to setup a unionfs setup myself. My logic is: PMS ties itself to the iNotify mechanism. iNotify doesn’t work over networks (NFS) and it definitely won’t work with FUSE mounts because they aren’t in kernel space (they’re user-space daemon API calls)

If you agree with my thinking, where / how is the best way to set up a unionfs test? I’ll punch up a bunch of music files to google and see what happens .

If we’re getting hit with inherent latency of unionfs and can demonstrate it what it takes to fix it, if it makes sense, I’ll go ask engineering what they can come up with.

Thoughts?

23

@GoingGaGa said:
also, if it fingerprints a file in a certain location, why exactly can it not just retrieve that information when the folder/file is included in a second/third library?

I had to ask that question too… It keeps every library’s entries isolated (Library ID / Metadata item pair). Dupes come up when it sees the same pair. It won’t search outside the current Library section. If it did and there exists a Movie and a TV series sharing the same name? Chaos. (IMDB will return based on name without differentiation… easy example)

24

@GoingGaGa said:

@ChuckPA said:
As for the ‘extras’, I have a secret script for that haha :slight_smile: (When ready for prime time, I’ll share)

scanning and emptying the trash isn’t enough to clear it out completely. You have to ‘Clean Bundles’. Only when the bundle files are gone does PMS remove the record from the database. Until that point, it’s progressively removing it from the different tables.

  1. Update — removes it from the section table and moves to Trash
  2. Empty Trash - does just that…
  3. Clean Bundles - removes the unused “Media Item” entries which still exist. These include the posters, etc and anything else PMS knows including the previous file’s fingerprint from the DB. ← finger prints are important… It’s part of duplicate detection.

cmd line perhaps?

I need to ask. The dev team is working on publishing the endpoints. I know some but not all. those key 3 should be the easiest but surprisingly aren’t on my list yet.

25

@ChuckPA said:
It’s an internal curl call … The default is 10 seconds. Most of the calls go direct to the public databases and is subject to their response time.
If you’re seeing “Error 28” in the agent logs (PMS Plug-in Logs) and/or PMS itself, those are from those servers.

PMS does movie matching based on Name / Title (if given in MP4) else it does purely by Filename and Year released (IMDB published)

If you’re adding/moving a lot of data or your DB is just that big (implied ‘slow’), a “SLOW QUERY” will cause the agents themselves to start timing out because they communicate with PMS using curl on the local socket.

What’s your upload/download speed? I presume you’re not maxing it out in any way when this is happening?

I think I need to setup a unionfs setup myself. My logic is: PMS ties itself to the iNotify mechanism. iNotify doesn’t work over networks (NFS) and it definitely won’t work with FUSE mounts because they aren’t in kernel space (they’re user-space daemon API calls)

If you agree with my thinking, where / how is the best way to set up a unionfs test? I’ll punch up a bunch of music files to google and see what happens .

If we’re getting hit with inherent latency of unionfs and can demonstrate it what it takes to fix it, if it makes sense, I’ll go ask engineering what they can come up with.

Thoughts?

_Can you perhaps give me the curl so I can test? Is there a way to configure that timeout?

I’ve added a few new ones plus renamed ones but left the files local, matching happened without any glitches whatsoever, so Plex is timing out reading my files, but reading even 100 new files when it already has scanned the others should be milliseconds

Server connected to GBit in a datacenter, it rarely even passes 2mb up or 4mb down while scanning…
UnionFS settings:::

unionfs-fuse -o max_files=16384 -o cow,allow_other,auto_cache,relaxed_permissions,nonempty,statfs_omit_ro -o fsname=AMAZONDATA -o dirs=/home/simba-chippie/amazon/local=RW:/home/simba-chippie/amazon/cloud=RO /mnt/Data

Google I’ve never tested, don’t have account and not planning on getting any, way too many issues with G-Man throwing hissy fits and blocking people for 24hours at a time…

Devs need to confirm what they changed between 1.5 and 1.6 versions as most of these issues seem to stem from the 1.6 variety…
_

@ChuckPA said:

@GoingGaGa said:

@ChuckPA said:
As for the ‘extras’, I have a secret script for that haha :slight_smile: (When ready for prime time, I’ll share)

scanning and emptying the trash isn’t enough to clear it out completely. You have to ‘Clean Bundles’. Only when the bundle files are gone does PMS remove the record from the database. Until that point, it’s progressively removing it from the different tables.

  1. Update — removes it from the section table and moves to Trash
  2. Empty Trash - does just that…
  3. Clean Bundles - removes the unused “Media Item” entries which still exist. These include the posters, etc and anything else PMS knows including the previous file’s fingerprint from the DB. ← finger prints are important… It’s part of duplicate detection.

cmd line perhaps?

I need to ask. The dev team is working on publishing the endpoints. I know some but not all. those key 3 should be the easiest but surprisingly aren’t on my list yet.

_Since going 1.6 I never get notified anymore about optimizations/bundle cleaning having been completed, we know now that offline optimization seems to work ok…

I can confirm that after doing offline optimization, starting plex and running optmization from gui, i don’t get feedback about optimization being completed, maybe something in there?_

@ChuckPA said:

@GoingGaGa said:
also, if it fingerprints a file in a certain location, why exactly can it not just retrieve that information when the folder/file is included in a second/third library?

I had to ask that question too… It keeps every library’s entries isolated (Library ID / Metadata item pair). Dupes come up when it sees the same pair. It won’t search outside the current Library section. If it did and there exists a Movie and a TV series sharing the same name? Chaos. (IMDB will return based on name without differentiation… easy example)

I agree on the Movie/TV same name dilemma, especially now that most popular movies are being produced as TV shows to drag it out… I know for myself it won’t affect me in the least, but that is only because I make the effort and spend the time to keep my libraries and folders organized properly… I know Plex does a lot of the metadata and figuring out BS to get a nice view on it, but I sleep better at night knowing I’ve made it so that if I decided to copy/paste my library to someone else, they would know exactly what was for what…

26

@ChuckPA no feedback yet?

27

Since last night? No. I’ve also had to deal with a bunch of other issues too . I’m not forgetting… just BUSY

I personally have setup FUSE (google-ocaml-fuse)… and it’s working. flawlessly but slow on 24 Mbps. I’m also challenging it to load a 900 series 'test library" and “5000” movies library

28

ok, good luck with Google, let me know how it turns out…

Here’s another thing pointing to Plex Agents going ape… some of my posters have come back either German or Russian or something other than English which my libraries are… Hide and Seek (2005) // Nanny McPhee (2005) // Pride & Prejudice (2005) // Stealth (2005) // Jarhead 2 - Field of Fire (2014) // The Librarian - Quest for the Spear (2004) // DodgeBall (2004) // etc etc etc…

29

Your getting posters in different languages sounds as if your not locking posters to your language in the Library properties.

Check your setting.

If your content is mixed language and you don’t lock it down, it will pull from the language tagging in the file.

My test library is still building (it’s big).

I also, just prior to answering your post here, started a simple directory listing. When it completes and gives file count, you’ll see my elapsed time and how difficult it is for me.

30

For laughs, Notice the size and time it took to pull at 24 Mbps.

It will take me time. I’m not seeing any bad posters either in any of the 900+ (I forgot how many) series I have defined.

[chuck@lizum google-drive.50]$ date ; ls -1R | wc -l ; date
Thu May 11 16:19:54 EDT 2017
188558
Thu May 11 17:08:20 EDT 2017
[chuck@lizum google-drive.51]$
31

@ChuckPa said:
Your getting posters in different languages sounds as if your not locking posters to your language in the Library properties.

Check your setting.

If your content is mixed language and you don’t lock it down, it will pull from the language tagging in the file.

My test library is still building (it’s big).

I also, just prior to answering your post here, started a simple directory listing. When it completes and gives file count, you’ll see my elapsed time and how difficult it is for me.

At times like these I tend to ask myself whats the difference between now and the previous 4000 that worked without issue, and if the answer is sweet blue buggerall, then I tend to point fingers…

32

TV Shows don’t have issues with posters at all, haven’t recently added new shows so it might just be that I haven’t seen a problem yet… (Futurama might be going in today - will let you know if there are issues)

All Movie libraries are locked to language, US…

33

The last thing I have up my sleeve is clearing the caches which doesn’t make any sense but what the heck. It’s non-destructive.

https://support.plex.tv/hc/en-us/articles/202967376-Clearing-Plugin-Channel-Agent-HTTP-Caches