Dear Plex team
How can I keep my PMS from rescanning pretty much every single Plex item once in a while (~daily?) while I request its metadata through GET /library/metadata/<itemid>?includeRelated=0&includeExtras=1&includeReviews=1&checkFiles=0? Since this scan takes a couple of seconds for each item, the initial sync process for PlexKodiConnect slows down to a crawl. I imagine that e.g. Plex Web or other Plex clients would suffer, too, just not as bad because they’d only be requesting a couple of dozens XMLs at most…
Once all XMLs have been requested (and the PMS scanned these items), the PMS won’t rescan the requested items for a while, instead serving up the XML instantly.
PlexKodiConnect uses a GET /library/sections/<viewid>/all request (split in ~200 items with X-Plex-Container-Size) to determine which Plex items changed since the last sync. To determine the last time an update to a particular Plex item happened, the updatedAt attribute of the item is used. For items that have a newer updatedAt then the one synced to PKC, PKC requests new metadata as described above.
It seems like updatedAt is periodically reset for an entire library (or maybe a particular media folder on the harddisk), even though the item itself did not change at all.
So…
- Is there a way to let that
updatedAt be, EXCEPT the item really did change?
- Alternatively, is there a way to let the PMS scan BEFORE I request an item’s metadata?
Happy to share debug logs via PM :-).
Cheers (and thanks for an awesome product!)
sync
Requesting metadata for each item individually is going to be much slower than getting items in bulk; needless to say.
I’d have to go read the code to see when updatedAt is changed, but I don’t believe it was intended to be used as a general purpose sync mechanism.
So what exactly are you trying to do here?
@mfeingol said:
So what exactly are you trying to do here?
PKC basically duplicates the Plex metadata and writes it to the Kodi native database. In theory, it should suffice to do that only once right after PKC-installation and then only update the metadata if it has changed.
In practice, this doesn’t work well because
- of the
updatedAt flag periodically being “reset” by the PMS for every single item of a Plex library. This would not be too bad except:
- the PMS periodically analyses an item first before serving up its metadata. It will do this for every item in a specific library for some reason. This takes 2-3 seconds, even on a fast PMS. I’ve noticed this lag on Plex for Android as well if one access e.g. a movie’s pre-play screen.
Requesting metadata for each item individually is going to be much slower than getting items in bulk; needless to say.
I’d have to go read the code to see when updatedAt is changed, but I don’t believe it was intended to be used as a general purpose sync mechanism.
GET /library/sections/<viewid>/all does not provide all the metadata that is needed, so PKC uses GET /library/metadata/<itemid>?includeRelated=0&includeExtras=1&includeReviews=1&checkFiles=0 for a new/changed item. It actually works very well except for the above :-).
I’m slightly surprised updatedAt gets bulk-updated, but maybe not entirely surprised.
As for the delays on /library/metadata/X requests, those tend to occur only when we notice something relating to media analysis is obsolete. Instead of doing a giant bulk scan, we fault things in on-demand. And there’s also a scheduled task to refresh these things periodically. So it shouldn’t happen in the general case, but your scenario is probably going to trigger it more frequently than you’d like.
Anyway, what metadata are you missing from the /all response? Do you need streams and such? Or is it just includeExtras and includeReviews? And have you tried using those parameters on the /all request?
@mfeingol said:
Anyway, what metadata are you missing from the /all response? Do you need streams and such? Or is it just includeExtras and includeReviews? And have you tried using those parameters on the /all request?
Thing is, PKC needs basically all the Plex metadata, including stream data and extras. includeExtras does not work on /all requests, nor does includeReviews. I’m stuck with GET /library/metadata/X - which is fine really except as described above…
I’m slightly surprised updatedAt gets bulk-updated, but maybe not entirely surprised.
I am, too - anything you could do to fix this? 
As for the delays on /library/metadata/X requests, those tend to occur only when we notice something relating to media analysis is obsolete. Instead of doing a giant bulk scan, we fault things in on-demand. And there’s also a scheduled task to refresh these things periodically. So it shouldn’t happen in the general case, but your scenario is probably going to trigger it more frequently than you’d like.
Seems weird that something being obsolete leads to a 2-3 seconds lag… Playing around with my Plex for Android, it seems like /library/metadata/X is called as soon as I access a preplay screen for the first time after an app reboot. So I’d venture that “for the first time” happens rather often :-D.
Since the updatedAt is periodically reset, the browsing experience will lag on every single item until it has been accessed once.
Again, anything you could do to fix this?
anything you could do to fix this?
If you can find me a repro scenario, I can attempt to reproduce locally and see what’s going on.
Seems weird that something being obsolete leads to a 2-3 seconds lag…
You might want to take a look at the logs and see where the 2-3 seconds are being spent. But odds are we’re spawning a scanner process to perform media analysis to update the item to the most recent version.
I’m trying to find a pattern to make this reproducable.
In the meantime, I can send you a PMS debug log with tons of scans being triggered, each and every one individually - if it helps?
@mfeingol, went ahead and sent you the PMS debug log.
It starts with PKC’s GET /library/sections/1/allLeaves to establish the currently available plex items for library section 1 (movies). Since every single item’s updatedAt is older than Plex’ directory change time,
DEBUG - Last refresh (2016-11-04 22:33:14) appears to be older than directory change time (2016-11-05 16:10:36), need to refresh local media agent.
every single item gets scanned by the PMS. Hope this helps
OK, so to reproduce the “daily scanning”, set the library settings like below. Use Win7, and a harddisk that sleeps after ~5min for your media. Have Plex point to the smb path (\\myserver\movies) even if the media is local.
Install Kodi with PlexKodiConnect on some client :-D. Let it do its initial sync; shut it down.
After >24-48 hours, fire up Kodi and let it do the startup sync. It should be painfully slow (#downloaded almost always = #processed) because of the PMS scanning every single item.
I get ~3 seconds delay on all Plex clients for every PMS item. Culprit seems to be
Nov 17, 2016 10:26:48.413 [6696] DEBUG - Last refresh (2016-11-16 13:14:47) appears to be older than directory change time (2016-11-17 09:47:28), need to refresh local media agent.
Here the PMS log for Plex for Android requesting metadata for item 428114: https://gist.github.com/croneter/48f2edbd0492241fb36a839dd7dbc2cd
EDIT: the need to rescan only affects movie libraries.TV shows are unaffected for some reason
EDIT2: Nope, episodes also get needlessly force-scanned. Logs: https://gist.github.com/croneter/0d65c7ff822193223c3f44eacd00ab60
I know it’s old but wondering if this is fixed or still not fixed?
