[REL] YouTube Metadata Agent

Hey @ZeroQI

Trying to get your scanner up and running and not getting anything showing in the library files. Any chance you could give me a hand? Be happy to donate :slight_smile: Great scanner, if I can get it to work!

It seems to be detecting the file, but my library stays empty.

Below is your scanners filelist.log

`
Library: ’ Lectures’, root: ‘/video/Shows/Lectures’, path: ‘Jordan B Peterson’, files: ‘0’, dirs: ‘1’, Plex scan date: 2018-05-17 09:25:10

[no files detected]
Library: ’ Lectures’, root: ‘/video/Shows/Lectures’, path: ‘Jordan B Peterson/Series [PL22J3VaeABQD_IZs7y60I3lUrrFTzkpat]’, files: ‘0’, dirs: ‘1’, Plex scan date: 2018-05-17 09:25:11

[no files detected]
Library: ’ Lectures’, root: ‘/video/Shows/Lectures’, path: ‘Jordan B Peterson/Series [PL22J3VaeABQD_IZs7y60I3lUrrFTzkpat]/The Psychological Significance of the Biblical Stories’, files: ‘1’, dirs: ‘0’, Plex scan date: 2018-05-17 09:25$

Jordan B Peterson/Series [PL22J3VaeABQD_IZs7y60I3lUrrFTzkpat]/The Psychological Significance of the Biblical Stories/E01 - Biblical Series I - Introduction to the Idea of God [f-wWBGo6a2w].mp4
`

@s1mpleman It’s simple man! :slight_smile:

The files untouched after downloading with youtube-dl are recognised by the scanner, and the playlist id need to be on hte parent folder, whether it is a season folder or better, the series folder. If you add the episode number it should pick it up however.

  • i need to check special cases like semicolon presence

Since scanner do not rescan folders that didn’t change contents, you may have to:

  • move your folders our of the root folder
  • rescan
  • rename folders
  • move them into your root folder
  • rescan

Here are the possible choices you have for naming from the most logical (playlist as series, gets the most metadata) to the most tortuous (playlist as season):

  • [Jordan B Peterson] The Psychological Significance of the Biblical Stories [PL22J3VaeABQD_IZs7y60I3lUrrFTzkpat]/Biblical Series I Introduction to the Idea of God [f-wWBGo6a2w].mp4
  • Jordan B Peterson/The Psychological Significance of the Biblical Stories [PL22J3VaeABQD_IZs7y60I3lUrrFTzkpat]/Biblical Series I: Introduction to the Idea of God [f-wWBGo6a2w].mp4
  • Jordan B Peterson/Season 1 - The Psychological Significance of the Biblical Stories [PL22J3VaeABQD_IZs7y60I3lUrrFTzkpat]/Biblical Series I: Introduction to the Idea of God [f-wWBGo6a2w].mp4
  • The Psychological Significance of the Biblical Stories/Season 1 - Jordan B Peterson [PL22J3VaeABQD_IZs7y60I3lUrrFTzkpat]/Biblical Series I: Introduction to the Idea of God [f-wWBGo6a2w].mp4
  • Jordan B Peterson/Series/The Psychological Significance of the Biblical Stories [PL22J3VaeABQD_IZs7y60I3lUrrFTzkpat]/E01 - Biblical Series I - Introduction to the Idea of God [f-wWBGo6a2w].mp4

I recommend playlist as series and grouping series in a Collection (user/channel or subject/game)
any issue, give me the relative path and filename and playlist id and i can reproduce it. will do tonight, no longer have access at my NAS from work…

@ZeroQI Totally was! Dunno what I was doing wrong! hahaha.

Went for the first format, but it’s working great now.

Had to include the episode number as it kept bringing up episode 501, dunno if that was suppose to happen only tested it on 1 ep. Doesn’t matter, my youtube script set’s it anyway :slight_smile:

Thanks for the great scanner, I’ll fly a donation your way!

@s1mpleman Thanks a lot for the donation.

501 is the default episode starting number when it search manually for an episode number in the title after failing all Regex, so it doesn’t conflicts with other episode numbers but YouTube folders use playlist listing to match so my code shouldn’t have been searching an episode number in the filename, now corrected, was due to ‘YouTube’ instead of ‘youtube’ in code…

Uploaded a new scanner version: https://github.com/ZeroQI/Absolute-Series-Scanner/blob/master/Scanners/Series/Absolute%20Series%20Scanner.py

  • As long as you put the playlist tag [PL…] in the folder containing the videos directly (series better but season folder also work) and the filename has the YouTube video id inside it will give the right episode number even if the title was changed so will work in most cases unlike before.

Uploaded a new YouTube agent: https://github.com/ZeroQI/YouTube-Agent.bundle/blob/master/Contents/Code/init.py

  • now videos without ‘standard’ thumbnail resolution will not crash agent, so this playlist will gather metadata like the summaries, rating, etc…

Please update both and refresh the series, it will look better, and please provide feedback

[Jordan B Peterson] The Psychological Significance of the Biblical Stories [PL22J3VaeABQD_IZs7y60I3lUrrFTzkpat]/Biblical Series I Introduction to the Idea of God [f-wWBGo6a2w].mp4

That’s the naming convention I used if it helps.

I’ve got Youtube-Dl setup on a script with a specific output format anyway, so it’s no trouble with the renaming, does it automatically when it pulls a playlist down.

My lectures library looks great anyway thank you!

Is it possible to auto pull the collection by anychance? That’s a thing that would make this extra awesome!

@s1mpleman Can you expand on auto pull the collection ?
There is a collection field i can fill, but what string do you want put in it ?
Uploaded new version of the agent using channel banner

@ZeroQI

I was thinking the first brackets, so the bold part here.

Not sure if that’s possible, or easier with a folder instead.

[Jordan B Peterson] The Psychological Significance of the Biblical Stories [PL22J3VaeABQD_IZs7y60I3lUrrFTzkpat]/Biblical Series I Introduction to the Idea of God [f-wWBGo6a2w].mp4

Do not forget the channel name “Jordan B Peterson” will also be as actor in series page with the channel logo and in episodes as a director…

if it’s important enough to make a folder to put the playlist series folder in it, it’s important enough for a collection tag…
I actually made it an agent setting for my HAMA agent so can implement fairly easily.

The filename below would create automatically a Jordan B Peterson collection if the collection agent setting (to be added) is enabled and more than one folder is present

  • Jordan B Peterson/The Psychological Significance of the Biblical Stories [PL22J3VaeABQD_IZs7y60I3lUrrFTzkpat]/Biblical Series I Introduction to the Idea of God [f-wWBGo6a2w].mp4

New agent and scanner version

  • fixes a crash if ‘normal’ url missing
  • if you put 2 or more series in the same grouping folder, creates a collection for you, great for channel|game / playlist [PL…] type organization

Used as test:

  • dr jordan b peterson\Dialogues with Interesting People [PL22J3VaeABQA_4uQfPdSEhiNhQM236_O-]
  • dr jordan b peterson\Psychological Significance of the Biblical Stories [PL22J3VaeABQD_IZs7y60I3lUrrFTzkpat]

Please test and report
@s1mpleman cannot use brackets as only if a collection has 2 members does the collection make sense. make sure to update the meta of all series to see them in the collection

@Craquehead said:
s1mpleman wrote on May 7 2015, 1:54 PM: »

Is anyone still working on this? Mines managed to just pull the video titles. Nothing else.

The problem is that Youtube stopped supporting the v2 API that this agent relies on.

I’ve managed to get the agent working with the v3 API, but it requires an API key. (anyone can get one, but the one I’m using is tied to my personal google account)

Unless the original author re-writes the agent to scrape the page instead of using the API, not sure there’s a way to get around the requirement for an API key tied to a person.

–Update–

I’ve submitted a pull request to the original author with my changes. It requires signing up for an API key and entering it through the agent preferences.

I’ve modifed this old change to be compatible with sander’s subsequent bugfixes, like the “Try/Catch empty summary” and the director metadata commits.

I’m not familiar enough with github to try and create a pull request, so here it is as a .zip.
Maybe later I’ll go some reading and try to get it setup properly.

As is specified in page 1 of this thread, you will need to go here and setup your own API key.

Sander’s original plugin is awesome, but I was having trouble with a lot of age gated videos tripping it up. Using APIv3 lets the plugin retrieve info even if the video itself is gated.

I know ZeroQI’s plugin uses APIv3 as well, but I much prefer this slimmed down OG version of the plugin; I personally have no need for the extra playlist features he’s brought in.

mine supports tv libraries and no need to enter a public API key …
Mine still works for movies and is actively developed

I keep trying to edit my post, but I can’t for some reason o_O Didn’t really want to double post an “edit” post, wtf is with this forum software?

Anyway, the only reasons I have for using this version instead of the original or your version are:

  • uses APIv3 to get around annoying age gates. I really don’t care if it’s age gated or otherwise restricted; if I have the video in plex already, I just need the metadata.
  • the original agent stumbled hard on “modern” descriptions filled with line breaks, links, emojis etc. APIv3 gets around this by default. Original method to pull description data could possibly be updated to parse these properly, but that’s well beyond my skill level.
  • literally just grabs video ID and pulls info. I personally don’t need all the extra stuff you’ve baked into yours, more stuff to go wrong IMO. Not saying there’s anything else wrong with yours, just feels a bit silly to “drive a nail with a nuclear bomb” when all one needs is a hammer, so to speak.

@ZeroQI said:
New agent and scanner version

  • fixes a crash if ‘normal’ url missing
  • if you put 2 or more series in the same grouping folder, creates a collection for you, great for channel|game / playlist [PL…] type organization

Used as test:

  • dr jordan b peterson\Dialogues with Interesting People [PL22J3VaeABQA_4uQfPdSEhiNhQM236_O-]
  • dr jordan b peterson\Psychological Significance of the Biblical Stories [PL22J3VaeABQD_IZs7y60I3lUrrFTzkpat]

Please test and report
@s1mpleman cannot use brackets as only if a collection has 2 members does the collection make sense. make sure to update the meta of all series to see them in the collection

When scanning files I can not get the scanner to pick up anything within a sub-directory, but the main directory scans just fine.

IE:

This will not work:

dr jordan b peterson\Dialogues with Interesting People [youtube-PL22J3VaeABQA_4uQfPdSEhiNhQM236_O-]

No videos are populated. Nothing shows up in plex. However if I remove the primary directory, this will work:

Dialogues with Interesting People [youtube-PL22J3VaeABQA_4uQfPdSEhiNhQM236_O-]

But since they are all in the top directory, it will not create the collections.

I honestly love using this agent…so many things off of Youtube now beautifully displayed in Plex. The only issue I’ve run across so far is it seems only the year is copying over for the date field…and then it gets “filled in” with today’s date and whatever year it found on youtube. If after a few days I refresh metadata, it again updates to the current day with whatever year it found on youtube in the first place. Is there some formatting bug with the date that you guys have seen and can recommend a fix for? Thanks!!!

@djmixman Please confirm you use the ‘Absolute Series Scanner’ as literally the first line in the read-me after the title says “-Please use the Absolute Series Scanner to scan your media and leave the YouTube id in the series/movie title” as my scanner (unlique default series scanner) handle:

  • support grouping folders which create automatically collections in the agent
  • support stock filenames from youtube-dl so no numbering needed for Playlist folders with PL id given

@G-Fresh Love it enough for a donation ? (link in my signature)
I suppose it is for a Series library with the playlist id on series folder name ?
I would need the agent logs to troubleshoot as all my aps have the correct published at date even after a refresh

https://github.com/ZeroQI/YouTube-Agent.bundle/blob/master/Contents/Code/init.py
Python agent code look good seen like that
Line 136 for YouTube movies

        date                             = Datetime.ParseDate(json_obj['snippet']['publishedAt']);  Log('date:  "{}"'.format(date))
        metadata.originally_available_at = date.date()

Line 206 for Playlists

        metadata.originally_available_at = Datetime.ParseDate(json_obj['snippet']['publishedAt']).date();              Log('[ ] publishedAt: '+ json_obj['snippet']['publishedAt'])

@ZeroQI said:
@djmixman Please confirm you use the ‘Absolute Series Scanner’ as literally the first line in the read-me after the title says “-Please use the Absolute Series Scanner to scan your media and leave the YouTube id in the series/movie title” as my scanner (unlique default series scanner) handle:

  • support grouping folders which create automatically collections in the agent
  • support stock filenames from youtube-dl so no numbering needed for Playlist folders with PL id given

@G-Fresh Love it enough for a donation ? (link in my signature)
I suppose it is for a Series library with the playlist id on series folder name ?
I would need the agent logs to troubleshoot as all my aps have the correct published at date even after a refresh

https://github.com/ZeroQI/YouTube-Agent.bundle/blob/master/Contents/Code/__init__.py
Python agent code look good seen like that
Line 136 for YouTube movies

        date                             = Datetime.ParseDate(json_obj['snippet']['publishedAt']);  Log('date:  "{}"'.format(date))
        metadata.originally_available_at = date.date()

Line 206 for Playlists

        metadata.originally_available_at = Datetime.ParseDate(json_obj['snippet']['publishedAt']).date();              Log('[ ] publishedAt: '+ json_obj['snippet']['publishedAt'])

@ZeroQI said:
@djmixman Please confirm you use the ‘Absolute Series Scanner’ as literally the first line in the read-me after the title says “-Please use the Absolute Series Scanner to scan your media and leave the YouTube id in the series/movie title” as my scanner (unlique default series scanner) handle:

  • support grouping folders which create automatically collections in the agent
  • support stock filenames from youtube-dl so no numbering needed for Playlist folders with PL id given

@G-Fresh Love it enough for a donation ? (link in my signature)
I suppose it is for a Series library with the playlist id on series folder name ?
I would need the agent logs to troubleshoot

https://github.com/ZeroQI/YouTube-Agent.bundle/blob/master/Contents/Code/__init__.py
Python agent code look good seen like that
Line 136 for YouTube movies

        date                             = Datetime.ParseDate(json_obj['snippet']['publishedAt']);  Log('date:  "{}"'.format(date))
        metadata.originally_available_at = date.date()

Line 206 for Playlists

        metadata.originally_available_at = Datetime.ParseDate(json_obj['snippet']['publishedAt']).date();              Log('[ ] publishedAt: '+ json_obj['snippet']['publishedAt'])

Confirming the scanner:

Only one episode is found (also note the last char of the playlist is stripped. [Missing a “s”] It seems to be doing it on all the playlists.):

Confirming agent:

It seems to be trying to call http://127.0.0.1 however, the server seems to only want to respond on https. I’ve tried changing the two lines I found in your code that points to http://127.0.0.1 to https but when viewing the logs it acts like it doesn’t even change. So this must be getting called from another source.

Update: Changing the lines of code did nothing. I’ve changed the lines back but it still is not detecting any videos. For both changes I removed the pyc file and restarted the server.

I have also deleted the lib and tried to recreate it but now nothing is coming up at all.

Log: >! Jun 03, 2018 15:13:38.708 [0x809406800] INFO - Plex Media Scanner v1.13.0.502 - Pastebin.com

Test Data:

https://www.youtube.com/playlist?list=PLkIGJ7Ljv6YzLKR6_vfN0sNDpJBQ0C8FJ
https://www.youtube.com/playlist?list=PLkIGJ7Ljv6YxjjBZjANiLbG8Kc1AQxzc_
https://www.youtube.com/playlist?list=PLyTnhjOTbyOV8yd5J_on6WjkBMr6_z1nf
https://www.youtube.com/playlist?list=PLo5whi14bKmiVIH82r4sFTTbazvnsC_94
https://www.youtube.com/playlist?list=PLo5whi14bKmit3qYs73oi_ZmmT3bmb1qg

Got the shows to show back up, however we are still having the grouping issue as well as the trimming of the last char of the playlist name.

If there are multiple playlists within a subdirectory they will be picked up by the scanner, but not grouped into a collection.

Reference Folder: https://i.imgur.com/7Ewdytr.png

If there is only one playlist within a sub directory the scanner will see it but not be added to plex:
Only 1 playlist in /CaRtOoNz: https://i.imgur.com/QrMmWbR.png

If there are two or more playlists within a sub directory the scanner will add the shows to plex but not create a collection:
Two or more playlist in /CaRtOoNz: https://i.imgur.com/aAZVp2B.png

No collections: https://i.imgur.com/llBY4pl.png

2nd Issue: (trimmed chars)
In the plex screen shots above, notice every playlist is missing the last char…

“GTA 5 Roleplay OCRP Episode” should be “GTA 5 Roleplay OCRP Episodes”
“GTA 5 Roleplay | OCR” should be “GTA 5 Roleplay | OCRP”
“House Flipper” should be “House Flippers”

Previous unable to connect issue and "fix"
In order to fix the http://127.0.0.1 issue I had to make sure “Secured Connections” was set to “Preferred” instead of required, which really sucks, but whatever…

@djmixman Please avoid putting the previous post in copy when you reply, it make it more difficult to seaparate the actual post from the vrevious, especially when including it twice.
The code i included was to show it does manage the date, i changed nothing in it.

If the files do not show in plex, it is a scanner issue
Scanner read-me: https://github.com/ZeroQI/Absolute-Series-Scanner/blob/master/README.md

Include the following logs (in any case, specify if file not present):
[...]/Plex Media Server/Logs/Plex Media Scanner.log (scanner crash info, no new files added, etc...)
[...]/Plex Media Server/Plug-in Support/Data/com.plexapp.agents.hama/DataItems/Library_name/Logs/root_folder_name.filelist.log
[...]/Plex Media Server/Plug-in Support/Data/com.plexapp.agents.hama/DataItems/Library_name/Logs/root_folder_name.scanner.log
[...]/Plex Media Server/Plug-in Support/Data/com.plexapp.agents.hama/DataItems/Logs/root_folder_name.filelist.log
[...]/Plex Media Server/Plug-in Support/Data/com.plexapp.agents.hama/DataItems/Logs/root_folder_name.scanner.log

whether or not you have HAMA installed the logs should be located here
My root scanner logs show for latest scanner version [github.com/ZeroQI/Absolute-Series-Scanner/blob/master/Scanners/Series/Absolute%20Series%20Scanner.py]:

Library: 'no valid X-Plex-Token.id', root: 'C:\Users\benja\Videos\_Youtube', path: '', files: '0', dirs: '1', Plex scan date: 2018-06-03 17:08:31
=============================================================================================================================================================
[no files detected]

root level manual call to Grouping folders
-------------------------------------------------------------------------------------------------------------------------------------------------------------
[dr jordan b peterson] Grouping folder (contain 2 dirs)
- dr jordan b peterson\Dialogues with Interesting People [PL22J3VaeABQA_4uQfPdSEhiNhQM236_O-], subdir_files:   1, reverse_path: ['Dialogues with Interesting People [PL22J3VaeABQA_4uQfPdSEhiNhQM236_O-]', 'dr jordan b peterson']
- dr jordan b peterson\Psychological Significance of the Biblical Stories [PL22J3VaeABQD_IZs7y60I3lUrrFTzkpat], subdir_files:  17, reverse_path: ['Psychological Significance of the Biblical Stories [PL22J3VaeABQD_IZs7y60I3lUrrFTzkpat]', 'dr jordan b peterson']

Please get me the filelist and scannerlist for all impacted series so i can reproduce your library, correct code if needed and publish a new version…

i submitted a code change on the YouTube agent for the last character trim

In order to fix the http://127.0.0.1 issue I had to make sure “Secured Connections” was set to “Preferred” instead of required, which really sucks, but whatever…
Where is that ?

Will reproduce now:

  • ensure collection info displayed in logs
  • grouping folder with one series folder with playlist id
  • grouping folder with two series folder with playlist id

@djmixman Please avoid putting the previous post in copy when you reply, it make it more difficult to seaparate the actual post from the vrevious, especially when including it twice.
Yeah I noticed it double posted after submitting my post and I can’t edit the post. Also it doesn’t help that this forum software is atrocious… :confused:


In order to fix the http://127.0.0.1 issue I had to make sure “Secured Connections” was set to “Preferred” instead of required, which really sucks, but whatever… Where is that ?
Settings -> Server (Tab) -> Network -> "Secure connections" dropdown...

I’m not sure what all is using the insecure side. I’ve noticed a few reference in your code but changing them to https does not seem to make a difference.


  • Looking at the root sanner log it seems like it is detecting the groups.

root.scanner.log:

Library: 'TestTube', root: '/mnt/Media_RW/Videos/TestTube', path: '', files: '0', dirs: '3', Plex scan date: 2018-06-03 17:00:28
=============================================================================================================================================================
[no files detected]

root level manual call to Grouping folders
-------------------------------------------------------------------------------------------------------------------------------------------------------------
[Bay Area Buggs] Grouping folder (contain 2 dirs)
- Bay Area Buggs/GTA 5 Roleplay OCRP Episodes [PLkIGJ7Ljv6YzLKR6_vfN0sNDpJBQ0C8FJ], subdir_files:   4, reverse_path: ['GTA 5 Roleplay OCRP Episodes [PLkIGJ7Ljv6YzLKR6_vfN0sNDpJBQ0C8FJ]', 'Bay Area Buggs']
- Bay Area Buggs/House Flipper Episodes [PLkIGJ7Ljv6YxjjBZjANiLbG8Kc1AQxzc_], subdir_files:   1, reverse_path: ['House Flipper Episodes [PLkIGJ7Ljv6YxjjBZjANiLbG8Kc1AQxzc_]', 'Bay Area Buggs']
-------------------------------------------------------------------------------------------------------------------------------------------------------------
[Jeff Favignano] Grouping folder (contain 2 dirs)
- Jeff Favignano/GTA 5 Roleplay _ OCRP [PLo5whi14bKmiVIH82r4sFTTbazvnsC_94], subdir_files:  10, reverse_path: ['GTA 5 Roleplay _ OCRP [PLo5whi14bKmiVIH82r4sFTTbazvnsC_94]', 'Jeff Favignano']
- Jeff Favignano/House Flipper [PLo5whi14bKmit3qYs73oi_ZmmT3bmb1qg], subdir_files:   4, reverse_path: ['House Flipper [PLo5whi14bKmit3qYs73oi_ZmmT3bmb1qg]', 'Jeff Favignano']

  • Will test pushed changes shortly