[REL] YouTube Metadata Agent

Dev Community Dev/API Corner
101

i submitted a code change on the YouTube agent for the last character trim
The changes [removing the stray rstrip] fixed the trim issue…

Found another issue:

Season/Episode is being incorrectly detected and listed as s00e501:

Library: 'TestTube', root: '/mnt/Media_RW/Videos/TestTube', path: 'Jeff Favignano/GTA 5 Roleplay _ OCRP [PLo5whi14bKmiVIH82r4sFTTbazvnsC_94]', files: '10', dirs: '0', Manual scan date: 2018-06-03 17:22:56
=============================================================================================================================================================
Forced ID in series folder: 'youtube' with id 'PLo5whi14bKmiVIH82r4sFTTbazvnsC_94'
exception: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>, url: https://www.googleapis.com/youtube/v3/playlistItems?part=snippet&maxResults=50&playlistId=PLo5whi14bKmiVIH82r4sFTTbazvnsC_94&key=AIzaSyC2q8yjciNdlYRNdvwbb7NEcDxBkv1Cass
misc_count: {'duty': 1, 'and': 1, 'seven': 1, 'family': 1, 'show': 1, 'cruise': 1, 'years': 1, '#9': 1, '#8': 1, 'reunion': 1, '#3': 1, '#2': 1, '#1': 1, '#7': 1, '#6': 1, '#4': 1, 'slow': 1, 'top': 1, 'bakery': 1, 'hat': 1, 'tour': 1, "what's": 1, 'juju': 1, 'drinking': 1, '#11': 1, 'off': 1, 'name': 1, 'diving': 1, 'shift': 1, 'bad': 1, 'car': 1, 'the': 1, '#10': 1, 'my': 1, 'scuba': 1}
"GTA 5 Roleplay _ OCRP [PLo5whi14bKmiVIH82r4sFTTbazvnsC_94]" s00e501                         "__" "Slow Shift" "GTA 5 Roleplay _ OCRP #1 - Slow Shift [JC0Km6FulkM].mkv"
<...>
"GTA 5 Roleplay _ OCRP [PLo5whi14bKmiVIH82r4sFTTbazvnsC_94]" s00e501                         "__" "Scuba Diving Tour" "GTA 5 Roleplay _ OCRP #11 - Scuba Diving Tour [3oxUtj33gow].mkv"
root level manual call to Grouping folders

Removing the “#” from the filename causes the episode numbers to be displayed correctly:

Library: 'TestTube', root: '/mnt/Media_RW/Videos/TestTube', path: 'Jeff Favignano/GTA 5 Roleplay _ OCRP [PLo5whi14bKmiVIH82r4sFTTbazvnsC_94]', files: '10', dirs: '0', Manual scan date: 2018-06-03 17:28:41
=============================================================================================================================================================
Forced ID in series folder: 'youtube' with id 'PLo5whi14bKmiVIH82r4sFTTbazvnsC_94'
exception: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)>, url: https://www.googleapis.com/youtube/v3/playlistItems?part=snippet&maxResults=50&playlistId=PLo5whi14bKmiVIH82r4sFTTbazvnsC_94&key=AIzaSyC2q8yjciNdlYRNdvwbb7NEcDxBkv1Cass
misc_count: {'duty': 1, 'and': 1, 'seven': 1, 'family': 1, 'show': 1, 'cruise': 1, 'years': 1, '#9': 1, '#8': 1, 'reunion': 1, 'diving': 1, '#1': 1, '#7': 1, '#6': 1, '#4': 1, 'slow': 1, 'top': 1, 'bakery': 1, '3': 1, '2': 1, 'hat': 1, 'tour': 1, "what's": 1, 'juju': 1, 'drinking': 1, '#11': 1, 'off': 1, 'name': 1, 'shift': 1, 'bad': 1, 'car': 1, 'the': 1, '#10': 1, 'my': 1, 'scuba': 1}
"GTA 5 Roleplay _ OCRP [PLo5whi14bKmiVIH82r4sFTTbazvnsC_94]" s01e002                         "__" "Top Hat" "GTA 5 Roleplay _ OCRP 2 - Top Hat [WC7dMOovFak].mkv"
"GTA 5 Roleplay _ OCRP [PLo5whi14bKmiVIH82r4sFTTbazvnsC_94]" s01e003                         "__" "Off-Duty Drinking" "GTA 5 Roleplay _ OCRP 3 - Off-Duty Drinking [htyctdVVg_I].mkv"
"GTA 5 Roleplay _ OCRP [PLo5whi14bKmiVIH82r4sFTTbazvnsC_94]" s00e501                         "__" "Slow Shift" "GTA 5 Roleplay _ OCRP #1 - Slow Shift [JC0Km6FulkM].mkv"
<...>
"GTA 5 Roleplay _ OCRP [PLo5whi14bKmiVIH82r4sFTTbazvnsC_94]" s00e501                         "__" "Scuba Diving Tour" "GTA 5 Roleplay _ OCRP #11 - Scuba Diving Tour [3oxUtj33gow].mkv"
root level manual call to Grouping folders

Feature Request

While thinking about it, instead of listing everything as season one, maybe have it detect the year and use that as the Season?

102

Please update to latest scanner, solve:

  • file added with no path, just filename
  • SSL cert error on Linux and OS-X while will use the video id in filename to number files
  • pasted the wrong file in scanner code, might have caused the file deletion…

Please update to latest agent, solve:

  • better logging, collection from grouping folders

Feature request:

  • i can add different playlist spanning potentially same year or several years as seasons so this would conflict
  • I can understand for long spanning series or time based series (John Oliver)and would need a tag like [youtube2-PLxxxxxxxxx] to know i have to initiate that. the will the ep number keep the playlist index (more logical if long series or weekly formats).
103

First off, thanks for the superfast response with all of this! :smile: It’s much appreciated.

Whats Fixed:

  • It appears that “# issue in filenames” is fixed and files are now being detected correctly.
  • It appears that the SSL issue is fixed.

Still have issues:

  • Same issues with grouping/collections

  1. Directories with only a single playlist are not being added. Adding an empty directory within the subfolder will allow the playlist to be added.

  2. Collections are still not being created.

_root_.*

Library: 'TestTube-3', root: '/mnt/Media_RW/Videos/TestTube-3', path: '', files: '4', dirs: '3', Plex scan date: 2018-06-03 19:11:40
plexignore_files: '[]', plexignore_dirs: '[]'
=============================================================================================================================================================
[no files detected]

Library: 'TestTube-3', root: '/mnt/Media_RW/Videos/TestTube-3', path: '', files: '0', dirs: '3', Plex scan date: 2018-06-03 19:11:40
=============================================================================================================================================================
[no files detected]

root level manual call to Grouping folders
-------------------------------------------------------------------------------------------------------------------------------------------------------------
[Bay Area Buggs] Grouping folder (contain 2 dirs)
- Bay Area Buggs/GTA 5 Roleplay OCRP Episodes [PLkIGJ7Ljv6YzLKR6_vfN0sNDpJBQ0C8FJ], subdir_files:   4, reverse_path: ['GTA 5 Roleplay OCRP Episodes [PLkIGJ7Ljv6YzLKR6_vfN0sNDpJBQ0C8FJ]', 'Bay Area Buggs']
- Bay Area Buggs/House Flipper Episodes [PLkIGJ7Ljv6YxjjBZjANiLbG8Kc1AQxzc_], subdir_files:   1, reverse_path: ['House Flipper Episodes [PLkIGJ7Ljv6YxjjBZjANiLbG8Kc1AQxzc_]', 'Bay Area Buggs']
-------------------------------------------------------------------------------------------------------------------------------------------------------------
[Jeff Favignano] Grouping folder (contain 2 dirs)
- Jeff Favignano/House Flipper [PLo5whi14bKmit3qYs73oi_ZmmT3bmb1qg], subdir_files:   4, reverse_path: ['House Flipper [PLo5whi14bKmit3qYs73oi_ZmmT3bmb1qg]', 'Jeff Favignano']
- Jeff Favignano/GTA 5 Roleplay _ OCRP [PLo5whi14bKmiVIH82r4sFTTbazvnsC_94], subdir_files:  10, reverse_path: ['GTA 5 Roleplay _ OCRP [PLo5whi14bKmiVIH82r4sFTTbazvnsC_94]', 'Jeff Favignano']

Let me know if you need any other log files.

104

Another new issue…

  • Different files with the same season/episode number show up as the same episode in Plex

Example:

Download two single videos from the same creator.

When the scanner/agent run it will not detect a video on a playlist and does not have any season/episode numbers to work with. So each video gets assigned the same episode s00e501.

1032×484 68.7 KB

A better way of handling single videos without numbers will be needed in order to keep Plex from combining these videos into one episode. Maybe make plex check for the highest episode number in that series and adding 1 when it cant find a season/episode number in the metadata? I don’t know of any decent way to handle this though at the moment.

Test case:

https://www.youtube.com/watch?v=DyqUw0WYwoc
https://www.youtube.com/watch?v=kW97CjpmrXw
105

Uploaded fix for scanner

  • fix dual ep 501, was resetting the counter at the wrong place. Now added but without metadata information. not sure how to fix…
  • fix single season in grouping folder which fixed the rest (collection needs 2+ folders in grouping folders)

Please test and report.

  • On scanner behaviour changes, you may have to remove, scan then add back in folders

Possible new features

106

Sweet, Nice work! So far everything seems to be working as planned… I’ll be pretty busy this week so if there is more testing needed I may be slow to respond.

107

Uploaded new version rewrote the whole series agent part with simpler code

  • support channel and playlist id (preferred method)
  • support channel search name (need to be unique)
  • support video id search if not in playlist or channel list, support ids without brackets
  • removed playlist id support on season folders, use collections to group playlist together

Please test and feedback
Might have to put back api personal key setting if reaching 1000 queries a day…

108

@djmixman @s1mpleman @Goof245 @Flakeinator @badog Please test the new version. loads of work went into it
I did remove playlist on season folders in the agent but it support channel ids and you just need the video id on each filename which you should have already…

109

It is on my to-do list… I’m so swamped with work right now i’ve only had time to eat and sleep… I hope to do some testing for you Sunday and will report back. :slight_smile:

110

@ZeroQI

I have a few questions about your scanner and agent. My need is very simple and I’m hoping this is the right place to find my answer. I download videos that are sorted by playlist and this is how I want to organize them in plex. My problem is that I have to manually add them to a collection and playlist. I also have a problem with the poster art and banner. What i’m looking for is a way to take my files and add a plex friendly poster, banner, tag to a collection, and add to a playlist. Is this possible with your setup?

111

@polayer2017 https://github.com/ZeroQI/YouTube-Agent.bundle

To do:

  • Download with YouTube-dl which add YouTube VideoId at the end of all filenames or include it manually (important point). You do not need to rename the files, the scanner will put the right episode number for you
  • Put files in a playlist folder with YouTube Playlist ID [PLxxxxxxxxxxxxx]
  • Put the Playlist folder in a Collection Tag folder/Playlist name [Youtube_Playlist_ID_PLxxxxxxxx]/YouTube_Title_xxxx_YoutubeVideoID.ext
  • if more than 1 Playlist in folder it will create a Collection tag with the name of the grouping folder
  • install the ASS scanner
  • install the YouTube Movie and Series Agent (1 agent does both libraries type)
  • Create a TV Series Library pointing to a root folder using the agent and scanner
  • Consider a donation

I don’t think the agent add banners but add poster and art.
I never saw the web interface use banners nor a client doing it. How do you see banners, which client?
Report missing metadata AND where in youtube you see in and link to channel/playlist/video

112

@ZeroQI

Thanks for the quick response! I followed the instructions and can’t seem to get posters, metadadata, or collections to work.

I’m using this with youtubedl to establish my folder structure.
%(uploader)s/%(playlist)s [%(playlist_id)s]/%(upload_date)s - %(title)s [%(id)s].%(ext)s"

My goal is to organize by uploader/playlist/videos. Some uploaders will have several playlist in the root folder of uploader. I would like to use your scanner to automate posters, metadata (as much as possible), and automate collections. I would also like to create a playlist from the collection if possible. I’m not sure if I need the seasons setup, would there be a way to simply load the files and sort by upload date?

This is what i’m getting.

zazq07eeamjn.png876×251 177 KB

t84uf4m6egwt.png725×251 7.85 KB

I believe I have selected the correct scanner/agent, but I can get it to work.

v8cujr0hclh3.png720×453 19.4 KB

113

Please give the path from the library root folder to one media file. The folder the files are in need to have the playlist id.

114

@ZeroQI said:
Please give the path from the library root folder to one media file. The folder the files are in need to have the playlist id.

W:\Ryan Scribner\Ryan Scribner - Financial Tips [PLXNL7ROldCKgWE-mtUgSYRjZFXroOLo-g]\20180502 - 5 BIGGEST MONEY LIES YOU BELIEVE! [a2kXRa0AC48].mp4

115

Grouping folder, Playlist id on series folder video id on file, numbering for me is perfect all ids valid:

YouTube agent logs show the series gets no title therefore no metadata updated.
Seem like a grouping folder with one series inside only trigger this.

Edit: Please update to latest and report, have reproduced it and solved it

116

@ZeroQI said:

Edit: Please update to latest and report, have reproduced it and solved it

The update to the code worked! Now I’m having a new issue. When importing the videos I’m getting nothing. Does the script allow for recursive scanning? I can get it to work from the playlist folder, but it skips when scanned via a group of uploader folders.

I’m looking to import similar to this.
W:\Ryan Scribner\Ryan Scribner - Financial Tips [PLXNL7ROldCKgWE-mtUgSYRjZFXroOLo-g]\20180502 - 5 BIGGEST MONEY LIES YOU BELIEVE! [a2kXRa0AC48].mp4
Uploader root folder (TV Show)
Playlist #1 from uploader (Season 1/Collection #1?) Sorted by date found in title of the video file
Playlist #2 from uploader (Season 2/Collection #2?) Sorted by date found in title of the video file

Folder structure of all files:
%(uploader)s/%(playlist)s [%(playlist_id)s]/%(upload_date)s - %(title)s [%(id)s].%(ext)s"

Few extra requests/questions.
*I will be adding additional videos in the future to most of the playlists. I’m looking to simply add to the existing collection and keep the order sorted by %(upload_date)s. Not sure how to address this particular setup in plex (Series, Collection, or other method?).
*Would it be possible to embed a picture into the poster of the uploader root folder? Each additional video file inside the playlist folder would simply default to the exisiting JPEG found in the folder or pulled from youtube (however the script accomplishes this). Maybe naming the files Poster, Banner, and Background for simplicity of use?
example:
uploader folder/poster.jpeg, banner.jpeg, and background.jpeg (user supplied) or pulled from youtube if possible.

117

You need to select the following library settings:

  • Collections: ‘Hide items which are in collections’
  • Seasons: ‘Hide for single-seasons only’ or ‘Hide’

I did create:

  • W:\Ryan Scribner\Ryan Scribner - Financial Tips [PLXNL7ROldCKgWE-mtUgSYRjZFXroOLo-g]\20180502 - 5 BIGGEST MONEY LIES YOU BELIEVE! [a2kXRa0AC48].mp4
  • W:\Ryan Scribner\Financial Education[PLGhRZdsPDX33JDEhNXxAkMMNWIuA56X7v]\QUIT MY $70,000 JOB AFTER LEARNING THIS LESSON ABOUT MONEY[_8M4HC19XDo].mkv

It did create a collection with the 2 shows…

Are you using Absolute Series Scanner as per the readme so it number the files as per the playlist order?

Already achieved by LocalMediaAssets agent integrated in Plex: https://support.plex.tv/articles/200220717-local-media-assets-tv-shows/

  • poster.jpg or poster-1.jpg in series (playlist) folder
  • banner.jpg or banner-1.jpg
  • background.jpg or background-1.jpg

If you set a poster it will be used for season posters as well
I do set the poster but a wide picture in cropped portrait mode doesn’t look good

118

I’m still not getting anything using the script. I uninstalled Plex and started from scratch. Here’s what I’m getting from the script after a fresh install of Plex.

Checked scanner/agent settings.

Importing from a network drive.
All files in Uploader/Playlist/Title format. (%(uploader)s/%(playlist)s [%(playlist_id)s]/%(upload_date)s - %(title)s [%(id)s].%(ext)s")
Folder’s import but no posters, metadata, and inconsistent naming of files.

6j2arrxws4iw.png1021×389 324 KB


I’m hoping to get this to work and I appreciate any help.

I’m going to send my logs via PM if that’s ok.

119

So I am running into an issue that I haven’t been able to track down yet that has been going on even before the re-write.

During the scan it will add the videos but will not add the path to the video file causing it to not be found when trying to play the video.

From what I can tell it seems to be adding the file to the database with no path, just the file name. It also only does it on some videos:

6gkf0fzhdsb5.png998×249 73.8 KB

z2sxyroxkcq4.png983×317 91.1 KB

Notice the one from Asian Boss has a full path and that video plays with no problems.

Edit:

I’m also not getting collection generation for folders that don’t contain playlists.

Since there is not a playlist id (there isn’t a playlist for these) this does not group the subfolders into a H2ODelirious collection.

120

Just noticed I’m having a similar issue to yours. Some of the files have a red unavailable. Do you store your files locally or use a network drive by chance?