[Rel] HTTP Anidb Metadata Agent (HAMA)

@hthuong09 Thanks for the update, it is not resolved like i thought. Guid is anidb-10891, so http://anidb.net/perl-bin/animedb.pl?show=anime&aid=10891

https://raw.githubusercontent.com/ScudLee/anime-lists/master/anime-list-master.xml:

  <anime anidbid="10891" tvdbid="102261" defaulttvdbseason="0" episodeoffset="" tmdbid="" imdbid="">
    <name>Tsukimonogatari: Yotsugi Doll</name>
    <mapping-list>
      <mapping anidbseason="1" tvdbseason="0">;1-15;2-15;3-16;4-17;5-18;</mapping>
    </mapping-list>
  </anime>

So these are the mappings:

  • AniDB s01e01-02 maps to TheTVDB s00e15
  • AniDB s01e03-05 maps to TheTVDB s00e16-18

Displayed are TVDB s01e01-04, so i need to correct it by correcting the agent wil ltake couple days.

How to rename the anime series using HAMA? such as one piece that has many series. Im using filebot. Thanks

Either follow thetvdb numbering if you have seasons or anidb numbering otherwise.
https://www.filebot.net/forums/viewtopic.php?t=4186

@hthuong09 issue reproduced

  • added to agent logs the new season and ep for easier t/s in the future
  • defaultseason ā€˜0’ it didn’t load the ep mappings (bad), and wouldn’t have due to season 0 restriction somewhere else (less bad)
  • could not handle start end (pretty bad)
    Now resolved. uploaded to beta branch.

Collections support in plex can now hide items in collection and show the collection’…

  • anidb collection regroup everything in scudlee movie collection increased by anidb related anime which adds series…
  • TheMovieDB has a collection field already supported

Questions:

  • Shall i develop a tvdb collection field for all items mapped under the same tvdbid ? developping a collection for 2 seasons anime seem convoluted…
  • shall i accumulate all collection fields from all sources like posters ?

hey @ZeroQI
I updated earlier this week to the newer beta version (thx for the overlord season fix)
I did notice to run into another problem now i have not seen before.
I have an anime called ā€œstrike the bloodā€ and when i do a fix match it always seems to mark my match to (tvdb-272129) while i have set up my collection to always use anidb.
The issue i have now is that thetvdb marks STB as 1 season and then the 2 ova + STB2 are specials. Anidb does have STB2 listed as a sequel and should normally find the correct matches on Season 2 episodes.

Is there any reason i can not get my fix match as (anidb-xxxx) for that show? I added another show to test (kill la kill) and that one still offers me the anidb results.

I want to include logs, and i know you mentioned before you should have a log / show now? But i can not seem to find your show separated logs, only the same old hama log as always. What is the path for logs you want me to add? I will include the only one i find for now.

@T-Blade I can see in the logs and need to check why anidb search isn’t tried. you have plex framework crashing with acess errors, may be linked
Here is the search code

 ### Check if a guid is specified "Show name [anidb-id]" ###
  import re
  match = re.search(u"(?P<show>.*?) ?\[(?P<source>([a-zA-Z0-9]*))-(tt)?(?P<guid>[0-9]{1,7})\]", orig_title, re.IGNORECASE) if ' [' in orig_title else False
  if match:
    guid=match.group('source') + '-' + match.group('guid')
    results.Append(MetadataSearchResult(id=guid, name=match.group('show')+" ["+guid+']', year=media.year, lang=lang, score=100))
    Log.Info("Forced ID - source: {}, id: {}, title: '{}'".format(match.group('source'), match.group('guid'), match.group('show')))
  else:  #if media.year is not None:  orig_title = orig_title + " (" + str(media.year) + ")"  ### Year - if present (manual search or from scanner but not mine), include in title ###
    maxi, n = 0, 0
    if movie or max(map(int, media.seasons.keys()))<=1:  maxi, n =       AniDB.Search(results, media, lang, manual, movie)
    if maxi<50 and movie:                                maxi    =  TheMovieDb.Search(results, media, lang, manual, movie)
    if maxi<80 and not movie or n>1:                     maxi    = max(TheTVDB.Search(results, media, lang, manual, movie), maxi)
  Log.Info("".ljust(157, '='))
  log.stop()

You wrote: ā€œAnidb does have STB2 listed as a sequel and should normally find the correct matches on Season 2 episodes.ā€

Anidb and tvdb both don’t have a season 2 so i am pretty sure you created your own issue there…
Please confirm, since you didn’t actually give me a folderlist or filelist, did you actually create a season 2 folder into into ā€œL:\Ecchi\Strike the Bloodā€ to put Strike the Blood II eps in it?

If so, the agent work exactly as written AND intended as multiple seasons series will only search tvdb.
Anidb only have single seasons and specials so i do not search it if there is multiple seasons.
If you did exactly that and feel bad, please make yourself feel better by donating…

If you follow anidb numbering it will look like:

  • strike the blood/strike the blood ep1.ext
  • Strike the Blood II / Strike the Blood II - ep01.ext
  • Strike the Blood Valkyria no Oukoku Hen/Strike the Blood Valkyria no Oukoku Hen.ext

If you follow tvdb numbering

  • strike the blood/Specials/s00e01-02 Strike the Blood - Valkyria no Oukoku Hen
  • strike the blood/Specials/s00e03-10 Strike the Blood II
  • strike the blood/01-24.ext

Ok,
So to summarize; It only searches anidb for single season shows and uses tvdb for multi (since i did have them listed in a season 02 folder) I thought if it was listed as a sequel on anidb it would give the sequel a new season and match it. (i guess that might be too hard and complex with anime due to the OVA - movies - and actual seasons listed)
I just renamed them to S00E3-10 as i found on tvdb. Thx for the explenation.

be careful, next time you re-create the library it will match to anidb since there is a single season now, so you should add " [tvdb-272129]" to the series folder name …

The code is complex enough as iit is, as i need to map anidb to tvdb eps regardless if the serie is anidb or tvdb numbered then also considered anidb series with seasons tvdb numbered… And i added tvdb2,3,5 modes and anidb2 and it is very tricky to have the code working smoothly…

That is the simplest i found so it detect anidb and tvdb series for people following naming conventions…
If people don’t follow them, then the metadata match will not work anyway…

Any suggestion are welcome…
I am thinking about collections since Plex made it cool now (hiding serie if in collections).
I would need examples when it doesn’t group as it should…

I was just about to write and mention they added the possibilities of collections to plex now.
Not sure how hard it would be, but collections + sorting title could make for a very easy to follow structure for some anime.
I used collection + sorting title for fast and the furious films as an example. They used to be all over the place with the different names.
I noticed it researched everyting on anidb now. Still did the job except for S00E01, it listed as special S1 instead of the actual title. I will try the rename folder with brackets and see if that finds all like it should

if you stick to a convention and make hama aware, it will all work like magic…

Already supported for collection field:

  • Scudlee xml movie collection complemented with anidb related anime list in the AniDB serie xml
  • TheMovieDb collection field…

Possible addition:

  • i could detect grouping folders like ā€œDragon Ball/Dragon Ball Super/ep01.extā€ as collection variable…
  • group all anidb ids sharing the same TheTVDB id…

For that, i need examples or series that should logically be grouped that aren’t currently getting a collection string…

I just did a full refresh metadata on my plex anime to see what it changes. Some titles got reverted to English instead of x-jat for some reason. And the only show i could think off to check for collections was Berserk. the 90’ version doesn’t group with the newer 2013+ ones.

on the side note, would you like access, as a normal user, to my plex? It might help you to find certain things and see what it does to most of my shows (and i have quite a bit of shows)

Off to bed now, refreshes should be done during the night.

Shouldn’t have changed any title…

https://raw.githubusercontent.com/ScudLee/anime-lists/master/anime-list-master.xml

  <anime anidbid="98" tvdbid="73752" defaulttvdbseason="1" episodeoffset="" tmdbid="" imdbid="">
    <name>Kenpuu Denki Berserk</name>
    <supplemental-info>
      <studio>OLM</studio>
    </supplemental-info>
  </anime>
  <anime anidbid="7991" tvdbid="307111" defaulttvdbseason="0" episodeoffset="" tmdbid="" imdbid="">
    <name>Berserk: Ougon Jidai Hen</name>
  </anime>
  <anime anidbid="11851" tvdbid="307111" defaulttvdbseason="1" episodeoffset="" tmdbid="" imdbid="">
    <name>Berserk</name>
  </anime>.
  <anime anidbid="12414" tvdbid="307111" defaulttvdbseason="2" episodeoffset="" tmdbid="" imdbid="">
    <name>Berserk (2017)</name>
  </anime>

Anidb relation graph: http://anidb.net/perl-bin/animedb.pl?show=rel&aid=11851

3 series one movie so not in ScudLee’s movie collection XML:

Won’t work until i code some better collection support…
the old and new series can’t be grouped per their tvdbid as 2 separate tvdb entries so:

  • need to list all related anime in anidb xml
  • need to find all tvdbid
  • from this tvdbid list to list all anidbids concerned in anime-list-master.xml
  • This list of anidbid + tvdbid is a collection.

Looking at a way to make this work at the moment, will take some time…

With the renaming part,
I noticed if i use thetvdb as the metadata, it uses sometimes English names instead of x-jat. If i change it back to match from anidb it uses x-jat names. problem with anidb is that sometimes i cannot link the special as i should.

example:

Mahouka Koukou no Rettousei, 26 episodes on anidb + specials 1-32 (actual specials - short additions to the anime that i do not have or want) This serie however does have a movie sequel. Separate entry on anidb so it cannot match it i think.
If i use thetvdb, i get 26 episode + special 1-8 where nr 8 is the movie.

So i have to choose between English title + matching special or x*jat title + wrong special name

It should behave the same title wise as you select the metadata source priority in agent settings so it doesn’t matter in an anidb or tvdb guid is used. Here are your configured agent settings from earlier log:

  • Prefs[SerieLanguagePriority ] = x-jat, main, en, ā– ā– ā– 
  • Prefs[EpisodeLanguagePriority] = en, main, x-jat, ā– ā– ā– 
  • Prefs[title ] = AniDB,TheTVDB

Right now you have episode title in **english ** first, from AniDB then TheTVDB for both series title and episode titles… and search in x-jat(romaji) then english and not in japanese since mispelt ('should be ā€˜ja’, made the error for months in default settings which don’t change upon upgrade)

Please update agent settings in your case to:

  • Prefs[SerieLanguagePriority ] = x-jat, en, ja
  • Prefs[EpisodeLanguagePriority] = x-jat, en, ja
  • Prefs[title ] = AniDB, TheTVDB | AniDB, TheTVDB

I reproduced the issue only when thetvdb is first if x-jat is first for ep title.
AniDB has xjat but not TheTVDB, and thetvdb loads in library language (which is the language for summaries) so ā€˜en’. Not ideal, need to use tvdb api v2 in json ans see if i can improve

New beta scanner and agent

  • Scanner: seem like grouping folders work with caching now and normal plex folder call if it contain only one folder to mimic plex normal scanner behaviour
  • Agent: using grouping folders as collection now

Due to recent Plex changes, Collections are more practical than ever, and i recommend when creating/editing the library:

  • Collections: ā€˜Hide items which are in collections’
  • Seasons: ā€˜Hide for single-seasons series’
    **This way you see collections with series inside and no season folders if movies or single season series
    **

Currently only using first avail collection field is used, not all.
Ideas:

  • Already done: using grouping folders
  • Already done: Scudlee’s AnimeList movie collection xml is supplemented by AniDB related anime, but mainly movies…
  • create a collection grouping all series with the same tvdbid, would help specials but not group same universe like Dragon ball much, use tvdb title then?
  • create a collection using pure AniDB related anime field in AniDB xml, but which unique string for collection ? could find tvdb title or anidbid for tvdb season 1 or lowest anidbid?
  • create a Universe grouping (Tenchi, Dragon Ball, Fullmetal) xml for Hama? which xml format? could use Studio Ghibli Collection and we have mapping files for matching anidbid to tvdbid…

Please let me know ideas

Downloaded and installed the new betas.
The grouping feature does look very nice but needs a lot of testing on my end to get it to work well.
While getting your new betas, i stumbled across this section:

anidb2 Absolute TVDB TVDB Map Anidb entries to the tvdb at the right season and ep number to show one Plex series entry only (need to be within a single tvdb entry)
Sword Art online [anidb2-8692]
Sword Art online II [anidb2-10376]
Sword Art Online Extra Edition [anidb2-10022]
Gekijouban Sword Art Online: Ordinal Scale [anidb2-11681]

i Have tried to use this exact naming for sword art online just to see what it would do.
Testing both Diskname\Anime\series\SAO\folders listed + anidb2 And Diskname\Anime\series\folders listed + anidb2 did not give the result i was hoping for. Not sure what i wanted it to do and what is possible with plex but i was hoping it made a sort of collection with everything listed separate. In both cases it managed to add 1 Entry called SAO split into S1, S2 and specials. The move seems to end outside of the collection and not in it. The specials did seem to end up right (using tvdb numbering) so the extra edition and soa2 special ended there.

So, not sure if this is achievable, but what i think i might prefer would be:

Having 1 entry in plex called SOA (either a collection or a single listed show) then however i would like to have stuff listed more seperated. For example:
Sword art online / sword art online 2 / extra edition / cardinal scale movie / specials.
A bit more like anidb rather than tvdb. specials and movies can get very very confusing to watch since specials contains both extra edition (s00e11 according to tvdb) and S00e12 (soa2 special) debrief (=episode name)
There is no way to tell if when to watch certain specials (sequel ova’s movies) or what order.
I guess this is a limitation by plex? It would be awsome if plex could somehow start listing shows as not having a season but rather a Sequel / prequel if wanted.

I am also playing around with Freezing and Freezing vibrations but also have issues getting them to work. i tried the same anidb2 naming there disk…\freezing\freezing + anidb2 and freezing vibration + anidb2
They seem to end up seperated in my list now. They do not turn into a collection either. I might be getting something wrong with the naming. Should it be disk…\freezing [tvdbid]\freezing anidb2 + freezing vib anidb2

And as mentioned before, if you want i can give you access to my plex library so you can have a look at how things turn out sometimes, and also feel free to watch something ofc :smiley:

@T-Blade

Your logs shows the grouping folder for freezing was detected, however not used due to agent settings.
Currently for collection the source order list is: ā€œAnimeLists, AniDB, TheTVDB, MyAnimeList, TheMovieDb, OMDbā€ā€¦
Please make agent setting for collections ā€˜Local, TheMovieDb, AniDB’ to use it.
When i change defaultPrefs.json default settings, it is not reflected in already set settings so am trying to code a default settings function but could not. If you can help making a default setting, please message me…

Plex cannot place specials between episodes you need Kodi for that but cannot have series+movies in one library, nor grouping folders, nor anidb2 mode

All anidb2 does is show anidb series as if you used tvdb numbering without having to rename and move the files. so you end up with pure tvdb numbering is correct. you need anidb numbering if using anidb2 mode.

You have numbered on your disks the movie as a tvdb special s00e10 so it cannot show as a series entry:

  • Sword Art Online\Specials\Sword Art Online - S00E10 - Sword Art Online Extra Edition.mkv
  • Sword Art Online\Specials\Sword Art Online - S00E22 - Sword Art Online The Movie Ordinal Scale.mkv

What you want is a collection indeed,so the same ā€˜AOA’ string in collections field in all series:

  • if you put all series below in ā€˜SOA’ then with ā€˜Local’ as source first on latest beta branch agent code, it will go in a collection
  • TheMovieDb collection field is used, just dunno how good it is…
  • AniDB currently complement the Scudlee anime movie list with related anime, whic is not great…
  • to implement: using anibd related anime only so all dragon ball are 1 collection
  • to implement: all anidb series sharing the same tvdbids, but separate dragon ball series would be 3 collections at least

I noticed collections list by release date by default or alphabetical order, which is cool

You can set up text in the collection, would be great if we could have the Viewing order there… Where could i find the information to populate ?

Can we access and populate from the agent side?

  • movie: metadata.collections.clear()
  • movie: metadata.collections.add(collection)
  • metadata.collections[collection].summary maybe?

I would do that:

  • set Collection field as ā€˜Local’ first to have freezing and SOA as collections, with library setting to hide items in collections so only the collection is showing and to not show seasons if a single season is present.
  • set Collection text to:

-Sword Art Online
-Sword Art Online - Extra Edition
-Sword Art Online II
-Sword Art Online - Ordinal Scale

  • SOA\Sword Art online [anidb-8692]\eps…ext (abs numbering or s01exx)
  • SOA\Sword Art online II [anidb-10376]\eps…ext (**no **season 2 on eps nor season folder 2)
  • SOA\Sword Art Online Extra Edition [anidb-10022]\Sword Art Online Extra Edition.mkv
  • SOA\Gekijouban Sword Art Online Ordinal Scale [anidb-11681]\Gekijouban Sword Art Online Ordinal Scale.mkv
  • Freezing/Freezing [anidb-7900]/
  • Freezing/Freezing vibrations [anidb-9789]/

Please change the setting, rename as indicated, refresh the metadata and reply back.

Any thoughts? I have pokemon arranged into 5 folders for the five shows and are using tvdb4 for the series

The posters are correct…the names of the files are correct

but there are no episode descriptions…at all - or any metadata beyond date and name

i know i’m missing something simple
Running current version of everything

@mcrommert You’re missing something simple: logs… They are mentioned in the troubleshooting section of the read-me, and links for both are in my signature and T-Blade attached 7 of them 2 posts higher… It should take absolutely no time to attach it to your post using your Verizon gigabit Fiber internet connection :D… You used Beta branch for both scanner and agent, right?

Thanks -
I stopped trying the method i had been (tvdb4 with folders already built) as Pokemon has the correct season structure in tvdb and i thought i would make use of that and i could fix my lack of episode descriptions

I moved to tvdb3 trying both Pocket monsters and Pokemon as the title and no cigar - I as not on beta releases…most recent normal releases. It will match title and pull correct posters, but only one season and no info on the episodes…also seems to be pulling the name only from the file title as far as i can tell

Was able to correctly do a couple shows in tvdb4 so the scanner and hama seem to be working otherwise - all requested log files are packaged in the attached zip

EDIT: you will see in the logs i have been coming at this a couple different ways with different files - but the last ones are the ones !

EDIT2: lol this is probably the problem - i might have to reevaluate how i do this

TVDB season mode (3) enabled, serie url: 'http://thetvdb.com/api/A27AD9BE0DA63333/series/76703/all/en.xml'
different abs number found on ep (s10e49) after starting to manually place our own abs numbers. all episodes will be added as season 1.