[Rel] HTTP Anidb Metadata Agent (HAMA)

Dev Community Dev/API Corner
101

@TazgodX: add new regex in scanner for your naming convention since you didn't use brackets... nothing to do with Hama. The XML update automatically now and cached for 2 weeks and saved in data folder

@dirk...: Xmls now update automatically but use 2 week cache and save a copy in data folder every tine so you know what the Xml content is... no need for anime-lists folder anymore, can't force local load anymore.

You don't need to save the Xml but wanted to see if content was loading in case you were temp banned
Honestly it's tricky to troubleshoot sine it works for me...

If the file show up correctly then the scanner is out of the loop. I actually used S - BABS.py since it's compatible in behaviour when no tvdb.id is present

 

I created Bible Black New Testament/Bible Black New Testament - 01.mkv to 06.mkv, i get serie title, anidb poster, no episode summary because tvdb doesn't have hentai... seem to be just you, and i don't know why...

Logs seem to point at my code crashing. sure you have the data folders created ?

102

@ZeroQI: Studio icons support is awesome! I think I can help you with that. just downloaded from https://sites.google.com/site/anidblist/AnimeStudios.zip%5D and I can rearrange them to the proper size. However their are about 120 studio images so I can't provide an estimate date.

103

Thanks ZeroQI, i just replaced all parentheses with brackets and it works now. I didn't know if you had that coded in, but thanks!

I was thinking about the Studio images, and I think if you made a HTM log file like you do for missing theme songs we could all contribute and find a couple that we need. I don't want to find 100+ studio graphics, but i don't mind trying to find 10 or 20 im missing.

Just like the themes file just link a place to send it, and the restrictions on what you want it to be.

104

I couldn't possibly have that coded in... To be precise it is the scanner (BABS or other) that translate filenames and folders into serie name, season, episodes, or more precisely the regular expressions (called regex) at the beginning of the scanner code. It is known that CRCs cause scanning problems and i noticed brackets at the end don't bother generally the detection... i generally use brackets for movies: "title (year) [special edition, crc, etc...].ext"

The agent is to take the file showing and finding the serie accuratly, and downloading metadata (serie summary, poster, season posters, episode summaries, studio, producers, etc...). for a good understanding read the framework PDF

The following Agent shows how to access files in local folder: https://forums.plex.tv/topic/38402-metadata-agents-for-exported-xbmc-library/. I am trying to find out how to access plex folders for the agent to check if the file exists then create a log

@sparanoid&TazgodX: If you redo logos, you might want to modify directly Plex database so it gets included in the next Plex release for everybody https://github.com/plexinc-plugins/Media-Flags.bundle . Since an xml file lists all studio icons, i would like to avoid an issue where two package exists (movies, documentaries, channels) and people have to manually mix the two xml together...

Also thanks for actually looking at the logs and giving feedback...

I took me some time to do and i was under the impression nobody really used the agent since i had only 68 downloads on it...

For logs, i wanted to allow common users to help the databases in small ways for everybody's benefits...

BTW, if you enable web links, you will have links to anidb, tvdb and relation graphs straight from the serie summary.

For the bible black hentai, naming works for me so advise you reinstall hama from the zip and update to latest code file on gihub and try again. try to use latest babs with tvdbid support even though you won't use that part

105

Logs seem to point at my code crashing. sure you have the data folders created ?

Yes, all folders in ./Plex Media Server/Plug-in Support/Data/com.plexapp.agents.hama/DataItems

I will try a reinstall and give feedback if it works.

Thanks for your work.

106

For studio icons, for a comparison, XBMC uses png file, white-on-clear, sized 161px x 109px, and are saved in 'skin.aeon.nox"/media/flags/studios/' for example. Already created ones: https://github.com/B...a/flags/studios. (http://forum.xbmc.org/showthread.php?tid=70398 show two sources for icons)

On Plex however, it uses 512x288px .png located in '/volume1/Plex/Library/Application Support/Plex Media Server/Plug-ins/Media-Flags.bundle/Contents/Resources/Studio/'. substitutions.xml file in parent folder contain the mappings and needs amendind

<match name="Animal_Planet" expression="animalplanet" />
<match name="Animal_Planet" expression="Animal_Planet" />
<match name="Animal_Planet" expression="animal planet" />
 
   . name is the name of the pnd, spaces and hyphens are mostly converted to underscores (remember reading about spaces not being allowed for filename but could spot exceptions in the xml)
   . expression is the name in the databases (since logos are used by plex, so movies and series agents will also use them) [one line per expression, can point to the same name]
Video and audio codecs also have logos (Hi10p and FLAC logos would be cool)
 
Found unrelated nice config file here: https://oldwiki.plexapp.com/index.php?title=Advancedsettings.xml
Didn't had time to improve logging but it's on the to do list
107

Excellent, that guide helped me get it rolling even with my FreeNAS plex version going ploink on me after the weekend!

I would strongly encourage that guide be made a little more visible, might help folks other than myself and prevent them from asking the same question later. (not the how to plug in from Plex mind you, though that link is handy)

Great work, my library's spooling up and getting all the right imagery now!

108

is 0.5 alpha still the latest release? just want to make sure since the date is 2013...

109

I am trying to add, http://anidb.net/perl-bin/animedb.pl?show=anime&aid=5868 and I just can't get it to recognize no matter what I do, I tried searching the main title, I tried searching the official title, I tried searching the aid(using aid:5868 aid=5868 5868 a5868), I tried removing the - from the title, I tried removing both the - and the ! from the title, I tried simply searching negima, I tried all combinations with and without dates (both 2008 and 2009) and all I get is "No matches found"

Edit: Actually it looks like I can't get any searches to go through, period(the data for most of my shows was downloaded by thetvdb), I had to resort to using myanimelist

110
Well i didn't really increment it when coding so i would go by the date at the beginning of the __code__.py page... The latest code is always on Github.
I did gather all process and info in the first post to increase the adoption rate and reduce the amount of useless questions, not that there have been that many but still... 
 
@arithine: "Mahou Sensei Negima Shiroki Tsubasa Ala Alba" is how i would label it
Went into the test "Bible Black New Testament" i had created for the hentai fanboy earlier in forum, went to incorrect match, searched, and voila, also "aid:5868" works as workaround as well. So in short, you did something wrong. Did you update the "__CODE__.py" file without creating the data folders ? I created an install procedure that should assist you. also, if no search go through, then looking at logs will tell you WHY.
 
I need to make it nor fail when folders are missing when i have time, and make it create a report in case of errors that you can attach here so i can reply in one post on how to solve or ammend the code. That is as soon as i have enough time to do that of course...
111

Hey,

So been using HAMA on and off, flicking between TVDB, MyAnimeList and old Anidb directly but noticed in the last few weeks its stopped pulling data, artwork, posters etc either when told to use AniDB or TVDB in the settings but it definitely is matching the series no problem.  Checking the logs only thing that stands out is;

2014-05-25 20:46:05,323 (1d2c) :  CRITICAL (core:561) - Exception in the update function of agent named 'HamaTV', called with guid 'com.plexapp.agents.hama://6645?lang=en' (most recent call last):
  File "Z:\Plex Server\Application Data\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\api\agentkit.py", line 970, in _update
    agent.update(obj, media, lang, force)
  File "Z:\Plex Server\Application Data\Plex Media Server\Plug-ins\Hama.bundle\Contents\Code\__init__.py", line 840, in update
    def update(self, metadata, media, lang, force ): self.parseAniDBXml(metadata, media, force,      False     )
  File "Z:\Plex Server\Application Data\Plex Media Server\Plug-ins\Hama.bundle\Contents\Code\__init__.py", line 541, in parseAniDBXml
    elif "s"+defaulttvdbseason+"e"+epNumVal in tvdbSummary:              tvdb_ep = "s"+defaulttvdbseason+"e"+epNumVal
ValueError
 

Anyone else seen something like this? I did a fresh install to try and get everything done by one agent and this is when its become an issue.  I have downloaded the latest from Github, created the folders etc

Running off Windows - 0.9.9.10.458-008ea34

2014-05-25 21:32:19,855 (19b4) :  DEBUG (networking:172) - Requesting 'http://api.anidb.net:9001/httpapi?request=anime&client=hama&clientver=1&protover=1&aid=7405'
2014-05-25 21:32:19,871 (13e8) :  DEBUG (logkit:13) - SearchByName: Local exact search for 'Nazo No Kanojo X' matched aid: 8853 Nazo no Kanojo X
2014-05-25 21:32:19,875 (13e8) :  DEBUG (runtime:918) - Response: [200] str, 632 bytes
2014-05-25 21:32:19,996 (13a4) :  DEBUG (logkit:13) - SearchByName: Local exact search for 'Nekomonogatari Black' matched aid: 9453 Nekomonogatari Black
2014-05-25 21:32:20,003 (13a4) :  DEBUG (runtime:918) - Response: [200] str, 640 bytes
2014-05-25 21:32:20,085 (1e54) :  DEBUG (runtime:717) - Handling request GET /:/plugins/com.plexapp.agents.hama/messaging/function/X0FnZW50S2l0OlNlYXJjaA__/Y2VyZWFsMQoxCmxpc3QKMApyMAo_/Y2VyZWFsMQoyCmRpY3QKZGljdAo0CnMyCmVuczQKbGFuZ3M3ClRWX1Nob3dzMTAKbWVkaWFfdHlwZWIwczYKbWFudWFscjEKczYKa3dhcmdzOQpzMTYKNWMxYWY2NTJkNzM4MmI2YXMxNwpvcGVuU3VidGl0bGVzSGFzaHMxCjVzNwplcGlzb2RlczEKMXM4CmVwaXNvZGljczIzCk51cnNlIFdpdGNoIEtvbXVnaSBDaGFuczQKc2hvd3MxCjFzNgpzZWFzb25zOTMKRSUzQSU1Q0FuaW1lJTVDTnVyc2UlMjBXaXRjaCUyMEtvbXVnaS1jaGFuJTVDTnVyc2UlMjBXaXRjaCUyMEtvbXVnaS1jaGFuJTIwLSUyME9WQSUyMDA1JTJFYXZpczgKZmlsZW5hbWVzNDAKMjk5NzUzZDRhOTk5MWQ0MTU0YzJlMjI3MDQ2MzcyYTEwM2U3MjhlOXM4CnBsZXhIYXNoczIKLTFzOApkdXJhdGlvbnM0CjgxOTJzMgppZHIwCg__
2014-05-25 21:32:20,085 (1e54) :  DEBUG (runtime:814) - Found route matching /:/plugins/com.plexapp.agents.hama/messaging/function/X0FnZW50S2l0OlNlYXJjaA__/Y2VyZWFsMQoxCmxpc3QKMApyMAo_/Y2VyZWFsMQoyCmRpY3QKZGljdAo0CnMyCmVuczQKbGFuZ3M3ClRWX1Nob3dzMTAKbWVkaWFfdHlwZWIwczYKbWFudWFscjEKczYKa3dhcmdzOQpzMTYKNWMxYWY2NTJkNzM4MmI2YXMxNwpvcGVuU3VidGl0bGVzSGFzaHMxCjVzNwplcGlzb2RlczEKMXM4CmVwaXNvZGljczIzCk51cnNlIFdpdGNoIEtvbXVnaSBDaGFuczQKc2hvd3MxCjFzNgpzZWFzb25zOTMKRSUzQSU1Q0FuaW1lJTVDTnVyc2UlMjBXaXRjaCUyMEtvbXVnaS1jaGFuJTVDTnVyc2UlMjBXaXRjaCUyMEtvbXVnaS1jaGFuJTIwLSUyME9WQSUyMDA1JTJFYXZpczgKZmlsZW5hbWVzNDAKMjk5NzUzZDRhOTk5MWQ0MTU0YzJlMjI3MDQ2MzcyYTEwM2U3MjhlOXM4CnBsZXhIYXNoczIKLTFzOApkdXJhdGlvbnM0CjgxOTJzMgppZHIwCg__
2014-05-25 21:32:20,085 (1e54) :  INFO (agentkit:915) - Searching for matches for {'openSubtitlesHash': '5c1af652d7382b6a', 'episode': '5', 'episodic': '1', 'show': 'Nurse Witch Komugi Chan', 'season': '1', 'filename': 'E%3A%5CAnime%5CNurse%20Witch%20Komugi-chan%5CNurse%20Witch%20Komugi-chan%20-%20OVA%2005%2Eavi', 'plexHash': '299753d4a9991d4154c2e227046372a103e728e9', 'duration': '-1', 'id': '8192'}
2014-05-25 21:32:20,114 (1e54) :  DEBUG (networking:172) - Requesting 'http://127.0.0.1:32400/library/metadata/8192/tree'
2014-05-25 21:32:20,201 (19b4) :  CRITICAL (core:561) - Exception in the update function of agent named 'HamaTV', called with guid 'com.plexapp.agents.hama://7405?lang=en' (most recent call last):
  File "Z:\Plex Server\Application Data\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\api\agentkit.py", line 970, in _update
    agent.update(obj, media, lang, force)
  File "Z:\Plex Server\Application Data\Plex Media Server\Plug-ins\Hama.bundle\Contents\Code\__init__.py", line 840, in update
    def update(self, metadata, media, lang, force ): self.parseAniDBXml(metadata, media, force,      False     )
  File "Z:\Plex Server\Application Data\Plex Media Server\Plug-ins\Hama.bundle\Contents\Code\__init__.py", line 314, in parseAniDBXml
    if movie and orig != "" and orig != metadata.original_title: metadata.original_title = orig # If it's a movie, Update original title in metadata http://forums.plexapp.com/index.php/topic/25584-setting-metadata-original-title-and-sort-title-still-not-possible/
ValueError

***Bah am now Banned for a bit on Anidb but as issue was present before starting afresh not sure what this issue is, will wait til ban clears.

112

By experience the line the error is reported on is quite off seem line 561 is where it failed when 541 is the latest reported... Please attach the full log on your prev post so I can spot which serie and episode triggered the error. I reproduce by creating same folder structure with sample video files since the content is irrelevant...

[EDIT:] PLEASE ATTACH FILES, not the content of them, it is not interesting for other peoples, it really tricky to see on a phone and i get less data

That being said, please document " I have downloaded the latest from Github" because... well... you didn't... Here is why:

This is line 334 on github:    if movie and orig != "" and orig != metadata.original_title: metadata.original_title = orig # If it's a movie,...

This is line 314, on your code   if movie and orig != "" and orig != metadata.original_title: metadata.original_title = orig # If it's a movie, 

Please update __code__.py with latest from git

113

Very odd, the log may of been when trying to compare and attempting the various versions.  But deleted plugin and redownloaded it. Same issue it matches series in this case Zombie Loan, it pulled two genres Horror and Manga both rated 600 but had present at 600 as well and several others over 200.

Update - Some reason plex isn't accessing https://rawgit.com/ScudLee/anime-lists/master/anime-movieset-list.xml but browsers can, so modified the code to just go to http://rawgit.com/ScudLee/anime-lists/master/anime-movieset-list.xml and it started pulling images.

114

Only two are ever directly displayed, pretty sure you see them all when clicking edit > Tags > Genre...

 
Now the log:
2014-05-26 01:22:33,010 (28c) :  DEBUG (logkit:13) - xmlElementFromFile - Loading XML file from url failed: https://rawgithub.com/ScudLee/anime-lists/master/anime-list-master.xml
2014-05-26 01:22:33,012 (28c) :  DEBUG (logkit:13) - xmlElementFromFile - Loading XML file from Data folder failed: XMLs/anime-list-master.xml
 
(logkit:13) - xmlElementFromFile - Loading XML file from url failed: https://rawgithub.com/ScudLee/anime-lists/master/anime-movieset-list.xml
2014-05-26 01:22:36,726 (1418) :  DEBUG (logkit:13) - xmlElementFromFile - Loading XML file from Data folder failed: XMLs/anime-movieset-list.xml
 
Well the agent cannot see https://rawgithub.com/ScudLee/anime-lists/master/anime-movieset-list.xml and cannot load it from data folder neither
i tried the URL and it works for me. try loading it and saving it in data folder > XMLs > anime-movieset-list.xml. possibly you have a firewall or no internet on the given machine
 
Have updated the source on git with:
   . download on http instead of https to reduce issues
   . It also can look in data folder and resource folder if you cannot download the base xml files
   . logging improvement
   . include "clear-cache" search title to clear the Plex cache
I have attached a recent version that is supposed to work without the data folders created, although i recommend them
Please let me know on this thread if anything goes wrong with that one
I will probably be working on the xml cache in data folder
115

Hey :)

Will attempt the latest you just posted, but since getting round the https http issue last night its been busy downloading metadata all day.  Gundam evolve is the one in question at the moment, there are others but attached the log just for it.

It's definitely on the internet its my "play with/attempt everything" machine so probably something done to it at some point.

Thanks

Note - attached bundle has error

File "Z:\Plex Server\Application Data\Plex Media Server\Plug-ins\Hama.bundle\Contents\Code\__init__.py", line 320
    except: raise ValueEr  ror

except: raise ValueEr  ror looks like you added some spaces to that line, removed on mine and confirmed it's launching again once spaces removed.
 

116

@ shikaiwraith: thanks for the feedback, uploaded the fixed version but 17 people downloaded it already. It also proved people download on first page at 40-50% proportion so will continue to amend the first page with stable release

new version use http only to download XMLs, fallback to latest downloaded XML if folder structure was done, and include a local copy just in case in ressources, in case people don't create folder structure in Data folder

Please let me know if it get banned during scraping after how many series it happens (important, but i did everything to prevent that), and if anything else happens.

This should be a stable release so please any of you feedback issues. updated agent in first page since many people used that version instead of latest, or didn't know how to update from git.

[Edit] I need to solve that, i have seen it pop up few times, but the line pasted is wrong, the error seem to be when loading the XML as the title searched should have output something before the error we see pointed out...

I posted edits on github to output a bit more after the file loading. could you update and paste the log again as you just did ?

Did you try to actually remove open subtitles from agent order by unticking it?: (just to make sure the issue is with HAMA)

Searching for matches for {'openSubtitlesHash': '7eb6de792040329c', 'episode': '10', 'episodic': '1', 'show': 'Gundam Evolve', 'season': '1', 'filename': 'G%3A%5CAnime%5CGundam%20Evolve%5CGundam%20Evolve%20-%20Omega%20(Episodes%206-10)%20[Solar][09D761B0]%2Emkv', 'plexHash': '2cb018f8afedab4f255c1ba84a7138af21cb9cdf', 'duration': '2693694', 'id': '23256'}

 
 
117

:) So progress is made.

Think the ban was totally my fault, where I had tried refreshing/wiping and starting again and pulling too many requests in single day, as when using my mates pc on different IP he never got banned.

So tried 3 series that failed before

Gundam Evolve - metadata pulled

Groove Rave Adventure - got genres and failed to do anything more

ikkitousen: xtreme xecutor - metadata pulled

Got quite a few pictures corrupted now as well, i.e grey bars etc but can't actually refresh poster to redownload new ones so manually redoing it.  They also if get series with follow on series they all get the same poster i.e. ikkitousen, initial d, votoms, macross etc but the I guess proper picture for the series or at least the unique picture tends to be the last one on the poster list.

New errors in the logs about files not being there and I don't use open subtitles as all, going to try a refresh of library and see what else sticks out.

Update - so think half issues are plex, refresh helped some items, others fix mismatch, doing a manual search and it pulled data but several series that still want to play ball;

Groove Rave Adventure

Kaikan Phrase

Kujakuou

Shin Kujakuou

I have tried with caching ticked in options and without.

118

THANKS! i could spot 3 things:

TypeError: not enough arguments for format string

========================================

   . Serie: Groove Adventure Rave:

   . TVDB XML empty http://thetvdb.com/api/A27AD9BE0DA63333/series/252688/banners.xml

   . TVDB serie page: http://thetvdb.com/?tab=series&id=252688&lid=7

   . serie has no posters... does this cause the issue?

not sure if issue related though but too much of a coincidence

IOError: [Errno 2] No such file or directory: u'\\\\?\\Z:\\Plex Server\\Application Data\\Plex Media Server\\Plug-in Support\\Data\\com.plexapp.agents.hama\\DataItems\\TVDB\\seasonswide\\._80158-4.jpg'

=======================================================================

In GetImagesFromTheTVDB: Data.Save(filename, poster), if folder not there, trouble

need to put a setting for local cache and set off by default

(content type 'image/jpeg' not cacheable in Agent plug-ins)

===============================================

Need to not cache again pictures in plex

119

Agreed, my wrong assumption if had AniDB ticked it would pull posters there and tvdb as well rather than one.  Bit weird they have pictures and things for the spanish version not english looking closer at it.

Refresh again didn't work once I unticked all thetvdb stuff, had to do fix incorrect match, manual search and click that way to pull even though it was the same link as automatic and it eventually pulled each episode just from anidb.

That has now fixed the remaining 4 with log attached, now to go through and fix the corrupted posters manually relinking them.

Cheers

120

@shikaiwraith: thanks for the logs, i can see it working ok now apart from one crash on "Groove Adventure Rave" only. 

When searching for "Groove Adventure Rave" on TheTVDB.com there is two hits:
. "Groove Adventure Rave", a Spanish title with tvdbid 72973 (and english title "Rave Master") with english posters
. "Groove Adventure Rave", an english title with tvdbid 252688 and no posters
 
I submitted a request on TVDB forum to have the duplicate issue reviewed: http://forums.thetvdb.com/viewtopic.php?f=7&t=18365

I have raised an issue on Github to have the mapping file amended to point to the series with posters: https://github.com/ScudLee/anime-lists/issues/37

From what you are saying, update didn't work. If i understand it right,update pull info if no info was there before, since it crashed previously, you need to do a full metadata pull, which you did doing the manual search. That means if you had this version you have now, it wouldn't have crashed in the first place.

I can only see an issue that popped up: TypeError: not enough arguments for format string

  File "Z:\Plex Server\Application Data\Plex Media Server\Plug-ins\Hama.bundle\Contents\Code\__init__.py", line 602, in parseAniDBXml
    elif "s"+defaulttvdbseason+"e"+str(epNumVal) in tvdbSummary:    tvdb_ep = "s"+defaulttvdbseason+"e"+str(epNumVal)
 
it seem like the very same serie works afterwards so i am puzzled
 
The first difference is: (cache/not cached) when the same serie was scrapped with no issue just afterwards
440: try: tvdbanime = self.urlLoadXml( TVDB_HTTP_API_URL % (TVDB_API_KEY, tvdbid), TVDB_SERIE_CACHE+"/"+tvdbid+".xml" ).xpath('/Data')[0
I have added logs in the urlLoadXml function to narrow down that bug i cannot narrow down more currently