[Rel] HTTP Anidb Metadata Agent (HAMA)

Also having the issue above. The scanner works great but metadata won’t download through HAMA.

@ZeroQI
It downloaded about 10 or shows worth of metadata and nothing else after that.

@Munchkin_ i found the reason… need to check if a recent update provoqued that though…

Fetching 'http://api.anidb.net:9001/httpapi?request=anime&client=hama&clientver=1&protover=1&aid=1695' from the HTTP cache
ERROR (__init__/xmlElementFromFile:944) - Not an XML file, AniDB banned possibly, result: '<error>Banned</error>'

@ZeroQI Except for him possibly being banned, what would be the reason for the latest version not downloading any metadata?

@chrno checking your logs, but there is so little logs from hama, it looks as if it can’t find the tvdbid due to mapping file, which i just changed code for… I can only see it see the serie has anidbid and that’s it

A line code “if metadata.seasons[media_season].episodes[media_episode].title == tvdb_table [ep] [‘EpisodeName’]: continue” was added to skip episode update if meta was already there but commend that line py prepending a hash sign “#” before qnd let me know if this fixes

Can you see if https://github.com/ZeroQI/Hama.bundle/commit/ab92eb755b9a453e93d5462dc458721d41178a30 does not have the issue? if it doesn’t then i know where to look

Will reproduce this evening if i don’t hear back from you…

@ZeroQI I am still at work for another 3 hours. I can check when I get home.

I created a test library with only 4 series in for my tests yesterday, might be why the log is so small.

@ZeroQI

Its downloading metadata again using the code from the link you provided, still feels very sluggish in how long it takes compared to the older version I tested yesterday though.

Side note…

  • its mapping Accel World’s OVAs in a very weird manner. Created a season 2012 and season 2013 for each OVA
  • Mapping A Certain Magical Index OVA 1 and 2 to the season’s episode 1 and 2

Never mind on the Accel World OVA, fixed it by moving it to its own folder and fixing naming

@chrno an sorry but file naming doesn’t follow anidb nor tvdb nor plex numbering. Single movies therefore s01e01 would include also oav and oad in the title. Can’t fix that i am aftraid.

@ZeroQI Yeah that’s not a problem, I fixed the file naming last night. I moved it straight from a downloads folder into the test library folder only fixing the series naming without paying attention to the extras

I was more concerned over the no metadata then the other stuff. Lost a drive and now needing to re-download what was lost on the drive, it will take some time to fix all the naming and folder structure to how it was before.

The filelist from the scanner should have a copy of serie and names if it helps
Will improve performance of the agent by not updating un-necessarily, but need to find the sweet spot
Just edit your post if you post 3 in a row in the same 5 minutes :wink:

Yeah, I use anidb file renamer to get everything sorted…just a tedious job when you have to go and work through almost 2tb of anime in one go again as the applet doesn’t always identify it or identify it correctly.

I have a wiki entry for Japanese Media Manager wit renaming scripts if it helps:

I will give that a go thanks a lot.

I’ll second Japanese Media Manager (JMM) as a great way to manage your anime. It scans files by hashing them, so it can take a bit of time, but you get nearly 100% match as long as the file has been hashed and entered into anidb. This is of course assuming you don’t alter the file (except renaming) in any way that would change the hash value and that would make it so it doesn’t match. Once it has matched, there is a bunch of metadata it can pull down and can use to put into your file names like resolution, codecs, CRC, etc.

Anyway, it is a good tool to keep your anidb account anime list up to date and also has pretty robust renaming script abilities. You can have it rename on import, when it scans and adds a file to your collection, or on demand for stuff you have already imported.

If interested, here’s my JMM rename script:

// Anime Name, use english name if it exists, otherwise use the Romaji name
IF I(eng) DO ADD '%eng '
IF I(ann);I(!eng) DO ADD '%ann '
// Episode Number (movies add # to assist with Plex/ASS identification)
IF T(!Movie) DO ADD '- Ep %enr '
IF T(Movie) DO ADD '%enr '
// If the file version is v2 or higher add it here
IF F(!1) DO ADD 'v%ver '
// If special, add the episode name
IF H(S) DO ADD '- %epr '
DO ADD '('
// If DVD or Blu-ray, add identifier
IF R(DVD) DO ADD 'DVD '
IF R(Blu-ray) DO ADD 'BD '
IF R(LD) DO ADD 'LD '
// Add resolution
DO ADD '%res'
// IF 10bit, add identifier (skip if also HEVC)
IF Z(10);C(!HEVC) DO ADD ' Hi10P'
// If HEVC, add identifier
IF C(HEVC) DO ADD ' HEVC'
// If dual language, add identifier
IF D(japanese);D(english) DO ADD ', JPN+ENG'
DO ADD ') '
// Before adding fansub group and CRC, clean up brackets
DO REPLACE '[' '- '
DO REPLACE ']' ''
// Add fansub group name - %grp is short name, %grl is long name
DO ADD '[%grl] '
// Add CRC
DO ADD '[%CRC]'

// Replacement rules (cleanup)
DO REPLACE '__' '_'
DO REPLACE '  ' ' '
// Fix special naming & numbering
DO REPLACE 'Ep S' 'Special '

// Generic cleanup of redundant labels in fansub group names
DO REPLACE ' Fansub]' ']'
DO REPLACE '-Fansub]' ']'
DO REPLACE ' Fansubs]' ']'
DO REPLACE '-Fansubs]' ']'
DO REPLACE ' fansub]' ']'
DO REPLACE '-fansub]' ']'
DO REPLACE ' fansubs]' ']'
DO REPLACE '-fansubs]' ']'
DO REPLACE ' Subs]' ']'
DO REPLACE '-Subs]' ']'
DO REPLACE ' subs]' ']'
DO REPLACE '-subs]' ']'
DO REPLACE ' Rips]' ']'
DO REPLACE '-Rips]' ']'
DO REPLACE ' Scans]' ']'
DO REPLACE '-Scans]' ']'

// Replace all illegal file name characters leftover
DO REPLACE '<' '('
DO REPLACE '>' ')'
DO REPLACE ':' ' -'
DO REPLACE '"' ''
DO REPLACE '/' '-'
DO REPLACE '\' ''
DO REPLACE '|' ''
DO REPLACE '?' '!'
// DO REPLACE '`' 'FIXME'
DO REPLACE '*' 'Bleep'

// Per fansub group special fixes
DO REPLACE '[Unlimited Translation Works]' '[UTW]'
DO REPLACE '[Cthuko-caffeine]' '[Cthune]'
DO REPLACE '[THORAnime]' '[THORA]'
DO REPLACE '[ShitimeSubs]' '[Shitime]'
DO REPLACE '[Irregular Hunter Productions]' '[IHP]'
DO REPLACE '[FFFpeeps]' '[FFF]'
DO REPLACE '[Chihiro-Senketsu]' '[Chihiro & Senketsu]'
DO REPLACE '[cornbreadman]' '[CBM]'
DO REPLACE '[GJM-FFF]' '[GJM & FFF]'
DO REPLACE '[μtw-scribbles]' '[mutw & scribbles]'
DO REPLACE '[Sailor Moon Center]' '[SMC]'
DO REPLACE '[Good Job! Media]' '[GJM]'
DO REPLACE '[Good Job! Media-Vivid]' '[Vivid & GJM]'
DO REPLACE '[naishosubs]' '[naisho]'
DO REPLACE '[OZC Anime]' '[OZC]'
DO REPLACE '[Sage]' '[sage]'
DO REPLACE '[Late Night Snack]' '[LNS]'
DO REPLACE '[DATS.US]' '[DATS]'
DO REPLACE '[AnimeSenshi]' '[Asenshi]'
DO REPLACE '[Asenshi-GJM]' '[Asenshi & GJM]'
DO REPLACE '[Gray-Phantom]' '[G_P]'
DO REPLACE '[Pikanet128]' '[Pn8]'
DO REPLACE '[Utsukushii Shujin Shujinowski]' '[USS]'
DO REPLACE '[WhyNot!]' '[WhyNot]'
DO REPLACE '[Good Job! Media-DameDesuYo]' '[GJM & DameDesuYo]'
DO REPLACE '[Hi10 Anime]' '[Hi10]'

// Per anime special fixes
DO REPLACE 'Bishoujo Senshi Sailor Moon Crystal Season III' 'Sailor Moon Crystal Season 3'
DO REPLACE 'Working!!!' 'Working!! 3'
DO REPLACE 'Mobile Suit Gundam Unicorn Re -0096' 'Mobile Suit Gundam Unicorn RE 0096'
DO REPLACE 'Idolmaster - Cinderella Girls (2015)' 'Idolmaster - Cinderella Girls Second Series'
DO REPLACE 'Glitter Force' 'Smile Precure!'
DO REPLACE 'Mahou Shoujo Nante Mouiidesukara' 'Mahou Shoujo Nante Mou Ii Desukara'
DO REPLACE 'Assassination Classroom - Second Season' 'Assassination Classroom (2016)'
DO REPLACE 'Re -Zero - Starting' 'Re-Zero - Starting'
// DO REPLACE 'High School Fleet' 'Haifuri'
DO REPLACE 'Big Order' 'Big Order (2016)'
DO REPLACE 'Kuromukuro' 'Black Relic'

Some of the stuff at the bottom is just me being fiddly with naming fixes, but I’ll leave them in there as examples.

@deusxqnime Updated JMM media page there https://github.com/ZeroQI/Absolute-Series-Scanner/wiki/Japanese-Media-Manager

yeah no meta downloaded on latest version if no custom mapping present… anidbTvdbMapping() code at fault, infinite loop:
while dir:
dir = os.path.dirname(dir)
Log.Info(“dir2: ‘%s’” % dir )
dir = “/” therefore loops

Edit: Would have caused major slowdowns when meta already there. no meta downloaded unless custom mapping file present

https://github.com/ZeroQI/Hama.bundle/blob/master/Contents/Code/init.py

I’m not sure if this is related to the issue above but this is happening a lot in the com.plexapp.system.log. Plex seems to do a lot of scanning and metadata stuff, it is constantly using a full core on my server.

2016-12-23 17:44:37,843 (7f5fab7fe700) :  ERROR (networking:219) - Error opening URL 'http://127.0.0.1:32400/:/plugins/com.plexapp.agents.hama/messaging/function/X0FnZW50S2l0OlVwZGF0ZU1ldGFkYXRh/Y2VyZWFsMQoxCmxpc3QKMApyMAo_/Y2VyZWFsMQoxCmRpY3QKMTAKczIKZW5zNApsYW5nYjFzNQpmb3JjZWIwczgKcGVyaW9kaWNzNAo2Mzg5czQKZGJpZGkwCnM3CnZlcnNpb25uczEwCnBhcmVudEdVSURuczgKcGFyZW50SURzNwpUVl9TaG93czEwCm1lZGlhX3R5cGVzNDUKY29tLnBsZXhhcHAuYWdlbnRzLmhhbWE6Ly9hbmlkYi0xMTkyMj9sYW5nPWVuczQKZ3VpZHMxMQphbmlkYi0xMTkyMnMyCmlkcjAK'
2016-12-23 17:44:37,843 (7f5fab7fe700) :  CRITICAL (agentservice:565) - Exception in update for com.plexapp.agents.hama://anidb-11922?lang=en (most recent call last):
  File "/usr/lib/plexmediaserver/Resources/Plug-ins-b38628e/System.bundle/Contents/Code/agentservice.py", line 407, in update_task
    self.agent_update_metadata(identifier, media_type, guid, id, lang, dbid, parentGUID, force, version, parentID, periodic)
  File "/usr/lib/plexmediaserver/Resources/Plug-ins-b38628e/System.bundle/Contents/Code/agentservice.py", line 1236, in agent_update_metadata
    kwargs=kwargs
  File "/usr/lib/plexmediaserver/Resources/Plug-ins-b38628e/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/components/messaging.py", line 86, in call_external_function
    packed_result = self._core.networking.http_request(url, cacheTime=0, timeout=None, immediate=True).content
  File "/usr/lib/plexmediaserver/Resources/Plug-ins-b38628e/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/components/networking.py", line 370, in http_request
    return HTTPRequest(self._core, url, data, h, url_cache, encoding, errors, timeout, immediate, sleep, opener, follow_redirects, method)
  File "/usr/lib/plexmediaserver/Resources/Plug-ins-b38628e/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/components/networking.py", line 141, in __init__
    self.load()
  File "/usr/lib/plexmediaserver/Resources/Plug-ins-b38628e/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/components/networking.py", line 181, in load
    f = self._opener.open(req, timeout=self._timeout)
  File "/usr/lib/plexmediaserver/Resources/Plug-ins-b38628e/Framework.bundle/Contents/Resources/Platforms/Shared/Libraries/urllib2_new.py", line 444, in open
    response = meth(req, response)
  File "/usr/lib/plexmediaserver/Resources/Plug-ins-b38628e/Framework.bundle/Contents/Resources/Platforms/Shared/Libraries/urllib2_new.py", line 557, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/plexmediaserver/Resources/Plug-ins-b38628e/Framework.bundle/Contents/Resources/Platforms/Shared/Libraries/urllib2_new.py", line 482, in error
    return self._call_chain(*args)
  File "/usr/lib/plexmediaserver/Resources/Plug-ins-b38628e/Framework.bundle/Contents/Resources/Platforms/Shared/Libraries/urllib2_new.py", line 416, in _call_chain
    result = func(*args)
  File "/usr/lib/plexmediaserver/Resources/Plug-ins-b38628e/Framework.bundle/Contents/Resources/Platforms/Shared/Libraries/urllib2_new.py", line 565, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 500: Internal Server Error

@F0rcen : Please update to latest code from minutes ago there was an infinite loop in the code, and i can see you were affected indeed.
Scanner code seem ancient…

@ZeroQI Great, thanks for the fix!

It should be the newest scanner? I update it with wget pretty often.
EDIT: Wait, one of the logfiles was from august… Not sure what happened.
The new correct log should be there at least.

Which of the update scripts for Ubuntu in the wiki is the most recent or the one we should use? I still used my own one but you added some extra chown stuff.