[Rel] HTTP Anidb Metadata Agent (HAMA)

scanner-agent-dev

#2

If you're going to use git and Github, just work on a master branch, and use tags and/or branches for the stable releases. Then other people can help out with development :)


#3

you probably forgot to upload the resources folder of the bundle to github.

2013-08-17 19:45:08,251 (-4e50b490) :  INFO (__init__:138) - SearchByName (MediaContainer(art=None, noHistory=False, title1=None, title2=None, replaceParent=False),en,Amagami Ss,None)

2013-08-17 19:45:08,273 (-4e50b490) :  INFO (__init__:794) - xmlElementFromFile (anime-titles.xml, anime-titles.xml.gz) None
2013-08-17 19:45:08,274 (-4e50b490) :  DEBUG (__init__:798) - xmlElementFromFile - Loading XML file from Resources folder failed:anime-titles.xml
2013-08-17 19:45:08,275 (-4e50b490) :  DEBUG (networking:172) - Requesting 'anime-titles.xml.gz'
2013-08-17 19:45:08,276 (-4e50b490) :  INFO (__init__:802) - xmlElementFromFile - Loading XML from url failed: anime-titles.xml.gz
2013-08-17 19:45:08,277 (-4e50b490) :  CRITICAL (agentkit:930) - Exception in the search function of agent named 'HamaTV', called with keyword arguments {'openSubtitlesHash': '55706e36c4ebc303', 'episode': '11', 'episodic': '1', 'show': 'Amagami Ss', 'season': '2', 'filename': '%2Fmnt%2Fuser%2FMedia%2FAnimeTest%2FAmagami%20SS%2FSeason%2002%2FAmagami%20SS%20Plus%20-%2011v2%20-%20Morishima%20Haruka%20-%20Part%201%20Sexy%20[UTW][1280x720][HDTV][8CB212D1]%2Emkv', 'plexHash': '24bca57b7f40dbb782f365c9b571c78868e4ac08', 'duration': '-1', 'id': '31243'} (most recent call last):
  File "/mnt/cache/usenet/apps/PlexLibrary/Application Support/Plex Media Server/Plug-ins/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/api/agentkit.py", line 921, in _search
    agent.search(results, media, lang, manual)
  File "/mnt/cache/usenet/apps/PlexLibrary/Application Support/Plex Media Server/Plug-ins/Hama.bundle/Contents/Code/__init__.py", line 855, in search
    self.searchByName(results, lang, media.show, media.year)
  File "/mnt/cache/usenet/apps/PlexLibrary/Application Support/Plex Media Server/Plug-ins/Hama.bundle/Contents/Code/__init__.py", line 200, in searchByName
    matchedWords[word]= [sample]
AttributeError: 'NoneType' object has no attribute 'iterdescendants'

the agent can't find anime-titles.xml.gz, which should be in the resources folder.

Update:

had another look at your code.

one error is, that you have two lines defining ANIDB_ANIME_TITLES_URL. you need to remove the second one...

but even then, the agent can't load from the correct url (in the browser it works). possibly python can't open an gz file as xml.

so you'd need to first download it, extract and then open it locally

Update2:

well, there seems to be more in error with opening remote xml files... after downloading anime-titles.xml manually, the next xml file gives the same error

2013-08-17 20:12:25,966 (-4e4ce490) :  DEBUG (__init__:798) - xmlElementFromFile - Loading XML file from Resources folder failed:anime-movieset-list.xml

2013-08-17 20:12:25,980 (-4e4ce490) :  DEBUG (networking:172) - Requesting 'https://github.com/ScudLee/anime-lists/raw/master/anime-movieset-list.xml'
2013-08-17 20:12:27,820 (-4e4ce490) :  DEBUG (networking:200) - Not caching 'https://github.com/ScudLee/anime-lists/raw/master/anime-movieset-list.xml' (content type 'text/plain; charset=utf-8' not cacheable in Agent plug-ins)
2013-08-17 20:12:27,829 (-4e4ce490) :  INFO (__init__:802) - xmlElementFromFile - Loading XML from url failed: https://github.com/ScudLee/anime-lists/raw/master/anime-movieset-list.xml

#4

Hi, thanks for the reply.

The resources folder needs:

   . anime-titles.xml, the AniDB title database, part of the AniDB HTTP API : http://anidb.net/api/anime-titles.xml.gz decompressed (xml extension)

   . anime-list-master.xml which is ScudLee's AniDB to TVDB xml mapping file, give studio and episode mapping list for te episode overview https://raw.github.com/ScudLee/anime-lists/master/anime-list-master.xml

   . anime-movieset-list.xml for the movie collection (Because XBMC only support movie collection and the files were for an XBMC plugin originaly) https://raw.github.com/ScudLee/anime-lists/master/anime-movieset-list.xml

 

I am still coding, and i did put the two xml mapping files and the AniDB title database in the resources Folder.

I did modify the source code to allow to pull from the web (using cache) the missing files. uploaded it just now


#5

To try:

   . will try to change the metadata object type to see if a movie scanned by the TV series plugin can have Movie only attributes updated

To those testing, please give me a feedback, performance wise, and bugs or series/animes not working. once loaded, this scrapper is meant to be pretty fast... Was thinking of using collections to put the studio name or other data tags... Any preference ?

New feature:

   . support for theme musics from plex TVDB theme support.

     You can add theme song here for thetvdb: http://wiki.plexapp.com/index.php/PlexNine_PMS_ThemeMusic

     Source code for PlexThemeMusic: [https://github.com/plexinc-agents/PlexThemeMusic.bundle/blob/master/Contents/Code/__init__.py]

anidbid =12345   ### id from TheTVDB.com
THEME_URL = 'http://tvthemes.plexapp.com/%s.mp3'
if THEME_URL % anidbid not in metadata.themes: ### can add latter a file search in serie or specific folder (if you don't want files in each folder)
  metadata.themes[THEME_URL % anidbid] = Proxy.Media(HTTP.Request(THEME_URL % anidbid))

If popular, we could have the following lookups:

. local folder

. specific folder with all theme songs, using main title or AnimeID for naming

. web server with files names with "AniDB id.mp3" pattern for movies an series grouped in TheTVDB.com. Length should be 30s max to avoid legal issues.. might cause bandwith issues if the plugin went to be massivelly adopted.

I am surprised there is no torrent packages of the sort yet... I guess there need to be a working implementation to start that, i hope i start a trend... if you have such data for test purposes (30s or full music, will trim myself) please contact me. i need data for tests and will develop the function, will help for exemple blind audio quiz for anime fans...


#6

Uploaded a new version, this one got no errors when scanning my anime series in progress folder.

It exports in 3 separate logs the alerts so you can update the databases to fix the issues (no poster, no description, tvdbid not in mapping xml file, etc...)

Please let me know of any bugs, and the performance..., pasting Hama logs with the error and last debug line if an error occured

[Edit: 2013-08-25]

Studio logos from http://forum.xbmc.org/showthread.php?tid=70398 [https://sites.google.com/site/anidblist/AnimeStudios.zip] seem to work but are tiny. after playing, it seems:

  . no logos name in folder include dots, dash, exclamation point but can contain + and some contain spaces

 . if a logo\\NAS\Plex\Library\Application Support\Plex Media Server\Plug-ins\Media-Flags.bundle\Contents\Resources\Studio

is detected, if you replace it in PMS, it won't refresh even after clearing the browser cache...

 

After some research, to update Plex, the Media-Flags.bundle is here: https://github.com/plexinc-plugins/Media-Flags.bundle/tree/master/Contents/Resources Source: http://forums.plexapp.com/index.php/topic/65514-studio-logos/

 

The substitutions.xml XML file [Plex\Library\Application Support\Plex Media Server\Plug-ins\Media-Flags.bundle\Contents\Resources\substitutions.xml] format is below, and can help mapping if the logo doesn't show:

   
        
        
        
                [...]
        [...]
   
   
       
        [...]
   
   
       
        [...]
   




#7

hello, i cant download the attachment?

i just need an agent that can recognise anime movies (all my anime tv show are done nicely), but not my movies.

example, i have Bleach - Memories of Nobody (2006).mkv

i tried anidb meta agent, but no success.


#8

New version with updated xml files attached to original post

Please let me know the issues you encounter.

There is separate html logs (contain links to update databases) in \\NAS\Plex\Library\Application Support\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama\DataItems for:

   . tv themes

   . anime-lists

   . TVDB

   . AniDB


#9

Hi guys, There have been few downloads, but no reply to the thread, so i do not know how well this works with other sets of series, or how it compares to the udp AniDB agent, so please, let me know how good it is...

I created separate logs that only keep unique information for feedback to thetvdb or anidb, so you can batch update...

The TV theme is using plex server that is based on thetvdb id...

You can request features, as i don't know what to do now...  Might consider automating the database and file download if enough people are using the agent, which i am not sure of...

So if you have any questions, or a serie has one of the titles in AniDB but isn't recognised, please let me know

You might need to update the database title and the mapping file manually for now... 


#10

don't know if its the scanner or the metadata, but I would like to the  HAMA to be able to separate the specials from the series if possible, like The Episode list on Anidb 

Otherwise I think it works very well, have some problems with my OAD files, but can be the scanner to 

It won't recognice Chaos Head for me, have tried to copy the name directly from anidb, but still don't work

 

#11

AniDB indicates the main title is "Chaos;Head" and none of the spellings, short title, synonims include "Chaos;Head".

the semi-colon not being forbidden by file systems, if you rename the anime, it should be fine...

Anidb title list: http://anidb.net/perl-bin/animedb.pl?show=anime&aid=6088

 

Do you use BABS scanner? The url is http://forums.plexapp.com/index.php/topic/31081-better-absolute-scanner-babs/

OAD file extention ?


#12

I use BABS to scan my anime with

OVA and OAD is the same, just diffrent names http://en.wikipedia.org/wiki/Original_video_animation


#13

Any instructions on installing this (Windows Server 2008 - Plex 0.9.86) would be greatly appreciated.


#14

Put it at the location below with the other.bundle files.

C:\Users\XXXXX\AppData\Local\Plex Media Server\Plug-ins


#15

I grabbed a zip of the master on git and also the latest xml from ScudLee.

Ran it through a few anime titles... one of which had >100 episodes using BABS and Hama

Parsed fine and loaded up all the metadata

Going to run it through part of my collection and see if any issues occur

Format - Worked no problems

   English Title/English Title - EP.container

Alt Format - Also worked without issues but ignored the OVAs... I'm assuming this is intended since they don't exist in the xmls

   English Title/Series/English Title - EP.container

   English Title/OVA/English Title - VolumeNo.container


#16

Thanks for your post Melator, i have very little feedback on my code, and previous buggy versions i release possibly didn't help its widespread use...

I am using a HP microserver N40L on which i installed DSM (the OS of synology NAS) that i downloaded on XPEnology.com (++ version has more drivers supported)

That allow to use alternate package servers to install 3rd party apps (sickbeard) but Plex is supported by default.

 

I have been moving houses and since i am a computer parts hoarder, and also have a day job, that is taking a lot of my time and i didn't do new development on the agent but on the other hand, i don't know what to improve as it pretty much work well for me, the studio icons are supported, multi language support is there, it logs the issue the user has a possibility to correct, etc...

The agent takes the posters from the tvdb not from anidb due to a fixed resolution hence fixed ratio. That also limited the anidb ban since we do one request per serie.

For windows the agent goes here: C:\Users\XXXXX\AppData\Local\Plex Media Server\Plug-ins

I am unsure if many of you use the clever html log generated (one instance of every warning present in them) to be able to update used databases easily as i wanted this agent to allow to benefit not only Plex users but all database users as it allow easy database update by including direct links when applicable

 Logs are located in DataItems since it's the only folder with write access available to the agents:

   . For me it's in \\NAS\Plex\Library\Application Support\Plex Media Server\Plug-in Support\Data\com.plexapp.agents.hama

   . For windows it should be in C:\Users\XXXXX\AppData\Local\Plex Media Server\Plug-ins\Data\com.plexapp.agents.hama

 
Here is the feedback logs description and one example of every feedback:

   . AniDB.htm        

        - Aid: 00002 No poster present

   . Anime-list.htm    ScudLee;s XML file feedback

       - Aid: 00002 '3x3 Eyes' AniDB and anime-list are both missing the studio

       - Aid: 00002 '3x3 Eyes' AniDB have studio 'xxx' and XML have 'yyy'

       - Aid: 00002 '3x3 Eyes' has no matching tvdbid ('OAV') in mapping file

       - Aid: 00002 anime-list is missing the anidbid

 

   . Themes.htm       Plex TV theme support whose filename is based on TheTVDB.com id and last 30s max. Wouldn't mind somebody a package will all complete mp3 theme songs whose name would be the AniDB ID (to account for the seasons different songs)...

        - Aid: 00002 '3x3 Eyes' tvdbid: 70973 '3x3 Eyes' Missing theme song 3x3 Eyes' No English poster

        - aid: 00002 tvdbid: 70973 '3x3 Eyes' Overview Empty

        - aid: 00002 tvdbid:70973 s1e1 Overview Empty


 


#17

On my debian box it's located here: /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-in Support/Data/com.plexapp.agents.hama/DataItems

I only see anime-list.htm and TVDB.htm in there though

I gave the Hama Movie scanner a whirl but it didn't pick up very many of my films.

I then updated from git today and re-ran the scan but nothing changed.. except when I hit refresh on the movie itself the metadata updated.

I've had other agents behave this way as well so I'm guessing something outside the agent or maybe plain old user error

Aside from the snag above everything worked out well.. Good job :)


#18

Thanks for giving the path on your box and for the nice feedback :D

It only create the logs if needed. i wanted to provide a way to feedback info easily to update databases so the users in OCD update the databases;)

Would you mind posting some of the movie titles and the filename associated so i try to reproduce?

Also, if you have (Implementable) feature requests... 


#19

I retested it and after a few manual name fixes, poorly named on my part, it picked up just about everything.

The only film it still can't find is Rao Den Jun'ai no Sho and for some films like Patlabor 1/2, Inuyasha, Tenchi Muyou and the two Evangelion films it's using the same cover art even though I have the AniDB > TVDB set in the config.

Raoh Den Junai no Sho - Director's Cut.mkv  (I've tried manual searches and did find it on TVDB through my browser but can't for the life of me get it to pick it up)

The only thing changed since the previous attempt has been a few PMS builds... one which had the new scanner so maybe that made the difference.

Can't think of anything more to add... you're pulling in all the data I typically look for in a series/movie.


#20

The TVDB is a database for TV series, not movies. It therefore cannot gather info that TheTVDB will not have, so no poster from thetvdb.

They can be included if anidb sets them as specials,and must be labeled as such.

For "Rao Den Jun'ai no Sho"  Anidb names it as S01e01 not a special from a serie.

did you try to label it as "Raoh Den Junai no Sho s01e01 [director's cut].mkv"

I do recommend BABS scanner...


#21

TVDB has it listed as a special which is why I thought it would get picked up

http://thetvdb.com/?tab=episode&seriesid=79156&seasonid=26601&id=2145591&lid=7

I'm using BABS as my scanner

I'm sure if I rename it to the english name it'll get picked up