MyAnimeList.net Metadata Agent

Dev Community Dev/API Corner
41

Hey Folks I'v got bad news. kind of.

Since MyAnimeList.net implemented whitelisting on their site to prevent DDOS attacks the unofficial API isn't working. As it says in the commits in their git repo the whitelisting should be implemented but I don't know why there are only blank responses.

So I've got a few options to choose:

  1. get back to the Official API and struggle with their response
  2. develope my own API which does the same as the Unofficial

first should be less time consuming, still the 1.0 version of the MALAgent uses this method and there are over 50% missmatches and this is really annoying. The second one would be really time consuming because I need to implement the services, setup server and other stuff I'm not even aware of but would be an alternative to the Official and Unofficial API. (and a better training for myself)

So actually there is only one thing you can do. Download the 1.0 version of the MALAgent and have many missmatches, but something that is working. Or you wait some more and the API comes back to live OR I will create my own API.

....

42

Quick Update:

I started building my own MAL-API based on Web services in Java which was easier to get together than I first thought. Still, I ran into the whitelisting problem, and the form to get my user-agent approved isn't accessable anymore which is quite sad because I'm really hyped to get this project running. I hope somebody actually send me a reply to get my own user-agent so that I can keep programming. Stay tuned.

43

Quick Update #2:

Finally I got my MAL-Whitelisting. Development ist on the way, searching for Anime works and I'm now working on parsing the anime page. Everything I wrote so far is experimental so it needs a lot of refactoring until it will be a lot of work. Still its progressing smoothly without any bigger problems so far.

When the backend is ready I will update the agent.

Also I will need to find a provider to host my backend which will require a tomcat server. I hope I find a reliable provider.

Thats all for now...

44

Quick Update #2:

Finally I got my MAL-Whitelisting. Development ist on the way, searching for Anime works and I'm now working on parsing the anime page. Everything I wrote so far is experimental so it needs a lot of refactoring until it will be a lot of work. Still its progressing smoothly without any bigger problems so far.

When the backend is ready I will update the agent.

Also I will need to find a provider to host my backend which will require a tomcat server. I hope I find a reliable provider.

Thats all for now...

Thanks for working on this! Really hope it comes to fruition.

Have you looked at Hummingbird? They recently introduced their API.

45

Thanks for working on this! Really hope it comes to fruition.

Have you looked at Hummingbird? They recently introduced their API.

nope didn't know about that yet. Still, need to finish the MAL Api and Agent and then I can think of other projects or in the words of a Peon "Work, Work" xD

46

Quick Update #3:

So after a few weeks of developing and finally finishing everything ... I can start over god damnit. well I knew it was all experimental but this was unexpected

Now I need to do the stuff with regular expressions to get specific results to add to the response, but I don't know anything about regular expressions at all so it will take some more time

Stay Tuned

Edit:

Okay found out that you shouldn't use regex in HTML but I use an XML parser now and I'm almost finished just need to change some things

47

Okay Guys, I got really far today and it's almost ready just need a few more improvements: the search scraping isn't as good as I thought it would be so I need to improve it and adjust it to the detail page scraping which is almost perfect. Still, some functionality isn't ready yet like the display of the release date or the episodes because this needs quite some work on the API itself to parse and modify the given strings to the final output.

And for all of you who wanted a separated Movie section for your Anime Movies, I implemented the Movie Section and with the first try it found 64 of 66 Movies I had in my DB (after a few adjustments and bugfixing) the last 2 weren't found because of an error on the search algorithm, like I said already it still needs some work.

Last but not least all the TV-Shows will make a request to theTVDB to get the Fanarts. All available Fanarts will be downloaded and used as background (only the backgrounds which are in the DB, because I can't access something which isn't existing)

48

First version of the API is done, Agent is adjusted to the changes still need to get it online but this is already on the way

49

Keep up the good work. I look forward to this agent, my Fist of the North Star collection is all messed up!

50

How can I do this on a Mac? I cannot find where to put the file on my Mac.

51

unfortunately this version isn't up-to-date yet because of the unofficial API. But rest assure I will release the newer version of the MALAgent this weekend. Its not perfect but its working quite nice (still didnt tested it with my whole database but I do this tomorrow)

To your Question: I have no clue how the folder structure works on a Mac, but If I need to install a Agent or Plugin I click with the right mouse button on the Plex media Server icon and select "open plugin folder" and if you got something like that too copy the agent into this folder.

Oh and this is what I found with a quick google search https://plexapp.zendesk.com/hc/en-us/articles/201106098-How-do-I-find-the-Plug-Ins-folder-

Edit:

Sorry to let you wait this long but you need to be patient a little bit more because I found a bug which prevents me from receiving metadata (request and response are working though) but the metadata isn't added. And another Bug is that the fetching of the URL for detailed information doesn't work in a special case. So before I release it I want to fix these bugs first

New version is up and a new thread is born here

52

Doesn't work very well.

Example:

Directory: Vanquished Queens. No match

Custom search: "Vanquished Queen": Displays 2 matches.

Plex Entry: Watashi Ga Motenai No Wa Dou Kangaetemo Omaera Ga Warui!. No match

Custom search: "Watashi ga Motenai No Wa Dou Kangaetemo Omaera Ga Warui!": Displays 2 matches.

Plex Entry: Usagi Drop. No match

Custom search: "Usagi Dro": Displays 2 matches.

For some reason I have to edit 1 letter for the match to work.

53

Doesn't work very well.

Example:

Directory: Vanquished Queens. No match

Custom search: "Vanquished Queen": Displays 2 matches.

Plex Entry: Watashi Ga Motenai No Wa Dou Kangaetemo Omaera Ga Warui!. No match

Custom search: "Watashi ga Motenai No Wa Dou Kangaetemo Omaera Ga Warui!": Displays 2 matches.

Plex Entry: Usagi Drop. No match

Custom search: "Usagi Dro": Displays 2 matches.

For some reason I have to edit 1 letter for the match to work.

please add the log file so that I can see what is happening exactly

54

thanks for the update! but i see this error in log. Can you see what is going on?

 

2014-04-22 20:42:08,733 (1688) :  DEBUG (networking:172) - Requesting 'http://daraku-mal-ap...e Darkness Oad'
2014-04-22 20:42:10,460 (1688) :  ERROR (networking:223) - Error opening URL 'http://daraku-mal-ap...e Darkness Oad'
2014-04-22 20:42:10,463 (1688) :  CRITICAL (core:561) - Exception in the search function of agent named 'MyAnimeList.net', called with keyword arguments {'show': 'To Love Ru Trouble Darkness Oad', 'id': '4492', 'year': '2012'} (most recent call last):
  File "C:\Users\userA\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\api\agentkit.py", line 924, in _search
    agent.search(results, media, lang)
  File "C:\Users\userA\AppData\Local\Plex Media Server\Plug-ins\MAL.bundle\Contents\Code\__init__.py", line 223, in search
    doSearchShow(results, media, lang)
  File "C:\Users\userA\AppData\Local\Plex Media Server\Plug-ins\MAL.bundle\Contents\Code\__init__.py", line 27, in doSearchShow
    json = JSON.ObjectFromURL(mal_url, timeout=100000)
  File "C:\Users\userA\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\api\parsekit.py", line 145, in ObjectFromURL
    method=method,
  File "C:\Users\userA\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\api etworkkit.py", line 67, in _http_request
    req = self._core.networking.http_request(url, *args, **kwargs)
  File "C:\Users\userA\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\components etworking.py", line 426, in http_request
    return HTTPRequest(self._core, url, data, h, url_cache, encoding, errors, timeout, immediate, sleep, opener, follow_redirects, method)
  File "C:\Users\userA\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\components etworking.py", line 147, in __init__
    self.load()
  File "C:\Users\userA\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\components etworking.py", line 187, in load
    f = self._opener.open(req, timeout=self._timeout)
  File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 410, in open
    response = meth(req, response)
  File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 500: Internal Server Error
 
2014-04-22 20:42:10,463 (1688) :  DEBUG (runtime:918) - Response: [200] str, 16 bytes
55

thanks for the update! but i see this error in log. Can you see what is going on?

 

2014-04-22 20:42:08,733 (1688) :  DEBUG (networking:172) - Requesting 'http://daraku-mal-ap...e Darkness Oad'
2014-04-22 20:42:10,460 (1688) :  ERROR (networking:223) - Error opening URL 'http://daraku-mal-ap...e Darkness Oad'
2014-04-22 20:42:10,463 (1688) :  CRITICAL (core:561) - Exception in the search function of agent named 'MyAnimeList.net', called with keyword arguments {'show': 'To Love Ru Trouble Darkness Oad', 'id': '4492', 'year': '2012'} (most recent call last):
  File "C:\Users\userA\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\api\agentkit.py", line 924, in _search
    agent.search(results, media, lang)
  File "C:\Users\userA\AppData\Local\Plex Media Server\Plug-ins\MAL.bundle\Contents\Code\__init__.py", line 223, in search
    doSearchShow(results, media, lang)
  File "C:\Users\userA\AppData\Local\Plex Media Server\Plug-ins\MAL.bundle\Contents\Code\__init__.py", line 27, in doSearchShow
    json = JSON.ObjectFromURL(mal_url, timeout=100000)
  File "C:\Users\userA\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\api\parsekit.py", line 145, in ObjectFromURL
    method=method,
  File "C:\Users\userA\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\api etworkkit.py", line 67, in _http_request
    req = self._core.networking.http_request(url, *args, **kwargs)
  File "C:\Users\userA\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\components etworking.py", line 426, in http_request
    return HTTPRequest(self._core, url, data, h, url_cache, encoding, errors, timeout, immediate, sleep, opener, follow_redirects, method)
  File "C:\Users\userA\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\components etworking.py", line 147, in __init__
    self.load()
  File "C:\Users\userA\AppData\Local\Plex Media Server\Plug-ins\Framework.bundle\Contents\Resources\Versions\2\Python\Framework\components etworking.py", line 187, in load
    f = self._opener.open(req, timeout=self._timeout)
  File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 410, in open
    response = meth(req, response)
  File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "C:\Program Files (x86)\Plex\Plex Media Server\python27.zip\urllib2.py", line 531, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 500: Internal Server Error
 
2014-04-22 20:42:10,463 (1688) :  DEBUG (runtime:918) - Response: [200] str, 16 bytes

The Name couldn't be found, please re-name the anime to any version which is used by myanimelist.net

56

Hmm not sure what I am doing wrong… new to plex. I can’t the mal.bundle agent or the hana agent to work at all. TV shows and movies are detecting fine. format is folder show name season name s/e numbers same as for tv shows. Fusion/xbmc seem to pick them up ok plex keep returning all blanks with either anime agent. I feel I must be missing something obvious with added agents… I set the right agent for the folder, stuff plays so the data is clearly working. Where should I look to debug? Scans seem to be logged but no meta data or art is fetched for anime.
I don’t see a way to add my Animelist.net credentials now, I’m gussing that went away with when you started to use your own API?
Mac mini osx mavericks running 0.9.9.7.429-f80a8d6
Thanks

57

Hmm not sure what I am doing wrong... new to plex. I can't the mal.bundle agent or the hana agent to work at all. TV shows and movies are detecting fine. format is folder show name season name s/e numbers same as for tv shows. Fusion/xbmc seem to pick them up ok plex keep returning all blanks with either anime agent. I feel I must be missing something obvious with added agents.... I set the right agent for the folder, stuff plays so the data is clearly working. Where should I look to debug? Scans seem to be logged but no meta data or art is fetched for anime.
I don't see a way to add my Animelist.net credentials now, I'm gussing that went away with when you started to use your own API?
Mac mini osx mavericks running 0.9.9.7.429-f80a8d6
Thanks

To help you i need the log files, you can look here: https://support.plex.tv/hc/en-us/articles/200250417-Plex-Media-Server-Log-Files to locate the log files on your system. try to match a anime manually and then upload the log file.

please be sure to use the latest version of the agent.

with the use of my own API i didn't needed an myanimelist account so I dont support it (yet?!) because the api doesnt support anything that would require an myanimelist account.

58

Hmm, lately myanimelist isn't working even when manually fixing titles.  IT shows up as no matches found.  Anyone else getting that?

59

I bet you are using an old version (3.5 or before). The API has changed so all previous versions probably do not function. You'll have to upgrade to version 3.6 (found https://forums.plex.tv/topic/105054-release-myanimelistnet-metadata-agent/ ) to fix your issue.

60

You can follow the Agent on my Blog http://devvsbugs.net/projects/pms-malagent/ or follow me on twitter https://twitter.com/D4r4ku to get new updates to the projects