Beta for OpenSubtitles metadata agent improvement

Dev Community Dev/API Corner
1

As you know, the current version of the OpenSubtitles metadata agent is not really efficient. After investigating and discussing with the community I've decide to code a prototype to demonstrate we can have better automatic results by making some consistency check on metadata to find the best subtitle to download.

 

You can fin in the following fork on Github, my code and a description of the new algorithm. If you want to participate to the improvement go download and test this fork. Instruction are in the README file and you can ask in this thread if you need help.

 

DON'T TEST THIS METADATA AGENT ON YOUR PRODUCTION SERVER. THERE IS A LITTLE RISK THIS SCREW-UP YOUR SUBS.

 

Github fork with explanations : https://github.com/oncleben31/OpenSubtitles.bundle

 

I'm listening your comments ant your tests results.

 

 

Release history:

 

# RC2 (02/04/2014):

   - release note:

        * Add a filter on the name of a movie or TV show if the primary agent use English language
        * Add a filter on the number of the episode.
        * Fix some bugs

   - Limitations:

        * No IMDB id search if hash/size don't return acceptable answers

        * Hearing Impared pref is useless for the moment

        * The API key for TheTVDB is not used due to XML.ElementFromString errors

 

 

# RC1 (20/03/2014):

​   - First release

   - Limitations:

        * No IMDB id search if hash/size don't return acceptable answers

        * Hearing Impared pref is useless for the moment

        * The API key for TheTVDB is not used due to XML.ElementFromString errors

2

Any body to beta test ?

3

If you don't want to take some risks by beta testing it. Can you help me by giving me some case where current official version didn't suceed to fetch the good subtitle. 

I need the following information :

 - hash size of the video file (in the Logs or by dragging it in the OpenSubtitles.org search form on the web site

 - file size Logs or file information

 - primary agent used and associated language

 - language of the subs

 - vidoe file name

 - movie or show/episode name

4

Hello. Great initiative. I always read about how weak the Opensubtitles plugin seem to be. :)

I haven't tried your agent improvement, but I'll give some feedback on what I've read.

I think you have done a thorough job analyzing how to interpret the data from Opensubtitles, and you have detailed it quite nicely in the readme. I think though that it's too complex for anyone not familiar with how the Opensubtitles and the site.

I think you should use another word instead of "mockup". Mockup is mostly used for non-working graphic prototypes. I was mislead first and thought that you had only detailed how you wanted it to work.

I think a really good thing to get some feedback would be to provide some examples on where this agent gives better results than the existing one.

Last but not least, I think sending a pull request to sander1 or sending a pm directly to him would give good response from someone who is knowledgeable in the code and the site. There is a developer chat available as well for easier communication. Sander will help you I think.

Finally. Good initiative! As far as I've understood, there is a lot of room for improvement in the original agent.

Cheers!

5

Thanks for your comments, and I agree I've change the title of the subject.

Sander and Elan are aware of my work but I don't want to ask for a pull request now since I want some beta testing to get an agent with more maturity. I've already identified limitations and bugs.

6

Hey guys!

I am really thrilled to find a thread like this. I also have problems with the current "official" agent, so I tried this one. I installed it according to your GitHub description. I have already registered at opensubtitles, my preferred languages are English and Hungarian.

I was trying to download subtitles for Sherlock S03E03, but I did not succeed. I have refreshed the show multiple times on the web inteface with the little round arrow, but unfortunately nothing happened with this particular episode. However, it did download the correct subtitles for S03E02, but the "official" agent was able to do that too. It's not like these subtitles are missing from opensubtitles or anything: if you do a manual search, you can clearly see that they are there.

I am uploading the log files (I hope these are the ones that you need). If you need any further information, I am happy to give those too.

Thanks again for your contribution! :) Looking forward to your reply.

ajsz

7

Thanks for your contribution, I will look into your logs and keep you informed.

8

I've look at the logs, and the metadata agent didn't find subs because OpenSubtitles seems to have no entries with the couple hash/size of your file.

Currently my metadata agent is only searching by using this method, but addition of other methods is in progress.

Are you sure you are using my metadata agent ? I didn't recognize the Logs messages.

9

Dear oncleben31,

Thank you for your reply. I'm fairly sure I did everything correctly: downloaded the zip from your github, extracted it to the folder, deleted the old one and renamed the new to the correct foldername, then restarted both the server and the client, and did a refresh on the series multiple times. Is this the correct way to do it?

I will reinstall your agent and upload some more logs later today maybe with different shows.

10

It seems you do it right.

11

RC2 released. See first post.

Next release will have other search methods.

12

Hi,

Just installed it on a fresh new install but apparently can't find anything... did i miss something ?

Looking for tV show "Vikings" S02E06 (Id 260449 on thetvdb)

Agent config is to french then english.

Some subtitles exist : http://www.opensubtitles.org/en/search/imdbid-3358060/sublanguageid-eng,fre/moviename-vikings

Here's the log

2014-04-04 14:12:08,134 (7f6abe7fc700) :  DEBUG (runtime:717) - Handling request GET /:/plugins/com.plexapp.agents.opensubtitles/messaging/function/X0FnZW50S2l0OlNlYXJjaA__/Y2VyZWFsMQoxCmxpc3QKMApyMAo_/Y2VyZWFsMQoyCmRpY3QKZGljdAo0CnMyCmZyczQKbGFuZ3M3ClRWX1Nob3dzMTAKbWVkaWFfdHlwZWIwczYKbWFudWFscjEKczYKa3dhcmdzMwpzNDcKY29tLnBsZXhhcHAuYWdlbnRzLnRoZXR2ZGI6Ly8yNjA0NDkvMi82P2xhbmc9ZnJzNApndWlkYjFzNQpmb3JjZXMyNgpjb20ucGxleGFwcC5hZ2VudHMudGhldHZkYnMxMwpwcmltYXJ5X2FnZW50cjAK
2014-04-04 14:12:08,138 (7f6abe7fc700) :  DEBUG (runtime:814) - Found route matching /:/plugins/com.plexapp.agents.opensubtitles/messaging/function/X0FnZW50S2l0OlNlYXJjaA__/Y2VyZWFsMQoxCmxpc3QKMApyMAo_/Y2VyZWFsMQoyCmRpY3QKZGljdAo0CnMyCmZyczQKbGFuZ3M3ClRWX1Nob3dzMTAKbWVkaWFfdHlwZWIwczYKbWFudWFscjEKczYKa3dhcmdzMwpzNDcKY29tLnBsZXhhcHAuYWdlbnRzLnRoZXR2ZGI6Ly8yNjA0NDkvMi82P2xhbmc9ZnJzNApndWlkYjFzNQpmb3JjZXMyNgpjb20ucGxleGFwcC5hZ2VudHMudGhldHZkYnMxMwpwcmltYXJ5X2FnZW50cjAK
2014-04-04 14:12:08,139 (7f6abe7fc700) :  INFO (agentkit:915) - Searching for matches for {'guid': 'com.plexapp.agents.thetvdb://260449/2/6?lang=fr', 'force': True, 'primary_agent': 'com.plexapp.agents.thetvdb'}
2014-04-04 14:12:08,141 (7f6abe7fc700) :  DEBUG (model:32) - Loading model with GUID com.plexapp.agents.thetvdb://260449/2/6?lang=fr
2014-04-04 14:12:08,142 (7f6abe7fc700) :  DEBUG (model:223) - Deserializing from /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata/TV Shows/1/dfe4797709a007e461a4b801a411064ead8bc32.bundle/Contents/com.plexapp.agents.thetvdb/Info.xml
2014-04-04 14:12:08,162 (7f6abe7fc700) :  DEBUG (runtime:918) - Response: [200] str, 600 bytes
2014-04-04 14:12:08,175 (7f6abe7fc700) :  DEBUG (runtime:717) - Handling request GET /:/plugins/com.plexapp.agents.opensubtitles/messaging/function/X0FnZW50S2l0OlVwZGF0ZU1ldGFkYXRh/Y2VyZWFsMQoxCmxpc3QKMApyMAo_/Y2VyZWFsMQoxCmRpY3QKNwpzMgpmcnM0CmxhbmdiMXM1CmZvcmNlczQ1CmNvbS5wbGV4YXBwLmFnZW50cy50aGV0dmRiOi8vMjYwNDQ5LzI%40bGFuZz1mcnMxMApwYXJlbnRHVUlEczcKVFZfU2hvd3MxMAptZWRpYV90eXBlczQKMTY3M3M0CmRiaWRzNDcKY29tLnBsZXhhcHAuYWdlbnRzLnRoZXR2ZGI6Ly8yNjA0NDkvMi82P2xhbmc9ZnJzNApndWlkczQKbnVsbHMyCmlkcjAK
2014-04-04 14:12:08,178 (7f6abe7fc700) :  DEBUG (runtime:814) - Found route matching /:/plugins/com.plexapp.agents.opensubtitles/messaging/function/X0FnZW50S2l0OlVwZGF0ZU1ldGFkYXRh/Y2VyZWFsMQoxCmxpc3QKMApyMAo_/Y2VyZWFsMQoxCmRpY3QKNwpzMgpmcnM0CmxhbmdiMXM1CmZvcmNlczQ1CmNvbS5wbGV4YXBwLmFnZW50cy50aGV0dmRiOi8vMjYwNDQ5LzI@bGFuZz1mcnMxMApwYXJlbnRHVUlEczcKVFZfU2hvd3MxMAptZWRpYV90eXBlczQKMTY3M3M0CmRiaWRzNDcKY29tLnBsZXhhcHAuYWdlbnRzLnRoZXR2ZGI6Ly8yNjA0NDkvMi82P2xhbmc9ZnJzNApndWlkczQKbnVsbHMyCmlkcjAK
2014-04-04 14:12:08,181 (7f6abe7fc700) :  DEBUG (model:32) - Loading model with GUID com.plexapp.agents.thetvdb://260449/2/6?lang=fr
2014-04-04 14:12:08,182 (7f6abe7fc700) :  DEBUG (model:223) - Deserializing from /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata/TV Shows/1/dfe4797709a007e461a4b801a411064ead8bc32.bundle/Contents/com.plexapp.agents.opensubtitles/Info.xml
2014-04-04 14:12:08,184 (7f6abe7fc700) :  DEBUG (networking:172) - Requesting 'http://127.0.0.1:32400/library/metadata/1673/tree'
2014-04-04 14:12:08,335 (7f6abe7fc700) :  INFO (__init__:46) - Looking for match for GUID 7dc91bfabbf718a0 and size 863274590
2014-04-04 14:12:08,410 (7f6abe7fc700) :  INFO (__init__:66) - No subtitles available for language fre
2014-04-04 14:12:08,411 (7f6abe7fc700) :  INFO (__init__:46) - Looking for match for GUID 7dc91bfabbf718a0 and size 863274590
2014-04-04 14:12:08,487 (7f6abe7fc700) :  INFO (__init__:66) - No subtitles available for language eng
2014-04-04 14:12:08,490 (7f6abe7fc700) :  DEBUG (model:218) - Serializing to /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Metadata/TV Shows/1/dfe4797709a007e461a4b801a411064ead8bc32.bundle/Contents/com.plexapp.agents.opensubtitles/Info.xml
2014-04-04 14:12:08,493 (7f6abe7fc700) :  DEBUG (runtime:918) - Response: [200] str, 16 bytes

and of course nothing in the Info.xml

<?xml version='1.0' encoding='utf-8'?>

Thanks.

Seb.

13

Hi,

Getting exception when trying to download

2014-04-05 22:29:30,752 (-b239470) :  CRITICAL (agentkit:974) - Exception in the update function of agent named 'OpenSubtitles.org', called with guid 'com.plexapp.agents.imdb://tt1408101?lang=sv' (most recent call last):
  File "/share/MD0_DATA/.qpkg/PlexMediaServer/Library/Plex Media Server/Plug-ins/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/api/agentkit.py", line 972, in _update
    agent.update(obj, media, lang)
  File "/root/Library/Plex Media Server/Plug-ins/OpenSubtitles.bundle/Contents/Code/__init__.py", line 86, in update
    fetchSubtitles(proxy, token, part, metadata.id)
  File "/root/Library/Plex Media Server/Plug-ins/OpenSubtitles.bundle/Contents/Code/__init__.py", line 63, in fetchSubtitles
    subData = Archive.GzipDecompress(subGz)
  File "/share/MD0_DATA/.qpkg/PlexMediaServer/Library/Plex Media Server/Plug-ins/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/api/utilkit.py", line 58, in GzipDecompress
    return self._core.data.archiving.gzip_decompress(data)
  File "/share/MD0_DATA/.qpkg/PlexMediaServer/Library/Plex Media Server/Plug-ins/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/components/data.py", line 100, in gzip_decompress
    ret = gzipper.read()
  File "/share/MD0_DATA/.qpkg/PlexMediaServer/Resources/Python/lib/python2.7/gzip.py", line 253, in read
    while self._read(readsize):
  File "/share/MD0_DATA/.qpkg/PlexMediaServer/Resources/Python/lib/python2.7/gzip.py", line 293, in _read
    self._read_gzip_header()
  File "/share/MD0_DATA/.qpkg/PlexMediaServer/Resources/Python/lib/python2.7/gzip.py", line 197, in _read_gzip_header
    raise IOError, 'Not a gzipped file'
IOError: Not a gzipped file

2014-04-05 22:29:30,756 (-b239470) :  DEBUG (model:218) - Serializing to /root/Library/Plex Media Server/Metadata/Movies/1/1e01d6db905fc7681be93bea91b75fd7e94e729.bundle/Contents/com.plexapp.agents.opensubtitles/Info.xml
2014-04-05 22:29:30,761 (-b239470) :  DEBUG (runtime:918) - Response: [200] str, 16 bytes
 

14

@Sebhash I'm not sure you have my code. Can you make a verification of the installation. And ressent your logs.

@mhugo I pretty sure this is due to OpenSubtitles.org which have some difficulties with their servers unavailable from time to time. I will add a catch in this part of the code to understand the reason of errors during this phase.

15

Hi,

I've been giving a try to your agent and it seems to work better than the original indeed !

Here's a CRITICAL I have encountered so far on different TV Shows :

2014-04-12 12:09:35,978 (7f6b016ee700) :  CRITICAL (agentkit:974) - Exception in the update function of agent named ‘OpenSubtitles.org’, called with guid ‘com.plexapp.agents.thetvdb://79334?lang=en’ (most recent call last):

  File “/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/api/agentkit.py”, line 972, in _update

    agent.update(obj, media, lang)

  File “/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/OpenSubtitles.bundle/Contents/Code/init.py”, line 412, in update

    subtitleResponse = fetchSubtitles(proxy, token, part, language, primaryAgentLanguage, ImdbEpisodeId, OS_Search_Methode.IMDB)

  File “/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/OpenSubtitles.bundle/Contents/Code/init.py”, line 183, in fetchSubtitles

    Log(‘Looking for match for name and language %s’ % ( language))

  File “/usr/lib/plexmediaserver/Resources/Python/lib/python2.7/xmlrpclib.py”, line 1224, in call

    return self.__send(self.__name, args)

  File “/usr/lib/plexmediaserver/Resources/Python/lib/python2.7/xmlrpclib.py”, line 1572, in __request

    allow_none=self.__allow_none)

  File “/usr/lib/plexmediaserver/Resources/Python/lib/python2.7/xmlrpclib.py”, line 1085, in dumps

    data = m.dumps(params)

  File “/usr/lib/plexmediaserver/Resources/Python/lib/python2.7/xmlrpclib.py”, line 632, in dumps

    dump(v, write)

  File “/usr/lib/plexmediaserver/Resources/Python/lib/python2.7/xmlrpclib.py”, line 654, in __dump

    f(self, value, write)

  File “/usr/lib/plexmediaserver/Resources/Python/lib/python2.7/xmlrpclib.py”, line 714, in dump_array

    dump(v, write)

  File “/usr/lib/plexmediaserver/Resources/Python/lib/python2.7/xmlrpclib.py”, line 654, in __dump

    f(self, value, write)

  File “/usr/lib/plexmediaserver/Resources/Python/lib/python2.7/xmlrpclib.py”, line 735, in dump_struct

    dump(v, write)

  File “/usr/lib/plexmediaserver/Resources/Python/lib/python2.7/xmlrpclib.py”, line 654, in __dump

    f(self, value, write)

  File “/usr/lib/plexmediaserver/Resources/Python/lib/python2.7/xmlrpclib.py”, line 658, in dump_nil

    raise TypeError, “cannot marshal None unless allow_none is enabled”

TypeError: cannot marshal None unless allow_none is enabled

Keep up the good work !

Alex

16

Thanks for your report. Can you add the logs before that message to help me understand the context of the error ? 

17

Hi, made a reply in direct message with log attached, hope you find something useful :)

Alex

18

Thanks I will look your logs.

19

Hello,

I built Korean metadata agent(com.plex.agent.daum_movie) and someone asked me how to download subtitles with OpenSubtitles.

It seems the first priority option is using file hash key which Plex framework tosses to the agent in special care, and searching with IMDB ID is the 2nd priority.

But the current OpenSubtitles is hard coded to run with IMDB & TVDB agent, even though no info from TVDB is used in the original OpenSubtitles agent.

I think you should remove this dependency for further compatibility.

Also there is a chance that new agent passes IMDB ID to let OpenSubtitles agent.work with. I added this option in my agent for test.

20

Some suggestions:

1. Don't fetch .sub subtitles, thay are generaly bad. In my  case every .sub substiles are out of sync or wrong episode.

2. Use HASH search, Im long time XBMC user and HASH search are almost 100% hit.

In my case all bulgarian subtitles for "24" fisrt two seasons episodes are wrong, it will be better if i pick them randomly :)