python-rarfile inclusion?

Dev Community Dev/API Corner
1

Hi,

I'm rewriting a subtitle metadata agent plugin and a good amount of subtitles that I'm downloading are archived as rar.

 

Is there a way for me to package it with my plugin bundle as a library?

 

If not, is it possible to add python-rarfile as a dependancy to my plugin so that plex would add it to site-packages?

 

If that is also not possible, would you please consider adding it in the next plexpass build as a python library?

 

python-rarfile is ISC licensed (BSD/MIT like) and its API documentation is here: https://rarfile.readthedocs.org/en/latest/api.html 

 

The "official" unrar (http://www.rarlab.com/rar/unrarsrc-5.1.6.tar.gz) is freeware and may be "freely distributed standalone or in other software packages".

 

If the freeware license would be too restrictive, this unrar is GPL (http://home.gna.org/unrar/) and for rar v3 there is unar ( https://code.google.com/p/theunarchiver/) which is LGPL, so there shouldn't be any licensing reasons that prevent this inclusion. If the user so chooses to replace the unrar version with 

 

I can imagine other plugins may also benefit from this. I'd really appreciate it.

2

To clarify:

  1. Your agent downloads the subtitles from some source where they are provided in a rar archive format.
  2. You want your agent (or Plex) to unrar the subtitles so that they can be used.
  3. (2) requires a python library for rar support which is not included in the existing Plex framework

Is that correct?

3

That is correct.

I would happily include rarfile.py itself in the bundle, but I'm not sure if that would work. Rarfile imports:

import sys, os, struct, errno
from struct import pack, unpack
from binascii import crc32
from tempfile import mkstemp
from subprocess import Popen, PIPE, STDOUT
from datetime import datetime
from Crypto.Cipher import AES
from hashlib import sha1
from sha import new as sha1
import array
from struct import Struct
from io import RawIOBase
import array
 
I have a standalone proof of concept done in python that I'm in the process of rewriting as a metadata agent. I may be able to drop some of them (don't need crypto probably). I have not tested it under Plex yet. Am I allowed to include it in a Bundle in the first place?
4

You _should_ be able to include the .py file directly in the code directory of the agent.bundle and import it in the __init__.py. You will likely need to make use of the elevated code policy if you are not already.

PlexPluginCodePolicy
Elevated

Add the above to the agent's plist file.

5

Finally got around to trying this, but running into a ton of problems.

I have tons of:

SyntaxError: Line 95: "_have_crypto" is an invalid variable name because it starts with "_"
 
But even after making a few dozen changes and renaming all these weak variables in rarfile.py, I still get:
 
SyntaxError: Line 703: Augmented assignment of attributes is not allowed.
The relevant code is:
                old.compress_size += item.compress_size

Full log:

2014-08-29 23:44:06,025 (7fa45e439700) :  INFO (core:347) - Starting framework core - Version: 2.5.0, Build: a51033b (Mon Jul 28 12:19:14 UTC 2014)
2014-08-29 23:44:06,025 (7fa45e439700) :  DEBUG (core:359) - Using the elevated policy
2014-08-29 23:44:06,025 (7fa45e439700) :  DEBUG (core:448) - Starting runtime component.
2014-08-29 23:44:06,026 (7fa45e439700) :  DEBUG (core:448) - Starting caching component.
2014-08-29 23:44:06,026 (7fa45e439700) :  DEBUG (core:448) - Starting data component.
2014-08-29 23:44:06,027 (7fa45e439700) :  DEBUG (core:448) - Starting networking component.
2014-08-29 23:44:06,027 (7fa45e439700) :  DEBUG (networking:288) - Loaded HTTP cookies
2014-08-29 23:44:06,027 (7fa45e439700) :  DEBUG (networking:458) - Setting the default network timeout to 20.0
2014-08-29 23:44:06,028 (7fa45e439700) :  DEBUG (core:448) - Starting localization component.
2014-08-29 23:44:06,028 (7fa45e439700) :  INFO (localization:408) - Setting the default locale to en-us
2014-08-29 23:44:06,028 (7fa45e439700) :  DEBUG (localization:426) - Loaded en strings
2014-08-29 23:44:06,028 (7fa45e439700) :  DEBUG (core:448) - Starting messaging component.
2014-08-29 23:44:06,029 (7fa45e439700) :  DEBUG (core:448) - Starting debugging component.
2014-08-29 23:44:06,029 (7fa45e439700) :  DEBUG (core:448) - Starting services component.
2014-08-29 23:44:06,029 (7fa452ee1700) :  DEBUG (networking:172) - Requesting 'http://127.0.0.1:32400/system/messaging/clear_events/com.plexapp.agents.hdbits'
2014-08-29 23:44:06,029 (7fa45e439700) :  DEBUG (core:448) - Starting myplex component.
2014-08-29 23:44:06,030 (7fa45e439700) :  DEBUG (core:448) - Starting notifications component.
2014-08-29 23:44:06,158 (7fa45e439700) :  DEBUG (accessor:68) - Creating a new model access point for provider com.plexapp.agents.hdbits in namespace 'metadata'
2014-08-29 23:44:06,161 (7fa45e439700) :  DEBUG (networking:172) - Requesting 'http://127.0.0.1:32400/:/plugins/com.plexapp.system/resourceHashes'
2014-08-29 23:44:06,195 (7fa452ee1700) :  DEBUG (services:265) - Plug-in is not daemonized - loading services from system
2014-08-29 23:44:06,195 (7fa45e439700) :  DEBUG (runtime:1111) - Created a thread named 'load_all_services'
2014-08-29 23:44:06,197 (7fa452ee1700) :  DEBUG (networking:172) - Requesting 'http://127.0.0.1:32400/:/plugins/com.plexapp.system/messaging/function/X0J1bmRsZVNlcnZpY2U6QWxsU2VydmljZXM_/Y2VyZWFsMQoxCmxpc3QKMApyMAo_/Y2VyZWFsMQoxCmRpY3QKMApyMAo_'
2014-08-29 23:44:06,197 (7fa45e439700) :  DEBUG (runtime:1111) - Created a thread named 'get_server_info'
2014-08-29 23:44:06,198 (7fa451edf700) :  DEBUG (networking:172) - Requesting 'http://127.0.0.1:32400'
2014-08-29 23:44:06,198 (7fa45e439700) :  DEBUG (core:150) - Finished starting framework core
2014-08-29 23:44:06,199 (7fa45e439700) :  DEBUG (core:558) - Loading plug-in code
2014-08-29 23:44:06,332 (7fa45e439700) :  DEBUG (core:564) - Finished loading plug-in code
2014-08-29 23:44:06,441 (7fa45e439700) :  CRITICAL (core:613) - Exception starting plug-in (most recent call last):
  File "/home/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/core.py", line 606, in start
    self.sandbox.execute(self.init_code)
  File "/home/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/code/sandbox.py", line 256, in execute
    exec(code) in self.environment
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/HdbitsSubtitles.bundle/Contents/Code/__init__.py", line 12, in 
    import rarfile
  File "/home/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/code/sandbox.py", line 333, in __import__
    return mod.load_module(_name)
  File "/home/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/code/sandbox.py", line 44, in load_module
    module = RestrictedModule(name, path, sandbox)
  File "/home/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/code/loader.py", line 20, in __init__
    code = sandbox._core.loader.load(filename, sandbox.policy.elevated_execution)
  File "/home/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/code/loader.py", line 47, in load
    code = self.compile(str(source), str(uni(filename)), elevated)
  File "/home/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/code/loader.py", line 52, in compile
    return RestrictedPython.compile_restricted(source, name, 'exec', elevated=elevated)
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Framework.bundle/Contents/Resources/Platforms/Shared/Libraries/RestrictedPython/RCompile.py", line 115, in compile_restricted
    gen.compile()
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Framework.bundle/Contents/Resources/Platforms/Shared/Libraries/RestrictedPython/RCompile.py", line 68, in compile
    tree = self._get_tree()
  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Framework.bundle/Contents/Resources/Platforms/Shared/Libraries/RestrictedPython/RCompile.py", line 62, in _get_tree
    raise SyntaxError, self.rm.errors[0]
SyntaxError: Line 703: Augmented assignment of attributes is not allowed.

2014-08-29 23:44:06,443 (7fa451edf700) :  DEBUG (core:536) - Machine identifier is 4ee8d72248097270436889eafea8b9dd66056fcc

2014-08-29 23:44:06,444 (7fa451edf700) :  DEBUG (core:537) - Server version is 0.9.9.13.525-197d5ed

2014-08-29 23:44:06,505 (7fa452ee1700) :  DEBUG (services:362) - Loaded services

2014-08-29 23:44:06,507 (7fa4526e0700) :  DEBUG (services:438) - No shared code to load

I've spent a good hour trying to rewrite rarfile.py to allow it to pass through RPython but still running into errors. I can understand the need for RPython parsing, but this makes including rarfile pretty difficult.

Is there a better way? If not, would you reconsider including it in main like zipfile so that it does not have to be included? Because rarfile is likely going to be doing some low level file operations that I don't even know  would work under RPython after making all the changes - I have no idea what other restrictions will apply once I get through the parsing errors.

6

Ah. That's a real pain. I'll see what I can find out in terms of a work-around.

7

I have rewritten most of rarfile.py and it sort of works when adding a lot of StringIO in there, but my biggest challenge at this point is the lack of fd handling, and addressing the 'unrar' binary. Zip file format can be handled purely in python but Rar needs the unrar binary for listing and unpacking.

At this stage the only workaround I can think of is an ugly one. I can have the user put unrar binaries in the Plugin Support data dir, and then simply directly unrar the DataItems/$key files and unpack them in a temp dir that I create. 

If none of this works I could throw up an external xmlrpc webservice that uploads rarfiles and returns zipfiles, but that's even less ideal because it adds an external dependency.

I'm open to suggestions for a better approach here.

8

I ran into this issue as well with my subtitle plugin. Unfortunately I found no good solution.

The best I could come up with as well was to use an online extractor, but I never implemented it.

It would be sweet if Plex could include unrar in one way or the other as part of the API.

9

I have this working now, on Linux at least. I hope to post a first release this weekend.

10

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.