i’ve been doing a bit more research regarding the issue we are seeing with errors such as:
UnicodeDecodeError: 'ascii' codec can't decode byte 0x??? in position n: ordinal not in range(128)
my hunch was it had to be something to do with the plugin process thinking its default codec for strings is ascii. Being a bit new to python I had a peep about in the sys package to see if it could help based upon some of the articles I'd read regarding the joys of and transpancey in programs. They make for some good background reading. Typical internet applications and services provide and consume data that is not locked onto a specific locale or character set.
[What I Thought I Knew About Unicode in Python Amounted To Nothing](http://farmdev.com/thoughts/23/what-i-thought-i-knew-about-unicode-in-python-amounted-to-nothing/)
[Do I hate Unicode, or Do I Hate ASCII?](http://blog.ianbicking.org/do-i-hate-unicode-or-do-i-hate-ascii.html)
so, if you make yourself a little python script like this
#!/usr/bin/python<br />
import sys<br />
import locale<br />
print "default encoding is '%s'" % sys.getdefaultencoding()<br />
print "default locale is '%s' '%s'" % locale.getdefaultlocale()
when run it outputs
default encoding is 'ascii'<br />
default locale is 'en_GB' 'UTF8'
or I just go at it in the python interepter I do pretty much the same thing...
idg@somewhere.local:~ $ python<br />
Python 2.5.1 (r251:54863, Feb 6 2009, 19:02:12)<br />
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin<br />
Type "help", "copyright", "credits" or "license" for more information.<br />
>>> import sys<br />
>>> import locale<br />
>>> print "default encoding is '%s'" % sys.getdefaultencoding()<br />
default encoding is 'ascii'<br />
>>> print "default locale is '%s' '%s'" % locale.getdefaultlocale()<br />
default locale is 'en_GB' 'UTF8'
I get the same.
this choice of encoding takes place down in
/System/Library/Frameworks/Python.framework/Versions/Current/lib/python2.5/site.py
If you take a peep at setencoding() on line 368 of the above file you'll see how a python process decides what character encoding and locale it will assume for and datatypes. As I am in the UK and my mac is configured with english language and United Kingdom region that is what I would expect. It would be good if somebody outside of the uk using a non-english language could report back what they see when they do the same.
I placed similar lines into my plugins initialisation to try and figure out what codec a PMS python plugin would be using (i.e. the very first thing in the Start() method) like so
In my plugins log I see.
15:24:05.620889: com.plexapp.plugins.flickr : (Framework) Attempting to start the plug-in...<br />
15:24:05.621274: com.plexapp.plugins.flickr : default encoding is 'ascii'<br />
15:24:05.622555: com.plexapp.plugins.flickr : default locale is 'None' 'None'
So - as suspected the plugin process thinks everything that happens to strings has to happen in an ascii way.
There are some nasty hacks you can do to get round this which you can read about on the above linked articles but changingn /usr/lib/python/site.py is bad as it effects everthing. The cleanest way seems to be to do something like
sys.setdefaultencoding('utf-8')
so we only effect ourselves and not everything that chooses to use the system python installation.
So next I modifed the first things my Start() method does to
sys.setdefaultencoding('utf-8')<br />
Log("default encoding is '%s'" % sys.getdefaultencoding())<br />
Log("default locale is '%s' '%s'" % locale.getdefaultlocale())
Now when the plugin initialises you see this in the log
15:31:59.146241: com.plexapp.plugins.flickr : (Framework) Bundle verification complete<br />
15:31:59.147052: com.plexapp.plugins.flickr : (Framework) Debugging is enabled<br />
15:31:59.164191: com.plexapp.plugins.flickr : (Framework) Configured framework modules<br />
15:31:59.178126: com.plexapp.plugins.flickr : (Framework) Loaded shared resource named 'trust.json'<br />
15:32:00.213398: com.plexapp.plugins.flickr : (Framework) Imported plug-in module<br />
15:32:00.219755: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'locale' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.226678: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'encodings' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.226836: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'utf_8' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.232664: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'codecs' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.237950: com.plexapp.plugins.flickr : (Framework) WARNING: The blacklisted module 'sys' has been imported. Future framework versions will abort at this point.<br />
15:32:00.238210: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'ascii' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.243939: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'types' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.255045: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'aliases' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.256646: com.plexapp.plugins.flickr : (Framework) WARNING: The blacklisted module 'socket' has been imported. Future framework versions will abort at this point.<br />
15:32:00.256769: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'flickrapi' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.259126: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'mimetools' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.259316: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'tempfile' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.259657: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module '_fcntl' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.260240: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module '_thread' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.261982: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module '_errno' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.262498: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module '_os' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.262965: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module '_copy_reg' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.263234: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'UserDict' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.263459: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'path' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.263780: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'stat' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.264242: com.plexapp.plugins.flickr : (Framework) WARNING: The blacklisted module 'os' has been imported. Future framework versions will abort at this point.<br />
15:32:00.264768: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'rfc822' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.271530: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'multipart' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.279436: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'base64' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.279916: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'struct' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.280281: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'binascii' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.280732: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'cache' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.281358: com.plexapp.plugins.flickr : (Framework) WARNING: The blacklisted module 'threading' has been imported. Future framework versions will abort at this point.<br />
15:32:00.283073: com.plexapp.plugins.flickr : (Framework) WARNING: The blacklisted module 'urllib' has been imported. Future framework versions will abort at this point.<br />
15:32:00.283214: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'tokencache' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.283491: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'logging' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.284518: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'cStringIO' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.284849: com.plexapp.plugins.flickr : (Framework) WARNING: The blacklisted module 'thread' has been imported. Future framework versions will abort at this point.<br />
15:32:00.284966: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'atexit' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.288504: com.plexapp.plugins.flickr : (Framework) WARNING: The blacklisted module 'traceback' has been imported. Future framework versions will abort at this point.<br />
15:32:00.294762: com.plexapp.plugins.flickr : (Framework) WARNING: The blacklisted module 'urllib2' has been imported. Future framework versions will abort at this point.<br />
15:32:00.301324: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'webbrowser' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.302898: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'subprocess' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.303132: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'select' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.304848: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'errno' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.305450: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'fcntl' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.305688: com.plexapp.plugins.flickr : (Framework) WARNING: The blacklisted module 'pickle' has been imported. Future framework versions will abort at this point.<br />
15:32:00.307668: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'ic' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.308094: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'Res' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.308959: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'Carbon' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.309099: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'Files' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.311139: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'warnings' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.311391: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'linecache' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.324079: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'File' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.324580: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'icglue' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.324722: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'macostools' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.324872: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'MacOS' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.339026: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'shlex' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.339270: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'reportinghttp' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.339377: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'httplib' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.339804: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'copy' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.340251: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'md5' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.340376: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'exceptions' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.340484: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'xmlnode' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.340583: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'xml' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.340684: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'dom' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.340913: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'xmlbuilder' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.341059: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'minidom' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.341296: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'domreg' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.341463: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'minicompat' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.344001: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'NodeFilter' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.344547: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'parsers' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.344677: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'expat' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.344811: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'errors' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.345079: com.plexapp.plugins.flickr : (Framework) NOTICE: The unknown module 'model' has been imported. This may be blacklisted in future and should be used with caution.<br />
15:32:00.345426: com.plexapp.plugins.flickr : (Framework) Checked module imports<br />
15:32:00.346487: com.plexapp.plugins.flickr : (Framework) Loaded the default preferences file<br />
15:32:00.346807: com.plexapp.plugins.flickr : (Framework) Loaded the user preferences file<br />
15:32:00.347062: com.plexapp.plugins.flickr : (Framework) No cookie jar found<br />
15:32:00.481684: com.plexapp.plugins.flickr : (Framework) Loaded en strings<br />
15:32:00.482134: com.plexapp.plugins.flickr : (Framework) Couldn't find en-us strings<br />
15:32:00.482406: com.plexapp.plugins.flickr : (Framework) Initialized framework modules<br />
15:32:00.482665: com.plexapp.plugins.flickr : (Framework) Attempting to start the plug-in...<br />
15:32:00.485858: com.plexapp.plugins.flickr : (Framework) An exception happened:<br />
Traceback (most recent call last):<br />
File "/Users/idg/Library/Application Support/Plex Media Server/Plug-ins/Framework.bundle/Contents/Resources/Versions/1/Python/PMS/Plugin.py", line 550, in __call<br />
return function(*args, **kwargs)<br />
File "/Users/idg/Library/Application Support/Plex Media Server/Plug-ins/Flickr.bundle/Contents/Code/__init__.py", line 75, in Start<br />
sys.setdefaultencoding('utf-8')<br />
AttributeError: 'module' object has no attribute 'setdefaultencoding'
the reason why this happens is because of the main() method of pythons site.py (see line 396) file. To protect a process once its been initialised the sys.setdefaultencoding function is removed so that it cannot be overriden to anything else. Once i'd figured this out this article gave a good explanation.
[sys.setdefaultencoding is evil](http://tarekziade.wordpress.com/2008/01/08/syssetdefaultencoding-is-evil/)
The hack around this method being deleted is to do something like this in your code:
import sys, locale<br />
reload(sys)<br />
sys.setdefaultencoding("utf-8")
I changed my Start() method to begin with...
reload(sys)<br />
sys.setdefaultencoding("utf-8")<br />
Log("default encoding is '%s'" % sys.getdefaultencoding())<br />
Log("default locale is '%s' '%s'" % locale.getdefaultlocale())
and when my plugin starts I now see all the nasty errors but importantly...
15:52:53.827983: com.plexapp.plugins.flickr : (Framework) Initialized framework modules<br />
15:52:53.828046: com.plexapp.plugins.flickr : (Framework) Attempting to start the plug-in...<br />
15:52:53.828135: com.plexapp.plugins.flickr : default encoding is 'utf-8'<br />
15:52:53.828242: com.plexapp.plugins.flickr : default locale is 'None' 'None'
Now when I run up the plugin and browse a tag list that would have massively caused problems due to accented non-ascii characters it doesn't explode and I get my data back in plex. The other important side effect of this is I no longer need to pass all data through my Encode() function because python and encode/decode mayhem is now seemingly correctly managed.
The only side effect I see now is that some completely valid characters have no corresponding character in the font map that is used back in the front end in Plex itself (this could be my font configuration though). Heres an example of what I mean (a correctly encoded response with correctly printed accented chracters and some with a missing *blank* in place of a character)
You can see it on this screenshot here:
I think this is a good explanation as to why we get encoding issues with character encoding in plugins - I need to do a bit more testing but I think this is the source of the issues we have. I know the Picasa plugin suffers the same fate as Flickr. I guess the best solution to this would be to deal with this higher up in the plugin framework so it becomes more abstract and plugin devs don't even need to be aware of things.
As I am very new to python can somebody please comment on my above thoughts.