Unicode (UTF-8)

Dev Community Dev/API Corner
1

One of the things I cannot get working in my Plug-in are the most exotic characters such as åäö. As I’m from Finland and the plug-in contents as well, this is vital. And also some of the videos don’t even play because of the errors caused by this. 



File "/Users/xxx/Library/Application Support/Plex/Plex Media Server.app/Contents/Resources/Python/Resources/PMS/MediaXML.py", line 59, in SetAttr<br />
	self.root.set(name, value)<br />
  File "lxml.etree.pyx", line 641, in lxml.etree._Element.set (src/lxml/lxml.etree.c:9596)<br />
  File "apihelpers.pxi", line 416, in lxml.etree._setAttributeValue (src/lxml/lxml.etree.c:31761)<br />
  File "apihelpers.pxi", line 1136, in lxml.etree._utf8 (src/lxml/lxml.etree.c:37215)<br />
ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes


It seems that the MediaItem is receiving some unacceptable data. This happened when a video title contained some of the aforementioned characters mentioned earlier. For example 'ä' is displayed like this in the ui: 'ä'. Even though I have selected Unicode from the display settings. Plug-ins like YouTube, for instance, do display the characters correctly. I just can't seem to grasp why.

The site that I'm playing with has declared:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"><br />
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="fi" lang="fi"><br />
...<br />
<meta http-equiv='Content-Type' content='text/html; charset=UTF-8' />


Maybe some of you who has more knowledge of these character encoding issues can say in what part of the code should I do what.
2


Sorry, I can't help, but Im having similar or exactly the same problems (depending if I read you correctly). Swedish characters are problematic to show in the infoList. If add for example:
dir.AppendItem(DirectoryItem("noje", "Nöje", ""))
... the App doesnt even show in the list. If changed to o from ö then the App shows in the list...
3


try to replace myvarwithumlaut with myvarwithumlaut.encode("utf-8")
4 



dir.AppendItem(DirectoryItem("noje", u'Nöje', ""))
5


I tried these:


dir.AppendItem(DirectoryItem('TEST$Test', u'åäö', ''))<br />
dir.AppendItem(DirectoryItem('TEST$Test', unicode('åäö', 'utf-8'), ''))


They both result in:

...(Framework) Couldn't find en strings<br />
...(Framework) Couldn't find en-us strings
6

Hm, which editor do you use?

Perhaps it saves the code in latin-1. If not, try adding the following line to the top



# -*- coding: utf-8 -*-
7


Am using TextMate and UTF8...

That line did it! Thank you!
8

Now I keep getting this when trying the decoding on further ui levels:


TypeError: decoding Unicode is not supported
9


Are you perhaps trying to convert a variable that already is unicode?
10

Just so you know, those log lines aren't UTF8 errors. They're framework messages related to localization. You can include strings files for multiple languages in your plug-in. This will allow the plug-in to be automatically localized based on language information provided by Plex (not yet implemented, but will be used within the next few versions). Strings files are JSON dictionaries using key-value pairs, and should be saved in the Strings folder, e.g. /Contents/Strings/en.json. You can then easily localize your plug-in using the _L method defined in PMS.Shorthand, e.g.

from PMS.Shorthand import _L<br />
<br />
localizedString = _L("KeyName")


The framework is logging those lines as it sets the default language to en-us, and your plug-in doesn't include strings for that language.
11


I have been trying to figure out what's the problem here. I found this googling: [http://www.red-mercury.com/blog/eclectic-t...ery-of-the-day/](http://www.red-mercury.com/blog/eclectic-tech/python-mystery-of-the-day/). It could explain the behavior that I'm experiencing. As the page author states:


So maybe the original site that I'm requesting data from is being read as iso-8559-1 and not utf-8. What made my suspicious is the fact that the site doesn't declare any xml document type, like <?xml version="1.0" encoding="UTF-8"?>. Could this be the reason?
12

This is how I got it working. I’m pretty sure it’s not the proper way to do it. But at least I get the ui to display some proper characters:



def utf8decode(s):<br />
  s = s.encode("iso-8859-1")<br />
  return s.decode("utf-8")<br />
<br />
for item in XML.ElementFromURL(PLUGIN_ROOT, True).xpath("//div[@class='foo']"):<br />
  item_name = utf8decode(item.xpath("a")[0].text)<br />
...<br />
  dir.AppendItem(DirectoryItem(item_id+"$"+item_name, item_name, Utils.EncodeStringToUrlPath(thumb), summary))


So basically first I encode the text to iso-8859-1 and then decode it back to utf-8 in order to be able to append the item. Weird? Well, I think so, too. This is what made me try it out: [http://groups.google.com/group/comp.lang.p...0bca2728df55bc6](http://groups.google.com/group/comp.lang.python/browse_thread/thread/00bca2728df55bc6)

I also tried to play with BeautifulSoup to no avail.

Someone has likely a more elegant solution to this.
13

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.