Non-ASCII characters in translations

Dev Community Dev/API Corner
1

My apologies if this is a simple question - I'm familiar with text encodings, but not so much with the Python/Plex ecosystem.

 

I'm preparing my plugin for localization - and what better way to do that than to translate it into your native language? In casu: Danish, which has its own, weird, non-ASCII characters "æ", "ø" and "å"

 

In my da.json translation file, I put in the following translation:

"CHILDREN" : "Børn"

This makes Plex/Python/... very unhappy:

  File "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server/Plug-ins/Framework.bundle/Contents/Resources/Versions/2/Python/Framework/modelling/objects.py", line 71, in _set_attribute
    el.set(convert_name(name), value)
  File "lxml.etree.pyx", line 699, in lxml.etree._Element.set (src/lxml/lxml.etree.c:34531)
  File "apihelpers.pxi", line 563, in lxml.etree._setAttributeValue (src/lxml/lxml.etree.c:15781)
  File "apihelpers.pxi", line 1366, in lxml.etree._utf8 (src/lxml/lxml.etree.c:22211)
ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters 

So I try Unicode:

"CHILDREN" : u'Børn'

Doesn't work either:

2015-01-29 12:50:20,713 (7f8af7fff700) :  WARNING (data:179) - Error decoding with simplejson, using demjson instead (this will cause a performance hit) - Expecting value: line 7 column 35 (char 285)
2015-01-29 12:50:20,715 (7f8af7fff700) :  ERROR (localization:456) - Exception when trying to load da strings 

So I try an HTML entity reference:

"CHILDREN" : "Børn"

At least this doesn't cause any error messages - but the translated text shows up exactly as written, i.e. in the UI (iPad) I see "Børn", not "Børn".

 

Garh! How do I get these weird characters in there?

2

OK, I think I got it figured out!

In the first, intuitive scenario

"CHILDREN" : "Børn"

it turns out that the translation is read OK - i.e.

localized_text = L("CHILDREN")

works just fine.

The problem comes when I create e.g. an ObjectContainer:

oc = ObjectContainer(title2=localized_text)

Instead, Python (or the ElementTree library?) wants me to do

oc = ObjectContainer(title2=localized_text.decode())

because the string returned by L() is a something-encoded unicode string, which decode() turns into a real unicode string.

My head spins, but the decode() call made it work...

3

@schaumburg

Thanks for the suggestion! I've been banging my head against this problem for quite some time, but it never occurred to me to try decode() without arguments.

I ended up implementing my own 'shortcut' method in the __init__.py in order to avoid changing calls to the framework's L() function everywhere in the file:

def L(string):
    local_string = Locale.LocalString(string)
    return str(local_string).decode()
4

Shouldn’t this be considered a bug on Plex?


We should be able to use the returned value of L() in any place including as an attribute of any Plex Object (title, summary, …, whatever).


That piece of code used on init.py solves the problem so if Plex core mofifies L() function code that should solve the problem too.


@schaumburg @czukowski Does localization works for you? I’m really using it on code just in case someday it is implemented correctly but anyway I have to use the en.json file to include the translations of the language I want on my plugins. There is no way to set the locale (with Plex Locale API) and at least in my case all the clients I tested (Plex Web, Plex Home Theater, Plex for Samsung TV) set the Locale of the plugin to the default en-us ignoring the configured language and ignoring the browser language too (in plex web). So Plex Web for example shows its GUI correctly translated but plugins are not translated at all. Same for Plex Home Theater. Same for Plex for Samsung TV. If you are able to set the locale, what exactly do you use to set it? Thanks.

5

@manfer

I had problems getting the localization to work (everything would be en-us), until I figured this out: you need to resolve the text in the context of a call, not in global variables.

I.e. in the following:

en_us_value = L("Some label")

def Start():

   localized_value = L(“Some label”)

the en_us_value will always be the default (en_us) localization, whereas localized_value will use the client locale.

Reason: the initialization of en_us_value = L("Some label") happens outside the context of any client call, whereas localized_value = L("Some label") is in the context of a specific call.

This is an under-documented feature of the localization mechanism... :)

EDIT: I've tested this with the iPad and Android clients - don't recall any others. Caveat lector.

6

Thanks @schaumburg, but sadly it is not the issue I’m facing.


I only use localization inside functions, in the starting point for the plugin that renders the main menu and inside callbacks. I’m almost 100% sure I do localization correctly.


Besides. In my case it is not that what I code is not localized. None of the channels is localized. If I install for example youtube channel and then I add a translation for my language it won’t localize either. When I look into any channel log I see how the localize module is always initialized with en-us.


I have the server installed on Windows and in the Plex Media Manager Configuration I have any option I can configure for Plex Web and Plex Server to my own language (I don’t know any other place for configuration and anyway in my opinion the localization should be a client side options so Plex Web running on a browser should already use the browser locale). The client GUIs are properly localized but channels do not. The clients I tested are: Plex Web, Plex Home Theater and Plex for Samsung TV. I have no other clients to test. None shows the channels with translations but in english.

7

Hmm, I got localization to work for my MythRecordings channel (the latest 0.5beta on github should have the working localization).

[My preferred client for testing is iPad, so the following is specific to that]

I had to set the device language to Danish on the iPad (and restart the client, I think), but then it worked.

Our primary "production use" client is a Roku3, but to be honest I don't remember if that works - being such a small language, we're pretty much used to English user interfaces (to the extent that it doesn't register)

But to sum up: it does work, at least from the iPad client.

(I'll try to remember testing with the PHT + web clients)

8

@czukowski
I also stumbled over a localise-problem yesterday. It was solved with your suggestion to use an own L-function (https://forums.plex.tv/discussion/comment/838061/#Comment_838061).
Against that: the suffix-solution by schaumburg (.decode()) don’t works for me.
Thanks!
/R

######Plugins: ARDMediathek2016, Plex-Plugin-KIKA_und_tivi, Plex-Plugin-3Sat_Mediathek, Plex-Plugin-Flickr, Plex-Plugin-TagesschauXL, Plex-Plugin-Phoenix_Mediathek, TuneIn2017