Another localization question

[czukowski]

Hi all, maybe this information will be useful:

 

After finding the Localization question thread, looking at the Strings directory docs and some debugging, I may have found out why my localization strings didn't work.

 

When the Locale object determines the current locale, it ultimately requests the locale property of ExecutionContext class instance (Framework.bundle/Contents/Resources/Versions/2/Python/Framework/code/context.py). Looking at the code reveals that it actually returns a Request header named 'X-Plex-Language':

  @property
  def locale(self):
    return self.get_header(Framework.constants.header.language)

Plex Home Theater (Windows) does send this header and in my case it's 'rus', when I would expect it to be 'ru' or 'ru-ru' from what it says in the docs. Indeed, renaming the JSON file helped immediately and all the strings are now localized.

 

Now the question is, whether is documentation wrong, or PHT sends invalid value. How other Plex clients use this header?

 

PS: Plex channels that I've seen all use the 'two-letter' language ISO, that makes me think it's rather the PHT problem.

[manfer]

Hi all, maybe this information will be useful:
 
After finding the Localization question thread, looking at the Strings directory docs and some debugging, I may have found out why my localization strings didn't work.
 
When the Locale object determines the current locale, it ultimately requests the locale property of ExecutionContext class instance (Framework.bundle/Contents/Resources/Versions/2/Python/Framework/code/context.py). Looking at the code reveals that it actually returns a Request header named 'X-Plex-Language':

  @property
  def locale(self):
    return self.get_header(Framework.constants.header.language)

Plex Home Theater (Windows) does send this header and in my case it's 'rus', when I would expect it to be 'ru' or 'ru-ru' from what it says in the docs. Indeed, renaming the JSON file helped immediately and all the strings are now localized.
 
Now the question is, whether is documentation wrong, or PHT sends invalid value. How other Plex clients use this header?
 
PS: Plex channels that I've seen all use the 'two-letter' language ISO, that makes me think it's rather the PHT problem.

So Plex Home Theater is sending the locales maybe on this format
https://msdn.microsoft.com/en-us/library/cdax410z(VS.71).aspx

instead of the expected 2 characters country code or what would be even more correct to use for a language the 4 characters language/region code.

The 4 characters (or similar) is the correct way to go (or the more common way to do it anyway) for language because there are lots of countries with more than one language and different countries with same language (with its own region differences) either.

I already tried creating a JSON file with the 3 characters once but I only tested in Plex Web. I'm going to try it now. These message had sed some light for me on what's going on with localization in Plex for Windows.

I agree that something should be done by Plex to correct how Windows Clients sends the locale (I bet *NIX ones sends it properly. At least from other comments I have read on these forums it looks like Plex for IOS and Plex for Android work correctly for example). In my opinion the best to do would be to unify criteria and use always a 4 characters language/region code to identify language:

en_US
en_AU
es_ES

with the region in uppercase or in lowercase would be fine either (the server could just turn the string into lowercase before analyzing the locale sended by client anyway).

And the server would serve the corresponding language looking for the 4 character (or whatever it receives as looking into w3org I realize it is not always 4 characters) language/region JSON with fallback to language JSON if not available and fallback to default language otherwise. So if client send es_AR first look for es_AR.JSON or es_ar.JSON falling back to es.JSON if not available and to en.JSON otherwise.

It looks that not only client developers have to correct how they send the locale but server developers enhance how to handle it either.

Well, it is not really an easy subject. The language/region is not always 4 character. The ideal would be to handle it in the same way browsers do. Maybe following what w3 standards have to say about it. This reading could be a starting point:
http://www.w3.org/International/articles/language-tags/
which explain the IANA language tags registry.

[manfer]

Plex Home Theater (Windows) does send this header and in my case it's 'rus', when I would expect it to be 'ru' or 'ru-ru' from what it says in the docs. Indeed, renaming the JSON file helped immediately and all the strings are now localized.

How do you capture the request coming from Plex Home Theater? I'm not able to think of a way of doing that just now. I tried a 3 character code for my language and sadly didn't work either so I would like to capture the headers sended by Plex Home Theater just to check which value it is sending.

And I can confirm that the X-Plex-Language header is the clue to all the issue. If I inject that header in Plex Web request with the locale I want (2 characters language code) then it receives the correct translations. Well, in fact I could just inject as value foobarbaz and it will correctly load the translations included in a foobarbaz.json file. So the server doesn't look to do many effort. Just looks for the sended value or fallbacks to default language if not json file available for that value.

I would like to know which value, if any, is sended by Plex for Samsung TV. So I'm insterested on the way to capture Plex Home Theater request just in case is valid to capture this one either. Though I bet in this case it sends nothing at all as the Plex for Samsung TV client doesn't have anything to configure region and I doubt it gets any value from the TV configuration. But at least if I can review its headers and confirm it is not included I can try requesting the developer to implement it.

[manfer]

How do you capture the request coming from Plex Home Theater? I'm not able to think of a way of doing that just now. I tried a 3 character code for my language and sadly didn't work either so I would like to capture the headers sended by Plex Home Theater just to check which value it is sending.

I don't know if there is an easier way but I have decided to use a packet sniffer that allowed me to sniff packets in the loopback and I have been able to read the value for the X-Plex-Language header sended by Plex Home Theater.

I can confirm Plex Home Theater uses ISO 3166-1 codes

https://en.wikipedia.org/wiki/ISO_3166-1_alpha-3

On my previous test I made a mistake with the 3 characters code for my language.

It would be great if Plex Home Theater for Windows solves this by sending a proper language code. The ISO 3166 code is not the one expected by Plex Server.

I will try to check the headers sended by Plex for Samsung TV later.

[czukowski]

I used something like Log(Request.Headers)   to see what client is sending.    

3-letter language codes are alright, there's a standard for that, but so there is for the 2-letter codes and the real issue is whether different Plex clients use it consistently. So far I can tell that Plex Home Theater uses 3-letter code in X-Plex-Language header and Plex Web uses 2-letter code in Accept-Language header.

From the Localization API I would say 2-letters code are the correct way to send the client language. And I think I even saw a conversion function for language codes between 2 and 3 letters somewhere inside the framework, so, in theory, the locale detection function in Plex could be somewhat 'smart' and understand either way.

[manfer]

I used something like Log(Request.Headers)   to see what client is sending.    

3-letter language codes are alright, there's a standard for that, but so there is for the 2-letter codes and the real issue is whether different Plex clients use it consistently. So far I can tell that Plex Home Theater uses 3-letter code in X-Plex-Language header and Plex Web uses 2-letter code in Accept-Language header.

From the Localization API I would say 2-letters code are the correct way to send the client language. And I think I even saw a conversion function for language codes between 2 and 3 letters somewhere inside the framework, so, in theory, the locale detection function in Plex could be somewhat 'smart' and understand either way.

You are right there is already a Plex API and it uses 2-letters so that's exactly what Plex Clients should be sending.

It is not the best for the reasons I said before and that is why w3 standards are a lot more complicated on the language codes standard they use and browsers Accept-Language header can be a lot more complicated that just 2-letters. The two letters language code (ISO 639-1) is not enough because it can't handle all world languages. It even doesn't have a language so largely spoken as brazilian portuguese, which would be pt_BR in other standards. Plex locale api had decided to use pb for that language, though as far as I know there isn't any standard using pb code. There would be still examples of languages not available in two letters code.

Anyway the locale API is what it is so it is just fine if Plex Client developers stick to it.

As for now and for the tests made it looks that clients are responsible of sending X-Plex-Language header and with proper two-letters as specified by Locale API.

* Home Plex Theater (Windows at least, I don't know if it is available for other platforms) has for that reason a bug. If sends the X-Plex-Language header but not correctly codified.

* Plex Web has a bug too. It doesn't send the header X-Plex-Language.  It makes no sense to force the server to handle Accept-Language if it ask for other header. Plex Web sends some other headers that look like X-Plex-xxxxx so there is no reason why it can't provide X-Plex-Language too, even if it is just the same value as Accept-Language header.

[manfer]

I would like to know which value, if any, is sended by Plex for Samsung TV. So I'm insterested on the way to capture Plex Home Theater request just in case is valid to capture this one either. Though I bet in this case it sends nothing at all as the Plex for Samsung TV client doesn't have anything to configure region and I doubt it gets any value from the TV configuration. But at least if I can review its headers and confirm it is not included I can try requesting the developer to implement it.

Plex for Samsung TV doesn't send X-Plex-Language header and that's why localization doesn't work on that client either.

I expected being that way as that client doesn't even have a localized GUI. So was very unlikely it were doing anything about channels localization.

[czukowski]

My personal guess and a little bit of experience tell me that it is not uncommon to see the localization part being less than perfect in frameworks like this, especially when it's developed by the US based company or a team. The other countries are not a large market for them and the feature doesn't seem to be so important for the framework developers.

You (and me) may either say "ok, having my channel in English in most of the players isn't the worst thing in the world" and live with it or try to come up with some workaround. For example, language selection in the channel settings (will probably be 'global' for all clients) or trying to save the language in a cookie (hopefully the common Plex players will keep them long enough), or maybe try some other tricks: would it be possible to set the X-Plex-Language header manually by guessing the language from the Accept-Language header or some other factors?

[manfer]

My personal guess and a little bit of experience tell me that it is not uncommon to see the localization part being less than perfect in frameworks like this, especially when it's developed by the US based company or a team. The other countries are not a large market for them and the feature doesn't seem to be so important for the framework developers.
 
You (and me) may either say "ok, having my channel in English in most of the players isn't the worst thing in the world" and live with it or try to come up with some workaround. For example, language selection in the channel settings (will probably be 'global' for all clients) or trying to save the language in a cookie (hopefully the common Plex players will keep them long enough), or maybe try some other tricks: would it be possible to set the X-Plex-Language header manually by guessing the language from the Accept-Language header or some other factors?

 
I don't think we can hook on the headers of requests sended by the client to the server in channel code.
 
Those request comes from our coded callbacks in channels. And the arguments you set to those callbacks other than the callback function name itself are the parameters you want your callback function to receive.

 
And I thought we wouldn't be able to try something in channel code either because I tried in the past with no luck.

The Request.Headers from API looks to me something for read-only purpose. Besides even if we could write that value I bet we would need to do it for each callback which doesn't look good practice.

Maybe I tried setting Locale.CurrentLocale in the past and that doesn't work or maybe Plex code have been enhanced or I didn't tried it correctly. Anyway now it seems we can do it by setting Locale.DefaultLocale
 
So, just an enum  in Prefs: 

[
	{
		"id": "language",
		"label": "Language",
		"type": "enum",
		"values": [
			"English/en",
			"Spanish/es",
			"Russian/ru",
			"French/fr"
		],
		"default": "English/en"
	}
]

 
And in the channel Start function: 

  Locale.DefaultLocale = Prefs["language"].split("/")[1]

Seems to work. And yes, doing it this way I bet it is a global setting for all clients. Besides if clients send correctly the X-Plex-Language (which I suppose sets Locale.CurrentLocale) should have preference over this setting.

[czukowski]

Well.. I was curious and apparently this works:

    Request.Headers['X-Plex-Language'] = 'Test'
    Log('Current locale: %s' % Locale.CurrentLocale)

Outputs:

2015-03-04 21:39:58,654 (1d5c) :  INFO (logkit:16) - Current locale: Test

[manfer]

Well.. I was curious and apparently this works:

    Request.Headers['X-Plex-Language'] = 'Test'
    Log('Current locale: %s' % Locale.CurrentLocale)

Outputs:

2015-03-04 21:39:58,654 (1d5c) :  INFO (logkit:16) - Current locale: Test

That would be even better if we could set it on Start function and it worked for any request to the channel. But I tested and as expected it won't. You would have to set the X-Plex-Language header at the beggining of each of your callbacks. Which is not practical.

Semantically looks better as the current locale is what we would like to change, not the default locale.

Anyway, depending of what you want to do can be necessary one approach or the other.

If you want the setting in the preferenes of your channel to be the one to be used, no matter what, then you would have to set the X-Plex-Language header at the beginning of all your callbacks:

  Request.Headers['X-Plex-Language'] = Prefs["language"].split("/")[1]

so you overwrite the value even if the client sends the header.

 

And if you only want the locale on the preferences of your channel to be used in case the X-Plex-Language header is not sended by the client then you would be ok just setting the default locale on the start function once or setting the X-Plex-Language header in each callback only if it is not set already. Just checking if Locale.CurrentLocale is None (Locale.CurrentLocale is read-only)

 

Setting the default locale in start function seems the most practical to me. It even works for Plex Home Theater that sets XXX wrong value for X-Plex-Header because if the Server doesn't find XXX.json it uses the default locale. Same for any other client sending a value for which there is no translation.

[czukowski]

I've been playing with the idea and I came up with a little "patch" that may be added to a channel and will try to fix this issue. Here's the link to the repo:

https://bitbucket.org/czukowski/plex-locale-patch

I've tried that with my channel and now it suddenly appears translated not only in Plex Home Theater and Plex Web, but even when I look at its raw XML in a browser! All of it after adding only one or two lines to the channel code.

See Readme (or the source code) in the repository for more detailed description.

p.s.: I took the liberty of marking this post as a solution.

[manfer]

I've been playing with the idea and I came up with a little "patch" that may be added to a channel and will try to fix this issue. Here's the link to the repo:

https://bitbucket.org/czukowski/plex-locale-patch

I've tried that with my channel and now it suddenly appears translated not only in Plex Home Theater and Plex Web, but even when I look at its raw XML in a browser! All of it after adding only one or two lines to the channel code.

See Readme (or the source code) in the repository for more detailed description.

p.s.: I took the liberty of marking this post as a solution.

Take in mind though the hard work is done only once you are doing a function call to initialize_locale on each call to L function. And besides, don't know exactly why, you have decided to use Request.Headers as a global store. Storing in it the value of a flag you wanted to check later in each call to initialize_locale. That doesn't look to me a good practice at all.

But that is not the worst part. You can try to correct that.

The worst part is that it is still not bulletproof.

If I recall correctly you said Plex Home Theater sends X-Plex-Language header and not Accept-Language. So only the X-Plex-Language is going to be taking into account in that case.

If I recall correctly too you said in your case it sends "rus" as the value.

So, indeed, it is working for you because Locale.Language.Match("rus") matches "ru" correctly. But that doesn't mean the same will happen to the rest of locales. For example "bra" will not match and returns "xx" instead of the "pb" value expected by Plex Locale API. It is not even able to match "esp" to "es". I have not tested but almost 100% sure won't match "prt" to "pt". In resume, Locale.Language.Match is not meant to turn ISO 3166 codes into Plex Locale API codes (which is mostly ISO 639-1 but not exactly).

So Plex Home Theater issue will not be solved with that code.  In some cases? Yes, probably there are more cases like "rus" that match correctly. But I bet many will fail as well. Probably more than the ones that work.

About Plex Web and if you parsed correctly Accept-Language into two characters it will. But still won't work in all cases like portuguese do brazil because "pb" used in Plex Local API is not a standard language code. Don't know if any other of the values in Plex Locale API are not standard codes. Though it seems only "pb" is the intruder and if you parse the first two characters of any language code in Accept-Language those will match the values in Plex Local API correctly.

So Plex Web issue is better mitigated by your code. Almost 100% I bet.

----------------------

I just prefer to include a configuration option in the channel code to choose from one of the languages the channel is translated into and use that preference to set the Locale.DefaultLocale just once on channel start function as described previously. Everything will work that way. And any client that has Plex Locale API implemented correctly will work as intended without interference by that code.

And anyway the only solution is that Plex Clients developers and/or Plex Core developers correct this to make it work as expected. Because, at least in my case, I don't want just any channel I code localized but any channel I install from other developer localized too and I'm not going to edit other channels code, amount other things, because my changes will be lost on each update. :(

[czukowski]

I agree, using Request.Headers is a hack, but it's a "storage" that exists only in the request context (and I haven't found any more suitable one), meaning that different clients connected to your channel at the same time will get correct translations even if they request different languages. I still have to use it to correct the language value though, and given the results, I could say that the ends justify the means. And by the way, even though the `initialize_locale()` function is called every time, only the first time being called during the request it processes the headers, then it just returns doing nothing.

Also, I'll stress that again, this is not a fix for general channel users, but the channel developers. In order to fix it for all channels globally, the framework itself has to be updated. I wouldn't mind sending a pull request, but I don't see the Plex framework out in the public on Github or such.

As for the "really incorrect" values that will not be matched by `Locale.Language.Match()`, it could be fixed easily for the known cases by adding a special case. Add a ticket to the issue tracker if you confirm that PHT sends locale values that can't be matched to a language code. If you haven't already, you can check how that function works by viewing the file `Framework.bundle/Contents/Resources/Versions/2/Python/Framework/components/localization.py`.

And of course, it does not prevent anyone from setting the default locale using the channel preferences, these approaches can work together perfectly.

[manfer]

I agree, using Request.Headers is a hack, but it's a "storage" that exists only in the request context (and I haven't found any more suitable one)

 
I thought it could be done differently but I can't think of other place either. I was thinking of a global or a static but those are not in the request context.
 
About the X-Plex-Language parsing I bet you have to code a match function from scratch. I don't know exactly why "rus" matches with Locale.Language.Match but taking into account it is not able to convert "esp" into "es" makes me think it won't work in many of the values sended by Plex Home Theater.
 
I suggest creating a dictionary just in the file or in a external json file with the matches you already know or the ones you want to include. You can include ISO 3166 codes -more specifically ISO 3166-1 alpha 3- as Home Plex Theater is using mostly that. Well, to be honest it is imposible to know what the hell is exactly using Plex Home Theater. It is not exactly 3166-1 alpha 3 (which is a country code and not a language code by the way), but it is not 639-2 alpha 3 either. So maybe starting with the values you know is just fine and let others add values if they want. You could test which value is sending Home Theater for each locale in its configuration options but that would be tedious. Here is an starting point if you want. 

matches = {
	"chi": "zh",
	"deu": "de",
	"eng": "en",
	"esp": "es",
	"fra": "fr",
	"hin": "hi",
	"ptg": "pt",
	"rus": "ru"
}

The match function would be a simple:

def match(code):
  if code in matches:
    return matches



  return "xx"</pre>
<p>For sure you need that matching function. I don't know what Locale.Language.Match do but it doesn't fit your needs. Your function needs to be able to convert Plex Home Theater values into Plex Locale API language codes.</p>

<p>PD: You would need to include in the dictionary all the Plex Locale API language codes too. "en": "en", "es": "es", ..., so it matches for clients that send a correct value in X-Plex-Language header.</p>

[czukowski]

I don't know exactly why "rus" matches with Locale.Language.Match but taking into account it is not able to convert "esp" into "es" makes me think it won't work in many of the values sended by Plex Home Theater...

It'll match many more, there is a conversion table in the file I've mentioned earlier (`localization.py`), although it doesn't have "esp" and "bra". It would make sense to create a dictionary for those codes that PHT or other clients are known to send, but the original `Locale.Language.Match()` function cannot recognize. But it doesn't substitute the need to try to reach the PHT developers with the bug report about that :)

[manfer]

It'll match many more, there is a conversion table in the file I've mentioned earlier (`localization.py`), although it doesn't have "esp" and "bra". It would make sense to create a dictionary for those codes that PHT or other clients are known to send, but the original `Locale.Language.Match()` function cannot recognize. But it doesn't substitute the need to try to reach the PHT developers with the bug report about that :)

Ok, I looked now into that file and I see the 639_3 dictionary (with a made up "pob" value, at least). It is very curious how I found without looking into that table and just thinking about common languages 3 that didn't match: esp, ptg, bra. And later even found deu, fra.

Now at least I know why those don't match and beside I can say almost 100% sure Plex Home Theater is using ISO3166 (which is a country code, not a language code) with some corrections to be able to code languages that are not available in ISO3166, like english.

So, what a mess.

Thank god the server is not using something more complex like BCP47 (RFC5646)

[czukowski]

Well, it's weird sending a country code instead of a language code, even though the software lets you select language and not country in its configuration (or is it? I'm not at home right now so can't check). I'll try to post in the PHT forum about it.