Locale related APIs -- language matching

[zbraniecki]

UPDATE: Latest Proposal - Nov 2015

I started working on a spec and polyfill for Intl.Locale API.

I should have some proposal for this next week. Currently I have five functions on it:

• isStructurallyValidLanguageTag(locale)
• canonicalizeLocaleList(locales)
• resolveLocale(availableLocales, requestedLocales, options, relevantExtensionKeys, localeData)
• prioritizeAvailableLocales(availableLocales, requestedLocales, defaultLocale)
• getDirection(locale)

The rationale for those five are:

• isStructurallyValidTag
  This allows testing language tag for sanitation/validation purposes.
• canonicalizeLocaleList
  This is the core function allowing all language negotiation operations
• resolveLocale

this exposes the core function used by all Intl formatters that allows people to write their own custom formatters or polyfills.

• prioritizeAvailableLocales

This is basic language negotiation function for all localization libraries. It takes requestedLocales list (navigator.languages or custom), availableLocales (app provided locales), and a fallback defaultLocale (default app locale, not default host env locale) and returns the list of prioritized available locales that best matches user requested locales.

• getDirection

Simple function allowing l10n libraries to set html.dir based on their negotiated locale.

@caridy - One question I'd like to get help with is about using Set instead of Array. In all cases where we carry a list of locales, we always want it deduplicated and Set does just that. It would be sweet to use it instead of an Array, but it's API seems to be significantly less user friendly than that of the Array - it's impossible to just retrieve locales[0], it's not easy to map etc.
Should I stick to Array or try to use Set?

[caridy]

Two things:

• You can certainly use a Set internally as part of the logic, but let's stick to an array for arguments, I haven't seen any precedent of using a Set as arguments in Ecma.
• Instead of implementing a new polyfill (once we settle on the api) you can just submit a PR for Intl.js, which already implements those abtract operations.

[zbraniecki]

sure! thanks for the feedback :)

[zbraniecki]

@caridy - I started playing with Intl.js and ended up separating src/locale.js out of src/core.js and then carved out the common functions into src/utils.js because src/core.js is huuge :)
Not sure if you're interested in any sort of refactor like that - if not, I'll bring it back into src/core.js.

Branch is here: https://github.com/zbraniecki/Intl.js/tree/locale

I also wrote a simple polyfill just for Intl.Locale based on Gecko implementation - https://github.com/zbraniecki/IntlLocale

Let's now talk about the API and get it agreed upon (before I start the daunting task of working on the spec)

[caridy]

@zbraniecki I added the module system into Intl.js to be able to split it into small chunks, I didn't get to split core into small chunks. In other words, go for it, we will appreciate it. Just open a PR there, and we can discuss the details.

In the other hand, I'm more interested on the API discussion :)

[zbraniecki]

@caridy Cool. Do you have any comments on the API I proposed in comment 1? I'd like to make Locale a good unified language negotiation API for other Intl APIs and localization frameworks.

I believe that with those five functions are enough to achieve that. On top of that, all of that code except of getDirection is already in Intl API, just not exposed, so it should not cause any cost to the vendors deploying it.

[zbraniecki]

I guess it's not the right idea to at the same time refactor a major piece of code and introduce new feature for debate in it :)

So I updated my locale branch to just introduce IntlLocale to Intl.js polyfill now - https://github.com/zbraniecki/Intl.js/tree/locale

@caridy - does it look like something you may be interested in?

[caridy]

update: we need to cook the proposal for the next meeting in two weeks. will try to coordinate some time to work on this.

[zbraniecki]

We've talked about this with @caridy and he mentioned that Intl formatters may be interested in switching to something like prioritizeLocales for language negotiation.

This would mean that we don't have to expose resolveLocale at all, and just exposing prioritizeLocales will be enough for both - localization frameworks and for custom intl formatters to match the behavior of the platform ones.

The only thing coming to my mind that we may want to use depth parameter to limit the length of the resolved chain - if Intl formatter will only take the first one anyway, there's no reason to resolve the whole chain. At the same time l10n frameworks operate in more uncertainty (l20n for example falls back when a single l10n string cannot be resolved in a given language at runtime) so they need full "depth".

That would leave us with:

• isStructurallyValidLanguageTag
• canonicalizeLocaleList
• prioritizeLocales(availableLocales, requestedLocales, defaultLocale, depth)
• getDirection

(and later)

• firstDayOfTheWeek (or weekendInfo returning firstDay, weekendStarts and weekendEnds)

wrt. prioritizeLocales you can see an implementation here: https://github.com/l20n/l20n.js/blob/v1.0.x/lib/l20n/intl.js#L434

It uses LookupAvailableLocales which is a new function that acts exactly like LookupSupportedLocales but instead of returning a requested locale that has a match in available ones, it returns the available locale that has a match in requested ones.
So if the user requests 'sv-SE' and we have 'sv', LookupSupportedLocales will return 'sv-SE', while the new function will return 'sv'. Which is what l10n and intl can later use to load the right resources.

[rxaviers]

I'm interested to know the exact algorithm and how this relates to https://github.com/rxaviers/ecma402-fix-lookup-matcher please.

[caridy]

@rxaviers that's precisely the reason why we will abstract the operation to produce the parent list, then we can update that to take into consideration the proper algo step.

[caridy]

Abstract Locale Operations - Nov 2015

[caridy]

@rxaviers Ideally we can get https://github.com/rxaviers/ecma402-fix-lookup-matcher to fit into getParentLocales(), in other words, make the lookup matcher fix so we can expose it via getParentLocales(). Can you work with @zbraniecki to get that flushed out?

[rxaviers]

@caridy, note https://github.com/rxaviers/ecma402-fix-lookup-matcher is about finding the bundle, not the parent, which is the first entry of the chain. For example, given the locale zh-TW, you start at the zh-Hant bundle, and then inherits from zh-Hant parents, probably using getParentLocales().

[caridy]

yes @rxaviers, the fact that all those algos will require the inheritance path to make a decision about the data availability, that's the piece we want to expose, so they can do that logic in user-land as well.

[rxaviers]

It's in my TO-DO, but I got few spare time at the moment to go through it. Please, alert me in case I'm blocking you, but I will try to catch up time permitting. Thanks

[zbraniecki]

Here's the initial POC of the patch for Intl.js to implement the abstract locale operations discussed in the slides @caridy linked on Nov 23rd: andyearnshaw/Intl.js@master...zbraniecki:locale

@caridy - does it look good? If so, I'll work on the spec patch.

[caridy]

@zbraniecki let's work on the spec, and then we can review the implementation. In principle, it looks good.

[zbraniecki]

Here's the first draft of the spec - master...zbraniecki:locale-api
I didn't write any algo for resolveLocaleInfo because I'm not sure how to approach accessing the necessary information from CLDR.

[rxaviers]

@caridy @zbraniecki sorry for the long delay on this reply and for my short worded comments above, but the end of the year has been intensive. First of all, great job with the Locale API, it's a good way of exposing these info and will definitely benefit custom libraries and polyfills.

How can I help you to get https://github.com/rxaviers/ecma402-fix-lookup-matcher into getParentLocales()? Note the first draft above gives wrong results [details].

[zbraniecki]

That's for week data:

• http://www.unicode.org/reports/tr35/tr35-dates.html#Week_Data
• http://www.unicode.org/reports/tr35/tr35.html#Key_Type_Definitions

[zbraniecki]

Heads up, this has been advanced to Stage 2 as of today. I'm going to work on updating the spec/polyfill now to @rxaviers and TC39 feedback and will submit it to reviewers in a couple weeks.

[stasm]

CLDR defines groups of supplemental data for each territory, like timeData or weekData. An incomplete table can be consulted here:

http://www.unicode.org/cldr/charts/28/supplemental/territory_information.html

And the full data set here:

http://unicode.org/repos/cldr/trunk/common/supplemental/supplementalData.xml

Perhaps a good future-proof solution would be to base the method names on these groups, e.g.:

Intl.Locale('fr-FR').getWeekData();

which would return:

{
    minDays: {
        count: 4
    },
    firstDay: {
        day: 'mon'
    },
    weekendStart: {
        day: 'sat'
    },
    weekendEnd: {
        day: 'sun'
    }
}

Or:

Intl.Locale('fr-FR').getTimeData();

which would return:

{
    hours: {
        preferred: 'H',
        allowed: 'H hB'
    }
}

And for a different French-speaking territory:

Intl.Locale('fr-CA').getTimeData();

which would return:

{
    hours: {
        preferred: 'h',
        allowed: 'h hb H hB'
    }
}

This is a bit verbose but it maps the CLDR data one-to-one which should make it much easier to extend it in the future.

[rxaviers]

FWIW, find CLDR in the JSON format here https://github.com/unicode-cldr/cldr-core/tree/master/supplemental.

[zbraniecki]

As I mentioned in #6 I landed mozIntl.getCalendarInfo for calendar information in Gecko.

Also, instead of getDirection, we're going to use mozIntl.getLocaleInfo which works very similarly to getCalendarInfo.

I'll be happy to revisit it and move away from that API toward any standard that ECMA402 will come up with.

[jacobrask]

Is there any further discussion on Intl.getCalendarInfo happening somewhere else? How did that API work out for you in Gecko?

[caridy]

We haven't discuss it recently, but probably something that we will get back to at some point.

[brettz9]

Some of the information in the IANA language subtag registry, a registry from which the following may be derived, may be incomplete, but I wonder whether the existing information present could be used to supply getLocaleInfo with info on macrolanguages of encompassed languages?

This would, I imagine, be useful information for i18n as e.g., a user with language "in" (an encompassed language with parent macrolanguage "ms") could be directed to a "ms" locale if an "in" one could not be found. The writing system could conceivably be different, but if the "Suppress-Script" information were available to languages too (see #205), compatibility here could be detected as well.)

[hugovdm]

I'm also looking for something like resolveLocale or prioritizeLocales: an API to call to help doing language matching (wanting to select a best locale based on the user's preferred list of locales and a developer-provided list of locales supported by an app, for example).

[opyh]

https://www.unicode.org/reports/tr35/#LanguageMatching proposes an algorithm to find a best fit for given sets of available and requested languages.

Could this be referenced in the spec - or maybe even integrated?

https://github.com/rxaviers/cldrjs/blob/master/doc/bundle_lookup_matcher.md describes more details about issues with this algorithm.

[sffc]

I see several discussions in this thread. Let's move discussion of Intl.getCalendarInfo() over to #6, and use this thread to focus on the language matcher issues.

[johan]

I'm developing an internal API for Gecko which will be used by various date/time calendars and pickers.

At the moment the proposal for the API to retrieve the calendar data is:

let info = Intl.getCalendarInfo(locales);

which returns:

{
  "firstDayOfWeek": 7,
  "weekendStart": 6,
  "weekendEnd": 1,
  "locale": "de",
  "calendar": "gregory" 
}

[...]

As I said - this is an internal API, but maybe it'll help us design ECMA 402 API for that dataset.

FWIW: if weekday references end up numbers instead of weekday name abbreviation names, as in CLDR (probably a good thing), it would be internally consistent to sync those numbers up with ECMA-262's data type Week Day (the return value from Date.prototype.getDay), ranging from Sun=0 to Sat=6.

[ryzokuken]

Intl.Locale is Stage 4 and the follow-on Intl LocaleInfo is Stage 3. Can this be closed?

[sffc]

Intl.Locale is Stage 4 and the follow-on Intl LocaleInfo is Stage 3. Can this be closed?

No. As suggested in #46 (comment), this issue is about language matching. It would be fixed by the Intl.LocaleMatcher proposal.

https://github.com/tc39/proposal-intl-localematcher