-
Notifications
You must be signed in to change notification settings - Fork 232
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WIP: start l10n tooling #486
Conversation
8fc7c4a
to
8082cb3
Compare
.env-dist
Outdated
@@ -4,6 +4,9 @@ SERVER_URL=http://localhost:6060 | |||
PORT=6060 | |||
COOKIE_SECRET=3895d33b5f9730f5eb2a2067fe0a690e298f55f5e382c032fd3656863412 | |||
|
|||
# available languages for l20n | |||
AVAILABLE_LANGUAGES="de,en,es,fr,it,ja,pt,ru,zh" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this account for en-US, en-CA, en-UK, etc?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should thoroughly test, but the accept-language
module says that get
will "Get the most likely language given an Accept-Language string".
.eslintignore
Outdated
@@ -3,3 +3,4 @@ coverage | |||
loadtests | |||
public/dist | |||
public/js/polyfills | |||
public/js/l20n.js |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oy vey! Any chance we can import via npm versus copy-pasta this file below?
middleware.js
Outdated
|
||
function pickLanguage (req, res, next) { | ||
acceptLanguage.languages(AppConstants.AVAILABLE_LANGUAGES.split(",")); | ||
req.app.locals.locale = acceptLanguage.get(req.headers["accept-language"]); |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code looks like it will default to whatever is the first language tag specified in acceptLanguage.languages
. So in our case - yes.
public/locales/en/app.ftl
Outdated
|
||
terms = Terms and Privacy | ||
|
||
copyright = Portions of this content are © 1998-2018 by individual mozilla.org contributors. Content available under a Creative Commons license |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actual markup below is:
<p data-l10n-id="copyright">
Portions of this content are © 1998-2018 by individual mozilla.org contributors.
<br />
Content available under a <a {{> analytics/link_event eventLabel="Creative Commons"}} href="https://www.mozilla.org/foundation/licensing/website-content/" target="_blank" rel="noopener">Creative Commons license</a>.
</p>
Not sure if the .ftl files should use © entity or if we need anything special for link preserving.
public/locales/de/app.ftl
Outdated
|
||
terms = Allgemeine Geschäftsbedingungen und Datenschutzerklärung | ||
|
||
copyright = Teile dieses Inhalts sind © 1998-2018 von einzelnen mozilla.org-Mitwirkenden. Inhalt verfügbar unter einer Creative Commons-Lizenz |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reasons to use a different repository:
- History: Pontoon will write in the repository every 20 minutes, once for each locale (if there are changes). History is going to become messy very quickly.
- Decouplement: You can develop on master and decide when strings are ready for translation.
Cons:
- You need scripts to import strings back.
- You need to expose strings in the l10n repository via PR.
Alternatively, we can also write in a branch instead of master, but then you need to deal with merges back and forth.
Side note: we'll need a dev server updated frequently for l10n testing. For that it might be easier to have the strings in the main repository.
From the point of view of configuration, if you want Pontoon to write here we just need to add its bot.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@flodolo can you point us to examples of each kind of setup? I.e., same-repo vs. separate-repo?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Screenshots and Test Pilot have strings in the main repo
https://github.com/mozilla-services/screenshots/
https://github.com/mozilla/testpilot/
For Test Pilot we use a l10n branch.
Other products use a repo for strings, but I don't have website examples at hands
strings: https://github.com/mozilla-l10n/firefoxios-l10n
code: https://github.com/mozilla-mobile/firefox-ios
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The one separate-repo I can remember off the top of my head is FxA's content server:
https://github.com/mozilla/fxa-content-server
vs
https://github.com/mozilla/fxa-content-server-l10n
I think the trick was putting a package.json in the *-l10n repo so you can include it as a dependency in your main repo's package.json. But of course, as I went to prove that, I notice that FxA team is using some Bash script[1] to download locales, so we may want to sync up with them to see why they switched to doing it that way (or maybe I'm misremembering how it was done originally).
[1] https://github.com/mozilla/fxa-content-server/blob/master/scripts/download_l10n.sh
Thanks for the CC and for kicking this off.
Instead, we have developed There's an experimental packaged called For language negotiation there's also |
Another good reference for fluent lib usage is the https://github.com/mozilla/send repo. |
0e140e5
to
29aef58
Compare
I looked thru send, screenshots, testpilot, and fluent-web and I prefer fluent-web by a large margin. I tried to simply replace How long would it take to get If not, I'll start digging into using |
does that work? |
I get |
I'm happy to move this over to a fluent-web issue. |
Opened projectfluent/fluent-web#2 |
53016bb
to
cb93013
Compare
Still waiting to see if we can use
This seems to work fine! And it feels simpler than some of the other approaches I saw in send, testpilot, screenshots, etc. Does it seem sane? |
One further wrench into the plan is that the presence of a directory/file isn't always an indicator of translation %%. In other repos (like mozilla/send) we had a script which would only include locales that are 100% translated. I think mozilla/fxa-content-server had a lower threshold (90%) before it would be considered ready for production. |
controllers/home.js
Outdated
@@ -22,7 +23,8 @@ function home(req, res) { | |||
|
|||
function notFound(req, res) { | |||
res.status(404); | |||
res.render("error", { message: "Page not found." }); | |||
const message = req.fluentBundle.format(req.fluentBundle.getMessage("notFound")); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since this is per-user and probably not suited for app.locals.*
, would it be possible to wrap this in some sort of helper function or middleware to cut down on verbosity?
function formatMessage(bundle, message) {
return bundle.format(bundle.getMessage(message));
}
...
const message = formatMessage(req.fluentBundle, "notFound");
email-utils.js
Outdated
@@ -45,6 +45,7 @@ const EmailUtils = { | |||
|
|||
sendEmail(aRecipient, aSubject, aTemplate, aContext) { | |||
if (!gTransporter) { | |||
// TODO: l10n? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this message even user facing, or is this purely server logging?
locale-utils.js
Outdated
|
||
|
||
function loadAvailableLanguages () { | ||
return fs.readdirSync( localesDir ); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure if this needs any footgun protection to capture stuff like .DS_Store or other files which may not be in the locales/* dir. Or if we need to add filter out any results that aren't a subdirectory, etc.
locale-utils.js
Outdated
function loadFluentBundles (availableLanguages) { | ||
const fluentBundles = {}; | ||
for (const lang of availableLanguages) { | ||
const langFTLSource = fs.readFileSync(path.join(localesDir, lang, "app.ftl"), "utf8"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Or if this needs to be in a try..catch
to handle any folders/results which may not be a subdirectory, or there may not be an "app.ftl" file.
views/layouts/default.hbs
Outdated
|
||
<meta name="defaultLanguage" content="en"> | ||
<meta name="availableLanguages" content="{{ AVAILABLE_LANGUAGES }}"> | ||
<link rel="localization" href="/locales/{{ locale }}/app.ftl"> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, because we're changing the views/layouts/default.hbs template, we'll need to decide what level we want to keep that views/error-429.html template in sync (since we have-to/should update the markup in both places).
bb8af38
to
85a5f95
Compare
Addressed all comments except the template-level ones. (Still checking if we'll be able to use Consolidated some of the server start-up code into a single Made the |
The %-checking heuristic seems neat, but how helpful is it? The fxa-content-server check just seems to check that a translated message is non-blank, which ignores up-to-date-ness of the translations; the send check for 100% no errors nor missing translations means (like you say in the PR) that any update to en-US invalidates all translations until they are back to 100%? Either tooling tactic seems to rely on a significant amount of manual translation support anyway. Does the tooling help notify the human translators of the updates needed? |
b82397e
to
0734f7f
Compare
public/locales/en/app.ftl
Outdated
That's good news, but data breaches can happen any time and there is still more you can do. | ||
Subscribe to Firefox Monitor for a full report, alerts when new breaches happen, and tips on protecting your passwords. | ||
|
||
featured-breach-results = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@stasm I'm still running into some selector trouble here in "featured-breach-results". As is, this throws a "ctor is not a constructor" error. If I change [one] to [1], the string will return correctly only if the value of $breachCount is exactly 1. I'm totally puzzled as I recreated this over in the Fluent playground where it appears to be working exactly as expected. Have you run into anything like this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This might be related to the Intl.PluralRules
API. It's still a fairly new feature in JavaScript. Looking at the browser support matrix, is there a chance you're testing on Safari? In order to support Safari as well as older browsers, you'll need to polyfill Intl.PluralRules
. I recommend intl-pluralrules
.
Background: Fluent relies on this API to match variant keys which are not numbers. When you use [one]
Fluent maps the numerical value of $breachCount
to a plural category. Categories are defined in CLDR. For English, they're one
and other
. Other languages have fewer or more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Aha, node.js is probably the culprit here - it doesn't have support for Intl.PluralRules
API. I'll see if I can add that to our package.json
and get this to work.
const path = require("path"); | ||
|
||
// node.js needs Intl.PluralRules polyfill | ||
require("intl-pluralrules"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can't even find a good enough emoji.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice! Happy to hear that was it!
e00ac27
to
d87b36a
Compare
Updated to have |
public/locales/en/app.ftl
Outdated
@@ -0,0 +1,35 @@ | |||
-product-name = Firefox Monitor |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you add a note similar to the one in the add-on, extending it to all brands in this section?
public/locales/en/app.ftl
Outdated
-brand-name = Firefox | ||
-brand-Quantum = Firefox Quantum | ||
-brand-Mozilla = Mozilla | ||
-brand-HIBP = HIBP |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add a note to explain what HIBP stands for.
public/locales/en/app.ftl
Outdated
# Descriptive headline for a column of links where users can give feedback, or get additional information about, Firefox Monitor. | ||
layout-support = Support | ||
# Link that takes the user to a Firefox Monitor survey. | ||
give-feedback = <span class="nowrap">Give Feedback</span> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it possible to avoid exposing this markup to localizers (same for the next string)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep, we can move this to template.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll remove it wherever possible. Unfortunately, for cases like L82 and a few others, it will need to remain as is.
public/locales/en/app.ftl
Outdated
error-not-subscribed = This email address is not subscribed to {-product-name}. | ||
error-hibp-throttled = Too many connections to {-brand-HIBP}. | ||
error-hibp-connect = Error connecting to {-brand-HIBP}. | ||
error-hibp-load-breaches = Could not load breaches. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
breaches -> data breaches?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think L16-L18 are actually log errors and may not need to be localized at all. @groovecoder do we want to localize these?
public/locales/en/app.ftl
Outdated
error-hibp-connect = Error connecting to HIBP. | ||
error-hibp-load-breaches = Could not load breaches. | ||
|
||
hibp-notify-email-subject = {-product-name} Alert : Your account was involved in a breach. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Still open question. Is it because it's the title of a web page?
public/locales/en/app.ftl
Outdated
And hackers rely on bad habits, like using the same password everywhere or using common phrases (p@ssw0rd, anyone?), | ||
so that if they hack one account, they can hack many. Here are six ways to protect your accounts. | ||
|
||
pwt-headline-1 = Use a Different Password for Every Account |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I pointed out before that this approach doesn't scale. Group headline and summary with meaningful IDs, don't use numbers.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think in this case the numbers are meaningful because the password tips are specifically presented in the order that they are numbered. Additionally, the tips are displayed alongside corresponding icons which are also numbered accordingly. I will happily defer to your judgement here if you think this is going to cause future problems though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Regarding the colon spacing on L13 (for some reason I can't respond to that comment directly): We just received updated copy for the emails and the extra space has been removed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Additionally, the tips are displayed alongside corresponding icons which are also numbered accordingly.
How is that done? Have you considered other types of numerals?
I will happily defer to your judgement here if you think this is going to cause future problems though.
Two likely scenarios:
- You want to change the order.
- You want to replace or update one of the tips. You'll need to change the message ID to do that.
In both cases you'll need to add a mapping afterward.
Also, strings are presented sequentially to localizers, so having title and description together help a bit.
public/locales/en/app.ftl
Outdated
-brand-Quantum = Firefox Quantum | ||
-brand-Mozilla = Mozilla | ||
-brand-HIBP = HIBP | ||
-product-name-nowrap = <span class="nowrap">Firefox Monitor</span> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wouldn't it be easier to use a non breaking space in the main brand name, and always use that?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Potentially - we've discussed this in previous pull requests and my understanding is that is deprecated and there are a11y concerns with using it.
public/locales/en/app.ftl
Outdated
form-signup-error = Must be a valid email. | ||
|
||
|
||
found-breaches-headline = This could be a problem... |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use single unicode character …
public/locales/en/app.ftl
Outdated
}. | ||
|
||
scan-results = Your account { $breachCount -> | ||
[0] Your email address did not appear in our basic scan. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@stasm @zbraniecki Thoughts on using Fluent like this, also in the previous string?
I really think we should split out the 0
case. If nothing else for practical reasons: it would be impossible to translate this in Pontoon, where there no text area for variants.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@flodolo thanks for the feedback. Is it possible to translate any variant in Pontoon, or is there a specific issue with the zero case?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is perfectly fine Fluent, it's just tricky for localizers to understand what we're asking them to translate.
It becomes easier if we're exposing the full phrase for each variant.
There's things we'll want to do on the l10n tooling side there in the next year, but right now, the extra repetition of the head and tail section would be good.
single-featured-breach-result = Your account appeared in the <span class="bold"> { $featuredBreach } </span> breach, but did not appear in any other known data breaches.
multiple-featured-breach-results =
{ $breachCount ->
[one] Your account appeared in the <span class="bold"> { $featuredBreach } </span> breach, as well as { $breachCount } other breach.
*[other] Your account appeared in the <span class="bold"> { $featuredBreach } </span> breach, as well as { $breachCount } other breaches.
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Pike thanks for the clarification
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What Pike said :)
The UI looks like this. The fragment before would become another input field, and you would need to drag the input field to see everything.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To be clear, I would do this
featured-breach-no-results =
Your account appeared in the <span class="bold"> { $featuredBreach } </span> breach,
but did not appear in any other known data breaches.
featured-breach-results =
{ $breachCount ->
[one] Your account appeared in the <span class="bold"> { $featuredBreach } </span> breach, as well as { $breachCount } other breach.
*[other] Your account appeared in the <span class="bold"> { $featuredBreach } </span> breach, as well as { $breachCount } other breaches.
}
scan-no-results =
Your email address did not appear in our basic scan.
That's good news, but data breaches can happen any time and there is still more you can do.
Subscribe to Firefox Monitor for a full report, alerts when new breaches happen, and tips on protecting your passwords.
scan-results =
{ $breachCount ->
[one] Your account appeared in the following breach.
*[other] Your account appeared in the following { $breachCount } breaches.
}
Note also that your current string would result in Your account Your email address did not appear in our basic scan. etc
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I really think we should split out the 0 case. If nothing else for practical reasons: it would be impossible to translate this in Pontoon, where there no text area for variants.
From the Fluent perspective, I think using [0]
like this is fine and it adds flavor to the translation. The Good Practices document has more discussion about this.
However, if Pontoon doesn't support extra variants right now, we should avoid it (for now). Sorry, @lesleyjanenorton! We'll get there eventually :)
public/locales/en/app.ftl
Outdated
|
||
# copyright-info (without markup) = Portions of this content are 1998-2018 by individual mozilla.org contributors. Content available under a Creative Commons license. | ||
copyright-info = | ||
Portions of this content are <span class="copyright-symbol">©</span> 1998-2018 by individual mozilla.org contributors. <br /> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please no HTML entities, use the utf-8 characters (©)
public/locales/en/data-classes.ftl
Outdated
email-messages = Email messages | ||
employers = Employers | ||
ethnicities = Ethnicities | ||
family-members'-names = Family members' names |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm pretty sure the '
in the ID is a parsing error. The string, on the other hand, should use an apostrophe ’
b0dc3a8
to
a36bf1c
Compare
After @lesleyjanenorton's updates, I added commits to remove the test I've also pushed this branch directly to https://fx-breach-alerts.herokuapp.com/ so we can see this code running live before our upcoming meeting. (Note: @lesleyjanenorton I'm getting |
I'm not exactly sure if that works reliably, or for Fluent projects. @mathjazz ? |
In theory it works, but requires additional testing. AFAICT, it hasn't been used in combination with a Fluent. I suggest we don't block enabling localization on it and enable in-place localization as a follow-up. |
# - Declined to adapt to grammatical case. | ||
# - Transliterated. | ||
# - Translated. | ||
-product-name = Firefox Monitor |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you're making this Firefox{"\u00A0"}Monitor
(aka nbsp), you probably get around the dom fragments and the nowrap, and that might also be good for a11y?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Filed #535 for a follow-up on this.
I'm merging this now, as it touches so many files and is preventing us from doing much else until it's merged. Filed #533 for email localization, #534 for the heroku/in-page localization follow-up, #535 for They are all in the |
No description provided.