Use UTF-8 Encoding by Default On Windows On Non English Languages #29755

nagilson · 2023-01-05T01:03:40Z

Fix for: #8833, #24757, #29151, and @rawieo's remark: microsoft/vstest#821

Before:

After:

The main fix is setting the CHCP and OutputEncoding to UTF8 by default on Windows, if a custom language is set.

Things to know:

Make sure this works in CMD and Powershell (Works in Powershell, no easy way to test it in CMD. Hasn't been tested, but Powershell seems to the trickier one because it has its own custom preference variable)
Make this take effect if the system language is not english as well. Note that this could be weird if VS has different behavior, but VSLANG is respected above OS lang.
Add tests if possible ( I did not add tests, they would be a massive pain. There is a fairly close attempt in PR history But we would have to change the SDK to make it testable.)
dotnet new duplicated a lot of the old code... but it's also fixed because this runs through Main in the CLI. We could consider removing that code, but I didn't to lower risk of breaks.
MSBuild does not respect the environment variables so that's still english ATM. We should do the work to align those once this is approved, I can migrate/copy this code to MSBuild. MSBuild should respect upstream tool language requests msbuild#1596
dotnet --debug will have broken encoding, because it runs before the encoding is set. Otherwise the debugger would not be able to attach to the encoding changing code. We don't expect many people to be impacted by this, probably not worth fixing.
UTF8 was not super-officially supported until Win10. We should only do this if it's supported to not break Win7 below, especially.

…e default

KalleOlaviNiemitalo · 2023-01-05T12:01:12Z

Does this affect the emoji that dotnet watch outputs? The OEM codepage of the culture might not include those characters.

sdk/src/BuiltInTools/dotnet-watch/Internal/ConsoleReporter.cs

Line 61 in 882a576

public virtual void Error(string message, string emoji = "❌")

nagilson · 2023-01-05T22:38:24Z

Does this affect the emoji that dotnet watch outputs? The OEM codepage of the culture might not include those characters.

sdk/src/BuiltInTools/dotnet-watch/Internal/ConsoleReporter.cs

Line 61 in 882a576

public virtual void Error(string message, string emoji = "❌")

@KalleOlaviNiemitalo That's a really good point. I was considering setting the page to UTF-8. It does indeed break the emoji in certain languages. UTF-8, meanwhile, is fine, though it's somewhat concerning in legacy scenarios.

KalleOlaviNiemitalo · 2023-01-05T23:26:14Z

I suspect any Console.OutputEncoding change should be limited to Windows only. Is the original problem reproducible on other operating systems?

nagilson · 2023-01-05T23:41:32Z

Not sure, but you're right in that InputEncoding doesn't work for certain System.Runtime.Versioning.UnsupportedOSPlatform, like tvos or android, so I am planning to see what happens when you try to use it on an unsupported OS before merging. Thanks for taking a look.

KalleOlaviNiemitalo · 2023-01-06T00:12:12Z

I mean, if I'm using Linux via a terminal from the 1980s and have set LANG=fi_FI.ISO-8859-1 to make the tty I/O compatible with that, then I'd like the dotnet CLI to keep the same encoding even if some other environment variable overrides the language. So if the original issues do not reproduce on Linux then it's not worth breaking this scenario just to match Windows.

nagilson · 2023-01-06T00:26:19Z

Yes, this does not appear to repro on Linux. I'll check other platforms I can access in a bit, but limiting it to Windows for now 👍

nagilson · 2023-01-06T17:19:35Z

/azp run

azure-pipelines · 2023-01-06T17:19:46Z

Azure Pipelines successfully started running 1 pipeline(s).

… appears to be rather difficult to get the changed encoding if it happens mid process. We would probably need to write more code for this. Let's see if we want to do that.

…cause it appears to be rather difficult to get the changed encoding if it happens mid process. We would probably need to write more code for this. Let's see if we want to do that." This reverts commit eada426.

dsplaisted

Looks good to me, but I don't know glob/loc details such as the different between CurrentCulture and CurrentUICulture, so we should get someone who does to review this.

nagilson · 2023-01-10T17:29:53Z

Thanks, do you have any contacts for loc?

dotnet/dotnet-api-docs#8767 The docs are kinda confusing so I filed an issue here to ask.

dsplaisted · 2023-01-10T22:18:40Z

@tarekgh can probably review this or point us to someone who can.

src/Cli/dotnet/UILanguageOverride.cs

tarekgh · 2023-01-10T23:05:30Z

src/Cli/dotnet/UILanguageOverride.cs

+            if (OperatingSystem.IsWindows() && Environment.OSVersion.Version.Major >= 10) // Encoding is only an issue on Windows, UTF-8 is only officially supported on 10+.
+            {
+                Console.OutputEncoding = DefaultMultilingualEncoding;
+                Console.InputEncoding = DefaultMultilingualEncoding; // Setting both encodings causes a change in the CHCP, making it so we dont need to P-Invoke ourselves.


@adamsitnik would setting UTF-8 encoding to the console be ok?

According to this, https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page, after the May 2019 Windows 10 update UTF-8 is officially supported. Since this only happens in Windows 10+, and when using languages besides English, which already have a bad encoding without this change, I think we can take this change for 8.

cc @dsplaisted for potential approval.

… dotnet to use OS language, but still change the encoding in any case where the language isn't en

nagilson · 2023-01-18T22:32:40Z

/azp run

azure-pipelines · 2023-01-18T22:32:46Z

Pull request contains merge conflicts.

dsplaisted

👍

…8503) Fixes #1596 Changes Made SetConsoleUI now calls into a helper which sets the encoding to support non-en languages and checks if an environment variable exists to change the language to. Testing Setting DOTNET_CLI_UI_LANGUAGE=ja now changes msbuild correctly: image Doing a complicated build (aka building MSBuild) to use multiple threads shows other threads seem to use the same UI culture: image See that chcp remains the same after execution: image (Was set to 65001 temporarily but back to the original page before execution.) Notes Much of this code is a port of this code: dotnet/sdk#29755 There are some details about the code here. [!] In addition, it will introduce a breaking change for msbuild just like the SDK. The break is documented here for the sdk: dotnet/docs#34250

nagilson added 2 commits January 4, 2023 15:52

Try setting the culture of the current thread and not just setting th…

ad7ea95

…e default

set the chcp page and not just the culture

df7120d

dotnet-issue-labeler bot added the Area-Infrastructure label Jan 5, 2023

nagilson changed the title ~~Set the CHCP Language Page to the Corresponding Culture~~ Use UTF-8 Encoding by Default On Windows Jan 6, 2023

Cleanup Code Around UILanguageOverride To Fix Garbled Chars

16a1b03

nagilson changed the title ~~Use UTF-8 Encoding by Default On Windows~~ Use UTF-8 Encoding by Default On Windows On Non English Languages Jan 6, 2023

nagilson added 4 commits January 6, 2023 14:06

Also do the encoding if the system language is set.

b1cd675

Only do the UI Language Override on Win10+ (UTF8 is unstable elsewise)

f5e48a2

(Revert This) WIP of test for this. I will remove the test because it…

eada426

… appears to be rather difficult to get the changed encoding if it happens mid process. We would probably need to write more code for this. Let's see if we want to do that.

nagilson changed the title ~~Use UTF-8 Encoding by Default On Windows On Non English Languages~~ Use UTF-8 Encoding by Default On Windows On Non English Languages & Change Culture For All Based on OS Main Language Jan 7, 2023

nagilson marked this pull request as ready for review January 7, 2023 00:24

nagilson requested a review from a team January 9, 2023 17:41

dsplaisted reviewed Jan 10, 2023

View reviewed changes

tarekgh reviewed Jan 10, 2023

View reviewed changes

src/Cli/dotnet/UILanguageOverride.cs Outdated Show resolved Hide resolved

tarekgh reviewed Jan 10, 2023

View reviewed changes

src/Cli/dotnet/UILanguageOverride.cs Outdated Show resolved Hide resolved

tarekgh reviewed Jan 10, 2023

View reviewed changes

nagilson mentioned this pull request Jan 11, 2023

dotnet publish shows garbled characters when running on PMC on OS with Traditional Chinese language #29151

Closed

nagilson added 2 commits January 11, 2023 09:31

No longer make the date/number formatting match the user's set language

86ea3fa

Respond to PR Feedback: Don't change CurrentCulture, and don't change…

c7c709c

… dotnet to use OS language, but still change the encoding in any case where the language isn't en

Fix an outdated comment to reflect changes from the previous commit

fdb24a7

nagilson changed the title ~~Use UTF-8 Encoding by Default On Windows On Non English Languages & Change Culture For All Based on OS Main Language~~ Use UTF-8 Encoding by Default On Windows On Non English Languages Jan 11, 2023

Merge remote-tracking branch 'primary/main' into nagilson-language-fix

8e35c7d

nagilson added 2 commits January 18, 2023 14:36

Merge remote-tracking branch 'primary/main' into nagilson-language-fix

28ef8cb

Use an older API for detecting if windows to support net 472

06d3426

dsplaisted approved these changes Jan 20, 2023

View reviewed changes

nagilson enabled auto-merge January 20, 2023 17:49

nagilson merged commit a23e88e into dotnet:main Jan 20, 2023

nagilson mentioned this pull request Jan 26, 2023

Build Output glitches on non-English locales #30143

Closed

This was referenced Feb 23, 2023

[Breaking Backport] Fix Encoding On Non EN Languages #30823

Merged

[Breaking change]: .NET 8 Uses UTF 8 Encoding in Non-EN Languages dotnet/docs#34250

Closed

This was referenced Feb 24, 2023

Set .net compilation output stream to utf8 godotengine/godot#73865

Closed

C#: Always decode dotnet output as UTF-8 godotengine/godot#74065

Merged

nagilson mentioned this pull request Feb 28, 2023

Detect .NET SDK & VSLANG Custom Language Settings & Apply To MSBuild dotnet/msbuild#8503

Merged

nagilson mentioned this pull request Mar 13, 2023

MSBuild output garbled godotengine/godot#64276

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use UTF-8 Encoding by Default On Windows On Non English Languages #29755

Use UTF-8 Encoding by Default On Windows On Non English Languages #29755

nagilson commented Jan 5, 2023 •

edited

Loading

KalleOlaviNiemitalo commented Jan 5, 2023

nagilson commented Jan 5, 2023

KalleOlaviNiemitalo commented Jan 5, 2023

nagilson commented Jan 5, 2023 •

edited

Loading

KalleOlaviNiemitalo commented Jan 6, 2023 •

edited

Loading

nagilson commented Jan 6, 2023

nagilson commented Jan 6, 2023

azure-pipelines bot commented Jan 6, 2023

dsplaisted left a comment

nagilson commented Jan 10, 2023

dsplaisted commented Jan 10, 2023

tarekgh Jan 10, 2023

nagilson Jan 18, 2023

nagilson commented Jan 18, 2023

azure-pipelines bot commented Jan 18, 2023

dsplaisted left a comment

Use UTF-8 Encoding by Default On Windows On Non English Languages #29755

Use UTF-8 Encoding by Default On Windows On Non English Languages #29755

Conversation

nagilson commented Jan 5, 2023 • edited Loading

KalleOlaviNiemitalo commented Jan 5, 2023

nagilson commented Jan 5, 2023

KalleOlaviNiemitalo commented Jan 5, 2023

nagilson commented Jan 5, 2023 • edited Loading

KalleOlaviNiemitalo commented Jan 6, 2023 • edited Loading

nagilson commented Jan 6, 2023

nagilson commented Jan 6, 2023

azure-pipelines bot commented Jan 6, 2023

dsplaisted left a comment

Choose a reason for hiding this comment

nagilson commented Jan 10, 2023

dsplaisted commented Jan 10, 2023

tarekgh Jan 10, 2023

Choose a reason for hiding this comment

nagilson Jan 18, 2023

Choose a reason for hiding this comment

nagilson commented Jan 18, 2023

azure-pipelines bot commented Jan 18, 2023

dsplaisted left a comment

Choose a reason for hiding this comment

nagilson commented Jan 5, 2023 •

edited

Loading

nagilson commented Jan 5, 2023 •

edited

Loading

KalleOlaviNiemitalo commented Jan 6, 2023 •

edited

Loading