Rust builds with rust-toolchain fail with image 20240317.1.0 but previous was fine 20240310.1.0

[FrancescElies]

Description

Rust builds fail with image 20240317.1.0 ❌ but previous was fine 20240310.1.0 ✅ (on same commit)

The current suspect is image ❌ 20240317.1.0
Previous image seems to work fine ✅20240310.1.0

In the release of 20240317.1.0 the only relevant rust note I see is the upgrade of Rustup from 1.26.0 to 1.27.0

Platforms affected

• Azure DevOps

Runner images affected

• Windows Server 2022

Image version and build link

I know you won't be able to access this builds (private) but we have rust builds on windows for the same commit green on Wednesday (20 march) but failing now.

Failing build here
For exactly the same commit green build here

Is it regression?

I think so

Expected behavior

Rust build should work on 20240317.1.0 too

Actual behavior

Last image fails to build rust

Repro steps

Run rust build for the images provided for the same commit

[FrancescElies]

Adding following is workaround to make our builds work again.

  - powershell: rustup update --force 1.76
    displayName: update 1.76

[Alexey-Ayupov]

Hello @FrancescElies. Glad to hear that you have a workaround. Meanwhile we will try to understand what's the problem.

[mikhailkoliada]

rust 1.77 will be released as a part of the runners next week.

[FrancescElies]

@mikhailkoliada the yaml workaround I posted had version 1.77 because of me doing copy pasta, sorry if that was confusing (just edited that to v1.76).

Both images 20240317 and 20240310 have rust 1.76.

In our links I added in the bug description we use 1.76, with one image fails with the other one didn't.

To be clear this issue is about 1.76 and the latest vm image that was released.

[FrancescElies]

I suspect something changed how rust stuff is installed in the image I would bet the same issue will be present for v.177, thus not sure awaiting-deployment will help.

[sergei-pyshnoi]

Hello @FrancescElies . Is problem still persist? If yes, can you please provide minimal repro?

[FrancescElies]

@sergei-pyshnoi

Is problem still persist?

If nothing changed with image 20240317.1.0, then the answer is yes.

If yes, can you please provide minimal repro?

I have provided you two links of two different builds in time that build the exact same commit, locally builds fine, so everything seems to point out in the direction of image 20240317.1.0.

A minimal repro is a fair request but given that there is not a clear easy way to do that with yaml and private projects this requires a non negligible effort from our side.

In all honesty, we are paying customers and I believe I have provided sufficient evidence that something might be wrong with your latest image.
Minimal repro is build a rust project for both images I provided, I guess the yaml for that is something you could put together.

Summarizing

We have a workaround and we don't feel the urgency for a fix, it's fine if is not high priority for you either.
It's obvious that we have done our research before opening this issue and asking for a minimal repro without showing me that you have done the same at your side feels a bit unfair to me.

[FrancescElies]

Clarifying our findings

In the links I provided the build fails with

info: syncing channel updates for '1.76-x86_64-pc-windows-msvc'
info: latest update on 2024-02-08, rust version 1.76.0 (07dca489a 2024-02-04)
error: component 'rust-std' for target 'stable-x86_64-pc-windows-msvc' is unavailable for download for channel '1.76'
If you don't need the component, you could try a minimal installation with:

    rustup toolchain add 1.76 --profile minimal

If you require these components, please install and use the latest successful build version,
which you can find at <[https://rust-lang.github.io/rustup-components-history>.](https://rust-lang.github.io/rustup-components-history%3E.)

After determining the correct date, install it with a command such as:

    rustup toolchain install nightly-2018-12-27

Then you can use the toolchain with commands such as:

If you run rustup toolchain list in your latest image you will see 1.76 is missing, the list is empty, thus the error.
I don't have access to your previous image anymore after the latest deployment but I would bet that listing the toolchains will show that 1.76 was avaialable back then.

if you run rustup toolchain add 1.76 the toolchain will be added and everything will work again, thus I doubt there is any kind of problem in the servers where the toolchain is downloaded from.

Please correct me if I see this wrong.

[FrancescElies]

NOTE
You can select the specific rust version adding this file to a rust project.

───────┬────────────────────────────────────────────────
       │ File: rust-toolchain.toml
───────┼────────────────────────────────────────────────
   1   │ [toolchain]
   2   │ channel = "1.76"
   3   │ targets = ["stable-x86_64-pc-windows-msvc"]

[sergei-pyshnoi]

After investigation I thinks that is not image related error. You starting gets error just in time when rust transfer stable tag to version 1.77 . So its can possible reason of conflicts while trying to download stable package from 1.76 channel.

Another moment, upcoming Windows-2022 update including 1.77 version of rust tools, so anyway you should keep workaround before you transfer rust project to newer version.

[FrancescElies]

I think the problem is that the readme for win2022 images promises rust 1.76 but the image doesn't provide it.

When you run rustup toolchain list in the image, the list is empty, IMO this means either the readme has wrong description or the image is missing rust 1.76.

Locally if I uninstall 1.76 toolchain, I can reproduce exactly the same error.

There is no way for me to prove that the previous image had rust 1.76 because I can't longer access it after last deployment but I would bet that is the case, and the reason it started failing it's because the new one misses the toolchain specified in the readme.

[erik-bershel]

Hey @FrancescElies!
Current images contain Rust 1.77.0 and Rustup 1.27.0. Correct links to the readme file: https://github.com/actions/runner-images/blob/main/images/windows/Windows2022-Readme.md
https://github.com/actions/runner-images/blob/main/images/windows/Windows2019-Readme.md

[erik-bershel]

According to the rustup, rustc and cargo:

Default host: x86_64-pc-windows-msvc
rustup home:  C:\Users\runneradmin\.rustup

installed targets for active toolchain
--------------------------------------
i686-pc-windows-msvc
x86_64-pc-windows-gnu
x86_64-pc-windows-msvc

active toolchain
----------------
stable-x86_64-pc-windows-msvc (default)
(timeout reading rustc version)
rustc 1.77.0 (aedd173a2 2024-03-17)
cargo 1.77.0 (3fe68eabf 2024-02-29)

[erik-bershel]

@FrancescElies do you still failing into any issues using rust compiler? If so, could you provide a bit more logs or at least error output. Also I'll be glad to understand the goal of the process. In this case we at least could try to build repro.

[FrancescElies]

@erik-bershel we have a workaround in place, if I remove it the issue is still there.

We already moved to 1.77, current image says it contains 1.77 but when I run rustup toolchain list I get a different result.

• Example build here.
• Image: windows-2022
• Version: 20240322.1.0
• Included Software: https://github.com/actions/runner-images/blob/win22/20240322.1/images/windows/Windows2022-Readme.md
• Image Release: https://github.com/actions/runner-images/releases/tag/win22%2F20240322.1

We still need to explicitly install 1.77 because in our project we specify the version we want to use.

───────┬─────────────────────────────────────────────
       │ File: rust-toolchain.toml
───────┼─────────────────────────────────────────────
   1   │ [toolchain]
   2   │ channel = "1.77"
   3   │ targets = ["stable-x86_64-pc-windows-msvc"]
───────┴─────────────────────────────────────────────

When list rust toolchains I would expect to see the following but we don't, are my expectations wrong?

> rustup toolchain list
stable-x86_64-pc-windows-msvc (default)
1.77-x86_64-pc-windows-msvc

I suspect this was the case for 20240310.1.0 because our same setup used to work without having to explicitely install the toolchain that was already specified in the readme but we have no way to prove it because we don't have access to that image any more.

[erik-bershel]

@FrancescElies according rustup documentation I may say that you expecting something what we never did.
Lets take a look here:

# Download the latest rustup-init.exe for Windows x64
# See https://rustup.rs/#
$rustupPath = Invoke-DownloadWithRetry "https://static.rust-lang.org/rustup/dist/x86_64-pc-windows-msvc/rustup-init.exe"

# Install Rust by running rustup-init.exe (disabling the confirmation prompt with -y)
& $rustupPath -y --default-toolchain=stable --profile=minimal
if ($LASTEXITCODE -ne 0) {
    throw "Rust installation failed with exit code $LASTEXITCODE"
}

During the installation process, we download the basic rustup installer itself, which then performs a minimal installation with one basic toolchain stable-msvc with the current stable version. Due to the fact that it is basic for the current stable build, I see no reason why it would have a name like 1.77-x86_64-pc-windows-msvc, unless we try to force install the same toolchain with a command like rustup toolchain install 1.77-msvc - but it's only us who would now that this is the same toolchain, but not rustup.
Preinstalled stable-x86_64-pc-windows-msvc (default) toolchain has always the same version as presented in readme.

[FrancescElies]

Not sure what the issue might be, in the issue description I provided two links to same commit with two different images one passing the other one failing (20240317.1.0) thus I know the problem is not caused because of a change in our code-base.

I wish I could choose the runner image version to see if it's because of the image or some external factor, but that's not possible.

The only related change I can see between both images is the upgrade of rustup 1.26->1.27.

I am not sure how you want to proceed with this one but it seems to me either something has changed on your side or on the rustup side.

By any chance could you run rustup toolchain list in the runner image version 20240310.1.0?

@erik-bershel how do you want to proceed with this?

[erik-bershel]

The only related change I can see between both images is the upgrade of rustup 1.26->1.27.

Yep, in that release there was a new rustup. Most likely it was tool behavior that changed.

By any chance could you run rustup toolchain list in the runner image version 20240310.1.0?
@erik-bershel how do you want to proceed with this?

I'll try to. I'll be there later with the information received.

[erik-bershel]

In the meantime. I think there is an error in your .toml.
You wrote next:

───────┬─────────────────────────────────────────────
       │ File: rust-toolchain.toml
───────┼─────────────────────────────────────────────
   1   │ [toolchain]
   2   │ channel = "1.77"
   3   │ targets = ["stable-x86_64-pc-windows-msvc"]
───────┴─────────────────────────────────────────────

But I think that the target and the toolchain is not the same matter. I suppose that you should use x86_64-pc-windows-msvc as a target. This is also reflected in documentation:

1. Toolchain file description.
2. Targets description.
3. Platforms description.

May I ask you to check your build using the next rust-toolchain.toml:

[toolchain]
channel = "1.77"
targets = ["x86_64-pc-windows-msvc"]

[FrancescElies]

We had this wrong in our rust-toolchain since version 1.64, for whatever reason it exploded now that's why we didn't even consider that, maybe the format changed at some point or we had it wrong all the way since the beginning.

Anyways, even if I was convinced the problem was not on our side, as it turns out, it was.

@erik-bershel thanks for your patience and help, issue resolved with your hint about rust-toolchain.toml

Closing.