Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix library dependency issues #3901

Merged
merged 6 commits into from
Nov 5, 2024

Conversation

ysaito1001
Copy link
Contributor

@ysaito1001 ysaito1001 commented Nov 4, 2024

Motivation and Context

Our CI and release pipeline have revealed several areas where a Cargo.lock file should be included but is currently missing. This has led to issues when building SDKs with our MSRV of 1.78.0, using the latest dependencies (specifically arbitrary 1.4.0 and idna 1.0.3). The following compilation error occurs:

error[E0658]: use of unstable library feature 'error_in_core'
  --> /opt/cargo/registry/src/index.crates.io-6f17d22bba15001f/idna-1.0.3/src/lib.rs:78:6
   |
78 | impl core::error::Error for Errors {}
   |      ^^^^^^^^^^^^^^^^^^
   |
   = note: see issue #103765 <https://github.com/rust-lang/rust/issues/103765> for more information

This issue arises because these versions of the arbitrary and idna crates depend on a recently stabilized feature that remains unstable in our MSRV.

Below is a list of additional places where dependencies need to be locked:

  1. sdk-adhoc-test
  2. client-codegen-test
  3. check-semver-hazards (this compares two versions of the SDKs: one from a feature branch and the other from the most recent release of the SDKs in aws-sdk-rust. Note that it does not use cargo-semver-checks)
  4. canary
  5. cargo semver-checks (this compares two different versions of runtime crates and SDKs between a feature branch and the main branch within the repository)

This PR adds lockfiles for the first three locations. However, we were unable to add lockfiles for the fourth and fifth locations.

canary

canary-runner generates Cargo.toml on the fly, so this place cannot have a lockfile. The fix is to pin the offending dependencies directly within the code that renders the Cargo.toml.

cargo semver-checks

This will remain broken until we upgrade MSRV to 1.81.0. cargo semver-checks runs as part of a PR workflow and is not a required check for CI. The reason we cannot fix this until we upgrade our MSRV to 1.81, which stabilizes error_in_core, is that cargo-semver-checks creates a temporary wrapper crate called rustdoc whose Cargo.toml looks like this:

[package]
name = "rustdoc"
edition = "2021"
version = "0.0.0"
publish = false

[workspace]
members = []

[dependencies.aws-sdk-timestreamwrite]
path = "/Users/awsaito/src/smithy-rs/tmp-codegen-diff/aws-sdk/sdk/timestreamwrite"
features = ["behavior-version-latest", "default", "rt-tokio", "rustls", "test-util"]

[lib]
path = "lib.rs"
required-features = []

This crate is placed under a directory looking like smithy-rs/target/semver-checks/local-aws_sdk_timestreamwrite-0_0_0_local-3131265f731d9f1d, and we cannot add a lockfile or manually edit its Cargo.toml. When cargo semver-checks builds this rustdoc crate, it pulls in offending dependencies such as idna 1.0.3 and arbitrary 1.4.0 (please refer to the diagram in this PR for illustration of how semver-checks.py organizes crates during execution).

In short, cargo semver-checks acts as a client of AWS SDKs, building their dependencies that require MSRV 1.81.0 using our MSRV 1.78.0, which causes the error.

Testing

  • Tests in CI

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@smithy-lang smithy-lang deleted a comment from github-actions bot Nov 4, 2024
Copy link

github-actions bot commented Nov 5, 2024

A new generated diff is ready to view.

  • No codegen difference in the AWS SDK
  • Client Test (ignoring whitespace)
  • No codegen difference in the Server Test
  • No codegen difference in the Server Test Python
  • No codegen difference in the Server Test Typescript

A new doc preview is ready to view.

@ysaito1001 ysaito1001 marked this pull request as ready for review November 5, 2024 03:37
@ysaito1001 ysaito1001 requested review from a team as code owners November 5, 2024 03:37
@ysaito1001 ysaito1001 changed the title Place SDK lockfile in adhoc tests Fix library dependency issues Nov 5, 2024
@landonxjames landonxjames added this pull request to the merge queue Nov 5, 2024
@landonxjames
Copy link
Contributor

Talked with @ysaito1001 and the semver check failure is expected (and details will be provided in the PR body later). Merging so work can be unblocked in the morning.

Merged via the queue into main with commit f648759 Nov 5, 2024
43 of 44 checks passed
@landonxjames landonxjames deleted the ysaito/place-lockfile-in-sdk-adhoc-tests branch November 5, 2024 04:20
@ysaito1001
Copy link
Contributor Author

Talked with @ysaito1001 and the semver check failure is expected (and details will be provided in the PR body later). Merging so work can be unblocked in the morning.

The PR description updated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants