Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: automate creation of GitHub releases and tags #2027

Closed
wants to merge 8 commits into from
+134 −5
Closed

ci: automate creation of GitHub releases and tags #2027

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
084c879
ci: automate creation of GitHub releases and tags
galargh Jul 23, 2024
c0aa24f
docs: update documentation on release creation
galargh Jul 23, 2024
c4fa7c0
feat: release individual packages
galargh Jul 27, 2024
2292bc1
fix: trigger release workflows on Cargo.toml changes
galargh Jul 27, 2024
85586da
chore: simplify cargo-publish job check
galargh Jul 28, 2024
50273e3
ci: limit duplication in release workflows
galargh Aug 5, 2024
a3fb5a9
docs: document the new release process and possible improvements
galargh Aug 13, 2024
423e826
chore: apply suggestions from code review
galargh Aug 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions .github/workflows/release-check.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
name: Release Checker

on:
pull_request_target:
paths:
- "**/Cargo.toml"
types: [ opened, synchronize, reopened, labeled, unlabeled ]
workflow_dispatch:

permissions:
contents: write
pull-requests: write

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
release-check:
uses: ipdxco/unified-github-workflows/.github/workflows/[email protected]
with:
sources: |
[
"fvm/Cargo.toml",
"testing/integration/Cargo.toml",
"ipld/amt/Cargo.toml",
"ipld/bitfield/Cargo.toml",
"ipld/blockstore/Cargo.toml",
"ipld/car/Cargo.toml",
"ipld/encoding/Cargo.toml",
"ipld/hamt/Cargo.toml",
"ipld/kamt/Cargo.toml",
"sdk/Cargo.toml",
"shared/Cargo.toml"
]
separator: "@"
cargo-publish:
needs: [release-check]
if: needs.release-check.outputs.json != '[]'
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
source: ${{ fromJSON(needs.release-check.outputs.json).*.source }}
steps:
- uses: actions/checkout@v4
- env:
source: ${{ matrix.source }}
run: |
pushd "$(dirname $source)"
cargo publish --dry-run
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this will fail because it won't publish in the right order.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I.e., if we cut a release of the sdk + shared at the same time (which we always do), we'd need to push a release of shared to some fake registry for the release of the sdk to pass a cargo publish dry run (I think).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see what you mean now! Yes, that would definitely show up as a failure in the proposed process. Would it make sense to split the release in that case, as in, you first release shared and only follow up with sdk release after shared is fully out?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's going to make releases take even longer, which defeats the point of having automation.

I think the answer here is manual dry-run and it's not going to be pretty.

  1. Figure out which crates need to be published.
  2. Either determine the correct order, or hard-code it in a config.
  3. Run cargo vendor --versioned-dirs, creating a .cargo/config.toml file to respect the vendor directory.
  4. In publish order (reverse dependency order):
  5. Run cargo package.
  6. Copy the package (the extracted one) out of target/packages into vendor.
  7. Generate the .cargo-checksum.json file for that manually vendored package. This is the most annoying step, but shouldn't be hard with find + jq...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is very helfpul, thank you. I included this in the improvements section of the release automation doc in a3fb5a9. I also described what happens inside the current state of the automation in more detail.

I think the big decision here is whether the partial automation would be useful here, or whether we should wait until we can fully handle cargo publishes with respect to the dependency graph.

cc @BigLep

popd
36 changes: 36 additions & 0 deletions .github/workflows/releaser.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
name: Releaser

on:
push:
paths:
- "**/Cargo.toml"
workflow_dispatch:

permissions:
contents: write

concurrency:
group: ${{ github.workflow }}-${{ github.sha }}
cancel-in-progress: true

jobs:
releaser:
uses: ipdxco/unified-github-workflows/.github/workflows/[email protected]
with:
sources: |
[
"fvm/Cargo.toml",
"testing/integration/Cargo.toml",
"ipld/amt/Cargo.toml",
"ipld/bitfield/Cargo.toml",
"ipld/blockstore/Cargo.toml",
"ipld/car/Cargo.toml",
"ipld/encoding/Cargo.toml",
"ipld/hamt/Cargo.toml",
"ipld/kamt/Cargo.toml",
"sdk/Cargo.toml",
"shared/Cargo.toml"
]
separator: "@"
secrets:
UCI_GITHUB_TOKEN: ${{ secrets.UCI_GITHUB_TOKEN }}
51 changes: 46 additions & 5 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,11 +76,7 @@ Once the release is prepared, it'll go through a review:
2. Make sure that we're correctly following semver.
3. Make sure that we're not missing anything in the changelogs.

Finally, an [FVM "owner"](https://github.com/orgs/filecoin-project/teams/fvm-crate-owners/members) will:

1. Merge the release PR to master.
2. For each released crate, create a git tag: `crate_name@crate_version`.
3. Run `cargo publish` for each released crate (in dependency order).
Finally, an [FVM "owner"](https://github.com/orgs/filecoin-project/teams/fvm-crate-owners/members) will follow the [release process](#release-process).

### Crate Dependency Graph

Expand Down Expand Up @@ -112,3 +108,48 @@ If that works, proceed with releasing these crates.
[chat]: https://docs.filecoin.io/about-filecoin/chat-and-discussion-forums/#chat
[discuss]: https://github.com/filecoin-project/fvm-specs/discussions
[issue]: https://github.com/filecoin-project/ref-fvm/issues

### Release Process

This section describes the automated parts of the release process and the manual steps FVM owners need to take.

#### Current State

1. Create a pull request which updates the [`version` in one or more `Cargo.toml` files](https://github.com/search?q=repo%3Afilecoin-project%2Fref-fvm+path%3ACargo.toml+%2F%5Eversion+%3D%2F&type=code).
- Title the PR `chore: release vX.Y.Z`
2. On such a release PR's creation, a [Release Checker](.github/workflows/release-check.yml) workflow will run. It will perform the following actions:
1. Extract the version from the modified `Cargo.toml` files, and process each crate in the workspace **independently**.
2. Check if a git tag for the version, using the `crate_name@version` as the pattern, already exists. Continue only if it does not.
3. Create a draft GitHub release with the version as the tag.
4. Comment on the pull request with a link to the draft release.
5. Run `cargo publish --dry-run` for the crate for which the release is proposed.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it intentional that it's automated here but manual in actors-utils? https://github.com/filecoin-project/actors-utils/pull/235/files#diff-2b1b69303b927a484e02c7fad9fc87d0d3ff0dc22ae1da0ecd0dc935d922a23cR12

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was explicitly requested here - that's why it was added. But it's not clear to me whether we want it at all without being able to incorporate dependency graph knowledge into the dry run publish process.

2. On pull request merge, a [Releaser](.github/workflows/release.yml) workflow will run. It will perform the following actions:
1. Extract the version from the modified `Cargo.toml` files, and process each crate in the workspace **independently**.
2. Check if a git tag for the version, using the `crate_name@version` as the pattern, already exists. Continue only if it does not.
3. Check if a draft GitHub release with the version as the tag exists.
4. If the draft release exists, publish it. Otherwise, create a new release with the version as the tag.
3. **[MANUAL]** Run `cargo publish` for each crate that has been released in the [reverse dependency order](#crate-dependency-graph).
- You will need to be part of [fvm-create-owners](https://github.com/orgs/filecoin-project/teams/fvm-crate-owners) to do this per https://crates.io/crates/fvm.

#### Known Limitations

1. `cargo publish --dry-run` will fail if a crate has a dependency on a crate that has not been published yet (i.e. that is being published in the same release).
2. `cargo publish` has to be run manually.

#### Possible Improvements

1. Run `cargo publish --dry-run` in the [reverse dependency order](#crate-dependency-graph). Use a local registry to simulate the dependencies that are not yet published.
2. Run `cargo publish` in the [**reverse dependency order**](#crate-dependency-graph) automatically after the merge.

<details>
<summary>How to?</summary>

1. Figure out which crates need to be published.
2. Either determine the correct order, or hard-code it in a config.
3. Run `cargo vendor --versioned-dirs`, creating a `.cargo/config.toml` file to respect the vendor directory.
4. In publish order (reverse dependency order):
1. Run `cargo package`.
2. Copy the package (the extracted one) out of `target/packages` into `vendor`.
3. Generate the `.cargo-checksum.json` file for that manually vendored package. This is the most annoying step, but shouldn't be hard with `find` + `jq`...

</details>
Loading