RFC30: Compile time benchmark

[thomas-k-cameron]

Motivation and Context

This PR implements ci-script for bench marking compile time.
Results are turned into a markdown file and compile time is normalized to a value relative to the compile time of S3's sdk on dev profile without any features.

Benchmark is triggered for every PR.

I considered using AWS batch, however, I decided not to move forward with for following reasons

• I do not have access to your AWS account, thus making the debugging extremely difficult
• Fork's github action will always fail if you block access

Testing

NA

Checklist

• I have updated CHANGELOG.next.toml if I made changes to the smithy-rs codegen or runtime crates
• I have updated CHANGELOG.next.toml if I made changes to the AWS SDK, generated SDK code, or SDK runtime crates

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[thomas-k-cameron]

At this point, the script does not upload the result markdown file.
Where should it be uploaded? (I think some where on github would be nice.)
I appreciate help.

[thomas-k-cameron]

The error is the regex unicode thing. Once the other branch is merged this should be fixed as well.

[thomas-k-cameron]

Gotta fix this.

Creating network "smithy-rs-buildoqvajw_default" with the default driver
Creating smithy-rs-buildoqvajw_smithy-rs-build_1 ... 
Creating smithy-rs-buildoqvajw_smithy-rs-build_1 ... done
bash: ./ci-scripts/compiletime-benchmark: Permission denied
Stopping smithy-rs-buildoqvajw_smithy-rs-build_1 ... 
Stopping smithy-rs-buildoqvajw_smithy-rs-build_1 ... done
Removing smithy-rs-buildoqvajw_smithy-rs-build_1 ... 
Removing smithy-rs-buildoqvajw_smithy-rs-build_1 ... done
Removing network smithy-rs-buildoqvajw_default

[thomas-k-cameron]

I added a script for bench marking the compile time, but it gives me Permission denied.

I have no idea how to fix it. I appreciate help!

bash: ./ci-scripts/compiletime-benchmark: Permission denied

[jdisanti]

It's probably just missing the execute permission. If you're on a Unix-like system, you just need to chmod +x compiletime-benchmark and commit it.

[thomas-k-cameron]

It's happening on the Github CI.
chmod on my local machine doesn't affect the CI right?
(Sorry I could be wrong.)

Here is a full message.

Run ./smithy-rs/tools/ci-build/ci-action compiletime-benchmark 
  ./smithy-rs/tools/ci-build/ci-action compiletime-benchmark 
  tar cfz artifacts-compiletime-benchmark.tar.gz -C artifacts .
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
  env:
    rust_version: 1.67.1
    rust_toolchain_components: clippy,rustfmt
    ENCRYPTED_DOCKER_PASSWORD: 
    DOCKER_LOGIN_TOKEN_PASSPHRASE: 
    CACHE_ON_FAILURE: false
    CARGO_INCREMENTAL: 0
Creating network "smithy-rs-buildoqvajw_default" with the default driver
Creating smithy-rs-buildoqvajw_smithy-rs-build_1 ... 
Creating smithy-rs-buildoqvajw_smithy-rs-build_1 ... done
bash: ./ci-scripts/compiletime-benchmark: Permission denied
Stopping smithy-rs-buildoqvajw_smithy-rs-build_1 ... 
Stopping smithy-rs-buildoqvajw_smithy-rs-build_1 ... done
Removing smithy-rs-buildoqvajw_smithy-rs-build_1 ... 
Removing smithy-rs-buildoqvajw_smithy-rs-build_1 ... done
Removing network smithy-rs-buildoqvajw_default
Error: Process completed with exit code 126.

[thomas-k-cameron]

chmod worked. I didn't know the permissions my local machine is transferred on git. TIL

[thomas-k-cameron]

@jdisanti
Where should I upload the benchmark result?

[jdisanti]

Sorry, I haven't had a chance to look at this closely. This looks really cool.

[jdisanti]

I think this should get added to the PR bot instead of to CI. That way, the results get posted as a comment on PRs (although this only works for maintainers right now, unfortunately).

The PR bot workflow basically works like this:

• Kicks off the generate-diff and generate-doc-preview jobs in parallel. You could just add a new compiletime-benchmark job in here.
• The post-bot-comment job depends on all the parallel jobs, grabs their output, and aggregates it into a human readable comment. Then it posts to the GitHub API to make the comment.

The GitHub Actions job outputs feature is being used to communicate between the jobs. For example:

    outputs:
      bot-message: ${{ steps.generate-preview.outputs.bot-message }}

and

${{ needs.generate-doc-preview.outputs.bot-message }}

Testing this is going to be difficult of course. I think what we could do is after reviewing it to see if it looks like it should work, we could copy it to a new PR so that it has the right permissions for testing.

[thomas-k-cameron]

Gotcha.

[thomas-k-cameron]

I initially thought about using AWS batch, since you can have better control over the hardware, I can just show the amount of time that it took to finish instead of showing relative values. Also, results are going to be more re-producible.

Since I don't have access to your AWS account, I concluded that it would be very difficult for me to do any debugging.

Just FYI.

[thomas-k-cameron]

@jdisanti CI is updated. Would you mind checking it out?

[Velfi]

@thomas-k-cameron John is out until next week, but this looks good to me. Thanks for all your work.

[Velfi]

@thomas-k-cameron It turns out this broke CI so we had to revert it. You'll need to open a new PR.