Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mrustc: init at 0.9 #113817

Merged
merged 2 commits into from
Apr 23, 2021
Merged

mrustc: init at 0.9 #113817

merged 2 commits into from
Apr 23, 2021

Conversation

r-burns
Copy link
Contributor

@r-burns r-burns commented Feb 20, 2021

This is rebased from the excellent work done by @progval in #85542, with the hopes that we can merge the underlying mrustc expressions and iterate on them while we sort out the bootstrapping logic.

Functional changes:

  • Needs compiler-rt patch for glibc ≥ 2.31

Cleanups:

  • Add pre/post hooks for all steps
  • Reuse src from main mrustc derivation
  • Use strictDeps
  • Override make variables with makeflags rather than patches + env vars

The mrustc-bootstrap builds rustc and cargo using mrustc. This is the main
motivator for mrustc and is very finicky so it's important that we
build this regularly and keep it green.

@progval says:

The initial build of Rust 1.29 (by mrustc) requires 2.5GB in /build
and 15GB of RAM and takes about 30 minutes on a modern x86 machine.

Can I tell hydra that this build will need a lot of RAM?
Compilation might take several hours on less beefy machines;
I hope Hydra doesn't timeout on this.

Motivation for this change
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@ofborg ofborg bot added 8.has: package (new) This PR adds a new package 11.by: package-maintainer This PR was created by the maintainer of the package it changes 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 1-10 labels Feb 20, 2021
@SuperSandro2000
Copy link
Member

Please squash all package related commits together.

cp run_rustc/output/prefix/bin/cargo $out/bin/cargo
cp run_rustc/output/prefix/bin/rustc_binary $out/bin/rustc

cp -r run_rustc/output/prefix/lib/* $out/lib/
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should move this to a separate output.

Copy link
Contributor Author

@r-burns r-burns Feb 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's definitely something we can look into, but I think for now I would prefer to just keep the bootstrap installation as simple as possible. I expect the final user-facing rustc derivations will disallow references to these bootstrapping stages anyway, so it shouldn't make any difference in terms of closure size whether this uses split outputs.

@r-burns
Copy link
Contributor Author

r-burns commented Feb 20, 2021

I've kept my fixes/cleanups in separate commits only for ease of review, and plan to squash before merging in accordance with CONTRIBUTING guidelines.

@SuperSandro2000 Is this all right with you? As a reviewer I sometimes like to see separation of concerns in commits, but I can just squash if it's not actually helpful.

@SuperSandro2000
Copy link
Member

Is this all right with you?

I bet 10 bucks on that I will probably forget it.

As a reviewer I sometimes like to see separation of concerns in commits, but I can just squash if it's not actually helpful.

If you don't rebase with master on force pushes I can see from the force push diff what you did. I am also using the mark-files-as-read-feature. Either works for me but keeping review comments around takes another force push before it can be merged.

@r-burns
Copy link
Contributor Author

r-burns commented Feb 21, 2021

I bet 10 bucks on that I will probably forget it.

Haha, fair enough :)
Squashed.

@r-burns
Copy link
Contributor Author

r-burns commented Feb 21, 2021

@ofborg build mrustc-minicargo

@progval
Copy link
Member

progval commented Apr 5, 2021

I started updating the nix expression to work with the current mrustc master, which supports rust 1.39: progval@07b59ad

But I still can't use it to build even build rust 1.29 (error: cannot satisfy dependencies so `std` only shows up once) and can't find how to fix it. Feel free to pick up where I stopped

@grahamc
Copy link
Member

grahamc commented Apr 22, 2021

Does the build use many CPU cores if they're available, or is it a memory-bound operation?

Copy link
Member

@alyssais alyssais left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM otherwise. I don't think we need to wait for the latest version to get an mrustc in Nixpkgs.

@progval
Copy link
Member

progval commented Apr 22, 2021

@grahamc It's partially parallel, but its bottleneck is by far the compilation of rustc (the main crate) by mrustc, which is not parallelizable

@SuperSandro2000
Copy link
Member

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.

Result of nixpkgs-review pr 113817 run on x86_64-linux 1

3 packages built:
  • mrustc
  • mrustc-bootstrap
  • mrustc-minicargo

mrustc is mostly patched to use shared LLVM sources but still uses
in-tree source for compiler-rt from LLVM 7. This needs to be patched to
compile under glibc 2.31 or later. It's easy enough to reapply all our
compiler-rt patches here.
@r-burns
Copy link
Contributor Author

r-burns commented Apr 23, 2021

@alyssais Thanks for the suggestions, updated.

@alyssais
Copy link
Member

alyssais commented Apr 23, 2021 via email

@alyssais alyssais merged commit 92c7773 into NixOS:master Apr 23, 2021
@alyssais
Copy link
Member

alyssais commented Apr 23, 2021 via email

@alyssais
Copy link
Member

alyssais commented Apr 23, 2021 via email

@r-burns r-burns deleted the mrustc branch April 23, 2021 11:03
@r-burns
Copy link
Contributor Author

r-burns commented Apr 24, 2021

Yay, it built! https://hydra.nixos.org/build/141770406#tabs-summary
Took about two hours, so I guess we don't need to set any special hydra rules for this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
8.has: package (new) This PR adds a new package 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 1-10 11.by: package-maintainer This PR was created by the maintainer of the package it changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants