[rustbuild] Implement testing and refactor configuration

[Mark-Simulacrum]

The important commits for review are marked with [VIC]. Each commit is intended to be relatively small, even though the overall patch is quite large. I would recommend reviewing by-commit for review purposes. I'm happy to split this PR up into smaller ones if that makes things easier -- the last commit (which implements the actual tests) is fairly small, though it builds on the refactoring done in the rest of the commits quite heavily.

Configuration refactor

This PR refactors the configuration to utilize Default impls for the structs and to directly include toml-deserialized structs in the final Config struct. This avoids quite a bit of boilerplate code, and, in my opinion, makes it considerably easier to understand what the default value for a given option is. It's intended that nothing has actually changed, but it's possible that I've accidentally forgotten some option -- I've tried to check manually and found nothing relevant.

Testing design (also included in relevant commit)

In order to run tests, various parts of rustbuild are cfg'd out,
generally speaking, these are filesystem-related operations and running
commands. Then, rustbuild is run "as normal" and the various steps that
where run are retrieved from the cache and checked against the expected
results.

Note that this means that the current implementation primarily tests
"what" we build, but doesn't actually test that what we build will
build. In other words, it doesn't do any form of dependency verification
for any crate. This is possible to implement, but is considered future
work.

This implementation strives to cfg out as little code as possible; it
also does not currently test anywhere near all of rustbuild. The current
tests are also not checked for "correctness," rather, they simply
represent what we do as of this commit, which may be wrong.

A fake fs module is also added to src/bootstrap/lib.rs and used
throughout rustbuild to make it so that most of the code doesn't have to
cfg out create_dir_all and remove_dir_all, etc.

Test cases are drawn from the old implementation of rustbuild, though
the expected results may vary.

r? @alexcrichton

[bors]

☔ The latest upstream changes (presumably #48520) made this pull request unmergeable. Please resolve the merge conflicts.

[bors]

☔ The latest upstream changes (presumably #48531) made this pull request unmergeable. Please resolve the merge conflicts.

[alexcrichton]

This is looking pretty nice, thanks for this @Mark-Simulacrum! I think I'm gonna have a pretty difficult time reviewing this, though. Would it be possible to split this apart in to separate PRs? At a high level I'm not sure that refactoring the configuration or testing would amount to 7k changes, so it seems like there's a lot more going on here as well?

In general I've found modifying rustc's build system to be extremely difficult on a large scale in the sense that there's always a regression lurking around the corner, so at least for me it's easiest to sit down with smaller chunks at a time and try to reason that through.

[petrochenkov]

I'm not sure that refactoring the configuration or testing would amount to 7k changes, so it seems like there's a lot more going on here as well?

There's a "rustfmt everything" commit hiding somewhere in the middle which is responsible for half of the diff 😄
(A good candidate for splitting.)

[Mark-Simulacrum]

#48599 and #48600 are now open. I'm going to close this for now -- I'll reopen once we've merged more of the parts, and work on trimming down the individual parts into separate PRs and open as I go.

[alexcrichton]

Ok thanks @Mark-Simulacrum!