binfmt_misc support for cross compiling

[lopsided98]

Issue description

I have been reading through the recent work on improving cross compiling, and one of the problems I have seen mentioned (and I have run into myself) is that many packages try to run binaries built for the host system, particularly while running tests. This means that most packages written in compiled languages need to have their tests disabled when cross compiling.

One possible solution to this is to use binfmt_misc and qemu to run binaries built for the host system. This would allow many tests to be run while cross compiling, and would require less work from package maintainers.

In NixOS, one can create a file in /etc/binfmt.d which will cause systemd to register a binfmt handler. This allowed my to build and test a few packages for aarch64, but it fails when sandboxing is enabled (because Nix does not add qemu to the sandbox). For this setup to be feasible, stdenv would have to include qemu while cross compiling, and there would have to be an automatic way of configuring binfmt.

I have not seen this solution discussed anywhere (though I may have missed it), so I wanted to know what others thought of this idea.

[bgamari]

Indeed this would be nice; I use Debian for my cross-compilation work and I have found that binfmt_misc is a great crutch to get things building.

[bjornfor]

Does user mode emulation work well? I remember Rob Landley calling QEMU user mode emulation "brittle", compared to system mode. But I have no first hand experience.

Another idea could be to use QEMU system mode emulation + Nix distributed builds mechanism, if made transparent to the user. ("Oh, there is no aarch64-linux system available, let's build one and run it in QEMU.")

[lopsided98]

I don't have a ton of experience with QEMU user mode, but I use it to build Docker images for armv6l and armv7l on Travis CI and it has worked reliably there. It might not work in all cases, but it seems better than disabling most tests while cross compiling.

I have considered using a full VM for compiling for ARM, and while it might be faster than any of my ARM hardware, I'd rather not have the overhead of emulation while running the compiler. QEMU user mode would allow the compiler to run natively, while only using emulation for things like tests.

[lopsided98]

I was doing some more testing, and I seem to have run into some of the "brittle" parts of QEMU. I tried to build libuv, and while most tests succeed, a few fail due to unimplemented features in QEMU.

Most of the errors are caused by unimplemented syscall options, for example here QEMU is missing an implementation of setsockopt(level=SOL_IPV6, optname=IPV6_MULTICAST_LOOP). This probably is missing from QEMU simply because nobody has needed it/gotten around to implement it (looking through the commit history, it seems that the few options for SOL_IPV6 that are implemented were added because someone wanted to make ping and traceroute work for IPv6).

not ok 297 - udp_options6
# exit code 6
# Output from process `udp_options6`:
# Unknown host QEMU_IFLA type: 43
# Unknown host QEMU_IFLA type: 43
# Unknown host QEMU_IFLA type: 43
# Unknown QEMU_IFLA_BR type 41
# Unknown QEMU_IFLA_BR type 42
# Unknown QEMU_IFLA_BR type 43
# Unknown QEMU_IFLA_BR type 44
# Unknown host QEMU_IFLA type: 43
# Unknown QEMU_IFLA_BR type 41
# Unknown QEMU_IFLA_BR type 42
# Unknown QEMU_IFLA_BR type 43
# Unknown QEMU_IFLA_BR type 44
# Unknown host QEMU_IFLA type: 43
# Unknown QEMU_IFLA_BRPORT type 28
# Unknown QEMU_IFLA_BRPORT type 27
# Unknown QEMU_IFLA_BRPORT type 30
# Unknown QEMU_IFLA_BRPORT type 29
# Unknown host QEMU_IFLA type: 43
# Unknown QEMU_IFLA_BRPORT type 28
# Unknown QEMU_IFLA_BRPORT type 27
# Unknown QEMU_IFLA_BRPORT type 30
# Unknown QEMU_IFLA_BRPORT type 29
# Unsupported setsockopt level=41 optname=19
# Unsupported setsockopt level=41 optname=19
# Unsupported setsockopt level=41 optname=19
# Unsupported setsockopt level=41 optname=19
# Assertion failed in test/test-udp-options.c on line 74: r == 0
# qemu: uncaught target signal 6 (Aborted) - core dumped

The other Unknown QEMU_IFLA_* warnings are also caused by missing implementations, but they do not seem to be as important (they don't appear to cause any syscalls to return errors).

[matthewbauer]

I am thinking it is not a good idea to integrate binfmt_misc into Nixpkgs. I don't think we want Nixpkgs to depend on any particular kernel feature or that it has been configured properly. However, I think something like "exeWrapper" I've started in #50212 could be very useful. We want to make this opt-in for package that really need to run code as build time.

I do think it would be cool to make boot.binfmtMiscRegistrations easier to use. Something like "boot.emulatedSystems" where you could list off arbitrary systems like so:

boot.emulatedSystems = [
  lib.systems.examples.aarch64-multiplatform
  lib.systems.examples.mingwW64
];

And it can figure out all of the right parts to configure things correctly. We would have to keep track of magic numbers in addition to the exeWrapper for each system, but I don't think it would be too bad. We could also add each of these to extraSystems in the nix.conf file!

[matthewbauer]

Related: NixOS/nix#2561 (comment)

Closing because:

• Support extra compatible architectures nix#1916 has been merged
• Add "emulator" function to systems #50212 is merged
• c64639b adds a binfmt module