NixOS cross-compile support

[ambrop72]

There is good support for cross-compilation in nixpkgs, but as far as I can tell, none in NixOS. This has a drastic impact on the usefulness of cross-compiling within Nix.

Ideally one would be able to:

• Build a cross-compiled system configuration.
• Deploy the system to a local storage medium (like nixos-install).
• Deploy the system via nixops (basic ssh deployment).
• Build a target-specific firmware image (where applicable).

I am prepared to work on this, but I would first like some guidence from people familiar with the NixOS code.

[vcunat]

I know no details, but this mostly sounds to me only as tweaking parameters and $NIX_PATH for nixos-rebuild. IMHO, for the actual building process it doesn't matter if you build NixOS config or something else.

[bjornfor]

(I'm interested in this; it has been a goal of mine since I started using NixOS. I think nixpkgs/NixOS can be (vastly) better than Buildroot/Yocto/Poky/T2/PTXdist/... I'm subscribed to the Buildroot mailing list and see them facing many of the issues that nix has already solved (build reproducability, build host isolation). Quite frustrating!

I've read https://nixos.org/wiki/CrossCompiling, and tried to understand what exactly @viric has been doing to cross-compile for the RaspberryPi etc. The wiki describes just cross-compiling single packages; it doesn't describe cross-compiling a whole NixOS system. So I'm still uncertain how @viric gets from building "bison" for a target, to getting something that can be used. (Btw, building the bootstrap tools for raspberryPi is broken in git master.)

I made an attempt to inject cross-compilation into nixos-rebuild like this:

  # configuration.nix snippet
  nixpkgs.config = {
    packageOverrides = pkgs: import ../cross-arm.nix;
  };

That doesn't work for at least one reason; when enabling cross-compilation, derivations get two extra attributes: .nativeDrv and .crossDrv. But the standard derivation attribute is still the "native" one.

I think it should be changed:

• Derivations should not have .nativeDrv and .crossDrv attributes that show up only when cross compiling.
• Derivations should always have a .hostDrv attribute, that is the derivation built for the build host.
• Derivation attributes should default to the "target" variant. When not cross-compiling target == host.

About cross-compiling a full NixOS system:
I personally think that cross-compiling a full GNU/Linux distro is a lot of pain. My idea has been to cross-compile a minimal NixOS system, and then build anything beyond that "natively" in VMs. Well, it's not really my idea, I've adopted it from Rob Landley. I recommend to read his http://landley.net/aboriginal/downloads/presentation.pdf (warning: 36 MiB PDF).

@viric: Do you have some thoughts about this?

[ambrop72]

So I assumed the nixos really is no different fromnixpkgs with regard to cross compiling, and ended up with this:

let
  pkgsFun = import <nixpkgs>;
  pkgsNoParams = pkgsFun {};
  crossSystem = {
    config = "armv6l-unknown-linux-gnueabi";
    bigEndian = false;
    arch = "arm";
    float = "hard";
    fpu = "vfp";
    withTLS = true;
    libc = "glibc";
    platform = pkgsNoParams.platforms.raspberrypi;
    openssl.system = "linux-generic32";
    gcc = {
      arch = "armv6";
      fpu = "vfp";
      float = "softfp";
      abi = "aapcs-linux";
    };
  };
in rec {
  nixos = (import <nixpkgs/nixos>) {
    system = "arm-linux";
    configuration = {
      services.openssh.enable = true;
      services.cron.enable = false;
      fileSystems."/" = {
        device = "/dev/sda1";
        fsType = "ext4";
      };
      boot.loader.grub.enable = false;
      nixpkgs.config = {
        packageOverrides = pkgs: (pkgsFun { inherit crossSystem; });
      };
    };
  };
}
/*
$ nix-build rpi.nix -A nixos.system.crossDrv
these derivations will be built:
  /nix/store/06paxqqkb7icfkry90z1cpya5a0fdasx-unit-script.drv
  /nix/store/0a62kbqqzpvv9d63jfivbjzal7d7kiiq-stage-2-init.sh.drv
  /nix/store/1c7vcv50ikii2bjmrpx2yw4vgbk4lknr-unit-cpufreq.service.drv
  /nix/store/1fs4v81bsiw5clc1zizhlin06ichdmf6-systemd-user.pam.drv
  /nix/store/1ihaizc9i6814z2k9qfwpx4pa3rnv4qd-libXmu-1.1.2.drv
 ...

From the names, most of the packages built are native not cross. Only a few are cross:

/nix/store/8p3g45iyrc1d39zcja45fsiwmywiyx22-binutils-2.23.1-armv6l-unknown-linux-gnueabi.drv
/nix/store/95bs36z32z35jzlkxl4hz9b7f3pmgz71-nixos-14.11.git.327c158-armv6l-unknown-linux-gnueabi.drv
/nix/store/b84yxhvw1szms3scagqnii2mq3s8l4w9-gcc-4.8.3-armv6l-unknown-linux-gnueabi-stage-static.drv
/nix/store/s7j4hqmlh3igrqb6mv01n0gcys3b541q-gcc-4.8.3-armv6l-unknown-linux-gnueabi-stage-final.drv
/nix/store/zv4m4m06s2rgz9lwyp2dk9qb8xr1js07-glibc-2.19-armv6l-unknown-linux-gnueabi.drv

[ambrop72]

"About cross-compiling a full NixOS system:
I personally think that cross-compiling a full GNU/Linux distro is a lot of pain. My idea has been to cross-compile a minimal NixOS system, and then build anything beyond that "natively" in VMs"

I would like a cross-compiled NixOS system to be "fully" supported, not just to help with bootstraping. This is for 2 reasons: compilation speed (VMs are slow) and practicality (not having to set up and manage a VM).

[bjornfor]

Yes, VMs are slower, but I'd rather have a slower build than to spend the rest of my life fixing cross-compilation issues in upstream packages. The VMs would be managed by nix, so it shouldn't really be noticed beyond the slowdown.

(Testing your rpi.nix file now.)

[nbp]

About the .nativeDrv and the .crossDrv, the reason was mainly to provide an integrated way to manage cross compilation without having to rewrite every expression. At the time @viric thought of making another tree of packages, which in-fine might have been better for knowing which packages are safe to cross-compile or not.

I also concur on the idea that cross compilation is a bad idea as it is poorly handled in many projects. Cross-compiling firefox would be a hell, because of all its dependencies. I still have made it for the SpiderMonkey back in 2012, but this was not easy to express it in Nix expression, and to get hydra understand that I wanted to build the project on one computer and getting it checked on an ARM board slave.

Cross-compiling is still needed, at least for building the bootstrap tools. I would love to have a NixOS profile which is only build on top of statically (cross-)compiled "busybox, uClibc, linux, binutils, gcc, make, bash" (slide 108) binaries.

One of the major thing about doing so, implies that we could make an /emulated-target/ system, which fetch everything locally, and use qemu as a chroot environment, where everything is copied in the disk image instead of being mounted, and copied back once finished. Doing so, would give us the ability to compile natively packages such as Nix.

Then with this first stage of packages, we can make a complete build including a compiled version of Nix, which can leave with its own nix store in the VM, and accept remote connections for building packages.

• stage 1: cross-compiled bootstrap image bootable from Qemu.
• stage 2: Use the bootstrap image as a chroot environment, to build an independent system image.
• final: Build natively (emulated or not)

The biggest advantage I see with such system, is that we minimize the number of packages to cross-compile for compiling natively on another architecture.

The 2 stages being extremely interesting as it effectively give us a comparable environment. Which means that we can effectively make a binary cache for another architecture, without referring to any cross-compilation.

[gitfoxi]

Did you ever come up with a way to cross-compile Nix?

If only Nix could be stripped of its many dependencies for the porting use-case. One idea I'm thinking about is to use Buildroot to build Nix's dependencies and Nix and then off to the races in the Qemu or maybe even compile on a development board with lots of RAM. If you had enough of them to use as build slaves it could go pretty quick to build a full NixOS.

[ambrop72]

I just installed ArchLinuxArm on my ARM boards (RPi, RPi2, BBB) and "manually" compiled Nix natively (with the AUR package really). Note, Nix depends on Perl so I think this (cross-compiling Nix) is too much effort to be worth pursuing.

[gitfoxi]

ArchLinuxArm is something I didn't know about. FWIW Buildroot can build Perl. Sometimes.

[bjornfor]

Reminds me, I have an old WIP branch which pulls in the Perl cross-compile fixes from Buildroot:

https://github.com/bjornfor/nixpkgs/tree/cross-compile-fixes

[tomberek]

Duplicate? #14965 and others.

[Ericson2314]

CC @bgamari, who has made great progress here.

[Ericson2314]

@shlevy we can close this now, right?

[shlevy]

Haven't tried in a few months, but it was working for me a while back.

[Ericson2314]

Thanks @shlevy. I guess I'll leave open until we have CI.

[Ericson2314]

Yes I think it high time we do proper CI for an entire cross package set.

[illegalprime]

are there updates / docs yet on how to cross compile nixos itself? I would be interested in furthering that goal..

[tomberek]

@illegalprime the last time I did this I had to cross-compile Nix, which then compiled (gcc itself, binutils, linux, everything) NixOS natively for a few days. I’m also interested in any progress in cross-compiling directly to NixOS and better binary cache support.

[bgamari]

I have uploaded the NixOS configuration that I have been trying to use for the MacchiatoBin AArch64 SBC here. It doesn't yet boot due to kernel issues but user-space (and consequently the NixOS configuration) is likely fine.

[illegalprime]

@bgamari I don't have that board available, but https://github.com/bgamari/mcbin.nix/blob/master/configuration.nix#L22 was really helpful to see how to get cross-compiled packages in a NixOS configuration. What I got is: Beaglebone Green, RPi 2, NanoPi M4, Odroid C1+, Odroid C2, RPi Zero.
I'll go try it on one of those & see!
NOTE: I can't build your repo because hardware-configuration.nix isn't committed, but it's a good inspiration.

[illegalprime]

I was able to cross compile a NixOS image for ARMv7 on my BeagleBone Green after quite a lot of cross-compilation fixes in nixpkgs:

<<< Welcome to NixOS 19.09.git.088cecc (armv7l) - ttyO0 >>>

Run `nixos-help` for the NixOS manual.

nixos-on-arm login: root (automatic login)


[root@nixos-on-arm:~]$ 

Unfortunately I believe patchShebangs or other leakage has occurred since I have x86 stuff in my /nix/store. #33956

[illegalprime]

Here's the progress I've made so far: https://github.com/illegalprime/nixos-on-arm

[stale]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[AleXoundOS]

Still important to me.

[stanipintjuk]

Me aswell

[Ericson2314]

@matthewbauer should we close? To be clear, I say that before it's mostly fixed---because it works well enough that specific problems should be tracked individually?