dockertools.buildLayeredImage fails with jq: Argument list too long

[koenw]

Describe the bug

dockertools.buildLayeredImage fails with /build/.attr-0: line 63: /nix/store/8v1x4xai13lj3q43b6ixgwgpvs66zmm3-jq-1.6-bin/bin/jq: Argument list too long when building an image with many layers. jq is used to build a JSON containing the image layers, and those layers are passed to jq through the command line. At some point we're passing the max allowed argument list length resulting in the error.

Steps To Reproduce

Steps to reproduce the behavior:
1.

cat > docker.nix <<EOF
{ pkgs ? import <nixpkgs> {} }:

pkgs.dockerTools.buildLayeredImage {
  name = "dockertools-test";
  tag = "latest";
  contents = [
    pkgs.texlive.combined.scheme-full
  ];
}
EOF

2. nix-build docker.nix

Expected behavior

A docker image is build successfully.

Notify maintainers

@utdemir

Metadata

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.10.63, NixOS, 21.05.3509.7daf35532d2 (Okapi)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.15`
 - channels(root): `"nixos-21.05.3509.7daf35532d2, nixpkgs-21.05, unstable-21.11pre321003.14aef06d9b3"`
 - nixpkgs: `/home/koen/.nix-defexpr/channels_root/nixpkgs`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: 
# a list of nixos modules affected by the problem
module:

[roberth]

Does this also happen with #122608?

[koenw]

It does not happen with #122608.

I created a patch (before I read your comment) to pass the layers via the store instead via the command line. I noticed #122608 does the same, except using the discouraged--argfile instead of --slurpfile.

I submitted the patch anyway so we can fix this without waiting for #122608 to land.

[roberth]

Don't worry about the potential conflict.

If you have that many layers, you'll probably benefit from #141050. Would you be interested in contributing to that?

[koenw]

If you have that many layers, you'll probably benefit from #141050. Would you be interested in contributing to that?

I thought so too, that's why I submitted it ;)

[roberth]

Oh 😆 copy paste error. I intended to ask about #122608.

[stale]

I marked this as stale due to inactivity. → More info