Get better content addressed store support into OCI image spec

[roberth]

OCI image concepts tend to work reasonably well as a store for Nix-built software, but we are running into limitations:

• 256 layer limit
• some backends do not store layers individually

Both of these follow from docker build's conceptual reliance on overlays, and by using OCI blobs as layers, we inherit these limitations for store paths.
Fortunately, store paths do not require overlay semantics, so an extension of the image spec could resolve these issues, and enable other non-Nix use cases in the process.

Concretely what I'm thinking of is to extend the OCI image spec so that as a sibling to the layers property, it supports mounts, which are handled after the layers have been merged, by mounting layer blobs at the intended locations in the root file system.

layers: [
  blob1
  blob2
],
mounts: [
  { "path": "/nix/store/...-foo": {
    readOnly: true,
    blob: blob3
  } }
]

Backends could implement this by merging the layers as usual (in whichever way, as usual), and then doing the same for the mounts.

Things that need to be done

• prototype implementation, docker/podman?
• are the names and schema consistent with existing OCI conventions?
• detailed proposal
• work with OCI folks

Don't make it Nix-specific, because it isn't.
All content-addressed stores (or similar) can be translated to mounts, and bypass the limitations of overlays and storage backends.

I don't believe this is an original idea, but I don't know who to credit.
My purpose for this issue is to promote the idea and serve as a crystallization point for those who are interested to get involved with this. (Personally I'm only tangentially involved with container stuff)

Other, discarded alternatives

• Alternate storage backend: by implementing a custom backend, a specific content-addressed store could be leveraged. This would perhaps be difficult to exploit by the wider OCI ecosystem, which can't make assumptions about backends.
• Alternate volume backend: seems feasible, but also creates a dependency on an external store which isn't the store of OCI blobs/layers
• Container bind mounts: widely supported, but requires a prepared host, defeating OCI images' distributability

Related

• pkgs.dockertools.buildLayeredImage: customisable layering strategy #122608 (comment)

[arianvp]

I think the erofs / https://github.com/dragonflyoss/nydus folks might also be interested in this.

From what I vaguely understand they keep a big erfos-metadata-image which records in which content-addressed blob each file-path in the final container lives and then combine that with file-backed mounts so that they don't have to worry about overlayfs ordering.

Having more of this functionality directly in OCI probably simplifies a lot for them.

[arianvp]

I actually think their erofs-metadata-image basically is the mounts stanza you propose. It's a big map from file paths to blobs

[arianvp]

That discussion seems to be already going on here sort of: opencontainers/image-spec#1190 (comment)

[arianvp]

Final reference: containerd/containerd#9361