Interop between two haskell.nix repositories

[tscholak]

Hi,

First, thank you very much for creating this library!
We have been using haskell.nix for https://github.com/hasktorch/hasktorch for some time now, and it has improved our productivity quite a bit!

I've got a basic question, though, and I can't seem to figure it out.

Suppose I have two repositories with two cabal projects, A and B, where B depends on A.
Both projects are already using haskell.nix and pretty much follow the cardano-skeleton example (in particular, the two projects are A, https://github.com/hasktorch/hasktorch, and B, https://github.com/hasktorch/hasktorch-skeleton).

In order to fulfil the dependency in B, I've added A to the source-repository section of the cabal.project file in B (e.g., https://github.com/hasktorch/hasktorch-skeleton/blob/master/cabal.project).

Now, when I start a nix-shell in B, cabal will have to build A fully before being able to do any work for B. My questions is:

How can I make it such that the build of A is part of building the nix-shell environment, and how I can benefit from A's cachix cache when launching the shell?

I've looked around in https://github.com/input-output-hk if I can find any example of that, but to no avail.

Thanks,
Torsten

[michaelpj]

To be clear, the workflow you want is:

• Go to A, build with Nix
• Go to B, build with Nix, get cache hit for A

First of all, the two instances of A might be built in different ways by haskell.nix, if:

• The sha you checked A out at is different from the one you pinned in B's cabal.project (or the source is different!)
• There is non-trivial configuration in A's cabal.project: when used as a source dependency the cabal.project is ignored.

However, even if that's all fine, ultimately I think Nix will think they are different because they get their source in different ways. That is, your local one will bottom out in src = ./. or similar, and the one from cabal.project will bottom out in builtins.fetchGit or similar. Maybe this would be different if you got the local source via a fixed-output derivation? Not sure.

So the summary is that a local copy of A is quite different from a source-dep on A, unfortunately. I would expect that you get cache hits for A's dependencies, though.

[tscholak]

Thank you @michaelpj, I understand.
I think though that you slightly misunderstood.
What I desire is that, in B, A should behave just like a hackage dependency. It should be built when I launch a nix-shell so that cabal and ghcide do not have to do the extra work of building A again for building B. The problem I have is that I don’t know or understand how I can make B’s cabalProject derivation use A’s cabalProject derivation to fulfil the A dependency.
I don’t really care when the A cache is populated, if during the build of A or when B is built. I just want a better developer experience for people working on B. They should not have to wait for A to be built again and again when working on only B.
Not sure if I explained this well :/

[michaelpj]

Ah I see. This is actually a long-standing cabal bug: haskell/cabal#5586 (apparently recently closed, so maybe it'll be better in the next version)

[tscholak]

Hi again :)

This is actually a long-standing cabal bug: haskell/cabal#5586

Not sure it is related.

I admit, I may have a completely wrong understanding of how haskell.nix works or is supposed to work, but I feel it should be possible to treat A as if it was coming from hackage.

Let's forget about source-repository-package, pretend I never mentioned it.

• I have a package set for project A, pkgSetA, made with cabalProject in repo A. A is neither on hackage nor stackage.
• I have a package set for project B, pkgSetB, made with cabalProject in repo B. B can't be built. It is missing a dependency, A.
• How can I use pkgSetA to fulfil the missing dependency in pkgSetB?

[michaelpj]

but I feel it should be possible to treat A as if it was coming from hackage.

The upshot of that cabal bug is that cabal does not treat it as if it came from Hackage. Cabal treats it as if it was another local package in your project. And we follow cabal.

(In fact, in a shellFor for such a project, we have built it and put it in the package database, but cabal ignores that and builds the "local" version anyway 🤷 )

How can I use pkgSetA to fulfil the missing dependency in pkgSetB?

The Haskell.nix Way is to ask: how would you do this with cabal? And the easiest way to do it with cabal is... source-repository-package. That works badly in this context, sure, but then we complain to upstream.

There is another alternative that might get you what you want, which is to use an internal hackage. I believe you can even use those with haskell.nix these days, but I don't know that much about them.

Now, if I understand you rightly, you don't care that your package won't build with cabal, and you want to hack together a setup that will only work in Nix. I think this is best avoided, but you're an adult ;)

I'm not totally sure, but I think you might be able to do that with pkg-def-extras, since I think that gets folded into the main package set. So if you put your package in there we may treat it as if it's in Hackage. You'll need to not include it as a source-repository-package, obviously.

[tscholak]

Ah, I get you now. This is gold, thank you!

[michaelpj]

I think we got to the bottom of this.