pkg: defer evaluating platform vars in commands

[gridbugs]

Package build and install commands are expressions that generate the command to run by evaluating a series of conditional statements. This allows the commands to be tailored to the current machine where the command is being run. Previously dune would evaluate platform-specific variables (e.g. os, arch) in these expressions at solve-time, producing lockfiles specialized to the machine where they are solved. This change defers evaluation of these variables until build-time.

The consequence of deferring evaluation of platform variables to build time is that lockfiles will preserve more of the conditional logic found in the corresponding opam file. Note that some variables are still evaluated, such as with-test, as these variables are not based on the platform. The benefit of deferring evaluation of platform variables is that users of computers that differ from the computer where the lockfile was generated will be able to run the build/install commands specialized for their computer, which is more likely to work than attempting to run the build/install commands specialized for the computer where the lockfile was generated.

Note that this change alone is not sufficient for making lockfiles portable, as it doesn't address the fact that the dependencies of a package can also vary depending on platform-specific variables.

[gridbugs]

@rgrinberg let me know if you'd like this change to be behind a feature flag.

[rgrinberg]

The last time we tried doing this, we weren't able to correctly model the semantics of opam's formula evaluation in the dune language. Did we solve those problems?

[gridbugs]

The last time we tried doing this, we weren't able to correctly model the semantics of opam's formula evaluation in the dune language. Did we solve those problems?

Do you have more details about which parts of opam's formulae we don't replicate in dune? Was it the way that opam's undefined values propagate through boolean ops? (If so we've solved that problem).

Once this feature gets added to the dev preview we'll be able to run an experiment to see if it reduced the number of opam packages we can build. Let me know if you'd rather I put the new behaviour behind a feature flag.

[rgrinberg]

One solution that would fork for dependencies as well as build actions and would side step the evaluation problem above is introducing a conditional expression that would allow us to dispatch based on the configuration tuples. E.g:

(depends
  (match %{os} %{arch}
    (win arch64) (...)
    (linux arch64) (linux x86) (...)
     ...))

We still print only the evaluated formulas, and rely on a new construct for deduplication.

EDIT: I don't know if the idea above is the right one, but I think this idea that you've proposed to solve for multiple configurations does not require the full generality of evaluating an opam formula during build time. In the end, we're going from 1 configuration to some relatively small n - far less the set of all valid configurations we tried to cover. We could perhaps make use of this to simplify our solution. I would also like some consistency between the handling of actions and dependencies.

[rgrinberg]

Do you have more details about which parts of opam's formulae we don't replicate in dune? Was it the way that opam's undefined values propagate through boolean ops? (If so we've solved that problem).

Yes, that is the problem I'm talking about. If we have solved it, I don't remember being satisfied with the end result.

Let me know if you'd rather I put the new behaviour behind a feature flag.

Given that it doesn't provide value until we generate build plans that take advantage of it, being behind a feature flag is probably the way to go.

[gridbugs]

One solution that would fork for dependencies as well as build actions and would side step the evaluation problem above is introducing a conditional expression that would allow us to dispatch based on the configuration tuples. E.g:

(depends
  (match %{os} %{arch}
    (win arch64) (...)
    (linux arch64) (linux x86) (...)
     ...))

We still print only the evaluated formulas, and rely on a new construct for deduplication.

EDIT: I don't know if the idea above is the right one, but I think this idea that you've proposed to solve for multiple configurations does not require the full generality of evaluating an opam formula during build time. In the end, we're going from 1 configuration to some relatively small n - far less the set of all valid configurations we tried to cover. We could perhaps make use of this to simplify our solution. I would also like some consistency between the handling of actions and dependencies.

Given your concerns about deferring evaluation of platform vars, and the fact that we'll need something like the above for depends anyway, I agree it makes sense to use the same mechanism for commands. I'll close this PR and experiment with something resembling a match statement for all 3 fields.

[rgrinberg]

One more note: maybe were my concerns about evaluating platform vars premature? In this case, we can use the fact that we know the values for all platform variables ahead of time to make sure we don't need to deal with any weirdness from undefined variables. Maybe that could be used to simplify our evaluation semantics - the fact that all variables are defined, and in fact known ahead of time.

So I wouldn't necessarily rule out the original approach just yet. I don't have a preference for a solution just yet, but I would like us to perhaps spend a bit more time to appreciate the difference between this problem that we're solving (limited portability) and how it differs that we've tried to solve before (full portability).

[gridbugs]

Sure, I'll explore match statements as we'll need them for depends anyway and see how that solution compares to defering expansions of platform vars.

Also I am a little unclear on how deferring the evaluation of platform variables will cause problems relating to possible differences in how dune and opam handle undefined vars. We already defer the evaluation of certain variables until build time (e.g. make, jobs), and not all variables are deferred in my solution (e.g. with-test and with-doc are still evaluated at solve-time). Platform variables are never undefined.