Copying a directory's contents and promoting the results

[cpitclaudel]

Consider the following setup: I'm writing a compiler xcomp from language X to C; I have an examples folder, which contains prog1 … progN. To run an example, I run xcomp examples/progN -o examples/_objects/progN/, which produces plenty of output files, including a progN.c file and a Makefile. Some of these example programs don't take further inputs; others need extra data, so there are a few folders progN.etc/, containing extra files.

In my Makefile-based setup, I run xcomp examples/progN -o examples/_objects/progN/, then cp -R examples/progN.etc/* examples/_objects/progN/, and finally make run in examples/_objects/progN/.

I'm moving this to Dune. I wrote a script to generate a long Dune file with one subdir stanza per example, like this:

(subdir "_objects/prog8"
 (rule
 (mode (promote (until-clean)))
 (deps (:prog ../../prog8)
       (:etc (glob_files ../../prog8.etc/*)))
 (targets
  prog8.cpp
  prog8.hpp
  prog8.v
  prog8.dot
  support.hpp
  Makefile)
 (action
  (chdir %{workspace_root}
   (run xcomp %{prog} -o examples/_objects/prog8)))))

But I can't find a way to also copy everything from examples/prog8.etc/ into examples/_objects/prog8. I could list all files from examples/prog8.etc in the target section, but that's going to get messy fairly quick (and unpleasant during development, when the set of files changes regularly), and Dune seems to already know how to do this for dependencies with glob_files. Basically I'd like something like this:

(subdir "_objects/prog8"
 (rule
 (mode (promote (until-clean)))
 (deps (:prog ../../prog8)
       (:etc (glob_files ../../prog8.etc/*)))
 (targets
  %{etc}
  prog8.cpp
  prog8.hpp
  prog8.v
  prog8.dot
  support.hpp
  Makefile)
 (action
  (chdir %{workspace_root}
   (copy_files %{etc} examples/_objects/prog8/)
   (run xcomp %{prog} -o examples/_objects/prog8)))))

… but obviously this doesn't work. I'm not too sure how to proceed. Is there an alternative approach I could take, using currently supported features? I think the key difficulty is telling dune to promote all files that end up in the examples/_objects/prog8 directory; that is, if I could just tell dune that my target is a complete folder, it would also work perfectly:

(subdir "_objects/prog8"
 (rule
 (mode (promote (until-clean)))
 (deps (:prog ../../prog8)
       (:etc (glob_files ../../prog8.etc/*)))
 (targets examples/_objects/prog8/)
 (action
  (chdir %{workspace_root}
   (bash "cp -R %{etc} examples/_objects/prog8/")
   (run xcomp %{prog} -o examples/_objects/prog8)))))

[ghost]

Directory targets are not supported yet, though @snowleopard has a plan for it.

In the meantime, you might be able to use the "foreign build sandboxing" method: https://dune.readthedocs.io/en/stable/foreign-code.html#foreign-build-sandboxing

[snowleopard]

Here is the current plan: #3316 (comment).

@cpitclaudel It would be helpful if you check whether the plan covers your use case and let us know.

[emillon]

A few years later, directory targets have landed and support promotion, see for example https://github.com/ocaml/dune/blob/3.13.0/test/blackbox-tests/test-cases/directory-targets/promote.t.