Add a MillBuildModule and improve BSP for build.sc

[lefou]

The idea is to have a Module that can read and build Mill projects.

With such a module, we can model the build files compilation with Mill instead of Ammonite.

This also opens the door for further improvements. Incremental compilation and better BSP support comes to mind.

• Fix [RFC] Create a module that can compile and run Ammonite scripts and use it for Mill itself #1675

I experimentally also included support for using directives as comments, as they are used in Scala CLI. This is to play with it, but also to eventually replace the Ammonite-specific import syntax.

Tasks:

• Translate file and line numbers in compile error messages
• Write some tests

Caveats:

• Ammonite imports are currently parsed with simple regular expressions, which definitely don't catch all legal combinations (but only the most common form). We can clearly improve them or integrate a real parser (that's not my home playground, PRs welcome). Instead, I think we should focus on supporting using directives.

Try it out

Additional to the trait, I also created an external module mill.scalalib.buildfile.MillBuildModule, so you can easily test it with any project you want.

$ mill -i dev.run ../mill-bsp -i showNamed "mill.scalalib.buildfile.MillBuildModule/{millVersion,sources,allSourceFiles,wrappedSourceFiles,millBuildSourceFiles,includedSourceFiles,compile,semanticDbData}"
[40/844] de.tobiasroeser.mill.vcs.version.VcsVersion.vcsState.overridden.de.tobiasroeser.mill.vcs.version.VcsVersion.vcsState 
[844/844] dev.run 
[1/1] showNamed 
[1/1] showNamed > [47/52] mill.scalalib.buildfile.MillBuildModule.semanticDbVersion 
{
  "mill.scalalib.buildfile.MillBuildModule.millVersion": "0.10.7",
  "mill.scalalib.buildfile.MillBuildModule.sources": [
    "ref:cb2806e2:/home/lefou/work/opensource/mill-bsp/.mill-version",
    "ref:c984eca8:/home/lefou/work/opensource/mill-bsp/.config/mill-version",
    "ref:c984eca8:/home/lefou/work/opensource/mill-bsp/.mill-jvm-opts",
    "ref:e0965e05:/home/lefou/work/opensource/mill-bsp/build.sc",
    "ref:60f41993:/home/lefou/work/opensource/mill-bsp/ci/shared.sc",
    "ref:5b62e47c:/home/lefou/work/opensource/mill-bsp/ci/upload.sc"
  ],
  "mill.scalalib.buildfile.MillBuildModule.allSourceFiles": [
    "ref:e0965e05:/home/lefou/work/opensource/mill-bsp/build.sc",
    "ref:60f41993:/home/lefou/work/opensource/mill-bsp/ci/shared.sc",
    "ref:5b62e47c:/home/lefou/work/opensource/mill-bsp/ci/upload.sc"
  ],
  "mill.scalalib.buildfile.MillBuildModule.wrappedSourceFiles": [
    {
      "orig": "ref:e0965e05:/home/lefou/work/opensource/mill-bsp/build.sc",
      "wrapped": [
        "ref:3f526faa:/home/lefou/work/opensource/mill-bsp/out/mill/scalalib/buildfile/MillBuildModule/wrappedSourceFiles.dest/build.sc.scala"
      ]
    },
    {
      "orig": "ref:60f41993:/home/lefou/work/opensource/mill-bsp/ci/shared.sc",
      "wrapped": [
        "ref:a59621d9:/home/lefou/work/opensource/mill-bsp/out/mill/scalalib/buildfile/MillBuildModule/wrappedSourceFiles.dest/shared.sc.scala"
      ]
    },
    {
      "orig": "ref:5b62e47c:/home/lefou/work/opensource/mill-bsp/ci/upload.sc",
      "wrapped": [
        "ref:03e8d55c:/home/lefou/work/opensource/mill-bsp/out/mill/scalalib/buildfile/MillBuildModule/wrappedSourceFiles.dest/upload.sc.scala"
      ]
    }
  ],
  "mill.scalalib.buildfile.MillBuildModule.millBuildSourceFiles": [
    "ref:e0965e05:/home/lefou/work/opensource/mill-bsp/build.sc",
    "ref:60f41993:/home/lefou/work/opensource/mill-bsp/ci/shared.sc",
    "ref:5b62e47c:/home/lefou/work/opensource/mill-bsp/ci/upload.sc"
  ],
  "mill.scalalib.buildfile.MillBuildModule.includedSourceFiles": [
    "ref:60f41993:/home/lefou/work/opensource/mill-bsp/ci/shared.sc",
    "ref:5b62e47c:/home/lefou/work/opensource/mill-bsp/ci/upload.sc"
  ],
  "mill.scalalib.buildfile.MillBuildModule.compile": {
    "analysisFile": "/home/lefou/work/opensource/mill-bsp/out/mill/scalalib/buildfile/MillBuildModule/compile.dest/zinc",
    "classes": "ref:c4b58a29:/home/lefou/work/opensource/mill-bsp/out/mill/scalalib/buildfile/MillBuildModule/compile.dest/classes"
  },
  "mill.scalalib.buildfile.MillBuildModule.semanticDbData": "ref:87bdfae1:/home/lefou/work/opensource/mill-bsp/out/mill/scalalib/buildfile/MillBuildModule/semanticDbData.dest/classes"
}

[lefou]

New release of using_directives is out. https://github.com/VirtusLab/using_directives/releases/tag/v0.0.9

[kubukoz]

@lefou my team is quite affected by Metals not working in build.sc (we're forced to use intellij, which has its well-known issues). Is there something I/we can do to help get this over the line?

[lefou]

@lefou my team is quite affected by Metals not working in build.sc (we're forced to use intellij, which has its well-known issues). Is there something I/we can do to help get this over the line?

This PR is currently not suitable for merging. I think I made some mistake in the latest rebase, so it fails most CI jobs. Also, the wrapping of the *.sc isn't correct.

I currently use IntelliJ via GenIdea/idea for all of my Mill-based projects, and I'm quite happy with it. What issues do you have? Are they reported? The only real issue I'm aware of is bad support for Cross modules, which use the same source files. You can only have one of each, which I typically control with the skipIdea target.

[kubukoz]

What issues do you have? Are they reported?

if you're asking about IDEA itself...

1. it's not Metals ;) personal preference, bloat etc.
2. still fails to type some things, that's been reported years ago I believe.

GenIdea/idea works fine.

[lefou]

@kubukoz I created PR #2291 which should improve the editing experience in Metals.

[kubukoz]

I tried that (after a somewhat confusing process of getting the locally published version work in the launcher), but it didn't seem to have much of an effect on our build :/

[lefou]

I tried that (after a somewhat confusing process of getting the locally published version work in the launcher), but it didn't seem to have much of an effect on our build :/

Oh, sorry to hear that. If I cut a 0.11.0-M4, would you be able to test with it more easily? You could then open a new discussion with the issues you see and maybe some details about your setup.

[kubukoz]

Yes, a release would help (I had issues getting millw to work with snapshots) :) I will then try to minimize the things that make it not work.

[lefou]

We already have a released version 0.11.0-M3-22-ac0ea4 which includes PR #2291.

[kubukoz]

okay, tried that and it still didn't work - posted a minimal reproduction in #2304.

[lefou]

BSP support is already improved by

• BSP: Improve support for mill-build module and the build.sc #2291

This PR needs more work, including a rebase, so I'm closing it for now.