Scala 3.3.0

.github/workflows/pr.yml

@@ -36,7 +36,7 @@ jobs:
           distribution: 'temurin'
           java-version: '19'
       - name: Check formatting
-        run: sbt ++2.13.8 scalafmtCheck test:scalafmtCheck
+        run: sbt scalafmtCheck Test/scalafmtCheck
       - run: echo "Previous step failed because code is not formatted. Run 'sbt scalafmt Test/scalafmt'"
         if: ${{ failure() }}
   test-scripts:

.scalafmt.conf

@@ -1,5 +1,5 @@
 version = 3.7.4
-runner.dialect = scala213
+runner.dialect = scala3
 preset = IntelliJ
 maxColumn = 120
 align.preset = true

README.md

@@ -23,6 +23,7 @@ Specification: https://cpg.joern.io
 
 ## News / Changelog
 
+- Joern v2.0.0 [upgrades from Scala2 to Scala3](changelog/2.0.0-scala3.md)
 - Joern v1.2.0 removes the `overflowdb.traversal.Traversal` class. This change is not completely backwards compatible. See [here](changelog/traversal_removal.md) for a detailed writeup.
 
 ## Requirements
@@ -42,20 +43,16 @@ chmod +x ./joern-install.sh
 sudo ./joern-install.sh
 joern
 
-Compiling (synthetic)/ammonite/predef/interpBridge.sc
-Compiling (synthetic)/ammonite/predef/replBridge.sc
-Compiling (synthetic)/ammonite/predef/DefaultPredef.sc
-Compiling /home/tmp/shiftleft/joern/(console)
-
      ██╗ ██████╗ ███████╗██████╗ ███╗   ██╗
      ██║██╔═══██╗██╔════╝██╔══██╗████╗  ██║
      ██║██║   ██║█████╗  ██████╔╝██╔██╗ ██║
 ██   ██║██║   ██║██╔══╝  ██╔══██╗██║╚██╗██║
 ╚█████╔╝╚██████╔╝███████╗██║  ██║██║ ╚████║
  ╚════╝  ╚═════╝ ╚══════╝╚═╝  ╚═╝╚═╝  ╚═══╝
+Version: 2.0.1
+Type `help` to begin
 
 joern>
-
 ```
 
 If the installation script fails for any reason, try
@@ -126,3 +123,4 @@ are:
 For more instructions on how to run benchmarks individually head over to the `benchmarks` subproject. If you would
 like the benchmark results to be written to a file instead of printed to STDOUT, set the path to the environment
 variable `JOERN_BENCHMARK_RESULT_FILE`.
+

benchmarks/build.sbt

@@ -1,7 +1,5 @@
 name := "benchmarks"
 
-crossScalaVersions := Seq("2.13.8", "3.3.0")
-
 dependsOn(Projects.dataflowengineoss)
 dependsOn(Projects.semanticcpg)
 dependsOn(Projects.console)

build.sbt

@@ -1,8 +1,8 @@
 name                     := "joern"
 ThisBuild / organization := "io.joern"
-ThisBuild / scalaVersion := "2.13.8"
+ThisBuild / scalaVersion := "3.3.0"
 
-val cpgVersion = "1.3.612"
+val cpgVersion = "1.4.1"
 
 lazy val joerncli          = Projects.joerncli
 lazy val querydb           = Projects.querydb

c2cpg.sh

@@ -0,0 +1 @@
+joern-cli/frontends/c2cpg/c2cpg.sh

changelog/2.0.0-scala3.md

@@ -0,0 +1,60 @@
+# 2.0.x: Upgrade to Scala 3
+Joern is based on Scala. As of Joern 2.0.x we [upgraded from Scala 2 to Scala 3](https://github.com/joernio/joern/pull/2595/files). 
+This is a major version upgrade and Scala 3 is essentially a new language with a new REPL implementation, so this may sound scary. 
+
+That being said, both the Scala as well as Joern maintainers have made an effort to minimize changes to the API, in order to ease the transition for users. Most importantly, the Joern workspace DSL (`importCode(...)` etc.) and the CPG Traversal DSL (e.g. `cpg.method.name("foo").l`) are unchanged. The latter is based on Scala collections API, which is actually identical (a shared library) between Scala 2 and Scala 3. 
+
+Depending on your use case you may not even notice a difference: we've tried to keep as much as possible just like it was - most importantly the query DSL.
+There are however a few changes - some that we believe are the better, and some were just unavoidable. Here's the most important ones as far as I can tell:
+
+1) The 'import a script' magic `$file.foo` from Ammonite was replaced by the `//> using file foo.sc` directive. This works in an active joern REPL as well as in scripts.
+
+2) Adding dependencies: the magic `$ivy.my-dependency` was replaced:
+* `--dep` parameter, e.g. `./joern --dep com.michaelpollmeier:versionsort:1.0.7`. Can be specified multiple times.
+* For scripts there's a slightly nicer alternative that let's you specify your dependencies within the script itself: `//> using dep com.michaelpollmeier:versionsort:1.0.7`
+* all dependencies need to be known when joern starts, i.e. you can not dynamically add more dependencies to an active joern REPL session
+
+3) Script parameters: pass multiple `--param` parameters rather than one comma-separated list. Example:
+```
+// old
+./joern --script foo.sc --params paramA=valueA,paramB=valueB
+// new
+./joern --script foo.sc --param paramA=valueA --param paramB=valueB
+```
+While that's slightly longer it is also less complex, easier to read, and you can pass values that contain commas :)
+
+Apart from that, Scala 3 is a bit stricter when it comes to adding or leaving out `()` for function application. The compiler messages are (on average) much better than before, so hopefully it'll guide you as good as possible. 
+
+Depending on your level of integration with Joern you might not even notice anything. If you do, please check the lists above and below, and if that doesn't help: open a [github issue](https://github.com/joernio/joern/issues/new) or hit us up on [discord](https://discord.gg/vv4MH284Hc).
+
+Some more generic Scala-issues when upgrading Scala 2 to Scala 3:
+
+1. anonymous functions need an extra parenthesis around their parameter list:
+```scala
+Seq(1,2,3).map { i: Int => i + 1 }   
+// error: parentheses are required around the parameter of a lambda
+
+// option 1: add parentheses, as suggested by compiler:
+Seq(1,2,3).map { (i: Int) => i + 1 }
+
+// option 2: drop type annotation (if possible):
+Seq(1,2,3).map { i => i + 1 }
+```
+
+2. `main` entrypoint: `def main` instead of `extends App`
+See https://docs.scala-lang.org/scala3/book/methods-main-methods.html
+```scala
+object Main extends App {
+  println("hello world")
+}
+
+// depending on usage, may lead to NullPointerExceptions
+// context: Scala3 doesn't support the 'magic' DelayedInit trait
+
+// rewrite to:
+object Main {
+  def main(args: Array[String]) = {
+    println("hello world")
+  }
+}
+```

console/build.sbt

@@ -5,7 +5,6 @@ enablePlugins(JavaAppPackaging)
 val ScoptVersion          = "4.1.0"
 val CaskVersion           = "0.8.3"
 val CirceVersion          = "0.14.5"
-val AmmoniteVersion       = "2.5.8"
 val ZeroturnaroundVersion = "1.15"
 
 dependsOn(
@@ -19,58 +18,17 @@ dependsOn(
 )
 
 libraryDependencies ++= Seq(
-  "io.shiftleft"      %% "codepropertygraph" % Versions.cpg,
-  "com.github.scopt"  %% "scopt"             % ScoptVersion,
-  "org.typelevel"     %% "cats-effect"       % Versions.cats,
-  "io.circe"          %% "circe-generic"     % CirceVersion,
-  "io.circe"          %% "circe-parser"      % CirceVersion,
-  "org.zeroturnaround" % "zt-zip"            % ZeroturnaroundVersion,
-  "com.lihaoyi"       %% "ammonite"          % AmmoniteVersion cross CrossVersion.full,
-  "com.lihaoyi"       %% "os-lib"            % "0.8.1",
-  "com.lihaoyi"       %% "cask"              % CaskVersion,
-  "org.scalatest"     %% "scalatest"         % Versions.scalatest % Test
+  "com.michaelpollmeier" %% "scala-repl-pp-all" % "0.1.32",
+  "io.shiftleft"         %% "codepropertygraph" % Versions.cpg,
+  "com.github.scopt"     %% "scopt"             % ScoptVersion,
+  "org.typelevel"        %% "cats-effect"       % Versions.cats,
+  "io.circe"             %% "circe-generic"     % CirceVersion,
+  "io.circe"             %% "circe-parser"      % CirceVersion,
+  "org.zeroturnaround"    % "zt-zip"            % ZeroturnaroundVersion,
+  "com.lihaoyi"          %% "os-lib"            % "0.8.1",
+  "com.lihaoyi"          %% "pprint"            % "0.7.3",
+  "com.lihaoyi"          %% "cask"              % CaskVersion,
+  "org.scalatest"        %% "scalatest"         % Versions.scalatest % Test
 )
 
-Test / compile := (Test / compile).dependsOn((Projects.c2cpg / stage)).value
-
-scalacOptions ++= Seq(
-  "-deprecation", // Emit warning and location for usages of deprecated APIs.
-  "-encoding",
-  "utf-8",                  // Specify character encoding used by source files.
-  "-explaintypes",          // Explain type errors in more detail.
-  "-feature",               // Emit warning and location for usages of features that should be imported explicitly.
-  "-language:existentials", // Existential types (besides wildcard types) can be written and inferred
-  "-language:experimental.macros", // Allow macro definition (besides implementation and application)
-  "-language:higherKinds",         // Allow higher-kinded types
-  "-language:implicitConversions", // Allow definition of implicit functions called views
-  "-unchecked",                    // Enable additional warnings where generated code depends on assumptions.
-  "-Xcheckinit",                   // Wrap field accessors to throw an exception on uninitialized access.
-  "-Xlint:adapted-args",           // Warn if an argument list is modified to match the receiver.
-  "-Xlint:constant",               // Evaluation of a constant arithmetic expression results in an error.
-  "-Xlint:delayedinit-select",     // Selecting member of DelayedInit.
-  "-Xlint:doc-detached",           // A Scaladoc comment appears to be detached from its element.
-  "-Xlint:inaccessible",           // Warn about inaccessible types in method signatures.
-  "-Xlint:infer-any",              // Warn when a type argument is inferred to be `Any`.
-  "-Xlint:missing-interpolator",   // A string literal appears to be missing an interpolator id.
-  "-Xlint:option-implicit",        // Option.apply used implicit view.
-  "-Xlint:package-object-classes", // Class or object defined in package object.
-  "-Xlint:poly-implicit-overload", // Parameterized overloaded implicit methods are not visible as view bounds.
-  "-Xlint:private-shadow",         // A private field (or class parameter) shadows a superclass field.
-  "-Xlint:stars-align",            // Pattern sequence wildcard must align with sequence component.
-  "-Xlint:type-parameter-shadow",  // A local type parameter shadows a type already in scope.
-  "-Ywarn-dead-code",              // Warn when dead code is identified.
-  "-Ywarn-extra-implicit",         // Warn when more than one implicit parameter section is defined.
-  "-Ywarn-numeric-widen",          // Warn when numerics are widened.
-  "-Ywarn-unused:implicits",       // Warn if an implicit parameter is unused.
-  "-Ywarn-unused:imports",         // Warn if an import selector is not referenced.
-  "-Ywarn-unused:locals",          // Warn if a local definition is unused.
-  "-Ywarn-unused:params",          // Warn if a value parameter is unused.
-  "-Ywarn-unused:patvars",         // Warn if a variable bound in a pattern is unused.
-  "-Ywarn-unused:privates",        // Warn if a private member is unused.
-  "-Yrangepos"
-)
-
-// would love to reenable, but somehow StorageBackend.scala triggers a strange `[warn] method with a single empty parameter list overrides method without any parameter list` that doesn't make sense to me...
-scalacOptions -= "-Xfatal-warnings"
-
-Test / fork := false
+Test/compile := (Test/compile).dependsOn((Projects.c2cpg/stage)).value