Webjars broken with Quarkus 3.16 due to an exception in Classgraph

[triceo]

In code that worked with Quarkus 3.15, I am now getting the following exception after upgrading to Quarkus 3.16:

java.lang.RuntimeException: Exception while calling findClassLoaderOrder for nonapi.io.github.classgraph.classloaderhandler.QuarkusClassLoaderHandler
at nonapi.io.github.classgraph.classloaderhandler.ClassLoaderHandlerRegistry$ClassLoaderHandlerRegistryEntry.findClasspathOrder(ClassLoaderHandlerRegistry.java:239)
at nonapi.io.github.classgraph.classpath.ClasspathFinder.<init>(ClasspathFinder.java:277)
at io.github.classgraph.Scanner.<init>(Scanner.java:170)
at io.github.classgraph.ClassGraph.scanAsync(ClassGraph.java:1562)
at io.github.classgraph.ClassGraph.scanAsync(ClassGraph.java:1590)
at io.github.classgraph.ClassGraph.scan(ClassGraph.java:1615)
at io.github.classgraph.ClassGraph.scan(ClassGraph.java:1654)
at io.github.classgraph.ClassGraph.scan(ClassGraph.java:1667)
at org.webjars.WebJarAssetLocator.scanForWebJars(WebJarAssetLocator.java:188)
at org.webjars.WebJarAssetLocator.<init>(WebJarAssetLocator.java:210)
at org.webjars.WebJarAssetLocator.<init>(WebJarAssetLocator.java:198)
...
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:115)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at nonapi.io.github.classgraph.classloaderhandler.ClassLoaderHandlerRegistry$ClassLoaderHandlerRegistryEntry.findClasspathOrder(ClassLoaderHandlerRegistry.java:236)
... 21 more
Caused by: java.lang.NullPointerException: Cannot invoke "java.util.Collection.iterator()" because the return value of "nonapi.io.github.classgraph.reflection.ReflectionUtils.getFieldVal(boolean, Object, String)" is null
at nonapi.io.github.classgraph.classloaderhandler.QuarkusClassLoaderHandler.findClasspathOrderForQuarkusClassloader(QuarkusClassLoaderHandler.java:120)
at nonapi.io.github.classgraph.classloaderhandler.QuarkusClassLoaderHandler.findClasspathOrder(QuarkusClassLoaderHandler.java:111)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
... 23 more

The following code in Classgraph seems to trigger the exception:

for (final Object element : (Collection<Object>) classpathOrder.reflectionUtils.getFieldVal(false,
            classLoader, "elements")) {
     ...
}

I'm assuming some behavior of the Quarkus ClassLoader changed in 3.16? I do not know any details about this though.

I attempted manually forcing the latest version of Classgraph (4.8.177), but exception persists.

$ java -version
openjdk version "21.0.5" 2024-10-15 LTS
OpenJDK Runtime Environment Temurin-21.0.5+11 (build 21.0.5+11-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.5+11 (build 21.0.5+11-LTS, mixed mode, sharing)

[lukehutch]

You correctly diagnosed this. classLoader.elements is null (or more likely no longer exists) in the new classloader.

To figure out where to find the class path elements in the new classloader, you would have to set a breakpoint at the line that you indicated, and then use the debugger to look at the fields and methods of the classloader.

You would then have to change the code in ClassGraph to work with both the old and the new classloader.

I don't know anything about Quarkus, so I would appreciate it if you could please put together a pull request for this. Thank you!

[triceo]

As I am not skilled in Quarkus either, I took the liberty of asking a question in their Discussions:
quarkusio/quarkus#44283

Let's see where that goes.

[gsmet]

The breakage was my doing. This field was internal and not exposed and I needed a more practical way to handle normal priority elements and lesser priority elements, thus why you have two fields now.

The QuarkusClassLoader is the one used in dev mode and test mode and is quite complex.

I think we could probably expose something that ClassGraph could use, but I'm not entirely sure what we should expose. The notion of bannedElements for instance wasn't implemented here AFAICS and also I'm not sure you are able to render the memory class path elements (i.e. elements that are solely in memory)?

Anyway, I'm interested in having a discussion around this to see if we can come up with some sort of SPI that you could consume.

[dmlloyd]

Have you tried a general solution like: classLoader.getResources("META-INF/MANIFEST.MF") or something like that to locate the JARs?

[lukehutch]

@dmlloyd that won't work in the general case. You would miss some classpath elements, when they don't include a manifest file (it's not required). But it may be helpful to add that as a fallback. Good suggestion.

[lukehutch]

@gsmet thanks for the productive reply. Are the in-memory classpath entries openable via URL, Path, or similar?

Even starting with the basic on-disk jars would be better than nothing.

[dmlloyd]

Other things that might work in different circumstances:

• classLoader.getResources("META-INF")
• classLoader.getResources("/") (works for some cases where the class path entry is backed by the file system)
• classLoader.getResources("module-info.class") (finds modules on the class path fairly reliably)

[michael-simons]

Neo4j-OGM falls over on this, too. We like to move forward, and I am gonna suggest a naive fix with #893 for the time being, which I tested with Neo4j-OGM both against Quarkus 3.15.x and 3.16.1 successfully.

This would help me and our customers quite a lot. I know it's not a long time solution, but most likely good enough and not worse than before.

[michael-simons]

I think we could probably expose something that ClassGraph could use, but I'm not entirely sure what we should expose. The notion of bannedElements for instance wasn't implemented here AFAICS

That's right.

Yeah, an API on it would be quite helpful, and I'd happily help using it, but cannot answer your question on what to expose with certainty.
Happy to see you here, though, @gsmet 👍

[gsmet]

Are the in-memory classpath entries openable via URL, Path, or similar?

Well, not exactly but... they are generated bytecode and basically a Map<String, byte[]> with the path of the resource and the generated bytecode.

The QuarkusClassLoader is used when running tests and in dev mode and doesn't write everything on disk (compared to the other class loaders where we could implement things a bit easier).

Even starting with the basic on-disk jars would be better than nothing.

So what @michael-simons did in https://github.com/classgraph/classgraph/pull/893/files as a workaround won't be worse than what you have atm.

But... keep in mind that the elements we have in memory can be additional classes but can also be classes that override an existing class. Thus you will have a partial and incorrect view of the classpath.

Would it be possible for you to also handle byte[] elements? We don't have that many of them but if we want to provide a clean API for you to integrate with the class loader, I don't think we should ignore them.

I wonder if some sort of ClasspathVisitor which would trigger events for String, byte[], String, Path/URL/whatever works best for jars, String, Path for local directories would work for you?

[dmlloyd]

They should actually be openable by URL instance I think (we set the handler IIRC), just not by synthetic URL that didn't come from our class loader.

[gsmet]

@dmlloyd my understanding is that they want to list all the resources/classes available in this class loader, not get a specific one.

[michael-simons]

Exactly! Am 08.11.2024 um 16:08 schrieb Guillaume Smet ***@***.***>:  @dmlloyd <https://github.com/dmlloyd> my understanding is that they want to list all the resources/classes available in this class loader, not get a specific one. — Reply to this email directly, view it on GitHub <#891 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEAQLYVBLMBVQMRQWLQWILZ7THWRAVCNFSM6AAAAABRDVFUIKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRUHE4TGMRTHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[lukehutch]

But... keep in mind that the elements we have in memory can be additional classes but can also be classes that override an existing class. Thus you will have a partial and incorrect view of the classpath.

Duly noted, and that will probably be the source of an obscure bug for somebody, far down the line, if in-memory class definitions are not supported by ClassGraph...

Unfortunately we'll probably have to just move forward with a best current effort here.

[lukehutch]

Would it be possible for you to also handle byte[] elements? We don't have that many of them but if we want to provide a clean API for you to integrate with the class loader, I don't think we should ignore them.

I wonder if some sort of ClasspathVisitor which would trigger events for String, byte[], String, Path/URL/whatever works best for jars, String, Path for local directories would work for you?

With the current ClassGraph API, there is always a URL that can be fetched for a classpath element that is known to exist. So having classpath elements from which classes can be loaded, but that don't have a valid URL, would not work with the current ClassGraph API.

There's another issue here though, which is that you are exposing individual classes with byte[], not classpath elements (where a classpath element is the URL or Path or File corresponding to the root of a directory tree full of classes). ClassGraph works with classpath elements, not individual classes. So yes, this is true:

They should actually be openable by URL instance I think (we set the handler IIRC), just not by synthetic URL that didn't come from our class loader.

@dmlloyd my understanding is that they want to list all the resources/classes available in this class loader, not get a specific one.

So we would need a URL corresponding to an implementation of Path, or a URL-handler that can find the class definition given a URL instance, not the URL for the individual class definition byte[] array.

I don't think it would be hard to write that URL-handler, but I don't have the bandwidth to do it right now.

[michael-simons]

I can provide the URL handler, and as it is that time pressing, add it to use any future API…

[lukehutch]

@dmlloyd @triceo I am pushing out the updated ClassGraph 4.8.178 now with the fix (which simply includes jars from the two new fields). @michael-simons thank you for your work on the fix!