Spring Boot 2.5 WAR app fails to deploy in Wildfly due to "ClassNotFoundException: # Licensed to the Apache ..."

[stevestorey]

#24744 switched the default EL implementation from glassfish to tomcat-el-embed. However, the tomcat implementation appears to violate the spec (https://docs.oracle.com/javaee/7/api/javax/el/ExpressionFactory.html#newInstance-- ):

Use the Services API (as detailed in the JAR specification). If a resource with the name of META-INF/services/javax.el.ExpressionFactory exists, then its first line, if present, is used as the UTF-8 encoded name of the implementation class.

The trouble being that the first line of that file in the TC implementation is indeed a comment indicating it's licenced under Apache 2.0 (I can't find a direct link to show this in their git repo so it may be generated, or a service to directly link to the file within the built JAR, but I'll repeat the whole file as I see it inside tomcat-embed-el-9.0.46.jar:

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

org.apache.el.ExpressionFactoryImpl

So I believe the issue is that the wildfly module system overrides the tomcat-embed-el implementation of javax.el.ExpressionFactory with the version from JBoss (https://github.com/jboss/jboss-jakarta-el-api_spec) which seems to be much stricter when it comes to the spec adherence (along with the Jakarta RI), and so ...

org.jboss.msc.service.StartException: java.lang.RuntimeException: javax.el.ELException: Provider # Licensed to the Apache Software Foundation (ASF) under one or more not found
        at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:81)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at [email protected]//org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at [email protected]//org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
        at [email protected]//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
        at [email protected]//org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
        at java.base/java.lang.Thread.run(Thread.java:831)
        at [email protected]//org.jboss.threads.JBossThread.run(JBossThread.java:513)
Caused by: java.lang.RuntimeException: javax.el.ELException: Provider # Licensed to the Apache Software Foundation (ASF) under one or more not found
        at [email protected]//io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:257)
        at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:96)
        at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:78)
        ... 8 common frames omitted
Caused by: javax.el.ELException: Provider # Licensed to the Apache Software Foundation (ASF) under one or more not found
        at [email protected]//javax.el.FactoryFinder.newInstance(FactoryFinder.java:64)
        at [email protected]//javax.el.FactoryFinder.find(FactoryFinder.java:103)
        at [email protected]//javax.el.ExpressionFactory.newInstance(ExpressionFactory.java:140)
        at [email protected]//javax.el.ELUtil.getExpressionFactory(ELUtil.java:185)
        at [email protected]//javax.el.ELManager.getExpressionFactory(ELManager.java:38)
        at [email protected]//org.apache.jasper.runtime.JspApplicationContextImpl.<init>(JspApplicationContextImpl.java:49)
        at [email protected]//org.apache.jasper.runtime.JspApplicationContextImpl.getInstance(JspApplicationContextImpl.java:78)
        at [email protected]//org.apache.jasper.runtime.JspFactoryImpl.getJspApplicationContext(JspFactoryImpl.java:218)
        at [email protected]//org.wildfly.extension.undertow.deployment.JspInitializationListener.contextInitialized(JspInitializationListener.java:51)
        at [email protected]//io.undertow.servlet.core.ApplicationListeners.contextInitialized(ApplicationListeners.java:187)
        at [email protected]//io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:219)
        at [email protected]//io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:187)
        at [email protected]//io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
        at [email protected]//io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
        at [email protected]//org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
        at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
        at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
        at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
        at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
        at [email protected]//org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1530)
        at [email protected]//io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:255)
        ... 10 common frames omitted
Caused by: java.lang.ClassNotFoundException: # Licensed to the Apache Software Foundation (ASF) under one or more from [Module "deployment.services-boot.war" from Service Module Loader]
        at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:255)
        at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:410)
        at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
        at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:116)
        at [email protected]//javax.el.FactoryFinder.newInstance(FactoryFinder.java:48)
        ... 30 common frames omitted

Underneath, this is likely a Tomcat issue (on the basis that both the JBoss impl + Jakarta impl strictly read the first line of the file and use that - https://github.com/jboss/jboss-jakarta-el-api_spec/blob/master/api/src/main/java/org/jboss/el/cache/FactoryFinderCache.java#L99), but SB 2.5 users deploying into an environment where javax.el.ExpressionFactory uses the JBoss/Jakarta version are going to notice the problem sooner I guess. I'm raising this
a. for visibility,
b. to pick up an updated Tomcat dependency as/when/if they move the licence down in the file or remove it entirely. (or if Spring wants to reverse the decision until Tomcat fixes its file?)
c. to list potential workarounds for others who run into the issue.

Possible workarounds:

1. effectively remove the Spring Boot provided tomcat-el-embed dependency and replace with the org.glassfish:jakarta-el implementation again, reversing the effects of Switch to Apache EL implementation by default #24744 - this works because the JBoss module system also has a fallback class of com.sun.el.ExpressionFactoryImpl defined which is provded by jakarta-el
2. Create your own META-INF/services/javax.el.ExpressionFactory file in the application WAR where the first line really is org.apache.el.ExpressionFactoryImpl (I would guess this carries some risks about which file will be picked up by the service loader first, but one would presume WAR resources will be picked ahead of resources inside WEB-INF/lib/*.jar. Seems to work for me for the mo ...)
3. Setup <local-last value="false" /> in a src/main/webapp/META-INF/jboss-deployment-structure.xml file and ensure that jakarta-el implementation is not also on the classpath of the WAR application itself.

[stevestorey]

I've kicked off a discussion in the Tomcat users mailing list as a starting point: https://www.mail-archive.com/[email protected]/msg137648.html

[snicoll]

That starting point is where this should have been raised in the first place. I can see some activity on that thread so I am going to close this one for now. If it turns out something needs to be done in Spring Boot we can reopen.

[stevestorey]

Fair enough on the close. Given the reply from the Tomcat devs is that it's not a spec violation (and that it is the EE7 RI that's broken) I can't see this being resolved any time soon by either side :). The latest EE spec has now been updated this morning to remove the offending specification line from the latest spec: jakartaee/expression-language@965f5ee - which was also already fixed in this regard: jakartaee/expression-language@6075bc9 as as and when people move to the 4.x spec, this problem will disappear anyway.