Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Race condition when initializing ClassAnalyzer with CdiInjecteeSkippingClassAnalyzer #559

Open
nhenneaux opened this issue Aug 17, 2021 · 14 comments

Comments

@nhenneaux
Copy link

For now several months, we are getting the following error for some of the builds we execute (I estimate it to 1-5% of all the builds).
It uses Jersey 3.0.2, Weld 4.0.2.Final with Java 16.
The error occurs when executing a JUnit test initializing CDI & JAX-RS server, similar to this test https://github.com/nhenneaux/jersey-httpclient-connector/blob/master/src/test/java/com/github/nhenneaux/jersey/connector/httpclient/JettyServerTest.java#L222

Could not find an implementation of ClassAnalyzer with name CdiInjecteeSkippingClassAnalyzer
	at org.jvnet.hk2.internal.ServiceLocatorImpl.getAnalyzer(ServiceLocatorImpl.java:2468)
	at org.jvnet.hk2.internal.Utilities.getClassAnalyzer(Utilities.java:143)
	at org.jvnet.hk2.internal.Utilities.justInject(Utilities.java:969)

I suspect a race condition occurs since the class analyzer uses a HashMap and not a thread safe map (java.util.concurrent.ConcurrentMap). Indeed one CPU core cache could see the map full and another the map empty.
https://github.com/eclipse-ee4j/glassfish-hk2/blob/master/hk2-locator/src/main/java/org/jvnet/hk2/internal/ServiceLocatorImpl.java#L2464

Full stacktrace

java.lang.IllegalStateException: jakarta.servlet.ServletException: org.glassfish.jersey.servlet.ServletContainer-f7acce1==org.glassfish.jersey.servlet.ServletContainer@e976386e{jsp=null,order=-1,inst=true,async=true,src=EMBEDDED:null,STARTED}
...
	at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: jakarta.servlet.ServletException: org.glassfish.jersey.servlet.ServletContainer-f7acce1==org.glassfish.jersey.servlet.ServletContainer@e976386e{jsp=null,order=-1,inst=true,async=true,src=EMBEDDED:null,STARTED}
	at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:651)
	at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:415)
	at org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$2(ServletHandler.java:690)
	at java.base/java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:357)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:312)
	at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:735)
	at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
	at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:714)
	at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:392)
	at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:891)
	at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:306)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89)
	at org.eclipse.jetty.server.handler.StatisticsHandler.doStart(StatisticsHandler.java:242)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:121)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:171)
	at org.eclipse.jetty.server.Server.start(Server.java:469)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
	at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:89)
	at org.eclipse.jetty.server.Server.doStart(Server.java:414)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:93)
...	... 9 more
Caused by: A MultiException has 1 exceptions.  They are:
1. java.lang.IllegalStateException: Could not find an implementation of ClassAnalyzer with name CdiInjecteeSkippingClassAnalyzer
	at org.jvnet.hk2.internal.Collector.throwIfErrors(Collector.java:65)
	at org.jvnet.hk2.internal.Utilities.justInject(Utilities.java:970)
	at org.jvnet.hk2.internal.ServiceLocatorImpl.inject(ServiceLocatorImpl.java:1014)
	at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.inject(AbstractHk2InjectionManager.java:207)
	at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.inject(ImmediateHk2InjectionManager.java:30)
	at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:769)
	at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:161)
	at org.jboss.weld.contexts.unbound.DependentContextImpl.get(DependentContextImpl.java:64)
	at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100)
	at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
	at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:694)
	at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:717)
	at org.jboss.weld.util.ForwardingBeanManager.getReference(ForwardingBeanManager.java:64)
	at org.jboss.weld.bean.builtin.BeanManagerProxy.getReference(BeanManagerProxy.java:87)
	at org.glassfish.jersey.ext.cdi1x.internal.CdiUtil.getBeanReference(CdiUtil.java:129)
	at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier$1.getInstance(AbstractCdiBeanSupplier.java:72)
	at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier._provide(AbstractCdiBeanSupplier.java:112)
	at org.glassfish.jersey.ext.cdi1x.internal.GenericCdiBeanSupplier.get(GenericCdiBeanSupplier.java:42)
	at org.glassfish.jersey.inject.hk2.InstanceSupplierFactoryBridge.provide(InstanceSupplierFactoryBridge.java:53)
	at org.jvnet.hk2.internal.FactoryCreator.create(FactoryCreator.java:129)
	at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:466)
	at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:46)
	at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2103)
	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:93)
	at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:67)
	at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.lambda$getAllServiceHolders$0(AbstractHk2InjectionManager.java:136)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
	at java.base/java.util.LinkedList$LLSpliterator.forEachRemaining(LinkedList.java:1242)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
	at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getAllServiceHolders(AbstractHk2InjectionManager.java:140)
	at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getAllServiceHolders(ImmediateHk2InjectionManager.java:30)
	at org.glassfish.jersey.internal.inject.Providers.getServiceHolders(Providers.java:307)
	at org.glassfish.jersey.internal.inject.Providers.getCustomProviders(Providers.java:151)
	at org.glassfish.jersey.server.internal.inject.ParamExtractorConfigurator.lambda$init$0(ParamExtractorConfigurator.java:49)
	at org.glassfish.jersey.internal.util.collection.Values$LazyValueImpl.get(Values.java:317)
	at org.glassfish.jersey.server.internal.inject.MultivaluedParameterExtractorFactory.get(MultivaluedParameterExtractorFactory.java:68)
	at org.glassfish.jersey.server.internal.inject.AbstractValueParamProvider.get(AbstractValueParamProvider.java:67)
	at org.glassfish.jersey.server.internal.inject.PathParamValueParamProvider.createValueProvider(PathParamValueParamProvider.java:71)
	at org.glassfish.jersey.server.internal.inject.AbstractValueParamProvider.getValueProvider(AbstractValueParamProvider.java:93)
	at org.glassfish.jersey.server.spi.internal.ParameterValueHelper.getParamValueProvider(ParameterValueHelper.java:148)
	at org.glassfish.jersey.server.spi.internal.ParameterValueHelper.createValueProviders(ParameterValueHelper.java:110)
	at org.glassfish.jersey.server.model.ResourceMethodValidator.checkValueProviders(ResourceMethodValidator.java:172)
	at org.glassfish.jersey.server.model.ResourceMethodValidator.checkMethod(ResourceMethodValidator.java:81)
	at org.glassfish.jersey.server.model.ResourceMethodValidator.visitJaxrsResourceMethod(ResourceMethodValidator.java:77)
	at org.glassfish.jersey.server.model.ResourceMethodValidator.visitResourceMethod(ResourceMethodValidator.java:67)
	at org.glassfish.jersey.server.model.ResourceMethod.accept(ResourceMethod.java:876)
	at org.glassfish.jersey.server.model.ComponentModelValidator.validateWithErrors(ComponentModelValidator.java:126)
	at org.glassfish.jersey.server.model.ComponentModelValidator.validateWithErrors(ComponentModelValidator.java:132)
	at org.glassfish.jersey.server.model.ComponentModelValidator.validateWithErrors(ComponentModelValidator.java:132)
	at org.glassfish.jersey.server.model.ComponentModelValidator.validateWithErrors(ComponentModelValidator.java:132)
	at org.glassfish.jersey.server.model.ComponentModelValidator.access$000(ComponentModelValidator.java:55)
	at org.glassfish.jersey.server.model.ComponentModelValidator$1.run(ComponentModelValidator.java:116)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.server.model.ComponentModelValidator.validate(ComponentModelValidator.java:111)
	at org.glassfish.jersey.server.ApplicationHandler.initialize(ApplicationHandler.java:366)
	at org.glassfish.jersey.server.ApplicationHandler.lambda$initialize$1(ApplicationHandler.java:293)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.processWithException(Errors.java:232)
	at org.glassfish.jersey.server.ApplicationHandler.initialize(ApplicationHandler.java:292)
	at org.glassfish.jersey.server.ApplicationHandler.<init>(ApplicationHandler.java:259)
	at org.glassfish.jersey.servlet.WebComponent.<init>(WebComponent.java:311)
	at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:154)
	at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:347)
	at jakarta.servlet.GenericServlet.init(GenericServlet.java:178)
	at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:633)
	... 38 more
Caused by: java.lang.IllegalStateException: Could not find an implementation of ClassAnalyzer with name CdiInjecteeSkippingClassAnalyzer
	at org.jvnet.hk2.internal.ServiceLocatorImpl.getAnalyzer(ServiceLocatorImpl.java:2468)
	at org.jvnet.hk2.internal.Utilities.getClassAnalyzer(Utilities.java:143)
	at org.jvnet.hk2.internal.Utilities.justInject(Utilities.java:969)
	... 110 more
@nhenneaux
Copy link
Author

#560 Merge request trying to fix this issue

@dmatej
Copy link
Contributor

dmatej commented Oct 21, 2021

Looks like something similar: payara/Payara#1160

@dmatej
Copy link
Contributor

dmatej commented Oct 21, 2021

It seems it affected much more users, year after year, but I didn't find any explanation except "something uses the container before it finished it's initialization" until now. I believe this is not the issue of HK2, but of the project using Jersey with HK2 where is something wrong.

The example project always passed all tests on my computer with JDK17 (except log messages from JaCoCo instrumentation; it passed also in eclipse without these messages).

But what finally attracted my eye is the class org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider which is in jersey-cdi1x project, while the example uses jersey-cdi2se. I would simply kick the cdi1x out of dependencies (is it doable?); I guess there may be some issue with your dependencies which causes that the container initializes services in random order (as the order is not defined), but I am not sure with it. I would really need some reproducer for that including all your dependencies.

However I am still convinced that using ConcurrentHashMap cannot resolve the problem, because all accesses to it are synchronized.

Simply said - it tries to inject something bound to the CdiInjecteeSkippingClassAnalyzer while CdiComponentProvider.initialize wan't called yet, did not finish or the beanManager was null.

The CdiServerComponentProvider is an interesting class too - no synchronization. And the ComponentProvider.initialize has suspicious javadoc saying

Initializes the component provider with a reference to a injection manager instance, which will get used in the application to manage individual components. Providers should keep a reference to the injection manager for later use. This method will be invoked prior to any bind method calls. The injection manager parameter will not be fully initialized at the time of invocation and should be used as a reference only.

But then the CdiComponentProvider.bindHk2ClassAnalyzer breaks this rule, because it uses it before the initialize method ended. Probably this is the real cause of the issue ...? Then the problem would be in Jersey, not in HK2. But again, I would simply run away from CDI 1.1 now.

@nhenneaux
Copy link
Author

Thanks for your follow-up!!

I am using Weld 4.0.2 with org.glassfish.jersey.ext.cdi::jersey-weld2-se which is under the hood depends on jersey-cdi1x. It is compatible with Weld 4.

I agree it probably comes with the order of initialization which is not deterministic. However it might also come with a concurrency issue with some value in cpu cache not matching the main memory.

I already tried to replace hk2 with pure cdi injection but it requires major rework in the way we use them now.

Here is all the dependencies

|  |  +- org.jboss.weld.se:weld-se-core:jar:4.0.2.Final:compile
|  |  |  +- org.jboss.weld.environment:weld-environment-common:jar:4.0.2.Final:compile
|  |  |  +- org.jboss.weld.probe:weld-probe-core:jar:4.0.2.Final:compile
|  |  |  \- org.jboss.classfilewriter:jboss-classfilewriter:jar:1.2.5.Final:compile
|  |  +- org.jboss.weld:weld-core-impl:jar:4.0.2.Final:compile
|  |  |  \- org.jboss.weld:weld-spi:jar:4.0.SP1:compile
|  |  +- org.jboss.weld:weld-api:jar:4.0.SP1:compile
|  |  \- jakarta.servlet:jakarta.servlet-api:jar:5.0.0:compile
|  \- jakarta.activation:jakarta.activation-api:jar:2.0.1:compile
|  +- jakarta.inject:jakarta.inject-api:jar:2.0.0:compile
|  +- jakarta.enterprise:jakarta.enterprise.cdi-api:jar:3.0.0:compile
|  |  \- jakarta.interceptor:jakarta.interceptor-api:jar:2.0.0:compile
|  +- jakarta.annotation:jakarta.annotation-api:jar:2.0.0:compile
|  \- org.slf4j:slf4j-api:jar:1.7.32:compile
|  |  +- jakarta.xml.bind:jakarta.xml.bind-api:jar:3.0.1:compile
|  |  +- com.sun.xml.bind:jaxb-impl:jar:3.0.2:compile
|  |  |  \- com.sun.xml.bind:jaxb-core:jar:3.0.2:compile
|  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.12.5:compile
|  |  \- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.12.5:compile
|  +- io.swagger:swagger-annotations:jar:1.6.2:compile
|  +- org.glassfish.jersey.media:jersey-media-json-jackson:jar:3.0.3:compile
|  |  +- org.glassfish.jersey.core:jersey-common:jar:3.0.3:compile
|  |  +- org.glassfish.jersey.ext:jersey-entity-filtering:jar:3.0.3:compile
|  |  +- com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.12.5:compile
|  |  \- javax.xml.bind:jaxb-api:jar:2.3.1:compile
|  \- jakarta.ws.rs:jakarta.ws.rs-api:jar:3.0.0:compile
|  |  \- org.ehcache:ehcache:jar:3.9.3:compile
|  +- org.glassfish.jersey.ext.cdi:jersey-weld2-se:jar:3.0.3:compile
|  |  +- org.glassfish.jersey.core:jersey-server:jar:3.0.3:compile
|  |  |  \- org.glassfish.jersey.core:jersey-client:jar:3.0.3:compile
|  |  \- org.glassfish.jersey.ext.cdi:jersey-cdi1x:jar:3.0.3:compile
|  +- org.glassfish.jersey.inject:jersey-hk2:jar:3.0.3:compile
|  |  +- org.glassfish.hk2:hk2-locator:jar:3.0.1:compile
|  |  |  +- org.glassfish.hk2.external:aopalliance-repackaged:jar:3.0.1:compile
|  |  |  +- org.glassfish.hk2:hk2-api:jar:3.0.1:compile
|  |  |  \- org.glassfish.hk2:hk2-utils:jar:3.0.1:compile
|  |  \- org.javassist:javassist:jar:3.25.0-GA:compile
|  +- org.eclipse.jetty:jetty-servlet:jar:11.0.7:compile
|  |  \- org.eclipse.jetty:jetty-security:jar:11.0.7:compile
|  +- org.glassfish.jersey.containers:jersey-container-jetty-servlet:jar:3.0.3:compile
|  |  +- org.glassfish.jersey.containers:jersey-container-servlet:jar:3.0.3:compile
|  |  +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:3.0.3:compile
|  |  \- org.glassfish.jersey.containers:jersey-container-jetty-http:jar:3.0.3:compile
|  +- org.glassfish.jersey.ext:jersey-proxy-client:jar:3.0.3:compile
|  +- org.glassfish.jersey.ext:jersey-bean-validation:jar:3.0.3:compile
|  +- org.hibernate.validator:hibernate-validator:jar:7.0.1.Final:compile
|  |  +- org.jboss.logging:jboss-logging:jar:3.4.1.Final:compile
|  |  \- com.fasterxml:classmate:jar:1.5.1:compile
|  +- org.glassfish.jersey.media:jersey-media-multipart:jar:3.0.3:compile
|  |  \- org.jvnet.mimepull:mimepull:jar:1.9.13:compile
|  +- org.eclipse.jetty.http2:http2-server:jar:11.0.7:compile
|  |  +- org.eclipse.jetty.http2:http2-common:jar:11.0.7:compile
|  |  |  \- org.eclipse.jetty.http2:http2-hpack:jar:11.0.7:compile
|  |  \- org.eclipse.jetty:jetty-server:jar:11.0.7:compile
|  |     \- org.eclipse.jetty:jetty-http:jar:11.0.7:compile
|  +- org.eclipse.jetty:jetty-alpn-java-server:jar:11.0.7:compile
|  |  +- org.eclipse.jetty:jetty-io:jar:11.0.7:compile
|  |  |  \- org.eclipse.jetty:jetty-util:jar:11.0.7:compile
|  |  \- org.eclipse.jetty:jetty-alpn-server:jar:11.0.7:compile
|  +- com.github.nhenneaux.jersey.connector.httpclient:jersey-httpclient-connector:jar:1.0.10:compile
|  \- com.fasterxml.jackson.core:jackson-annotations:jar:2.12.5:compile
|  \- com.github.stefanbirkner:system-lambda:jar:1.1.0:compile
+- org.mockito:mockito-core:jar:3.12.4:compile
|  +- net.bytebuddy:byte-buddy:jar:1.11.13:compile
|  +- net.bytebuddy:byte-buddy-agent:jar:1.11.13:compile
|  \- org.objenesis:objenesis:jar:3.2:compile
+- org.slf4j:slf4j-simple:jar:1.7.32:test
+- jakarta.validation:jakarta.validation-api:jar:3.0.0:compile
+- com.fasterxml.jackson.core:jackson-databind:jar:2.12.5:compile
|  \- com.fasterxml.jackson.core:jackson-core:jar:2.12.5:compile
|  +- org.reflections:reflections:jar:0.9.11:test
|  |  \- com.google.guava:guava:jar:30.1-jre:test
|  |     +- com.google.guava:failureaccess:jar:1.0.1:test
|  |     +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:test
|  |     +- com.google.code.findbugs:jsr305:jar:3.0.2:test
|  |     +- org.checkerframework:checker-qual:jar:3.5.0:test
|  |     +- com.google.errorprone:error_prone_annotations:jar:2.3.4:test
|  |     \- com.google.j2objc:j2objc-annotations:jar:1.3:test
|  \- org.apache.commons:commons-lang3:jar:3.11:test
+- jakarta.el:jakarta.el-api:jar:4.0.0:compile
+- org.glassfish:jakarta.el:jar:4.0.2:compile
+- org.junit.jupiter:junit-jupiter-engine:jar:5.8.1:test
|  +- org.junit.platform:junit-platform-engine:jar:1.8.1:test
|  |  +- org.opentest4j:opentest4j:jar:1.2.0:test
|  |  \- org.junit.platform:junit-platform-commons:jar:1.8.1:test
|  +- org.junit.jupiter:junit-jupiter-api:jar:5.8.1:test
|  \- org.apiguardian:apiguardian-api:jar:1.1.2:test
+- org.awaitility:awaitility:jar:3.1.6:test
|  +- org.hamcrest:hamcrest-library:jar:1.3:test
|  \- org.hamcrest:hamcrest-core:jar:1.3:test
+- org.hamcrest:hamcrest:jar:2.2:test
\- org.mockito:mockito-junit-jupiter:jar:3.12.4:test



@dmatej
Copy link
Contributor

dmatej commented Oct 21, 2021

Hmm, the referred example uses jersey-cdi2-se instead of jersey-weld2-se ... btw ... HK2 is nearly a concurrent of Weld, so that really can cause some issues ... it should use just one of them.

Also I see that in service files of cdi1x project is the CdiComponentProvider twice, once as an extension and once as a ComponentProvider impl - that can probably cause redundant initialization too. Or two standalone instances of the same class used as two standalone services ...

If cpu caches would work like this, nothing would work ever ... prove it (the map is still protected by synchronized methods).

Btw I was digging in my old examples and I found this:

        <dependency>
            <groupId>org.glassfish.jersey.core</groupId>
            <artifactId>jersey-server</artifactId>
            <scope>provided</scope>
        </dependency>

        <!-- If we skip this dependency, jersey-client throws java.lang.IllegalStateException: InjectionManagerFactory not found. -->
        <!-- With that we must have META-INF/beans.xml in src/test/resources -->
        <dependency>
            <groupId>org.glassfish.jersey.inject</groupId>
            <artifactId>jersey-cdi2-se</artifactId>
        </dependency>
        <dependency>
            <groupId>org.jboss.weld.se</groupId>
            <artifactId>weld-se-core</artifactId>
        </dependency>

@nhenneaux
Copy link
Author

I'm also getting this error while the applications runs in production. It occurred the 11/01/2022 for a service started the 20/12/2021.
I include a truncated stack of the error.

jakarta.ws.rs.ProcessingException: A MultiException has 1 exceptions.  They are:
1. java.lang.IllegalStateException: Could not find an implementation of ClassAnalyzer with name CdiInjecteeSkippingClassAnalyzer

	at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:309) ~[app.jar:x.x.x]
	at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$2(JerseyInvocation.java:687) ~[app.jar:x.x.x]
	at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697) ~[app.jar:x.x.x]
	at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691) ~[app.jar:x.x.x]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[app.jar:x.x.x]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[app.jar:x.x.x]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:205) ~[app.jar:x.x.x]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) ~[app.jar:x.x.x]
	at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691) ~[app.jar:x.x.x]
	at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:686) ~[app.jar:x.x.x]
	at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:461) ~[app.jar:x.x.x]
	at org.glassfish.jersey.client.proxy.WebResourceFactory.invoke(WebResourceFactory.java:329) ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at jdk.proxy2.$Proxy134.partitionKeyCaptureBatchAcquireLock(Unknown Source) ~[?:?]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
	at java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:720) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) ~[?:?]
	at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616) ~[?:?]
	at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:622) ~[?:?]
	at java.util.stream.ReferencePipeline.toList(ReferencePipeline.java:627) ~[?:?]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class [app.jar:x.x.x]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.glassfish.hk2.api.MultiException: A MultiException has 1 exceptions.  They are:
1. java.lang.IllegalStateException: Could not find an implementation of ClassAnalyzer with name CdiInjecteeSkippingClassAnalyzer

	at org.jvnet.hk2.internal.Collector.throwIfErrors(Collector.java:65) ~[app.jar:x.x.x]
	at org.jvnet.hk2.internal.Utilities.justInject(Utilities.java:970) ~[app.jar:x.x.x]
	at org.jvnet.hk2.internal.ServiceLocatorImpl.inject(ServiceLocatorImpl.java:1014) ~[app.jar:x.x.x]
	at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.inject(AbstractHk2InjectionManager.java:212) ~[app.jar:x.x.x]
	at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.inject(ImmediateHk2InjectionManager.java:30) ~[app.jar:x.x.x]
	at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:776) ~[app.jar:x.x.x]
	at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:161) ~[app.jar:x.x.x]
	at org.jboss.weld.contexts.unbound.DependentContextImpl.get(DependentContextImpl.java:64) ~[app.jar:x.x.x]
	at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) ~[app.jar:x.x.x]
	at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) ~[app.jar:x.x.x]
	at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:694) ~[app.jar:x.x.x]
	at org.jboss.weld.bean.builtin.InstanceImpl.getBeanInstance(InstanceImpl.java:252) ~[app.jar:x.x.x]
	at org.jboss.weld.bean.builtin.InstanceImpl.get(InstanceImpl.java:114) ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at jdk.proxy2.$Proxy177.signature(Unknown Source) ~[?:?]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at org.glassfish.jersey.client.ClientFilteringStages$RequestFilteringStage.apply(ClientFilteringStages.java:144) ~[app.jar:x.x.x]
	at org.glassfish.jersey.client.ClientFilteringStages$RequestFilteringStage.apply(ClientFilteringStages.java:132) ~[app.jar:x.x.x]
	at org.glassfish.jersey.process.internal.Stages.process(Stages.java:147) ~[app.jar:x.x.x]
	at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[app.jar:x.x.x]
	... 36 more
Caused by: java.lang.IllegalStateException: Could not find an implementation of ClassAnalyzer with name CdiInjecteeSkippingClassAnalyzer
	at org.jvnet.hk2.internal.ServiceLocatorImpl.getAnalyzer(ServiceLocatorImpl.java:2468) ~[app.jar:x.x.x]
	at org.jvnet.hk2.internal.Utilities.getClassAnalyzer(Utilities.java:143) ~[app.jar:x.x.x]
	at org.jvnet.hk2.internal.Utilities.justInject(Utilities.java:969) ~[app.jar:x.x.x]
	at org.jvnet.hk2.internal.ServiceLocatorImpl.inject(ServiceLocatorImpl.java:1014) ~[app.jar:x.x.x]
	at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.inject(AbstractHk2InjectionManager.java:212) ~[app.jar:x.x.x]
	at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.inject(ImmediateHk2InjectionManager.java:30) ~[app.jar:x.x.x]
	at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:776) ~[app.jar:x.x.x]
	at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:161) ~[app.jar:x.x.x]
	at org.jboss.weld.contexts.unbound.DependentContextImpl.get(DependentContextImpl.java:64) ~[app.jar:x.x.x]
	at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) ~[app.jar:x.x.x]
	at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) ~[app.jar:x.x.x]
	at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:694) ~[app.jar:x.x.x]
	at org.jboss.weld.bean.builtin.InstanceImpl.getBeanInstance(InstanceImpl.java:252) ~[app.jar:x.x.x]
	at org.jboss.weld.bean.builtin.InstanceImpl.get(InstanceImpl.java:114) ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at jdk.proxy2.$Proxy177.signature(Unknown Source) ~[?:?]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at internal.class ~[app.jar:x.x.x]
	at org.glassfish.jersey.client.ClientFilteringStages$RequestFilteringStage.apply(ClientFilteringStages.java:144) ~[app.jar:x.x.x]
	at org.glassfish.jersey.client.ClientFilteringStages$RequestFilteringStage.apply(ClientFilteringStages.java:132) ~[app.jar:x.x.x]
	at org.glassfish.jersey.process.internal.Stages.process(Stages.java:147) ~[app.jar:x.x.x]
	at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) ~[app.jar:x.x.x]
	... 36 more

@nhenneaux
Copy link
Author

@dmatej I think I found where the problem lie. It occurs when there is a concurrent initialization from Jersey server but also from each new client. Each new client triggers the initialization of injection which makes things worse.

@nhenneaux
Copy link
Author

Hi @dmatej
Were you able to dig further on this issue? We are still impacted by it from time to time. Especially in CI but also for some use case in live code.
I'm not sure the associated MR would help but I think it provides an extra guarantee about the consistency of the store in case of race condition.

@matthjes
Copy link

Hi,
We're getting this error quite suddenly now, first occurrence was May 11th, before this we never saw this error. It's our production environment, we've had no new deployments since mid-April.

Any suggestions how to deal with this?

@dmatej
Copy link
Contributor

dmatej commented May 25, 2023

Hi, We're getting this error quite suddenly now, first occurrence was May 11th, before this we never saw this error. It's our production environment, we've had no new deployments since mid-April.

Any suggestions how to deal with this?

With the same version of Jetty, HK2 and Jersey? Could you create a reproducer?

@matthjes
Copy link

Hi @dmatej,

I'm afraid thats not possible: the error does not occur for every request, but maybe 4 or 5 times a day, with over a hundred requests per day.

We are using Payara Server Enterprise 5.49.0 #badassfish (build 839).

@nhenneaux
Copy link
Author

The issue is still happening on a regular basis in CI and production with Jersey 3.1.1 & Jetty 11.0.15.

It happens more when the Jersey client is instantiated concurrently.
What about merging MR #560 since it is quite harmless and might fix the issue?

@avpinchuk
Copy link
Contributor

@nhenneaux, issue happening in this project too?

@nhenneaux
Copy link
Author

I don't remember so. It happens when doing concurrent client instantiations and I don't on this project. https://github.com/nhenneaux/jersey-httpclient-connector/blob/master/src/test/java/com/github/nhenneaux/jersey/connector/httpclient/JettyServerTest.java#L412-L413

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants