NoClassDefFoundError: OkHttpClientBuilderAdvisor$ running with DataDog instrumentation

[aakture]

We're running a Java akka app using akka v 2.6.10, and akkaHttp v 10.2.1 and recently upgrading to kamon 2.1.9. I can startup the app locally fine, but when I deploy it to a kubernetes cluster, I get the error

    com.zaxxer.hikari.HikariDataSource       : PrimaryPool - Starting...
    [dd-trace-writer] ERROR 2021-01-05 21:21:39  Logger : An error occurred while trying to apply an advisor: java.lang.NoClassDefFoundError: kamon/okhttp3/instrumentation/OkHttpClientBuilderAdvisor$
        at okhttp3.OkHttpClient.<init>(OkHttpClient.java:234)
        at okhttp3.OkHttpClient$Builder.build(OkHttpClient.java:1040)
        at datadog.trace.agent.common.writer.ddagent.DDAgentApi.buildHttpClient(DDAgentApi.java:191)
        at datadog.trace.agent.common.writer.ddagent.DDAgentApi.endpointAvailable(DDAgentApi.java:164)
        at 

I suppose it could be a problem with datadog getting in the way, but wanted to see if anyone had any helpful ideas to trouble shoot.

[dpsoft]

Hi @aakture you are using kanela agent + dd agent?

[dpsoft]

in which order? kanela - > dd or dd - > kanela?

[aakture]

We have dd as a vm opt: " -javaagent:dd-java-agent.jar " and have the kamon bundle dependency in the build. So not 100% sure how it's ordered, but I do see in the trace that it's kanela->dd

jdk.internal.loader.ClassLoaders$AppClassLoader@368239c8: java.lang.IllegalStateException: 
  Cannot resolve type description for datadog.trace.bootstrap.instrumentation.context.FieldBackedProvider$ContextAccessor$scala$concurrent$impl$Promise$Transformation$datadog$trace$bootstrap$instrumentation$java$concurrent$State 
    at kanela.agent.libs.net.bytebuddy.pool.TypePool$Resolution$Illegal.resolve(TypePool.java:157)    
      at kanela.agent.libs.net.bytebuddy.pool.TypePool$Default$LazyTypeDescription$TokenizedGenericType.toErasure(TypePool.java:6475) 
        at kanela.agent.libs.net.bytebuddy.pool.TypePool$Default$LazyTypeDescription$LazyTypeList.get(TypePool.java:6283) 
          at kanela.agent.libs.net.bytebuddy.pool.TypePool$Default$LazyTypeDescription$LazyTypeList.get(TypePool.java:6256) 
            at java.base/java.util.AbstractList$Itr.next(AbstractList.java:371)

[dpsoft]

You can try pass the kanela agent as javaagent:kanela.jar?

[aakture]

I'll give it a try, reply with results. thanks!

[stewart-shawn]

@aakture any luck resolving? We also pass the dd-agent as a VM opt while the kanela agent is configured via our build config file. I tried setting the kanela agent via javaagent as @dpsoft suggested, but I am still seeing the following

An error occurred while trying to apply an advisor: java.lang.NoClassDefFoundError: kamon/okhttp3/instrumentation/OkHttpClientBuilderAdvisor$ at okhttp3.OkHttpClient.<init>(OkHttpClient.java:234) at okhttp3.OkHttpClient$Builder.build(OkHttpClient.java:1040) at datadog.communication.http.OkHttpUtils.buildHttpClient(OkHttpUtils.java:80) at datadog.communication.http.OkHttpUtils.buildHttpClient(OkHttpUtils.java:51) at datadog.communication.ddagent.SharedCommunicationObjects.createRemaining(SharedCommunicationObjects.java:28) at datadog.trace.agent.core.CoreTracer.<init>(CoreTracer.java:491) at datadog.trace.agent.core.CoreTracer.<init>(CoreTracer.java:86) at datadog.trace.agent.core.CoreTracer$CoreTracerBuilder.build(CoreTracer.java:380) at datadog.trace.agent.tooling.TracerInstaller.installGlobalTracer(TracerInstaller.java:19) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at datadog.trace.bootstrap.Agent.installDatadogTracer(Agent.java:443) at datadog.trace.bootstrap.Agent.access$400(Agent.java:45) at datadog.trace.bootstrap.Agent$InstallDatadogTracerCallback.execute(Agent.java:369) at datadog.trace.bootstrap.Agent.start(Agent.java:237) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at datadog.trace.bootstrap.AgentBootstrap.agentmain(AgentBootstrap.java:60) at datadog.trace.bootstrap.AgentBootstrap.premain(AgentBootstrap.java:47) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513) at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)