Deploying two apps with APM 1.17.0 cause performance issues

[paulstearns91]

No description provided.

[paulstearns91]

Hi Michael

When we deploy 2 Apps on same runtime with APM 1.17.0. we starts to gets the performance issue and one of the app won't record the traces correctly.I saw the following error in the logs

Error from mule_ee log file
ERROR co.elastic.apm.agent.plugin.api.HeaderInjectorBridge - Failed to add trace context headers
java.lang.ClassCastException: Cannot cast co.elastic.apm.mule.utils.SpanUtils$$Lambda$92/1785346483 to co.elastic.apm.api.HeaderInjector
at java.lang.invoke.MethodHandleImpl.newClassCastException(MethodHandleImpl.java:361)
at java.lang.invoke.MethodHandleImpl.castReference(MethodHandleImpl.java:356)
at co.elastic.apm.agent.plugin.api.HeaderInjectorBridge.setHeader(HeaderInjectorBridge.java:61)
at co.elastic.apm.agent.plugin.api.HeaderInjectorBridge.setHeader(HeaderInjectorBridge.java:35)
at co.elastic.apm.agent.impl.transaction.TraceContext.setOutgoingTraceContextHeaders(TraceContext.java:522)
at co.elastic.apm.api.AbstractSpanImpl.doInjectTraceHeaders(AbstractSpanImpl.java:153)
at co.elastic.apm.api.AbstractSpanImpl.injectTraceHeaders(AbstractSpanImpl.java:148)
at co.elastic.apm.mule.utils.SpanUtils.startSpan(SpanUtils.java:39)
at co.elastic.apm.mule.listeners.TraceMessageProcessorNotificationListener.onNotification(TraceMessageProcessorNotificationListener.java:31)
at co.elastic.apm.mule.listeners.TraceMessageProcessorNotificationListener.onNotification(TraceMessageProcessorNotificationListener.java:18)
at org.mule.context.notification.Sender.dispatch(Sender.java:36)
at org.mule.context.notification.Policy.dispatchToSenders(Policy.java:171)
at org.mule.context.notification.Policy.doDispatch(Policy.java:136)
at org.mule.context.notification.Policy.dispatch(Policy.java:117)
at org.mule.context.notification.ServerNotificationManager.notifyListeners(ServerNotificationManager.java:255)
at org.mule.context.notification.ServerNotificationManager.fireNotification(ServerNotificationManager.java:202)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.fireNotification(MessageProcessorNotificationExecutionInterceptor.java:144)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:55)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:88)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.processor.chain.DefaultMessageProcessorChain.doProcess(DefaultMessageProcessorChain.java:81)
at org.mule.processor.chain.AbstractMessageProcessorChain.process(AbstractMessageProcessorChain.java:74)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:98)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.processor.AbstractInterceptingMessageProcessorBase.processNext(AbstractInterceptingMessageProcessorBase.java:99)
at org.mule.processor.AsyncInterceptingMessageProcessor.process(AsyncInterceptingMessageProcessor.java:119)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:112)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.chain.SimpleMessageProcessorChain.doProcess(SimpleMessageProcessorChain.java:43)
at org.mule.processor.chain.AbstractMessageProcessorChain.process(AbstractMessageProcessorChain.java:74)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:112)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.BlockingProcessorExecutor.executeNext(BlockingProcessorExecutor.java:88)
at org.mule.processor.BlockingProcessorExecutor.execute(BlockingProcessorExecutor.java:59)
at org.mule.processor.chain.DefaultMessageProcessorChain.doProcess(DefaultMessageProcessorChain.java:81)
at org.mule.processor.chain.AbstractMessageProcessorChain.process(AbstractMessageProcessorChain.java:74)
at org.mule.processor.chain.InterceptingChainLifecycleWrapper.doProcess(InterceptingChainLifecycleWrapper.java:50)
at org.mule.processor.chain.AbstractMessageProcessorChain.process(AbstractMessageProcessorChain.java:74)
at org.mule.processor.chain.InterceptingChainLifecycleWrapper.access$001(InterceptingChainLifecycleWrapper.java:22)
at org.mule.processor.chain.InterceptingChainLifecycleWrapper$1.process(InterceptingChainLifecycleWrapper.java:66)
at org.mule.execution.ExceptionToMessagingExceptionExecutionInterceptor.execute(ExceptionToMessagingExceptionExecutionInterceptor.java:27)
at org.mule.execution.MessageProcessorNotificationExecutionInterceptor.execute(MessageProcessorNotificationExecutionInterceptor.java:112)
at org.mule.execution.MessageProcessorExecutionTemplate.execute(MessageProcessorExecutionTemplate.java:44)
at org.mule.processor.chain.InterceptingChainLifecycleWrapper.process(InterceptingChainLifecycleWrapper.java:61)

[michaelhyatt]

Hi Paul,

I did a new release with a more defensive way of handling the header parsing. It used to be a lambda but I turned it into a more explicit anonymous class. I also added try/catch on the property setting method. I am not exactly sure why lambda failed sometimes, but this hopefully will produce a bit more logging if it does.

Also, if you are running two projects in the same domain, have a look at the example in the repo that shows a domain project. You can specify the tracer at the domain level only once and it should take care of all the projects deployed into the Mule 3 server.

Can you please check it out?