Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] ConfigurationException on creating OpenTelemetry SDK object when using AzureMonitorExporterBuilder #41859

Closed
3 tasks done
greatvovan opened this issue Sep 14, 2024 · 12 comments
Assignees
Labels
customer-reported Issues that are reported by GitHub users external to the Azure organization. Monitor - Exporter Monitor OpenTelemetry Exporter needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team OpenTelemetry OpenTelemetry instrumentation question The issue doesn't require a change to the product in order to be resolved. Most issues start as that

Comments

@greatvovan
Copy link

Describe the bug
When executing telemetry configuration code more than once in a Scala Databricks notebook / cluster, an "Unexpected configuration error" is thrown.

Exception or Stack Trace

ConfigurationException: Unexpected configuration error
ConfigurationException: Unexpected configuration error
Caused by: RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@3df452e6 rejected from java.util.concurrent.ScheduledThreadPoolExecutor@19a3c7c7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
	at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.build(AutoConfiguredOpenTelemetrySdkBuilder.java:525)
	at $linedfd7570c873044a4acb88457562b72e943.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-4089470955889137:7)
	at $linedfd7570c873044a4acb88457562b72e943.$read$$iw$$iw$$iw$$iw$$iw.<init>(command-4089470955889137:60)
	at $linedfd7570c873044a4acb88457562b72e943.$read$$iw$$iw$$iw$$iw.<init>(command-4089470955889137:62)
	at $linedfd7570c873044a4acb88457562b72e943.$read$$iw$$iw$$iw.<init>(command-4089470955889137:64)
	at $linedfd7570c873044a4acb88457562b72e943.$read$$iw$$iw.<init>(command-4089470955889137:66)
	at $linedfd7570c873044a4acb88457562b72e943.$read$$iw.<init>(command-4089470955889137:68)
	at $linedfd7570c873044a4acb88457562b72e943.$read.<init>(command-4089470955889137:70)
	at $linedfd7570c873044a4acb88457562b72e943.$read$.<init>(command-4089470955889137:74)
	at $linedfd7570c873044a4acb88457562b72e943.$read$.<clinit>(command-4089470955889137)
	at $linedfd7570c873044a4acb88457562b72e943.$eval$.$print$lzycompute(<notebook>:7)
	at $linedfd7570c873044a4acb88457562b72e943.$eval$.$print(<notebook>:6)
	at $linedfd7570c873044a4acb88457562b72e943.$eval.$print(<notebook>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:747)
	at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1020)
	at scala.tools.nsc.interpreter.IMain.$anonfun$interpret$1(IMain.scala:568)
	at scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:36)
	at scala.reflect.internal.util.ScalaClassLoader.asContext$(ScalaClassLoader.scala:116)
	at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:41)
	at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:567)
	at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:594)
	at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:564)
	at com.databricks.backend.daemon.driver.DriverILoop.execute(DriverILoop.scala:201)
	at com.databricks.backend.daemon.driver.ScalaDriverLocal.$anonfun$repl$3(ScalaDriverLocal.scala:294)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at com.databricks.backend.daemon.driver.DriverLocal$TrapExitInternal$.threadSafeTrapExit(DriverLocal.scala:1578)
	at com.databricks.backend.daemon.driver.DriverLocal$TrapExitInternal$.trapExit(DriverLocal.scala:1536)
	at com.databricks.backend.daemon.driver.DriverLocal$TrapExit$.apply(DriverLocal.scala:1460)
	at com.databricks.backend.daemon.driver.ScalaDriverLocal.executeCommand$1(ScalaDriverLocal.scala:294)
	at com.databricks.backend.daemon.driver.ScalaDriverLocal.$anonfun$repl$2(ScalaDriverLocal.scala:264)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
	at scala.Console$.withErr(Console.scala:196)
	at com.databricks.backend.daemon.driver.ScalaDriverLocal.$anonfun$repl$1(ScalaDriverLocal.scala:261)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
	at scala.Console$.withOut(Console.scala:167)
	at com.databricks.backend.daemon.driver.ScalaDriverLocal.repl(ScalaDriverLocal.scala:261)
	at com.databricks.backend.daemon.driver.DriverLocal.$anonfun$execute$30(DriverLocal.scala:1141)
	at com.databricks.unity.UCSEphemeralState$Handle.runWith(UCSEphemeralState.scala:51)
	at com.databricks.unity.HandleImpl.runWith(UCSHandle.scala:104)
	at com.databricks.backend.daemon.driver.DriverLocal.$anonfun$execute$25(DriverLocal.scala:1132)
	at com.databricks.logging.AttributionContextTracing.$anonfun$withAttributionContext$1(AttributionContextTracing.scala:48)
	at com.databricks.logging.AttributionContext$.$anonfun$withValue$1(AttributionContext.scala:276)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
	at com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:272)
	at com.databricks.logging.AttributionContextTracing.withAttributionContext(AttributionContextTracing.scala:46)
	at com.databricks.logging.AttributionContextTracing.withAttributionContext$(AttributionContextTracing.scala:43)
	at com.databricks.backend.daemon.driver.DriverLocal.withAttributionContext(DriverLocal.scala:88)
	at com.databricks.logging.AttributionContextTracing.withAttributionTags(AttributionContextTracing.scala:95)
	at com.databricks.logging.AttributionContextTracing.withAttributionTags$(AttributionContextTracing.scala:76)
	at com.databricks.backend.daemon.driver.DriverLocal.withAttributionTags(DriverLocal.scala:88)
	at com.databricks.backend.daemon.driver.DriverLocal.$anonfun$execute$1(DriverLocal.scala:1076)
	at com.databricks.backend.daemon.driver.DriverLocal$.$anonfun$maybeSynchronizeExecution$4(DriverLocal.scala:1488)
	at com.databricks.backend.daemon.driver.DriverLocal.execute(DriverLocal.scala:767)
	at com.databricks.backend.daemon.driver.DriverWrapper.$anonfun$tryExecutingCommand$2(DriverWrapper.scala:931)
	at scala.util.Try$.apply(Try.scala:213)
	at com.databricks.backend.daemon.driver.DriverWrapper.$anonfun$tryExecutingCommand$1(DriverWrapper.scala:920)
	at com.databricks.backend.daemon.driver.DriverWrapper.$anonfun$tryExecutingCommand$3(DriverWrapper.scala:966)
	at com.databricks.logging.UsageLogging.executeThunkAndCaptureResultTags$1(UsageLogging.scala:631)
	at com.databricks.logging.UsageLogging.$anonfun$recordOperationWithResultTags$4(UsageLogging.scala:651)
	at com.databricks.logging.AttributionContextTracing.$anonfun$withAttributionContext$1(AttributionContextTracing.scala:48)
	at com.databricks.logging.AttributionContext$.$anonfun$withValue$1(AttributionContext.scala:276)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
	at com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:272)
	at com.databricks.logging.AttributionContextTracing.withAttributionContext(AttributionContextTracing.scala:46)
	at com.databricks.logging.AttributionContextTracing.withAttributionContext$(AttributionContextTracing.scala:43)
	at com.databricks.backend.daemon.driver.DriverWrapper.withAttributionContext(DriverWrapper.scala:73)
	at com.databricks.logging.AttributionContextTracing.withAttributionTags(AttributionContextTracing.scala:95)
	at com.databricks.logging.AttributionContextTracing.withAttributionTags$(AttributionContextTracing.scala:76)
	at com.databricks.backend.daemon.driver.DriverWrapper.withAttributionTags(DriverWrapper.scala:73)
	at com.databricks.logging.UsageLogging.recordOperationWithResultTags(UsageLogging.scala:626)
	at com.databricks.logging.UsageLogging.recordOperationWithResultTags$(UsageLogging.scala:536)
	at com.databricks.backend.daemon.driver.DriverWrapper.recordOperationWithResultTags(DriverWrapper.scala:73)
	at com.databricks.backend.daemon.driver.DriverWrapper.tryExecutingCommand(DriverWrapper.scala:966)
	at com.databricks.backend.daemon.driver.DriverWrapper.executeCommandAndGetError(DriverWrapper.scala:717)
	at com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(DriverWrapper.scala:784)
	at com.databricks.backend.daemon.driver.DriverWrapper.$anonfun$runInnerLoop$1(DriverWrapper.scala:590)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at com.databricks.logging.AttributionContextTracing.$anonfun$withAttributionContext$1(AttributionContextTracing.scala:48)
	at com.databricks.logging.AttributionContext$.$anonfun$withValue$1(AttributionContext.scala:276)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
	at com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:272)
	at com.databricks.logging.AttributionContextTracing.withAttributionContext(AttributionContextTracing.scala:46)
	at com.databricks.logging.AttributionContextTracing.withAttributionContext$(AttributionContextTracing.scala:43)
	at com.databricks.backend.daemon.driver.DriverWrapper.withAttributionContext(DriverWrapper.scala:73)
	at com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(DriverWrapper.scala:590)
	at com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWrapper.scala:512)
	at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:306)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@3df452e6 rejected from java.util.concurrent.ScheduledThreadPoolExecutor@19a3c7c7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
	at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
	at java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:597)
	at java.util.concurrent.Executors$DelegatedScheduledExecutorService.scheduleWithFixedDelay(Executors.java:738)
	at com.azure.monitor.opentelemetry.exporter.implementation.statsbeat.AzureMetadataService.scheduleWithFixedDelay(AzureMetadataService.java:54)
	at com.azure.monitor.opentelemetry.exporter.implementation.statsbeat.StatsbeatModule.start(StatsbeatModule.java:94)
	at com.azure.monitor.opentelemetry.exporter.AzureMonitorExporterBuilder.startStatsbeatModule(AzureMonitorExporterBuilder.java:463)
	at com.azure.monitor.opentelemetry.exporter.AzureMonitorExporterBuilder.internalBuildAndFreeze(AzureMonitorExporterBuilder.java:348)
	at com.azure.monitor.opentelemetry.exporter.AzureMonitorExporterBuilder.lambda$install$2(AzureMonitorExporterBuilder.java:310)
	at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.lambda$mergeCustomizer$13(AutoConfiguredOpenTelemetrySdkBuilder.java:648)
	at io.opentelemetry.sdk.autoconfigure.MetricExporterConfiguration.configureReader(MetricExporterConfiguration.java:78)
	at io.opentelemetry.sdk.autoconfigure.MeterProviderConfiguration.lambda$configureMetricReaders$2(MeterProviderConfiguration.java:92)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1580)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
	at io.opentelemetry.sdk.autoconfigure.MeterProviderConfiguration.configureMetricReaders(MeterProviderConfiguration.java:99)
	at io.opentelemetry.sdk.autoconfigure.MeterProviderConfiguration.configureMeterProvider(MeterProviderConfiguration.java:61)
	at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.build(AutoConfiguredOpenTelemetrySdkBuilder.java:452)
	at $linedfd7570c873044a4acb88457562b72e943.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command-4089470955889137:7)
	at $linedfd7570c873044a4acb88457562b72e943.$read$$iw$$iw$$iw$$iw$$iw.<init>(command-4089470955889137:60)
	at $linedfd7570c873044a4acb88457562b72e943.$read$$iw$$iw$$iw$$iw.<init>(command-4089470955889137:62)
	at $linedfd7570c873044a4acb88457562b72e943.$read$$iw$$iw$$iw.<init>(command-4089470955889137:64)
	at $linedfd7570c873044a4acb88457562b72e943.$read$$iw$$iw.<init>(command-4089470955889137:66)
	at $linedfd7570c873044a4acb88457562b72e943.$read$$iw.<init>(command-4089470955889137:68)
	at $linedfd7570c873044a4acb88457562b72e943.$read.<init>(command-4089470955889137:70)
	at $linedfd7570c873044a4acb88457562b72e943.$read$.<init>(command-4089470955889137:74)
	at $linedfd7570c873044a4acb88457562b72e943.$read$.<clinit>(command-4089470955889137)
	at $linedfd7570c873044a4acb88457562b72e943.$eval$.$print$lzycompute(<notebook>:7)
	at $linedfd7570c873044a4acb88457562b72e943.$eval$.$print(<notebook>:6)
	at $linedfd7570c873044a4acb88457562b72e943.$eval.$print(<notebook>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:747)
	at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1020)
	at scala.tools.nsc.interpreter.IMain.$anonfun$interpret$1(IMain.scala:568)
	at scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:36)
	at scala.reflect.internal.util.ScalaClassLoader.asContext$(ScalaClassLoader.scala:116)
	at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:41)
	at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:567)
	at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:594)
	at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:564)
	at com.databricks.backend.daemon.driver.DriverILoop.execute(DriverILoop.scala:201)
	at com.databricks.backend.daemon.driver.ScalaDriverLocal.$anonfun$repl$3(ScalaDriverLocal.scala:294)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at com.databricks.backend.daemon.driver.DriverLocal$TrapExitInternal$.threadSafeTrapExit(DriverLocal.scala:1578)
	at com.databricks.backend.daemon.driver.DriverLocal$TrapExitInternal$.trapExit(DriverLocal.scala:1536)
	at com.databricks.backend.daemon.driver.DriverLocal$TrapExit$.apply(DriverLocal.scala:1460)
	at com.databricks.backend.daemon.driver.ScalaDriverLocal.executeCommand$1(ScalaDriverLocal.scala:294)
	at com.databricks.backend.daemon.driver.ScalaDriverLocal.$anonfun$repl$2(ScalaDriverLocal.scala:264)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
	at scala.Console$.withErr(Console.scala:196)
	at com.databricks.backend.daemon.driver.ScalaDriverLocal.$anonfun$repl$1(ScalaDriverLocal.scala:261)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
	at scala.Console$.withOut(Console.scala:167)
	at com.databricks.backend.daemon.driver.ScalaDriverLocal.repl(ScalaDriverLocal.scala:261)
	at com.databricks.backend.daemon.driver.DriverLocal.$anonfun$execute$30(DriverLocal.scala:1141)
	at com.databricks.unity.UCSEphemeralState$Handle.runWith(UCSEphemeralState.scala:51)
	at com.databricks.unity.HandleImpl.runWith(UCSHandle.scala:104)
	at com.databricks.backend.daemon.driver.DriverLocal.$anonfun$execute$25(DriverLocal.scala:1132)
	at com.databricks.logging.AttributionContextTracing.$anonfun$withAttributionContext$1(AttributionContextTracing.scala:48)
	at com.databricks.logging.AttributionContext$.$anonfun$withValue$1(AttributionContext.scala:276)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
	at com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:272)
	at com.databricks.logging.AttributionContextTracing.withAttributionContext(AttributionContextTracing.scala:46)
	at com.databricks.logging.AttributionContextTracing.withAttributionContext$(AttributionContextTracing.scala:43)
	at com.databricks.backend.daemon.driver.DriverLocal.withAttributionContext(DriverLocal.scala:88)
	at com.databricks.logging.AttributionContextTracing.withAttributionTags(AttributionContextTracing.scala:95)
	at com.databricks.logging.AttributionContextTracing.withAttributionTags$(AttributionContextTracing.scala:76)
	at com.databricks.backend.daemon.driver.DriverLocal.withAttributionTags(DriverLocal.scala:88)
	at com.databricks.backend.daemon.driver.DriverLocal.$anonfun$execute$1(DriverLocal.scala:1076)
	at com.databricks.backend.daemon.driver.DriverLocal$.$anonfun$maybeSynchronizeExecution$4(DriverLocal.scala:1488)
	at com.databricks.backend.daemon.driver.DriverLocal.execute(DriverLocal.scala:767)
	at com.databricks.backend.daemon.driver.DriverWrapper.$anonfun$tryExecutingCommand$2(DriverWrapper.scala:931)
	at scala.util.Try$.apply(Try.scala:213)
	at com.databricks.backend.daemon.driver.DriverWrapper.$anonfun$tryExecutingCommand$1(DriverWrapper.scala:920)
	at com.databricks.backend.daemon.driver.DriverWrapper.$anonfun$tryExecutingCommand$3(DriverWrapper.scala:966)
	at com.databricks.logging.UsageLogging.executeThunkAndCaptureResultTags$1(UsageLogging.scala:631)
	at com.databricks.logging.UsageLogging.$anonfun$recordOperationWithResultTags$4(UsageLogging.scala:651)
	at com.databricks.logging.AttributionContextTracing.$anonfun$withAttributionContext$1(AttributionContextTracing.scala:48)
	at com.databricks.logging.AttributionContext$.$anonfun$withValue$1(AttributionContext.scala:276)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
	at com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:272)
	at com.databricks.logging.AttributionContextTracing.withAttributionContext(AttributionContextTracing.scala:46)
	at com.databricks.logging.AttributionContextTracing.withAttributionContext$(AttributionContextTracing.scala:43)
	at com.databricks.backend.daemon.driver.DriverWrapper.withAttributionContext(DriverWrapper.scala:73)
	at com.databricks.logging.AttributionContextTracing.withAttributionTags(AttributionContextTracing.scala:95)
	at com.databricks.logging.AttributionContextTracing.withAttributionTags$(AttributionContextTracing.scala:76)
	at com.databricks.backend.daemon.driver.DriverWrapper.withAttributionTags(DriverWrapper.scala:73)
	at com.databricks.logging.UsageLogging.recordOperationWithResultTags(UsageLogging.scala:626)
	at com.databricks.logging.UsageLogging.recordOperationWithResultTags$(UsageLogging.scala:536)
	at com.databricks.backend.daemon.driver.DriverWrapper.recordOperationWithResultTags(DriverWrapper.scala:73)
	at com.databricks.backend.daemon.driver.DriverWrapper.tryExecutingCommand(DriverWrapper.scala:966)
	at com.databricks.backend.daemon.driver.DriverWrapper.executeCommandAndGetError(DriverWrapper.scala:717)
	at com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(DriverWrapper.scala:784)
	at com.databricks.backend.daemon.driver.DriverWrapper.$anonfun$runInnerLoop$1(DriverWrapper.scala:590)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at com.databricks.logging.AttributionContextTracing.$anonfun$withAttributionContext$1(AttributionContextTracing.scala:48)
	at com.databricks.logging.AttributionContext$.$anonfun$withValue$1(AttributionContext.scala:276)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
	at com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:272)
	at com.databricks.logging.AttributionContextTracing.withAttributionContext(AttributionContextTracing.scala:46)
	at com.databricks.logging.AttributionContextTracing.withAttributionContext$(AttributionContextTracing.scala:43)
	at com.databricks.backend.daemon.driver.DriverWrapper.withAttributionContext(DriverWrapper.scala:73)
	at com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(DriverWrapper.scala:590)
	at com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWrapper.scala:512)
	at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:306)
	at java.lang.Thread.run(Thread.java:750)

To Reproduce

  1. Have a Databricks workspace
  2. Have an App Insights instance
  3. Create a smallest compute cluster with default configuration (I used "Personal Compute" policy).
  4. Install com.azure:azure-monitor-opentelemetry-exporter:1.0.0-beta.28 in the cluster.
  5. Create a Scala notebook
  6. Run any notebook of these: 1, 2 (all cells)
  7. Wait 1 minute
  8. Select Run -> Detach & re-attach to compute resource
  9. Run the same notebook again

With auto configuration (gist 1) the code fails on

val openTelemetry = sdkBuilder.build().getOpenTelemetrySdk()

With manual configuration (gist 2), the code fails on

val openTelemetry = OpenTelemetrySdk.builder()
    .setTracerProvider(sdkTracerProvider)
    .setMeterProvider(sdkMeterProvider)
    .setLoggerProvider(sdkLoggerProvider)
    .buildAndRegisterGlobal()

Code Snippet
See gists in repro steps.

Expected behavior
Detaching a notebook is expected to reset the state (equivalent to running a new notebook), however the cluster falls into a broken state and can be recovered only by a restart.

Screenshots
N/A

Setup (please complete the following information):

  • OS: Linux-like (unknown)
  • IDE: Databricks Notebook
  • Library/Libraries: com.azure:azure-monitor-opentelemetry-exporter:1.0.0-beta.28
  • Java version: 1.8.0.412
  • App Server/Environment: Databricks runtime 15.4 LTS ML (no GPU)
  • Frameworks: No

Additional context
Related bug: #41856.

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

  • Bug Description Added
  • Repro Steps Added
  • Setup information Added
@github-actions github-actions bot added customer-reported Issues that are reported by GitHub users external to the Azure organization. needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Sep 14, 2024
@greatvovan greatvovan changed the title [BUG] Unexpected configuration error on creating OpenTelemetry SDK object when using AzureMonitorExporterBuilder [BUG] ConfigurationException on creating OpenTelemetry SDK object when using AzureMonitorExporterBuilder Sep 14, 2024
@joshfree joshfree added the OpenTelemetry OpenTelemetry instrumentation label Sep 18, 2024
@joshfree
Copy link
Member

@lmolkova could you please take a look?

@github-actions github-actions bot removed the needs-triage Workflow: This is a new issue that needs to be triaged to the appropriate team. label Sep 18, 2024
@lmolkova
Copy link
Member

@greatvovan do you happen to have the full exception and the stack trace?

Could you please share it? And please also enable OTel internal logging and share the logs.

Thank you!

@jeanbisutti @heyams or @trask should be able to investigate further.

@lmolkova lmolkova assigned jeanbisutti and unassigned lmolkova Sep 18, 2024
@lmolkova lmolkova added the Monitor - Exporter Monitor OpenTelemetry Exporter label Sep 18, 2024
@github-actions github-actions bot added the needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team label Sep 18, 2024
@heyams
Copy link
Member

heyams commented Sep 18, 2024

@greatvovan do you happen to have the full exception and the stack trace?

Could you please share it? And please also enable OTel internal logging and share the logs.

Thank you!

@jeanbisutti @heyams or @trask should be able to investigate further.

Yes, please share logs. @greatvovan

@greatvovan
Copy link
Author

@lmolkova I shared the full stack trace in the original message.
Did you happen to expand the collapse block?
Screenshot 2024-09-18 at 2 42 32 PM

@heyams
Copy link
Member

heyams commented Sep 18, 2024

I will try to repro it.

@heyams
Copy link
Member

heyams commented Sep 19, 2024

@greatvovan I am not able to repro using 1.0.0-beta.28.
Here is my test sample. Can you provide a repro app? Please take a look at my sample app if that offers some insights.

based on the stack trace, it's to do with metadata service scheduler. it seemed that ScheduledExecutorService might be shut down before it's invoking scheduleWithFixedDelay. We do have certain cases where it needs to be shutdown. It might be this case. Can you share your full log?

@greatvovan
Copy link
Author

@heyams I cannot provide a repro app, because the essence of the bug is Databricks environment. In fact the both gists send telemetry just fine if wrapped by object Main extends App {...} and ran as standalone Scala programs (with zero code changes).

Regarding logs – as I cannot put a properties file next to a Scala notebook, I tried

import java.util.logging.{Level => JulLevel}
import java.util.logging.{Logger => JulLogger}
JulLogger.getLogger("io.opentelemetry").setLevel(JulLevel.FINEST)

But it did not produce any output. My guess is that JUL needs to be additionally configured to log to console? At this point I have nothing more to share as there is no any additional output anywhere. How else can I check/validate any ideas?

@heyams
Copy link
Member

heyams commented Sep 19, 2024

When my PR is merged, I can give you a new exporter to try it out. I suspected that would resolve your issue.

@heyams
Copy link
Member

heyams commented Oct 4, 2024

@greatvovan please try https://central.sonatype.com/artifact/com.azure/azure-monitor-opentelemetry-exporter v1.0.0-beta.29 and let us know the outcome.

@heyams heyams added needs-author-feedback Workflow: More information is needed from author to address the issue. and removed needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team labels Oct 4, 2024
Copy link

github-actions bot commented Oct 4, 2024

Hi @greatvovan. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

@greatvovan
Copy link
Author

This version seems broken, the code does not even compile complaining on missing AzureMonitorExporterBuilder symbol.

See below (not putting it in a collapse section this time):

import com.azure.monitor.opentelemetry.exporter.AzureMonitorExporterBuilder;

public class My
{
    public static void main(String[] args) {
        System.out.println("Hello");
    }
}
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.example</groupId>
    <artifactId>otel-exporter</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>17</maven.compiler.source>
        <maven.compiler.target>17</maven.compiler.target>
    </properties>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.1</version>
                <configuration>
                    <source>17</source>
                    <target>17</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>exec-maven-plugin</artifactId>
                <version>3.0.0</version>
                <configuration>
                    <mainClass>My</mainClass>
                </configuration>
            </plugin>
        </plugins>
    </build>

    <dependencies>    
        <dependency>
            <groupId>com.azure</groupId>
            <artifactId>azure-monitor-opentelemetry-exporter</artifactId>
            <version>1.0.0-beta.29</version>
        </dependency>
    </dependencies>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>io.opentelemetry.instrumentation</groupId>
                <artifactId>opentelemetry-instrumentation-bom</artifactId>
                <version>2.8.0</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
</project>
 % docker run -it --rm  -v $(pwd):/proj maven bash
root@94d2353ed224:/# cd /proj
root@94d2353ed224:/proj# mvn clean install
...
[INFO] Compiling 1 source file to /proj/target/classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /proj/src/main/java/My.java:[1,48] cannot find symbol
  symbol:   class AzureMonitorExporterBuilder
  location: package com.azure.monitor.opentelemetry.exporter
...

If I change 29 to 28, it compiles.

Did you move this class to another package or something?

@github-actions github-actions bot added needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team and removed needs-author-feedback Workflow: More information is needed from author to address the issue. labels Oct 4, 2024
@greatvovan
Copy link
Author

Okay, it looks like you renamed it into AzureMonitorExporter and changed API.

I tried the original sample with

AzureMonitorExporter.customize(sdkBuilder, aiConnStr)

instead of

new AzureMonitorExporterBuilder().connectionString(aiConnStr).install(sdkBuilder)

and the error does not reproduce.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
customer-reported Issues that are reported by GitHub users external to the Azure organization. Monitor - Exporter Monitor OpenTelemetry Exporter needs-team-attention Workflow: This issue needs attention from Azure service team or SDK team OpenTelemetry OpenTelemetry instrumentation question The issue doesn't require a change to the product in order to be resolved. Most issues start as that
Projects
None yet
Development

No branches or pull requests

5 participants