v2.5.2 - System Metrics on Scala 3 & Network Metrics Fixes

Improvements:

• system-metrics": This module is now built and published for Scala 3 as well. Contributed by @ornicar via #1155
• otel: The OpenTelemetry reporter now supports the OTEL_RESOURCE_ATTRIBUTES environment variable. Contributed by @hughsimpson via #1157
• otel: The OpenTelemetry exporter dependencies are now bumped to 1.13.0. Contributed by @hughsimpson via #1158

Fixes:

• system-metrics: No network metrics were reported by the System Metrics module when running inside docker containers. Fixed by @jypma via #1165

v2.5.1 - OpenTelemetry over HTTP & Datadog Propagation

New Features

• core: This release ships with a new datadog Span propagation scheme that uses the x-datadog-trace-id, x-datadog-parent-id, and x-datadog-sampling-priority headers for all tracing information, exactly as Datadog does it. Contributed by @seglo and @leonwlaw via #1145
• core: It is now possible to filter the metrics sent to a metrics reporter via configuration. You can use kamon.modules.reporter-name.metric-filters to configure an includes/excludes filter with the metrics you want to be exported. Contributed by @ivantopo via #1148
• otel: The OpenTelemetry traces reporter now supports http/protobuf protocol. The entire OTel reporter was rewritten on top of the official OTel exporter and should make it really easy to get into exporting metrics soon 🎉. Contributed by @hughsimpson via #1102

Fixes

• twitter-future: Fixed context propagation issues that were haunting us for ages in the Twitter Future. Many thanks to @dispalt for providing reproducibles for the issue and helping debug this issue. Fixed by @ivantopo via #1147
• akka-http: We were unable to trace HTTP/1 requests on Akka HTTP servers that had HTTP/2 enabled, now we can. Many thanks to @jtjeferreira for writing a failing test for the issue. Fixed by @ivantopo via #1094

v2.5.0 - Akka gRPC Server Support & Improvements

Akka gRPC Server Support

Starting on this release, Kamon Telemetry ships with support for tracing server-side Akka gRPC requests implemented with Akka HTTP, Play Framework, and Lagom. This instrumentation is the first step towards full Akka gRPC support and we will soon add:

• gRPC client tracing
• Context propagation across gRPC channels

Improvements

• core: Add the error.type tag to traces, getting us closer to follow the OpenTelemetry semantic conventions. Contributed by @hughsimpson via #1112
• kafka: Prevent the Kafka producer instrumentation from starting traces when there is no current trace on the application. There is a new kamon.instrumentation.kafka.client.tracing.start-trace-on-producer setting to control this behavior. Contributed by @ivantopo via #1125
• akka-grpc: New instrumentation for server-side Akka gRPC. Contributed by @ivantopo via #1119
• akka: #1093 New instrumentation for Akka's Scheduler scheduler.scheduleOnce function, which should bring proper context propagation for pattern.retry and pattern.after as well. Contributed by @ivantopo via #1135
• executors: #641 Avoid reflective access on the executors instrumentation. Contributed by @ivantopo via #1136

Fixes

• influxdb: #1129, #1131 There two bugs regarding encoding and configuration settings on the InfluxDB Authorization Token support introduced on Kamon Telemetry 2.4.8, those are gone now. Fixed by @duese via #1130
• redis: #1082 The Jedis instrumentation wasn't measuring the entire roundtrip time to Redis, only the time it took to send out the commands. Fixed by @ivantopo via #1117
• redis: #1072 Ensure the Akka instrumentation takes the Rediscala actors' exclusions into account. Reported by @jtjeferreira and fixed by @ivantopo via #1122
• akka:" #1116 Stop generating spans for Akka Streams-related actors. Fixed by @ivantopo via #1133
• akka: #1132 Fix module registration warnings for Akka Dispatchers. Fixed by @ivantopo via #1138

Breaking Changes 🚨

• We removed the Play gRPC-specific instrumentation that was included in kamon-play to the new kamon-akka-grpc hooks at a lower level that will work with both, coming with two important changes:
  • If you are not using the Kamon Bundle but instead adding dependencies one by one, you should add the new kamon-akka-grpc dependency for to your classpath
  • There is no custom operation name generator support in kamon-play anymore. All gRPC operations are now instrumented with the tag names defined by the OpenTelemetry semantic conventions and there shouldn't be any need to deviate from that with a custom generator
• The operation tag in Play gRPC requests will no longer have a / prefix, meaning that operations that previously were reported as /ReplyService/SayHello will now show up as ReplyService/SayHello. You might need to update your dashboards and alert conditions to account for that
• Kamon can no longer instrument Executors created by these helper functions from java.util.concurrent.Executors:
  • newSingleThreadExecutor
  • newSingleThreadScheduledExecutor
  • unconfigurableExecutorService
  • unconfigurableScheduledExecutorService

v2.4.8 - Token Authentication for InfluxDB

Starting on this release you can use API Token authentication when sending data to InfluxDB using this setting:

kamon.influxdb.authentication {
  token = "your-glorious-token-here"
}

Improvements

• influxdb. Introduced a new authentication.token configuration setting for cases where the InfluxDB reporter needs to authenticate with an API token. Contributed by @ivantopo via #1110.

v2.4.7 - Controlling the db.statement tag in JDBC calls

You can now use the kamon.instrumentation.jdbc.add-db-statement-as-span-tag setting to control the db.statement tag in JDBC calls. This is especially important when your JDBC calls might contain sensitive information that you don't want to share with third parties, like your tracing vendor. The possible setting values are:

• always: (default) will always add the db.statement tag to spans
• prepared: will only add the the db.statement tag when the traced execution was made with a PreparedStatement, which should ensure no placeholder values are leaked to the tracing backend
• never: completely disables adding the db.statement tag to spans

Improvements

• jdbc: control what kind of JDBC statements will be added to the db.statement tag in JDBC spans. Contributed by @ivantopo via #1103

v2.4.6 - Namespaced Environment Tags

Namespaced Environment Tags

Now you can add namespaced (i.e. tags with dots in their name) environment tags to the kamon.environment.tags settings in your configuration file.

kamon.environment.tags {
  container.id = "4444444444"
  k8s {
    namespace.name = "production"
    cluster {
      name = "strong-cluster"
    }
  } 
}

For example, the settings above in your application.conf file would add the container.id, k8s.namespace.name, and k8s.cluster.name tags to the environment. Most reporters will automatically add all of these tags to metrics and spans before they get sent out.

Improvements

• core: It is now possible to have namespaced tags inside the kamon.environment.tags configuration. Contributed by @ivantopo via #1101.

v2.4.5 - Support for Finagle Client & Lagom Circuit Breakers

New Instrumentation

• finagle: Now you can add a new HTTP filter to trace Finagle Client requests. Contributed by @seglo via #1096
• lagom: We are now shipping a Lagom CircuitBreakerMetricsProvider that keeps track of all circuit breakers by default. Contributed by @ihostage via #760

New Features

• runtime-attacher: We are now shipping a new artifact called kamon-runtime-attacher that contains the Kanela agent and the logic to attach it in a running JVM. It is basically the bundle, but without the dependencies. Contributed by @ivantopo via d5ce436
• apm: The Kamon APM reporter now sends environment tags as part for metrics and spans ingestion requests. Contributed by @ivantopo via 2235b79

v2.4.2 - Maintenance Release

Fixes

• akka-http: Fix a "requirement failed: HTTP/1.0 responses must not have a chunked entity" error message when using HttpEntity.Default in responses. Fixed by @ivantopo via d8a5791

v2.4.1 - JVM Thread States Metrics

This improvement was originally meant to go out with v2.4.0, but it slipped through the cracks. Sorry!

Improvements

• system: Add a new jvm.threads.states gauge tracking the number of threads on each possible state. Contributed by @getArtemUsername via #1075

v2.4.0 - Local Tail Sampling & More!

Introducing Local Tail Sampling

This release comes with a new way of ensuring you will get traces for slow and failed requests, at least locally: the Local Tail-based Sampler.

The quick summary is that the local tail sampler can keep finished spans in memory for a few seconds and then override their sampling decision if they reach a configurable latency or error count threshold. If you want the full story check out our post about how to keep traces for slow and failed requests.

You can enabled the local tail sampler with these settings:

kamon.trace {
  span-reporting-delay = 20 seconds

  local-tail-sampler {
    enabled = yes
    error-count-threshold = 1
    latency-threshold = 1 second
  }
}

Improvements

• datadog: The OkHttp dependency is now shaded into the Datadog reporter jar to avoid classpath clashes. Contributed by @seglo via #1069
• prometheus: Get easy access to the Prometheus scrape data through PrometheusReporter.latestScrapeData() in case you need it for exposing on your own HTTP server. Contributed by @ivantopo via #1079
• redis: The Jedis instrumentation now works with Jedis 2.x as well. There is still a known issue in #1082 that should get solved soon after. Contributed by @jtjeferreira via #1076
• apm: The Kamon APM reporter now drops old Spans that might be accumulated in the reporters' executor and comes with lower retries and backoff default settings. Contributed by @ivantopo via eeb0596 and 49aef33

Fixes

• bundle: Remove a misplaced logback.xml from kamon-mongo that was ending up in the bundle. Thanks @PavelPenkov for reporting the issue. Fixed by @ivantopo via d37cb48
• general: @Philippus helped us get a bunch of typos fixed!