Skip to content

Flaky OTel conformance runs: Dash0 trace lookup timeout, Datadog span timestamp precision #121

Description

@yaythomas

Two OTel conformance jobs on aws-durable-execution-sdk-js main failed this week for backend-side reasons. Both passed on re-run with no code change.

1. Dash0: trace not found within 60s

OpenTelemetry backend validation failed: No correlated trace was found in backend 'dash0' after 27 attempt(s) within 60s

2. Datadog: child span starts "before" parent by sub-second skew

span_assertions[4].expect.parent: child span 'otel-retry attempt 1' starts at 2026-09-09T22:14:46+00:00, before parent span 'otel-retry' starts at 2026-09-09T22:14:46.697000+00:00

The child timestamp has whole-second precision. The parent has millisecond precision. So the backend appears to truncate one of them on ingest.

Possible fixes

  • Raise the Dash0 trace-lookup timeout above 60s, or make it backend-configurable.
  • Make the parent/child start-time assertion tolerate sub-second skew when a backend reports whole-second timestamps.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpkg:otelPackage: aws-durable-execution-conformance-tests-otel

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions