Skip to content

3.x: ShardAwarenessTest.correctShardInTracingTest flakes when tracing lacks querying locally event #946

Description

@dkropachev

Problem

ShardAwarenessTest.correctShardInTracingTest intermittently fails in the 3.x Scylla integration-test matrix because the query trace sometimes does not include any event whose description contains querying locally.

Observed on PR #943, head efe29bf2e24b357417d7134b6ac39c9659f9cd38, in Scylla ITs (LATEST, 8):

https://github.com/scylladb/java-driver/actions/runs/28683913680/job/85072817714

Failure:

java.lang.AssertionError: No 'querying locally' trace event was observed for the query expected [true] but found [false]
    at com.datastax.driver.core.ShardAwarenessTest.verifyCorrectShardSingleRow(ShardAwarenessTest.java:95)
    at com.datastax.driver.core.ShardAwarenessTest.correctShardInTracingTest(ShardAwarenessTest.java:115)

The same surefire result shows that this failed only on the first attempt and then passed on retry:

Run 1: ShardAwarenessTest.correctShardInTracingTest:115->verifyCorrectShardSingleRow:95 No 'querying locally' trace event was observed for the query expected [true] but found [false]
Run 2: PASS

A similar failure was also seen in an earlier PR #943 run, Scylla ITs (LTS-LATEST, 8), where the first invocation failed at ShardAwarenessTest.java:113 and the retry passed:

https://github.com/scylladb/java-driver/actions/runs/28670230744/job/85031593100

Notes

This does not look directly related to PR #943: that PR changes only Connection, RequestHandler, and RequestHandlerTest, while the failing test is ShardAwarenessTest.

There is a related merged fix in #931 for a different ShardAwarenessTest.correctShardInTracingTest failure on Scylla 2025.1. That fix made the anyLocal assertion real, so this missing-querying locally case now fails visibly, but I did not find an existing GitHub or Jira issue tracking this exact flaky symptom.

Expected

ShardAwarenessTest.correctShardInTracingTest should be deterministic in CI, or the test should wait/retry trace-event collection if the querying locally event can appear asynchronously.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions