Skip to content

CCM add node fails with exit code 2 — 5 tests broken on master #133

Description

@dkropachev

Problem

5 tests that dynamically add nodes to a running CCM cluster are consistently failing on master. The ccm add command exits with code 2, and the Java driver CCMBridge does not capture or log CCM stderr, so the Scylla-side error is not visible.

Build: #1930 (2026-03-18)
Build node: minipc3.cloudius-systems.com
Scylla version: 2026.2.0~dev (release 0.20260318.88b98fac3a0f)
Affected driver versions: Both datastax v3 (5/993 failures) and datastax v4 (5/995 failures)

Failing tests

All 5 tests fail identically with the same exception chain:

Test Source Line Node Being Added
SessionLeakTest.connectionLeakTest :70 (v3) / :69 (v4) node2
NodeRefreshDebouncerTest.should_call_onAdd_with_bootstrap_stop_start :52 node2
MetadataTest.should_update_metadata_on_topology_change :91 node4
StateListenerTest.should_receive_events_when_node_states_change :43 node2
RefreshConnectedHostTest.should_refresh_single_connected_host :63 node3

Error details

com.datastax.driver.core.CCMException: The command ccm add node2 -d dc1 -i 127.0.1.2 
  -t 127.0.1.2:42753 -l 127.0.1.2:34231 --binary-itf 127.0.1.2:33635 -j 34983 
  -r 127.0.1.2:40367 -s -b --scylla --config-dir=/tmp/2162260084678700738 failed to execute
    at com.datastax.driver.core.CCMBridge.execute(CCMBridge.java:894)
    at com.datastax.driver.core.CCMBridge.execute(CCMBridge.java:906)
    at com.datastax.driver.core.CCMBridge.add(CCMBridge.java:740)
    at com.datastax.driver.core.CCMBridge.add(CCMBridge.java:729)
    at com.datastax.driver.core.CCMCache$CachedCCMAccess.add(CCMCache.java:211)
    ...
Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 2 (Exit value: 2)
    at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
    at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:166)
    at com.datastax.driver.core.CCMBridge.execute(CCMBridge.java:876)

Likely root cause

This is possibly the same issue as scylladb/scylla-ccm#725 — the Scylla maintenance socket path exceeds the 108-character Unix domain socket limit (sun_path in sockaddr_un). The python-driver-matrix test confirmed this exact error for the same Scylla build:

Startup failed: std::runtime_error (Maintenance socket path is too long:
.../node1/cql.m. Change it to string shorter than 108 chars.)

The Java driver's CCMBridge.execute() does not capture or forward CCM's stderr output, making it impossible to confirm from the Jenkins log alone. The only visible information is exit code 2.

Additional note

The Java driver CCMBridge should ideally capture and include CCM stderr in the CCMException message to make such failures diagnosable without SSH access to the build node.

Impact

These 5 tests have been failing consistently for 3+ days on master across both driver versions.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions