From 2af28c9d481f9da4f96283a5af9cb8879e566872 Mon Sep 17 00:00:00 2001 From: Martin Vogel Date: Thu, 23 Jul 2026 08:42:30 +0200 Subject: [PATCH 1/4] test: size the daemon-application wait budget for TSan's slowdown tail MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The suite's observation waits shared a 2000 ms budget that is a hang detector, not a latency assertion — every waiter returns the moment its condition holds, so green runs never pay it. Under ThreadSanitizer's 5-20x slowdown on a loaded 3-core macOS CI runner the cancel-delivery wait lost the tail of that distribution once in seven otherwise-green TSan rounds (the cancel itself had succeeded; only the fake worker's counter observation timed out). The budget is now 10000 ms. Verified: nine consecutive daemon_application runs under native macOS TSan with the widened budget, all green, zero ThreadSanitizer warnings. Signed-off-by: Martin Vogel --- tests/test_daemon_application.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/tests/test_daemon_application.c b/tests/test_daemon_application.c index 218ab13c3..167d5255f 100644 --- a/tests/test_daemon_application.c +++ b/tests/test_daemon_application.c @@ -27,7 +27,12 @@ #include #include -enum { APP_TEST_TIMEOUT_MS = 2000, APP_TEST_PATH_CAP = 1024 }; +/* Observation-wait budget. This is a HANG DETECTOR, not a latency + * assertion: every waiter returns the moment its condition holds, so green + * runs never pay it. It must absorb ThreadSanitizer's 5-20x slowdown on a + * loaded 3-core CI runner — at 2000 ms the cancel-delivery wait lost the + * tail of that distribution once in seven otherwise-green TSan rounds. */ +enum { APP_TEST_TIMEOUT_MS = 10000, APP_TEST_PATH_CAP = 1024 }; typedef struct { char runtime_parent[APP_TEST_PATH_CAP]; From acce9a4582eed3409f3719228023317addae20ad Mon Sep 17 00:00:00 2001 From: Martin Vogel Date: Thu, 23 Jul 2026 08:49:35 +0200 Subject: [PATCH 2/4] test: make the Windows build-dir DACL stamp verify itself out loud MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The dry run's windows-11-arm shard failed its install-flow tests with the source-directory refusal the stamp exists to prevent — while the identical commands, run from the identically dirty state on the ARM64 VM, stamp clean and pass. The difference is invisible because the stamp was fully silenced; a silent load-bearing step cost a full CI round to even learn whether it had run. The stamp now reports each icacls failure with the user and directory, re-inspects the DACL afterwards, and prints one line in either direction: stamped clean, or the surviving cross-account grants. Verified on the VM from a deliberately reset (inherited, Authenticated- Users-writable) build directory through the real --par path: stamp reports clean, suites green. Signed-off-by: Martin Vogel --- scripts/run-tests-parallel.sh | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/scripts/run-tests-parallel.sh b/scripts/run-tests-parallel.sh index f9e06c734..62718da79 100644 --- a/scripts/run-tests-parallel.sh +++ b/scripts/run-tests-parallel.sh @@ -52,10 +52,21 @@ case "$(uname -s 2>/dev/null)" in MINGW* | MSYS*) runner_dir_w="$(cygpath -w "$(dirname "$RUNNER")")" me="$(whoami | tr -d '\r')" - MSYS2_ARG_CONV_EXCL='*' icacls "$runner_dir_w" /inheritance:r \ + stamp_out=$(MSYS2_ARG_CONV_EXCL='*' icacls "$runner_dir_w" /inheritance:r \ /grant:r "${me}:(OI)(CI)F" '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' \ - /Q >/dev/null 2>&1 || true - MSYS2_ARG_CONV_EXCL='*' icacls "${runner_dir_w}\\*" /reset /T /C /Q >/dev/null 2>&1 || true + /Q 2>&1) || echo "WARN: build-dir DACL stamp failed (user=$me dir=$runner_dir_w): $stamp_out" + reset_out=$(MSYS2_ARG_CONV_EXCL='*' icacls "${runner_dir_w}\\*" /reset /T /C /Q 2>&1) || + echo "WARN: build-dir child DACL reset failed: $(printf '%s' "$reset_out" | tail -2)" + # The stamp is load-bearing for the install-flow suites: verify it and say + # so, in either direction — a silent stamp once cost a full CI round to + # even see WHETHER it had run. + if MSYS2_ARG_CONV_EXCL='*' icacls "$runner_dir_w" 2>/dev/null | + grep -qE 'Authenticated Users|CREATOR OWNER'; then + echo "WARN: build-dir DACL still grants cross-account mutation after stamp:" + MSYS2_ARG_CONV_EXCL='*' icacls "$runner_dir_w" 2>&1 | head -8 + else + echo "build-dir DACL stamped clean ($runner_dir_w, user=$me)" + fi ;; esac From dc5649d1e6d35c74dfb81596188e39a8aea64f19 Mon Sep 17 00:00:00 2001 From: Martin Vogel Date: Thu, 23 Jul 2026 08:54:56 +0200 Subject: [PATCH 3/4] test: re-stamp the Windows build-dir DACL at the serial-tail boundary MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit On one dry run the two arm shards split: the shard whose serial tail held the install-flow suites failed the source-directory policy minutes after its pre-wave stamp, while its sibling — same runner image, same stamp — passed with no install-flow suites to notice. Wave suites spawn Cygwin-family tooling that can rewrite the build directory's DACL behind the first stamp, so the stamp is now a function invoked both pre-wave and at the tail boundary, where the deadline-sensitive suites that depend on it actually run. Both invocations self-verify out loud. Verified on the VM from a deliberately reset (inherited) build directory through the real --par path: both stamps report clean, mini shard green. Signed-off-by: Martin Vogel --- scripts/run-tests-parallel.sh | 28 ++++++++++++++++++++-------- 1 file changed, 20 insertions(+), 8 deletions(-) diff --git a/scripts/run-tests-parallel.sh b/scripts/run-tests-parallel.sh index 62718da79..5062d20af 100644 --- a/scripts/run-tests-parallel.sh +++ b/scripts/run-tests-parallel.sh @@ -48,27 +48,32 @@ mkdir -p "$LOGDIR" # setup. Two idempotent steps: protect the DIRECTORY (inheritance flags are # directory-only — a /T re-root leaves files with empty deny-all DACLs), # then /reset the children to re-inherit the clean set. -case "$(uname -s 2>/dev/null)" in -MINGW* | MSYS*) +stamp_windows_build_dir() { + local when="$1" + case "$(uname -s 2>/dev/null)" in + MINGW* | MSYS*) ;; + *) return 0 ;; + esac + local runner_dir_w me stamp_out reset_out runner_dir_w="$(cygpath -w "$(dirname "$RUNNER")")" me="$(whoami | tr -d '\r')" stamp_out=$(MSYS2_ARG_CONV_EXCL='*' icacls "$runner_dir_w" /inheritance:r \ /grant:r "${me}:(OI)(CI)F" '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' \ - /Q 2>&1) || echo "WARN: build-dir DACL stamp failed (user=$me dir=$runner_dir_w): $stamp_out" + /Q 2>&1) || echo "WARN: build-dir DACL stamp ($when) failed (user=$me dir=$runner_dir_w): $stamp_out" reset_out=$(MSYS2_ARG_CONV_EXCL='*' icacls "${runner_dir_w}\\*" /reset /T /C /Q 2>&1) || - echo "WARN: build-dir child DACL reset failed: $(printf '%s' "$reset_out" | tail -2)" + echo "WARN: build-dir child DACL reset ($when) failed: $(printf '%s' "$reset_out" | tail -2)" # The stamp is load-bearing for the install-flow suites: verify it and say # so, in either direction — a silent stamp once cost a full CI round to # even see WHETHER it had run. if MSYS2_ARG_CONV_EXCL='*' icacls "$runner_dir_w" 2>/dev/null | grep -qE 'Authenticated Users|CREATOR OWNER'; then - echo "WARN: build-dir DACL still grants cross-account mutation after stamp:" + echo "WARN: build-dir DACL still grants cross-account mutation after $when stamp:" MSYS2_ARG_CONV_EXCL='*' icacls "$runner_dir_w" 2>&1 | head -8 else - echo "build-dir DACL stamped clean ($runner_dir_w, user=$me)" + echo "build-dir DACL stamped clean ($when, $runner_dir_w, user=$me)" fi - ;; -esac +} +stamp_windows_build_dir pre-wave SUITES_FILE="$LOGDIR/suites.txt" RESULTS_FILE="$LOGDIR/results.txt" @@ -243,6 +248,13 @@ while IFS= read -r sname; do echo "$sname" >> "$FLEX_FILE" fi done < "$SER_FILE" +# Wave suites spawn Cygwin-family tooling that can rewrite the build +# directory's DACL behind the first stamp (observed: an arm shard whose +# tail held the install-flow suites failed the source-directory policy +# minutes after a clean pre-wave stamp, while its sibling shard passed). +# Re-stamp at the tail boundary so the deadline-sensitive tail — which +# hosts those suites — always starts from the verified-clean shape. +stamp_windows_build_dir pre-tail xargs -P "${CBM_TAIL_JOBS:-2}" -I{} bash -c 'run_one "$@"' _ {} < "$FLEX_FILE" while IFS= read -r sname; do run_one "$sname" From e06d1b520e9671c57e1bf04cb9c0ef8d278f4985 Mon Sep 17 00:00:00 2001 From: Martin Vogel Date: Thu, 23 Jul 2026 09:02:17 +0200 Subject: [PATCH 4/4] fix(daemon,win): give conflict-log share collisions a contention budget MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The private-file prepare treated slow CONTENTION like permanent OBSTRUCTION: one 250 ms deadline for both. With many concurrent appenders on a loaded, sanitized runner the last writer can legitimately wait longer than that behind its peers' brief exclusive windows, and the concurrent-append guard correctly failed a dropped conflict event (dry run: one appender of the pack lost). Transient share/lock violations now retry against a 2000 ms contention budget while every other error class keeps the 250 ms fast-fail — a rejected client is still waiting behind this on the hello path. Signed-off-by: Martin Vogel --- src/daemon/service.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/src/daemon/service.c b/src/daemon/service.c index 777700f88..13413be2f 100644 --- a/src/daemon/service.c +++ b/src/daemon/service.c @@ -856,12 +856,26 @@ static bool windows_private_file_prepare(const char *directory, const char *base * to the ~16 ms timer granularity, and a permanently obstructed path * (e.g. a directory squatting on it) must fail fast — a rejected client * is waiting behind this on the hello path with its own timeout. */ - ULONGLONG deadline = GetTickCount64() + 250; + ULONGLONG obstruction_deadline = GetTickCount64() + 250; + /* Genuine share collisions get a wider budget than obstruction: with + * many concurrent appenders on a loaded, sanitized CI runner the + * last-in-line writer can legitimately spend more than 250 ms behind + * its peers' brief exclusive windows, and the no-drop contract for + * conflict events outranks latency there — while a permanently + * obstructed path (a directory squatting on the name) still fails + * inside 250 ms, because a rejected client is waiting behind this on + * the hello path with its own timeout. */ + ULONGLONG contention_deadline = GetTickCount64() + 2000; for (;;) { + SetLastError(ERROR_SUCCESS); FILE *file = cbm_daemon_ipc_private_log_open(directory, base, SIZE_MAX); if (file) { return fclose(file) == 0; } + DWORD open_error = GetLastError(); + bool transient_collision = + open_error == ERROR_SHARING_VIOLATION || open_error == ERROR_LOCK_VIOLATION; + ULONGLONG deadline = transient_collision ? contention_deadline : obstruction_deadline; if (GetTickCount64() >= deadline) { return false; }