Skip to content

Commit d1b3fe4

Browse files
bradygasterbrady gasterCopilot
authored
fix(test): raise Aspire dashboard health-check timeout to absorb CI runner contention (#2132)
* fix(test): pin Aspire dashboard image to verified-good 13.5.2 build Root cause of the #2123/#2132 Aspire integration test CI failures was NOT CPU contention: `mcr.microsoft.com/dotnet/aspire-dashboard:latest` (and the major-version tag `:13`, confirmed to currently resolve to the identical freshly-published `13.6.0` build/digest) is broken — the container exits immediately (exitCode=145) because /app/Aspire.Dashboard.dll is missing from the image, so the dashboard health check can never succeed no matter the timeout. Fix: pin docker pull/run to `:13.5.2`, a distinct ~3-week-older build (different layer digests, verified via the MCR manifest API) that predates the broken `13.6.0` release and does not exhibit the missing-DLL problem. Diagnostics hardening made while chasing this down (kept, as they are independently valuable for any future flake in this suite): - Removed --rm from the dashboard `docker run` so a crashed container survives long enough for failure-path `docker inspect`/`docker logs` diagnostics to run (relying on existing explicit removeContainer() cleanup instead). - Captured container stderr in the `docker logs` diagnostics call (`2>&1`), since .NET unhandled-exception output goes to stderr and was previously silently dropped, hiding the real crash reason. - Raised isDockerAvailable()'s `docker info` timeout from 5s to 20s to prevent false-skip of the whole Docker-dependent suite under CI CPU contention (which would otherwise look like a false pass). - Raised the dashboard health-check timeout to 120s as defense-in-depth against genuine contention-induced slow starts, independent of the image-pinning fix above. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(test): widen beforeAll hook budget for diagnostics margin Addresses reviewer feedback on PR #2132: - The 240s beforeAll hook budget had zero margin above the bounded 120s docker-pull timeout + 120s health-check timeout, so Vitest could abort the hook with a generic timeout before waitForHealthy's more informative crash diagnostics could be thrown. Widened to 300s (~60s margin for container startup, diagnostics collection, and Playwright browser launch). - The --rm-vs-diagnostics concern on the same thread was already resolved in a prior commit (docker run no longer passes --rm); no further change needed there. Also trims two overly verbose/speculative inline comments (image-pin rationale, health-check rationale) down to the essential facts. Closes #2133 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: brady gaster <bradygaster@github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
1 parent 10a8931 commit d1b3fe4

2 files changed

Lines changed: 49 additions & 11 deletions

File tree

‎test/aspire-integration.test.ts‎

Lines changed: 41 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -36,13 +36,20 @@ const SKIP_REASON = dockerSkipReason() ?? playwrightBrowserSkipReason();
3636
const CONTAINER_NAME = 'squad-aspire-dashboard';
3737
const DASHBOARD_URL = 'http://localhost:18888';
3838
const OTLP_GRPC_TARGET = 'http://localhost:4317';
39+
// Pin to a verified-good patch version, not a floating tag: `:latest` and
40+
// the major-version tag `:13` both currently resolve to a broken same-day
41+
// upstream build (missing /app/Aspire.Dashboard.dll, confirmed via MCR
42+
// manifest digest comparison). `13.5.2` is a distinct, older build that
43+
// does not exhibit the problem. Re-pin forward once a newer verified-good
44+
// patch tag is published upstream.
45+
const DASHBOARD_IMAGE = 'mcr.microsoft.com/dotnet/aspire-dashboard:13.5.2';
3946

4047
// ============================================================================
4148
// Helpers
4249
// ============================================================================
4350

4451
/** Poll a URL until it responds with 200 or timeout expires. */
45-
async function waitForHealthy(url: string, timeoutMs = 60_000): Promise<void> {
52+
async function waitForHealthy(url: string, timeoutMs = 120_000): Promise<void> {
4653
const start = Date.now();
4754
while (Date.now() - start < timeoutMs) {
4855
try {
@@ -53,7 +60,24 @@ async function waitForHealthy(url: string, timeoutMs = 60_000): Promise<void> {
5360
}
5461
await new Promise((r) => setTimeout(r, 1_000));
5562
}
56-
throw new Error(`Dashboard at ${url} did not become healthy within ${timeoutMs}ms`);
63+
// Surface container diagnostics so a slow/crashed start is distinguishable
64+
// from a genuinely unreachable dashboard without needing to re-run in CI.
65+
let diagnostics = '';
66+
try {
67+
const state = execSync(
68+
`docker inspect --format="{{.State.Status}} (exitCode={{.State.ExitCode}})" ${CONTAINER_NAME}`,
69+
{ encoding: 'utf8' },
70+
).trim();
71+
// `docker logs` relays container stdout/stderr as two separate OS
72+
// streams; execSync only returns stdout by default, so a crash reported
73+
// via stderr (the common case for unhandled .NET exceptions) would be
74+
// silently dropped. Redirect stderr into stdout so it's captured too.
75+
const logs = execSync(`docker logs --tail 50 ${CONTAINER_NAME} 2>&1`, { encoding: 'utf8' });
76+
diagnostics = `\nContainer state: ${state}\nLast logs:\n${logs}`;
77+
} catch (err) {
78+
diagnostics = `\n(failed to collect container diagnostics: ${(err as Error).message})`;
79+
}
80+
throw new Error(`Dashboard at ${url} did not become healthy within ${timeoutMs}ms${diagnostics}`);
5781
}
5882

5983
/** Force-remove the test container (ignore errors). */
@@ -122,33 +146,40 @@ describe.skipIf(SKIP_REASON !== null)(
122146
// Clean up any leftover container from a prior run
123147
removeContainer();
124148

125-
// Pull latest Aspire dashboard image
149+
// Pull the pinned Aspire dashboard image (see DASHBOARD_IMAGE above)
126150
execSync(
127-
'docker pull mcr.microsoft.com/dotnet/aspire-dashboard:latest',
151+
`docker pull ${DASHBOARD_IMAGE}`,
128152
{ stdio: 'inherit', timeout: 120_000 },
129153
);
130154

131-
// Start the Aspire dashboard container
155+
// Start the Aspire dashboard container.
156+
// Deliberately NOT using --rm: if the container exits/crashes instead
157+
// of merely starting slowly, --rm would auto-delete it before
158+
// waitForHealthy's failure-path diagnostics (docker inspect/logs) can
159+
// run, masking the real cause as a generic timeout. removeContainer()
160+
// (called above and in afterAll) already guarantees explicit cleanup.
132161
execSync(
133162
[
134-
'docker run --rm -d',
163+
'docker run -d',
135164
`-p 18888:18888 -p 4317:18889`,
136165
'-e ASPIRE_DASHBOARD_UNSECURED_ALLOW_ANONYMOUS=true',
137166
`--name ${CONTAINER_NAME}`,
138-
'mcr.microsoft.com/dotnet/aspire-dashboard:latest',
167+
DASHBOARD_IMAGE,
139168
].join(' '),
140169
{ stdio: 'inherit' },
141170
);
142171

143-
// Wait for dashboard UI to respond
144-
await waitForHealthy(DASHBOARD_URL, 60_000);
172+
// 120s gives generous headroom over the dashboard's typical
173+
// few-second cold start, as defense-in-depth against CI runner
174+
// contention (independent of the image-pinning fix above).
175+
await waitForHealthy(DASHBOARD_URL, 120_000);
145176

146177
// Initialize OTel gRPC exporters targeting the dashboard
147178
initOTelForAspire();
148179

149180
// Launch Playwright browser
150181
browser = await chromium.launch({ headless: true });
151-
}, 180_000); // 3 min timeout for pull + start
182+
}, 300_000); // 5 min timeout: 120s pull + 120s health-check budget, plus ~60s margin for container startup, diagnostics, and browser launch
152183

153184
// ------------------------------------------------------------------
154185
// Teardown: shutdown OTel, close browser, remove container

‎test/helpers/skip-guards.ts‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,14 @@ import { execSync } from 'node:child_process';
1313
*/
1414
export function isDockerAvailable(): boolean {
1515
try {
16-
execSync('docker info', { stdio: 'ignore', timeout: 5000 });
16+
// 5s was too tight: under full-parallelism CI load (~10k tests running
17+
// concurrently), `docker info`'s round-trip to the daemon can occasionally
18+
// exceed 5s purely from CPU contention, causing this check to time out
19+
// and silently SKIP the whole Docker-dependent suite instead of running
20+
// it — a false "pass" that masks real failures (see Aspire dashboard CI
21+
// investigation, #2123/#2132). 20s gives enough headroom to reliably
22+
// detect a genuinely-available daemon under the same load.
23+
execSync('docker info', { stdio: 'ignore', timeout: 20_000 });
1724
return true;
1825
} catch {
1926
return false;

0 commit comments

Comments
 (0)