Skip to content

Commit 374269e

Browse files
brady gasterCopilot
andcommitted
fix(test): pin Aspire dashboard image to verified-good 13.5.2 build
Root cause of the #2123/#2132 Aspire integration test CI failures was NOT CPU contention: `mcr.microsoft.com/dotnet/aspire-dashboard:latest` (and the major-version tag `:13`, confirmed to currently resolve to the identical freshly-published `13.6.0` build/digest) is broken — the container exits immediately (exitCode=145) because /app/Aspire.Dashboard.dll is missing from the image, so the dashboard health check can never succeed no matter the timeout. Fix: pin docker pull/run to `:13.5.2`, a distinct ~3-week-older build (different layer digests, verified via the MCR manifest API) that predates the broken `13.6.0` release and does not exhibit the missing-DLL problem. Diagnostics hardening made while chasing this down (kept, as they are independently valuable for any future flake in this suite): - Removed --rm from the dashboard `docker run` so a crashed container survives long enough for failure-path `docker inspect`/`docker logs` diagnostics to run (relying on existing explicit removeContainer() cleanup instead). - Captured container stderr in the `docker logs` diagnostics call (`2>&1`), since .NET unhandled-exception output goes to stderr and was previously silently dropped, hiding the real crash reason. - Raised isDockerAvailable()'s `docker info` timeout from 5s to 20s to prevent false-skip of the whole Docker-dependent suite under CI CPU contention (which would otherwise look like a false pass). - Raised the dashboard health-check timeout to 120s as defense-in-depth against genuine contention-induced slow starts, independent of the image-pinning fix above. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
1 parent 10a8931 commit 374269e

2 files changed

Lines changed: 60 additions & 11 deletions

File tree

‎test/aspire-integration.test.ts‎

Lines changed: 52 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -36,13 +36,25 @@ const SKIP_REASON = dockerSkipReason() ?? playwrightBrowserSkipReason();
3636
const CONTAINER_NAME = 'squad-aspire-dashboard';
3737
const DASHBOARD_URL = 'http://localhost:18888';
3838
const OTLP_GRPC_TARGET = 'http://localhost:4317';
39+
// Pin to a specific, verified-good patch version rather than a floating tag.
40+
// Confirmed in CI: `:latest` AND the floating major tag `:13` currently both
41+
// resolve to the SAME freshly-published `13.6.0` image (built 2026-10-01,
42+
// same layer digests/sha256:239e58... as `:latest`), and that image's
43+
// entrypoint fails immediately with "Aspire.Dashboard.dll does not exist" /
44+
// "No .NET SDKs were found" — the app binary is missing from the image
45+
// itself (confirmed via `docker inspect`/`docker logs`, not a slow/contended
46+
// start). `13.5.2` is a distinct, ~3-week-older build (different layer
47+
// digests, verified via the MCR manifest API) that predates this broken
48+
// release and does not exhibit the problem. Re-pin forward once a newer
49+
// verified-good patch tag is published upstream.
50+
const DASHBOARD_IMAGE = 'mcr.microsoft.com/dotnet/aspire-dashboard:13.5.2';
3951

4052
// ============================================================================
4153
// Helpers
4254
// ============================================================================
4355

4456
/** Poll a URL until it responds with 200 or timeout expires. */
45-
async function waitForHealthy(url: string, timeoutMs = 60_000): Promise<void> {
57+
async function waitForHealthy(url: string, timeoutMs = 120_000): Promise<void> {
4658
const start = Date.now();
4759
while (Date.now() - start < timeoutMs) {
4860
try {
@@ -53,7 +65,24 @@ async function waitForHealthy(url: string, timeoutMs = 60_000): Promise<void> {
5365
}
5466
await new Promise((r) => setTimeout(r, 1_000));
5567
}
56-
throw new Error(`Dashboard at ${url} did not become healthy within ${timeoutMs}ms`);
68+
// Surface container diagnostics so a slow/crashed start is distinguishable
69+
// from a genuinely unreachable dashboard without needing to re-run in CI.
70+
let diagnostics = '';
71+
try {
72+
const state = execSync(
73+
`docker inspect --format="{{.State.Status}} (exitCode={{.State.ExitCode}})" ${CONTAINER_NAME}`,
74+
{ encoding: 'utf8' },
75+
).trim();
76+
// `docker logs` relays container stdout/stderr as two separate OS
77+
// streams; execSync only returns stdout by default, so a crash reported
78+
// via stderr (the common case for unhandled .NET exceptions) would be
79+
// silently dropped. Redirect stderr into stdout so it's captured too.
80+
const logs = execSync(`docker logs --tail 50 ${CONTAINER_NAME} 2>&1`, { encoding: 'utf8' });
81+
diagnostics = `\nContainer state: ${state}\nLast logs:\n${logs}`;
82+
} catch (err) {
83+
diagnostics = `\n(failed to collect container diagnostics: ${(err as Error).message})`;
84+
}
85+
throw new Error(`Dashboard at ${url} did not become healthy within ${timeoutMs}ms${diagnostics}`);
5786
}
5887

5988
/** Force-remove the test container (ignore errors). */
@@ -122,33 +151,46 @@ describe.skipIf(SKIP_REASON !== null)(
122151
// Clean up any leftover container from a prior run
123152
removeContainer();
124153

125-
// Pull latest Aspire dashboard image
154+
// Pull the pinned Aspire dashboard image (see DASHBOARD_IMAGE above)
126155
execSync(
127-
'docker pull mcr.microsoft.com/dotnet/aspire-dashboard:latest',
156+
`docker pull ${DASHBOARD_IMAGE}`,
128157
{ stdio: 'inherit', timeout: 120_000 },
129158
);
130159

131-
// Start the Aspire dashboard container
160+
// Start the Aspire dashboard container.
161+
// Deliberately NOT using --rm: if the container exits/crashes instead
162+
// of merely starting slowly, --rm would auto-delete it before
163+
// waitForHealthy's failure-path diagnostics (docker inspect/logs) can
164+
// run, masking the real cause as a generic timeout. removeContainer()
165+
// (called above and in afterAll) already guarantees explicit cleanup.
132166
execSync(
133167
[
134-
'docker run --rm -d',
168+
'docker run -d',
135169
`-p 18888:18888 -p 4317:18889`,
136170
'-e ASPIRE_DASHBOARD_UNSECURED_ALLOW_ANONYMOUS=true',
137171
`--name ${CONTAINER_NAME}`,
138-
'mcr.microsoft.com/dotnet/aspire-dashboard:latest',
172+
DASHBOARD_IMAGE,
139173
].join(' '),
140174
{ stdio: 'inherit' },
141175
);
142176

143-
// Wait for dashboard UI to respond
144-
await waitForHealthy(DASHBOARD_URL, 60_000);
177+
// Wait for dashboard UI to respond. 120s gives generous headroom over
178+
// the dashboard's typical few-second cold start, absorbing real CPU
179+
// contention on CI runners where this suite runs alongside ~10k other
180+
// tests in full parallelism. (The repeated timeouts originally
181+
// investigated in #2123/#2132 turned out to have a separate root
182+
// cause — a transiently broken `:latest` image tag, fixed above by
183+
// pinning to DASHBOARD_IMAGE — but a generous timeout remains
184+
// worthwhile defense-in-depth against genuine contention-induced
185+
// slow starts.)
186+
await waitForHealthy(DASHBOARD_URL, 120_000);
145187

146188
// Initialize OTel gRPC exporters targeting the dashboard
147189
initOTelForAspire();
148190

149191
// Launch Playwright browser
150192
browser = await chromium.launch({ headless: true });
151-
}, 180_000); // 3 min timeout for pull + start
193+
}, 240_000); // 4 min timeout for pull + start (120s health-check budget + overhead)
152194

153195
// ------------------------------------------------------------------
154196
// Teardown: shutdown OTel, close browser, remove container

‎test/helpers/skip-guards.ts‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,14 @@ import { execSync } from 'node:child_process';
1313
*/
1414
export function isDockerAvailable(): boolean {
1515
try {
16-
execSync('docker info', { stdio: 'ignore', timeout: 5000 });
16+
// 5s was too tight: under full-parallelism CI load (~10k tests running
17+
// concurrently), `docker info`'s round-trip to the daemon can occasionally
18+
// exceed 5s purely from CPU contention, causing this check to time out
19+
// and silently SKIP the whole Docker-dependent suite instead of running
20+
// it — a false "pass" that masks real failures (see Aspire dashboard CI
21+
// investigation, #2123/#2132). 20s gives enough headroom to reliably
22+
// detect a genuinely-available daemon under the same load.
23+
execSync('docker info', { stdio: 'ignore', timeout: 20_000 });
1724
return true;
1825
} catch {
1926
return false;

0 commit comments

Comments
 (0)