You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Our TypeScript AppHost integration tests (used across many integrations — Meilisearch, Floci, Minio, RustFs, Ollama, etc.) have been a recurring source of CI flakiness on main, usually surfacing as exit code 7 (or 134) after a ~7 minute hang-dump timeout.
Important: in every case we've investigated, this is not an actual test failure. The test body — including aspire wait <resource> for every resource being waited on — completes successfully. The hang happens afterwards, in our test harness's cleanup step, inside aspire stop. aspire stop never returns, the CI hang-dump watchdog eventually identifies it as the still-alive leaf process, and the whole job gets killed and reported as a failure even though the test itself passed.
We estimate this pattern accounts for roughly half of our main branch test failures over the past two weeks.
Root cause
This is an upstream Aspire CLI bug (a socket leak/race in AuxiliaryBackchannelService during AppHost shutdown), not something specific to our test harness or any one integration. Full analysis, evidence, and code walkthrough are in the upstream issue — see below rather than duplicating it here.
Related work
Improve diagnostics for TypeScript AppHost test failures #1607 adds diagnostics to our TypeScript AppHost test harness (capturing the real Aspire CLI/AppHost log files as CI artifacts, and dumping resource state/logs on aspire wait failure). This is not a complete fix for this issue — it only improves diagnosability for aspire wait timeouts, and does not address the aspire stop hang described here at all, since aspire stop never logs a wait failure in this case.
Filed upstream: microsoft/aspire#20316 — full root-cause analysis, evidence, and discussion of a partial upstream mitigation (#19832, not yet released) that reduces the hang duration but does not fix aspire stop itself.
Expected behavior
TypeScript AppHost tests should not fail (or should fail with a clear, fast, distinguishable error) when the only problem is aspire stop hanging during cleanup after a successful test run.
Additional context
Until the upstream issue is fixed/released, options worth considering here include:
Wrapping aspire stop (and other CLI invocations in our harness) with an explicit timeout so a hung invocation fails fast with a clear message instead of consuming the full CI hang-dump budget.
Tracking whether the underlying test assertions passed separately from cleanup success, so a hung aspire stop doesn't mask a real pass.
Describe the bug
Our TypeScript AppHost integration tests (used across many integrations — Meilisearch, Floci, Minio, RustFs, Ollama, etc.) have been a recurring source of CI flakiness on
main, usually surfacing as exit code 7 (or 134) after a ~7 minute hang-dump timeout.Important: in every case we've investigated, this is not an actual test failure. The test body — including
aspire wait <resource>for every resource being waited on — completes successfully. The hang happens afterwards, in our test harness's cleanup step, insideaspire stop.aspire stopnever returns, the CI hang-dump watchdog eventually identifies it as the still-alive leaf process, and the whole job gets killed and reported as a failure even though the test itself passed.Example failures
We estimate this pattern accounts for roughly half of our
mainbranch test failures over the past two weeks.Root cause
This is an upstream Aspire CLI bug (a socket leak/race in
AuxiliaryBackchannelServiceduring AppHost shutdown), not something specific to our test harness or any one integration. Full analysis, evidence, and code walkthrough are in the upstream issue — see below rather than duplicating it here.Related work
aspire waitfailure). This is not a complete fix for this issue — it only improves diagnosability foraspire waittimeouts, and does not address theaspire stophang described here at all, sinceaspire stopnever logs a wait failure in this case.aspire stopitself.Expected behavior
TypeScript AppHost tests should not fail (or should fail with a clear, fast, distinguishable error) when the only problem is
aspire stophanging during cleanup after a successful test run.Additional context
Until the upstream issue is fixed/released, options worth considering here include:
aspire stop(and other CLI invocations in our harness) with an explicit timeout so a hung invocation fails fast with a clear message instead of consuming the full CI hang-dump budget.aspire stopdoesn't mask a real pass.