Skip to content

Impove TypeScript test failure output. #1608

Description

@afscrome

Describe the bug

Our TypeScript AppHost integration tests (used across many integrations — Meilisearch, Floci, Minio, RustFs, Ollama, etc.) have been a recurring source of CI flakiness on main, usually surfacing as exit code 7 (or 134) after a ~7 minute hang-dump timeout.

Important: in every case we've investigated, this is not an actual test failure. The test body — including aspire wait <resource> for every resource being waited on — completes successfully. The hang happens afterwards, in our test harness's cleanup step, inside aspire stop. aspire stop never returns, the CI hang-dump watchdog eventually identifies it as the still-alive leaf process, and the whole job gets killed and reported as a failure even though the test itself passed.

Example failures

We estimate this pattern accounts for roughly half of our main branch test failures over the past two weeks.

Root cause

This is an upstream Aspire CLI bug (a socket leak/race in AuxiliaryBackchannelService during AppHost shutdown), not something specific to our test harness or any one integration. Full analysis, evidence, and code walkthrough are in the upstream issue — see below rather than duplicating it here.

Related work

  • Improve diagnostics for TypeScript AppHost test failures #1607 adds diagnostics to our TypeScript AppHost test harness (capturing the real Aspire CLI/AppHost log files as CI artifacts, and dumping resource state/logs on aspire wait failure). This is not a complete fix for this issue — it only improves diagnosability for aspire wait timeouts, and does not address the aspire stop hang described here at all, since aspire stop never logs a wait failure in this case.
  • Filed upstream: microsoft/aspire#20316 — full root-cause analysis, evidence, and discussion of a partial upstream mitigation (#19832, not yet released) that reduces the hang duration but does not fix aspire stop itself.

Expected behavior

TypeScript AppHost tests should not fail (or should fail with a clear, fast, distinguishable error) when the only problem is aspire stop hanging during cleanup after a successful test run.

Additional context

Until the upstream issue is fixed/released, options worth considering here include:

  • Wrapping aspire stop (and other CLI invocations in our harness) with an explicit timeout so a hung invocation fails fast with a clear message instead of consuming the full CI hang-dump budget.
  • Tracking whether the underlying test assertions passed separately from cleanup success, so a hung aspire stop doesn't mask a real pass.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions