Skip to content

main: three tests fail Test (ubuntu-latest) on every PR (stale providers golden, half-installed context-window fixtures) #7154

Closed

Description

@luochen211

What happened

Test (ubuntu-latest) fails on every open PR with the same three tests, independent of what the PR changes:

route::export::tests::golden_providers_export_matches_registry                      (codewhale-config)
commands::config_policy_status_tests::tests::status_report_localizes_the_complete_japanese_surface  (codewhale-tui)
tui::app::tests::test_update_model_compaction_budget                                (codewhale-tui)

Seen on #7149, #7150 (run 41016661593) and #7151 (run 41019507563), all with PR base 6dd9af54c. Nothing after that base on current main (c96f4edc5) touches the files involved, so the failures should still reproduce there.

The main compile error from #7148 is fixed by 7198de9 (Lint passes on these PRs again). These three test failures are separate from it.

1. golden_providers_export_matches_registry: golden file out of date

panicked at crates/config/src/route/export.rs:311:9:
assertion `left == right` failed: update providers-export.golden.json from ProvidersExport::from_registry

0e3677b feat(catalog): pin kimi-k3.1-pro, MiniMax-M3.1-Flash and gpt-6.1-terra in the offline seed added models to the bundled catalog but did not update crates/config/src/route/providers-export.golden.json. I compared the left and right values in the log route by route. The only differences are models that the registry lists and the golden file does not:

route missing from golden
minimax, minimax-anthropic MiniMax-M3.1-Flash
moonshot kimi-k3.1-pro, kimi-k3.1-pro-highspeed
openai gpt-6.1-terra

That commit verified with cargo test -p codewhale-config --lib --locked catalog, and that filter does not select route::export::tests.

Fix: regenerate with the existing helper,

WRITE_GOLDEN=1 scripts/dev-cargo.sh test -p codewhale-config --lib --locked \
  write_golden_providers_export_when_requested -- --ignored

2. status_report_localizes_the_complete_japanese_surface: fixture sets only the source

panicked at crates/tui/src/commands/config_policy_status_tests.rs:538:13:
missing StatusContextSourceProviderReported: ...

da026a2 fix(tui): carry the resolved context window with its source ... made effective_context_window_source() read the source from active_context_window when one is installed. The fixture at config_policy_status_tests.rs:507 sets only

app.active_context_window_source = crate::route_runtime::ContextWindowSource::ProviderReported;

so /status shows a different provenance label and the Japanese provider-reported label is missing.

5e9cefd already fixed three fixtures for this reason, including status_report_names_context_window_source_and_override_key in the same file, but missed this one. Installing the window next to its source, as 5e9cefd did, should fix it:

app.active_context_window = Some(crate::route_runtime::ContextWindowResolution {
    tokens: /* any window */,
    source: crate::route_runtime::ContextWindowSource::ProviderReported,
});

3. test_update_model_compaction_budget: probably the same change (not verified locally)

panicked at crates/tui/src/tui/app/tests.rs:4935:5:
a large-context model (61440) should budget more than an unknown model (62976)

The test at app/tests.rs:4910 clears active_route_limits and active_context_window_override, then switches app.model between deepseek-v4-pro and unknown-test-model. It never clears active_context_window. After da026a2, effective_budget_route() prefers the installed window. If App::new leaves one installed, both models are budgeted against the same window and the per-model comparison no longer tests anything. I have not confirmed this by running the test. Clearing app.active_context_window in the fixture (or installing a window per model) is the likely fix.

Steps to reproduce

git checkout origin/main
scripts/dev-test.sh config golden_providers_export_matches_registry
scripts/dev-test.sh tui status_report_localizes_the_complete_japanese_surface
scripts/dev-test.sh tui test_update_model_compaction_budget

Why this reached PRs

ci.yml does not run on pushes to main. da026a2 also carries [skip ci]. Because of both, the first runs of these tests were on contributor PRs, where they look like regressions caused by the PR.

Activity

  1. added
    needs-triageNew external report awaiting maintainer triage; repro, logs and version output help
    on Oct 30, 2026
  2. moved this from Backlog to In progress in CodeWhaleon Oct 30, 2026
  3. luochen211 commented on Oct 31, 2026

    @luochen211
    ContributorAuthor

    Fixed on main by 84acb83.

  4. moved this from In progress to Done in CodeWhaleon Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-triageNew external report awaiting maintainer triage; repro, logs and version output help

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions