Skip to content

test(meshjob): gate tc16 on real job status instead of a fixed 8s sleep - #1459

Merged
dhyansraj merged 1 commit into
mainfrom
fix/1458-tc16-status-gate
Aug 1, 2026
Merged

test(meshjob): gate tc16 on real job status instead of a fixed 8s sleep#1459
dhyansraj merged 1 commit into
mainfrom
fix/1458-tc16-status-gate

Conversation

@dhyansraj

@dhyansraj dhyansraj commented Jul 31, 2026

Copy link
Copy Markdown
Owner

Summary

Both tc16 variants slept a flat 8 seconds waiting for a long-running job, then asserted it was completed. Under --parallel 8 the job outlives that budget and the bystanders read status=working:

assertion 4 (bystander-x should read the actual completed status (no auth gating))
  failed: does not contain "\"completed\""

Failed in 2 of 3 full parallel runs; passes scoped — which is what made it look flaky rather than under-specified.

The Java twin already had this exact gate (uc23/tc16 ~line 141), which is precisely why it never failed. Python and TypeScript were the outliers, so this brings all three into line rather than introducing a new pattern.

Two things beyond "wait longer"

It polls via the consumer, not a bystander. The bystanders reading job status without auth is the assertion under test — polling through one would pre-empt the very thing being proven. The comment says so, so a future editor doesn't "simplify" it.

It distinguishes a broken job from a slow one. Previously, a job that genuinely failed produced the identical message as one that was merely slow — does not contain "completed" — with the case timeout cutting off any diagnostics before they landed. The gate now exits immediately on failed/cancelled with the response body, dumps the last status and the provider log tail on timeout, and the case timeout moves 120 → 180 so that output can actually surface.

That second point is why this isn't just a flake fix: the old failure mode was uninformative by construction.

Verification

Full suite at --parallel 8: 559 passed, 0 failed, 9 skipped.

[PASS] uc21_meshjob/tc16_third_party_can_read_status        (38.9s)
[PASS] uc22_meshjob_ts/tc16_third_party_can_read_status_ts  (50.4s)
[PASS] uc23_meshjob_java/tc16_third_party_can_read_status_java (120.0s)
[PASS] uc20_tutorial/tc11_day10_bonus_streaming             (303.4s)

Assertions 4 and 5 are unchanged — they are the point of the test.

Follow-up, not in this PR

46 steps across the meshjob suites use handler: wait, several standing in for real readiness conditions. tsuite 0.7.0 ships a probe handler (interval, timeout, until, success_threshold, on_failure) that the suite uses nowhere today. Worth migrating as one reviewed change — on_failure alone would have short-circuited two separate investigations today by attaching meshctl logs <agent> | tail -50 to the failure output.

Boundary for that work: probe belongs on infrastructure readiness — agent serving HTTP, registry answering, job reaching terminal state — and never on dependency or provider resolution. The settling window (#1193) exists so integration tests don't wait on DI, and a probe waiting for the mesh to converge would have hidden #1456 entirely. That test failing is what surfaced a real runtime race.

Closes #1458

Summary by CodeRabbit

  • Tests
    • Improved mesh job status test reliability by polling for completion instead of relying on a fixed wait.
    • Added clearer handling for failed, cancelled, and timed-out jobs.
    • Enhanced timeout diagnostics with captured status details and provider logs.

Closes #1458

Both tc16 variants slept a flat 8 seconds and then asserted the job was
completed. Under --parallel 8 the long-running job outlives that budget, so the
bystanders read status=working and assertion 4 fails. Observed in 2 of 3 full
parallel runs; passes scoped, which is what made it look flaky rather than
under-specified.

The Java twin already had this gate, which is why it never failed. Python and
TypeScript were the outliers, so this brings all three into line rather than
introducing a new pattern.

Polls via the CONSUMER, not a bystander: the bystanders reading status without
auth is the assertion under test, so polling through one would pre-empt it.

Also distinguishes a broken job from a slow one. Previously a genuinely failed
job produced the identical message as a slow one -- does not contain
"completed" -- with the case timeout cutting off any diagnostics. The gate now
exits immediately on failed/cancelled with the response body, dumps the last
status and provider log tail on timeout, and the case timeout moves 120 -> 180
so that output can land.

Full suite at --parallel 8: 559 passed, 0 failed, 9 skipped.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hsym5TX4sFTLUv9LrxgSjq
@coderabbitai

coderabbitai Bot commented Jul 31, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 1cc981d6-310c-4bea-b61c-ff43fd9239ef

📥 Commits

Reviewing files that changed from the base of the PR and between ae4c19d and 80c18d9.

📒 Files selected for processing (2)
  • tests/integration/suites/uc21_meshjob/tc16_third_party_can_read_status/test.yaml
  • tests/integration/suites/uc22_meshjob_ts/tc16_third_party_can_read_status_ts/test.yaml

📝 Walkthrough

Walkthrough

The Python and TypeScript MeshJob status tests replace fixed 8-second waits with consumer-side polling gates. They detect completion, failed or cancelled jobs, and timeout diagnostics. The Python test timeout increases from 120 to 180 seconds.

Changes

MeshJob status polling

Layer / File(s) Summary
Consumer-side completion gates
tests/integration/suites/uc21_meshjob/tc16_third_party_can_read_status/test.yaml, tests/integration/suites/uc22_meshjob_ts/tc16_third_party_can_read_status_ts/test.yaml
Both tests poll __mesh_job_status until completion. They fail immediately for failed or cancelled states and report status and provider logs on timeout. The Python test timeout increases to 180 seconds.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes replacing the fixed 8-second wait with real job-status polling in the meshjob tc16 tests.
Linked Issues check ✅ Passed The changes satisfy issue #1458 by adding consumer-side status polling, terminal-state handling, diagnostics, and the 120-to-180-second timeout increase.
Out of Scope Changes check ✅ Passed The changes are limited to the two Python and TypeScript tc16 tests and implement the linked issue objectives without unrelated scope.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/1458-tc16-status-gate

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@dhyansraj
dhyansraj merged commit 25c6322 into main Aug 1, 2026
17 checks passed
@dhyansraj
dhyansraj deleted the fix/1458-tc16-status-gate branch August 1, 2026 00:44
dhyansraj added a commit that referenced this pull request Aug 5, 2026
…3 runtimes (#1485)

## Summary

Health-check withdrawal shipped in Python (#1473), Java (#1475) and
TypeScript (#1481) with **no integration coverage in any runtime**.
Everything asserted was either a unit test with a stubbed publish or a
Rust test that a command was enqueued. Nothing exercised the chain the
feature exists for:

```
verdict → heartbeat suppression → registry withdrawal → consumer failover → recovery via 410 Gone
```

`uc41` adds six cases — withdrawal-through-recovery and a throwing-check
negative, per runtime.

**Six cases, not nine.** Withdrawal and recovery are one continuous
timeline with separately-asserted phases, because the load-bearing claim
is the **same pid** across both the outage and the restore — a test that
redoes the withdrawal from scratch cannot make it. Java reads its pid
from inside the process (`ProcessHandle.current().pid()`): meshctl's pid
file names the `mvn spring-boot:run` wrapper, so a restarted JVM under a
surviving wrapper would look identical from outside.

**The negative is the case that matters.** A suite that only tests
withdrawal passes just as happily against a runtime that withdraws on
*any* non-healthy verdict — which is the bug, since `degraded` must keep
heartbeating. It watches the invocation log to prove the check kept
running, rather than inferring that from the absence of a withdrawal.

## Review notes

**Nothing sleeps through a state transition** (the #1459 lesson). Phases
gate on the registry's actual state and on the consumer naming the other
provider. `/livez` is curled once per second across the whole outage
with `LIVEZ_FAILURES: 0` asserted, rather than sampled at two convenient
moments — "withdrawn, not dead" gets continuous coverage.

The registry runs at `HEALTH_CHECK_INTERVAL=2` /
`DEFAULT_TIMEOUT_THRESHOLD=5`, turning a ~30s withdrawal into ~7s. The
routine greps the registry's own startup line and **exits 1 if the
tuning did not take**, so a renamed env var fails loudly instead of
degrading into confusing 60s poll timeouts.

**Every case proven red against a neutered runtime**, with the neuters
applied to the artifacts rather than the test files:

| neuter | result |
|---|---|
| `fail` branch returns healthy (≡ pre-#1472) | 3 withdrawal cases red |
| `throw` branch returns unhealthy (≡ a throw withdraws) | 3 negatives
red, `WITHDRAWN at ~7s` |
| `throw` returns healthy and tracing stops (≡ refresh loop died) | 3
negatives red |

The third is the important one: the agent is **not** withdrawn — the
outcome a naive negative test scores as a pass — and all three still
fail.

Two things found while building this, worth recording:

- The `tsuite-mesh:local` image on the cluster predated all three
health-check merges, so nothing had ever been validated against a build
containing the feature. Rebuilt before any run here.
- Under `--parallel 6`, the assumption that provider A wins the initial
resolution only holds once *both* providers exist — B sometimes
registered first and the test silently lost the property it is built on.
Fixed with explicit ordering gates; a general trap for any
multi-provider case added to this suite later.

Scoped to the mesh machinery only. The scaffold's emitted vendor probe
is **not** covered — it hardcodes its API host with no base-URL
override, so it cannot be pointed at a stub without a template change.
Deferred to #1483, noted in `routines.yaml`.

TypeScript's `/health` is FastMCP's built-in and carries no verdict
(#1478), so tc03/tc04 read it from the `[mesh-health]` log line, with a
header note marking the assertion to move once #1478 lands.

Closes #1480

## Test plan

- [x] `tsuite run --suite-path tests/src-tests` — 12/12, 376s,
`tsuite-mesh:local` rebuilt from `fa6d2e7a2`
- [x] `tsuite run --suite-path tests/integration --uc
uc41_health_check_withdrawal --parallel 6` — **6 passed, 0 failed,
136.5s**
- [x] Full integration suite green across four consecutive runs; 559 →
565 tests
- [x] Each case verified red under three separate runtime neuters, all
reverted byte-identically


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added Python, TypeScript, and Java health-check consumer and provider
test applications.
- Added healthy, unhealthy, and error health states with provider
identity reporting.
- Added automatic provider failover and same-process recovery
validation.
- Added consumer responses identifying the active provider or reporting
unavailable dependencies.

- **Tests**
- Added integration coverage for withdrawal, failover, recovery, and
degraded health behavior across all three languages.
- Added validation that liveness remains available during withdrawal and
that throwing checks degrade without removing providers.
- Added registry readiness, timing, diagnostics, and cleanup validation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

meshjob tc16: fixed 8s sleep races job completion under parallel load (Java twin already had the gate)

1 participant