Test flakiness hygiene: scope retries to live-API specs, enable random ordering, tier timeouts

[justin808]

Summary

Flakiness controls are blunt instruments applied suite-wide: every spec (including deterministic unit specs) retries up to 3×, test ordering is fixed (random ordering commented out), and every example gets a 10-minute timeout. Retries on unit specs mask real intermittent bugs; fixed ordering hides inter-spec state leaks; a 10-minute per-example timeout means a hung integration spec blocks CI for 10 minutes before failing.

Evidence (all in spec/spec_helper.rb)

• #L138-L140: config.default_retry_count = ENV["RSPEC_RETRY_COUNT"] || 3 — applies to all specs. Note: ENV[...] returns a String, so RSPEC_RETRY_COUNT=0 would be truthy and "0" — worth normalizing with Integer(...) while here.
• #L124: config.order = :random commented out — suite runs in fixed file order.
• #L155-L158: Timeout.timeout(600) around every example.

Proposed fix

1. Scope retries to live-API specs only (e.g. config.default_retry_count = 1 globally; retry: 3 via metadata on :integration/:slow specs, or retry_count_condition keyed off the tag). Depends on the spec taxonomy introduced in Test suite cannot load without CPLN_ORG — even pure unit specs require live-org configuration #373.
2. Normalize the env override: Integer(ENV.fetch("RSPEC_RETRY_COUNT", "3")).
3. Enable config.order = :random (+ Kernel.srand config.seed), fix whatever ordering issues surface, and keep the seed in CI output for reproduction.
4. Tier the example timeout: e.g. 60s for unit specs, 600s only for :slow/integration specs, via tag-conditional around hook.

Acceptance criteria

• Unit specs run with 0–1 retries; only live-API specs keep multi-retry
• Random ordering enabled and the suite is green across several seeds (document the seeds tried)
• Timeouts tiered by suite type
• CI prints the seed for failed-run reproduction

Priority / size / dependencies

P3. Size: S–M (the random-ordering fallout is the unknown). Blocked by #373 (uses its :integration tagging to scope retries/timeouts).

---

Origin: automated codebase health review, 2026-07-02, against 7c96df9.