feat(ci): tune system-tests compile concurrency (workers 128->64, timeout 15->20min)#1929
Conversation
The outer precompile pool (128) multiplied with each case's kernel-compile fan-out to spawn ~100 concurrent ccec subprocesses, exhausting process/ thread limits and segfaulting mid-compile. Halve the outer pool so it composes with the device_runner per-case cap instead of overwhelming the runner.
Lowering precompile-workers to 64 to avoid ccec compile-thread exhaustion reduces compile throughput; the ~440-case suite (13.5min at 128 workers) now runs to 99% but crosses the old 15min step budget. Give it headroom.
|
Note Gemini is unable to generate a review for this pull request due to the file types involved not being currently supported. |
|
Warning Review limit reached
Next review available in: 59 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Little-oil
changed the title
2 participants
What
CI workflow half of the
system-tests(a2a3) compile-stability work, split outfrom #1926 per review. The paired runtime/backend changes (per-case fan-out cap,
configurable ptoas timeout) are in #1928.
Changes
.github/workflows/ci.yml,Test system testsstep:--precompile-workers128 → 64 — the outer precompile pool multiplied witheach case's kernel-compile fan-out to spawn ~100 concurrent ccec subprocesses,
exhausting process/thread limits and segfaulting mid-compile. Halving it lets
the outer pool compose with the per-case cap in fix(runtime,backend): bound device compile fan-out; configurable ptoas timeout #1928.
timeout-minutes15 → 20 — the lower concurrency reduces compile throughput,so the ~440-case suite crosses the old 15min budget; give it headroom.
Test
Validated on this branch's own CI run —
system-tests(a2a3) completed greenwithin the 20min budget.
Note
Best paired with #1928:
--precompile-workers=64is calibrated to compose withthat PR's per-case fan-out cap. Landing this alone still lowers concurrency
safely, but the two are designed to work together.