fix(runtime,backend): bound device compile fan-out; configurable ptoas timeout#1928
fix(runtime,backend): bound device compile fan-out; configurable ptoas timeout#1928Little-oil wants to merge 2 commits into
Conversation
…austion Each test case compiled its kernels + orchestration with up to 64 threads, each spawning a libgomp-parallel ccec subprocess. Under the precompile pool this fan-out multiplies with the outer concurrency and exhausts process/ thread limits, surfacing as libgomp thread-creation failures and segfaults during compilation. Cap the per-case fan-out at min(16, available_cpus), overridable via PYPTO_DEVICE_COMPILE_WORKERS.
…to 120s Heavy kernels (multi-stage pipelines, looped scatter) can push a single ptoas invocation past the old hardcoded 60s wall-clock under high compile-pool concurrency, surfacing as spurious PartialCodegenError timeouts. Read the per-kernel timeout from PYPTO_PTOAS_TIMEOUT (default 120s) so CI can tune it without affecting local runs.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThis PR adds environment-variable-based configuration for two compilation resource limits: a per-kernel ChangesCompilation Resource Limits
Estimated code review effort: 2 (Simple) | ~10 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces configurable timeouts and worker limits for compilation processes. It adds the PYPTO_PTOAS_TIMEOUT environment variable to customize the ptoas compilation timeout and dynamically calculates the thread count for kernel compilation based on available CPUs, capped by the PYPTO_DEVICE_COMPILE_WORKERS environment variable. The review feedback suggests capping the worker count by the number of compilation units even when the environment variable override is active to ensure resource efficiency.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| if env_val: | ||
| try: | ||
| return max(1, int(env_val)) |
There was a problem hiding this comment.
When PYPTO_DEVICE_COMPILE_WORKERS is set to a value larger than the number of compilation units (n_units), the returned worker count is not capped by n_units. While ThreadPoolExecutor only spawns threads as tasks are submitted, it is cleaner and more resource-efficient to cap the maximum workers at n_units even when the environment variable override is active.
| if env_val: | |
| try: | |
| return max(1, int(env_val)) | |
| if env_val: | |
| try: | |
| return max(1, min(int(env_val), n_units)) |
…eout 15->20min) (#1929) ## What CI workflow half of the `system-tests` (a2a3) compile-stability work, split out from #1926 per review. The paired runtime/backend changes (per-case fan-out cap, configurable ptoas timeout) are in #1928. ## Changes `.github/workflows/ci.yml`, `Test system tests` step: - `--precompile-workers` 128 → 64 — the outer precompile pool multiplied with each case's kernel-compile fan-out to spawn ~100 concurrent ccec subprocesses, exhausting process/thread limits and segfaulting mid-compile. Halving it lets the outer pool compose with the per-case cap in #1928. - `timeout-minutes` 15 → 20 — the lower concurrency reduces compile throughput, so the ~440-case suite crosses the old 15min budget; give it headroom. ## Test Validated on this branch's own CI run — `system-tests` (a2a3) completed green within the 20min budget. ## Note Best paired with #1928: `--precompile-workers=64` is calibrated to compose with that PR's per-case fan-out cap. Landing this alone still lowers concurrency safely, but the two are designed to work together. --------- Co-authored-by: Youhezhen <youhezhen@huawei.com>
1 participant
What
Runtime/backend half of the
system-tests(a2a3) compile-stability work,split out from #1926 per review. The paired CI workflow tuning
(
--precompile-workers, steptimeout-minutes) lives in a separate PR.Under the
system-testsjob, each case's per-kernel compile fan-out spun upmany
ccecsubprocesses, and heavy kernels could be killed by a hardcoded 60sptoas wall-clock under high compile-pool concurrency (spurious timeouts, not
real work limits).
Changes
device_runner.py— cap per-case device compile fan-out atmin(16, available_cpus)(overridable viaPYPTO_DEVICE_COMPILE_WORKERS) soit composes with the outer precompile pool instead of multiplying into
process/thread exhaustion (libgomp thread-creation failures / segfaults). The
affinity probe also tolerates
OSError(restricted containers/sandboxes),falling back to
os.cpu_count().pto_backend.py— make the per-kernel ptoas compile timeout configurablevia
PYPTO_PTOAS_TIMEOUT, default raised 60s → 120s.Test
Python-only; no behavior change on the default (unset-env) local path other
than the higher default timeout. Validated end-to-end alongside the CI tuning
PR (system-tests a2a3 stabilized).