Commit 86e276f
CI: identify hanging tests in macOS unittest job
Summary:
The macOS unittest job has been hitting its timeout for several runs in a
row with the progress bar frozen partway through pytest. Add
`pytest-timeout` so a stuck test fails with a nodeid and per-thread stack
trace, and set `faulthandler_timeout=180` so every xdist worker dumps its
threads every 3 minutes while tests are still running -- this surfaces the
hung test as it develops, not just at termination.
Diagnostic runs identified the hang fingerprint: tests stuck inside
`torch._inductor.package._package.__call__` (line 736), the AOTI-packaged
`.so` invocation. The same stack was observed for tests in
`examples/models/llama3_2_vision/{preprocess,vision_encoder}` and the
local M1 reruns of these tests against the CI-pinned torch wheel all
pass quickly, so the hang is CI-environment-specific.
Experiment: drop the macOS pytest run to `-n 1` to test the hypothesis
that the hang is caused by xdist-worker contention (parallel clang/ld
during AOTI compile, dlopen lock on darwin, libomp oversubscription)
rather than a true deadlock. Bump the job timeout to 90 minutes so a
serial run has room to finish, and keep the per-test `--timeout=1500`.
If the previously-hung AOTI tests now pass under `-n 1`, the right
permanent fix is splitting macOS into a serial AOTI lane plus a parallel
everything-else lane (or simply dropping the parallelism cap).
Co-Authored-By: Claude <noreply@anthropic.com>1 parent 4de16d0 commit 86e276f
3 files changed
Lines changed: 11 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
| 14 | + | |
14 | 15 | | |
15 | 16 | | |
16 | 17 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
15 | | - | |
16 | | - | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
17 | 24 | | |
18 | 25 | | |
19 | 26 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
49 | 49 | | |
50 | 50 | | |
51 | 51 | | |
| 52 | + | |
52 | 53 | | |
53 | 54 | | |
54 | 55 | | |
| |||
0 commit comments