Skip to content

Commit 86e276f

Browse files
SS-JIAclaude
andcommitted
CI: identify hanging tests in macOS unittest job
Summary: The macOS unittest job has been hitting its timeout for several runs in a row with the progress bar frozen partway through pytest. Add `pytest-timeout` so a stuck test fails with a nodeid and per-thread stack trace, and set `faulthandler_timeout=180` so every xdist worker dumps its threads every 3 minutes while tests are still running -- this surfaces the hung test as it develops, not just at termination. Diagnostic runs identified the hang fingerprint: tests stuck inside `torch._inductor.package._package.__call__` (line 736), the AOTI-packaged `.so` invocation. The same stack was observed for tests in `examples/models/llama3_2_vision/{preprocess,vision_encoder}` and the local M1 reruns of these tests against the CI-pinned torch wheel all pass quickly, so the hang is CI-environment-specific. Experiment: drop the macOS pytest run to `-n 1` to test the hypothesis that the hang is caused by xdist-worker contention (parallel clang/ld during AOTI compile, dlopen lock on darwin, libomp oversubscription) rather than a true deadlock. Bump the job timeout to 90 minutes so a serial run has room to finish, and keep the per-test `--timeout=1500`. If the previously-hung AOTI tests now pass under `-n 1`, the right permanent fix is splitting macOS into a serial AOTI lane plus a parallel everything-else lane (or simply dropping the parallelism cap). Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 4de16d0 commit 86e276f

3 files changed

Lines changed: 11 additions & 2 deletions

File tree

.ci/docker/requirements-ci.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ zstd==1.5.5.1
1111
pandas>=2.2.2; python_version >= '3.10'
1212
pytest==7.2.0
1313
pytest-cov==4.1.0
14+
pytest-timeout==2.2.0
1415
expecttest==0.1.6
1516
hypothesis==6.84.2
1617
parameterized==0.9.0

.ci/scripts/unittest-macos-cmake.sh

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,15 @@ set -eux
1212
export TORCHINDUCTOR_CACHE_DIR="$(mktemp -d "${RUNNER_TEMP:-/tmp}/torchinductor_cache_XXXXXX")"
1313
trap 'rm -rf "${TORCHINDUCTOR_CACHE_DIR}"' EXIT
1414

15-
# Run pytest with coverage
16-
${CONDA_RUN} pytest -n auto --cov=./ --cov-report=xml
15+
# EXPERIMENT: drop xdist (`-n 1`) on macOS to test whether AOTI hangs are
16+
# caused by parallel-worker contention (clang/ld, dlopen lock, libomp
17+
# oversubscription) rather than a true deadlock. AOTI skips removed so we
18+
# can observe whether the previously-hung tests now pass serially.
19+
# --timeout surfaces hung tests with a thread dump and faulthandler_timeout
20+
# periodically dumps every worker's threads while tests are still running.
21+
${CONDA_RUN} pytest -n 1 --cov=./ --cov-report=xml \
22+
--timeout=1500 --timeout-method=thread \
23+
-o faulthandler_timeout=180
1724
# Run gtest
1825
LLVM_PROFDATA="xcrun llvm-profdata" LLVM_COV="xcrun llvm-cov" \
1926
${CONDA_RUN} test/run_oss_cpp_tests.sh

.github/workflows/_unittest.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,7 @@ jobs:
4949
python-version: '3.11'
5050
submodules: 'recursive'
5151
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
52+
timeout: 90
5253
script: |
5354
set -eux
5455
# This is needed to get the prebuilt PyTorch wheel from S3

0 commit comments

Comments
 (0)