ci(mlx): test stream race — dump stream ids + eval-before-gather #14
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: MLX PR check (fork) | |
| # Fait tourner les tests MLX pertinents pour la PR #20685 sur un runner | |
| # macOS Apple Silicon GRATUIT (macos-14 = M1, arm64, illimité sur dépôt public). | |
| # | |
| # Pourquoi ce fichier et pas .github/workflows/mlx.yml : le workflow officiel | |
| # demande `runner: macos-14-xlarge`, un "larger runner" réservé à l'org pytorch. | |
| # Sur un fork perso, ce label ne résout pas et le job reste en attente | |
| # indéfiniment. Ici on cible `macos-14`, le runner standard gratuit. | |
| # | |
| # INSTALLATION : | |
| # 1. Sur ton fork, onglet Actions -> activer les workflows (désactivés par | |
| # défaut sur les forks). | |
| # 2. Vérifier que le fork est PUBLIC (le gratuit illimité ne vaut que pour | |
| # les dépôts publics). | |
| # 3. Placer ce fichier en .github/workflows/mlx-fork-check.yml sur la branche | |
| # ci-mlx-check (jamais sur la branche PR), commit + push. | |
| # 4. Actions -> "MLX PR check (fork)" -> Run workflow. | |
| # | |
| # NE PAS inclure ce fichier dans la PR upstream : il est là pour ta validation | |
| # locale uniquement. Le supprimer avant le push final vers pytorch/executorch. | |
| on: | |
| workflow_dispatch: | |
| push: | |
| branches: | |
| - ci-mlx-check | |
| paths-ignore: | |
| - mlx_gather_mm_repro.py | |
| - .github/workflows/mlx-only-repro.yml | |
| concurrency: | |
| group: ${{ github.workflow }}-${{ github.ref }} | |
| # false: a repro-only push must not cancel an in-flight ExecuTorch build | |
| cancel-in-progress: false | |
| permissions: | |
| contents: read | |
| jobs: | |
| test-mlx-moe: | |
| # Runner GitHub-hosted standard : M1, 3 vCPU, 7 Go RAM, 14 Go disque. | |
| # Gratuit et illimité sur les dépôts publics. | |
| runs-on: macos-14 | |
| timeout-minutes: 300 | |
| steps: | |
| - name: Checkout | |
| uses: actions/checkout@v4 | |
| with: | |
| submodules: recursive | |
| - name: Setup Python 3.12 | |
| uses: actions/setup-python@v5 | |
| with: | |
| python-version: "3.12" | |
| - name: Diagnostics machine (avant build) | |
| run: | | |
| set -eux | |
| uname -m # doit afficher arm64 | |
| sysctl -n machdep.cpu.brand_string | |
| sysctl -n hw.ncpu | |
| df -h / | |
| xcodebuild -version || true | |
| - name: Install ExecuTorch | |
| run: | | |
| set -eux | |
| python -m pip install --upgrade pip | |
| python install_executorch.py | |
| - name: Configure (preset mlx-release) | |
| run: | | |
| set -eux | |
| cmake --preset mlx-release \ | |
| -DEXECUTORCH_BUILD_TESTS=ON \ | |
| -DEXECUTORCH_MLX_ENABLE_SANITIZERS=OFF | |
| - name: Build op_test_runner | |
| id: build | |
| run: | | |
| set -eux | |
| # -j2 volontairement bas : 7 Go de RAM seulement, l'édition de liens | |
| # d'ExecuTorch est gourmande. Monter à $(sysctl -n hw.ncpu) si OK. | |
| cmake --build cmake-out --target op_test_runner -j2 | |
| df -h / | |
| # What did we actually link into op_test_runner? (not the git tag alone) | |
| - name: Inspect MLX linked into op_test_runner | |
| if: steps.build.outcome == 'success' | |
| continue-on-error: true | |
| run: | | |
| set -eux | |
| RUNNER=$(find cmake-out -type f -name op_test_runner | head -n 1) | |
| echo "op_test_runner=$RUNNER" | |
| ls -la "$RUNNER" | |
| otool -L "$RUNNER" | grep -i mlx || true | |
| strings "$RUNNER" | grep -iE 'mlx.*0\.[0-9]|MLX_VERSION|mlx version' | head -n 40 || true | |
| # Submodule tip vs tag | |
| git -C backends/mlx/third-party/mlx rev-parse HEAD | |
| git -C backends/mlx/third-party/mlx describe --tags --always || true | |
| ls backends/mlx/third-party/mlx/build 2>/dev/null | head || true | |
| # A) Control: PyPI wheel (fast). Already known mostly-green; keep for A/B. | |
| - name: Repro gather_mm — PyPI mlx wheel | |
| if: always() | |
| continue-on-error: true | |
| env: | |
| ET_MLX_REPRO_SOURCE: pypi-wheel | |
| run: | | |
| set -eux | |
| pip install -q 'mlx==0.32.0' numpy | |
| python -c "import mlx.core as mx; print('mlx', mx.__version__, mx.__file__)" | |
| python mlx_gather_mm_repro.py | |
| # B) Decisive: Python bindings built from the SAME submodule tree the | |
| # C++ runner links. Same tag as PyPI can still differ (Metal SDK / flags). | |
| - name: Repro gather_mm — submodule-built mlx | |
| if: steps.build.outcome == 'success' | |
| continue-on-error: true | |
| env: | |
| ET_MLX_REPRO_SOURCE: submodule-pip-install | |
| run: | | |
| set -eux | |
| pip install -q numpy cmake ninja | |
| # Reuse / rebuild Python extension from the checked-out submodule. | |
| # Prefer in-tree build dir if present from the ET cmake mlx build. | |
| cd backends/mlx/third-party/mlx | |
| echo "submodule HEAD=$(git rev-parse HEAD)" | |
| pip install --no-build-isolation -v . | |
| python -c "import mlx.core as mx; print('mlx', mx.__version__, mx.__file__)" | |
| cd "$GITHUB_WORKSPACE" | |
| python mlx_gather_mm_repro.py | |
| # Instrumentation dump: shapes/strides/indices + stream ids + Y compare. | |
| # ET_MLX_DUMP_GATHER=1 enables stderr dumps in exec_gather_mm/qmm. | |
| - name: Dump gather_mm (baseline — no pre-eval) | |
| if: steps.build.outcome == 'success' | |
| continue-on-error: true | |
| env: | |
| ET_MLX_DUMP_GATHER: "1" | |
| run: | | |
| set -eux | |
| python -m executorch.backends.mlx.test.run_all_tests \ | |
| gather_mm_e4_i64_o128 \ | |
| gather_mm_e4_i64_o128_sorted \ | |
| gather_mm_e4_i64_o128_sorted_literal \ | |
| gather_qmm_e4_i64_o128_g32_sorted \ | |
| -v --clean-after | |
| # Decisive stream-race test: eval(A,B,idx) immediately before gather_mm. | |
| # If max_diff≈33 disappears, missing sync — not a compute bug. | |
| - name: Dump gather_mm (EVAL inputs before gather) | |
| if: steps.build.outcome == 'success' | |
| continue-on-error: true | |
| env: | |
| ET_MLX_DUMP_GATHER: "1" | |
| ET_MLX_EVAL_BEFORE_GATHER: "1" | |
| run: | | |
| set -eux | |
| python -m executorch.backends.mlx.test.run_all_tests \ | |
| gather_mm_e4_i64_o128 \ | |
| gather_mm_e4_i64_o128_sorted \ | |
| gather_mm_e4_i64_o128_sorted_literal \ | |
| -v --clean-after | |
| # A/B: Vid path (ItemIntNode) vs literal flag — isolates resolve_int chain. | |
| - name: Op tests — gather_mm sorted Vid vs literal | |
| if: steps.build.outcome == 'success' | |
| continue-on-error: true | |
| run: | | |
| set -eux | |
| python -m executorch.backends.mlx.test.run_all_tests \ | |
| gather_mm_e4_i64_o128_sorted \ | |
| gather_mm_e4_i64_o128_sorted_literal \ | |
| -v --clean-after | |
| # Le cœur de la validation : uniquement les ops touchées par la PR. | |
| # run_all_tests.py accepte des noms de tests en positionnel, ce qui évite | |
| # de faire tourner la suite complète (trop lourde pour le runner gratuit). | |
| # continue-on-error: recover MoE + pytest results even if gather is red. | |
| - name: Op tests — gather_mm / gather_qmm (chemin sorted_indices Optional) | |
| if: steps.build.outcome == 'success' | |
| continue-on-error: true | |
| run: | | |
| set -eux | |
| python -m executorch.backends.mlx.test.run_all_tests \ | |
| gather_mm gather_qmm -v --clean-after | |
| - name: Op tests — moe_gather_inputs / moe_scatter_outputs (feature #20554) | |
| if: steps.build.outcome == 'success' | |
| continue-on-error: true | |
| run: | | |
| set -eux | |
| python -m executorch.backends.mlx.test.run_all_tests \ | |
| moe_gather_inputs moe_scatter_outputs -v --clean-after | |
| # Utile pour recalibrer expected_node_counts si un assert échoue : | |
| # la sortie -v liste les nœuds réellement émis. | |
| - name: Liste des configs de test disponibles (debug) | |
| if: always() | |
| run: | | |
| python -m executorch.backends.mlx.test.run_all_tests --list || true | |
| - name: Tests Python du backend (non-runtime) | |
| if: steps.build.outcome == 'success' | |
| continue-on-error: true | |
| run: | | |
| set -eux | |
| python -m pytest \ | |
| backends/mlx/test/test_passes.py \ | |
| backends/mlx/test/test_partitioner.py \ | |
| backends/mlx/test/test_serialization_dedup.py \ | |
| -v | |
| - name: Diagnostics machine (après) | |
| if: always() | |
| run: df -h / |