ci(mlx): fork-check v6 — resolve paths via module, tee harnais, A/B p… #23
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: MLX PR check (fork) | |
| # CAMPAGNE v6 — plus aucune résolution de chemin à la main. | |
| # | |
| # Post-mortem v5 : `python -m executorch...` résout le module INSTALLÉ | |
| # (site-packages). get_test_dir() est relatif à __file__ (sans fallback) | |
| # -> generate écrit dans site-packages ; mon `D=backends/mlx/test/...` | |
| # pointait dans le vide, le step direct est mort au `ls`, avant le runner. | |
| # find_executorch_root(), lui, retombe sur le cwd (le dépôt) -> le harnais | |
| # a bien exécuté le binaire sondé de cmake-out (probe-in-binary: 3). | |
| # Les lignes [GMM-PROBE] sont donc peut-être déjà dans les blobs | |
| # "C++ binary output:" du run précédent — à vérifier avant re-run. | |
| # | |
| # Changements v6 : | |
| # 1. D et R sont demandés AU MODULE lui-même (même résolution que le | |
| # harnais) : plus de chemins devinés. | |
| # 2. Identité du binaire prouvée : shasum(R) vs shasum(cmake-out) + | |
| # strings(R). | |
| # 3. A/B perturbation : même binaire, mêmes entrées, PROBE=1 vs PROBE=0, | |
| # cmp des sorties. Si la sonde change le RÉSULTAT, le bug est | |
| # sensible à l'ordre d'exécution — indice majeur état/provenance. | |
| # 4. Harnais tee-é et greppé DANS le step : le verdict ne peut plus | |
| # être enfoui. | |
| # 5. Échelle réduite à L0 + L3-exact (l'image statique est innocentée | |
| # par le L3 de v5 ; on les garde en contrôle de régression). | |
| # | |
| # Verdict ([GMM-PROBE] du step direct ou du harnais) : | |
| # wrapped >> 0 && natcopies ~ 0 -> PROVENANCE -> fix ET dans la PR. | |
| # wrapped >> 0 && natcopies >> 0 -> état de processus créé AVANT l'appel | |
| # (l'image statique étant innocentée) -> prochaine sonde : raw call au | |
| # démarrage du runner vs après load/execute. | |
| # EXCEPTION: ... -> la sonde explique. | |
| # | |
| # Prérequis : probe v1+v2 appliqués ; repro_gather_mm.cpp à la racine. | |
| on: | |
| workflow_dispatch: | |
| push: | |
| branches: | |
| - ci-mlx-check | |
| concurrency: | |
| group: ${{ github.workflow }}-${{ github.ref }} | |
| cancel-in-progress: true | |
| permissions: | |
| contents: read | |
| jobs: | |
| provenance-vs-link: | |
| runs-on: macos-14 | |
| timeout-minutes: 300 | |
| steps: | |
| - uses: actions/checkout@v4 | |
| with: | |
| submodules: recursive | |
| - uses: actions/setup-python@v5 | |
| with: | |
| python-version: "3.12" | |
| - name: Preuve — commit, sonde v2 présente | |
| run: | | |
| set -eux | |
| git log --oneline -3 | |
| grep -c "GMM-PROBE" backends/mlx/runtime/MLXInterpreter.h | |
| - name: Install ExecuTorch | |
| run: | | |
| set -eux | |
| python -m pip install --upgrade pip | |
| python install_executorch.py | |
| - name: Configure + build op_test_runner | |
| id: build | |
| run: | | |
| set -eux | |
| cmake --preset mlx-release \ | |
| -DEXECUTORCH_BUILD_TESTS=ON \ | |
| -DEXECUTORCH_MLX_ENABLE_SANITIZERS=OFF | |
| cmake --build cmake-out --target op_test_runner -j2 | |
| df -h / | |
| # ------------------------------------------------------------------ | |
| # SONDE P directe — chemins résolus par le module, binaire prouvé, | |
| # flux bruts, A/B perturbation. | |
| # ------------------------------------------------------------------ | |
| - name: SONDE P directe — verdict | |
| if: steps.build.outcome == 'success' | |
| continue-on-error: true | |
| run: | | |
| set -eux | |
| CFG=gather_mm_e4_i64_o128_sorted | |
| R=$(python -c "import executorch.backends.mlx.test.test_utils as t; print(t.find_op_test_runner())") | |
| D=$(python -c "import executorch.backends.mlx.test.test_utils as t, pathlib; print(pathlib.Path(t.__file__).parent/'op_tests'/'$CFG')") | |
| echo "runner=$R" | |
| echo "testdir=$D" | |
| shasum "$R" "$(find cmake-out -name op_test_runner -type f | head -1)" | |
| echo "probe-in-binary: $(strings "$R" | grep -c 'GMM-PROBE')" | |
| python -m executorch.backends.mlx.test.test_utils "$CFG" generate | |
| ls -la "$D" | |
| set +e | |
| ET_MLX_GMM_PROBE=1 "$R" \ | |
| --pte "$D/model.pte" --input "$D/input.bin" \ | |
| --output /tmp/out_probe.bin --verbose \ | |
| > /tmp/p_out.txt 2> /tmp/p_err.txt | |
| rc1=$? | |
| "$R" \ | |
| --pte "$D/model.pte" --input "$D/input.bin" \ | |
| --output /tmp/out_noprobe.bin --verbose \ | |
| > /tmp/n_out.txt 2> /tmp/n_err.txt | |
| rc2=$? | |
| set -e | |
| echo "=== rc probe=$rc1 / noprobe=$rc2 ===" | |
| echo "=== STDOUT (probe) ==="; cat /tmp/p_out.txt | |
| echo "=== STDERR (probe) ==="; cat /tmp/p_err.txt | |
| echo "=== lignes GMM-PROBE ===" | |
| grep -h 'GMM-PROBE' /tmp/p_out.txt /tmp/p_err.txt || echo "AUCUNE" | |
| echo "=== A/B perturbation : sortie identique avec/sans sonde ? ===" | |
| cmp /tmp/out_probe.bin /tmp/out_noprobe.bin \ | |
| && echo "IDENTIQUES (la sonde ne perturbe pas)" \ | |
| || echo "DIFFERENTES (bug sensible a l'ordre d'execution !)" | |
| # ------------------------------------------------------------------ | |
| # Corroboration harnais — tee + grep dans le step. | |
| # ------------------------------------------------------------------ | |
| - name: SONDE P harnais — famille gather_mm, log greppé | |
| if: steps.build.outcome == 'success' | |
| continue-on-error: true | |
| run: | | |
| set -eux | |
| export ET_MLX_GMM_PROBE=1 | |
| python -m executorch.backends.mlx.test.run_all_tests gather_mm -v \ | |
| 2>&1 | tee /tmp/harness.log | |
| echo "=== GMM-PROBE dans le harnais ===" | |
| grep -n 'GMM-PROBE' /tmp/harness.log || echo "AUCUNE" | |
| echo "=== contexte des configs sorted ===" | |
| grep -n -B3 -A14 'e4_i64_o128_sorted' /tmp/harness.log | head -120 | |
| - name: CENSUS — symboles mlx/Metal en double dans la closure | |
| if: steps.build.outcome == 'success' | |
| continue-on-error: true | |
| run: | | |
| set -eux | |
| RUNNER=$(find cmake-out -name op_test_runner -type f | head -1) | |
| otool -L "$RUNNER" | |
| LINKTXT=$(find cmake-out -path "*op_test_runner.dir/link.txt" | head -1) | |
| rm -f /tmp/defs.txt | |
| for a in $(tr ' ' '\n' < "$LINKTXT" | grep '\.a$' | sort -u); do | |
| [ -f "$a" ] || continue | |
| nm -gU "$a" 2>/dev/null | awk -v A="$a" \ | |
| '$2 ~ /^[TSDW]$/ {print $3, A}' >> /tmp/defs.txt || true | |
| done | |
| grep -iE 'mlx|metal|MTL' /tmp/defs.txt | sort | \ | |
| awk '{print $1}' | uniq -d | head -80 || true | |
| - name: ÉCHELLE — L0 mlx seul (contrôle) | |
| if: steps.build.outcome == 'success' | |
| continue-on-error: true | |
| run: | | |
| set -eux | |
| MLXA=$(find cmake-out -name libmlx.a | head -1) | |
| METALLIB=$(find cmake-out -name mlx.metallib | head -1) | |
| cp "$METALLIB" . | |
| clang++ -std=c++17 -O2 repro_gather_mm.cpp \ | |
| -I backends/mlx/third-party/mlx \ | |
| -I "$(dirname "$MLXA")" \ | |
| "$MLXA" \ | |
| -framework Metal -framework Foundation -framework QuartzCore \ | |
| -o rung_L0 | |
| ./rung_L0 && echo "L0: OK" || echo "L0: DIVERGE/CRASH rc=$?" | |
| - name: ÉCHELLE — L3 closure exacte (contrôle de régression) | |
| if: steps.build.outcome == 'success' | |
| continue-on-error: true | |
| run: | | |
| set -eux | |
| MLXA=$(find cmake-out -name libmlx.a | head -1) | |
| LINKTXT=$(find cmake-out -path "*op_test_runner.dir/link.txt" | head -1) | |
| BLD=$(dirname "$(dirname "$(dirname "$LINKTXT")")") | |
| clang++ -std=c++17 -O2 -c repro_gather_mm.cpp \ | |
| -I backends/mlx/third-party/mlx \ | |
| -I "$(dirname "$MLXA")" \ | |
| -o /tmp/repro.o | |
| sed -E \ | |
| -e "s#[^ ]*op_test_runner\.cpp\.o#/tmp/repro.o#" \ | |
| -e "s# -o [^ ]*op_test_runner# -o $PWD/rung_L3#" \ | |
| "$LINKTXT" > /tmp/l3cmd.sh | |
| (cd "$BLD" && bash /tmp/l3cmd.sh) 2>&1 | tail -40 | |
| ./rung_L3 && echo "L3: OK" || echo "L3: DIVERGE/CRASH rc=$?" | |
| - name: Diagnostics finaux | |
| if: always() | |
| run: df -h / |