diff --git a/.github/workflows/pyre-ci.yml b/.github/workflows/pyre-ci.yml index 4758dcbf98e..f13874307a7 100644 --- a/.github/workflows/pyre-ci.yml +++ b/.github/workflows/pyre-ci.yml @@ -457,6 +457,13 @@ jobs: --lib -- --ignored --test-threads=1 \ compiler::tests::test_host_loop_external_jump_to + - name: Run cel's release-only probe gates (Linux only) + # Debug builds ignore these tests because their arithmetic is intentionally + # wrapping. Run only their correctness gates here; benchmark loops remain + # available through `cargo run -p cel --release`. + if: runner.os == 'Linux' + run: cargo test --release -p cel --no-default-features --features dynasm + # The CPython suite rides this job's Linux copy rather than a job of its # own: it needs the same checkout, toolchain, Cargo cache and prepared # Charon/LLBC set, and those cost a job's worth of setup to assemble a diff --git a/AGENTS.md b/AGENTS.md index 33595089b01..934e14d2875 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -218,8 +218,13 @@ comment at the site citing both sides. ## Before committing -- `cargo test --all --features dynasm`. The feature flag is mandatory: without it - `majit-metainterp` emits `compile_error!` and every error after it is noise. +- `cargo test --all --no-default-features --features dynasm`. Both halves matter. + `--features dynasm` selects a backend; `--no-default-features` is what keeps the + selection consistent across crates. With default features on, the workspace's + example interpreters default to `cranelift` and supply it to the one shared + `majit-metainterp`, while `pyre-jit` compiles in its dynasm registrations — so + the binary dispatches through one backend and holds the other's hooks. Nothing + emits an error; the tests simply run against a mismatched pair. - `python3 pyre/check.py` — every backend the host can build. A perf regression is a finding to explain, not an automatic veto: if the slower code is the line-by-line port and the faster was a shortcut, **the port stands** — record diff --git a/majit/charon-corpus/README.md b/majit/charon-corpus/README.md index 5863d164c8b..cbae13d0d00 100644 --- a/majit/charon-corpus/README.md +++ b/majit/charon-corpus/README.md @@ -81,7 +81,8 @@ out in issue #97: The corpus also includes a header-first object model. These functions pin lowering decisions that otherwise fail by leaving a residual call or an -untyped field access rather than raising an error: +untyped field access rather than raising an error, so each is pinned by an +assertion in `majit-translate/tests/test_mir_frontend.rs`: | Function | Premise it pins | |-----------------------|-------------------------------------------------------| @@ -97,11 +98,23 @@ only data field is `typeptr`. The two-word allocation can fuse when its allocation can fuse from its declared layout because no class word exists to disagree with the type pointer. +Both shapes are carried because `resolve_header_plan` has an arm for each, and +a corpus holding only the two-word shape can never reach the second: the match +lands on the stored `w_class` before `header_declares_no_class_word` is +consulted. + The fixture spells `pyre_object::pyobject::get_instantiate` literally because `model.rs` currently recognises that path suffix. The one-word header does not use this helper. `_immutable_fields_W_IntObject` preserves the marker shape -harvested by `front::llbc_hints`. - +harvested by `front::llbc_hints`; `lower_function` runs upstream of that +consumer, so no assertion in `test_mir_frontend.rs` covers it. Every other +identity in the corpus is named after no host, so a fixture cannot tell a +general recogniser apart from one conformed to a particular host. + +`corpus.ullbc` is regenerated in place whenever this fixture is edited, and it +embeds the source verbatim, so counts taken against an older artefact — op +censuses, lowered-graph totals, local-fn counts — are not comparable across a +regeneration. Re-measure rather than subtract. ## Findings diff --git a/majit/examples/cel/src/colscalar.rs b/majit/examples/cel/src/colscalar.rs index 916eed9bd3f..9b2c263c75f 100644 --- a/majit/examples/cel/src/colscalar.rs +++ b/majit/examples/cel/src/colscalar.rs @@ -1,13 +1,8 @@ -//! Diagnostic: does a SCALAR `int` state-field base for raw_load compile, and -//! does WRITING that field every iteration (as the wasmi kernel does with -//! `state.mem_base = __mb`) change the outcome vs a READ-ONLY scalar field? -//! -//! The `column` probe proved base-in-REGISTER compiles (13-16x). This isolates the -//! VirtualStatesCantMatch root cause: base-in-SCALAR-FIELD. Two programs, -//! identical except one re-writes the scalar field each iteration. -//! RDONLY keeps `state.col_base` read-only after initialization; WRITTEN -//! reassigns it from a register every iteration. -//! Prints compiles/aborts for each. RELEASE ONLY. +//! Raw-load regression probe with the base address in a scalar `int` state +//! field. One program reads a base initialized before the loop; the other +//! rewrites the same field on every iteration. Both must compile. This protects +//! `VirtualizableConfig::identity_input_index`: treating scalar input slot 0 as +//! the virtualizable identity makes the read-only variant fail loop closing. use crate::common::*; use std::sync::atomic::Ordering; @@ -23,6 +18,14 @@ const OP_SET_BASE: i64 = 6; // [SET_BASE, src_reg] state.col_base = regs[src] struct VmState { regs: Vec, col_base: i64, + /// What `OP_RETURN` hands back. + /// + /// The `; state` merge point leaves the loop through `break` before it + /// assigns the walk's resume pc, so after the loop `pc` still names the + /// position the walk started from and `program[pc + 1]` — the operand + /// saying which register holds the result — cannot be read there. The + /// result has to arrive in `state`. + ret: i64, } #[majit_macros::jit_interp( @@ -32,6 +35,7 @@ struct VmState { state_fields = { regs: [int; virt], col_base: int, + ret: int, }, )] fn mainloop(program: &Code, num_regs: usize, col_base: i64, threshold: u32) -> i64 { @@ -48,6 +52,7 @@ fn mainloop(program: &Code, num_regs: usize, col_base: i64, threshold: u32) -> i let mut state = VmState { regs: vec![0; num_regs], col_base, + ret: 0, }; { @@ -58,7 +63,11 @@ fn mainloop(program: &Code, num_regs: usize, col_base: i64, threshold: u32) -> i } loop { - jit_merge_point!(); + // `; state` selects the single-executor close: the walk's final state is + // transferred into `state` here and the native loop resumes at the close + // pc, instead of discarding the walk outcome and re-running the circuit + // the walk already executed. + jit_merge_point!(driver, program, pc; state); let opcode = program[pc]; match opcode { OP_LOAD => { @@ -106,14 +115,29 @@ fn mainloop(program: &Code, num_regs: usize, col_base: i64, threshold: u32) -> i } pc += 4; } + // Stores into `ret` and then leaves through an in-arm `return`, + // never `{ store; break }`: `classify.rs` `is_break_expr` requires + // the arm body to be exactly `break`, so a composite body classifies + // `Lowerable` and its tail `break` reaches `lower_stmt_fallback`, + // which guards an enclosed `return` but not an enclosed `break` — + // the statement is inert and is silently dropped, leaving the + // lowered arm to fall through to the dispatch back-edge. OP_RETURN => { let r = program[pc + 1] as usize; - return state.regs[r]; + state.ret = state.regs[r]; + return state.ret; } - _ => break, + // Was `_ => break` with the panic below the loop. The loop now has a + // second way out — the merge point's own `break` on a walk that + // reached a terminal return — so falling out of it no longer + // identifies a bad opcode, and the panic moves into the arm that + // actually saw one. + _ => panic!("fell off end of code"), } } - panic!("fell off end of code"); + // Reached only when the merge point broke out on a walk that already ran the + // terminal opcode, so the result is whatever that opcode parked in `ret`. + state.ret } fn clean_interp(program: &Code, num_regs: usize, col_base: i64) -> i64 { @@ -266,7 +290,14 @@ fn make_col(n: i64) -> Vec { v } -fn run_case(label: &str, prog: &[i64], col_base: i64) { +/// One case's correctness + tier-liveness gate. There is no timing half in this +/// probe — it reports compile/abort counts, not ns/row — so this is the whole +/// of the work `run` does. +/// +/// `on_c >= 1` says a trace was minted for the case's loop. It does not say the +/// trace has a real body (an empty dispatch still compiles a bare `Finish()`), +/// so it is the liveness half only. +fn gate_case(label: &str, prog: &[i64], col_base: i64) { COMPILES.store(0, Ordering::Relaxed); ABORTS.store(0, Ordering::Relaxed); let clean = clean_interp(prog, NUM_REGS, col_base); @@ -279,23 +310,35 @@ fn run_case(label: &str, prog: &[i64], col_base: i64) { let on_a = ABORTS.load(Ordering::Relaxed); assert_eq!(clean, off, "{label}: clean vs off"); assert_eq!(clean, on, "{label}: clean vs on -> miscompile"); - let verdict = if on_c >= 1 { - "COMPILES" - } else { - "ABORTS (never compiles)" - }; - println!("[{label}] compiles: off={off_c} on={on_c} aborts(on)={on_a} -> {verdict}"); + assert_eq!(off_c, 0, "{label}: JIT-off must never compile"); + // Was a printed verdict — "COMPILES" / "ABORTS (never compiles)" — because + // RDONLY genuinely never compiled before the `identity_input_index` fix the + // header describes. Both cases compile now, so the outcome the probe exists + // to watch is a fixed one and belongs in an assertion: a printed verdict + // cannot fail a run, and the two equality gates above stay green with the + // JIT tier entirely inert. + assert!( + on_c >= 1, + "{label}: JIT-on compiled nothing (aborts={on_a}) — the scalar-field \ + base no longer closes the trace" + ); + println!("[{label}] compiles: off={off_c} on={on_c} aborts(on)={on_a}"); } -pub fn run() { +/// Both cases' gates. This is the whole probe; `run` adds only the banner. +pub(crate) fn run_gates() { let n: i64 = std::env::var("CELN") .ok() .and_then(|s| s.parse().ok()) .unwrap_or(300_000); let col = make_col(n); let base = col.as_ptr() as i64; - println!("scalar-int-state-field base for raw_load: read-only vs written\n"); - run_case("RDONLY ", &rdonly_program(n), base); - run_case("WRITTEN", &written_program(n, base), base); + gate_case("RDONLY ", &rdonly_program(n), base); + gate_case("WRITTEN", &written_program(n, base), base); std::hint::black_box(&col); } + +pub fn run() { + println!("scalar-int-state-field base for raw_load: read-only vs written\n"); + run_gates(); +} diff --git a/majit/examples/cel/src/column.rs b/majit/examples/cel/src/column.rs index 0b59400d91b..2ba946c3866 100644 --- a/majit/examples/cel/src/column.rs +++ b/majit/examples/cel/src/column.rs @@ -20,6 +20,14 @@ const OP_GE: i64 = 6; // [GE, a, b, dst] dst = (regs[a] >= regs[b]) as {0,1} struct VmState { regs: Vec, + /// What `OP_RETURN` hands back. + /// + /// The `; state` merge point leaves the loop through `break` before it + /// assigns the walk's resume pc, so after the loop `pc` still names the + /// position the walk started from and `program[pc + 1]` — the operand + /// saying which register holds the result — cannot be read there. The + /// result has to arrive in `state`. + ret: i64, } #[majit_macros::jit_interp( @@ -28,6 +36,7 @@ struct VmState { greens = [pc, program], state_fields = { regs: [int; virt], + ret: int, }, )] fn mainloop(program: &Code, num_regs: usize, threshold: u32) -> i64 { @@ -43,6 +52,7 @@ fn mainloop(program: &Code, num_regs: usize, threshold: u32) -> i64 { let _stacksize: i32 = 0; let mut state = VmState { regs: vec![0; num_regs], + ret: 0, }; { @@ -53,7 +63,11 @@ fn mainloop(program: &Code, num_regs: usize, threshold: u32) -> i64 { } loop { - jit_merge_point!(); + // `; state` selects the single-executor close: the walk's final state is + // transferred into `state` here and the native loop resumes at the close + // pc, instead of discarding the walk outcome and re-running the circuit + // the walk already executed. + jit_merge_point!(driver, program, pc; state); let opcode = program[pc]; match opcode { OP_LOAD => { @@ -105,14 +119,29 @@ fn mainloop(program: &Code, num_regs: usize, threshold: u32) -> i64 { } pc += 4; } + // Stores into `ret` and then leaves through an in-arm `return`, + // never `{ store; break }`: `classify.rs` `is_break_expr` requires + // the arm body to be exactly `break`, so a composite body classifies + // `Lowerable` and its tail `break` reaches `lower_stmt_fallback`, + // which guards an enclosed `return` but not an enclosed `break` — + // the statement is inert and is silently dropped, leaving the + // lowered arm to fall through to the dispatch back-edge. OP_RETURN => { let r = program[pc + 1] as usize; - return state.regs[r]; + state.ret = state.regs[r]; + return state.ret; } - _ => break, + // Was `_ => break` with the panic below the loop. The loop now has a + // second way out — the merge point's own `break` on a walk that + // reached a terminal return — so falling out of it no longer + // identifies a bad opcode, and the panic moves into the arm that + // actually saw one. + _ => panic!("fell off end of code"), } } - panic!("fell off end of code"); + // Reached only when the merge point broke out on a walk that already ran the + // terminal opcode, so the result is whatever that opcode parked in `ret`. + state.ret } /// Clean interpreter of the identical bytecode — the honest "good non-JIT @@ -297,18 +326,22 @@ fn time_ns_per_row i64>(n: i64, f: F) -> f64 { t.elapsed().as_nanos() as f64 / n as f64 } -/// Run one program: 3-way equality gate (small n) then interleaved A/B/C -/// timing (large n). `hold` keeps the backing column buffers alive. -fn run_program( - label: &str, - num_regs: usize, - prog_at: &dyn Fn(i64) -> Vec, - hold: &[&Vec], -) { - let gn: i64 = std::env::var("CELGATE_N") +/// How many rows the equality gate runs. Far below the timing row count: the +/// gate's three properties hold at any length past the trace threshold. +fn gate_n() -> i64 { + std::env::var("CELGATE_N") .ok() .and_then(|s| s.parse().ok()) - .unwrap_or(300_000); + .unwrap_or(300_000) +} + +/// One program's correctness + tier-liveness gate, with no timing in it. +/// +/// `on_c >= 1` says a trace was minted for the loop. It does not say the trace +/// has a real body — an empty dispatch still compiles one whose whole optimized +/// body is `Finish()` — so this is the liveness half only. +fn gate_program(label: &str, num_regs: usize, prog_at: &dyn Fn(i64) -> Vec) { + let gn = gate_n(); let gprog = prog_at(gn); COMPILES.store(0, Ordering::Relaxed); ABORTS.store(0, Ordering::Relaxed); @@ -326,15 +359,29 @@ fn run_program( "{label}: clean vs JIT-on divergence -> miscompile" ); assert_eq!(off_c, 0, "{label}: JIT-off must never compile"); + // Was a printed `!!` note and an early return, which is a diagnostic and not + // a gate: the three assertions above are all satisfied by the interpreter + // answering alone, so with the JIT tier inert this probe printed its warning + // and still exited 0. + assert!( + on_c >= 1, + "{label}: JIT-on compiled nothing (aborts={on_a}) — the raw_load \ + red-index read does not close the trace" + ); println!( "[{label} gate n={gn}] result={on} (clean==off==on ok) compiles: off={off_c} on={on_c} aborts(on)={on_a}" ); - if on_c == 0 { - println!( - " !! {label}: JIT-on did NOT compile (aborts={on_a}) — raw_load red-index read does not close the trace" - ); - return; - } +} + +/// Run one program: the gate above, then interleaved A/B/C timing (large n). +/// `hold` keeps the backing column buffers alive. +fn run_program( + label: &str, + num_regs: usize, + prog_at: &dyn Fn(i64) -> Vec, + hold: &[&Vec], +) { + gate_program(label, num_regs, prog_at); let n: i64 = 20_000_000; let prog = prog_at(n); @@ -355,21 +402,62 @@ fn run_program( black_box(hold); } +/// REAL data columns: LCG-filled, distinct, non-foldable. The LCG runs from a +/// fixed seed, so a short column is a prefix of a long one and the gate reads +/// the same rows whatever length was allocated for it. +fn make_cols(n: i64) -> (Vec, Vec) { + ( + make_col(n, 0x2545F4914F6CDD1D), + make_col(n, 0x9E3779B97F4A7C15u64 as i64), + ) +} + +type ProgramSpec = (&'static str, usize, Box Vec>); + +/// The programs this probe covers, as `(label, num_regs, builder)`. `run` gates +/// and then times each; [`run_gates`] gates each and stops there. Both walk this +/// one list, so a program added here reaches the test as well as the binary. +fn programs(base_a: i64, base_b: i64) -> Vec { + vec![ + ( + "SUM ", + SUM_REGS, + Box::new(move |k| sum_program(k, base_a)), + ), + ( + "POLICY", + POL_REGS, + Box::new(move |k| policy_program(k, base_a, base_b)), + ), + ] +} + +/// Every program's gate at [`gate_n`] rows, with none of the timing. The columns +/// are allocated to the gate's own length rather than the timing length — two +/// 20M-row buffers is 320 MB the gate never reads past the first `gate_n` rows +/// of. +#[cfg(test)] +pub(crate) fn run_gates() { + let (col_a, col_b) = make_cols(gate_n()); + let base_a = col_a.as_ptr() as i64; + let base_b = col_b.as_ptr() as i64; + for (label, num_regs, prog_at) in programs(base_a, base_b) { + gate_program(label, num_regs, &prog_at); + } + black_box((&col_a, &col_b)); +} + pub fn run() { let n: i64 = 20_000_000; - // REAL data columns: LCG-filled, distinct, non-foldable. - let col_a = make_col(n, 0x2545F4914F6CDD1D); - let col_b = make_col(n, 0x9E3779B97F4A7C15u64 as i64); + let (col_a, col_b) = make_cols(n); let base_a = col_a.as_ptr() as i64; let base_b = col_b.as_ptr() as i64; println!("columnar red-index reads via raw_load_i (base in register file)\n"); - run_program("SUM ", SUM_REGS, &|k| sum_program(k, base_a), &[&col_a]); - println!(); - run_program( - "POLICY", - POL_REGS, - &|k| policy_program(k, base_a, base_b), - &[&col_a, &col_b], - ); + for (i, (label, num_regs, prog_at)) in programs(base_a, base_b).into_iter().enumerate() { + if i > 0 { + println!(); + } + run_program(label, num_regs, &prog_at, &[&col_a, &col_b]); + } } diff --git a/majit/examples/cel/src/float.rs b/majit/examples/cel/src/float.rs index d25132e3a0d..b4f0c10c651 100644 --- a/majit/examples/cel/src/float.rs +++ b/majit/examples/cel/src/float.rs @@ -1,18 +1,23 @@ -//! Float register machine — the honest fair-win probe for the float macro path. +//! Float register-machine benchmark for the macro-generated JIT path. //! -//! This machine originally fused raw_load+add+i++ into ONE opcode; that made the -//! clean interpreter a single-op memory-bandwidth-bound loop with no dispatch -//! for the JIT to eliminate, so the JIT could not win (0.62x). That was a -//! kernel artifact, not a float-JIT defect. This rewrite mirrors the `column` -//! probe's honest structure: a PER-OP bytecode machine whose clean interpreter -//! pays real dispatch cost, so the win is attributable to compilation. +//! Loads, arithmetic, index updates, and branches use separate opcodes so the +//! interpreter and generated JIT execute the same dispatch structure. +//! +//! The per-op structure is deliberate, and this probe has already been fooled +//! once without it. An earlier version fused raw_load+add+i++ into a single +//! opcode, which left the clean interpreter a one-op, memory-bandwidth-bound +//! loop with no dispatch for the JIT to remove; the JIT then read 0.62x, a +//! loss. That figure measured the kernel rather than the float JIT, and +//! re-fusing these opcodes reproduces it. The shape kept here is the `column` +//! probe's: a clean interpreter that pays real dispatch cost, so a win is +//! attributable to compilation. //! //! Addressing (i, n, base_a, base_b) lives in SCALAR int state fields; values //! (column reads, accumulator) live in a `[float; virt]` register bank. Three //! programs sweep the compute:memory ratio: -//! ACC — sum(a[i]) 1 load, 1 float add (memory-bound ref) -//! DOT — sum(a[i]*b[i]) 2 loads, 1 mul + 1 add -//! COMPUTE — sum(a*b - a*a + b*b) 2 loads, 5 float ops (compute-bound) +//! ACC: sum(a[i]), one load and one floating-point add. +//! DOT: sum(a[i] * b[i]), two loads, one multiply, and one add. +//! COMPUTE: sum(a*b - a*a + b*b), two loads and five arithmetic operations. use crate::common::{ABORTS, COMPILES, Code, JIT_OFF, JIT_ON, majit_raw_load_f, median}; use std::hint::black_box; @@ -37,6 +42,13 @@ const R4: usize = 5; const R5: usize = 6; const R6: usize = 7; const NUM_REGS: usize = 8; +/// Result slot, one register past the program's own file. The `; state` merge +/// point can leave the dispatch loop with `pc` still at the walk-segment start, +/// so the result has to come out of `state` after the loop rather than off +/// `program[pc + 1]`. It rides the virtualizable bank because a `ret: float` +/// state field is rejected outright ("state_fields float scalars are not +/// supported with recursive portal fresh allocation yet"). +const RET: usize = NUM_REGS; const BODY_PC: usize = 0; @@ -76,7 +88,7 @@ fn mainloop(program: &Code, base_a: i64, base_b: i64, n: i64, threshold: u32) -> n, base_a, base_b, - regs: vec![0.0; NUM_REGS], + regs: vec![0.0; NUM_REGS + 1], }; { @@ -87,7 +99,7 @@ fn mainloop(program: &Code, base_a: i64, base_b: i64, n: i64, threshold: u32) -> } loop { - jit_merge_point!(); + jit_merge_point!(driver, program, pc; state); let opcode = program[pc]; match opcode { OP_LOAD_A => { @@ -136,10 +148,23 @@ fn mainloop(program: &Code, base_a: i64, base_b: i64, n: i64, threshold: u32) -> } pc += 2; } - OP_RETURN => return state.regs[program[pc + 1] as usize], + // Stores into `RET` and then leaves through an in-arm `return`, never + // `{ store; break }`: `classify.rs` `is_break_expr` requires the arm + // body to be exactly `break`, so a composite body classifies + // `Lowerable` and its tail `break` reaches `lower_stmt_fallback`, + // which guards an enclosed `return` but not an enclosed `break` — + // the statement is inert and is silently dropped, leaving the + // lowered arm to fall through to the dispatch back-edge. + OP_RETURN => { + state.regs[RET] = state.regs[program[pc + 1] as usize]; + return state.regs[RET]; + } _ => panic!("bad opcode {opcode}"), } } + // Reached only when the merge point broke out on a walk that already ran the + // terminal opcode, so the result is whatever that opcode parked in `RET`. + state.regs[RET] } /// Clean interpreter of the identical bytecode — the honest per-op non-JIT @@ -251,7 +276,7 @@ mod count { } loop { - jit_merge_point!(); + jit_merge_point!(driver, program, pc; state); let opcode = program[pc]; match opcode { OP_COUNT_GE => { @@ -274,10 +299,14 @@ mod count { } pc += 2; } - OP_RETURN_COUNT => return state.count, + // The `; state` merge point can leave the loop with `pc` still at + // the walk-segment start, so the result has to come out of + // `state` after the loop rather than off `program[pc + ..]`. + OP_RETURN_COUNT => break, _ => panic!("bad opcode {opcode}"), } } + state.count } fn clean_interp_count(program: &Code, base_a: i64, base_b: i64, n: i64) -> i64 { @@ -331,7 +360,7 @@ mod count { assert_eq!(clean, off, "{label}: clean vs JIT-off count"); assert_eq!(clean, on, "{label}: clean vs JIT-on count"); assert_eq!(off_c, 0, "{label}: JIT-off must not compile"); - assert!(on_c >= 1, "{label}: JIT-on must compile at least one trace"); + assert_eq!(on_c, 1, "{label}: JIT-on must compile exactly one trace"); println!("[{label} n={n}] count={on} compiles off={off_c} on={on_c} aborts on={on_a}"); on } @@ -383,12 +412,12 @@ mod twobank { const R_I: usize = 0; const R_ACC: usize = 1; const R_N: usize = 2; - const R_ONE: usize = 3; + pub(super) const R_ONE: usize = 3; const R_STRIDE: usize = 4; const R_EA: usize = 5; const R_BASE_A: usize = 6; const R_BASE_B: usize = 7; - const R_BOOL: usize = 8; + pub(super) const R_BOOL: usize = 8; const NUM_INT: usize = 9; const F_A: usize = 0; @@ -400,6 +429,16 @@ mod twobank { struct TwoBankState { regs: Vec, fregs: Vec, + /// What `OP_RETURN` hands back. + /// + /// The `; state` merge point leaves the loop through `break` before it + /// assigns the walk's resume pc, so after the loop `pc` still names the + /// position the walk started from and `program[pc + 1]` — the operand + /// saying which register holds the result — cannot be read there. An + /// `int` scalar is the direct carrier; it is only available since the + /// `VirtualizableConfig::identity_input_index` fix, before which a + /// scalar declared beside a `[.. ; virt]` array aborted every trace. + ret: i64, } #[majit_macros::jit_interp( @@ -409,6 +448,7 @@ mod twobank { state_fields = { regs: [int; virt], fregs: [float; virt], + ret: int, }, )] fn mainloop_twobank(program: &Code, threshold: u32) -> i64 { @@ -425,6 +465,7 @@ mod twobank { let mut state = TwoBankState { regs: vec![0; NUM_INT], fregs: vec![0.0; NUM_FLOAT], + ret: 0, }; { @@ -435,7 +476,11 @@ mod twobank { } loop { - jit_merge_point!(); + // `; state` selects the single-executor close: the walk's final + // state is transferred into `state` here and the native loop resumes + // at the close pc, instead of discarding the walk outcome and + // re-running the circuit the walk already executed. + jit_merge_point!(driver, program, pc; state); let opcode = program[pc]; match opcode { OP_LOAD => { @@ -483,10 +528,24 @@ mod twobank { } pc += 4; } - OP_RETURN => return state.regs[program[pc + 1] as usize], + // Stores into `ret` and then leaves through an in-arm `return`, + // never `{ store; break }`: `classify.rs` `is_break_expr` + // requires the arm body to be exactly `break`, so a composite + // body classifies `Lowerable` and its tail `break` reaches + // `lower_stmt_fallback`, which guards an enclosed `return` but + // not an enclosed `break` — the statement is inert and is + // silently dropped, leaving the lowered arm to fall through to + // the dispatch back-edge. + OP_RETURN => { + state.ret = state.regs[program[pc + 1] as usize]; + return state.ret; + } _ => panic!("bad opcode {opcode}"), } } + // Reached only when the merge point broke out on a walk that already ran + // the terminal opcode, so the result is whatever it parked in `ret`. + state.ret } fn clean_twobank(program: &Code) -> i64 { @@ -536,7 +595,12 @@ mod twobank { } // count rows where a[i] >= b[i] - fn program(n: i64, base_a: i64, base_b: i64) -> Vec { + /// The counting program, over `accumuland`. + /// + /// `R_BOOL` implements `acc += a[i] >= b[i]`. `R_ONE` instead increments on + /// every row, which makes duplicated or skipped iterations observable even + /// when the affected row's comparison is false. + fn program(n: i64, base_a: i64, base_b: i64, accumuland: usize) -> Vec { let mut p = vec![ OP_LOAD, 0, @@ -580,7 +644,7 @@ mod twobank { R_BOOL as i64, OP_ADD, R_ACC as i64, - R_BOOL as i64, + accumuland as i64, R_ACC as i64, OP_ADD, R_I as i64, @@ -596,10 +660,16 @@ mod twobank { p } - pub(super) fn run_gate(label: &str, n: i64, col_a: &[f64], col_b: &[f64]) -> i64 { + pub(super) fn run_gate( + label: &str, + n: i64, + col_a: &[f64], + col_b: &[f64], + accumuland: usize, + ) -> i64 { let base_a = col_a.as_ptr() as i64; let base_b = col_b.as_ptr() as i64; - let prog = program(n, base_a, base_b); + let prog = program(n, base_a, base_b, accumuland); COMPILES.store(0, Ordering::Relaxed); ABORTS.store(0, Ordering::Relaxed); @@ -617,7 +687,7 @@ mod twobank { assert_eq!(clean, off, "{label}: clean vs JIT-off"); assert_eq!(clean, on, "{label}: clean vs JIT-on"); assert_eq!(off_c, 0, "{label}: JIT-off must not compile"); - assert!(on_c >= 1, "{label}: JIT-on must compile at least one trace"); + assert_eq!(on_c, 1, "{label}: JIT-on must compile exactly one trace"); println!( "[twobank {label} n={n}] count={on} compiles off={off_c} on={on_c} aborts on={on_a}" ); @@ -631,7 +701,7 @@ mod twobank { .unwrap_or(1_000_000); let base_a = col_a.as_ptr() as i64; let base_b = col_b.as_ptr() as i64; - let prog = program(n, base_a, base_b); + let prog = program(n, base_a, base_b, R_BOOL); let mut clean = Vec::new(); let mut jit = Vec::new(); for _ in 0..5 { @@ -763,7 +833,7 @@ fn run_gate(label: &str, prog: &Code, n: i64, col_a: &[f64], col_b: &[f64]) -> f assert_eq!(clean.to_bits(), off.to_bits(), "{label}: clean vs JIT-off"); assert_eq!(clean.to_bits(), on.to_bits(), "{label}: clean vs JIT-on"); assert_eq!(off_c, 0, "{label}: JIT-off must not compile"); - assert!(on_c >= 1, "{label}: JIT-on must compile at least one trace"); + assert_eq!(on_c, 1, "{label}: JIT-on must compile exactly one trace"); println!( "[{label} n={n}] bits={:#018x} value={on:.17e} compiles off={off_c} on={on_c} aborts off={off_a} on={on_a}", on.to_bits() @@ -800,10 +870,29 @@ fn perf_probe(label: &str, prog: &Code, col_a: &[f64], col_b: &[f64]) { ); } -pub fn run() { - let max_n = 1_100_000usize; - let col_a = make_col(max_n, 0x2545_F491_4F6C_DD1D); - let col_b = make_col(max_n, 0x9E37_79B9_7F4A_7C15); +/// The two columns, `n` rows each. `make_col`'s LCG runs from a fixed seed, so a +/// short column is a prefix of a long one and a gate reads the same values +/// whatever length was allocated for it. +fn make_cols(n: usize) -> (Vec, Vec) { + ( + make_col(n, 0x2545_F491_4F6C_DD1D), + make_col(n, 0x9E37_79B9_7F4A_7C15), + ) +} + +/// The longest column any gate below reads. +const GATE_MAX_N: usize = 200_017; + +/// Every gate in this probe and none of the timing. Allocates its own columns at +/// the gate row count rather than the perf probes' 1.1M rows. +/// +/// The `on_c >= 1` inside each `run_gate` says a trace was minted for that loop. +/// It does not say the trace has a real body — an empty dispatch still compiles +/// one whose whole optimized body is `Finish()` — so that half is tier liveness +/// only. The absolute trip-count assertions are a separate property (they catch +/// a loop that ran the wrong number of rows) and do not certify the body either. +pub(crate) fn run_gates() { + let (col_a, col_b) = make_cols(GATE_MAX_N); // Bit-exact correctness gate on every program, plus a guard-resume length // variant (steady-state accumulate never hits a float resume otherwise). @@ -829,16 +918,53 @@ pub fn run() { count_primary, count_resume, "count: guard-resume variant should use a distinct length" ); + // The same loop with an absolute assertion on the trip count. `count` above + // accumulates a predicate, so a duplicated iteration only moves it when that + // row's `a[i] >= b[i]` happens to hold — and the duplicated row is always the + // one the trace closes on, so the gate can agree with the clean oracle while + // the loop runs an extra row. Passing one column as BOTH operands makes the + // predicate true for every row, which turns `count` into the trip count and + // makes that extra row visible. + for n in [200_000i64, 200_017] { + let rows = count::run_gate_count(&format!("count-rows n={n}"), n, &col_a, &col_a); + assert_eq!( + rows, n, + "count: the loop ran {rows} rows for n={n} — one extra iteration is \ + the signature of a terminal arm whose `break` was dropped" + ); + } + // Two-bank de-risk: int virt-array + float virt-array in one state. + let tb_primary = twobank::run_gate("count-ge", 200_000, &col_a, &col_b, twobank::R_BOOL); + let tb_resume = twobank::run_gate("count-ge-resume", 200_017, &col_a, &col_b, twobank::R_BOOL); + assert_ne!(tb_primary, tb_resume, "twobank: distinct lengths"); + // The same loop with an every-row accumulator, and an absolute assertion on + // the trip count rather than only agreement with the clean interpreter. The + // gate above compares against a clean oracle and still cannot see a + // duplicated iteration (see `twobank::program`); this one can, because a + // trip count is the thing being asserted. + for n in [200_000i64, 200_017] { + let rows = twobank::run_gate(&format!("rows n={n}"), n, &col_a, &col_b, twobank::R_ONE); + assert_eq!( + rows, n, + "twobank: the loop ran {rows} rows for n={n} — one extra iteration \ + is the signature of a terminal arm whose `break` was dropped" + ); + } + + black_box(col_a); + black_box(col_b); +} + +pub fn run() { + run_gates(); + + let max_n = 1_100_000usize; + let (col_a, col_b) = make_cols(max_n); perf_probe("acc", &acc_program(), &col_a, &col_b); perf_probe("dot", &dot_program(), &col_a, &col_b); perf_probe("compute", &compute_program(), &col_a, &col_b); count::perf_probe_count(&col_a, &col_b); - - // Two-bank de-risk: int virt-array + float virt-array in one state. - let tb_primary = twobank::run_gate("count-ge", 200_000, &col_a, &col_b); - let tb_resume = twobank::run_gate("count-ge-resume", 200_017, &col_a, &col_b); - assert_ne!(tb_primary, tb_resume, "twobank: distinct lengths"); twobank::perf_probe(&col_a, &col_b); black_box(col_a); diff --git a/majit/examples/cel/src/main.rs b/majit/examples/cel/src/main.rs index 9bf1a1c2cce..746e1c27d4e 100644 --- a/majit/examples/cel/src/main.rs +++ b/majit/examples/cel/src/main.rs @@ -1,6 +1,4 @@ -//! cel-majit de-risk probes (issue #357) — one binary gathering the -//! meta-tracing kill-tests / prototypes that used to be separate examples -//! (celprobe / celpolicy / celcolumn / celcolscalar / celfloat). +//! CEL-shaped majit probes collected into one binary. //! //! Each probe is a self-contained `#[jit_interp]` register machine over an //! i64-word bytecode, gated 3-way (clean interp == JIT-off == JIT-on) and then @@ -19,7 +17,9 @@ //! * float — float register machines (acc/dot/compute, count, two-bank) //! //! RELEASE ONLY: the i64-wrap and bit-exact-float equality gates need overflow -//! checks off. +//! checks off. That applies to the `#[test]`s at the bottom of this file as much +//! as to the binary, so they self-ignore under `debug_assertions` and CI runs +//! them from a dedicated `cargo test --release -p cel` step. mod colscalar; mod column; @@ -32,7 +32,7 @@ mod probe; /// the `#[jit_interp]` macro recognizes, the JIT thresholds, the compile/abort /// counters the drivers bump, and the timing median. pub mod common { - use std::sync::atomic::AtomicUsize; + use std::sync::atomic::{AtomicBool, AtomicUsize}; /// The env: an i64-word bytecode stream (8-byte elements). pub type Code = [i64]; @@ -49,6 +49,16 @@ pub mod common { pub static COMPILES: AtomicUsize = AtomicUsize::new(0); pub static ABORTS: AtomicUsize = AtomicUsize::new(0); + /// Shape of the most recently compiled loop body, from + /// `LoopBodyShape::of(opcodes)` in the `on_compile_loop` hook. + /// + /// `COMPILES` counts that a trace was minted; these two say whether the + /// body it minted does anything. A dispatch that lowers nothing still + /// compiles a loop whose whole optimized body is `Finish()`, so a count + /// alone cannot separate a working tier from a dead one. + pub static LAST_HAS_JUMP: AtomicBool = AtomicBool::new(false); + pub static LAST_ALWAYS_FAILS: AtomicBool = AtomicBool::new(false); + /// Raw native-memory load intrinsics recognized by the `#[jit_interp]` proc /// macro (lowered to `raw_load_i` / `raw_load_f`); at the interpreter tier /// these real fns run. `base`/`ea` are an address and a byte offset. @@ -95,3 +105,171 @@ fn main() { std::process::exit(2); } } + +/// Each probe's correctness + tier-liveness gate, run as a test. +/// +/// Every gate the five probes carry already existed; until this module they were +/// reachable only through `main`, so `cargo test -p cel` compiled the crate and +/// ran nothing. These tests call the gate halves only — the timing loops +/// (`probe` alone times 9 interleaved rounds over 20M rows) stay in the binary. +/// +/// The gates assert three-way agreement (clean interpreter == JIT-off == +/// JIT-on), that JIT-off compiles nothing, and that JIT-on compiles at least one +/// loop. That last one is tier liveness and nothing more: `compiles >= 1` counts +/// TRACES, not work, and an empty dispatch still compiles a trace whose whole +/// optimized body is `Finish()`. No assertion here inspects a compiled body. +#[cfg(test)] +mod tests { + /// `common::COMPILES` / `common::ABORTS` are process-global and every gate + /// brackets its runs with `store(0)` … `load()`, so two gates on libtest's + /// parallel threads read each other's compiles. The guard is held across the + /// whole gate call, which is what puts the counter loads — they happen + /// inside the gate — inside the lock. A load taken after the guard dropped + /// would observe a concurrent test's compile. + /// + /// Poison is discarded: a gate that fails an assertion panics with the lock + /// held, and a poisoned mutex would turn one real failure into five. + static PROBE_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(()); + + /// The unroll-retry channel, read as a delta inside `PROBE_LOCK` — the same + /// window the gates bracket their own counters in. The counters are + /// process-global and CUMULATIVE, so an absolute read would carry every + /// other gate's events, which is what the lock exists to prevent. + /// + /// The two unroll counters are stages of one retry sequence and must not be + /// summed. Their interpretation is defined beside `MC_DIAG_LABELS`. + /// + /// The slots are looked up by LABEL and their indices fall out, because an + /// index re-stated beside a hand-written name is free to drift from it and + /// these indices have moved before. The printed name comes from + /// `MC_DIAG_LABELS` for the same reason, as does the gate name from + /// libtest's thread name — a harness that does not name its threads reports + /// `unknown` rather than a plausible-looking guess. + const CENSUS_SLOTS: [usize; 3] = [ + mc_diag_slot("unroll_cancelled_invalid_loop"), + mc_diag_slot("unroll_free_retry_rescued"), + mc_diag_slot("unroll_free_retry_failed"), + ]; + + /// The index of `label` in `MC_DIAG_LABELS`, resolved at compile time. + /// + /// A label renamed or removed upstream fails the BUILD here. The census + /// prints `MC_DIAG_LABELS[slot]` beside every count, so a wrong index reads + /// back as internally consistent and cannot announce itself at runtime. + const fn mc_diag_slot(label: &str) -> usize { + let mut slot = 0; + while slot < majit_metainterp::MC_DIAG_LABELS.len() { + if str_eq(majit_metainterp::MC_DIAG_LABELS[slot], label) { + return slot; + } + slot += 1; + } + panic!("no MC_DIAG slot carries this label"); + } + + /// `str` carries no const `==`. + const fn str_eq(a: &str, b: &str) -> bool { + let a = a.as_bytes(); + let b = b.as_bytes(); + if a.len() != b.len() { + return false; + } + let mut i = 0; + while i < a.len() { + if a[i] != b[i] { + return false; + } + i += 1; + } + true + } + + fn exclusive(gate: impl FnOnce() -> T) -> T { + let _guard = PROBE_LOCK.lock().unwrap_or_else(|e| e.into_inner()); + let before = CENSUS_SLOTS.map(majit_metainterp::mc_diag); + let r = gate(); + let after = CENSUS_SLOTS.map(majit_metainterp::mc_diag); + let thread = std::thread::current(); + let counts = CENSUS_SLOTS + .iter() + .enumerate() + .map(|(i, &slot)| { + format!( + "{}={}", + majit_metainterp::MC_DIAG_LABELS[slot], + after[i] - before[i] + ) + }) + .collect::>() + .join(" "); + eprintln!( + "[unroll-census] gate={} {counts}", + thread.name().unwrap_or("unknown") + ); + + // CEL's generated dispatch arms should not fall back to abort stubs. + // Check the shared registry once after each gate rather than duplicating + // the set of generated interpreter names in every probe. + let degraded: Vec<(&str, &str, &str)> = majit_metainterp::degraded_dispatch_arms() + .iter() + .map(|a| (a.interp, a.arm, a.reason)) + .collect(); + assert_eq!( + degraded, + Vec::new(), + "dispatch arms degraded to abort stubs, seen after gate {}: \ + {degraded:?}. The registry is cumulative and never cleared, so the \ + gate named is where this was NOTICED and not necessarily the one \ + that installed the arm — every later gate reports it too.", + thread.name().unwrap_or("unknown") + ); + r + } + + // Why every test below self-ignores under `debug_assertions`: the gates + // compare i64 results that wrap and floats that must match bit-for-bit + // across three execution paths, and in a debug profile the arithmetic panics + // on overflow instead of wrapping — see the module header. `cargo test --all` + // is debug, so that leg reports these ignored; the dedicated release step in + // `.github/workflows/pyre-ci.yml` is what actually runs them. Left out of CI + // the crate is back to being unable to fail, only now with a green badge. + // + // The reason string is spelled out at each site rather than shared through a + // `macro_rules!`: `#[ignore = mac!()]` compiles and then silently reports a + // bare `ignored` with no reason at all. + + #[test] + #[cfg_attr(debug_assertions, ignore = "release only: wrapping i64 arithmetic")] + fn probe_gates() { + // 20M rows in the binary; the gate proves the same three properties at + // any row count past the trace threshold of 8. + exclusive(|| crate::probe::run_gates(300_000)); + } + + #[test] + #[cfg_attr(debug_assertions, ignore = "release only: wrapping i64 arithmetic")] + fn policy_gates() { + exclusive(crate::policy::run_gates); + } + + #[test] + #[cfg_attr(debug_assertions, ignore = "release only: wrapping i64 arithmetic")] + fn column_gates() { + exclusive(crate::column::run_gates); + } + + #[test] + #[cfg_attr(debug_assertions, ignore = "release only: wrapping i64 arithmetic")] + fn colscalar_gates() { + exclusive(crate::colscalar::run_gates); + } + + #[test] + #[cfg_attr( + debug_assertions, + ignore = "release only: bit-exact float and wrapping i64 arithmetic" + )] + fn float_gates() { + exclusive(crate::float::run_gates); + } +} diff --git a/majit/examples/cel/src/policy.rs b/majit/examples/cel/src/policy.rs index 537ed0eded5..40f3f594016 100644 --- a/majit/examples/cel/src/policy.rs +++ b/majit/examples/cel/src/policy.rs @@ -25,6 +25,14 @@ const HI: i64 = i64::MAX; // `draw >= HI` ~always false, not foldable struct VmState { regs: Vec, + /// What `OP_RETURN` hands back. + /// + /// The `; state` merge point leaves the loop through `break` before it + /// assigns the walk's resume pc, so after the loop `pc` still names the + /// position the walk started from and `program[pc + 1]` — the operand + /// saying which register holds the result — cannot be read there. The + /// result has to arrive in `state`. + ret: i64, } #[majit_macros::jit_interp( @@ -33,6 +41,7 @@ struct VmState { greens = [pc, program], state_fields = { regs: [int; virt], + ret: int, }, )] fn mainloop(program: &Code, num_regs: usize, threshold: u32) -> i64 { @@ -45,6 +54,7 @@ fn mainloop(program: &Code, num_regs: usize, threshold: u32) -> i64 { let _stacksize: i32 = 0; let mut state = VmState { regs: vec![0; num_regs], + ret: 0, }; { @@ -55,7 +65,11 @@ fn mainloop(program: &Code, num_regs: usize, threshold: u32) -> i64 { } loop { - jit_merge_point!(); + // `; state` selects the single-executor close: the walk's final state is + // transferred into `state` here and the native loop resumes at the close + // pc, instead of discarding the walk outcome and re-running the circuit + // the walk already executed. + jit_merge_point!(driver, program, pc; state); let opcode = program[pc]; match opcode { OP_LOAD => { @@ -112,14 +126,29 @@ fn mainloop(program: &Code, num_regs: usize, threshold: u32) -> i64 { } pc += 4; } + // Stores into `ret` and then leaves through an in-arm `return`, + // never `{ store; break }`: `classify.rs` `is_break_expr` requires + // the arm body to be exactly `break`, so a composite body classifies + // `Lowerable` and its tail `break` reaches `lower_stmt_fallback`, + // which guards an enclosed `return` but not an enclosed `break` — + // the statement is inert and is silently dropped, leaving the + // lowered arm to fall through to the dispatch back-edge. OP_RETURN => { let r = program[pc + 1] as usize; - return state.regs[r]; + state.ret = state.regs[r]; + return state.ret; } - _ => break, + // Was `_ => break` with the panic below the loop. The loop now has a + // second way out — the merge point's own `break` on a walk that + // reached a terminal return — so falling out of it no longer + // identifies a bad opcode, and the panic moves into the arm that + // actually saw one. + _ => panic!("fell off end of code"), } } - panic!("fell off end of code"); + // Reached only when the merge point broke out on a walk that already ran the + // terminal opcode, so the result is whatever that opcode parked in `ret`. + state.ret } fn clean_interp(program: &Code, num_regs: usize) -> i64 { @@ -470,7 +499,24 @@ fn time_ns i64>(n: i64, f: F) -> f64 { t.elapsed().as_nanos() as f64 / n as f64 } -fn run_regime(name: &str, prog: &[i64], n: i64, rounds: usize) { +/// One regime's correctness + tier-liveness gate, with no timing in it. +/// +/// `on_c == expect_compiles` says a trace was minted for each loop the regime +/// has. It does not say a trace has a real body — an empty dispatch still +/// compiles one whose whole optimized body is `Finish()` — so this is the +/// liveness half only. +/// +/// The count is pinned exactly rather than as `>= 1`. That is what separates the +/// two comprehension regimes: `LOOP` compiles both the inner loop and the outer +/// row loop, `UNROLL` compiles only the outer one because the inner is unrolled +/// against the green length. A lower bound reads 1 as a pass for both and so +/// cannot see the unroll stop happening — which is the property these two +/// regimes exist to compare. +/// +/// The `pass-rate` line moved here from below the timing loop with it: it +/// reports the gate's own `on`, and is now printed before the timing rather than +/// after. +fn gate_regime(name: &str, prog: &[i64], n: i64, expect_compiles: usize) { COMPILES.store(0, Ordering::Relaxed); let clean = clean_interp(prog, NUM_REGS); let off = mainloop(prog, NUM_REGS, JIT_OFF); @@ -484,6 +530,23 @@ fn run_regime(name: &str, prog: &[i64], n: i64, rounds: usize) { "{name}: clean vs JIT-on divergence -> miscompile" ); assert_eq!(off_c, 0, "{name}: JIT-off must not compile"); + // Unlike `column` and `colscalar`, which are diagnostics that report either + // outcome, this probe's numbers only mean anything if a trace ran: a JIT + // that compiled nothing still answers correctly through the interpreter, so + // the three equality gates above stay green and the kill-bar below just + // prints FAIL. Nothing else here would change the exit code. + assert_eq!( + on_c, expect_compiles, + "{name}: JIT-on must compile exactly this regime's loops" + ); + println!( + "[{name}] pass-rate={:.1}% compiles(on)={on_c}", + 100.0 * on as f64 / n as f64 + ); +} + +fn run_regime(name: &str, prog: &[i64], n: i64, rounds: usize, expect_compiles: usize) { + gate_regime(name, prog, n, expect_compiles); let (mut a, mut b, mut c) = (Vec::new(), Vec::new(), Vec::new()); for _ in 0..rounds { @@ -492,10 +555,6 @@ fn run_regime(name: &str, prog: &[i64], n: i64, rounds: usize) { a.push(time_ns(n, || mainloop(prog, NUM_REGS, JIT_ON))); } let (a, b, c) = (median(a), median(b), median(c)); - println!( - "[{name}] pass-rate={:.1}% compiles(on)={on_c}", - 100.0 * on as f64 / n as f64 - ); println!(" (a) JIT-on {a:.3} (b) clean {b:.3} (c) JIT-off {c:.3} ns/eval"); println!( " (b)/(a) clean-vs-trace = {:.2}x kill-bar>=3x: {}", @@ -504,36 +563,107 @@ fn run_regime(name: &str, prog: &[i64], n: i64, rounds: usize) { ); } -pub fn run() { - let n: i64 = 5_000_000; - println!("policy: account.balance >= txn.amount && !account.frozen (slot-resolved)\n"); - run_regime("SKEWED ", &skewed_program(n), n, 5); - println!(); - run_regime("UNBIASED", &unbiased_program(n), n, 5); +/// One regime: the banner printed before it (if it opens a group), its name, +/// its program, its row count, and how many timing rounds `run` gives it. +struct Regime { + banner: Option<&'static str>, + name: String, + prog: Vec, + n: i64, + rounds: usize, + /// How many loops this regime's program compiles under JIT-on. Pinned per + /// regime rather than as a lower bound because the unrolled comprehension + /// differs from the looping one by exactly this count. + compiles: usize, +} + +/// Every regime the probe covers, with the row counts divided by `scale`. +/// +/// `run` walks this at `scale = 1`; [`run_gates`] walks the SAME list at a +/// larger scale, so a regime added here is gated by the test as well as timed by +/// the binary and the two cannot drift apart. Dividing is sound because what the +/// gate asserts — three-way equality and a compiled loop — holds at any row +/// count past the trace threshold. +fn regimes(scale: i64) -> Vec { + let n = 5_000_000 / scale; + let mut out = vec![ + Regime { + banner: None, + name: "SKEWED ".to_string(), + prog: skewed_program(n), + n, + rounds: 5, + compiles: 1, + }, + Regime { + banner: Some(""), + name: "UNBIASED".to_string(), + prog: unbiased_program(n), + n, + rounds: 5, + compiles: 1, + }, + ]; // Comprehension: nested loop, ns/eval is per-ROW (each row runs `listlen` // inner iterations). Sweep listlen to find where the inner loop amortizes // its trace entry/exit. Lean (few rounds, small total work) because JIT-on // may thrash. Total inner work n*listlen kept ~1M. - println!("\ncomprehension (INNER LOOP): size(list.filter(x, x >= 0)) — sweep list length\n"); + let mut banner = + Some("\ncomprehension (INNER LOOP): size(list.filter(x, x >= 0)) — sweep list length\n"); for &listlen in &[8_i64, 64, 1024] { - let nc = (1_000_000 / listlen).max(4_000); - run_regime( - &format!("LOOP len={listlen:<4}"), - &comprehension_program(nc, listlen), - nc, - 3, - ); + let nc = (1_000_000 / scale / listlen).max(4_000 / scale); + out.push(Regime { + banner: banner.take(), + name: format!("LOOP len={listlen:<4}"), + prog: comprehension_program(nc, listlen), + n: nc, + rounds: 3, + // The inner comprehension loop and the outer row loop. + compiles: 2, + }); } - println!("\ncomprehension (UNROLLED, green length) — same work, inner loop unrolled\n"); + let mut banner = + Some("\ncomprehension (UNROLLED, green length) — same work, inner loop unrolled\n"); for &listlen in &[8_i64, 64] { - let nc = (8_000_000 / listlen).max(50_000); - run_regime( - &format!("UNROLL len={listlen:<4}"), - &comprehension_unrolled_program(nc, listlen), - nc, - 5, - ); + let nc = (8_000_000 / scale / listlen).max(50_000 / scale); + out.push(Regime { + banner: banner.take(), + name: format!("UNROLL len={listlen:<4}"), + prog: comprehension_unrolled_program(nc, listlen), + n: nc, + rounds: 5, + // The outer row loop only: the inner one is unrolled. + compiles: 1, + }); + } + out +} + +/// How much smaller the gate-only walk is than the binary's. Chosen so the whole +/// walk gates in well under a second; the trace threshold is 8, which every +/// scaled row count still clears by three orders of magnitude. +#[cfg(test)] +const GATE_SCALE: i64 = 25; + +/// Every regime's gate, with none of the timing. +#[cfg(test)] +pub(crate) fn run_gates() { + for r in regimes(GATE_SCALE) { + if let Some(banner) = r.banner { + println!("{banner}"); + } + gate_regime(&r.name, &r.prog, r.n, r.compiles); + } +} + +pub fn run() { + println!("policy: account.balance >= txn.amount && !account.frozen (slot-resolved)\n"); + for r in regimes(1) { + if let Some(banner) = r.banner { + println!("{banner}"); + } + run_regime(&r.name, &r.prog, r.n, r.rounds, r.compiles); } } diff --git a/majit/examples/cel/src/probe.rs b/majit/examples/cel/src/probe.rs index 7c1d90fc525..40af90999ea 100644 --- a/majit/examples/cel/src/probe.rs +++ b/majit/examples/cel/src/probe.rs @@ -18,6 +18,14 @@ const OP_SUB: i64 = 5; // [SUB, a, b, dst] struct VmState { regs: Vec, + /// What `OP_RETURN` hands back. + /// + /// The `; state` merge point leaves the loop through `break` before it + /// assigns the walk's resume pc, so after the loop `pc` still names the + /// position the walk started from and `program[pc + 1]` — the operand + /// saying which register holds the result — cannot be read there. The + /// result has to arrive in `state`. + ret: i64, } #[majit_macros::jit_interp( @@ -26,18 +34,23 @@ struct VmState { greens = [pc, program], state_fields = { regs: [int; virt], + ret: int, }, )] fn mainloop(program: &Code, num_regs: usize, threshold: u32) -> i64 { let mut driver: majit_metainterp::JitDriver = majit_metainterp::JitDriver::new(threshold); - driver.set_on_compile_loop(|_green_key, _ops_before, _ops_after, _opcodes| { + driver.set_on_compile_loop(|_green_key, _ops_before, _ops_after, opcodes| { COMPILES.fetch_add(1, Ordering::Relaxed); + let shape = majit_metainterp::LoopBodyShape::of(opcodes); + LAST_HAS_JUMP.store(shape.has_jump, Ordering::Relaxed); + LAST_ALWAYS_FAILS.store(shape.has_always_fails, Ordering::Relaxed); }); let mut pc: usize = 0; let _stacksize: i32 = 0; let mut state = VmState { regs: vec![0; num_regs], + ret: 0, }; { @@ -48,7 +61,11 @@ fn mainloop(program: &Code, num_regs: usize, threshold: u32) -> i64 { } loop { - jit_merge_point!(); + // `; state` selects the single-executor close: the walk's final state is + // transferred into `state` here and the native loop resumes at the close + // pc, instead of discarding the walk outcome and re-running the circuit + // the walk already executed. + jit_merge_point!(driver, program, pc; state); let opcode = program[pc]; match opcode { OP_LOAD => { @@ -91,18 +108,32 @@ fn mainloop(program: &Code, num_regs: usize, threshold: u32) -> i64 { } pc += 4; } + // Stores into `ret` and then leaves through an in-arm `return`, + // never `{ store; break }`: `classify.rs` `is_break_expr` requires + // the arm body to be exactly `break`, so a composite body classifies + // `Lowerable` and its tail `break` reaches `lower_stmt_fallback`, + // which guards an enclosed `return` but not an enclosed `break` — + // the statement is inert and is silently dropped, leaving the + // lowered arm to fall through to the dispatch back-edge. OP_RETURN => { let r = program[pc + 1] as usize; - return state.regs[r]; + state.ret = state.regs[r]; + return state.ret; } - _ => break, + // Was `_ => break` with the panic below the loop. The loop now has a + // second way out — the merge point's own `break` on a walk that + // reached a terminal return — so falling out of it no longer + // identifies a bad opcode, and the panic moves into the arm that + // actually saw one. + _ => panic!("fell off end of code"), } } - panic!("fell off end of code"); + // Reached only when the merge point broke out on a walk that already ran the + // terminal opcode, so the result is whatever that opcode parked in `ret`. + state.ret } -/// A CLEAN interpreter of the identical bytecode: no jit_merge_point, no -/// can_enter_jit, no driver — the honest "good non-JIT implementation" baseline. +/// Interpreter for the same bytecode without JIT hooks. fn clean_interp(program: &Code, num_regs: usize) -> i64 { let mut regs = vec![0i64; num_regs]; let mut pc = 0usize; @@ -231,29 +262,74 @@ fn time_ns_per_eval i64>(n: i64, f: F) -> f64 { t.elapsed().as_nanos() as f64 / n as f64 } -pub fn run() { +/// The correctness half, with no timing in it: all three paths must produce the +/// identical (release-wrapped) accumulator, JIT-off must compile nothing, and +/// JIT-on must compile the hot loop. +/// +/// `on_compiles == 1` says a trace was minted for the loop. It does not say the +/// trace has a real body — an empty dispatch still compiles one whose whole +/// optimized body is `Finish()` — so the count is the tier-liveness half only. +/// The shape assertion below is the other half: it reads the opcode kinds of +/// the body that count refers to, and fails when the body does not close a +/// loop. +/// +/// The count is pinned exactly because this program has one hot loop. The +/// `exclusive` wrapper makes an extra compilation observable rather than +/// allowing a concurrent probe's trace to satisfy a loose lower bound. +/// +/// `n` is a parameter because the gate proves the same three properties at any +/// row count past the trace threshold: [`run`] gates at the row count it then +/// times, and the test gates at a fraction of it. +pub(crate) fn run_gates(n: i64) { // Sanity: the body really starts at BODY_PC (the backward JIA target). assert_eq!(batch_program(1)[BODY_PC], OP_MUL, "BODY_PC out of sync"); - let n: i64 = 20_000_000; let prog = batch_program(n); - // Correctness gate: all three paths must produce the identical - // (release-wrapped) accumulator. COMPILES.store(0, Ordering::Relaxed); let clean = clean_interp(&prog, NUM_REGS); let off = mainloop(&prog, NUM_REGS, JIT_OFF); let off_compiles = COMPILES.load(Ordering::Relaxed); COMPILES.store(0, Ordering::Relaxed); + LAST_HAS_JUMP.store(false, Ordering::Relaxed); + LAST_ALWAYS_FAILS.store(false, Ordering::Relaxed); let on = mainloop(&prog, NUM_REGS, JIT_ON); let on_compiles = COMPILES.load(Ordering::Relaxed); + let shape = majit_metainterp::LoopBodyShape { + has_jump: LAST_HAS_JUMP.load(Ordering::Relaxed), + has_always_fails: LAST_ALWAYS_FAILS.load(Ordering::Relaxed), + }; assert_eq!(clean, off, "clean vs JIT-off divergence"); assert_eq!(clean, on, "clean vs JIT-on divergence -> miscompile"); assert_eq!(off_compiles, 0, "JIT-off must never compile"); - assert!(on_compiles >= 1, "JIT-on must compile the hot loop"); + assert_eq!( + on_compiles, 1, + "JIT-on must compile the hot loop exactly once" + ); + // Ordered after the compile count so a tier that never ran fails on the + // count, which names the cause. Reaching this line means a body exists, + // so a false here is a real statement about that body — and the reset + // values above are the failing ones, so a hook that never fired cannot + // pass this by leaving the flags untouched. + // `{shape:?}` carries both `LoopBodyShape` fields. `why_not()`'s string is + // decoration: a rendered reason is a lossy encoding of a compound state, + // and what it loses is the discrimination — so the failure output must not + // depend on it alone. + assert!( + shape.closes_a_loop(), + "JIT-on compiled {on_compiles} loop(s) but the body {} ({shape:?}) — a \ + trace was minted for a dispatch that lowers nothing", + shape.why_not().unwrap_or("closes a loop") + ); println!( - "n = {n}, acc = {on} (clean==off==on ok), compiles: off={off_compiles} on={on_compiles}" + "n = {n}, acc = {on} (clean==off==on ok), compiles: off={off_compiles} on={on_compiles}, body closes a loop" ); +} + +pub fn run() { + let n: i64 = 20_000_000; + run_gates(n); + let prog = batch_program(n); // Interleaved A/B/C, several rounds (interleave per round to average drift). let rounds = 9; diff --git a/majit/examples/dualtape/src/jit_interp.rs b/majit/examples/dualtape/src/jit_interp.rs index 3405a1aaba6..48650d8243c 100644 --- a/majit/examples/dualtape/src/jit_interp.rs +++ b/majit/examples/dualtape/src/jit_interp.rs @@ -18,10 +18,11 @@ pub static LAST_OPS_AFTER: AtomicUsize = AtomicUsize::new(0); /// that gate). These two can: they are `LoopBodyShape`'s fields, recorded off /// the same hook. /// -/// One integer represents at least three states: `1` is an empty dispatch whose -/// whole body is `Finish()`, `5` is a segmented runaway, and a useful loop has -/// some other count. The count therefore reports a body's size, while these -/// flags report its shape; `COMPILES > 0` alone proves neither. +/// The count was once documented as sufficient on its own; the shape gate +/// refuted that. One integer represents at least three states: `1` is an empty +/// dispatch whose whole body is `Finish()`, `5` is a segmented runaway, and a +/// useful loop has some other count. The count therefore reports a body's size, +/// while these flags report its shape; `COMPILES > 0` alone proves neither. pub static LAST_HAS_JUMP: AtomicBool = AtomicBool::new(false); pub static LAST_ALWAYS_FAILS: AtomicBool = AtomicBool::new(false); diff --git a/majit/examples/spcount/src/main.rs b/majit/examples/spcount/src/main.rs index c16a9d396d2..3adb0a9cb4e 100644 --- a/majit/examples/spcount/src/main.rs +++ b/majit/examples/spcount/src/main.rs @@ -28,32 +28,6 @@ const TOUCH: u8 = 30; // residual: side-effecting, result-neutral stack touch #[cfg(test)] static TOUCH_CALLS: core::sync::atomic::AtomicU32 = core::sync::atomic::AtomicU32::new(0); -/// Number of loops the driver compiled/closed, observed by the tests. The -/// residual-count canary only exercises the single-pass close if a trace -/// actually compiled; this counter lets the test assert that it did, so a run -/// that never starts tracing (or aborts before the `; state` close) fails -/// loudly instead of passing vacuously. Mirrors tl's `SPIKE_COMPILES`. -static SPCOUNT_COMPILES: core::sync::atomic::AtomicU32 = core::sync::atomic::AtomicU32::new(0); - -/// Optimized op count of the most recently compiled loop body. -/// -/// A compile *count* says a trace closed; it does not say the trace did any -/// work. An entirely empty dispatch still compiles one trace whose whole -/// optimized body is `Finish()` — `ops_after == 1` — and that degenerate body -/// satisfies every inequality a real loop satisfies. `SPCOUNT_COMPILES` alone -/// therefore cannot tell a live tier from a hollow one, which is why the third -/// callback parameter is captured here instead of discarded. -static SPCOUNT_LAST_OPS_AFTER: core::sync::atomic::AtomicUsize = - core::sync::atomic::AtomicUsize::new(0); - -/// Loop-shape flags recorded with [`SPCOUNT_LAST_OPS_AFTER`]. They distinguish -/// a body that reaches its back edge from an empty or always-failing body -/// without relying only on a measured operation count. -static SPCOUNT_LAST_HAS_JUMP: core::sync::atomic::AtomicBool = - core::sync::atomic::AtomicBool::new(false); -static SPCOUNT_LAST_ALWAYS_FAILS: core::sync::atomic::AtomicBool = - core::sync::atomic::AtomicBool::new(false); - /// Side-effecting residual, `@dont_look_inside` — the JIT does not trace into /// it; it emits a residual CALL. `#[dont_look_inside]` is non-elidable and may /// raise, so the optimizer keeps the call. It is result-neutral (its only @@ -100,16 +74,12 @@ pub fn mainloop(program: &Bytecode, inputarg: i64, threshold: u32) -> i64 { // The third parameter is the optimized op count of the closed body. It is // captured rather than discarded because the count alone cannot separate a // real compiled loop from a bare `Finish()` — see SPCOUNT_LAST_OPS_AFTER. - driver.set_on_compile_loop(|_gk, _before, after, opcodes| { - SPCOUNT_COMPILES.fetch_add(1, core::sync::atomic::Ordering::Relaxed); - SPCOUNT_LAST_OPS_AFTER.store(after, core::sync::atomic::Ordering::Relaxed); - let shape = majit_metainterp::LoopBodyShape::of(opcodes); - SPCOUNT_LAST_HAS_JUMP.store(shape.has_jump, core::sync::atomic::Ordering::Relaxed); - SPCOUNT_LAST_ALWAYS_FAILS.store( - shape.has_always_fails, - core::sync::atomic::Ordering::Relaxed, - ); - }); + // The residual-count canary only exercises the single-pass close if a trace + // actually compiled, and a compile COUNT alone cannot say the trace did any + // work — an empty dispatch still compiles one body of a bare `Finish()`. The + // census carries both the count and the last body's op count and shape, so + // the canary can tell a live tier from a hollow one. + majit_metainterp::embed::Census::install(&mut driver); let mut pc: usize = 0; let stacksize: i32 = 0; let mut state = StackState { @@ -420,7 +390,8 @@ fn main() { mod tests { use super::*; use core::sync::atomic::Ordering; - use majit_metainterp::{RefusalKind, refusal_kind}; + use majit_metainterp::RefusalKind; + use majit_metainterp::embed::{self, Census}; /// Serializes the tier probe against every other test that runs the JIT. /// @@ -432,45 +403,32 @@ mod tests { /// can report another fixture's body size, which is a *plausible* number and /// therefore will not look wrong. /// - /// The lock only works if EVERY test that enters the JIT takes it, not + /// The window only works if EVERY test that enters the JIT opens one, not /// just the probe — a one-sided lock serializes nothing. This was not /// hypothetical: with `jit_output_matches_interp` still calling `mainloop` /// directly, the probe read `2 compile(s)` for a fixture that compiles /// exactly one, and could have pinned that test's 13-op body instead of this - /// one's 17. Hence [`run_jit`]. Neither helper may call the other — a plain - /// mutex re-entered on one thread deadlocks. - static PROBE_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(()); - - /// For tests that assert only on the result. They still compile, so they - /// must not run inside the probe's window. See [`PROBE_LOCK`]. + /// one's 17. Hence [`run_jit`]. Neither helper may call the other — + /// [`Census::begin`] is a plain mutex and re-entering it deadlocks. fn run_jit(program: &[u8], inputarg: i64) -> i64 { - let _guard = PROBE_LOCK.lock().unwrap_or_else(|e| e.into_inner()); + let _census = Census::begin(); mainloop(program, inputarg, 3) } + /// Run inside a census window, returning `(result, touches, counts)`. + /// + /// [`TOUCH_CALLS`] is this crate's own instrument and stays here: the census + /// counts what the JIT did, and a residual call the interpreter makes is not + /// that. It is reset inside the window all the same, so the two readings + /// describe the same run. fn compile_probe( program: &[u8], inputarg: i64, - ) -> (i64, u32, u32, usize, majit_metainterp::LoopBodyShape) { - let _guard = PROBE_LOCK.lock().unwrap_or_else(|e| e.into_inner()); + ) -> (i64, u32, majit_metainterp::embed::CensusCounts) { + let census = Census::begin(); TOUCH_CALLS.store(0, Ordering::Relaxed); - SPCOUNT_COMPILES.store(0, Ordering::Relaxed); - SPCOUNT_LAST_OPS_AFTER.store(0, Ordering::Relaxed); - // Reset to the values `closes_a_loop()` rejects, so a hook that never - // fires fails the shape assertion instead of passing it untouched. - SPCOUNT_LAST_HAS_JUMP.store(false, Ordering::Relaxed); - SPCOUNT_LAST_ALWAYS_FAILS.store(false, Ordering::Relaxed); let got = mainloop(program, inputarg, 3); - ( - got, - TOUCH_CALLS.load(Ordering::Relaxed), - SPCOUNT_COMPILES.load(Ordering::Relaxed), - SPCOUNT_LAST_OPS_AFTER.load(Ordering::Relaxed), - majit_metainterp::LoopBodyShape { - has_jump: SPCOUNT_LAST_HAS_JUMP.load(Ordering::Relaxed), - has_always_fails: SPCOUNT_LAST_ALWAYS_FAILS.load(Ordering::Relaxed), - }, - ) + (got, TOUCH_CALLS.load(Ordering::Relaxed), census.counts()) } /// The plain interpreter and the single-pass JIT mainloop must compute the @@ -493,7 +451,9 @@ mod tests { let n: i64 = 50; let expected = interp(&program, n); - let (got, jit_touches, compiles, ops_after, shape) = compile_probe(&program, n); + let (got, jit_touches, counts) = compile_probe(&program, n); + let (compiles, ops_after) = (counts.loops_compiled, counts.last_ops_after); + let shape = counts.last_loop_body_shape; // The residual-count canary is only meaningful if a trace actually // compiled and closed via the `; state` single-pass path. Without this @@ -527,39 +487,18 @@ mod tests { // be to weaken it. Pinning the set instead means a SECOND arm degrading // is a failure rather than a silent addition, and PUSHARG lowering again // is also a failure — the prompt to re-measure `ops_after` above. - let mut sp_arms: Vec<_> = majit_metainterp::degraded_dispatch_arms() - .into_iter() - .filter(|a| a.interp == "StackState") - .collect(); - sp_arms.sort_unstable_by_key(|a| a.arm); - let degraded: Vec<&str> = sp_arms.iter().map(|a| a.arm).collect(); - assert_eq!( - degraded, - ["PUSHARG"], - "the degraded-arm set moved; every trace reaching an abort stub aborts" - ); + embed::assert_degraded_dispatch_arms("StackState", &["PUSHARG"]); // The CAUSE, which the name set above cannot see: the comment on the // name pin says PUSHARG degrades because the lowerer cannot express the - // store of a loop-external input. That was prose; this asserts it. - let causes: Vec<(&str, RefusalKind)> = sp_arms - .iter() - .map(|a| (a.arm, refusal_kind(a.reason))) - .collect(); - assert_eq!( - causes, - [("PUSHARG", RefusalKind::UnlowerableStmt)], - "PUSHARG still degrades but a different mechanism is refusing it. \ - `RefusalKind::Unclassified` means majit grew a refusal family the \ - classifier does not know — add it in `majit-metainterp`, do not \ - re-record this pin" - ); - assert!( - sp_arms[0].reason.contains("inputarg"), - "PUSHARG's refusal no longer names the loop-external input it \ - stores: {}", - sp_arms[0].reason + // store of a loop-external input. That was prose; these assert it. + // `RefusalKind::Unclassified` here means majit grew a refusal family the + // classifier does not know — add it there, do not re-record this pin. + embed::assert_degraded_dispatch_arm_causes( + "StackState", + &[("PUSHARG", RefusalKind::UnlowerableStmt)], ); + embed::assert_degraded_dispatch_arm_reason_contains("StackState", "PUSHARG", "inputarg"); assert_eq!(got, expected, "JIT result diverged from interp"); // One TOUCH per iteration; N iterations before the counter hits 0. @@ -572,7 +511,8 @@ mod tests { ); println!( "[tier-alive] touch_loop({n}) = {got}, compiled {compiles} loop(s) of \ - {ops_after} ops, {jit_touches} residual calls, degraded {degraded:?}" + {ops_after} ops, {jit_touches} residual calls, degraded {:?}", + embed::degraded_dispatch_arm_names("StackState") ); } @@ -612,7 +552,8 @@ mod tests { #[ignore = "end-state gate: the outer loop does not trace once the inner loop is compiled"] fn nested_loops_compile_two_keys_in_one_run() { let program = nested_loop_program(3); - let (got, _touches, compiles, _ops_after, _shape) = compile_probe(&program, 8); + let (got, _touches, counts) = compile_probe(&program, 8); + let compiles = counts.loops_compiled; assert_eq!(got, 36, "sum(8) = 36"); assert!( compiles >= 2, diff --git a/majit/examples/tinyframe/src/jit_interp.rs b/majit/examples/tinyframe/src/jit_interp.rs index d93dfc6a183..65f51658a2f 100644 --- a/majit/examples/tinyframe/src/jit_interp.rs +++ b/majit/examples/tinyframe/src/jit_interp.rs @@ -3,32 +3,10 @@ /// Greens: [pc, bytecode] /// Reds: [regs] (tracked via state_fields) use crate::interp::{ADD, JUMP_IF_ABOVE, LOAD, RETURN}; -use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; +use majit_metainterp::embed::Census; pub type Bytecode = [u8]; -/// Hot loops majit compiled. The only positive evidence the JIT tier is alive: -/// a green suite, agreement with the plain interpreter and an exact trip count -/// are all satisfied by an interpreter answering alone. -pub static COMPILES: AtomicUsize = AtomicUsize::new(0); - -/// Ops in the last compiled loop body after optimization. -/// -/// `COMPILES > 0` is necessary but NOT sufficient: an entirely empty dispatch -/// still compiles a trace — one whose whole optimized body is `Finish()`, i.e. -/// `ops_after == 1`. A compile counter counts TRACES, not WORK. This is the -/// term that separates a compiled loop from a compiled nothing. -pub static LAST_OPS_AFTER: AtomicUsize = AtomicUsize::new(0); - -/// Shape of the last compiled loop body — see [`majit_metainterp::LoopBodyShape`]. -/// -/// Held as two flags rather than the struct itself so the recording stays -/// lock-free on the compile path; the probe rebuilds the struct inside the same -/// lock window it reads the counters in, because this is as process-global as -/// they are. -pub static LAST_HAS_JUMP: AtomicBool = AtomicBool::new(false); -pub static LAST_ALWAYS_FAILS: AtomicBool = AtomicBool::new(false); - #[expect( dead_code, reason = "the jit_interp macro resolves bytecode reads through this trait surface" @@ -73,13 +51,10 @@ fn mainloop( ) -> i64 { let mut driver: majit_metainterp::JitDriver = majit_metainterp::JitDriver::new(threshold); - driver.set_on_compile_loop(|_green_key, _ops_before, ops_after, opcodes| { - COMPILES.fetch_add(1, Ordering::Relaxed); - LAST_OPS_AFTER.store(ops_after, Ordering::Relaxed); - let shape = majit_metainterp::LoopBodyShape::of(opcodes); - LAST_HAS_JUMP.store(shape.has_jump, Ordering::Relaxed); - LAST_ALWAYS_FAILS.store(shape.has_always_fails, Ordering::Relaxed); - }); + // Every counter the tier gate reads, plus the last body's op count and + // shape, off the driver callbacks. `Census::begin` is what opens a window + // over them. + Census::install(&mut driver); let mut pc: usize = 0; let _stacksize: i32 = 0; let mut state = TinyFrameState { @@ -205,19 +180,15 @@ mod tests { use super::*; use crate::interp; - /// [`COMPILES`] is process-global, so under the default parallel libtest - /// runner a concurrent `run` lands inside [`compile_probe`]'s - /// store/run/load window and the probe reads someone else's compile. The - /// lock therefore covers *every* call that can compile, not just the - /// probe's own — [`run_jit`] and [`compile_probe`] are the only two ways a - /// test may enter the JIT, and neither may call the other (a plain mutex - /// re-entered on one thread deadlocks). - static PROBE_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(()); - /// For tests that assert only on the result. They still compile, so they - /// must not run inside the probe's window. See [`PROBE_LOCK`]. + /// must not run inside another test's census window — the window is what + /// makes [`compile_probe`]'s numbers this run's rather than the process's. + /// + /// [`Census::begin`] is the only lock here, and it is not reentrant: + /// [`run_jit`] and [`compile_probe`] are the only two ways a test may enter + /// the JIT, and neither may call the other. fn run_jit(code: &interp::Code, init_regs: &[(usize, i64)]) -> i64 { - let _guard = PROBE_LOCK.lock().unwrap_or_else(|e| e.into_inner()); + let _census = Census::begin(); JitTinyFrameInterp::new().run(code, init_regs) } @@ -225,41 +196,24 @@ mod tests { /// see the block at its assertion. `MC_DIAG` slot 73. const EXPECT_UNPEELED: u64 = 0; - /// Run with both counters reset, returning `(result, compiles, ops_after)`. + /// Run inside a census window, returning `(result, counts, unpeeled)`. /// - /// [`LAST_OPS_AFTER`] is read here rather than at the call site, and reset - /// here rather than nowhere. Both counters are process-global, so both need - /// the same treatment [`PROBE_LOCK`] exists to give [`COMPILES`]: a load - /// taken after the guard drops can observe a concurrent test's compile, and - /// a counter that is never stored to zero retains whatever the last compile - /// anywhere in the process left behind. Unreset, a zero from this probe is + /// The window is what separates this run's compiles from every other test's: + /// the counters behind it are process-global, so an absolute read carries + /// whatever the rest of the binary left behind, and a zero from it would be /// indistinguishable from an inherited value. fn compile_probe( code: &interp::Code, init_regs: &[(usize, i64)], - ) -> (i64, usize, usize, u64, majit_metainterp::LoopBodyShape) { - let _guard = PROBE_LOCK.lock().unwrap_or_else(|e| e.into_inner()); - COMPILES.store(0, Ordering::Relaxed); - LAST_OPS_AFTER.store(0, Ordering::Relaxed); - LAST_HAS_JUMP.store(false, Ordering::Relaxed); - LAST_ALWAYS_FAILS.store(false, Ordering::Relaxed); - // Slot 72 is process-global and CUMULATIVE, so it is read as a delta - // across this run and inside `PROBE_LOCK`, alongside the counters this - // probe resets. An absolute read would carry every other test's - // compiles — the inherited-value defect this probe already guards. + ) -> (i64, majit_metainterp::embed::CensusCounts, u64) { + let census = Census::begin(); + // Slot 73 is process-global and CUMULATIVE with no reset of its own, so + // it is read as a delta across this run and inside the census window, + // for the same reason the counters are. let unpeeled_before = majit_metainterp::mc_diag(73); let got = JitTinyFrameInterp::new().run(code, init_regs); let unpeeled = majit_metainterp::mc_diag(73) - unpeeled_before; - ( - got, - COMPILES.load(Ordering::Relaxed), - LAST_OPS_AFTER.load(Ordering::Relaxed), - unpeeled, - majit_metainterp::LoopBodyShape { - has_jump: LAST_HAS_JUMP.load(Ordering::Relaxed), - has_always_fails: LAST_ALWAYS_FAILS.load(Ordering::Relaxed), - }, - ) + (got, census.counts(), unpeeled) } /// A real loop body was compiled — the one property no assertion on a @@ -271,20 +225,20 @@ mod tests { /// /// All three parts are needed and none implies another: /// - /// 1. `COMPILES` 0 → non-zero. A green suite, agreement with - /// `interp::Frame::interpret` and even an exact absolute trip count are - /// all satisfied by the interpreter answering alone. - /// 2. `ops_after` pinned by equality. `compiles >= 1` is necessary but NOT - /// sufficient: an entirely empty dispatch still compiles a trace — one - /// whose whole optimized body is `Finish()`, i.e. `ops_after == 1`. A - /// compile counter counts TRACES, not WORK, and every inequality a real - /// loop satisfies that degenerate body satisfies too. - /// 3. `degraded_dispatch_arms()` empty. An arm whose body did not lower is - /// an abort stub, so any trace reaching it aborts. The list names the - /// arm, which an abort count cannot: `trace action at pc=N -> Abort` - /// reports the trace-START pc, not the arm that caused it. + /// 1. `loops_compiled` non-zero over the window. A green suite, agreement + /// with `interp::Frame::interpret` and even an exact absolute trip count + /// are all satisfied by the interpreter answering alone. + /// 2. `last_ops_after` pinned by equality. `compiles >= 1` is necessary but + /// NOT sufficient: an entirely empty dispatch still compiles a trace — + /// one whose whole optimized body is `Finish()`, i.e. `ops_after == 1`. + /// A compile counter counts TRACES, not WORK, and every inequality a + /// real loop satisfies that degenerate body satisfies too. + /// 3. No degraded dispatch arm. An arm whose body did not lower is an abort + /// stub, so any trace reaching it aborts. The assertion names the arm, + /// which an abort count cannot: `trace action at pc=N -> Abort` reports + /// the trace-START pc, not the arm that caused it. /// - /// The registry is process-wide, so it is filtered to this machine's + /// The arm registry is process-wide, so it is asked about this machine's /// `state = TinyFrameState`, and read *after* a run because nothing /// installs the dispatch JitCode until the interpreter is entered. /// @@ -306,7 +260,9 @@ mod tests { RETURN r0 ", ); - let (got, compiles, ops_after, unpeeled, shape) = compile_probe(&code, &[(2, N)]); + let (got, counts, unpeeled) = compile_probe(&code, &[(2, N)]); + let (compiles, ops_after) = (counts.loops_compiled, counts.last_ops_after); + let shape = counts.last_loop_body_shape; // `unpeeled` counts loops this run compiled through the unroll-free // fallback: the unrolled compile raised `InvalidLoop`, and the retry // WITHOUT the peel succeeded. The retry succeeding is what makes a @@ -344,16 +300,10 @@ mod tests { describe ran {got} passes rather than {N}" ); - let degraded: Vec<&str> = majit_metainterp::degraded_dispatch_arms() - .iter() - .filter(|a| a.interp == "TinyFrameState") - .map(|a| a.arm) - .collect(); - assert!( - degraded.is_empty(), - "dispatch arms degraded to abort stubs: {degraded:?} — every trace \ - reaching one aborts" - ); + // Stronger than filtering `degraded_dispatch_arms()` by hand: an empty + // list is also what a portal that was never installed produces, and + // this settles which of the two happened. + majit_metainterp::assert_no_degraded_dispatch_arms("TinyFrameState"); // Zero-vs-nonzero is the property; a later change that legitimately // mints more than one artifact is not this regression. @@ -434,7 +384,7 @@ mod tests { #[test] fn jit_trace_reads_input_written_after_the_arming_pc() { - let _guard = PROBE_LOCK.lock().unwrap_or_else(|e| e.into_inner()); + let _census = Census::begin(); let code = straight_line_program(); // Threshold 1: the entry door arms on its single hit at pc 0. let got = mainloop(&code.code, code.regno, &[], 1); diff --git a/majit/examples/tla/src/jit_interp.rs b/majit/examples/tla/src/jit_interp.rs index 06e6c988919..297e19080bf 100644 --- a/majit/examples/tla/src/jit_interp.rs +++ b/majit/examples/tla/src/jit_interp.rs @@ -12,33 +12,10 @@ /// trace of the countdown loop aborts and nothing is ever compiled. Declaring /// the greens is also what gives the merge point a green pc to report, which the /// `; state` close needs to name a resume position. -use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; +use majit_metainterp::embed::Census; pub type Bytecode = [u8]; -/// Hot loops majit compiled. The only positive evidence the JIT tier is alive: -/// a green suite, byte-identical output and an exact trip count are all -/// satisfied by an interpreter answering alone. Before `greens = [pc, program]` -/// was declared this example ran its whole suite green at `Traces compiled: 0`. -pub static COMPILES: AtomicUsize = AtomicUsize::new(0); - -/// Ops in the last compiled loop body after optimization. -/// -/// `COMPILES > 0` is necessary but NOT sufficient: an entirely empty dispatch -/// still compiles a trace — one whose whole optimized body is `Finish()`, i.e. -/// `ops_after == 1`. A compile counter counts TRACES, not WORK. This is the -/// term that separates a compiled loop from a compiled nothing. -pub static LAST_OPS_AFTER: AtomicUsize = AtomicUsize::new(0); - -/// Shape of the last compiled loop body — see [`majit_metainterp::LoopBodyShape`]. -/// -/// Held as two flags rather than the struct itself so the recording stays -/// lock-free on the compile path; the probe rebuilds the struct inside the same -/// lock window it reads the counters in, because this is as process-global as -/// they are. -pub static LAST_HAS_JUMP: AtomicBool = AtomicBool::new(false); -pub static LAST_ALWAYS_FAILS: AtomicBool = AtomicBool::new(false); - #[expect( dead_code, reason = "the jit_interp macro resolves bytecode reads through this trait surface" @@ -87,13 +64,9 @@ const NEWSTR: u8 = 7; pub fn mainloop(program: &Bytecode, initial_value: i64, threshold: u32) -> i64 { let mut driver: majit_metainterp::JitDriver = majit_metainterp::JitDriver::new(threshold); - driver.set_on_compile_loop(|_green_key, _ops_before, ops_after, opcodes| { - COMPILES.fetch_add(1, Ordering::Relaxed); - LAST_OPS_AFTER.store(ops_after, Ordering::Relaxed); - let shape = majit_metainterp::LoopBodyShape::of(opcodes); - LAST_HAS_JUMP.store(shape.has_jump, Ordering::Relaxed); - LAST_ALWAYS_FAILS.store(shape.has_always_fails, Ordering::Relaxed); - }); + // Every counter the tier gate reads, plus the last body's op count and + // shape, off the driver callbacks. `Census::begin` opens a window over them. + Census::install(&mut driver); let mut pc: usize = 0; let stacksize: i32 = 0; let mut state = TlaState { @@ -220,49 +193,30 @@ mod tests { vec![DUP, CONST_INT, 1, SUB, DUP, JUMP_IF, 1, POP, RETURN] } - /// [`COMPILES`] is process-global, so under the default parallel libtest - /// runner a concurrent `run` lands inside [`compile_probe`]'s - /// store/run/load window and the probe reads someone else's compile. The - /// lock therefore covers *every* call that can compile, not just the - /// probe's own — [`run_jit`] and [`compile_probe`] are the only two ways a - /// test may enter the JIT, and neither may call the other (a plain mutex - /// re-entered on one thread deadlocks). - static PROBE_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(()); - /// For tests that assert only on the result. They still compile, so they - /// must not run inside the probe's window. See [`PROBE_LOCK`]. + /// must not run inside another test's census window — the window is what + /// makes [`compile_probe`]'s numbers this run's rather than the process's. + /// + /// [`Census::begin`] is the only lock here, and it is not reentrant: + /// [`run_jit`] and [`compile_probe`] are the only two ways a test may enter + /// the JIT, and neither may call the other. fn run_jit(bc: &[u8], arg: i64) -> i64 { - let _guard = PROBE_LOCK.lock().unwrap_or_else(|e| e.into_inner()); + let _census = Census::begin(); let mut jit = JitTlaInterp::new(); jit.run(bc, interp::WObject::Int(arg)).int_value() } - /// Run with both counters reset, returning `(result, compiles, ops_after)`. + /// Run inside a census window, returning `(result, counts)`. /// - /// [`LAST_OPS_AFTER`] is read here rather than at the call site, and reset - /// here rather than nowhere. Both counters are process-global, so both need - /// the same treatment [`PROBE_LOCK`] exists to give [`COMPILES`]: a load - /// taken after the guard drops can observe a concurrent test's compile, and - /// a counter that is never stored to zero retains whatever the last compile - /// anywhere in the process left behind. Unreset, a zero from this probe is + /// The window is what separates this run's compiles from every other test's: + /// the counters behind it are process-global, so an absolute read carries + /// whatever the rest of the binary left behind, and a zero from it would be /// indistinguishable from an inherited value. - fn compile_probe(bc: &[u8], arg: i64) -> (i64, usize, usize, majit_metainterp::LoopBodyShape) { - let _guard = PROBE_LOCK.lock().unwrap_or_else(|e| e.into_inner()); - COMPILES.store(0, Ordering::Relaxed); - LAST_OPS_AFTER.store(0, Ordering::Relaxed); - LAST_HAS_JUMP.store(false, Ordering::Relaxed); - LAST_ALWAYS_FAILS.store(false, Ordering::Relaxed); + fn compile_probe(bc: &[u8], arg: i64) -> (i64, majit_metainterp::embed::CensusCounts) { + let census = Census::begin(); let mut jit = JitTlaInterp::new(); let got = jit.run(bc, interp::WObject::Int(arg)).int_value(); - ( - got, - COMPILES.load(Ordering::Relaxed), - LAST_OPS_AFTER.load(Ordering::Relaxed), - majit_metainterp::LoopBodyShape { - has_jump: LAST_HAS_JUMP.load(Ordering::Relaxed), - has_always_fails: LAST_ALWAYS_FAILS.load(Ordering::Relaxed), - }, - ) + (got, census.counts()) } #[test] @@ -294,15 +248,15 @@ mod tests { /// and even an exact absolute trip count are all satisfied by the /// interpreter answering alone; this example ran its entire suite green /// at `Traces compiled: 0` before `greens = [pc, program]` was declared. - /// 2. `degraded_dispatch_arms()` empty. An arm whose body did not lower is - /// an abort stub, so any trace reaching it aborts. The list is populated - /// at dispatch-JitCode install time and names the arm, which an abort - /// count cannot: `trace action at pc=N -> Abort` reports the trace-START - /// pc, not the arm that caused it. + /// 2. No degraded dispatch arm. An arm whose body did not lower is an abort + /// stub, so any trace reaching it aborts. The registry is populated at + /// dispatch-JitCode install time and names the arm, which an abort count + /// cannot: `trace action at pc=N -> Abort` reports the trace-START pc, + /// not the arm that caused it. /// - /// The list is a process-wide registry, so it is filtered to this machine's - /// `state = TlaState`. It is read *after* a run, because nothing installs - /// the dispatch JitCode until the interpreter is entered. + /// The registry is process-wide, so the assertion is asked about this + /// machine's `state = TlaState`. It runs *after* a run, because nothing + /// installs the dispatch JitCode until the interpreter is entered. /// /// The subject is `count_to`, not `countdown`, so that the *result* assertion /// is itself an absolute trip count: `count_to(n)` returns the number of @@ -313,7 +267,9 @@ mod tests { #[test] fn jit_tier_is_alive() { const N: i64 = 1001; - let (got, compiles, ops_after, shape) = compile_probe(&count_to_bytecode(N), 0); + let (got, counts) = compile_probe(&count_to_bytecode(N), 0); + let (compiles, ops_after) = (counts.loops_compiled, counts.last_ops_after); + let shape = counts.last_loop_body_shape; // The body actually closes a loop — see `LoopBodyShape`. A compile // count and an op count together still accept a body that bails out on // its first pass; this is the term that does not. Sound HERE because @@ -330,22 +286,14 @@ mod tests { describe ran {got} passes rather than {N}" ); - let degraded: Vec<&str> = majit_metainterp::degraded_dispatch_arms() - .iter() - .filter(|a| a.interp == "TlaState") - .map(|a| a.arm) - .collect(); // `greens = [pc, program]` means a degraded arm aborts only traces that - // reach that arm, rather than disabling the whole dispatch loop. This - // assertion pins the stronger property that every exercised arm lowers. - assert!( - degraded.is_empty(), - "dispatch arms degraded to abort stubs: {degraded:?} — every trace \ - that reaches one aborts. With `greens = [pc, program]` declared, \ - that costs the traces reaching those arms, not the dispatch loop \ - as a whole; the arms this crate exercises are still expected to \ - lower, so a non-empty set is a regression and not a trade-off" - ); + // reach that arm, rather than disabling the whole dispatch loop. The + // empty pin is still the right one here: the arms this crate exercises + // are all expected to lower, so a non-empty set is a regression and not + // a trade-off. The pin also carries the denominator, which a bare + // emptiness check over the registry cannot — an uninstalled portal + // produces the same empty list a healthy one does. + majit_metainterp::embed::assert_degraded_dispatch_arms("TlaState", &[]); // Zero-vs-nonzero is the property; a later change that legitimately // mints more than one artifact is not this regression. diff --git a/majit/examples/tlr/src/jit_interp.rs b/majit/examples/tlr/src/jit_interp.rs index 8835e3d77c1..12c6fa50398 100644 --- a/majit/examples/tlr/src/jit_interp.rs +++ b/majit/examples/tlr/src/jit_interp.rs @@ -4,32 +4,10 @@ /// /// Greens: [pc, bytecode] /// Reds: [a, regs] (tracked via state_fields) -use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; +use majit_metainterp::embed::Census; pub type Bytecode = [u8]; -/// Hot loops majit compiled. The only positive evidence the JIT tier is alive: -/// a green suite, agreement with `interp::interpret` and an exact result are -/// all satisfied by an interpreter answering alone. -pub static COMPILES: AtomicUsize = AtomicUsize::new(0); - -/// Ops in the last compiled loop body after optimization. -/// -/// `COMPILES > 0` is necessary but NOT sufficient: an entirely empty dispatch -/// still compiles a trace — one whose whole optimized body is `Finish()`, i.e. -/// `ops_after == 1`. A compile counter counts TRACES, not WORK. This is the -/// term that separates a compiled loop from a compiled nothing. -pub static LAST_OPS_AFTER: AtomicUsize = AtomicUsize::new(0); - -/// Shape of the last compiled loop body — see [`majit_metainterp::LoopBodyShape`]. -/// -/// Held as two flags rather than the struct itself so the recording stays -/// lock-free on the compile path; the probe rebuilds the struct inside the same -/// lock window it reads the counters in, because this is as process-global as -/// they are. -pub static LAST_HAS_JUMP: AtomicBool = AtomicBool::new(false); -pub static LAST_ALWAYS_FAILS: AtomicBool = AtomicBool::new(false); - #[expect( dead_code, reason = "the jit_interp macro resolves bytecode reads through this trait surface" @@ -90,13 +68,9 @@ const DEFAULT_THRESHOLD: u32 = 3; fn mainloop(program: &Bytecode, initial_a: i64, threshold: u32) -> i64 { let mut driver: majit_metainterp::JitDriver = majit_metainterp::JitDriver::new(threshold); - driver.set_on_compile_loop(|_green_key, _ops_before, ops_after, opcodes| { - COMPILES.fetch_add(1, Ordering::Relaxed); - LAST_OPS_AFTER.store(ops_after, Ordering::Relaxed); - let shape = majit_metainterp::LoopBodyShape::of(opcodes); - LAST_HAS_JUMP.store(shape.has_jump, Ordering::Relaxed); - LAST_ALWAYS_FAILS.store(shape.has_always_fails, Ordering::Relaxed); - }); + // Every counter the tier gate reads, plus the last body's op count and + // shape, off the driver callbacks. `Census::begin` opens a window over them. + Census::install(&mut driver); let mut pc: usize = 0; let _stacksize: i32 = 0; let mut state = TlrState { @@ -207,7 +181,7 @@ impl JitTlrInterp { mod tests { use super::*; use crate::interp; - use majit_metainterp::{RefusalKind, refusal_kind}; + use majit_metainterp::{RefusalKind, embed}; fn square_bytecode() -> Vec { vec![ @@ -266,7 +240,9 @@ mod tests { const PASSES: i64 = 20; let narrow_bc = imm_loop_bytecode(false); - let (narrow_got, narrow_compiles, narrow_ops, ..) = compile_probe(&narrow_bc, PASSES); + let (narrow_got, narrow_counts) = compile_probe(&narrow_bc, PASSES); + let (narrow_compiles, narrow_ops) = + (narrow_counts.loops_compiled, narrow_counts.last_ops_after); assert_eq!( narrow_got, NARROW * PASSES, @@ -280,7 +256,8 @@ mod tests { ); let wide_bc = imm_loop_bytecode(true); - let (wide_got, wide_compiles, wide_ops, ..) = compile_probe(&wide_bc, PASSES); + let (wide_got, wide_counts) = compile_probe(&wide_bc, PASSES); + let (wide_compiles, wide_ops) = (wide_counts.loops_compiled, wide_counts.last_ops_after); // Decoding, not just compiling: a dropped high byte yields 2*PASSES // instead of 258*PASSES, and a dropped statement yields 0. assert_eq!( @@ -348,7 +325,9 @@ mod tests { const PASSES: i64 = 20; let control_bc = realloc_loop_bytecode(false); - let (control_got, control_compiles, control_ops, ..) = compile_probe(&control_bc, PASSES); + let (control_got, control_counts) = compile_probe(&control_bc, PASSES); + let (control_compiles, control_ops) = + (control_counts.loops_compiled, control_counts.last_ops_after); assert_eq!(control_got, interp::interpret(&control_bc, PASSES)); assert!( control_compiles >= 1, @@ -361,7 +340,8 @@ mod tests { // catches a stale-mirror miscompile, and it must not be guarded by a // compile count that a miscompiling build would still satisfy. let expected = interp::interpret(&bc, PASSES); - let (got, compiles, ops_after, ..) = compile_probe(&bc, PASSES); + let (got, counts) = compile_probe(&bc, PASSES); + let (compiles, ops_after) = (counts.loops_compiled, counts.last_ops_after); assert_eq!( got, expected, "realloc-in-loop disagrees with the interpreter ({got} vs \ @@ -387,50 +367,28 @@ mod tests { ); } - /// [`COMPILES`] is process-global, so under the default parallel libtest - /// runner a concurrent `run` lands inside [`compile_probe`]'s - /// store/run/load window and the probe reads someone else's compile. The - /// lock therefore covers *every* call that can compile, not just the - /// probe's own — [`run_jit`] and [`compile_probe`] are the only two ways a - /// test may enter the JIT, and neither may call the other (a plain mutex - /// re-entered on one thread deadlocks). - static PROBE_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(()); - /// For tests that assert only on the result. They still compile, so they - /// must not run inside the probe's window. See [`PROBE_LOCK`]. + /// must not run inside another test's census window — the window is what + /// makes [`compile_probe`]'s numbers this run's rather than the process's. + /// + /// [`Census::begin`] is the only lock here, and it is not reentrant: + /// [`run_jit`] and [`compile_probe`] are the only two ways a test may enter + /// the JIT, and neither may call the other. fn run_jit(bc: &[u8], initial_a: i64) -> i64 { - let _guard = PROBE_LOCK.lock().unwrap_or_else(|e| e.into_inner()); + let _census = Census::begin(); JitTlrInterp::new().run(bc, initial_a) } - /// Run with both counters reset, returning `(result, compiles, ops_after)`. + /// Run inside a census window, returning `(result, counts)`. /// - /// [`LAST_OPS_AFTER`] is read here rather than at the call site, and reset - /// here rather than nowhere. Both counters are process-global, so both need - /// the same treatment [`PROBE_LOCK`] exists to give [`COMPILES`]: a load - /// taken after the guard drops can observe a concurrent test's compile, and - /// a counter that is never stored to zero retains whatever the last compile - /// anywhere in the process left behind. Unreset, a zero from this probe is + /// The window is what separates this run's compiles from every other test's: + /// the counters behind it are process-global, so an absolute read carries + /// whatever the rest of the binary left behind, and a zero from it would be /// indistinguishable from an inherited value. - fn compile_probe( - bc: &[u8], - initial_a: i64, - ) -> (i64, usize, usize, majit_metainterp::LoopBodyShape) { - let _guard = PROBE_LOCK.lock().unwrap_or_else(|e| e.into_inner()); - COMPILES.store(0, Ordering::Relaxed); - LAST_OPS_AFTER.store(0, Ordering::Relaxed); - LAST_HAS_JUMP.store(false, Ordering::Relaxed); - LAST_ALWAYS_FAILS.store(false, Ordering::Relaxed); + fn compile_probe(bc: &[u8], initial_a: i64) -> (i64, majit_metainterp::embed::CensusCounts) { + let census = Census::begin(); let got = JitTlrInterp::new().run(bc, initial_a); - ( - got, - COMPILES.load(Ordering::Relaxed), - LAST_OPS_AFTER.load(Ordering::Relaxed), - majit_metainterp::LoopBodyShape { - has_jump: LAST_HAS_JUMP.load(Ordering::Relaxed), - has_always_fails: LAST_ALWAYS_FAILS.load(Ordering::Relaxed), - }, - ) + (got, census.counts()) } /// A real loop body was compiled, and exactly one known arm is degraded. @@ -463,7 +421,9 @@ mod tests { /// until the interpreter is entered. #[test] fn jit_tier_is_alive() { - let (got, compiles, ops_after, shape) = compile_probe(&square_bytecode(), 100); + let (got, counts) = compile_probe(&square_bytecode(), 100); + let (compiles, ops_after) = (counts.loops_compiled, counts.last_ops_after); + let shape = counts.last_loop_body_shape; // The body actually closes a loop — see `LoopBodyShape`. A compile // count and an op count together still accept a body that bails out on // its first pass; this is the term that does not. Sound HERE because @@ -476,37 +436,18 @@ mod tests { ); assert_eq!(got, 10_000, "square(100) must still answer 10000"); - let tlr_arms: Vec<_> = majit_metainterp::degraded_dispatch_arms() - .into_iter() - .filter(|a| a.interp == "TlrState") - .collect(); - let degraded: Vec<&str> = tlr_arms.iter().map(|a| a.arm).collect(); - assert_eq!( - degraded, - ["ALLOCATE"], - "the degraded-arm set moved. A NEW name means an arm silently \ - stopped lowering and every trace reaching it now aborts; a MISSING \ - name means that arm lowers again" - ); - - let causes: Vec<(&str, RefusalKind)> = tlr_arms - .iter() - .map(|a| (a.arm, refusal_kind(a.reason))) - .collect(); - assert_eq!( - causes, - [("ALLOCATE", RefusalKind::GreenWriteback)], - "ALLOCATE still degrades but a different mechanism is refusing it. \ - `RefusalKind::Unclassified` means majit grew a refusal family the \ - classifier does not know — add it in `majit-metainterp`, do not \ - re-record this pin" - ); - assert!( - tlr_arms[0].reason.contains("pc += 1"), - "ALLOCATE's refusal no longer names the green write that stops \ - lowering before the reallocation: {}", - tlr_arms[0].reason + // The set, then the mechanism refusing it, then the source it refuses + // on. Each is a different fact and each fails with its own message: + // `RefusalKind::Unclassified` at the second means majit grew a refusal + // family the classifier does not know — add it there, do not re-record + // this pin. + embed::assert_degraded_dispatch_arms("TlrState", &["ALLOCATE"]); + embed::assert_degraded_dispatch_arm_causes( + "TlrState", + &[("ALLOCATE", RefusalKind::GreenWriteback)], ); + // The green write that stops lowering before the reallocation. + embed::assert_degraded_dispatch_arm_reason_contains("TlrState", "ALLOCATE", "pc += 1"); // Zero-vs-nonzero is the property; a later change that legitimately // mints more than one artifact is not this regression. @@ -523,7 +464,9 @@ mod tests { lowered nothing at all" ); println!( - "[tier-alive] square(100) = {got}, compiled {compiles} loop(s) of {ops_after} ops, degraded {degraded:?}" + "[tier-alive] square(100) = {got}, compiled {compiles} loop(s) of \ + {ops_after} ops, degraded {:?}", + embed::degraded_dispatch_arm_names("TlrState") ); } diff --git a/majit/gate-triage.md b/majit/gate-triage.md index 6753d3d2592..05e997f308b 100644 --- a/majit/gate-triage.md +++ b/majit/gate-triage.md @@ -76,6 +76,13 @@ Each entry records its reader, purpose, and retirement condition. `UNRECORDED` m - What it does: **UNRECORDED** — no doc comment at the read site. - Retirement condition: **UNRECORDED** — owed by this gate's owner. +### `MAJIT_DECLINE_LOG` + +- Read sites: 1 — `majit/majit-translate/src/decline.rs` +- Accessor: `level()` +- What it does: Census of the lowering gates' silent declines. Unset, `0`, or empty disables it; any other value counts declines per (gate, reason) and prints runtime reasons; `2` additionally prints one line per decline event. `PYRE_MIR_FRONTEND_DEBUG` is accepted as an alias at the counter level. +- Retirement condition: retire when the decline counts are no longer needed to steer cel lowering coverage. + ### `MAJIT_DIAG` - Read sites: 1 — `majit/majit-metainterp/src/lib.rs` @@ -99,8 +106,8 @@ Each entry records its reader, purpose, and retirement condition. `UNRECORDED` m ### `MAJIT_DUMP_CLIF` -- Read sites: 2 — `majit/majit-backend-cranelift/src/compiler.rs` -- Accessor: read inline in `do_compile()`, at both sites +- Read sites: 4 — `majit/majit-backend-cranelift/src/compiler.rs` +- Accessor: read inline in `do_compile()`, at all four sites — two for the trace body, two for the host-callable entry wrapper - What it does: **UNRECORDED** — no doc comment at the read site. - Retirement condition: **UNRECORDED** — owed by this gate's owner. @@ -129,7 +136,7 @@ Each entry records its reader, purpose, and retirement condition. `UNRECORDED` m - Read sites: 2 — `majit/majit-ir/src/descr.rs`, `pyre/pyre-jit-trace/build.rs` - Accessor: `field_mint_trace_enabled()`; the build script also declares it as a rerun input and bypasses its code-generation cache while enabled -- What it does: Setting it to `1` prints descriptor-mint disagreements and keeps the analyzer live so those diagnostics cannot be hidden by a restored artifact cache. Unset is inert. +- What it does: Setting it to `1` prints field-descriptor mint disagreements (`cache_hit_disagree`, `ei_descr_mint_disagree`) and keeps the analyzer live so those diagnostics cannot be hidden by a restored artifact cache. Unset is inert. - Retirement condition: Remove when field and size descriptor identity no longer has fallback or disagreement paths to diagnose. ### `MAJIT_FIELD_POS_UNRESOLVED` @@ -202,6 +209,13 @@ Each entry records its reader, purpose, and retirement condition. `UNRECORDED` m - What it does: Whether `MAJIT_J2PLAN_LOG` is set, cached at first access. - Retirement condition: **UNRECORDED** — owed by this gate's owner. +### `MAJIT_JITFRAME_POOL` + +- Read sites: 1 — `majit/majit-backend/src/deadframe.rs` +- Accessor: `seed_jitframe_pool_arm()`, behind `jitframe_pool_enabled()` +- What it does: Selects which arm allocates the jitframe a compiled entry runs on, for backends that build frames out of the Rust heap rather than the GC nursery. `0` selects `FrameHeapOwner::OWNED`, one `calloc`/`free` pair per entry; anything else, including leaving it unset, selects the pooled per-thread free list. Read once and latched, so it names a strategy for the process rather than a per-entry state; `set_jitframe_pool` overrides it for a harness that can call in. +- Retirement condition: when the owned arm is retired — it exists to be differenced against the pooled one, and a build with no second arm has nothing to select. + ### `MAJIT_LEAF3_PROV` - Read sites: 1 — `majit/majit-metainterp/src/resume.rs` @@ -361,7 +375,7 @@ Each entry records its reader, purpose, and retirement condition. `UNRECORDED` m - Read sites: 2 — `majit/majit-translate/src/lib.rs`, `pyre/pyre-jit-trace/build.rs` - Accessor: `struct_layout_census_enabled()`; the build script also declares it as a rerun input and bypasses its code-generation cache while enabled -- What it does: Setting it to `1` reports structure IDs that resolve to multiple spellings or conflicting concrete layouts during translation. Unset is inert. +- What it does: Setting it to `1` reports structure IDs that resolve to multiple spellings or conflicting concrete layouts during translation, as `conflict`, `variant` and `summary` lines. Unset is inert. - Retirement condition: Remove when one structure ID cannot collect conflicting layouts by construction. ### `MAJIT_TLDBG` diff --git a/majit/majit-backend-cranelift/Cargo.toml b/majit/majit-backend-cranelift/Cargo.toml index 84c52f420d4..315d0753b98 100644 --- a/majit/majit-backend-cranelift/Cargo.toml +++ b/majit/majit-backend-cranelift/Cargo.toml @@ -6,6 +6,14 @@ license.workspace = true repository.workspace = true description = "Cranelift code generation backend for majit JIT compiler" +[features] +# A measurement arm, never a shipping one. It adds one run-time selectable loop +# to `run_compiled_code_inner` that repeats the frame allocation and the input +# argument writes into a frame nothing enters, so the repeatable PREFIX of a +# compiled entry's call can be priced against the call it prefixes. Off by +# default; the count it reads lives in `majit_backend::deadframe`. +execute-stage-probe = [] + [dependencies] indexmap = { workspace = true } majit-ir = { workspace = true } diff --git a/majit/majit-backend-cranelift/src/compiler.rs b/majit/majit-backend-cranelift/src/compiler.rs index 5e219b03007..86151274cb4 100644 --- a/majit/majit-backend-cranelift/src/compiler.rs +++ b/majit/majit-backend-cranelift/src/compiler.rs @@ -278,6 +278,7 @@ fn match_metainterp_finish_descr( // JitFrame layout constants (`jitframe.py:61-83`) // The canonical layout lives in `majit_backend::jitframe`; re-export the // byte offsets here so uses inside this file stay terse. +use majit_backend::deadframe::{FrameHeapOwner, jitframe_pool_enabled}; use majit_backend::jitframe::{ BASEITEMOFS, JF_DESCR_OFS, JF_FORCE_DESCR_OFS, JF_FORWARD_OFS, JF_FRAME_OFS, JF_GCMAP_OFS, JF_GUARD_EXC_OFS, JF_SAVEDATA_OFS, @@ -3176,7 +3177,7 @@ pub fn force_token_to_dead_frame(force_token: GcRef) -> DeadFrame { fn deadframe_from_jitframe( jf_gcref: GcRef, fail_descr: DescrRef, - heap_owner: Option>, + heap_owner: Option, ) -> DeadFrame { DeadFrame::JitFrame(JitFrameDeadFrame::new( jf_gcref, fail_descr, None, heap_owner, @@ -7947,7 +7948,7 @@ impl FrameInputs<'_> { /// the JitFrame GcRef is returned directly. Values stay in place. struct JitExecResult { jf_gcref: GcRef, - heap_owner: Option>, + heap_owner: Option, fail_index: u32, direct_descr: Option, } @@ -8065,7 +8066,7 @@ fn run_compiled_code_inner( // SSA/ref_root_slots afterward. let runtime_jitframe_tid = cranelift_jitframe_type_id(); let use_gc_alloc = runtime_jitframe_tid.is_some(); - let (jf_gcref, heap_owner): (GcRef, Option>) = if use_gc_alloc { + let (jf_gcref, heap_owner): (GcRef, Option) = if use_gc_alloc { let type_id = runtime_jitframe_tid.unwrap(); let gcref = with_cranelift_gc_required(|gc| { gc.alloc_nursery_no_collect_typed(type_id, payload_bytes) @@ -8087,7 +8088,11 @@ fn run_compiled_code_inner( // pointer, so without it that load reads outside the buffer. const HEADER_WORDS: usize = majit_gc::header::GcHeader::SIZE / 8; const _: () = assert!(HEADER_WORDS * 8 == majit_gc::header::GcHeader::SIZE); - let mut buf = vec![0i64; HEADER_WORDS + jf_total]; + // Off the per-thread free list by default, or one `calloc`/`free` pair + // per compiled entry if the owned arm was selected — `FrameHeapOwner` + // is where the two differ, and the buffer it hands back is zeroed to + // `words` either way. + let mut buf = FrameHeapOwner::new(HEADER_WORDS + jf_total, jitframe_pool_enabled()); let gcref = GcRef(unsafe { buf.as_mut_ptr().add(HEADER_WORDS) } as usize); // jitframe.py:84 parity — `jf_frame.length` is the count of `Signed` // payload slots after the length word. The GC-alloc branch above @@ -8104,6 +8109,30 @@ fn run_compiled_code_inner( // llmodel.py:306-315: set arguments in frame unsafe { inputs.write_into(jf_ptr.add(header_words)) }; + // The one repeatable part of a compiled entry's call. Everything past this + // point runs the trace and cannot be made to happen twice, but allocating + // the frame and writing the arguments into it produces a frame NOTHING has + // entered, so it can be done N more times and thrown away. Scoped to the + // Rust-heap arm: the nursery arm would be allocating collectable frames + // that no root names, which is a different question and an unsafe way to + // ask it. Each pass drops its frame before the next, so the pool cycles one + // buffer and the loop prices a build rather than a pool miss. + #[cfg(feature = "execute-stage-probe")] + if !use_gc_alloc { + let repeats = majit_backend::deadframe::frame_build_repeats(); + majit_backend::deadframe::count_frame_build_passes(repeats); + for _ in 0..repeats { + const HEADER_WORDS: usize = majit_gc::header::GcHeader::SIZE / 8; + let mut scratch = FrameHeapOwner::new(HEADER_WORDS + jf_total, jitframe_pool_enabled()); + unsafe { + let base = scratch.as_mut_ptr().add(HEADER_WORDS); + *((base as usize + JF_FRAME_LENGTH_OFS as usize) as *mut usize) = frame_depth; + inputs.write_into(base.add(header_words)); + } + // Without this the writes are dead and the loop prices nothing. + std::hint::black_box(&mut scratch); + } + } if majit_ir::debug::have_debug_prints() { let preview: Vec = (0..inputs.len().min(10)).map(|i| inputs.raw(i)).collect(); majit_ir::debug::log_one("jit-running", &format!("pre-call-inputs {preview:?}")); @@ -9378,13 +9407,25 @@ impl CraneliftBackend { let call_conv = self.module.target_config().default_call_conv; // The body uses CallConv::Tail so it can `return_call_indirect` to // another body's entry — `assembler.py closing_jump` raw - // `JMP imm(target)` parity. The host (Rust) caller cannot speak - // Tail ABI (it clobbers AppleAarch64 callee-saves x19-x28/x29), so a - // separate `trace_N_entry` wrapper carrying `default_call_conv` is - // declared alongside the body and forwards the jitframe pointer to - // the body via `call_indirect`. Wrapper is what host code calls + // `JMP imm(target)` parity. A separate `trace_N_entry` wrapper + // carrying `default_call_conv` is declared alongside it and forwards + // the jitframe pointer via `call_indirect`, because the pinned + // register `enable_pinned_reg` hands the JIT is NOT callee-saved in + // Cranelift and IS callee-saved under AAPCS: something has to park the + // host's value across the run. Wrapper is what host code calls // (`CompiledLoop.code_ptr` / `LoopTargetEntry.code_ptr`); body is // what in-code dispatch tail-calls (`LoopTargetDescr.ll_loop_code`). + // + // Not because Tail trashes the callee-saved set: `get_regs_clobbered_by_call` + // gives `(Tail, true)` — the EXCEPTION path, which emits no exception + // tables here — `ALL_CLOBBERS`, while an ordinary Tail call falls to + // `DEFAULT_AAPCS_CLOBBERS` and preserves x19-x28 like any other. So + // the hop is cheap, and measured as emitted it is 13 aarch64 + // instructions (52 bytes): 4 memory ops over x19 and the fp/lr frame + // record, no callee-save block, no fp saves. An instruction count is + // an upper bound on a superscalar core and not a measured cost, but it + // bounds this entry at a few nanoseconds — not somewhere to look for + // entry overhead. `MAJIT_DUMP_CLIF` prints it as `[jit][disasm-entry]`. let body_call_conv = cranelift_codegen::isa::CallConv::Tail; let mut sig = Signature::new(body_call_conv); @@ -15390,6 +15431,13 @@ impl CraneliftBackend { self.func_ctx = wrapper_ctx; } let mut wrapper_compile_ctx = Context::for_function(wrapper_func); + if std::env::var_os("MAJIT_DUMP_CLIF").is_some() { + // The body's dump above shows what the trace costs; this one shows + // what reaching it costs. The wrapper's own conv is the host's, and + // the body's is `Tail`, so whatever the two conventions disagree + // about is emitted HERE and is paid once per compiled entry. + wrapper_compile_ctx.set_disasm(true); + } if let Err(e) = self .module .define_function(entry_id, &mut wrapper_compile_ctx) @@ -15409,6 +15457,18 @@ impl CraneliftBackend { .compiled_code() .map(|code| code.code_info().total_size as usize) .unwrap_or(0); + if std::env::var_os("MAJIT_DUMP_CLIF").is_some() { + // Bytes and not just the text: on a fixed-width instruction set the + // size IS the instruction count, so the entry price of the ABI hop + // is readable without parsing the disassembly. + eprintln!("[jit][code-size] trace_id={trace_id} wrapper_bytes={wrapper_code_bytes}"); + if let Some(text) = wrapper_compile_ctx + .compiled_code() + .and_then(|code| code.vcode.as_deref()) + { + eprintln!("[jit][disasm-entry] trace_id={trace_id}\n{text}"); + } + } self.module.clear_context(&mut wrapper_compile_ctx); self.module.finalize_definitions().unwrap(); diff --git a/majit/majit-backend/src/deadframe.rs b/majit/majit-backend/src/deadframe.rs index 27d4b2821e6..81007b17bb4 100644 --- a/majit/majit-backend/src/deadframe.rs +++ b/majit/majit-backend/src/deadframe.rs @@ -6,12 +6,290 @@ //! JITFRAMEPTR and reads `jf_frame[index]` in place, and `get_latest_descr` //! (`llmodel.py:411-419`) does the same for `jf_descr`. +use std::cell::RefCell; +use std::sync::atomic::{AtomicU8, AtomicU32, AtomicU64, Ordering}; + use majit_gc::shadow_stack::OwnerRootGuard; use majit_ir::{DescrRef, GcRef}; use crate::ExitRecoveryLayout; use crate::jitframe::{FIRST_ITEM_OFFSET, JF_GUARD_EXC_OFS, JF_SAVEDATA_OFS, JitFrame}; +/// The Rust-heap backing of a jitframe, and where the memory goes when the +/// deadframe holding it dies. +/// +/// A backend with no JITFRAME type id to allocate under builds its frames out +/// of the Rust heap instead of the nursery, one per compiled entry. The buffer +/// is not read through this handle — the frame pointer the compiled code was +/// handed points into it — so all this type does is decide the lifetime, which +/// is why it is `_heap_owner` on the deadframe and never read there either. +/// +/// Two arms, chosen per allocation by [`jitframe_pool_enabled`], because what +/// one costs against the other is a difference that has to be taken inside one +/// binary: +/// +/// * [`FrameHeapOwner::POOLED`] — the DEFAULT — takes the buffer off a +/// per-thread free list and puts it back on drop, so a steady entry rate pays +/// the allocator nothing after the first few calls. +/// * [`FrameHeapOwner::OWNED`] is the probe arm: `vec![0i64; words]` in, free on +/// drop. One `calloc`/`free` pair per entry. +/// +/// Pooled is the default because it is the closer shape to the arm this whole +/// type stands in for. `jitframe_allocate` builds the frame out of the GC +/// nursery, which is a bump allocator with no per-frame release at all — that +/// is the `use_gc_alloc` branch, live wherever a JITFRAME type id is +/// registered. The allocator round trip is the deviation, not the baseline, so +/// a build that reaches this type gets the free list unless it asks otherwise. +/// +/// Measured, on a compiled cel entry: pooled is 2.62 ns/entry faster, negative +/// in 24 of 24 shape-runs, and it makes the steady compiled tier +/// allocation-free (`allocs_per_eval` 1.000 -> 0.000). +pub struct FrameHeapOwner { + /// Never empty for a live frame; `Drop` takes it out to hand it back. + buf: Vec, + /// Which arm allocated it, and therefore which one has to release it. A + /// buffer must go back to the arm it came from: pushing an `OWNED` one onto + /// the free list would be sound but would silently convert the probe arm + /// into the pooled one after its first entry. + pooled: bool, +} + +impl FrameHeapOwner { + pub const OWNED: bool = false; + pub const POOLED: bool = true; + + /// A zeroed `words`-word buffer for one frame. + /// + /// Zeroed and not merely sized: the frame's header starts at word 0 and + /// `GuardNotForced` reads `jf_descr != 0`, so a reused buffer carrying the + /// previous entry's descr would fail a guard that did not fail. The owned + /// arm gets that from `calloc`; the pooled arm has to spell it. + /// + /// `pooled` is [`FrameHeapOwner::POOLED`] unless a caller selected the + /// other arm; the single call site reads [`jitframe_pool_enabled`]. + pub fn new(words: usize, pooled: bool) -> Self { + let buf = if pooled { + take_pooled_frame_buf(words) + } else { + count_owned_frame_buf(); + vec![0i64; words] + }; + FrameHeapOwner { buf, pooled } + } + + /// The base of the buffer — the word the frame's own header sits behind. + #[inline] + pub fn as_mut_ptr(&mut self) -> *mut i64 { + self.buf.as_mut_ptr() + } +} + +impl Drop for FrameHeapOwner { + /// Release the buffer, which for the pooled arm means handing it back. + /// + /// This runs when the deadframe holding it drops, and the deadframe is the + /// last thing that reads the frame: the compiled run finished before the + /// deadframe was built, and every accessor on it goes through + /// `jf_gcref()`. So the interior pointer compiled code was handed is dead + /// by the time the buffer is offered to the next entry. The pool hands a + /// buffer out by REMOVING it from the free list, so two live frames can + /// never be looking at one. + fn drop(&mut self) { + if self.pooled { + give_back_pooled_frame_buf(std::mem::take(&mut self.buf)); + } + } +} + +/// Frame buffers a thread keeps rather than frees. +/// +/// Small because the count that matters is the number of frames live at once, +/// not the entry rate: entries are nested only by `execute_bridge` recursion +/// and the CALL_ASSEMBLER hop, so the steady state is one or two. Anything past +/// this is released to the allocator, which bounds a pathological trace's +/// footprint without costing the ordinary one anything. +const FRAME_POOL_CAPACITY: usize = 8; + +#[derive(Default)] +struct FramePool { + free: Vec>, + /// Buffers the owned arm asked the allocator for. + owned: u64, + /// Buffers the pooled arm handed out. + taken: u64, + /// …of which the free list was empty for, so the allocator was asked after + /// all. `taken - misses` is what pooling actually saved. + misses: u64, +} + +thread_local! { + static FRAME_POOL: RefCell = const { + RefCell::new(FramePool { free: Vec::new(), owned: 0, taken: 0, misses: 0 }) + }; +} + +fn take_pooled_frame_buf(words: usize) -> Vec { + // `try_with` and not `with`: a frame outliving its thread's TLS teardown + // would otherwise panic inside a `Drop`, and falling back to the allocator + // is the right answer there anyway. + let pooled = FRAME_POOL.try_with(|pool| { + let mut pool = pool.borrow_mut(); + pool.taken += 1; + match pool.free.pop() { + Some(buf) => Some(buf), + None => { + pool.misses += 1; + None + } + } + }); + match pooled { + Ok(Some(mut buf)) => { + // Grow-only. A frame is 21-22 words on the shapes measured so far + // and 16 more on the tall ones, so the list converges on the tallest + // frame the thread has run and stops resizing. + if buf.len() < words { + buf.resize(words, 0); + } + buf[..words].fill(0); + buf + } + _ => vec![0i64; words], + } +} + +fn give_back_pooled_frame_buf(buf: Vec) { + let _ = FRAME_POOL.try_with(|pool| { + let mut pool = pool.borrow_mut(); + if pool.free.len() < FRAME_POOL_CAPACITY { + pool.free.push(buf); + } + }); +} + +/// Tally one owned allocation. +/// +/// A thread-local access and a `RefCell` borrow per frame, which the pooled arm +/// pays too but only alongside a free-list pop it was already going to take. +/// This is charged to the OWNED arm alone, and the default arm is not OWNED — +/// so it is a cost of the probe, not of the shipping path. Anyone reading an +/// owned-arm figure should read it as the allocator round trip plus this. +fn count_owned_frame_buf() { + let _ = FRAME_POOL.try_with(|pool| pool.borrow_mut().owned += 1); +} + +/// `(owned, pooled takes, pooled misses)` for this thread since it started. +/// +/// The witness that an arm selector actually reached the allocation: a timing +/// difference between two arms that allocated the same way is measuring +/// something else. +pub fn jitframe_pool_counts() -> (u64, u64, u64) { + FRAME_POOL + .try_with(|pool| { + let pool = pool.borrow(); + (pool.owned, pool.taken, pool.misses) + }) + .unwrap_or((0, 0, 0)) +} + +const POOL_ARM_UNSEEDED: u8 = 0; +const POOL_ARM_OFF: u8 = 1; +const POOL_ARM_ON: u8 = 2; + +/// Process-wide, because the thing it selects is a strategy and not a state: +/// a caller flipping arms between two timed batches wants the flip to hold for +/// whichever thread the next entry runs on. +static POOL_ARM: AtomicU8 = AtomicU8::new(POOL_ARM_UNSEEDED); + +/// Select the frame-allocation arm for subsequent compiled entries. +/// +/// Either direction: `false` selects [`FrameHeapOwner::OWNED`] on a build whose +/// default is pooled. Overrides `MAJIT_JITFRAME_POOL`, which is only how a +/// harness that cannot call this — a test binary with no hook of its own — +/// picks an arm. +pub fn set_jitframe_pool(on: bool) { + POOL_ARM.store( + if on { POOL_ARM_ON } else { POOL_ARM_OFF }, + Ordering::Relaxed, + ); +} + +/// Whether the next frame comes off the pool. One relaxed load on the entry +/// path, which BOTH arms pay, so it cancels out of their difference. +#[inline] +pub fn jitframe_pool_enabled() -> bool { + match POOL_ARM.load(Ordering::Relaxed) { + POOL_ARM_OFF => false, + POOL_ARM_ON => true, + _ => seed_jitframe_pool_arm(), + } +} + +/// First-touch arm selection: pooled, unless `MAJIT_JITFRAME_POOL` is set to +/// `0`, which is how the owned arm is asked for. +/// +/// The env var reads BOTH ways rather than only switching the pool on, because +/// the pool is what a build gets without asking — a harness that wants the +/// allocator round trip has to be able to say so. +#[cold] +#[inline(never)] +fn seed_jitframe_pool_arm() -> bool { + let on = std::env::var_os("MAJIT_JITFRAME_POOL").is_none_or(|v| v != "0"); + set_jitframe_pool(on); + on +} + +// ── compiled-entry frame-build probe ───────────────────────────────────── +// +// The count lives here, one crate below the backend that reads it and one +// below the metainterp that sets it, because those two do not see each other: +// `majit-metainterp` depends on a backend, never the reverse. The LOOP it +// drives is gated by the reading backend's own feature; this side is a handful +// of atomics that cost nothing until something loads them. + +/// Extra frame builds per compiled entry, for splitting what the entry spends +/// before it reaches compiled code. +/// +/// The compiled call cannot be repeated — it runs the trace — but its PREFIX +/// can: allocating the jitframe and writing the input arguments into it +/// produces a frame nothing has entered, which is thrown away. That is the only +/// part of the call this can price, and it is the part upstream pays +/// differently (`jitframe_allocate` bump-allocates out of the nursery). +static FRAME_BUILD_REPEATS: AtomicU32 = AtomicU32::new(0); + +/// Frame builds the probe actually performed. The witness that an armed count +/// reached the allocation rather than being set on a path nothing ran. +static FRAME_BUILD_PASSES: AtomicU64 = AtomicU64::new(0); + +/// Set the extra frame builds per compiled entry, answering what it was. +/// +/// Process-wide for the reason [`set_jitframe_pool`] is: it selects a strategy, +/// and a harness flipping arms between two timed batches wants the flip to hold +/// for whichever thread the next entry runs on. +pub fn set_frame_build_repeats(repeats: u32) -> u32 { + FRAME_BUILD_REPEATS.swap(repeats, Ordering::Relaxed) +} + +/// One relaxed load on the entry path, which both arms pay. +#[inline] +pub fn frame_build_repeats() -> u32 { + FRAME_BUILD_REPEATS.load(Ordering::Relaxed) +} + +/// Tally a call's worth of extra frame builds. Once per entry, not once per +/// pass, so the read-modify-write does not scale with the repeat count. +#[inline] +pub fn count_frame_build_passes(passes: u32) { + if passes != 0 { + FRAME_BUILD_PASSES.fetch_add(u64::from(passes), Ordering::Relaxed); + } +} + +/// Extra frame builds performed since the process started. +pub fn frame_build_passes() -> u64 { + FRAME_BUILD_PASSES.load(Ordering::Relaxed) +} + /// Where a held deadframe keeps its jitframe pointer. /// /// The frame is an ordinary GC object (`jitframe.py` makes JITFRAME a @@ -79,8 +357,9 @@ pub struct JitFrameDeadFrame { /// overlay descr synthesis — the deadframe's `fail_descr` keeps the /// callee's own Arc identity rather than being swapped for a synthetic one. pub call_assembler_caller_layout: Option, - /// Keeps the frame memory alive for non-GC allocations. - _heap_owner: Option>, + /// Keeps the frame memory alive for non-GC allocations, and decides where + /// it goes when this deadframe dies. See [`FrameHeapOwner`]. + _heap_owner: Option, /// Whether dropping this deadframe should release the frame's `jf_gcmap`. /// /// True for the deadframe a compiled run returns — the exit established @@ -102,7 +381,7 @@ impl JitFrameDeadFrame { jf_gcref: GcRef, fail_descr: DescrRef, latest_descr: Option, - heap_owner: Option>, + heap_owner: Option, ) -> Self { let jf_root = if heap_owner.is_some() { JitFrameRoot::Unrooted(jf_gcref) diff --git a/majit/majit-charon-reader/tests/corpus.rs b/majit/majit-charon-reader/tests/corpus.rs index 99624fcab14..2dd5e170137 100644 --- a/majit/majit-charon-reader/tests/corpus.rs +++ b/majit/majit-charon-reader/tests/corpus.rs @@ -30,7 +30,8 @@ fn loads_fixture_corpus() { // `w_new_type_only_int`, `w_number_add`, `w_int_add`, // `lltype::malloc_typed`, the fixture's `pyobject::get_instantiate`, and // the initializer bodies for `INT_CLASS`, `DOUBLE_CLASS`, and - // `_immutable_fields_W_IntObject`. + // `_immutable_fields_W_IntObject` — a `static`/`const` carries its + // initializer as a function body, so it lands in `iter_local_fns` too. // // + 2 for the host-registered callback table: `host_registry_dispatch` // and `host_registry_dispatch_optional`. `HostCallback` is a type alias, diff --git a/majit/majit-macros/src/jit_interp/jitcode_lower/lower_value.rs b/majit/majit-macros/src/jit_interp/jitcode_lower/lower_value.rs index dcb81f65af3..24426dfc2b2 100644 --- a/majit/majit-macros/src/jit_interp/jitcode_lower/lower_value.rs +++ b/majit/majit-macros/src/jit_interp/jitcode_lower/lower_value.rs @@ -2592,6 +2592,129 @@ mod tests { assert!(emitted.contains("true")); } + /// One lowering of `source`, seeded with the same three int bindings every + /// spelling gets, reduced to the facts a caller can compare. + fn lower_source(source: &str) -> (Vec, Vec>, Vec>, String) { + let mut lowerer = Lowerer::new(None); + for (name, reg) in [("base", 3u16), ("ea", 4), ("val", 5)] { + lowerer + .bindings + .insert(name.to_string(), binding(reg, BindingKind::Int)); + } + if source.ends_with(';') { + let stmt: syn::Stmt = syn::parse_str(source).expect("parse statement"); + lowerer.lower_stmt(&stmt).expect("statement lowers"); + } else { + let expr: Expr = syn::parse_str(source).expect("parse expression"); + lowerer.lower_value_expr(&expr).expect("expression lowers"); + } + ( + lowerer.op_metadata.iter().map(|m| m.kind).collect(), + lowerer + .op_metadata + .iter() + .map(|m| m.reads.clone()) + .collect(), + lowerer + .op_metadata + .iter() + .map(|m| m.writes.clone()) + .collect(), + lowerer + .statements + .iter() + .map(ToString::to_string) + .collect::(), + ) + } + + /// An IMPORTED intrinsic lowers exactly as a locally-defined one does. + /// + /// The lowerer matches the last segment of the call expression's path and + /// never looks at where the function is defined, so a `use` of an intrinsic + /// from another crate is recognized as readily as a `fn` in this module. + /// The two are indistinguishable here on purpose: an imported name is + /// spelled as a bare call, which is the same token stream a local + /// definition produces, so the bare arm below IS the imported spelling and + /// the qualified arm covers the other way to reach the same import. + /// + /// This matters because the untraced bodies these names need can then live + /// in one library instead of being rewritten in every module that traces + /// them — a duplication that silently diverges the two tiers the day one + /// copy is fixed and the others are not. + /// + /// The op-kind assertion is load-bearing beyond naming the right op: the + /// intrinsic arms run ahead of the generic call path, so a spelling that + /// failed to be recognized would lower as a residual `Call` — and a + /// residual needs a call-policy the imported spelling has no local + /// definition to derive one from. + #[test] + fn an_imported_intrinsic_lowers_like_a_local_one() { + for (bare, qualified, expected_kind, expected_emit) in [ + ( + "majit_raw_load_i64(base, ea)", + "majit_metainterp::intrinsics::majit_raw_load_i64(base, ea)", + OpKind::RawLoad, + "raw_load_i", + ), + ( + "majit_raw_store_i64(base, ea, val);", + "majit_metainterp::intrinsics::majit_raw_store_i64(base, ea, val);", + OpKind::RawStore, + "raw_store_i", + ), + ( + "majit_uint_lt(base, ea)", + "majit_metainterp::intrinsics::majit_uint_lt(base, ea)", + OpKind::BinopI, + "UintLt", + ), + ] { + let local = lower_source(bare); + let imported = lower_source(qualified); + assert_eq!( + local.0, + vec![expected_kind], + "{bare} lowered to {:?}", + local.0 + ); + assert_eq!(local, imported, "`{qualified}` lowered unlike `{bare}`"); + assert!( + local.3.contains(expected_emit), + "`{bare}` emitted no `{expected_emit}`: {}", + local.3 + ); + } + } + + /// The control for the test above: the match really is on the LAST segment. + /// + /// Without this, "the qualified spelling lowered the same" would also hold + /// of a lowerer that ignored the path and matched on argument shape alone. + #[test] + fn only_the_last_path_segment_selects_an_intrinsic() { + fn raw_load_lowers(source: &str) -> bool { + let mut lowerer = Lowerer::new(None); + for (name, reg) in [("base", 3u16), ("ea", 4)] { + lowerer + .bindings + .insert(name.to_string(), binding(reg, BindingKind::Int)); + } + let Expr::Call(call) = syn::parse_str::(source).expect("parse call") else { + unreachable!("the sources below are all call expressions"); + }; + lowerer.lower_raw_load_call(&call).is_some() + } + + // An owner segment nobody declared still lowers: only the tail is read. + assert!(raw_load_lowers("whatever::majit_raw_load_i64(base, ea)")); + // A tail that is not an intrinsic name declines, however it is spelled. + assert!(!raw_load_lowers( + "majit_metainterp::intrinsics::majit_raw_load_i128(base, ea)" + )); + assert!(!raw_load_lowers("majit_raw_load_i128(base, ea)")); + } + #[test] fn record_exact_class_statement_uses_ref_int_argcodes() { let mut lowerer = Lowerer::new(None); diff --git a/majit/majit-macros/src/jit_interp/mod.rs b/majit/majit-macros/src/jit_interp/mod.rs index 9701c908ad9..0efa8bb50ea 100644 --- a/majit/majit-macros/src/jit_interp/mod.rs +++ b/majit/majit-macros/src/jit_interp/mod.rs @@ -2856,13 +2856,13 @@ fn rewrite_body( &mut #state, ); // Push the walk-final loop-carried virt-array - // element values into native `state` too. The - // walk mutates the array on the trace-ctx - // shadow only; native `state`'s array is - // frozen at trace-start (synchronize_ - // virtualizable skips the RustVec write-back - // during tracing). Without this the compiled- - // loop seed (extract_live reads native state) + // element values into native `state` too. A + // field embedding a Rust `Vec` by value is + // carved out of `synchronize_virtualizable`, + // so its array stays frozen at trace-start + // and this is its only writer; without it the + // compiled-loop seed (extract_live reads + // native state) // reflects the trace-start array and the loop // re-executes the peeled iteration, double- // firing any side-effecting residual. No-op @@ -3150,6 +3150,20 @@ fn rewrite_body( { let __back_edge_resume = #call; #finish_drain + // The back edge's spelling of the + // `jit_merge_point!` hook's terminal-return + // exit. A guard that bridges enters the walk + // at the guard's own position, so the walk + // reaches the interpreted function's return + // here just as it can at a merge point; the + // position reported alongside names nothing + // to resume at, and assigning it to + // `#pc_expr` decodes an out-of-range index in + // any dispatch loop not bottom-tested on its + // program length. + if #driver_expr.take_single_pass_finish() { + break; + } if let Some(__resume_pc) = __back_edge_resume { #pc_expr = __resume_pc; continue; diff --git a/majit/majit-metainterp/Cargo.toml b/majit/majit-metainterp/Cargo.toml index d1c53f412ad..100182b1fac 100644 --- a/majit/majit-metainterp/Cargo.toml +++ b/majit/majit-metainterp/Cargo.toml @@ -11,6 +11,29 @@ description = "Meta-interpreter and optimizer for majit JIT compiler" dynasm = ["dep:majit-backend-dynasm"] cranelift = ["dep:majit-backend-cranelift"] jit-audits = ["majit-ir/jit-audits"] +# A measurement arm, never a shipping one. It adds run-time selectable +# amplification loops to three stages of the warm compiled entry in +# `jitdriver::JitDriver::back_edge_internal`, so what one stage costs +# against another is a difference taken inside ONE binary. Off by default, +# and a build without it has neither the loops nor the atomic load that +# selects them. See `jitdriver::BackEdgeStageRepeats`, which also says which +# stages CANNOT be amplified and are therefore left in the residual. +back-edge-stage-probe = [] +# The second half of the same split, and never a shipping feature either. It +# amplifies the two repeatable stages of `execute_assembler_at_dispatch_key` +# and clocks the one that cannot repeat, and forwards to the cranelift arm +# that repeats the frame build. See `pyjitpl::ExecuteStageRepeats`, which +# says which numbers are amplified and which is single-shot. +execute-stage-probe = ["majit-backend-cranelift?/execute-stage-probe"] +# The third of the same family, for the sibling-door yield scan an embedder +# runs ahead of a warm entry. That scan's predicate is a conjunction over three +# state sources, and the public API fuses two of them: `has_compiled_loop` is +# `entry_procedure_token(..).is_some()`, and `get_procedure_token` performs the +# `Weak::upgrade` AND reads `is_invalidated` inside one function. This feature +# adds the one reader that separates them -- the cell lookup and the upgrade +# with NO flag read after it -- so an amplified arm can price the refcount pair +# on its own. See `WarmState::probe_cell_token_upgrades`. +yield-stage-probe = [] [dependencies] majit-ir = { workspace = true } diff --git a/majit/majit-metainterp/src/blackhole.rs b/majit/majit-metainterp/src/blackhole.rs index facba68abd7..2e9751cf6f5 100644 --- a/majit/majit-metainterp/src/blackhole.rs +++ b/majit/majit-metainterp/src/blackhole.rs @@ -8917,6 +8917,14 @@ pub fn build_inline_call_only_bh_builder() -> BlackholeInterpBuilder { "arraylen_vable/rdd>i".to_string(), majit_translate::insns::BC_ARRAYLEN_VABLE, ); + // Registered for the same reason the array-build family below is: the + // abort this op provokes resumes through the blackhole over the jitcode + // that contains it, so leaving the byte unresolved would panic + // `dispatch_step` on the op's own failure path. + insns.insert( + "arraybase_vable/rdd>i".to_string(), + majit_translate::insns::BC_ARRAYBASE_VABLE, + ); // GC array-build family — `BuildTuple` / `BuildList` / `BuildMap` / // `BuildSet` / `BuildString` lower to `new_array_clear` (alloc) + // unrolled `setarrayitem_gc_r` (fill) + a `new*_from_array` residual @@ -9589,6 +9597,7 @@ pub fn wire_bhimpl_handlers(builder: &mut BlackholeInterpBuilder) { builder.wire_handler("setarrayitem_vable_i/riidd", handler_setarrayitem_vable_i); builder.wire_handler("setarrayitem_vable_r/rirdd", handler_setarrayitem_vable_r); builder.wire_handler("arraylen_vable/rdd>i", handler_arraylen_vable); + builder.wire_handler("arraybase_vable/rdd>i", handler_arraybase_vable); builder.wire_handler("getarrayitem_raw_i/iid>i", handler_getarrayitem_raw_i); builder.wire_handler("setarrayitem_raw_i/iiid", handler_setarrayitem_raw_i); builder.wire_handler("conditional_call_ir_v/iiIRd", handler_conditional_call_ir_v); @@ -10405,6 +10414,32 @@ fn handler_arraylen_vable( Ok(p + 1) } +/// `arraybase_vable/rdd>i` — item-0 address of a virtualizable array. +/// +/// Wiring this is not optional even though a trace carrying the op always +/// aborts: the abort resumes through the blackhole over this very jitcode, so +/// an unwired byte panics `dispatch_step` on exactly the path the op creates. +/// Operand layout is `arraylen_vable`'s, so the decode is line-for-line its +/// sibling above. +/// +/// No escape signal is raised here. The signal exists to abort a *trace*, and +/// the blackhole is already running because that abort happened. +fn handler_arraybase_vable( + bh: &mut BlackholeInterpreter, + code: &[u8], + p: usize, +) -> Result { + let vable = bh.registers_r[code[p] as usize]; + let vinfo = vable_clear_token_and_get_vinfo(bh, vable); + let (field_descr, p) = read_descr(bh, code, p + 1); + let array_idx = field_descr.as_vable_array_index(); + let (_, p) = read_descr(bh, code, p); + let ainfo = &vinfo.array_fields[array_idx]; + let base = unsafe { crate::virtualizable::bhimpl_arraybase_vable(vable as *const u8, ainfo) }; + bh.registers_i[code[p] as usize] = base as usize as i64; + Ok(p + 1) +} + // ── getarrayitem_raw / setarrayitem_raw (blackhole.py:1343-1365) ──── /// RPython `blackhole.py` `bhimpl_getarrayitem_raw_i`: /// `return cpu.bh_getarrayitem_raw_i(array, index, arraydescr)`. diff --git a/majit/majit-metainterp/src/embed.rs b/majit/majit-metainterp/src/embed.rs new file mode 100644 index 00000000000..8fd1aefa5ea --- /dev/null +++ b/majit/majit-metainterp/src/embed.rs @@ -0,0 +1,753 @@ +//! Per-run JIT census for an embedder. +//! +//! An interpreter that drives [`JitDriver`] needs to know what its JIT actually +//! did: a green test suite, agreement with the untraced tier and an exact +//! answer are all satisfied by an interpreter that never compiled anything. +//! The evidence lives behind four driver callbacks and a set of process-global +//! diagnostic slots. [`Census`] wires those up once so an embedder does not +//! rebuild the counters, the callback closures and the serializing lock per +//! module. +//! +//! # The window is the point +//! +//! The counters behind this module are process-global and cumulative, because +//! the callbacks are `'static` closures that cannot borrow a caller's local +//! state. An absolute read therefore answers "since this process started", +//! which is not the question a run asks. [`Census::begin`] opens a window and +//! [`Census::counts`] reports the DELTA across it. +//! +//! Two rules make that delta trustworthy, and both were learned from getting +//! them wrong: +//! +//! * The serializing lock is a process-global [`Mutex`], not a `thread_local!`. +//! Its job is to keep two concurrently running consumers — parallel test +//! threads, most often — out of each other's window. A thread-local lock is +//! uncontended by construction and silently does nothing at all. +//! * The cumulative counters are diffed, never zeroed, so a window cannot +//! destroy an enclosing reader's totals. The "last compiled body" fields +//! cannot be diffed, so [`Census::begin`] resets those instead — and it holds +//! the window lock across the reset AND the read, which is what makes that +//! reset safe. A reset outside a held lock hands each of two overlapping +//! consumers a fraction of the truth. + +use std::fmt; +use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; +use std::sync::{Mutex, MutexGuard}; + +use crate::{JitDriver, JitState, LoopBodyShape}; + +/// Loops compiled, from the compile-loop callback. +static COMPILES: AtomicUsize = AtomicUsize::new(0); +/// Guards that failed back into the runtime, from the guard-failure callback. +static GUARD_FAILURES: AtomicUsize = AtomicUsize::new(0); +/// Traces abandoned before they closed, from the trace-abort callback. +static TRACE_ABORTS: AtomicUsize = AtomicUsize::new(0); +/// Calls that entered compiled code, from the compiled-entry callback. +static COMPILED_ENTRIES: AtomicUsize = AtomicUsize::new(0); +/// Recorded and optimized op counts, summed over every compiled loop. +static TRACE_OPS_BEFORE: AtomicUsize = AtomicUsize::new(0); +static TRACE_OPS_AFTER: AtomicUsize = AtomicUsize::new(0); +/// Driver-internal totals taken off a driver by [`Census::absorb`]. Neither has +/// a callback, so a driver that is dropped without being absorbed takes its +/// tallies with it. +static ABSORBED_BRIDGES: AtomicUsize = AtomicUsize::new(0); +static ABSORBED_PANICS: AtomicUsize = AtomicUsize::new(0); + +/// The last compiled body's optimized op count and shape. +/// +/// Not cumulative, so [`Census::begin`] resets them rather than diffing them. +/// The shape is held as its two flags rather than the struct so the recording +/// stays lock-free on the compile path. +static LAST_OPS_AFTER: AtomicUsize = AtomicUsize::new(0); +static LAST_HAS_JUMP: AtomicBool = AtomicBool::new(false); +static LAST_ALWAYS_FAILS: AtomicBool = AtomicBool::new(false); + +/// Serializes measurement windows against each other. +/// +/// Process-global on purpose: the counters it protects are, and the consumer +/// this exists for is a parallel test runner entering the same JIT from several +/// threads at once. +static WINDOW_LOCK: Mutex<()> = Mutex::new(()); + +/// One reading of what the JIT did. +/// +/// Every field except the two `last_*` ones is a count over the window it was +/// read in. `bridges_compiled` and `internal_compile_panics` are the exception +/// in a second way as well: they have no callback and only appear here for a +/// driver that was handed to [`Census::absorb`]. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct CensusCounts { + pub loops_compiled: usize, + /// Only what [`Census::absorb`] collected. Zero otherwise, which is not the + /// same as "no bridge was compiled". + pub bridges_compiled: usize, + pub loops_aborted: usize, + pub guard_failures: usize, + /// Non-zero means a trace was dropped by a panic inside compilation and the + /// tier silently fell back to the untraced path for it. Nothing else + /// reports this: a run that stops compiling still answers correctly, so + /// every other counter here stays plausible. Only what + /// [`Census::absorb`] collected, as with `bridges_compiled`. + pub internal_compile_panics: usize, + pub trace_ops_before: usize, + pub trace_ops_after: usize, + /// The only field that separates "an artifact exists" from "an artifact + /// ran". + pub compiled_entries: usize, + /// Optimized op count of the LAST compiled body in the window. + /// + /// `loops_compiled > 0` is necessary but not sufficient evidence of a + /// working tier: an entirely empty dispatch still compiles a trace, one + /// whose whole optimized body is `Finish()`. A compile counter counts + /// TRACES, not WORK. + pub last_ops_after: usize, + /// Shape of the LAST compiled body in the window. Its [`Default`] value is + /// also what "nothing compiled in this window" reads as — see + /// [`LoopBodyShape::why_not`], whose no-`Jump` arm is phrased for + /// both cases. + pub last_loop_body_shape: LoopBodyShape, +} + +impl fmt::Display for CensusCounts { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!( + f, + "loops_compiled={} bridges_compiled={} loops_aborted={} \ + guard_failures={} internal_compile_panics={} trace_ops_before={} \ + trace_ops_after={} compiled_entries={} last_ops_after={} \ + last_closes_a_loop={}", + self.loops_compiled, + self.bridges_compiled, + self.loops_aborted, + self.guard_failures, + self.internal_compile_panics, + self.trace_ops_before, + self.trace_ops_after, + self.compiled_entries, + self.last_ops_after, + self.last_loop_body_shape.closes_a_loop(), + ) + } +} + +/// An open measurement window over the process-global JIT counters. +/// +/// Hold one for as long as the run being measured, then read it: +/// +/// ```ignore +/// let census = Census::begin(); +/// let answer = run_the_interpreter(); +/// assert!(census.counts().loops_compiled > 0, "{census}"); +/// ``` +/// +/// ⚠ NOT REENTRANT. [`Census::begin`] takes a plain [`Mutex`], so opening a +/// second window on a thread that already holds one deadlocks. Where a test +/// helper opens the window, every test that can compile has to go through that +/// helper and none of them may call another. +/// +/// ⚠ Bind it to a NAMED local, `_census` included. `let _ = Census::begin();` +/// drops the window on the spot and leaves the run unserialized, which fails +/// only under contention and only sometimes. +pub struct Census { + /// Cumulative counters at [`Census::begin`], subtracted at read time. + base: RawCounts, + /// Abort-reason tallies at [`Census::begin`], for the same reason. + aborts_before: Vec<(&'static str, u64)>, + /// Held for the window's whole life. Dropping the [`Census`] closes it. + _window: MutexGuard<'static, ()>, +} + +/// The cumulative half of a reading, snapshotted for the later subtraction. +#[derive(Clone, Copy, Debug, Default)] +struct RawCounts { + compiles: usize, + bridges: usize, + aborts: usize, + guard_failures: usize, + panics: usize, + ops_before: usize, + ops_after: usize, + compiled_entries: usize, +} + +impl RawCounts { + fn read() -> Self { + Self { + compiles: COMPILES.load(Ordering::Relaxed), + bridges: ABSORBED_BRIDGES.load(Ordering::Relaxed), + aborts: TRACE_ABORTS.load(Ordering::Relaxed), + guard_failures: GUARD_FAILURES.load(Ordering::Relaxed), + panics: ABSORBED_PANICS.load(Ordering::Relaxed), + ops_before: TRACE_OPS_BEFORE.load(Ordering::Relaxed), + ops_after: TRACE_OPS_AFTER.load(Ordering::Relaxed), + compiled_entries: COMPILED_ENTRIES.load(Ordering::Relaxed), + } + } +} + +impl Census { + /// Wire every driver callback the census reads into the process-global + /// counters. + /// + /// The four hooks are `set_on_compile_loop`, `set_on_guard_failure`, + /// `set_on_trace_abort` and `set_on_compiled_entry`. Each holds ONE + /// closure, so this replaces whatever was installed before it; an embedder + /// that was recording the compiled body's shape by hand reads + /// [`CensusCounts::last_loop_body_shape`] instead of installing its own. + /// + /// Install it on every driver whose activity should be counted — the + /// counters are shared, the callbacks are not. + pub fn install(driver: &mut JitDriver) { + driver.set_on_compile_loop(|_green_key, ops_before, ops_after, opcodes| { + COMPILES.fetch_add(1, Ordering::Relaxed); + TRACE_OPS_BEFORE.fetch_add(ops_before, Ordering::Relaxed); + TRACE_OPS_AFTER.fetch_add(ops_after, Ordering::Relaxed); + LAST_OPS_AFTER.store(ops_after, Ordering::Relaxed); + let shape = LoopBodyShape::of(opcodes); + LAST_HAS_JUMP.store(shape.has_jump, Ordering::Relaxed); + LAST_ALWAYS_FAILS.store(shape.has_always_fails, Ordering::Relaxed); + }); + driver.set_on_guard_failure(|_green_key, _fail_index, _count| { + GUARD_FAILURES.fetch_add(1, Ordering::Relaxed); + }); + driver.set_on_trace_abort(|_green_key, _permanent| { + TRACE_ABORTS.fetch_add(1, Ordering::Relaxed); + }); + driver.set_on_compiled_entry(|_green_key, _target_pc| { + COMPILED_ENTRIES.fetch_add(1, Ordering::Relaxed); + }); + } + + /// Open a measurement window: serialize against any other window, snapshot + /// the cumulative counters, and reset the non-cumulative ones. + /// + /// Blocks until any window open on another thread closes. See the type's + /// reentrancy and binding warnings. + #[must_use = "the window closes when the returned Census is dropped"] + pub fn begin() -> Self { + let window = WINDOW_LOCK.lock().unwrap_or_else(|e| e.into_inner()); + LAST_OPS_AFTER.store(0, Ordering::Relaxed); + LAST_HAS_JUMP.store(false, Ordering::Relaxed); + LAST_ALWAYS_FAILS.store(false, Ordering::Relaxed); + Self { + base: RawCounts::read(), + aborts_before: abort_reasons(), + _window: window, + } + } + + /// What the JIT did since [`Census::begin`]. + pub fn counts(&self) -> CensusCounts { + let now = RawCounts::read(); + CensusCounts { + loops_compiled: now.compiles.saturating_sub(self.base.compiles), + bridges_compiled: now.bridges.saturating_sub(self.base.bridges), + loops_aborted: now.aborts.saturating_sub(self.base.aborts), + guard_failures: now.guard_failures.saturating_sub(self.base.guard_failures), + internal_compile_panics: now.panics.saturating_sub(self.base.panics), + trace_ops_before: now.ops_before.saturating_sub(self.base.ops_before), + trace_ops_after: now.ops_after.saturating_sub(self.base.ops_after), + compiled_entries: now + .compiled_entries + .saturating_sub(self.base.compiled_entries), + last_ops_after: LAST_OPS_AFTER.load(Ordering::Relaxed), + last_loop_body_shape: LoopBodyShape { + has_jump: LAST_HAS_JUMP.load(Ordering::Relaxed), + has_always_fails: LAST_ALWAYS_FAILS.load(Ordering::Relaxed), + }, + } + } + + /// The abort reasons that fired since [`Census::begin`], as `label=delta`. + /// + /// Empty when nothing aborted, so a caller can print it unconditionally in + /// a failure message and a quiet window stays quiet. See [`abort_reasons`] + /// for what the labels do and do not distinguish. + pub fn abort_reasons_since(&self) -> String { + render_abort_delta(&self.aborts_before, &abort_reasons()) + } + + /// Take a driver's own tallies before it is dropped. + /// + /// Bridge compiles and swallowed compilation panics have no callback, so + /// they live on the driver and die with it. This adds them to the shared + /// counters, which makes them visible to any window still open. + /// + /// ⚠ Call it ONCE, at the driver's end of life. The driver's tallies are + /// cumulative, so absorbing a driver that will keep running counts its + /// history again at the next call. + pub fn absorb(driver: &JitDriver) { + let stats = driver.get_stats(); + ABSORBED_BRIDGES.fetch_add(stats.bridges_compiled, Ordering::Relaxed); + ABSORBED_PANICS.fetch_add(stats.internal_compile_panics as usize, Ordering::Relaxed); + } + + /// Absolute counters since process start, or since the last + /// [`Census::reset`]. Takes no lock, so it is safe to call from inside an + /// open window — and is a total, not a window, which is why the assertions + /// a run makes belong on [`Census::counts`] instead. + pub fn totals() -> CensusCounts { + let now = RawCounts::read(); + CensusCounts { + loops_compiled: now.compiles, + bridges_compiled: now.bridges, + loops_aborted: now.aborts, + guard_failures: now.guard_failures, + internal_compile_panics: now.panics, + trace_ops_before: now.ops_before, + trace_ops_after: now.ops_after, + compiled_entries: now.compiled_entries, + last_ops_after: LAST_OPS_AFTER.load(Ordering::Relaxed), + last_loop_body_shape: LoopBodyShape { + has_jump: LAST_HAS_JUMP.load(Ordering::Relaxed), + has_always_fails: LAST_ALWAYS_FAILS.load(Ordering::Relaxed), + }, + } + } + + /// Zero every counter this module owns. + /// + /// For a caller that reads totals rather than windows. It moves the ground + /// under any window that is already open, so a windowed reader should have + /// no reason to call it — [`Census::begin`] already gives that reader a + /// fresh zero without disturbing anyone else. + pub fn reset() { + for counter in [ + &COMPILES, + &GUARD_FAILURES, + &TRACE_ABORTS, + &COMPILED_ENTRIES, + &TRACE_OPS_BEFORE, + &TRACE_OPS_AFTER, + &ABSORBED_BRIDGES, + &ABSORBED_PANICS, + &LAST_OPS_AFTER, + ] { + counter.store(0, Ordering::Relaxed); + } + LAST_HAS_JUMP.store(false, Ordering::Relaxed); + LAST_ALWAYS_FAILS.store(false, Ordering::Relaxed); + } +} + +impl fmt::Display for Census { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(f, "{}", self.counts()) + } +} + +/// The degraded dispatch arms recorded for `interp`, sorted by arm name. +/// +/// The process-wide registry holds every machine's arms, so a reader that wants +/// one machine's has to filter; sorting makes the result comparable against a +/// written-down set whatever order the portal happened to install them in. +/// +/// ⚠ This is NOT a windowed reading and must not be made one, which is what +/// separates it from every counter in this module. The registry records a +/// per-ARM fact at portal-install time and deduplicates by content, so it +/// describes what the lowerer can express about a machine, not what one run +/// did. Diffing it across a window would report "installed during this window", +/// a different and far less useful property. +pub fn degraded_dispatch_arm_names(interp: &str) -> Vec<&'static str> { + let mut names: Vec<&'static str> = crate::degraded_dispatch_arms() + .into_iter() + .filter(|arm| arm.interp == interp) + .map(|arm| arm.arm) + .collect(); + names.sort_unstable(); + names +} + +/// The degraded arms recorded for `interp` paired with their refusal family, +/// sorted by arm name. +/// +/// Strictly more than [`degraded_dispatch_arm_names`]: an arm can keep +/// degrading under a different mechanism, which moves the family while the name +/// set sits still. Only the FIRST refusal in each arm's reason is classified — +/// that is the outermost blocker, the one that stopped lowering. +pub fn degraded_dispatch_arm_causes(interp: &str) -> Vec<(&'static str, crate::RefusalKind)> { + let mut causes: Vec<(&'static str, crate::RefusalKind)> = crate::degraded_dispatch_arms() + .into_iter() + .filter(|arm| arm.interp == interp) + .map(|arm| (arm.arm, crate::refusal_kind(arm.reason))) + .collect(); + causes.sort_unstable_by_key(|(arm, _)| *arm); + causes +} + +/// The refusal reason recorded for one of `interp`'s arms, if that arm +/// degraded. +/// +/// Looked up by arm name rather than by position: the registry's order is the +/// portal's install order, so an index is a pin on something nobody chose. +pub fn degraded_dispatch_arm_reason(interp: &str, arm: &str) -> Option<&'static str> { + crate::degraded_dispatch_arms() + .into_iter() + .find(|entry| entry.interp == interp && entry.arm == arm) + .map(|entry| entry.reason) +} + +/// Panic unless `interp`'s `arm` degraded AND its reason contains `needle`. +/// +/// One level below [`assert_degraded_dispatch_arm_causes`]: the family says +/// which MECHANISM refused the arm, and this says which SOURCE it refused on. +/// A refusal can keep its family while moving to a different statement, which +/// means the gap being tracked is not the one the pin was written for. +/// +/// A substring, never the whole reason — the refusal is rendered with the +/// lowerer's own spacing, which is not a thing a gate should be pinned to. +pub fn assert_degraded_dispatch_arm_reason_contains(interp: &str, arm: &str, needle: &str) { + match degraded_dispatch_arm_reason(interp, arm) { + Some(reason) => assert!( + reason.contains(needle), + "`{interp}`'s `{arm}` no longer refuses on `{needle}`, so the pin \ + tracks a gap that has moved: {reason}" + ), + None => panic!( + "`{interp}` has no degraded arm named `{arm}`, so there is no \ + refusal to read `{needle}` out of. Degraded arms: {:?}", + degraded_dispatch_arm_names(interp) + ), + } +} + +/// Panic unless `interp`'s portal was installed AND its degraded arms are +/// EXACTLY `expected`. +/// +/// **An equality over a named set, never an emptiness check.** A machine with a +/// known lowering gap has a non-empty degraded set today, so demanding +/// emptiness there fails on day one and the only available response is to +/// weaken the gate into uselessness. Pinning the set keeps both directions +/// live: a NEW name means an arm silently stopped lowering and every trace +/// reaching it aborts, and a MISSING name means that arm lowers again, so the +/// gap the pin was tracking is closed and the surrounding gate can be +/// strengthened. An emptiness check reports only the first of those, and only +/// where the answer is already zero. +/// +/// Pass `&[]` for a machine that should have no degraded arm at all; +/// [`crate::assert_no_degraded_dispatch_arms`] is that spelling. +/// +/// The portal check is what makes an empty `expected` mean anything. An empty +/// registry is also what a machine whose portal was never built produces — the +/// same shape as the defect the gate exists to report, one level up — so the +/// arm census supplies the denominator and the two failures get two different +/// messages instead of one silent pass. +/// +/// Call it after whatever installs the portal. The facts are recorded at +/// install, not at trace time, so running the machine is not required — but +/// nothing is recorded until the portal is built at least once. +pub fn assert_degraded_dispatch_arms(interp: &str, expected: &[&str]) { + let arms = assert_portal_installed(interp, expected.is_empty()); + let degraded = degraded_dispatch_arm_names(interp); + assert!( + degraded == expected, + "`{interp}`'s arms that lowered to an abort stub are {degraded:?}, not \ + the pinned {expected:?} — out of the {arms} arm(s) its portal emitted. \ + Every trace that reaches a stub aborts, once per threshold, forever. A \ + NEW name means an arm silently stopped lowering; a MISSING name means \ + that arm lowers again, so the pin is stale and whatever it was blocking \ + may now be possible." + ); +} + +/// Panic unless `interp`'s portal was installed AND its degraded arms and their +/// refusal families are EXACTLY `expected`. +/// +/// The companion to [`assert_degraded_dispatch_arms`] and strictly stronger, +/// since the pairs pin the names too. Both are worth asserting where a machine +/// has a known gap: they fail for different reasons and the two messages say so +/// — one reports that the SET moved, this one that a set which did not move is +/// now being refused by a different mechanism. +pub fn assert_degraded_dispatch_arm_causes(interp: &str, expected: &[(&str, crate::RefusalKind)]) { + assert_portal_installed(interp, expected.is_empty()); + let causes = degraded_dispatch_arm_causes(interp); + let matches = causes.len() == expected.len() + && causes + .iter() + .zip(expected) + .all(|((arm, kind), (want_arm, want_kind))| arm == want_arm && kind == want_kind); + assert!( + matches, + "`{interp}`'s degraded dispatch arms are refused by {causes:?}, not the \ + pinned {expected:?}. A family that moved while its arm name did not \ + means a different mechanism is refusing that arm now; \ + `RefusalKind::Unclassified` means a refusal family exists that nothing \ + classifies yet." + ); +} + +/// The arm count `interp`'s portal recorded, or a panic naming the machines +/// that did record one. +/// +/// `expecting_none` only shapes the message: a pin that expects degraded arms +/// would fail anyway on the empty registry, but it would fail saying "the set +/// moved" when the truth is that nothing was ever measured. +fn assert_portal_installed(interp: &str, expecting_none: bool) -> usize { + let census = crate::dispatch_arm_census(); + match census.iter().find(|entry| entry.interp == interp) { + Some(entry) => entry.arms, + None => { + let consequence = if expecting_none { + "so an empty degraded list says nothing about it" + } else { + "so the degraded list below is empty because nothing was measured, \ + not because the pinned arms started lowering" + }; + panic!( + "no dispatch-arm census for `{interp}`: its portal was never \ + installed in this process, {consequence}. Build the dispatch \ + JitCode (or run the machine) first. Recorded machines: {:?}", + census.iter().map(|e| e.interp).collect::>() + ); + } + } +} + +/// Snapshot the abort-reason tallies as `(label, count)` pairs. +/// +/// [`CensusCounts::loops_aborted`] counts aborts without saying why, and the +/// reason is not carried on the trace-abort callback — it survives only in the +/// `MC_DIAG` slots. Without this a gate can assert `loops_aborted == 0`, watch +/// it fail, and have no way to learn which reason fired. +/// +/// Selection is by the `abrt_` label prefix rather than by the slot range those +/// labels currently occupy: a hard-coded range names the wrong counters the +/// moment a slot is added, and it does so silently. +/// +/// ⚠ `abrt_bridge` is the UNCLASSIFIED bucket wearing a specific-sounding name. +/// Every abort whose reason was never staged falls back to the generic reason, +/// whose id is that slot, so a count there is not evidence of bridge activity. +/// [`Census::abort_reasons_since`] relabels it in its output for that reason. +/// +/// ⚠ The slots are process-global, cumulative, and have no reset, so diff two +/// snapshots rather than reading one. +pub fn abort_reasons() -> Vec<(&'static str, u64)> { + crate::MC_DIAG_LABELS + .iter() + .enumerate() + .filter(|(_, label)| label.starts_with("abrt_")) + .map(|(slot, label)| (*label, crate::mc_diag(slot))) + .collect() +} + +/// Render `after - before` over two [`abort_reasons`] snapshots, dropping the +/// slots that did not move. +fn render_abort_delta(before: &[(&'static str, u64)], after: &[(&'static str, u64)]) -> String { + after + .iter() + .zip(before) + .filter_map(|((label, now), (_, then))| { + let delta = now.saturating_sub(*then); + if delta == 0 { + return None; + } + let name = if *label == "abrt_bridge" { + "unclassified(abrt_bridge)" + } else { + label + }; + Some(format!("{name}={delta}")) + }) + .collect::>() + .join(" ") +} + +#[cfg(test)] +mod tests { + use super::*; + + /// The window read is a DELTA, so activity that happened before it opened + /// belongs to nobody's window. + #[test] + fn a_window_reports_only_what_happened_inside_it() { + let census = Census::begin(); + assert_eq!(census.counts(), CensusCounts::default()); + + COMPILES.fetch_add(2, Ordering::Relaxed); + TRACE_OPS_AFTER.fetch_add(11, Ordering::Relaxed); + assert_eq!(census.counts().loops_compiled, 2); + assert_eq!(census.counts().trace_ops_after, 11); + + drop(census); + // A second window opened after the first closed starts from zero + // without either of them having zeroed a shared counter. + let next = Census::begin(); + assert_eq!(next.counts().loops_compiled, 0); + assert!(Census::totals().loops_compiled >= 2); + } + + /// The `last_*` fields cannot be diffed, so the window resets them — and a + /// window with no compile in it must not inherit the previous one's body. + #[test] + fn the_last_body_fields_do_not_leak_across_windows() { + { + let _census = Census::begin(); + LAST_OPS_AFTER.store(42, Ordering::Relaxed); + LAST_HAS_JUMP.store(true, Ordering::Relaxed); + } + let census = Census::begin(); + let counts = census.counts(); + assert_eq!(counts.last_ops_after, 0); + assert!(!counts.last_loop_body_shape.closes_a_loop()); + assert!(counts.last_loop_body_shape.why_not().is_some()); + } + + #[test] + fn abort_reason_labels_are_selected_by_prefix_not_by_slot_range() { + let reasons = abort_reasons(); + assert!(!reasons.is_empty()); + assert!(reasons.iter().all(|(label, _)| label.starts_with("abrt_"))); + } + + #[test] + fn a_quiet_abort_window_renders_empty_and_a_busy_one_names_its_slot() { + let before = vec![("abrt_too_long", 3u64), ("abrt_bridge", 1u64)]; + assert_eq!(render_abort_delta(&before, &before), ""); + + let after = vec![("abrt_too_long", 5u64), ("abrt_bridge", 4u64)]; + assert_eq!( + render_abort_delta(&before, &after), + "abrt_too_long=2 unclassified(abrt_bridge)=3" + ); + } + + /// Register a synthetic machine so the registry assertions have a subject. + /// + /// The registry is process-wide, deduplicated by content and never cleared, + /// so each test uses a name of its own rather than trying to reset it. + fn record_machine( + interp: &'static str, + arms: usize, + degraded: &[(&'static str, &'static str)], + ) { + crate::record_dispatch_arm_census(interp, arms); + for (arm, reason) in degraded { + crate::record_degraded_dispatch_arm(interp, arm, reason); + } + } + + /// The finding this gate exists for, as an executable claim: a machine with + /// a known lowering gap can pin its set, and the emptiness-only spelling can + /// only fail on it. + #[test] + fn a_known_gap_is_expressible_as_a_pinned_set() { + record_machine( + "PinnedGapState", + 4, + &[("PUSHARG", "arm body cannot express this statement")], + ); + assert_degraded_dispatch_arms("PinnedGapState", &["PUSHARG"]); + assert_degraded_dispatch_arm_causes( + "PinnedGapState", + &[("PUSHARG", crate::RefusalKind::UnlowerableStmt)], + ); + } + + #[test] + #[should_panic(expected = "not the pinned []")] + fn the_zero_spelling_cannot_express_a_known_gap() { + record_machine( + "ZeroSpellingState", + 4, + &[("PUSHARG", "arm body cannot express this statement")], + ); + crate::assert_no_degraded_dispatch_arms("ZeroSpellingState"); + } + + /// Install order is not the pin's business, so the reader sorts. + #[test] + fn arm_names_are_sorted_whatever_order_the_portal_installed_them() { + record_machine( + "SortedState", + 3, + &[ + ("ROLL", "arm body cannot express this statement"), + ("ALLOCATE", "arm body cannot express this statement"), + ], + ); + assert_eq!( + degraded_dispatch_arm_names("SortedState"), + ["ALLOCATE", "ROLL"] + ); + assert_degraded_dispatch_arms("SortedState", &["ALLOCATE", "ROLL"]); + } + + /// A machine whose portal was never built has an empty degraded list, which + /// is the same shape as a healthy one. The denominator is what tells them + /// apart, and it must do so for a NON-empty pin too — there the empty list + /// would otherwise read as "the pinned arms started lowering". + #[test] + #[should_panic(expected = "nothing was measured")] + fn an_uninstalled_portal_does_not_read_as_a_closed_gap() { + assert_degraded_dispatch_arms("NeverInstalledState", &["PUSHARG"]); + } + + #[test] + #[should_panic(expected = "says nothing about it")] + fn an_uninstalled_portal_does_not_read_as_a_clean_one() { + crate::assert_no_degraded_dispatch_arms("NeverInstalledEmptyState"); + } + + /// The family says which mechanism refused; the reason says which source. + #[test] + fn a_refusal_is_readable_by_arm_name_not_by_install_position() { + record_machine( + "ReasonState", + 2, + &[ + ("ROLL", "arm body cannot express this statement: pc += 1"), + ( + "ALLOCATE", + "arm body cannot express this statement: regs[n]", + ), + ], + ); + // Looked up by name, so the portal's install order is not a pin. + assert_degraded_dispatch_arm_reason_contains("ReasonState", "ALLOCATE", "regs[n]"); + assert_degraded_dispatch_arm_reason_contains("ReasonState", "ROLL", "pc += 1"); + assert!(degraded_dispatch_arm_reason("ReasonState", "NOSUCH").is_none()); + } + + #[test] + #[should_panic(expected = "no degraded arm named")] + fn a_reason_pin_on_an_arm_that_lowers_reports_that_rather_than_a_mismatch() { + record_machine("LoweredArmState", 2, &[]); + assert_degraded_dispatch_arm_reason_contains("LoweredArmState", "ROLL", "pc += 1"); + } + + /// A family can move while the name set sits still, which is the whole + /// reason the causes gate is separate from the names one. + #[test] + #[should_panic(expected = "refused by")] + fn a_cause_that_moved_under_an_unchanged_name_is_a_failure() { + record_machine( + "MovedCauseState", + 2, + &[("ALLOCATE", "arm body cannot express this statement")], + ); + assert_degraded_dispatch_arms("MovedCauseState", &["ALLOCATE"]); + assert_degraded_dispatch_arm_causes( + "MovedCauseState", + &[("ALLOCATE", crate::RefusalKind::GreenWriteback)], + ); + } + + #[test] + fn display_names_every_field_it_reports() { + let text = CensusCounts::default().to_string(); + for key in [ + "loops_compiled=", + "bridges_compiled=", + "loops_aborted=", + "guard_failures=", + "internal_compile_panics=", + "trace_ops_before=", + "trace_ops_after=", + "compiled_entries=", + "last_ops_after=", + "last_closes_a_loop=", + ] { + assert!(text.contains(key), "{key} missing from {text}"); + } + } +} diff --git a/majit/majit-metainterp/src/intrinsics.rs b/majit/majit-metainterp/src/intrinsics.rs new file mode 100644 index 00000000000..e7a48fc9dd5 --- /dev/null +++ b/majit/majit-metainterp/src/intrinsics.rs @@ -0,0 +1,326 @@ +//! Untraced-path bodies for the intrinsics `#[jit_interp]` rewrites. +//! +//! A `#[jit_interp]` mainloop runs at two tiers. While tracing, the macro +//! rewrites a call to one of the names below into a trace op — the body is +//! never entered. At the interpreter tier the same source calls a real Rust +//! function, and the two tiers must agree bit-for-bit or a compiled loop +//! answers differently from the interpreter that fed it. This module is that +//! function, once, so an interpreter does not hand-write it per module. +//! +//! **The macro matches the LAST PATH SEGMENT of the call expression**, not the +//! definition site. All three spellings therefore lower identically: +//! +//! ```ignore +//! use majit_metainterp::intrinsics::majit_raw_load_i64; +//! majit_raw_load_i64(base, ea); // imported +//! majit_metainterp::intrinsics::majit_raw_load_i64(base, ea); // qualified +//! fn majit_raw_load_i64(base: i64, ea: i64) -> i64 { .. } // local +//! ``` +//! +//! `majit_uint_mul_high` is the one exception and is documented at its +//! definition: it has no hard-coded name in the macro and is reached only +//! through a `native_int_binops` alias. +//! +//! # Addresses and safety +//! +//! Raw-memory intrinsics take their address as an `i64`, the `support` +//! `AddressAsInt` convention, and `ea` is a BYTE offset from it — as +//! `rawstorage.py` `raw_storage_getitem` takes an `index` it `ptradd`s onto a +//! `CCHARP`. +//! +//! ⚠ These are safe `fn`s that dereference a caller-supplied integer, and that +//! is forced, not chosen: the lowerer matches a bare call expression and has no +//! rule for an `unsafe` block, so an `unsafe fn` spelling would stop lowering +//! and silently leave the traced tier calling into the interpreter body. The +//! obligation is the caller's on every one of them — `base + ea` must address +//! a live, correctly sized allocation for the whole call, and the traced tier +//! repeats the access with no check of its own. + +/// `rawstorage.py` `raw_storage_getitem` at one byte, sign-extended. +/// +/// Loads widen into the 64-bit int register bank, so the intrinsic's own +/// signedness — not the register's — decides whether the high bits are the +/// sign or zero. The traced tier reads the same width and signedness off the +/// array descr the lowerer attaches. +pub fn majit_raw_load_i8(base: i64, ea: i64) -> i64 { + unsafe { core::ptr::read_unaligned(raw_addr(base, ea) as *const i8) as i64 } +} + +/// `rawstorage.py` `raw_storage_getitem` at one byte, zero-extended. +pub fn majit_raw_load_u8(base: i64, ea: i64) -> i64 { + unsafe { core::ptr::read_unaligned(raw_addr(base, ea) as *const u8) as i64 } +} + +/// `rawstorage.py` `raw_storage_getitem` at two bytes, sign-extended. +pub fn majit_raw_load_i16(base: i64, ea: i64) -> i64 { + unsafe { core::ptr::read_unaligned(raw_addr(base, ea) as *const i16) as i64 } +} + +/// `rawstorage.py` `raw_storage_getitem` at two bytes, zero-extended. +pub fn majit_raw_load_u16(base: i64, ea: i64) -> i64 { + unsafe { core::ptr::read_unaligned(raw_addr(base, ea) as *const u16) as i64 } +} + +/// `rawstorage.py` `raw_storage_getitem` at four bytes, sign-extended. +pub fn majit_raw_load_i32(base: i64, ea: i64) -> i64 { + unsafe { core::ptr::read_unaligned(raw_addr(base, ea) as *const i32) as i64 } +} + +/// `rawstorage.py` `raw_storage_getitem` at four bytes, zero-extended. +pub fn majit_raw_load_u32(base: i64, ea: i64) -> i64 { + unsafe { core::ptr::read_unaligned(raw_addr(base, ea) as *const u32) as i64 } +} + +/// `rawstorage.py` `raw_storage_getitem` at eight bytes. +pub fn majit_raw_load_i64(base: i64, ea: i64) -> i64 { + unsafe { core::ptr::read_unaligned(raw_addr(base, ea) as *const i64) } +} + +/// `rawstorage.py` `raw_storage_getitem` at eight bytes, unsigned descr. +/// +/// At the register width there is no extension gap, so this reads the same +/// bits as [`majit_raw_load_i64`]. It exists because the lowerer accepts the +/// spelling and stamps an unsigned descr for it. +pub fn majit_raw_load_u64(base: i64, ea: i64) -> i64 { + unsafe { core::ptr::read_unaligned(raw_addr(base, ea) as *const u64) as i64 } +} + +/// `rawstorage.py` `raw_storage_getitem` at eight bytes into the FLOAT bank. +/// +/// The only load whose result is a float register; the lowerer stamps a raw +/// float array descr for it rather than a width/signedness pair. +pub fn majit_raw_load_f(base: i64, ea: i64) -> f64 { + unsafe { core::ptr::read_unaligned(raw_addr(base, ea) as *const f64) } +} + +/// `rawstorage.py` `raw_storage_setitem` at one byte. +/// +/// The stored value arrives in an int register and is truncated to the +/// intrinsic's width. Signedness cannot change which bits land, so the signed +/// and unsigned spellings at a given width write identically; they differ only +/// in the descr the traced tier carries. +pub fn majit_raw_store_i8(base: i64, ea: i64, val: i64) { + unsafe { core::ptr::write_unaligned(raw_addr(base, ea) as *mut i8, val as i8) } +} + +/// `rawstorage.py` `raw_storage_setitem` at one byte, unsigned descr. +pub fn majit_raw_store_u8(base: i64, ea: i64, val: i64) { + unsafe { core::ptr::write_unaligned(raw_addr(base, ea) as *mut u8, val as u8) } +} + +/// `rawstorage.py` `raw_storage_setitem` at two bytes. +pub fn majit_raw_store_i16(base: i64, ea: i64, val: i64) { + unsafe { core::ptr::write_unaligned(raw_addr(base, ea) as *mut i16, val as i16) } +} + +/// `rawstorage.py` `raw_storage_setitem` at two bytes, unsigned descr. +pub fn majit_raw_store_u16(base: i64, ea: i64, val: i64) { + unsafe { core::ptr::write_unaligned(raw_addr(base, ea) as *mut u16, val as u16) } +} + +/// `rawstorage.py` `raw_storage_setitem` at four bytes. +pub fn majit_raw_store_i32(base: i64, ea: i64, val: i64) { + unsafe { core::ptr::write_unaligned(raw_addr(base, ea) as *mut i32, val as i32) } +} + +/// `rawstorage.py` `raw_storage_setitem` at four bytes, unsigned descr. +pub fn majit_raw_store_u32(base: i64, ea: i64, val: i64) { + unsafe { core::ptr::write_unaligned(raw_addr(base, ea) as *mut u32, val as u32) } +} + +/// `rawstorage.py` `raw_storage_setitem` at eight bytes. +pub fn majit_raw_store_i64(base: i64, ea: i64, val: i64) { + unsafe { core::ptr::write_unaligned(raw_addr(base, ea) as *mut i64, val) } +} + +/// `rawstorage.py` `raw_storage_setitem` at eight bytes, unsigned descr. +pub fn majit_raw_store_u64(base: i64, ea: i64, val: i64) { + unsafe { core::ptr::write_unaligned(raw_addr(base, ea) as *mut u64, val as u64) } +} + +/// `longlong2float.py` `float2longlong` — a float's 64-bit pattern read as an +/// int, no value change. +/// +/// Lowers to `convert_float_bytes_to_longlong`. A branchless float select uses +/// this pair to stay bit-exact where an arithmetic blend cannot. +pub fn majit_f64_to_bits(x: f64) -> i64 { + x.to_bits() as i64 +} + +/// `longlong2float.py` `longlong2float` — the inverse bitcast, lowering to +/// `convert_longlong_bytes_to_float`. +pub fn majit_bits_to_f64(x: i64) -> f64 { + f64::from_bits(x as u64) +} + +/// Unsigned `<` over the int bank, lowering to the `uint_lt` resop. +/// +/// A `rarithmetic.py` `r_uint` value travels in an ordinary int register as its +/// raw 64-bit pattern, and the lowerer collapses every Rust comparison operator +/// to its SIGNED opcode. An explicit intrinsic is therefore the only way to +/// select the unsigned comparison from the tracing frontend; a bare `<` +/// disagrees with this body for any operand at or above `2^63`. +pub fn majit_uint_lt(a: i64, b: i64) -> i64 { + ((a as u64) < (b as u64)) as i64 +} + +/// Unsigned `<=` over the int bank, lowering to the `uint_le` resop. See +/// [`majit_uint_lt`]. +pub fn majit_uint_le(a: i64, b: i64) -> i64 { + ((a as u64) <= (b as u64)) as i64 +} + +/// Unsigned `/` over the int bank — `rint.py` `ll_uint_py_div`. +/// +/// This lowers to the `int.udiv` oopspec residual call rather than to a trace +/// opcode: `UINT_FLOORDIV` was removed from the resop set, and unsigned +/// division routes through that elidable call instead. +/// +/// The caller must guarantee `b != 0`, exactly the precondition +/// `ll_uint_py_div_zer` wraps: this body divides unconditionally and the +/// compiled tier does too. +pub fn majit_uint_div(a: i64, b: i64) -> i64 { + ((a as u64) / (b as u64)) as i64 +} + +/// Unsigned `%` over the int bank — `rint.py` `ll_uint_py_mod`, lowering to the +/// `int.umod` oopspec residual call. Carries [`majit_uint_div`]'s `b != 0` +/// precondition. +pub fn majit_uint_mod(a: i64, b: i64) -> i64 { + ((a as u64) % (b as u64)) as i64 +} + +/// `rarithmetic.py` `uint_mul_high` — the high 64 bits of the 128-bit unsigned +/// product, zero exactly when `a * b` fits in a `u64`. That makes it the +/// unsigned multiply-overflow test. +/// +/// ⚠ UNLIKE EVERY OTHER NAME HERE, this one is not hard-coded in the lowerer. +/// It is reached only through a `native_int_binops` alias, which matches the +/// call's FULL path against the configured one: +/// +/// ```ignore +/// use majit_metainterp::intrinsics::majit_uint_mul_high; +/// // #[jit_interp(.., native_int_binops = { majit_uint_mul_high => UintMulHigh })] +/// ``` +/// +/// So the alias key and the call site must be spelled the same way — an +/// imported bare name in both, or the same qualified path in both. Configured +/// under one spelling and called under the other, the call lowers as an +/// ordinary residual and the `u128` below is what the trace executes. +pub fn majit_uint_mul_high(a: i64, b: i64) -> i64 { + (((a as u64 as u128) * (b as u64 as u128)) >> 64) as u64 as i64 +} + +/// The effective address a raw-memory intrinsic touches. +/// +/// Wrapping rather than checked because the pair is an address and a byte +/// offset that the caller has already resolved; an overflow here would be a +/// caller defect that a panic in the interpreter tier would report and the +/// traced tier, which computes the same sum in a register, would not. +#[inline] +fn raw_addr(base: i64, ea: i64) -> usize { + (base as usize).wrapping_add(ea as usize) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// A scratch buffer addressed the way an intrinsic addresses it, with no + /// shared reference aliasing the writes: the store tests below mutate + /// through this address. + fn base_of(buf: &mut [u8]) -> i64 { + buf.as_mut_ptr() as usize as i64 + } + + #[test] + fn raw_loads_extend_by_the_intrinsics_own_signedness() { + // Every byte is 0xff, so a signed load reads -1 at any width and an + // unsigned one reads that width's mask — whatever the byte order is. + let mut cell = [0xffu8; 8]; + let base = base_of(&mut cell); + assert_eq!(majit_raw_load_i8(base, 0), -1); + assert_eq!(majit_raw_load_u8(base, 0), 0xff); + assert_eq!(majit_raw_load_i16(base, 0), -1); + assert_eq!(majit_raw_load_u16(base, 0), 0xffff); + assert_eq!(majit_raw_load_i32(base, 0), -1); + assert_eq!(majit_raw_load_u32(base, 0), 0xffff_ffff); + assert_eq!(majit_raw_load_i64(base, 0), -1); + assert_eq!(majit_raw_load_u64(base, 0), -1); + } + + #[test] + fn ea_is_a_byte_offset_not_an_element_index() { + let cells: [i64; 3] = [10, 20, 30]; + let base = cells.as_ptr() as usize as i64; + assert_eq!(majit_raw_load_i64(base, 0), 10); + assert_eq!(majit_raw_load_i64(base, 16), 30); + + // Byte-addressed at one-byte width, so the offset is the index there. + let mut bytes: [u8; 4] = [7, 8, 9, 10]; + let byte_base = base_of(&mut bytes); + for (i, expected) in bytes.iter().enumerate() { + assert_eq!(majit_raw_load_u8(byte_base, i as i64), i64::from(*expected)); + } + } + + #[test] + fn raw_stores_truncate_to_their_width() { + let mut buf = [0u8; 8]; + let base = base_of(&mut buf); + + // A value wider than the intrinsic keeps only the low byte, and the + // neighbouring byte is untouched. + majit_raw_store_u8(base, 0, 0x1ff); + assert_eq!(majit_raw_load_u8(base, 0), 0xff); + assert_eq!(majit_raw_load_u8(base, 1), 0); + + // Signedness cannot change which bits land: two bytes of 0xff either + // way, and nothing beyond them. + majit_raw_store_i64(base, 0, 0); + majit_raw_store_i16(base, 0, -1); + assert_eq!(majit_raw_load_u16(base, 0), 0xffff); + assert_eq!(majit_raw_load_u16(base, 2), 0); + majit_raw_store_i64(base, 0, 0); + majit_raw_store_u16(base, 0, 0xffff); + assert_eq!(majit_raw_load_u16(base, 0), 0xffff); + + // Full width writes the whole cell, and the load side reads it back. + majit_raw_store_i64(base, 0, -1); + assert_eq!(majit_raw_load_i64(base, 0), -1); + } + + #[test] + fn float_load_and_bitcasts_round_trip() { + let cells: [f64; 2] = [1.5, -0.25]; + let base = cells.as_ptr() as usize as i64; + assert_eq!(majit_raw_load_f(base, 8), -0.25); + assert_eq!(majit_bits_to_f64(majit_f64_to_bits(-0.25)), -0.25); + + // The bitcast is a pattern, not a value conversion: the float load and + // the int-bank load of the same eight bytes hold the same bits. + assert_eq!( + majit_f64_to_bits(majit_raw_load_f(base, 8)), + majit_raw_load_i64(base, 8) + ); + + // NaN survives, which is why a branchless float select uses the pair + // instead of an arithmetic blend. + assert!(majit_bits_to_f64(majit_f64_to_bits(f64::NAN)).is_nan()); + } + + #[test] + fn unsigned_intrinsics_disagree_with_the_signed_operators() { + // -1 is the largest u64, which is where every signed operator is wrong + // — the reason these intrinsics exist at all. + assert_eq!(majit_uint_lt(-1, 1), 0); + assert!(-1 < 1); + assert_eq!(majit_uint_le(-1, -1), 1); + assert_eq!(majit_uint_div(-1, 2), (u64::MAX / 2) as i64); + assert_eq!(majit_uint_mod(-1, 10), (u64::MAX % 10) as i64); + + // A zero high word is exactly "the product fits in a u64". + assert_eq!(majit_uint_mul_high(3, 5), 0); + assert_eq!(majit_uint_mul_high(-1, 2), 1); + } +} diff --git a/majit/majit-metainterp/src/jitcode/assembler.rs b/majit/majit-metainterp/src/jitcode/assembler.rs index c6c867875ab..a0590aa7cbc 100644 --- a/majit/majit-metainterp/src/jitcode/assembler.rs +++ b/majit/majit-metainterp/src/jitcode/assembler.rs @@ -1661,6 +1661,27 @@ impl JitCodeBuilder { self.push_reg_u8(dest, "arraylen_vable result"); } + /// Address of item 0 of virtualizable array `array_idx`, as an int. + /// + /// Operand layout is byte-for-byte the `arraylen_vable` triple above, so + /// both decode through `read_vable_arraylen`. + /// + /// Emitting this is an escape: the address outlives the op and lets a + /// callee mutate items represented by the trace's SSA values. Callers must + /// pair it with the `TraceCtx` raw-base escape signal — the macro's + /// lowering is the only caller, and it does. + pub fn vable_arraybase_with_base(&mut self, dest: u16, vable_reg: u16, array_idx: u16) { + self.touch_ref_reg(vable_reg); + self.touch_reg(dest); + let field_descr = self.add_vable_array_field_descr(array_idx); + let array_descr = self.add_vable_array_descr(majit_ir::value::Type::Ref, false); + self.write_insn("arraybase_vable/rdd>i"); + self.push_reg_u8(vable_reg, "arraybase_vable base"); + self.push_u16(field_descr); + self.push_u16(array_descr); + self.push_reg_u8(dest, "arraybase_vable result"); + } + pub fn vable_force_with_base(&mut self, vable_reg: u16) { self.touch_ref_reg(vable_reg); self.write_insn("hint_force_virtualizable/r"); diff --git a/majit/majit-metainterp/src/jitdriver.rs b/majit/majit-metainterp/src/jitdriver.rs index 01aa1dacdb4..89cdfefdcb8 100644 --- a/majit/majit-metainterp/src/jitdriver.rs +++ b/majit/majit-metainterp/src/jitdriver.rs @@ -535,6 +535,185 @@ fn portal_rca_enabled() -> bool { *FLAG.get_or_init(|| std::env::var_os("PYRE_PORTAL_RCA").is_some()) } +// ── warm-entry stage probe ─────────────────────────────────────────────── +// +// A measurement arm, never a shipping one, and behind a cargo feature for that +// reason: a default build has neither the load below nor the loops it feeds. +// Inside the feature the arms are selected at RUN time, because what one stage +// costs against another is a difference that has to be taken inside ONE binary +// — two `cargo build` invocations admit compile drift and stale binaries, and +// neither is visible in the numbers they produce. + +/// Extra passes of one stage of a warm compiled entry. +/// +/// A compiled entry that finds its artifact already there spends ~100 ns +/// arriving, and four fifths of that is inside [`JitDriver::back_edge_internal`] +/// below. No clock can be put inside it: `Instant::now()` costs a fifth of the +/// budget on the box these were taken on, so five of them would report the +/// clock. Each stage is timed by REPETITION instead — the same entry with every +/// count at zero, then again with one count at `k`, and the difference over `k` +/// is that stage. Everything the two arms share cancels out of it. +/// +/// Only stages that are IDEMPOTENT under repetition can be read this way, and +/// the warm entry has one that is emphatically not: **the call**. +/// `execute_assembler_at_dispatch_key` builds the frame, enters the compiled +/// trace and decodes the deadframe it returns (`get_latest_descr_arc` and +/// `decode_exit_slots` are inside it, not after it). It RUNS THE PROGRAM: a +/// second pass would execute the trace again from state the first one already +/// advanced, and would leave through a different exit. There is no honest +/// repeatable form of it, so none is offered — the call is what the residual +/// is, together with the handful of one-shot moves named on the fields below. +#[cfg(feature = "back-edge-stage-probe")] +#[derive(Clone, Copy, Default, Debug, PartialEq, Eq)] +pub struct BackEdgeStageRepeats { + /// Extra gate consultations: `entry_procedure_token` (only for a caller + /// that did not arrive carrying one, which is the shape the gate itself + /// takes), `get_compiled_meta(..).cloned()`, [`JitDriver::driver_descriptor_for`] + /// and `is_compatible`. All four answer the same thing every time they are + /// asked on one entry, which is what makes them repeatable. + /// + /// NOT in this stage, and left in the residual: `resolve_cell_key`, which a + /// carrying caller skips altogether, and + /// `take_single_pass_label_entry_dispatch_key_for_back_edge`, which TAKES + /// the slot — a second call finds it empty, answers `None`, and prices the + /// other branch rather than this one. + pub gate: u16, + /// Extra refills of the entry-argument buffers: `sync_before`, + /// `extract_live_values_into`, `live_values_match_descriptor`, + /// `extend_compiled_live_values_into`, and the buffer clears. + /// + /// The clears belong to the stage rather than to the machinery: + /// [`JitDriver::take_entry_scratch`] does exactly them once per entry, and + /// a refill that skipped them would append to the previous pass instead of + /// repeating it. What is NOT in the stage is that function's own + /// `mem::take` — a second take hands back the defaulted struct. + /// + /// Armed only for an entry that extracted its arguments from state. One + /// that arrived with `direct_live_values` ran none of this. + pub marshal_in: u16, + /// Extra `restore_values` + `sync_after` on the values a completed run left + /// behind. + /// + /// Armed only on the arms that reach them — a FINISH or a normal back-edge + /// JUMP. A guard failure goes to `handle_fail` and reaches neither. + /// `restore_values` is skipped for a FINISH exactly as the shipping arm + /// skips it: those exit slots are the portal's return value and not the + /// loop-carried state. + /// + /// NOT in this stage: `entry_scratch_out`, which moves the buffers back + /// into the driver, and the `drop(result)` on the guard-failure path. Both + /// run once by construction. + pub marshal_out: u16, + /// Extra passes of the same loop with NO stage in it: the counter, and the + /// one optimization barrier every stage's loop also carries. Subtracting it + /// is what leaves a stage's own cost rather than its cost plus the + /// machinery that made it repeat. + pub barrier: u16, +} + +#[cfg(feature = "back-edge-stage-probe")] +impl BackEdgeStageRepeats { + fn pack(self) -> u64 { + (self.gate as u64) + | (self.marshal_in as u64) << 16 + | (self.marshal_out as u64) << 32 + | (self.barrier as u64) << 48 + } + + fn unpack(packed: u64) -> Self { + Self { + gate: packed as u16, + marshal_in: (packed >> 16) as u16, + marshal_out: (packed >> 32) as u16, + barrier: (packed >> 48) as u16, + } + } +} + +/// The counts the next warm entry runs with, on every thread. +/// +/// Process-wide rather than thread-local, for the reason the jitframe pool's +/// arm is (`set_jitframe_pool`): what it selects is a strategy, not a state, and +/// a harness flipping arms between two timed batches wants the flip to hold for +/// whichever thread the next entry runs on. +/// +/// Four counts packed into one word so the entry path takes exactly ONE relaxed +/// load, which every arm pays and which therefore cancels out of the difference +/// between two of them. There is no unseeded state and no environment gate to +/// seed one from: all-zero is the shipping shape, and +/// [`set_back_edge_stage_repeats`] is the only way out of it. +#[cfg(feature = "back-edge-stage-probe")] +static BACK_EDGE_STAGE_REPEATS: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); + +/// Amplified passes actually executed, indexed by the constants below. +#[cfg(feature = "back-edge-stage-probe")] +static BACK_EDGE_STAGE_PASSES: [std::sync::atomic::AtomicU64; 4] = [ + std::sync::atomic::AtomicU64::new(0), + std::sync::atomic::AtomicU64::new(0), + std::sync::atomic::AtomicU64::new(0), + std::sync::atomic::AtomicU64::new(0), +]; + +#[cfg(feature = "back-edge-stage-probe")] +const BACK_EDGE_STAGE_GATE: usize = 0; +#[cfg(feature = "back-edge-stage-probe")] +const BACK_EDGE_STAGE_MARSHAL_IN: usize = 1; +#[cfg(feature = "back-edge-stage-probe")] +const BACK_EDGE_STAGE_MARSHAL_OUT: usize = 2; +#[cfg(feature = "back-edge-stage-probe")] +const BACK_EDGE_STAGE_BARRIER: usize = 3; + +/// Set the repeat counts for subsequent warm entries, answering what they were. +/// +/// A setter and not a wrapper around one entry: the entry's whole cost is read +/// through the frontend's own door, so the stages have to be read through it +/// too, or the parts and the whole are about different doors. +#[cfg(feature = "back-edge-stage-probe")] +pub fn set_back_edge_stage_repeats(repeats: BackEdgeStageRepeats) -> BackEdgeStageRepeats { + BackEdgeStageRepeats::unpack( + BACK_EDGE_STAGE_REPEATS.swap(repeats.pack(), std::sync::atomic::Ordering::Relaxed), + ) +} + +/// Amplified passes since the process started, as +/// `[gate, marshal_in, marshal_out, barrier]` — the field order of +/// [`BackEdgeStageRepeats`]. +/// +/// The witness that an arm was REACHED. Two arms that ran the same code differ +/// by nothing but the box, and a stage figure taken off such a pair describes +/// the box; these counts are what tells the two cases apart without a clock. +#[cfg(feature = "back-edge-stage-probe")] +pub fn back_edge_stage_passes() -> [u64; 4] { + let read = |i: usize| BACK_EDGE_STAGE_PASSES[i].load(std::sync::atomic::Ordering::Relaxed); + [ + read(BACK_EDGE_STAGE_GATE), + read(BACK_EDGE_STAGE_MARSHAL_IN), + read(BACK_EDGE_STAGE_MARSHAL_OUT), + read(BACK_EDGE_STAGE_BARRIER), + ] +} + +/// One relaxed load per warm entry. Both arms pay it. +#[cfg(feature = "back-edge-stage-probe")] +#[inline] +fn back_edge_stage_repeats() -> BackEdgeStageRepeats { + BackEdgeStageRepeats::unpack(BACK_EDGE_STAGE_REPEATS.load(std::sync::atomic::Ordering::Relaxed)) +} + +/// Tally one call's worth of amplified passes for a stage. +/// +/// Once per call rather than once per pass, and the barrier arm pays it too, so +/// the read-modify-write cancels out of every stage's difference instead of +/// being amplified `k`-fold into it. +#[cfg(feature = "back-edge-stage-probe")] +#[inline] +fn count_back_edge_stage_passes(stage: usize, repeats: u16) { + if repeats != 0 { + BACK_EDGE_STAGE_PASSES[stage] + .fetch_add(repeats as u64, std::sync::atomic::Ordering::Relaxed); + } +} + fn format_rca_live_values(labels: Option<&[String]>, values: &[Value]) -> String { let mut out = String::new(); for (idx, value) in values.iter().enumerate() { @@ -4881,10 +5060,15 @@ impl JitDriver { self.arm_single_pass_label_entry_on_next_back_edge(state); self.discard_single_pass_resume(); // A terminal dispatch return means the interpreted function has - // returned. The hook `break`s out of the native loop for it; the - // back edge's own spelling of that is the out-of-range position - // its caller assigns to `pc`, which fails the loop's `pc < len`. - if self.take_single_pass_finish() { + // returned, so the native dispatch loop must EXIT rather than + // resume at a position. `single_pass_finish` is how that is said, + // and it is deliberately left standing here for the caller to + // consume: the `#[jit_interp]` expansion of both markers reads it + // straight after the call and `break`s on it, which is the only + // exit a dispatch loop is required to have. The `usize::MAX` + // beside it is a position no program holds, so a caller that + // ignores the flag fails loudly instead of resuming somewhere. + if self.meta.single_pass_finish { return Some(usize::MAX); } return Some(pc); @@ -5023,6 +5207,15 @@ impl JitDriver { // `EnterJitAssembler` carries `:483`'s read past `maybe_compile_and_run` // and into the executor. The meta below is still fetched here because // its VALUE is needed, not just its presence. + // Read once per entry, ahead of every stage, so no stage's difference + // carries it. See [`BackEdgeStageRepeats`]; a default build has neither + // this load nor the loops it feeds. + #[cfg(feature = "back-edge-stage-probe")] + let stage_repeats = back_edge_stage_repeats(); + // Which shape the gate below takes, recorded before the token is moved + // into it, so the amplified gate can take the same one. + #[cfg(feature = "back-edge-stage-probe")] + let carried_token = carried_procedure_token.is_some(); if let Some(procedure_token) = carried_procedure_token.or_else(|| self.meta.entry_procedure_token(green_key)) && let Some(compiled_meta) = self.meta.get_compiled_meta(green_key).cloned() @@ -5037,6 +5230,36 @@ impl JitDriver { self.meta.invalidate_loop(green_key); return None; } + // Stage E1, and the barrier the other stages are differenced + // against. The gate above has just run, so every consultation it + // makes is repeated here in the same order and under the same + // condition rather than being reordered into a shape the entry + // never takes. + #[cfg(feature = "back-edge-stage-probe")] + { + count_back_edge_stage_passes(BACK_EDGE_STAGE_GATE, stage_repeats.gate); + for _ in 0..stage_repeats.gate { + let token = if carried_token { + None + } else { + self.meta.entry_procedure_token(green_key) + }; + let repeat_meta = self.meta.get_compiled_meta(green_key).cloned(); + let repeat_descriptor = self.driver_descriptor_for(state, &compiled_meta); + let compatible = state.is_compatible(&compiled_meta); + // Without this the four answers are dead and the loop + // prices nothing. + std::hint::black_box((token, repeat_meta, repeat_descriptor, compatible)); + std::hint::black_box(&mut *state); + } + // The same loop and the same barrier with no stage in them, so + // what the amplification itself costs is subtracted rather than + // reported as a stage. + count_back_edge_stage_passes(BACK_EDGE_STAGE_BARRIER, stage_repeats.barrier); + for _ in 0..stage_repeats.barrier { + std::hint::black_box(&mut *state); + } + } if !self.sync_before(state, &compiled_meta, vable) { return None; } @@ -5053,6 +5276,49 @@ impl JitDriver { // order; state-extracted ones are in state-field order and have to be // mapped before a dispatch-key entry can use them. let values_are_label_ordered = direct_live_values.is_some(); + // Stage E2, ahead of the shipping refill rather than after it: each + // pass ENDS with the clears `take_entry_scratch` performs, so the + // buffers the real extraction below fills are in exactly the state + // it would have found them in, whatever any pass answered. + #[cfg(feature = "back-edge-stage-probe")] + if !values_are_label_ordered { + count_back_edge_stage_passes(BACK_EDGE_STAGE_MARSHAL_IN, stage_repeats.marshal_in); + for _ in 0..stage_repeats.marshal_in { + let synced = self.sync_before(state, &compiled_meta, vable); + state.extract_live_values_into( + &compiled_meta, + &mut scratch.live_values, + &mut scratch.raw, + &mut scratch.types, + ); + let matched = Self::live_values_match_descriptor( + descriptor.as_deref(), + &scratch.live_values, + state.state_field_layout().total_live_values(), + ); + let extended = self.extend_compiled_live_values_into( + green_key, + state, + &compiled_meta, + vable, + &mut scratch.live_values, + &mut scratch.vable_static, + &mut scratch.vable_arrays, + ); + // Answered and not acted on: the shipping pass below takes + // the decision, and a probe pass that declined would only + // mean this stage cannot be amplified on this shape. + std::hint::black_box((synced, matched, extended)); + std::hint::black_box(&mut *state); + scratch.live_values.clear(); + scratch.raw.clear(); + scratch.types.clear(); + scratch.vable_static.clear(); + for array in &mut scratch.vable_arrays { + array.clear(); + } + } + } if let Some(values) = direct_live_values { scratch.live_values.extend(values); } else { @@ -5214,6 +5480,24 @@ impl JitDriver { ); } + // Stage E4, before the arm split, so it prices one pair of calls + // rather than a different pair per outcome — and before the FINISH + // arm takes the exit values out from under it. + #[cfg(feature = "back-edge-stage-probe")] + if result.is_finish || result.fail_index == u32::MAX { + count_back_edge_stage_passes( + BACK_EDGE_STAGE_MARSHAL_OUT, + stage_repeats.marshal_out, + ); + for _ in 0..stage_repeats.marshal_out { + if !result.is_finish && !result.typed_values.is_empty() { + state.restore_values(&result.meta, &result.typed_values); + } + self.sync_after(state, &result.meta, vable); + std::hint::black_box(&mut *state); + } + } + if result.is_finish { // compile.py `_DoneWithThisFrameDescr.final_descr = True`: // the compiled run ended in FINISH, so the traced function has @@ -6990,6 +7274,17 @@ impl JitDriver { self.meta.has_compiled_loop(green_key) } + /// [`Self::has_compiled_loop`] with both flag reads removed, so an + /// amplified arm can price the cell lookup and the `Weak::upgrade`/drop + /// pair without them. A cost probe with a deliberately weaker answer than + /// the door's -- see `WarmState::probe_cell_token_upgrades`. Nothing may + /// route on it. + #[cfg(feature = "yield-stage-probe")] + #[inline] + pub fn probe_cell_token_upgrades(&self, green_key: u64) -> bool { + self.meta.probe_cell_token_upgrades(green_key) + } + /// Whether the warm-entry runner can actually execute the code at this /// green key. Stronger than `has_compiled_loop`: it also requires a /// frontend `compiled_loops` meta (`get_compiled_meta`). diff --git a/majit/majit-metainterp/src/lib.rs b/majit/majit-metainterp/src/lib.rs index 0c5b9b1c7a3..2ac35c560be 100644 --- a/majit/majit-metainterp/src/lib.rs +++ b/majit/majit-metainterp/src/lib.rs @@ -35,6 +35,15 @@ //! threaded through metainterp optimizers. //! * `io_buffer`, `jit_state`, `trace_ctx`, and `parity` are pyre //! runtime/test boundaries with no same-named upstream file. +//! * `intrinsics` has no same-named upstream file: it is the untraced half of +//! the intrinsics `#[jit_interp]` rewrites while tracing, whose traced half +//! lives in the macro lowerer. RPython needs no such module because it +//! lowers `rawstorage.py` / `longlong2float.py` / `rarithmetic.py` at rtype +//! time from the one definition each already has. +//! * `embed` has no same-named upstream file: it holds the per-run census an +//! interpreter would otherwise rebuild around the driver callbacks. The +//! upstream equivalent is split between `warmspot.py`'s test harness and +//! `jitprof.py`, neither of which is a library surface an embedder calls. //! * `jitcode` and `recorder` are transitional runtime ABI boundaries //! around canonical translate-side `jitcode.py` / `opencoder.py` //! ports; their module docs describe the remaining migration path. @@ -106,12 +115,14 @@ pub mod counter; pub use majit_backend::model as cpu; pub use majit_ir::Value; pub use majit_ir::debug; +pub mod embed; pub mod executor; pub mod gc; pub mod graphpage; pub mod greenfield; pub mod heapcache; pub mod history; +pub mod intrinsics; pub(crate) mod io_buffer; pub mod jit; mod jit_state; @@ -183,7 +194,22 @@ pub use jitdriver::{ TraceContinuationSuspendGuard, current_state_field_fvc_epoch, drive_multi_frame_blackhole, drive_single_frame_blackhole, no_bridge_enabled, trace_continuation_suspended, }; +// The warm-entry stage probe, which an embedder drives from its own harness — +// the split has to be read through the frontend's own door, so the counts are +// set from outside and the door is left the shipping one. +#[cfg(feature = "back-edge-stage-probe")] +pub use jitdriver::{BackEdgeStageRepeats, back_edge_stage_passes, set_back_edge_stage_repeats}; +// The compiled-run split, which reaches one crate further down than the rest of +// this probe: the frame build it prices is the backend's, so the count for that +// one arm is set through `majit_backend` and only its loop is in the backend. pub use majit_backend::CompiledTraceInfo; +#[cfg(feature = "execute-stage-probe")] +pub use majit_backend::deadframe::{frame_build_passes, set_frame_build_repeats}; +#[cfg(feature = "execute-stage-probe")] +pub use pyjitpl::{ + ExecuteStageRepeats, call_shot_totals, execute_stage_clock_floor_ns, execute_stage_passes, + reset_call_shot_totals, set_execute_stage_repeats, +}; pub use pyjitpl::{eval_binop_f, eval_binop_i, eval_float_cmp, eval_unary_f, eval_unary_i}; // Re-export the canonical translate-side Assembler so macro-emitted // state-field JIT setup (e.g. `__JitMeta_::install_canonical_liveness`) @@ -655,39 +681,16 @@ pub fn dispatch_arm_census() -> Vec { /// Panic unless `interp`'s portal was installed AND none of its arms degraded. /// -/// This is the gate every consumer would otherwise write, denominator and all. -/// Reading `degraded_dispatch_arms()` alone cannot be that gate: it passes on -/// an empty registry, and an empty registry is also what a portal that was -/// never built produces. The census settles which one happened, so the two -/// failures get two different messages instead of one silent pass. +/// The zero case of [`embed::assert_degraded_dispatch_arms`], which is where +/// the implementation and the reasoning live. Reach for the general form +/// whenever the answer is not zero: a machine with a known lowering gap has a +/// non-empty degraded set today, and this spelling can only fail on it. /// -/// Call it after whatever installs the portal. `#[jit_interp]` records both -/// facts at install, not at trace time, so running the machine is not required -/// — but nothing is recorded until the portal is built at least once. +/// Call it after whatever installs the portal. Both facts are recorded at +/// install, not at trace time, so running the machine is not required — but +/// nothing is recorded until the portal is built at least once. pub fn assert_no_degraded_dispatch_arms(interp: &str) { - let census = dispatch_arm_census(); - let Some(entry) = census.iter().find(|e| e.interp == interp) else { - panic!( - "no dispatch-arm census for `{interp}`: its portal was never \ - installed in this process, so an empty degraded list says nothing \ - about it. Build the dispatch JitCode (or run the machine) first. \ - Recorded machines: {:?}", - census.iter().map(|e| e.interp).collect::>() - ); - }; - let degraded: Vec = degraded_dispatch_arms() - .into_iter() - .filter(|e| e.interp == interp) - .collect(); - assert!( - degraded.is_empty(), - "{} of `{interp}`'s {} dispatch arms lowered to an abort stub. Every \ - trace that reaches one of these opcodes aborts, once per threshold, \ - forever: {:#?}", - degraded.len(), - entry.arms, - degraded - ); + embed::assert_degraded_dispatch_arms(interp, &[]); } /// A declared field key that no access site asked about. @@ -1255,6 +1258,12 @@ pub fn green_key_hash_typed(values: &[i64], types: &[majit_ir::GreenType]) -> u6 // Re-exported from majit-codegen so both meta and backend can access it. pub use majit_backend::{JittedGuard, set_jitted, we_are_jitted}; +// ── jitframe allocation arm ── +// An embedder measuring the compiled entry needs to select the arm from its own +// harness, and `majit-backend` is not one of its dependencies — every embedder +// reaches the backends through this crate. +pub use majit_backend::deadframe::{jitframe_pool_counts, set_jitframe_pool}; + // ── rstack criticalcode hooks ── // rpython/translator/c/src/stack.h:42-43 LL_stack_criticalcode_start/stop. // Used by blackhole_from_resumedata / handle_async_forcing / diff --git a/majit/majit-metainterp/src/pyjitpl.rs b/majit/majit-metainterp/src/pyjitpl.rs index 52509f9b4f7..3e0a7bd53b0 100644 --- a/majit/majit-metainterp/src/pyjitpl.rs +++ b/majit/majit-metainterp/src/pyjitpl.rs @@ -167,6 +167,149 @@ fn guardlog_enabled() -> bool { *ENABLED.get_or_init(|| std::env::var_os("MAJIT_GUARDLOG").is_some()) } +// ── compiled-run stage probe ───────────────────────────────────────────── +// +// A measurement arm, never a shipping one, behind a cargo feature so a default +// build has neither the load below nor the loops it feeds. Inside the feature +// the arms are selected at RUN time, because what one stage costs against +// another is a difference that has to be taken inside ONE binary. + +/// Extra passes of one stage of `execute_assembler_at_dispatch_key`. +/// +/// That function is the residual the entry split left behind — the largest +/// single term in a compiled entry — and it is NOT one thing. Two of its parts +/// answer the same thing every time they are asked and can be amplified; one +/// runs the trace and can never be. +/// +/// ⚠ THE TWO KINDS OF NUMBER THIS PRODUCES ARE NOT COMPARABLE, and a reader +/// must be told which is which: +/// +/// * `prologue` and `decode` are AMPLIFIED. Run k extra times, subtract the +/// barrier, divide by k. The usual reading, and a lower bound: the passes +/// after the first find the caches the first one filled. +/// * `call_shot` is SINGLE-SHOT. The compiled call cannot be repeated — it runs +/// the trace, consumes the inputs and produces a deadframe, so a second pass +/// would execute the program again from state the first one advanced. It is +/// instead CLOCKED, one reading per entry, accumulated. Two `Instant::now()` +/// calls land inside the entry when this arm is armed, so the arm's own +/// whole-entry figure is inflated and must not be quoted; only the +/// accumulated call figure is, and [`execute_stage_clock_floor_ns`] is what +/// the clock's own cost is subtracted with. +/// +/// The frame build and the input-argument writes are the repeatable PREFIX of +/// that call and are amplified on the backend's side of the boundary instead — +/// `majit_backend::deadframe::set_frame_build_repeats`, whose loop lives in +/// `run_compiled_code_inner`, because that is where the frame is. +#[cfg(feature = "execute-stage-probe")] +#[derive(Clone, Copy, Default, Debug, PartialEq, Eq)] +pub struct ExecuteStageRepeats { + /// Extra `compiled_loops` probes and meta clones, plus + /// `prepare_compiled_run_io`. All answer the same thing on one entry. + pub prologue: u16, + /// Extra deadframe decodes: `get_latest_descr_arc`, the four descr reads, + /// `fail_arg_types`, and `decode_exit_slots`. These read the frame the run + /// returned and build two fresh lists; they do not touch it, so they + /// repeat. + pub decode: u16, + /// Extra passes of the same loop with NO stage in it. + pub barrier: u16, + /// Nonzero clocks the compiled call itself, once per entry. SINGLE-SHOT — + /// see the type's own note. + pub call_shot: u16, +} + +#[cfg(feature = "execute-stage-probe")] +impl ExecuteStageRepeats { + fn pack(self) -> u64 { + (self.prologue as u64) + | (self.decode as u64) << 16 + | (self.barrier as u64) << 32 + | (self.call_shot as u64) << 48 + } + + fn unpack(packed: u64) -> Self { + Self { + prologue: packed as u16, + decode: (packed >> 16) as u16, + barrier: (packed >> 32) as u16, + call_shot: (packed >> 48) as u16, + } + } +} + +/// Four counts packed into one word, so the compiled-run path takes exactly ONE +/// relaxed load that every arm pays and which therefore cancels out of the +/// difference between two of them. +#[cfg(feature = "execute-stage-probe")] +static EXECUTE_STAGE_REPEATS: AtomicU64 = AtomicU64::new(0); + +/// Amplified passes actually performed, `[prologue, decode, barrier]`. +#[cfg(feature = "execute-stage-probe")] +static EXECUTE_STAGE_PASSES: [AtomicU64; 3] = + [AtomicU64::new(0), AtomicU64::new(0), AtomicU64::new(0)]; + +/// Nanoseconds accumulated across single-shot readings of the compiled call, +/// and how many readings that is. +#[cfg(feature = "execute-stage-probe")] +static CALL_SHOT_NS: AtomicU64 = AtomicU64::new(0); +#[cfg(feature = "execute-stage-probe")] +static CALL_SHOT_COUNT: AtomicU64 = AtomicU64::new(0); + +/// Set the repeat counts for subsequent compiled runs, answering what they were. +#[cfg(feature = "execute-stage-probe")] +pub fn set_execute_stage_repeats(repeats: ExecuteStageRepeats) -> ExecuteStageRepeats { + ExecuteStageRepeats::unpack(EXECUTE_STAGE_REPEATS.swap(repeats.pack(), Ordering::Relaxed)) +} + +/// `[prologue, decode, barrier]` amplified passes since the process started. +#[cfg(feature = "execute-stage-probe")] +pub fn execute_stage_passes() -> [u64; 3] { + [ + EXECUTE_STAGE_PASSES[0].load(Ordering::Relaxed), + EXECUTE_STAGE_PASSES[1].load(Ordering::Relaxed), + EXECUTE_STAGE_PASSES[2].load(Ordering::Relaxed), + ] +} + +/// `(nanoseconds, readings)` accumulated by the single-shot call arm. +/// +/// The mean is the call PLUS one clock pair's own cost; subtract +/// [`execute_stage_clock_floor_ns`] to get the call. +#[cfg(feature = "execute-stage-probe")] +pub fn call_shot_totals() -> (u64, u64) { + ( + CALL_SHOT_NS.load(Ordering::Relaxed), + CALL_SHOT_COUNT.load(Ordering::Relaxed), + ) +} + +/// Reset the single-shot accumulators, so a harness can bracket one measured +/// window rather than reading the whole process's history. +#[cfg(feature = "execute-stage-probe")] +pub fn reset_call_shot_totals() { + CALL_SHOT_NS.store(0, Ordering::Relaxed); + CALL_SHOT_COUNT.store(0, Ordering::Relaxed); +} + +/// What an empty `Instant::now()` / `elapsed()` pair reads on this box, in +/// nanoseconds — the floor under every single-shot figure. +/// +/// Measured HERE rather than in the harness so it is the same clock, the same +/// crate and the same optimization settings as the reading it corrects. Taking +/// the minimum rather than the mean: the pair cannot run faster than it is, so +/// the smallest of many readings is the least contaminated one, and subtracting +/// a mean inflated by scheduler noise would under-report the call. +#[cfg(feature = "execute-stage-probe")] +pub fn execute_stage_clock_floor_ns() -> f64 { + let mut best = u128::MAX; + for _ in 0..4096 { + let start = std::time::Instant::now(); + let seen = start.elapsed().as_nanos(); + best = best.min(seen); + } + best as f64 +} + /// compile.py `forget_optimization_info` — discard optimizer-only /// forwarding state before handing a trace to the backend. The /// `reset_values` arm is unported because neither send-to-backend call site @@ -11007,12 +11150,53 @@ impl MetaInterp { ) -> Option> { let meta = self.compiled_loops.get(&green_key)?.meta.clone(); + // Read once per run, ahead of every stage, so no stage's difference + // carries it. See [`ExecuteStageRepeats`]; a default build has neither + // this load nor the loops it feeds. + #[cfg(feature = "execute-stage-probe")] + let stage_repeats = + ExecuteStageRepeats::unpack(EXECUTE_STAGE_REPEATS.load(Ordering::Relaxed)); + #[cfg(feature = "execute-stage-probe")] + { + if stage_repeats.prologue != 0 { + EXECUTE_STAGE_PASSES[0] + .fetch_add(u64::from(stage_repeats.prologue), Ordering::Relaxed); + } + for _ in 0..stage_repeats.prologue { + let repeat_meta = self.compiled_loops.get(&green_key).map(|c| c.meta.clone()); + Self::prepare_compiled_run_io(); + std::hint::black_box(repeat_meta); + } + // The same loop and the same barrier with no stage in them, so what + // the amplification itself costs is subtracted rather than reported + // as a stage. + if stage_repeats.barrier != 0 { + EXECUTE_STAGE_PASSES[2] + .fetch_add(u64::from(stage_repeats.barrier), Ordering::Relaxed); + } + for _ in 0..stage_repeats.barrier { + std::hint::black_box(&green_key); + } + } + Self::prepare_compiled_run_io(); + // SINGLE-SHOT, and the only stage here that is: the call runs the + // trace, so it is clocked once rather than repeated. Both arms take the + // same branch shape; only the armed one pays the two clock reads, and + // that cost stays inside the figure this accumulates rather than + // leaking into the amplified stages above. + #[cfg(feature = "execute-stage-probe")] + let call_started = (stage_repeats.call_shot != 0).then(std::time::Instant::now); let frame = self.backend.execute_token_with_dispatch_key( procedure_token, live_values, dispatch_key, ); + #[cfg(feature = "execute-stage-probe")] + if let Some(started) = call_started { + CALL_SHOT_NS.fetch_add(started.elapsed().as_nanos() as u64, Ordering::Relaxed); + CALL_SHOT_COUNT.fetch_add(1, Ordering::Relaxed); + } // RPython: bridge compilation happens synchronously inside // assembler_call_helper (called from compiled code). No deferred queue. @@ -11036,6 +11220,32 @@ impl MetaInterp { // The exit slots are read for both outcomes, so they are decoded before // the split rather than once in each arm. let (values, typed_values) = Self::decode_exit_slots(&self.backend, &frame, exit_types); + // The deadframe decode, amplified. It READS the frame the run returned + // and builds two fresh lists; it does not touch the frame, so it + // repeats. Each pass drops what it built, which is the same drop the + // shipping pass eventually pays for its own. + #[cfg(feature = "execute-stage-probe")] + { + if stage_repeats.decode != 0 { + EXECUTE_STAGE_PASSES[1] + .fetch_add(u64::from(stage_repeats.decode), Ordering::Relaxed); + } + for _ in 0..stage_repeats.decode { + let repeat_descr = self.backend.get_latest_descr_arc(&frame); + let repeat_fail = repeat_descr + .as_fail_descr() + .expect("get_latest_descr_arc returned a non-FailDescr Descr"); + let repeat_types: &[Type] = repeat_fail.fail_arg_types(); + let decoded = Self::decode_exit_slots(&self.backend, &frame, repeat_types); + std::hint::black_box(( + repeat_fail.fail_index(), + repeat_fail.trace_id(), + repeat_fail.is_finish(), + repeat_fail.is_exit_frame_with_exception(), + &decoded, + )); + } + } // `warmstate.py:404-418`: "First, a fast path to avoid raising and // immediately catching a DoneWithThisFrame exception". On a final @@ -11955,6 +12165,14 @@ impl MetaInterp { .filter(|token| token.has_compiled_code()) } + /// [`Self::has_compiled_loop`] with both flag reads removed, for pricing + /// the refcount pair alone. See `WarmState::probe_cell_token_upgrades` -- + /// a cost probe, never a decision. + #[cfg(feature = "yield-stage-probe")] + pub fn probe_cell_token_upgrades(&self, green_key: u64) -> bool { + self.warm_state.probe_cell_token_upgrades(green_key) + } + /// `warmstate.py:458-464` — the same code-presence gate as /// [`Self::has_compiled_loop`], resolved on the full green key instead of /// on the bucket head. diff --git a/majit/majit-metainterp/src/pyjitpl/dispatch.rs b/majit/majit-metainterp/src/pyjitpl/dispatch.rs index 2ab3b0751a5..dcc1c9f74ce 100644 --- a/majit/majit-metainterp/src/pyjitpl/dispatch.rs +++ b/majit/majit-metainterp/src/pyjitpl/dispatch.rs @@ -1437,10 +1437,30 @@ where /// escaped during this CALL_MAY_FORCE. Returns the still-rooted /// [`ActiveStandardVirtualizable`] so the caller can read the heap object /// before dropping the shadow-stack root. + /// + /// Two signals meet here. The token check is the upstream one and answers + /// "did the callee force the virtualizable". It cannot answer for a callee + /// handed the array's raw base address: such a callee never touches the + /// token, and an `#[jit_interp]` machine has no token to touch + /// (`without_vable_token`), so the check returns `false` for precisely the + /// calls most able to invalidate the trace. `TraceCtx` carries that case + /// from the array-base lowering to this post-call check. fn escaped_standard_virtualizable( + ctx: &mut TraceCtx, active: Option, ) -> Option { - active.filter(|active| unsafe { active.info.tracing_after_residual_call(active.obj_ptr()) }) + // Drained before the `filter`, and unconditionally: with no active + // virtualizable there is nothing to filter, and a signal left standing + // would abort the next, unrelated residual call. + let raw_base_escape = ctx.take_raw_vable_base_escape(); + active.filter(|active| { + // Evaluated first and not short-circuited past: on a token-bearing + // virtualizable this call is what clears the token back to NONE, + // and skipping it would trip the `== 0` assert in + // `tracing_before_residual_call` at the next call instead. + let forced = unsafe { active.info.tracing_after_residual_call(active.obj_ptr()) }; + forced || raw_base_escape + }) } fn finalize_standard_virtualizable_may_force( @@ -1449,7 +1469,7 @@ where sym: &mut S, active: Option, ) -> TraceAction { - if let Some(active) = Self::escaped_standard_virtualizable(active) { + if let Some(active) = Self::escaped_standard_virtualizable(ctx, active) { // pyjitpl.py `self.load_fields_from_virtualizable()` runs // BEFORE the abort: the residual call forced the virtualizable and // wrote its fields through the heap object, so the tracing-time @@ -4958,6 +4978,28 @@ where .unwrap_or(0); self.set_int_reg(dest, Some(result), Some(len as i64)); } + jitcode::insns::BC_ARRAYBASE_VABLE => { + // Same `rdd>i` operand triple as `arraylen_vable` above, hence + // the shared decoder. + let (vable_reg, array_idx, dest) = { + let frame = self.frames.current_mut(); + frame.read_vable_arraylen() + }; + let Some((_vable_opref, fdescr, _adescr)) = + self.vable_array_descrs(ctx, vable_reg, array_idx) + else { + return TraceAction::Abort; + }; + let vable_struct_ptr = self.read_ref_reg(vable_reg).1; + // An unresolvable base aborts rather than defaulting: the walk + // really executes the residual call this address feeds, so a + // placeholder would be handed to a live callee. + let Some((result, addr)) = ctx.vable_arraybase_vable(vable_struct_ptr, fdescr) + else { + return TraceAction::Abort; + }; + self.set_int_reg(dest, Some(result), Some(addr)); + } jitcode::insns::BC_HINT_FORCE_VIRTUALIZABLE => { let vable_reg = self.frames.current_mut().next_reg() as usize; let vable_opref = self.resolve_vable_box(vable_reg); diff --git a/majit/majit-metainterp/src/resume_box_reader.rs b/majit/majit-metainterp/src/resume_box_reader.rs index 1725f77f67d..b0241cb0513 100644 --- a/majit/majit-metainterp/src/resume_box_reader.rs +++ b/majit/majit-metainterp/src/resume_box_reader.rs @@ -1074,16 +1074,11 @@ pub fn seed_bridge_virtualizable_boxes( // compile.py:27), which is strictly more conservative than upstream: the // guard keeps deopting through the blackhole exactly as it did before. // - // The two writes upstream pairs with the assignment are both inert for this - // seed's only consumer, so neither is ported: - // * `reset_token_gcref` — the state-field vinfo is built by - // `VirtualizableInfo::without_vable_token()`, whose token protocol - // no-ops (`codegen_state.rs` `__build_virtualizable_info`). - // * `synchronize_virtualizable()` — `TraceCtx::synchronize_virtualizable` - // deliberately skips the write-back for `RustVec` array storage, which - // is what every `[.. ; virt]` state field is: the macro-generated - // mainloop owns that struct and writes it on every opcode, so the heap - // is authoritative and flushing the shadow back would clobber it. + // Of the two writes upstream pairs with the assignment, `reset_token_gcref` + // is inert for this seed's only consumer — the state-field vinfo is built by + // `VirtualizableInfo::without_vable_token()`, whose token protocol no-ops + // (`codegen_state.rs` `__build_virtualizable_info`) — so it is not ported. + // `synchronize_virtualizable()` is ported, below the seed. match ctx.virtualizable_heap_ptr() { Some(live) if live == vable_ptr => {} Some(_) | None => return false, @@ -1121,5 +1116,20 @@ pub fn seed_bridge_virtualizable_boxes( )); values.push(identity_value); ctx.set_virtualizable_boxes_with_info(boxes, values, info, &array_lengths); + // `rebuild_state_after_failure`'s trailing `self.synchronize_virtualizable()` + // (pyjitpl.py) — the object and the shadow have to agree before the bridge + // replays a single vable op. + // + // A token-less vinfo is a `#[jit_interp]` `state` struct, and it is the + // family with no other writer for this: the compiled loop carries its banks + // in machine registers, so the guard leaves the struct holding whatever the + // run was entered with, and the macro-generated mainloop that otherwise + // keeps it current ran no opcode of that entry. A token-bearing one is a + // host object whose slots have a boxing protocol the generic + // `value_to_raw_bits` cannot serve, and whose own field-aware guard-failure + // writer has already synchronized it. + if !info.has_vable_token() { + ctx.synchronize_virtualizable(); + } true } diff --git a/majit/majit-metainterp/src/trace_ctx.rs b/majit/majit-metainterp/src/trace_ctx.rs index 5e9c8ab52f9..405f073ca32 100644 --- a/majit/majit-metainterp/src/trace_ctx.rs +++ b/majit/majit-metainterp/src/trace_ctx.rs @@ -305,6 +305,9 @@ pub struct TraceCtx { /// (virtualizable.py write_boxes parity). `None` disables the /// write — unit-test or init-before-run path. virtualizable_heap_ptr: Option<*const u8>, + /// Set when the current residual call receives a virtualizable array's raw + /// base address. Drained by the post-call escape check. + raw_vable_base_escape_pending: bool, /// Header PC at which this trace started (0 = function entry). pub header_pc: usize, /// When a cross-loop cut occurs (trace closes at inner loop header), @@ -1707,6 +1710,7 @@ impl TraceCtx { virtualizable_info: None, virtualizable_array_lengths: None, virtualizable_heap_ptr: None, + raw_vable_base_escape_pending: false, header_pc: 0, cut_inner_green_key: None, inline_loop_abort_pending: false, @@ -1801,6 +1805,7 @@ impl TraceCtx { virtualizable_info: None, virtualizable_array_lengths: None, virtualizable_heap_ptr: None, + raw_vable_base_escape_pending: false, header_pc: 0, cut_inner_green_key: None, inline_loop_abort_pending: false, @@ -5491,6 +5496,50 @@ impl TraceCtx { self.record_op_with_descr(OpCode::ArraylenGc, &[array_opref], adescr) } + /// Address of item 0 of a virtualizable array field, as a constant int, + /// and raise the raw-base escape signal. + /// + /// `None` when the array's layout cannot be resolved. That is not a + /// recoverable miss to paper over with a zero or a recorded op: the walker + /// really performs the residual call this address is an argument to, so an + /// unresolved base would hand a live callee a wrong pointer and corrupt + /// memory. The caller aborts the trace instead. + /// + /// The address is read from the live heap object rather than recorded as + /// an operation because the trace does not survive: raising the escape + /// signal here means the enclosing CALL_MAY_FORCE aborts with + /// ABORT_ESCAPE before this constant can reach an optimizer or a backend. + /// Returns the trace-side constant and the same address as a concrete, so + /// the caller can stamp the destination register with both. + pub fn vable_arraybase_vable( + &mut self, + vable_struct_ptr: i64, + fdescr: DescrRef, + ) -> Option<(OpRef, i64)> { + if vable_struct_ptr == 0 { + return None; + } + let info = self.virtualizable_info.as_ref()?; + let array_idx = info.array_field_by_descr(&fdescr)?; + let array = info.array_fields.get(array_idx)?; + let base = unsafe { + crate::virtualizable::bhimpl_arraybase_vable(vable_struct_ptr as *const u8, array) + }; + if base.is_null() { + return None; + } + // Raised only once the address is known to be real, so an aborted + // resolution above cannot leave a signal behind for the next call. + self.raw_vable_base_escape_pending = true; + let addr = base as usize as i64; + Some((self.const_int(addr), addr)) + } + + /// Consume the raw-base escape signal for the current residual call. + pub(crate) fn take_raw_vable_base_escape(&mut self) -> bool { + std::mem::take(&mut self.raw_vable_base_escape_pending) + } + /// Compute the flat index into virtualizable_boxes for an array element. /// Returns `None` if standard virtualizable is not active or the array field is unknown. fn vable_array_flat_index(&self, fdescr: &DescrRef, item_index: usize) -> Option { diff --git a/majit/majit-metainterp/src/warmstate.rs b/majit/majit-metainterp/src/warmstate.rs index 47c79bc4631..5991934b614 100644 --- a/majit/majit-metainterp/src/warmstate.rs +++ b/majit/majit-metainterp/src/warmstate.rs @@ -1414,6 +1414,28 @@ impl WarmEnterState { .and_then(|cell| cell.get_procedure_token()) } + /// [`Self::get_procedure_token`] with the two flag reads taken OUT: the + /// cell lookup, the `Weak::upgrade`, and the drop of the `Arc` it + /// produced, and nothing else. + /// + /// A measurement reader with no production caller and no upstream + /// counterpart. It exists because the shipping predicate fuses two + /// separable costs into one function -- `get_procedure_token` performs the + /// refcount pair AND reads `invalidated` -- so no arm built from the public + /// API can say which of them a sibling-door scan is paying for. Differenced + /// against `has_compiled_loop`, this one leaves the flag reads. + /// + /// The answer is deliberately WEAKER than the shipping predicate's: an + /// invalidated token still upgrades, so this returns `true` where the door + /// would decide `false`. That is why it is a cost probe and never a + /// decision -- nothing may route on it. + #[cfg(feature = "yield-stage-probe")] + pub fn probe_cell_token_upgrades(&self, cell_key: u64) -> bool { + self.cell_by_key(cell_key) + .and_then(|cell| cell.loop_token.as_ref()) + .is_some_and(|weak| weak.upgrade().is_some()) + } + /// `warmstate.py` — resolve the cell by `comparekey`, then read its /// procedure token. /// diff --git a/majit/majit-metainterp/tests/arraybase_vable_has_no_emitter.rs b/majit/majit-metainterp/tests/arraybase_vable_has_no_emitter.rs new file mode 100644 index 00000000000..4acd979fc67 --- /dev/null +++ b/majit/majit-metainterp/tests/arraybase_vable_has_no_emitter.rs @@ -0,0 +1,152 @@ +//! `BC_ARRAYBASE_VABLE` is defined, decoded, walked, wired into the blackhole +//! and assemblable — and **nothing emits it**. This test is what keeps that +//! sentence true. +//! +//! The opcode exists ahead of its only intended producer: a `#[jit_interp]` +//! lowering for `state..as_mut_ptr()`, held back because the walk +//! resumes at the residual's own argument byte rather than at the next opcode +//! (see the `BC_ARRAYBASE_VABLE` comment in `majit-translate`'s `insns.rs`). +//! Until that is fixed, an emitter would turn a dormant opcode into a wrong +//! answer. +//! +//! ## Why a test and not a comment +//! +//! The opcode already carries a comment saying nothing emits it. Prose +//! describing an **absence** is uniquely fragile: it stays on the page, +//! reading as current, at the exact moment someone adds the emitter that makes +//! it false. Nothing about writing that emitter brings the reader past the +//! sentence. A test converts "this claim quietly went stale" into a red. +//! +//! ## Why the emitter and not the byte +//! +//! `vable_arraybase_with_base` is the single choke point — the only way the +//! opcode's byte reaches a jitcode. Scanning for `BC_ARRAYBASE_VABLE` instead +//! would match its definition, its name-map registration and its handler +//! wiring, all of which are supposed to exist, so the interesting signal would +//! have to be separated from three legitimate ones. +//! +//! The emitter is located by **symbol, not by path**: the file it lives in has +//! moved once already and may again. + +use std::path::{Path, PathBuf}; + +/// The only way to put `BC_ARRAYBASE_VABLE` into a jitcode. +const EMITTER: &str = "vable_arraybase_with_base"; + +/// Crates scanned. The Rust workspace lives under these two roots; everything +/// else at the repository root is Python, fixtures, or untracked scratch. +const ROOTS: [&str; 2] = ["majit", "pyre"]; + +fn repo_root() -> PathBuf { + // `majit/majit-metainterp` -> `majit` -> repository root. + Path::new(env!("CARGO_MANIFEST_DIR")) + .ancestors() + .nth(2) + .expect("manifest dir has two ancestors") + .to_path_buf() +} + +fn collect_rust_sources(dir: &Path, out: &mut Vec) { + let Ok(entries) = std::fs::read_dir(dir) else { + return; + }; + for entry in entries.flatten() { + let path = entry.path(); + let name = entry.file_name(); + let name = name.to_string_lossy(); + if path.is_dir() { + // `target` holds generated copies of the sources being scanned; + // counting them would report the same call site many times over. + if name == "target" || name.starts_with('.') { + continue; + } + collect_rust_sources(&path, out); + } else if path.extension().is_some_and(|e| e == "rs") { + out.push(path); + } + } +} + +/// Every reference to [`EMITTER`], split into its definition and its callers. +/// +/// Comment lines are dropped first: a doc comment naming the emitter (this +/// file's own header does it, and the opcode's documentation may come to) +/// is prose about the symbol, not a use of it. +fn scan() -> (Vec, Vec) { + let root = repo_root(); + let mut files = Vec::new(); + for r in ROOTS { + collect_rust_sources(&root.join(r), &mut files); + } + assert!( + files.len() > 100, + "source walk found only {} files under {ROOTS:?} — the scan is \ + mis-rooted at {}, so a zero result would mean nothing", + files.len(), + root.display(), + ); + + let mut definitions = Vec::new(); + let mut callers = Vec::new(); + for file in files { + // This file names the emitter in prose and in `EMITTER`; excluding it + // by path keeps the const from counting as a call site. + if file.ends_with("arraybase_vable_has_no_emitter.rs") { + continue; + } + let Ok(text) = std::fs::read_to_string(&file) else { + continue; + }; + if !text.contains(EMITTER) { + continue; + } + for (i, line) in text.lines().enumerate() { + if !line.contains(EMITTER) || line.trim_start().starts_with("//") { + continue; + } + let site = format!( + "{}:{}: {}", + file.strip_prefix(&root).unwrap_or(&file).display(), + i + 1, + line.trim(), + ); + if line.contains(&format!("fn {EMITTER}")) { + definitions.push(site); + } else { + callers.push(site); + } + } + } + (definitions, callers) +} + +#[test] +fn arraybase_vable_has_no_emitter() { + let (definitions, callers) = scan(); + + // A missing definition fails as loudly as an extra caller: it means the + // opcode's producer was deleted or renamed, and this test would otherwise + // keep passing while guarding nothing. + assert_eq!( + definitions.len(), + 1, + "expected exactly one `{EMITTER}` definition, found {}:\n {}", + definitions.len(), + definitions.join("\n "), + ); + + assert!( + callers.is_empty(), + "`{EMITTER}` now has {} caller(s):\n {}\n\n\ + `BC_ARRAYBASE_VABLE` is deliberately unreachable — it is defined, \ + decoded and wired, but no lowering emits it, because the walk resumes \ + at the residual's argument byte instead of the next opcode and the \ + program returns a wrong answer rather than aborting.\n\n\ + If you are landing the lowering as part of fixing that: delete this \ + test in the same commit, and say in the message that the opcode now \ + has a producer. If you are not, this is the bug this test exists to \ + catch.", + callers.len(), + callers.join("\n "), + ); +} diff --git a/majit/majit-translate/src/codewriter/call.rs b/majit/majit-translate/src/codewriter/call.rs index f519844a592..496c55085e3 100644 --- a/majit/majit-translate/src/codewriter/call.rs +++ b/majit/majit-translate/src/codewriter/call.rs @@ -20,6 +20,23 @@ use crate::model::{CallTarget, FunctionGraph, LinkArg, OpKind, SpaceOperation}; use crate::parse::CallPath; use crate::policy::JitPolicy; +// Decline-census gate names. Declared in `crate::decline::gate` so a +// gate name cannot exist without the recorder that consumes it; aliased +// here for readability at the call sites. +// +// `FIND_ALL_GRAPHS` is the discovery walk that decides which callees +// become candidates, and so which can become a `JitCode` at all — a +// callee it skips never reaches the codewriter, and every later gate is +// silent about it. `GUESS_CALL_KIND` is the per-call-site half: where +// the first answers "was this callee ever a candidate", this one answers +// "was this particular call site allowed to enter it". `WRAPPER_FAMILY` +// seeds the BFS, so a wrapper missing from it is a whole gateway body +// discovery never starts from. +use crate::decline::gate::{ + FIND_ALL_GRAPHS as BFS_GATE, GUESS_CALL_KIND as CALLKIND_GATE, + WRAPPER_FAMILY as WRAPPER_FAMILY_GATE, +}; + // ── Graph-based analyzers (RPython effectinfo.py + canraise.py) ──── // // RPython uses BoolGraphAnalyzer subclasses that traverse call graphs @@ -3835,7 +3852,14 @@ impl CallControl { while let Some(path) = todo.pop() { let graph = match self.function_graphs.get(&path) { Some(g) => g.clone(), - None => continue, + None => { + crate::decline::record( + BFS_GATE, + "seeded-path-has-no-graph", + format_args!("{path}"), + ); + continue; + } }; // RPython call.py:77-90: scan all Call ops in the graph. // For each call, check guess_call_kind (with BFS-aware @@ -3855,7 +3879,14 @@ impl CallControl { OpKind::IndirectCall { graphs, .. } => match graphs { Some(graphs) if graphs.is_empty() => builtin_wrappers.clone(), Some(graphs) => graphs.clone(), - None => continue, + None => { + crate::decline::record( + BFS_GATE, + "indirect-family-unknown", + format_args!("in {path}"), + ); + continue; + } }, // Same indirect_call site, spelled the way it // exists *before* `rpbc::lower_indirect_calls` @@ -3897,7 +3928,27 @@ impl CallControl { OpKind::Call { target, .. } => { let callee_path = match self.target_to_path(target) { Some(path) => path, - None => continue, + None => { + // The single widest silent refusal in the + // pipeline: a call whose target resolves to + // no registered path at all. Upstream has + // no analogue — `funcobj.graph` is an + // object reference that either exists or is + // `None` (call.py:127), never a name lookup + // that can miss — so a miss here means the + // callee was never lowered into + // `function_graphs`, not that a gate judged + // it. Every gate downstream of this point + // is therefore never consulted for this + // callee, which is exactly the reading that + // a bare `continue` cannot support. + crate::decline::record( + BFS_GATE, + "callee-target-unresolvable", + format_args!("{target:?} in {path}"), + ); + continue; + } }; // `call.py:119-120` // jitdriver_sd_from_portal_runner_ptr → recursive. @@ -3906,6 +3957,15 @@ impl CallControl { .iter() .any(|jd| jd.portal_graph == callee_path) { + // Not a refusal — the portal is already a + // candidate and re-walking it would loop — + // but recorded so the BFS's skip rows add up + // to every call site it saw. + crate::decline::record( + BFS_GATE, + "callee-is-portal-recursive", + format_args!("{callee_path} in {path}"), + ); continue; } // `call.py:129-134` @@ -3917,6 +3977,11 @@ impl CallControl { .func_effects(&callee_path) .is_some_and(|f| f.close_stack) { + crate::decline::record( + BFS_GATE, + "callee-close-stack-residual", + format_args!("{callee_path} in {path}"), + ); continue; } // `call.py:135-136` @@ -3925,6 +3990,11 @@ impl CallControl { .func_effects(&callee_path) .is_some_and(|f| f.oopspec.is_some()) { + crate::decline::record( + BFS_GATE, + "callee-oopspec-builtin", + format_args!("{callee_path} in {path}"), + ); continue; } // `#[pyre_class]`'s `allocate`/`allocate_stable` @@ -3943,10 +4013,19 @@ impl CallControl { callee_path.last_segment(), Some("allocate") | Some("allocate_stable") ) { + crate::decline::record( + BFS_GATE, + "callee-pyre-class-ctor", + format_args!("{callee_path} in {path}"), + ); continue; } vec![callee_path] } + // Not a call operation. This arm is the population + // filter, not a decline: recording it would count + // every arithmetic op in every graph and drown the + // rows that are about call sites. _ => continue, }; for callee_path in callees { @@ -3956,9 +4035,25 @@ impl CallControl { } // A target with no registered graph is upstream's // `funcobj.graph is None` → residual (call.py:127). + // + // In upstream that condition is a property of the + // callable (an `external`/`llhelper` funcptr genuinely + // has no graph). Here it also covers a callee whose + // body the front end never lowered — the two are + // indistinguishable from inside this loop, which is + // precisely why the count has to exist: it turns "no + // jitcode appeared" into "this named path had no + // registered graph at BFS time". let graph_ref = match self.function_graphs.get(&callee_path) { Some(g) => g, - None => continue, + None => { + crate::decline::record( + BFS_GATE, + "callee-has-no-registered-graph", + format_args!("{callee_path} in {path}"), + ); + continue; + } }; // RPython call.py:84,87: callee must satisfy // policy.look_inside_graph(graph). Synthesize a @@ -3982,6 +4077,19 @@ impl CallControl { if policy.look_inside_graph(&func) { self.candidate_graphs.insert(callee_path.clone()); todo.push(callee_path); + } else { + // `policy.py look_inside_graph` said no — + // a `dont_look_inside` / `elidable` hint, or a + // loop without `unroll_safe`. Upstream and pyre + // agree on this one, so it is the decline row a + // reader wants to see NON-empty: a zero here with + // a non-zero `callee-has-no-registered-graph` + // means the policy never got a say. + crate::decline::record( + BFS_GATE, + "callee-policy-declined", + format_args!("{callee_path} in {path}"), + ); } } } @@ -4403,6 +4511,11 @@ impl CallControl { } // call.py:129-134 _gctransformer_hint_close_stack_ → 'residual' if self.func_effects(p).is_some_and(|f| f.close_stack) { + crate::decline::record( + CALLKIND_GATE, + "residual-close-stack", + format_args!("{p}"), + ); return CallKind::Residual; } // call.py `hasattr(targetgraph.func, 'oopspec')` → 'builtin' @@ -4414,6 +4527,35 @@ impl CallControl { // RPython `call.py:137-139` — both direct_call (fall-through) // and indirect_call reach this final classification. if self.graphs_from(op).is_none() { + // THE residual/JitCode fork. `graphs_from` answers `None` for + // three structurally different reasons and the caller cannot + // tell them apart from the `CallKind::Residual` it gets back, + // so re-derive which one it was — but only when the census is + // on, so the classification never runs on the hot path it + // measures. + if crate::decline::enabled() { + let reason = match &op.kind { + OpKind::Call { target, .. } => match self.target_to_path(target) { + // The path resolved but `find_all_graphs` never put + // it in the candidate set. Cross-reference the + // `find_all_graphs_bfs` rows to see which of its + // gates dropped it — or, if none did, that the BFS + // never reached this call site at all. + Some(_) => "residual-callee-not-a-candidate", + // No registered path for the target: nothing was + // ever lowered under this name. + None => "residual-target-unresolvable", + }, + OpKind::IndirectCall { graphs: None, .. } => "residual-indirect-family-unknown", + OpKind::IndirectCall { .. } => "residual-indirect-family-no-candidate", + // `graphs_from` answers `None` for every non-call op. + // Callers only classify call sites, so reaching here + // means a caller asked about something else; name it + // rather than folding it into a call-shaped reason. + _ => "residual-not-a-call-op", + }; + crate::decline::record(CALLKIND_GATE, reason, format_args!("{:?}", op.kind)); + } CallKind::Residual } else { CallKind::Regular @@ -5202,9 +5344,28 @@ impl CallControl { std::collections::BTreeMap::new(); for (path, &fnaddr) in &self.function_fnaddrs { let Some(leaf) = path.last_segment() else { + crate::decline::record( + WRAPPER_FAMILY_GATE, + "fnaddr-path-has-no-leaf", + format_args!("{fnaddr:#x}"), + ); continue; }; - if !leaf.starts_with("__pyre_wrap_") || !self.function_graphs.contains_key(path) { + // The `__pyre_wrap_` test is the population filter — every + // non-wrapper fnaddr in the binary fails it — so it is not + // recorded. The missing-graph test that follows IS a decline: + // a generated wrapper published an address but no graph, so it + // cannot join the PBC family and every indirect site that would + // have dispatched to it stays residual. + if !leaf.starts_with("__pyre_wrap_") { + continue; + } + if !self.function_graphs.contains_key(path) { + crate::decline::record( + WRAPPER_FAMILY_GATE, + "wrapper-has-no-registered-graph", + format_args!("{path}"), + ); continue; } by_address.entry(fnaddr).or_default().push(path.clone()); diff --git a/majit/majit-translate/src/codewriter/codewriter.rs b/majit/majit-translate/src/codewriter/codewriter.rs index 208f3a8df4e..2ca09f204ac 100644 --- a/majit/majit-translate/src/codewriter/codewriter.rs +++ b/majit/majit-translate/src/codewriter/codewriter.rs @@ -340,6 +340,15 @@ impl CodeWriter { Ok(crate::translator::rtyper::cutover::DualGateOutcome::Match { real_value_to_var, }) => { + // The accept arm, recorded so this gate's rows sum to a + // denominator. "N graphs Skipped" is not a finding on its + // own — "N of M" is, and M has to come from the same run + // rather than from a figure written down elsewhere. + crate::decline::observe_accept( + crate::decline::gate::DUAL_GATE, + "match-real-rtyper (ACCEPT, not a decline)", + &graph.name, + ); // Commit each real-rtyper Variable's `concretetype` // (LowLevelType) onto its placeholder on the graph's // value table. Mirrors RPython `rtyper.py:258 v.concretetype = ...` @@ -369,6 +378,57 @@ impl CodeWriter { Some(real_value_to_var) } Ok(crate::translator::rtyper::cutover::DualGateOutcome::Skip(reason)) => { + // A Skip is a decline: the real rtyper refused this graph + // and the legacy walker types it instead. This is the fork + // that decides whether a host gets the real rtyper or the + // flattening fallback, and it is keyed on substring-matching + // a diagnostic string (`cutover::unported_category`) — so + // WHICH arm matched is the finding, not that one did. A + // single "Skip" count cannot say whether a host is dominated + // by registry misses or by unimplemented operations, and + // those imply completely different work. + // + // The classification already exists at this point and was + // being discarded; the outcome was visible only behind + // `PYRE_RTYPER_VERBOSE=1`. Record the arm as the count key + // and the graph as the subject, so the row carries both an + // event count and a distinct-graph count. + // + // WHICH ROUTE the Skip took is decidable here, and the two + // must not share a bucket. `unported_category` classifies + // a CAUGHT PANIC PAYLOAD; the arm above is the only + // producer of that shape and it stamps a + // `registry build panicked: ` prefix. Every other Skip + // arrives as a direct `Ok(DualGateOutcome::Skip(..))` and + // never reaches that predicate at all — the two + // populations are disjoint by construction, not + // overlapping. + // + // So an unmatched direct Skip is NOT "unknown shape, the + // arm list is incomplete". It is "this route never + // consults the arm list". Those prescribe opposite next + // actions — extend the arms, versus instrument a route + // nobody had drawn — so the row is labelled for its + // mechanism. Same defect as a zero that two causes can + // produce. + const PANIC_PREFIX: &str = "registry build panicked: "; + let class = if reason.starts_with(PANIC_PREFIX) { + crate::translator::rtyper::cutover::unported_category(&reason) + // Unreachable while the arm above is the sole + // producer (it only builds this shape when + // `is_known_unported` said yes), so a row here + // means that invariant broke. + .unwrap_or("panic-route-unmatched-by-any-arm") + } else { + crate::translator::rtyper::cutover::non_arm_skip_category(&reason) + .unwrap_or("direct-skip (never reaches unported_category)") + }; + crate::decline::record_reason( + crate::decline::gate::DUAL_GATE, + class, + &reason, + &graph.name, + ); if std::env::var_os("PYRE_RTYPER_VERBOSE").is_some_and(|v| v == "1") { eprintln!( "[PYRE_RTYPER skip] graph {diag_label:?} ({:?}): {reason}", diff --git a/majit/majit-translate/src/codewriter/insns.rs b/majit/majit-translate/src/codewriter/insns.rs index 26036f32fd5..9125d6e02a3 100644 --- a/majit/majit-translate/src/codewriter/insns.rs +++ b/majit/majit-translate/src/codewriter/insns.rs @@ -401,6 +401,29 @@ pub const BC_INT_ADD_JUMP_IF_OVF: u8 = 231; pub const BC_INT_SUB_JUMP_IF_OVF: u8 = 232; pub const BC_INT_MUL_JUMP_IF_OVF: u8 = 233; +// pyre-only `arraybase_vable/rdd>i` — the address of item 0 of a +// virtualizable array field, as an integer. Argcode and operand layout are +// the same `rdd>i` triple as its sibling `arraylen_vable` (BC_ARRAYLEN_VABLE = +// 74), so `read_vable_arraylen` decodes both. +// +// Nothing emits this opcode. It is defined, decoded, walked, and wired into the +// blackhole, but it is not a working feature. +// The only intended emitter is a `#[jit_interp]` macro lowering for +// `state..as_mut_ptr()`, which is deliberately held back: with it +// applied, tl's `jit_residual_not_double_executed` runs the ROLL residual 7 +// times where the program contains 40, and returns 17 where the interpreter +// returns 0. The remaining resume-position defect must be fixed before adding +// the emitter. +// +// RPython has no counterpart, and the reason is the interesting part: upstream +// never needs the base as a *value* because every `*_vable` op resolves it +// internally from `(fdescr, adescr)` off the live virtualizable. The op exists +// only to hand that address to a residual call, which is an escape — the callee +// can permute items that are the trace's own SSA values. Executing it +// therefore also marks the current `TraceCtx` as escaped, and the enclosing +// CALL_MAY_FORCE aborts with ABORT_ESCAPE before the trace reaches a backend. +pub const BC_ARRAYBASE_VABLE: u8 = 234; + // `switch/id` — RPython `blackhole.py` `bhimpl_switch` — // table-of-cases dispatch keyed by an int register + a descr selecting // the case table. @@ -1275,6 +1298,13 @@ pub fn pyre_extension_insns() -> IndexMap<&'static str, u8> { // pyre's Rust port hits this only on `dyn Trait` calls, where the // backend epic must look up the vtable slot itself. m.insert("vtable_method_ptr/rd>i", BC_VTABLE_METHOD_PTR); + // pyre-only `arraybase_vable/rdd>i` — address of item 0 of a + // virtualizable array field. Registered here rather than in + // `wellknown_bh_insns` beside `arraylen_vable` because RPython has no + // such op: upstream resolves the base inside each `*_vable` op and never + // materialises it as a value. See [`BC_ARRAYBASE_VABLE`] for why + // producing it is by definition an escape. + m.insert("arraybase_vable/rdd>i", BC_ARRAYBASE_VABLE); m } diff --git a/majit/majit-translate/src/codewriter/policy.rs b/majit/majit-translate/src/codewriter/policy.rs index f56ccd0252b..79e77a08151 100644 --- a/majit/majit-translate/src/codewriter/policy.rs +++ b/majit/majit-translate/src/codewriter/policy.rs @@ -192,6 +192,34 @@ pub trait JitPolicy { func.name ); } + // A `false` here is the policy's refusal to let this callee become + // a JitCode, and it reaches the caller as a bare `bool` — four + // structurally different clauses collapsed into one answer. + // `unsafe_loopy_graphs` already records ONE of them (and only when + // `res` was still true at that point), so it cannot stand in for + // the rest. Re-derive which clause refused; guarded on the census + // so the extra predicate calls never run on the decision path they + // measure. + if !res && crate::decline::enabled() { + let reason = if !see_function { + if jit_look_inside_hint(&func.hints) == Some(false) { + "dont_look_inside-hint" + } else if self._reject_function(func) { + "elidable-hint" + } else { + "look_inside_function-said-no" + } + } else if contains_loop { + "loop-without-unroll_safe" + } else { + "unsupported-variable-type" + }; + crate::decline::record( + crate::decline::gate::LOOK_INSIDE_GRAPH, + reason, + format_args!("{}", func.name), + ); + } res } } diff --git a/majit/majit-translate/src/decline.rs b/majit/majit-translate/src/decline.rs new file mode 100644 index 00000000000..3e5ac43cda4 --- /dev/null +++ b/majit/majit-translate/src/decline.rs @@ -0,0 +1,384 @@ +//! Decline census: make a silently-refused lowering countable. +//! +//! Every lowering gate in this pipeline refuses with a bare `continue`, +//! `return 0`, or `false`. None of them raises, none of them warns, and +//! none of them counts — so when a probe does not move after a change, +//! nothing in the tree can say whether the gate the change targeted was +//! consulted and said no, or was never reached at all. Reading a jitcode +//! list by hand is currently the only way to tell those apart. +//! +//! Upstream is loud exactly where this pipeline is silent: +//! `rpython/jit/codewriter/jtransform.py`'s `_handle_list_call` raises +//! `NotImplementedError("prebuilt lists cannot be virtual")` rather than +//! falling through to a residual, so an unhandled shape stops translation +//! with its own name attached. This module does NOT turn any decline into +//! an error — every gate keeps its exact current control flow — it only +//! records that the decline happened and why, so the same information +//! upstream would have raised is at least countable here. +//! +//! # Contract +//! +//! - **Nothing reads these counters to decide anything.** The only +//! consumers are [`snapshot`] and [`dump_to_stderr`], both of which +//! write to stderr / return data to a test. No gate branches on a +//! count, so an instrumented build takes exactly the lowering decisions +//! an uninstrumented one takes. +//! - **Off by default.** With neither switch set, every recorder returns +//! before touching the map or formatting anything; [`record`] takes its +//! `subject` as `fmt::Arguments`, so the caller's `format_args!` +//! allocates nothing on the disabled path. +//! - **A count names its gate AND its reason.** A bare "something +//! declined" count would reproduce the problem this module exists to +//! fix, one level up. Where a gate classifies its own refusal — the +//! dual gate's `cutover::unported_category` — the count key is the ARM +//! that matched, since which arm dominates is what decides the next +//! stretch of work. +//! - **Events and subjects are counted separately.** [`record_named`] +//! and [`record_reason`] also record the distinct subject, so a row +//! reports "N events over M distinct graphs". Reading an event count as +//! a graph count is wrong wherever a gate can revisit one graph. +//! +//! # Switches +//! +//! `MAJIT_DECLINE_LOG` is the switch; `PYRE_MIR_FRONTEND_DEBUG` (the +//! existing front-end debug switch, `front/checked_arith.rs` et al.) is +//! accepted as an alias at level 1 so a reader who already knows this +//! codebase's debug channel does not have to learn a second one. +//! +//! | value | effect | +//! |---|---| +//! | unset | disabled — no counting, no output, no formatting | +//! | `1` (or `PYRE_MIR_FRONTEND_DEBUG` set) | count per (gate, reason); [`record_reason`] also prints its runtime reason | +//! | `2` | the above, plus one stderr line per individual decline | +//! +//! Level 1 does not print per-event lines because the instrumented gates +//! include per-op ones (`guess_call_kind` runs once per call operation in +//! every graph); a whole-program run would emit tens of thousands of +//! lines and bury the summary. [`record_reason`] is the exception: its +//! callers are per-graph, and its reason is a runtime string that the +//! bounded count key cannot carry. +//! +//! # Reading a decline count +//! +//! A gate that reports zero declines is *not* the same as a gate that +//! accepted everything: it can equally mean the gate was never reached. +//! Where that distinction matters the gate also records the reached-but- +//! declined case under a distinct reason, so a missing row and a zero row +//! stay different observations. +//! +//! Population filters are deliberately NOT recorded. `fuse_boxing_alloc` +//! inspects every operation in a graph and skips the ones that are not +//! `malloc_typed` calls at all; counting those would make the instrument +//! part of the population it measures — the count would be dominated by +//! operations that were never candidates. Recording starts once a site +//! has been identified as the kind of thing the gate exists to lower. + +use std::borrow::Cow; +use std::collections::{BTreeMap, BTreeSet}; +use std::fmt::Arguments; +use std::sync::{LazyLock, Mutex, OnceLock}; + +/// Every instrumented gate's census name, in one place. +/// +/// These live here, beside the recorder, rather than next to each gate. +/// A name defined at its call site can be referenced while this module +/// is absent — which is exactly how a half-applied edit produced a tree +/// that named `model::fuse_boxing_alloc`'s gates and could not build. +/// Declared here, a gate name cannot exist without the module that +/// consumes it, so the two cannot go out of sync. +pub mod gate { + /// `model::fuse_boxing_alloc`'s site loop — a `malloc_typed` call + /// the pass identified but did not rewrite. + pub const FUSE_BOXING_ALLOC: &str = "model::fuse_boxing_alloc"; + /// The `resolve_header_plan` closure inside it, whose `None` the site + /// loop can only report as `vtable-unresolved`. + pub const RESOLVE_HEADER_PLAN: &str = "model::resolve_header_plan"; + /// The `Result` callee rule. + pub const RESULT_EXC_CALLEE: &str = "result_exc::lower_result_exc_returns"; + /// The `?`-site caller rule. + pub const RESULT_EXC_CALLER: &str = "result_exc::rewire_result_exc_call_sites"; + /// The builtin gateway PBC family that seeds graph discovery. + pub const WRAPPER_FAMILY: &str = "call::compute_builtin_wrapper_indirect_graphs"; + /// Registry population — a callable skipped here resolves as a host + /// builtin or residual stub rather than as a user graph. + pub const CALL_REGISTRY: &str = "cutover::populate_call_registry_from_call_graphs"; + /// The real-rtyper-versus-legacy-walker fork. + pub const DUAL_GATE: &str = "codewriter::dual_gate_publish_concretetypes"; + /// Graph discovery: which callees can become a `JitCode` at all. + pub const FIND_ALL_GRAPHS: &str = "call::find_all_graphs_bfs"; + /// Per-call-site emission: `residual_call_*` versus an inlined entry. + pub const GUESS_CALL_KIND: &str = "call::guess_call_kind"; + /// The policy clause behind a discovery refusal. + pub const LOOK_INSIDE_GRAPH: &str = "policy::look_inside_graph"; +} + +/// Verbosity of the decline census. Resolved once, from the environment. +#[derive(Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Debug)] +pub enum Level { + /// No counting and no output. The default. + Off, + /// Count per (gate, reason); print only runtime reason strings. + Counters, + /// Also print one line per individual decline. + Events, +} + +fn level() -> Level { + static LEVEL: OnceLock = OnceLock::new(); + *LEVEL.get_or_init(|| match std::env::var_os("MAJIT_DECLINE_LOG") { + Some(value) if value == "2" => Level::Events, + Some(value) if value == "0" || value.is_empty() => Level::Off, + Some(_) => Level::Counters, + // Alias: the front end's established debug switch turns the + // census on at counter level, so a reader who already reaches + // for `PYRE_MIR_FRONTEND_DEBUG` gets the decline rows too + // rather than finding a second, parallel channel for the same + // job. + None if std::env::var_os("PYRE_MIR_FRONTEND_DEBUG").is_some() => Level::Counters, + None => Level::Off, + }) +} + +/// Whether the census is on. +/// +/// Call sites that would have to compute a reason (walk a call target, +/// re-derive which of three conditions refused) guard that work on this +/// so the disabled path stays free of it. +#[inline] +pub fn enabled() -> bool { + level() != Level::Off +} + +/// `(gate, reason) -> count`, ordered so a dump is stable across runs. +/// +/// `Cow` because most reasons are `&'static str` tags fixed by the call +/// site, while a few gates classify a runtime message into a bounded set +/// of owned strings. Unbounded keys (a graph name, a full panic payload) +/// belong in `subject`, never here. +static COUNTS: LazyLock), Row>>> = + LazyLock::new(|| Mutex::new(BTreeMap::new())); + +/// One `(gate, reason)` row. +/// +/// `events` and `subjects` can differ and the difference is the point: +/// a gate called once per graph can decline the same graph repeatedly +/// (`dual_gate_publish_concretetypes` runs per graph but a shared callee +/// is re-visited), so an event count alone cannot be read as "how many +/// graphs". A per-graph figure needs a denominator anyone can +/// reproduce, which means counting distinct subjects, not calls. +#[derive(Default)] +struct Row { + events: u64, + /// Distinct subjects, when the call site named one. Empty for the + /// gates whose subject is an operation rather than a nameable graph. + subjects: BTreeSet, +} + +fn bump(gate: &'static str, reason: Cow<'static, str>, subject: Option<&str>) { + // A poisoned map means some other thread panicked mid-record. The + // census must never turn that into a second failure, so recover the + // guard: a lost count is strictly better than an instrument that + // aborts the run it is measuring. + let mut counts = COUNTS.lock().unwrap_or_else(|e| e.into_inner()); + let row = counts.entry((gate, reason)).or_default(); + row.events += 1; + if let Some(subject) = subject { + row.subjects.insert(subject.to_string()); + } +} + +/// Record one decline whose reason is fixed by the call site. +/// +/// `gate` names the function that refused, `reason` names which of its +/// refusal paths ran, and `subject` names what was refused (a graph name, +/// a call path). Only `(gate, reason)` is counted; `subject` is printed +/// at [`Level::Events`] and otherwise never formatted. +#[inline] +pub fn record(gate: &'static str, reason: &'static str, subject: Arguments<'_>) { + let level = level(); + if level == Level::Off { + return; + } + bump(gate, Cow::Borrowed(reason), None); + if level == Level::Events { + eprintln!("[decline] {gate} {reason}: {subject}"); + } +} + +/// Record one decline against a NAMED subject, so the row carries a +/// distinct-subject count alongside its event count. +/// +/// Use this wherever the subject is a graph (or anything else with a +/// stable identity a reader can count): it is what makes a figure like +/// "88 of 95 graphs" reproducible from the instrument rather than from +/// one person's notes. `subject` is formatted on every call once the +/// census is on, so keep it to an identity, not a dump. +#[inline] +pub fn record_named(gate: &'static str, reason: &'static str, subject: &str) { + let level = level(); + if level == Level::Off { + return; + } + bump(gate, Cow::Borrowed(reason), Some(subject)); + if level == Level::Events { + eprintln!("[decline] {gate} {reason}: {subject}"); + } +} + +/// Record one decline that already carries a formatted reason string, +/// against a named subject. +/// +/// `class` is the bounded count key; `reason` is the gate's own message, +/// which is printed at [`Level::Counters`] — unconditionally, i.e. without +/// consulting whatever narrower switch the gate's own logging used to sit +/// behind. Use this only for per-graph gates: it formats on every call +/// once the census is on. +#[inline] +pub fn record_reason(gate: &'static str, class: &'static str, reason: &str, subject: &str) { + let level = level(); + if level == Level::Off { + return; + } + bump(gate, Cow::Borrowed(class), Some(subject)); + eprintln!("[decline] {gate} {class} {subject}: {reason}"); +} + +/// Record a gate's ACCEPT arm, so its rows sum to a denominator. +/// +/// Only for a gate where the accept/decline ratio is the finding — the +/// dual gate, where "N graphs Skipped" is meaningless without "out of +/// how many". Named `observe` rather than `record` because it is not a +/// decline, and the dump labels it so a reader cannot add it into one. +#[inline] +pub fn observe_accept(gate: &'static str, class: &'static str, subject: &str) { + if level() == Level::Off { + return; + } + bump(gate, Cow::Borrowed(class), Some(subject)); +} + +/// Every `(gate, reason, events, distinct_subjects)` recorded so far, +/// gate-then-reason ordered. +/// +/// `distinct_subjects` is 0 where the call site named no subject; it is +/// NOT a claim that one subject was involved. +pub fn snapshot() -> Vec<(&'static str, String, u64, usize)> { + let counts = COUNTS.lock().unwrap_or_else(|e| e.into_inner()); + counts + .iter() + .map(|((gate, reason), row)| (*gate, reason.to_string(), row.events, row.subjects.len())) + .collect() +} + +/// The distinct subjects recorded under one `(gate, reason)` row. +/// +/// The list, not the count — for the reader who needs to know WHICH +/// graphs, not how many. Empty for rows whose call site named none. +pub fn subjects_of(gate: &str, reason: &str) -> Vec { + let counts = COUNTS.lock().unwrap_or_else(|e| e.into_inner()); + counts + .iter() + .filter(|((g, r), _)| *g == gate && r == reason) + .flat_map(|(_, row)| row.subjects.iter().cloned()) + .collect() +} + +/// Print the census to stderr. A no-op when the census is off, so a +/// caller can wire this in unconditionally. +/// +/// `label` identifies the run, since a test binary may census more than +/// one pipeline. Counters are cumulative across the whole process — the +/// map is never cleared, because clearing it would let one run's dump +/// silently omit declines that a previous run in the same binary had +/// already recorded. +pub fn dump_to_stderr(label: &str) { + if !enabled() { + return; + } + let rows = snapshot(); + let total: u64 = rows.iter().map(|(_, _, n, _)| n).sum(); + eprintln!( + "=== majit decline census [{label}]: {total} events, {rows_len} (gate, reason) rows ===", + rows_len = rows.len() + ); + eprintln!( + " events subjects reason (subjects = DISTINCT named subjects; '-' = call site named none)" + ); + if rows.is_empty() { + // Distinguish "no gate declined" from "the census was on but no + // instrumented gate ran": both print this line, and neither is + // evidence that a lowering succeeded. + eprintln!(" (no instrumented gate recorded a decline in this process)"); + return; + } + let mut current = ""; + for (gate, reason, events, subjects) in rows { + if gate != current { + eprintln!(" {gate}"); + current = gate; + } + // A row whose subjects were never named prints `-` rather than + // `0`: zero distinct subjects and "this gate does not name its + // subject" are different facts and must not share a spelling. + let subjects = if subjects == 0 { + "-".to_string() + } else { + subjects.to_string() + }; + eprintln!(" {events:6} {subjects:>8} {reason}"); + } +} + +/// Dump the census when this value is dropped, including during an +/// unwind. +/// +/// A translation run that panics is exactly the run whose refusals matter +/// most — the pipeline is designed to fail loud on a shape it cannot +/// digest — so the dump must not be an ordinary statement at the end of +/// the happy path, where a panic would skip it. +pub struct CensusScope { + label: &'static str, +} + +impl CensusScope { + /// `label` is `&'static str` so an off census pays nothing to name a + /// run it will not print. + pub fn new(label: &'static str) -> Self { + Self { label } + } +} + +impl Drop for CensusScope { + fn drop(&mut self) { + dump_to_stderr(self.label); + } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// The disabled path must not touch the map. This is the property + /// that keeps the instrument out of the decisions it measures: with + /// the switch off there is no state for anything to read. + /// + /// The test binary inherits whatever the environment set, and + /// `level()` is resolved once per process, so assert the invariant + /// that holds either way: recording bumps this gate's own row exactly + /// when the census is on. The gate name is private to this test so + /// the count is unaffected by whatever the rest of the test binary is + /// declining in parallel. + #[test] + fn record_touches_the_map_exactly_when_enabled() { + const GATE: &str = "decline::tests::record_touches_the_map_exactly_when_enabled"; + let row = |rows: Vec<(&'static str, String, u64, usize)>| -> u64 { + rows.iter() + .filter(|(gate, reason, _, _)| *gate == GATE && reason == "probe") + .map(|(_, _, n, _)| *n) + .sum() + }; + let before = row(snapshot()); + record(GATE, "probe", format_args!("subject")); + assert_eq!(row(snapshot()), before + u64::from(enabled())); + } +} diff --git a/majit/majit-translate/src/front/mod.rs b/majit/majit-translate/src/front/mod.rs index 42656cbbc2e..6d424b0f6ff 100644 --- a/majit/majit-translate/src/front/mod.rs +++ b/majit/majit-translate/src/front/mod.rs @@ -88,6 +88,7 @@ pub(crate) mod result_exc; pub(crate) mod saturating_sub; pub mod semantic; pub(crate) mod slice_first; +pub(crate) mod slice_get; pub(crate) mod slice_index; pub(crate) mod str_find; pub mod typestr; diff --git a/majit/majit-translate/src/front/result_exc.rs b/majit/majit-translate/src/front/result_exc.rs index 13d84e651b0..34961bf2c9e 100644 --- a/majit/majit-translate/src/front/result_exc.rs +++ b/majit/majit-translate/src/front/result_exc.rs @@ -361,6 +361,43 @@ pub(crate) fn result_ctor_kind(target: &CallTarget) -> Option { pub(crate) fn lower_result_exc_returns( graph: &mut FunctionGraph, tail_forwarded_returns: usize, +) -> Result { + // Every `Err` below declines the WHOLE callee to a residual call. The + // message says why, but it travels out as `LowerError::Unsupported` and + // the front end's coverage gate reports only a category tally, so the + // per-graph reason is not recoverable from any output. Record it here, + // where the reason still exists. + // + // Upstream is loud in the equivalent position: + // `rpython/jit/codewriter/jtransform.py`'s `_handle_list_call` raises + // `NotImplementedError("prebuilt lists cannot be virtual")` rather than + // falling through to a residual. This does not change the decline into + // an error — the fail-safe residual is deliberate here — it only makes + // the refusal countable. + let outcome = lower_result_exc_returns_inner(graph, tail_forwarded_returns); + if let Err(msg) = &outcome { + crate::decline::record_reason( + RESULT_EXC_CALLEE_GATE, + "callee-declined-to-residual", + msg, + &graph.name, + ); + } + outcome +} + +// Decline-census gate names: the callee rule (a scoped +// `Result` graph the exception-link lowering refused whole) +// and the caller rule (one `?`-site the diamond rewrite refused). +// Declared in `crate::decline::gate` so a name cannot outlive the +// recorder that consumes it. +use crate::decline::gate::{ + RESULT_EXC_CALLEE as RESULT_EXC_CALLEE_GATE, RESULT_EXC_CALLER as RESULT_EXC_CALLER_GATE, +}; + +fn lower_result_exc_returns_inner( + graph: &mut FunctionGraph, + tail_forwarded_returns: usize, ) -> Result { let nblocks = graph.blocks.len(); let mut rewritten = 0usize; @@ -1100,13 +1137,30 @@ pub(crate) fn rewire_result_exc_call_sites( fused: 0, }; for (r, suffix, payload_ty) in results { - match rewire_one_call_site( + let site = rewire_one_call_site( graph, r, suffix.as_deref().unwrap_or(""), payload_ty, enclosing_scoped, - )? { + ); + let site = match site { + Ok(site) => site, + Err(msg) => { + // Same disposition as the callee rule above: the message + // is the only statement of why this `?`-site could not be + // lowered, and it is about to be flattened into the front + // end's category tally. Count it with its reason intact. + crate::decline::record_reason( + RESULT_EXC_CALLER_GATE, + "call-site-declined-to-residual", + &msg, + &graph.name, + ); + return Err(msg); + } + }; + match site { SiteOutcome::Diamond => outcome.diamonds += 1, SiteOutcome::TailForward => outcome.tail_forwards += 1, SiteOutcome::Rewrapped => outcome.rewrapped += 1, @@ -1164,8 +1218,22 @@ fn rewire_one_call_site( // `catch_and_rewrap`. The fusion is fail-safe: an `Err` from // `try_fuse_drain_match` MUST NOT propagate (that would decline the // whole graph); it converts here into the existing rewrap path. - if try_fuse_drain_match(graph, a, r).is_ok() { - return Ok(SiteOutcome::Fused); + match try_fuse_drain_match(graph, a, r) { + Ok(()) => return Ok(SiteOutcome::Fused), + Err(msg) => { + // The one refusal in this file that discards a fully + // formed reason string: `is_ok()` threw the message away, + // so a site that ALMOST matched the drain shape and a site + // that never resembled it produced identical evidence — + // none. The fail-safe fallthrough to `catch_and_rewrap` is + // deliberate and unchanged; only the reason is now kept. + crate::decline::record_reason( + RESULT_EXC_CALLER_GATE, + "drain-match-fusion-declined", + &msg, + &name, + ); + } } catch_and_rewrap(graph, a, r, suffix, payload_ty)?; return Ok(SiteOutcome::Rewrapped); diff --git a/majit/majit-translate/src/front/slice_get.rs b/majit/majit-translate/src/front/slice_get.rs new file mode 100644 index 00000000000..8cf2a281200 --- /dev/null +++ b/majit/majit-translate/src/front/slice_get.rs @@ -0,0 +1,542 @@ +//! `<[T]>::get(slice, i)` → bounds-checked `Option<&T>` diamond. +//! +//! ## Positioning +//! +//! `core::slice::::get` is a foreign leaf whose body is Opaque in the +//! LLBC (Charon cannot extract `core`), so the caller emits a residual `get` +//! call — an unregistered callee the rtyper census Skips, which Skips the +//! CALLING graph with it. Its `Self` is the primitive slice `[T]` (not an +//! ADT), so `lower_call` keeps the raw `FunctionPath` segments +//! `["core","slice","","get"]`, receiver in `args[0]` and index in +//! `args[1]`. `get` returns `Some(&slice[i])` iff `i` is in bounds, so this +//! pass *synthesizes* the guard `i < len(slice)`: +//! +//! ```text +//! opt = get(slice, i) // residual `get` call +//! becomes +//! if i < len(slice) { opt = Some(&slice[i]) } else { opt = None } +//! ``` +//! +//! The branch is mandatory — the element read must not run when `i` is out of +//! range (`slice[i]` would read past the block), so a single-block always-read +//! encoding is unsound. This is exactly why an *inline* fold (the +//! `emit_tagged_pair_aggregate` path, which writes `__pos_0` unconditionally in +//! the call's own block before the consumer's discriminant switch) cannot +//! express `get`; a post-pass that splits block A into a guarded Some/None +//! diamond can. It is the same "diamond is unavoidable" shape as +//! [`crate::front::slice_first`], of which `get` is the general case: `first` +//! is `get` at a fixed index 0, where `i < len` collapses to `len > 0`. +//! +//! Only the upper bound is tested. The index is a `usize` — the scalar +//! `SliceIndex` instantiation is the only one this pass fires for (see below) +//! — so it is non-negative by type and needs no lower-bound compare. +//! +//! ## The `SliceIndex` instantiation gate +//! +//! `<[T]>::get` is generic over `SliceIndex`, and only its scalar +//! instantiation has the shape this diamond encodes. `get(0..2)` returns +//! `Option<&[T]>` — a sub-slice, not an element — and an `ArrayRead` at the +//! range's start would hand the consumer a `T` where a `[T]` is expected. +//! `front::mir` `recognize_slice_get_site` therefore pins the instantiation by +//! the index operand's declared `usize` before a site is recorded at all, so +//! every range form falls through and keeps its residual call. +//! +//! ## Payload representation +//! +//! `get` returns `Option<&T>`, but the `Some` payload is materialised by +//! `OpKind::ArrayRead`, which yields the element VALUE, not a pointer-to-slot. +//! In the list model a `&T` and a `T` are the same one GC pointer word, and no +//! consumer derefs a slot-pointer (there is no `copied` pass that would; the +//! front has no pointer-to-slot op at all). So the `&T`-vs-`T` distinction +//! collapses harmlessly, and the value payload is both the only representable +//! and the correct choice. The receiver is a `SomeList`-modelled slice (its +//! `Input` op carries the list-container `class_root`), so `__len` and the +//! element read repr-dispatch to `arraylen_gc` / `getarrayitem` on the +//! underlying length-prefixed GcArray. +//! +//! ## The rewrite (`rewire_one_slice_get_site`) +//! +//! Block A holds the residual `get` call producing `opt`. Because +//! `Option<&PyObjectRef>` triggers `option_residual_narrow_root`, `lower_call` +//! appends a trailing `__pyre_cast_instance` after the call, so the call is NOT +//! A's last op — the block-A skeleton absorbs that optional cast, exactly as +//! [`crate::front::option_map_or`] does. The rewrite: +//! 1. drops the `get` call (+ absorbed cast) and closes A with a +//! `bool(i < len(slice))` branch to two fresh arms; +//! 2. the `then_bb` arm reads `slice[i]` and wraps it in `Some` +//! (`__discriminant = 1` / `__pos_0 = elem`); +//! 3. the `else_bb` arm builds `None` (`__discriminant = 0`); +//! 4. both arms re-apply the absorbed narrowing and forward to B, reproducing +//! A's original exit args with the `opt` slot sourced from the arm's +//! `Some`/`None` value and every other live value threaded through. +//! +//! It is **fail-safe**: any structural mismatch returns `Err`, the caller +//! leaves the residual call untouched, and the unregistered `get` callee keeps +//! the rtyper census Skip (no regression vs the legacy walker). + +use crate::flowspace::model::Variable; +use crate::front::bool_then::{ + close_goto_mixed, emit_option_variant, map_source, reproduce_exit_args, +}; +use crate::front::option_map_or::emit_narrow; +use crate::model::{CallTarget, FunctionGraph, LinkArg, OpKind, SpaceOperation, ValueType}; + +/// A recognized `<[T]>::get(slice, i)` call site captured during body lowering +/// (`front::mir` `recognize_slice_get_site`). The owner strings are resolved +/// at the recording site where the destination `Option<&T>` type is in hand; +/// the post-pass only needs them to spell the `Some`/`None` aggregates. +#[derive(Clone)] +pub(crate) struct SliceGetSite { + /// The `get` call result (the `Option<&T>` value) — locates block A. The + /// slice and index operands are read from that located call's + /// `args[0]`/`args[1]`. + pub result_var: Variable, + /// The `Option` enum root `name_path` (per-instantiation, suffixed) — the + /// ctor owner for the `Some`/`None` aggregates. + pub option_owner: String, + /// The `Option::Some` variant `name_path` — the `__pos_0` payload field + /// owner (matching the variant-qualified `resolve_adt_field` read owner). + pub some_owner: String, + /// The `Option`'s payload `&T` projected to a [`ValueType`] — the + /// `Some::__pos_0` field kind (`Ref(None)` for `Option<&PyObjectRef>`). + pub payload_ty: ValueType, +} + +/// Rewrite every recorded `<[T]>::get` call site into the bounds-checked +/// `Option` diamond. Fail-safe: a site whose block does not fit the +/// residual-call shape is left untouched (Skip), so a mismatch never regresses +/// a graph the legacy walker already handled. Returns the number of sites +/// rewritten. +pub(crate) fn rewire_slice_get_call_sites( + graph: &mut FunctionGraph, + sites: &[SliceGetSite], +) -> usize { + let mut rewritten = 0; + for site in sites { + match rewire_one_slice_get_site(graph, site) { + Ok(()) => rewritten += 1, + Err(_decline) => { + // Leave the residual `get` call; the unregistered callee keeps + // the rtyper census Skip for this graph. + } + } + } + rewritten +} + +fn rewire_one_slice_get_site(graph: &mut FunctionGraph, site: &SliceGetSite) -> Result<(), String> { + let name = graph.name.clone(); + // Block A: the `get` residual call producing `result_var`. + let a = graph + .blocks + .iter() + .position(|b| { + b.operations + .iter() + .any(|op| op.result.as_ref() == Some(&site.result_var)) + }) + .ok_or_else(|| format!("{name}: slice::get result var has no producer block"))?; + + // Locate the `get` call op by its result (not assuming it is the block + // tail — the trailing `__pyre_cast_instance` may follow, see below). + let ci = graph.blocks[a] + .operations + .iter() + .position(|op| op.result.as_ref() == Some(&site.result_var)) + .ok_or_else(|| format!("{name}: slice::get call op not found in block {a}"))?; + let ops_len = graph.blocks[a].operations.len(); + + // `Option<&PyObjectRef>` gains a trailing `__pyre_cast_instance` narrowing + // op (`result_narrow_root`) whose output is what the block forwards on. + // Absorb that optional cast: `flow_result` is the value B consumes, + // `narrow_root` re-applies the narrowing per arm, `remove_upto` bounds the + // ops to drop. Any other trailing shape declines (fail-safe). + let (flow_result, narrow_root, remove_upto) = if ci + 1 == ops_len { + (site.result_var.clone(), None, ci) + } else if ci + 2 == ops_len { + let cast = &graph.blocks[a].operations[ci + 1]; + match (&cast.kind, cast.result.as_ref()) { + ( + OpKind::Call { + target: CallTarget::FunctionPath { segments }, + args, + .. + }, + Some(narrowed), + ) if segments.len() == 2 + && segments[0] == crate::pyre_names::shims::CAST_INSTANCE + && args.len() == 1 + && args[0] == site.result_var => + { + (narrowed.clone(), Some(segments[1].clone()), ci + 1) + } + _ => { + return Err(format!( + "{name}: slice::get call is not the last op of block {a}" + )); + } + } + } else { + return Err(format!( + "{name}: slice::get call is not the last op of block {a}" + )); + }; + + // Capture the slice receiver and the index operand. + let (slice, index) = match &graph.blocks[a].operations[ci].kind { + OpKind::Call { args, .. } if args.len() == 2 => (args[0].clone(), args[1].clone()), + other => { + return Err(format!( + "{name}: slice::get producer op is not a 2-arg call: {other:?}" + )); + } + }; + + // A's single exit → B (the continuation consuming the Option). Must be a + // plain goto — `lower_call` closes with exactly this shape. + let [exit] = graph.blocks[a].exits.as_slice() else { + return Err(format!( + "{name}: slice::get call block {a} does not have a single exit" + )); + }; + if exit.exitcase.is_some() || exit.last_exception.is_some() || exit.last_exc_value.is_some() { + return Err(format!( + "{name}: slice::get call block {a} exit is not a plain goto" + )); + } + let saved_exit = exit.clone(); + let b_target = saved_exit.target; + + // `carried` = the distinct live Values A forwards to B other than the + // Option itself (`flow_result`); each must be threaded through the diamond + // arms to reach B (a fresh block cannot see A-scope Variables directly). + let mut carried: Vec = Vec::new(); + for arg in &saved_exit.args { + if let LinkArg::Value(v) = arg + && *v != flow_result + && !carried.contains(v) + { + carried.push(v.clone()); + } + } + + // --- All structural validation passed; mutate the graph. --- + + // `then_bb` (`Some`) carries `carried` plus `slice` and `index` (the base + // and subscript of the element read); `else_bb` (`None`) carries only + // `carried`. The source-var lists double as the branch link args. + let mut then_sources = carried.clone(); + for v in [&slice, &index] { + if !then_sources.contains(v) { + then_sources.push(v.clone()); + } + } + let (then_bb, then_inputs) = graph.create_block_with_arg_vars(then_sources.len()); + let (else_bb, else_inputs) = graph.create_block_with_arg_vars(carried.len()); + + // `then_bb`: elem = slice[i]; opt = Some(elem). + let slice_in_then = map_source(&then_sources, &then_inputs, &slice) + .ok_or_else(|| format!("{name}: slice not threaded into Some arm"))?; + let index_in_then = map_source(&then_sources, &then_inputs, &index) + .ok_or_else(|| format!("{name}: index not threaded into Some arm"))?; + let elem = graph.alloc_value_var(); + graph.block_mut(then_bb).operations.push(SpaceOperation { + result: Some(elem.clone()), + kind: OpKind::ArrayRead { + base: slice_in_then, + // The element read and the `Some::__pos_0` field write below + // consume the same `elem`, so the read's declared element type + // must be the payload type the field carries (`site.payload_ty`, + // `Ref(None)` for `Option<&PyObjectRef>`) — a hardcoded `Ref(None)` + // would disagree for any instantiation whose payload projects to + // another `ValueType`. + item_ty: site.payload_ty.clone(), + index: index_in_then, + array_type_id: None, + nolength: false, + pure: false, + }, + }); + let some_var = emit_option_variant( + graph, + then_bb, + &site.option_owner, + 1, + Some((&site.some_owner, elem, site.payload_ty.clone())), + ); + let then_result = emit_narrow(graph, then_bb, some_var, &narrow_root); + let then_link_args = reproduce_exit_args( + &saved_exit, + &flow_result, + &then_result, + &then_sources, + &then_inputs, + &name, + )?; + close_goto_mixed(graph, then_bb, b_target, then_link_args); + + // `else_bb`: opt = None. + let none_var = emit_option_variant(graph, else_bb, &site.option_owner, 0, None); + let else_result = emit_narrow(graph, else_bb, none_var, &narrow_root); + let else_link_args = reproduce_exit_args( + &saved_exit, + &flow_result, + &else_result, + &carried, + &else_inputs, + &name, + )?; + close_goto_mixed(graph, else_bb, b_target, else_link_args); + + // A: drop the residual `get` call (+ absorbed cast), synthesize the guard + // `i < len(slice)`, branch on it. `__len` on the `SomeList`-modelled slice + // routes through the rtyper `len` op → `AbstractBaseListRepr.rtype_len` → + // `arraylen_gc`. The compare's result is a Bool, so it declares `Int` (as + // `range_contains` and `slice_first` do) and never the `Unsigned` that + // would wrap the flag in `r_uint`; the annotator derives the `lt` llop from + // the OPERAND annotations, where the `usize` index and the non-negative + // `arraylen_gc` both admit the unsigned compare. `set_branch` appends the + // idempotent `bool(cond)` hop and installs the Bool(false)/Bool(true) arm + // links, so the true (in-bounds) arm is `then_bb`. + let a_id = graph.blocks[a].id; + for _ in ci..=remove_upto { + graph.blocks[a].operations.remove(ci); + } + let len = graph.alloc_value_var(); + graph.block_mut(a_id).operations.push(SpaceOperation { + result: Some(len.clone()), + kind: OpKind::Call { + target: CallTarget::FunctionPath { + segments: vec!["__len".to_string()], + }, + args: vec![slice], + result_ty: ValueType::Int, + }, + }); + let cond = graph.alloc_value_var(); + graph.block_mut(a_id).operations.push(SpaceOperation { + result: Some(cond.clone()), + kind: OpKind::BinOp { + op: "lt".to_string(), + lhs: index, + rhs: len, + result_ty: ValueType::Int, + }, + }); + graph.set_branch(a_id, cond, then_bb, then_sources, else_bb, carried); + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn slice_get_site(result_var: Variable) -> SliceGetSite { + SliceGetSite { + result_var, + option_owner: "core::option::Option".into(), + some_owner: "core::option::Option::Some".into(), + payload_ty: ValueType::Ref(None), + } + } + + fn emit_call(g: &mut FunctionGraph, a: crate::model::BlockId, args: Vec) -> Variable { + g.push_op_var( + a, + OpKind::Call { + target: CallTarget::FunctionPath { + segments: vec!["core".into(), "slice".into(), "".into(), "get".into()], + }, + args, + result_ty: ValueType::Ref(None), + }, + true, + ) + .unwrap() + } + + fn residual_get_survives(g: &FunctionGraph, a: crate::model::BlockId) -> bool { + g.blocks[a.0].operations.iter().any(|op| { + matches!( + &op.kind, + OpKind::Call { target: CallTarget::FunctionPath { segments }, .. } + if segments.last().map(String::as_str) == Some("get") + ) + }) + } + + /// Build the minimal `opt = get(slice, i)` shape — block A = the residual + /// call closed by a single goto to B (which consumes the Option) — and + /// assert the rewrite drops the call, synthesizes `i < len(slice)`, and + /// branches to a `Some` arm (`slice[i]` → `Some`) and a `None` arm, both + /// merging to B. + #[test] + fn rewrite_lifts_get_to_bounds_checked_option() { + let mut g = FunctionGraph::new("test_slice_get"); + let a = g.startblock; + let slice = g.push_op_var(a, OpKind::ConstInt(0), true).unwrap(); + let index = g.push_op_var(a, OpKind::ConstInt(3), true).unwrap(); + let opt = emit_call(&mut g, a, vec![slice.clone(), index.clone()]); + + // B: the continuation consuming the get() result. + let (b, _b_args) = g.create_block_with_arg_vars(1); + g.set_return(b, None); + g.set_goto(a, b, vec![opt.clone()]); + + let rewritten = rewire_slice_get_call_sites(&mut g, &[slice_get_site(opt)]); + assert_eq!(rewritten, 1, "the slice::get site must be rewritten"); + + // The residual `get` call is gone from block A. + assert!( + !residual_get_survives(&g, a), + "residual get call removed from A" + ); + // A synthesizes the `__len` guard and an `lt` compare, then branches. + assert!( + g.blocks[a.0].operations.iter().any(|op| matches!( + &op.kind, + OpKind::Call { target: CallTarget::FunctionPath { segments }, .. } + if segments.first().map(String::as_str) == Some("__len") + )), + "A synthesizes the __len guard" + ); + assert!( + g.blocks[a.0] + .operations + .iter() + .any(|op| matches!(&op.kind, OpKind::BinOp { op, .. } if op == "lt")), + "A compares i < len" + ); + assert_eq!(g.blocks[a.0].exits.len(), 2, "A branches to Some/None arms"); + // Exactly one arm reads an array element (the Some payload), and it + // subscripts with the arm's threaded index — NOT a synthesized 0, which + // is what would make this pass a mis-indexed `slice_first`. + let elem_reads: Vec<&Variable> = g + .blocks + .iter() + .flat_map(|blk| &blk.operations) + .filter_map(|op| match &op.kind { + OpKind::ArrayRead { index, .. } => Some(index), + _ => None, + }) + .collect(); + assert_eq!(elem_reads.len(), 1, "the Some arm reads slice[i]"); + let then_inputs = &g.block(g.blocks[a.0].exits[1].target).inputargs; + assert!( + then_inputs.contains(elem_reads[0]), + "the element read subscripts the arm's threaded index" + ); + // Both arms write an Option `__discriminant` (Some=1, None=0). + let disc_writes = g + .blocks + .iter() + .flat_map(|blk| &blk.operations) + .filter(|op| { + matches!(&op.kind, OpKind::FieldWrite { field, .. } if field.name == "__discriminant") + }) + .count(); + assert_eq!(disc_writes, 2, "both arms write a discriminant"); + } + + /// The production shape: an `Option<&RegisteredStruct>` result appends a + /// trailing `__pyre_cast_instance` narrowing op, so the call is NOT the + /// block tail. The rewrite absorbs the cast, fires, and re-applies the + /// narrowing in both arms. + #[test] + fn rewrite_absorbs_trailing_narrow_cast() { + let mut g = FunctionGraph::new("test_slice_get_narrow"); + let a = g.startblock; + let slice = g.push_op_var(a, OpKind::ConstInt(0), true).unwrap(); + let index = g.push_op_var(a, OpKind::ConstInt(3), true).unwrap(); + let opt = emit_call(&mut g, a, vec![slice, index]); + let narrowed = g + .push_op_var( + a, + OpKind::Call { + target: CallTarget::FunctionPath { + segments: vec![ + crate::pyre_names::shims::CAST_INSTANCE.into(), + "PyObject".into(), + ], + }, + args: vec![opt.clone()], + result_ty: ValueType::Ref(Some("PyObject".into())), + }, + true, + ) + .unwrap(); + + // B consumes the NARROWED value (what the block actually forwards). + let (b, _b_args) = g.create_block_with_arg_vars(1); + g.set_return(b, None); + g.set_goto(a, b, vec![narrowed]); + + // Site records the CALL result; the rewrite discovers the trailing cast. + let rewritten = rewire_slice_get_call_sites(&mut g, &[slice_get_site(opt)]); + assert_eq!(rewritten, 1, "the slice::get site must be rewritten"); + + // The residual call and the trailing cast are gone from block A. + assert!( + !residual_get_survives(&g, a), + "residual get call removed from A" + ); + // Two arms each re-emit a `__pyre_cast_instance` narrowing. + let narrow_casts = g + .blocks + .iter() + .flat_map(|blk| &blk.operations) + .filter(|op| { + matches!( + &op.kind, + OpKind::Call { target: CallTarget::FunctionPath { segments }, .. } + if segments.first().map(String::as_str) == Some(crate::pyre_names::shims::CAST_INSTANCE) + ) + }) + .count(); + assert_eq!(narrow_casts, 2, "each diamond arm re-applies the narrowing"); + } + + /// A call block whose trailing shape is neither the bare call nor a single + /// absorbed cast declines (fail-safe): the residual call survives untouched. + #[test] + fn rewrite_declines_on_unexpected_trailing_shape() { + let mut g = FunctionGraph::new("test_slice_get_decline"); + let a = g.startblock; + let slice = g.push_op_var(a, OpKind::ConstInt(0), true).unwrap(); + let index = g.push_op_var(a, OpKind::ConstInt(3), true).unwrap(); + let opt = emit_call(&mut g, a, vec![slice, index]); + // Two trailing ops break both the "call is last" and "single cast" shapes. + g.push_op_var(a, OpKind::ConstInt(9), true).unwrap(); + g.push_op_var(a, OpKind::ConstInt(8), true).unwrap(); + g.set_return(a, None); + + let rewritten = rewire_slice_get_call_sites(&mut g, &[slice_get_site(opt)]); + assert_eq!(rewritten, 0, "an unexpected trailing shape declines"); + assert!( + residual_get_survives(&g, a), + "residual call survives on decline" + ); + } + + /// The arity guard: `get` is recorded with two operands, so a located + /// producer that is a 1-arg call (the `first` shape) cannot supply an + /// index. It declines rather than reading `slice[?]`. + #[test] + fn rewrite_declines_on_one_arg_producer() { + let mut g = FunctionGraph::new("test_slice_get_arity"); + let a = g.startblock; + let slice = g.push_op_var(a, OpKind::ConstInt(0), true).unwrap(); + let opt = emit_call(&mut g, a, vec![slice]); + + let (b, _b_args) = g.create_block_with_arg_vars(1); + g.set_return(b, None); + g.set_goto(a, b, vec![opt.clone()]); + + let rewritten = rewire_slice_get_call_sites(&mut g, &[slice_get_site(opt)]); + assert_eq!(rewritten, 0, "a 1-arg producer declines"); + assert!( + residual_get_survives(&g, a), + "residual call survives on decline" + ); + } +} diff --git a/majit/majit-translate/src/lib.rs b/majit/majit-translate/src/lib.rs index 5677b1c5a50..802beb2f0dd 100644 --- a/majit/majit-translate/src/lib.rs +++ b/majit/majit-translate/src/lib.rs @@ -32,6 +32,10 @@ pub mod annotator; )] pub mod codewriter; pub mod config; +// Decline census — no upstream counterpart. Upstream gates that cannot +// lower a shape raise a named error (`jtransform.py _handle_list_call`); +// every gate here declines silently, so the refusals are counted instead. +pub mod decline; #[cfg_attr( test, expect( @@ -886,6 +890,12 @@ fn analyze_pipeline_from_module_paths( impl_fnaddr_bindings: &ImplFnAddrBindings<'_>, static_addrs: HostStaticAddrs<'_>, ) -> pipeline::ProgramPipelineResult { + // Dump the decline census when this run ends — on the normal return + // AND on the unwind, since a pipeline that panics on an undigestible + // shape is exactly the run whose silent refusals a reader needs. + // Silent unless `MAJIT_DECLINE_LOG` / `PYRE_MIR_FRONTEND_DEBUG` is + // set, so default output is unchanged. + let _decline_census = decline::CensusScope::new("analyze_pipeline"); let mut prof = PhaseProfiler::new(); macro_rules! mark_phase { ($name:literal) => { diff --git a/majit/majit-translate/src/model.rs b/majit/majit-translate/src/model.rs index 23da81c5177..58e3d808cea 100644 --- a/majit/majit-translate/src/model.rs +++ b/majit/majit-translate/src/model.rs @@ -3038,11 +3038,24 @@ pub fn remove_dead_aggregates(graph: &mut FunctionGraph) -> usize { /// The orphaned aggregate ctor + header `FieldWrite`s become dead and are swept /// by the `remove_dead_aggregates` + `prune_dead_phis` passes that follow in /// `simplify_lowered_graph`. +/// +/// Every refusal below is a bare `continue` that leaves `malloc_typed` +/// residual, and none of them says so anywhere. They are counted through +/// [`crate::decline`] under the two gate names declared next; see that +/// module for why the shape filters at the top of the site loop are +/// deliberately not among them. pub fn fuse_boxing_alloc( graph: &mut FunctionGraph, struct_field_attrs: &std::collections::HashMap>, ) -> usize { use crate::flowspace::model::Variable; + // Gate names come from `crate::decline::gate` rather than being + // spelled here: a name defined at its call site can be referenced + // while the recorder module is absent, which is how a half-applied + // edit left this function naming two gates the tree could not build. + use crate::decline::gate::{ + FUSE_BOXING_ALLOC as FUSE_GATE, RESOLVE_HEADER_PLAN as VTABLE_GATE, + }; // Derive a boxing struct's scalar payload fields from its registered field // layout, in struct-declaration order, skipping the `ob_header` (PyObject // base): the header's type pointer is captured separately into @@ -3058,20 +3071,14 @@ pub fn fuse_boxing_alloc( // already-computed layout map the front end owns, so this pass performs no // front-end (Llbc) reads. // - // The ctor carries the bare struct leaf (`W_FloatObject`) while the map is - // keyed by the crate-stripped qualified path (`floatobject::W_FloatObject`), - // so fall back to a leaf match when the exact key misses. An unknown struct - // yields `None` and is left unfused — the same fail-safe the old hardcoded - // set applied to any struct outside the numeric four. + // An unknown struct yields `None` and is left unfused — the same fail-safe + // the old hardcoded set applied to any struct outside the numeric four. // - // The `PyObject` base field carries the type pointer, which - // `resolve_vtable_addr` lifts into `NewWithVtable.vtable`, so it is skipped - // here (the runtime stamps `ob_type`/`w_class` from the descriptor). Two - // spellings reach this pass: the hand-written boxing structs - // (`W_FloatObject` etc.) name it `ob_header`, while `#[pyre_class]` injects - // it as `ob` (pyre-macros `expand_pyre_class`). Both name the same base, - // so both are recognised as the header and neither is re-emitted as a - // payload setfield. + // Two spellings of the `PyObject` base reach this pass: the hand-written + // boxing structs (`W_FloatObject` etc.) name it `ob_header`, while + // `#[pyre_class]` injects it as `ob` (pyre-macros `expand_pyre_class`). + // Both name the same base, so both are recognised as the header and + // neither is re-emitted as a payload setfield. fn is_header_field(name: &str) -> bool { name == "ob_header" || name == "ob" } @@ -3377,10 +3384,23 @@ pub fn fuse_boxing_alloc( w_class: Option, } let resolve_header_plan = |graph: &FunctionGraph, agg: &Variable| -> Option { - let header = - store_value(graph, agg, "ob_header").or_else(|| store_value(graph, agg, "ob"))?; + let Some(header) = + store_value(graph, agg, "ob_header").or_else(|| store_value(graph, agg, "ob")) + else { + crate::decline::record( + VTABLE_GATE, + "no-unique-ob_header-store", + format_args!("{}", graph.name), + ); + return None; + }; let mut roots = Vec::new(); if !store_roots(graph, &header, 8, &mut roots) { + crate::decline::record( + VTABLE_GATE, + "header-roots-unresolvable", + format_args!("{}", graph.name), + ); return None; } let mut resolved: Option = None; @@ -3388,15 +3408,36 @@ pub fn fuse_boxing_alloc( // folds into the vtable, `Some(Some(_))` once one does not. let mut w_class: Option> = None; for root in &roots { - let vtable = store_value(graph, root, "ob_type") - .and_then(|obtype| const_ref_addr(graph, &obtype, 8))?; + let Some(vtable) = store_value(graph, root, "ob_type") + .and_then(|obtype| const_ref_addr(graph, &obtype, 8)) + else { + // The commonest reason in a test fixture: the `PyType` + // singleton addresses were not supplied, so the `&T` read + // stayed a residual call rather than a `ConstRefAddr`. It is + // indistinguishable here from a graph that genuinely stores no + // type pointer, which is why the count names the resolution + // step rather than guessing the cause. + crate::decline::record( + VTABLE_GATE, + "ob_type-not-a-constant-address", + format_args!("{}", graph.name), + ); + return None; + }; let declares_no_class_word = header_declares_no_class_word(graph, root, struct_field_attrs); let store = match unique_store(graph, root, "w_class") { // A store to a field absent from the registered layout makes // the layout evidence self-contradictory, so keep the original // allocation rather than guessing which source is correct. - Some(_) if declares_no_class_word => return None, + Some(_) if declares_no_class_word => { + crate::decline::record( + VTABLE_GATE, + "w_class-store-contradicts-layout", + format_args!("{}", graph.name), + ); + return None; + } Some((field, value, ty)) => { let folds = value .as_variable() @@ -3406,11 +3447,22 @@ pub fn fuse_boxing_alloc( } // RPython's root OBJECT declares only `typeptr`. With no // per-instance class word, there is nothing that can disagree - // with the vtable carried by `NewWithVtable`. + // with the vtable carried by `NewWithVtable`, and nothing + // downstream can read one either: every `w_class` consumer + // keys on a field descriptor named `w_class` + // (`descr.rs FieldDescr::is_w_class`), which a struct that + // never declares the field cannot produce. None if declares_no_class_word => None, // A layout that declares `w_class` still needs a unique store // proving that the vtable stands for that per-instance class. - None => return None, + None => { + crate::decline::record( + VTABLE_GATE, + "no-unique-w_class-store", + format_args!("{}", graph.name), + ); + return None; + } }; match &w_class { None => w_class = Some(store), @@ -3427,6 +3479,11 @@ pub fn fuse_boxing_alloc( _ => false, }; if !agrees { + crate::decline::record( + VTABLE_GATE, + "header-roots-disagree-about-w_class", + format_args!("{}", graph.name), + ); return None; } } @@ -3436,9 +3493,20 @@ pub fn fuse_boxing_alloc( Some(seen) if seen == vtable => {} // Predecessors building headers for different types merge into // one malloc: no single vtable stands for the whole cluster. - Some(_) => return None, + Some(_) => { + crate::decline::record( + VTABLE_GATE, + "header-roots-name-different-vtables", + format_args!("{}", graph.name), + ); + return None; + } } } + // `store_roots` only answers `true` for a non-empty `roots`, and every + // iteration either sets `resolved` or returns, so the `?` below is + // unreachable rather than a decline path — it is not recorded, and a + // `None` from there would be a bug in that invariant, not a refusal. Some(HeaderPlan { vtable: resolved?, w_class: w_class @@ -3464,13 +3532,27 @@ pub fn fuse_boxing_alloc( let mut sites: Vec = Vec::new(); for (bi, block) in graph.blocks.iter().enumerate() { for (oi, op) in block.operations.iter().enumerate() { + // The two shape tests below are the population filter, not a + // decline: every operation in the graph is offered to them and + // almost all fail. Recording here would make the instrument + // part of the population it measures — the fuse's count would + // be dominated by ops that were never boxing clusters. + // Recording starts once the op IS a `malloc_typed` call, i.e. + // once the pass has committed to lowering it. let OpKind::Call { target, args, .. } = &op.kind else { continue; }; if !is_malloc_typed(target) || args.len() != 1 { continue; } - let Some(result) = &op.result else { continue }; + let Some(result) = &op.result else { + crate::decline::record( + FUSE_GATE, + "malloc-call-has-no-result-var", + format_args!("{}", graph.name), + ); + continue; + }; // The aggregate itself can reach the malloc as a `Block.inputargs` // phi, not only its header: a constructor that builds the struct up // front and then branches — `w_float_new` builds the `W_FloatObject` @@ -3484,12 +3566,27 @@ pub fn fuse_boxing_alloc( // `sink_fused_boxing_aggregates_at_raw_writes` matches on. let mut agg_roots = Vec::new(); if !store_roots(graph, &args[0], 8, &mut agg_roots) { + crate::decline::record( + FUSE_GATE, + "aggregate-roots-unresolvable", + format_args!("{}", graph.name), + ); continue; } let Some((agg, other_roots)) = agg_roots.split_first() else { + crate::decline::record( + FUSE_GATE, + "aggregate-has-no-roots", + format_args!("{}", graph.name), + ); continue; }; if other_roots.iter().any(|root| root != agg) { + crate::decline::record( + FUSE_GATE, + "aggregate-roots-disagree", + format_args!("{}", graph.name), + ); continue; } // `%agg` must be a `SyntheticTransparentCtor` for a known boxing @@ -3510,8 +3607,24 @@ pub fn fuse_boxing_alloc( ) if r == agg => Some(name.clone()), _ => None, }); - let Some(owner) = owner else { continue }; + let Some(owner) = owner else { + crate::decline::record( + FUSE_GATE, + "aggregate-is-not-a-synthetic-ctor", + format_args!("{}", graph.name), + ); + continue; + }; let Some(fields) = payload_fields(&owner, struct_field_attrs) else { + // Either the struct has no registered field layout at all, + // or its leaf name is ambiguous across the layout map. Both + // leave `malloc_typed` residual; `registered_layout` is where + // they part, and neither is visible in the fused count. + crate::decline::record( + FUSE_GATE, + "struct-layout-unregistered-or-ambiguous", + format_args!("{owner} in {}", graph.name), + ); continue; }; // Resolve every payload field's store: `FieldWrite { base: %agg, @@ -3536,6 +3649,11 @@ pub fn fuse_boxing_alloc( } } if !complete { + crate::decline::record( + FUSE_GATE, + "payload-store-missing-or-conflicting", + format_args!("{owner} in {}", graph.name), + ); continue; } // Leave the cluster unfused when the `ob_header.ob_type` store @@ -3552,7 +3670,17 @@ pub fn fuse_boxing_alloc( // `ConstRefAddr`; the production driver supplies those addresses, // so there the pointer resolves. Predecessors that disagree about // the type or the class also decline — see `HeaderPlan`. + // + // Which of those it was is not knowable from the `None`; the + // `model::resolve_header_plan` rows are where that is recorded, + // and this row is the count of clusters the fuse gave up on for + // any header reason at all. let Some(header) = resolve_header_plan(graph, agg) else { + crate::decline::record( + FUSE_GATE, + "vtable-unresolved", + format_args!("{owner} in {}", graph.name), + ); continue; }; sites.push(Site { diff --git a/majit/majit-translate/src/parse.rs b/majit/majit-translate/src/parse.rs index 0fc0cf9c977..a1783348ae8 100644 --- a/majit/majit-translate/src/parse.rs +++ b/majit/majit-translate/src/parse.rs @@ -72,6 +72,25 @@ impl CallPath { } } +/// The `canonical_key` spelling, written straight into a formatter. +/// +/// Lets a diagnostic name a path through `format_args!` without the +/// `canonical_key()` `String` allocation, which matters where the +/// diagnostic is disabled: `format_args!` evaluates its arguments +/// eagerly, so an allocating accessor there would cost on the path that +/// prints nothing. +impl std::fmt::Display for CallPath { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + for (i, segment) in self.segments.iter().enumerate() { + if i > 0 { + f.write_str("::")?; + } + f.write_str(segment)?; + } + Ok(()) + } +} + /// Strip the module prefix and return the trailing identifier. /// /// Accepts both spellings: a `::`-joined Rust path and the `.`-joined diff --git a/majit/majit-translate/src/translator/driver.rs b/majit/majit-translate/src/translator/driver.rs index 9c79df04ac5..28fc0de0983 100644 --- a/majit/majit-translate/src/translator/driver.rs +++ b/majit/majit-translate/src/translator/driver.rs @@ -2602,6 +2602,10 @@ mod tests { ); } + /// Re-enabling this test requires injectable input and output as well as an + /// injectable `CheckpointRuntime`. On the no-fork path the `auto` token is + /// discarded and the prompt reads input before consulting the runtime, so + /// a runtime seam alone cannot prevent a blocking or repeated EOF read. #[test] fn task_database_c_sets_translator_frozen_before_c_backend_leaf() { let td = TranslationDriver::new_default().expect("driver"); diff --git a/majit/majit-translate/src/translator/rtyper/cutover.rs b/majit/majit-translate/src/translator/rtyper/cutover.rs index 2a7733d50d2..ee050e72d07 100644 --- a/majit/majit-translate/src/translator/rtyper/cutover.rs +++ b/majit/majit-translate/src/translator/rtyper/cutover.rs @@ -44,7 +44,7 @@ //! The anchor corpus will surface which //! followup is the next priority. `Ref`-typed operands now route //! through `valuetype_to_someshell(Ref) → SomeInstance(classdef=None)` -//! (`codewriter/annotation_state.rs`), so the rtyper picks +//! (`codewriter/annotation_state.rs:69`), so the rtyper picks //! `getinstancerepr(rtyper, None, Gc) → InstanceRepr::new_rootinstance //! → Ptr(GcStruct(OBJECT))` and the projection collapses to //! `ConcreteType::GcRef` matching the legacy resolver — the previous @@ -989,12 +989,12 @@ fn dead_op_result_vars(graph: &LegacyGraph) -> std::collections::HashSet std::collections::HashS /// `FieldWrite` of the same field name — the extract-then-repack of an /// identity `match` re-wrap (`match step { Return(v) => Ok(Return(v)), /// CloseLoop { jump_args, loop_header_pc } => Ok(CloseLoop { .. }), … }`, -/// `pyopcode.rs`). Each arm reads the incoming variant's payload +/// `pyopcode.rs:1945`). Each arm reads the incoming variant's payload /// (`FieldRead("__pos_0" | "loop_header_pc", owner = StepResult::Variant)`) /// and immediately writes it into a freshly-built outgoing variant /// (`FieldWrite(same field, owner = StepResult<…>::Variant)`). @@ -1241,12 +1241,12 @@ fn collect_divergences( // `JUMP_*` delta — all bare bytecode operands the rtyper types // `Signed` (matching upstream, where a bytecode oparg is a plain // int). The legacy walker's `GcRef` is the pyre-only conservative - // `Unknown → GcRef` backfill (`legacy_resolve.rs`), which has + // `Unknown → GcRef` backfill (`legacy_resolve.rs:374-378`), which has // no RPython analogue (upstream's rtyper never leaves a value // untyped, so it never defaults to ref); it is the divergent side. // // This pair once appeared unacceptable because accepting it crashed - // `emit_list_of_kind` (`assembler.rs`). That crash was a phase + // `emit_list_of_kind` (`assembler.rs:2169`). That crash was a phase // ordering artifact, not a real mistype: a residual `dont_look_inside` // decode helper's argument list is partitioned by kind at jtransform // time (`make_three_lists_from_vars`), and the real path's kind was @@ -1417,7 +1417,8 @@ fn compare_real_against_legacy( /// The returned tag identifies the matching fallback condition. Keeping the /// category and the boolean decision in one function prevents diagnostic code /// from maintaining a second, drifting copy of the substring table. Lowering -/// only tests whether the result is `Some`. +/// only tests whether the result is `Some`; the tags are stable identifiers +/// for the decline census (`crate::decline`). pub(crate) fn unported_category(msg: &str) -> Option<&'static str> { if msg.contains("not registered in PyreCallRegistry") { return Some("call-registry-miss"); @@ -1764,6 +1765,30 @@ pub(crate) fn unported_category(msg: &str) -> Option<&'static str> { None } +/// Category for a Skip that never passes through [`unported_category`]. +/// +/// [`unported_category`] classifies a PANIC payload: the dual gate +/// catches an unwind and asks whether the failure is a known-unported +/// shape. A Skip returned directly as `Ok(DualGateOutcome::Skip(..))` +/// never reaches that predicate, so a census that only broke down +/// `unported_category`'s arms would file every one of them under +/// "unclassified" — and on cel's closure the direct Skips are the ones +/// that fire. The two populations are disjoint, which is why this is a +/// separate function rather than more arms: adding these to +/// `unported_category` would make `is_known_unported` answer `true` for +/// them and change what the panic handler does. +/// +/// Census classification only. No lowering decision reads it. +pub(crate) fn non_arm_skip_category(msg: &str) -> Option<&'static str> { + if msg.contains("two-phase: graph was never a prepass subject") { + return Some("two-phase-never-a-subject"); + } + if msg.contains("two-phase: subject rtype-skipped in prepass") { + return Some("two-phase-rtype-skipped"); + } + None +} + /// Whether the dual gate should Skip this failure to the legacy walker. /// /// The predicate is [`unported_category`] with its answer erased, so there is @@ -1789,7 +1814,7 @@ pub(crate) fn is_known_unported(msg: &str) -> bool { /// 1. `get_or_register` every entry's signature so HostObjects exist /// before any callee body lifts. /// 2. `lift_callee_to_pygraph` + `prefill_default_cache` per entry -/// so `cachedgraph` (`description.rs`) hits at the +/// so `cachedgraph` (`description.rs:1037-1039`) hits at the /// rtyper's `direct_call`. pub(crate) fn populate_call_registry_from_call_graphs( function_graphs: &crate::codewriter::call::GraphStore, @@ -1797,6 +1822,12 @@ pub(crate) fn populate_call_registry_from_call_graphs( foreign_opaque_method_externals: &[(Vec, Signature, crate::model::ValueType)], registry: &PyreCallRegistry, ) -> Result<(), TyperError> { + // Decline-census gate name for this function's registration skips. + // A callable skipped here has no registry entry, so every callsite + // resolves it as a host builtin or a residual stub rather than as a + // user graph — silently, with the four reasons below indistinguishable + // from one another at the callsite. + use crate::decline::gate::CALL_REGISTRY as REGISTRY_GATE; // Dedupe by canonical path — RPython `Bookkeeper.getdesc(pyobj)` // (`bookkeeper.py:353-409`) returns the *same* FunctionDesc for // any reference to the same callable, keyed by `Constant(pyobj)` @@ -1809,7 +1840,7 @@ pub(crate) fn populate_call_registry_from_call_graphs( // contract. // // The dedupe key strips a leading crate-prefix segment so the - // five alias-explosion shapes registered by `lib.rs` for + // five alias-explosion shapes registered by `lib.rs:438-490` for // each free function (`[mod, foo]`, `[crate, mod, foo]`, // `[pyre_interpreter, mod, foo]`, `[pyre_object, mod, foo]`, // `[pyre_jit, mod, foo]`) collapse to a single canonical entry @@ -1868,7 +1899,7 @@ pub(crate) fn populate_call_registry_from_call_graphs( // `W_ComplexObject`/`W_LongObject`, per `model.rs payload_fields`) that // `fuse_boxing_alloc` rewrites to a native // `NewWithVtable` during MIR `simplify_lowered_graph` - // (`front/mir.rs`, `model.rs` `payload_fields`) — *before* the + // (`front/mir.rs:1409`, `model.rs` `payload_fields`) — *before* the // rtyper runs, so a numeric `malloc_typed` never reaches Layer-3b. // Upstream `jtransform.rewrite_op_malloc` (`jtransform.py`) lowers // EVERY mallocable GC struct to `new`/`new_with_vtable`; pyre has not @@ -1889,6 +1920,11 @@ pub(crate) fn populate_call_registry_from_call_graphs( if canonical_strip == ["lltype", "malloc_typed"] || canonical_strip == ["lltype", "malloc_typed_managed"] { + crate::decline::record( + REGISTRY_GATE, + "skip-malloc-typed-intrinsic", + format_args!("{path}"), + ); continue; } // `#[pyre_class]`'s generated allocation constructors @@ -1909,6 +1945,11 @@ pub(crate) fn populate_call_registry_from_call_graphs( canonical_strip.last().map(String::as_str), Some("allocate") | Some("allocate_stable") ) { + crate::decline::record( + REGISTRY_GATE, + "skip-pyre-class-allocate-ctor", + format_args!("{path}"), + ); continue; } // `pyre_object::lltype::malloc_raw` is the raw (non-GC) allocation @@ -1921,6 +1962,11 @@ pub(crate) fn populate_call_registry_from_call_graphs( // `malloc_typed`: callsites resolve to the HOST_ENV builtin // (translate_op Layer-3b) instead of this failed user-graph entry. if canonical_strip == ["lltype", "malloc_raw"] { + crate::decline::record( + REGISTRY_GATE, + "skip-malloc-raw-intrinsic", + format_args!("{path}"), + ); continue; } // The `pyobject::ll_issubclass` / `ll_issubclass_const` / `ll_isinstance` @@ -1940,6 +1986,11 @@ pub(crate) fn populate_call_registry_from_call_graphs( || canonical_strip == ["pyobject", "ll_isinstance"] || canonical_strip == ["pyobject", "subclass_range_read"] { + crate::decline::record( + REGISTRY_GATE, + "skip-issubclass-helper-body", + format_args!("{path}"), + ); continue; } let signature = function_graphs @@ -1974,7 +2025,7 @@ pub(crate) fn populate_call_registry_from_call_graphs( // fail nor what each computes — only the attribution determinism. pending.sort_by(|a, b| a.0.segments().cmp(b.0.segments())); // Register `unsafe fn` stubs between Pass 1 (alias explosion) and - // Pass 2 (callee lift). `build_flow.rs` rejects unsafe bodies so + // Pass 2 (callee lift). `build_flow.rs:215` rejects unsafe bodies so // they never enter `function_graphs`; without a stub a safe-fn body // lifted in Pass 2 that calls an unsafe callee (`is_cell`, // `is_exception`, …) records a "not registered" lift error and the @@ -2022,7 +2073,7 @@ pub(crate) fn populate_call_registry_from_call_graphs( // `cachedgraph` consumer surfaces the actual producer-side // failure instead of falling through to `buildflowgraph`'s // generic "missing code object" message - // (`translator/translator.rs`). This keeps the lazy-failure + // (`translator/translator.rs:439`). This keeps the lazy-failure // *point of observation* aligned with upstream // `description.py:228` while preserving pyre's eager prefill // shape for the success path. Without per-entry error capture a @@ -2125,7 +2176,7 @@ pub(crate) fn populate_call_registry_from_call_graphs( // minted with no members (`Bookkeeper::intern_class_by_qualname`), // so a receiver method call — `CallTarget::Method` lowers to // `getattr(recv, name)` + `simple_call` - // (`flowspace_adapter.rs`) — found no attribute source and + // (`flowspace_adapter.rs:1357`) — found no attribute source and // blocked. Registering each `[owner, method]` registry entry's // user-function HostObject as a class member completes the // analogue: `find_source_for` imports it into the classdict as a @@ -2172,9 +2223,9 @@ pub(crate) fn populate_call_registry_from_call_graphs( /// Lift a pyre `model::FunctionGraph` (a callee that may appear on a /// `OpKind::Call::FunctionPath` callsite of some other graph) into a /// `Rc` suitable for pre-filling the callee's -/// `FunctionDesc.cache` (`description.rs`). +/// `FunctionDesc.cache` (`description.rs:794`). /// -/// `cachedgraph` (`description.rs`) returns the cached +/// `cachedgraph` (`description.rs:1037-1039`) returns the cached /// `Rc` as soon as the lookup key matches, skipping the /// `buildgraph` path that delegates to /// `translator.buildflowgraph(pyobj, false)` — that delegation @@ -2529,7 +2580,7 @@ fn declared_funcptr_type_from_legacy( /// while the rest of the batch lands. /// /// Mirrors `populate_call_registry_from_call_graphs`'s -/// "register and prefill" contract but feeds +/// "register and prefill" contract (`cutover.rs:856-875`) but feeds /// from the LLBC-sourced stub-spec list (`collect_unsafe_fn_stubs_from_llbc`) /// instead of pyre's /// `function_graphs: HashMap` (which excludes @@ -2545,7 +2596,7 @@ fn declared_funcptr_type_from_legacy( /// is never present in `function_graphs`. `CallControl:: /// find_all_graphs` walks `function_graphs.keys()` only and resolves /// each call target via `target_to_path_and_graph` -/// (`codewriter/call.rs`) which returns `None` for any +/// (`codewriter/call.rs:2601`) which returns `None` for any /// path absent from `function_graphs` — so an unsafe-stub target /// triggers `continue` and is never added to `candidate_graphs`, /// never reaches `transform_graph_to_jitcode`, and never compiles @@ -2943,7 +2994,7 @@ fn drive_subject( // queue, mirroring how callees enter through // `pycall -> recursivecall -> addpendingblock` // (`description.py:283-305`, `annrpython.py:315-336`). - // `addpendingblock` (`annrpython.rs`) writes + // `addpendingblock` (`annrpython.rs:1245-1302`) writes // `Variable.annotation` for each inputarg via // `bindinputargs.setbinding`, inserts `all_blocks[startblock]` // and `annotated[startblock] = None`, then @@ -2952,7 +3003,7 @@ fn drive_subject( // `processblock` flips `annotated[block] = Some(graph)`, // `flowin` walks every op and recurses into successors via // `process_link -> addpendingblock(target, inputs_s)` - // (`annrpython.rs/1690`). Transitive blocks register + // (`annrpython.rs:1502/1690`). Transitive blocks register // themselves into `all_blocks`/`annotated` through the same // bindinputargs path on first arrival. let subject_inputcells = @@ -2972,7 +3023,7 @@ fn drive_subject( // never reach it through a `Link`, but `specialize_more_blocks` // must walk it so its inputargs receive an exception-typed // `concretetype`. The rtyper's `setup_block_entry` - // exception-block branch (`rtyper.rs`) writes + // exception-block branch (`rtyper.rs:1915-1942`) writes // `Variable.concretetype = ExceptionData.lltype_of_exception_*` // without reading `Variable.annotation`, so the block needs no // `flowin`; it only needs to appear in `annotator.annotated`. @@ -2983,7 +3034,7 @@ fn drive_subject( // same exceptblock via `addpendingblock` with `seen_before=true` // and routes to `mergeinputargs` — does not panic on the unbound // `seed_variable(legacy_v)` inputargs from - // `flowspace_adapter.rs`. `setbinding` widens via the + // `flowspace_adapter.rs:1839`. `setbinding` widens via the // lattice's `contains` check, so an `Integer`-already-annotated // slot stays `Integer`; a `None`-annotated slot becomes // `Impossible`, which any later `mergeinputargs` widens to the @@ -3034,7 +3085,7 @@ fn drive_subject( // → policy hook → exit-on-empty. Without this drain, the // `addpendingblock(startblock, inputcells)` queued just above // stays in `genpendingblocks`, `annotated[block]` remains the - // `None` sentinel, and `specialize_block` (`rtyper.rs`) + // `None` sentinel, and `specialize_block` (`rtyper.rs:1656`) // panics on "annotator.annotated[block] is False sentinel". annotator .complete_pending_blocks() @@ -3054,9 +3105,9 @@ fn drive_subject( // Populate per-callsite call-family / calltable state // by walking the seeded blocks' call_ops. `compute_at_fixpoint` - // (`bookkeeper.py:108-118`, pyre `bookkeeper.rs`) drains + // (`bookkeeper.py:108-118`, pyre `bookkeeper.rs:627-648`) drains // `annotator.call_sites()` through `consider_call_site` - // (`bookkeeper.py:152-166`, pyre `bookkeeper.rs`); each + // (`bookkeeper.py:152-166`, pyre `bookkeeper.rs:675`); each // `simple_call(callable_const, *args)` op resolves the callable // to a `SomePBC` (via `immutablevalue_hostobject` for the // pre-registered `HostObject::UserFunction`), then records the @@ -3075,7 +3126,7 @@ fn drive_subject( // `try`/`except`, so a failed `consider_call_site` terminates // `simplify` and unwinds out of the annotator driver. Pyre's // port surfaces the same condition through `?`-propagation - // here rather than swallowing it at `bookkeeper.rs`. + // here rather than swallowing it at `bookkeeper.rs:627-648`. call_registry .bookkeeper() .compute_at_fixpoint() @@ -3570,7 +3621,7 @@ fn run_phase_b_rtype_isolated( // linked into Phase B. The annotator never followed those links — a // constant exitswitch makes the mismatched-exitcase arm carry // `s_ImpossibleValue` through `follow_link`, which returns before - // `links_followed[link] = True` (annrpython.rs), leaving the + // `links_followed[link] = True` (annrpython.rs:1762-1768), leaving the // arm's target unannotated — but the surviving structural link reaches // `specialize_block` → `insert_link_conversions` → `bindingrepr`, which // then KeyErrors on the dead target's unbound inputargs. @@ -3861,7 +3912,7 @@ fn run_phase_b_rtype_isolated( /// as a `None` twin, reached when the rtyper defaulted the un-narrowed /// projection to unit. It would otherwise be colored by /// `emit_call_result_arg` (the op's declared non-void `result_kind` -/// forces the `>X` result argcode, `assembler.rs`) and panic in +/// forces the `>X` result argcode, `assembler.rs:2226`) and panic in /// `lookup_coloring`. A twin the rtyper *positively* typed `Signed` / /// `Float` is left untouched — that is a real kind conflict → Skip. #[expect( @@ -3905,21 +3956,30 @@ pub(crate) fn dual_gate_outcome_from_cache( ) -> Result { // Clone the cached real types out so the cache borrow drops before the // legacy baseline runs (the baseline never touches the cache). + // Two causes reach the Skip below and they are not the same finding: + // the prepass may never have had this graph as a subject at all, or it + // may have had it and rtype-skipped it. The first says the closure + // walk did not reach the graph; the second says it did and the real + // rtyper refused it. They shared one message until now, so a census + // of Skip reasons could not separate "never attempted" from + // "attempted and declined" — the same conflation the decline census + // exists to remove. Neither spelling matches any + // `unported_category` arm, so classification is unchanged. let cached = { let tp = call_registry.two_phase(); match tp.subjects.get(diag_key) { - Some(subj) if !tp.rtype_skipped.contains(&subj.graph_key) => Some(( + Some(subj) if !tp.rtype_skipped.contains(&subj.graph_key) => Ok(( subj.value_to_var.clone(), subj.value_to_var_candidates.clone(), subj.constant_concretetypes.clone(), )), - _ => None, + Some(_) => Err("two-phase: subject rtype-skipped in prepass"), + None => Err("two-phase: graph was never a prepass subject"), } }; - let Some((mut value_to_var, value_to_var_candidates, constants)) = cached else { - return Ok(DualGateOutcome::Skip( - "two-phase: subject not annotated/rtyped in prepass".to_string(), - )); + let (mut value_to_var, value_to_var_candidates, constants) = match cached { + Ok(cached) => cached, + Err(reason) => return Ok(DualGateOutcome::Skip(reason.to_string())), }; select_rtyped_representatives(&mut value_to_var, &value_to_var_candidates) .map_err(|e| e.to_string())?; @@ -5383,7 +5443,7 @@ mod tests { // `prefill_default_cache`, bypassing `buildflowgraph`, so the // graph would otherwise never enter `translator.graphs`. // `FunctionDesc.cachedgraph`'s hit path restores the invariant - // (description.rs). This pins that the registration + // (description.rs:1058-1082). This pins that the registration // actually fires through the shared-bookkeeper session and that // `funcobj.graph` (= the cached PyGraph's `graph`) is the exact // `Rc` now resolvable by the flowspace effect analyzers @@ -5885,7 +5945,7 @@ mod tests { /// `function_graphs.keys()` and resolves each call op's target via /// `target_to_path_and_graph` which requires the target to be /// present in `function_graphs`. This test mirrors the - /// `codewriter.rs` production call shape (registry + + /// `codewriter.rs:192-195` production call shape (registry + /// `callcontrol.unsafe_fn_stubs`) and asserts the path is reachable /// via the registry but not via `CallControl::function_graphs`. #[test] diff --git a/majit/majit-translate/tests/test_cel_census.rs b/majit/majit-translate/tests/test_cel_census.rs new file mode 100644 index 00000000000..278c1706f75 --- /dev/null +++ b/majit/majit-translate/tests/test_cel_census.rs @@ -0,0 +1,1014 @@ +//! Front-end B census over cel's Charon LLBC. +//! +//! This is a measurement rather than an acceptance gate: it prints what the +//! translator can and cannot lower and does not assert census totals. +//! +//! The probes answer independent questions: +//! +//! * `cel_census_call_sites` lowers **every** local body with +//! `lower_fun_decl` and classifies every call site it produces. It needs no +//! portal, so its coverage is the whole artefact rather than one BFS +//! closure. This is the probe that answers which call sites are walls. +//! * `cel_census_pipeline_*` runs the production analyzer from a portal seed, +//! which additionally exercises the codewriter and annotator. Coverage is +//! the portal's graph closure only. +//! +//! ```sh +//! cargo test --release -p majit-translate --test test_cel_census \ +//! -- --nocapture --test-threads=1 +//! ``` +//! +//! Do not pass `--ignored`: `cfg_attr(debug_assertions, ignore)` makes these +//! ordinary tests in release builds, so `--ignored` would select none of them. +//! +//! `--test-threads=1` is not decoration: each pipeline invocation re-seeds +//! process-global registries (`STRUCT_ORIGIN_REGISTRY`, …), so two probes +//! running concurrently race on them. + +use std::collections::{BTreeMap, BTreeSet}; +use std::path::PathBuf; + +use majit_charon_reader::Llbc; +use majit_translate::front::mir::{LowerError, lower_fun_decl}; +use majit_translate::{ + AnalyzeConfig, CallPath, CallTarget, HostStaticAddrs, JitDriverSpec, OpKind, PipelineConfig, +}; + +/// `cel-jit/build/llbc/cel.ullbc`, or `CEL_CENSUS_LLBC`. Returns `None` when +/// absent so the test skips instead of failing in a checkout that has never +/// run the extractor. +fn cel_llbc_path() -> Option { + if let Ok(p) = std::env::var("CEL_CENSUS_LLBC") { + return Some(PathBuf::from(p)); + } + let path = PathBuf::from(concat!( + env!("CARGO_MANIFEST_DIR"), + "/../../cel-jit/build/llbc/cel.ullbc" + )); + path.exists().then_some(path) +} + +/// Assert the pipeline will read the **cel** artefact, and say which one. +/// +/// `PYRE_MIR_FRONTEND_LLBC` is level 2 of a three-level resolution order +/// (`build_semantic_program_via_active_frontend`): explicit analyzer paths, +/// then this variable, then auto-discovery of +/// `/build/llbc/{pyre-object,pyre-interpreter,pyre-jit}.ullbc`. +/// +/// Level 3 is a documented success path, and those files exist in this +/// workspace. A census that failed to set the variable would +/// read pyre's corpus instead, raise nothing, and print cells indistinguishable +/// from a real reading. That is why this asserts rather than prints: a +/// diagnostic nobody reads would be the same failure one layer up. +/// +/// `--test-threads=1` is required because the variable is +/// process-global, so tests running in parallel would interleave it and the +/// failure would present as one census silently reading another's corpus. +/// Whoever speeds this suite up will reach for that flag first; the post-run +/// call below is what catches them, because it re-checks after the pipeline has +/// had a chance to observe a value some other test replaced. +/// FNV-1a 64 over the artefact's bytes. +/// +/// Inlined rather than taken as a dependency because the whole point is that +/// two *different runs* can be compared, so the function must be stable across +/// Rust versions and machines. `DefaultHasher` explicitly promises the +/// opposite, and a hash that silently changes meaning between runs is worse +/// than none: it reads like a corpus that moved. +fn content_hash(bytes: &[u8]) -> u64 { + let mut h: u64 = 0xcbf2_9ce4_8422_2325; + for b in bytes { + h ^= *b as u64; + h = h.wrapping_mul(0x0000_0100_0000_01b3); + } + h +} + +/// Returns the artefact's content hash, so the caller can compare the +/// before-run and after-run readings. +/// +/// Path and hash are printed on one line, and the path is not replaced by a +/// label. A hash captured under a label like `corpus sha256` and a +/// hash captured from a file are indistinguishable once they are two hex +/// strings in a message: nothing about them says they came from different +/// files, and a before/after "transition" assembled that way is two quantities +/// each measured once. Only the path travelling next to the digest prevents it. +fn assert_llbc_is_cel(expected: &PathBuf, when: &str) -> u64 { + let resolved = std::env::var("PYRE_MIR_FRONTEND_LLBC").unwrap_or_else(|_| { + panic!( + "{when}: PYRE_MIR_FRONTEND_LLBC is unset — the pipeline would auto-discover pyre's LLBC" + ) + }); + assert_eq!( + PathBuf::from(&resolved), + *expected, + "{when}: PYRE_MIR_FRONTEND_LLBC does not name the artefact this census set" + ); + // Content, not filename: a basename check would pass on a pyre corpus that + // happened to be copied to this path. + let provenance = expected.with_extension("ullbc.provenance"); + // Fail closed when the sidecar is absent. This is an anticipated state: + // `llbc_extract.py` reports that an artefact may predate provenance or may + // have been written by another extractor. Accepting that state would make + // this check unable to establish which corpus produced the census. + let text = std::fs::read_to_string(&provenance).unwrap_or_else(|e| { + panic!( + "{when}: {} has no readable provenance sidecar ({e}) — refusing to call this a cel \ + extraction on the strength of its filename", + expected.display() + ) + }); + assert!( + text.lines().any(|l| l.trim() == "crate=cel"), + "{when}: {} is not a cel extraction — {} does not say `crate=cel`", + expected.display(), + provenance.display() + ); + let bytes = std::fs::read(expected) + .unwrap_or_else(|e| panic!("{when}: cannot read {} — {e}", expected.display())); + let hash = content_hash(&bytes); + eprintln!( + "[census] {when}: {} len={} fnv1a64={:016x}", + expected.display(), + bytes.len(), + hash + ); + hash +} + +fn skip_note() { + eprintln!( + "skipping: cel.ullbc missing — run `python3 scripts/extract-llbc.py cel` \ + in cel-jit, or set CEL_CENSUS_LLBC" + ); +} + +fn bump(counts: &mut BTreeMap<&'static str, usize>, key: &'static str) { + *counts.entry(key).or_default() += 1; +} + +/// Lower every local body and classify every call site in the result. +/// +/// The wall classes are named as such: a wall stops the graph, whereas an +/// ordinary residual call is a normal lowering that the codewriter continues +/// past. `__dyn_call` is documented at `front/mir.rs:16542` as "not a +/// lowering, it is a placeholder: an unregistered synthetic path that stops +/// whatever graph reaches it". +#[test] +#[cfg_attr( + debug_assertions, + ignore = "release-only: lowers the whole cel LLBC; use `cargo test --release --test test_cel_census`" +)] +fn cel_census_call_sites() { + let Some(path) = cel_llbc_path() else { + skip_note(); + return; + }; + let llbc = Llbc::load(&path).expect("load cel llbc"); + + let mut fns_total = 0usize; + let mut fns_bodyless = 0usize; + let mut fns_failed: BTreeMap = BTreeMap::new(); + let mut graphs = Vec::new(); + // Leaf names of bodies that lowered. Used only to split FunctionPath call + // sites into "callee has a graph here" and "callee does not"; leaf-keyed, + // so the ambiguity count below is reported alongside it rather than + // silently absorbed. + let mut lowered_leaves: BTreeMap = BTreeMap::new(); + + for fd in llbc.iter_local_fns() { + if fd.is_global_initializer.is_some() { + continue; + } + fns_total += 1; + if fd.unstructured().is_none() { + fns_bodyless += 1; + continue; + } + match lower_fun_decl(&llbc, fd) { + Ok(graph) => { + if let Some(leaf) = graph.name.rsplit("::").next() { + *lowered_leaves.entry(leaf.to_string()).or_default() += 1; + } + graphs.push(graph); + } + Err(err) => { + let class = match &err { + LowerError::FunctionNotFound(_) => "FunctionNotFound".to_string(), + LowerError::Schema(_) => "Schema".to_string(), + // Keep the leading clause only: the tail carries block and + // local numbers, which would make every failure unique and + // turn the histogram into a list. + LowerError::Unsupported(msg) => { + let head: String = msg.chars().take(60).collect(); + format!("Unsupported: {head}") + } + }; + *fns_failed.entry(class).or_default() += 1; + } + } + } + + let mut sites = BTreeMap::new(); + let mut residual_callees: BTreeMap = BTreeMap::new(); + let mut indirect_sites: BTreeMap = BTreeMap::new(); + // Name wall owners and indirect sites so equal totals cannot hide a change + // in which graphs or trait methods make up the population. + let mut wall_owners: BTreeMap = BTreeMap::new(); + for graph in &graphs { + for block in &graph.blocks { + for op in &block.operations { + match &op.kind { + OpKind::Call { target, .. } => match target { + CallTarget::FunctionPath { segments } => { + let leaf = segments.last().map(String::as_str).unwrap_or(""); + if leaf == "__dyn_call" { + bump(&mut sites, "WALL call __dyn_call (graph-stopping)"); + *wall_owners.entry(graph.name.clone()).or_default() += 1; + } else if lowered_leaves.contains_key(leaf) { + bump(&mut sites, " call FunctionPath, callee lowered here"); + } else { + bump(&mut sites, " call FunctionPath, no local graph"); + *residual_callees.entry(leaf.to_string()).or_default() += 1; + } + } + CallTarget::Method { .. } => { + bump(&mut sites, " call Method (receiver dispatch)") + } + CallTarget::SyntheticTransparentCtor { .. } => bump( + &mut sites, + " call SyntheticTransparentCtor (enum shell)", + ), + CallTarget::Indirect { + trait_root, + method_name, + } => { + bump(&mut sites, " call Indirect (vtable arm)"); + // Lowering to `Indirect` and resolving its family are + // separate stages, so preserve the member names. + *indirect_sites + .entry(format!("{trait_root}::{method_name}")) + .or_default() += 1; + } + CallTarget::UnsupportedExpr => { + bump(&mut sites, "WALL call UnsupportedExpr"); + *wall_owners.entry(graph.name.clone()).or_default() += 1; + } + }, + OpKind::IndirectCall { graphs, .. } => match graphs { + Some(candidates) if !candidates.is_empty() => { + bump(&mut sites, " indirect-call, candidate graphs present") + } + Some(_) => bump(&mut sites, " indirect-call, empty candidate list"), + None => { + bump(&mut sites, "WALL indirect-call, graphs=None"); + *wall_owners.entry(graph.name.clone()).or_default() += 1; + } + }, + _ => {} + } + } + } + } + + let ambiguous_leaves = lowered_leaves.values().filter(|n| **n > 1).count(); + let total_sites: usize = sites.values().sum(); + let walls: usize = sites + .iter() + .filter(|(k, _)| k.starts_with("WALL")) + .map(|(_, n)| *n) + .sum(); + + eprintln!("=== cel front-end B census: {} ===", path.display()); + // Include process identity and the relevant gate value so separate census + // runs can be distinguished without assuming anything about caches. + eprintln!( + "pid {} PYRE_FNPTR_INDIRECT={}", + std::process::id(), + std::env::var("PYRE_FNPTR_INDIRECT").unwrap_or_else(|_| "".into()) + ); + eprintln!("local fns visited {fns_total}"); + eprintln!(" no body (opaque) {fns_bodyless}"); + eprintln!(" lowered {}", graphs.len()); + eprintln!( + " refused {}", + fns_failed.values().sum::() + ); + for (class, n) in &fns_failed { + eprintln!(" {n:6} {class}"); + } + eprintln!("call sites in lowered graphs {total_sites}"); + for (class, n) in &sites { + eprintln!(" {n:6} {class}"); + } + eprintln!("walls {walls} of {total_sites}"); + eprintln!( + "leaf-keyed control: {ambiguous_leaves} of {} lowered leaves are owned by >1 body, \ + so the FunctionPath split above is approximate by that much", + lowered_leaves.len() + ); + eprintln!("wall sites, by owning graph:"); + for (name, n) in &wall_owners { + eprintln!(" {n:6} {name}"); + } + eprintln!("vtable (Indirect) sites, by trait::method:"); + for (name, n) in &indirect_sites { + eprintln!(" {n:6} {name}"); + } + let mut top: Vec<(&String, &usize)> = residual_callees.iter().collect(); + top.sort_by(|a, b| b.1.cmp(a.1).then(a.0.cmp(b.0))); + eprintln!("top callees with no local graph:"); + for (name, n) in top.into_iter().take(20) { + eprintln!(" {n:6} {name}"); + } +} + +/// Opnames in a `JitCode::dump()`, one per assembled instruction. +/// +/// `format_assembler` (`codewriter/format.rs:112-`) writes one line per +/// `FlatOp`, optionally prefixed by a `%4d ` bytecode position, and every +/// non-label arm opens with the opname. Labels are `L:` and are the only +/// lines that are not instructions, so they are the only thing filtered. +fn dump_opnames(dump: &str) -> impl Iterator { + dump.lines().filter_map(|line| { + let mut tokens = line.split_whitespace(); + let first = tokens.next()?; + // The position prefix is a bare integer; the opname is the next token. + let name = if first.bytes().all(|b| b.is_ascii_digit()) { + tokens.next()? + } else { + first + }; + (!name.ends_with(':')).then_some(name) + }) +} + +/// Print reproducible pipeline-shape measurements for one portal. +/// +/// Two measurements are deliberately absent rather than approximated: +/// +/// * the ULLBC `Drop` / `Call` terminator / fn counts are over the portal's +/// ULLBC closure, which this result does not carry — a whole-artefact count +/// would be a different denominator wearing the same name; +/// * a "real-computation ops" percentage needs a separately specified opname +/// classification, so the full histogram is printed instead. +fn section1_cells(label: &str, result: &majit_translate::pipeline::ProgramPipelineResult) { + let mut hist: BTreeMap = BTreeMap::new(); + for jitcode in &result.jitcodes { + for name in dump_opnames(&jitcode.dump()) { + *hist.entry(name.to_string()).or_default() += 1; + } + } + let ops: usize = hist.values().sum(); + // Print counts both with and without the liveness and end-of-block + // pseudo-ops so consumers can choose an explicit instruction definition. + let live_markers = hist.get("-live-").copied().unwrap_or(0); + let block_markers = hist.get("---").copied().unwrap_or(0); + let count_prefix = |p: &str| -> usize { + hist.iter() + .filter(|(k, _)| k.starts_with(p)) + .map(|(_, n)| n) + .sum() + }; + let exact = |k: &str| -> usize { hist.get(k).copied().unwrap_or(0) }; + + eprintln!("--- §1 cells [{label}] ---"); + eprintln!(" jitcodes {}", result.jitcodes.len()); + eprintln!(" ops (all dump lines) {ops}"); + eprintln!( + " ops less `-live-` {} <- the rule that reproduces §1", + ops - live_markers + ); + eprintln!( + " ops less `-live-` + `---` {}", + ops - live_markers - block_markers + ); + eprintln!( + " residual_call* : inline_call* {} : {}", + count_prefix("residual_call"), + count_prefix("inline_call") + ); + eprintln!(" guard_class {}", exact("guard_class")); + // The dump and instruction table use different spellings for this op, so + // print both rather than silently reporting zero for one representation. + eprintln!( + " vtablemethodptr (dump) {} [insns-table spelling `vtable_method_ptr`: {}]", + exact("vtablemethodptr"), + exact("vtable_method_ptr") + ); + // Same split as `vtablemethodptr` above, and it read as a hard zero until + // it was checked: the dump spells this op `newwithvtable`, the insns table + // spells it `new_with_vtable`, and this cell used to look up the latter in + // the former's histogram. Every `new_with_vtable 0` recorded from this + // harness before that was fixed is the lookup missing, not a fuse that + // declined — so print both and let the reader see which one is populated. + eprintln!( + " new / newwithvtable {} / {} [insns-table spelling `new_with_vtable`: {}]", + exact("new"), + exact("newwithvtable"), + exact("new_with_vtable") + ); + eprintln!( + " indirectcalltarget_indices {}", + result.indirectcalltarget_indices.len() + ); + eprintln!(" opname histogram ({} distinct):", hist.len()); + let mut rows: Vec<(&String, &usize)> = hist.iter().collect(); + rows.sort_by(|a, b| b.1.cmp(a.1).then(a.0.cmp(b.0))); + for (name, n) in rows { + eprintln!(" {n:6} {name}"); + } +} + +fn run_pipeline_census(label: &str, portal: CallPath) { + run_pipeline_census_with_pytypes(label, portal, &[]); +} + +/// [`run_pipeline_census`], with the class statics' addresses supplied. +/// +/// `pytypes` is not decoration for a boxing census — it is the gate. +/// `resolve_vtable_addr` has to turn `&CEL_INT_CLASS` into an address to put in +/// `NewWithVtable.vtable`, and `HostStaticAddrs.pytypes` is the only channel +/// that carries one (`test_mir_frontend.rs +/// boxing_cluster_fuses_from_the_host_supplied_class_address` asserts that +/// supplying it is by itself sufficient). With the default empty slice the fuse +/// declines with a bare `continue`, so a `new_with_vtable` of `0` reports the +/// missing address rather than anything about the constructor — which is why +/// the two entry points are separate rather than one with a default. +fn run_pipeline_census_with_pytypes<'a>( + label: &str, + portal: CallPath, + pytypes: &'a [(&'a str, i64)], +) { + let Some(path) = cel_llbc_path() else { + skip_note(); + return; + }; + // SAFETY: serialized test binary (`--test-threads=1`); set before the + // pipeline reads it and before any worker spawns. + unsafe { std::env::set_var("PYRE_MIR_FRONTEND_LLBC", &path) }; + // Supersedes a bare "LLBC in use: " line: the path alone cannot + // distinguish this corpus from one rewritten under the same name. + let hash_before = assert_llbc_is_cel(&path, "before the pipeline"); + + let config = AnalyzeConfig { + pipeline: PipelineConfig { + transform: Default::default(), + jit_drivers: vec![JitDriverSpec { + portal, + greens: Vec::new(), + reds: Vec::new(), + green_kinds: Vec::new(), + red_kinds: Vec::new(), + autoreds: false, + virtualizables: Vec::new(), + red_types: Vec::new(), + }], + register_trait_families: Vec::new(), + }, + }; + + // A panic here is expected output, not a bug to be silenced. The pipeline + // is designed to fail loud on a shape it cannot digest, and it prints its + // census histograms before reaching that point — so catching the unwind + // and reporting the message is what makes this a measurement at all. + // Anyone "fixing" this into a quiet fallback destroys the exact signal the + // harness exists to collect. + let outcome = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { + majit_translate::analyze_multiple_pipeline_with_modules( + &[], + &config, + None, + &|_, _| None, + &[], + HostStaticAddrs { + pytypes, + ..Default::default() + }, + ) + })); + + // Re-checked AFTER the run, not only before: the value the pipeline + // actually observed is the one live during it, and a parallel test could + // only have replaced it in that window. + // + // ⭐ Comparing the two HASHES closes a hazard the variable check cannot see + // at all. `PYRE_MIR_FRONTEND_LLBC` can name the same path start to finish + // while a concurrent extraction rewrites the bytes underneath it — a real + // near-miss on this tree, where a re-extraction landed ~110s after a census + // had read the artefact. Nothing recorded the ordering, so it was + // reconstructible only from a log mtime that happened to sit near the + // rewrite. This makes it decidable instead. + let hash_after = assert_llbc_is_cel(&path, "after the pipeline"); + assert_eq!( + hash_before, hash_after, + "the LLBC was rewritten while this census was reading it — the cells above describe \ + no single corpus" + ); + + match outcome { + Ok(result) => { + eprintln!("=== cel pipeline census [{label}]: completed ==="); + eprintln!("jitcodes emitted: {}", result.jitcodes.len()); + let names: BTreeSet = result + .jitcodes_by_path + .keys() + .map(|k| k.canonical_key()) + .collect(); + eprintln!("jitcode paths ({}): {names:#?}", names.len()); + let mut insns: Vec<&String> = result.insns.keys().collect(); + insns.sort_unstable(); + eprintln!("insn vocabulary ({}): {insns:?}", insns.len()); + section1_cells(label, &result); + } + Err(err) => { + let msg = err + .downcast_ref::() + .map(String::as_str) + .or_else(|| err.downcast_ref::<&str>().copied()) + .unwrap_or(""); + eprintln!("=== cel pipeline census [{label}]: panicked ==="); + eprintln!("panic: {msg}"); + } + } +} + +/// Census the typed bytecode VM portal `clean_interp_seeded_f`. +#[test] +#[cfg_attr( + debug_assertions, + ignore = "release-only: runs full LLBC translation; use `cargo test --release --test test_cel_census`" +)] +fn cel_census_pipeline_typed_vm() { + run_pipeline_census( + "clean_interp_seeded_f", + CallPath::from_segments(["majit", "bytecode", "float_bank", "clean_interp_seeded_f"]), + ); +} + +/// Census the float-bank main loop `run_mainloop_f`. +#[test] +#[cfg_attr( + debug_assertions, + ignore = "release-only: runs full LLBC translation; use `cargo test --release --test test_cel_census`" +)] +fn cel_census_pipeline_mainloop() { + run_pipeline_census( + "run_mainloop_f", + CallPath::from_segments(["majit", "bytecode", "float_bank", "run_mainloop_f"]), + ); +} + +/// `cel::vm::eval`, the bytecode VM entry. A free function, which matters: +/// `register_configured_jitdrivers` (`lib.rs:1942`) asserts the portal +/// resolves to an exact graph in `call_control.function_graphs()`, and the +/// walker's own entry (`cel::objects::::resolve_value`) is an inherent +/// method rather than a free function. +#[test] +#[cfg_attr( + debug_assertions, + ignore = "release-only: runs full LLBC translation; use `cargo test --release --test test_cel_census`" +)] +fn cel_census_pipeline_vm_eval() { + run_pipeline_census("vm::eval", CallPath::from_segments(["vm", "eval"])); +} + +/// Census the AST walker. Associated functions must include their owning type +/// because only free functions receive widened alias paths. +#[test] +#[cfg_attr( + debug_assertions, + ignore = "release-only: runs full LLBC translation; use `cargo test --release --test test_cel_census`" +)] +fn cel_census_pipeline_walker() { + run_pipeline_census( + "objects::Value::resolve_value", + CallPath::from_segments(["objects", "Value", "resolve_value"]), + ); +} + +/// Census the class-family arithmetic chain `runtime::binop::cel_add`. +/// +/// This portal exists to read one cell: `new / new_with_vtable`. The class +/// family in `cel::runtime` is built on the premise that its constructors fuse +/// — that `fuse_boxing_alloc` recognises the `lltype::malloc_typed` body and +/// mints a `NewWithVtable` the optimizer can delete — and all three of the +/// conditions that premise rests on fail with a bare `continue`, so nothing +/// reports a constructor that did not fuse. A portal seeded here puts the +/// constructors in a closure where the cell is countable. +/// +/// `cel_add` reaches `new_int`, `new_uint`, `new_double`, `new_duration` and +/// `new_timestamp` through its arms; `new_bool` is under the ordering chain, +/// which is why the next probe exists rather than this one standing alone. +#[test] +#[cfg_attr( + debug_assertions, + ignore = "release-only: runs full LLBC translation; use `cargo test --release --test test_cel_census`" +)] +fn cel_census_pipeline_runtime_add() { + run_pipeline_census_with_pytypes( + "runtime::binop::cel_add", + CallPath::from_segments(["runtime", "binop", "cel_add"]), + CEL_CLASS_ADDRS, + ); +} + +/// Census the class-family ordering chain `runtime::binop::cel_less`. +/// +/// The `new_bool` half of the fuse question, plus the only chain whose arms +/// return an `i64` code rather than a value. See +/// [`cel_census_pipeline_runtime_add`]. +#[test] +#[cfg_attr( + debug_assertions, + ignore = "release-only: runs full LLBC translation; use `cargo test --release --test test_cel_census`" +)] +fn cel_census_pipeline_runtime_less() { + run_pipeline_census_with_pytypes( + "runtime::binop::cel_less", + CallPath::from_segments(["runtime", "binop", "cel_less"]), + CEL_CLASS_ADDRS, + ); +} + +/// Census the optional constructors through `optional.ofNonZeroValue`. +/// +/// The family's first MANAGED payload, and so the case least entitled to be +/// assumed from the five scalar constructors [`cel_census_pipeline_runtime_add`] +/// covers: `new_optional` stores a `CelRef` *argument* into the allocation where +/// every scalar leaf stores an `i64`. +/// +/// This portal rather than `cel_optional_of` because its closure holds BOTH +/// constructors — the zero arm allocates through `new_optional_none`, the other +/// through `new_optional` — so one run says whether each fuses. They are +/// separate functions on purpose (`new_optional_none` spells its null at the +/// allocation site), which is exactly why one fusing is not evidence about the +/// other. +#[test] +#[cfg_attr( + debug_assertions, + ignore = "release-only: runs full LLBC translation; use `cargo test --release --test test_cel_census`" +)] +fn cel_census_pipeline_runtime_optional() { + run_pipeline_census_with_pytypes( + "runtime::optional::cel_optional_of_non_zero_value", + CallPath::from_segments(["runtime", "optional", "cel_optional_of_non_zero_value"]), + CEL_CLASS_ADDRS, + ); +} + +/// Census the variable-length leaf `new_list`. +/// +/// `W_ListObject` is the leaf whose fuse result the encoding argued in +/// `runtime::object_array`'s module doc rests on: the payload is a separately +/// allocated block precisely so the leaf stays a fixed-size struct the fuse can +/// match. If it does not fuse, the block split bought nothing. +/// +/// Its payload pointer is NOT a managed edge — the block is allocated outside +/// the traced heap, and `runtime::registration` registers no offset for it. The +/// family's only managed edge is `W_OptionalObject::w_value`. +/// +/// The seed is the constructor itself rather than a caller, unlike +/// [`cel_census_pipeline_runtime_add`] and [`cel_census_pipeline_runtime_optional`], +/// because the three variable-length constructors have no production call site +/// yet — every use in the crate is under `#[cfg(test)]`, which the extractor +/// does not carry. There is no `cel_*` chain that reaches them, so a chain +/// portal would report a closure that never allocates one. `fuse_boxing_alloc` +/// runs per graph over the constructor's own body, so the constructor's graph +/// being in the closure is what makes the cell countable; a caller is not +/// needed for that, only for the wider closure the other probes also measure. +/// +/// Two cells answer the question. `new / newwithvtable` counts the leaf +/// allocations that fused. The block allocation cannot fuse at all — +/// `new_items_block` is size-parameterised, lives outside `runtime::lltype` and +/// is not spelled `malloc_typed`, so the matcher never even inspects it — so it +/// survives as a call, and `residual_call* : inline_call*` is which KIND of call +/// it survived as. Those are different outcomes: a residual call stops the +/// closure at the block allocator, an inline call carries it in. +#[test] +#[cfg_attr( + debug_assertions, + ignore = "release-only: runs full LLBC translation; use `cargo test --release --test test_cel_census`" +)] +fn cel_census_pipeline_runtime_new_list() { + run_pipeline_census_with_pytypes( + "runtime::object::new_list", + CallPath::from_segments(["runtime", "object", "new_list"]), + CEL_CLASS_ADDRS, + ); +} + +/// Census the variable-length leaf `new_bytes`. +/// +/// Its own seed rather than a widening of [`cel_census_pipeline_runtime_new_list`]: +/// the two closures are disjoint below the leaf — `new_list` reaches +/// `new_items_block`, `new_bytes` reaches `new_bytes_block` — so neither run says +/// anything about the other's block call, and the leaves are separate structs +/// with separate field layouts the fuse resolves independently. +#[test] +#[cfg_attr( + debug_assertions, + ignore = "release-only: runs full LLBC translation; use `cargo test --release --test test_cel_census`" +)] +fn cel_census_pipeline_runtime_new_bytes() { + run_pipeline_census_with_pytypes( + "runtime::object::new_bytes", + CallPath::from_segments(["runtime", "object", "new_bytes"]), + CEL_CLASS_ADDRS, + ); +} + +/// Census the variable-length leaf `new_string`. +/// +/// `new_string` shares `new_bytes_block` with `new_bytes` but not its leaf: +/// `W_StringObject` is a distinct struct with distinct field names, and the fuse +/// resolves the payload stores by field name off that layout. See +/// [`cel_census_pipeline_runtime_new_bytes`] for why each leaf gets its own seed. +#[test] +#[cfg_attr( + debug_assertions, + ignore = "release-only: runs full LLBC translation; use `cargo test --release --test test_cel_census`" +)] +fn cel_census_pipeline_runtime_new_string() { + run_pipeline_census_with_pytypes( + "runtime::object::new_string", + CallPath::from_segments(["runtime", "object", "new_string"]), + CEL_CLASS_ADDRS, + ); +} + +/// Census the items block's own constructor, seeded AT it. +/// +/// [`cel_census_pipeline_runtime_new_list`] cannot answer anything about this +/// function's body. `new_items_block` copies its elements in a `while` loop, so +/// its graph carries a backedge, and `JitPolicy::look_inside_graph` rejects any +/// loopy graph — `CallControl::find_all_graphs_bfs` therefore never adds it to +/// `candidate_graphs`, `CallControl::graphs_from` answers `None`, and +/// `guess_call_kind` classifies the call `Residual`. Seeded from `new_list` it +/// is a `residual_call_r_r` and nothing inside it is ever dumped, so that census +/// is silent about the element store rather than negative about it. +/// +/// A portal seed bypasses exactly that gate: `find_all_graphs_bfs` inserts the +/// jitdrivers' own portal graphs into `candidate_graphs` directly, with no +/// `look_inside_graph` call, because a portal is the thing being compiled rather +/// than a callee being judged. So this seed makes the loopy body dumpable +/// WITHOUT relaxing the policy, changing cel, or touching the front end. +/// +/// **Read the jitcode count first.** A portal that produced no jitcode and an +/// element store that lowered to nothing are the same empty histogram, and only +/// the count separates them. `register_configured_jitdrivers` asserts the portal +/// path resolves, so a typo fails loudly rather than quietly measuring nothing — +/// but a resolved portal that still emits zero jitcodes would not, and that is +/// the reading this comment exists to prevent. +/// +/// `CEL_CLASS_ADDRS` is passed although this closure allocates no class-headed +/// object: the block has no vtable, so nothing here should fuse. Supplying the +/// table anyway keeps a `new_with_vtable` of `0` meaning "nothing fused" rather +/// than "no address was available to fuse with". +/// +/// # The reading of record, and what it can be attributed to +/// +/// First run: 3 jitcodes (`new_items_block`, `items_block_items_base`, +/// `alloc_block`), vocabulary carrying `setarrayitem_gc_r`, histogram with one +/// `arraywrite`, one `arrayread` and two `arraylen` — the loop body. The +/// element store lowers to a reference array store. +/// +/// That reading was taken against a tree where `majit-translate` was **dirty**, +/// and the split matters more than the fact: +/// +/// * Reconstructable — `codewriter/policy.rs`, `model.rs`, `lib.rs`, `parse.rs`, +/// `codewriter/call.rs`, `codewriter/codewriter.rs` and +/// `front/result_exc.rs` were in the state that became the decline-instrument +/// commit, so their measured content is recoverable from it. +/// * **Not** reconstructable — `decline.rs` and `translator/rtyper/cutover.rs` +/// were edited again after the rlib was built, so whatever they held at build +/// time is gone. `cutover::is_known_unported` participates in the +/// residual-versus-candidate decision, so this is not a nil concern. +/// * Pinned regardless — `front/mir.rs`, which holds +/// `is_list_items_elem_ptr_add_parts` and therefore actually decides the +/// store, was clean; and the input was `cel.ullbc` extracted from a clean +/// cel-jit tree (`dirty_status=ok`). +/// +/// So the result is strong enough to act on and **not yet citable as a grade**: +/// a confirming re-run against a clean tree is outstanding. Record the same +/// split for any future reading rather than the bare cells — a number whose +/// tree nobody can reconstruct expires the moment someone asks which code +/// produced it. +#[test] +#[cfg_attr( + debug_assertions, + ignore = "release-only: runs full LLBC translation; use `cargo test --release --test test_cel_census`" +)] +fn cel_census_pipeline_runtime_new_items_block() { + run_pipeline_census_with_pytypes( + "runtime::object_array::new_items_block", + CallPath::from_segments(["runtime", "object_array", "new_items_block"]), + CEL_CLASS_ADDRS, + ); +} + +/// The class statics of `cel::runtime::object`, at placeholder addresses. +/// +/// Translation-time values only: the census lowers and never executes, and +/// these reach nothing but `NewWithVtable.vtable`. They are distinct so that a +/// fused site names which class it captured rather than one shared constant +/// standing in for twelve. +/// +/// The table has to name every class the seeded closure can allocate: a class +/// missing here has no address for `resolve_vtable_addr` to resolve, and the +/// fuse declines with a bare `continue` — a zero that is about this table +/// rather than about the constructor. +/// +/// The last three are the variable-length leaves. They are appended at the next +/// free addresses rather than inserted in declaration order, because these +/// values are placeholders whose only requirement is distinctness — renumbering +/// the nine above would change every existing row for no reading. +const CEL_CLASS_ADDRS: &[(&str, i64)] = &[ + ("CEL_INT_CLASS", 0x0001_0000), + ("CEL_UINT_CLASS", 0x0001_0100), + ("CEL_DOUBLE_CLASS", 0x0001_0200), + ("CEL_BOOL_CLASS", 0x0001_0300), + ("CEL_NULL_CLASS", 0x0001_0400), + ("CEL_DURATION_CLASS", 0x0001_0500), + ("CEL_TIMESTAMP_CLASS", 0x0001_0600), + ("CEL_TYPE_CLASS", 0x0001_0700), + ("CEL_OPTIONAL_CLASS", 0x0001_0800), + ("CEL_BYTES_CLASS", 0x0001_0900), + ("CEL_STRING_CLASS", 0x0001_0a00), + ("CEL_LIST_CLASS", 0x0001_0b00), +]; + +/// Per-symbol lowering-wall table for the `cel::vm` evaluator. +/// +/// The whole-crate census reports aggregate opacity; this table instead shows +/// which evaluator bodies lower and where each one stops. +/// +/// It is a census, not a gate. The only assertion is that the module was found +/// at all — a filter that silently matches nothing would otherwise print an +/// empty table and read as "no walls". +#[test] +#[cfg_attr( + debug_assertions, + ignore = "release-only: lowers the whole cel LLBC; use `cargo test --release --test test_cel_census`" +)] +fn cel_census_vm_walls() { + let Some(path) = cel_llbc_path() else { + skip_note(); + return; + }; + let llbc = Llbc::load(&path).expect("load cel llbc"); + + // Keep one entry per body, not per name: Charon emits a separate `FunDecl` + // per monomorphization, and a name-keyed map would overwrite siblings. + let mut rows: Vec<(String, String)> = Vec::new(); + let mut clean = 0usize; + let mut bodyless = 0usize; + let mut walled = 0usize; + let mut wall_classes: BTreeMap = BTreeMap::new(); + // Call sites INSIDE bodies that lowered. A body can lower and still be + // useless to trace if every arm leaves through an opaque call, so "clean" + // is reported with its residual/indirect counts rather than alone. + let mut dyn_call_sites = 0usize; + let mut indirect_sites = 0usize; + let mut method_sites = 0usize; + let mut unsupported_sites = 0usize; + + for fd in llbc.iter_local_fns() { + if fd.is_global_initializer.is_some() { + continue; + } + let name = fd.item_meta.name_path(); + if !name.contains("cel::vm") { + continue; + } + if fd.unstructured().is_none() { + bodyless += 1; + rows.push((name, "bodyless (no MIR in this artefact)".to_string())); + continue; + } + match lower_fun_decl(&llbc, fd) { + Ok(graph) => { + let mut dyns = 0usize; + let mut inds = 0usize; + let mut meths = 0usize; + let mut residuals = 0usize; + // Counted on its own rather than folded into the residual + // bucket: a call whose TARGET EXPRESSION did not lower is a + // different fact from a call to a callee with no local graph, + // and only the first says the front end could not read the + // call at all. + let mut unsupported = 0usize; + for block in &graph.blocks { + for op in &block.operations { + if let OpKind::Call { target, .. } = &op.kind { + match target { + CallTarget::FunctionPath { segments } => { + if segments.last().map(String::as_str) == Some("__dyn_call") { + dyns += 1; + } else { + residuals += 1; + } + } + CallTarget::Indirect { .. } => inds += 1, + CallTarget::Method { .. } => meths += 1, + CallTarget::UnsupportedExpr => unsupported += 1, + CallTarget::SyntheticTransparentCtor { .. } => {} + } + } + } + } + clean += 1; + dyn_call_sites += dyns; + indirect_sites += inds; + method_sites += meths; + unsupported_sites += unsupported; + rows.push(( + name, + format!( + "lowered blocks={:<4} dyn_call={dyns:<3} indirect={inds:<3} \ + method={meths:<3} unsupported={unsupported:<3} residual={residuals}", + graph.blocks.len(), + ), + )); + } + Err(err) => { + // The leading clause only. The tail carries block and local + // numbers, which would make every wall unique and turn the + // class histogram below into a copy of the row list. + let class = match &err { + LowerError::FunctionNotFound(_) => "FunctionNotFound".to_string(), + LowerError::Schema(_) => "Schema".to_string(), + LowerError::Unsupported(msg) => { + format!("Unsupported: {}", msg.chars().take(70).collect::()) + } + }; + walled += 1; + *wall_classes.entry(class.clone()).or_default() += 1; + rows.push((name, format!("WALL {class}"))); + } + } + } + + assert!( + !rows.is_empty(), + "no `cel::vm` bodies in {}: the name filter matched nothing, which is a \ + broken instrument rather than a clean module. Check the artefact's \ + naming (whole-crate vs --start-from) before reading any zero here.", + path.display() + ); + + println!( + "=== cel::vm lowering walls, artefact {} ===", + path.display() + ); + rows.sort(); + for (name, verdict) in &rows { + println!(" {name}\n {verdict}"); + } + rows.sort(); + let distinct = rows.iter().map(|(n, _)| n).collect::>().len(); + println!( + "--- totals over {} `cel::vm` bodies ({distinct} distinct names) ---", + rows.len() + ); + println!(" lowered : {clean}"); + println!(" walled : {walled}"); + println!(" bodyless : {bodyless}"); + println!(" call sites inside the lowered bodies:"); + println!(" __dyn_call : {dyn_call_sites}"); + println!(" Indirect (vtable): {indirect_sites}"); + println!(" Method (receiver): {method_sites}"); + println!(" UnsupportedExpr : {unsupported_sites}"); + if !wall_classes.is_empty() { + println!("--- wall classes ---"); + for (class, n) in &wall_classes { + println!(" {n:>4} {class}"); + } + } +} + +/// Seed the pipeline directly at the dispatch loop. +/// +/// `cel_census_pipeline_vm_eval` seeds at `vm::eval` and its jitcode closure +/// contains `cel_eval_loop`, `Vm::new` and `Vm::public_error` — everything +/// `cel_eval_loop` calls except `Vm::run`, and so nothing of the loop or the +/// opcode match. Meanwhile `cel_census_vm_walls` shows both bodies lower with +/// zero walls in isolation. Two causes fit that pair: the method call does not +/// carry the closure across, or the annotator failures recorded on that run +/// truncated it before it got there. +/// +/// Seeding here removes the first hop, so it separates them: a closure that +/// now contains `Vm::step` puts the fault on reachability from `vm::eval`, +/// which is what the portal reshape (Step 2a) exists to fix; a closure that +/// still does not, or a run that dies the same way, puts it on the annotator +/// and Step 2a would not have helped. +#[test] +#[cfg_attr( + debug_assertions, + ignore = "release-only: runs full LLBC translation; use `cargo test --release --test test_cel_census`" +)] +fn cel_census_pipeline_vm_run() { + run_pipeline_census( + "vm::interp::Vm::run", + CallPath::from_segments(["vm", "interp", "Vm", "run"]), + ); +} diff --git a/majit/majit-translate/tests/test_indirect_family_post_rtyper.rs b/majit/majit-translate/tests/test_indirect_family_post_rtyper.rs new file mode 100644 index 00000000000..bb4db2d693f --- /dev/null +++ b/majit/majit-translate/tests/test_indirect_family_post_rtyper.rs @@ -0,0 +1,334 @@ +//! What cel's `CallTarget::Indirect` vtable sites become **after** +//! `translator::rtyper::rpbc::lower_indirect_calls` runs. +//! +//! `test_cel_census.rs` classifies **pre**-rtyper: it counts `CallTarget::Indirect` +//! as emitted by `front::mir::lower_fun_decl`, before any pass has resolved a +//! family. `IndirectCall { graphs: None }` stops graph traversal, so this probe +//! checks whether the rtyper resolves each site to a non-empty family. +//! +//! ```sh +//! cargo test --release -p majit-translate --test test_indirect_family_post_rtyper \ +//! -- --nocapture --test-threads=1 +//! ``` +//! +//! Do not pass `--ignored`: the conditional ignore is absent in release builds. +//! +//! ## What is real and what is reconstructed +//! +//! * **Real**: `lower_fun_decl`, `lower_indirect_calls`, `CallControl`, +//! `register_trait_method`, and the `family.is_empty() -> None` fold — all the +//! production functions, called directly. +//! * **Reconstructed**: the *registration order* the production pipeline performs +//! at `lib.rs:1236-1311`, replayed here from the same `SemanticProgram` the +//! pipeline builds. That chain has exactly one writer end to end: +//! `all_impls_for_indirect` reads `trait_method_impls` (`call.rs:4678-4685`), +//! written only by `register_trait_method` when `trait_root.is_some()` +//! (`call.rs:2700-2709`), whose sole production caller is `lib.rs:1311` over +//! `canonical_trait_impls`, itself written at exactly one site — `lib.rs:1038`, +//! the `(self_ty_root: None, trait_root: Some(_))` arm, i.e. **trait default +//! bodies only**. `replay_trait_method_registration` below is that chain. +//! +//! The report includes the number of registered families and distinct traits, +//! so an empty reconstruction is visible instead of looking like an artefact +//! with no indirect sites. +//! +//! Two findings, both structural rather than cel-specific: +//! +//! 1. **A concrete `impl Trait for Type` never enters `trait_method_impls`.** +//! `lib.rs:1002-1031` routes it to `canonical_inherent_methods` on purpose — +//! the comment there says registering it through `register_trait_method` would +//! also seed `method_to_impl_types` and flip `resolve_method`'s name-based +//! lookup for every same-named method. So a trait method reached through a +//! vtable gets an EMPTY family unless something else filed a body for it, and +//! `rpbc.rs:417` folds empty to `None`, which stops the graph. +//! 2. **What does get filed under `` is not a trait default.** +//! None of `AsDebug` / `OpaqueEq` / `VariableResolver` declares a default body +//! (`cel/src/objects.rs:552-583`, `cel/src/context.rs:281-283` are all bare +//! signatures). The bodies filed are BLANKET impls — `impl AsDebug for T`, +//! `impl OpaqueEq for T`, `impl VariableResolver for +//! &T` — whose Charon self type is a type parameter, so `self_ty_root` is +//! `None` and they land in the trait-default arm. A blanket implementation can +//! therefore appear complete while concrete implementations remain absent. +//! The report lists both registered and concrete candidates so this +//! under-description is visible. + +use std::collections::{BTreeMap, BTreeSet}; +use std::path::PathBuf; + +use majit_charon_reader::Llbc; +use majit_translate::codewriter::call::CallControl; +use majit_translate::front::mir::{build_semantic_program_from_llbcs, lower_fun_decl}; +use majit_translate::translator::rtyper::rpbc::lower_indirect_calls; +use majit_translate::{CallTarget, FunctionGraph, OpKind}; + +/// `cel-jit/build/llbc/cel.ullbc`, or `CEL_CENSUS_LLBC` — the same resolution +/// `test_cel_census.rs` uses, so both probes read the same artefact by default. +fn cel_llbc_path() -> Option { + if let Ok(p) = std::env::var("CEL_CENSUS_LLBC") { + return Some(PathBuf::from(p)); + } + let path = PathBuf::from(concat!( + env!("CARGO_MANIFEST_DIR"), + "/../../cel-jit/build/llbc/cel.ullbc" + )); + path.exists().then_some(path) +} + +/// Replay `lib.rs:1038` + `lib.rs:1236-1311` — the only chain that ever writes +/// `trait_method_impls`. Returns `(families_registered, traits_covered)` so the +/// caller can report the replay's own cardinality. +fn replay_trait_method_registration( + call_control: &mut CallControl, + program: &majit_translate::SemanticProgram, + // Every replayed registration, as `(trait, method) -> [module_path::name]`, so + // the report can say WHICH body was filed under ``. + provenance: &mut BTreeMap<(String, String), Vec>, +) -> (usize, BTreeSet) { + let mut families = 0usize; + let mut traits = BTreeSet::new(); + for func in &program.functions { + // `lib.rs:1037` — the trait default-body arm is the only one that reaches + // `canonical_trait_impls`. A concrete `impl Trait for Type` is + // `(Some(owner), Some(trait_leaf))` and is registered as an *inherent* + // method instead (`lib.rs:1024-1031`), deliberately, so that it does not + // seed `method_to_impl_types`. + let (None, Some(trait_leaf)) = (&func.self_ty_root, &func.trait_root) else { + continue; + }; + // `lib.rs:1040` mints the pseudo impl type; `lib.rs:1237-1248` reads it + // back as `impl_type` (because `self_ty_root` is `None`) and passes + // `trait_root = Some(trait_leaf)`. + let impl_type = format!(""); + call_control.register_trait_method( + &func.name, + Some(trait_leaf.as_str()), + &impl_type, + func.graph.clone(), + ); + families += 1; + traits.insert(trait_leaf.clone()); + provenance + .entry((trait_leaf.clone(), func.name.clone())) + .or_default() + .push(format!("{}::{}", func.module_path, func.name)); + } + (families, traits) +} + +#[test] +#[cfg_attr( + debug_assertions, + ignore = "release-only: lowers the whole cel LLBC; use `cargo test --release --test test_indirect_family_post_rtyper`" +)] +fn cel_indirect_sites_after_lower_indirect_calls() { + let Some(path) = cel_llbc_path() else { + eprintln!( + "skipping: cel.ullbc missing — run `python3 scripts/extract-llbc.py cel` \ + in cel-jit, or set CEL_CENSUS_LLBC" + ); + return; + }; + let llbc = Llbc::load(&path).expect("load cel llbc"); + + let program = build_semantic_program_from_llbcs(std::slice::from_ref(&llbc)) + .expect("build semantic program from cel llbc"); + let mut call_control = CallControl::new(); + let mut provenance: BTreeMap<(String, String), Vec> = BTreeMap::new(); + let (registered, traits_covered) = + replay_trait_method_registration(&mut call_control, &program, &mut provenance); + + // The concrete impls of each `(trait, method)` — `lib.rs:1002-1031`'s + // `(Some(owner), Some(trait_leaf))` arm. These are registered as *inherent* + // methods and deliberately never reach `trait_method_impls`, so they are + // exactly the callees a vtable family does NOT list. Counted here so the + // report can say whether a `Some(family)` under-describes the callee set. + let mut concrete_impls: BTreeMap<(String, String), Vec> = BTreeMap::new(); + for func in &program.functions { + if let (Some(owner), Some(trait_leaf)) = (&func.self_ty_root, &func.trait_root) { + concrete_impls + .entry((trait_leaf.clone(), func.name.clone())) + .or_default() + .push(owner.clone()); + } + } + + // Lower every local body, exactly as the census does. + let mut graphs: Vec = Vec::new(); + for fd in llbc.iter_local_fns() { + if fd.is_global_initializer.is_some() { + continue; + } + if fd.unstructured().is_none() { + continue; + } + if let Ok(graph) = lower_fun_decl(&llbc, fd) { + graphs.push(graph); + } + } + + // PRE-rtyper: the census's own classification. + let mut pre: BTreeMap = BTreeMap::new(); + for graph in &graphs { + for block in &graph.blocks { + for op in &block.operations { + if let OpKind::Call { + target: + CallTarget::Indirect { + trait_root, + method_name, + }, + .. + } = &op.kind + { + *pre.entry(format!("{trait_root}::{method_name}")) + .or_default() += 1; + } + } + } + } + + // The family each pre-rtyper site will be handed, read from the same + // accessor `lower_indirect_calls` reads (`rpbc.rs:416`). + let mut family_sizes: BTreeMap)> = BTreeMap::new(); + for graph in &graphs { + for block in &graph.blocks { + for op in &block.operations { + if let OpKind::Call { + target: + CallTarget::Indirect { + trait_root, + method_name, + }, + .. + } = &op.kind + { + let family = call_control.all_impls_for_indirect(trait_root, method_name); + family_sizes + .entry(format!("{trait_root}::{method_name}")) + .or_insert_with(|| { + ( + family.len(), + family.iter().map(|p| p.segments.join("::")).collect(), + ) + }); + } + } + } + } + + // POST-rtyper: run the production pass and classify what it produced. + let mut post_none = 0usize; + let mut post_some_nonempty = 0usize; + let mut post_some_empty = 0usize; + let mut surviving_indirect_targets = 0usize; + // Pre-existing `IndirectCall` ops (the fn-pointer arm) would otherwise be + // counted as if the vtable pass had produced them, so subtract the baseline. + let mut baseline_indirect_ops = 0usize; + for graph in &graphs { + for block in &graph.blocks { + for op in &block.operations { + if matches!(op.kind, OpKind::IndirectCall { .. }) { + baseline_indirect_ops += 1; + } + } + } + } + let mut lowered = graphs.clone(); + for graph in &mut lowered { + lower_indirect_calls(graph, &call_control); + } + for graph in &lowered { + for block in &graph.blocks { + for op in &block.operations { + match &op.kind { + OpKind::IndirectCall { graphs, .. } => match graphs { + None => post_none += 1, + Some(c) if c.is_empty() => post_some_empty += 1, + Some(_) => post_some_nonempty += 1, + }, + OpKind::Call { + target: CallTarget::Indirect { .. }, + .. + } => surviving_indirect_targets += 1, + _ => {} + } + } + } + } + + let pre_total: usize = pre.values().sum(); + eprintln!("=== cel post-rtyper vtable census: {} ===", path.display()); + eprintln!( + "pid {} PYRE_FNPTR_INDIRECT={}", + std::process::id(), + std::env::var("PYRE_FNPTR_INDIRECT").unwrap_or_else(|_| "".into()), + ); + eprintln!("bodies lowered {}", graphs.len()); + eprintln!( + "replayed trait-default registrations {registered} over {} traits", + traits_covered.len() + ); + eprintln!("PRE-rtyper CallTarget::Indirect {pre_total}"); + for (name, n) in &pre { + let (size, members) = family_sizes.get(name).cloned().unwrap_or((0, Vec::new())); + let (trait_leaf, method) = name.split_once("::").unwrap_or((name.as_str(), "")); + let owners = concrete_impls + .get(&(trait_leaf.to_string(), method.to_string())) + .cloned() + .unwrap_or_default(); + let verdict = if size == 0 { + "family EMPTY -> graphs: None == the census's WALL class".to_string() + } else if owners.is_empty() { + "family non-empty -> graphs: Some(..)".to_string() + } else { + format!( + "family non-empty -> graphs: Some(..) BUT omits {} concrete impl(s)", + owners.len() + ) + }; + eprintln!(" {n:6} {name} family={size} {members:?} {verdict}"); + eprintln!( + " concrete impls NOT in the family ({}): {owners:?}", + owners.len() + ); + let filed = provenance + .get(&(trait_leaf.to_string(), method.to_string())) + .cloned() + .unwrap_or_default(); + eprintln!( + " bodies filed under `` ({}): {filed:?}", + filed.len() + ); + } + eprintln!( + "traits with a replayed `` registration ({}): {:?}", + traits_covered.len(), + traits_covered + ); + eprintln!("baseline IndirectCall ops (fn-pointer arm, pre-pass) {baseline_indirect_ops}"); + eprintln!("POST-rtyper IndirectCall ops:"); + eprintln!(" graphs=None (WALL) {post_none}"); + eprintln!(" graphs=Some(non-empty) {post_some_nonempty}"); + eprintln!(" graphs=Some(empty) {post_some_empty}"); + eprintln!(" CallTarget::Indirect surviving {surviving_indirect_targets}"); + eprintln!( + "delta attributable to the vtable pass: graphs=None {} (was {baseline_indirect_ops} total before)", + post_none as i64 - baseline_indirect_ops as i64 + ); + + // Non-vacuity, asserted rather than eyeballed. Both are properties of the + // instrument, not of the answer: they fail if the probe measured nothing. + assert!( + !graphs.is_empty(), + "instrument vacuous: no cel body lowered" + ); + assert_eq!( + surviving_indirect_targets, 0, + "lower_indirect_calls left {surviving_indirect_targets} CallTarget::Indirect behind — \ + the pass did not run over every site" + ); + assert!( + pre_total > 0, + "instrument vacuous: no CallTarget::Indirect site to classify" + ); +} diff --git a/pyre/cpython_tests/run.py b/pyre/cpython_tests/run.py index 1145ca690f0..e10417570b2 100644 --- a/pyre/cpython_tests/run.py +++ b/pyre/cpython_tests/run.py @@ -11,8 +11,8 @@ * script (default): `pyre /test_xxx.py` — runs the test file directly as `__main__` so its `if __name__ == "__main__": unittest.main()` block - fires. Needs only `unittest` plus the module's own imports, so it remains - the most robust mode today. A package, and a file carrying no such block, + fires. Needs only `unittest` plus the module's own imports, making it the + most robust default. A package, and a file carrying no such block, go through a synthesized unittest entry instead — running those directly exits 0 without testing anything. Runner metadata gives resource-heavy or dotted-identity-sensitive modules the corresponding @@ -709,20 +709,116 @@ def host_baseline_path(path: Path) -> Path: return path.with_name(f"{path.stem}.{HOST_TAG}{path.suffix}") -def expected_status(baseline: dict, overlay: dict, module: str, - backend: str) -> str | None: - """Recorded verdict for `module`, the host overlay winning over the shared - file. Within one file `dynasm` stands in for a backend with no entry.""" - for source in (overlay, baseline): +# Metadata keys stored beside per-backend verdicts in a module entry. +NON_BACKEND_ENTRY_KEYS = ("reason", "provenance") + + +def cell_sources(baseline: dict, overlay: dict) -> tuple[dict, dict]: + """The two files a cell can come from, in the order they are consulted. + + One helper rather than the pair spelled at each site: `expected_cell` and + `cell_provenance` have to agree about precedence, or a status read from the + host overlay is graded against an axis read from the shared file. + """ + return (overlay, baseline) + + +def expected_cell(baseline: dict, overlay: dict, module: str, + backend: str) -> tuple[str | None, str | None]: + """Return a status and the backend that recorded it. + + The host overlay is consulted before the shared file, and a cell it holds + wins — including when it answers by borrowing `dynasm`. An overlay row that + holds no cell for this backend is not an answer, though: `write_baseline` + mints a row carrying only the backend whose status diverged on this host, + so a row exists for every module any backend ever diverged on. Treating + that as an answer would drop the shared file's expectation for every OTHER + backend, and a module recorded PASS there would stop being one — no longer + selected by the gate, and no longer a regression when it fails. + + Missing backend cells fall back to dynasm; returning the source makes that + substitution visible to selection and reporting. + """ + for source in cell_sources(baseline, overlay): + entry = source.get("modules", {}).get(module) + if entry is None: + continue + own = entry.get(backend) + if own: + return own, backend + borrowed = entry.get("dynasm") + if borrowed: + return borrowed, "dynasm" + return None, None + + +def cell_provenance(baseline: dict, overlay: dict, module: str, + backend: str) -> dict | None: + """Return the recorded measurement axis, or None for legacy cells. + + Reads the same source order as `expected_cell`, and falls through the same + way, so the axis returned is the one under the status that lookup returned. + Reading only the shared file here would pair an overlay host's status with + the axis of a cell it overrode; stopping at a row that holds no cell for + this backend would pair a shared-file status with no axis at all. + """ + for source in cell_sources(baseline, overlay): entry = source.get("modules", {}).get(module) if entry is None: continue - status = entry.get(backend) or entry.get("dynasm") - if status is not None: - return status + prov = entry.get("provenance", {}).get(backend) + if prov is not None: + return prov return None +def run_axis(args: argparse.Namespace, binary: Path) -> dict: + """Return the configuration that gives this run's statuses meaning.""" + return { + "backend": args.backend, + # Name only, not the resolved path: the absolute path is host state and + # would rewrite every cell on a different machine without any of them + # having been re-measured. + "binary": binary.name, + "mode": args.mode, + "jit": not args.no_jit, + "stdlib_version": stdlib_version(), + } + + +def axis_drift_report(baseline: dict, overlay: dict, selected: list[str], + axis: dict) -> list[str]: + """Report axis drift for cells recorded by the requested backend. + + Borrowed cells are reported separately. Drift is informational because + legacy cells do not carry provenance. + """ + unrecorded = 0 + differing: dict[str, int] = {} + compared = 0 + for m in selected: + _status, src = expected_cell(baseline, overlay, m, axis["backend"]) + if src != axis["backend"]: + continue + prov = cell_provenance(baseline, overlay, m, src) + if prov is None: + unrecorded += 1 + continue + compared += 1 + for key, value in axis.items(): + if key == "backend": + continue + if prov.get(key) != value: + differing[key] = differing.get(key, 0) + 1 + own = compared + unrecorded + lines = [f"{compared} of {own} own-backend cells carry a recorded axis, " + f"{unrecorded} predate it"] + for key in sorted(differing): + lines.append(f" axis drift: {differing[key]} of {compared} recorded " + f"under a different {key} (this run: {axis[key]!r})") + return lines + + # ── main ───────────────────────────────────────────────────────────── def positive_int(value: str) -> int: @@ -784,6 +880,8 @@ def main() -> int: overlay_path = host_baseline_path(args.baseline) overlay = load_baseline(overlay_path) if overlay_path.exists() else {"modules": {}} modules = discover_modules(args.filter) + # Derive once for reports, baseline updates, and drift checks. + axis = run_axis(args, binary) if args.list: for m in modules: @@ -816,6 +914,8 @@ def main() -> int: skipped: list[str] = [] off_platform: list[tuple[str, str]] = [] deselected = 0 + # Selected modules whose expected status came from another backend. + borrowed_gated: list[str] = [] for m in modules: # Even full/update runs must not overwrite another platform's result. gate_reason = platform_gate(m) @@ -823,7 +923,7 @@ def main() -> int: off_platform.append((m, gate_reason)) skipped.append(m) continue - exp = expected_status(baseline, overlay, m, args.backend) + exp, exp_src = expected_cell(baseline, overlay, m, args.backend) is_skip = (exp == "SKIP") or (m in KNOWN_SKIPS) if is_skip and not args.full and not args.update_baseline: skipped.append(m) @@ -832,9 +932,12 @@ def main() -> int: deselected += 1 continue to_run.append(m) + if exp_src is not None and exp_src != args.backend: + borrowed_gated.append(m) - print(f"pyre CPython suite — backend={args.backend} mode={args.mode} " - f"jit={'off' if args.no_jit else 'on'} jobs={args.jobs}") + # Print the same derived axis that baseline updates record. + print(f"pyre CPython suite — backend={axis['backend']} mode={axis['mode']} " + f"jit={'on' if axis['jit'] else 'off'} jobs={args.jobs}") print(f"binary: {binary}") overlay_count = len(overlay.get("modules", {})) print(f"baseline: {args.baseline.name} + " @@ -844,7 +947,13 @@ def main() -> int: print(f" off-platform on {sys.platform}: {m} ({reason})") extra = f", {deselected} not gated (non-PASS)" if deselected else "" print(f"{len(to_run)} to run, {len(skipped)} skipped{extra}, " - f"timeout={args.timeout}s\n") + f"timeout={args.timeout}s") + # Expose how much of this run was gated against another backend. + print(f"{len(borrowed_gated)} of {len(to_run)} gated against another " + f"backend's recorded status") + for line in axis_drift_report(baseline, overlay, to_run, axis): + print(line) + print() results: dict[str, tuple[str, str]] = {} done = 0 @@ -912,11 +1021,16 @@ def record_result(module: str, result: tuple[str, str]) -> None: regressions: list[str] = [] improvements: list[str] = [] for m, (status, detail) in sorted(results.items()): - exp = expected_status(baseline, overlay, m, args.backend) + exp, exp_src = expected_cell(baseline, overlay, m, args.backend) + # Name the backend a verdict is measured against whenever it is not the + # one under test. Without it a regression line reads as this backend + # having gone from PASS to FAIL, when what happened may be that this + # backend was never recorded passing. + via = "" if exp_src in (None, args.backend) else f" (expected from {exp_src})" if exp == "PASS" and status != "PASS": - regressions.append(f"{m}: PASS -> {status} {detail}") + regressions.append(f"{m}: PASS -> {status}{via} {detail}") elif exp != "PASS" and status == "PASS": - improvements.append(f"{m}: {exp or 'new'} -> PASS") + improvements.append(f"{m}: {exp or 'new'} -> PASS{via}") if improvements: print(f"\n── improvements ({len(improvements)}) ──") @@ -925,10 +1039,7 @@ def record_result(module: str, result: tuple[str, str]) -> None: if args.report: report = { - "backend": args.backend, - "mode": args.mode, - "jit": not args.no_jit, - "stdlib_version": stdlib_version(), + **axis, "counts": counts, "modules": {m: {"status": s, "detail": d} for m, (s, d) in sorted(results.items())}, @@ -939,11 +1050,18 @@ def record_result(module: str, result: tuple[str, str]) -> None: print(f"\nreport written: {args.report}") if args.update_baseline: - written = write_baseline(args.baseline, baseline, overlay, results, - args.backend) + # `axis`, not `args.backend`: the cells this writes record the axis + # they were measured along, and `run_axis` is the one place that + # derives it. + written = write_baseline(args.baseline, baseline, overlay, results, axis) recorded = sum(1 for s, _ in results.values() if s == "PASS") for target in written: print(f"\nbaseline written: {target} ({recorded} PASS recorded)") + # Read the file as written, not as loaded: this bless repairs some of + # what these would otherwise name, and a report naming a cell the same + # run just corrected is wrong the moment it prints. + for line in phantom_row_report(baseline) + stale_curated_cell_report(baseline): + print(line) return 0 if regressions: @@ -964,8 +1082,41 @@ def record_result(module: str, result: tuple[str, str]) -> None: return 0 +def phantom_row_report(baseline: dict) -> list[str]: + """Lines naming baseline rows that no longer have a discoverable module. + + Rows are reported rather than deleted because the caller may be running a + filtered subset. Discovery is repeated without that filter here. + """ + rows = baseline.get("modules", {}) + phantom = sorted(set(rows) - set(discover_modules(None))) + if not phantom: + return [] + return [f"{len(phantom)} of {len(rows)} baseline rows have no discoverable " + f"module (carried unchanged, not gated):"] + [f" ? {m}" for m in phantom] + + +def stale_curated_cell_report(baseline: dict) -> list[str]: + """Lines naming cells on a curated-skip row that still record a measurement. + + `KNOWN_SKIPS` already governs selection, so these cells are inert. Preserve + their last measured status in case the module leaves the curated skip set. + """ + rows = baseline.get("modules", {}) + stale = [(m, backend, status) + for m, entry in sorted(rows.items()) if m in KNOWN_SKIPS + for backend, status in sorted(entry.items()) + if backend not in NON_BACKEND_ENTRY_KEYS and status != "SKIP"] + if not stale: + return [] + return [f"{len(stale)} cell(s) on {len({m for m, _, _ in stale})} curated-skip " + f"row(s) still record a measurement (reported, not synced — the " + f"curated table governs selection, so these are stale and inert):"] + [ + f" * {m} [{backend}] {status}" for m, backend, status in stale] + + def write_baseline(path: Path, baseline: dict, overlay: dict, results: dict, - backend: str) -> list[Path]: + axis: dict) -> list[Path]: """Record `results`, splitting them between the shared baseline and this host's overlay. Returns the files actually written. @@ -974,11 +1125,20 @@ def write_baseline(path: Path, baseline: dict, overlay: dict, results: dict, that a host writes to its own overlay only where it disagrees, and a run that comes back into agreement drops the overlay entry again -- so an overlay never outlives the divergence that created it. + + A cell's provenance travels with the status into whichever of the two files + ends up holding it, so an overlay cell is a measurement with an axis under + it rather than a bare status. """ + backend = axis["backend"] modules = baseline.setdefault("modules", {}) overlay_modules = overlay.setdefault("modules", {}) + # Keep the legacy file-level value; per-cell provenance is authoritative for + # a measurement's stdlib version. baseline["stdlib_version"] = stdlib_version() overlay_dirty = False + # The containing cell already identifies the backend. + cell_axis = {k: v for k, v in axis.items() if k != "backend"} for m, (status, _detail) in results.items(): # Defensive: never overwrite another platform's recorded result. if platform_gate(m) is not None: @@ -991,24 +1151,56 @@ def write_baseline(path: Path, baseline: dict, overlay: dict, results: dict, if m in KNOWN_SKIPS: entry = modules.setdefault(m, {}) entry[backend] = "SKIP" - entry.setdefault("reason", KNOWN_SKIPS[m]) + # Curated reasons are row-wide and refreshed from their source of + # truth. Warn before replacing a different recorded rationale. + recorded = entry.get("reason") + if recorded is not None and recorded != KNOWN_SKIPS[m]: + # Leading blank line: the improvements block prints immediately + # above, and an unseparated ` ! ` line reads as one of its rows. + print(f"\n ! {m}: recorded `reason` replaced by the curated text") + print(f" was : {recorded}") + print(f" curated: {KNOWN_SKIPS[m]}") + entry["reason"] = KNOWN_SKIPS[m] + # A curated skip is a decision, not a measurement, so it has no + # measurement provenance. + prov = entry.get("provenance") + if prov is not None: + prov.pop(backend, None) + if not prov: + del entry["provenance"] + # The skip is a decision about the module rather than about this + # host, so it is shared and any host entry for it is dropped. overlay_dirty |= overlay_modules.pop(m, None) is not None continue shared = modules.get(m, {}).get(backend) if shared is None: - modules.setdefault(m, {})[backend] = status + entry = modules.setdefault(m, {}) + entry[backend] = status + entry.setdefault("provenance", {})[backend] = dict(cell_axis) continue if status == shared: host_entry = overlay_modules.get(m) if host_entry is not None and host_entry.pop(backend, None) is not None: overlay_dirty = True - # `reason` is prose about the divergence, so it cannot keep the - # entry alive once no backend still records one. - if not any(k != "reason" for k in host_entry): + # The axis goes with the cell it described. Left behind it would + # be provenance for a status this file no longer records. + host_prov = host_entry.get("provenance") + if host_prov is not None: + host_prov.pop(backend, None) + if not host_prov: + del host_entry["provenance"] + # Neither `reason` nor a leftover `provenance` is a verdict, so + # neither can keep the entry alive once no backend still records + # one. Spelled through the constant rather than as a literal: + # this is the second site walking an entry generically, and two + # spellings of "which keys are not backends" is how they drift. + if not any(k not in NON_BACKEND_ENTRY_KEYS for k in host_entry): del overlay_modules[m] continue - if overlay_modules.setdefault(m, {}).get(backend) != status: - overlay_modules[m][backend] = status + host_entry = overlay_modules.setdefault(m, {}) + if host_entry.get(backend) != status: + host_entry[backend] = status + host_entry.setdefault("provenance", {})[backend] = dict(cell_axis) overlay_dirty = True written = [path] diff --git a/pyre/gate-triage.md b/pyre/gate-triage.md index f075dae46b3..3d58d409219 100644 --- a/pyre/gate-triage.md +++ b/pyre/gate-triage.md @@ -74,7 +74,8 @@ Kept as-is; listed for completeness. `PYRE_RTYPER_VERBOSE`, `PYRE_JTRANSFORM_SHADOW`, `PYRE_DIAG124C`, `_51C`, `_GIN`, `_INLINE_RECOG`, `PYRE_WASM_DUMP_ALL_TRACES`, `_DUMP_BAD_TRACE`, `_EXEC_TRACE`, `_JIT_STATS`, `PYRE_INTERP_RETURN_LOG`, `PYRE_NBODY_DEBUG`, - `PYRE_DEBUG_CALL`, `PYRE_DEBUG_CLASS`, `PYRE_DESCR_DEMAND`. + `PYRE_DEBUG_CALL`, `PYRE_DEBUG_CLASS`, `PYRE_ALLOCSITES`, + `PYRE_CELL_CENSUS`, `PYRE_DEOPT_PROBE`, `PYRE_DESCR_DEMAND`. `PYRE_DESCR_DEMAND` records the distinct dense descriptor indices a run actually resolves, so the per-index pool loader can be measured against the pool size; the resolve path reads it through a `OnceLock` and pays nothing @@ -209,10 +210,11 @@ the folds it selects, not before them. `PYRE_FBW_SPEC_CENSUS` in §6c is its read-only half: the per-fold consulted/fired tallies. -### §6c — Default-OFF diagnostics, censuses and probes (71): keep, cost nothing +### §6c — Default-OFF diagnostics, censuses and probes (70): keep, cost nothing -Each is inert unless set, so none is a removal target by this file's -already-ON criterion. They are listed so they cannot be missed again. +Deleting one of these environment reads does not change behavior when the +variable is unset. They remain listed so diagnostics are not mistaken for dead +configuration. `PYRE_ALLOCSITES`, `PYRE_BH_NULL_ARG`, `PYRE_BRIDGE_LATCH_AUDIT`, `PYRE_CALLEE_RCA`, `PYRE_CATCH_LIVE_CENSUS`, @@ -229,7 +231,7 @@ already-ON criterion. They are listed so they cannot be missed again. `PYRE_FORITER_INFLIGHT_CENSUS`, `PYRE_FOR_ITER_GATE_DIAG`, `PYRE_GC_DIAG`, `PYRE_GC_FREELIST_DIAG`, `PYRE_GEN_ENTRY_DIAG`, `PYRE_JD1_DEBUG`, `PYRE_JD1_DUMP`, -`PYRE_LB_SITE`, `PYRE_LLBC_SKIP_FINGERPRINT_CHECK`, `PYRE_LLBC_STRICT`, +`PYRE_LB_SITE`, `PYRE_LLBC_SKIP_FINGERPRINT_CHECK`, `PYRE_M73_BACKXLAT_TWIN_AUDIT`, `PYRE_M73_EMPTYTWIN_CENSUS`, `PYRE_M73_LASTINSTR_AUDIT`, `PYRE_M73_MIDBODY_CARRY_AUDIT`, `PYRE_MAJIT_STATS_ANCESTOR`, `PYRE_MAJIT_STATS_ROOT_ONLY`, `PYRE_MC_DIAG`, @@ -283,6 +285,13 @@ gate restores the whole-environment copy, so the size of that effect stays measurable on one binary. It goes when the allowlist stops being the thing under measurement. +### §6d — Default-ON safety controls + +| gate | behavior | retirement condition | +|---|---|---| +| `PYRE_LLBC_STRICT` | treats stale frozen LLBC artifacts as an error; setting it to `0` demotes the error to a warning | retain while the build consumes frozen LLBC artifacts | + + ## §7 — General MAJIT gates These translator and metainterpreter controls are inert unless explicitly set,