feat(experiment-runner): add deterministic trial engine with immutable evaluator and journal replay

AGENTS.md

@@ -18,6 +18,7 @@ Start with `docs/README.md`. Read `docs/IMPLEMENTATION_PLAN.md`, `docs/ARCHITECT
 - Keep provider lifecycle behavior in `crates/provider-sdk`.
 - Keep the capability registry, policy, broker lifecycle, and experiment journal in `crates/control-plane`.
 - Keep the independent crash and lease recovery path in `apps/watchdog`; it reads the journal owned by `crates/control-plane` and writes only its own restore-outcome records, never the schema.
+- Keep the measurement model, immutable evaluator, and replayable trial records in `apps/experiment-runner`; the evaluator stays a pure function of recorded samples and fixed bounds.
 - Put provider-specific code in one `sidecars/<provider>` package; sidecars may not import each other.
 - Put non-Rust compatibility processes under `bridges/` and isolate them behind the sidecar protocol.
 - Do not vendor third-party binaries without confirmed redistribution rights.

README.md

@@ -88,7 +88,8 @@ The repository currently includes:
 - A working stdio MCP gateway that serves the mock path end to end
 - A CLI `doctor` command that reports gateway and journal status
 - An independent watchdog that restores prior state from the journal after a crash or lease expiry, on the Linux-safe mock path
-- Scaffolds for the privileged broker and experiment runner
+- A deterministic experiment runner that gates measured trials through an immutable evaluator and replays them from the journal alone
+- Scaffolds for the privileged broker
 - OSS governance, security policy, issue templates, and CI
 - An organized [documentation index](docs/README.md) with architecture, plans, threat model, and provider guides
 
@@ -98,6 +99,7 @@ Try the read-only alpha:
 cargo test --workspace
 cargo run -p fpsmaxxing-cli -- doctor
 cargo run -p fpsmaxxing-mock-provider
+cargo run -p fpsmaxxing-experiment-runner
 printf '%s\n' \
   '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}' \
   '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"fpsmaxxing.run_mock_lifecycle","arguments":{"value":42,"lease_seconds":30}}}' \
@@ -110,6 +112,9 @@ Override the journal location with `--journal <path>` or the `FPSMAXXING_JOURNAL
 Run the watchdog against the same journal to reclaim leaked experiments: `cargo run -p fpsmaxxing-watchdog -- --once` performs a single expired-lease pass and `--recover-all` rolls back every unclosed experiment after a crash.
 It accepts the same `--journal <path>` and `FPSMAXXING_JOURNAL_PATH` overrides, plus `--interval <seconds>` for its steady-state poll loop.
 
+The experiment runner measures a baseline and a candidate against a deterministic stand-in for live telemetry, gates the candidate's lifecycle on the immutable evaluator's verdict, journals the trial with its spec, samples, and verdict, then replays it from the journal alone and checks the re-evaluated verdict against the recorded one.
+It is a demonstration binary rather than an MCP tool, takes no arguments, and journals to an in-memory SQLite database, so it leaves nothing on disk and exits non-zero if a replay diverges from the journal, falls outside the policy gate, or the broker refuses a promoted lifecycle.
+
 ## Architecture
 
 ```text
@@ -133,7 +138,7 @@ Start with the [documentation index](docs/README.md). The core references are th
 
 ### Can Claude optimize my PC for higher FPS?
 
-That is the intended workflow. A Claude or Codex agent should be able to inspect available capabilities, propose a bounded change, run a controlled game or benchmark workload, and keep the change only when frame time, latency, thermals, and correctness remain within policy. The alpha already runs capability discovery and the bounded snapshot-to-rollback lifecycle over MCP against a mock provider; the measurement-driven keep-or-rollback decision and real hardware providers are not implemented yet.
+That is the intended workflow. A Claude or Codex agent should be able to inspect available capabilities, propose a bounded change, run a controlled game or benchmark workload, and keep the change only when frame time, latency, thermals, and correctness remain within policy. The alpha already runs capability discovery and the bounded snapshot-to-rollback lifecycle over MCP against a mock provider, and it promotes or rejects one measured experiment through an immutable evaluator that decides from recorded samples and fixed bounds alone. Real hardware providers, live frame-time measurement, and a promotion that survives its lease are not implemented yet: the alpha measures a deterministic stand-in for telemetry, and the verdict gates whether a candidate is applied at all rather than whether it persists.
 
 ### Can an AI safely overclock a GPU?
 

apps/experiment-runner/Cargo.toml

@@ -7,5 +7,18 @@ repository.workspace = true
 rust-version.workspace = true
 publish = false
 
+[dependencies]
+fpsmaxxing-contracts.workspace = true
+fpsmaxxing-control-plane = { path = "../../crates/control-plane" }
+fpsmaxxing-mock-provider = { path = "../../sidecars/mock-provider" }
+serde.workspace = true
+serde_json.workspace = true
+thiserror.workspace = true
+
+[dev-dependencies]
+fpsmaxxing-provider-sdk.workspace = true
+rusqlite.workspace = true
+tempfile = "3"
+
 [lints]
 workspace = true

apps/experiment-runner/src/evaluator.rs

@@ -0,0 +1,260 @@
+//! The immutable experiment evaluator.
+//!
+//! [`evaluate`] is a pure function: given recorded baseline and candidate
+//! samples plus fixed [`DecisionBounds`] it returns a deterministic
+//! [`Verdict`]. It performs no I/O, reads no clock, holds no state, and never
+//! mutates its inputs, so a trial can be re-evaluated from the journal alone
+//! and always yield an identical verdict.
+//!
+//! # Decision rule (fixed and ordered)
+//!
+//! 1. **Minimum samples.** Both the baseline and candidate sets must hold at
+//!    least `bounds.min_samples` samples, otherwise the verdict is a
+//!    [`Reject`](Decision::Reject) with
+//!    [`InsufficientSamples`](VerdictReason::InsufficientSamples).
+//! 2. **Constraint bounds.** The candidate's worst temperature, then worst
+//!    power draw, then total errors must each stay within the inclusive
+//!    ceilings. The first ceiling exceeded rejects, in that order.
+//! 3. **Improvement threshold.** The candidate mean FPS must beat the baseline
+//!    mean FPS by at least `bounds.min_fps_improvement`. If it does, the
+//!    verdict is a [`Promote`](Decision::Promote); otherwise a
+//!    [`Reject`](Decision::Reject) with
+//!    [`InsufficientImprovement`](VerdictReason::InsufficientImprovement).
+
+use fpsmaxxing_contracts::{
+    Decision, DecisionBounds, MetricSample, MetricSummary, Verdict, VerdictReason,
+};
+
+/// Applies the immutable decision rule to recorded samples.
+///
+/// The returned [`Verdict`] carries the baseline and candidate aggregates the
+/// decision used, so a journaled trial is self-describing and re-evaluation can
+/// be checked against it.
+#[must_use]
+pub fn evaluate(
+    baseline: &[MetricSample],
+    candidate: &[MetricSample],
+    bounds: &DecisionBounds,
+) -> Verdict {
+    let baseline_summary = summarize(baseline);
+    let candidate_summary = summarize(candidate);
+    let fps_improvement = candidate_summary.mean_fps - baseline_summary.mean_fps;
+    let min_samples = u64::from(bounds.min_samples.get());
+
+    let reason =
+        if baseline_summary.samples < min_samples || candidate_summary.samples < min_samples {
+            VerdictReason::InsufficientSamples
+        } else if candidate_summary.max_temperature_c > bounds.max_temperature_c {
+            VerdictReason::TemperatureExceeded
+        } else if candidate_summary.max_power_w > bounds.max_power_w {
+            VerdictReason::PowerExceeded
+        } else if candidate_summary.total_errors > bounds.max_errors {
+            VerdictReason::ErrorsExceeded
+        } else if fps_improvement >= bounds.min_fps_improvement {
+            VerdictReason::Promoted
+        } else {
+            VerdictReason::InsufficientImprovement
+        };
+
+    let decision = if matches!(reason, VerdictReason::Promoted) {
+        Decision::Promote
+    } else {
+        Decision::Reject
+    };
+
+    Verdict {
+        decision,
+        reason,
+        fps_improvement,
+        baseline: baseline_summary,
+        candidate: candidate_summary,
+    }
+}
+
+/// Aggregates one measurement set deterministically.
+///
+/// The FPS mean is the arithmetic mean, while the temperature and power fields
+/// report the worst (highest) observed value and errors are summed. An empty
+/// set reports zero throughout, having observed nothing. Counting the divisor
+/// as an `f64` avoids a lossy integer-to-float cast without changing the result
+/// for realistic sample counts. Samples may come from a journaled record this
+/// build did not write, so the aggregates state only what the samples hold: the
+/// error total saturates rather than overflowing, and the maxima fold from
+/// negative infinity so an all-negative set reports its true worst observation
+/// instead of a value the journal never recorded.
+fn summarize(samples: &[MetricSample]) -> MetricSummary {
+    if samples.is_empty() {
+        return MetricSummary {
+            samples: 0,
+            mean_fps: 0.0,
+            max_temperature_c: 0.0,
+            max_power_w: 0.0,
+            total_errors: 0,
+        };
+    }
+    let mut sum_fps = 0.0_f64;
+    let mut divisor = 0.0_f64;
+    let mut max_temperature_c = f64::NEG_INFINITY;
+    let mut max_power_w = f64::NEG_INFINITY;
+    let mut total_errors = 0_u64;
+    for sample in samples {
+        sum_fps += sample.fps;
+        divisor += 1.0;
+        max_temperature_c = max_temperature_c.max(sample.temperature_c);
+        max_power_w = max_power_w.max(sample.power_w);
+        total_errors = total_errors.saturating_add(sample.errors);
+    }
+    MetricSummary {
+        samples: samples.len() as u64,
+        mean_fps: sum_fps / divisor,
+        max_temperature_c,
+        max_power_w,
+        total_errors,
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use std::num::NonZeroU32;
+
+    use super::{Decision, DecisionBounds, MetricSample, VerdictReason, evaluate};
+
+    fn bounds() -> DecisionBounds {
+        DecisionBounds {
+            min_samples: NonZeroU32::new(3).expect("min samples is non-zero"),
+            min_fps_improvement: 5.0,
+            max_temperature_c: 80.0,
+            max_power_w: 200.0,
+            max_errors: 0,
+        }
+    }
+
+    fn samples(
+        fps: f64,
+        temperature_c: f64,
+        power_w: f64,
+        errors: u64,
+        count: usize,
+    ) -> Vec<MetricSample> {
+        (0..count)
+            .map(|_| MetricSample {
+                fps,
+                temperature_c,
+                power_w,
+                errors,
+            })
+            .collect()
+    }
+
+    #[test]
+    fn promotes_when_every_bound_is_met() {
+        let baseline = samples(100.0, 60.0, 150.0, 0, 5);
+        let candidate = samples(120.0, 70.0, 180.0, 0, 5);
+        let verdict = evaluate(&baseline, &candidate, &bounds());
+        assert_eq!(verdict.decision, Decision::Promote);
+        assert_eq!(verdict.reason, VerdictReason::Promoted);
+        assert!((verdict.fps_improvement - 20.0).abs() < f64::EPSILON);
+        assert_eq!(verdict.baseline.samples, 5);
+        assert!((verdict.candidate.mean_fps - 120.0).abs() < f64::EPSILON);
+    }
+
+    #[test]
+    fn rejects_when_a_set_is_below_minimum_samples() {
+        let baseline = samples(100.0, 60.0, 150.0, 0, 5);
+        let candidate = samples(120.0, 70.0, 180.0, 0, 2);
+        let verdict = evaluate(&baseline, &candidate, &bounds());
+        assert_eq!(verdict.decision, Decision::Reject);
+        assert_eq!(verdict.reason, VerdictReason::InsufficientSamples);
+    }
+
+    #[test]
+    fn temperature_ceiling_is_checked_before_improvement() {
+        let baseline = samples(100.0, 60.0, 150.0, 0, 5);
+        let candidate = samples(140.0, 80.1, 180.0, 0, 5);
+        let verdict = evaluate(&baseline, &candidate, &bounds());
+        assert_eq!(verdict.decision, Decision::Reject);
+        assert_eq!(verdict.reason, VerdictReason::TemperatureExceeded);
+    }
+
+    #[test]
+    fn power_ceiling_rejects_even_with_a_large_gain() {
+        let baseline = samples(100.0, 60.0, 150.0, 0, 5);
+        let candidate = samples(140.0, 70.0, 200.1, 0, 5);
+        let verdict = evaluate(&baseline, &candidate, &bounds());
+        assert_eq!(verdict.decision, Decision::Reject);
+        assert_eq!(verdict.reason, VerdictReason::PowerExceeded);
+    }
+
+    #[test]
+    fn errors_ceiling_rejects_a_regressing_candidate() {
+        let baseline = samples(100.0, 60.0, 150.0, 0, 5);
+        let candidate = samples(140.0, 70.0, 180.0, 1, 5);
+        let verdict = evaluate(&baseline, &candidate, &bounds());
+        assert_eq!(verdict.decision, Decision::Reject);
+        assert_eq!(verdict.reason, VerdictReason::ErrorsExceeded);
+    }
+
+    #[test]
+    fn rejects_when_improvement_is_below_threshold() {
+        let baseline = samples(100.0, 60.0, 150.0, 0, 5);
+        let candidate = samples(104.0, 70.0, 180.0, 0, 5);
+        let verdict = evaluate(&baseline, &candidate, &bounds());
+        assert_eq!(verdict.decision, Decision::Reject);
+        assert_eq!(verdict.reason, VerdictReason::InsufficientImprovement);
+        assert!((verdict.fps_improvement - 4.0).abs() < f64::EPSILON);
+    }
+
+    #[test]
+    fn improvement_exactly_at_threshold_promotes() {
+        let baseline = samples(100.0, 60.0, 150.0, 0, 3);
+        let candidate = samples(105.0, 70.0, 180.0, 0, 3);
+        let verdict = evaluate(&baseline, &candidate, &bounds());
+        assert_eq!(verdict.decision, Decision::Promote);
+        assert_eq!(verdict.reason, VerdictReason::Promoted);
+    }
+
+    #[test]
+    fn an_error_total_beyond_u64_saturates_and_rejects() {
+        // Journaled samples can come from a record this build did not write, so
+        // aggregation must not overflow on hostile counts.
+        let baseline = samples(100.0, 60.0, 150.0, 0, 3);
+        let candidate = samples(140.0, 70.0, 180.0, u64::MAX, 3);
+        let verdict = evaluate(&baseline, &candidate, &bounds());
+        assert_eq!(verdict.candidate.total_errors, u64::MAX);
+        assert_eq!(verdict.decision, Decision::Reject);
+        assert_eq!(verdict.reason, VerdictReason::ErrorsExceeded);
+    }
+
+    #[test]
+    fn maxima_report_the_worst_observation_even_when_all_are_negative() {
+        // The summary is journaled verbatim in the verdict, so it must not
+        // claim a temperature or power draw the samples never held.
+        let baseline = samples(100.0, -40.0, -10.0, 0, 3);
+        let candidate = samples(140.0, -30.0, -5.0, 0, 3);
+        let verdict = evaluate(&baseline, &candidate, &bounds());
+        assert!((verdict.baseline.max_temperature_c - -40.0).abs() < f64::EPSILON);
+        assert!((verdict.baseline.max_power_w - -10.0).abs() < f64::EPSILON);
+        assert!((verdict.candidate.max_temperature_c - -30.0).abs() < f64::EPSILON);
+        assert!((verdict.candidate.max_power_w - -5.0).abs() < f64::EPSILON);
+    }
+
+    #[test]
+    fn an_empty_set_reports_zero_throughout() {
+        let verdict = evaluate(&[], &[], &bounds());
+        assert_eq!(verdict.baseline.samples, 0);
+        assert!((verdict.baseline.mean_fps - 0.0).abs() < f64::EPSILON);
+        assert!((verdict.baseline.max_temperature_c - 0.0).abs() < f64::EPSILON);
+        assert!((verdict.baseline.max_power_w - 0.0).abs() < f64::EPSILON);
+        assert_eq!(verdict.decision, Decision::Reject);
+        assert_eq!(verdict.reason, VerdictReason::InsufficientSamples);
+    }
+
+    #[test]
+    fn evaluation_is_deterministic_across_repeated_calls() {
+        let baseline = samples(100.0, 60.0, 150.0, 0, 4);
+        let candidate = samples(118.0, 72.0, 190.0, 0, 4);
+        let first = evaluate(&baseline, &candidate, &bounds());
+        let second = evaluate(&baseline, &candidate, &bounds());
+        assert_eq!(first, second);
+    }
+}

apps/experiment-runner/src/lib.rs

@@ -0,0 +1,22 @@
+//! Deterministic experiment engine.
+//!
+//! This crate provides the pieces that turn a typed [`ExperimentSpec`] into an
+//! auditable, replayable trial:
+//!
+//! - a deterministic measurement model that stands in for a live telemetry
+//!   source on the Linux safe-alpha path,
+//! - the immutable [`evaluate`] decision rule, and
+//! - a trial runner that journals every sample and verdict so a trial can be
+//!   replayed and re-evaluated from the journal alone.
+//!
+//! [`ExperimentSpec`]: fpsmaxxing_contracts::ExperimentSpec
+
+mod evaluator;
+mod model;
+mod runner;
+
+pub use evaluator::evaluate;
+pub use runner::{
+    LifecycleFailure, LifecycleOutcome, ReplayOutcome, RunnerError, StoredTrial,
+    TRIAL_RECORD_VERSION, TrialRecord, replay_trial, run_trial,
+};