Validation date: 2026-07-24. Scope: controlled local infrastructure only. These results are not
external-project validation, cloud-deployment evidence, production scale, adoption, or time saved.
Raw evidence is retained in validation/platform/results-strict.json,
results-benchmark.json, and results-infrastructure.json.
| Requirement | Status | Executed evidence / limitation |
|---|---|---|
| Signed webhook through completed replay | passed | Real HMAC webhook, PostgreSQL tenant/event, encrypted MinIO artifact, durable queue, Docker worker, and Python replay completed in one attempt. |
| Tenant isolation and duplicate delivery/request handling | passed | Cross-tenant read returned 404; duplicate webhook returned the recorded duplicate response; duplicate replay idempotency key returned 409. |
| Artifact integrity and signatures | passed | Valid encrypted object replayed; incorrect upload digest returned 422; overwritten object failed verification on every attempt. |
| Cause-preserving failure classification | passed | A real Docker API permission failure returned exit code 1 against an artifact expecting exit code 1 and was correctly classified reproduced: false; a regression test prevents infrastructure text from matching a test failure. |
| Secret redaction | passed | A known GITHUB_TOKEN canary was absent from every archived member. |
| Retries, backoff, corruption, and DLQ | passed | Corrupted MinIO object made five real attempts and reached dead_letter in 30.442 seconds with artifact digest mismatch. |
| Lease expiry, worker crash, checkpoint recovery | passed | Worker was killed during a five-second replay; the expired lease was reclaimed and checkpoint advanced from worker_claimed to completed on attempt 2 in 5.383 seconds. |
| PostgreSQL restart recovery | passed | PostgreSQL was restarted; readiness, webhook intake, upload, queueing, and replay recovered in 1.477 seconds. |
| Prometheus observability | passed | Real control-plane and worker endpoints were scraped; the final worker replay incremented reproci_worker_success_total and execution latency. |
| OpenTelemetry export | passed | Official Collector Contrib 0.130.1 received one resource-span batch containing three spans from a real smoke replay. No hosted backend claim. |
| Retention controls | failed | MinIO lifecycle export confirmed the one-day expiration rule, but actual deletion was not observed because the 24-hour boundary was not awaited. Configuration alone is not execution evidence. |
| Strict integration mode | passed | REPROCI_VALIDATION_REQUIRED=1 go test -tags=integration ./... used real PostgreSQL and MinIO; missing dependency variables call t.Fatal, never t.Skip. |
| Controlled load at 1, 10, 25, and 50 concurrent jobs | passed | Two complete runs per level; 172/172 controlled jobs reproduced. Raw per-job records are retained. |
| Local Kubernetes | passed | Kind v0.31.0/Kubernetes 1.34.3: two control-plane replicas Ready and a numeric-non-root replay Job completed with the expected signature. |
| Terraform | passed | Official Terraform 1.15.8 container, AWS provider 5.100.0: format, init, and validate completed with zero errors and zero warnings. No plan/apply or cloud claim. |
| GitHub App installation and real deliveries | passed for owner-controlled dogfood | Least-privilege App 4386569 was installed on exactly two owner repositories; 36/36 expected signed workflow_job deliveries were accepted after final configuration. The temporary webhook is now disabled. |
| AWS plan/apply and hosted observability | externally pending | Requires AWS credentials, budget, DNS/TLS decisions, and deployment approval. |
| Two-repository owner dogfood | passed with qualification | Four public workflow runs across two owner-controlled repositories produced 12/12 controlled replays. This is not independent adoption or flaky-test validation. |
| Independent beta users | externally pending | No person was contacted and no independent repository was modified. |
Each level ran twice against the real Compose stack. Reproducibility was 100% for this single controlled artifact only.
| Concurrent jobs | Throughput jobs/s | Queue p95 (s) | Completion p95 (s) |
|---|---|---|---|
| 1 | 3.557 / 3.614 | 0.031 / 0.046 | 0.223 / 0.229 |
| 10 | 13.873 / 17.548 | 0.071 / 0.031 | 0.680 / 0.553 |
| 25 | 18.342 / 18.093 | 0.121 / 0.085 | 1.309 / 1.323 |
| 50 | 17.640 / 18.501 | 0.208 / 0.149 | 2.781 / 2.638 |
The 1,316-byte plaintext artifact occupied 1,332 bytes after AES-256-GCM encryption: 16 bytes
of object-data overhead. The replays PostgreSQL relation grew 32,768 bytes during the retained
172-job run. Sampled maximum single-container memory was 386,400,256 bytes; sampled maximum
single-container CPU was 16.79%. These are Docker Desktop observations, not host-wide resource
totals or capacity claims.
| Exact phrase | Status | Permitted interpretation |
|---|---|---|
| “Built an open-source system” | passed | Implemented repository and locally executed platform. |
| “captures failed CI jobs” | passed | The released Action captured controlled failures in 12 successful public GitHub Actions trials, in addition to local validation. |
| “deterministically replays them in isolated containers” | passed | Docker and Kind replays reproduced controlled artifacts; 12/12 post-adaptation owner-dogfood artifacts reproduced in one attempt. No cross-project generalization claim. |
| “helping developers reproduce intermittent test failures” | externally pending | Product intent only; no external user or debugging-time study. |
| “distributed execution” | passed | PostgreSQL queue and 50-worker local execution were exercised. |
| “content-addressed caching” | passed | Existing SHA-256 cache is preserved; hosted artifacts are digest-verified. |
| “sandboxing” | passed for declared controls | Read-only filesystem, non-root UID, dropped capabilities, no-new-privileges, resource limits, and no-network Docker replay were exercised; not a complete security boundary. |
| “checkpointing” | passed | A killed worker resumed from an expired lease and completed with a persisted checkpoint. |
| “filesystem/network fault injection” | passed | Supported only by the existing controlled engine benchmark; not evidence of external failure coverage. |
| “replay 50 concurrent jobs with 100% reproducibility” | passed | Two 50-job controlled runs completed 50/50; do not quote as a real-world reproduction rate. |
| “Piloted with [N] open-source projects” | externally pending | Two public owner repositories were dogfooded, but that is not an independent pilot or adoption. |
| “reduced median failure-reproduction time by [X]%” | externally pending | No user baseline or time study exists. |
| “leading to [N] resolved flaky tests and [N] external contributions” | externally pending | No external repository was modified and nobody was contacted. |
docker compose -f compose.platform.yaml up -d --build
python validation/platform/strict_validation.py
python validation/platform/benchmark_platform.py
cd controlplane
REPROCI_VALIDATION_REQUIRED=1 \
DATABASE_URL=postgres://reproci:local-only-change-me@127.0.0.1:5432/reproci \
REPROCI_S3_BUCKET=reproci REPROCI_S3_ENDPOINT=http://127.0.0.1:9000 \
AWS_ACCESS_KEY_ID=local-reproci AWS_SECRET_ACCESS_KEY=local-only-change-me \
AWS_REGION=us-east-1 go test -tags=integration ./... -count=1