Skip to content

feat: add quarterly validation suite with CI integration and local ru… - #722

Merged
pugsley76 merged 1 commit into
pugsley76:mainfrom
directorfloo:Establishaquarterlyendtoendproductlidationsuite
Jul 30, 2026
Merged

feat: add quarterly validation suite with CI integration and local ru…#722
pugsley76 merged 1 commit into
pugsley76:mainfrom
directorfloo:Establishaquarterlyendtoendproductlidationsuite

Conversation

@directorfloo

Copy link
Copy Markdown
Contributor

Closes #626

PR title
Validation: add quarterly end-to-end validation suite and deterministic fixtures

PR body (copy this into the PR description)

Summary

  • Add a scheduled, versioned validation suite that runs deterministic, sanitized journeys exercising the primary user flows (debug, trace, profile, session, audit, protocol, release verification).
  • Run quarterly on Linux + Windows; artifacts are redacted and uploaded for 30 days.
  • Provide tools to aggregate run results into quarterly compatibility reports.

What this change adds

  • Runner and helpers
    • run-validation-suite.sh — validation runner (journeys + summary.json + report.md)
    • generate-compat-report.sh — aggregate summary.json into compatibility_report.md
    • VALIDATION_RUN.md — local & CI run instructions
  • CI
    • validation-suite.yml — scheduled workflow (quarterly; ubuntu + windows) uploading ci-artifacts/validation/
  • Docs
    • validation-suite.md — spec, acceptance criteria, retention and reporting notes
  • Deterministic fixtures (minimal, test-only)
    • sample.trace.json
    • sample_tx.json
    • sample.registry.json
    • placeholder.wasm
    • sample.session.json
    • payload.json
    • testonly_key.pem
    • README.md
  • Minor run/report fixes: improved redaction invocation and reliable report.md generation

Acceptance checklist

  • Every journey has a deterministic fixture under fixtures
  • Failures output per-journey logs and ci-artifacts/validation/summary.json
  • Workflow scheduled quarterly and targets Linux + Windows (validation-suite.yml)
  • Artifacts are redacted by redact-logs.sh before upload
  • generate-compat-report.sh exists for quarterly aggregation
  • Local run instructions added in VALIDATION_RUN.md

Notes & scope

  • The runner exercises CLI-level journeys (dry-runs, verify-dir, loading traces). Full E2E simulator replay or KMS signing across networks may require additional CI environment setup or test harnesses; those can be added in follow-up PRs.
  • The audit fixture uses testonly_ semantics and the runner runs redact-logs.sh before uploads; maintainers should verify redaction rules when adding new fixtures.

How to run locally (copyable)

# build binary
go build -o bin/glassbox ./cmd/glassbox

# install node deps (if needed)
npm ci --prefer-offline
npm run build --if-present

# run suite (writes artifacts to ci-artifacts/validation/)
chmod +x scripts/run-validation-suite.sh scripts/generate-compat-report.sh
GLASSBOX_BINARY=./bin/glassbox scripts/run-validation-suite.sh

# generate compatibility report from runs
scripts/generate-compat-report.sh ci-artifacts/validation ci-artifacts/validation/compatibility_report.md

Files changed

  • run-validation-suite.sh (new/updated)
  • generate-compat-report.sh (new)
  • validation-suite.yml (new)
  • validation-suite.md (new/updated)
  • VALIDATION_RUN.md (new)
  • test/validation/... (fixtures added)
  • README.md (new)

Suggested reviewers / labels

  • Reviewers: maintainers, ci
  • Labels: ci, test, regression

@drips-wave

drips-wave Bot commented Jul 30, 2026

Copy link
Copy Markdown

@directorfloo Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

@pugsley76
pugsley76 merged commit 41aa7c0 into pugsley76:main Jul 30, 2026
0 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Issue 119: Establish a quarterly end-to-end product validation suite

3 participants