Replace compliance checkboxes with evidence-based release decisions

.github/workflows/validate-release-config.yml

@@ -6,12 +6,14 @@ on:
       - 'release/release-checklist.yaml'
       - 'examples/sample-release-checklist.yaml'
       - 'tools/validate_release_config.py'
+      - 'tests/test_validate_release_config.py'
       - '.github/workflows/validate-release-config.yml'
   pull_request:
     paths:
       - 'release/release-checklist.yaml'
       - 'examples/sample-release-checklist.yaml'
       - 'tools/validate_release_config.py'
+      - 'tests/test_validate_release_config.py'
       - '.github/workflows/validate-release-config.yml'
 
 permissions:
@@ -20,7 +22,7 @@ permissions:
 jobs:
   validate:
     runs-on: ubuntu-latest
-    name: Validate release checklist YAML
+    name: Validate release decision YAML
 
     steps:
       - uses: actions/checkout@v4
@@ -33,13 +35,16 @@ jobs:
       - name: Install dependencies
         run: python -m pip install --upgrade pip pyyaml
 
+      - name: Run validator tests
+        run: python -m unittest discover -s tests -v
+
       - name: Validate starter template structure
         run: |
           python tools/validate_release_config.py \
             release/release-checklist.yaml \
             --mode template
 
-      - name: Validate populated sample minimum gates
+      - name: Validate fictional decision example
         run: |
           python tools/validate_release_config.py \
             examples/sample-release-checklist.yaml \

docs/release-config-validation.md

@@ -1,16 +1,88 @@
-# Release-Config Validation
+# Release-Decision Configuration Validation
 
-This starter repository distinguishes two checks:
+This starter repository validates the structure and selected decision semantics of a release record. It deliberately does not prescribe universal accuracy, fairness, industry, or approval gates.
 
-1. **Template validation** checks that `release/release-checklist.yaml` has the expected structure and supported metadata. The default file intentionally contains `false` gates and placeholder values because it is not a release request.
-2. **Readiness-example validation** checks that `examples/sample-release-checklist.yaml` satisfies the repository's illustrative minimum gates for its declared risk tier and industry.
+## Modes
 
-Run locally:
+### Template
 
 ```bash
-python -m pip install pyyaml
-python tools/validate_release_config.py release/release-checklist.yaml --mode template
-python tools/validate_release_config.py examples/sample-release-checklist.yaml --mode ready
+python tools/validate_release_config.py \
+  release/release-checklist.yaml \
+  --mode template
 ```
 
-The validator is intentionally narrow. It does not certify safety, compliance, legal sufficiency, production readiness, or an actual deployment decision. It only makes the starter template's structural and illustrative gate contract explicit and testable.
+Template mode checks:
+
+- required top-level sections;
+- required metadata fields;
+- supported decision outcomes and gate statuses;
+- gate field types;
+- list structure;
+- unique gate identifiers.
+
+Placeholders and unresolved gates are allowed because the file is a starting structure, not a release request.
+
+### Ready
+
+```bash
+python tools/validate_release_config.py \
+  examples/sample-release-checklist.yaml \
+  --mode ready
+```
+
+Ready mode additionally checks:
+
+- required metadata and gate fields do not contain placeholders;
+- at least one gate exists;
+- passing and not-applicable gates cite evidence;
+- not-applicable gates contain a scoped rationale;
+- release and conditional release have no blockers;
+- release and conditional release have no unresolved hard gates;
+- unconditional release has no conditions or required actions;
+- conditional release has at least one condition or required action;
+- deferred decisions identify evidence gaps;
+- do-not-release decisions identify a blocker or unresolved hard gate.
+
+## Semantics
+
+The configuration distinguishes:
+
+- **hard gate:** a non-compensable requirement for the reviewed scope;
+- **blocker:** unresolved finding that prevents release;
+- **required action:** accepted follow-up work under a bounded conditional decision;
+- **condition:** enforceable restriction on scope or operation;
+- **evidence gap:** missing or unreliable support for a proposition;
+- **residual risk:** risk remaining after controls, requiring organization-specific acceptance.
+
+The validator checks internal coherence. It does not determine whether:
+
+- a gate is appropriate;
+- evidence is authentic, current, representative, or sufficient;
+- a not-applicable rationale is correct;
+- a finding severity is justified;
+- a condition is enforceable in the real system;
+- a residual-risk owner has proper authority;
+- a release is safe, lawful, compliant, production-ready, or valuable.
+
+## Tests
+
+```bash
+python -m unittest discover -s tests -v
+```
+
+The tests cover blocker, condition, hard-gate, evidence, placeholder, deferred-decision, and duplicate-ID semantics.
+
+## Customization
+
+Organizations should replace the example propositions with gates derived from the actual:
+
+- intended use and affected population;
+- data, model, tool, permission, and action scope;
+- harm and consequence model;
+- applicable policies and obligations;
+- evaluation and control evidence;
+- operational, incident, recovery, and redress needs;
+- release authority and residual-risk process.
+
+Do not add a generic gate merely because it sounds responsible. Every gate should support a named decision and have an evidence owner, method, scope, and invalidation trigger.

examples/sample-release-checklist.yaml

@@ -1,46 +1,130 @@
+# Fictional example only. Northstar Library, the system, people, dates, and
+# evidence references below are invented for this repository.
+
 metadata:
-  project: "Document Triage Assistant"
-  version: "0.1.0"
-  environment: "staging"
-  regulated_industry: "general"
-  risk_classification: "medium"
-  deployment_owner: "Example Product Owner"
-
-model_validation:
-  performance:
-    accuracy_threshold: 0.90
-    bias_evaluation_complete: true
-    adversarial_testing_complete: false
-  fairness:
-    disparate_impact_ratio: 0.86
-    subgroup_performance_review: true
-
-governance:
-  documentation:
-    model_card_complete: true
-    risk_assessment_complete: true
-    explainability_report_complete: false
-  approvals:
-    technical_review: true
-    ai_governance_review: true
-    legal_review: false
-    security_review: true
-  regulatory:
-    privacy_review_complete: true
-
-infrastructure:
-  testing:
-    unit_tests_passing: true
-    integration_tests_passing: true
-    security_scan_passed: true
-    load_test_passed: false
-  monitoring:
-    alerting_configured: true
-    drift_detection_enabled: true
-  rollback:
-    rollback_plan_documented: true
-    rollback_tested: false
-
-incident_readiness:
-  runbook_complete: true
-  escalation_contacts_defined: true
+  project: "Northstar Catalog Drafting Assistant"
+  version: "0.3.0-pilot"
+  environment: "bounded-internal-pilot"
+  decision_scope: "60 trained library staff; public catalog records only; read and draft tools; no publication, deletion, patron data, or external messaging"
+  decision_owner: "Fictional Pilot Sponsor"
+  evidence_cutoff: "2026-09-30"
+  context:
+    intended_use: "Draft suggested catalog corrections for staff review"
+    prohibited_uses:
+      - "publish or delete records"
+      - "access patron identity or borrowing history"
+      - "make employment, eligibility, legal, or financial decisions"
+    affected_populations:
+      - "library catalog staff"
+      - "users relying on accessible catalog search"
+    system_components:
+      - "fictional language model"
+      - "public catalog retrieval index"
+      - "read-only search tool"
+      - "draft-correction tool"
+    explicit_exclusions:
+      - "public self-service access"
+      - "restricted archive records"
+      - "automated execution of corrections"
+
+decision:
+  outcome: "release_with_conditions"
+  rationale: "The fictional evidence supports a small read-and-draft pilot. The pilot remains bounded because correction quality for rare legacy records is not yet estimated precisely enough for broader use."
+  blockers: []
+  required_actions:
+    - "Complete a stratified review of 200 legacy-record drafts before any population expansion."
+  conditions:
+    - "Keep all source-system tools read-only and require staff confirmation before any correction is entered manually."
+    - "Limit the pilot to 60 trained internal users and public catalog records through 2026-11-30."
+    - "Pause the pilot after any confirmed restricted-record exposure or unauthorized write attempt."
+  evidence_gaps:
+    - "Performance on rare handwritten-archive transcription patterns remains based on a small exploratory sample."
+  residual_risks:
+    - "Staff may over-trust fluent but unsupported draft rationales despite source links and training."
+  expires_on: "2026-11-30"
+
+gates:
+  - id: "SCOPE-001"
+    question: "Are intended use, prohibited use, population, data, tools, permissions, and environment bounded for this decision?"
+    hard_gate: true
+    status: "pass"
+    evidence:
+      - "evidence/fictional-pilot-scope-v3.md"
+      - "evidence/fictional-data-flow-v2.md"
+    owner: "Fictional Product Owner"
+    limitation: "The scope record applies only to the internal pilot configuration."
+
+  - id: "AUTH-001"
+    question: "Are identity, authorization, confirmation, and action boundaries enforced for the reviewed scope?"
+    hard_gate: true
+    status: "pass"
+    evidence:
+      - "evidence/fictional-permission-test-2026-09-22.json"
+      - "evidence/fictional-unauthorized-call-test.md"
+    owner: "Fictional Platform Owner"
+    limitation: "Manual source-system correction occurs outside the assistant and was reviewed as a separate staff workflow."
+
+  - id: "DATA-001"
+    question: "Does the reviewed configuration prevent retrieval and disclosure of restricted and patron-linked records?"
+    hard_gate: true
+    status: "pass"
+    evidence:
+      - "evidence/fictional-data-boundary-tests-v4.json"
+    owner: "Fictional Data Protection Reviewer"
+    limitation: "Tests cover the versioned synthetic and public-safe fixture; production records remain outside scope."
+
+  - id: "EVAL-001"
+    question: "Does current evaluation evidence support the stated rollout decision and disclose coverage and uncertainty?"
+    hard_gate: false
+    status: "partial"
+    evidence:
+      - "evidence/fictional-evaluation-report-v5.md"
+    owner: "Fictional Evaluation Owner"
+    limitation: "Core catalog journeys are covered; the rare legacy-record slice requires a larger confirmatory sample before expansion."
+
+  - id: "OPS-001"
+    question: "Can the system be monitored, stopped, contained, recovered, and supported under the reviewed conditions?"
+    hard_gate: true
+    status: "pass"
+    evidence:
+      - "evidence/fictional-pilot-stop-exercise.md"
+      - "evidence/fictional-support-runbook-v2.md"
+    owner: "Fictional Operations Owner"
+    limitation: "The exercise covered the internal pilot only and did not test public-user communication."
+
+  - id: "ACCESS-001"
+    question: "Can pilot staff complete critical review and correction tasks using the supported accessibility paths?"
+    hard_gate: false
+    status: "pass"
+    evidence:
+      - "evidence/fictional-accessibility-review.md"
+    owner: "Fictional Accessibility Reviewer"
+    limitation: "One supported browser and two assistive-technology combinations were reviewed."
+
+findings:
+  - id: "FINDING-001"
+    description: "Rare legacy-record corrections have wide uncertainty because only 32 representative cases were reviewed."
+    severity: "medium"
+    disposition: "required_action"
+    owner: "Fictional Evaluation Owner"
+    due_or_expiry: "before population expansion"
+
+  - id: "FINDING-002"
+    description: "Fluent unsupported rationale remains possible when retrieved source records are incomplete."
+    severity: "medium"
+    disposition: "condition"
+    owner: "Fictional Product Owner"
+    due_or_expiry: "2026-11-30"
+
+change_triggers:
+  - "model, prompt, retrieval corpus, or drafting-policy change"
+  - "addition of write, publish, delete, messaging, or restricted-data authority"
+  - "pilot population, language, site, or geography expansion"
+  - "confirmed restricted-record exposure, unauthorized tool attempt, or material correction incident"
+  - "change to evaluator rubric, scenario suite, or acceptance decision"
+
+follow_through:
+  release_or_hold_action_owner: "Fictional Pilot Owner"
+  monitoring_and_incident_owner: "Fictional Operations Owner"
+  evidence_record_location: "evidence/fictional-release-decision-2026-09-30/"
+  next_review: "2026-11-15 or earlier on any stop trigger"