Skip to content

Commit 873c9cc

Browse files
committed
Implemented the Phase 1 evidence governance artifacts in the repo.
1 parent ba65866 commit 873c9cc

11 files changed

Lines changed: 2479 additions & 0 deletions

File tree

audit-logs/README.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# Audit Log Boundary
2+
3+
This directory is reserved for append-only audit-log schemas, local buffers,
4+
decision ledgers, access logs, signing events, and transparency-log records.
5+
6+
Audit inclusion proves that a record was logged under the stated hash and
7+
canonicalization rules. It does not prove that the underlying evidence is
8+
valid, representative, true, or sufficient for deployment.
Lines changed: 363 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,363 @@
1+
# Evidence Governance Repository Phase 1
2+
3+
This document specifies the Phase 1 constitutional layer for an evidence
4+
governance repository integrated with `cc-framework`. It is an evidence and
5+
assurance compilation substrate, not a deployment approval system, not proof of
6+
deployment safety, and not causal inference without assumptions.
7+
8+
## Repository Directory Structure
9+
10+
```text
11+
cc-framework/
12+
├── docs/
13+
│ ├── architecture/
14+
│ │ ├── evidence-governance-phase-1.md
15+
│ │ ├── AUDIT_EVIDENCE_BOUNDARIES.md
16+
│ │ └── STRICT_KERNEL_CONTRACT.md
17+
│ ├── design-specs/
18+
│ ├── governance/
19+
│ ├── research/
20+
│ └── runbooks/
21+
├── schemas/
22+
│ ├── cc_report.schema.json
23+
│ ├── common/
24+
│ │ ├── digest.schema.json
25+
│ │ ├── actor.schema.json
26+
│ │ ├── timestamp.schema.json
27+
│ │ ├── signature.schema.json
28+
│ │ └── confidence.schema.json
29+
│ ├── evidence/
30+
│ │ ├── evidence-item.schema.json
31+
│ │ ├── custody-event.schema.json
32+
│ │ ├── review-decision.schema.json
33+
│ │ ├── extraction-report.schema.json
34+
│ │ └── contradiction-link.schema.json
35+
│ ├── ontology/
36+
│ │ ├── namespace-manifest.schema.json
37+
│ │ ├── ontology-node.schema.json
38+
│ │ ├── ontology-edge.schema.json
39+
│ │ ├── concept-scheme.schema.json
40+
│ │ └── mapping-rule.schema.json
41+
│ ├── storage/
42+
│ │ ├── bag-manifest.schema.json
43+
│ │ ├── ocfl-object-extension.schema.json
44+
│ │ ├── retention-policy.schema.json
45+
│ │ ├── embedding-record.schema.json
46+
│ │ └── cold-tier-restore.schema.json
47+
│ └── audit/
48+
│ ├── access-event.schema.json
49+
│ ├── pipeline-event.schema.json
50+
│ ├── transparency-entry.schema.json
51+
│ └── signing-event.schema.json
52+
├── ontology/
53+
│ ├── contexts/
54+
│ │ ├── evidence.context.jsonld
55+
│ │ ├── provenance.context.jsonld
56+
│ │ └── review.context.jsonld
57+
│ ├── namespaces/
58+
│ │ ├── registry.yaml
59+
│ │ ├── reserved-prefixes.yaml
60+
│ │ └── deprecation-map.yaml
61+
│ ├── schemes/
62+
│ │ ├── domains/
63+
│ │ ├── document-types.ttl
64+
│ │ ├── evidence-tiers.ttl
65+
│ │ └── extraction-methods.ttl
66+
│ ├── owl/
67+
│ │ ├── evidence-core.owl.ttl
68+
│ │ ├── provenance-axioms.owl.ttl
69+
│ │ └── governance-axioms.owl.ttl
70+
│ ├── mappings/
71+
│ │ ├── prov-mapping.yaml
72+
│ │ ├── premis-mapping.yaml
73+
│ │ ├── dcat-mapping.yaml
74+
│ │ └── external-taxonomy-crosswalks/
75+
│ └── catalog/
76+
│ ├── catalog.ttl
77+
│ └── datasets/
78+
├── pipelines/
79+
│ ├── ingest/
80+
│ │ ├── package_submission.py
81+
│ │ ├── verify_manifest.py
82+
│ │ ├── capture_provenance.py
83+
│ │ └── submit_to_queue.py
84+
│ ├── sanitation/
85+
│ │ ├── mime_validate.py
86+
│ │ ├── schema_validate.py
87+
│ │ ├── canonicalize_json.py
88+
│ │ ├── normalize_text.py
89+
│ │ ├── detect_malware.py
90+
│ │ ├── compute_hashes.py
91+
│ │ └── score_extraction_quality.py
92+
│ ├── dedupe/
93+
│ │ ├── exact_hash_match.py
94+
│ │ ├── near_duplicate_text.py
95+
│ │ ├── near_duplicate_media.py
96+
│ │ └── contradiction_probe.py
97+
│ ├── review/
98+
│ │ ├── route_quarantine.py
99+
│ │ ├── assign_reviewers.py
100+
│ │ ├── merge_decisions.py
101+
│ │ └── emit_review_attestation.py
102+
│ ├── promote/
103+
│ │ ├── build_ocfl_object.py
104+
│ │ ├── write_premis_event.py
105+
│ │ ├── lock_worm_retention.py
106+
│ │ └── publish_catalog_record.py
107+
│ ├── observability/
108+
│ │ ├── emit_pipeline_event.py
109+
│ │ ├── export_otel_logs.py
110+
│ │ └── anomaly_alerts.py
111+
│ ├── lib/
112+
│ │ ├── hashing.py
113+
│ │ ├── signatures.py
114+
│ │ ├── policy_client.py
115+
│ │ ├── ontology_client.py
116+
│ │ └── time_stamping.py
117+
│ └── tests/
118+
├── policies/
119+
│ ├── governance/
120+
│ │ ├── evidence-lifecycle.md
121+
│ │ ├── confidence-scoring.md
122+
│ │ ├── reviewer-consensus.md
123+
│ │ └── deprecation-policy.md
124+
│ ├── access/
125+
│ │ ├── roles.yaml
126+
│ │ ├── attributes.yaml
127+
│ │ ├── data-classification.yaml
128+
│ │ └── separation-of-duties.yaml
129+
│ ├── retention/
130+
│ │ ├── worm-retention.yaml
131+
│ │ ├── legal-hold.yaml
132+
│ │ ├── cold-archive.yaml
133+
│ │ └── restore-sla.yaml
134+
│ ├── rego/
135+
│ │ ├── admission/
136+
│ │ ├── promotion/
137+
│ │ │ ├── tiering.rego
138+
│ │ │ ├── conflict.rego
139+
│ │ │ └── retention.rego
140+
│ │ └── access/
141+
│ └── exceptions/
142+
├── storage/
143+
│ ├── manifests/
144+
│ │ ├── storage-classes.yaml
145+
│ │ ├── ocfl-storage-root.yaml
146+
│ │ ├── bucket-layout.yaml
147+
│ │ └── encryption-kms.yaml
148+
│ ├── bagit/
149+
│ │ ├── bag-info-template.txt
150+
│ │ ├── tagmanifest-sha256-template.txt
151+
│ │ └── submission-profile.yaml
152+
│ ├── ocfl/
153+
│ │ ├── extensions/
154+
│ │ ├── validation-profiles/
155+
│ │ └── object-template/
156+
│ │ ├── 0=ocfl_object_1.1
157+
│ │ ├── inventory.json
158+
│ │ └── v1/
159+
│ │ ├── inventory.json
160+
│ │ └── content/
161+
│ ├── tiers/
162+
│ │ ├── quarantine/
163+
│ │ ├── raw/
164+
│ │ ├── normalized/
165+
│ │ ├── derivatives/
166+
│ │ ├── embeddings/
167+
│ │ ├── catalog/
168+
│ │ └── cold/
169+
│ └── restore/
170+
├── audit-logs/
171+
│ ├── schemas/
172+
│ │ ├── decision-ledger.schema.json
173+
│ │ ├── access-log.schema.json
174+
│ │ ├── pipeline-log.schema.json
175+
│ │ └── transparency-log.schema.json
176+
│ ├── pipeline/
177+
│ ├── access/
178+
│ ├── decisions/
179+
│ ├── signing/
180+
│ └── transparency/
181+
└── src/cc/
182+
├── kernel/
183+
│ ├── frechet_classes.py
184+
│ ├── sensitivity.py
185+
│ └── frechet_sensitivity.py
186+
├── evidence/
187+
│ ├── claim_governance.py
188+
│ ├── confirmatory_protocol.py
189+
│ ├── decay.py
190+
│ ├── merkle_log.py
191+
│ └── role_ontology.py
192+
├── reporting/
193+
│ ├── canonical.py
194+
│ └── report.py
195+
└── redteam/
196+
└── dependence_search.py
197+
```
198+
199+
`src/cc/claims/` is intentionally absent from Phase 1. Claim lifecycle states
200+
remain quarantined until a separate claim compiler is designed.
201+
202+
## RACI Matrix
203+
204+
Legend: R = responsible for normal work, A = accountable approver, C =
205+
consulted reviewer, I = informed observer, E = exception authority.
206+
207+
| Area | Ingestion bot | Synthesis engine | Policy sentinel | Submitter | Curator | Reviewer | Policy-admin | Auditor |
208+
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
209+
| `/schemas` | I | C | R | I | C | C | A/E | C |
210+
| `/ontology` | I | C | R | I | A | C | E | C |
211+
| `/pipelines/ingest` | R | I | C | C | A | I | E | C |
212+
| `/pipelines/sanitation` | R | C | R | I | A | C | E | C |
213+
| `/pipelines/dedupe` | R | C | C | I | A | C | E | C |
214+
| `/pipelines/review` | C | C | R | I | A | R | E | C |
215+
| `/pipelines/promote` | R | I | R | I | C | A | E | C |
216+
| `/policies/governance` | I | C | R | I | C | C | A/E | C |
217+
| `/policies/rego` | I | I | R | I | C | C | A/E | C |
218+
| `/storage/bagit` | R | I | C | C | A | I | E | C |
219+
| `/storage/ocfl` | R | I | C | I | A | C | E | C |
220+
| `/storage/tiers/quarantine` | R | I | R | I | A | R | E | C |
221+
| `/audit-logs` | R | I | R | I | I | C | E | A |
222+
| `/src/cc/kernel` | I | C | C | I | C | C | A | C |
223+
| `/src/cc/evidence` | I | C | R | I | C | A | E | C |
224+
| `/src/cc/reporting` | I | R | R | I | C | A | E | C |
225+
| `/src/cc/redteam` | I | R | C | I | C | C | E | I |
226+
227+
Authority constraints:
228+
229+
- Submitters can create submissions and read their own submission status. They
230+
cannot promote, edit normalized evidence, alter policy, or approve exceptions.
231+
- Curators own metadata completeness, ontology routing, and quarantine
232+
triage. They cannot override Rego denials without a policy-admin exception.
233+
- Reviewers approve human-review requirements but cannot modify confidence
234+
scores or statistical weights.
235+
- Policy-admins own policy and exception mechanics. Exception records require
236+
owner, expiry, affected policy, reason, and attestation.
237+
- Auditors have read access to all evidence, audit logs, policy bundles, and
238+
storage inventories. They do not mutate evidence records.
239+
240+
## Confidence Model
241+
242+
Let:
243+
244+
- `P` = provenance completeness score in `[0, 1]`
245+
- `I` = integrity score in `[0, 1]`
246+
- `R` = source reliability score in `[0, 1]`
247+
- `C` = corroboration score in `[0, 1]`
248+
- `T` = temporal validity or freshness score in `[0, 1]`
249+
- `E` = extraction quality score in `[0, 1]`
250+
251+
The confidence score is:
252+
253+
```text
254+
confidence(e) = round(0.24P + 0.24I + 0.18R + 0.16C + 0.10T + 0.08E, 3)
255+
```
256+
257+
Promotion thresholds:
258+
259+
| Tier | Rule |
260+
| --- | --- |
261+
| Tier 1 | `P >= 0.95`, `I == 1.0`, and `confidence >= 0.85` |
262+
| Tier 2 | `P >= 0.80`, `I >= 0.90`, and `confidence >= 0.70` |
263+
| Tier 3 | `P >= 0.55`, `I >= 0.70`, and `confidence >= 0.45` |
264+
| Tier 4 | Otherwise; raw, unverified, or retained only for review |
265+
266+
Human review may satisfy a policy gate. It does not change `P`, `I`, `R`,
267+
`C`, `T`, `E`, or `confidence(e)`.
268+
269+
## Ingestion, Sanitation, And Quarantine Workflow
270+
271+
```mermaid
272+
flowchart TD
273+
A["Raw BagIt or object submission arrives"] --> B["Create submission envelope"]
274+
B --> C["Assign submission UUID and intake timestamp"]
275+
C --> D["Compute raw SHA-256 and SHA-512"]
276+
D --> E["Verify BagIt manifest and MIME consistency"]
277+
E --> F{"Manifest or MIME failure?"}
278+
F -- Yes --> Q1["Immediate quarantine: package integrity failure"]
279+
F -- No --> G["Capture W3C PROV fields"]
280+
G --> H{"Required provenance present?"}
281+
H -- No --> Q2["Quarantine: missing required provenance"]
282+
H -- Yes --> I["Extract content"]
283+
I --> J["NFC normalize text and canonicalize JSON"]
284+
J --> K["Run JSON Schema 2020-12 validation"]
285+
K --> L{"Schema valid?"}
286+
L -- No --> Q3["Quarantine: schema violation"]
287+
L -- Yes --> M["Verify signatures and timestamps"]
288+
M --> N{"Integrity failure?"}
289+
N -- Yes --> Q4["Immediate quarantine: signature, timestamp, or hash breach"]
290+
N -- No --> O["Malware and parser exploit scan"]
291+
O --> P{"Malicious payload?"}
292+
P -- Yes --> DROP["Automatic drop: malicious active content"]
293+
P -- No --> R["Exact SHA-256 deduplication"]
294+
R --> S["Near-duplicate clustering"]
295+
S --> T["Ontology tagging and namespace resolution"]
296+
T --> U{"Unresolved tags or anomalies?"}
297+
U -- Yes --> HR1["Human review queue"]
298+
U -- No --> V["Contradiction and corroboration analysis"]
299+
V --> W{"Conflict with higher-confidence evidence?"}
300+
W -- Yes --> HR2["Human review queue"]
301+
W -- No --> X["Compute confidence score"]
302+
X --> Y["Evaluate OPA admission and promotion policy"]
303+
Y --> Z{"Confidence and policy gates satisfied?"}
304+
Z -- No --> HR3["Human review queue or Tier 4 retention"]
305+
Z -- Yes --> AA["Build OCFL object version"]
306+
AA --> AB["Write PREMIS preservation event"]
307+
AB --> AC["Write append-only audit event"]
308+
AC --> AD["Sign SLSA/Sigstore attestation"]
309+
AD --> AE["Record transparency-log leaf"]
310+
AE --> AF["Apply WORM retention or legal hold"]
311+
AF --> AG["Promote to immutable evidence store"]
312+
```
313+
314+
## Failure-State Specification
315+
316+
Automatic drop:
317+
318+
- Malware, parser exploit payload, or active content that violates ingestion
319+
safety policy.
320+
- Payload type that cannot be safely parsed in a sandbox and has no approved
321+
manual handling profile.
322+
- Attempted overwrite or deletion of a WORM-protected object.
323+
324+
Immediate quarantine:
325+
326+
- Raw package hash mismatch.
327+
- BagIt manifest mismatch or missing required payload manifest.
328+
- Declared MIME type inconsistent with detected MIME type.
329+
- Missing required provenance fields.
330+
- JSON Schema failure on required fields or forbidden extra fields.
331+
- Required signature absent, revoked, unverifiable, or failed.
332+
- Timestamp from the future outside the accepted clock-skew window.
333+
- Chain-of-custody event hash mismatch.
334+
- Empty `non_claims` array.
335+
- Overclaiming text that asserts deployment approval, absolute safety, model
336+
truth, or causal inference without assumptions.
337+
- Exploratory red-team evidence routed to confirmatory promotion without a
338+
pre-registered protocol binding and model version hash increment.
339+
340+
Human-review exception queue:
341+
342+
- Incomplete but plausibly reconstructible provenance.
343+
- OCR or extraction quality below policy threshold.
344+
- Near duplicate with numeric or temporal drift.
345+
- Unresolved ontology mapping or namespace ambiguity.
346+
- Legal, privacy, or data-classification ambiguity.
347+
- Contradiction with higher-confidence evidence.
348+
- Confidence score below requested tier threshold but above raw-retention
349+
threshold.
350+
- Clustered, adaptive, or post-selection evidence that needs an explicit
351+
confirmatory-protocol determination.
352+
353+
Immutable promotion writes:
354+
355+
1. OCFL object version with inventory.
356+
2. PREMIS preservation event.
357+
3. Evidence-item metadata record with storage references.
358+
4. Signed pipeline attestation and transparency-log entry.
359+
360+
Each promotion receipt must append the caveat that cryptographic integrity
361+
proves byte identity and provenance continuity only; it does not prove
362+
statistical validity, representativeness, empirical truth, deployment safety,
363+
or causal validity.

ontology/README.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# Ontology Boundary
2+
3+
This directory is reserved for namespace registries, JSON-LD contexts, SKOS
4+
schemes, OWL axioms, DCAT catalog records, and crosswalk mappings.
5+
6+
Namespace identifiers use:
7+
8+
```text
9+
ev:<domain>:<subdomain>:<entity-type>:<slug>
10+
```
11+
12+
Ontology assets define vocabulary and graph publication semantics. They do not
13+
promote evidence, approve deployment, or alter statistical confidence.

pipelines/README.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
# Pipeline Boundary
2+
3+
This directory is reserved for ingestion, sanitation, deduplication, review,
4+
promotion, and observability pipeline code.
5+
6+
Pipelines must fail closed. Missing provenance, integrity mismatches, schema
7+
violations, malware signals, overclaims, and red-team-to-confirmatory leakage
8+
route to quarantine or review according to the Phase 1 architecture spec.

0 commit comments

Comments
 (0)