Skip to content

README.md

Latest commit

 

History

History
450 lines (346 loc) · 18.2 KB

README.md

File metadata and controls

450 lines (346 loc) · 18.2 KB
Raw

CC-Framework

Dependence-aware composition of AI guardrails: Frechet-Hoeffding bounds on stacked-system failure, not a safety certification.

CI Python License: MIT

CC-Framework is a Python research framework for dependence-aware analysis of composed binary guardrail failures. It computes what available guardrail evidence supports under stated dependence assumptions. It does not certify that a stacked system is safe.

Public Boundary

Surface Role Claim boundary
Paper Core Manuscript, deterministic paper artifacts, theorem ledger, and validation matrix. Supports the v0.3 finite-atom partial-identification claim only.
Research Kernel cc.kernel.strict, atom LPs, Frechet helpers, canonical metrics, endpoint witnesses, and finite-sample helpers. Bounds composed binary failure probability under declared assumptions.
Evidence Governance Claim envelopes, evidence roles, Merkle logs, receipts, confirmatory protocol, and decay semantics. Supports replayable evidence records; hashes and signatures do not prove validity or safety.
Ecosystem Bridges Optional adapters for external guardrail or evaluation tools. Interop only; not paper-core evidence unless promoted by a release note.
Enterprise Reference AWS/KMS/S3/DynamoDB/API reference and moto-backed smoke tests. Experimental evidence-integrity reference; not an enterprise product or compliance claim.
Dashboard/Demo Next.js dashboard and Claim Observatory visual material. Demonstrates reports and evidence bundles; it does not define the research kernel.
Experimental/Archive Experiments, notebooks, historical generated results, and preserved side branches. Useful research material outside the release claim path unless explicitly promoted.

Business & Assurance Interpretation

CC-Framework can also be read as a prototype for AI safety disclosure controls. In high-stakes organizational settings, a major failure mode is not only technical miscalculation, but institutional overclaiming: presenting narrow, procedural, stale, or brittle evaluation evidence as broad proof of deployment safety.

The framework applies a disclosure-control lens to AI assurance reports. It keeps statistical assumptions, evidence artifacts, cryptographic receipts, human review, validation lanes, and deployment claims conceptually separate. This is analogous to internal-control thinking in financial reporting: the goal is to make public claims traceable to assumptions, evidence, validation procedures, and explicit limitations.

This interpretation does not expand the project's claim boundary. CC-Framework remains a research prototype. It does not certify deployed systems, prove compliance, validate dataset representativeness, or turn receipt integrity into empirical truth. Its narrower purpose is to make AI assurance claims more evidence-bound, reproducible, and harder to overstate accidentally.

Claim Boundary Manifest

CC-Framework now maintains a claim-boundary manifest that maps public claims to validation lanes, supporting files, tests or commands, and explicit non-claims. The manifest is intended to prevent documentation, demos, receipts, dashboards, or enterprise references from being interpreted as broader assurance claims than the repository supports. See docs/claims/CLAIM_BOUNDARY_MANIFEST.md.

Open Core Strategy

CC-Framework is currently a public research prototype. Its core mathematical, reporting, and evidence-boundary primitives are intended to remain inspectable for credibility and reproducibility. Potential hosted, enterprise, or customer-specific workflows should remain separate from the research claim boundary. See docs/product/OPEN_CORE_STRATEGY.md.

60-second quickstart

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip wheel setuptools
python -m pip install -e ".[dev]"
python - <<'PY'
from cc.kernel.strict import frechet_bounds

failure_rates = {
    "input_filter": 0.10,
    "policy_judge": 0.10,
}
bounds = frechet_bounds(list(failure_rates.values()), event="and")
independent = failure_rates["input_filter"] * failure_rates["policy_judge"]

print(f"Stacked failure is bounded by [{bounds.lower:.2%}, {bounds.upper:.2%}]")
print(f"Independence would estimate {independent:.2%}")
PY

Expected output:

Stacked failure is bounded by [0.00%, 10.00%]
Independence would estimate 1.00%

Run focused release-core smoke tests from the activated environment:

PYTHONPATH=src python -m pytest -q tests/unit/kernel tests/unit/evidence

For deeper context, start with the public API contract, validation matrix, the finite-sample identification note, and the runnable minimal example.

If you use this software, cite it with the metadata in CITATION.cff. Contributions should follow CONTRIBUTING.md. GitHub Discussions can be enabled from the repository settings before launch.

Research Status

This repository is an active research prototype. Its first-paper core is a finite binary atom kernel for sharp Frechet composition intervals, metric diagnostics, and endpoint witness distributions. It is not a deployment safety certificate, not a product platform, and not an AI alignment solution.

The validation tracks are intentionally separated:

  • Paper Core v0.3 is release-candidate quality in v0.3-rc1 for the finite-atom kernel, canonical metrics, endpoint witnesses, deterministic paper artifacts, and documentation spine.
  • Enterprise Reference v0.1 is an experimental reference architecture for preserving evidence integrity around bundles. It is not the paper core and does not certify deployment safety.

See docs/validation_matrix.md for the command matrix that states which evidence supports each scoped claim.

For the v0.3-rc1 release narrative and checklist, see docs/release/V0_3_RC1_CHECKLIST.md.

Core Thesis

Composed AI safety should be evaluated as a dependence-aware partial-identification problem, not as a product of independent guardrail scores.

The central claim is narrow: many composition claims are underidentified unless dependence among guardrail failures is measured, bounded, or explicitly assumed. The kernel computes what follows from declared assumptions and evidence; it does not validate the upstream data collection process.

What This Is / What This Is Not

What this is:

  • A Python research kernel for finite-atom guardrail failure composition.
  • A partial-identification framework for unknown dependence.
  • A metrics layer for identified-set and product-baseline diagnostics.
  • A witness-oriented reproducibility scaffold for mathematical verification.
  • A research program with explicit non-claims and future-work boundaries.

What this is not:

  • Not a deployment approval system for deployed models.
  • Not proof of deployment safety.
  • Not causal inference without causal assumptions.
  • Not evidence of dataset representativeness or future performance.
  • Not a replacement for red teaming or human governance.
  • Not an enterprise, dashboard, AWS, or adapter-centered project.

Mathematical Object

Let

Z = (Z_1, ..., Z_m),  Z_i in {0, 1}

with the repository-wide convention:

Z_i = 1 means guardrail failure / unsafe pass.

The unknown object is the joint law

pi(z) = P(Z = z),  z in {0, 1}^m.

Evaluation evidence may include singleton failure marginals

p_i = P(Z_i = 1)

and optional pairwise constraints

q_ij = P(Z_i = 1, Z_j = 1).

For a declared Boolean composition event phi(Z), define the feasible Frechet class F as the set of atom distributions satisfying the supplied assumptions and evidence. The kernel computes:

L_phi = inf_{pi in F} E_pi[phi(Z)]
U_phi = sup_{pi in F} E_pi[phi(Z)]

These are sharp bounds conditional on supplied assumptions and evidence.

Implemented Kernel Surface

The current publication-facing kernel surface is intentionally small:

Other modules remain useful but should be described more carefully:

  • src/cc/core, src/cc/exp, and src/cc/cartographer are protocol, workflow, or legacy surfaces.
  • experiments/, scripts/, and notebooks/ are experimental surfaces.
  • apps/dashboard, vendor adapters, and enterprise references are application or demonstration surfaces, not the first-paper core.

Minimal Example

This example uses the current kernel API to identify the probability that at least one declared guardrail failure occurs, given exact singleton failure marginals and no dependence assumption.

from cc.kernel.strict import (
    AssumptionSet,
    LinearQuery,
    fh_width,
    independence_regret,
    independent_event_probability,
)

labels = ("input_filter_failure", "policy_judge_failure")
marginals = {
    "input_filter_failure": 0.08,
    "policy_judge_failure": 0.05,
}

assumptions = AssumptionSet.empty(labels)
for label, value in marginals.items():
    assumptions = assumptions.with_marginal_interval(label, value, value)

query = LinearQuery.union(
    labels,
    labels,
    name="P(any guardrail failure)",
)

result = assumptions.identify(query)
width = fh_width(result.lower_bound, result.upper_bound)
product_baseline = independent_event_probability(marginals, query, labels=labels)
endpoint_regrets = (
    independence_regret(result.lower_bound, product_baseline),
    independence_regret(result.upper_bound, product_baseline),
)

print(result.lower_bound, result.upper_bound)
print(width, product_baseline, endpoint_regrets)

For a runnable reviewer-facing script with witness checks, see examples/minimal/run_bounds.py.

Metrics

The canonical metric taxonomy has four categories:

Category Metrics Interpretation
Identified-set diagnostics fh_width, fh_position Describe the sharp interval [L_phi, U_phi] and where a selected event risk lies inside it.
Assumption-comparison diagnostics independent_event_probability, independence_regret Compare a declared event risk with an explicit product-coupling baseline.
One-world normalization diagnostics cc_gain Normalize a composition failure risk by the largest singleton failure risk in the same setting.
Two-world movement diagnostics cc_shift Summarize movement in composition failure risk relative to singleton failure-rate movement across two settings.

Older names such as cc_max, cc_rel, delta_add, and delta_mult are legacy/deprecated compatibility surfaces. They should not be presented as the front-door theory; see docs/theory/metric_taxonomy.md.

Witnesses and Reproducibility

If the LP reports [L_phi, U_phi], a reproducible result should expose endpoint witness distributions pi_L and pi_U that satisfy the declared constraints and achieve the lower and upper endpoints.

Witnesses verify mathematical feasibility relative to supplied constraints. They do not verify dataset representativeness, causal validity, semantic coverage, or deployment safety.

The repository already exposes endpoint solutions through IdentificationResult.lower_solution and IdentificationResult.upper_solution. Paper-core artifacts can be regenerated with make reproduce-paper and checked with make verify-paper-artifacts. This pipeline verifies deterministic kernel artifacts and endpoint witnesses; it is not a claim that empirical benchmark inputs are representative or deployment-valid.

The current Paper 1 LaTeX source is paper/main.tex. Run make paper-smoke for static source checks and an optional LaTeX build when latexmk is available.

Research Program Documents

Repository Structure

src/cc/kernel/              canonical finite-atom and metric kernel
src/cc/core/                protocol and legacy workflow support
src/cc/exp/                 two-setting experiment runners
src/cc/cartographer/        workflow, reporting, and older atlas utilities
examples/minimal/           smallest runnable atom-LP example
docs/theory/                metric taxonomy, theorem ledger, derivations
docs/research/              research spine, paper core, non-claims, roadmap
experiments/                experimental studies and demonstrations
apps/dashboard/             application surface, not first-paper core

Installation and Validation

Create a local environment and install development dependencies:

python3 -m venv .venv
.venv/bin/pip install --upgrade pip wheel setuptools
.venv/bin/pip install -e '.[dev,docs]'

Runtime support:

  • Local package floor: Python 3.10 or newer.
  • CI code/docs matrix: Python 3.10, 3.11, 3.12, and 3.13.

Run the minimal example:

PYTHONPATH=src .venv/bin/python examples/minimal/run_bounds.py

Kernel validation commands:

PYTHONPATH=src .venv/bin/pytest tests/unit/kernel -q
PYTHONPATH=src .venv/bin/mypy src/cc/kernel --strict
.venv/bin/ruff check src/cc/kernel tests/unit/kernel
.venv/bin/mkdocs build --strict
make paper-smoke

Release-facing validation lanes are documented in docs/validation_matrix.md and summarized in the v0.3-rc1 checklist. The short version: make test-kernel, make test-release, make docs, and PYTHONPATH=src .venv/bin/pytest -q support the v0.3-rc1 evidence record; make reproduce-paper and make verify-paper-artifacts support deterministic paper artifacts. Enterprise, dashboard, security, and vendor checks are separate lanes with their own optional dependencies and non-claims.

Paper 1 benchmark summaries can be generated with:

PYTHONPATH=src .venv/bin/python -m cc.evals.dependence_benchmark \
  --dataset tests/fixtures/dependence_benchmark_harmful.csv \
  --adapters keyword_blocker \
  --keyword-terms jailbreak,exploit \
  --out /tmp/cc-dependence-summary.json

Experimental Surfaces

Experimental and historical material remains in the repository, but it is not the README's central theory.

  • Two-setting diagnostics and older Composition Coefficient ratios are protocol or legacy material. Use cc_shift for the canonical two-setting movement diagnostic.
  • Correlation-cliff experiments are research demonstrations; see experiments/correlation_cliff/README.md and docs/theory/correlation_cliffs.md.
  • Older audit packets, manifests, adapters, dashboards, and cloud references are supporting or application surfaces unless a specific document promotes them into a reviewed contract.

Limitations and Non-Claims

The kernel operates on finite binary failure events and explicit linear constraints. Its conclusions are conditional on those inputs.

Key limitations:

  • Binary event reductions may hide score-level or semantic structure.
  • Estimated marginals and pairwise constraints require their own statistical justification.
  • Pairwise evidence generally does not identify the full joint law.
  • Explicit atom enumeration has scaling limits as m grows.
  • Static finite-atom bounds are not a sequential agent framework.

Non-claims:

  • The project does not prove deployment safety.
  • The project does not certify deployed models.
  • The project does not infer causality without causal assumptions.
  • The project does not establish dataset representativeness or future behavior.
  • The project does not make cryptographic integrity equivalent to statistical validity.

See docs/research/NON_CLAIMS.md for the stricter claim boundary.

Citation

No archival citation exists yet. Until a preprint or release artifact exists, cite the repository URL and commit hash used for the analysis.

License

MIT. See LICENSE.