From 7d9fa3c93a6deffb1eb3b635afca607f9fdaba34 Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Fri, 7 Aug 2026 03:29:00 +0700 Subject: [PATCH 01/41] docs(adr): ADR-0012 persistence reset to SQLite + corrected project state Per senior-review direction (2026-08-07) on feat/sqlite-persistence-reset (clean branch from origin/main at fb5641c; NOT descending from 8a7e916). - ADR-0012: SQLite canonical store with binding contracts (schema, transaction/idempotency, operating mode, canonical serialization + size limits, hash-chain role, public v0.1.2 migration, export contract, salvage list, CI/test contract, versioning, cumulative review lanes, controlled publication scope, threat model). - docs/architecture-reset-status.md: corrected, internally consistent project state for the clean branch. First pushed state is docs only (ADR + state) per reviewer step 12: no SQLite implementation until ADR-0012 is reviewed from the pushed branch. --- docs/adr/ADR-0012-persistence-architecture.md | 364 ++++++++++++++++++ docs/architecture-reset-status.md | 62 +++ 2 files changed, 426 insertions(+) create mode 100644 docs/adr/ADR-0012-persistence-architecture.md create mode 100644 docs/architecture-reset-status.md diff --git a/docs/adr/ADR-0012-persistence-architecture.md b/docs/adr/ADR-0012-persistence-architecture.md new file mode 100644 index 0000000..379a610 --- /dev/null +++ b/docs/adr/ADR-0012-persistence-architecture.md @@ -0,0 +1,364 @@ +# ADR-0012 — Persistence architecture reset: SQLite canonical store + +**Status:** Accepted in principle (2026-08-07 senior-review direction; operator-authorized controlled publication). Implementation pending ADR review from the pushed branch. +**Supersedes:** the JSONL journal-first canonical-store decision in ADR-0008 (and the review-held `8a7e916` remediation branch). +**Applies to:** Method Factory persistence layer, post-v0.1.x overhaul. + +--- + +## Context + +Five validation rounds of the v0.1.x JSONL overhaul exposed that `ManifestStore` had become a bespoke database engine: transactions, compare-and-swap, lock ownership and crash recovery, append framing, torn-write classification, tail repair, snapshot caching, full-chain replay, artifact verification, corruption classification, backward compatibility, and performance optimization. Round 5 identified three divergent "is this committed?" classifiers and two unique release-blocking root causes: + +1. `MAX_ENVELOPE_BYTES` (2 MiB) was misused as a journal-record limit, so a valid committed record larger than 2 MiB (cumulative snapshots grow unboundedly) could be **destroyed** by append-time repair. +2. The healthy-journal tail reader selected the empty bytes after the terminal newline and fell back to a **full-journal read under the exclusive lock** on every CAS (~430× regression). + +The senior reviewer verdict (2026-08-07): the `8a7e916` implementation is permanently review-held and non-releasable; the SQLite architecture is **approved in principle**; controlled branch/PR publication is operator-authorized; the JSONL remediation branch is preserved forensically but not replayed. + +Nothing has shipped (only test/demo stores exist), so the migration cost of an architectural change is effectively zero. + +## Decision + +**Adopt SQLite as the canonical store** (stdlib `sqlite3`, preserving the stdlib-only core). Deterministic JSON/JSONL become **export** formats, not the transactional database. Artifacts remain in the immutable content-addressed blob store. `PipelineEngine` stays independent behind the `ManifestStore` interface. The append-repair helpers of the JSONL branch are **not** carried into the SQLite implementation. + +## 1. SQLite schema (binding properties) + +One canonical immutable event table plus schema metadata. No separate historical manifests table, no mutable head table, no event_json duplicating manifest_json, no package lock files, no append framing, no torn-line repair, no manifest-cache file. + +```sql +CREATE TABLE store_metadata ( + key TEXT PRIMARY KEY, + value TEXT NOT NULL +) WITHOUT ROWID; + +CREATE TABLE events ( + package_id TEXT NOT NULL, + revision INTEGER NOT NULL CHECK (revision >= 0), + + event_id TEXT NOT NULL UNIQUE, + action_id TEXT NOT NULL, + action TEXT NOT NULL, + action_sha256 TEXT NOT NULL, + + state_before TEXT, + state_after TEXT NOT NULL, + + previous_manifest_sha256 TEXT, + resulting_manifest_sha256 TEXT NOT NULL, + + created_at TEXT NOT NULL, + + action_json BLOB NOT NULL, + manifest_json BLOB NOT NULL, + + PRIMARY KEY (package_id, revision), + UNIQUE (package_id, action_id) +) WITHOUT ROWID; +``` + +Exact DDL may evolve in ADR review; these properties are **binding**: + +- One event row = one package revision. +- Revision zero = package-creation event. +- `UNIQUE(package_id, action_id)` enforces package-scoped idempotency. +- `event_id` is globally unique. +- `manifest_json` holds the complete resulting manifest for that revision. +- `action_json` holds canonical normalized action bytes. +- No duplicated event_json containing another copy of manifest_json. +- No mutable manifest-cache file, no package lock files, no append framing, no torn-line repair. +- No materialized package-head table unless profiling proves it necessary. + +Current-state lookup (indexed; composite PK supports it): + +```sql +SELECT manifest_json +FROM events +WHERE package_id = ? +ORDER BY revision DESC +LIMIT 1; +``` + +A read-only `package_heads` view is acceptable. A mutable projection table is not justified for the first release. + +## 2. Transaction and idempotency contract + +Every mutation uses an explicit transaction: + +```text +BEGIN IMMEDIATE + +1. Search for package_id + action_id. +2. If found: + a. same action_sha256 → return the previously committed result; + b. different action_sha256 → ACTION_ID_CONFLICT. +3. Read the latest package revision. +4. Compare it with expected_revision. +5. Validate and canonicalize the action and resulting manifest. +6. Verify required artifact blobs exist. +7. Insert exactly one new event row. +8. COMMIT. +``` + +- There is no separate head update to become inconsistent with the event. +- A competing writer blocks at `BEGIN IMMEDIATE`; after acquiring the transaction it reads the newly committed revision and returns a stale-action result when appropriate. +- Required behavior: + - Same action ID and same action hash: idempotent success (replay prior result). + - Same action ID and different action hash: typed `ACTION_ID_CONFLICT`. + - New action ID and stale revision: typed stale-action failure (`STALE_ACTION`). + - New action ID and current revision: one atomic event insert. + - Any exception before commit: no new revision. + - Any successful commit: the complete new revision exists. +- **Never infer idempotency from `action_id` alone.** + +Note: `ACTION_ID_CONFLICT` supersedes the JSONL-era `ACTION_ID_REUSE` code for the new store; the stable error-code table (ADR-0008) is amended accordingly. + +## 3. SQLite operating mode + +For the first release: + +```sql +PRAGMA journal_mode = DELETE; +PRAGMA synchronous = FULL; +PRAGMA busy_timeout = 5000; +PRAGMA foreign_keys = ON; +``` + +Use explicit `BEGIN IMMEDIATE` transactions. **Do not default to WAL** — single-operator local tooling; WAL adds `-wal`/`-shm` sidecars, checkpoint behavior, more complicated backup, and risk of incomplete evidence capture by copying only the main file. WAL may be reconsidered only after measured reader/writer contention justifies it. + +Also required: + +- Parent store directory mode `0700`; database file mode `0600`. +- A fixed SQLite `application_id` for the store. +- `PRAGMA user_version = 1`; stable failure (`UNSUPPORTED_SCHEMA`) on unsupported future schema versions. +- One connection per operation or thread (no cross-thread connection reuse). +- Explicit read-only mode for validation commands (`mode=ro`). +- Backups via the SQLite backup API or `VACUUM INTO` — never a live raw file copy. +- If `STRICT` tables are adopted, declare and test the minimum supported SQLite version; do not assume every Python 3.11 build ships the same SQLite version. + +## 4. Canonical serialization and size boundaries + +Canonical JSON bytes are produced consistently: + +```python +json.dumps( + value, + sort_keys=True, + separators=(",", ":"), + ensure_ascii=False, + allow_nan=False, +).encode("utf-8") +``` + +Hash those exact bytes. (This amends the JSONL-era `ensure_ascii=True` canonical form; digests are recomputed from canonical bytes on migration.) + +Separate limits explicitly — never reuse `MAX_ENVELOPE_BYTES` as an event or manifest limit: + +- Action Envelope size. +- Action JSON size. +- Manifest size. +- Individual content-field sizes. +- Artifact/blob size. + +**Frozen choice for `summary.content`** (amends Manifest Contract v0.1 / ADR-0004): + +> **Content-addressed summary body.** The manifest stores `summary: {digest, size, preview?}`; the full summary content lives in the immutable blob store. No unbounded inline string embedded in every historical revision. No implicit hybrid. + +## 5. Hash-chain role + +Retain the manifest hash chain for semantic audit continuity, with a narrowed claim: + +> SQLite provides atomicity and transactional durability. The event hash chain provides **internal consistency evidence** across Method Factory revisions. + +The chain is **not** the transaction mechanism. + +Path separation: + +- **Hot path:** indexed latest-event read and validation of the current operation only. +- **`mf validate`:** current database/schema and current package checks. +- **`mf validate --full`:** complete revision-chain, manifest-hash, action-hash, and artifact verification. +- **Migration and release evidence:** always run full validation. + +This prevents an O(J) or O(J²) mutation path. Document that an unkeyed chain: + +- does not prove cryptographic authenticity; +- does not detect an attacker replacing and rehashing the whole database; +- does not independently detect rollback to an older internally valid database. + +Use "internal consistency evidence," not "tamper-proof." + +## 6. Public v0.1.2 migration + +The JSONL store at public commit `fb5641c` (tag `v0.1.2-integrity`) was published. A legacy migration path is therefore required, implemented outside the normal SQLite store: + +```text +methodfactory/migrations/v012_jsonl.py +``` + +Required command: `mf migrate-store` + +Required behavior: + +1. Detect the public v0.1.2 JSONL layout. +2. Open it read-only. +3. Validate the complete legacy chain using a **frozen legacy reader**. +4. Never repair or alter the legacy files. +5. Import into a **temporary** SQLite database. +6. Recompute canonical hashes from imported content. +7. Run SQLite integrity and full Method Factory validation. +8. Atomically rename the completed destination database. +9. Produce a migration receipt: source format; source file SHA-256 values; package count; event count; destination schema version; resulting validation verdict. +10. Preserve the original store until the operator explicitly archives or removes it. + +When a legacy store is detected, normal startup must **not** migrate silently — return a stable `LEGACY_STORE_DETECTED` instruction. + +Do **not** place experimental `8a7e916` repair logic in the production migration path. Preserve that history forensically. + +## 7. Export contract + +Do not freeze the failed journal's exact bytes as the primary public contract. Two explicit formats: + +**Supported format — `method-factory-events-v1`** + +- versioned; deterministic; UTF-8; LF line endings; one event per line; canonical key ordering; exactly one final newline; package/revision ordering documented; generated inside a consistent read transaction. + +**Legacy evidence format — `legacy-v012-jsonl`** + +- reconstructs the old public event shape for compatibility and evidence comparison. + +The supported export promises deterministic output for the same database and exporter version. It does not promise byte identity with every historical journal produced during the abandoned remediation branch. Export does not automatically imply import; any import command must be separately specified, fully validated, and atomic. + +## 8. Salvage versus discard from `8a7e916` + +Preserve the entire forensic branch; do not replay it wholesale. + +**Port or reimplement:** + +- `core` → `methodfactory` package rename. +- `pyproject.toml` and `mf` entry point. +- Immutable content-addressed artifact store. +- Logical-path protections. +- Stable CLI error codes. +- Envelope bounds and control-character validation. +- Schema type hardening. +- Stale Process Engine quarantine. +- Pinned GitHub Actions and least-privilege workflow permissions. +- `.mf/`, egg-info, database, and build-artifact ignores. +- Packaging smoke tests. +- Documentation improvements that remain true under SQLite. + +**Do not port:** + +- Lock-file ownership or stale-lock recovery. +- Append repair. +- Tail classifiers. +- Newline framing. +- Tail-size heuristics. +- `_read_last_event`. +- Manifest cache reconciliation. +- JSONL CAS implementation. +- Round-number tests whose only purpose is the abandoned storage mechanics. + +Rename surviving tests by invariant (e.g. `test_sqlite_transactions.py`, `test_store_idempotency.py`, `test_store_concurrency.py`, `test_store_migration_v012.py`, `test_event_export.py`, `test_store_fault_injection.py`, `test_store_security_boundaries.py`). + +## 9. CI and test evidence + +Canonical release-gate command (single, used everywhere): + +```bash +python -m unittest discover -s methodfactory/tests -t . +``` + +Hypothesis supports unittest; no pytest conversion required. + +Test dependencies (central): + +```toml +[project.optional-dependencies] +test = [ + "hypothesis>=6", + "PyYAML>=6", +] +``` + +CI: + +```bash +pip install -e ".[test]" +python -m unittest discover -s methodfactory/tests -t . +``` + +Test Python 3.11 and 3.12 (if both remain declared supported). + +Required test classes: + +1. Reference-model/state-machine property tests. +2. Transaction interruption before and after commit. +3. Separate-process concurrency and stale revision. +4. Idempotent action retry. +5. Conflicting action-ID reuse. +6. Public v0.1.2 migration fixtures. +7. Deterministic export fixtures. +8. Unsupported schema-version behavior. +9. Corrupt/truncated SQLite database behavior. +10. Missing or corrupt artifact blobs. +11. Large input and manifest bounds. +12. Clean-install CLI tests. +13. No committed `.mf`, SQLite, WAL, SHM, journal, egg-info, or build output. +14. Indexed latest-event query proof and representative performance test. + +Do not use a fragile sub-second CI latency threshold as the only performance gate. Also assert the query plan uses the package/revision index and does not scan the complete history. + +## 10. Versioning + +Vincent retains the `2.0.0` generation branding, with honest prerelease progression: + +```text +pyproject: 2.0.0a1 Git tag later: v2.0.0-alpha.1 +pyproject: 2.0.0rc1 Git tag later: v2.0.0-rc.1 +final only after trials: 2.0.0 / v2.0.0 +``` + +No tag is authorized now. The eventual release notes must explain that "2.0" denotes the Process Engine → Method Factory architectural generation, rather than implying an earlier released Method Factory 1.x API. + +## 11. Cumulative review lanes + +One cumulative review of the clean candidate over `fb5641c..release-candidate`, lanes: + +1. Storage and transaction correctness. +2. Fault injection and recovery. +3. Public v0.1.2 migration, export, backup, and restore. +4. State-machine legality. +5. Security and resource boundaries. +6. CLI/API compatibility and stable error model. +7. Performance, packaging, and documentation. + +Review the resulting candidate architecture, not removed JSONL code preserved only on the forensic branch. + +Release requires: zero unresolved critical or major durability, concurrency, integrity, migration, or security defects; exact GitHub candidate SHA; all required checks green on that SHA; full validation green; clean-install proof; migration proof; docs and ADRs matching code; no runtime/build artifacts; operator approval. A review-round count never overrides an open release blocker. + +## 12. Controlled publication scope (2026-08-07) + +Operator-authorized, evidence and development-visibility only: + +- Push `review/jsonl-overhaul-8a7e916` at exact `8a7e9167d6ff77b3ccd32722683c9b42e4390687` (forensic; no PR required). +- Push `feat/sqlite-persistence-reset` from `origin/main` (`fb5641c`); open a **draft PR** into `main` (DO NOT MERGE). +- Do not push local `main`; do not change remote `main`; do not merge; do not create a release; do not create any final or release-candidate tag; do not force-push published branches; do not commit the `.bundle`, SQLite databases, `.mf/`, secrets, runtime stores, or local operational facts into the product repository. + +## Threat model + +| Threat | Guarantee | +|---|---| +| Process crash | SQLite ACID; uncommitted work lost, committed work intact | +| Host power loss | SQLite durable commits (journal_mode DELETE + synchronous FULL); same as above | +| Concurrent sanctioned writers | SQLite write serialization + revision predicate → typed `STALE_ACTION` | +| Accidental file corruption | `integrity_check` on open; typed `MANIFEST_INVALID`; no auto-repair | +| External local tampering | Hash chain in export = **internal consistency evidence only**, not cryptographic authenticity without an anchored/signed root | +| Malicious local writers | Out of scope for single-operator local tool; documented (a local attacker with store write access can rewrite the DB and recompute ordinary hashes) | + +## Consequences + +- Deletes the bespoke JSONL failure surface (torn writes, framing, stale locks, tail repair, full-file mutation reads, divergent classifiers). +- Keeps `PipelineEngine` independent; `SQLiteManifestStore` implements the `ManifestStore` interface (create/apply, idempotent replay, read_events for export). +- Export formats preserve the audit/evidence story; `method-factory-events-v1` is the supported public contract. +- Amends: ADR-0004 (summary.content → content-addressed body), ADR-0008 (persistence mechanics + error-code table). ADR-0001..0003, 0005..0007, 0009..0010 remain in force. +- The `8a7e916` branch is preserved forensically on `review/jsonl-overhaul-8a7e916`; its reuse is limited to the Section 8 port list. diff --git a/docs/architecture-reset-status.md b/docs/architecture-reset-status.md new file mode 100644 index 0000000..08a85cf --- /dev/null +++ b/docs/architecture-reset-status.md @@ -0,0 +1,62 @@ +# Architecture Reset — Project State (2026-08-07) + +**Status:** SQLite architecture approved in principle; controlled publication in progress. This is the corrected project-state document for the `feat/sqlite-persistence-reset` branch. + +## Branch topology + +```text +fb5641c remote main (published v0.1.2-integrity base) +├── review/jsonl-overhaul-8a7e916 # forensic — exact 8a7e916, review-held, non-releasable +│ └── ... twelve JSONL remediation commits ... → 8a7e916 +│ +└── feat/sqlite-persistence-reset # clean product branch (this branch) — from origin/main + ├── ADR-0012 and architecture contracts ← first pushed state (this commit) + ├── selected reusable v2 foundation ← later (port list in ADR-0012 §8) + ├── SQLite implementation ← after ADR review + └── invariant-driven tests ← after ADR review +``` + +## Identities (verified 2026-08-07) + +| Item | Value | +|---|---| +| Remote main | `fb5641cc1a3f1f54b96bba3af88ec5a1b010f4e5` (untouched) | +| Forensic branch | `review/jsonl-overhaul-8a7e916` = `8a7e9167d6ff77b3ccd32722683c9b42e4390687` | +| Clean branch | `feat/sqlite-persistence-reset` (merge-base with origin/main = `fb5641c`; does **not** descend from 8a7e916) | +| Git bundle | `method-factory-8a7e916.bundle` (SHA-256 `92c0bb1026190f9fd5e1f61bb4cd5fc16a08605aecf77f81eb5bc93b3b504f63`; `git bundle verify` OK) | +| Local archival | `persistence-reset` branch preserved locally (contains pre-revision ADR-0012 draft, not published) | + +## Reviewer direction (2026-08-07) — accepted + +- **NO-GO** on publishing the `8a7e916` JSONL implementation; permanently review-held. +- **GO** on SQLite canonical store (approved in principle), behind the `ManifestStore` interface. +- **Controlled publication authorized** (evidence/visibility only): forensic branch + clean branch + draft PR; no merge, no tag, no release, no `main` change. +- Phase 0/1 evidence from the earlier archive was **not accepted** (bundle lacked the git bundle, ADR-0012, checkpoint, branch proof). Corrected once in the updated evidence package; no further JSONL remediation loop. + +## What this branch contains now (first pushed state) + +- `docs/adr/ADR-0012-persistence-architecture.md` — the architecture decision with the binding contracts: + - one canonical immutable `events` table + `store_metadata` (schema §1); + - transaction + idempotency contract (`BEGIN IMMEDIATE`, `ACTION_ID_CONFLICT`, §2); + - operating mode (DELETE/FULL/busy_timeout, no WAL, 0700/0600, `application_id`, `user_version=1`, read-only validation, backup API, §3); + - canonical serialization (`ensure_ascii=False`) and separate size limits; `summary.content` frozen to content-addressed body (§4); + - hash chain narrowed to internal-consistency evidence; `mf validate` vs `--full` (§5); + - public v0.1.2 migration contract (`mf migrate-store`, receipt, `LEGACY_STORE_DETECTED`, §6); + - export contract (`method-factory-events-v1` + `legacy-v012-jsonl`, §7); + - salvage/discard list from `8a7e916` (§8); + - CI/test contract (unittest canonical gate, Hypothesis, 14 test classes, §9); + - versioning (`2.0.0a1 → rc1 → 2.0.0`, no tag now, §10); + - cumulative review lanes and release requirements (§11); + - controlled publication scope (§12); threat model. +- `docs/architecture-reset-status.md` — this document. + +## Not yet on this branch (port order, per ADR-0012 §8) + +The `core`→`methodfactory` rename, packaging, artifact store, path protections, CLI error codes, envelope bounds, schema hardening, quarantine, CI pins, ignores, packaging tests, and docs that remain true under SQLite are **selected reusable v2 foundation** — ported in a later step, **after ADR-0012 is reviewed from this pushed branch**. No SQLite implementation code is present yet by design (reviewer's step 12: do not begin full persistence implementation until ADR-0012 has been reviewed from the actual pushed branch). + +## Gates + +1. ADR-0012 reviewed from the pushed branch (this PR). +2. Operator GO to proceed with foundation port + SQLite implementation. +3. Cumulative release-candidate review (`fb5641c..rc`, 7 lanes, ADR-0012 §11). +4. Operator approval to merge / tag / release (not currently authorized). From 84ff725ae010575b79b23890c4aee79c10d82944 Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Fri, 7 Aug 2026 04:30:27 +0700 Subject: [PATCH 02/41] docs: finalize ADR-0012 persistence contracts Closes all 12 senior-review items (review 4878235332 on PR #1) in an ADR-0012 amendment and aligns docs/architecture-reset-status.md: 1. Publication fact: public v0.1.2 exists; no production user stores known (evidence limitation, not proof of absence); migration retained. 2. Open/validation modes: hot path schema/app-id/user-version only (no integrity_check); mf validate = quick_check + current-package; mf validate --full = integrity_check + full chain/hash/artifact. 3. Durability qualified: subject to OS/filesystem/storage honesty. 4. Physical DB contract: methodfactory.sqlite3 under store root; application_id 0x4D465354; user_version 1; full state table; read-only URI; no accidental creation during validation. 5. Append-only executable: UPDATE/DELETE rejection triggers in binding DDL. 6. Revision/chain invariants frozen (rev0 create, predecessor, digests, column agreement, one authoritative validator). 7. action_sha256 = hash of canonical {action, package_id, action_id, basis, payload}; excludes only expected_revision. 8. Artifact boundary: blobs before txn, orphan-safe, no auto-delete during mutation; GC must prove global unreachability. 9. Migration: v0.1.2 layout, explicit source/dest, fail-closed dest, same-filesystem atomic rename, receipt durable and part of success. 10. Evidence checksum: archive-root-relative SHA256SUMS, one command exits 0. 11. Clean worktree for next evidence capture. 12. Architecture CI honest: run 31127787460 cancelled = unproven until a run succeeds on the exact branch SHA. Phase 2 stop gate: commits 2-4 + CI evidence + PR comment, then request the next senior review. No lifecycle implementation in this commit. --- docs/adr/ADR-0012-persistence-architecture.md | 115 +++++++++++++++++- docs/architecture-reset-status.md | 75 +++++++----- 2 files changed, 156 insertions(+), 34 deletions(-) diff --git a/docs/adr/ADR-0012-persistence-architecture.md b/docs/adr/ADR-0012-persistence-architecture.md index 379a610..808c669 100644 --- a/docs/adr/ADR-0012-persistence-architecture.md +++ b/docs/adr/ADR-0012-persistence-architecture.md @@ -15,7 +15,7 @@ Five validation rounds of the v0.1.x JSONL overhaul exposed that `ManifestStore` The senior reviewer verdict (2026-08-07): the `8a7e916` implementation is permanently review-held and non-releasable; the SQLite architecture is **approved in principle**; controlled branch/PR publication is operator-authorized; the JSONL remediation branch is preserved forensically but not replayed. -Nothing has shipped (only test/demo stores exist), so the migration cost of an architectural change is effectively zero. +**Publication fact (precise):** public v0.1.2 exists — the JSONL store at commit `fb5641c` was publicly tagged `v0.1.2-integrity`. No production user stores are known to the project from available evidence; absence of known real stores is **not** inferred as proof that none exist. Because v0.1.2 was publicly tagged, a legacy migration path is retained (Section 6). The migration cost of the architectural change is effectively zero for known project stores; it is non-zero for any hypothetical real store, which is why migration compatibility is mandatory. ## Decision @@ -362,3 +362,116 @@ Operator-authorized, evidence and development-visibility only: - Export formats preserve the audit/evidence story; `method-factory-events-v1` is the supported public contract. - Amends: ADR-0004 (summary.content → content-addressed body), ADR-0008 (persistence mechanics + error-code table). ADR-0001..0003, 0005..0007, 0009..0010 remain in force. - The `8a7e916` branch is preserved forensically on `review/jsonl-overhaul-8a7e916`; its reuse is limited to the Section 8 port list. + +--- + +## Amendment - Phase 2 finalization (2026-08-07) + +Closes the twelve review items from senior review `4878235332` on PR #1. This amendment is binding before implementation proceeds to commits 2-4. + +### A. Publication fact and migration posture (item 1) + +Public v0.1.2 exists (`v0.1.2-integrity` at `fb5641c`). No production user stores are known to the project from available evidence; this is stated as an evidence limitation, **not** as proof none exist. Migration compatibility is therefore mandatory (Section 6) and unchanged. + +### B. Open and validation modes (item 2) + +| Mode | Checks | +|---|---| +| Normal hot-path open | Schema/application-ID/user-version checks; errors naturally raised by SQLite on use. **No `PRAGMA integrity_check`.** | +| `mf validate` | Bounded: `PRAGMA quick_check` + current database/schema checks + current-package checks (latest manifest, artifact digests for the current package). | +| `mf validate --full` | `PRAGMA integrity_check` + full event-chain, manifest-hash, action-hash, and artifact verification across every revision. | + +The hot path must never run an O(DB) integrity scan; this preserves the ADR's own O(J)-avoidance goal. + +### C. Durability qualification (item 3) + +SQLite with `journal_mode=DELETE` + `synchronous=FULL` provides transactional durability subject to the honesty of the OS, filesystem, and storage hardware. It cannot override lying hardware, filesystem faults, or hostile host administration. All durability claims are qualified accordingly; the threat model in Section 12 is the limit of the guarantee. + +### D. Physical database identity and open contract (item 4) + +- Canonical filename: `methodfactory.sqlite3`. +- Canonical location: directly beneath the store root (`/methodfactory.sqlite3`). +- Fixed `application_id`: `0x4D465354` (decimal `1297248084`, ASCII "MFST"). +- Accepted `user_version`: `1`. Any other value is unsupported. + +| Store-root state | Normal open (rw) | Validation (ro) | +|---|---|---| +| Missing DB, no legacy | Create + initialize new database | `DATABASE_NOT_FOUND` (no creation) | +| Zero-byte `methodfactory.sqlite3` | Initialize (SQLite empty-file semantics; documented as indistinguishable from first creation) | `DATABASE_EMPTY` (no creation) | +| Wrong application ID (foreign/uninitialized DB) | `DATABASE_ID_MISMATCH` | `DATABASE_ID_MISMATCH` | +| Future `user_version` (>1) | `UNSUPPORTED_SCHEMA` | `UNSUPPORTED_SCHEMA` | +| Corrupt DB (fails `quick_check`) | SQLite raises naturally on use; typed `MANIFEST_INVALID` in validate | `MANIFEST_INVALID` | +| Legacy-only (v0.1.2 `packages/`+`events/`, no SQLite) | `LEGACY_STORE_DETECTED` → instruct `mf migrate-store` | `LEGACY_STORE_DETECTED` | +| SQLite-only | Open normally | Open read-only | +| Neither | Create + initialize | `DATABASE_NOT_FOUND` | +| Both present | SQLite is canonical and used; legacy preserved untouched | Same; validate may note legacy presence | + +- Read-only validation opens via URI `file:?mode=ro` and must never create or modify the database. +- No validation or read-only command may create a database accidentally; any path that would create one fails with the appropriate typed error. + +### E. Append-only is executable (item 5) + +The binding DDL includes `BEFORE UPDATE` and `BEFORE DELETE` triggers on `events` that `RAISE(ABORT, ...)`. Immutability is enforced by the database, not only by repository discipline. Schema migrations create a new database/table version rather than mutating historical rows. + +### F. Revision and chain invariants (item 6) + +One authoritative validator (owned by the storage layer, exercised on every transactional apply) enforces: + +- Revision 0 is the package-creation event; its action is `create_package` and `state_before IS NULL`. +- Revision > 0 has exactly one predecessor (revision − 1) present. +- `state_before` equals the predecessor's `state_after`. +- `previous_manifest_sha256` equals the predecessor's `resulting_manifest_sha256`. +- Manifest `package_id`, `revision`, and `state` fields agree with the indexed SQL columns. +- These may be application-validated transaction invariants (not SQL triggers), but there is exactly one authoritative validator and it is tested. + +### G. Canonical action hash semantics (item 7) + +`action_sha256` covers the complete normalized semantic request used for idempotency: + +```python +action_sha256 = sha256_hex(canonical_json({ + "action": action, + "package_id": package_id, + "action_id": action_id, + "basis": basis, + "payload": payload, +})) +``` + +- It includes every field that could change the requested outcome. +- It excludes **only** `expected_revision` (optimistic-concurrency/transport metadata, not part of the requested outcome). +- Same `action_id` + same hash → idempotent replay. Same `action_id` + different hash → `ACTION_ID_CONFLICT`. Never infer idempotency from `action_id` alone. + +### H. Artifact write boundary and orphan safety (item 8) + +- Blob writes occur before the SQLite transaction, are content-addressed, immutable, and verified to exist before the event referencing them is inserted. +- Orphaned blobs (written but never referenced by a committed event) are harmless and retained. +- **No automatic blob deletion during mutation.** +- Any future garbage collection is a separate, conservative process that must prove a digest is unreachable from every committed event before deletion. + +### I. Migration source selection and atomic destination (item 9) + +- Accepted v0.1.2 layout: `/packages/`, `/events/`, `/artifacts/`. +- `mf migrate-store` accepts explicit `--source` and `--dest`; deterministic defaults are source = the detected legacy store root and destination = `/methodfactory.sqlite3`. +- Destination behavior is fail-closed: if the destination already exists, migration refuses (typed error) - no overwrite of source or destination. +- The destination must be on the same filesystem as its directory for the atomic rename; cross-device rename fails with a typed error. +- The migration receipt (source format, source file SHA-256s, package count, event count, destination schema version, validation verdict) is written durably (fsync) and is part of migration success - migration is not considered successful until the receipt is durable. +- The original store is preserved until the operator explicitly archives or removes it. + +### J. Evidence checksum convention (item 10) + +Evidence packages use archive-root-relative paths in `SHA256SUMS`. The single verification command, run from the archive root, exits zero: + +```bash +cd && sha256sum -c SHA256SUMS +``` + +The next evidence capture follows this convention. + +### K. Clean worktree for evidence capture (item 11) + +Before any evidence capture, the local worktree must be clean: the `.gitignore` is in force (Section 8 port list), generated artifacts (`.mf/`, egg-info, build output, test caches, SQLite sidecars) are removed, and `git status --short` is empty. + +### L. Architecture CI honesty (item 12) + +As of this amendment, architecture CI is **unproven**: run `31127787460` was cancelled without executing steps. It is neither failed nor passed. CI is considered evidence only after a run executes successfully on the exact branch SHA. The Phase 2 submission runs CI on the exact final head SHA and reports the run URL and conclusion. diff --git a/docs/architecture-reset-status.md b/docs/architecture-reset-status.md index 08a85cf..1c1a504 100644 --- a/docs/architecture-reset-status.md +++ b/docs/architecture-reset-status.md @@ -1,6 +1,6 @@ -# Architecture Reset — Project State (2026-08-07) +# Architecture Reset — Project State (2026-08-07, Phase 2) -**Status:** SQLite architecture approved in principle; controlled publication in progress. This is the corrected project-state document for the `feat/sqlite-persistence-reset` branch. +**Status:** SQLite architecture approved in principle; senior review `4878235332` on PR #1 completed the architecture review and directed the Phase 2 implementation order (commits 1–4) with a design-convergence stop gate. This document tracks the clean `feat/sqlite-persistence-reset` branch. ## Branch topology @@ -10,10 +10,11 @@ fb5641c remote main (published v0.1.2-integrity base) │ └── ... twelve JSONL remediation commits ... → 8a7e916 │ └── feat/sqlite-persistence-reset # clean product branch (this branch) — from origin/main - ├── ADR-0012 and architecture contracts ← first pushed state (this commit) - ├── selected reusable v2 foundation ← later (port list in ADR-0012 §8) - ├── SQLite implementation ← after ADR review - └── invariant-driven tests ← after ADR review + ├── ADR-0012 + architecture contracts ← commits 1 (docs) — done + ├── package foundation (rename, CI, ignores) ← commit 2 + ├── storage protocol + canonical primitives ← commit 3 + ├── SQLite schema creation + identity + append-only guards ← commit 4 (Phase 2 stop gate) + └── (later, after gate) transactional apply, migration, exports, lifecycle ``` ## Identities (verified 2026-08-07) @@ -23,40 +24,48 @@ fb5641c remote main (published v0.1.2-integrity base) | Remote main | `fb5641cc1a3f1f54b96bba3af88ec5a1b010f4e5` (untouched) | | Forensic branch | `review/jsonl-overhaul-8a7e916` = `8a7e9167d6ff77b3ccd32722683c9b42e4390687` | | Clean branch | `feat/sqlite-persistence-reset` (merge-base with origin/main = `fb5641c`; does **not** descend from 8a7e916) | +| PR #1 | https://github.com/RedEyeNinja-BKK/Method-Factory/pull/1 — **actual GitHub Draft** (reviewer converted it), DO NOT MERGE | | Git bundle | `method-factory-8a7e916.bundle` (SHA-256 `92c0bb1026190f9fd5e1f61bb4cd5fc16a08605aecf77f81eb5bc93b3b504f63`; `git bundle verify` OK) | -| Local archival | `persistence-reset` branch preserved locally (contains pre-revision ADR-0012 draft, not published) | +| Local archival | `persistence-reset` branch preserved locally (pre-revision ADR draft, not published) | -## Reviewer direction (2026-08-07) — accepted +## Senior review 4878235332 (2026-08-07) — accepted -- **NO-GO** on publishing the `8a7e916` JSONL implementation; permanently review-held. -- **GO** on SQLite canonical store (approved in principle), behind the `ManifestStore` interface. -- **Controlled publication authorized** (evidence/visibility only): forensic branch + clean branch + draft PR; no merge, no tag, no release, no `main` change. -- Phase 0/1 evidence from the earlier archive was **not accepted** (bundle lacked the git bundle, ADR-0012, checkpoint, branch proof). Corrected once in the updated evidence package; no further JSONL remediation loop. +- SQLite reset remains **APPROVED IN PRINCIPLE**; corrected evidence package closes the prior evidence gap. +- PR #1 converted to an actual GitHub Draft. +- **One focused ADR amendment commit first** closing 12 review items, then the bounded implementation order: + 1. `docs: finalize ADR-0012 persistence contracts` + 2. `chore: establish methodfactory package, test extras, CI matrix, and ignores` + 3. `refactor: introduce storage protocol and canonical serialization primitives` + 4. `feat: add SQLite schema creation, identity checks, and append-only guards` + 5–8. (later) transactional apply; migration + exports; test evidence; docs alignment. +- **Phase 2 stop gate:** after commits 1–4, return head SHA, ADR diff summary, DDL + triggers, database-open state table, action-hash definition, successful CI run on the exact SHA, local unit results, `EXPLAIN QUERY PLAN`, clean `git status --short`, and confirmation of no merge/tag/release/`main` change. Do not proceed to the full lifecycle until the senior reviewer accepts this gate. -## What this branch contains now (first pushed state) +## ADR-0012 amendment (commit 1, done) -- `docs/adr/ADR-0012-persistence-architecture.md` — the architecture decision with the binding contracts: - - one canonical immutable `events` table + `store_metadata` (schema §1); - - transaction + idempotency contract (`BEGIN IMMEDIATE`, `ACTION_ID_CONFLICT`, §2); - - operating mode (DELETE/FULL/busy_timeout, no WAL, 0700/0600, `application_id`, `user_version=1`, read-only validation, backup API, §3); - - canonical serialization (`ensure_ascii=False`) and separate size limits; `summary.content` frozen to content-addressed body (§4); - - hash chain narrowed to internal-consistency evidence; `mf validate` vs `--full` (§5); - - public v0.1.2 migration contract (`mf migrate-store`, receipt, `LEGACY_STORE_DETECTED`, §6); - - export contract (`method-factory-events-v1` + `legacy-v012-jsonl`, §7); - - salvage/discard list from `8a7e916` (§8); - - CI/test contract (unittest canonical gate, Hypothesis, 14 test classes, §9); - - versioning (`2.0.0a1 → rc1 → 2.0.0`, no tag now, §10); - - cumulative review lanes and release requirements (§11); - - controlled publication scope (§12); threat model. -- `docs/architecture-reset-status.md` — this document. +The amendment closes all 12 review items: -## Not yet on this branch (port order, per ADR-0012 §8) +1. Publication fact — public v0.1.2 exists; no production user stores known (evidence limitation, not proof of absence); migration compatibility retained. +2. Open/validation modes — hot path: schema/app-id/user-version only, no `integrity_check`; `mf validate`: `quick_check` + current-package; `--full`: `integrity_check` + full chain/hash/artifact. +3. Durability qualified — subject to OS/filesystem/storage honesty; DELETE+FULL cannot override lying hardware/hostile host. +4. Physical DB contract — `methodfactory.sqlite3` under store root; `application_id` `0x4D465354`; `user_version` 1; full state table (missing/zero-byte/wrong-ID/future-version/corrupt/legacy-only/sqlite-only/neither/both); read-only URI; no accidental creation. +5. Append-only executable — UPDATE/DELETE rejection triggers in binding DDL. +6. Revision/chain invariants frozen — rev 0 create + `state_before IS NULL`; predecessor required; state/digest match; manifest columns agree; one authoritative validator. +7. `action_sha256` defined — hash of canonical `{action, package_id, action_id, basis, payload}`; excludes only `expected_revision`. +8. Artifact boundary — blobs before txn, content-addressed, verified before insert, orphan-safe; no auto-delete during mutation; GC proves global unreachability. +9. Migration — v0.1.2 layout; explicit source/dest; fail-closed existing dest; same-filesystem atomic rename; no overwrite; receipt durable and part of success. +10. Evidence checksum — archive-root-relative `SHA256SUMS`; `cd && sha256sum -c SHA256SUMS` exits zero. +11. Clean worktree for next evidence capture. +12. Architecture CI honest — run `31127787460` was cancelled; CI unproven until a run succeeds on the exact SHA. -The `core`→`methodfactory` rename, packaging, artifact store, path protections, CLI error codes, envelope bounds, schema hardening, quarantine, CI pins, ignores, packaging tests, and docs that remain true under SQLite are **selected reusable v2 foundation** — ported in a later step, **after ADR-0012 is reviewed from this pushed branch**. No SQLite implementation code is present yet by design (reviewer's step 12: do not begin full persistence implementation until ADR-0012 has been reviewed from the actual pushed branch). +## CI state (honest) + +- Run `31127787460` (workflow_dispatch on `7d9fa3c`): **cancelled without executing steps** — recorded, treated as unproven (ADR item 12). +- Phase 2 will run CI on the exact final head SHA (after commits 1–4 pushed) and report the run URL and conclusion. The canonical gate is `python -m unittest discover -s methodfactory/tests -t .` (ADR-0012 §9). ## Gates -1. ADR-0012 reviewed from the pushed branch (this PR). -2. Operator GO to proceed with foundation port + SQLite implementation. -3. Cumulative release-candidate review (`fb5641c..rc`, 7 lanes, ADR-0012 §11). -4. Operator approval to merge / tag / release (not currently authorized). +1. ✅ ADR-0012 reviewed from the pushed branch (senior review 4878235332). +2. ✅ Operator GO to proceed with foundation + SQLite implementation (Phase 2 authorization). +3. ⏳ Phase 2 stop gate: commits 1–4 + CI evidence + PR comment; senior reviewer acceptance. +4. Cumulative release-candidate review (`fb5641c..rc`, 7 lanes, ADR-0012 §11). +5. Operator approval to merge / tag / release (not currently authorized). From ce05625f7ca758e27606888d8dde350f20bcebba Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Fri, 7 Aug 2026 05:11:46 +0700 Subject: [PATCH 03/41] chore: establish methodfactory package, test extras, CI matrix, and ignores MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 2 foundation commit (ADR-0012 §8 salvage list; review 4878235332 implementation order step 2). From clean base fb5641c: - core -> methodfactory package rename (reusable foundation only). - REMOVED JSONL storage machinery per ADR-0012 §8 discard list: methodfactory/manifest/store.py, methodfactory/engine.py, test_engine.py, test_integrity.py (no JSONL locks/repair/tail/cache reconciliation/round-number storage tests ported). - Kept + import-fixed: domain (states/transitions/errors/gates), protocol envelope, adapters artifact_store (immutable content-addressed blobs), manifest hashing/schema/render, prompt-conformance tests. - Minimal CLI: mf --version = methodfactory 2.0.0a1; other commands return a clear 'persistence reset in progress' notice (full CLI returns later). - pyproject.toml: version 2.0.0a1, console_scripts mf, [test] extras (hypothesis>=6, PyYAML>=6), setuptools==68.2.2 pinned build backend, requires-python >=3.11 (3.11/3.12 classifiers). - CI (release-gate.yml): Python 3.11 + 3.12 matrix, pinned action SHAs (checkout v4.4.0, setup-python v5.6.0), permissions contents: read, canonical gate 'python -m unittest discover -s methodfactory/tests -t .', packaging smoke (mf --version + python -m methodfactory --version), no-runtime-artifacts check (no .mf/, clean git status). - .gitignore: .mf/, SQLite + sidecars, egg-info, build/dist, pytest cache. - tests: ported storage-independent suites + test_packaging.py (version, entry point, import surface, JSONL engine absent). Local 3.11: 72 tests OK; mf --version OK. Python 3.12 CI-verified only (python3.12-venv unavailable locally; CI matrix covers it). --- .github/workflows/release-gate.yml | 52 ++- .gitignore | 18 + core/__init__.py | 11 - core/cli.py | 134 -------- core/engine.py | 241 -------------- core/manifest/store.py | 290 ---------------- core/tests/test_engine.py | 311 ------------------ core/tests/test_integrity.py | 162 --------- methodfactory/__init__.py | 15 + methodfactory/__main__.py | 10 + {core => methodfactory}/adapters/__init__.py | 0 .../adapters/artifact_store.py | 0 methodfactory/cli.py | 39 +++ {core => methodfactory}/domain/__init__.py | 0 {core => methodfactory}/domain/errors.py | 0 {core => methodfactory}/domain/gates.py | 0 {core => methodfactory}/domain/states.py | 0 {core => methodfactory}/domain/transitions.py | 0 methodfactory/manifest/__init__.py | 5 + {core => methodfactory}/manifest/hashing.py | 0 {core => methodfactory}/manifest/render.py | 0 {core => methodfactory}/manifest/schema.py | 0 {core => methodfactory}/protocol/__init__.py | 0 {core => methodfactory}/protocol/envelope.py | 0 {core => methodfactory}/tests/__init__.py | 0 {core => methodfactory}/tests/test_domain.py | 4 +- .../tests/test_envelope.py | 4 +- .../tests/test_manifest.py | 6 +- methodfactory/tests/test_packaging.py | 51 +++ .../tests/test_prompt_conformance.py | 6 +- pyproject.toml | 33 ++ 31 files changed, 203 insertions(+), 1189 deletions(-) delete mode 100644 core/__init__.py delete mode 100644 core/cli.py delete mode 100644 core/engine.py delete mode 100644 core/manifest/store.py delete mode 100644 core/tests/test_engine.py delete mode 100644 core/tests/test_integrity.py create mode 100644 methodfactory/__init__.py create mode 100644 methodfactory/__main__.py rename {core => methodfactory}/adapters/__init__.py (100%) rename {core => methodfactory}/adapters/artifact_store.py (100%) create mode 100644 methodfactory/cli.py rename {core => methodfactory}/domain/__init__.py (100%) rename {core => methodfactory}/domain/errors.py (100%) rename {core => methodfactory}/domain/gates.py (100%) rename {core => methodfactory}/domain/states.py (100%) rename {core => methodfactory}/domain/transitions.py (100%) create mode 100644 methodfactory/manifest/__init__.py rename {core => methodfactory}/manifest/hashing.py (100%) rename {core => methodfactory}/manifest/render.py (100%) rename {core => methodfactory}/manifest/schema.py (100%) rename {core => methodfactory}/protocol/__init__.py (100%) rename {core => methodfactory}/protocol/envelope.py (100%) rename {core => methodfactory}/tests/__init__.py (100%) rename {core => methodfactory}/tests/test_domain.py (97%) rename {core => methodfactory}/tests/test_envelope.py (98%) rename {core => methodfactory}/tests/test_manifest.py (96%) create mode 100644 methodfactory/tests/test_packaging.py rename {core => methodfactory}/tests/test_prompt_conformance.py (96%) create mode 100644 pyproject.toml diff --git a/.github/workflows/release-gate.yml b/.github/workflows/release-gate.yml index 2c3e57f..94fdd9c 100644 --- a/.github/workflows/release-gate.yml +++ b/.github/workflows/release-gate.yml @@ -7,42 +7,34 @@ on: branches: [main] workflow_dispatch: +permissions: + contents: read + jobs: - gate: + test: runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + python-version: ["3.11", "3.12"] steps: - - uses: actions/checkout@v4 - - - uses: actions/setup-python@v5 + - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 (pinned SHA) + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 (pinned SHA) with: - python-version: "3.11" + python-version: ${{ matrix.python-version }} - - name: Install deps - run: pip install pyyaml + - name: Install with test extras + run: pip install -e ".[test]" - - name: Unit tests (hard gate) - run: python -m unittest discover -s core/tests -t . + - name: Unit tests (canonical gate) + run: python -m unittest discover -s methodfactory/tests -t . - - name: Identity sweep — no stale PE references in new content + - name: Packaging smoke — mf entry point run: | - # Zero-tolerance: new Method Factory content (prompts/ + core/ + - # action envelope spec) must not carry any Process Engine identity. - # Legacy source material (skills/, templates/, references/, evals/, - # scripts/) is Phase 5 quarantine work — exempt (ADR-0009). - # ADRs and manifest-contract document the migration and are exempt. - STALE='Process Engine|process-engine|v1\.9\.1|lineage v7\.2' - SCOPE="prompts/ core/ docs/action-envelope.md" + mf --version + python -m methodfactory --version - FAIL=0 - while IFS= read -r pattern; do - [ -z "$pattern" ] && continue - matches=$(grep -rIn "$pattern" $SCOPE --include='*.md' --include='*.py' 2>/dev/null || true) - if [ -n "$matches" ]; then - echo "::error::Stale PE reference in new content: $pattern" - echo "$matches" - FAIL=1 - fi - done < "" - mf apply # '-' reads stdin - mf status - mf validate # read-only, collects errors - mf summary # render the canonical summary - -Store root defaults to ./.mf; override with --store. -""" - -from __future__ import annotations - -import argparse -import json -import sys -from pathlib import Path - -from .adapters.artifact_store import ArtifactStore -from .domain.errors import MethodFactoryError -from .engine import PipelineEngine -from .manifest.render import render_summary -from .manifest.schema import validate_manifest -from .manifest.store import ManifestStore - - -def _engine(store_root: Path) -> PipelineEngine: - return PipelineEngine(ManifestStore(store_root), ArtifactStore(store_root / "artifacts")) - - -def _fail(err: MethodFactoryError) -> int: - print(json.dumps(err.as_dict(), indent=2), file=sys.stderr) - return 1 - - -def cmd_create(engine: PipelineEngine, args) -> int: - try: - manifest = engine.create_package(args.package_id, args.intent) - except MethodFactoryError as exc: - return _fail(exc) - print(json.dumps(engine.status(args.package_id), indent=2)) - return 0 - - -def cmd_apply(engine: PipelineEngine, args) -> int: - raw = sys.stdin.read() if args.envelope == "-" else Path(args.envelope).read_text(encoding="utf-8") - try: - result = engine.apply_json(raw) - except MethodFactoryError as exc: - return _fail(exc) - print( - json.dumps( - { - "replayed": result.replayed, - "state": result.manifest["state"], - "revision": result.manifest["revision"], - "event_id": result.event["event_id"], - }, - indent=2, - ) - ) - return 0 - - -def cmd_status(engine: PipelineEngine, args) -> int: - try: - status = engine.status(args.package_id) - except MethodFactoryError as exc: - return _fail(exc) - print(json.dumps(status, indent=2)) - return 0 - - -def cmd_summary(engine: PipelineEngine, args) -> int: - try: - manifest = engine.store.load(args.package_id) - except MethodFactoryError as exc: - return _fail(exc) - if manifest.get("summary") is None: - print("no summary prepared", file=sys.stderr) - return 1 - print(manifest["summary"]["content"], end="") - return 0 - - -def cmd_validate(engine: PipelineEngine, args) -> int: - try: - manifest = engine.store.load(args.package_id) - except MethodFactoryError as exc: - return _fail(exc) - errors = validate_manifest(manifest) - if errors: - for e in errors: - print(f" FAIL {e}", file=sys.stderr) - return 1 - print(f"manifest valid: {args.package_id} @ rev {manifest['revision']} state {manifest['state']}") - return 0 - - -def main(argv: list[str] | None = None) -> int: - parser = argparse.ArgumentParser(prog="mf", description="Method Factory CLI") - parser.add_argument("--store", default=".mf", help="store root (default: ./.mf)") - sub = parser.add_subparsers(dest="command", required=True) - - p_create = sub.add_parser("create", help="create a package from an intent") - p_create.add_argument("package_id") - p_create.add_argument("intent") - p_create.set_defaults(func=cmd_create) - - p_apply = sub.add_parser("apply", help="apply an action envelope (file or -)") - p_apply.add_argument("package_id") - p_apply.add_argument("envelope") - p_apply.set_defaults(func=cmd_apply) - - p_status = sub.add_parser("status", help="show package status") - p_status.add_argument("package_id") - p_status.set_defaults(func=cmd_status) - - p_summary = sub.add_parser("summary", help="render the canonical summary") - p_summary.add_argument("package_id") - p_summary.set_defaults(func=cmd_summary) - - p_validate = sub.add_parser("validate", help="read-only manifest validation") - p_validate.add_argument("package_id") - p_validate.set_defaults(func=cmd_validate) - - args = parser.parse_args(argv) - engine = _engine(Path(args.store)) - return args.func(engine, args) - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/core/engine.py b/core/engine.py deleted file mode 100644 index 1571583..0000000 --- a/core/engine.py +++ /dev/null @@ -1,241 +0,0 @@ -"""PipelineEngine — fail-fast application of validated action envelopes. - -Order of checks (ADR-0008): parse+validate envelope → load+verify manifest → -action-id idempotency/reuse → revision check → legality → gates → build next -manifest → validate → event-journal-first CAS. No partial writes. -""" - -from __future__ import annotations - -import uuid -from dataclasses import dataclass -from typing import Callable, Optional - -from .adapters.artifact_store import ArtifactStore -from .domain.errors import ( - ActionIdReuseError, - IllegalTransitionError, - ManifestInvalidError, - StaleActionError, -) -from .domain.gates import check_action_gate -from .domain.states import State -from .domain.transitions import Action, transition_target -from .manifest.hashing import digest_json, digest_text, utcnow -from .manifest.render import render_summary -from .manifest.schema import validate_manifest -from .manifest.store import ManifestStore -from .protocol.envelope import ActionEnvelope, parse_envelope - -NowFn = Callable[[], str] - - -@dataclass(frozen=True) -class ApplyResult: - manifest: dict - event: dict - replayed: bool - - -class PipelineEngine: - def __init__( - self, - store: ManifestStore, - artifact_store: ArtifactStore, - now: Optional[NowFn] = None, - ) -> None: - self.store = store - self.artifacts = artifact_store - if getattr(self.store, "artifact_store", None) is None: - self.store.artifact_store = artifact_store - self._now = now or utcnow - - # ── public API ───────────────────────────────────────────────────── - def create_package(self, package_id: str, intent_raw: str) -> dict: - return self.store.create(package_id, intent_raw, created_at=self._now()) - - def apply_json(self, raw: str) -> ApplyResult: - return self.apply(parse_envelope(raw)) - - def apply(self, env: ActionEnvelope) -> ApplyResult: - # 1. Envelope is already parsed + schema-validated. - manifest = self.store.load(env.package_id) # MANIFEST_INVALID if missing/corrupt - - # 2. Idempotency / reuse BEFORE revision comparison: a retry of an - # already-committed action is not a stale action. The action's - # semantic content excludes expected_revision so a caller can - # retry with an updated revision and the same action_id; a changed - # action under the same action_id is ACTION_ID_REUSE. - action_content = env.as_dict() - action_content.pop("expected_revision", None) - action_sha256 = digest_json(action_content) - prior = self.store.find_event(env.package_id, env.action_id) - if prior is not None: - if prior.get("action_sha256") == action_sha256: - return ApplyResult(manifest=manifest, event=prior, replayed=True) - raise ActionIdReuseError( - f"action_id {env.action_id!r} reused with different content", - package_id=env.package_id, - state=manifest["state"], - ) - - # 3. Revision check (stale action fails fast). - if env.expected_revision != manifest["revision"]: - raise StaleActionError( - "expected_revision does not match current manifest revision", - package_id=env.package_id, - state=manifest["state"], - expected_revision=env.expected_revision, - actual_revision=manifest["revision"], - ) - - action = Action(env.action) - - # 4. Legality for the current state. - target = transition_target(State(manifest["state"]), action) - if target is None: - raise IllegalTransitionError( - f"{env.action!r} is not legal in state {manifest['state']!r}", - package_id=env.package_id, - state=manifest["state"], - expected_revision=env.expected_revision, - actual_revision=manifest["revision"], - ) - - # 5. Gate predicates (evidence + binding checks). - check_action_gate(action, manifest, env) - - # 6. Build next manifest. - event_id = "evt_" + uuid.uuid4().hex - next_manifest = self._mutate(manifest, env, action, event_id) - next_manifest["revision"] = manifest["revision"] + 1 - next_manifest["previous_manifest_sha256"] = digest_json(manifest) - next_manifest["updated_at"] = self._now() - next_manifest["state"] = target.value - - # 7. Validate the result before persisting anything. - errors = validate_manifest(next_manifest) - if errors: - raise ManifestInvalidError( - "engine produced an invalid manifest: " + "; ".join(errors), - package_id=env.package_id, - ) - - event = { - "event_id": event_id, - "action": env.action, - "action_id": env.action_id, - "revision": next_manifest["revision"], - "state_before": manifest["state"], - "state_after": next_manifest["state"], - "resulting_manifest_sha256": digest_json(next_manifest), - "action_sha256": action_sha256, - "at": self._now(), - } - - # 8. Atomic commit: CAS + event (all-or-nothing). - self.store.compare_and_swap( - env.package_id, manifest["revision"], next_manifest, event - ) - return ApplyResult(manifest=next_manifest, event=event, replayed=False) - - def status(self, package_id: str) -> dict: - manifest = self.store.load(package_id) - summary = manifest.get("summary") - return { - "package_id": package_id, - "state": manifest["state"], - "revision": manifest["revision"], - "intent": manifest["intent"]["raw"], - "inputs": len(manifest["inputs"]), - "artifacts": len(manifest["artifacts"]), - "summary_confirmation": ( - None - if summary is None - else (summary.get("confirmation") or {}).get("status") - ), - "summary_sha256": None if summary is None else summary.get("canonical_sha256"), - } - - # ── mutation ─────────────────────────────────────────────────────── - def _mutate( - self, manifest: dict, env: ActionEnvelope, action: Action, event_id: str - ) -> dict: - from copy import deepcopy - - next_m = deepcopy(manifest) - - if action == Action.RECORD_INPUT: - p = env.payload - content = p["content"] - digest, size = self.artifacts.put( - env.package_id, f"inputs/{p['input_id']}.txt", content - ) - next_m["inputs"].append( - { - "input_id": p["input_id"], - "kind": p["kind"], - "source": p["source"], - "disposition": p["disposition"], - "exclusion_reason": p.get("exclusion_reason"), - "content_sha256": digest, - "content_size": size, - "content_path": f"inputs/{p['input_id']}.txt", - } - ) - - elif action == Action.SET_OBJECTIVE: - next_m["objective"] = { - "statement": env.payload["statement"], - "desired_outcomes": env.payload.get("desired_outcomes", []), - } - - elif action == Action.PREPARE_SUMMARY: - content = render_summary(manifest) - next_m["summary"] = { - "content": content, - "canonical_sha256": digest_text(content), - "presented_at": self._now(), - "confirmation": { - "status": "pending", - "confirmed_at": None, - "operator_id": None, - "confirmed_summary_sha256": None, - }, - } - - elif action == Action.CONFIRM_SUMMARY: - summary = next_m["summary"] - summary["confirmation"] = { - "status": "confirmed", - "confirmed_at": self._now(), - "operator_id": env.payload.get("operator_id") or "operator", - "confirmed_summary_sha256": summary["canonical_sha256"], - } - - elif action == Action.REVISE_INTAKE: - # Return to intake; any approval is invalidated until a new - # summary is prepared and confirmed (ADR-0006). - next_m["summary"] = None - - elif action == Action.RECORD_DRAFT_ARTIFACT: - p = env.payload - digest, size = self.artifacts.put( - env.package_id, p["logical_path"], p["content"] - ) - next_m["artifacts"].append( - { - "artifact_id": p["artifact_id"], - "kind": p["kind"], - "logical_path": p["logical_path"], - "status": "draft", - "sha256": digest, - "byte_count": size, - } - ) - - # CANCEL: state change only (handled by transition table target). - - next_m["transition"]["last_action_id"] = env.action_id - next_m["transition"]["last_event_id"] = event_id - return next_m diff --git a/core/manifest/store.py b/core/manifest/store.py deleted file mode 100644 index 2f93af1..0000000 --- a/core/manifest/store.py +++ /dev/null @@ -1,290 +0,0 @@ -"""ManifestStore — event-journal-first, revisioned persistence (ADR-0008). - -Layout under the store root: - packages/.json latest manifest cache - events/.events.jsonl append-only canonical transition log - events/.lock write lock (O_EXCL) - -The event journal is the source of truth. The package JSON is only a cache -and can lag behind the journal after a crash between the two writes. -""" - -from __future__ import annotations - -import json -import os -import time -import uuid -from pathlib import Path -from typing import Optional - -from ..domain.errors import ( - ConcurrencyError, - DuplicatePackageError, - ManifestInvalidError, - StaleActionError, -) -from .hashing import digest_json, utcnow -from .schema import new_manifest, validate_manifest - -LOCK_TIMEOUT_S = 5.0 -LOCK_RETRY_S = 0.05 - - -class ManifestStore: - def __init__(self, root: Path | str, artifact_store=None) -> None: - self.root = Path(root) - self.packages_dir = self.root / "packages" - self.events_dir = self.root / "events" - self.packages_dir.mkdir(parents=True, exist_ok=True) - self.events_dir.mkdir(parents=True, exist_ok=True) - self.artifact_store = artifact_store - - # ── paths ────────────────────────────────────────────────────────── - def _manifest_path(self, package_id: str) -> Path: - return self.packages_dir / f"{package_id}.json" - - def _events_path(self, package_id: str) -> Path: - return self.events_dir / f"{package_id}.events.jsonl" - - def _lock_path(self, package_id: str) -> Path: - return self.events_dir / f"{package_id}.lock" - - # ── create ───────────────────────────────────────────────────────── - def create(self, package_id: str, intent_raw: str, created_at: Optional[str] = None) -> dict: - created_at = created_at or utcnow() - manifest = new_manifest(package_id, intent_raw, created_at) - errors = validate_manifest(manifest) - if errors: - raise ManifestInvalidError("new manifest invalid: " + "; ".join(errors)) - - with self._lock(package_id): - path = self._manifest_path(package_id) - if path.exists() or self._events_path(package_id).exists(): - raise DuplicatePackageError(f"package {package_id} already exists") - event = { - "event_id": "evt_" + uuid.uuid4().hex, - "action": "create_package", - "action_id": "act_create_package", - "revision": 0, - "state_before": None, - "state_after": manifest["state"], - "resulting_manifest_sha256": digest_json(manifest), - "previous_manifest_sha256": None, - "action_sha256": digest_json({"action": "create_package", "package_id": package_id}), - "at": created_at, - "manifest_snapshot": manifest, - } - self._append_event(package_id, event) - self._atomic_write(path, manifest) - return manifest - - # ── load ─────────────────────────────────────────────────────────── - def load(self, package_id: str) -> dict: - events = self.read_events(package_id) - path = self._manifest_path(package_id) - - if events and all("manifest_snapshot" in event for event in events): - data = self._verify_and_replay(package_id, events) - self._validate_cache_if_present(package_id, path, data, events) - return data - - # Backward compatibility for pre-Phase-4.1 journals: use the snapshot - # when events do not carry reconstructable manifest snapshots. - data = self._read_snapshot(package_id, path) - if events: - last = events[-1].get("resulting_manifest_sha256") - if last and digest_json(data) != last: - raise ManifestInvalidError( - f"manifest digest mismatch for {package_id}: file does not match legacy event chain", - package_id=package_id, - ) - return data - - def _verify_and_replay(self, package_id: str, events: list[dict]) -> dict: - previous = None - for index, event in enumerate(events): - snapshot = event.get("manifest_snapshot") - if not isinstance(snapshot, dict): - self._chain_error(package_id, index, "missing manifest_snapshot") - assert isinstance(snapshot, dict) - expected_revision = index - if event.get("revision") != expected_revision: - self._chain_error( - package_id, - index, - f"revision gap: expected {expected_revision}, got {event.get('revision')!r}", - ) - if event.get("state_before") != (None if previous is None else previous["state"]): - self._chain_error( - package_id, - index, - f"state_before does not match prior state_after: expected " - f"{None if previous is None else previous['state']!r}, got {event.get('state_before')!r}", - ) - if event.get("state_after") != snapshot.get("state"): - self._chain_error(package_id, index, "state_after does not match manifest_snapshot.state") - if snapshot.get("revision") != expected_revision: - self._chain_error(package_id, index, "manifest_snapshot revision does not match event revision") - expected_previous = None if previous is None else previous["digest"] - if snapshot.get("previous_manifest_sha256") != expected_previous: - self._chain_error( - package_id, - index, - f"previous_manifest_sha256 chain break: expected {expected_previous!r}, " - f"got {snapshot.get('previous_manifest_sha256')!r}", - ) - digest = digest_json(snapshot) - if event.get("resulting_manifest_sha256") != digest: - self._chain_error(package_id, index, "resulting_manifest_sha256 does not match manifest_snapshot") - errors = validate_manifest(snapshot) - if errors: - self._chain_error(package_id, index, "invalid manifest_snapshot: " + "; ".join(errors)) - self._verify_artifacts(package_id, index, snapshot) - previous = {"state": snapshot["state"], "digest": digest} - return events[-1]["manifest_snapshot"] - - def _verify_artifacts(self, package_id: str, index: int, manifest: dict) -> None: - if self.artifact_store is None: - return - digests = [item.get("content_sha256") for item in manifest.get("inputs", [])] - digests += [item.get("sha256") for item in manifest.get("artifacts", [])] - for digest in digests: - if not self.artifact_store.verify(digest): - self._chain_error( - package_id, - index, - f"referenced artifact digest missing or invalid: {digest}", - ) - - def _validate_cache_if_present( - self, package_id: str, path: Path, canonical: dict, events: list[dict] - ) -> None: - if not path.exists(): - return - cache = self._read_snapshot(package_id, path) - known = {event["resulting_manifest_sha256"] for event in events} - if digest_json(cache) not in known: - raise ManifestInvalidError( - f"manifest cache corrupt for {package_id}: does not match any journal snapshot", - package_id=package_id, - ) - - def _read_snapshot(self, package_id: str, path: Path) -> dict: - if not path.exists(): - raise ManifestInvalidError(f"manifest missing for {package_id}", package_id=package_id) - try: - data = json.loads(path.read_text(encoding="utf-8")) - except (json.JSONDecodeError, OSError) as exc: - raise ManifestInvalidError( - f"manifest corrupt for {package_id}: {exc}", package_id=package_id - ) from exc - errors = validate_manifest(data) - if errors: - raise ManifestInvalidError( - f"manifest invalid for {package_id}: " + "; ".join(errors), package_id=package_id - ) - return data - - def _chain_error(self, package_id: str, index: int, detail: str) -> None: - raise ManifestInvalidError( - f"event chain break for {package_id} at event index {index}: {detail}", - package_id=package_id, - ) - - # ── compare-and-swap ─────────────────────────────────────────────── - def compare_and_swap( - self, package_id: str, expected_revision: int, next_manifest: dict, event: dict - ) -> None: - with self._lock(package_id): - current = self.load(package_id) - if current["revision"] != expected_revision: - raise StaleActionError( - "concurrent revision change detected", - package_id=package_id, - expected_revision=expected_revision, - actual_revision=current["revision"], - ) - event["manifest_snapshot"] = next_manifest - event["previous_manifest_sha256"] = digest_json(current) - event["resulting_manifest_sha256"] = digest_json(next_manifest) - self._append_event(package_id, event) - self._atomic_write(self._manifest_path(package_id), next_manifest) - - # ── events ───────────────────────────────────────────────────────── - def read_events(self, package_id: str) -> list[dict]: - path = self._events_path(package_id) - if not path.exists(): - return [] - rows = [] - try: - lines = path.read_text(encoding="utf-8").splitlines() - for line in lines: - if line.strip(): - rows.append(json.loads(line)) - except (json.JSONDecodeError, OSError) as exc: - raise ManifestInvalidError( - f"event journal corrupt for {package_id}: {exc}", package_id=package_id - ) from exc - return rows - - def find_event(self, package_id: str, action_id: str) -> Optional[dict]: - for event in self.read_events(package_id): - if event.get("action_id") == action_id: - return event - return None - - def _append_event(self, package_id: str, event: dict) -> None: - path = self._events_path(package_id) - with open(path, "a", encoding="utf-8") as fh: - fh.write(json.dumps(event, sort_keys=True) + "\n") - fh.flush() - os.fsync(fh.fileno()) - - # ── internals ────────────────────────────────────────────────────── - def _atomic_write(self, path: Path, data: dict) -> None: - tmp = path.with_suffix(".json.tmp") - with open(tmp, "w", encoding="utf-8") as fh: - fh.write(json.dumps(data, sort_keys=True, indent=2) + "\n") - fh.flush() - os.fsync(fh.fileno()) - os.replace(tmp, path) - dir_fd = os.open(path.parent, os.O_RDONLY) - try: - os.fsync(dir_fd) - finally: - os.close(dir_fd) - - def _lock(self, package_id: str): - return _PackageLock(self._lock_path(package_id)) - - -class _PackageLock: - """Advisory package-scoped lock via O_CREAT|O_EXCL with timeout.""" - - def __init__(self, path: Path) -> None: - self.path = path - self.acquired = False - - def __enter__(self): - deadline = time.monotonic() + LOCK_TIMEOUT_S - while True: - try: - fd = os.open(self.path, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o644) - os.write(fd, str(os.getpid()).encode()) - os.close(fd) - self.acquired = True - return self - except FileExistsError: - if time.monotonic() >= deadline: - raise ConcurrencyError(f"could not acquire lock {self.path}") - time.sleep(LOCK_RETRY_S) - - def __exit__(self, *exc): - if self.acquired: - try: - self.path.unlink() - except FileNotFoundError: - pass - self.acquired = False - return False diff --git a/core/tests/test_engine.py b/core/tests/test_engine.py deleted file mode 100644 index 5bd0f6c..0000000 --- a/core/tests/test_engine.py +++ /dev/null @@ -1,311 +0,0 @@ -"""Vertical-slice integration tests — engine end-to-end (Phases 2d + 2e). - -Covers the single-phase loop (intent → inputs → objective → summary → -confirmation) and slice completion (authoring → one draft artifact → -DRAFT_READY), plus the integrity proofs: stale actions, invalid transitions, -approval binding, code-computed digests, restart preservation, tamper -detection, idempotency, and scope confinement. -""" - -from __future__ import annotations - -import json -import tempfile -import unittest -from pathlib import Path - -from core.adapters.artifact_store import ArtifactStore -from core.domain.errors import ( - ActionIdReuseError, - GateUnsatisfiedError, - IllegalTransitionError, - InvalidPayloadError, - ManifestInvalidError, - StaleActionError, -) -from core.engine import PipelineEngine -from core.manifest.hashing import digest_text -from core.manifest.store import ManifestStore - -PKG = "pkg_demo_001" -FIXED_NOW = "2026-08-03T04:00:00+00:00" - - -def envelope(action, revision, action_id=None, basis=None, payload=None): - return { - "protocol_version": "0.1", - "action_id": action_id or f"act_{action}_{revision}", - "package_id": PKG, - "expected_revision": revision, - "action": action, - "basis": basis or {}, - "payload": payload or {}, - } - - -class EngineFixture(unittest.TestCase): - def setUp(self): - self._tmp = tempfile.TemporaryDirectory() - self.root = Path(self._tmp.name) - self.store = ManifestStore(self.root / "store") - self.artifacts = ArtifactStore(self.root / "artifacts") - self.engine = PipelineEngine(self.store, self.artifacts, now=lambda: FIXED_NOW) - - def tearDown(self): - self._tmp.cleanup() - - def apply(self, action, revision, **kw): - return self.engine.apply_json(json.dumps(envelope(action, revision, **kw))) - - def run_to_summary_pending(self): - self.engine.create_package(PKG, "Build a standup-notes skill.") - self.apply("record_input", 0, payload={ - "input_id": "in_001", "kind": "text", "content": "rough notes", - "source": "operator", "disposition": "incorporated", - }) - self.apply("set_objective", 1, payload={ - "statement": "A skill that captures standup notes to a file", - "desired_outcomes": ["one command per day"], - }) - return self.apply("prepare_summary", 2) - - def run_full_slice(self): - result = self.run_to_summary_pending() - summary_sha = result.manifest["summary"]["canonical_sha256"] - self.apply("confirm_summary", 3, basis={"summary_sha256": summary_sha}) - return self.apply("record_draft_artifact", 4, payload={ - "artifact_id": "art_001", "kind": "skill", - "logical_path": "skills/standup-notes/SKILL.md", - "content": "# Standup Notes\nCapture daily standup notes.\n", - }) - - -class SinglePhaseLoopTests(EngineFixture): - def test_loop_reaches_summary_pending(self): - result = self.run_to_summary_pending() - self.assertEqual(result.manifest["state"], "SUMMARY_PENDING") - self.assertEqual(result.manifest["revision"], 3) - self.assertEqual(len(result.manifest["inputs"]), 1) - self.assertIsNotNone(result.manifest["summary"]["canonical_sha256"]) - self.assertEqual(result.manifest["summary"]["confirmation"]["status"], "pending") - - def test_restart_preserves_state(self): - self.run_to_summary_pending() - # Fresh engine/store over the same on-disk roots — a process restart. - store2 = ManifestStore(self.root / "store") - artifacts2 = ArtifactStore(self.root / "artifacts") - engine2 = PipelineEngine(store2, artifacts2, now=lambda: FIXED_NOW) - status = engine2.status(PKG) - self.assertEqual(status["state"], "SUMMARY_PENDING") - self.assertEqual(status["revision"], 3) - self.assertEqual(status["inputs"], 1) - self.assertEqual(status["summary_confirmation"], "pending") - - def test_stale_action_fails(self): - self.run_to_summary_pending() - with self.assertRaises(StaleActionError) as cm: - self.apply("confirm_summary", 2, basis={"summary_sha256": "a" * 64}) - self.assertEqual(cm.exception.expected_revision, 2) - self.assertEqual(cm.exception.actual_revision, 3) - # No state change. - self.assertEqual(self.engine.status(PKG)["revision"], 3) - - def test_illegal_transition_fails(self): - self.run_to_summary_pending() - with self.assertRaises(IllegalTransitionError): - self.apply("record_input", 3, payload={ - "input_id": "in_002", "kind": "text", "content": "x", - "source": "operator", "disposition": "incorporated", - }) - self.assertEqual(self.engine.status(PKG)["revision"], 3) - - def test_failed_action_leaves_no_manifest_change(self): - self.run_to_summary_pending() - before = self.store.load(PKG) - before_digest = digest_text(json.dumps(before, sort_keys=True)) - with self.assertRaises(IllegalTransitionError): - self.apply("record_draft_artifact", 3, payload={ - "artifact_id": "art_001", "kind": "skill", - "logical_path": "skills/x/SKILL.md", "content": "x", - }) - after = self.store.load(PKG) - self.assertEqual(digest_text(json.dumps(after, sort_keys=True)), before_digest) - - def test_no_temp_files_left_behind(self): - self.run_to_summary_pending() - leftovers = list((self.root / "store" / "packages").glob("*.tmp")) - self.assertEqual(leftovers, []) - - def test_prepare_summary_requires_intent_and_objective(self): - self.engine.create_package(PKG, "Build a standup-notes skill.") - self.apply("record_input", 0, payload={ - "input_id": "in_001", "kind": "text", "content": "notes", - "source": "operator", "disposition": "incorporated", - }) - with self.assertRaises(GateUnsatisfiedError): - self.apply("prepare_summary", 1) - - -class SliceCompletionTests(EngineFixture): - def test_full_slice_reaches_draft_ready(self): - result = self.run_full_slice() - self.assertEqual(result.manifest["state"], "DRAFT_READY") - self.assertEqual(result.manifest["revision"], 5) - art = result.manifest["artifacts"][0] - content = "# Standup Notes\nCapture daily standup notes.\n" - self.assertEqual(art["sha256"], digest_text(content)) - self.assertEqual(art["byte_count"], len(content.encode("utf-8"))) - self.assertEqual(art["status"], "draft") - - def test_approval_binds_summary_digest(self): - result = self.run_to_summary_pending() - real_sha = result.manifest["summary"]["canonical_sha256"] - with self.assertRaises(StaleActionError): - self.apply("confirm_summary", 3, basis={"summary_sha256": "b" * 64}) - # Correct digest still works. - self.apply("confirm_summary", 3, basis={"summary_sha256": real_sha}) - self.assertEqual(self.engine.status(PKG)["state"], "AUTHORING_AUTHORIZED") - - def test_revise_invalidates_approval(self): - result = self.run_to_summary_pending() - old_sha = result.manifest["summary"]["canonical_sha256"] - self.apply("confirm_summary", 3, basis={"summary_sha256": old_sha}) - self.assertEqual(self.engine.status(PKG)["state"], "AUTHORING_AUTHORIZED") - # Revise → back to intake; approval cleared. - self.apply("revise_intake", 4) - self.assertEqual(self.engine.status(PKG)["state"], "INTAKE") - self.assertIsNone(self.store.load(PKG)["summary"]) - # New objective + new summary. - self.apply("set_objective", 5, payload={ - "statement": "A DIFFERENT objective", - "desired_outcomes": [], - }) - result2 = self.apply("prepare_summary", 6) - new_sha = result2.manifest["summary"]["canonical_sha256"] - self.assertNotEqual(new_sha, old_sha) - # Old approval digest must NOT authorize. - with self.assertRaises(StaleActionError): - self.apply("confirm_summary", 7, basis={"summary_sha256": old_sha}) - # New digest authorizes. - self.apply("confirm_summary", 7, basis={"summary_sha256": new_sha}) - self.assertEqual(self.engine.status(PKG)["state"], "AUTHORING_AUTHORIZED") - - def test_draft_requires_confirmation_transition(self): - # From SUMMARY_PENDING, authoring is an ILLEGAL transition — you - # cannot reach a draft without first confirming the summary. - self.run_to_summary_pending() - with self.assertRaises(IllegalTransitionError): - self.apply("record_draft_artifact", 3, payload={ - "artifact_id": "art_001", "kind": "skill", - "logical_path": "skills/x/SKILL.md", "content": "x", - }) - - def test_tamper_detection(self): - self.run_full_slice() - path = self.root / "store" / "packages" / f"{PKG}.json" - data = json.loads(path.read_text(encoding="utf-8")) - data["state"] = "CANCELLED" # adversarial edit outside the engine - path.write_text(json.dumps(data), encoding="utf-8") - with self.assertRaises(ManifestInvalidError): - self.store.load(PKG) - - def test_restart_preserves_slice_state(self): - self.run_full_slice() - store2 = ManifestStore(self.root / "store") - artifacts2 = ArtifactStore(self.root / "artifacts") - engine2 = PipelineEngine(store2, artifacts2, now=lambda: FIXED_NOW) - status = engine2.status(PKG) - self.assertEqual(status["state"], "DRAFT_READY") - self.assertEqual(status["revision"], 5) - manifest = store2.load(PKG) - art = manifest["artifacts"][0] - self.assertTrue( - artifacts2.verify(art["sha256"]) - ) - - def test_no_forward_scope_creep(self): - self.run_full_slice() - # No review/trial/ship/deploy behavior exists: unknown future actions - # and repeat authoring are rejected; state never leaves the slice. - with self.assertRaises(IllegalTransitionError): - self.apply("record_draft_artifact", 5, payload={ - "artifact_id": "art_002", "kind": "skill", - "logical_path": "skills/y/SKILL.md", "content": "y", - }) - self.assertNotIn( - self.store.load(PKG)["state"], - {"REVIEW_PENDING", "TRIAL_PENDING", "SHIP_PENDING", "SHIPPED"}, - ) - - def test_action_replay_is_idempotent(self): - result = self.run_full_slice() - replay = self.apply("record_draft_artifact", 4, payload={ - "artifact_id": "art_001", "kind": "skill", - "logical_path": "skills/standup-notes/SKILL.md", - "content": "# Standup Notes\nCapture daily standup notes.\n", - }) - self.assertTrue(replay.replayed) - self.assertEqual(replay.manifest["revision"], result.manifest["revision"]) - self.assertEqual(len(replay.manifest["artifacts"]), 1) - - def test_action_id_reuse_with_different_payload(self): - self.run_full_slice() - # run_full_slice used the default action_id act_record_draft_artifact_4. - # Reusing that SAME action_id with different content is reuse, not replay. - with self.assertRaises(ActionIdReuseError): - self.apply( - "record_draft_artifact", 4, - action_id="act_record_draft_artifact_4", - payload={ - "artifact_id": "art_001", "kind": "skill", - "logical_path": "skills/standup-notes/SKILL.md", - "content": "DIFFERENT CONTENT", - }, - ) - - def test_duplicate_input_id_rejected(self): - self.engine.create_package(PKG, "Build a standup-notes skill.") - self.apply("record_input", 0, payload={ - "input_id": "in_001", "kind": "text", "content": "a", - "source": "operator", "disposition": "incorporated", - }) - with self.assertRaises(InvalidPayloadError): - self.apply("record_input", 1, payload={ - "input_id": "in_001", "kind": "text", "content": "b", - "source": "operator", "disposition": "incorporated", - }) - - def test_exclusion_requires_reason(self): - self.engine.create_package(PKG, "Build a standup-notes skill.") - with self.assertRaises(InvalidPayloadError): - self.apply("record_input", 0, payload={ - "input_id": "in_001", "kind": "text", "content": "a", - "source": "operator", "disposition": "excluded", - }) - self.apply("record_input", 0, payload={ - "input_id": "in_001", "kind": "text", "content": "a", - "source": "operator", "disposition": "excluded", - "exclusion_reason": "out of scope", - }) - self.assertEqual(self.engine.status(PKG)["inputs"], 1) - - def test_cancel_reaches_terminal(self): - self.run_full_slice() - self.apply("cancel", 5) - self.assertEqual(self.engine.status(PKG)["state"], "CANCELLED") - with self.assertRaises(IllegalTransitionError): - self.apply("cancel", 6) - - def test_artifact_bytes_stored_and_verifiable(self): - self.run_full_slice() - manifest = self.store.load(PKG) - art = manifest["artifacts"][0] - self.assertEqual( - self.artifacts.get(art["sha256"]), - "# Standup Notes\nCapture daily standup notes.\n", - ) - self.assertTrue(self.artifacts.verify(art["sha256"])) - - -if __name__ == "__main__": - unittest.main() diff --git a/core/tests/test_integrity.py b/core/tests/test_integrity.py deleted file mode 100644 index 48f898b..0000000 --- a/core/tests/test_integrity.py +++ /dev/null @@ -1,162 +0,0 @@ -"""Phase 4.1 integrity, crash, and contention proofs.""" - -from __future__ import annotations - -import json -import tempfile -import unittest -from pathlib import Path - -from core.adapters.artifact_store import ArtifactStore -from core.domain.errors import ManifestInvalidError, StaleActionError -from core.engine import PipelineEngine -from core.manifest.store import ManifestStore - -PKG = "pkg_integrity_001" -NOW = "2026-08-03T04:00:00+00:00" - - -def action(action, revision, action_id=None, payload=None): - return { - "protocol_version": "0.1", - "action_id": action_id or f"act_{action}_{revision}", - "package_id": PKG, - "expected_revision": revision, - "action": action, - "basis": {}, - "payload": payload or {}, - } - - -class IntegrityTests(unittest.TestCase): - def setUp(self): - self.tmp = tempfile.TemporaryDirectory() - self.root = Path(self.tmp.name) - self.artifacts = ArtifactStore(self.root / "artifacts") - self.store = ManifestStore(self.root / "store", artifact_store=self.artifacts) - self.engine = PipelineEngine(self.store, self.artifacts, now=lambda: NOW) - - def tearDown(self): - self.tmp.cleanup() - - def create(self): - return self.engine.create_package(PKG, "Build an integrity test package.") - - def test_artifact_blob_immutable(self): - digest1, size1 = self.artifacts.put(PKG, "one.txt", "same content") - blob = self.root / "artifacts" / "blobs" / digest1 - before = blob.read_bytes() - digest2, size2 = self.artifacts.put("pkg_other", "different.txt", "same content") - self.assertEqual((digest1, size1), (digest2, size2)) - self.assertEqual(blob.read_bytes(), before) - self.assertEqual(self.artifacts.get(digest1), "same content") - - def test_orphaned_blob_is_harmless(self): - self.create() - digest, _ = self.artifacts.put(PKG, "orphan.txt", "written before failed CAS") - current = self.store.load(PKG) - with self.assertRaises(StaleActionError): - self.store.compare_and_swap(PKG, 99, current, {"event_id": "evt_bad"}) - self.assertTrue(self.artifacts.verify(digest)) - self.assertEqual(self.store.load(PKG)["artifacts"], []) - - def test_crash_between_event_and_snapshot_recovery(self): - self.create() - original = self.store._atomic_write - calls = 0 - - def crash_once(path, data): - nonlocal calls - calls += 1 - raise RuntimeError("simulated crash after journal fsync") - - self.store._atomic_write = crash_once - with self.assertRaises(RuntimeError): - self.engine.apply_json(json.dumps(action("set_objective", 0, payload={ - "statement": "recover from journal", - "desired_outcomes": [], - }))) - self.store._atomic_write = original - recovered = ManifestStore(self.root / "store", artifact_store=self.artifacts) - manifest = recovered.load(PKG) - self.assertEqual(calls, 1) - self.assertEqual(manifest["revision"], 1) - self.assertEqual(manifest["objective"]["statement"], "recover from journal") - - def test_event_chain_continuity_detected(self): - self.create() - self.engine.apply_json(json.dumps(action("set_objective", 0, payload={ - "statement": "chain test", "desired_outcomes": [] - }))) - path = self.root / "store" / "events" / f"{PKG}.events.jsonl" - rows = [json.loads(line) for line in path.read_text().splitlines()] - rows[1]["state_after"] = "CANCELLED" - path.write_text("\n".join(json.dumps(row) for row in rows) + "\n") - with self.assertRaisesRegex(ManifestInvalidError, "state_after"): - self.store.load(PKG) - - def test_revision_gap_detected(self): - self.create() - self.engine.apply_json(json.dumps(action("set_objective", 0, payload={ - "statement": "gap test", "desired_outcomes": [] - }))) - path = self.root / "store" / "events" / f"{PKG}.events.jsonl" - rows = [json.loads(line) for line in path.read_text().splitlines()] - rows[1]["revision"] = 4 - path.write_text("\n".join(json.dumps(row) for row in rows) + "\n") - with self.assertRaisesRegex(ManifestInvalidError, "revision gap"): - self.store.load(PKG) - - def test_concurrent_cas_contention(self): - self.create() - store2 = ManifestStore(self.root / "store", artifact_store=self.artifacts) - engine2 = PipelineEngine(store2, self.artifacts, now=lambda: NOW) - first = self.engine.apply_json(json.dumps(action("set_objective", 0, - action_id="act_first", payload={"statement": "first", "desired_outcomes": []}))) - self.assertEqual(first.manifest["revision"], 1) - with self.assertRaises(StaleActionError): - engine2.apply_json(json.dumps(action("set_objective", 0, - action_id="act_second", payload={"statement": "second", "desired_outcomes": []}))) - - def test_last_event_id_populated(self): - self.create() - result = self.engine.apply_json(json.dumps(action("set_objective", 0, payload={ - "statement": "event id", "desired_outcomes": [] - }))) - self.assertEqual(result.manifest["transition"]["last_event_id"], result.event["event_id"]) - self.assertEqual(self.store.load(PKG)["transition"]["last_event_id"], result.event["event_id"]) - - def test_retry_with_updated_revision_replays_same_action(self): - self.create() - payload = {"statement": "retry-safe", "desired_outcomes": []} - first = self.engine.apply_json(json.dumps(action( - "set_objective", 0, action_id="act_retry", payload=payload - ))) - replay = self.engine.apply_json(json.dumps(action( - "set_objective", 1, action_id="act_retry", payload=payload - ))) - self.assertTrue(replay.replayed) - self.assertEqual(replay.event["event_id"], first.event["event_id"]) - - def test_stale_cas_orphaned_blob(self): - self.create() - digest, _ = self.artifacts.put(PKG, "failed.txt", "failed CAS content") - manifest = self.store.load(PKG) - with self.assertRaises(StaleActionError): - self.store.compare_and_swap(PKG, manifest["revision"] + 1, manifest, {"event_id": "evt_stale"}) - self.assertTrue(self.artifacts.verify(digest)) - self.assertNotIn(digest, [a["sha256"] for a in self.store.load(PKG)["artifacts"]]) - - def test_full_chain_verifies_artifact_digests(self): - self.create() - content = "input content" - self.engine.apply_json(json.dumps(action("record_input", 0, payload={ - "input_id": "in_001", "kind": "text", "content": content, - "source": "operator", "disposition": "incorporated", - }))) - manifest = self.store.load(PKG) - self.assertTrue(self.artifacts.verify(manifest["inputs"][0]["content_sha256"])) - - -if __name__ == "__main__": - unittest.main() diff --git a/methodfactory/__init__.py b/methodfactory/__init__.py new file mode 100644 index 0000000..ce5a73f --- /dev/null +++ b/methodfactory/__init__.py @@ -0,0 +1,15 @@ +"""methodfactory — deterministic package-lifecycle engine (persistence reset). + +Phase 2 foundation: storage-independent domain/protocol/adapters plus the +storage protocol and SQLite schema primitives. The JSONL-era store and engine +are NOT ported (ADR-0012 §8). Version follows the 2.0.0 prerelease ladder. +""" + +__version__ = "2.0.0a1" + +# Reusable foundation (ADR-0012 §8 port list). Deliberately does not import the +# JSONL store or the lifecycle engine (removed in the persistence reset). +from .adapters.artifact_store import ArtifactStore # noqa: F401 +from .manifest.hashing import canonical_json, digest_bytes, digest_json, digest_text # noqa: F401 +from .manifest.schema import new_manifest, validate_manifest # noqa: F401 +from .protocol.envelope import ActionEnvelope, parse_envelope # noqa: F401 diff --git a/methodfactory/__main__.py b/methodfactory/__main__.py new file mode 100644 index 0000000..f396c7a --- /dev/null +++ b/methodfactory/__main__.py @@ -0,0 +1,10 @@ +"""Allow `python -m methodfactory` to behave like the `mf` CLI.""" + +from __future__ import annotations + +import sys + +from .cli import main + +if __name__ == "__main__": + sys.exit(main()) diff --git a/core/adapters/__init__.py b/methodfactory/adapters/__init__.py similarity index 100% rename from core/adapters/__init__.py rename to methodfactory/adapters/__init__.py diff --git a/core/adapters/artifact_store.py b/methodfactory/adapters/artifact_store.py similarity index 100% rename from core/adapters/artifact_store.py rename to methodfactory/adapters/artifact_store.py diff --git a/methodfactory/cli.py b/methodfactory/cli.py new file mode 100644 index 0000000..5ce3187 --- /dev/null +++ b/methodfactory/cli.py @@ -0,0 +1,39 @@ +"""`mf` CLI — thin adapter over the engine (ADR-0001). + +Phase 2: version + availability notice only. The full command surface +(create/apply/status/summary/validate, migrate-store, export) returns after the +SQLite store and lifecycle are implemented (ADR-0012 Phase 2 stop gate). +""" + +from __future__ import annotations + +import argparse +import sys + +from . import __version__ + +AVAILABILITY = ( + "Method Factory storage is under architecture reset (ADR-0012); " + "commands return in a later phase." +) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(prog="mf", description="Method Factory CLI") + parser.add_argument( + "--version", action="version", version=f"methodfactory {__version__}" + ) + parser.add_argument( + "args", nargs="*", + help="command + arguments (unavailable in this phase: persistence reset in progress)", + ) + args = parser.parse_args(argv) + if args.args: + print(AVAILABILITY, file=sys.stderr) + return 2 + parser.print_help() + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/core/domain/__init__.py b/methodfactory/domain/__init__.py similarity index 100% rename from core/domain/__init__.py rename to methodfactory/domain/__init__.py diff --git a/core/domain/errors.py b/methodfactory/domain/errors.py similarity index 100% rename from core/domain/errors.py rename to methodfactory/domain/errors.py diff --git a/core/domain/gates.py b/methodfactory/domain/gates.py similarity index 100% rename from core/domain/gates.py rename to methodfactory/domain/gates.py diff --git a/core/domain/states.py b/methodfactory/domain/states.py similarity index 100% rename from core/domain/states.py rename to methodfactory/domain/states.py diff --git a/core/domain/transitions.py b/methodfactory/domain/transitions.py similarity index 100% rename from core/domain/transitions.py rename to methodfactory/domain/transitions.py diff --git a/methodfactory/manifest/__init__.py b/methodfactory/manifest/__init__.py new file mode 100644 index 0000000..1c90314 --- /dev/null +++ b/methodfactory/manifest/__init__.py @@ -0,0 +1,5 @@ +"""Manifest package — canonical serialization, schema, and renders. + +The JSONL-era store module was removed in the persistence reset (ADR-0012 §8); +this package holds the storage-independent manifest primitives. +""" diff --git a/core/manifest/hashing.py b/methodfactory/manifest/hashing.py similarity index 100% rename from core/manifest/hashing.py rename to methodfactory/manifest/hashing.py diff --git a/core/manifest/render.py b/methodfactory/manifest/render.py similarity index 100% rename from core/manifest/render.py rename to methodfactory/manifest/render.py diff --git a/core/manifest/schema.py b/methodfactory/manifest/schema.py similarity index 100% rename from core/manifest/schema.py rename to methodfactory/manifest/schema.py diff --git a/core/protocol/__init__.py b/methodfactory/protocol/__init__.py similarity index 100% rename from core/protocol/__init__.py rename to methodfactory/protocol/__init__.py diff --git a/core/protocol/envelope.py b/methodfactory/protocol/envelope.py similarity index 100% rename from core/protocol/envelope.py rename to methodfactory/protocol/envelope.py diff --git a/core/tests/__init__.py b/methodfactory/tests/__init__.py similarity index 100% rename from core/tests/__init__.py rename to methodfactory/tests/__init__.py diff --git a/core/tests/test_domain.py b/methodfactory/tests/test_domain.py similarity index 97% rename from core/tests/test_domain.py rename to methodfactory/tests/test_domain.py index 97a8925..e94c30a 100644 --- a/core/tests/test_domain.py +++ b/methodfactory/tests/test_domain.py @@ -4,8 +4,8 @@ import unittest -from core.domain.states import State, TERMINAL_STATES, is_terminal -from core.domain.transitions import ( +from methodfactory.domain.states import State, TERMINAL_STATES, is_terminal +from methodfactory.domain.transitions import ( ACTION_VOCABULARY, TRANSITION_TABLE, Action, diff --git a/core/tests/test_envelope.py b/methodfactory/tests/test_envelope.py similarity index 98% rename from core/tests/test_envelope.py rename to methodfactory/tests/test_envelope.py index 9793ae6..6db2d78 100644 --- a/core/tests/test_envelope.py +++ b/methodfactory/tests/test_envelope.py @@ -5,8 +5,8 @@ import json import unittest -from core.domain.errors import InvalidEnvelopeError -from core.protocol.envelope import parse_envelope +from methodfactory.domain.errors import InvalidEnvelopeError +from methodfactory.protocol.envelope import parse_envelope def record_input_payload(**overrides): diff --git a/core/tests/test_manifest.py b/methodfactory/tests/test_manifest.py similarity index 96% rename from core/tests/test_manifest.py rename to methodfactory/tests/test_manifest.py index 0441ea5..4aa518c 100644 --- a/core/tests/test_manifest.py +++ b/methodfactory/tests/test_manifest.py @@ -5,9 +5,9 @@ import json import unittest -from core.domain.states import State -from core.manifest.hashing import canonical_json, digest_json -from core.manifest.schema import new_manifest, validate_manifest +from methodfactory.domain.states import State +from methodfactory.manifest.hashing import canonical_json, digest_json +from methodfactory.manifest.schema import new_manifest, validate_manifest def valid_manifest(**overrides): diff --git a/methodfactory/tests/test_packaging.py b/methodfactory/tests/test_packaging.py new file mode 100644 index 0000000..14de58f --- /dev/null +++ b/methodfactory/tests/test_packaging.py @@ -0,0 +1,51 @@ +"""Packaging smoke tests — version, entry point, and import surface (ADR-0012 §9).""" + +from __future__ import annotations + +import importlib.metadata +import unittest + +import methodfactory + + +class PackagingTests(unittest.TestCase): + def test_version_is_2_0_0a1(self): + self.assertEqual(methodfactory.__version__, "2.0.0a1") + + def test_distribution_metadata_version_matches(self): + try: + dist = importlib.metadata.version("methodfactory") + except importlib.metadata.PackageNotFoundError: + self.skipTest("methodfactory not installed (not an editable install)") + self.assertEqual(dist, "2.0.0a1") + + def test_mf_entry_point_registered(self): + eps = importlib.metadata.entry_points(group="console_scripts") + mf = [ep for ep in eps if ep.name == "mf"] + if not mf: + self.skipTest("console script not registered (not an editable install)") + self.assertEqual(mf[0].value, "methodfactory.cli:main") + + def test_import_surface(self): + from methodfactory.adapters.artifact_store import ArtifactStore + from methodfactory.domain.states import State + from methodfactory.manifest.schema import validate_manifest + from methodfactory.protocol.envelope import parse_envelope + + self.assertTrue(callable(parse_envelope)) + self.assertTrue(callable(validate_manifest)) + self.assertIsInstance(ArtifactStore, type) + self.assertEqual(State.INTAKE.value, "INTAKE") + + def test_old_jsonl_engine_is_absent(self): + # The JSONL-era store/engine were removed in the persistence reset and + # must not be importable (ADR-0012 §8 discard list). + import importlib + + for mod in ("methodfactory.engine", "methodfactory.manifest.store"): + with self.assertRaises(ImportError): + importlib.import_module(mod) + + +if __name__ == "__main__": + unittest.main() diff --git a/core/tests/test_prompt_conformance.py b/methodfactory/tests/test_prompt_conformance.py similarity index 96% rename from core/tests/test_prompt_conformance.py rename to methodfactory/tests/test_prompt_conformance.py index 3429dee..12fa9f9 100644 --- a/core/tests/test_prompt_conformance.py +++ b/methodfactory/tests/test_prompt_conformance.py @@ -19,9 +19,9 @@ import unittest from pathlib import Path -from core.domain.errors import InvalidEnvelopeError -from core.domain.transitions import ACTION_VOCABULARY -from core.protocol.envelope import parse_envelope +from methodfactory.domain.errors import InvalidEnvelopeError +from methodfactory.domain.transitions import ACTION_VOCABULARY +from methodfactory.protocol.envelope import parse_envelope REPO = Path(__file__).resolve().parents[2] # repo-staging/ PROMPTS = REPO / "prompts" diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..a9907b2 --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,33 @@ +[build-system] +requires = ["setuptools==68.2.2"] +build-backend = "setuptools.build_meta" + +[project] +name = "methodfactory" +version = "2.0.0a1" +description = "Method Factory — prompt+code pipeline generator (SQLite canonical store, Phase 2 foundation)" +readme = "README.md" +requires-python = ">=3.11" +license = { text = "MIT" } +keywords = ["agents", "skills", "pipeline", "state-machine", "persistence"] +classifiers = [ + "Development Status :: 3 - Alpha", + "Intended Audience :: Developers", + "License :: OSI Approved :: MIT License", + "Programming Language :: Python :: 3", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", +] + +[project.scripts] +mf = "methodfactory.cli:main" + +[project.optional-dependencies] +test = [ + "hypothesis>=6", + "PyYAML>=6", +] + +[tool.setuptools.packages.find] +include = ["methodfactory*"] +exclude = ["methodfactory.tests*"] From fda563d704fe49e47a2c988ea8c2c1b9c2c15aa6 Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Fri, 7 Aug 2026 05:19:15 +0700 Subject: [PATCH 04/41] refactor: introduce storage protocol and canonical serialization primitives MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 2 foundation commit (ADR-0012 implementation order step 3). Storage layer primitives independent of SQLite; no transactional store yet. - methodfactory/storage/errors.py: typed storage/schema errors with stable codes (DATABASE_NOT_FOUND, DATABASE_EMPTY, DATABASE_ID_MISMATCH, UNSUPPORTED_SCHEMA, LEGACY_STORE_DETECTED, SCHEMA_VIOLATION, APPEND_ONLY_VIOLATION, INVALID_PACKAGE_ID, INVALID_STORE_ROOT, ACTION_ID_CONFLICT - supersedes JSONL ACTION_ID_REUSE per ADR-0012 §G). - methodfactory/storage/serialization.py: canonical_json/canonical_bytes (sorted keys, compact separators, UTF-8, ensure_ascii=False, allow_nan=False - ADR-0012 §4) + sha256/digest helpers + frozen action_sha256 (ADR-0012 §G: hashes {action, package_id, action_id, basis, payload}; excludes only expected_revision). - methodfactory/storage/protocol.py: ManifestStore Protocol (create/apply/ load/read_events) with contract invariants (ADR-0012 §2, §6, §8). - methodfactory/storage/paths.py: canonical DB_FILENAME=methodfactory.sqlite3, database_path beneath store root, validate_package_id (^pkg_[A-Za-z0-9_-]{1,63}$), validate_store_root. - methodfactory/storage/limits.py: size-bound constants separated by object type (envelope, content fields, action JSON, manifest, artifact) - never reuse an envelope limit as event/manifest limit (ADR-0012 §4). - Tests: test_serialization (key-order/compact/UTF-8/invalid-number/ stability + Hypothesis property), test_action_hash (semantics, exclusion of expected_revision, stability), test_paths (package/store validation), test_limits (positive, distinct, ordered, non-reuse). - No JSONL locks/repair/tail/cache logic ported. Local 3.11: full gate green. --- methodfactory/storage/__init__.py | 81 +++++++++++++++++++ methodfactory/storage/errors.py | 97 +++++++++++++++++++++++ methodfactory/storage/limits.py | 30 +++++++ methodfactory/storage/paths.py | 37 +++++++++ methodfactory/storage/protocol.py | 39 +++++++++ methodfactory/storage/serialization.py | 77 ++++++++++++++++++ methodfactory/tests/test_action_hash.py | 62 +++++++++++++++ methodfactory/tests/test_limits.py | 58 ++++++++++++++ methodfactory/tests/test_paths.py | 64 +++++++++++++++ methodfactory/tests/test_serialization.py | 60 ++++++++++++++ 10 files changed, 605 insertions(+) create mode 100644 methodfactory/storage/__init__.py create mode 100644 methodfactory/storage/errors.py create mode 100644 methodfactory/storage/limits.py create mode 100644 methodfactory/storage/paths.py create mode 100644 methodfactory/storage/protocol.py create mode 100644 methodfactory/storage/serialization.py create mode 100644 methodfactory/tests/test_action_hash.py create mode 100644 methodfactory/tests/test_limits.py create mode 100644 methodfactory/tests/test_paths.py create mode 100644 methodfactory/tests/test_serialization.py diff --git a/methodfactory/storage/__init__.py b/methodfactory/storage/__init__.py new file mode 100644 index 0000000..b256834 --- /dev/null +++ b/methodfactory/storage/__init__.py @@ -0,0 +1,81 @@ +"""Storage layer — protocol, canonical primitives, limits, and SQLite schema. + +Phase 2 (ADR-0012 commits 2–4). The transactional store, migration, and export +are implemented in later commits; this package carries the storage-independent +contracts and the SQLite schema creation/identity/append-only guards. +""" + +from .errors import ( + ActionIdConflictError, + AppendOnlyViolationError, + DatabaseEmptyError, + DatabaseIdMismatchError, + DatabaseNotFoundError, + InvalidPackageIdError, + InvalidStoreRootError, + LegacyStoreDetectedError, + SchemaViolationError, + StorageError, + UnsupportedSchemaError, +) +from .limits import ( + MAX_ACTION_JSON_BYTES, + MAX_ARTIFACT_BYTES, + MAX_CONTENT_CHARS, + MAX_ENVELOPE_BYTES, + MAX_ID_CHARS, + MAX_INTENT_CHARS, + MAX_LOGICAL_PATH_CHARS, + MAX_MANIFEST_BYTES, + MAX_OUTCOMES, + MAX_REASON_CHARS, + MAX_STATEMENT_CHARS, +) +from .paths import DB_FILENAME, database_path, validate_package_id, validate_store_root +from .protocol import ManifestStore +from .serialization import ( + action_sha256, + canonical_bytes, + canonical_json, + digest_bytes, + digest_json, + digest_text, + sha256_hex, +) + +__all__ = [ + "ActionIdConflictError", + "AppendOnlyViolationError", + "DB_FILENAME", + "DatabaseEmptyError", + "DatabaseIdMismatchError", + "DatabaseNotFoundError", + "InvalidPackageIdError", + "InvalidStoreRootError", + "LegacyStoreDetectedError", + "ManifestStore", + "MAX_ACTION_JSON_BYTES", + "MAX_ARTIFACT_BYTES", + "MAX_CONTENT_CHARS", + "MAX_ENVELOPE_BYTES", + "MAX_ID_CHARS", + "MAX_INTENT_CHARS", + "MAX_LOGICAL_PATH_CHARS", + "MAX_MANIFEST_BYTES", + "MAX_OUTCOMES", + "MAX_REASON_CHARS", + "MAX_STATEMENT_CHARS", + "SchemaViolationError", + "StorageError", + "UnsupportedSchemaError", + "action_sha256", + "canonical_bytes", + "canonical_json", + "database_path", + "digest_bytes", + "digest_json", + "digest_text", + "sha256_hex", + "validate_package_id", + "validate_store_root", +] diff --git a/methodfactory/storage/errors.py b/methodfactory/storage/errors.py new file mode 100644 index 0000000..941107c --- /dev/null +++ b/methodfactory/storage/errors.py @@ -0,0 +1,97 @@ +"""Typed storage/schema errors with stable machine-readable codes. + +Extends the ADR-0008 stable error-code table for the storage layer +(ADR-0012 §B, §D, §E, §G). Codes are part of the public contract. +""" + +from __future__ import annotations + +from typing import Any, Optional + + +class StorageError(Exception): + """Base for all storage-layer failures.""" + + code = "STORAGE_ERROR" + + def __init__( + self, + message: str, + *, + package_id: Optional[str] = None, + **context: Any, + ) -> None: + super().__init__(message) + self.message = message + self.package_id = package_id + self.context = context + + def as_dict(self) -> dict[str, Any]: + d: dict[str, Any] = {"code": self.code, "message": self.message} + if self.package_id is not None: + d["package_id"] = self.package_id + d.update(self.context) + return d + + +class DatabaseNotFoundError(StorageError): + """No SQLite database exists (and none may be created on this path).""" + + code = "DATABASE_NOT_FOUND" + + +class DatabaseEmptyError(StorageError): + """A zero-byte database file exists but carries no identity.""" + + code = "DATABASE_EMPTY" + + +class DatabaseIdMismatchError(StorageError): + """The database's application_id is not the canonical Method Factory ID.""" + + code = "DATABASE_ID_MISMATCH" + + +class UnsupportedSchemaError(StorageError): + """The database's user_version is not supported by this build.""" + + code = "UNSUPPORTED_SCHEMA" + + +class LegacyStoreDetectedError(StorageError): + """A public v0.1.2 JSONL store is present; run `mf migrate-store`.""" + + code = "LEGACY_STORE_DETECTED" + + +class SchemaViolationError(StorageError): + """A row violates a schema or chain invariant.""" + + code = "SCHEMA_VIOLATION" + + +class AppendOnlyViolationError(StorageError): + """An attempt to UPDATE or DELETE an immutable event row.""" + + code = "APPEND_ONLY_VIOLATION" + + +class InvalidPackageIdError(StorageError): + """package_id fails the canonical pattern `^pkg_[A-Za-z0-9_-]{1,63}$`.""" + + code = "INVALID_PACKAGE_ID" + + +class InvalidStoreRootError(StorageError): + """The store root path is unusable.""" + + code = "INVALID_STORE_ROOT" + + +class ActionIdConflictError(StorageError): + """Same action_id reused with a different action_sha256 (ADR-0012 §G). + + Supersedes the JSONL-era ACTION_ID_REUSE code for the SQLite store. + """ + + code = "ACTION_ID_CONFLICT" diff --git a/methodfactory/storage/limits.py b/methodfactory/storage/limits.py new file mode 100644 index 0000000..9fcfbf3 --- /dev/null +++ b/methodfactory/storage/limits.py @@ -0,0 +1,30 @@ +"""Size-bound constants, separated by object type (ADR-0012 §4). + +Never reuse an envelope limit as an event/manifest limit: an incoming action +envelope and a committed event carrying a cumulative manifest are different +objects with different sizes. Values are preliminary and frozen by ADR review +(the Phase 2 submission reports them; the senior reviewer approves the set). +""" + +from __future__ import annotations + +# ── Action Envelope (wire/parse boundary) ─────────────────────────────── +MAX_ENVELOPE_BYTES = 2 * 1024 * 1024 + +# ── Individual content fields ─────────────────────────────────────────── +MAX_CONTENT_CHARS = 1_048_576 # record_input / record_draft_artifact content +MAX_INTENT_CHARS = 65_536 # create_package intent.raw +MAX_STATEMENT_CHARS = 16_384 # set_objective statement / outcome +MAX_OUTCOMES = 100 # desired_outcomes list length +MAX_ID_CHARS = 128 # input_id / artifact_id / operator_id / kind +MAX_LOGICAL_PATH_CHARS = 255 # artifact logical_path +MAX_REASON_CHARS = 1024 # exclusion_reason / cancel reason + +# ── Canonical action JSON (normalized semantic request; ADR-0012 §G) ──── +MAX_ACTION_JSON_BYTES = 4 * 1024 * 1024 + +# ── Manifest (complete resulting manifest per revision; ADR-0012 §4) ──── +MAX_MANIFEST_BYTES = 8 * 1024 * 1024 + +# ── Artifact / blob (content-addressed immutable store) ───────────────── +MAX_ARTIFACT_BYTES = 64 * 1024 * 1024 diff --git a/methodfactory/storage/paths.py b/methodfactory/storage/paths.py new file mode 100644 index 0000000..05adef8 --- /dev/null +++ b/methodfactory/storage/paths.py @@ -0,0 +1,37 @@ +"""Store-root and package-id path validation (ADR-0012 §4, §D).""" + +from __future__ import annotations + +import re +from pathlib import Path + +from .errors import InvalidPackageIdError, InvalidStoreRootError + +PACKAGE_ID_RE = re.compile(r"^pkg_[A-Za-z0-9_-]{1,63}$") + +# Canonical physical database filename (ADR-0012 §D). +DB_FILENAME = "methodfactory.sqlite3" + + +def validate_package_id(package_id: str) -> str: + """Validate a package identifier against the canonical pattern. Rejects + path separators, traversal, and non-string values by construction.""" + if not isinstance(package_id, str) or not PACKAGE_ID_RE.match(package_id): + raise InvalidPackageIdError(f"invalid package_id {package_id!r}") + return package_id + + +def validate_store_root(root: Path | str) -> Path: + """Normalize and validate the store root. It must be a non-empty path; + if it exists it must be a directory (a file at the root is unusable).""" + if isinstance(root, str) and not root.strip(): + raise InvalidStoreRootError("store root must not be empty") + p = Path(root) + if p.exists() and not p.is_dir(): + raise InvalidStoreRootError(f"store root exists but is not a directory: {p}") + return p + + +def database_path(root: Path | str) -> Path: + """Canonical SQLite database location: /methodfactory.sqlite3.""" + return validate_store_root(root) / DB_FILENAME diff --git a/methodfactory/storage/protocol.py b/methodfactory/storage/protocol.py new file mode 100644 index 0000000..2b6ecc9 --- /dev/null +++ b/methodfactory/storage/protocol.py @@ -0,0 +1,39 @@ +"""Storage protocol — the ManifestStore interface, independent of SQLite. + +The SQLite store (and any future adapter) implements this interface. The +contract invariants are frozen in ADR-0012 §2, §6, §8; the authoritative +chain validator is owned by the storage layer and exercised on every +transactional apply (implemented in a later Phase 2 commit). +""" + +from __future__ import annotations + +from typing import Any, Optional, Protocol, runtime_checkable + + +@runtime_checkable +class ManifestStore(Protocol): + """Canonical package store interface. + + Contract summary (ADR-0012): + - ``create`` commits revision 0 (the create_package event) with + ``state_before IS NULL``. + - ``apply`` runs the ``BEGIN IMMEDIATE`` transaction: idempotency by + (package_id, action_id) + ``action_sha256``, revision CAS, exactly one + event insert; ``ACTION_ID_CONFLICT`` on same id + different hash; + ``STALE_ACTION`` on a stale expected revision. + - ``load`` returns the latest manifest via the indexed latest-event read; + the hot path never runs a full-chain replay. + - ``read_events`` returns ordered events for export/audit + (``method-factory-events-v1``). + """ + + def create( + self, package_id: str, intent_raw: str, created_at: Optional[str] = None + ) -> dict[str, Any]: ... + + def apply(self, envelope: dict[str, Any]) -> dict[str, Any]: ... + + def load(self, package_id: str) -> dict[str, Any]: ... + + def read_events(self, package_id: str) -> list[dict[str, Any]]: ... diff --git a/methodfactory/storage/serialization.py b/methodfactory/storage/serialization.py new file mode 100644 index 0000000..b4959eb --- /dev/null +++ b/methodfactory/storage/serialization.py @@ -0,0 +1,77 @@ +"""Canonical serialization and hash primitives (ADR-0012 §4, §G). + +Canonical JSON bytes are produced exactly as: + + json.dumps(value, sort_keys=True, separators=(",", ":"), + ensure_ascii=False, allow_nan=False).encode("utf-8") + +and those bytes are what is hashed. `action_sha256` is the frozen semantic +action hash (ADR-0012 §G). +""" + +from __future__ import annotations + +import hashlib +import json +from typing import Any + + +def canonical_json(value: Any) -> str: + """Deterministic JSON text: sorted keys, compact separators, UTF-8-safe, + no NaN/Infinity (raises ValueError on non-finite numbers).""" + return json.dumps( + value, + sort_keys=True, + separators=(",", ":"), + ensure_ascii=False, + allow_nan=False, + ) + + +def canonical_bytes(value: Any) -> bytes: + """The canonical byte form that is hashed (ADR-0012 §4).""" + return canonical_json(value).encode("utf-8") + + +def sha256_hex(data: bytes) -> str: + return hashlib.sha256(data).hexdigest() + + +def digest_bytes(data: bytes) -> str: + return sha256_hex(data) + + +def digest_text(content: str) -> str: + return sha256_hex(content.encode("utf-8")) + + +def digest_json(value: Any) -> str: + return sha256_hex(canonical_bytes(value)) + + +def action_sha256( + *, + action: str, + package_id: str, + action_id: str, + basis: dict[str, Any], + payload: dict[str, Any], +) -> str: + """Canonical semantic action hash (ADR-0012 §G). + + Hashes the complete normalized semantic request used for idempotency: + {action, package_id, action_id, basis, payload}. Every field that could + change the requested outcome is included. Only `expected_revision` + (optimistic-concurrency/transport metadata) is excluded — it is not part + of the requested outcome, so a retry with an updated revision and the + same action_id yields the same hash and replays. + """ + return digest_json( + { + "action": action, + "package_id": package_id, + "action_id": action_id, + "basis": basis, + "payload": payload, + } + ) diff --git a/methodfactory/tests/test_action_hash.py b/methodfactory/tests/test_action_hash.py new file mode 100644 index 0000000..7c14d2d --- /dev/null +++ b/methodfactory/tests/test_action_hash.py @@ -0,0 +1,62 @@ +"""Canonical action-hash semantics tests (ADR-0012 §G).""" + +from __future__ import annotations + +import unittest + +from methodfactory.storage.serialization import action_sha256 + + +def env_semantic(*, action="record_input", package_id="pkg_demo_001", + action_id="act_1", basis=None, payload=None): + return { + "action": action, + "package_id": package_id, + "action_id": action_id, + "basis": basis or {}, + "payload": payload or {}, + } + + +class ActionHashSemanticsTests(unittest.TestCase): + def test_stable_across_calls(self): + a = env_semantic() + self.assertEqual(action_sha256(**a), action_sha256(**a)) + + def test_payload_change_changes_hash(self): + a = env_semantic(payload={"content": "x"}) + b = env_semantic(payload={"content": "y"}) + self.assertNotEqual(action_sha256(**a), action_sha256(**b)) + + def test_basis_change_changes_hash(self): + a = env_semantic(action="confirm_summary", basis={"summary_sha256": "0" * 64}) + b = env_semantic(action="confirm_summary", basis={"summary_sha256": "1" * 64}) + self.assertNotEqual(action_sha256(**a), action_sha256(**b)) + + def test_action_id_is_semantic(self): + a = env_semantic(action_id="act_1") + b = env_semantic(action_id="act_2") + self.assertNotEqual(action_sha256(**a), action_sha256(**b)) + + def test_expected_revision_excluded(self): + # The function has no expected_revision parameter by design (ADR-0012 §G): + # a retry with an updated revision and the same semantic fields must + # hash identically so it replays instead of conflicting. + semantic = env_semantic() + h = action_sha256(**semantic) + # Simulate the transport carrying different expected_revision values: + # the semantic dict is unchanged, so the hash is unchanged. + self.assertEqual(h, action_sha256(**semantic)) + + def test_unicode_payload_stable(self): + a = env_semantic(payload={"content": "สวัสดี"}) + self.assertEqual(action_sha256(**a), action_sha256(**a)) + + def test_is_64_hex(self): + h = action_sha256(**env_semantic()) + self.assertEqual(len(h), 64) + int(h, 16) + + +if __name__ == "__main__": + unittest.main() diff --git a/methodfactory/tests/test_limits.py b/methodfactory/tests/test_limits.py new file mode 100644 index 0000000..22b7110 --- /dev/null +++ b/methodfactory/tests/test_limits.py @@ -0,0 +1,58 @@ +"""Size-bound constant tests (ADR-0012 §4): limits separated by object type.""" + +from __future__ import annotations + +import unittest + +from methodfactory.storage.limits import ( + MAX_ACTION_JSON_BYTES, + MAX_ARTIFACT_BYTES, + MAX_CONTENT_CHARS, + MAX_ENVELOPE_BYTES, + MAX_ID_CHARS, + MAX_INTENT_CHARS, + MAX_LOGICAL_PATH_CHARS, + MAX_MANIFEST_BYTES, + MAX_OUTCOMES, + MAX_REASON_CHARS, + MAX_STATEMENT_CHARS, +) + + +class LimitsTests(unittest.TestCase): + def test_all_positive(self): + for v in ( + MAX_ENVELOPE_BYTES, + MAX_ACTION_JSON_BYTES, + MAX_MANIFEST_BYTES, + MAX_ARTIFACT_BYTES, + MAX_CONTENT_CHARS, + MAX_INTENT_CHARS, + MAX_STATEMENT_CHARS, + MAX_ID_CHARS, + MAX_LOGICAL_PATH_CHARS, + MAX_REASON_CHARS, + MAX_OUTCOMES, + ): + self.assertGreater(v, 0, f"{v} must be positive") + + def test_envelope_limit_never_reused_for_event_or_manifest(self): + # ADR-0012 §4: never reuse MAX_ENVELOPE_BYTES as an event/manifest limit. + self.assertNotEqual(MAX_ENVELOPE_BYTES, MAX_MANIFEST_BYTES) + self.assertNotEqual(MAX_ENVELOPE_BYTES, MAX_ACTION_JSON_BYTES) + self.assertLess(MAX_ENVELOPE_BYTES, MAX_MANIFEST_BYTES) + + def test_ordering_by_object_type(self): + # Action JSON is smaller than a cumulative manifest; artifacts are + # separately bounded and can be the largest object. + self.assertLessEqual(MAX_ACTION_JSON_BYTES, MAX_MANIFEST_BYTES) + self.assertGreaterEqual(MAX_ARTIFACT_BYTES, MAX_MANIFEST_BYTES) + + def test_content_field_limits_are_sane(self): + self.assertLess(MAX_ID_CHARS, MAX_STATEMENT_CHARS) + self.assertLess(MAX_STATEMENT_CHARS, MAX_CONTENT_CHARS) + self.assertLess(MAX_LOGICAL_PATH_CHARS, MAX_ID_CHARS * 3) + + +if __name__ == "__main__": + unittest.main() diff --git a/methodfactory/tests/test_paths.py b/methodfactory/tests/test_paths.py new file mode 100644 index 0000000..c29f5c1 --- /dev/null +++ b/methodfactory/tests/test_paths.py @@ -0,0 +1,64 @@ +"""Store-root and package-id path validation tests (ADR-0012 §D).""" + +from __future__ import annotations + +import unittest +from pathlib import Path + +from methodfactory.storage.errors import InvalidPackageIdError, InvalidStoreRootError +from methodfactory.storage.paths import ( + DB_FILENAME, + database_path, + validate_package_id, + validate_store_root, +) + + +class PackageIdValidationTests(unittest.TestCase): + def test_accepts_valid_ids(self): + for pid in ("pkg_demo_001", "pkg_a", "pkg_x-y_9"): + self.assertEqual(validate_package_id(pid), pid) + + def test_rejects_invalid_ids(self): + for pid in ( + "../evil", + "core", + "pkg", # too short after prefix + "pkg_" + "x" * 64, # too long + "pkg/a/b", + "pkg with space", + "", + None, + 123, + ): + with self.subTest(pid=pid): + with self.assertRaises(InvalidPackageIdError): + validate_package_id(pid) # type: ignore[arg-type] + + +class StoreRootValidationTests(unittest.TestCase): + def test_database_path_is_beneath_root(self): + root = Path("/tmp/mf-store-test") + self.assertEqual(database_path(root), root / DB_FILENAME) + self.assertEqual(DB_FILENAME, "methodfactory.sqlite3") + + def test_empty_root_rejected(self): + with self.assertRaises(InvalidStoreRootError): + validate_store_root("") + + def test_file_root_rejected(self): + import tempfile + + with tempfile.NamedTemporaryFile() as fh: + with self.assertRaises(InvalidStoreRootError): + validate_store_root(Path(fh.name)) + + def test_directory_root_accepted(self): + import tempfile + + with tempfile.TemporaryDirectory() as td: + self.assertEqual(validate_store_root(Path(td)), Path(td)) + + +if __name__ == "__main__": + unittest.main() diff --git a/methodfactory/tests/test_serialization.py b/methodfactory/tests/test_serialization.py new file mode 100644 index 0000000..65d0c19 --- /dev/null +++ b/methodfactory/tests/test_serialization.py @@ -0,0 +1,60 @@ +"""Canonical serialization tests (ADR-0012 §4).""" + +from __future__ import annotations + +import unittest + +from hypothesis import given, strategies as st + +from methodfactory.storage.serialization import ( + canonical_bytes, + canonical_json, + digest_bytes, + digest_json, + digest_text, +) + + +class CanonicalSerializationTests(unittest.TestCase): + def test_key_order_invariant(self): + a = {"z": 1, "a": {"nested": 2, "list": [3, 1]}} + b = {"a": {"list": [3, 1], "nested": 2}, "z": 1} + self.assertEqual(digest_json(a), digest_json(b)) + self.assertEqual(canonical_json(a), canonical_json(b)) + + def test_compact_separators(self): + self.assertEqual(canonical_json({"a": 1, "b": [1, 2]}), '{"a":1,"b":[1,2]}') + + def test_ensure_ascii_false(self): + # Canonical form is UTF-8 (ensure_ascii=False): raw non-ASCII, not \\uXXXX. + s = canonical_json({"s": "สวัสดี"}) + self.assertIn("สวัสดี", s) + self.assertNotIn("\\u0e2a", s) + # ...and the byte form round-trips as UTF-8. + self.assertEqual(canonical_bytes({"s": "สวัสดี"}).decode("utf-8"), s) + + def test_invalid_numbers_rejected(self): + with self.assertRaises(ValueError): + canonical_json({"x": float("nan")}) + with self.assertRaises(ValueError): + canonical_json({"x": float("inf")}) + + def test_digests_are_64_hex(self): + for d in (digest_bytes(b"x"), digest_text("x"), digest_json({"a": 1})): + self.assertEqual(len(d), 64) + int(d, 16) # hex + + def test_digest_text_stable(self): + self.assertEqual(digest_text("สวัสดี"), digest_text("สวัสดี")) + self.assertNotEqual(digest_text("สวัสดี"), digest_text("hello")) + + @given(st.integers(min_value=-10**6, max_value=10**6), st.integers(min_value=-10**6, max_value=10**6)) + def test_key_order_invariant_property(self, x: int, y: int): + self.assertEqual( + digest_json({"a": x, "b": y}), + digest_json({"b": y, "a": x}), + ) + + +if __name__ == "__main__": + unittest.main() From e442929b34663b2b0ca56b6df2187cf345b3501a Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Fri, 7 Aug 2026 05:28:48 +0700 Subject: [PATCH 05/41] feat: add SQLite schema creation, identity checks, and append-only guards MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 2 foundation commit (ADR-0012 implementation order step 4). Implements only the schema/open/identity/guards surface; NOT the transactional create/apply, migration, or export (later commits). - methodfactory/storage/sqlite.py: * canonical filename methodfactory.sqlite3 beneath store root (ADR-0012 §D); * fixed application_id 0x4D465354 ('MFST'), user_version 1; * binding DDL: store_metadata (WITHOUT ROWID) + events (WITHOUT ROWID, PK (package_id, revision), UNIQUE (package_id, action_id), event_id UNIQUE, action_json + manifest_json BLOBs, no event_json duplicate); * append-only triggers events_no_update / events_no_delete (RAISE ABORT) - immutability enforced by the database (ADR-0012 §E); * operating mode: journal_mode DELETE, synchronous FULL, busy_timeout 5000, foreign_keys ON (ADR-0012 §3); parent dir 0700, db file 0600; * state detection (detect_presence): no_store / legacy_only / sqlite_only / both (ADR-0012 §D state table); * open contract: rw+NO_STORE creates+initializes; rw+LEGACY_ONLY -> LEGACY_STORE_DETECTED (no silent migration); zero-byte rw initializes / ro DATABASE_EMPTY; wrong application_id -> DATABASE_ID_MISMATCH; user_version >1 -> UNSUPPORTED_SCHEMA; both -> SQLite canonical; * read-only URI (mode=ro) never creates; validation never touches the file; * schema initialization in one transaction (idempotent); * latest_event query (SELECT manifest_json ... ORDER BY revision DESC LIMIT 1) + explain_latest_event_plan helper. - Tests: * test_sqlite_open: full state table (fresh/missing/zero-byte/wrong-id/ future-version/legacy-only/sqlite-only/both/neither), file modes, read-only non-mutation; * test_sqlite_append_only: INSERT ok, UPDATE/DELETE rejected by triggers, PK + action_id unique constraints; * test_sqlite_query_plan: EXPLAIN QUERY PLAN uses PRIMARY KEY, no SCAN. Local 3.11: 115 tests OK. EXPLAIN QUERY PLAN: 'SEARCH events USING PRIMARY KEY (package_id=?)'. --- methodfactory/storage/sqlite.py | 279 ++++++++++++++++++ .../tests/test_sqlite_append_only.py | 108 +++++++ methodfactory/tests/test_sqlite_open.py | 170 +++++++++++ methodfactory/tests/test_sqlite_query_plan.py | 43 +++ 4 files changed, 600 insertions(+) create mode 100644 methodfactory/storage/sqlite.py create mode 100644 methodfactory/tests/test_sqlite_append_only.py create mode 100644 methodfactory/tests/test_sqlite_open.py create mode 100644 methodfactory/tests/test_sqlite_query_plan.py diff --git a/methodfactory/storage/sqlite.py b/methodfactory/storage/sqlite.py new file mode 100644 index 0000000..8fd39c0 --- /dev/null +++ b/methodfactory/storage/sqlite.py @@ -0,0 +1,279 @@ +"""SQLite schema creation, identity checks, and append-only guards. + +Phase 2 (ADR-0012 commits 2–4 scope; §1 schema, §3 operating mode, §D physical +database identity, §E append-only). Implements only: + +- database creation/opening; +- canonical filename/location; +- parent mode 0700, database mode 0600; +- fixed application_id and user_version=1; +- journal_mode=DELETE, synchronous=FULL, busy_timeout, foreign_keys; +- binding store_metadata + events DDL with append-only triggers; +- database-state detection (missing, zero-byte, wrong-ID, future-version, + corrupt, legacy-only, sqlite-only, neither, both); +- read-only URI opening with no accidental creation; +- schema initialization transaction; +- latest-event query + EXPLAIN QUERY PLAN helper. + +The transactional create/apply, idempotent replay, v0.1.2 migration, and +deterministic export are intentionally NOT implemented here (later Phase 2 +commits). +""" + +from __future__ import annotations + +import os +import sqlite3 +from datetime import datetime, timezone +from enum import Enum +from pathlib import Path + +from .errors import ( + DatabaseEmptyError, + DatabaseIdMismatchError, + DatabaseNotFoundError, + LegacyStoreDetectedError, + StorageError, + UnsupportedSchemaError, +) +from .paths import DB_FILENAME, database_path, validate_store_root + +# ── Physical database identity (ADR-0012 §D) ──────────────────────────── +APPLICATION_ID = 0x4D465354 # "MFST" — canonical Method Factory application id +USER_VERSION = 1 # accepted schema version; >1 is unsupported +APPLICATION_ID_DECIMAL = int(APPLICATION_ID) # for documentation/tests + +# ── Binding DDL (ADR-0012 §1, §E) ─────────────────────────────────────── +STORE_METADATA_DDL = """ +CREATE TABLE IF NOT EXISTS store_metadata ( + key TEXT PRIMARY KEY, + value TEXT NOT NULL +) WITHOUT ROWID; +""" + +EVENTS_DDL = """ +CREATE TABLE IF NOT EXISTS events ( + package_id TEXT NOT NULL, + revision INTEGER NOT NULL CHECK (revision >= 0), + + event_id TEXT NOT NULL UNIQUE, + action_id TEXT NOT NULL, + action TEXT NOT NULL, + action_sha256 TEXT NOT NULL, + + state_before TEXT, + state_after TEXT NOT NULL, + + previous_manifest_sha256 TEXT, + resulting_manifest_sha256 TEXT NOT NULL, + + created_at TEXT NOT NULL, + + action_json BLOB NOT NULL, + manifest_json BLOB NOT NULL, + + PRIMARY KEY (package_id, revision), + UNIQUE (package_id, action_id) +) WITHOUT ROWID; +""" + +APPEND_ONLY_TRIGGERS_DDL = """ +CREATE TRIGGER IF NOT EXISTS events_no_update +BEFORE UPDATE ON events +FOR EACH ROW +BEGIN + SELECT RAISE(ABORT, 'events are append-only: UPDATE not permitted'); +END; + +CREATE TRIGGER IF NOT EXISTS events_no_delete +BEFORE DELETE ON events +FOR EACH ROW +BEGIN + SELECT RAISE(ABORT, 'events are append-only: DELETE not permitted'); +END; +""" + +SCHEMA_DDL = STORE_METADATA_DDL + EVENTS_DDL + APPEND_ONLY_TRIGGERS_DDL + +# Current-state lookup (indexed by the composite primary key). +LATEST_EVENT_SQL = """ +SELECT manifest_json +FROM events +WHERE package_id = ? +ORDER BY revision DESC +LIMIT 1; +""" + +# Public v0.1.2 legacy layout directories (ADR-0012 §I). +LEGACY_DIRS = ("packages", "events", "artifacts") + + +class StorePresence(str, Enum): + NO_STORE = "no_store" + LEGACY_ONLY = "legacy_only" + SQLITE_ONLY = "sqlite_only" + BOTH = "both" + + +def _utcnow() -> str: + return datetime.now(timezone.utc).isoformat() + + +def detect_presence(root: Path | str) -> StorePresence: + """Classify the store root by physical presence (ADR-0012 §D).""" + r = validate_store_root(root) + db = r / DB_FILENAME + has_db = db.exists() + legacy = any((r / d).exists() for d in LEGACY_DIRS) + if has_db and legacy: + return StorePresence.BOTH + if has_db: + return StorePresence.SQLITE_ONLY + if legacy: + return StorePresence.LEGACY_ONLY + return StorePresence.NO_STORE + + +def _connect(db: Path, read_only: bool, timeout: float = 5.0) -> sqlite3.Connection: + if read_only: + conn = sqlite3.connect(f"file:{db}?mode=ro", uri=True, timeout=timeout) + else: + conn = sqlite3.connect(str(db), timeout=timeout) + conn.row_factory = sqlite3.Row + conn.execute("PRAGMA busy_timeout = 5000") + conn.execute("PRAGMA foreign_keys = ON") + return conn + + +def _identity(conn: sqlite3.Connection) -> tuple[int, int]: + app_id = conn.execute("PRAGMA application_id").fetchone()[0] + user_version = conn.execute("PRAGMA user_version").fetchone()[0] + return int(app_id), int(user_version) + + +def initialize_database(conn: sqlite3.Connection) -> None: + """Initialize the schema and identity in one transaction (idempotent).""" + with conn: # implicit BEGIN ... COMMIT / ROLLBACK + conn.executescript(SCHEMA_DDL) + conn.execute(f"PRAGMA application_id = {APPLICATION_ID}") + conn.execute(f"PRAGMA user_version = {USER_VERSION}") + conn.execute( + "INSERT OR IGNORE INTO store_metadata (key, value) VALUES ('schema_version', ?)", + (str(USER_VERSION),), + ) + conn.execute( + "INSERT OR IGNORE INTO store_metadata (key, value) VALUES ('created_at', ?)", + (_utcnow(),), + ) + + +def open_database(root: Path | str, read_only: bool = False) -> sqlite3.Connection: + """Open (and, on the read-write path, create/initialize) the canonical DB. + + Contract (ADR-0012 §D): + + - read-write + NO_STORE: create parent (0700), create DB (0600), initialize. + - read-write + LEGACY_ONLY: raise LegacyStoreDetectedError (never migrate + silently; instruct `mf migrate-store`). + - read-write + SQLITE_ONLY/BOTH: open, verify identity; zero-byte initializes; + wrong application_id -> DatabaseIdMismatchError; user_version > 1 -> + UnsupportedSchemaError. + - read-only: never creates; NO_STORE -> DatabaseNotFoundError; LEGACY_ONLY + -> LegacyStoreDetectedError; zero-byte -> DatabaseEmptyError; identity + checks as above. + """ + r = validate_store_root(root) + db = r / DB_FILENAME + presence = detect_presence(r) + + if read_only: + if presence == StorePresence.LEGACY_ONLY: + raise LegacyStoreDetectedError( + "v0.1.2 JSONL store detected; run `mf migrate-store`", + ) + if presence in (StorePresence.NO_STORE,): + raise DatabaseNotFoundError( + f"no database at {db}", + ) + try: + conn = _connect(db, read_only=True) + except sqlite3.OperationalError as exc: + raise DatabaseNotFoundError(f"cannot open {db} read-only: {exc}") from exc + _verify_identity(conn, db, allow_zero_byte=False) + return conn + + # read-write path + if presence == StorePresence.LEGACY_ONLY: + raise LegacyStoreDetectedError( + "v0.1.2 JSONL store detected; run `mf migrate-store`", + ) + if presence in (StorePresence.NO_STORE,): + r.mkdir(parents=True, exist_ok=True) + try: + os.chmod(r, 0o700) + except OSError: + pass + conn = _connect(db, read_only=False) + initialize_database(conn) + try: + os.chmod(db, 0o600) + except OSError: + pass + return conn + + # SQLITE_ONLY or BOTH: SQLite is canonical and used (ADR-0012 §D). + conn = _connect(db, read_only=False) + _verify_identity(conn, db, allow_zero_byte=True) + return conn + + +def _verify_identity(conn: sqlite3.Connection, db: Path, allow_zero_byte: bool) -> None: + size = db.stat().st_size if db.exists() else 0 + if size == 0: + if allow_zero_byte: + initialize_database(conn) + try: + os.chmod(db, 0o600) + except OSError: + pass + return + conn.close() + raise DatabaseEmptyError(f"database {db} is zero bytes") + + app_id, user_version = _identity(conn) + if app_id != APPLICATION_ID: + conn.close() + raise DatabaseIdMismatchError( + f"database {db} has application_id {app_id}, expected {APPLICATION_ID}", + ) + if user_version > USER_VERSION: + conn.close() + raise UnsupportedSchemaError( + f"database {db} has user_version {user_version}, supported <= {USER_VERSION}", + ) + if user_version == 0: + # Partially initialized (identity set but version not): initialize. + initialize_database(conn) + return + + +def latest_event(conn: sqlite3.Connection, package_id: str) -> dict | None: + """Return the latest manifest for a package (indexed latest-event read).""" + row = conn.execute(LATEST_EVENT_SQL, (package_id,)).fetchone() + if row is None: + return None + import json + + return json.loads(row["manifest_json"]) + + +def explain_latest_event_plan(conn: sqlite3.Connection, package_id: str) -> list[tuple]: + """EXPLAIN QUERY PLAN for the latest-event lookup (ADR-0012 §9 item 14).""" + rows = conn.execute( + f"EXPLAIN QUERY PLAN {LATEST_EVENT_SQL}", (package_id,) + ).fetchall() + return [tuple(r) for r in rows] + + +def close_database(conn: sqlite3.Connection) -> None: + conn.close() diff --git a/methodfactory/tests/test_sqlite_append_only.py b/methodfactory/tests/test_sqlite_append_only.py new file mode 100644 index 0000000..6dc2399 --- /dev/null +++ b/methodfactory/tests/test_sqlite_append_only.py @@ -0,0 +1,108 @@ +"""Append-only guard tests (ADR-0012 §E): UPDATE/DELETE rejected by triggers.""" + +from __future__ import annotations + +import sqlite3 +import tempfile +import unittest +from pathlib import Path + +from methodfactory.storage.paths import DB_FILENAME +from methodfactory.storage.sqlite import close_database, open_database + + +def make_event(package_id="pkg_demo_001", revision=0, action_id="act_1"): + return ( + package_id, + revision, + f"evt_{package_id}_{revision}", + action_id, + "create_package", + "0" * 64, + None, + "INTAKE", + None, + "0" * 64, + "2026-08-07T00:00:00+00:00", + b'{"action":"create_package"}', + b'{"schema_version":"0.1"}', + ) + + +INSERT_EVENT_SQL = """ +INSERT INTO events ( + package_id, revision, event_id, action_id, action, action_sha256, + state_before, state_after, previous_manifest_sha256, + resulting_manifest_sha256, created_at, action_json, manifest_json +) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) +""" + + +class AppendOnlyTests(unittest.TestCase): + def _db(self, td): + root = Path(td) + conn = open_database(root, read_only=False) + return root, conn + + def test_insert_works(self): + with tempfile.TemporaryDirectory() as td: + _, conn = self._db(td) + try: + conn.execute(INSERT_EVENT_SQL, make_event()) + conn.commit() + n = conn.execute("SELECT COUNT(*) FROM events").fetchone()[0] + self.assertEqual(n, 1) + finally: + close_database(conn) + + def test_update_rejected_by_trigger(self): + with tempfile.TemporaryDirectory() as td: + _, conn = self._db(td) + try: + conn.execute(INSERT_EVENT_SQL, make_event()) + conn.commit() + with self.assertRaises(sqlite3.IntegrityError) as ctx: + conn.execute( + "UPDATE events SET state_after = 'CANCELLED' WHERE package_id = 'pkg_demo_001'" + ) + self.assertIn("append-only", str(ctx.exception).lower()) + finally: + close_database(conn) + + def test_delete_rejected_by_trigger(self): + with tempfile.TemporaryDirectory() as td: + _, conn = self._db(td) + try: + conn.execute(INSERT_EVENT_SQL, make_event()) + conn.commit() + with self.assertRaises(sqlite3.IntegrityError) as ctx: + conn.execute("DELETE FROM events WHERE package_id = 'pkg_demo_001'") + self.assertIn("append-only", str(ctx.exception).lower()) + finally: + close_database(conn) + + def test_primary_key_duplicate_rejected(self): + with tempfile.TemporaryDirectory() as td: + _, conn = self._db(td) + try: + conn.execute(INSERT_EVENT_SQL, make_event()) + with self.assertRaises(sqlite3.IntegrityError): + conn.execute(INSERT_EVENT_SQL, make_event(revision=0, action_id="act_2")) + conn.rollback() + finally: + close_database(conn) + + def test_action_id_unique_rejected(self): + with tempfile.TemporaryDirectory() as td: + _, conn = self._db(td) + try: + conn.execute(INSERT_EVENT_SQL, make_event(revision=0)) + with self.assertRaises(sqlite3.IntegrityError): + conn.execute(INSERT_EVENT_SQL, make_event(revision=1, action_id="act_1")) + conn.rollback() + finally: + close_database(conn) + + +if __name__ == "__main__": + unittest.main() diff --git a/methodfactory/tests/test_sqlite_open.py b/methodfactory/tests/test_sqlite_open.py new file mode 100644 index 0000000..cc87bde --- /dev/null +++ b/methodfactory/tests/test_sqlite_open.py @@ -0,0 +1,170 @@ +"""SQLite open/identity/state tests (ADR-0012 §D state table).""" + +from __future__ import annotations + +import os +import sqlite3 +import tempfile +import unittest +from pathlib import Path + +from methodfactory.storage.errors import ( + DatabaseEmptyError, + DatabaseIdMismatchError, + DatabaseNotFoundError, + LegacyStoreDetectedError, + UnsupportedSchemaError, +) +from methodfactory.storage.paths import DB_FILENAME +from methodfactory.storage.sqlite import ( + APPLICATION_ID, + USER_VERSION, + close_database, + detect_presence, + initialize_database, + open_database, +) + + +def _make_legacy_dirs(root: Path) -> None: + for d in ("packages", "events", "artifacts"): + (root / d).mkdir(parents=True, exist_ok=True) + + +class SqliteOpenTests(unittest.TestCase): + def test_fresh_rw_creates_and_initializes(self): + with tempfile.TemporaryDirectory() as td: + root = Path(td) + conn = open_database(root, read_only=False) + try: + app_id = conn.execute("PRAGMA application_id").fetchone()[0] + uv = conn.execute("PRAGMA user_version").fetchone()[0] + self.assertEqual(int(app_id), APPLICATION_ID) + self.assertEqual(int(uv), USER_VERSION) + tables = {r[0] for r in conn.execute( + "SELECT name FROM sqlite_master WHERE type='table'")} + self.assertIn("store_metadata", tables) + self.assertIn("events", tables) + triggers = {r[0] for r in conn.execute( + "SELECT name FROM sqlite_master WHERE type='trigger'")} + self.assertIn("events_no_update", triggers) + self.assertIn("events_no_delete", triggers) + meta = {r["key"]: r["value"] for r in conn.execute( + "SELECT key, value FROM store_metadata")} + self.assertEqual(meta.get("schema_version"), str(USER_VERSION)) + finally: + close_database(conn) + + def test_file_and_dir_modes(self): + with tempfile.TemporaryDirectory() as td: + root = Path(td) + conn = open_database(root, read_only=False) + close_database(conn) + self.assertEqual(os.stat(root).st_mode & 0o777, 0o700) + self.assertEqual(os.stat(root / DB_FILENAME).st_mode & 0o777, 0o600) + + def test_ro_missing_does_not_create(self): + with tempfile.TemporaryDirectory() as td: + root = Path(td) + with self.assertRaises(DatabaseNotFoundError): + open_database(root, read_only=True) + self.assertFalse((root / DB_FILENAME).exists()) + + def test_ro_neither_raises_not_found(self): + with tempfile.TemporaryDirectory() as td: + with self.assertRaises(DatabaseNotFoundError): + open_database(Path(td), read_only=True) + + def test_legacy_only_rw_and_ro_raise(self): + with tempfile.TemporaryDirectory() as td: + root = Path(td) + _make_legacy_dirs(root) + with self.assertRaises(LegacyStoreDetectedError): + open_database(root, read_only=False) + with self.assertRaises(LegacyStoreDetectedError): + open_database(root, read_only=True) + self.assertFalse((root / DB_FILENAME).exists()) + + def test_zero_byte_rw_initializes(self): + with tempfile.TemporaryDirectory() as td: + root = Path(td) + (root / DB_FILENAME).write_bytes(b"") + conn = open_database(root, read_only=False) + try: + self.assertEqual(int(conn.execute("PRAGMA user_version").fetchone()[0]), USER_VERSION) + finally: + close_database(conn) + + def test_zero_byte_ro_raises_empty(self): + with tempfile.TemporaryDirectory() as td: + root = Path(td) + (root / DB_FILENAME).write_bytes(b"") + with self.assertRaises(DatabaseEmptyError): + open_database(root, read_only=True) + + def test_wrong_application_id_raises(self): + with tempfile.TemporaryDirectory() as td: + root = Path(td) + db = root / DB_FILENAME + c = sqlite3.connect(str(db)) + c.execute(f"PRAGMA application_id = {0xDEADBEEF}") + c.execute("CREATE TABLE junk (x)") + c.commit() + c.close() + with self.assertRaises(DatabaseIdMismatchError): + open_database(root, read_only=False) + with self.assertRaises(DatabaseIdMismatchError): + open_database(root, read_only=True) + + def test_future_user_version_raises(self): + with tempfile.TemporaryDirectory() as td: + root = Path(td) + db = root / DB_FILENAME + c = sqlite3.connect(str(db)) + c.execute(f"PRAGMA application_id = {APPLICATION_ID}") + c.execute(f"PRAGMA user_version = {USER_VERSION + 1}") + c.execute("CREATE TABLE t (x)") + c.commit() + c.close() + with self.assertRaises(UnsupportedSchemaError): + open_database(root, read_only=False) + with self.assertRaises(UnsupportedSchemaError): + open_database(root, read_only=True) + + def test_both_present_uses_sqlite(self): + with tempfile.TemporaryDirectory() as td: + root = Path(td) + conn0 = open_database(root, read_only=False) # create SQLite first + close_database(conn0) + _make_legacy_dirs(root) # then legacy dirs -> BOTH + conn = open_database(root, read_only=False) + try: + self.assertEqual(int(conn.execute("PRAGMA user_version").fetchone()[0]), USER_VERSION) + finally: + close_database(conn) + + def test_detect_presence(self): + with tempfile.TemporaryDirectory() as td: + root = Path(td) + self.assertEqual(detect_presence(root).value, "no_store") + conn = open_database(root, read_only=False) + close_database(conn) + self.assertEqual(detect_presence(root).value, "sqlite_only") + _make_legacy_dirs(root) + self.assertEqual(detect_presence(root).value, "both") + + def test_ro_open_on_valid_db_works_and_does_not_mutate(self): + with tempfile.TemporaryDirectory() as td: + root = Path(td) + conn0 = open_database(root, read_only=False) + initialize_database(conn0) + close_database(conn0) + before = (root / DB_FILENAME).stat().st_mtime_ns + conn = open_database(root, read_only=True) + close_database(conn) + after = (root / DB_FILENAME).stat().st_mtime_ns + self.assertEqual(before, after) # read-only open did not touch the file + + +if __name__ == "__main__": + unittest.main() diff --git a/methodfactory/tests/test_sqlite_query_plan.py b/methodfactory/tests/test_sqlite_query_plan.py new file mode 100644 index 0000000..724445d --- /dev/null +++ b/methodfactory/tests/test_sqlite_query_plan.py @@ -0,0 +1,43 @@ +"""Indexed latest-event query evidence (ADR-0012 §9 item 14). + +Asserts the query plan uses the package/revision primary key and does not +scan the complete history — the hot path must stay bounded. +""" + +from __future__ import annotations + +import tempfile +import unittest +from pathlib import Path + +from methodfactory.storage.sqlite import ( + close_database, + explain_latest_event_plan, + latest_event, + open_database, +) + + +class QueryPlanTests(unittest.TestCase): + def test_latest_event_plan_uses_primary_key(self): + with tempfile.TemporaryDirectory() as td: + conn = open_database(Path(td), read_only=False) + try: + plan = explain_latest_event_plan(conn, "pkg_demo_001") + text = " ".join(" ".join(str(c) for c in row) for row in plan) + self.assertIn("SEARCH events USING", text) + self.assertNotIn("SCAN events", text) + finally: + close_database(conn) + + def test_latest_event_empty_returns_none(self): + with tempfile.TemporaryDirectory() as td: + conn = open_database(Path(td), read_only=False) + try: + self.assertIsNone(latest_event(conn, "pkg_demo_001")) + finally: + close_database(conn) + + +if __name__ == "__main__": + unittest.main() From b0dd2ca922006c63287b7ade39e60722eff3eecf Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Fri, 7 Aug 2026 07:00:09 +0700 Subject: [PATCH 06/41] fix(storage): Finding 1 - SQLite open and initialization contract executable Closes senior review 4878620791 Finding 1 (Phase 2 gate blocker). - Apply AND read back every binding PRAGMA: journal_mode=DELETE (actively re-established on rw open so a previously-WAL DB is reset), synchronous=FULL (2), busy_timeout=5000, foreign_keys=ON (rw; ro connections verify connection-scoped pragmas where meaningful and accept the per-connection default for foreign_keys, which cannot be set read-only). Any required mode that cannot be established raises StorageError (typed), never silent. - Read-only connections verify without attempting mutation. - initialize_database is now ONE explicit atomic operation: BEGIN IMMEDIATE + individual execute() (no executescript() implicit commit boundary); any failure rolls back so no partial schema is accepted. Triggers execute individually (execute() allows one statement at a time). - Only a genuinely new/zero-byte file is initialized. A non-zero DB with the canonical application_id and user_version=0 is REJECTED (UnsupportedSchema) - no recovery path is specified yet (ADR-0012 accepts user_version=1 only). - One authoritative schema verifier (_verify_schema) checks: required tables + columns + WITHOUT ROWID, primary key, unique constraints (incl. event_id autoindex), append-only triggers, metadata (schema_version/created_at), application ID, and exact user_version. Drift -> SchemaViolationError. - Read-only SQLite URIs built with urllib.parse.quote (spaces/Unicode/?/#/% escaped); proven to open correctly and create no sibling/alternate file. - Legacy detection requires the COMPLETE frozen v0.1.2 layout (packages/ AND events/ AND artifacts/); single-dir partial layouts are NOT legacy. - Store-root 0700 and database 0600 modes enforced or fail typed (chmod failure -> StorageError, not silent). Tests: 18 (PRAGMA read-back incl. WAL-reopen reset + ro typed failure, atomic-init fault, version-0 rejection, schema/trigger/metadata drift, URI-significant paths, complete/partial legacy layouts, both-present, mode enforcement + incorrect-mode correction). Full suite 121 OK. --- methodfactory/storage/sqlite.py | 415 +++++++++++++++++++----- methodfactory/tests/test_sqlite_open.py | 289 ++++++++++++----- 2 files changed, 542 insertions(+), 162 deletions(-) diff --git a/methodfactory/storage/sqlite.py b/methodfactory/storage/sqlite.py index 8fd39c0..a792abb 100644 --- a/methodfactory/storage/sqlite.py +++ b/methodfactory/storage/sqlite.py @@ -1,23 +1,27 @@ """SQLite schema creation, identity checks, and append-only guards. -Phase 2 (ADR-0012 commits 2–4 scope; §1 schema, §3 operating mode, §D physical -database identity, §E append-only). Implements only: - -- database creation/opening; -- canonical filename/location; -- parent mode 0700, database mode 0600; -- fixed application_id and user_version=1; -- journal_mode=DELETE, synchronous=FULL, busy_timeout, foreign_keys; -- binding store_metadata + events DDL with append-only triggers; -- database-state detection (missing, zero-byte, wrong-ID, future-version, - corrupt, legacy-only, sqlite-only, neither, both); -- read-only URI opening with no accidental creation; -- schema initialization transaction; -- latest-event query + EXPLAIN QUERY PLAN helper. +Phase 2 corrections (senior review 4878620791, Finding 1). Applies the +binding first-release SQLite contract exactly as specified: + +- every binding connection PRAGMA is applied AND read back; a required mode + that cannot be established fails typed (StorageError), never silently; +- read-only connections VERIFY PRAGMAs without attempting mutation; +- first initialization is one explicit atomic operation (no implicit + executescript() commit boundary); only a genuinely new/zero-byte file is + initialized; +- a non-zero database with application_id MFST and user_version=0 is REJECTED + (no recovery path is specified yet; ADR-0012 accepts only user_version=1); +- one authoritative schema verifier checks tables, columns, constraints/ + indexes, triggers, metadata, application ID, and exact version; +- read-only SQLite URIs are built with correct path escaping (uri_quote) + so spaces/Unicode/?/#/% cannot change the target; read-only opens never + create any file (proven by tests); +- legacy detection requires the complete frozen v0.1.2 layout, not any() + single directory; +- store-root (0700) and database (0600) modes are enforced or fail typed. The transactional create/apply, idempotent replay, v0.1.2 migration, and -deterministic export are intentionally NOT implemented here (later Phase 2 -commits). +deterministic export are intentionally NOT implemented here. """ from __future__ import annotations @@ -27,12 +31,14 @@ from datetime import datetime, timezone from enum import Enum from pathlib import Path +from urllib.parse import quote from .errors import ( DatabaseEmptyError, DatabaseIdMismatchError, DatabaseNotFoundError, LegacyStoreDetectedError, + SchemaViolationError, StorageError, UnsupportedSchemaError, ) @@ -43,16 +49,24 @@ USER_VERSION = 1 # accepted schema version; >1 is unsupported APPLICATION_ID_DECIMAL = int(APPLICATION_ID) # for documentation/tests +# ── Binding first-release operating mode (ADR-0012 §3) ────────────────── +BINDING_PRAGMAS = { + "journal_mode": "DELETE", + "synchronous": "FULL", + "busy_timeout": 5000, + "foreign_keys": 1, +} + # ── Binding DDL (ADR-0012 §1, §E) ─────────────────────────────────────── STORE_METADATA_DDL = """ -CREATE TABLE IF NOT EXISTS store_metadata ( +CREATE TABLE store_metadata ( key TEXT PRIMARY KEY, value TEXT NOT NULL ) WITHOUT ROWID; """ EVENTS_DDL = """ -CREATE TABLE IF NOT EXISTS events ( +CREATE TABLE events ( package_id TEXT NOT NULL, revision INTEGER NOT NULL CHECK (revision >= 0), @@ -77,23 +91,49 @@ ) WITHOUT ROWID; """ -APPEND_ONLY_TRIGGERS_DDL = """ -CREATE TRIGGER IF NOT EXISTS events_no_update +APPEND_ONLY_TRIGGERS_DDL = [ + """ +CREATE TRIGGER events_no_update BEFORE UPDATE ON events FOR EACH ROW BEGIN SELECT RAISE(ABORT, 'events are append-only: UPDATE not permitted'); END; - -CREATE TRIGGER IF NOT EXISTS events_no_delete +""", + """ +CREATE TRIGGER events_no_delete BEFORE DELETE ON events FOR EACH ROW BEGIN SELECT RAISE(ABORT, 'events are append-only: DELETE not permitted'); END; -""" - -SCHEMA_DDL = STORE_METADATA_DDL + EVENTS_DDL + APPEND_ONLY_TRIGGERS_DDL +""", +] + +SCHEMA_DDL = STORE_METADATA_DDL + EVENTS_DDL + "".join(APPEND_ONLY_TRIGGERS_DDL) + +# Authoritative schema expectations (Finding 1 item 4). +REQUIRED_TABLES = { + "store_metadata": { + "columns": {"key", "value"}, + "without_rowid": True, + }, + "events": { + "columns": { + "package_id", "revision", "event_id", "action_id", "action", + "action_sha256", "state_before", "state_after", + "previous_manifest_sha256", "resulting_manifest_sha256", + "created_at", "action_json", "manifest_json", + }, + "without_rowid": True, + "primary_key": ("package_id", "revision"), + "unique": {("package_id", "action_id"), ("event_id",)}, + }, +} + +REQUIRED_TRIGGERS = {"events_no_update", "events_no_delete"} + +REQUIRED_METADATA = {"schema_version", "created_at"} # Current-state lookup (indexed by the composite primary key). LATEST_EVENT_SQL = """ @@ -104,7 +144,8 @@ LIMIT 1; """ -# Public v0.1.2 legacy layout directories (ADR-0012 §I). +# Public v0.1.2 legacy layout directories (ADR-0012 §I). ALL must be present +# for a legacy store (Finding 1 item 6). LEGACY_DIRS = ("packages", "events", "artifacts") @@ -120,11 +161,16 @@ def _utcnow() -> str: def detect_presence(root: Path | str) -> StorePresence: - """Classify the store root by physical presence (ADR-0012 §D).""" + """Classify the store root by physical presence (ADR-0012 §D). + + A legacy store is detected only when the COMPLETE frozen v0.1.2 layout is + present (packages/ AND events/ AND artifacts/), not any single directory + (Finding 1 item 6). + """ r = validate_store_root(root) db = r / DB_FILENAME has_db = db.exists() - legacy = any((r / d).exists() for d in LEGACY_DIRS) + legacy = all((r / d).is_dir() for d in LEGACY_DIRS) if has_db and legacy: return StorePresence.BOTH if has_db: @@ -134,17 +180,103 @@ def detect_presence(root: Path | str) -> StorePresence: return StorePresence.NO_STORE +def _readonly_uri(db: Path) -> str: + """Build a read-only SQLite URI that correctly escapes path-significant + characters (spaces, Unicode, ?, #, %) — Finding 1 item 5.""" + # quote() with safe='' percent-encodes everything including ? # %; + # sqlite3 URI parsing then unquotes the path component. The 'file:' scheme + # requires an absolute path with forward slashes for the authority form. + path_part = str(db.resolve()) + if os.sep == "\\": + path_part = path_part.replace("\\", "/") + return f"file:{quote(path_part, safe='/')}?mode=ro" + + def _connect(db: Path, read_only: bool, timeout: float = 5.0) -> sqlite3.Connection: if read_only: - conn = sqlite3.connect(f"file:{db}?mode=ro", uri=True, timeout=timeout) + conn = sqlite3.connect(_readonly_uri(db), uri=True, timeout=timeout) else: conn = sqlite3.connect(str(db), timeout=timeout) conn.row_factory = sqlite3.Row - conn.execute("PRAGMA busy_timeout = 5000") - conn.execute("PRAGMA foreign_keys = ON") + _apply_or_verify_pragmas(conn, read_only=read_only) return conn +def _apply_or_verify_pragmas(conn: sqlite3.Connection, *, read_only: bool) -> None: + """Apply (rw) or verify (ro) the binding first-release PRAGMAs. + + A required mode that cannot be established raises StorageError (typed), + never silently proceeds. Read-only connections only verify: attempting to + set a PRAGMA on a read-only connection would mutate or fail, so we read + back and compare. busy_timeout/foreign_keys are per-connection; journal + and synchronous are database-level and must match the binding values. + """ + for pragma, expected in BINDING_PRAGMAS.items(): + if pragma == "journal_mode": + if read_only: + # Verify only: cannot set on ro; the binding value must already + # hold (a WAL DB fails typed rather than being silently accepted). + actual = conn.execute("PRAGMA journal_mode").fetchone()[0] + if str(actual).lower() != "delete": + raise StorageError( + f"binding journal_mode=DELETE not established (got {actual!r})" + ) + else: + # Actively establish DELETE (a previously-WAL DB must be reset). + conn.execute("PRAGMA journal_mode = DELETE") + actual = conn.execute("PRAGMA journal_mode").fetchone()[0] + if str(actual).lower() != "delete": + raise StorageError( + f"binding journal_mode=DELETE not established (got {actual!r})" + ) + continue + if pragma == "synchronous": + if read_only: + actual = conn.execute("PRAGMA synchronous").fetchone()[0] + if int(actual) != 2: # FULL == 2 + raise StorageError( + f"binding synchronous=FULL not established (got {actual!r})" + ) + else: + conn.execute("PRAGMA synchronous = FULL") + actual = conn.execute("PRAGMA synchronous").fetchone()[0] + if int(actual) != 2: + raise StorageError( + f"binding synchronous=FULL not established (got {actual!r})" + ) + continue + if pragma == "busy_timeout": + if read_only: + actual = conn.execute("PRAGMA busy_timeout").fetchone()[0] + if int(actual) != 5000: + raise StorageError( + f"binding busy_timeout=5000 not established (got {actual!r})" + ) + else: + conn.execute("PRAGMA busy_timeout = 5000") + actual = conn.execute("PRAGMA busy_timeout").fetchone()[0] + if int(actual) != 5000: + raise StorageError( + f"binding busy_timeout=5000 not established (got {actual!r})" + ) + continue + if pragma == "foreign_keys": + if read_only: + # foreign_keys is per-connection and defaults OFF; a ro + # connection cannot set it. It is not a database property, so + # verification accepts the per-connection default (the binding + # applies to rw connections which set it explicitly). + pass + else: + conn.execute("PRAGMA foreign_keys = ON") + actual = conn.execute("PRAGMA foreign_keys").fetchone()[0] + if int(actual) != 1: + raise StorageError( + f"binding foreign_keys=ON not established (got {actual!r})" + ) + continue + + def _identity(conn: sqlite3.Connection) -> tuple[int, int]: app_id = conn.execute("PRAGMA application_id").fetchone()[0] user_version = conn.execute("PRAGMA user_version").fetchone()[0] @@ -152,9 +284,18 @@ def _identity(conn: sqlite3.Connection) -> tuple[int, int]: def initialize_database(conn: sqlite3.Connection) -> None: - """Initialize the schema and identity in one transaction (idempotent).""" - with conn: # implicit BEGIN ... COMMIT / ROLLBACK - conn.executescript(SCHEMA_DDL) + """Initialize the schema and identity in ONE explicit atomic operation. + + Uses individual execute() calls inside an explicit transaction (no + executescript(), which has an implicit commit boundary). Any failure + rolls back so no partially initialized schema is accepted. + """ + conn.execute("BEGIN IMMEDIATE") + try: + conn.execute(STORE_METADATA_DDL) + conn.execute(EVENTS_DDL) + for trigger_ddl in APPEND_ONLY_TRIGGERS_DDL: + conn.execute(trigger_ddl) conn.execute(f"PRAGMA application_id = {APPLICATION_ID}") conn.execute(f"PRAGMA user_version = {USER_VERSION}") conn.execute( @@ -165,22 +306,143 @@ def initialize_database(conn: sqlite3.Connection) -> None: "INSERT OR IGNORE INTO store_metadata (key, value) VALUES ('created_at', ?)", (_utcnow(),), ) + conn.execute("COMMIT") + except BaseException: + try: + conn.execute("ROLLBACK") + except sqlite3.Error: + pass + raise + + +def _verify_schema(conn: sqlite3.Connection) -> None: + """Authoritative schema verifier (Finding 1 item 4). + + Checks required tables + columns, WITHOUT ROWID, primary key, unique + constraints, append-only triggers, metadata, application ID, and exact + user_version. Raises SchemaViolationError on any drift. + """ + app_id, user_version = _identity(conn) + if app_id != APPLICATION_ID: + raise DatabaseIdMismatchError( + f"database application_id {app_id}, expected {APPLICATION_ID}" + ) + if user_version != USER_VERSION: + raise UnsupportedSchemaError( + f"database user_version {user_version}, supported {USER_VERSION}" + ) + + tables = {} + for row in conn.execute( + "SELECT name, sql FROM sqlite_master WHERE type='table'" + ): + tables[row["name"]] = row["sql"] or "" + for tname, spec in REQUIRED_TABLES.items(): + if tname not in tables: + raise SchemaViolationError(f"required table {tname!r} missing") + # columns + cols = { + r["name"] + for r in conn.execute(f"PRAGMA table_info({tname})") + } + missing_cols = spec["columns"] - cols + if missing_cols: + raise SchemaViolationError( + f"table {tname!r} missing columns {sorted(missing_cols)}" + ) + # WITHOUT ROWID + if spec.get("without_rowid") and "WITHOUT ROWID" not in tables[tname].upper(): + raise SchemaViolationError(f"table {tname!r} is not WITHOUT ROWID") + + # primary key + unique constraints (via PRAGMA index_list / table_info) + for tname, spec in REQUIRED_TABLES.items(): + pk = spec.get("primary_key") + if pk is not None: + pk_cols = [ + r["name"] + for r in conn.execute(f"PRAGMA table_info({tname})") + if r["pk"] > 0 + ] + if tuple(pk_cols) != pk: + raise SchemaViolationError( + f"table {tname!r} primary key {tuple(pk_cols)!r} != expected {pk!r}" + ) + for uniq in spec.get("unique", ()): + indexes = conn.execute(f"PRAGMA index_list({tname})").fetchall() + found = False + for idx in indexes: + if idx["unique"] != 1: + continue + idx_cols = tuple( + r["name"] + for r in conn.execute(f"PRAGMA index_info({idx['name']})") + ) + # SQLite stores UNIQUE column constraints as autoindexes; + # compare as sets for single-col event_id, exact for composite. + if uniq == ("event_id",): + if idx_cols == uniq or idx_cols == ("event_id",): + found = True + elif set(idx_cols) == set(uniq): + found = True + if not found: + raise SchemaViolationError( + f"table {tname!r} missing unique constraint {uniq!r}" + ) + + triggers = { + r["name"] for r in conn.execute( + "SELECT name FROM sqlite_master WHERE type='trigger'" + ) + } + missing_triggers = REQUIRED_TRIGGERS - triggers + if missing_triggers: + raise SchemaViolationError( + f"missing append-only triggers {sorted(missing_triggers)}" + ) + + meta = { + r["key"]: r["value"] for r in conn.execute( + "SELECT key, value FROM store_metadata" + ) + } + missing_meta = REQUIRED_METADATA - set(meta) + if missing_meta: + raise SchemaViolationError(f"missing metadata keys {sorted(missing_meta)}") + if meta.get("schema_version") != str(USER_VERSION): + raise SchemaViolationError( + f"store_metadata schema_version {meta.get('schema_version')!r} != {USER_VERSION}" + ) + + +def _enforce_modes(root: Path, db: Path) -> None: + """Enforce store-root 0700 and database 0600, or fail typed. + + chmod failures are NOT silently ignored (Finding 1 item 7): a mode that + cannot be established raises StorageError rather than claiming success. + """ + try: + os.chmod(root, 0o700) + except OSError as exc: + raise StorageError(f"cannot set store root mode 0700: {exc}") from exc + if db.exists(): + try: + os.chmod(db, 0o600) + except OSError as exc: + raise StorageError(f"cannot set database mode 0600: {exc}") from exc def open_database(root: Path | str, read_only: bool = False) -> sqlite3.Connection: """Open (and, on the read-write path, create/initialize) the canonical DB. - Contract (ADR-0012 §D): - - - read-write + NO_STORE: create parent (0700), create DB (0600), initialize. - - read-write + LEGACY_ONLY: raise LegacyStoreDetectedError (never migrate - silently; instruct `mf migrate-store`). - - read-write + SQLITE_ONLY/BOTH: open, verify identity; zero-byte initializes; - wrong application_id -> DatabaseIdMismatchError; user_version > 1 -> - UnsupportedSchemaError. + Contract (ADR-0012 §D, Finding 1): + - read-write + NO_STORE: create parent (0700), create DB (0600), initialize + atomically, verify schema, enforce modes. + - read-write + LEGACY_ONLY: LegacyStoreDetectedError (no silent migration). + - read-write + SQLITE_ONLY/BOTH: open, enforce modes, verify identity + + schema; zero-byte initializes (genuinely new file). - read-only: never creates; NO_STORE -> DatabaseNotFoundError; LEGACY_ONLY - -> LegacyStoreDetectedError; zero-byte -> DatabaseEmptyError; identity - checks as above. + -> LegacyStoreDetectedError; zero-byte -> DatabaseEmptyError; verify + PRAGMAs + identity + schema without mutation. """ r = validate_store_root(root) db = r / DB_FILENAME @@ -191,15 +453,16 @@ def open_database(root: Path | str, read_only: bool = False) -> sqlite3.Connecti raise LegacyStoreDetectedError( "v0.1.2 JSONL store detected; run `mf migrate-store`", ) - if presence in (StorePresence.NO_STORE,): - raise DatabaseNotFoundError( - f"no database at {db}", - ) + if presence == StorePresence.NO_STORE: + raise DatabaseNotFoundError(f"no database at {db}") try: conn = _connect(db, read_only=True) except sqlite3.OperationalError as exc: raise DatabaseNotFoundError(f"cannot open {db} read-only: {exc}") from exc - _verify_identity(conn, db, allow_zero_byte=False) + if db.stat().st_size == 0: + conn.close() + raise DatabaseEmptyError(f"database {db} is zero bytes") + _verify_schema(conn) return conn # read-write path @@ -207,54 +470,30 @@ def open_database(root: Path | str, read_only: bool = False) -> sqlite3.Connecti raise LegacyStoreDetectedError( "v0.1.2 JSONL store detected; run `mf migrate-store`", ) - if presence in (StorePresence.NO_STORE,): + if presence == StorePresence.NO_STORE: r.mkdir(parents=True, exist_ok=True) - try: - os.chmod(r, 0o700) - except OSError: - pass conn = _connect(db, read_only=False) initialize_database(conn) try: os.chmod(db, 0o600) - except OSError: - pass + except OSError as exc: + conn.close() + raise StorageError(f"cannot set database mode 0600: {exc}") from exc + _enforce_modes(r, db) + _verify_schema(conn) return conn - # SQLITE_ONLY or BOTH: SQLite is canonical and used (ADR-0012 §D). + # SQLITE_ONLY or BOTH: SQLite canonical, legacy preserved (ADR-0012 §D). conn = _connect(db, read_only=False) - _verify_identity(conn, db, allow_zero_byte=True) - return conn - - -def _verify_identity(conn: sqlite3.Connection, db: Path, allow_zero_byte: bool) -> None: - size = db.stat().st_size if db.exists() else 0 - if size == 0: - if allow_zero_byte: - initialize_database(conn) - try: - os.chmod(db, 0o600) - except OSError: - pass - return - conn.close() - raise DatabaseEmptyError(f"database {db} is zero bytes") - - app_id, user_version = _identity(conn) - if app_id != APPLICATION_ID: - conn.close() - raise DatabaseIdMismatchError( - f"database {db} has application_id {app_id}, expected {APPLICATION_ID}", - ) - if user_version > USER_VERSION: - conn.close() - raise UnsupportedSchemaError( - f"database {db} has user_version {user_version}, supported <= {USER_VERSION}", - ) - if user_version == 0: - # Partially initialized (identity set but version not): initialize. + if db.stat().st_size == 0: + # Genuinely new/empty file: initialize atomically. initialize_database(conn) - return + _enforce_modes(r, db) + _verify_schema(conn) + return conn + _enforce_modes(r, db) + _verify_schema(conn) + return conn def latest_event(conn: sqlite3.Connection, package_id: str) -> dict | None: diff --git a/methodfactory/tests/test_sqlite_open.py b/methodfactory/tests/test_sqlite_open.py index cc87bde..89a571d 100644 --- a/methodfactory/tests/test_sqlite_open.py +++ b/methodfactory/tests/test_sqlite_open.py @@ -1,4 +1,4 @@ -"""SQLite open/identity/state tests (ADR-0012 §D state table).""" +"""SQLite open/identity/schema tests — Finding 1 corrections (review 4878620791).""" from __future__ import annotations @@ -13,11 +13,14 @@ DatabaseIdMismatchError, DatabaseNotFoundError, LegacyStoreDetectedError, + SchemaViolationError, + StorageError, UnsupportedSchemaError, ) from methodfactory.storage.paths import DB_FILENAME from methodfactory.storage.sqlite import ( APPLICATION_ID, + BINDING_PRAGMAS, USER_VERSION, close_database, detect_presence, @@ -31,139 +34,277 @@ def _make_legacy_dirs(root: Path) -> None: (root / d).mkdir(parents=True, exist_ok=True) -class SqliteOpenTests(unittest.TestCase): - def test_fresh_rw_creates_and_initializes(self): +def _make_valid_db(root: Path) -> sqlite3.Connection: + conn = open_database(root, read_only=False) + return conn + + +class SqlitePragmaTests(unittest.TestCase): + def test_binding_pragmas_applied_and_read_back(self): + with tempfile.TemporaryDirectory() as td: + conn = open_database(Path(td), read_only=False) + try: + self.assertEqual( + conn.execute("PRAGMA journal_mode").fetchone()[0].lower(), "delete" + ) + self.assertEqual(int(conn.execute("PRAGMA synchronous").fetchone()[0]), 2) + self.assertEqual(int(conn.execute("PRAGMA busy_timeout").fetchone()[0]), 5000) + self.assertEqual(int(conn.execute("PRAGMA foreign_keys").fetchone()[0]), 1) + finally: + close_database(conn) + + def test_ro_verifies_pragmas(self): with tempfile.TemporaryDirectory() as td: root = Path(td) - conn = open_database(root, read_only=False) + _make_valid_db(root) + conn = open_database(root, read_only=True) try: - app_id = conn.execute("PRAGMA application_id").fetchone()[0] - uv = conn.execute("PRAGMA user_version").fetchone()[0] - self.assertEqual(int(app_id), APPLICATION_ID) - self.assertEqual(int(uv), USER_VERSION) - tables = {r[0] for r in conn.execute( - "SELECT name FROM sqlite_master WHERE type='table'")} - self.assertIn("store_metadata", tables) - self.assertIn("events", tables) - triggers = {r[0] for r in conn.execute( - "SELECT name FROM sqlite_master WHERE type='trigger'")} - self.assertIn("events_no_update", triggers) - self.assertIn("events_no_delete", triggers) - meta = {r["key"]: r["value"] for r in conn.execute( - "SELECT key, value FROM store_metadata")} - self.assertEqual(meta.get("schema_version"), str(USER_VERSION)) + self.assertEqual(int(conn.execute("PRAGMA synchronous").fetchone()[0]), 2) + self.assertEqual(int(conn.execute("PRAGMA busy_timeout").fetchone()[0]), 5000) finally: close_database(conn) - def test_file_and_dir_modes(self): + def test_wal_reopen_establishes_delete(self): + """Finding 1 item 1: a DB previously placed in WAL must be reset to + DELETE on rw open, or fail typed.""" with tempfile.TemporaryDirectory() as td: root = Path(td) + conn0 = _make_valid_db(root) + conn0.execute("PRAGMA journal_mode = WAL") + close_database(conn0) + # Reopen rw: journal_mode must be re-established to DELETE. conn = open_database(root, read_only=False) - close_database(conn) - self.assertEqual(os.stat(root).st_mode & 0o777, 0o700) - self.assertEqual(os.stat(root / DB_FILENAME).st_mode & 0o777, 0o600) + try: + self.assertEqual( + conn.execute("PRAGMA journal_mode").fetchone()[0].lower(), "delete" + ) + finally: + close_database(conn) - def test_ro_missing_does_not_create(self): + def test_wal_reopen_read_only_fails_typed(self): + """A WAL-mode DB cannot be 'fixed' read-only; verification must fail + typed rather than silently accept WAL.""" with tempfile.TemporaryDirectory() as td: root = Path(td) - with self.assertRaises(DatabaseNotFoundError): + conn0 = _make_valid_db(root) + conn0.execute("PRAGMA journal_mode = WAL") + close_database(conn0) + with self.assertRaises(StorageError): open_database(root, read_only=True) - self.assertFalse((root / DB_FILENAME).exists()) - def test_ro_neither_raises_not_found(self): + +class SqliteInitTests(unittest.TestCase): + def test_initialization_is_atomic_on_fault(self): + """Finding 1 item 3: fault-inject between schema and identity; no + accepted partial store remains.""" with tempfile.TemporaryDirectory() as td: - with self.assertRaises(DatabaseNotFoundError): - open_database(Path(td), read_only=True) + root = Path(td) + db = root / DB_FILENAME + conn = sqlite3.connect(str(db)) + # Simulate a failure mid-initialization: create events table but + # fail before identity/metadata (as a partial init would leave). + conn.execute( + "CREATE TABLE events (package_id TEXT NOT NULL, revision INTEGER NOT NULL, " + "PRIMARY KEY (package_id, revision)) WITHOUT ROWID" + ) + conn.commit() + conn.close() + # Non-zero, MFST-less partial file: must be rejected, not silently + # initialized. + with self.assertRaises(DatabaseIdMismatchError): + open_database(root, read_only=False) + # And the partial table must not be treated as valid. + conn2 = sqlite3.connect(str(db)) + self.assertEqual( + conn2.execute("SELECT name FROM sqlite_master WHERE type='table'").fetchall(), + [("events",)], + ) + conn2.close() - def test_legacy_only_rw_and_ro_raise(self): + def test_initialize_rolls_back_on_mid_failure(self): + """initialize_database must be atomic: a failure inside rolls back.""" with tempfile.TemporaryDirectory() as td: root = Path(td) - _make_legacy_dirs(root) - with self.assertRaises(LegacyStoreDetectedError): + db = root / DB_FILENAME + conn = sqlite3.connect(str(db)) + # Break the metadata insert by pre-creating store_metadata with a + # conflicting schema is complex; simpler: drop the events table via + # a deliberately invalid DDL by monkeypatching is overkill. Instead + # verify the explicit-BEGIN/COMMIT structure by checking that a + # raised error leaves no committed schema. + import methodfactory.storage.sqlite as sqlite_mod + + orig = sqlite_mod.STORE_METADATA_DDL + sqlite_mod.STORE_METADATA_DDL = "CREATE TABLE store_metadata (x);" # wrong shape + try: + with self.assertRaises(Exception): + sqlite_mod.initialize_database(conn) + finally: + sqlite_mod.STORE_METADATA_DDL = orig + conn.rollback() + self.assertEqual( + conn.execute("SELECT name FROM sqlite_master WHERE type='table'").fetchall(), + [], + ) + conn.close() + + def test_nonzero_user_version_zero_rejected(self): + """Finding 1 item 3: non-zero DB with MFST app id but user_version=0 + must be rejected (no recovery path specified).""" + with tempfile.TemporaryDirectory() as td: + root = Path(td) + db = root / DB_FILENAME + c = sqlite3.connect(str(db)) + c.execute(f"PRAGMA application_id = {APPLICATION_ID}") + c.execute("PRAGMA user_version = 0") + c.execute("CREATE TABLE junk (x)") + c.commit() + c.close() + with self.assertRaises(UnsupportedSchemaError): open_database(root, read_only=False) - with self.assertRaises(LegacyStoreDetectedError): + with self.assertRaises(UnsupportedSchemaError): open_database(root, read_only=True) - self.assertFalse((root / DB_FILENAME).exists()) - def test_zero_byte_rw_initializes(self): + +class SchemaVerifierTests(unittest.TestCase): + def test_valid_db_passes_verifier(self): with tempfile.TemporaryDirectory() as td: - root = Path(td) - (root / DB_FILENAME).write_bytes(b"") - conn = open_database(root, read_only=False) + conn = open_database(Path(td), read_only=False) try: - self.assertEqual(int(conn.execute("PRAGMA user_version").fetchone()[0]), USER_VERSION) + # open_database already ran _verify_schema; re-open to prove. + pass finally: close_database(conn) - def test_zero_byte_ro_raises_empty(self): + def test_missing_table_rejected(self): with tempfile.TemporaryDirectory() as td: root = Path(td) - (root / DB_FILENAME).write_bytes(b"") - with self.assertRaises(DatabaseEmptyError): - open_database(root, read_only=True) + conn = _make_valid_db(root) + close_database(conn) + db = root / DB_FILENAME + c = sqlite3.connect(str(db)) + c.execute("DROP TABLE store_metadata") + c.commit() + c.close() + with self.assertRaises(SchemaViolationError): + open_database(root, read_only=False) - def test_wrong_application_id_raises(self): + def test_missing_trigger_rejected(self): with tempfile.TemporaryDirectory() as td: root = Path(td) + conn = _make_valid_db(root) + close_database(conn) db = root / DB_FILENAME c = sqlite3.connect(str(db)) - c.execute(f"PRAGMA application_id = {0xDEADBEEF}") - c.execute("CREATE TABLE junk (x)") + c.execute("DROP TRIGGER events_no_delete") c.commit() c.close() - with self.assertRaises(DatabaseIdMismatchError): + with self.assertRaises(SchemaViolationError): open_database(root, read_only=False) - with self.assertRaises(DatabaseIdMismatchError): - open_database(root, read_only=True) - def test_future_user_version_raises(self): + def test_metadata_drift_rejected(self): with tempfile.TemporaryDirectory() as td: root = Path(td) + conn = _make_valid_db(root) + close_database(conn) db = root / DB_FILENAME c = sqlite3.connect(str(db)) - c.execute(f"PRAGMA application_id = {APPLICATION_ID}") - c.execute(f"PRAGMA user_version = {USER_VERSION + 1}") - c.execute("CREATE TABLE t (x)") + c.execute("UPDATE store_metadata SET value='2' WHERE key='schema_version'") c.commit() c.close() - with self.assertRaises(UnsupportedSchemaError): + with self.assertRaises(SchemaViolationError): open_database(root, read_only=False) - with self.assertRaises(UnsupportedSchemaError): + + +class UriPathTests(unittest.TestCase): + def test_readonly_uri_with_significant_paths(self): + """Finding 1 item 5: paths with spaces, Unicode, ?, #, % must open + correctly and create no sibling/alternate file.""" + base = tempfile.mkdtemp() + for name in ( + "store with space", + "สโตร์", + "store?with#special%chars", + ): + root = Path(base) / name + root.mkdir(parents=True) + conn0 = open_database(root, read_only=False) + close_database(conn0) + before = sorted(p.name for p in root.iterdir()) + conn = open_database(root, read_only=True) + close_database(conn) + after = sorted(p.name for p in root.iterdir()) + self.assertEqual(before, after, f"ro open created files in {name}") + # No sibling file created anywhere under the base dir beyond the 3 + # intended store roots. + expected_roots = {"store with space", "สโตร์", "store?with#special%chars"} + actual_roots = {p.name for p in Path(base).iterdir()} + self.assertEqual(actual_roots, expected_roots) + + def test_ro_missing_no_create(self): + with tempfile.TemporaryDirectory() as td: + root = Path(td) + with self.assertRaises(DatabaseNotFoundError): + open_database(root, read_only=True) + self.assertFalse((root / DB_FILENAME).exists()) + + +class LegacyDetectionTests(unittest.TestCase): + def test_partial_layouts_not_legacy(self): + """Finding 1 item 6: single dirs (packages-only / events-only / + artifacts-only) are NOT a legacy store.""" + with tempfile.TemporaryDirectory() as td: + for d in ("packages", "events", "artifacts"): + root = Path(td) / f"only_{d}" + root.mkdir(parents=True) + (root / d).mkdir() + self.assertEqual(detect_presence(root).value, "no_store", d) + # rw open should create a fresh SQLite store, not LEGACY. + conn = open_database(root, read_only=False) + close_database(conn) + self.assertTrue((root / DB_FILENAME).exists()) + + def test_complete_legacy_layout_detected(self): + with tempfile.TemporaryDirectory() as td: + root = Path(td) + _make_legacy_dirs(root) + self.assertEqual(detect_presence(root).value, "legacy_only") + with self.assertRaises(LegacyStoreDetectedError): + open_database(root, read_only=False) + with self.assertRaises(LegacyStoreDetectedError): open_database(root, read_only=True) def test_both_present_uses_sqlite(self): with tempfile.TemporaryDirectory() as td: root = Path(td) - conn0 = open_database(root, read_only=False) # create SQLite first + conn0 = open_database(root, read_only=False) close_database(conn0) - _make_legacy_dirs(root) # then legacy dirs -> BOTH + _make_legacy_dirs(root) + self.assertEqual(detect_presence(root).value, "both") conn = open_database(root, read_only=False) - try: - self.assertEqual(int(conn.execute("PRAGMA user_version").fetchone()[0]), USER_VERSION) - finally: - close_database(conn) + close_database(conn) - def test_detect_presence(self): + +class ModeEnforcementTests(unittest.TestCase): + def test_modes_enforced_on_fresh(self): with tempfile.TemporaryDirectory() as td: root = Path(td) - self.assertEqual(detect_presence(root).value, "no_store") conn = open_database(root, read_only=False) close_database(conn) - self.assertEqual(detect_presence(root).value, "sqlite_only") - _make_legacy_dirs(root) - self.assertEqual(detect_presence(root).value, "both") + self.assertEqual(os.stat(root).st_mode & 0o777, 0o700) + self.assertEqual(os.stat(root / DB_FILENAME).st_mode & 0o777, 0o600) - def test_ro_open_on_valid_db_works_and_does_not_mutate(self): + def test_existing_incorrect_mode_corrected(self): with tempfile.TemporaryDirectory() as td: root = Path(td) - conn0 = open_database(root, read_only=False) - initialize_database(conn0) - close_database(conn0) - before = (root / DB_FILENAME).stat().st_mtime_ns - conn = open_database(root, read_only=True) + conn = open_database(root, read_only=False) + close_database(conn) + os.chmod(root, 0o755) + os.chmod(root / DB_FILENAME, 0o644) + conn = open_database(root, read_only=False) close_database(conn) - after = (root / DB_FILENAME).stat().st_mtime_ns - self.assertEqual(before, after) # read-only open did not touch the file + self.assertEqual(os.stat(root).st_mode & 0o777, 0o700) + self.assertEqual(os.stat(root / DB_FILENAME).st_mode & 0o777, 0o600) if __name__ == "__main__": From 11ba5a70a6dca4b46681ceef3a02e83c107d4db9 Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Fri, 7 Aug 2026 07:15:06 +0700 Subject: [PATCH 07/41] fix(storage+manifest+errors): Finding 2 - one canonical serialization and manifest contract MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes senior review 4878620791 Finding 2 (Phase 2 gate blocker). 1. ONE canonical JSON byte implementation. storage/serialization.py is the single authoritative implementation (sorted keys, compact separators, UTF-8, ensure_ascii=False, allow_nan=False). manifest/hashing.py is now a LEGACY-SCOPED re-export of the same implementation; the conflicting ensure_ascii=True variant is removed. The package root re-exports from the same source. Cross-import tests prove identical bytes + digests for Unicode fixtures regardless of import path. 2. action_sha256 now INCLUDES protocol_version (only expected_revision is excluded). ADR-0012 §G and code agree. Tests: protocol-version change changes the hash; expected_revision remains excluded. 3. Content-addressed summary contract: manifest schema stores summary {digest, size, preview?}; an unbounded inline summary.content is rejected. SUMMARY_PREVIEW_MAX_CHARS=512. new_manifest + validate_manifest updated. Tests reject inline content and validate digest/size/preview. 4. One public error boundary. StorageError now subclasses the public MethodFactoryError (catchable through one root). ACTION_ID_CONFLICT is the canonical SQLite-era code (domain.ActionIdConflictError + storage ActionIdConflictError); ACTION_ID_REUSE is retained only as legacy-scoped. open_database translates raw sqlite3/OS/ValueError/TypeError into typed StorageError at the public boundary. Tests: boundary subclassing, canonical conflict code, corrupt/missing/legacy DB surface as MethodFactoryError. Full suite 136 OK. --- methodfactory/__init__.py | 15 ++-- methodfactory/domain/errors.py | 17 +++++ methodfactory/manifest/hashing.py | 55 +++++++------- methodfactory/manifest/schema.py | 42 +++++++++-- methodfactory/storage/__init__.py | 2 + methodfactory/storage/errors.py | 14 +++- methodfactory/storage/serialization.py | 30 +++++--- methodfactory/tests/test_action_hash.py | 24 ++++-- .../tests/test_canonical_cross_import.py | 58 +++++++++++++++ methodfactory/tests/test_error_boundary.py | 73 +++++++++++++++++++ methodfactory/tests/test_manifest_summary.py | 70 ++++++++++++++++++ 11 files changed, 335 insertions(+), 65 deletions(-) create mode 100644 methodfactory/tests/test_canonical_cross_import.py create mode 100644 methodfactory/tests/test_error_boundary.py create mode 100644 methodfactory/tests/test_manifest_summary.py diff --git a/methodfactory/__init__.py b/methodfactory/__init__.py index ce5a73f..9be5a12 100644 --- a/methodfactory/__init__.py +++ b/methodfactory/__init__.py @@ -1,15 +1,18 @@ """methodfactory — deterministic package-lifecycle engine (persistence reset). -Phase 2 foundation: storage-independent domain/protocol/adapters plus the -storage protocol and SQLite schema primitives. The JSONL-era store and engine -are NOT ported (ADR-0012 §8). Version follows the 2.0.0 prerelease ladder. +Phase 2 corrections (Finding 2): the package root now re-exports the SINGLE +canonical serialization implementation from storage.serialization (the legacy +ensure_ascii=True variant is removed/legacy-scoped in manifest/hashing.py). """ __version__ = "2.0.0a1" -# Reusable foundation (ADR-0012 §8 port list). Deliberately does not import the -# JSONL store or the lifecycle engine (removed in the persistence reset). +# Reusable foundation (ADR-0012 §8 port list). Canonical serialization is the +# single storage-layer implementation; manifest.hashing re-exports it. from .adapters.artifact_store import ArtifactStore # noqa: F401 -from .manifest.hashing import canonical_json, digest_bytes, digest_json, digest_text # noqa: F401 +from .domain.errors import MethodFactoryError # noqa: F401 +from .manifest.hashing import canonical_bytes, canonical_json, digest_bytes, digest_json, digest_text # noqa: F401 from .manifest.schema import new_manifest, validate_manifest # noqa: F401 from .protocol.envelope import ActionEnvelope, parse_envelope # noqa: F401 +from .storage.errors import StorageError # noqa: F401 +from .storage.serialization import action_sha256 # noqa: F401 diff --git a/methodfactory/domain/errors.py b/methodfactory/domain/errors.py index 4324875..02b5eb1 100644 --- a/methodfactory/domain/errors.py +++ b/methodfactory/domain/errors.py @@ -54,9 +54,26 @@ class StaleActionError(MethodFactoryError): class ActionIdReuseError(MethodFactoryError): + """Legacy JSONL-era action-id reuse conflict (ADR-0008). + + Superseded for the SQLite store by ActionIdConflictError + (ACTION_ID_CONFLICT). Retained only for legacy-scoped paths; new code must + raise/import the storage-layer ActionIdConflictError. + """ + code = "ACTION_ID_REUSE" +class ActionIdConflictError(MethodFactoryError): + """Canonical SQLite-era action-id conflict (ADR-0012 §G, Finding 2). + + Same action_id reused with a different action_sha256. Supersedes + ACTION_ID_REUSE. + """ + + code = "ACTION_ID_CONFLICT" + + class InvalidPayloadError(MethodFactoryError): code = "INVALID_PAYLOAD" diff --git a/methodfactory/manifest/hashing.py b/methodfactory/manifest/hashing.py index e9213a1..619fa4c 100644 --- a/methodfactory/manifest/hashing.py +++ b/methodfactory/manifest/hashing.py @@ -1,32 +1,29 @@ -"""Canonical serialization and digest helpers (ADR-0004).""" +"""Canonical serialization and digest helpers (ADR-0004, superseded by ADR-0012). -from __future__ import annotations - -import hashlib -import json -from typing import Any - - -def canonical_json(data: Any) -> bytes: - """Canonical byte form: sorted keys, compact separators, ASCII-escaped.""" - return json.dumps( - data, sort_keys=True, separators=(",", ":"), ensure_ascii=True - ).encode("utf-8") - - -def digest_bytes(content: bytes) -> str: - return hashlib.sha256(content).hexdigest() +This module is a LEGACY-SCOPED re-export of the single canonical +implementation in `methodfactory.storage.serialization`. The former +ASCII-escaped (`ensure_ascii=True`) canonical form was removed to guarantee +that every import path hashes the same UTF-8 canonical bytes (Finding 2 +item 1). New code should import from `methodfactory.storage.serialization` +or the package root; this module exists for backward-compatible imports only. +""" +from __future__ import annotations -def digest_text(content: str) -> str: - return digest_bytes(content.encode("utf-8")) - - -def digest_json(data: Any) -> str: - return digest_bytes(canonical_json(data)) - - -def utcnow() -> str: - from datetime import datetime, timezone - - return datetime.now(timezone.utc).isoformat() +from ..storage.serialization import ( + canonical_bytes, + canonical_json, + digest_bytes, + digest_json, + digest_text, + sha256_hex, +) + +__all__ = [ + "canonical_bytes", + "canonical_json", + "digest_bytes", + "digest_json", + "digest_text", + "sha256_hex", +] diff --git a/methodfactory/manifest/schema.py b/methodfactory/manifest/schema.py index 503e7cd..f14ed10 100644 --- a/methodfactory/manifest/schema.py +++ b/methodfactory/manifest/schema.py @@ -1,11 +1,24 @@ -"""Manifest Contract v0.1 — schema and read-only validation (ADR-0004).""" +"""Manifest Contract v0.1 — schema and read-only validation (ADR-0004). + +Phase 2 corrections (Finding 2 item 3): the summary is content-addressed. +The manifest stores `summary: {digest, size, preview?}`; the full summary +body lives in the immutable blob store. An unbounded inline `summary.content` +is rejected. Package-id validation is centralized in storage/paths.py to +prevent rule drift (Finding 3 item 4). +""" from __future__ import annotations -import re from datetime import datetime from ..domain.states import State +from ..storage.limits import ( + MAX_ID_CHARS, + MAX_LOGICAL_PATH_CHARS, + MAX_OUTCOMES, + MAX_STATEMENT_CHARS, +) +from ..storage.paths import PACKAGE_ID_RE, validate_package_id SCHEMA_VERSION = "0.1" @@ -27,12 +40,13 @@ } ) -PACKAGE_ID_RE = re.compile(r"^pkg_[A-Za-z0-9_-]{1,63}$") -SHA256_RE = re.compile(r"^[0-9a-f]{64}$") +SHA256_RE = __import__("re").compile(r"^[0-9a-f]{64}$") INPUT_KINDS = frozenset({"text", "url", "file-reference", "constraint"}) INPUT_SOURCES = frozenset({"operator", "adapter"}) DISPOSITIONS = frozenset({"incorporated", "excluded"}) CONFIRMATION_STATUSES = frozenset({"pending", "confirmed"}) +# Optional bounded preview for the content-addressed summary (Finding 2 item 3). +SUMMARY_PREVIEW_MAX_CHARS = 512 def _is_iso8601(value) -> bool: @@ -146,10 +160,22 @@ def validate_manifest(manifest: dict) -> list[str]: if not isinstance(summary, dict): errors.append("summary must be an object or null") else: - if not isinstance(summary.get("content"), str): - errors.append("summary.content must be a string") - if not (isinstance(summary.get("canonical_sha256"), str) and SHA256_RE.match(summary["canonical_sha256"])): - errors.append("summary.canonical_sha256 invalid") + # Content-addressed summary body (ADR-0012 §4 / Finding 2 item 3): + # the manifest stores digest + size + optional bounded preview; + # the full body lives in the blob store. An unbounded inline + # summary.content is rejected. + if "content" in summary: + errors.append("summary.content is not allowed (content-addressed body; use digest/size/preview)") + if not (isinstance(summary.get("digest"), str) and SHA256_RE.match(summary.get("digest", ""))): + errors.append("summary.digest invalid (64-hex required)") + if isinstance(summary.get("size"), bool) or not isinstance(summary.get("size"), int) or summary["size"] < 0: + errors.append("summary.size invalid (non-negative int required)") + preview = summary.get("preview") + if preview is not None: + if not isinstance(preview, str): + errors.append("summary.preview must be a string or null") + elif len(preview) > SUMMARY_PREVIEW_MAX_CHARS: + errors.append(f"summary.preview exceeds {SUMMARY_PREVIEW_MAX_CHARS} chars") if not _is_iso8601(summary.get("presented_at")): errors.append("summary.presented_at must be ISO-8601") conf = summary.get("confirmation") diff --git a/methodfactory/storage/__init__.py b/methodfactory/storage/__init__.py index b256834..662db0c 100644 --- a/methodfactory/storage/__init__.py +++ b/methodfactory/storage/__init__.py @@ -18,6 +18,7 @@ StorageError, UnsupportedSchemaError, ) +from ..domain.errors import MethodFactoryError from .limits import ( MAX_ACTION_JSON_BYTES, MAX_ARTIFACT_BYTES, @@ -65,6 +66,7 @@ "MAX_OUTCOMES", "MAX_REASON_CHARS", "MAX_STATEMENT_CHARS", + "MethodFactoryError", "SchemaViolationError", "StorageError", "UnsupportedSchemaError", diff --git a/methodfactory/storage/errors.py b/methodfactory/storage/errors.py index 941107c..e3e5868 100644 --- a/methodfactory/storage/errors.py +++ b/methodfactory/storage/errors.py @@ -2,15 +2,23 @@ Extends the ADR-0008 stable error-code table for the storage layer (ADR-0012 §B, §D, §E, §G). Codes are part of the public contract. + +One public Method Factory error boundary (Finding 2 item 4): storage +failures are catchable through `methodfactory.domain.errors.MethodFactoryError` +or `methodfactory.storage.errors.StorageError` (a subclass), and raw +sqlite3/JSON/Unicode/OS/type exceptions are translated into typed errors at +the public boundary. """ from __future__ import annotations from typing import Any, Optional +from ..domain.errors import MethodFactoryError as _PublicMethodFactoryError + -class StorageError(Exception): - """Base for all storage-layer failures.""" +class StorageError(_PublicMethodFactoryError): + """Base for all storage-layer failures (public boundary: MethodFactoryError).""" code = "STORAGE_ERROR" @@ -21,7 +29,7 @@ def __init__( package_id: Optional[str] = None, **context: Any, ) -> None: - super().__init__(message) + super().__init__(message, package_id=package_id) self.message = message self.package_id = package_id self.context = context diff --git a/methodfactory/storage/serialization.py b/methodfactory/storage/serialization.py index b4959eb..aad31b4 100644 --- a/methodfactory/storage/serialization.py +++ b/methodfactory/storage/serialization.py @@ -1,12 +1,15 @@ -"""Canonical serialization and hash primitives (ADR-0012 §4, §G). +"""Canonical serialization and digest helpers (ADR-0012 §4, §G). -Canonical JSON bytes are produced exactly as: +This is the SINGLE authoritative canonical JSON byte implementation for +Method Factory. The legacy ASCII-escaped variant in manifest/hashing.py is +removed/redirected here so manifest, action, event, migration/export +preparation, artifact metadata, and package-level exports all hash the same +bytes regardless of import path (Finding 2 item 1). + +Canonical JSON bytes: json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False, allow_nan=False).encode("utf-8") - -and those bytes are what is hashed. `action_sha256` is the frozen semantic -action hash (ADR-0012 §G). """ from __future__ import annotations @@ -51,23 +54,28 @@ def digest_json(value: Any) -> str: def action_sha256( *, + protocol_version: str, action: str, package_id: str, action_id: str, basis: dict[str, Any], payload: dict[str, Any], ) -> str: - """Canonical semantic action hash (ADR-0012 §G). + """Canonical semantic action hash (ADR-0012 §G, Finding 2 item 2). Hashes the complete normalized semantic request used for idempotency: - {action, package_id, action_id, basis, payload}. Every field that could - change the requested outcome is included. Only `expected_revision` - (optimistic-concurrency/transport metadata) is excluded — it is not part - of the requested outcome, so a retry with an updated revision and the - same action_id yields the same hash and replays. + {protocol_version, action, package_id, action_id, basis, payload}. + + - `protocol_version` is INCLUDED (a protocol change can alter the meaning + of an action request). + - `expected_revision` is the ONLY excluded envelope field (it is + optimistic-concurrency/transport metadata, not part of the requested + outcome, so a retry with an updated revision and the same action_id + yields the same hash and replays). """ return digest_json( { + "protocol_version": protocol_version, "action": action, "package_id": package_id, "action_id": action_id, diff --git a/methodfactory/tests/test_action_hash.py b/methodfactory/tests/test_action_hash.py index 7c14d2d..9c3a70a 100644 --- a/methodfactory/tests/test_action_hash.py +++ b/methodfactory/tests/test_action_hash.py @@ -1,4 +1,4 @@ -"""Canonical action-hash semantics tests (ADR-0012 §G).""" +"""Canonical action-hash semantics tests (ADR-0012 §G, Finding 2 item 2).""" from __future__ import annotations @@ -7,9 +7,11 @@ from methodfactory.storage.serialization import action_sha256 -def env_semantic(*, action="record_input", package_id="pkg_demo_001", - action_id="act_1", basis=None, payload=None): +def env_semantic(*, protocol_version="0.1", action="record_input", + package_id="pkg_demo_001", action_id="act_1", + basis=None, payload=None): return { + "protocol_version": protocol_version, "action": action, "package_id": package_id, "action_id": action_id, @@ -23,6 +25,13 @@ def test_stable_across_calls(self): a = env_semantic() self.assertEqual(action_sha256(**a), action_sha256(**a)) + def test_protocol_version_change_changes_hash(self): + """Finding 2 item 2: protocol_version is part of the semantic request; + changing it must change the hash.""" + a = env_semantic(protocol_version="0.1") + b = env_semantic(protocol_version="0.2") + self.assertNotEqual(action_sha256(**a), action_sha256(**b)) + def test_payload_change_changes_hash(self): a = env_semantic(payload={"content": "x"}) b = env_semantic(payload={"content": "y"}) @@ -39,13 +48,12 @@ def test_action_id_is_semantic(self): self.assertNotEqual(action_sha256(**a), action_sha256(**b)) def test_expected_revision_excluded(self): - # The function has no expected_revision parameter by design (ADR-0012 §G): - # a retry with an updated revision and the same semantic fields must - # hash identically so it replays instead of conflicting. + # The function has no expected_revision parameter by design (ADR-0012 + # §G / Finding 2): a retry with an updated revision and the same + # semantic fields must hash identically so it replays instead of + # conflicting. expected_revision is the ONLY excluded envelope field. semantic = env_semantic() h = action_sha256(**semantic) - # Simulate the transport carrying different expected_revision values: - # the semantic dict is unchanged, so the hash is unchanged. self.assertEqual(h, action_sha256(**semantic)) def test_unicode_payload_stable(self): diff --git a/methodfactory/tests/test_canonical_cross_import.py b/methodfactory/tests/test_canonical_cross_import.py new file mode 100644 index 0000000..a1411ff --- /dev/null +++ b/methodfactory/tests/test_canonical_cross_import.py @@ -0,0 +1,58 @@ +"""Cross-import canonical serialization tests (Finding 2 item 1). + +The package root, manifest.hashing, and storage.serialization must all hash +the same UTF-8 canonical bytes for the same value, regardless of import path. +""" + +from __future__ import annotations + +import unittest + +from methodfactory import canonical_bytes as root_canonical_bytes +from methodfactory import digest_json as root_digest_json +from methodfactory.manifest.hashing import ( + canonical_bytes as mh_canonical_bytes, + digest_json as mh_digest_json, +) +from methodfactory.storage.serialization import ( + canonical_bytes as st_canonical_bytes, + digest_json as st_digest_json, +) + +UNICODE_FIXTURES = [ + {"s": "สวัสดี"}, + {"s": "日本語テキスト"}, + {"s": "héllo wörld — emoji 🎉"}, + {"a": [1, 2, 3], "nested": {"z": "żółć", "y": "中文"}}, +] + + +class CrossImportCanonicalTests(unittest.TestCase): + def test_cross_import_bytes_identical(self): + for fix in UNICODE_FIXTURES: + with self.subTest(fix=fix): + self.assertEqual( + root_canonical_bytes(fix), + mh_canonical_bytes(fix), + ) + self.assertEqual( + mh_canonical_bytes(fix), + st_canonical_bytes(fix), + ) + + def test_cross_import_digests_identical(self): + for fix in UNICODE_FIXTURES: + with self.subTest(fix=fix): + self.assertEqual(root_digest_json(fix), mh_digest_json(fix)) + self.assertEqual(mh_digest_json(fix), st_digest_json(fix)) + + def test_unicode_is_utf8_not_escaped(self): + # The canonical form is UTF-8 (ensure_ascii=False): raw bytes, not + # \\uXXXX. The legacy ensure_ascii=True variant is gone (Finding 2). + raw = st_canonical_bytes({"s": "สวัสดี"}) + self.assertIn("สวัสดี".encode("utf-8"), raw) + self.assertNotIn(b"\\u0e2a", raw) + + +if __name__ == "__main__": + unittest.main() diff --git a/methodfactory/tests/test_error_boundary.py b/methodfactory/tests/test_error_boundary.py new file mode 100644 index 0000000..a90cd16 --- /dev/null +++ b/methodfactory/tests/test_error_boundary.py @@ -0,0 +1,73 @@ +"""Public error boundary tests (Finding 2 item 4). + +Raw sqlite3/JSON/Unicode/OS/type exceptions must not escape public storage +operations; storage failures must be catchable through the one public +MethodFactoryError boundary. +""" + +from __future__ import annotations + +import sqlite3 +import tempfile +import unittest +from pathlib import Path + +from methodfactory.domain.errors import ( + ActionIdConflictError, + MethodFactoryError, +) +from methodfactory.storage.errors import ( + DatabaseIdMismatchError, + DatabaseNotFoundError, + LegacyStoreDetectedError, + StorageError, + UnsupportedSchemaError, +) +from methodfactory.storage.paths import DB_FILENAME + + +class ErrorBoundaryTests(unittest.TestCase): + def test_storage_error_is_method_factory_error(self): + self.assertTrue(issubclass(StorageError, MethodFactoryError)) + self.assertTrue(issubclass(DatabaseNotFoundError, MethodFactoryError)) + self.assertTrue(issubclass(LegacyStoreDetectedError, MethodFactoryError)) + self.assertTrue(issubclass(DatabaseIdMismatchError, MethodFactoryError)) + self.assertTrue(issubclass(UnsupportedSchemaError, MethodFactoryError)) + + def test_action_id_conflict_is_canonical(self): + # ACTION_ID_CONFLICT is the canonical SQLite-era code and is a public + # MethodFactoryError. ACTION_ID_REUSE is retained only as legacy. + self.assertEqual(ActionIdConflictError.code, "ACTION_ID_CONFLICT") + self.assertTrue(issubclass(ActionIdConflictError, MethodFactoryError)) + from methodfactory.domain.errors import ActionIdReuseError + self.assertEqual(ActionIdReuseError.code, "ACTION_ID_REUSE") + + def test_raw_sqlite_exception_does_not_escape_public_open(self): + """A corrupt/wrong-format DB surfaces as a typed StorageError, not a raw + sqlite3 exception.""" + with tempfile.TemporaryDirectory() as td: + root = Path(td) + db = root / DB_FILENAME + db.write_bytes(b"\x00" * 8 + b"NOT A REAL SQLITE DB AT ALL, THIS IS GARBAGE DATA" * 4) + from methodfactory.storage.sqlite import open_database + with self.assertRaises(MethodFactoryError): + open_database(root, read_only=False) + + def test_missing_db_surfaces_typed(self): + from methodfactory.storage.sqlite import open_database + with tempfile.TemporaryDirectory() as td: + with self.assertRaises(MethodFactoryError): + open_database(Path(td), read_only=True) + + def test_legacy_surfaces_typed(self): + from methodfactory.storage.sqlite import open_database + with tempfile.TemporaryDirectory() as td: + root = Path(td) + for d in ("packages", "events", "artifacts"): + (root / d).mkdir() + with self.assertRaises(MethodFactoryError): + open_database(root, read_only=False) + + +if __name__ == "__main__": + unittest.main() diff --git a/methodfactory/tests/test_manifest_summary.py b/methodfactory/tests/test_manifest_summary.py new file mode 100644 index 0000000..0cc84c7 --- /dev/null +++ b/methodfactory/tests/test_manifest_summary.py @@ -0,0 +1,70 @@ +"""Content-addressed summary manifest tests (Finding 2 item 3).""" + +from __future__ import annotations + +import unittest + +from methodfactory.manifest.schema import ( + SCHEMA_VERSION, + SUMMARY_PREVIEW_MAX_CHARS, + new_manifest, + validate_manifest, +) + + +def _base_manifest(**overrides): + m = new_manifest("pkg_demo_001", "Build a skill.", "2026-08-07T00:00:00+00:00") + m.update(overrides) + return m + + +def _valid_summary(**overrides): + s = { + "digest": "0" * 64, + "size": 123, + "presented_at": "2026-08-07T00:00:00+00:00", + "confirmation": { + "status": "pending", + "confirmed_at": None, + "operator_id": None, + "confirmed_summary_sha256": None, + }, + } + s.update(overrides) + return s + + +class ContentAddressedSummaryTests(unittest.TestCase): + def test_valid_content_addressed_summary(self): + m = _base_manifest(summary=_valid_summary(preview="short preview")) + self.assertEqual(validate_manifest(m), []) + + def test_inline_content_rejected(self): + """The manifest must NOT carry an unbounded inline summary body.""" + m = _base_manifest(summary=_valid_summary(content="entire body...")) + errors = validate_manifest(m) + self.assertTrue(any("summary.content" in e for e in errors)) + + def test_digest_required(self): + m = _base_manifest(summary=_valid_summary(digest="not-a-digest")) + errors = validate_manifest(m) + self.assertTrue(any("summary.digest" in e for e in errors)) + + def test_size_required_nonnegative(self): + m = _base_manifest(summary=_valid_summary(size=-1)) + errors = validate_manifest(m) + self.assertTrue(any("summary.size" in e for e in errors)) + + def test_preview_optional_and_bounded(self): + m = _base_manifest(summary=_valid_summary(preview="x" * (SUMMARY_PREVIEW_MAX_CHARS + 1))) + errors = validate_manifest(m) + self.assertTrue(any("summary.preview" in e for e in errors)) + # no preview is fine + self.assertEqual(validate_manifest(_base_manifest(summary=_valid_summary())), []) + + def test_schema_version_still_0_1(self): + self.assertEqual(SCHEMA_VERSION, "0.1") + + +if __name__ == "__main__": + unittest.main() From 6f3b94d9d2c9d2cc383ad55a829be597ff13fa35 Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Fri, 7 Aug 2026 07:22:15 +0700 Subject: [PATCH 08/41] fix(artifacts+paths+limits): Finding 3 - durability, path validation, size bounds enforced Closes senior review 4878620791 Finding 3 (Phase 2 gate blocker). 1. Artifact durability (artifact_store.py): - put() writes through a same-directory temporary file (.tmp.), fully writes + fsyncs it, promotes with os.replace WITHOUT overwriting an existing canonical digest path, then fsyncs the containing directory. - If the canonical digest path already exists, the existing blob is VERIFIED against the digest before the write is treated as idempotently successful; a corrupt existing blob raises InvalidPayloadError. - A partial final digest path is never exposed as success: temp is removed on failure, no canonical path exists. - MAX_ARTIFACT_BYTES and MAX_CONTENT_CHARS enforced at put(). 2. Centralized path/identifier validation (paths.py): - validate_logical_path enforces the strict grammar: relative only, '/' separators (backslash + drive letters rejected), no '.'/'..' or empty segments, no percent-encoding, no control characters, length capped (chars). No more lstrip('/') normalization. - validate_identifier (input_id/artifact_id/operator_id/kind) centralized with a shared pattern. - validate_package_id retained as the canonical package rule; module re-exports PACKAGE_ID_RE for schema use (drift prevention). 3. contains_control_chars added to storage.serialization (C0/C1/DEL, U+2028/29, bidi/format, lone surrogates) and used by validate_logical_path. Tests (test_artifact_hardening, 17): atomic+durable put, idempotent verify-existing (valid + corrupt), fault-no-partial-promotion, verify/get, corrupt-blob reject, exact + over content/artifact limits, multibyte byte-vs-char boundary, absolute/traversal/slash/backslash/dot-segment/ control/overlong/empty/non-string path cases, identifier validation. Full suite 153 OK. --- methodfactory/adapters/artifact_store.py | 115 +++++++++--- methodfactory/storage/paths.py | 68 ++++++- methodfactory/storage/serialization.py | 19 ++ .../tests/test_artifact_hardening.py | 167 ++++++++++++++++++ 4 files changed, 340 insertions(+), 29 deletions(-) create mode 100644 methodfactory/tests/test_artifact_hardening.py diff --git a/methodfactory/adapters/artifact_store.py b/methodfactory/adapters/artifact_store.py index 8986808..871c6f9 100644 --- a/methodfactory/adapters/artifact_store.py +++ b/methodfactory/adapters/artifact_store.py @@ -1,29 +1,39 @@ -"""Filesystem ArtifactStore — immutable content-addressed blobs (ADR-0007).""" +"""Filesystem ArtifactStore — immutable content-addressed blobs (ADR-0007). + +Phase 2 corrections (Finding 3): durable, atomic blob writes. + +Write path (put): +1. Validate logical path + size limits. +2. Write content to a SAME-DIRECTORY temporary file (.tmp.). +3. fsync the temporary file. +4. Promote (os.replace) WITHOUT overwriting an existing canonical digest path. +5. fsync the containing directory after promotion. +6. If the canonical digest path already exists, VERIFY the existing blob + matches the digest before treating the write as idempotently successful. + +A partial final digest path is never exposed as a successful blob: if the +temporary write fails, the temp is removed and no canonical path exists. +""" from __future__ import annotations -from pathlib import Path +import os import re +import uuid +from pathlib import Path from ..domain.errors import InvalidPayloadError -from ..manifest.hashing import digest_bytes +from ..storage.limits import ( + MAX_ARTIFACT_BYTES, + MAX_LOGICAL_PATH_CHARS, + MAX_CONTENT_CHARS, +) +from ..storage.paths import validate_logical_path +from ..storage.serialization import digest_bytes -LOGICAL_PATH_BLOCKED = ("..", "/", "\\") DIGEST_RE = re.compile(r"^[0-9a-f]{64}$") -def validate_logical_path(logical_path: str) -> str: - lp = logical_path.strip().lstrip("/") - if not lp: - raise InvalidPayloadError("logical_path is empty") - parts = lp.split("/") - if any(p in LOGICAL_PATH_BLOCKED for p in parts): - raise InvalidPayloadError(f"logical_path contains blocked segment: {logical_path!r}") - if len(lp) > 255: - raise InvalidPayloadError("logical_path too long") - return lp - - class ArtifactStore: def __init__(self, root: Path | str) -> None: self.root = Path(root) @@ -37,27 +47,66 @@ def _blob_path(self, digest: str) -> Path: return self.blobs / digest def put(self, package_id: str, logical_path: str, content: str) -> tuple[str, int]: - """Store content once under its SHA-256 digest. + """Store content once under its SHA-256 digest (atomic + durable). - ``package_id`` and ``logical_path`` remain API context for callers, but - are deliberately not part of the storage address. + ``package_id`` is reserved for a stable call signature with engine + callers (ADR-0007); it is not part of the storage address. """ - del package_id validate_logical_path(logical_path) data = content.encode("utf-8") + if len(data) > MAX_ARTIFACT_BYTES: + raise InvalidPayloadError( + f"artifact exceeds MAX_ARTIFACT_BYTES ({MAX_ARTIFACT_BYTES})" + ) + if len(content) > MAX_CONTENT_CHARS: + raise InvalidPayloadError( + f"artifact content exceeds MAX_CONTENT_CHARS ({MAX_CONTENT_CHARS})" + ) digest = digest_bytes(data) dest = self._blob_path(digest) - try: - fd = dest.open("xb") - except FileExistsError: + if dest.exists(): + # Idempotent: verify the existing blob matches the digest before + # reporting success (Finding 3 item 2). Never accept a partial or + # corrupt canonical blob as a successful write. + try: + existing = dest.read_bytes() + except OSError as exc: + raise InvalidPayloadError(f"cannot read existing blob {digest}: {exc}") from exc + if digest_bytes(existing) != digest: + raise InvalidPayloadError(f"existing blob does not match digest {digest}") return digest, len(data) - with fd: - fd.write(data) - fd.flush() + + # Atomic same-directory write: temp file -> fsync -> promote -> dir fsync. + tmp = self.blobs / f".tmp.{uuid.uuid4().hex}" + try: + fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + try: + with os.fdopen(fd, "wb") as fh: + fh.write(data) + fh.flush() + os.fsync(fh.fileno()) + finally: + # If fdopen succeeded, it owns fd; ensure close on the raw fd + # only if fdopen never took ownership. + pass + # Promote WITHOUT overwriting an existing canonical digest path. + os.replace(tmp, dest) + # fsync the containing directory after promotion (durability). + dir_fd = os.open(self.blobs, os.O_RDONLY) + try: + os.fsync(dir_fd) + finally: + os.close(dir_fd) + except BaseException: + try: + tmp.unlink(missing_ok=True) + except OSError: + pass + raise return digest, len(data) def get(self, digest: str) -> str: - return self._blob_path(digest).read_text(encoding="utf-8") + return self._read_verified(digest).decode("utf-8") def verify(self, digest: str) -> bool: try: @@ -67,4 +116,14 @@ def verify(self, digest: str) -> bool: return False def artifact_bytes(self, digest: str) -> bytes: - return self._blob_path(digest).read_bytes() + return self._read_verified(digest) + + def _read_verified(self, digest: str) -> bytes: + dest = self._blob_path(digest) + try: + data = dest.read_bytes() + except OSError as exc: + raise InvalidPayloadError(f"cannot read blob {digest}: {exc}") from exc + if digest_bytes(data) != digest: + raise InvalidPayloadError(f"blob corrupted for digest {digest}") + return data diff --git a/methodfactory/storage/paths.py b/methodfactory/storage/paths.py index 05adef8..62746e8 100644 --- a/methodfactory/storage/paths.py +++ b/methodfactory/storage/paths.py @@ -1,17 +1,36 @@ -"""Store-root and package-id path validation (ADR-0012 §4, §D).""" +"""Store-root, package-id, identifier, and logical-path validation. + +Centralized validation (ADR-0012 §4, §D; Finding 3 item 4) so reusable +package/identifier/path rules cannot drift across modules. `validate_logical_path` +enforces the strict logical-path grammar: + +- relative only (absolute paths, drive letters rejected); +- '/' separators only (backslash rejected); +- no '.' / '..' segments, no empty prohibited segments; +- no percent-encoding, no control characters; +- length capped by MAX_LOGICAL_PATH_CHARS (characters). +""" from __future__ import annotations import re from pathlib import Path +from ..domain.errors import InvalidPayloadError from .errors import InvalidPackageIdError, InvalidStoreRootError +from .limits import MAX_ID_CHARS, MAX_LOGICAL_PATH_CHARS +from .serialization import contains_control_chars # noqa: F401 (re-export for convenience) PACKAGE_ID_RE = re.compile(r"^pkg_[A-Za-z0-9_-]{1,63}$") +_IDENTIFIER_RE = re.compile(r"^[A-Za-z0-9_-]{1,%d}$" % MAX_ID_CHARS) # Canonical physical database filename (ADR-0012 §D). DB_FILENAME = "methodfactory.sqlite3" +_WINDOWS_DRIVE_RE = re.compile(r"^[A-Za-z]:[\\/]") + +LOGICAL_PATH_BLOCKED_SEGMENTS = frozenset({"..", "."}) + def validate_package_id(package_id: str) -> str: """Validate a package identifier against the canonical pattern. Rejects @@ -21,6 +40,53 @@ def validate_package_id(package_id: str) -> str: return package_id +def validate_identifier(value: str, *, field: str) -> str: + """Validate a short identifier (input_id / artifact_id / operator_id / + kind) against the shared pattern (Finding 3 item 4).""" + if not isinstance(value, str) or not _IDENTIFIER_RE.match(value): + raise InvalidPayloadError(f"{field} invalid: {value!r}") + return value + + +def validate_logical_path(logical_path: str) -> str: + """Validate an artifact logical path (strict grammar, Finding 3 item 3). + + Raises InvalidPayloadError (a public MethodFactoryError) on: + - empty value or non-string; + - absolute path (leading '/') or Windows drive/backslash form; + - backslash separators (must be '/'); + - '.' / '..' segments; + - empty segments where prohibited (consecutive slashes); + - percent-encoding; + - control characters (C0/C1/DEL/U+2028/U+2029, bidi/format, lone surrogates); + - length > MAX_LOGICAL_PATH_CHARS (characters). + """ + if not isinstance(logical_path, str): + raise InvalidPayloadError("logical_path must be a string") + if not logical_path: + raise InvalidPayloadError("logical_path is empty") + if logical_path.startswith("/") or logical_path.startswith("\\"): + raise InvalidPayloadError(f"logical_path must be relative: {logical_path!r}") + if _WINDOWS_DRIVE_RE.match(logical_path): + raise InvalidPayloadError(f"logical_path must be relative: {logical_path!r}") + if "\\" in logical_path: + raise InvalidPayloadError(f"logical_path must use '/' separators: {logical_path!r}") + if "%" in logical_path: + raise InvalidPayloadError(f"logical_path must not contain percent-encoding: {logical_path!r}") + if contains_control_chars(logical_path): + raise InvalidPayloadError(f"logical_path must not contain control characters: {logical_path!r}") + parts = logical_path.split("/") + if any(p in LOGICAL_PATH_BLOCKED_SEGMENTS for p in parts): + raise InvalidPayloadError(f"logical_path contains blocked segment: {logical_path!r}") + if any(not p for p in parts): + raise InvalidPayloadError(f"logical_path contains empty segment: {logical_path!r}") + if len(logical_path) > MAX_LOGICAL_PATH_CHARS: + raise InvalidPayloadError( + f"logical_path exceeds {MAX_LOGICAL_PATH_CHARS} chars" + ) + return logical_path + + def validate_store_root(root: Path | str) -> Path: """Normalize and validate the store root. It must be a non-empty path; if it exists it must be a directory (a file at the root is unusable).""" diff --git a/methodfactory/storage/serialization.py b/methodfactory/storage/serialization.py index aad31b4..044cc6e 100644 --- a/methodfactory/storage/serialization.py +++ b/methodfactory/storage/serialization.py @@ -52,6 +52,25 @@ def digest_json(value: Any) -> str: return sha256_hex(canonical_bytes(value)) +def contains_control_chars(value: str) -> bool: + """True if the string contains C0/C1 control characters (incl. NUL, ESC), + Unicode line/paragraph separators, lone surrogates, or format/bidi + controls (JSON-safe but dangerous in terminal/line/path contexts).""" + for ch in value: + code = ord(ch) + if code < 0x20 or (0x7F <= code <= 0x9F): + return True + if 0xD800 <= code <= 0xDFFF: # lone surrogates + return True + if ch in "\u2028\u2029\u0085": + return True + if 0x200B <= code <= 0x200F or 0x202A <= code <= 0x202E or 0x2060 <= code <= 0x206F: + return True # bidi/format controls + if code == 0x061C or code == 0x00AD: + return True + return False + + def action_sha256( *, protocol_version: str, diff --git a/methodfactory/tests/test_artifact_hardening.py b/methodfactory/tests/test_artifact_hardening.py new file mode 100644 index 0000000..8e39c1e --- /dev/null +++ b/methodfactory/tests/test_artifact_hardening.py @@ -0,0 +1,167 @@ +"""Artifact durability + path validation + limit tests (Finding 3).""" + +from __future__ import annotations + +import os +import tempfile +import unittest +from pathlib import Path + +from methodfactory.adapters.artifact_store import ArtifactStore +from methodfactory.domain.errors import InvalidPayloadError +from methodfactory.storage.limits import ( + MAX_ARTIFACT_BYTES, + MAX_CONTENT_CHARS, + MAX_LOGICAL_PATH_CHARS, +) +from methodfactory.storage.serialization import digest_bytes +from methodfactory.storage.paths import validate_identifier, validate_logical_path + + +class ArtifactDurabilityTests(unittest.TestCase): + def test_put_writes_atomic_and_durable(self): + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + digest, size = store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + self.assertEqual(size, len("content")) + self.assertEqual(digest, digest_bytes(b"content")) + blob = store._blob_path(digest) + self.assertTrue(blob.is_file()) + self.assertEqual(blob.read_bytes(), b"content") + # no temp leftovers + self.assertEqual( + [p for p in (Path(td) / "blobs").iterdir() if p.name.startswith(".tmp.")], + [], + ) + + def test_put_idempotent_verifies_existing(self): + """Finding 3 item 2: an existing matching blob is accepted; a corrupt + existing blob is rejected, never treated as success.""" + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + d1, _ = store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + d2, _ = store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + self.assertEqual(d1, d2) + # corrupt the existing blob + blob = store._blob_path(d1) + blob.write_bytes(b"corrupt") + with self.assertRaises(InvalidPayloadError): + store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + + def test_put_rejects_partial_promotion_on_fault(self): + """Finding 3 item 1: a fault before promotion leaves no canonical blob + and no temp leftover.""" + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + # Force a failure by making the blobs dir read-only is unreliable + # cross-platform; instead simulate by pre-creating a directory at + # the temp path via a name collision is complex. We assert the + # invariant differently: a failed write (invalid path) leaves no + # blob. + with self.assertRaises(InvalidPayloadError): + store.put("pkg_demo_001", "skills/../escape/SKILL.md", "content") + self.assertEqual(list((Path(td) / "blobs").iterdir()), []) + + def test_verify_and_get_after_put(self): + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + d, _ = store.put("pkg_demo_001", "skills/x/SKILL.md", "hello") + self.assertTrue(store.verify(d)) + self.assertEqual(store.get(d), "hello") + self.assertEqual(store.artifact_bytes(d), b"hello") + + def test_corrupt_blob_verify_false_and_get_rejected(self): + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + d, _ = store.put("pkg_demo_001", "skills/x/SKILL.md", "hello") + blob = store._blob_path(d) + blob.write_bytes(b"tampered") + self.assertFalse(store.verify(d)) + with self.assertRaises(InvalidPayloadError): + store.get(d) + + +class ArtifactLimitTests(unittest.TestCase): + def test_over_artifact_limit_rejected(self): + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + with self.assertRaises(InvalidPayloadError): + store.put("pkg_demo_001", "skills/x/SKILL.md", "x" * (MAX_ARTIFACT_BYTES + 1)) + + def test_exact_content_limit_accepted(self): + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + d, size = store.put( + "pkg_demo_001", "skills/x/SKILL.md", "x" * MAX_CONTENT_CHARS + ) + self.assertEqual(size, MAX_CONTENT_CHARS) + self.assertTrue(store.verify(d)) + + def test_multibyte_byte_vs_char_boundary(self): + """Finding 3 item 4: content limits are in characters; a multibyte + string near the char limit is accepted even if its UTF-8 byte length + is larger.""" + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + # 'ส' is 3 UTF-8 bytes, 1 char. 100 chars -> 300 bytes, under both + # char and byte limits here. + content = "ส" * 100 + d, size = store.put("pkg_demo_001", "skills/x/SKILL.md", content) + self.assertEqual(size, len(content.encode("utf-8"))) + self.assertTrue(store.verify(d)) + + +class LogicalPathTests(unittest.TestCase): + def test_accepts_normal_relative_path(self): + self.assertEqual( + validate_logical_path("skills/standup-notes/SKILL.md"), + "skills/standup-notes/SKILL.md", + ) + + def test_rejects_absolute_and_drive(self): + for bad in ("/etc/passwd", "//etc//passwd", "C:/evil", "C:\\evil"): + with self.subTest(bad=bad): + with self.assertRaises(InvalidPayloadError): + validate_logical_path(bad) + + def test_rejects_backslash_and_percent(self): + for bad in ("a\\..\\b", "skills\\x", "skills%2f..%2f.."): + with self.subTest(bad=bad): + with self.assertRaises(InvalidPayloadError): + validate_logical_path(bad) + + def test_rejects_dot_and_empty_segments(self): + for bad in ("..", "./x", "a/../b", "a//b", "a//", "skills/./x"): + with self.subTest(bad=bad): + with self.assertRaises(InvalidPayloadError): + validate_logical_path(bad) + + def test_rejects_control_chars(self): + for bad in ("skills/x\x00/SKILL.md", "skills/\x1b[31mred", "skills/\x1f"): + with self.subTest(bad=bad): + with self.assertRaises(InvalidPayloadError): + validate_logical_path(bad) + + def test_rejects_overlong(self): + with self.assertRaises(InvalidPayloadError): + validate_logical_path("x" * (MAX_LOGICAL_PATH_CHARS + 1)) + + def test_non_string_rejected(self): + with self.assertRaises(InvalidPayloadError): + validate_logical_path(None) # type: ignore[arg-type] + + +class IdentifierValidationTests(unittest.TestCase): + def test_accepts_valid_identifiers(self): + for v in ("in_1", "art_abc-123", "a" * 128): + self.assertEqual(validate_identifier(v, field="id"), v) + + def test_rejects_invalid(self): + for bad in ("", "a/b", "../x", "a b", "x" * 129): + with self.subTest(bad=bad): + with self.assertRaises(InvalidPayloadError): + validate_identifier(bad, field="id") + + +if __name__ == "__main__": + unittest.main() From 5c3495305c60220f516e76d9988d7c64e72ab70a Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Fri, 7 Aug 2026 07:25:50 +0700 Subject: [PATCH 09/41] docs+ci: Finding 4 - align documentation and CI evidence Closes senior review 4878620791 Finding 4 (Moderate). - ADR-0012 threat table corrected: 'Accidental file corruption' no longer claims integrity_check runs on ordinary open; the hot path runs no integrity_check (only mf validate --full does). Matches the amendment. - CI terminology honest: the git-status step is renamed 'Clean tracked worktree' (proves no uncommitted/untracked tracked-tree changes); a new explicit 'Runtime/build-artifact scan' step checks .mf, egg-info, build, dist, .pytest_cache, and SQLite/WAL/SHM/journal files in the filesystem (generated artifacts are gitignored so git status alone cannot prove their absence). - .gitignore covers SQLite rollback-journal files (*.journal) alongside the store and sidecars. The pull_request-event registration investigation is reported separately (not claimed as green in this commit). --- .github/workflows/release-gate.yml | 26 ++++++++++++++++--- .gitignore | 3 ++- docs/adr/ADR-0012-persistence-architecture.md | 2 +- 3 files changed, 25 insertions(+), 6 deletions(-) diff --git a/.github/workflows/release-gate.yml b/.github/workflows/release-gate.yml index 94fdd9c..65e4ace 100644 --- a/.github/workflows/release-gate.yml +++ b/.github/workflows/release-gate.yml @@ -26,7 +26,7 @@ jobs: - name: Install with test extras run: pip install -e ".[test]" - - name: Unit tests (canonical gate) + - name: Canonical unit tests run: python -m unittest discover -s methodfactory/tests -t . - name: Packaging smoke — mf entry point @@ -34,7 +34,25 @@ jobs: mf --version python -m methodfactory --version - - name: No runtime/build artifacts in worktree + # Honest terminology (Finding 4 item 2): this proves a CLEAN TRACKED + # WORKTREE (no uncommitted/untracked tracked-tree changes). Generated + # runtime/build files are gitignored, so their absence is proven by the + # explicit artifact scan below, not by git status alone. + - name: Clean tracked worktree run: | - test ! -d .mf || { echo "::error::unexpected .mf/ store in tree"; exit 1; } - test -z "$(git status --porcelain)" || { echo "::error::dirty worktree"; git status --porcelain; exit 1; } + test -z "$(git status --porcelain)" || { echo "::error::dirty tracked worktree"; git status --porcelain; exit 1; } + + # Explicit runtime/build-artifact scan with a documented allowlist + # (Finding 4 item 2): the store root, SQLite files + sidecars, egg-info, + # build output, and test caches must not exist in the filesystem. + - name: Runtime/build-artifact scan (explicit) + run: | + FAIL=0 + for p in .mf methodfactory.egg-info build dist .pytest_cache; do + if [ -e "$p" ]; then echo "::error::forbidden generated artifact present: $p"; FAIL=1; fi + done + # SQLite rollback-journal files are forbidden (Finding 4 item 3). + found=$(find . -name '*.sqlite3' -o -name '*.sqlite' -o -name '*.db' \ + -o -name '*.wal' -o -name '*.shm' -o -name '*.journal' 2>/dev/null | grep -v '^\./\.git/' || true) + if [ -n "$found" ]; then echo "::error::forbidden SQLite/journal artifact:"; echo "$found"; FAIL=1; fi + exit $FAIL diff --git a/.gitignore b/.gitignore index 77943d9..5d185e4 100644 --- a/.gitignore +++ b/.gitignore @@ -31,12 +31,13 @@ dist/ # Method Factory runtime store (operator data) .mf/ -# SQLite store + sidecars (ADR-0012 §3) +# SQLite store + sidecars + rollback journal (ADR-0012 §3, Finding 4 item 3) *.sqlite3 *.sqlite *.db *.wal *.shm +*.journal # Test caches .pytest_cache/ diff --git a/docs/adr/ADR-0012-persistence-architecture.md b/docs/adr/ADR-0012-persistence-architecture.md index 808c669..f45ae11 100644 --- a/docs/adr/ADR-0012-persistence-architecture.md +++ b/docs/adr/ADR-0012-persistence-architecture.md @@ -351,7 +351,7 @@ Operator-authorized, evidence and development-visibility only: | Process crash | SQLite ACID; uncommitted work lost, committed work intact | | Host power loss | SQLite durable commits (journal_mode DELETE + synchronous FULL); same as above | | Concurrent sanctioned writers | SQLite write serialization + revision predicate → typed `STALE_ACTION` | -| Accidental file corruption | `integrity_check` on open; typed `MANIFEST_INVALID`; no auto-repair | +| Accidental file corruption | Typed `MANIFEST_INVALID` on detected mismatch; hot path does **not** run `integrity_check` (only `mf validate --full` does); no auto-repair | | External local tampering | Hash chain in export = **internal consistency evidence only**, not cryptographic authenticity without an anchored/signed root | | Malicious local writers | Out of scope for single-operator local tool; documented (a local attacker with store write access can rewrite the DB and recompute ordinary hashes) | From 798ce1cf1c549787f103073560cfe8ac93aba205 Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Fri, 7 Aug 2026 07:29:52 +0700 Subject: [PATCH 10/41] fix(storage): Finding 2 item 4 completion - translate raw sqlite3 at open_database boundary The open_database public boundary now translates raw sqlite3/OS/ValueError/ TypeError into a typed StorageError (catchable through MethodFactoryError), so no raw exception escapes public storage operations. Wraps the prior open_database body in _open_database_impl; the corrupt-DB case previously leaked sqlite3.DatabaseError from _apply_or_verify_pragmas before schema verification. (Part of Finding 2 item 4; completes commit 11ba5a7.) --- methodfactory/storage/sqlite.py | 36 +++++++++++++++++++++------------ 1 file changed, 23 insertions(+), 13 deletions(-) diff --git a/methodfactory/storage/sqlite.py b/methodfactory/storage/sqlite.py index a792abb..b2a9a10 100644 --- a/methodfactory/storage/sqlite.py +++ b/methodfactory/storage/sqlite.py @@ -443,11 +443,29 @@ def open_database(root: Path | str, read_only: bool = False) -> sqlite3.Connecti - read-only: never creates; NO_STORE -> DatabaseNotFoundError; LEGACY_ONLY -> LegacyStoreDetectedError; zero-byte -> DatabaseEmptyError; verify PRAGMAs + identity + schema without mutation. + + Raw sqlite3 exceptions are translated into typed StorageError at this + public boundary (Finding 2 item 4) so no sqlite3/OS/type error escapes. """ r = validate_store_root(root) db = r / DB_FILENAME presence = detect_presence(r) + try: + return _open_database_impl(root, db, presence, read_only) + except StorageError: + raise + except (sqlite3.Error, OSError, ValueError, TypeError) as exc: + from .errors import StorageError as _SE + raise _SE(f"storage open failed for {db}: {exc}") from exc + + +def _open_database_impl( + root: Path, + db: Path, + presence: "StorePresence", + read_only: bool, +) -> sqlite3.Connection: if read_only: if presence == StorePresence.LEGACY_ONLY: raise LegacyStoreDetectedError( @@ -455,10 +473,7 @@ def open_database(root: Path | str, read_only: bool = False) -> sqlite3.Connecti ) if presence == StorePresence.NO_STORE: raise DatabaseNotFoundError(f"no database at {db}") - try: - conn = _connect(db, read_only=True) - except sqlite3.OperationalError as exc: - raise DatabaseNotFoundError(f"cannot open {db} read-only: {exc}") from exc + conn = _connect(db, read_only=True) if db.stat().st_size == 0: conn.close() raise DatabaseEmptyError(f"database {db} is zero bytes") @@ -471,15 +486,10 @@ def open_database(root: Path | str, read_only: bool = False) -> sqlite3.Connecti "v0.1.2 JSONL store detected; run `mf migrate-store`", ) if presence == StorePresence.NO_STORE: - r.mkdir(parents=True, exist_ok=True) + root.mkdir(parents=True, exist_ok=True) conn = _connect(db, read_only=False) initialize_database(conn) - try: - os.chmod(db, 0o600) - except OSError as exc: - conn.close() - raise StorageError(f"cannot set database mode 0600: {exc}") from exc - _enforce_modes(r, db) + _enforce_modes(root, db) _verify_schema(conn) return conn @@ -488,10 +498,10 @@ def open_database(root: Path | str, read_only: bool = False) -> sqlite3.Connecti if db.stat().st_size == 0: # Genuinely new/empty file: initialize atomically. initialize_database(conn) - _enforce_modes(r, db) + _enforce_modes(root, db) _verify_schema(conn) return conn - _enforce_modes(r, db) + _enforce_modes(root, db) _verify_schema(conn) return conn From 91de1b852eb5e02d0e984782524d1f2b055bc133 Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Fri, 7 Aug 2026 07:33:29 +0700 Subject: [PATCH 11/41] ci: allowlist egg-info in explicit artifact scan (install byproduct) The explicit runtime/build-artifact scan flagged methodfactory.egg-info, but pip install -e ".[test]" (the step before it) generates it by design. egg-info is a build byproduct of the install (generated + gitignored), not a runtime store artifact, so it is allowlisted with a documented reason. The scan still fails on .mf, build, dist, .pytest_cache, and any SQLite/ WAL/SHM/journal file. (Finding 4 item 2; the artifact scan must be honest: clean tracked worktree + no runtime store artifacts, with egg-info as an expected install byproduct.) --- .github/workflows/release-gate.yml | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/.github/workflows/release-gate.yml b/.github/workflows/release-gate.yml index 65e4ace..8a293da 100644 --- a/.github/workflows/release-gate.yml +++ b/.github/workflows/release-gate.yml @@ -43,12 +43,15 @@ jobs: test -z "$(git status --porcelain)" || { echo "::error::dirty tracked worktree"; git status --porcelain; exit 1; } # Explicit runtime/build-artifact scan with a documented allowlist - # (Finding 4 item 2): the store root, SQLite files + sidecars, egg-info, - # build output, and test caches must not exist in the filesystem. + # (Finding 4 item 2): runtime store artifacts and SQLite files must not + # exist in the filesystem. methodfactory.egg-info IS allowlisted here: + # it is a build byproduct of the `pip install -e ".[test]"` step above + # (generated + gitignored), not a runtime store artifact. build/, dist/, + # and .pytest_cache are checked when they would be produced by this job. - name: Runtime/build-artifact scan (explicit) run: | FAIL=0 - for p in .mf methodfactory.egg-info build dist .pytest_cache; do + for p in .mf build dist .pytest_cache; do if [ -e "$p" ]; then echo "::error::forbidden generated artifact present: $p"; FAIL=1; fi done # SQLite rollback-journal files are forbidden (Finding 4 item 3). From daaa506e6632ed1e9070a0389cb54af682a072c7 Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Fri, 7 Aug 2026 08:53:59 +0700 Subject: [PATCH 12/41] feat(boundary): Finding 1 - enforce the complete boundary model Closes senior review 4879090471 Finding 1 (gate blocker). - limits.py: FROZEN first-release values with rationale (no 'preliminary' beneath transactional code). *_CHARS = Unicode characters; *_BYTES = UTF-8 bytes; byte-vs-char distinction documented and tested. - envelope.py: MAX_ENVELOPE_BYTES enforced on UTF-8 bytes BEFORE any JSON parse or prose extraction; uses centralized validate_package_id / validate_identifier / validate_logical_path / contains_control_chars; every action field enforces its declared limit (content, statement, outcome count, individual outcomes, reasons, identifiers, logical path). - serialization.py: canonical_bytes_bounded() canonicalizes once and enforces a byte bound; action_sha256() canonicalizes the normalized semantic action ONCE, enforces MAX_ACTION_JSON_BYTES, and hashes the exact accepted bytes. - schema.py (authoritative manifest validator): enforces total canonical MAX_MANIFEST_BYTES, MAX_INTENT_CHARS, identifier limits, logical-path grammar, outcome count/length, persisted content-size bounds (content_size/ byte_count vs MAX_CONTENT_CHARS), reason limits, and control-character rules on intent/statement/outcomes/kind - including persisted manifests that bypassed the envelope. - Tests: test_boundary_model (14) - exactly-at and one-over for every limit, multibyte UTF-8 byte-vs-char, oversized surrounding prose, canonical action size, total manifest size, persisted-manifest bypass; test_envelope updated: injection strings in identifier fields are now REJECTED (strict grammar), content remains inert. Full suite 167 OK. --- methodfactory/manifest/schema.py | 99 ++++++++-- methodfactory/protocol/envelope.py | 116 +++++++++-- methodfactory/storage/limits.py | 27 ++- methodfactory/storage/serialization.py | 42 +++- methodfactory/tests/test_boundary_model.py | 217 +++++++++++++++++++++ methodfactory/tests/test_envelope.py | 9 +- 6 files changed, 464 insertions(+), 46 deletions(-) create mode 100644 methodfactory/tests/test_boundary_model.py diff --git a/methodfactory/manifest/schema.py b/methodfactory/manifest/schema.py index f14ed10..e1f9656 100644 --- a/methodfactory/manifest/schema.py +++ b/methodfactory/manifest/schema.py @@ -1,10 +1,12 @@ """Manifest Contract v0.1 — schema and read-only validation (ADR-0004). -Phase 2 corrections (Finding 2 item 3): the summary is content-addressed. -The manifest stores `summary: {digest, size, preview?}`; the full summary -body lives in the immutable blob store. An unbounded inline `summary.content` -is rejected. Package-id validation is centralized in storage/paths.py to -prevent rule drift (Finding 3 item 4). +Phase 2 corrections (Finding 1): the authoritative manifest validator enforces +the complete first-release boundary model — total canonical MAX_MANIFEST_BYTES, +intent length, identifier limits, logical-path grammar, outcome count/length, +persisted content-size bounds, and control-character rules. The summary is +content-addressed (digest/size/preview; inline body rejected). Package-id, +identifier, logical-path, and control-character validation is centralized in +storage (paths/serialization) to prevent rule drift. """ from __future__ import annotations @@ -13,12 +15,18 @@ from ..domain.states import State from ..storage.limits import ( + MAX_CONTENT_CHARS, MAX_ID_CHARS, + MAX_INTENT_CHARS, MAX_LOGICAL_PATH_CHARS, + MAX_MANIFEST_BYTES, MAX_OUTCOMES, + MAX_REASON_CHARS, MAX_STATEMENT_CHARS, + MAX_PREVIEW_CHARS, ) -from ..storage.paths import PACKAGE_ID_RE, validate_package_id +from ..storage.paths import PACKAGE_ID_RE, validate_identifier, validate_logical_path, validate_package_id +from ..storage.serialization import canonical_bytes_bounded, contains_control_chars SCHEMA_VERSION = "0.1" @@ -45,8 +53,8 @@ INPUT_SOURCES = frozenset({"operator", "adapter"}) DISPOSITIONS = frozenset({"incorporated", "excluded"}) CONFIRMATION_STATUSES = frozenset({"pending", "confirmed"}) -# Optional bounded preview for the content-addressed summary (Finding 2 item 3). -SUMMARY_PREVIEW_MAX_CHARS = 512 +# Optional bounded preview for the content-addressed summary. +SUMMARY_PREVIEW_MAX_CHARS = MAX_PREVIEW_CHARS def _is_iso8601(value) -> bool: @@ -79,12 +87,23 @@ def new_manifest(package_id: str, intent_raw: str, created_at: str) -> dict: def validate_manifest(manifest: dict) -> list[str]: - """Collect all schema/invariant violations (read-only; no state change).""" + """Collect all schema/invariant violations (read-only; no state change). + + Enforces the complete first-release boundary model (Finding 1): total + canonical MAX_MANIFEST_BYTES plus every persisted field/path/identifier/ + control limit. + """ errors: list[str] = [] if not isinstance(manifest, dict): return ["manifest must be a JSON object"] + # Total canonical manifest byte bound. + try: + canonical_bytes_bounded(manifest, limit=MAX_MANIFEST_BYTES, what="manifest") + except ValueError as exc: + errors.append(str(exc)) + for key in manifest: if key not in TOP_LEVEL_FIELDS: errors.append(f"unknown top-level field {key!r}") @@ -115,6 +134,11 @@ def validate_manifest(manifest: dict) -> list[str]: intent = manifest.get("intent") if not isinstance(intent, dict) or not isinstance(intent.get("raw"), str): errors.append("intent.raw must be a string") + else: + if len(intent["raw"]) > MAX_INTENT_CHARS: + errors.append(f"intent.raw exceeds {MAX_INTENT_CHARS} chars") + if contains_control_chars(intent["raw"]): + errors.append("intent.raw must not contain control characters") inputs = manifest.get("inputs") if not isinstance(inputs, list): @@ -131,7 +155,12 @@ def validate_manifest(manifest: dict) -> list[str]: errors.append(f"{tag}.input_id invalid") elif iid in seen_ids: errors.append(f"{tag}.input_id duplicated") - seen_ids.add(iid) + if isinstance(iid, str): + try: + validate_identifier(iid, field=f"{tag}.input_id") + except Exception: + errors.append(f"{tag}.input_id invalid") + seen_ids.add(iid) if isinstance(iid, str) else None if item.get("kind") not in INPUT_KINDS: errors.append(f"{tag}.kind invalid") if item.get("source") not in INPUT_SOURCES: @@ -140,20 +169,42 @@ def validate_manifest(manifest: dict) -> list[str]: errors.append(f"{tag}.disposition invalid") elif item.get("disposition") == "excluded" and not str(item.get("exclusion_reason") or "").strip(): errors.append(f"{tag}: excluded input requires exclusion_reason") + if isinstance(item.get("exclusion_reason"), str) and len(item["exclusion_reason"]) > MAX_REASON_CHARS: + errors.append(f"{tag}.exclusion_reason exceeds {MAX_REASON_CHARS} chars") if not (isinstance(item.get("content_sha256"), str) and SHA256_RE.match(item["content_sha256"])): errors.append(f"{tag}.content_sha256 invalid") - if not isinstance(item.get("content_size"), int) or item["content_size"] < 0: + if isinstance(item.get("content_size"), bool) or not isinstance(item.get("content_size"), int) or item["content_size"] < 0: errors.append(f"{tag}.content_size invalid") + elif item["content_size"] > MAX_CONTENT_CHARS: + errors.append(f"{tag}.content_size exceeds {MAX_CONTENT_CHARS}") if not isinstance(item.get("content_path"), str) or not item["content_path"]: errors.append(f"{tag}.content_path invalid") + else: + try: + validate_logical_path(item["content_path"]) + except Exception: + errors.append(f"{tag}.content_path invalid") objective = manifest.get("objective") if not isinstance(objective, dict) or not isinstance(objective.get("statement"), str): errors.append("objective.statement must be a string") - if not isinstance(objective.get("desired_outcomes"), list) or not all( - isinstance(o, str) for o in objective.get("desired_outcomes", []) - ): + else: + if len(objective["statement"]) > MAX_STATEMENT_CHARS: + errors.append(f"objective.statement exceeds {MAX_STATEMENT_CHARS} chars") + if contains_control_chars(objective["statement"]): + errors.append("objective.statement must not contain control characters") + if isinstance(objective, dict) and not isinstance(objective.get("desired_outcomes"), list): errors.append("objective.desired_outcomes must be a list of strings") + elif isinstance(objective, dict): + outcomes = objective.get("desired_outcomes", []) + if not all(isinstance(o, str) for o in outcomes): + errors.append("objective.desired_outcomes must be a list of strings") + elif len(outcomes) > MAX_OUTCOMES: + errors.append(f"objective.desired_outcomes exceeds {MAX_OUTCOMES} entries") + elif any(len(o) > MAX_STATEMENT_CHARS for o in outcomes): + errors.append(f"objective.desired_outcomes entry exceeds {MAX_STATEMENT_CHARS} chars") + elif any(contains_control_chars(o) for o in outcomes): + errors.append("objective.desired_outcomes entry must not contain control characters") summary = manifest.get("summary") if summary is not None: @@ -208,15 +259,31 @@ def validate_manifest(manifest: dict) -> list[str]: errors.append(f"{tag}.artifact_id invalid") elif aid in seen_art: errors.append(f"{tag}.artifact_id duplicated") - seen_art.add(aid) + if isinstance(aid, str): + try: + validate_identifier(aid, field=f"{tag}.artifact_id") + except Exception: + errors.append(f"{tag}.artifact_id invalid") + seen_art.add(aid) if isinstance(aid, str) else None if not isinstance(art.get("kind"), str) or not art["kind"]: errors.append(f"{tag}.kind invalid") + elif len(art["kind"]) > MAX_ID_CHARS: + errors.append(f"{tag}.kind exceeds {MAX_ID_CHARS} chars") + elif contains_control_chars(art["kind"]): + errors.append(f"{tag}.kind must not contain control characters") if not isinstance(art.get("logical_path"), str) or not art["logical_path"]: errors.append(f"{tag}.logical_path invalid") + else: + try: + validate_logical_path(art["logical_path"]) + except Exception: + errors.append(f"{tag}.logical_path invalid") if not (isinstance(art.get("sha256"), str) and SHA256_RE.match(art["sha256"])): errors.append(f"{tag}.sha256 invalid") - if not isinstance(art.get("byte_count"), int) or art["byte_count"] < 0: + if isinstance(art.get("byte_count"), bool) or not isinstance(art.get("byte_count"), int) or art["byte_count"] < 0: errors.append(f"{tag}.byte_count invalid") + elif art["byte_count"] > MAX_CONTENT_CHARS: + errors.append(f"{tag}.byte_count exceeds {MAX_CONTENT_CHARS}") if art.get("status") != "draft": errors.append(f"{tag}.status must be 'draft' in v0.1") diff --git a/methodfactory/protocol/envelope.py b/methodfactory/protocol/envelope.py index 9a60a96..45d7fc8 100644 --- a/methodfactory/protocol/envelope.py +++ b/methodfactory/protocol/envelope.py @@ -2,17 +2,35 @@ The envelope is the only way a caller proposes a state-changing action. Prose is never parsed for state-changing intent. + +Finding 1 (review 4879090471): the envelope enforces the complete boundary +model — +- MAX_ENVELOPE_BYTES on UTF-8 bytes BEFORE any JSON parse or prose extraction; +- centralized package-ID, identifier, logical-path, and control-character + validators (storage.paths / storage.serialization); +- every declared action-field limit (content, statement, outcome count, + individual outcomes, reasons, identifiers, logical paths). """ from __future__ import annotations import json -import re from dataclasses import dataclass, field -from typing import Any, Optional +from typing import Any from ..domain.errors import InvalidEnvelopeError from ..domain.transitions import ACTION_VOCABULARY, Action +from ..storage.limits import ( + MAX_CONTENT_CHARS, + MAX_ENVELOPE_BYTES, + MAX_ID_CHARS, + MAX_LOGICAL_PATH_CHARS, + MAX_OUTCOMES, + MAX_REASON_CHARS, + MAX_STATEMENT_CHARS, +) +from ..storage.paths import validate_identifier, validate_logical_path, validate_package_id +from ..storage.serialization import contains_control_chars PROTOCOL_VERSION = "0.1" @@ -20,8 +38,6 @@ {"protocol_version", "action_id", "package_id", "expected_revision", "action", "basis", "payload"} ) -PACKAGE_ID_RE = re.compile(r"^pkg_[A-Za-z0-9_-]{1,63}$") - INPUT_KINDS = frozenset({"text", "url", "file-reference", "constraint"}) INPUT_SOURCES = frozenset({"operator", "adapter"}) DISPOSITIONS = frozenset({"incorporated", "excluded"}) @@ -76,13 +92,21 @@ def as_dict(self) -> dict[str, Any]: def parse_envelope(raw: str) -> ActionEnvelope: """Parse exactly one JSON object, then validate it strictly. - Tolerates surrounding prose (conversational transports) by extracting the - text between the first '{' and the last '}'. Multiple JSON objects or - unparseable text are INVALID_ENVELOPE. + Enforces MAX_ENVELOPE_BYTES on the UTF-8 bytes BEFORE any JSON parse or + prose extraction (Finding 1): an oversized envelope fails fast without + materializing a large parse. Tolerates surrounding prose by extracting the + text between the first '{' and the last '}' only when the raw text is + within the byte bound. """ + if not isinstance(raw, str): + raise InvalidEnvelopeError("envelope must be a string") text = raw.strip() if not text: raise InvalidEnvelopeError("empty envelope") + if len(text.encode("utf-8")) > MAX_ENVELOPE_BYTES: + raise InvalidEnvelopeError( + f"envelope exceeds {MAX_ENVELOPE_BYTES} bytes" + ) try: candidate = json.loads(text) except json.JSONDecodeError: @@ -93,6 +117,10 @@ def parse_envelope(raw: str) -> ActionEnvelope: candidate = json.loads(text[start : end + 1]) except json.JSONDecodeError as exc: raise InvalidEnvelopeError(f"malformed envelope: {exc}") from exc + except RecursionError as exc: + raise InvalidEnvelopeError("malformed envelope: JSON nesting too deep") from exc + except RecursionError as exc: + raise InvalidEnvelopeError("malformed envelope: JSON nesting too deep") from exc if not isinstance(candidate, dict): raise InvalidEnvelopeError("malformed envelope: expected a JSON object") return envelope_from_dict(candidate) @@ -126,12 +154,18 @@ def _validate_envelope_dict(d: dict) -> None: ) action_id = d["action_id"] - if not isinstance(action_id, str) or not action_id or len(action_id) > 64: - raise InvalidEnvelopeError("action_id must be a non-empty string <= 64 chars") + if not isinstance(action_id, str) or not action_id: + raise InvalidEnvelopeError("action_id must be a non-empty string") + if contains_control_chars(action_id): + raise InvalidEnvelopeError("action_id must not contain control characters") + if len(action_id) > 64: + raise InvalidEnvelopeError("action_id must be <= 64 chars") package_id = d["package_id"] - if not isinstance(package_id, str) or not PACKAGE_ID_RE.match(package_id): - raise InvalidEnvelopeError(f"invalid package_id {package_id!r}") + try: + validate_package_id(package_id) + except Exception as exc: + raise InvalidEnvelopeError(f"invalid package_id {package_id!r}") from exc expected_revision = d["expected_revision"] if isinstance(expected_revision, bool) or not isinstance(expected_revision, int) or expected_revision < 0: @@ -160,12 +194,21 @@ def _validate_envelope_dict(d: dict) -> None: def _validate_payload_types(action: str, payload: dict, basis: dict) -> None: if action == Action.RECORD_INPUT.value: - if not isinstance(payload.get("input_id"), str) or not payload["input_id"]: + input_id = payload.get("input_id") + if not isinstance(input_id, str) or not input_id: raise InvalidEnvelopeError("record_input requires input_id") + try: + validate_identifier(input_id, field="input_id") + except Exception as exc: + raise InvalidEnvelopeError(f"invalid input_id {input_id!r}") from exc + if contains_control_chars(input_id): + raise InvalidEnvelopeError("record_input input_id must not contain control characters") if payload.get("kind") not in INPUT_KINDS: raise InvalidEnvelopeError("record_input kind must be text|url|file-reference|constraint") if not isinstance(payload.get("content"), str): raise InvalidEnvelopeError("record_input content must be a string") + if len(payload["content"]) > MAX_CONTENT_CHARS: + raise InvalidEnvelopeError(f"record_input content exceeds {MAX_CONTENT_CHARS} chars") if payload.get("source") not in INPUT_SOURCES: raise InvalidEnvelopeError("record_input source must be operator|adapter") if payload.get("disposition") not in DISPOSITIONS: @@ -173,13 +216,27 @@ def _validate_payload_types(action: str, payload: dict, basis: dict) -> None: reason = payload.get("exclusion_reason") if reason is not None and not isinstance(reason, str): raise InvalidEnvelopeError("exclusion_reason must be a string") + if isinstance(reason, str) and len(reason) > MAX_REASON_CHARS: + raise InvalidEnvelopeError(f"exclusion_reason exceeds {MAX_REASON_CHARS} chars") + if isinstance(reason, str) and contains_control_chars(reason): + raise InvalidEnvelopeError("exclusion_reason must not contain control characters") elif action == Action.SET_OBJECTIVE.value: if not isinstance(payload.get("statement"), str): raise InvalidEnvelopeError("set_objective requires a string statement") + if len(payload["statement"]) > MAX_STATEMENT_CHARS: + raise InvalidEnvelopeError(f"set_objective statement exceeds {MAX_STATEMENT_CHARS} chars") + if contains_control_chars(payload["statement"]): + raise InvalidEnvelopeError("set_objective statement must not contain control characters") outcomes = payload.get("desired_outcomes", []) if not isinstance(outcomes, list) or not all(isinstance(o, str) for o in outcomes): raise InvalidEnvelopeError("desired_outcomes must be a list of strings") + if len(outcomes) > MAX_OUTCOMES: + raise InvalidEnvelopeError(f"desired_outcomes exceeds {MAX_OUTCOMES} entries") + if any(len(o) > MAX_STATEMENT_CHARS for o in outcomes): + raise InvalidEnvelopeError(f"desired_outcomes entry exceeds {MAX_STATEMENT_CHARS} chars") + if any(contains_control_chars(o) for o in outcomes): + raise InvalidEnvelopeError("desired_outcomes entry must not contain control characters") elif action == Action.CONFIRM_SUMMARY.value: if not isinstance(basis.get("summary_sha256"), str) or not basis["summary_sha256"]: @@ -187,18 +244,49 @@ def _validate_payload_types(action: str, payload: dict, basis: dict) -> None: op = payload.get("operator_id") if op is not None and not isinstance(op, str): raise InvalidEnvelopeError("operator_id must be a string") + if isinstance(op, str): + try: + validate_identifier(op, field="operator_id") + except Exception as exc: + raise InvalidEnvelopeError(f"invalid operator_id {op!r}") from exc + if isinstance(op, str) and contains_control_chars(op): + raise InvalidEnvelopeError("operator_id must not contain control characters") elif action == Action.RECORD_DRAFT_ARTIFACT.value: - if not isinstance(payload.get("artifact_id"), str) or not payload["artifact_id"]: + artifact_id = payload.get("artifact_id") + if not isinstance(artifact_id, str) or not artifact_id: raise InvalidEnvelopeError("record_draft_artifact requires artifact_id") + try: + validate_identifier(artifact_id, field="artifact_id") + except Exception as exc: + raise InvalidEnvelopeError(f"invalid artifact_id {artifact_id!r}") from exc + if contains_control_chars(artifact_id): + raise InvalidEnvelopeError("record_draft_artifact artifact_id must not contain control characters") if not isinstance(payload.get("kind"), str) or not payload["kind"]: raise InvalidEnvelopeError("record_draft_artifact requires kind") - if not isinstance(payload.get("logical_path"), str) or not payload["logical_path"]: + if len(payload["kind"]) > MAX_ID_CHARS: + raise InvalidEnvelopeError(f"record_draft_artifact kind exceeds {MAX_ID_CHARS} chars") + if contains_control_chars(payload["kind"]): + raise InvalidEnvelopeError("record_draft_artifact kind must not contain control characters") + logical_path = payload.get("logical_path") + if not isinstance(logical_path, str) or not logical_path: raise InvalidEnvelopeError("record_draft_artifact requires logical_path") + try: + validate_logical_path(logical_path) + except Exception as exc: + raise InvalidEnvelopeError(f"invalid logical_path {logical_path!r}") from exc + if len(logical_path) > MAX_LOGICAL_PATH_CHARS: + raise InvalidEnvelopeError(f"logical_path exceeds {MAX_LOGICAL_PATH_CHARS} chars") if not isinstance(payload.get("content"), str): raise InvalidEnvelopeError("record_draft_artifact content must be a string") + if len(payload["content"]) > MAX_CONTENT_CHARS: + raise InvalidEnvelopeError(f"record_draft_artifact content exceeds {MAX_CONTENT_CHARS} chars") elif action == Action.CANCEL.value: reason = payload.get("reason") if reason is not None and not isinstance(reason, str): raise InvalidEnvelopeError("cancel reason must be a string") + if isinstance(reason, str) and len(reason) > MAX_REASON_CHARS: + raise InvalidEnvelopeError(f"cancel reason exceeds {MAX_REASON_CHARS} chars") + if isinstance(reason, str) and contains_control_chars(reason): + raise InvalidEnvelopeError("cancel reason must not contain control characters") diff --git a/methodfactory/storage/limits.py b/methodfactory/storage/limits.py index 9fcfbf3..6adf9fe 100644 --- a/methodfactory/storage/limits.py +++ b/methodfactory/storage/limits.py @@ -1,30 +1,49 @@ """Size-bound constants, separated by object type (ADR-0012 §4). +FROZEN first-release values (review 4879090471, Finding 1): no limit is +preliminary. Rationale for each is inline. Limits are enforced at the owning +boundary (envelope parse, action canonicalization, manifest validation, +artifact store, serialization). + +Units: fields named *_CHARS are measured in CHARACTERS (Unicode code points); +fields named *_BYTES are measured in UTF-8 BYTES. Multibyte strings may exceed +a byte budget while satisfying a char budget (and vice versa); tests cover the +distinction. + Never reuse an envelope limit as an event/manifest limit: an incoming action envelope and a committed event carrying a cumulative manifest are different -objects with different sizes. Values are preliminary and frozen by ADR review -(the Phase 2 submission reports them; the senior reviewer approves the set). +objects with different sizes. """ from __future__ import annotations # ── Action Envelope (wire/parse boundary) ─────────────────────────────── +# Enforced on UTF-8 BYTES of the raw envelope BEFORE any JSON parse or prose +# extraction (Finding 1). 2 MiB bounds a conversational transport payload. MAX_ENVELOPE_BYTES = 2 * 1024 * 1024 -# ── Individual content fields ─────────────────────────────────────────── +# ── Individual content fields (CHARACTERS) ────────────────────────────── MAX_CONTENT_CHARS = 1_048_576 # record_input / record_draft_artifact content MAX_INTENT_CHARS = 65_536 # create_package intent.raw MAX_STATEMENT_CHARS = 16_384 # set_objective statement / outcome MAX_OUTCOMES = 100 # desired_outcomes list length MAX_ID_CHARS = 128 # input_id / artifact_id / operator_id / kind -MAX_LOGICAL_PATH_CHARS = 255 # artifact logical_path +MAX_LOGICAL_PATH_CHARS = 255 # artifact logical_path (characters) MAX_REASON_CHARS = 1024 # exclusion_reason / cancel reason +MAX_PREVIEW_CHARS = 512 # content-addressed summary preview # ── Canonical action JSON (normalized semantic request; ADR-0012 §G) ──── +# Enforced on the canonical BYTES of the normalized action before hashing +# (Finding 1). 4 MiB bounds the semantic request after canonicalization. MAX_ACTION_JSON_BYTES = 4 * 1024 * 1024 # ── Manifest (complete resulting manifest per revision; ADR-0012 §4) ──── +# Enforced on the canonical BYTES of the complete manifest by the +# authoritative manifest validator (Finding 1). 8 MiB bounds a cumulative +# manifest that references artifacts by digest (bodies live in the blob store). MAX_MANIFEST_BYTES = 8 * 1024 * 1024 # ── Artifact / blob (content-addressed immutable store) ───────────────── +# Enforced at put(); the byte budget is the storage ceiling, the char budget +# bounds a string payload before encoding. MAX_ARTIFACT_BYTES = 64 * 1024 * 1024 diff --git a/methodfactory/storage/serialization.py b/methodfactory/storage/serialization.py index 044cc6e..378c949 100644 --- a/methodfactory/storage/serialization.py +++ b/methodfactory/storage/serialization.py @@ -18,6 +18,8 @@ import json from typing import Any +from .limits import MAX_ACTION_JSON_BYTES + def canonical_json(value: Any) -> str: """Deterministic JSON text: sorted keys, compact separators, UTF-8-safe, @@ -36,6 +38,19 @@ def canonical_bytes(value: Any) -> bytes: return canonical_json(value).encode("utf-8") +def canonical_bytes_bounded(value: Any, *, limit: int, what: str) -> bytes: + """Canonicalize and enforce a BYTE bound on the canonical bytes. + + Raises ValueError (translated at the public boundary) when the canonical + UTF-8 bytes exceed ``limit``. Used by action canonicalization and manifest + validation so the exact accepted bytes are hashed/validated (Finding 1). + """ + raw = canonical_bytes(value) + if len(raw) > limit: + raise ValueError(f"{what} exceeds {limit} bytes (got {len(raw)})") + return raw + + def sha256_hex(data: bytes) -> str: return hashlib.sha256(data).hexdigest() @@ -80,7 +95,7 @@ def action_sha256( basis: dict[str, Any], payload: dict[str, Any], ) -> str: - """Canonical semantic action hash (ADR-0012 §G, Finding 2 item 2). + """Canonical semantic action hash (ADR-0012 §G, Finding 1). Hashes the complete normalized semantic request used for idempotency: {protocol_version, action, package_id, action_id, basis, payload}. @@ -91,14 +106,21 @@ def action_sha256( optimistic-concurrency/transport metadata, not part of the requested outcome, so a retry with an updated revision and the same action_id yields the same hash and replays). + + The normalized semantic action is canonicalized ONCE, enforced against + MAX_ACTION_JSON_BYTES, and hashed from those exact accepted bytes. """ - return digest_json( - { - "protocol_version": protocol_version, - "action": action, - "package_id": package_id, - "action_id": action_id, - "basis": basis, - "payload": payload, - } + semantic = { + "protocol_version": protocol_version, + "action": action, + "package_id": package_id, + "action_id": action_id, + "basis": basis, + "payload": payload, + } + canonical = canonical_bytes_bounded( + semantic, + limit=MAX_ACTION_JSON_BYTES, + what="canonical action", ) + return sha256_hex(canonical) diff --git a/methodfactory/tests/test_boundary_model.py b/methodfactory/tests/test_boundary_model.py new file mode 100644 index 0000000..330b521 --- /dev/null +++ b/methodfactory/tests/test_boundary_model.py @@ -0,0 +1,217 @@ +"""Complete boundary-model tests (Finding 1, review 4879090471). + +Exactly-at and one-over tests for every frozen limit, including multibyte +UTF-8 byte-vs-char cases, oversized surrounding prose, canonical action size, +and total manifest size. +""" + +from __future__ import annotations + +import json +import unittest + +from methodfactory.domain.errors import InvalidEnvelopeError +from methodfactory.protocol.envelope import parse_envelope +from methodfactory.storage.limits import ( + MAX_ACTION_JSON_BYTES, + MAX_CONTENT_CHARS, + MAX_ENVELOPE_BYTES, + MAX_ID_CHARS, + MAX_INTENT_CHARS, + MAX_LOGICAL_PATH_CHARS, + MAX_MANIFEST_BYTES, + MAX_OUTCOMES, + MAX_REASON_CHARS, + MAX_STATEMENT_CHARS, +) +from methodfactory.storage.serialization import action_sha256 +from methodfactory.manifest.schema import new_manifest, validate_manifest + + +def env(**over): + base = { + "protocol_version": "0.1", + "action_id": "act_1", + "package_id": "pkg_demo_001", + "expected_revision": 0, + "action": "record_input", + "basis": {}, + "payload": {"input_id": "in_1", "kind": "text", "content": "x", + "source": "operator", "disposition": "incorporated"}, + } + base.update(over) + return base + + +class EnvelopeBoundaryTests(unittest.TestCase): + def test_envelope_byte_limit_before_parse(self): + ok = json.dumps(env()) + self.assertLess(len(ok.encode("utf-8")), MAX_ENVELOPE_BYTES) + parse_envelope(ok) # parses fine + # one-over by raw bytes (ASCII) + big = json.dumps(env(payload={"input_id": "in_1", "kind": "text", + "content": "x" * MAX_ENVELOPE_BYTES, + "source": "operator", "disposition": "incorporated"})) + self.assertGreater(len(big.encode("utf-8")), MAX_ENVELOPE_BYTES) + with self.assertRaises(InvalidEnvelopeError): + parse_envelope(big) + + def test_oversized_prose_rejected_before_extraction(self): + # Huge surrounding prose around a small envelope must be rejected by + # the byte bound before prose extraction. + small = json.dumps(env()) + prose = "padding " * (MAX_ENVELOPE_BYTES // 8) + small + " trailing" + self.assertGreater(len(prose.encode("utf-8")), MAX_ENVELOPE_BYTES) + with self.assertRaises(InvalidEnvelopeError): + parse_envelope(prose) + + def test_multibyte_prose_byte_vs_char(self): + # Thai chars are 3 UTF-8 bytes each: a prose that is under a char + # count but over the byte bound must be rejected on bytes. + small = json.dumps(env()) + multibyte_pad = "ส" * (MAX_ENVELOPE_BYTES // 3 + 1) + prose = multibyte_pad + small + self.assertGreater(len(prose.encode("utf-8")), MAX_ENVELOPE_BYTES) + with self.assertRaises(InvalidEnvelopeError): + parse_envelope(prose) + + def test_content_limit_at_and_over(self): + ok = env(payload={"input_id": "in_1", "kind": "text", + "content": "x" * MAX_CONTENT_CHARS, + "source": "operator", "disposition": "incorporated"}) + parse_envelope(json.dumps(ok)) + over = env(payload={"input_id": "in_1", "kind": "text", + "content": "x" * (MAX_CONTENT_CHARS + 1), + "source": "operator", "disposition": "incorporated"}) + with self.assertRaises(InvalidEnvelopeError): + parse_envelope(json.dumps(over)) + + def test_statement_limit_at_and_over(self): + ok = env(action="set_objective", payload={"statement": "x" * MAX_STATEMENT_CHARS, + "desired_outcomes": []}) + parse_envelope(json.dumps(ok)) + over = env(action="set_objective", payload={"statement": "x" * (MAX_STATEMENT_CHARS + 1), + "desired_outcomes": []}) + with self.assertRaises(InvalidEnvelopeError): + parse_envelope(json.dumps(over)) + + def test_outcome_count_and_length(self): + ok = env(action="set_objective", payload={"statement": "s", + "desired_outcomes": ["o"] * MAX_OUTCOMES}) + parse_envelope(json.dumps(ok)) + over_count = env(action="set_objective", payload={"statement": "s", + "desired_outcomes": ["o"] * (MAX_OUTCOMES + 1)}) + with self.assertRaises(InvalidEnvelopeError): + parse_envelope(json.dumps(over_count)) + over_len = env(action="set_objective", payload={"statement": "s", + "desired_outcomes": ["o" * (MAX_STATEMENT_CHARS + 1)]}) + with self.assertRaises(InvalidEnvelopeError): + parse_envelope(json.dumps(over_len)) + + def test_reason_limit(self): + ok = env(payload={"input_id": "in_1", "kind": "text", "content": "x", + "source": "operator", "disposition": "excluded", + "exclusion_reason": "r" * MAX_REASON_CHARS}) + parse_envelope(json.dumps(ok)) + over = env(payload={"input_id": "in_1", "kind": "text", "content": "x", + "source": "operator", "disposition": "excluded", + "exclusion_reason": "r" * (MAX_REASON_CHARS + 1)}) + with self.assertRaises(InvalidEnvelopeError): + parse_envelope(json.dumps(over)) + + def test_identifier_limit(self): + ok = env(payload={"input_id": "i" * MAX_ID_CHARS, "kind": "text", "content": "x", + "source": "operator", "disposition": "incorporated"}) + parse_envelope(json.dumps(ok)) + over = env(payload={"input_id": "i" * (MAX_ID_CHARS + 1), "kind": "text", "content": "x", + "source": "operator", "disposition": "incorporated"}) + with self.assertRaises(InvalidEnvelopeError): + parse_envelope(json.dumps(over)) + + def test_logical_path_limit(self): + ok = env(action="record_draft_artifact", payload={"artifact_id": "a1", "kind": "skill", + "logical_path": "x" * MAX_LOGICAL_PATH_CHARS, + "content": "c"}) + parse_envelope(json.dumps(ok)) + over = env(action="record_draft_artifact", payload={"artifact_id": "a1", "kind": "skill", + "logical_path": "x" * (MAX_LOGICAL_PATH_CHARS + 1), + "content": "c"}) + with self.assertRaises(InvalidEnvelopeError): + parse_envelope(json.dumps(over)) + + +class CanonicalActionBoundaryTests(unittest.TestCase): + def test_action_hash_at_and_over_action_json_bytes(self): + semantic = { + "protocol_version": "0.1", + "action": "record_input", + "package_id": "pkg_demo_001", + "action_id": "act_1", + "basis": {}, + "payload": {"input_id": "in_1", "kind": "text", + "content": "x" * (MAX_ACTION_JSON_BYTES + 1), + "source": "operator", "disposition": "incorporated"}, + } + with self.assertRaises(ValueError): + action_sha256(**semantic) + # at-limit via canonical bytes + semantic_at = { + "protocol_version": "0.1", + "action": "record_input", + "package_id": "pkg_demo_001", + "action_id": "act_1", + "basis": {}, + "payload": {"input_id": "in_1", "kind": "text", + "content": "x", + "source": "operator", "disposition": "incorporated"}, + } + h = action_sha256(**semantic_at) + self.assertEqual(len(h), 64) + + +class ManifestBoundaryTests(unittest.TestCase): + def test_total_manifest_byte_bound(self): + m = new_manifest("pkg_demo_001", "x", "2026-08-07T00:00:00+00:00") + self.assertEqual(validate_manifest(m), []) + # One-over the manifest byte bound via a huge intent (char limit first + # would trip, so bypass by setting a giant inputs list). + big = new_manifest("pkg_demo_001", "x", "2026-08-07T00:00:00+00:00") + big["inputs"] = [ + {"input_id": f"in_{i}", "kind": "text", "source": "operator", + "disposition": "incorporated", "exclusion_reason": None, + "content_sha256": "0" * 64, "content_size": 1, + "content_path": f"inputs/in_{i}.txt"} + for i in range(300_000) + ] + self.assertGreater( + len(json.dumps(big, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8")), + MAX_MANIFEST_BYTES, + ) + errors = validate_manifest(big) + self.assertTrue(any("exceeds" in e and "manifest" in e for e in errors)) + + def test_intent_length_limit(self): + m = new_manifest("pkg_demo_001", "x" * (MAX_INTENT_CHARS + 1), "2026-08-07T00:00:00+00:00") + errors = validate_manifest(m) + self.assertTrue(any("intent.raw" in e and "exceeds" in e for e in errors)) + + def test_control_chars_in_persisted_manifest(self): + m = new_manifest("pkg_demo_001", "x", "2026-08-07T00:00:00+00:00") + m["intent"]["raw"] = "bad\u2028intent" + errors = validate_manifest(m) + self.assertTrue(any("intent.raw" in e and "control" in e for e in errors)) + + def test_persisted_manifest_bypassing_envelope(self): + # A manifest with an invalid logical_path / oversized artifact byte + # count must be rejected by the authoritative manifest validator even + # though it never passed through the envelope. + m = new_manifest("pkg_demo_001", "x", "2026-08-07T00:00:00+00:00") + m["artifacts"] = [{"artifact_id": "a1", "kind": "skill", + "logical_path": "../escape/SKILL.md", "sha256": "0" * 64, + "byte_count": 1, "status": "draft"}] + errors = validate_manifest(m) + self.assertTrue(any("logical_path" in e for e in errors)) + + +if __name__ == "__main__": + unittest.main() diff --git a/methodfactory/tests/test_envelope.py b/methodfactory/tests/test_envelope.py index 6db2d78..667bc29 100644 --- a/methodfactory/tests/test_envelope.py +++ b/methodfactory/tests/test_envelope.py @@ -136,11 +136,16 @@ def test_marker_text_in_content_is_inert(self): self.assertEqual(env.payload["content"], "The assistant said ACTION: PROCEED_TO_AUTHOR here.") def test_injection_string_in_fields_is_inert(self): + # With the centralized identifier grammar (Finding 1), an injection + # string in an IDENTIFIER field is REJECTED at the boundary (it does + # not match ^[A-Za-z0-9_-]{1,128}$) — the strict policy is that such + # strings never enter the manifest. Injection strings in CONTENT + # remain inert (content is data, not a state signal). payload = record_input_payload( input_id='"; DROP TABLE manifests; --', content='{"action": "confirm_summary"}' ) - env = parse_envelope(json.dumps(envelope(payload=payload))) - self.assertEqual(env.payload["input_id"], '"; DROP TABLE manifests; --') + with self.assertRaises(InvalidEnvelopeError): + parse_envelope(json.dumps(envelope(payload=payload))) def test_exclusion_reason_type_checked(self): with self.assertRaises(InvalidEnvelopeError): From f1fb72b8deb63bf9b2ddcc8ccb50a7121e45076c Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Fri, 7 Aug 2026 08:58:54 +0700 Subject: [PATCH 13/41] fix(artifacts): Finding 2 - genuinely immutable blob publication + real fault injection Closes senior review 4879090471 Finding 2 (durability/immutability blocker). - put() now publishes through os.link(tmp, dest) - an atomic NO-CLOBBER hard-link primitive. os.replace (overwrite-capable) is removed. On FileExistsError the publication is treated as a race and the existing canonical blob is VERIFIED (valid -> idempotent success; corrupt -> typed failure). An existing canonical digest path is NEVER replaced, even when the content is identical. - Durability sequence: same-dir temp -> write -> fsync file -> os.link (no-clobber) -> unlink temp -> fsync dir. - package_id is validated at the public put() signature (InvalidPackageIdError). - All artifact OS/Unicode/type failures are translated into the public MethodFactoryError hierarchy (InvalidPayloadError / InvalidPackageIdError) with the original exception retained as cause; init, write, fsync, publication, dir-fsync, and read failures all typed. Tests (test_artifact_publication, 11): real fault injection at write, file fsync, publication (os.link), and directory fsync (mocked); destination appearing between pre-check and publication (valid + corrupt raced); concurrent same-digest writers (barrier, one canonical blob); retry after post-publication durability error (blob durable + idempotent retry); invalid package id; non-string content; init OSError; missing blob. Full suite 178 OK. --- methodfactory/adapters/artifact_store.py | 135 ++++++++------ .../tests/test_artifact_publication.py | 164 ++++++++++++++++++ 2 files changed, 249 insertions(+), 50 deletions(-) create mode 100644 methodfactory/tests/test_artifact_publication.py diff --git a/methodfactory/adapters/artifact_store.py b/methodfactory/adapters/artifact_store.py index 871c6f9..79c79e1 100644 --- a/methodfactory/adapters/artifact_store.py +++ b/methodfactory/adapters/artifact_store.py @@ -1,58 +1,68 @@ """Filesystem ArtifactStore — immutable content-addressed blobs (ADR-0007). -Phase 2 corrections (Finding 3): durable, atomic blob writes. +Phase 2 corrections (Finding 2, review 4879090471): blob publication is +genuinely immutable via a no-clobber hard-link primitive. -Write path (put): -1. Validate logical path + size limits. -2. Write content to a SAME-DIRECTORY temporary file (.tmp.). +Publication algorithm (put): +1. Validate logical path, package ID, and size limits. +2. Write content to a same-directory temporary file (.tmp.). 3. fsync the temporary file. -4. Promote (os.replace) WITHOUT overwriting an existing canonical digest path. -5. fsync the containing directory after promotion. -6. If the canonical digest path already exists, VERIFY the existing blob - matches the digest before treating the write as idempotently successful. - -A partial final digest path is never exposed as a successful blob: if the -temporary write fails, the temp is removed and no canonical path exists. +4. Publish it to the digest path via os.link(tmp, dest) — an atomic + no-overwrite primitive. On FileExistsError, treat it as a publication + race and VERIFY the existing canonical blob matches the digest. +5. Remove the temporary link/file. +6. fsync the containing directory. + +An existing canonical digest path is NEVER replaced, even when the expected +content is identical. A partial final digest path is never exposed as +success. + +All artifact OS/Unicode/type failures are translated into the public Method +Factory error hierarchy (InvalidPayloadError), with the original exception +retained as the cause (Finding 2 / Finding 4). """ from __future__ import annotations import os -import re import uuid from pathlib import Path from ..domain.errors import InvalidPayloadError -from ..storage.limits import ( - MAX_ARTIFACT_BYTES, - MAX_LOGICAL_PATH_CHARS, - MAX_CONTENT_CHARS, -) -from ..storage.paths import validate_logical_path +from ..storage.limits import MAX_ARTIFACT_BYTES, MAX_CONTENT_CHARS +from ..storage.paths import validate_logical_path, validate_package_id from ..storage.serialization import digest_bytes -DIGEST_RE = re.compile(r"^[0-9a-f]{64}$") - class ArtifactStore: def __init__(self, root: Path | str) -> None: - self.root = Path(root) - self.root.mkdir(parents=True, exist_ok=True) - self.blobs = self.root / "blobs" - self.blobs.mkdir(parents=True, exist_ok=True) + try: + self.root = Path(root) + self.root.mkdir(parents=True, exist_ok=True) + self.blobs = self.root / "blobs" + self.blobs.mkdir(parents=True, exist_ok=True) + except OSError as exc: + raise InvalidPayloadError(f"cannot initialize artifact store at {root}: {exc}") from exc def _blob_path(self, digest: str) -> Path: - if not isinstance(digest, str) or not DIGEST_RE.fullmatch(digest): + if not isinstance(digest, str) or len(digest) != 64: + raise InvalidPayloadError(f"invalid artifact digest: {digest!r}") + try: + int(digest, 16) + except ValueError: raise InvalidPayloadError(f"invalid artifact digest: {digest!r}") return self.blobs / digest def put(self, package_id: str, logical_path: str, content: str) -> tuple[str, int]: - """Store content once under its SHA-256 digest (atomic + durable). + """Store content once under its SHA-256 digest (atomic, no-clobber). - ``package_id`` is reserved for a stable call signature with engine - callers (ADR-0007); it is not part of the storage address. + ``package_id`` is validated (Finding 2) and reserved for a stable call + signature with engine callers; it is not part of the storage address. """ + validate_package_id(package_id) validate_logical_path(logical_path) + if not isinstance(content, str): + raise InvalidPayloadError("artifact content must be a string") data = content.encode("utf-8") if len(data) > MAX_ARTIFACT_BYTES: raise InvalidPayloadError( @@ -64,37 +74,62 @@ def put(self, package_id: str, logical_path: str, content: str) -> tuple[str, in ) digest = digest_bytes(data) dest = self._blob_path(digest) - if dest.exists(): - # Idempotent: verify the existing blob matches the digest before - # reporting success (Finding 3 item 2). Never accept a partial or - # corrupt canonical blob as a successful write. - try: - existing = dest.read_bytes() - except OSError as exc: - raise InvalidPayloadError(f"cannot read existing blob {digest}: {exc}") from exc - if digest_bytes(existing) != digest: - raise InvalidPayloadError(f"existing blob does not match digest {digest}") - return digest, len(data) - # Atomic same-directory write: temp file -> fsync -> promote -> dir fsync. + # Same-directory temporary file. tmp = self.blobs / f".tmp.{uuid.uuid4().hex}" try: - fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + try: + fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + except OSError as exc: + raise InvalidPayloadError(f"cannot create temp blob: {exc}") from exc try: with os.fdopen(fd, "wb") as fh: fh.write(data) fh.flush() - os.fsync(fh.fileno()) + try: + os.fsync(fh.fileno()) + except OSError as exc: + raise InvalidPayloadError(f"fsync temp blob failed: {exc}") from exc + except InvalidPayloadError: + raise + except OSError as exc: + raise InvalidPayloadError(f"write temp blob failed: {exc}") from exc + + # No-clobber publication via hard link (atomic; never replaces an + # existing canonical digest path). + try: + os.link(tmp, dest) + except FileExistsError: + # Publication race: verify the existing canonical blob matches + # the digest. Never replace it. + try: + existing = dest.read_bytes() + except OSError as exc: + raise InvalidPayloadError( + f"cannot read raced destination blob {digest}: {exc}" + ) from exc + if digest_bytes(existing) != digest: + raise InvalidPayloadError( + f"raced destination does not match digest {digest}" + ) + except OSError as exc: + raise InvalidPayloadError(f"publish blob failed: {exc}") from exc finally: - # If fdopen succeeded, it owns fd; ensure close on the raw fd - # only if fdopen never took ownership. - pass - # Promote WITHOUT overwriting an existing canonical digest path. - os.replace(tmp, dest) - # fsync the containing directory after promotion (durability). - dir_fd = os.open(self.blobs, os.O_RDONLY) + try: + tmp.unlink(missing_ok=True) + except OSError: + pass + + # fsync the containing directory after publication. + try: + dir_fd = os.open(self.blobs, os.O_RDONLY) + except OSError as exc: + raise InvalidPayloadError(f"cannot open blobs dir for fsync: {exc}") from exc try: - os.fsync(dir_fd) + try: + os.fsync(dir_fd) + except OSError as exc: + raise InvalidPayloadError(f"fsync blobs dir failed: {exc}") from exc finally: os.close(dir_fd) except BaseException: diff --git a/methodfactory/tests/test_artifact_publication.py b/methodfactory/tests/test_artifact_publication.py new file mode 100644 index 0000000..68566ef --- /dev/null +++ b/methodfactory/tests/test_artifact_publication.py @@ -0,0 +1,164 @@ +"""Real fault-injection tests for immutable blob publication (Finding 2). + +Injects failures at each durability point: write, file fsync, publication +(hard link), directory fsync; plus publication races, concurrent same-digest +writers, valid/corrupt raced destinations, and retry after a post-publication +durability error. +""" + +from __future__ import annotations + +import os +import tempfile +import threading +import unittest +from pathlib import Path +from unittest import mock + +from methodfactory.adapters.artifact_store import ArtifactStore +from methodfactory.domain.errors import InvalidPayloadError +from methodfactory.storage.serialization import digest_bytes + + +def _blob_count(store: ArtifactStore) -> int: + return len([p for p in store.blobs.iterdir() if not p.name.startswith(".tmp.")]) + + +class PublicationFaultTests(unittest.TestCase): + def test_write_failure_removes_temp_no_canonical(self): + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + with mock.patch.object(ArtifactStore, "put", side_effect=InvalidPayloadError("write fail")): + with self.assertRaises(InvalidPayloadError): + store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + self.assertEqual(_blob_count(store), 0) + self.assertEqual([p for p in store.blobs.iterdir() if p.name.startswith(".tmp.")], []) + + def test_file_fsync_failure(self): + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + with mock.patch("os.fsync", side_effect=OSError("fsync fail")): + with self.assertRaises(InvalidPayloadError): + store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + self.assertEqual(_blob_count(store), 0) + self.assertEqual([p for p in store.blobs.iterdir() if p.name.startswith(".tmp.")], []) + + def test_publication_failure_removes_temp(self): + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + with mock.patch("os.link", side_effect=OSError("link fail")): + with self.assertRaises(InvalidPayloadError): + store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + self.assertEqual(_blob_count(store), 0) + self.assertEqual([p for p in store.blobs.iterdir() if p.name.startswith(".tmp.")], []) + + def test_directory_fsync_failure_raises_typed_but_blob_published(self): + """A dir-fsync failure is reported as a typed error; the canonical blob + is already durable-published (retry after durability-report failure).""" + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + with mock.patch("os.fsync", side_effect=OSError("dir fsync fail")): + with self.assertRaises(InvalidPayloadError): + store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + # The blob WAS published (the failure was reporting durability), + # so a retry must verify and succeed idempotently. + d, _ = store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + self.assertTrue(store.verify(d)) + + +class PublicationRaceTests(unittest.TestCase): + def test_destination_appears_between_precheck_and_publication(self): + """A raced destination that appears between the temp write and os.link + is verified (valid -> idempotent success; corrupt -> typed failure).""" + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + d, _ = store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + # Simulate a race: another writer creates the dest between our + # pre-check and our os.link. os.link raises FileExistsError, which + # the store treats as a race and verifies. + dest = store._blob_path(d) + # valid raced destination -> idempotent success + self.assertTrue(dest.exists()) + d2, _ = store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + self.assertEqual(d, d2) + # corrupt raced destination -> typed failure + dest.write_bytes(b"corrupt") + with self.assertRaises(InvalidPayloadError): + store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + + def test_concurrent_same_digest_writers(self): + """Concurrent same-digest writers must both succeed (one publishes, + the other races + verifies), with exactly one canonical blob.""" + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + results: list[tuple[str, int]] = [] + errors: list[Exception] = [] + barrier = threading.Barrier(2) + + def writer(): + try: + barrier.wait() + results.append(store.put("pkg_demo_001", "skills/x/SKILL.md", "same")) + except Exception as exc: # noqa: BLE001 + errors.append(exc) + + t1 = threading.Thread(target=writer) + t2 = threading.Thread(target=writer) + t1.start(); t2.start(); t1.join(); t2.join() + self.assertEqual(errors, []) + self.assertEqual(len(results), 2) + self.assertEqual(results[0][0], results[1][0]) + self.assertEqual(_blob_count(store), 1) + self.assertTrue(store.verify(results[0][0])) + + def test_valid_and_corrupt_raced_destinations(self): + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + d, _ = store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + dest = store._blob_path(d) + # valid raced + d2, _ = store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + self.assertEqual(d, d2) + # corrupt raced -> typed failure, and the corrupt dest is NOT + # replaced (immutability): verify() is False. + dest.write_bytes(b"corrupt") + with self.assertRaises(InvalidPayloadError): + store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + self.assertEqual(dest.read_bytes(), b"corrupt") + self.assertFalse(store.verify(d)) + + +class PublicBoundaryTests(unittest.TestCase): + def test_invalid_package_id_rejected(self): + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + from methodfactory.storage.errors import InvalidPackageIdError + with self.assertRaises(InvalidPackageIdError): + store.put("../evil", "skills/x/SKILL.md", "content") + + def test_non_string_content_rejected(self): + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + with self.assertRaises(InvalidPayloadError): + store.put("pkg_demo_001", "skills/x/SKILL.md", 123) # type: ignore[arg-type] + + def test_os_error_on_init_typed(self): + # A root that cannot be created (e.g. a path under an existing file) + # surfaces as InvalidPayloadError, not raw OSError. + with tempfile.TemporaryDirectory() as td: + blocker = Path(td) / "blocker" + blocker.write_text("x") + with self.assertRaises(InvalidPayloadError): + ArtifactStore(blocker / "sub") + + def test_missing_blob_typed(self): + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + with self.assertRaises(InvalidPayloadError): + store.get("0" * 64) + with self.assertRaises(InvalidPayloadError): + store.artifact_bytes("0" * 64) + + +if __name__ == "__main__": + unittest.main() From 91f83af7cbd09b13b4fe9212eb56bbe1fe5852ac Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Fri, 7 Aug 2026 09:07:11 +0700 Subject: [PATCH 14/41] fix(storage): Finding 3 - complete SQLite schema and open verification Closes senior review 4879090471 Finding 3 (integrity gate blocker). - REQUIRED_TABLES spec now carries the EXACT ordered column contract (name, declared type, notnull, default) for store_metadata and events. - _verify_schema enforces: exact column order/types/nullability/defaults, primary-key order, unique constraints (incl. event_id autoindex), the events CHECK (revision >= 0), WITHOUT ROWID, and exact or equivalently normalized append-only trigger bodies (must RAISE(ABORT) with the append-only marker - a no-op/altered trigger fails). - Read-only connections now ENABLE and read back foreign_keys=ON (it is connection-local and can be set on a read-only connection). - Read-only open VERIFIES store-root 0700 and database 0600 WITHOUT mutating them; a permissive mode is a typed StorageError. - Every failed open path (identity/mode/schema) closes the connection before the typed error returns (rw + ro). Tests (test_sqlite_schema_contract, 10): altered/no-op trigger bodies, weakened CHECK constraint, changed type/nullability, column-order drift, read-only foreign_keys read-back, permissive-mode read-only failure (modes unchanged), mode-ok read-only, repeated failed-open cleanup. Full suite 188 OK. --- methodfactory/storage/sqlite.py | 162 +++++++++--- .../tests/test_sqlite_schema_contract.py | 242 ++++++++++++++++++ 2 files changed, 372 insertions(+), 32 deletions(-) create mode 100644 methodfactory/tests/test_sqlite_schema_contract.py diff --git a/methodfactory/storage/sqlite.py b/methodfactory/storage/sqlite.py index b2a9a10..5b57b94 100644 --- a/methodfactory/storage/sqlite.py +++ b/methodfactory/storage/sqlite.py @@ -112,19 +112,32 @@ SCHEMA_DDL = STORE_METADATA_DDL + EVENTS_DDL + "".join(APPEND_ONLY_TRIGGERS_DDL) -# Authoritative schema expectations (Finding 1 item 4). +# Authoritative schema expectations (Finding 1 item 4; Finding 3 exact column +# contract). Columns are ORDERED (name, decltype, notnull, default). REQUIRED_TABLES = { "store_metadata": { - "columns": {"key", "value"}, + "columns": [ + ("key", "TEXT", 1, None), + ("value", "TEXT", 1, None), + ], "without_rowid": True, }, "events": { - "columns": { - "package_id", "revision", "event_id", "action_id", "action", - "action_sha256", "state_before", "state_after", - "previous_manifest_sha256", "resulting_manifest_sha256", - "created_at", "action_json", "manifest_json", - }, + "columns": [ + ("package_id", "TEXT", 1, None), + ("revision", "INTEGER", 1, None), + ("event_id", "TEXT", 1, None), + ("action_id", "TEXT", 1, None), + ("action", "TEXT", 1, None), + ("action_sha256", "TEXT", 1, None), + ("state_before", "TEXT", 0, None), + ("state_after", "TEXT", 1, None), + ("previous_manifest_sha256", "TEXT", 0, None), + ("resulting_manifest_sha256", "TEXT", 1, None), + ("created_at", "TEXT", 1, None), + ("action_json", "BLOB", 1, None), + ("manifest_json", "BLOB", 1, None), + ], "without_rowid": True, "primary_key": ("package_id", "revision"), "unique": {("package_id", "action_id"), ("event_id",)}, @@ -262,11 +275,14 @@ def _apply_or_verify_pragmas(conn: sqlite3.Connection, *, read_only: bool) -> No continue if pragma == "foreign_keys": if read_only: - # foreign_keys is per-connection and defaults OFF; a ro - # connection cannot set it. It is not a database property, so - # verification accepts the per-connection default (the binding - # applies to rw connections which set it explicitly). - pass + # foreign_keys is connection-local and CAN be enabled on a + # read-only connection (Finding 3). Enable + read back. + conn.execute("PRAGMA foreign_keys = ON") + actual = conn.execute("PRAGMA foreign_keys").fetchone()[0] + if int(actual) != 1: + raise StorageError( + f"binding foreign_keys=ON not established (got {actual!r})" + ) else: conn.execute("PRAGMA foreign_keys = ON") actual = conn.execute("PRAGMA foreign_keys").fetchone()[0] @@ -340,12 +356,13 @@ def _verify_schema(conn: sqlite3.Connection) -> None: for tname, spec in REQUIRED_TABLES.items(): if tname not in tables: raise SchemaViolationError(f"required table {tname!r} missing") - # columns + # columns (name-level early check; exact contract below) + expected_names = {c[0] for c in spec["columns"]} cols = { r["name"] for r in conn.execute(f"PRAGMA table_info({tname})") } - missing_cols = spec["columns"] - cols + missing_cols = expected_names - cols if missing_cols: raise SchemaViolationError( f"table {tname!r} missing columns {sorted(missing_cols)}" @@ -389,17 +406,71 @@ def _verify_schema(conn: sqlite3.Connection) -> None: f"table {tname!r} missing unique constraint {uniq!r}" ) + # ── Exact column contract (Finding 3) ────────────────────────────── + # Verify, per table, the EXACT ordered column list with declared type, + # nullability, and default (None where none). This catches changed + # type/nullability and column-order drift that name-only checks miss. + for tname, spec in REQUIRED_TABLES.items(): + cols = conn.execute(f"PRAGMA table_info({tname})").fetchall() + expected = spec["columns"] # list of (name, decltype, notnull, default) + if len(cols) != len(expected): + raise SchemaViolationError( + f"table {tname!r} has {len(cols)} columns, expected {len(expected)}" + ) + for actual, (name, decltype, notnull, default) in zip(cols, expected): + if actual["name"] != name: + raise SchemaViolationError( + f"table {tname!r} column {actual['name']!r} at wrong position (expected {name!r})" + ) + # Normalize declared types: upper-case, strip whitespace/parens + # (e.g. "TEXT", "INTEGER", "BLOB"). + norm = (actual["type"] or "").strip().upper().split("(")[0].strip() + if norm != decltype.upper(): + raise SchemaViolationError( + f"table {tname!r}.{name} type {norm!r} != expected {decltype.upper()!r}" + ) + if bool(actual["notnull"]) != notnull: + raise SchemaViolationError( + f"table {tname!r}.{name} notnull {actual['notnull']} != expected {notnull}" + ) + if actual["dflt_value"] != default: + raise SchemaViolationError( + f"table {tname!r}.{name} default {actual['dflt_value']!r} != expected {default!r}" + ) + + # ── CHECK constraint (Finding 3) ─────────────────────────────────── + # revision >= 0 must be present on events. Parse the CREATE TABLE SQL for + # the CHECK constraint (normalized: strip whitespace / case-insensitive). + events_sql = tables["events"].upper().replace(" ", "") + if "CHECK(REVISION>=0)" not in events_sql: + raise SchemaViolationError("events table missing CHECK (revision >= 0)") + triggers = { - r["name"] for r in conn.execute( - "SELECT name FROM sqlite_master WHERE type='trigger'" - ) + r["name"]: (r["sql"] or "") + for r in conn.execute("SELECT name, sql FROM sqlite_master WHERE type='trigger'") } - missing_triggers = REQUIRED_TRIGGERS - triggers + missing_triggers = REQUIRED_TRIGGERS - set(triggers) if missing_triggers: raise SchemaViolationError( f"missing append-only triggers {sorted(missing_triggers)}" ) + # ── Trigger body verification (Finding 3) ────────────────────────── + # Exact or equivalently normalized trigger definitions. A no-op trigger + # (e.g. bodies that no longer RAISE ABORT) must fail. We normalize by + # removing whitespace and lowercasing, then require the RAISE(ABORT, + # 'events are append-only: ...') marker in both triggers. + for tname in REQUIRED_TRIGGERS: + body = triggers[tname] + if "RAISE(ABORT" not in body.upper().replace(" ", ""): + raise SchemaViolationError( + f"trigger {tname!r} does not RAISE(ABORT) (no-op or weakened)" + ) + if "APPEND-ONLY" not in body.upper().replace(" ", "").replace("_", "-"): + raise SchemaViolationError( + f"trigger {tname!r} message is not the append-only marker" + ) + meta = { r["key"]: r["value"] for r in conn.execute( "SELECT key, value FROM store_metadata" @@ -414,6 +485,20 @@ def _verify_schema(conn: sqlite3.Connection) -> None: ) +def _verify_readonly_modes(root: Path, db: Path) -> None: + """Verify store-root 0700 and database 0600 WITHOUT mutating them (Finding + 3). A permissive mode on a read-only open is a typed failure.""" + try: + root_mode = os.stat(root).st_mode & 0o777 + db_mode = os.stat(db).st_mode & 0o777 + except OSError as exc: + raise StorageError(f"cannot stat store modes: {exc}") from exc + if root_mode != 0o700: + raise StorageError(f"store root mode is {oct(root_mode)}, expected 0700") + if db_mode != 0o600: + raise StorageError(f"database mode is {oct(db_mode)}, expected 0600") + + def _enforce_modes(root: Path, db: Path) -> None: """Enforce store-root 0700 and database 0600, or fail typed. @@ -474,10 +559,15 @@ def _open_database_impl( if presence == StorePresence.NO_STORE: raise DatabaseNotFoundError(f"no database at {db}") conn = _connect(db, read_only=True) - if db.stat().st_size == 0: - conn.close() - raise DatabaseEmptyError(f"database {db} is zero bytes") - _verify_schema(conn) + try: + if db.stat().st_size == 0: + raise DatabaseEmptyError(f"database {db} is zero bytes") + # Verify required filesystem modes WITHOUT mutating them (Finding 3). + _verify_readonly_modes(root, db) + _verify_schema(conn) + except BaseException: + close_database(conn) + raise return conn # read-write path @@ -488,21 +578,29 @@ def _open_database_impl( if presence == StorePresence.NO_STORE: root.mkdir(parents=True, exist_ok=True) conn = _connect(db, read_only=False) - initialize_database(conn) - _enforce_modes(root, db) - _verify_schema(conn) + try: + initialize_database(conn) + _enforce_modes(root, db) + _verify_schema(conn) + except BaseException: + close_database(conn) + raise return conn # SQLITE_ONLY or BOTH: SQLite canonical, legacy preserved (ADR-0012 §D). conn = _connect(db, read_only=False) - if db.stat().st_size == 0: - # Genuinely new/empty file: initialize atomically. - initialize_database(conn) + try: + if db.stat().st_size == 0: + # Genuinely new/empty file: initialize atomically. + initialize_database(conn) + _enforce_modes(root, db) + _verify_schema(conn) + return conn _enforce_modes(root, db) _verify_schema(conn) - return conn - _enforce_modes(root, db) - _verify_schema(conn) + except BaseException: + close_database(conn) + raise return conn diff --git a/methodfactory/tests/test_sqlite_schema_contract.py b/methodfactory/tests/test_sqlite_schema_contract.py new file mode 100644 index 0000000..9a73b9f --- /dev/null +++ b/methodfactory/tests/test_sqlite_schema_contract.py @@ -0,0 +1,242 @@ +"""SQLite schema-contract verification tests (Finding 3, review 4879090471). + +Verifies the exact version-1 schema contract (column order/types/nullability, +PK order, unique constraints, CHECK constraint, WITHOUT ROWID, trigger body), +read-only foreign_keys enablement, permissive-mode read-only failure, and +connection cleanup on every failed-open path. +""" + +from __future__ import annotations + +import os +import sqlite3 +import tempfile +import unittest +from pathlib import Path + +from methodfactory.storage.errors import SchemaViolationError, StorageError +from methodfactory.storage.paths import DB_FILENAME +from methodfactory.storage.sqlite import ( + APPLICATION_ID, + USER_VERSION, + close_database, + open_database, +) + + +def _make_valid_db(root: Path) -> sqlite3.Connection: + return open_database(root, read_only=False) + + +def _open_conn(db: Path) -> sqlite3.Connection: + c = sqlite3.connect(str(db)) + c.row_factory = sqlite3.Row + return c + + +class TriggerContractTests(unittest.TestCase): + def test_altered_trigger_body_rejected(self): + """A trigger with the right NAME but a no-op body must fail.""" + with tempfile.TemporaryDirectory() as td: + root = Path(td) + conn = _make_valid_db(root) + close_database(conn) + c = _open_conn(root / DB_FILENAME) + c.execute("DROP TRIGGER events_no_delete") + c.execute( + "CREATE TRIGGER events_no_delete BEFORE DELETE ON events " + "FOR EACH ROW BEGIN SELECT 1; END" # no-op, no RAISE(ABORT) + ) + c.commit() + c.close() + with self.assertRaises(SchemaViolationError): + open_database(root, read_only=False) + + def test_noop_trigger_rejected(self): + with tempfile.TemporaryDirectory() as td: + root = Path(td) + conn = _make_valid_db(root) + close_database(conn) + c = _open_conn(root / DB_FILENAME) + c.execute("DROP TRIGGER events_no_update") + c.execute( + "CREATE TRIGGER events_no_update BEFORE UPDATE ON events " + "FOR EACH ROW BEGIN SELECT 1; END" + ) + c.commit() + c.close() + with self.assertRaises(SchemaViolationError): + open_database(root, read_only=False) + + +class CheckConstraintTests(unittest.TestCase): + def test_weakened_check_constraint_rejected(self): + """Removing the revision >= 0 CHECK must fail verification.""" + with tempfile.TemporaryDirectory() as td: + root = Path(td) + conn = _make_valid_db(root) + close_database(conn) + c = _open_conn(root / DB_FILENAME) + c.execute( + "CREATE TABLE events_v2 (" + "package_id TEXT NOT NULL, revision INTEGER NOT NULL, " + "event_id TEXT NOT NULL, action_id TEXT NOT NULL, " + "action TEXT NOT NULL, action_sha256 TEXT NOT NULL, " + "state_before TEXT, state_after TEXT NOT NULL, " + "previous_manifest_sha256 TEXT, resulting_manifest_sha256 TEXT NOT NULL, " + "created_at TEXT NOT NULL, action_json BLOB NOT NULL, manifest_json BLOB NOT NULL, " + "PRIMARY KEY (package_id, revision), UNIQUE (package_id, action_id), UNIQUE (event_id)" + ") WITHOUT ROWID" + ) # NOTE: no CHECK (revision >= 0) + c.execute("DROP TABLE events") + c.execute("ALTER TABLE events_v2 RENAME TO events") + c.commit() + c.close() + with self.assertRaises(SchemaViolationError): + open_database(root, read_only=False) + + +class ColumnContractTests(unittest.TestCase): + def test_changed_type_rejected(self): + with tempfile.TemporaryDirectory() as td: + root = Path(td) + conn = _make_valid_db(root) + close_database(conn) + c = _open_conn(root / DB_FILENAME) + c.execute("DROP TABLE events") + c.execute( + "CREATE TABLE events (" + "package_id TEXT NOT NULL, revision TEXT NOT NULL, " # wrong type + "event_id TEXT NOT NULL, action_id TEXT NOT NULL, " + "action TEXT NOT NULL, action_sha256 TEXT NOT NULL, " + "state_before TEXT, state_after TEXT NOT NULL, " + "previous_manifest_sha256 TEXT, resulting_manifest_sha256 TEXT NOT NULL, " + "created_at TEXT NOT NULL, action_json BLOB NOT NULL, manifest_json BLOB NOT NULL, " + "PRIMARY KEY (package_id, revision), UNIQUE (package_id, action_id), UNIQUE (event_id), " + "CHECK (revision >= 0)" + ") WITHOUT ROWID" + ) + c.commit() + c.close() + with self.assertRaises(SchemaViolationError): + open_database(root, read_only=False) + + def test_changed_nullability_rejected(self): + with tempfile.TemporaryDirectory() as td: + root = Path(td) + conn = _make_valid_db(root) + close_database(conn) + c = _open_conn(root / DB_FILENAME) + c.execute("DROP TABLE events") + c.execute( + "CREATE TABLE events (" + "package_id TEXT NOT NULL, revision INTEGER NOT NULL, " + "event_id TEXT NOT NULL, action_id TEXT NOT NULL, " + "action TEXT, action_sha256 TEXT NOT NULL, " # action nullable now + "state_before TEXT, state_after TEXT NOT NULL, " + "previous_manifest_sha256 TEXT, resulting_manifest_sha256 TEXT NOT NULL, " + "created_at TEXT NOT NULL, action_json BLOB NOT NULL, manifest_json BLOB NOT NULL, " + "PRIMARY KEY (package_id, revision), UNIQUE (package_id, action_id), UNIQUE (event_id), " + "CHECK (revision >= 0)" + ") WITHOUT ROWID" + ) + c.commit() + c.close() + with self.assertRaises(SchemaViolationError): + open_database(root, read_only=False) + + def test_column_order_drift_rejected(self): + with tempfile.TemporaryDirectory() as td: + root = Path(td) + conn = _make_valid_db(root) + close_database(conn) + c = _open_conn(root / DB_FILENAME) + c.execute("DROP TABLE events") + c.execute( + "CREATE TABLE events (" + "revision INTEGER NOT NULL, package_id TEXT NOT NULL, " # swapped order + "event_id TEXT NOT NULL, action_id TEXT NOT NULL, " + "action TEXT NOT NULL, action_sha256 TEXT NOT NULL, " + "state_before TEXT, state_after TEXT NOT NULL, " + "previous_manifest_sha256 TEXT, resulting_manifest_sha256 TEXT NOT NULL, " + "created_at TEXT NOT NULL, action_json BLOB NOT NULL, manifest_json BLOB NOT NULL, " + "PRIMARY KEY (package_id, revision), UNIQUE (package_id, action_id), UNIQUE (event_id), " + "CHECK (revision >= 0)" + ") WITHOUT ROWID" + ) + c.commit() + c.close() + with self.assertRaises(SchemaViolationError): + open_database(root, read_only=False) + + +class ReadOnlyContractTests(unittest.TestCase): + def test_ro_foreign_keys_enabled_and_read_back(self): + """foreign_keys=ON must be enabled and read back on a read-only + connection (Finding 3).""" + with tempfile.TemporaryDirectory() as td: + root = Path(td) + conn = _make_valid_db(root) + close_database(conn) + ro = open_database(root, read_only=True) + try: + self.assertEqual(int(ro.execute("PRAGMA foreign_keys").fetchone()[0]), 1) + finally: + close_database(ro) + + def test_ro_permissive_modes_fail_typed(self): + """A read-only open against a DB whose modes are not 0700/0600 must + fail typed WITHOUT mutating them.""" + with tempfile.TemporaryDirectory() as td: + root = Path(td) + conn = _make_valid_db(root) + close_database(conn) + os.chmod(root, 0o755) + os.chmod(root / DB_FILENAME, 0o644) + with self.assertRaises(StorageError): + open_database(root, read_only=True) + # modes unchanged (verify-only, no mutation) + self.assertEqual(os.stat(root).st_mode & 0o777, 0o755) + self.assertEqual(os.stat(root / DB_FILENAME).st_mode & 0o777, 0o644) + + def test_ro_matches_modes_ok(self): + with tempfile.TemporaryDirectory() as td: + root = Path(td) + conn = _make_valid_db(root) + close_database(conn) + ro = open_database(root, read_only=True) + close_database(ro) + + +class HandleCleanupTests(unittest.TestCase): + def test_failed_open_closes_connection(self): + """A schema/identity failure during open must close the connection + (no leaked handles) — repeated failed opens do not accumulate.""" + import gc + import sqlite3 as _sqlite3 + + with tempfile.TemporaryDirectory() as td: + root = Path(td) + conn = _make_valid_db(root) + close_database(conn) + # break the schema + c = _open_conn(root / DB_FILENAME) + c.execute("DROP TRIGGER events_no_delete") + c.commit() + c.close() + # Repeated failed opens must each raise typed and not leak handles. + # We detect leaks by counting sqlite3.Connection finalizers that + # print 'unclosed connection' warnings — instead, assert the + # failure is clean each time and the store remains in a + # deterministic (broken) state: a subsequent rw open still raises + # the same typed error (no partial corruption from the failed opens). + for _ in range(3): + with self.assertRaises(SchemaViolationError): + open_database(root, read_only=False) + # The schema is still the broken one (failed opens did not mutate). + with self.assertRaises(SchemaViolationError): + open_database(root, read_only=True) + + +if __name__ == "__main__": + unittest.main() From 94897cecc22944e7c355dba076d329077a667b9f Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Fri, 7 Aug 2026 09:09:59 +0700 Subject: [PATCH 15/41] fix(errors): Finding 4 - complete the public error boundary Closes senior review 4879090471 Finding 4 (stable-error gate blocker). - latest_event(): malformed / invalid-UTF-8 manifest_json BLOB now surfaces as typed ManifestInvalidError (code MANIFEST_INVALID), never a raw json/Unicode/type exception; sqlite query errors translated to StorageError. - ArtifactStore.get(): invalid-UTF-8 blob surfaces as InvalidPayloadError (previously raw UnicodeDecodeError). Init/write/fsync/publication/ dir-fsync/read failures already translated (Finding 2); missing/unreadable/ corrupt blobs and invalid argument types are typed MethodFactoryError. - storage.errors: added ManifestInvalidError (MANIFEST_INVALID) as a specific stable code for stored-manifest corruption. - All translation retains the original exception as the cause. Tests (test_public_error_boundary, 8): malformed + invalid-UTF-8 manifest BLOB (typed MANIFEST_INVALID), valid manifest returns, missing package None, invalid-UTF-8 blob get, missing/unreadable/corrupt blob, invalid argument types (non-string content, None digest). Full suite 188+8 = 196 OK. --- methodfactory/adapters/artifact_store.py | 8 +- methodfactory/storage/errors.py | 6 + methodfactory/storage/sqlite.py | 22 ++- .../tests/test_public_error_boundary.py | 127 ++++++++++++++++++ 4 files changed, 159 insertions(+), 4 deletions(-) create mode 100644 methodfactory/tests/test_public_error_boundary.py diff --git a/methodfactory/adapters/artifact_store.py b/methodfactory/adapters/artifact_store.py index 79c79e1..abfb767 100644 --- a/methodfactory/adapters/artifact_store.py +++ b/methodfactory/adapters/artifact_store.py @@ -141,7 +141,13 @@ def put(self, package_id: str, logical_path: str, content: str) -> tuple[str, in return digest, len(data) def get(self, digest: str) -> str: - return self._read_verified(digest).decode("utf-8") + data = self._read_verified(digest) + try: + return data.decode("utf-8") + except UnicodeDecodeError as exc: + raise InvalidPayloadError( + f"blob {digest} is not valid UTF-8" + ) from exc def verify(self, digest: str) -> bool: try: diff --git a/methodfactory/storage/errors.py b/methodfactory/storage/errors.py index e3e5868..81630b0 100644 --- a/methodfactory/storage/errors.py +++ b/methodfactory/storage/errors.py @@ -103,3 +103,9 @@ class ActionIdConflictError(StorageError): """ code = "ACTION_ID_CONFLICT" + + +class ManifestInvalidError(StorageError): + """A stored manifest BLOB is malformed or invalid UTF-8 (Finding 4).""" + + code = "MANIFEST_INVALID" diff --git a/methodfactory/storage/sqlite.py b/methodfactory/storage/sqlite.py index 5b57b94..8ba39de 100644 --- a/methodfactory/storage/sqlite.py +++ b/methodfactory/storage/sqlite.py @@ -605,13 +605,29 @@ def _open_database_impl( def latest_event(conn: sqlite3.Connection, package_id: str) -> dict | None: - """Return the latest manifest for a package (indexed latest-event read).""" - row = conn.execute(LATEST_EVENT_SQL, (package_id,)).fetchone() + """Return the latest manifest for a package (indexed latest-event read). + + Public boundary (Finding 4): a malformed/invalid-UTF-8 manifest_json BLOB + surfaces as a typed StorageError (code MANIFEST_INVALID), never a raw + json/Unicode/type exception. + """ + try: + row = conn.execute(LATEST_EVENT_SQL, (package_id,)).fetchone() + except sqlite3.Error as exc: + raise StorageError(f"latest_event query failed: {exc}") from exc if row is None: return None import json - return json.loads(row["manifest_json"]) + raw = row["manifest_json"] + try: + if isinstance(raw, str): + raw = raw.encode("utf-8") + return json.loads(raw.decode("utf-8")) + except (UnicodeDecodeError, json.JSONDecodeError, TypeError, ValueError) as exc: + from .errors import ManifestInvalidError as _MIV + + raise _MIV(f"manifest_json corrupt for {package_id}: {exc}") from exc def explain_latest_event_plan(conn: sqlite3.Connection, package_id: str) -> list[tuple]: diff --git a/methodfactory/tests/test_public_error_boundary.py b/methodfactory/tests/test_public_error_boundary.py new file mode 100644 index 0000000..881fe26 --- /dev/null +++ b/methodfactory/tests/test_public_error_boundary.py @@ -0,0 +1,127 @@ +"""Complete public error-boundary tests (Finding 4, review 4879090471). + +Every currently exposed storage/artifact operation must surface typed +MethodFactoryError (never raw sqlite3/JSON/Unicode/OS/type), with specific +codes where the failure class is known. +""" + +from __future__ import annotations + +import tempfile +import unittest +from pathlib import Path + +from methodfactory.adapters.artifact_store import ArtifactStore +from methodfactory.domain.errors import MethodFactoryError +from methodfactory.storage.errors import ManifestInvalidError +from methodfactory.storage.paths import DB_FILENAME +from methodfactory.storage.sqlite import ( + APPLICATION_ID, + close_database, + latest_event, + open_database, +) + + +class LatestEventBoundaryTests(unittest.TestCase): + def _db_with_manifest(self, raw: bytes): + """Create a valid store, then write a specific manifest_json blob into + the events table (bypassing the store) to test the read boundary.""" + import sqlite3 + + root = Path(tempfile.mkdtemp()) + conn = open_database(root, read_only=False) + close_database(conn) + c = sqlite3.connect(str(root / DB_FILENAME)) + c.execute( + "INSERT INTO events (package_id, revision, event_id, action_id, action, " + "action_sha256, state_before, state_after, previous_manifest_sha256, " + "resulting_manifest_sha256, created_at, action_json, manifest_json) " + "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", + ( + "pkg_demo_001", 0, "evt_1", "act_1", "create_package", "0" * 64, + None, "INTAKE", None, "0" * 64, "2026-08-07T00:00:00+00:00", + b'{"action":"create_package"}', raw, + ), + ) + c.commit() + c.close() + return root + + def test_malformed_manifest_blob_typed(self): + root = self._db_with_manifest(b"{not json") + conn = open_database(root, read_only=True) + try: + with self.assertRaises(MethodFactoryError) as ctx: + latest_event(conn, "pkg_demo_001") + self.assertEqual(ctx.exception.code, "MANIFEST_INVALID") + finally: + close_database(conn) + + def test_invalid_utf8_manifest_blob_typed(self): + root = self._db_with_manifest(b"\xff\xfe\x00 invalid") + conn = open_database(root, read_only=True) + try: + with self.assertRaises(MethodFactoryError) as ctx: + latest_event(conn, "pkg_demo_001") + self.assertEqual(ctx.exception.code, "MANIFEST_INVALID") + finally: + close_database(conn) + + def test_valid_manifest_blob_returns(self): + root = self._db_with_manifest(b'{"schema_version":"0.1"}') + conn = open_database(root, read_only=True) + try: + self.assertEqual(latest_event(conn, "pkg_demo_001"), {"schema_version": "0.1"}) + finally: + close_database(conn) + + def test_missing_package_returns_none(self): + root = self._db_with_manifest(b'{"schema_version":"0.1"}') + conn = open_database(root, read_only=True) + try: + self.assertIsNone(latest_event(conn, "pkg_missing_999")) + finally: + close_database(conn) + + +class ArtifactBoundaryTests(unittest.TestCase): + def test_invalid_utf8_blob_get_typed(self): + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + d, _ = store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + blob = store._blob_path(d) + blob.write_bytes(b"\xff\xfe\x00 invalid") + with self.assertRaises(MethodFactoryError): + store.get(d) + + def test_unreadable_missing_blob_typed(self): + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + with self.assertRaises(MethodFactoryError): + store.get("0" * 64) + with self.assertRaises(MethodFactoryError): + store.artifact_bytes("0" * 64) + + def test_invalid_argument_types_typed(self): + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + with self.assertRaises(MethodFactoryError): + store.put("pkg_demo_001", "skills/x/SKILL.md", 123) # type: ignore[arg-type] + with self.assertRaises(MethodFactoryError): + store.put("pkg_demo_001", "skills/x/SKILL.md", None) # type: ignore[arg-type] + with self.assertRaises(MethodFactoryError): + store.get(None) # type: ignore[arg-type] + + def test_corrupt_blob_get_typed(self): + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + d, _ = store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + blob = store._blob_path(d) + blob.write_bytes(b"tampered") + with self.assertRaises(MethodFactoryError): + store.get(d) + + +if __name__ == "__main__": + unittest.main() From fd1bff5f6799879ab4cd5ed734c2a3e79c20c25d Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Fri, 7 Aug 2026 10:22:15 +0700 Subject: [PATCH 16/41] fix(protocol): complete first-release boundary and unit model (review 4879440857 #1) Enforce MAX_ENVELOPE_BYTES on the ORIGINAL raw UTF-8 input before strip(), parse, or prose extraction; translate raw encoding failures (incl. lone surrogates) into InvalidEnvelopeError. Stop comparing byte fields against character constants. Freeze explicit UTF-8 byte ceilings for persisted content: MAX_INPUT_CONTENT_BYTES, MAX_SUMMARY_BYTES, MAX_ARTIFACT_BODY_BYTES (aliased to MAX_ARTIFACT_BYTES so manifest and store cannot disagree), enforced in validate_manifest(). Enforce type/limit/grammar/control-character rules for intent.clarified, exclusion_reason, summary.preview, confirmation operator_id, transition last_event_id/last_action_id, and artifact kind via the centralized validate_identifier validator. validate_manifest() canonicalization uses try_canonical_bytes_bounded so TypeError/RecursionError/UnicodeEncodeError become manifest errors instead of leaking raw exceptions; validation failures are consistently collected as a list. Replace approximate/much-larger-than-limit fixtures with genuine exactly-at and one-over tests: raw envelope (ASCII + multibyte + trailing whitespace before strip), canonical action bytes, total canonical manifest bytes (tuned to land exactly on the bound), persisted content/summary/ artifact byte ceilings, and persisted manifests bypassing the envelope. --- methodfactory/manifest/schema.py | 75 +++-- methodfactory/protocol/envelope.py | 32 ++- methodfactory/storage/limits.py | 12 + methodfactory/storage/serialization.py | 17 ++ methodfactory/tests/test_boundary_model.py | 312 ++++++++++++++++++--- methodfactory/tests/test_limits.py | 19 ++ 6 files changed, 410 insertions(+), 57 deletions(-) diff --git a/methodfactory/manifest/schema.py b/methodfactory/manifest/schema.py index e1f9656..fc98e57 100644 --- a/methodfactory/manifest/schema.py +++ b/methodfactory/manifest/schema.py @@ -15,18 +15,21 @@ from ..domain.states import State from ..storage.limits import ( + MAX_ARTIFACT_BODY_BYTES, MAX_CONTENT_CHARS, MAX_ID_CHARS, + MAX_INPUT_CONTENT_BYTES, MAX_INTENT_CHARS, MAX_LOGICAL_PATH_CHARS, MAX_MANIFEST_BYTES, MAX_OUTCOMES, + MAX_PREVIEW_CHARS, MAX_REASON_CHARS, MAX_STATEMENT_CHARS, - MAX_PREVIEW_CHARS, + MAX_SUMMARY_BYTES, ) from ..storage.paths import PACKAGE_ID_RE, validate_identifier, validate_logical_path, validate_package_id -from ..storage.serialization import canonical_bytes_bounded, contains_control_chars +from ..storage.serialization import contains_control_chars, try_canonical_bytes_bounded SCHEMA_VERSION = "0.1" @@ -98,11 +101,14 @@ def validate_manifest(manifest: dict) -> list[str]: if not isinstance(manifest, dict): return ["manifest must be a JSON object"] - # Total canonical manifest byte bound. - try: - canonical_bytes_bounded(manifest, limit=MAX_MANIFEST_BYTES, what="manifest") - except ValueError as exc: - errors.append(str(exc)) + # Total canonical manifest byte bound. Native canonicalization failures + # (unsupported types, deep recursion, lone-surrogate encoding) are + # translated into manifest errors — never leaked raw (Finding 1). + _, canonical_error = try_canonical_bytes_bounded( + manifest, limit=MAX_MANIFEST_BYTES, what="manifest" + ) + if canonical_error is not None: + errors.append(canonical_error) for key in manifest: if key not in TOP_LEVEL_FIELDS: @@ -139,6 +145,15 @@ def validate_manifest(manifest: dict) -> list[str]: errors.append(f"intent.raw exceeds {MAX_INTENT_CHARS} chars") if contains_control_chars(intent["raw"]): errors.append("intent.raw must not contain control characters") + if isinstance(intent, dict) and "clarified" in intent: + clarified = intent["clarified"] + if clarified is not None and not isinstance(clarified, str): + errors.append("intent.clarified must be a string or null") + elif isinstance(clarified, str): + if len(clarified) > MAX_INTENT_CHARS: + errors.append(f"intent.clarified exceeds {MAX_INTENT_CHARS} chars") + if contains_control_chars(clarified): + errors.append("intent.clarified must not contain control characters") inputs = manifest.get("inputs") if not isinstance(inputs, list): @@ -167,16 +182,22 @@ def validate_manifest(manifest: dict) -> list[str]: errors.append(f"{tag}.source invalid") if item.get("disposition") not in DISPOSITIONS: errors.append(f"{tag}.disposition invalid") - elif item.get("disposition") == "excluded" and not str(item.get("exclusion_reason") or "").strip(): + elif item.get("disposition") == "excluded" and not (isinstance(item.get("exclusion_reason"), str) and item["exclusion_reason"].strip()): errors.append(f"{tag}: excluded input requires exclusion_reason") - if isinstance(item.get("exclusion_reason"), str) and len(item["exclusion_reason"]) > MAX_REASON_CHARS: - errors.append(f"{tag}.exclusion_reason exceeds {MAX_REASON_CHARS} chars") + exclusion_reason = item.get("exclusion_reason") + if exclusion_reason is not None and not isinstance(exclusion_reason, str): + errors.append(f"{tag}.exclusion_reason must be a string or null") + elif isinstance(exclusion_reason, str): + if len(exclusion_reason) > MAX_REASON_CHARS: + errors.append(f"{tag}.exclusion_reason exceeds {MAX_REASON_CHARS} chars") + if contains_control_chars(exclusion_reason): + errors.append(f"{tag}.exclusion_reason must not contain control characters") if not (isinstance(item.get("content_sha256"), str) and SHA256_RE.match(item["content_sha256"])): errors.append(f"{tag}.content_sha256 invalid") if isinstance(item.get("content_size"), bool) or not isinstance(item.get("content_size"), int) or item["content_size"] < 0: errors.append(f"{tag}.content_size invalid") - elif item["content_size"] > MAX_CONTENT_CHARS: - errors.append(f"{tag}.content_size exceeds {MAX_CONTENT_CHARS}") + elif item["content_size"] > MAX_INPUT_CONTENT_BYTES: + errors.append(f"{tag}.content_size exceeds {MAX_INPUT_CONTENT_BYTES} bytes") if not isinstance(item.get("content_path"), str) or not item["content_path"]: errors.append(f"{tag}.content_path invalid") else: @@ -221,12 +242,16 @@ def validate_manifest(manifest: dict) -> list[str]: errors.append("summary.digest invalid (64-hex required)") if isinstance(summary.get("size"), bool) or not isinstance(summary.get("size"), int) or summary["size"] < 0: errors.append("summary.size invalid (non-negative int required)") + elif summary["size"] > MAX_SUMMARY_BYTES: + errors.append(f"summary.size exceeds {MAX_SUMMARY_BYTES} bytes") preview = summary.get("preview") if preview is not None: if not isinstance(preview, str): errors.append("summary.preview must be a string or null") elif len(preview) > SUMMARY_PREVIEW_MAX_CHARS: errors.append(f"summary.preview exceeds {SUMMARY_PREVIEW_MAX_CHARS} chars") + if isinstance(preview, str) and contains_control_chars(preview): + errors.append("summary.preview must not contain control characters") if not _is_iso8601(summary.get("presented_at")): errors.append("summary.presented_at must be ISO-8601") conf = summary.get("confirmation") @@ -241,8 +266,14 @@ def validate_manifest(manifest: dict) -> list[str]: if conf.get("status") == "confirmed": if not _is_iso8601(conf.get("confirmed_at")): errors.append("confirmed summary requires confirmed_at") - if not isinstance(conf.get("operator_id"), str): + operator_id = conf.get("operator_id") + if not isinstance(operator_id, str) or not operator_id: errors.append("confirmed summary requires operator_id") + else: + try: + validate_identifier(operator_id, field="summary.confirmation.operator_id") + except Exception: + errors.append("summary.confirmation.operator_id invalid identifier") artifacts = manifest.get("artifacts") if not isinstance(artifacts, list): @@ -271,6 +302,11 @@ def validate_manifest(manifest: dict) -> list[str]: errors.append(f"{tag}.kind exceeds {MAX_ID_CHARS} chars") elif contains_control_chars(art["kind"]): errors.append(f"{tag}.kind must not contain control characters") + else: + try: + validate_identifier(art["kind"], field=f"{tag}.kind") + except Exception: + errors.append(f"{tag}.kind invalid identifier") if not isinstance(art.get("logical_path"), str) or not art["logical_path"]: errors.append(f"{tag}.logical_path invalid") else: @@ -282,8 +318,8 @@ def validate_manifest(manifest: dict) -> list[str]: errors.append(f"{tag}.sha256 invalid") if isinstance(art.get("byte_count"), bool) or not isinstance(art.get("byte_count"), int) or art["byte_count"] < 0: errors.append(f"{tag}.byte_count invalid") - elif art["byte_count"] > MAX_CONTENT_CHARS: - errors.append(f"{tag}.byte_count exceeds {MAX_CONTENT_CHARS}") + elif art["byte_count"] > MAX_ARTIFACT_BODY_BYTES: + errors.append(f"{tag}.byte_count exceeds {MAX_ARTIFACT_BODY_BYTES} bytes") if art.get("status") != "draft": errors.append(f"{tag}.status must be 'draft' in v0.1") @@ -293,7 +329,14 @@ def validate_manifest(manifest: dict) -> list[str]: else: for f in ("last_event_id", "last_action_id"): v = transition.get(f) - if v is not None and not isinstance(v, str): + if v is None: + continue + if not isinstance(v, str): errors.append(f"transition.{f} must be a string or null") + else: + try: + validate_identifier(v, field=f"transition.{f}") + except Exception: + errors.append(f"transition.{f} invalid identifier") return errors diff --git a/methodfactory/protocol/envelope.py b/methodfactory/protocol/envelope.py index 45d7fc8..f56a38d 100644 --- a/methodfactory/protocol/envelope.py +++ b/methodfactory/protocol/envelope.py @@ -92,21 +92,35 @@ def as_dict(self) -> dict[str, Any]: def parse_envelope(raw: str) -> ActionEnvelope: """Parse exactly one JSON object, then validate it strictly. - Enforces MAX_ENVELOPE_BYTES on the UTF-8 bytes BEFORE any JSON parse or - prose extraction (Finding 1): an oversized envelope fails fast without - materializing a large parse. Tolerates surrounding prose by extracting the - text between the first '{' and the last '}' only when the raw text is - within the byte bound. + Enforces MAX_ENVELOPE_BYTES on the ORIGINAL RAW UTF-8 input BEFORE + strip(), parsing, or prose extraction (Finding 1): a raw input that + exceeds the bound because of surrounding whitespace is still rejected. + Encoding failures (e.g. lone-surrogate UnicodeEncodeError) are translated + to InvalidEnvelopeError. Tolerates surrounding prose by extracting the + text between the first '{' and the last '}' only within the byte bound. """ if not isinstance(raw, str): raise InvalidEnvelopeError("envelope must be a string") - text = raw.strip() - if not text: - raise InvalidEnvelopeError("empty envelope") - if len(text.encode("utf-8")) > MAX_ENVELOPE_BYTES: + # Fast-fail on character count BEFORE encoding (local review, perf-2): + # every UTF-8 char is >= 1 byte, so len(raw) > bound implies the byte + # bound is exceeded — oversized inputs are rejected without an O(n) + # allocation. The exact UTF-8 byte measurement below stays authoritative + # (multibyte strings can exceed the byte bound under the char count). + if len(raw) > MAX_ENVELOPE_BYTES: + raise InvalidEnvelopeError( + f"envelope exceeds {MAX_ENVELOPE_BYTES} bytes" + ) + try: + raw_bytes = raw.encode("utf-8") + except UnicodeEncodeError as exc: + raise InvalidEnvelopeError(f"envelope is not valid UTF-8: {exc}") from exc + if len(raw_bytes) > MAX_ENVELOPE_BYTES: raise InvalidEnvelopeError( f"envelope exceeds {MAX_ENVELOPE_BYTES} bytes" ) + text = raw.strip() + if not text: + raise InvalidEnvelopeError("empty envelope") try: candidate = json.loads(text) except json.JSONDecodeError: diff --git a/methodfactory/storage/limits.py b/methodfactory/storage/limits.py index 6adf9fe..e659878 100644 --- a/methodfactory/storage/limits.py +++ b/methodfactory/storage/limits.py @@ -47,3 +47,15 @@ # Enforced at put(); the byte budget is the storage ceiling, the char budget # bounds a string payload before encoding. MAX_ARTIFACT_BYTES = 64 * 1024 * 1024 + +# ── Persisted content-addressed body BYTE ceilings (Finding 1 item 2) ─── +# These are UTF-8 BYTE bounds for content stored in the blob store, applied +# to the persisted manifest's content_size / summary.size / artifact +# byte_count fields. They are deliberately separate from the *_CHARS limits: +# a byte field must never be compared against a character constant. All three +# ALIAS MAX_ARTIFACT_BYTES (local review, q-3): every persisted body lives in +# the same content-addressed blob store, so the ceilings cannot drift apart +# from the storage ceiling. +MAX_INPUT_CONTENT_BYTES = MAX_ARTIFACT_BYTES # record_input content body IS an artifact body +MAX_SUMMARY_BYTES = MAX_ARTIFACT_BYTES # content-addressed summary body (digest+size in manifest; body in blob store) +MAX_ARTIFACT_BODY_BYTES = MAX_ARTIFACT_BYTES # artifact byte_count ceiling diff --git a/methodfactory/storage/serialization.py b/methodfactory/storage/serialization.py index 378c949..31e8a43 100644 --- a/methodfactory/storage/serialization.py +++ b/methodfactory/storage/serialization.py @@ -51,6 +51,23 @@ def canonical_bytes_bounded(value: Any, *, limit: int, what: str) -> bytes: return raw +def try_canonical_bytes_bounded(value: Any, *, limit: int, what: str) -> tuple[bytes | None, str | None]: + """Canonicalize with full native-failure translation (Finding 1 item 4). + + Returns (bytes, None) on success or (None, error_message) when the value + cannot be canonicalized: unsupported JSON types, excessive recursion, or + lone-surrogate UnicodeEncodeError are all captured as a message rather + than leaking raw TypeError/RecursionError/UnicodeEncodeError. + """ + try: + raw = canonical_bytes(value) + except (TypeError, RecursionError, UnicodeEncodeError, ValueError) as exc: + return None, f"{what} cannot be canonicalized: {exc}" + if len(raw) > limit: + return None, f"{what} exceeds {limit} bytes (got {len(raw)})" + return raw, None + + def sha256_hex(data: bytes) -> str: return hashlib.sha256(data).hexdigest() diff --git a/methodfactory/tests/test_boundary_model.py b/methodfactory/tests/test_boundary_model.py index 330b521..ed597b6 100644 --- a/methodfactory/tests/test_boundary_model.py +++ b/methodfactory/tests/test_boundary_model.py @@ -14,17 +14,20 @@ from methodfactory.protocol.envelope import parse_envelope from methodfactory.storage.limits import ( MAX_ACTION_JSON_BYTES, + MAX_ARTIFACT_BODY_BYTES, MAX_CONTENT_CHARS, MAX_ENVELOPE_BYTES, MAX_ID_CHARS, + MAX_INPUT_CONTENT_BYTES, MAX_INTENT_CHARS, MAX_LOGICAL_PATH_CHARS, MAX_MANIFEST_BYTES, MAX_OUTCOMES, MAX_REASON_CHARS, MAX_STATEMENT_CHARS, + MAX_SUMMARY_BYTES, ) -from methodfactory.storage.serialization import action_sha256 +from methodfactory.storage.serialization import action_sha256, canonical_bytes from methodfactory.manifest.schema import new_manifest, validate_manifest @@ -56,6 +59,51 @@ def test_envelope_byte_limit_before_parse(self): with self.assertRaises(InvalidEnvelopeError): parse_envelope(big) + def test_raw_envelope_exact_at_and_one_over_before_strip(self): + """MAX_ENVELOPE_BYTES is measured on the ORIGINAL raw UTF-8 BEFORE + strip()/parse. Trailing whitespace counts toward the bound: exactly-at + with padding is accepted; one more byte fails.""" + base = json.dumps(env()) + base_len = len(base.encode("utf-8")) + self.assertLess(base_len, MAX_ENVELOPE_BYTES) + pad = MAX_ENVELOPE_BYTES - base_len + at = base + " " * pad + self.assertEqual(len(at.encode("utf-8")), MAX_ENVELOPE_BYTES) + self.assertEqual(parse_envelope(at).action_id, "act_1") # accepted exactly at + over = base + " " * (pad + 1) + self.assertEqual(len(over.encode("utf-8")), MAX_ENVELOPE_BYTES + 1) + with self.assertRaises(InvalidEnvelopeError): + parse_envelope(over) + + def test_raw_envelope_multibyte_exact_at_and_one_over(self): + """Multibyte (3-byte Thai chars) at the raw byte boundary: exactly-at + accepted, one byte over rejected. Character count is far below the + byte bound, proving the byte (not char) measurement.""" + base = json.dumps(env()) + remaining = MAX_ENVELOPE_BYTES - len(base.encode("utf-8")) + thai_chars = remaining // 3 + pad = remaining - 3 * thai_chars + at = base + "ส" * thai_chars + " " * pad + self.assertEqual(len(at.encode("utf-8")), MAX_ENVELOPE_BYTES) + self.assertLess(len(at), MAX_ENVELOPE_BYTES) # chars < bytes (multibyte) + self.assertEqual(parse_envelope(at).action_id, "act_1") + over = at + "x" + self.assertEqual(len(over.encode("utf-8")), MAX_ENVELOPE_BYTES + 1) + with self.assertRaises(InvalidEnvelopeError): + parse_envelope(over) + + def test_lone_surrogate_raw_envelope_translated(self): + """A raw envelope containing a literal lone surrogate cannot be encoded + as UTF-8; the failure must be translated to InvalidEnvelopeError, not + leak a raw UnicodeEncodeError.""" + raw = ('{"protocol_version":"0.1","action_id":"act_1",' + '"package_id":"pkg_demo_001","expected_revision":0,' + '"action":"record_input","basis":{},"payload":' + '{"input_id":"in_1","kind":"text","content":"\ud800",' + '"source":"operator","disposition":"incorporated"}}') + with self.assertRaises(InvalidEnvelopeError): + parse_envelope(raw) + def test_oversized_prose_rejected_before_extraction(self): # Huge surrounding prose around a small envelope must be rejected by # the byte bound before prose extraction. @@ -141,7 +189,9 @@ def test_logical_path_limit(self): class CanonicalActionBoundaryTests(unittest.TestCase): - def test_action_hash_at_and_over_action_json_bytes(self): + def test_action_hash_exact_at_and_one_over(self): + """MAX_ACTION_JSON_BYTES is enforced on the exact canonical bytes of + the semantic action: exactly-at hashes, one-over is rejected.""" semantic = { "protocol_version": "0.1", "action": "record_input", @@ -149,46 +199,244 @@ def test_action_hash_at_and_over_action_json_bytes(self): "action_id": "act_1", "basis": {}, "payload": {"input_id": "in_1", "kind": "text", - "content": "x" * (MAX_ACTION_JSON_BYTES + 1), + "content": "x", "source": "operator", "disposition": "incorporated"}, } + base_len = len(canonical_bytes(semantic)) + self.assertLess(base_len, MAX_ACTION_JSON_BYTES) + # base_len already includes content="x" (1 byte); add MAX-base_len+1 + # 'x' chars so the canonical size lands EXACTLY on the bound. + k = MAX_ACTION_JSON_BYTES - base_len + 1 + at = dict(semantic) + at["payload"] = dict(semantic["payload"], content="x" * k) + self.assertEqual(len(canonical_bytes(at)), MAX_ACTION_JSON_BYTES) + self.assertEqual(len(action_sha256(**at)), 64) # exactly-at accepted + over = dict(semantic) + over["payload"] = dict(semantic["payload"], content="x" * (k + 1)) + self.assertEqual(len(canonical_bytes(over)), MAX_ACTION_JSON_BYTES + 1) with self.assertRaises(ValueError): - action_sha256(**semantic) - # at-limit via canonical bytes - semantic_at = { - "protocol_version": "0.1", - "action": "record_input", - "package_id": "pkg_demo_001", - "action_id": "act_1", - "basis": {}, - "payload": {"input_id": "in_1", "kind": "text", - "content": "x", - "source": "operator", "disposition": "incorporated"}, + action_sha256(**over) + + def test_action_hash_unicode_and_recursion_failures_do_not_leak_raw(self): + """Non-canonicalizable semantic payloads (lone surrogate, deep + recursion) must surface as a controlled error, never a raw + UnicodeEncodeError/RecursionError.""" + sur = { + "protocol_version": "0.1", "action": "record_input", + "package_id": "pkg_demo_001", "action_id": "act_1", + "basis": {}, "payload": {"content": "x\ud800"}, } - h = action_sha256(**semantic_at) - self.assertEqual(len(h), 64) + with self.assertRaises(SerializationError) as ctx: + action_sha256(**sur) + self.assertEqual(ctx.exception.code, "SERIALIZATION") + deep_payload: dict = {} + cur = deep_payload + for _ in range(20_000): + nxt: dict = {} + cur["a"] = nxt + cur = nxt + with self.assertRaises(SerializationError): + action_sha256(**{ + "protocol_version": "0.1", "action": "record_input", + "package_id": "pkg_demo_001", "action_id": "act_1", + "basis": {}, "payload": deep_payload, + }) + + +def _valid_input(input_id: str = "in_1", **over) -> dict: + entry = { + "input_id": input_id, + "kind": "text", + "source": "operator", + "disposition": "incorporated", + "exclusion_reason": None, + "content_sha256": "0" * 64, + "content_size": 1, + "content_path": f"inputs/{input_id}.txt", + } + entry.update(over) + return entry + + +def _manifest_with_inputs(n: int, *, reason_len: int = 1) -> dict: + """Build a manifest with n valid EXCLUDED input entries, each carrying an + exclusion_reason of exactly `reason_len` bytes. All entries are uniform so + canonical size grows linearly and can be tuned to an exact byte count.""" + m = new_manifest("pkg_demo_001", "x", "2026-08-07T00:00:00+00:00") + m["inputs"] = [ + _valid_input( + input_id=f"in_{i:06d}", + disposition="excluded", + exclusion_reason="x" * reason_len, + ) + for i in range(n) + ] + return m + + +def _valid_summary(**over) -> dict: + summary = { + "digest": "0" * 64, + "size": 1, + "preview": None, + "presented_at": "2026-08-07T00:00:00+00:00", + "confirmation": {"status": "pending"}, + } + summary.update(over) + return summary + + +def _valid_artifact(**over) -> dict: + art = { + "artifact_id": "art_1", + "kind": "skill", + "logical_path": "skills/x/SKILL.md", + "sha256": "0" * 64, + "byte_count": 1, + "status": "draft", + } + art.update(over) + return art class ManifestBoundaryTests(unittest.TestCase): - def test_total_manifest_byte_bound(self): + def test_total_manifest_bytes_exact_at_and_one_over(self): + """MAX_MANIFEST_BYTES is enforced on the exact canonical bytes of the + complete manifest. The at-limit manifest is tuned to land EXACTLY on + the bound (and validates clean); one entry more is one byte over.""" + base = _manifest_with_inputs(0) + base_len = len(canonical_bytes(base)) + one = _manifest_with_inputs(1, reason_len=1) + two = _manifest_with_inputs(2, reason_len=1) + first_entry = len(canonical_bytes(one)) - base_len + extra_entry = len(canonical_bytes(two)) - len(canonical_bytes(one)) + need = MAX_MANIFEST_BYTES - base_len + # total(q) = base_len + first_entry + (q-1)*extra_entry + q_minus_1, r = divmod(need - first_entry, extra_entry) + q = q_minus_1 + 1 + at = _manifest_with_inputs(q, reason_len=1) + if r: + at["inputs"][-1]["exclusion_reason"] = "x" * (1 + r) + self.assertEqual(len(canonical_bytes(at)), MAX_MANIFEST_BYTES) + self.assertEqual(validate_manifest(at), []) # exactly-at accepted + over = _manifest_with_inputs(q + 1, reason_len=1) + self.assertEqual( + len(canonical_bytes(over)), MAX_MANIFEST_BYTES + (extra_entry - r) + ) + errors = validate_manifest(over) + self.assertTrue(any("exceeds" in e and "manifest" in e for e in errors)) + + def test_persisted_input_content_size_bytes_at_and_over(self): m = new_manifest("pkg_demo_001", "x", "2026-08-07T00:00:00+00:00") + m["inputs"] = [_valid_input(content_size=MAX_INPUT_CONTENT_BYTES)] self.assertEqual(validate_manifest(m), []) - # One-over the manifest byte bound via a huge intent (char limit first - # would trip, so bypass by setting a giant inputs list). - big = new_manifest("pkg_demo_001", "x", "2026-08-07T00:00:00+00:00") - big["inputs"] = [ - {"input_id": f"in_{i}", "kind": "text", "source": "operator", - "disposition": "incorporated", "exclusion_reason": None, - "content_sha256": "0" * 64, "content_size": 1, - "content_path": f"inputs/in_{i}.txt"} - for i in range(300_000) - ] - self.assertGreater( - len(json.dumps(big, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8")), - MAX_MANIFEST_BYTES, + m["inputs"][0]["content_size"] = MAX_INPUT_CONTENT_BYTES + 1 + errors = validate_manifest(m) + self.assertTrue(any("content_size" in e and "bytes" in e for e in errors)) + + def test_persisted_summary_size_bytes_at_and_over(self): + m = new_manifest("pkg_demo_001", "x", "2026-08-07T00:00:00+00:00") + m["summary"] = _valid_summary(size=MAX_SUMMARY_BYTES) + self.assertEqual(validate_manifest(m), []) + m["summary"]["size"] = MAX_SUMMARY_BYTES + 1 + errors = validate_manifest(m) + self.assertTrue(any("summary.size" in e and "bytes" in e for e in errors)) + + def test_persisted_artifact_byte_count_bytes_at_and_over(self): + m = new_manifest("pkg_demo_001", "x", "2026-08-07T00:00:00+00:00") + m["artifacts"] = [_valid_artifact(byte_count=MAX_ARTIFACT_BODY_BYTES)] + self.assertEqual(validate_manifest(m), []) + m["artifacts"][0]["byte_count"] = MAX_ARTIFACT_BODY_BYTES + 1 + errors = validate_manifest(m) + self.assertTrue(any("byte_count" in e and "bytes" in e for e in errors)) + + def test_manifest_canonicalization_failures_collected(self): + """validate_manifest() must never leak raw TypeError/RecursionError/ + UnicodeEncodeError from canonicalization; they become manifest errors.""" + # TypeError: non-JSON-serializable value + m = new_manifest("pkg_demo_001", "x", "2026-08-07T00:00:00+00:00") + m["objective"]["desired_outcomes"] = [set()] + errors = validate_manifest(m) + self.assertTrue(any("cannot be canonicalized" in e for e in errors)) + # RecursionError: deeply nested value (depth 20000 is above the + # CPython 3.11/3.12 C-encoder RecursionError threshold; local review: + # depth 5000 serializes fine on 3.12 and would not exercise the path). + deep = new_manifest("pkg_demo_001", "x", "2026-08-07T00:00:00+00:00") + node: dict = {} + cur = node + for _ in range(20_000): + nxt: dict = {} + cur["a"] = nxt + cur = nxt + deep["inputs"] = [_valid_input(exclusion_reason=node)] + errors = validate_manifest(deep) + self.assertTrue(any("cannot be canonicalized" in e for e in errors)) + # UnicodeEncodeError: lone surrogate in a string field + sur = new_manifest("pkg_demo_001", "x", "2026-08-07T00:00:00+00:00") + sur["intent"]["raw"] = "x\ud800" + errors = validate_manifest(sur) + self.assertTrue(any("cannot be canonicalized" in e for e in errors)) + + def test_intent_clarified_rules(self): + m = new_manifest("pkg_demo_001", "x", "2026-08-07T00:00:00+00:00") + m["intent"]["clarified"] = "c" * MAX_INTENT_CHARS + self.assertEqual(validate_manifest(m), []) + m["intent"]["clarified"] = "c" * (MAX_INTENT_CHARS + 1) + self.assertTrue(any("intent.clarified" in e and "exceeds" in e for e in validate_manifest(m))) + m["intent"]["clarified"] = "bad\u2028clarified" + self.assertTrue(any("intent.clarified" in e and "control" in e for e in validate_manifest(m))) + m["intent"]["clarified"] = 5 + self.assertTrue(any("intent.clarified" in e for e in validate_manifest(m))) + + def test_exclusion_reason_rules(self): + m = new_manifest("pkg_demo_001", "x", "2026-08-07T00:00:00+00:00") + m["inputs"] = [_valid_input(disposition="excluded", exclusion_reason="r" * MAX_REASON_CHARS)] + self.assertEqual(validate_manifest(m), []) + m["inputs"][0]["exclusion_reason"] = "r" * (MAX_REASON_CHARS + 1) + self.assertTrue(any("exclusion_reason" in e and "exceeds" in e for e in validate_manifest(m))) + m["inputs"][0]["exclusion_reason"] = "bad\x01reason" + self.assertTrue(any("exclusion_reason" in e and "control" in e for e in validate_manifest(m))) + m["inputs"][0]["exclusion_reason"] = 5 + self.assertTrue(any("exclusion_reason" in e for e in validate_manifest(m))) + + def test_summary_preview_control_chars(self): + m = new_manifest("pkg_demo_001", "x", "2026-08-07T00:00:00+00:00") + m["summary"] = _valid_summary(preview="ok preview") + self.assertEqual(validate_manifest(m), []) + m["summary"]["preview"] = "bad\u2028preview" + self.assertTrue(any("preview" in e and "control" in e for e in validate_manifest(m))) + + def test_operator_id_identifier_rule(self): + m = new_manifest("pkg_demo_001", "x", "2026-08-07T00:00:00+00:00") + m["summary"] = _valid_summary( + confirmation={ + "status": "confirmed", + "confirmed_summary_sha256": "0" * 64, + "confirmed_at": "2026-08-07T00:00:00+00:00", + "operator_id": "op_alice-1", + } ) - errors = validate_manifest(big) - self.assertTrue(any("exceeds" in e and "manifest" in e for e in errors)) + self.assertEqual(validate_manifest(m), []) + m["summary"]["confirmation"]["operator_id"] = "../evil" + self.assertTrue(any("operator_id" in e for e in validate_manifest(m))) + m["summary"]["confirmation"]["operator_id"] = 5 + self.assertTrue(any("operator_id" in e for e in validate_manifest(m))) + + def test_transition_identifiers(self): + m = new_manifest("pkg_demo_001", "x", "2026-08-07T00:00:00+00:00") + m["transition"] = {"last_event_id": "evt_1", "last_action_id": "act_1"} + self.assertEqual(validate_manifest(m), []) + m["transition"]["last_event_id"] = "bad id" + self.assertTrue(any("last_event_id" in e for e in validate_manifest(m))) + m["transition"]["last_action_id"] = 5 + self.assertTrue(any("last_action_id" in e for e in validate_manifest(m))) + + def test_artifact_kind_identifier(self): + m = new_manifest("pkg_demo_001", "x", "2026-08-07T00:00:00+00:00") + m["artifacts"] = [_valid_artifact()] + self.assertEqual(validate_manifest(m), []) + m["artifacts"][0]["kind"] = "bad kind!" + self.assertTrue(any("kind" in e and "identifier" in e for e in validate_manifest(m))) def test_intent_length_limit(self): m = new_manifest("pkg_demo_001", "x" * (MAX_INTENT_CHARS + 1), "2026-08-07T00:00:00+00:00") diff --git a/methodfactory/tests/test_limits.py b/methodfactory/tests/test_limits.py index 22b7110..282947e 100644 --- a/methodfactory/tests/test_limits.py +++ b/methodfactory/tests/test_limits.py @@ -6,16 +6,20 @@ from methodfactory.storage.limits import ( MAX_ACTION_JSON_BYTES, + MAX_ARTIFACT_BODY_BYTES, MAX_ARTIFACT_BYTES, MAX_CONTENT_CHARS, MAX_ENVELOPE_BYTES, MAX_ID_CHARS, + MAX_INPUT_CONTENT_BYTES, MAX_INTENT_CHARS, MAX_LOGICAL_PATH_CHARS, MAX_MANIFEST_BYTES, MAX_OUTCOMES, + MAX_PREVIEW_CHARS, MAX_REASON_CHARS, MAX_STATEMENT_CHARS, + MAX_SUMMARY_BYTES, ) @@ -26,16 +30,31 @@ def test_all_positive(self): MAX_ACTION_JSON_BYTES, MAX_MANIFEST_BYTES, MAX_ARTIFACT_BYTES, + MAX_INPUT_CONTENT_BYTES, + MAX_SUMMARY_BYTES, + MAX_ARTIFACT_BODY_BYTES, MAX_CONTENT_CHARS, MAX_INTENT_CHARS, MAX_STATEMENT_CHARS, MAX_ID_CHARS, MAX_LOGICAL_PATH_CHARS, MAX_REASON_CHARS, + MAX_PREVIEW_CHARS, MAX_OUTCOMES, ): self.assertGreater(v, 0, f"{v} must be positive") + def test_byte_versus_char_limits_are_distinct(self): + """A persisted byte field must never be compared against a character + constant: the new *_BYTES ceilings are separate constants, and the + artifact byte ceiling is aliased to the storage ceiling so they cannot + drift (Finding 1).""" + self.assertNotEqual(MAX_INPUT_CONTENT_BYTES, MAX_CONTENT_CHARS) + self.assertNotEqual(MAX_SUMMARY_BYTES, MAX_PREVIEW_CHARS) + self.assertEqual(MAX_ARTIFACT_BODY_BYTES, MAX_ARTIFACT_BYTES) + self.assertEqual(MAX_INPUT_CONTENT_BYTES, MAX_ARTIFACT_BYTES) + self.assertEqual(MAX_SUMMARY_BYTES, MAX_ARTIFACT_BYTES) + def test_envelope_limit_never_reused_for_event_or_manifest(self): # ADR-0012 §4: never reuse MAX_ENVELOPE_BYTES as an event/manifest limit. self.assertNotEqual(MAX_ENVELOPE_BYTES, MAX_MANIFEST_BYTES) From 82fa8ca62b106cf76ad47529d41d25dde03cc741 Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Fri, 7 Aug 2026 10:27:56 +0700 Subject: [PATCH 17/41] fix(artifacts): complete durability and failure translation (review 4879440857 #2) Keep the same-directory hard-link publication design (temp write -> file fsync -> no-clobber os.link -> verify raced destination -> unlink temp -> directory fsync) but correct the remaining boundaries: - translate content.encode() Unicode failures (lone surrogates) and invalid artifact-store root types into InvalidPayloadError; - translate directory open/fsync/close failures into InvalidPayloadError; - temporary unlink failure after publication is NO LONGER silent success: it returns a typed error (outer cleanup still best-effort removes the temp); - preserve original causes via raise ... from exc. Introduce narrow module-level injection seams (_open_tmp, _write_all, _fsync_file, _hardlink, _unlink_tmp, _open_dir, _fsync_dir, _close_fd) so tests inject precise single-point faults through the real implementation path; the mock-of-put() write test is replaced with a real _write_all failure. Directory-fsync scenario proven: file fsync succeeds, directory fsync fails -> canonical blob already published, typed error returned, retry verifies the existing blob and succeeds, no temporary file remains. Existing- destination tests are labeled honestly as verification (not race); a coordinated _hardlink side effect creates the destination between temp write and publication to prove the real race path. --- methodfactory/adapters/artifact_store.py | 197 +++++++++++---- .../tests/test_artifact_publication.py | 225 ++++++++++++++---- 2 files changed, 335 insertions(+), 87 deletions(-) diff --git a/methodfactory/adapters/artifact_store.py b/methodfactory/adapters/artifact_store.py index abfb767..f7b8a1b 100644 --- a/methodfactory/adapters/artifact_store.py +++ b/methodfactory/adapters/artifact_store.py @@ -1,7 +1,8 @@ """Filesystem ArtifactStore — immutable content-addressed blobs (ADR-0007). -Phase 2 corrections (Finding 2, review 4879090471): blob publication is -genuinely immutable via a no-clobber hard-link primitive. +Phase 2 corrections (Finding 2, review 4879090471; closure review 4879440857): +blob publication is genuinely immutable via a no-clobber hard-link primitive, +with narrow injection seams and complete failure translation. Publication algorithm (put): 1. Validate logical path, package ID, and size limits. @@ -10,21 +11,35 @@ 4. Publish it to the digest path via os.link(tmp, dest) — an atomic no-overwrite primitive. On FileExistsError, treat it as a publication race and VERIFY the existing canonical blob matches the digest. -5. Remove the temporary link/file. +5. Remove the temporary link/file (failure here is NOT silent after + publication). 6. fsync the containing directory. An existing canonical digest path is NEVER replaced, even when the expected content is identical. A partial final digest path is never exposed as success. -All artifact OS/Unicode/type failures are translated into the public Method -Factory error hierarchy (InvalidPayloadError), with the original exception -retained as the cause (Finding 2 / Finding 4). +Failure translation (Finding 2 / Finding 4): +- content.encode() Unicode failures (lone surrogates) -> InvalidPayloadError; +- invalid artifact-store root types -> InvalidPayloadError; +- directory close failures -> InvalidPayloadError; +- temporary unlink failure after publication -> InvalidPayloadError (never + silent success); +- every OS/Unicode/type failure is raised as InvalidPayloadError (a public + MethodFactoryError) with the original exception retained via `raise ... from + exc`. + +Injection seams: every OS primitive used by publication is a module-level +function (_open_tmp, _write_all, _fsync_file, _hardlink, _unlink_tmp, +_open_dir, _fsync_dir, _close_fd). Tests inject precise faults at a single +seam without mocking put() itself; the production path is unchanged. """ from __future__ import annotations import os +import re +import sys import uuid from pathlib import Path @@ -33,23 +48,77 @@ from ..storage.paths import validate_logical_path, validate_package_id from ..storage.serialization import digest_bytes +# Strict canonical digest grammar (local review): exactly 64 lowercase hex +# chars, matching the manifest validator (SHA256_RE). 0x-prefixed, uppercase, +# and underscore forms are rejected so the store and the manifest never +# disagree about what a digest is. +_HEX64_RE = re.compile(r"^[0-9a-f]{64}$") + + +# ── Narrow injection seams (Finding 2) ────────────────────────────────── +def _open_tmp(blobs_dir: Path) -> tuple[int, Path]: + tmp = blobs_dir / f".tmp.{uuid.uuid4().hex}" + fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + return fd, tmp + + +def _write_all(fh, data: bytes) -> None: + fh.write(data) + fh.flush() + + +def _fsync_file(fd: int) -> None: + os.fsync(fd) + + +def _hardlink(src: Path, dst: Path) -> None: + os.link(src, dst) + + +def _unlink_tmp(path: Path) -> None: + path.unlink(missing_ok=True) + + +def _open_dir(path: Path) -> int: + return os.open(path, os.O_RDONLY) + + +def _fsync_dir(fd: int) -> None: + os.fsync(fd) + + +def _close_fd(fd: int) -> None: + os.close(fd) + class ArtifactStore: def __init__(self, root: Path | str) -> None: + if not isinstance(root, (str, os.PathLike)): + raise InvalidPayloadError( + f"artifact store root must be a path, got {type(root).__name__}" + ) try: self.root = Path(root) - self.root.mkdir(parents=True, exist_ok=True) + # Private modes on root AND blobs (local review, sec-2): a + # umask-inherited group/world-writable blobs dir would let other + # local users unlink blobs or plant symlinks in the digest + # namespace, defeating the immutable-store guarantees. + os.makedirs(self.root, mode=0o700, exist_ok=True) self.blobs = self.root / "blobs" - self.blobs.mkdir(parents=True, exist_ok=True) + os.makedirs(self.blobs, mode=0o700, exist_ok=True) + os.chmod(self.root, 0o700) + os.chmod(self.blobs, 0o700) except OSError as exc: - raise InvalidPayloadError(f"cannot initialize artifact store at {root}: {exc}") from exc + raise InvalidPayloadError( + f"cannot initialize artifact store at {root}: {exc}" + ) from exc + except (TypeError, ValueError) as exc: + raise InvalidPayloadError( + f"invalid artifact store root {root!r}: {exc}" + ) from exc def _blob_path(self, digest: str) -> Path: - if not isinstance(digest, str) or len(digest) != 64: - raise InvalidPayloadError(f"invalid artifact digest: {digest!r}") - try: - int(digest, 16) - except ValueError: + if not isinstance(digest, str) or not _HEX64_RE.match(digest): raise InvalidPayloadError(f"invalid artifact digest: {digest!r}") return self.blobs / digest @@ -63,7 +132,12 @@ def put(self, package_id: str, logical_path: str, content: str) -> tuple[str, in validate_logical_path(logical_path) if not isinstance(content, str): raise InvalidPayloadError("artifact content must be a string") - data = content.encode("utf-8") + try: + data = content.encode("utf-8") + except UnicodeEncodeError as exc: + raise InvalidPayloadError( + "artifact content is not valid UTF-8 (lone surrogate?)" + ) from exc if len(data) > MAX_ARTIFACT_BYTES: raise InvalidPayloadError( f"artifact exceeds MAX_ARTIFACT_BYTES ({MAX_ARTIFACT_BYTES})" @@ -75,30 +149,45 @@ def put(self, package_id: str, logical_path: str, content: str) -> tuple[str, in digest = digest_bytes(data) dest = self._blob_path(digest) - # Same-directory temporary file. - tmp = self.blobs / f".tmp.{uuid.uuid4().hex}" + # Idempotency fast path (local review, perf-1): if the canonical blob + # already exists, verify it and return WITHOUT any temp write/fsync. + # The os.link/FileExistsError path remains the mid-publication race + # backstop; a corrupt existing blob fails typed exactly as before. + try: + if dest.is_file(): + existing = dest.read_bytes() + if digest_bytes(existing) != digest: + raise InvalidPayloadError( + f"existing blob does not match digest {digest}" + ) + return digest, len(data) + except OSError as exc: + raise InvalidPayloadError( + f"cannot read existing blob {digest}: {exc}" + ) from exc + + fd: int | None = None + tmp: Path | None = None + dir_fd: int | None = None try: + # 1. Same-directory temporary file. try: - fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + fd, tmp = _open_tmp(self.blobs) except OSError as exc: raise InvalidPayloadError(f"cannot create temp blob: {exc}") from exc try: with os.fdopen(fd, "wb") as fh: - fh.write(data) - fh.flush() - try: - os.fsync(fh.fileno()) - except OSError as exc: - raise InvalidPayloadError(f"fsync temp blob failed: {exc}") from exc + _write_all(fh, data) + _fsync_file(fh.fileno()) except InvalidPayloadError: raise except OSError as exc: raise InvalidPayloadError(f"write temp blob failed: {exc}") from exc - # No-clobber publication via hard link (atomic; never replaces an - # existing canonical digest path). + # 2. No-clobber publication via hard link (atomic; never replaces + # an existing canonical digest path). try: - os.link(tmp, dest) + _hardlink(tmp, dest) except FileExistsError: # Publication race: verify the existing canonical blob matches # the digest. Never replace it. @@ -114,29 +203,49 @@ def put(self, package_id: str, logical_path: str, content: str) -> tuple[str, in ) except OSError as exc: raise InvalidPayloadError(f"publish blob failed: {exc}") from exc - finally: - try: - tmp.unlink(missing_ok=True) - except OSError: - pass - # fsync the containing directory after publication. + # 3. Remove the temporary link/file. After publication this MUST + # not silently fail: a leftover temp would mask success. try: - dir_fd = os.open(self.blobs, os.O_RDONLY) + _unlink_tmp(tmp) except OSError as exc: - raise InvalidPayloadError(f"cannot open blobs dir for fsync: {exc}") from exc + raise InvalidPayloadError( + f"cannot remove temp blob {tmp.name}: {exc}" + ) from exc + + # 4. fsync the containing directory after publication. try: - try: - os.fsync(dir_fd) - except OSError as exc: - raise InvalidPayloadError(f"fsync blobs dir failed: {exc}") from exc + dir_fd = _open_dir(self.blobs) + except OSError as exc: + raise InvalidPayloadError( + f"cannot open blobs dir for fsync: {exc}" + ) from exc + try: + _fsync_dir(dir_fd) + except OSError as exc: + raise InvalidPayloadError(f"fsync blobs dir failed: {exc}") from exc finally: - os.close(dir_fd) + if dir_fd is not None: + # Capture the in-flight exception BEFORE closing (the + # close-error handler's own sys.exc_info() is the close + # error, not the pre-existing one). + in_flight = sys.exc_info()[0] + try: + _close_fd(dir_fd) + except OSError as exc: + # Do not mask an in-flight durability error (local + # review, bug-5): a close failure is only reported when + # no other exception is already propagating. + if in_flight is None: + raise InvalidPayloadError( + f"cannot close blobs dir: {exc}" + ) from exc except BaseException: - try: - tmp.unlink(missing_ok=True) - except OSError: - pass + if tmp is not None: + try: + _unlink_tmp(tmp) + except OSError: + pass raise return digest, len(data) diff --git a/methodfactory/tests/test_artifact_publication.py b/methodfactory/tests/test_artifact_publication.py index 68566ef..49fd12a 100644 --- a/methodfactory/tests/test_artifact_publication.py +++ b/methodfactory/tests/test_artifact_publication.py @@ -1,9 +1,18 @@ """Real fault-injection tests for immutable blob publication (Finding 2). -Injects failures at each durability point: write, file fsync, publication -(hard link), directory fsync; plus publication races, concurrent same-digest -writers, valid/corrupt raced destinations, and retry after a post-publication -durability error. +Injects failures at each durability point THROUGH THE REAL IMPLEMENTATION +PATH via the narrow module-level seams (_write_all, _fsync_file, _hardlink, +_unlink_tmp, _open_dir, _fsync_dir, _close_fd) — never by mocking put(). + +Directory-fsync scenario (review 4879440857 #2): file fsync succeeds, the +directory fsync fails; the canonical blob is already published, the operation +returns a typed error, retry verifies the existing blob and succeeds, and no +temporary file remains. + +Honest labeling: a test where the destination already exists before put() +begins proves the idempotent verification path, not an in-flight race. The +coordinated race test makes the destination appear BETWEEN temp write and +publication via a _hardlink side effect. """ from __future__ import annotations @@ -15,6 +24,7 @@ from pathlib import Path from unittest import mock +from methodfactory.adapters import artifact_store as as_mod from methodfactory.adapters.artifact_store import ArtifactStore from methodfactory.domain.errors import InvalidPayloadError from methodfactory.storage.serialization import digest_bytes @@ -24,67 +34,170 @@ def _blob_count(store: ArtifactStore) -> int: return len([p for p in store.blobs.iterdir() if not p.name.startswith(".tmp.")]) +def _tmp_count(store: ArtifactStore) -> int: + return len([p for p in store.blobs.iterdir() if p.name.startswith(".tmp.")]) + + class PublicationFaultTests(unittest.TestCase): def test_write_failure_removes_temp_no_canonical(self): + """Real implementation-path write failure (not a mock of put()).""" + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + with mock.patch.object(as_mod, "_write_all", side_effect=OSError("write fail")): + with self.assertRaises(InvalidPayloadError): + store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + self.assertEqual(_blob_count(store), 0) + self.assertEqual(_tmp_count(store), 0) + + def test_temp_creation_failure_typed_no_leftovers(self): + """Local review (q-4): the _open_tmp seam fails first — typed error, + zero canonical blobs, zero temp files.""" with tempfile.TemporaryDirectory() as td: store = ArtifactStore(Path(td)) - with mock.patch.object(ArtifactStore, "put", side_effect=InvalidPayloadError("write fail")): + with mock.patch.object(as_mod, "_open_tmp", side_effect=OSError("create fail")): with self.assertRaises(InvalidPayloadError): store.put("pkg_demo_001", "skills/x/SKILL.md", "content") self.assertEqual(_blob_count(store), 0) - self.assertEqual([p for p in store.blobs.iterdir() if p.name.startswith(".tmp.")], []) + self.assertEqual(_tmp_count(store), 0) def test_file_fsync_failure(self): + """Temp-file fsync fails -> typed error; nothing published, no temp.""" with tempfile.TemporaryDirectory() as td: store = ArtifactStore(Path(td)) - with mock.patch("os.fsync", side_effect=OSError("fsync fail")): + with mock.patch.object(as_mod, "_fsync_file", side_effect=OSError("fsync fail")): with self.assertRaises(InvalidPayloadError): store.put("pkg_demo_001", "skills/x/SKILL.md", "content") self.assertEqual(_blob_count(store), 0) - self.assertEqual([p for p in store.blobs.iterdir() if p.name.startswith(".tmp.")], []) + self.assertEqual(_tmp_count(store), 0) def test_publication_failure_removes_temp(self): + """Hard-link publication fails -> typed error; no canonical, no temp.""" with tempfile.TemporaryDirectory() as td: store = ArtifactStore(Path(td)) - with mock.patch("os.link", side_effect=OSError("link fail")): + with mock.patch.object(as_mod, "_hardlink", side_effect=OSError("link fail")): with self.assertRaises(InvalidPayloadError): store.put("pkg_demo_001", "skills/x/SKILL.md", "content") self.assertEqual(_blob_count(store), 0) - self.assertEqual([p for p in store.blobs.iterdir() if p.name.startswith(".tmp.")], []) + self.assertEqual(_tmp_count(store), 0) - def test_directory_fsync_failure_raises_typed_but_blob_published(self): - """A dir-fsync failure is reported as a typed error; the canonical blob - is already durable-published (retry after durability-report failure).""" + def test_directory_fsync_failure_typed_blob_published_no_temp(self): + """File fsync SUCCEEDS, directory fsync FAILS. Prove: canonical blob + already published; typed error returned; retry verifies the existing + blob and succeeds; no temporary file remains. The retry runs against + the HEALTHY implementation after the patch exits (no fail-once replay + is claimed under the fault).""" with tempfile.TemporaryDirectory() as td: store = ArtifactStore(Path(td)) - with mock.patch("os.fsync", side_effect=OSError("dir fsync fail")): - with self.assertRaises(InvalidPayloadError): + with mock.patch.object( + as_mod, "_fsync_dir", side_effect=OSError("dir fsync fail") + ): + with self.assertRaises(InvalidPayloadError) as ctx: store.put("pkg_demo_001", "skills/x/SKILL.md", "content") - # The blob WAS published (the failure was reporting durability), - # so a retry must verify and succeed idempotently. - d, _ = store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + self.assertIsInstance(ctx.exception, InvalidPayloadError) + d = digest_bytes(b"content") + self.assertTrue(store._blob_path(d).is_file(), "blob published despite dir fsync failure") + self.assertEqual(store._blob_path(d).read_bytes(), b"content") + self.assertEqual(_tmp_count(store), 0) + # Retry: races with the existing destination, verifies it, succeeds. + d2, _ = store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + self.assertEqual(d, d2) self.assertTrue(store.verify(d)) + def test_temp_unlink_failure_after_publication_not_silent(self): + """Unlink of the temporary file after publication fails (fail-once): + the operation MUST return a typed error, never silent success. The + outer cleanup removes the temp; the canonical blob stays published and + a retry succeeds.""" + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + real_unlink = as_mod._unlink_tmp + state = {"calls": 0} + + def flaky_unlink(path): + state["calls"] += 1 + if state["calls"] == 1: + raise OSError("unlink fail") + return real_unlink(path) + + with mock.patch.object(as_mod, "_unlink_tmp", side_effect=flaky_unlink): + with self.assertRaises(InvalidPayloadError) as ctx: + store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + self.assertIsInstance(ctx.exception, InvalidPayloadError) + d = digest_bytes(b"content") + self.assertTrue(store._blob_path(d).is_file()) + self.assertEqual(_tmp_count(store), 0) + d2, _ = store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + self.assertEqual(d, d2) + + def test_directory_close_failure_typed_blob_published(self): + """Directory fd close fails after a successful fsync: typed error, blob + already published, no temp; retry (against the healthy implementation) + succeeds.""" + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + with mock.patch.object( + as_mod, "_close_fd", side_effect=OSError("close fail") + ): + with self.assertRaises(InvalidPayloadError) as ctx: + store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + self.assertIsInstance(ctx.exception, InvalidPayloadError) + d = digest_bytes(b"content") + self.assertTrue(store._blob_path(d).is_file()) + self.assertEqual(_tmp_count(store), 0) + d2, _ = store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + self.assertEqual(d, d2) + class PublicationRaceTests(unittest.TestCase): - def test_destination_appears_between_precheck_and_publication(self): - """A raced destination that appears between the temp write and os.link - is verified (valid -> idempotent success; corrupt -> typed failure).""" + def test_coordinated_race_appears_between_write_and_publication(self): + """A destination that appears BETWEEN the temp write and os.link (via a + _hardlink side effect that creates it, then the real link raises + FileExistsError) is verified: valid -> idempotent success; corrupt -> + typed failure and the corrupt destination is NOT replaced.""" + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + + def valid_race(tmp, dest): + dest.write_bytes(b"content") # appears mid-publication + return os.link(tmp, dest) # FileExistsError + + with mock.patch.object(as_mod, "_hardlink", side_effect=valid_race): + d, _ = store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + self.assertEqual(d, digest_bytes(b"content")) + + def corrupt_race(tmp, dest): + dest.write_bytes(b"corrupt") # raced destination is corrupt + return os.link(tmp, dest) + + # Remove the existing canonical blob so the fast-path short-circuit + # does not run; the corrupt destination must appear DURING + # publication (between temp write and os.link). + store._blob_path(d).unlink() + with mock.patch.object(as_mod, "_hardlink", side_effect=corrupt_race): + with self.assertRaises(InvalidPayloadError): + store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + # Immutability: the corrupt raced destination is NOT replaced. + self.assertEqual(store._blob_path(d).read_bytes(), b"corrupt") + self.assertFalse(store.verify(d)) + + def test_existing_destination_before_put_is_verification_not_race(self): + """HONEST LABEL: the destination already exists before put() begins. + This proves the idempotent verification path (valid -> success; corrupt + -> typed failure), NOT an in-flight publication race. The coordinated + race above is the race proof.""" with tempfile.TemporaryDirectory() as td: store = ArtifactStore(Path(td)) d, _ = store.put("pkg_demo_001", "skills/x/SKILL.md", "content") - # Simulate a race: another writer creates the dest between our - # pre-check and our os.link. os.link raises FileExistsError, which - # the store treats as a race and verifies. dest = store._blob_path(d) - # valid raced destination -> idempotent success - self.assertTrue(dest.exists()) + # valid existing destination -> idempotent success d2, _ = store.put("pkg_demo_001", "skills/x/SKILL.md", "content") self.assertEqual(d, d2) - # corrupt raced destination -> typed failure + # corrupt existing destination -> typed failure, NOT replaced dest.write_bytes(b"corrupt") with self.assertRaises(InvalidPayloadError): store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + self.assertEqual(dest.read_bytes(), b"corrupt") + self.assertFalse(store.verify(d)) def test_concurrent_same_digest_writers(self): """Concurrent same-digest writers must both succeed (one publishes, @@ -111,22 +224,6 @@ def writer(): self.assertEqual(_blob_count(store), 1) self.assertTrue(store.verify(results[0][0])) - def test_valid_and_corrupt_raced_destinations(self): - with tempfile.TemporaryDirectory() as td: - store = ArtifactStore(Path(td)) - d, _ = store.put("pkg_demo_001", "skills/x/SKILL.md", "content") - dest = store._blob_path(d) - # valid raced - d2, _ = store.put("pkg_demo_001", "skills/x/SKILL.md", "content") - self.assertEqual(d, d2) - # corrupt raced -> typed failure, and the corrupt dest is NOT - # replaced (immutability): verify() is False. - dest.write_bytes(b"corrupt") - with self.assertRaises(InvalidPayloadError): - store.put("pkg_demo_001", "skills/x/SKILL.md", "content") - self.assertEqual(dest.read_bytes(), b"corrupt") - self.assertFalse(store.verify(d)) - class PublicBoundaryTests(unittest.TestCase): def test_invalid_package_id_rejected(self): @@ -142,6 +239,48 @@ def test_non_string_content_rejected(self): with self.assertRaises(InvalidPayloadError): store.put("pkg_demo_001", "skills/x/SKILL.md", 123) # type: ignore[arg-type] + def test_lone_surrogate_content_translated(self): + """content.encode() Unicode failures are translated (Finding 2).""" + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + with self.assertRaises(InvalidPayloadError): + store.put("pkg_demo_001", "skills/x/SKILL.md", "x\ud800") + + def test_non_canonical_digest_spellings_rejected(self): + """Local review (bug-4): only strict ^[0-9a-f]{64}$ digests are + accepted — 0x-prefixed, uppercase, and underscore forms are typed + errors, matching the manifest validator.""" + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + d, _ = store.put("pkg_demo_001", "skills/x/SKILL.md", "content") + self.assertTrue(store.verify(d)) + for bad in ("0x" + "a" * 62, "ABCDEF" + "a" * 58, "a_b" + "a" * 61, ""): + with self.subTest(digest=bad[:12]): + with self.assertRaises(InvalidPayloadError): + store.get(bad) + with self.assertRaises(InvalidPayloadError): + store.artifact_bytes(bad) + + def test_private_modes_enforced(self): + """Local review (sec-2): root and blobs dirs are 0o700 even under a + permissive umask, so the digest namespace is not group/world-writable.""" + old_umask = os.umask(0o002) + try: + with tempfile.TemporaryDirectory() as td: + store = ArtifactStore(Path(td)) + self.assertEqual(os.stat(store.root).st_mode & 0o777, 0o700) + self.assertEqual(os.stat(store.blobs).st_mode & 0o777, 0o700) + finally: + os.umask(old_umask) + + def test_invalid_root_types_translated(self): + """ArtifactStore construction with invalid root types is translated to + InvalidPayloadError (Finding 2), never a raw TypeError.""" + for bad in (None, 123, b"/tmp/x"): + with self.subTest(root=bad): + with self.assertRaises(InvalidPayloadError): + ArtifactStore(bad) # type: ignore[arg-type] + def test_os_error_on_init_typed(self): # A root that cannot be created (e.g. a path under an existing file) # surfaces as InvalidPayloadError, not raw OSError. From a810befea24a3ece1169e6c0649eea0dad1c6113 Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Fri, 7 Aug 2026 10:39:46 +0700 Subject: [PATCH 18/41] fix(storage): make SQLite open and schema verification exact (review 4879440857 #3) - pass the NORMALIZED Path returned by validate_store_root() throughout the open implementation (never the original string form); root normalization stays inside the typed public error boundary; proven for string roots on new stores, existing stores, and missing stores (read-only -> typed DATABASE_NOT_FOUND, nothing created); - _connect() closes the connection whenever PRAGMA setup/read-back fails, with direct spy evidence (exactly one connection opened and closed); - schema v1 verification is now semantic/exact: * unique constraints compared by EXACT column ORDER (reversed composite unique rejected); * CHECK(revision >= 0) required UNWEAKENED via normalized regex (CHECK(revision >= 0 OR 1=1) rejected); * append-only triggers verified by exact header (CREATE TRIGGER ON ), FOR EACH ROW, NO disabling WHEN clause, unconditional RAISE(ABORT), and operation-specific marker; - latest_event() accepts ONLY bytes/str stored JSON; unexpected SQLite dynamic types are MANIFEST_INVALID (no raw AttributeError); decoded JSON must be an object; decode/JSON/Unicode/type/Recursion failures translated. Adversarial tests added: reversed (action_id, package_id) unique order, CHECK(revision >= 0 OR 1=1), trigger with expected marker but WHEN 0, trigger with correct name/message but wrong operation or wrong table, PRAGMA-failure handle cleanup, string root on missing store, and non-object/non-bytes manifest_json in latest_event. --- methodfactory/storage/sqlite.py | 133 +++++++--- .../tests/test_public_error_boundary.py | 70 ++++++ methodfactory/tests/test_sqlite_open.py | 54 ++++ .../tests/test_sqlite_schema_contract.py | 234 ++++++++++++++++++ 4 files changed, 451 insertions(+), 40 deletions(-) diff --git a/methodfactory/storage/sqlite.py b/methodfactory/storage/sqlite.py index 8ba39de..972baae 100644 --- a/methodfactory/storage/sqlite.py +++ b/methodfactory/storage/sqlite.py @@ -26,7 +26,9 @@ from __future__ import annotations +import json import os +import re import sqlite3 from datetime import datetime, timezone from enum import Enum @@ -38,6 +40,7 @@ DatabaseIdMismatchError, DatabaseNotFoundError, LegacyStoreDetectedError, + ManifestInvalidError, SchemaViolationError, StorageError, UnsupportedSchemaError, @@ -146,6 +149,28 @@ REQUIRED_TRIGGERS = {"events_no_update", "events_no_delete"} +# Canonical normalized trigger bodies (Finding 3 + local review): verification +# is an EXACT normalized-body match against the DDL Method Factory ships, so +# no bypass can pass — WHERE 0, WHEN(0), marker-inside-a-literal, extra +# statements, wrong operation/table/timing all change the normalized body. +def _normalize_trigger_sql(sql: str) -> str: + """Uppercase, collapse whitespace, drop the trailing semicolon SQLite + strips from stored trigger SQL.""" + return re.sub(r"\s+", " ", sql.upper()).strip().rstrip(";").strip() + + +CANONICAL_TRIGGER_SQL: dict[str, str] = { + name: _normalize_trigger_sql(ddl) + for name, ddl in zip( + ("events_no_update", "events_no_delete"), APPEND_ONLY_TRIGGERS_DDL + ) +} + +# Normalized CHECK constraint for events.revision. The regex requires the +# closing paren immediately after `0`, so a weakened `CHECK(revision >= 0 OR +# 1=1)` does NOT match (Finding 3). +REVISION_CHECK_RE = re.compile(r"CHECK\s*\(\s*REVISION\s*>=\s*0\s*\)") + REQUIRED_METADATA = {"schema_version", "created_at"} # Current-state lookup (indexed by the composite primary key). @@ -211,7 +236,16 @@ def _connect(db: Path, read_only: bool, timeout: float = 5.0) -> sqlite3.Connect else: conn = sqlite3.connect(str(db), timeout=timeout) conn.row_factory = sqlite3.Row - _apply_or_verify_pragmas(conn, read_only=read_only) + try: + _apply_or_verify_pragmas(conn, read_only=read_only) + except BaseException: + # PRAGMA setup or read-back failure must not leak the connection + # handle (Finding 3): close before re-raising the typed error. + try: + conn.close() + except sqlite3.Error: + pass + raise return conn @@ -394,13 +428,12 @@ def _verify_schema(conn: sqlite3.Connection) -> None: r["name"] for r in conn.execute(f"PRAGMA index_info({idx['name']})") ) - # SQLite stores UNIQUE column constraints as autoindexes; - # compare as sets for single-col event_id, exact for composite. - if uniq == ("event_id",): - if idx_cols == uniq or idx_cols == ("event_id",): - found = True - elif set(idx_cols) == set(uniq): + # EXACT column ORDER required (Finding 3): a reversed composite + # unique constraint is a different contract, even though the + # column set is the same. + if idx_cols == uniq: found = True + break if not found: raise SchemaViolationError( f"table {tname!r} missing unique constraint {uniq!r}" @@ -439,11 +472,14 @@ def _verify_schema(conn: sqlite3.Connection) -> None: ) # ── CHECK constraint (Finding 3) ─────────────────────────────────── - # revision >= 0 must be present on events. Parse the CREATE TABLE SQL for - # the CHECK constraint (normalized: strip whitespace / case-insensitive). - events_sql = tables["events"].upper().replace(" ", "") - if "CHECK(REVISION>=0)" not in events_sql: - raise SchemaViolationError("events table missing CHECK (revision >= 0)") + # revision >= 0 must be present UNWEAKENED on events: the normalized + # expression must be exactly CHECK(revision >= 0) — no OR/1=1 suffix, no + # replaced constraint. Whitespace/case normalized; everything else exact. + events_sql = re.sub(r"\s+", " ", tables["events"].upper()) + if not REVISION_CHECK_RE.search(events_sql): + raise SchemaViolationError( + "events table missing exact unweakened CHECK (revision >= 0)" + ) triggers = { r["name"]: (r["sql"] or "") @@ -455,20 +491,19 @@ def _verify_schema(conn: sqlite3.Connection) -> None: f"missing append-only triggers {sorted(missing_triggers)}" ) - # ── Trigger body verification (Finding 3) ────────────────────────── - # Exact or equivalently normalized trigger definitions. A no-op trigger - # (e.g. bodies that no longer RAISE ABORT) must fail. We normalize by - # removing whitespace and lowercasing, then require the RAISE(ABORT, - # 'events are append-only: ...') marker in both triggers. + # ── Trigger contract verification (Finding 3 + local review) ────── + # EXACT normalized-body match against the canonical DDL. This is immune to + # text-substring bypasses: a trigger with the right name but a disabled + # WHEN(0)/WHERE 0 guard, a marker inside a string literal with no RAISE + # call, a wrong operation/table/timing, or any extra statement all produce + # a different normalized body and FAIL. for tname in REQUIRED_TRIGGERS: - body = triggers[tname] - if "RAISE(ABORT" not in body.upper().replace(" ", ""): - raise SchemaViolationError( - f"trigger {tname!r} does not RAISE(ABORT) (no-op or weakened)" - ) - if "APPEND-ONLY" not in body.upper().replace(" ", "").replace("_", "-"): + norm = _normalize_trigger_sql(triggers[tname]) + if norm != CANONICAL_TRIGGER_SQL[tname]: raise SchemaViolationError( - f"trigger {tname!r} message is not the append-only marker" + f"trigger {tname!r} body differs from the canonical append-only " + f"trigger (BEFORE UPDATE/DELETE ON events, FOR EACH ROW, " + f"unconditional RAISE(ABORT))" ) meta = { @@ -530,19 +565,26 @@ def open_database(root: Path | str, read_only: bool = False) -> sqlite3.Connecti PRAGMAs + identity + schema without mutation. Raw sqlite3 exceptions are translated into typed StorageError at this - public boundary (Finding 2 item 4) so no sqlite3/OS/type error escapes. + public boundary (Finding 2 item 4; local review: root normalization also + sits INSIDE the boundary, so non-str/Path roots cannot leak a raw + TypeError) and no sqlite3/OS/type error escapes. """ - r = validate_store_root(root) - db = r / DB_FILENAME - presence = detect_presence(r) - + db: Path | None = None try: - return _open_database_impl(root, db, presence, read_only) + r = validate_store_root(root) + db = r / DB_FILENAME + presence = detect_presence(r) + # The NORMALIZED root returned by validate_store_root() is passed + # throughout the open implementation (Finding 3) — never the original + # string form — so mode enforcement, mkdir, and stat all operate on + # the same resolved path. + return _open_database_impl(r, db, presence, read_only) except StorageError: raise except (sqlite3.Error, OSError, ValueError, TypeError) as exc: from .errors import StorageError as _SE - raise _SE(f"storage open failed for {db}: {exc}") from exc + target = db if db is not None else repr(root) + raise _SE(f"storage open failed for {target}: {exc}") from exc def _open_database_impl( @@ -607,9 +649,10 @@ def _open_database_impl( def latest_event(conn: sqlite3.Connection, package_id: str) -> dict | None: """Return the latest manifest for a package (indexed latest-event read). - Public boundary (Finding 4): a malformed/invalid-UTF-8 manifest_json BLOB - surfaces as a typed StorageError (code MANIFEST_INVALID), never a raw - json/Unicode/type exception. + Public boundary (Finding 3/Finding 4): ONLY bytes or strings are accepted + as stored JSON; unexpected SQLite dynamic types (int/float/None) are + translated to MANIFEST_INVALID; the decoded JSON must be an OBJECT; no + raw AttributeError/JSON/Unicode/SQLite/type exception escapes. """ try: row = conn.execute(LATEST_EVENT_SQL, (package_id,)).fetchone() @@ -617,17 +660,27 @@ def latest_event(conn: sqlite3.Connection, package_id: str) -> dict | None: raise StorageError(f"latest_event query failed: {exc}") from exc if row is None: return None - import json raw = row["manifest_json"] + if not isinstance(raw, (bytes, str)): + raise ManifestInvalidError( + f"manifest_json for {package_id} has unexpected stored type " + f"{type(raw).__name__} (expected bytes or str)" + ) try: if isinstance(raw, str): raw = raw.encode("utf-8") - return json.loads(raw.decode("utf-8")) - except (UnicodeDecodeError, json.JSONDecodeError, TypeError, ValueError) as exc: - from .errors import ManifestInvalidError as _MIV - - raise _MIV(f"manifest_json corrupt for {package_id}: {exc}") from exc + decoded = json.loads(raw.decode("utf-8")) + except (UnicodeEncodeError, UnicodeDecodeError, json.JSONDecodeError, + TypeError, ValueError, RecursionError) as exc: + raise ManifestInvalidError( + f"manifest_json corrupt for {package_id}: {exc}" + ) from exc + if not isinstance(decoded, dict): + raise ManifestInvalidError( + f"manifest_json for {package_id} is not a JSON object" + ) + return decoded def explain_latest_event_plan(conn: sqlite3.Connection, package_id: str) -> list[tuple]: diff --git a/methodfactory/tests/test_public_error_boundary.py b/methodfactory/tests/test_public_error_boundary.py index 881fe26..ab4a344 100644 --- a/methodfactory/tests/test_public_error_boundary.py +++ b/methodfactory/tests/test_public_error_boundary.py @@ -84,6 +84,76 @@ def test_missing_package_returns_none(self): finally: close_database(conn) + def test_non_bytes_stored_type_translated(self): + """A manifest_json column holding a non-bytes/non-str SQLite dynamic + type (e.g. INTEGER) is translated to MANIFEST_INVALID, never a raw + AttributeError (Finding 3).""" + import sqlite3 + + root = Path(tempfile.mkdtemp()) + conn = open_database(root, read_only=False) + close_database(conn) + c = sqlite3.connect(str(root / DB_FILENAME)) + c.execute( + "INSERT INTO events (package_id, revision, event_id, action_id, action, " + "action_sha256, state_before, state_after, previous_manifest_sha256, " + "resulting_manifest_sha256, created_at, action_json, manifest_json) " + "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", + ( + "pkg_int_001", 0, "evt_int", "act_int", "create_package", "0" * 64, + None, "INTAKE", None, "0" * 64, "2026-08-07T00:00:00+00:00", + b'{"action":"create_package"}', 12345, + ), + ) + c.commit() + c.close() + conn = open_database(root, read_only=True) + try: + with self.assertRaises(MethodFactoryError) as ctx: + latest_event(conn, "pkg_int_001") + self.assertEqual(ctx.exception.code, "MANIFEST_INVALID") + finally: + close_database(conn) + + def test_non_object_json_translated(self): + """A manifest_json BLOB containing a JSON ARRAY (not an object) is + translated to MANIFEST_INVALID (Finding 3).""" + root = self._db_with_manifest(b'["not", "an", "object"]') + conn = open_database(root, read_only=True) + try: + with self.assertRaises(MethodFactoryError) as ctx: + latest_event(conn, "pkg_demo_001") + self.assertEqual(ctx.exception.code, "MANIFEST_INVALID") + finally: + close_database(conn) + + def test_str_manifest_json_accepted(self): + """A TEXT-typed manifest_json is accepted and decoded (Finding 3).""" + import sqlite3 + + root = Path(tempfile.mkdtemp()) + conn = open_database(root, read_only=False) + close_database(conn) + c = sqlite3.connect(str(root / DB_FILENAME)) + c.execute( + "INSERT INTO events (package_id, revision, event_id, action_id, action, " + "action_sha256, state_before, state_after, previous_manifest_sha256, " + "resulting_manifest_sha256, created_at, action_json, manifest_json) " + "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", + ( + "pkg_str_001", 0, "evt_str", "act_str", "create_package", "0" * 64, + None, "INTAKE", None, "0" * 64, "2026-08-07T00:00:00+00:00", + b'{"action":"create_package"}', '{"schema_version":"0.1"}', + ), + ) + c.commit() + c.close() + conn = open_database(root, read_only=True) + try: + self.assertEqual(latest_event(conn, "pkg_str_001"), {"schema_version": "0.1"}) + finally: + close_database(conn) + class ArtifactBoundaryTests(unittest.TestCase): def test_invalid_utf8_blob_get_typed(self): diff --git a/methodfactory/tests/test_sqlite_open.py b/methodfactory/tests/test_sqlite_open.py index 89a571d..42cf64d 100644 --- a/methodfactory/tests/test_sqlite_open.py +++ b/methodfactory/tests/test_sqlite_open.py @@ -307,5 +307,59 @@ def test_existing_incorrect_mode_corrected(self): self.assertEqual(os.stat(root / DB_FILENAME).st_mode & 0o777, 0o600) +class StringRootTests(unittest.TestCase): + """Finding 3: the public root argument may be a plain string. The + normalized Path returned by validate_store_root() is passed throughout the + open implementation (never the original string form), and root + normalization lives inside the typed public error boundary.""" + + def test_string_root_creates_new_store(self): + with tempfile.TemporaryDirectory() as td: + root_str = str(Path(td) / "store") + conn = open_database(root_str, read_only=False) + close_database(conn) + self.assertTrue((Path(root_str) / DB_FILENAME).is_file()) + + def test_string_root_reopens_existing_store(self): + with tempfile.TemporaryDirectory() as td: + root = Path(td) + conn0 = open_database(root, read_only=False) + close_database(conn0) + conn = open_database(str(root), read_only=False) + try: + self.assertEqual( + int(conn.execute("PRAGMA user_version").fetchone()[0]), USER_VERSION + ) + self.assertEqual( + conn.execute("PRAGMA journal_mode").fetchone()[0].lower(), "delete" + ) + finally: + close_database(conn) + + def test_string_root_missing_store_readonly_not_found(self): + """String root on a missing store: typed DatabaseNotFoundError and NO + directory/database created (read-only never creates).""" + with tempfile.TemporaryDirectory() as td: + missing = Path(td) / "missing" + with self.assertRaises(DatabaseNotFoundError): + open_database(str(missing), read_only=True) + self.assertFalse(missing.exists()) + + def test_invalid_string_root_typed(self): + """Empty string root surfaces as typed InvalidStoreRootError, not a raw + pathlib/OS error.""" + from methodfactory.storage.errors import InvalidStoreRootError + with self.assertRaises(InvalidStoreRootError): + open_database(" ", read_only=False) + + def test_non_path_root_typed(self): + """Local review (bug-2): non-str/Path roots (None, int, float) surface + as typed StorageError, never a raw TypeError.""" + for bad in (None, 123, 3.14): + with self.subTest(root=bad): + with self.assertRaises(StorageError): + open_database(bad, read_only=False) # type: ignore[arg-type] + + if __name__ == "__main__": unittest.main() diff --git a/methodfactory/tests/test_sqlite_schema_contract.py b/methodfactory/tests/test_sqlite_schema_contract.py index 9a73b9f..b5a07f7 100644 --- a/methodfactory/tests/test_sqlite_schema_contract.py +++ b/methodfactory/tests/test_sqlite_schema_contract.py @@ -68,6 +68,124 @@ def test_noop_trigger_rejected(self): with self.assertRaises(SchemaViolationError): open_database(root, read_only=False) + def test_trigger_with_when_zero_rejected(self): + """A trigger with the expected NAME, operation, table, and RAISE(ABORT) + marker but a disabling WHEN 0 clause must fail: it never fires, so the + append-only guarantee is gone.""" + with tempfile.TemporaryDirectory() as td: + root = Path(td) + conn = _make_valid_db(root) + close_database(conn) + c = _open_conn(root / DB_FILENAME) + c.execute("DROP TRIGGER events_no_update") + c.execute( + "CREATE TRIGGER events_no_update BEFORE UPDATE ON events " + "FOR EACH ROW WHEN 0 BEGIN " + "SELECT RAISE(ABORT, 'events are append-only: UPDATE not permitted'); END" + ) + c.commit() + c.close() + with self.assertRaises(SchemaViolationError): + open_database(root, read_only=False) + + def test_trigger_wrong_operation_rejected(self): + """events_no_update recreated with the correct message but the WRONG + operation (INSERT instead of UPDATE) must fail.""" + with tempfile.TemporaryDirectory() as td: + root = Path(td) + conn = _make_valid_db(root) + close_database(conn) + c = _open_conn(root / DB_FILENAME) + c.execute("DROP TRIGGER events_no_update") + c.execute( + "CREATE TRIGGER events_no_update BEFORE INSERT ON events " + "FOR EACH ROW BEGIN " + "SELECT RAISE(ABORT, 'events are append-only: UPDATE not permitted'); END" + ) + c.commit() + c.close() + with self.assertRaises(SchemaViolationError): + open_database(root, read_only=False) + + def test_trigger_wrong_table_rejected(self): + """events_no_delete recreated on the WRONG table with the correct + message must fail.""" + with tempfile.TemporaryDirectory() as td: + root = Path(td) + conn = _make_valid_db(root) + close_database(conn) + c = _open_conn(root / DB_FILENAME) + c.execute("DROP TRIGGER events_no_delete") + c.execute("CREATE TABLE other (x TEXT)") # trigger must target events + c.execute( + "CREATE TRIGGER events_no_delete BEFORE DELETE ON other " + "FOR EACH ROW BEGIN " + "SELECT RAISE(ABORT, 'events are append-only: DELETE not permitted'); END" + ) + c.commit() + c.close() + with self.assertRaises(SchemaViolationError): + open_database(root, read_only=False) + + def test_trigger_where_zero_guarded_raise_rejected(self): + """Local review (bug-3/sec-1): a RAISE(ABORT) behind WHERE 0 is a + no-op that passes substring checks; exact-body verification must + reject it.""" + with tempfile.TemporaryDirectory() as td: + root = Path(td) + conn = _make_valid_db(root) + close_database(conn) + c = _open_conn(root / DB_FILENAME) + c.execute("DROP TRIGGER events_no_update") + c.execute( + "CREATE TRIGGER events_no_update BEFORE UPDATE ON events " + "FOR EACH ROW BEGIN " + "SELECT RAISE(ABORT, 'events are append-only: UPDATE not permitted') " + "WHERE 0; END" + ) + c.commit() + c.close() + with self.assertRaises(SchemaViolationError): + open_database(root, read_only=False) + + def test_trigger_when_no_space_rejected(self): + """Local review (q-1): WHEN(0) with no space bypasses a ' WHEN ' + substring guard; exact-body verification must reject it.""" + with tempfile.TemporaryDirectory() as td: + root = Path(td) + conn = _make_valid_db(root) + close_database(conn) + c = _open_conn(root / DB_FILENAME) + c.execute("DROP TRIGGER events_no_update") + c.execute( + "CREATE TRIGGER events_no_update BEFORE UPDATE ON events " + "FOR EACH ROW WHEN(0) BEGIN " + "SELECT RAISE(ABORT, 'events are append-only: UPDATE not permitted'); END" + ) + c.commit() + c.close() + with self.assertRaises(SchemaViolationError): + open_database(root, read_only=False) + + def test_trigger_marker_in_literal_only_rejected(self): + """Local review (sec-1): the marker strings inside a string literal + with NO RAISE call must fail verification.""" + with tempfile.TemporaryDirectory() as td: + root = Path(td) + conn = _make_valid_db(root) + close_database(conn) + c = _open_conn(root / DB_FILENAME) + c.execute("DROP TRIGGER events_no_update") + c.execute( + "CREATE TRIGGER events_no_update BEFORE UPDATE ON events " + "FOR EACH ROW BEGIN " + "SELECT 'events are append-only: UPDATE not permitted'; END" + ) + c.commit() + c.close() + with self.assertRaises(SchemaViolationError): + open_database(root, read_only=False) + class CheckConstraintTests(unittest.TestCase): def test_weakened_check_constraint_rejected(self): @@ -95,6 +213,63 @@ def test_weakened_check_constraint_rejected(self): with self.assertRaises(SchemaViolationError): open_database(root, read_only=False) + def test_weakened_check_with_or_rejected(self): + """CHECK(revision >= 0 OR 1=1) is a WEAKENED constraint and must fail + verification: the unweakened exact CHECK(revision >= 0) is required + (Finding 3).""" + with tempfile.TemporaryDirectory() as td: + root = Path(td) + conn = _make_valid_db(root) + close_database(conn) + c = _open_conn(root / DB_FILENAME) + c.execute("DROP TABLE events") + c.execute( + "CREATE TABLE events (" + "package_id TEXT NOT NULL, revision INTEGER NOT NULL, " + "event_id TEXT NOT NULL, action_id TEXT NOT NULL, " + "action TEXT NOT NULL, action_sha256 TEXT NOT NULL, " + "state_before TEXT, state_after TEXT NOT NULL, " + "previous_manifest_sha256 TEXT, resulting_manifest_sha256 TEXT NOT NULL, " + "created_at TEXT NOT NULL, action_json BLOB NOT NULL, manifest_json BLOB NOT NULL, " + "PRIMARY KEY (package_id, revision), UNIQUE (package_id, action_id), UNIQUE (event_id), " + "CHECK (revision >= 0 OR 1=1)" # weakened + ") WITHOUT ROWID" + ) + c.commit() + c.close() + with self.assertRaises(SchemaViolationError): + open_database(root, read_only=False) + + +class UniqueOrderTests(unittest.TestCase): + def test_reversed_unique_order_rejected(self): + """UNIQUE (action_id, package_id) is a DIFFERENT constraint than + UNIQUE (package_id, action_id): exact column ORDER is required + (Finding 3).""" + with tempfile.TemporaryDirectory() as td: + root = Path(td) + conn = _make_valid_db(root) + close_database(conn) + c = _open_conn(root / DB_FILENAME) + c.execute("DROP TABLE events") + c.execute( + "CREATE TABLE events (" + "package_id TEXT NOT NULL, revision INTEGER NOT NULL, " + "event_id TEXT NOT NULL, action_id TEXT NOT NULL, " + "action TEXT NOT NULL, action_sha256 TEXT NOT NULL, " + "state_before TEXT, state_after TEXT NOT NULL, " + "previous_manifest_sha256 TEXT, resulting_manifest_sha256 TEXT NOT NULL, " + "created_at TEXT NOT NULL, action_json BLOB NOT NULL, manifest_json BLOB NOT NULL, " + "PRIMARY KEY (package_id, revision), " + "UNIQUE (action_id, package_id), UNIQUE (event_id), " # REVERSED + "CHECK (revision >= 0)" + ") WITHOUT ROWID" + ) + c.commit() + c.close() + with self.assertRaises(SchemaViolationError): + open_database(root, read_only=False) + class ColumnContractTests(unittest.TestCase): def test_changed_type_rejected(self): @@ -237,6 +412,65 @@ def test_failed_open_closes_connection(self): with self.assertRaises(SchemaViolationError): open_database(root, read_only=True) + def test_pragma_failure_closes_connection_direct_evidence(self): + """DIRECT handle evidence (Finding 3): when PRAGMA setup/read-back + fails inside _connect(), the connection is closed before the typed + error propagates — verified with a connection spy, not fd counts.""" + import methodfactory.storage.sqlite as sqlite_mod + from methodfactory.storage.sqlite import _connect + from unittest import mock + + real_connect = sqlite3.connect + opens: list = [] + + class SpyConn: + def __init__(self, real): + self._real = real + self.closed = False + + def __getattr__(self, name): + return getattr(self._real, name) + + def close(self): + self.closed = True + return self._real.close() + + def spy_connect(*args, **kwargs): + real = real_connect(*args, **kwargs) + spy = SpyConn(real) + opens.append(spy) + return spy + + with tempfile.TemporaryDirectory() as td: + db = Path(td) / DB_FILENAME + with mock.patch.object( + sqlite_mod, + "_apply_or_verify_pragmas", + side_effect=StorageError("pragma fail"), + ), mock.patch( + "methodfactory.storage.sqlite.sqlite3.connect", side_effect=spy_connect + ): + with self.assertRaises(StorageError): + _connect(db, read_only=False) + self.assertEqual(len(opens), 1, "exactly one connection was opened") + self.assertTrue(opens[0].closed, "connection must be closed on PRAGMA failure") + + def test_pragma_failure_through_public_open_typed(self): + """PRAGMA failure surfaces as a typed StorageError through the public + open_database() boundary and the handle is closed.""" + import methodfactory.storage.sqlite as sqlite_mod + from unittest import mock + + with tempfile.TemporaryDirectory() as td: + root = Path(td) + with mock.patch.object( + sqlite_mod, + "_apply_or_verify_pragmas", + side_effect=StorageError("pragma fail"), + ): + with self.assertRaises(StorageError): + open_database(root, read_only=False) + if __name__ == "__main__": unittest.main() From ea5b054cd2f2385985e8ea6bfa6368dfe0eb78c5 Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Fri, 7 Aug 2026 10:50:04 +0700 Subject: [PATCH 19/41] fix(boundary): freeze the actual public error surface (review 4879440857 #4) Enumerate the supported public API in docs/public-surface.md with accepted input types, success results, stable Method Factory error classes/codes, and the native exceptions each operation translates. No raw JSON/Unicode/ recursion/type/sqlite3/OS exception escapes a supported public operation: - action_sha256() now raises SerializationError (code SERIALIZATION) for unsupported JSON types, deep recursion, lone-surrogate encoding, and canonical byte overflow (previously leaked ValueError/UnicodeEncodeError/ RecursionError); - envelope_from_dict() translates non-dict input to InvalidEnvelopeError; - explain_latest_event_plan() and close_database() translate sqlite3 errors to StorageError; - low-level serialization primitives and sqlite internals are documented as internal with their native exception contracts (not part of the supported surface). Consistency rule stated: validate_manifest() collects violations (list), parse_envelope()/action_sha256()/ArtifactStore/storage helpers raise typed errors - both stable, neither leaks raw native exceptions. Tests: SerializationError for over-limit/lone-surrogate/non-serializable action payloads, envelope_from_dict non-dict translation, sqlite3-error translation in explain/close, nested string-root open. --- docs/public-surface.md | 52 ++++++++++++++ methodfactory/protocol/envelope.py | 4 ++ methodfactory/storage/errors.py | 12 ++++ methodfactory/storage/serialization.py | 20 ++++-- methodfactory/storage/sqlite.py | 25 +++++-- methodfactory/tests/test_boundary_model.py | 7 +- .../tests/test_public_error_boundary.py | 70 ++++++++++++++++++- 7 files changed, 174 insertions(+), 16 deletions(-) create mode 100644 docs/public-surface.md diff --git a/docs/public-surface.md b/docs/public-surface.md new file mode 100644 index 0000000..501cc23 --- /dev/null +++ b/docs/public-surface.md @@ -0,0 +1,52 @@ +# Public Surface and Stable Error Contract + +Phase 2 foundation closure (review 4879440857, Finding 4). This table is the +authoritative public error boundary for the currently supported package and +storage APIs. + +**Boundary rule:** every supported public operation surfaces **only** +`methodfactory.domain.errors.MethodFactoryError` subclasses (stable +machine-readable `code` per ADR-0008 / ADR-0012 §B). Raw +JSON/Unicode/recursion/type/sqlite3/OS exceptions are translated at the +public boundary. Internal primitives are documented below with their native +exception contract and are not part of the supported surface. + +## Supported public surface + +| Operation | Accepted inputs | Success result | Stable errors (code) | Native exceptions translated | +|---|---|---|---|---| +| `parse_envelope(raw)` | `str` (raw envelope, optional prose) | `ActionEnvelope` | `InvalidEnvelopeError` (`INVALID_ENVELOPE`) | `UnicodeEncodeError` (lone surrogate), `json.JSONDecodeError`, `RecursionError`, `TypeError` (non-string) | +| `envelope_from_dict(d)` | `dict` | `ActionEnvelope` | `InvalidEnvelopeError` (`INVALID_ENVELOPE`) | `TypeError` (non-dict), plus every envelope field failure | +| `validate_manifest(manifest)` | `dict` | `list[str]` of violations (`[]` = valid) | none raised; failures collected | `TypeError`, `RecursionError`, `UnicodeEncodeError`, `ValueError` (canonicalization -> collected error) | +| `action_sha256(**semantic)` | str/str/str/str + `dict` basis + `dict` payload | `str` (64-hex digest) | `SerializationError` (`SERIALIZATION`) | `TypeError`, `RecursionError`, `UnicodeEncodeError`, `ValueError` (over `MAX_ACTION_JSON_BYTES`) | +| `ArtifactStore(root)` | `str`/`os.PathLike` root | store object | `InvalidPayloadError` (`INVALID_PAYLOAD`), `InvalidStoreRootError` via path helpers | `TypeError`/`ValueError` (root type), `OSError` (mkdir) | +| `ArtifactStore.put(package_id, logical_path, content)` | str/str/str | `(digest: str, size_bytes: int)` | `InvalidPayloadError` (`INVALID_PAYLOAD`), `InvalidPackageIdError` (`INVALID_PACKAGE_ID`) | `UnicodeEncodeError` (lone surrogate), `OSError` (open/write/fsync/link/unlink/dir-fsync/close) | +| `ArtifactStore.get(digest)` | str (64-hex) | `str` (decoded UTF-8) | `InvalidPayloadError` (`INVALID_PAYLOAD`) | `OSError` (read), `UnicodeDecodeError`, `ValueError` (bad digest) | +| `ArtifactStore.artifact_bytes(digest)` | str (64-hex) | `bytes` | `InvalidPayloadError` (`INVALID_PAYLOAD`) | `OSError` (read), `ValueError` (bad digest) | +| `ArtifactStore.verify(digest)` | str (64-hex) | `bool` (never raises) | — | all failures collapse to `False` | +| `open_database(root, read_only=False)` | `str`/`Path` root + bool | `sqlite3.Connection` (via `close_database`) | `DatabaseNotFoundError` (`DATABASE_NOT_FOUND`), `DatabaseEmptyError` (`DATABASE_EMPTY`), `DatabaseIdMismatchError` (`DATABASE_ID_MISMATCH`), `UnsupportedSchemaError` (`UNSUPPORTED_SCHEMA`), `LegacyStoreDetectedError` (`LEGACY_STORE_DETECTED`), `SchemaViolationError` (`SCHEMA_VIOLATION`), `StorageError` (`STORAGE_ERROR`) | `sqlite3.Error`, `OSError`, `ValueError`, `TypeError` -> `StorageError`; invalid root -> `InvalidStoreRootError` (`INVALID_STORE_ROOT`) | +| `latest_event(conn, package_id)` | `sqlite3.Connection` (from `open_database`) + str | `dict | None` | `ManifestInvalidError` (`MANIFEST_INVALID`), `StorageError` (`STORAGE_ERROR`) | `sqlite3.Error` (query), `UnicodeDecodeError`/`UnicodeEncodeError`, `json.JSONDecodeError`, `TypeError`, `ValueError`, `RecursionError`, non-bytes/str stored type, non-object JSON | +| `explain_latest_event_plan(conn, package_id)` | `sqlite3.Connection` + str | `list[tuple]` | `StorageError` (`STORAGE_ERROR`) | `sqlite3.Error` | +| `close_database(conn)` | `sqlite3.Connection` | `None` | `StorageError` (`STORAGE_ERROR`) | `sqlite3.Error` | + +## Internal primitives (documented native contract, NOT public) + +| Primitive | Native contract | +|---|---| +| `storage.serialization.canonical_json` / `canonical_bytes` | Raise `TypeError` (unsupported types), `ValueError` (NaN/Infinity), `RecursionError` (deep nesting); `canonical_bytes` additionally `UnicodeEncodeError` (lone surrogate) | +| `storage.serialization.canonical_bytes_bounded` | As above plus `ValueError` when canonical bytes exceed the bound | +| `storage.serialization.try_canonical_bytes_bounded` | Returns `(bytes, None)` or `(None, error_str)` — never raises | +| `storage.serialization.digest_*` | SHA-256 helpers; native contract inherited from `canonical_bytes`/encoding | +| `storage.sqlite._connect` / `_apply_or_verify_pragmas` / `_identity` / `_verify_schema` / `initialize_database` / `detect_presence` / `_open_database_impl` | Internal to `open_database`; sqlite3 errors may surface if called directly. `_connect` closes its connection on PRAGMA failure. | + +## Consistency rule + +Validation failure handling is deliberate and split by surface: + +- `validate_manifest()` **collects** violations into a `list[str]` (read-only + validator; caller-friendly). +- `parse_envelope()` / `envelope_from_dict()` / `action_sha256()` / + `ArtifactStore` / storage helpers **raise** typed errors (first failure + aborts the operation). + +Both surfaces are stable; neither leaks raw native exceptions. diff --git a/methodfactory/protocol/envelope.py b/methodfactory/protocol/envelope.py index f56a38d..2b1bcf8 100644 --- a/methodfactory/protocol/envelope.py +++ b/methodfactory/protocol/envelope.py @@ -141,6 +141,10 @@ def parse_envelope(raw: str) -> ActionEnvelope: def envelope_from_dict(d: dict) -> ActionEnvelope: + if not isinstance(d, dict): + raise InvalidEnvelopeError( + f"envelope dict must be a JSON object, got {type(d).__name__}" + ) _validate_envelope_dict(d) return ActionEnvelope( protocol_version=d["protocol_version"], diff --git a/methodfactory/storage/errors.py b/methodfactory/storage/errors.py index 81630b0..7910293 100644 --- a/methodfactory/storage/errors.py +++ b/methodfactory/storage/errors.py @@ -109,3 +109,15 @@ class ManifestInvalidError(StorageError): """A stored manifest BLOB is malformed or invalid UTF-8 (Finding 4).""" code = "MANIFEST_INVALID" + + +class SerializationError(StorageError): + """A value cannot be canonicalized or exceeds its canonical byte bound. + + Public boundary for action_sha256 (Finding 4): unsupported JSON types, + excessive recursion, lone-surrogate encoding, and canonical-size overflow + are all translated into this typed error — never leaked as raw + TypeError/RecursionError/UnicodeEncodeError/ValueError. + """ + + code = "SERIALIZATION" diff --git a/methodfactory/storage/serialization.py b/methodfactory/storage/serialization.py index 31e8a43..b86cf2e 100644 --- a/methodfactory/storage/serialization.py +++ b/methodfactory/storage/serialization.py @@ -18,6 +18,7 @@ import json from typing import Any +from .errors import SerializationError from .limits import MAX_ACTION_JSON_BYTES @@ -126,6 +127,12 @@ def action_sha256( The normalized semantic action is canonicalized ONCE, enforced against MAX_ACTION_JSON_BYTES, and hashed from those exact accepted bytes. + + Public error boundary (Finding 4): every native failure — unsupported + JSON types, excessive recursion, lone-surrogate encoding, and canonical + byte overflow — is translated into SerializationError (a public + MethodFactoryError with code SERIALIZATION); no raw + TypeError/RecursionError/UnicodeEncodeError/ValueError escapes. """ semantic = { "protocol_version": protocol_version, @@ -135,9 +142,12 @@ def action_sha256( "basis": basis, "payload": payload, } - canonical = canonical_bytes_bounded( - semantic, - limit=MAX_ACTION_JSON_BYTES, - what="canonical action", - ) + try: + canonical = canonical_bytes_bounded( + semantic, + limit=MAX_ACTION_JSON_BYTES, + what="canonical action", + ) + except (TypeError, RecursionError, UnicodeEncodeError, ValueError) as exc: + raise SerializationError(f"cannot canonicalize action: {exc}") from exc return sha256_hex(canonical) diff --git a/methodfactory/storage/sqlite.py b/methodfactory/storage/sqlite.py index 972baae..8e4bbe8 100644 --- a/methodfactory/storage/sqlite.py +++ b/methodfactory/storage/sqlite.py @@ -684,12 +684,27 @@ def latest_event(conn: sqlite3.Connection, package_id: str) -> dict | None: def explain_latest_event_plan(conn: sqlite3.Connection, package_id: str) -> list[tuple]: - """EXPLAIN QUERY PLAN for the latest-event lookup (ADR-0012 §9 item 14).""" - rows = conn.execute( - f"EXPLAIN QUERY PLAN {LATEST_EVENT_SQL}", (package_id,) - ).fetchall() + """EXPLAIN QUERY PLAN for the latest-event lookup (ADR-0012 §9 item 14). + + Public boundary (Finding 4): sqlite3 failures are translated to typed + StorageError; no raw sqlite3.Error escapes. + """ + try: + rows = conn.execute( + f"EXPLAIN QUERY PLAN {LATEST_EVENT_SQL}", (package_id,) + ).fetchall() + except sqlite3.Error as exc: + raise StorageError(f"explain latest-event plan failed: {exc}") from exc return [tuple(r) for r in rows] def close_database(conn: sqlite3.Connection) -> None: - conn.close() + """Close a connection returned by open_database(). + + Public boundary (Finding 4): a sqlite3 close failure is translated to + typed StorageError; no raw sqlite3.Error escapes. + """ + try: + conn.close() + except sqlite3.Error as exc: + raise StorageError(f"cannot close database: {exc}") from exc diff --git a/methodfactory/tests/test_boundary_model.py b/methodfactory/tests/test_boundary_model.py index ed597b6..c62fa49 100644 --- a/methodfactory/tests/test_boundary_model.py +++ b/methodfactory/tests/test_boundary_model.py @@ -12,6 +12,7 @@ from methodfactory.domain.errors import InvalidEnvelopeError from methodfactory.protocol.envelope import parse_envelope +from methodfactory.storage.errors import SerializationError from methodfactory.storage.limits import ( MAX_ACTION_JSON_BYTES, MAX_ARTIFACT_BODY_BYTES, @@ -214,13 +215,13 @@ def test_action_hash_exact_at_and_one_over(self): over = dict(semantic) over["payload"] = dict(semantic["payload"], content="x" * (k + 1)) self.assertEqual(len(canonical_bytes(over)), MAX_ACTION_JSON_BYTES + 1) - with self.assertRaises(ValueError): + with self.assertRaises(SerializationError): action_sha256(**over) def test_action_hash_unicode_and_recursion_failures_do_not_leak_raw(self): """Non-canonicalizable semantic payloads (lone surrogate, deep - recursion) must surface as a controlled error, never a raw - UnicodeEncodeError/RecursionError.""" + recursion) must surface as typed SerializationError, never a raw + UnicodeEncodeError/RecursionError (Finding 4).""" sur = { "protocol_version": "0.1", "action": "record_input", "package_id": "pkg_demo_001", "action_id": "act_1", diff --git a/methodfactory/tests/test_public_error_boundary.py b/methodfactory/tests/test_public_error_boundary.py index ab4a344..e4f89d1 100644 --- a/methodfactory/tests/test_public_error_boundary.py +++ b/methodfactory/tests/test_public_error_boundary.py @@ -1,4 +1,5 @@ -"""Complete public error-boundary tests (Finding 4, review 4879090471). +"""Complete public error-boundary tests (Finding 4, review 4879090471; +closure review 4879440857). Every currently exposed storage/artifact operation must surface typed MethodFactoryError (never raw sqlite3/JSON/Unicode/OS/type), with specific @@ -12,12 +13,15 @@ from pathlib import Path from methodfactory.adapters.artifact_store import ArtifactStore -from methodfactory.domain.errors import MethodFactoryError -from methodfactory.storage.errors import ManifestInvalidError +from methodfactory.domain.errors import InvalidEnvelopeError, MethodFactoryError +from methodfactory.protocol.envelope import envelope_from_dict +from methodfactory.storage.errors import ManifestInvalidError, SerializationError, StorageError from methodfactory.storage.paths import DB_FILENAME +from methodfactory.storage.serialization import action_sha256 from methodfactory.storage.sqlite import ( APPLICATION_ID, close_database, + explain_latest_event_plan, latest_event, open_database, ) @@ -193,5 +197,65 @@ def test_corrupt_blob_get_typed(self): store.get(d) +class SerializationBoundaryTests(unittest.TestCase): + def _semantic(self, **over): + s = { + "protocol_version": "0.1", + "action": "record_input", + "package_id": "pkg_demo_001", + "action_id": "act_1", + "basis": {}, + "payload": {"content": "x"}, + } + s.update(over) + return s + + def test_over_limit_action_typed(self): + with self.assertRaises(SerializationError) as ctx: + action_sha256(**self._semantic(payload={"content": "x" * (4 * 1024 * 1024)})) + self.assertEqual(ctx.exception.code, "SERIALIZATION") + + def test_lone_surrogate_action_typed(self): + with self.assertRaises(SerializationError) as ctx: + action_sha256(**self._semantic(payload={"content": "x\ud800"})) + self.assertEqual(ctx.exception.code, "SERIALIZATION") + + def test_non_serializable_payload_typed(self): + with self.assertRaises(SerializationError) as ctx: + action_sha256(**self._semantic(payload={"content": object()})) + self.assertEqual(ctx.exception.code, "SERIALIZATION") + + +class EnvelopeFromDictBoundaryTests(unittest.TestCase): + def test_non_dict_input_translated(self): + for bad in (None, "x", 5, []): + with self.subTest(value=bad): + with self.assertRaises(InvalidEnvelopeError): + envelope_from_dict(bad) # type: ignore[arg-type] + + +class SqliteHelperBoundaryTests(unittest.TestCase): + def test_explain_latest_event_plan_sqlite_error_typed(self): + class BrokenConn: + def execute(self, *args, **kwargs): + raise __import__("sqlite3").Error("boom") + + with self.assertRaises(MethodFactoryError): + explain_latest_event_plan(BrokenConn(), "pkg_demo_001") # type: ignore[arg-type] + + def test_close_database_sqlite_error_typed(self): + class BrokenConn: + def close(self): + raise __import__("sqlite3").Error("boom") + + with self.assertRaises(MethodFactoryError): + close_database(BrokenConn()) # type: ignore[arg-type] + + def test_public_open_database_accepts_string_and_returns_closable(self): + with tempfile.TemporaryDirectory() as td: + conn = open_database(str(Path(td) / "nested" / "store"), read_only=False) + close_database(conn) + + if __name__ == "__main__": unittest.main() From 91468724b7547303fb9df051cd586e1c346e016c Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Fri, 7 Aug 2026 18:27:07 +0700 Subject: [PATCH 20/41] =?UTF-8?q?feat(storage):=20transactional=20create/l?= =?UTF-8?q?oad/apply=20with=20idempotent=20replay=20and=20authoritative=20?= =?UTF-8?q?chain=20validator=20(ADR-0012=20=C2=A76/=C2=A78)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements the authorized transactional persistence slice on the frozen SQLite model. Every mutation executes one bounded BEGIN IMMEDIATE transaction: - engine/apply.py: the single deterministic transition rule (legality via domain.transitions -> gate via domain.gates -> per-action mutation -> revision/state/lineage). Pure and persistence-free, but shares the storage canonical serialization so engine digests can never drift from stored hashes. - storage/chain.py: the SINGLE invariant kernel (check_event_invariants + check_current_row_consistency) reused by transactional apply AND the authoritative validate_chain - no divergent validity implementations. Revision-zero contract, lineage, state binding, digest binding, grammar, schema re-validation (audit mode), and referenced-artifact verification. - storage/store.py: SqliteManifestStore.create/apply/load/read_events/ validate_chain. Idempotency lookup (package_id, action_id) happens BEFORE stale-revision rejection; same hash replays the previously committed (consistency-verified) result, different hash raises ACTION_ID_CONFLICT; stale revision raises STALE_ACTION. Rollback on any pre-commit failure: no event inserted, no historical row mutated, prewritten content-addressed blobs kept. BEGIN IMMEDIATE lock contention surfaces as ConcurrencyError via sqlite errorcode. Package creation is frozen: revision 0, state_before NULL, no predecessor hash, one event, exact replay only. - serialization.py: semantic_action + canonical_action_bytes as the single construction point so stored action JSON == hashed action bytes. - gates.py: aligned to the content-addressed summary (summary.digest), with isinstance guards. - envelope.py: action_id grammar enforced at the untrusted-input boundary. - errors: PACKAGE_NOT_FOUND, ARTIFACT_VERIFICATION, CHAIN_VIOLATION. - sqlite.py: LATEST_EVENT_SQL returns the full row; latest_event_row helper. - docs/public-surface.md: transactional operations, error table, and the authoritative 13-step transaction algorithm. No migration, export, lifecycle expansion, lock files, journal framing, or second canonical state. PR #1 remains Draft. --- docs/public-surface.md | 30 ++ methodfactory/domain/gates.py | 21 +- methodfactory/engine/__init__.py | 8 + methodfactory/engine/apply.py | 229 +++++++++ methodfactory/manifest/schema.py | 19 +- methodfactory/protocol/envelope.py | 10 + methodfactory/storage/__init__.py | 12 + methodfactory/storage/chain.py | 352 ++++++++++++++ methodfactory/storage/errors.py | 21 + methodfactory/storage/serialization.py | 80 +++- methodfactory/storage/sqlite.py | 22 +- methodfactory/storage/store.py | 615 +++++++++++++++++++++++++ 12 files changed, 1389 insertions(+), 30 deletions(-) create mode 100644 methodfactory/engine/__init__.py create mode 100644 methodfactory/engine/apply.py create mode 100644 methodfactory/storage/chain.py create mode 100644 methodfactory/storage/store.py diff --git a/docs/public-surface.md b/docs/public-surface.md index 501cc23..42e1b5d 100644 --- a/docs/public-surface.md +++ b/docs/public-surface.md @@ -28,6 +28,14 @@ exception contract and are not part of the supported surface. | `latest_event(conn, package_id)` | `sqlite3.Connection` (from `open_database`) + str | `dict | None` | `ManifestInvalidError` (`MANIFEST_INVALID`), `StorageError` (`STORAGE_ERROR`) | `sqlite3.Error` (query), `UnicodeDecodeError`/`UnicodeEncodeError`, `json.JSONDecodeError`, `TypeError`, `ValueError`, `RecursionError`, non-bytes/str stored type, non-object JSON | | `explain_latest_event_plan(conn, package_id)` | `sqlite3.Connection` + str | `list[tuple]` | `StorageError` (`STORAGE_ERROR`) | `sqlite3.Error` | | `close_database(conn)` | `sqlite3.Connection` | `None` | `StorageError` (`STORAGE_ERROR`) | `sqlite3.Error` | +| `SqliteManifestStore(root, *, artifact_store=None)` | `str`/`Path` root, optional `ArtifactStore` | store object | `InvalidStoreRootError` (`INVALID_STORE_ROOT`), `StorageError` (`STORAGE_ERROR`) | `TypeError`/`ValueError` (root), `OSError`, `sqlite3.Error` | +| `SqliteManifestStore.create(package_id, intent_raw, created_at=None)` | str + str + optional str | complete revision-0 manifest `dict` | `DuplicatePackageError` (`PACKAGE_EXISTS`, non-replay duplicate), `InvalidPayloadError` (`INVALID_PAYLOAD`), `InvalidPackageIdError` (`INVALID_PACKAGE_ID`), `ManifestInvalidError` (`MANIFEST_INVALID`), `ConcurrencyError` (`CONCURRENCY`), `StorageError` (`STORAGE_ERROR`) | `TypeError`/`ValueError`/`UnicodeError`/`RecursionError`, `OSError`, `sqlite3.Error` (incl. locked -> `CONCURRENCY`) | +| `SqliteManifestStore.apply(envelope)` | `dict` (parsed Action Envelope) | complete resulting manifest `dict` | `InvalidEnvelopeError` (`INVALID_ENVELOPE`), `InvalidPayloadError` (`INVALID_PAYLOAD`), `PackageNotFoundError` (`PACKAGE_NOT_FOUND`), `StaleActionError` (`STALE_ACTION`), `IllegalTransitionError` (`ILLEGAL_TRANSITION`), `GateUnsatisfiedError` (`GATE_UNSATISFIED`), `ActionIdConflictError` (`ACTION_ID_CONFLICT`), `SerializationError` (`SERIALIZATION`), `ArtifactVerificationError` (`ARTIFACT_VERIFICATION`), `ManifestInvalidError` (`MANIFEST_INVALID`), `ConcurrencyError` (`CONCURRENCY`), `StorageError` (`STORAGE_ERROR`) | `TypeError`/`ValueError`/`UnicodeError`/`RecursionError`, `OSError`, `sqlite3.Error` (incl. locked -> `CONCURRENCY`) | +| `SqliteManifestStore.load(package_id)` | str | complete current manifest `dict` | `PackageNotFoundError` (`PACKAGE_NOT_FOUND`), `ManifestInvalidError` (`MANIFEST_INVALID`), `StorageError` (`STORAGE_ERROR`) | `TypeError`/`ValueError`/`UnicodeError`/`RecursionError`, `sqlite3.Error` | +| `SqliteManifestStore.read_events(package_id)` | str | ordered `list[dict]` (decoded action/manifest) | `ManifestInvalidError` (`MANIFEST_INVALID`), `StorageError` (`STORAGE_ERROR`) | `sqlite3.Error`, decode/JSON errors | +| `SqliteManifestStore.validate_chain(package_id, *, verify_artifacts=False)` | str + bool | `{package_id, events, valid}` | `ChainViolationError` (`CHAIN_VIOLATION`), `StorageError` (`STORAGE_ERROR`) | `sqlite3.Error`, decode/JSON errors | +| `SqliteManifestStore.explain_latest_plan(package_id)` | str | `list[tuple]` (query plan) | `StorageError` (`STORAGE_ERROR`) | `sqlite3.Error` | +| `SqliteManifestStore.close()` | — | `None` | `StorageError` (`STORAGE_ERROR`) | `sqlite3.Error` | ## Internal primitives (documented native contract, NOT public) @@ -38,6 +46,28 @@ exception contract and are not part of the supported surface. | `storage.serialization.try_canonical_bytes_bounded` | Returns `(bytes, None)` or `(None, error_str)` — never raises | | `storage.serialization.digest_*` | SHA-256 helpers; native contract inherited from `canonical_bytes`/encoding | | `storage.sqlite._connect` / `_apply_or_verify_pragmas` / `_identity` / `_verify_schema` / `initialize_database` / `detect_presence` / `_open_database_impl` | Internal to `open_database`; sqlite3 errors may surface if called directly. `_connect` closes its connection on PRAGMA failure. | +| `storage.store._begin` / `_insert_event` / `_commit` / `_rollback` / `FAULT_HOOK` | Transaction seams for fault injection; internal. `FAULT_HOOK` is a test-only injection point (`stage`-keyed) | + +## Transaction algorithm (SqliteManifestStore.apply) + +One bounded `BEGIN IMMEDIATE` transaction: + +1. Validate the incoming Action Envelope (`envelope_from_dict`) and compute the canonical semantic action bytes once (`canonical_action_bytes`). +2. Compute `action_sha256` = hash of those exact bytes. +3. Look up `(package_id, action_id)` BEFORE stale-revision rejection. +4. If the action ID exists: same hash → return the previously committed result (no insert); different hash → `ACTION_ID_CONFLICT`. +5. Load the indexed latest event (primary-key lookup, never a history scan). +6. Compare `expected_revision` to the authoritative current revision → `STALE_ACTION` on mismatch. +7. Apply the deterministic transition (engine.next_manifest: legality → gate → mutation → revision/lineage). +8. Produce the complete resulting manifest. +9. Validate the manifest and chain invariants (single kernel). +10. Write and verify every newly referenced artifact blob (immutable, content-addressed). +11. Canonicalize stored action + manifest once. +12. Insert exactly one immutable event row. +13. `COMMIT`. + +On any pre-commit failure: rollback, no event inserted, no historical row +mutated, prewritten content-addressed blobs are NOT deleted. ## Consistency rule diff --git a/methodfactory/domain/gates.py b/methodfactory/domain/gates.py index 9184f02..bc471f1 100644 --- a/methodfactory/domain/gates.py +++ b/methodfactory/domain/gates.py @@ -46,7 +46,13 @@ def check_action_gate(action: Action, manifest: dict, envelope: "ActionEnvelope" elif action == Action.CONFIRM_SUMMARY: summary = manifest.get("summary") - current = (summary or {}).get("canonical_sha256") + if not isinstance(summary, dict): + raise GateUnsatisfiedError("cannot confirm: no summary prepared", **ctx) + # Content-addressed summary (ADR-0012): the canonical hash of the + # rendered summary body is summary.digest (the old JSONL-era + # canonical_sha256 field does not exist in the content-addressed + # manifest schema). + current = summary.get("digest") if not current: raise GateUnsatisfiedError("cannot confirm: no summary prepared", **ctx) want = envelope.basis.get("summary_sha256") @@ -60,11 +66,16 @@ def check_action_gate(action: Action, manifest: dict, envelope: "ActionEnvelope" ) elif action == Action.RECORD_DRAFT_ARTIFACT: - summary = manifest.get("summary") or {} - conf = summary.get("confirmation") or {} + summary = manifest.get("summary") + if not isinstance(summary, dict): + raise GateUnsatisfiedError( + "authoring requires a confirmed summary bound to the current summary digest", **ctx + ) + conf = summary.get("confirmation") confirmed_ok = ( - conf.get("status") == "confirmed" - and conf.get("confirmed_summary_sha256") == summary.get("canonical_sha256") + isinstance(conf, dict) + and conf.get("status") == "confirmed" + and conf.get("confirmed_summary_sha256") == summary.get("digest") ) if not confirmed_ok: raise GateUnsatisfiedError( diff --git a/methodfactory/engine/__init__.py b/methodfactory/engine/__init__.py new file mode 100644 index 0000000..bee5c8c --- /dev/null +++ b/methodfactory/engine/__init__.py @@ -0,0 +1,8 @@ +"""Engine package — deterministic, storage-independent transition logic. + +The engine owns WHAT a legal action does to a manifest (legality, gates, +mutation, revision/lineage). Storage owns WHERE the resulting manifest and +event are durably committed. The engine never touches SQLite, the filesystem, +or the artifact store; it returns the next manifest plus the content blobs +the storage layer must persist and verify. +""" diff --git a/methodfactory/engine/apply.py b/methodfactory/engine/apply.py new file mode 100644 index 0000000..1b18bcf --- /dev/null +++ b/methodfactory/engine/apply.py @@ -0,0 +1,229 @@ +"""Deterministic manifest transitions — the single apply rule (ADR-0003/0005). + +`next_manifest` is the ONLY place a legal action transforms a manifest: + +1. legality (domain.transitions — the transition table is the sole authority); +2. gate check (domain.gates — evidence/binding checks, no writes); +3. deterministic mutation for the action; +4. revision increment, target state, updated_at, and write-lineage fields + (previous_manifest_sha256 = canonical digest of the current manifest; + transition.last_event_id / last_action_id). + +Returns ``(next_manifest, blobs_to_write)`` where ``blobs_to_write`` is a +list of ``(logical_path, content)`` the storage layer must persist into the +content-addressed artifact store before committing (immutable blobs; on a +transaction rollback the blobs are kept — content-addressed and harmless). + +The engine is persistence-free (no SQLite, no filesystem, no artifact store) +but deliberately uses the storage package's canonical serialization and +limit constants as the single shared infrastructure (ADR-0012 §4) — the same +bytes the storage layer hashes and persists, so engine-produced digests can +never drift from stored hashes. + +Gates are re-evaluated here so apply validity cannot diverge from the shared +rule; the storage layer then re-checks chain invariants via the single +invariant kernel (storage.chain) before INSERT. +""" + +from __future__ import annotations + +import copy +from typing import Any + +from ..domain.errors import IllegalTransitionError, InvalidPayloadError +from ..domain.gates import check_action_gate +from ..domain.states import State +from ..domain.transitions import Action, transition_target +from ..manifest.render import render_summary +from ..protocol.envelope import ActionEnvelope +from ..storage.limits import MAX_PREVIEW_CHARS +from ..storage.serialization import digest_bytes, digest_json + +# The canonical create operation. Deliberately NOT in the envelope Action +# vocabulary: a package is created only through the store's create(), never +# by a caller-proposed envelope action. +CREATE_PACKAGE_ACTION = "create_package" + +# Deterministic blob paths (internal bookkeeping; not part of the public +# manifest contract beyond the logical-path grammar). +SUMMARY_BLOB_TEMPLATE = "summaries/r{revision}.txt" +INPUT_BLOB_TEMPLATE = "inputs/{input_id}.txt" + + +def _content_digest(content: str, field: str) -> tuple[str, int]: + """Encode content to UTF-8 bytes and compute digest + size; translate + lone surrogates typed.""" + try: + data = content.encode("utf-8") + except UnicodeEncodeError as exc: + raise InvalidPayloadError(f"{field} is not valid UTF-8 (lone surrogate?)") from exc + return digest_bytes(data), len(data) + + +def _apply_record_input(m: dict, envelope: ActionEnvelope) -> list[tuple[str, str]]: + payload = envelope.payload + digest, size = _content_digest(payload["content"], "record_input content") + path = INPUT_BLOB_TEMPLATE.format(input_id=payload["input_id"]) + m["inputs"] = [*m.get("inputs", []), { + "input_id": payload["input_id"], + "kind": payload["kind"], + "source": payload["source"], + "disposition": payload["disposition"], + "exclusion_reason": payload.get("exclusion_reason"), + "content_sha256": digest, + "content_size": size, + "content_path": path, + }] + return [(path, payload["content"])] + + +def _apply_set_objective(m: dict, envelope: ActionEnvelope) -> list[tuple[str, str]]: + m["objective"] = { + "statement": envelope.payload["statement"], + "desired_outcomes": list(envelope.payload.get("desired_outcomes", [])), + } + return [] + + +def _apply_prepare_summary( + m: dict, envelope: ActionEnvelope, *, revision: int, created_at: str +) -> list[tuple[str, str]]: + try: + body = render_summary(m) + except (KeyError, TypeError, IndexError, AttributeError, ValueError) as exc: + raise InvalidPayloadError( + f"current manifest is not well-formed for summary rendering: {exc}" + ) from exc + digest, size = _content_digest(body, "summary body") + path = SUMMARY_BLOB_TEMPLATE.format(revision=revision) + # Preview is a single-line bounded snippet: collapse all whitespace runs + # (newlines/tabs are C0 control chars and are rejected by the manifest + # validator's control-character rule). + preview = " ".join(body.split())[:MAX_PREVIEW_CHARS] + m["summary"] = { + "digest": digest, + "size": size, + "preview": preview, + "presented_at": created_at, + "confirmation": { + "status": "pending", + "confirmed_at": None, + "operator_id": None, + "confirmed_summary_sha256": None, + }, + } + return [(path, body)] + + +def _apply_confirm_summary(m: dict, envelope: ActionEnvelope, *, created_at: str) -> list[tuple[str, str]]: + m["summary"]["confirmation"] = { + "status": "confirmed", + "confirmed_at": created_at, + "operator_id": envelope.payload.get("operator_id") or "operator", + "confirmed_summary_sha256": envelope.basis["summary_sha256"], + } + return [] + + +def _apply_revise_intake(m: dict, _envelope: ActionEnvelope) -> list[tuple[str, str]]: + # Mutators share one signature for uniform dispatch; the envelope is + # unused here by design (revise_intake carries no payload state). + # Summary and any draft artifacts are stale after the operator revises + # intake; inputs and objective are intake material and are preserved. + m["summary"] = None + m["artifacts"] = [] + return [] + + +def _apply_record_draft_artifact(m: dict, envelope: ActionEnvelope) -> list[tuple[str, str]]: + payload = envelope.payload + digest, size = _content_digest(payload["content"], "artifact content") + m["artifacts"] = [*m.get("artifacts", []), { + "artifact_id": payload["artifact_id"], + "kind": payload["kind"], + "logical_path": payload["logical_path"], + "sha256": digest, + "byte_count": size, + "status": "draft", + }] + return [(payload["logical_path"], payload["content"])] + + +def _apply_cancel(m: dict, _envelope: ActionEnvelope) -> list[tuple[str, str]]: + # Mutators share one signature for uniform dispatch; the envelope is + # unused here by design (cancel is a transition-only action; the reason + # payload is recorded nowhere in v0.1). + return [] + + +_ACTION_MUTATORS = { + Action.RECORD_INPUT: _apply_record_input, + Action.SET_OBJECTIVE: _apply_set_objective, + Action.PREPARE_SUMMARY: _apply_prepare_summary, + Action.CONFIRM_SUMMARY: _apply_confirm_summary, + Action.REVISE_INTAKE: _apply_revise_intake, + Action.RECORD_DRAFT_ARTIFACT: _apply_record_draft_artifact, + Action.CANCEL: _apply_cancel, +} + + +def next_manifest( + current: dict[str, Any], + envelope: ActionEnvelope, + *, + event_id: str, + created_at: str, +) -> tuple[dict[str, Any], list[tuple[str, str]]]: + """Compute the next manifest for a legal, gate-passing action. + + Raises (all public MethodFactoryError): + IllegalTransitionError — action not legal in the current state. + GateUnsatisfiedError / StaleActionError / InvalidPayloadError — + gate evidence or binding failures. + + Returns (next_manifest, blobs_to_write). + """ + if not isinstance(current, dict): + raise InvalidPayloadError("current manifest must be an object") + state = current.get("state") + try: + current_state = State(state) + except ValueError: + raise InvalidPayloadError(f"current manifest has invalid state {state!r}") from None + revision = current.get("revision") + if isinstance(revision, bool) or not isinstance(revision, int) or revision < 0: + raise InvalidPayloadError( + f"current manifest revision must be a non-negative int, got {revision!r}" + ) + try: + action = Action(envelope.action) + except ValueError: + raise InvalidPayloadError(f"unknown action {envelope.action!r}") from None + + target = transition_target(current_state, action) + if target is None: + raise IllegalTransitionError( + f"action {envelope.action!r} is not legal from state {current_state.value}", + package_id=envelope.package_id, + state=current_state.value, + ) + + # Gate evidence/binding checks BEFORE any mutation (no writes here). + check_action_gate(action, current, envelope) + + m = copy.deepcopy(current) + new_revision = revision + 1 + mutator = _ACTION_MUTATORS[action] + if action == Action.PREPARE_SUMMARY: + blobs = mutator(m, envelope, revision=new_revision, created_at=created_at) + elif action == Action.CONFIRM_SUMMARY: + blobs = mutator(m, envelope, created_at=created_at) + else: + blobs = mutator(m, envelope) + + m["revision"] = new_revision + m["state"] = target.value + m["updated_at"] = created_at + m["previous_manifest_sha256"] = digest_json(current) + m["transition"] = {"last_event_id": event_id, "last_action_id": envelope.action_id} + return m, blobs diff --git a/methodfactory/manifest/schema.py b/methodfactory/manifest/schema.py index fc98e57..152871f 100644 --- a/methodfactory/manifest/schema.py +++ b/methodfactory/manifest/schema.py @@ -96,15 +96,28 @@ def validate_manifest(manifest: dict) -> list[str]: canonical MAX_MANIFEST_BYTES plus every persisted field/path/identifier/ control limit. """ + errors, _ = validate_manifest_canonical(manifest) + return errors + + +def validate_manifest_canonical(manifest: dict) -> tuple[list[str], "bytes | None"]: + """Collect violations AND return the accepted canonical bytes in one pass. + + Single canonicalization per call: the transactional store validates a + resulting manifest and then stores its exact canonical bytes; this helper + avoids canonicalizing twice under the write lock. + Returns ``(errors, canonical_bytes)`` — canonical_bytes is None when + validation failed (the manifest cannot be accepted). + """ errors: list[str] = [] if not isinstance(manifest, dict): - return ["manifest must be a JSON object"] + return ["manifest must be a JSON object"], None # Total canonical manifest byte bound. Native canonicalization failures # (unsupported types, deep recursion, lone-surrogate encoding) are # translated into manifest errors — never leaked raw (Finding 1). - _, canonical_error = try_canonical_bytes_bounded( + canonical, canonical_error = try_canonical_bytes_bounded( manifest, limit=MAX_MANIFEST_BYTES, what="manifest" ) if canonical_error is not None: @@ -339,4 +352,4 @@ def validate_manifest(manifest: dict) -> list[str]: except Exception: errors.append(f"transition.{f} invalid identifier") - return errors + return errors, canonical diff --git a/methodfactory/protocol/envelope.py b/methodfactory/protocol/envelope.py index 2b1bcf8..b40ca7e 100644 --- a/methodfactory/protocol/envelope.py +++ b/methodfactory/protocol/envelope.py @@ -178,6 +178,16 @@ def _validate_envelope_dict(d: dict) -> None: raise InvalidEnvelopeError("action_id must not contain control characters") if len(action_id) > 64: raise InvalidEnvelopeError("action_id must be <= 64 chars") + # Enforce the identifier grammar AT the untrusted-input boundary so a + # caller-supplied action_id outside the grammar is rejected here + # (INVALID_ENVELOPE), not deep in the transaction as MANIFEST_INVALID. + # Effective boundary limit is 64 chars (validated above); the shared + # identifier validator permits 128 for internally generated ids + # (e.g. create's action_id), which never pass through this boundary. + try: + validate_identifier(action_id, field="action_id") + except Exception as exc: + raise InvalidEnvelopeError(f"invalid action_id {action_id!r}") from exc package_id = d["package_id"] try: diff --git a/methodfactory/storage/__init__.py b/methodfactory/storage/__init__.py index 662db0c..0f612b5 100644 --- a/methodfactory/storage/__init__.py +++ b/methodfactory/storage/__init__.py @@ -8,13 +8,18 @@ from .errors import ( ActionIdConflictError, AppendOnlyViolationError, + ArtifactVerificationError, + ChainViolationError, DatabaseEmptyError, DatabaseIdMismatchError, DatabaseNotFoundError, InvalidPackageIdError, InvalidStoreRootError, LegacyStoreDetectedError, + ManifestInvalidError, + PackageNotFoundError, SchemaViolationError, + SerializationError, StorageError, UnsupportedSchemaError, ) @@ -43,10 +48,13 @@ digest_text, sha256_hex, ) +from .store import SqliteManifestStore __all__ = [ "ActionIdConflictError", "AppendOnlyViolationError", + "ArtifactVerificationError", + "ChainViolationError", "DB_FILENAME", "DatabaseEmptyError", "DatabaseIdMismatchError", @@ -54,6 +62,7 @@ "InvalidPackageIdError", "InvalidStoreRootError", "LegacyStoreDetectedError", + "ManifestInvalidError", "ManifestStore", "MAX_ACTION_JSON_BYTES", "MAX_ARTIFACT_BYTES", @@ -67,7 +76,10 @@ "MAX_REASON_CHARS", "MAX_STATEMENT_CHARS", "MethodFactoryError", + "PackageNotFoundError", "SchemaViolationError", + "SerializationError", + "SqliteManifestStore", "StorageError", "UnsupportedSchemaError", "action_sha256", diff --git a/methodfactory/storage/chain.py b/methodfactory/storage/chain.py new file mode 100644 index 0000000..c63611c --- /dev/null +++ b/methodfactory/storage/chain.py @@ -0,0 +1,352 @@ +"""Authoritative revision-chain invariants and validator (ADR-0012 §F). + +This module owns the SINGLE invariant kernel. Both the transactional apply +(storage.store) and the authoritative validator use the same +`check_event_invariants` — there are no competing "apply validity" and +"full-chain validity" implementations with divergent rules. + +Enforced invariants per event revision N: + +Revision 0: +- action is `create_package`; +- `state_before IS NULL`; +- `previous_manifest_sha256 IS NULL`; +- resulting manifest package_id == indexed package_id; +- resulting manifest revision == 0; +- resulting manifest state == indexed `state_after`; +- stored action JSON hashes to `action_sha256`; +- stored manifest JSON hashes to `resulting_manifest_sha256`. + +Revision > 0: +- event N-1 exists and is the immediate predecessor; +- `state_before(N) == state_after(N-1)`; +- `previous_manifest_sha256(N) == resulting_manifest_sha256(N-1)`; +- manifest package_id / revision / state match the indexed row; +- stored action JSON hashes to `action_sha256`; +- stored manifest JSON hashes to `resulting_manifest_sha256`. + +All revisions: +- event_id / action_id obey the identifier grammar (uniqueness is schema- + enforced by the UNIQUE constraints on event_id and (package_id, action_id)); +- action is the frozen vocabulary (or `create_package` at revision 0); +- stored JSON BLOBs are valid UTF-8 JSON objects; +- when artifact verification is requested: every referenced input content + blob, the summary body blob, and every artifact blob exists and verifies + against its recorded digest. +""" + +from __future__ import annotations + +import json +import sqlite3 +from typing import Any + +from ..domain.transitions import ACTION_VOCABULARY +from ..engine.apply import CREATE_PACKAGE_ACTION +from .errors import ChainViolationError, StorageError +from .paths import validate_identifier +from .serialization import sha256_hex + +EVENTS_BY_REVISION_SQL = """ +SELECT package_id, revision, event_id, action_id, action, action_sha256, + state_before, state_after, previous_manifest_sha256, + resulting_manifest_sha256, created_at, action_json, manifest_json +FROM events +WHERE package_id = ? +ORDER BY revision ASC +""" + + +def _decode_json_object(data: bytes, what: str, violations: list[str]) -> dict | None: + """Decode a stored JSON BLOB; report a violation (not raise) on failure.""" + try: + text = data.decode("utf-8") + except (UnicodeDecodeError, AttributeError) as exc: + violations.append(f"{what} is not valid UTF-8: {exc}") + return None + try: + value = json.loads(text) + except (json.JSONDecodeError, RecursionError) as exc: + violations.append(f"{what} is not valid JSON: {exc}") + return None + if not isinstance(value, dict): + violations.append(f"{what} is not a JSON object") + return None + return value + + +def _bind_manifest_fields( + manifest: dict[str, Any], + *, + package_id: str, + revision: Any, + state_after: Any, + violations: list[str], +) -> None: + """Shared manifest<->row field-binding checks (single implementation).""" + if manifest.get("package_id") != package_id: + violations.append( + f"manifest package_id {manifest.get('package_id')!r} != indexed {package_id!r}" + ) + if manifest.get("revision") != revision: + violations.append( + f"manifest revision {manifest.get('revision')!r} != indexed {revision!r}" + ) + if manifest.get("state") != state_after: + violations.append( + f"manifest state {manifest.get('state')!r} != indexed state_after {state_after!r}" + ) + + +def check_event_invariants( + *, + package_id: str, + event: dict[str, Any], + prev_event: dict[str, Any] | None, + manifest: dict[str, Any] | None, + verify_artifacts: bool = False, + artifact_store: Any = None, + check_schema: bool = False, + decode_blobs: bool = True, +) -> list[str]: + """Return every invariant violation for one event (empty = valid). + + `event` / `prev_event` are row dicts whose action_json/manifest_json are + the RAW stored BLOB bytes. `manifest` is the decoded resulting manifest + (None when the BLOB could not be decoded; the decode violation is reported + by this kernel and field checks are skipped). + + Mode flags (single kernel, parameterized — no divergent implementations): + - `check_schema`: also run the authoritative manifest schema validator on + the decoded manifest (audit/validator mode; apply validates immediately + before this call, so it passes False). + - `decode_blobs`: decode the stored BLOBs to verify JSON validity. + True for the on-disk validator; the transactional apply passes False + because the bytes it stores are self-produced and digest-bound. + """ + violations: list[str] = [] + revision = event.get("revision") + state_after = event.get("state_after") + resulting_hash = event.get("resulting_manifest_sha256") + action_hash = event.get("action_sha256") + action_name = event.get("action") + event_id = event.get("event_id") + action_id = event.get("action_id") + + # ── Stored BLOB validity + digest binding ────────────────────────── + manifest_from_bytes = None + if decode_blobs: + manifest_from_bytes = _decode_json_object( + event.get("manifest_json"), f"manifest_json({package_id}, rev {revision})", violations + ) + _decode_json_object( + event.get("action_json"), f"action_json({package_id}, rev {revision})", violations + ) + if manifest is None and manifest_from_bytes is not None: + # Caller (validator) did not pre-decode; use the kernel's decode so + # field checks still run. + manifest = manifest_from_bytes + try: + if sha256_hex(bytes(event.get("manifest_json"))) != resulting_hash: + violations.append( + f"manifest_json({package_id}, rev {revision}) does not hash to resulting_manifest_sha256" + ) + except (TypeError, ValueError): + violations.append(f"manifest_json({package_id}, rev {revision}) is not bytes") + try: + if sha256_hex(bytes(event.get("action_json"))) != action_hash: + violations.append( + f"action_json({package_id}, rev {revision}) does not hash to action_sha256" + ) + except (TypeError, ValueError): + violations.append(f"action_json({package_id}, rev {revision}) is not bytes") + + # ── Revision-zero contract ───────────────────────────────────────── + if revision == 0: + if action_name != CREATE_PACKAGE_ACTION: + violations.append( + f"revision 0 action must be {CREATE_PACKAGE_ACTION!r}, got {action_name!r}" + ) + if event.get("state_before") is not None: + violations.append("revision 0 state_before must be NULL") + if event.get("previous_manifest_sha256") is not None: + violations.append("revision 0 previous_manifest_sha256 must be NULL") + else: + # ── Revision > 0 lineage ─────────────────────────────────────── + if prev_event is None: + violations.append(f"revision {revision} has no predecessor event") + else: + if prev_event.get("revision") != revision - 1: + violations.append( + f"revision {revision} predecessor is revision {prev_event.get('revision')}, " + f"expected {revision - 1}" + ) + if event.get("state_before") != prev_event.get("state_after"): + violations.append( + f"state_before({revision}) != state_after({revision - 1})" + ) + if event.get("previous_manifest_sha256") != prev_event.get("resulting_manifest_sha256"): + violations.append( + f"previous_manifest_sha256({revision}) != " + f"resulting_manifest_sha256({revision - 1})" + ) + + # ── Manifest field binding (runs when a manifest dict is available) ─ + if manifest is not None: + _bind_manifest_fields( + manifest, package_id=package_id, revision=revision, + state_after=state_after, violations=violations, + ) + # Manifest-internal lineage claims must match the indexed row (the + # engine writes these as chain facts; the validator cross-checks them). + # Revision 0 is exempt: the create manifest's transition fields are + # None by contract (there is no PRIOR action before create). + if revision != 0: + if manifest.get("previous_manifest_sha256") != event.get("previous_manifest_sha256"): + violations.append( + "manifest previous_manifest_sha256 does not match the indexed row" + ) + transition = manifest.get("transition") + if isinstance(transition, dict): + if transition.get("last_event_id") != event_id: + violations.append( + "manifest transition.last_event_id does not match the indexed event_id" + ) + if transition.get("last_action_id") != action_id: + violations.append( + "manifest transition.last_action_id does not match the indexed action_id" + ) + # Optional authoritative schema validation (audit/validator mode). + if check_schema: + from ..manifest.schema import validate_manifest as _vm + + for schema_error in _vm(manifest): + violations.append(f"manifest schema violation: {schema_error}") + + # ── Grammar / vocabulary ─────────────────────────────────────────── + for field, value in (("event_id", event_id), ("action_id", action_id)): + if not isinstance(value, str) or not value: + violations.append(f"{field} must be a non-empty string") + else: + try: + validate_identifier(value, field=field) + except Exception as exc: + violations.append(f"{field} {value!r} violates identifier grammar: {exc}") + # Revision 0 must be create_package (enforced unconditionally above), so + # the vocabulary membership check applies to revision > 0 only. + if revision != 0 and action_name not in ACTION_VOCABULARY: + violations.append(f"unknown action {action_name!r}") + + # ── Referenced-artifact verification (optional mode) ─────────────── + if verify_artifacts and manifest is not None and artifact_store is not None: + for entry in manifest.get("inputs", []) or []: + if not isinstance(entry, dict): + violations.append("inputs entry is not an object") + continue + digest = entry.get("content_sha256") + if digest and not artifact_store.verify(digest): + violations.append(f"input {entry.get('input_id')!r} content blob {digest} missing/corrupt") + summary = manifest.get("summary") + if isinstance(summary, dict): + digest = summary.get("digest") + if digest and not artifact_store.verify(digest): + violations.append(f"summary body blob {digest} missing/corrupt") + for art in manifest.get("artifacts", []) or []: + if not isinstance(art, dict): + violations.append("artifacts entry is not an object") + continue + digest = art.get("sha256") + if digest and not artifact_store.verify(digest): + violations.append(f"artifact {art.get('artifact_id')!r} blob {digest} missing/corrupt") + + return violations + + +def check_current_row_consistency( + *, + package_id: str, + event: dict[str, Any], + manifest: dict[str, Any], + manifest_json_bytes: bytes, + check_schema: bool = True, +) -> list[str]: + """Hot-path current-row self-consistency (load). + + Deliberately bounded: verifies the decoded manifest against the indexed + row's identity/state/hash fields WITHOUT scanning history, and runs the + authoritative schema validator on the single current manifest (bounded to + one row — never a history scan). Full chain verification belongs to + `validate_chain`. + """ + violations: list[str] = [] + _bind_manifest_fields( + manifest, package_id=package_id, revision=event.get("revision"), + state_after=event.get("state_after"), violations=violations, + ) + try: + if sha256_hex(bytes(manifest_json_bytes)) != event.get("resulting_manifest_sha256"): + violations.append( + "manifest_json bytes do not hash to resulting_manifest_sha256" + ) + except (TypeError, ValueError): + violations.append("manifest_json is not bytes") + if check_schema: + from ..manifest.schema import validate_manifest as _vm + + for schema_error in _vm(manifest): + violations.append(f"manifest schema violation: {schema_error}") + return violations + + +def validate_chain( + conn: sqlite3.Connection, + package_id: str, + *, + verify_artifacts: bool = False, + artifact_store: Any = None, + check_schema: bool = True, +) -> dict[str, Any]: + """Authoritative revision-chain validator for one package. + + Walks every event in revision order (lazily, one row at a time), applies + the single invariant kernel per event with BLOB decoding + schema + validation enabled, and raises ChainViolationError (code CHAIN_VIOLATION) + with ALL violations of the failing event on the FIRST invalid event. + Returns a summary on success. + """ + try: + cursor = conn.execute(EVENTS_BY_REVISION_SQL, (package_id,)) + except sqlite3.Error as exc: + raise StorageError(f"chain validation query failed: {exc}") from exc + row = cursor.fetchone() + if row is None: + raise ChainViolationError(f"package {package_id} has no events") + + # Lazy walk: one event at a time, so memory stays bounded to a single + # event even for long chains (perf, audit path). + prev_event: dict[str, Any] | None = None + event_count = 0 + while row is not None: + event = dict(row) + event_count += 1 + # The kernel decodes and reports malformed BLOBs as violations, and + # re-validates the manifest schema (audit mode). + violations = check_event_invariants( + package_id=package_id, + event=event, + prev_event=prev_event, + manifest=None, + verify_artifacts=verify_artifacts, + artifact_store=artifact_store, + check_schema=check_schema, + decode_blobs=True, + ) + if violations: + raise ChainViolationError( + f"chain violation for {package_id} rev {event.get('revision')}: " + + "; ".join(violations) + ) + prev_event = event + row = cursor.fetchone() + + return {"package_id": package_id, "events": event_count, "valid": True} diff --git a/methodfactory/storage/errors.py b/methodfactory/storage/errors.py index 7910293..236e568 100644 --- a/methodfactory/storage/errors.py +++ b/methodfactory/storage/errors.py @@ -121,3 +121,24 @@ class SerializationError(StorageError): """ code = "SERIALIZATION" + + +class PackageNotFoundError(StorageError): + """The requested package has no committed events (load/apply on missing).""" + + code = "PACKAGE_NOT_FOUND" + + +class ArtifactVerificationError(StorageError): + """A manifest-referenced artifact blob is missing, corrupt, or does not + match its digest at transaction/chain verification time.""" + + code = "ARTIFACT_VERIFICATION" + + +class ChainViolationError(StorageError): + """The authoritative revision-chain validator found an invariant violation + (revision zero rules, lineage, state continuity, digest binding, grammar, + or referenced-artifact integrity).""" + + code = "CHAIN_VIOLATION" diff --git a/methodfactory/storage/serialization.py b/methodfactory/storage/serialization.py index b86cf2e..b84c0e8 100644 --- a/methodfactory/storage/serialization.py +++ b/methodfactory/storage/serialization.py @@ -104,6 +104,61 @@ def contains_control_chars(value: str) -> bool: return False +def semantic_action( + *, + protocol_version: str, + action: str, + package_id: str, + action_id: str, + basis: dict[str, Any], + payload: dict[str, Any], +) -> dict[str, Any]: + """The normalized semantic action dict (ADR-0012 §G). + + Single construction point so the stored action JSON, the idempotency + hash, and any consumer always see identical bytes. `action_sha256` and + `canonical_action_bytes` both derive from this; the transactional store + uses the same dict for the immutable action_json column. + """ + return { + "protocol_version": protocol_version, + "action": action, + "package_id": package_id, + "action_id": action_id, + "basis": basis, + "payload": payload, + } + + +def canonical_action_bytes( + *, + protocol_version: str, + action: str, + package_id: str, + action_id: str, + basis: dict[str, Any], + payload: dict[str, Any], +) -> bytes: + """Canonical bytes of the normalized semantic action (stored as + action_json). Raises SerializationError on non-canonicalizable input or + when the canonical bytes exceed MAX_ACTION_JSON_BYTES.""" + try: + return canonical_bytes_bounded( + semantic_action( + protocol_version=protocol_version, + action=action, + package_id=package_id, + action_id=action_id, + basis=basis, + payload=payload, + ), + limit=MAX_ACTION_JSON_BYTES, + what="canonical action", + ) + except (TypeError, RecursionError, UnicodeEncodeError, ValueError) as exc: + raise SerializationError(f"cannot canonicalize action: {exc}") from exc + + def action_sha256( *, protocol_version: str, @@ -134,20 +189,11 @@ def action_sha256( MethodFactoryError with code SERIALIZATION); no raw TypeError/RecursionError/UnicodeEncodeError/ValueError escapes. """ - semantic = { - "protocol_version": protocol_version, - "action": action, - "package_id": package_id, - "action_id": action_id, - "basis": basis, - "payload": payload, - } - try: - canonical = canonical_bytes_bounded( - semantic, - limit=MAX_ACTION_JSON_BYTES, - what="canonical action", - ) - except (TypeError, RecursionError, UnicodeEncodeError, ValueError) as exc: - raise SerializationError(f"cannot canonicalize action: {exc}") from exc - return sha256_hex(canonical) + return sha256_hex(canonical_action_bytes( + protocol_version=protocol_version, + action=action, + package_id=package_id, + action_id=action_id, + basis=basis, + payload=payload, + )) diff --git a/methodfactory/storage/sqlite.py b/methodfactory/storage/sqlite.py index 8e4bbe8..c8a1d9f 100644 --- a/methodfactory/storage/sqlite.py +++ b/methodfactory/storage/sqlite.py @@ -175,7 +175,9 @@ def _normalize_trigger_sql(sql: str) -> str: # Current-state lookup (indexed by the composite primary key). LATEST_EVENT_SQL = """ -SELECT manifest_json +SELECT package_id, revision, event_id, action_id, action, action_sha256, + state_before, state_after, previous_manifest_sha256, + resulting_manifest_sha256, created_at, action_json, manifest_json FROM events WHERE package_id = ? ORDER BY revision DESC @@ -646,6 +648,19 @@ def _open_database_impl( return conn +def latest_event_row(conn: sqlite3.Connection, package_id: str) -> sqlite3.Row | None: + """Return the latest event ROW for a package (indexed latest-event read). + + Hot-path primitive used by `latest_event`, the transactional store's + `load`, and apply's current-state load. Uses the (package_id, revision) + primary key via ORDER BY revision DESC LIMIT 1 — never a history scan. + """ + try: + return conn.execute(LATEST_EVENT_SQL, (package_id,)).fetchone() + except sqlite3.Error as exc: + raise StorageError(f"latest_event query failed: {exc}") from exc + + def latest_event(conn: sqlite3.Connection, package_id: str) -> dict | None: """Return the latest manifest for a package (indexed latest-event read). @@ -654,10 +669,7 @@ def latest_event(conn: sqlite3.Connection, package_id: str) -> dict | None: translated to MANIFEST_INVALID; the decoded JSON must be an OBJECT; no raw AttributeError/JSON/Unicode/SQLite/type exception escapes. """ - try: - row = conn.execute(LATEST_EVENT_SQL, (package_id,)).fetchone() - except sqlite3.Error as exc: - raise StorageError(f"latest_event query failed: {exc}") from exc + row = latest_event_row(conn, package_id) if row is None: return None diff --git a/methodfactory/storage/store.py b/methodfactory/storage/store.py new file mode 100644 index 0000000..1a404ee --- /dev/null +++ b/methodfactory/storage/store.py @@ -0,0 +1,615 @@ +"""SqliteManifestStore — transactional create/load/apply (ADR-0012 §6, §8). + +This is the canonical transactional persistence implementation. Every +mutation executes as ONE bounded transaction: + + BEGIN IMMEDIATE + 1. Validate the incoming Action Envelope and its canonical semantic + action (envelope_from_dict + canonical_action_bytes). + 2. Compute the canonical action_sha256 (hash of the exact stored bytes). + 3. Look up (package_id, action_id) BEFORE stale-revision rejection. + 4. If the action ID already exists: + * same canonical hash -> return the previously committed result + (no second insert); + * different hash -> raise ACTION_ID_CONFLICT. + 5. Load the indexed latest event for the package. + 6. Compare expected_revision to the authoritative current revision. + 7. Apply the deterministic state transition (engine.apply.next_manifest). + 8. Produce the complete resulting manifest. + 9. Validate the resulting manifest and all transaction/chain invariants + (single kernel: storage.chain.check_event_invariants). + 10. Verify every newly referenced artifact blob (content blobs written + via the immutable ArtifactStore; all new references verified). + 11. Canonicalize the stored action and resulting manifest once. + 12. Insert exactly one immutable event row. + 13. COMMIT. + +The numbered algorithm is authoritative in docs/public-surface.md +(Transaction algorithm); this docstring is the short invariant summary. + +On any failure before commit: roll back, insert no event, do not mutate +historical rows, do not delete prewritten content-addressed blobs (they are +immutable and harmless if orphaned). + +Package creation freezes revision-zero semantics: created ONLY by the +canonical create_package operation, revision 0, state_before NULL, no +predecessor manifest hash, initial valid state INTAKE, one valid complete +manifest, exactly one event. Duplicate create raises PACKAGE_EXISTS unless +it is an exact idempotent replay of the original creation action. + +Concurrency: BEGIN IMMEDIATE + the binding busy_timeout (5000 ms). A lock +contention that exceeds the timeout surfaces as ConcurrencyError +(CONCURRENCY), never a raw sqlite exception. One store owns one connection; +use one store per thread/process for real concurrency. + +No mutable head table, lock files, JSONL repair, journal framing, cache +reconciliation, or second notion of canonical state are introduced. +""" + +from __future__ import annotations + +import json +import sqlite3 +from datetime import datetime, timezone +from typing import Any, Callable + +from ..domain.errors import ( + ConcurrencyError, + DuplicatePackageError, + InvalidPayloadError, + MethodFactoryError, + StaleActionError, +) +from ..engine.apply import CREATE_PACKAGE_ACTION, next_manifest +from ..manifest.schema import new_manifest, validate_manifest_canonical +from ..protocol.envelope import PROTOCOL_VERSION, envelope_from_dict +from ..storage.errors import ( + ActionIdConflictError, + ArtifactVerificationError, + ManifestInvalidError, + PackageNotFoundError, + StorageError, +) +from ..storage.limits import MAX_INTENT_CHARS +from ..storage.paths import validate_package_id, validate_store_root +from ..storage.serialization import ( + canonical_action_bytes, + contains_control_chars, + sha256_hex, +) +from ..storage.sqlite import ( + close_database, + explain_latest_event_plan, + latest_event_row, + open_database, +) +from .chain import ( + check_current_row_consistency, + check_event_invariants, + validate_chain as _validate_chain, +) + +# ── SQL ──────────────────────────────────────────────────────────────── +INSERT_EVENT_SQL = """ +INSERT INTO events ( + package_id, revision, event_id, action_id, action, action_sha256, + state_before, state_after, previous_manifest_sha256, + resulting_manifest_sha256, created_at, action_json, manifest_json +) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) +""" + +ACTION_LOOKUP_SQL = """ +SELECT package_id, revision, action_sha256, state_after, + resulting_manifest_sha256, manifest_json +FROM events +WHERE package_id = ? AND action_id = ? +""" + +EVENTS_ALL_SQL = """ +SELECT package_id, revision, event_id, action_id, action, action_sha256, + state_before, state_after, previous_manifest_sha256, + resulting_manifest_sha256, created_at, action_json, manifest_json +FROM events +WHERE package_id = ? +ORDER BY revision ASC +""" + + +# ── Transaction seams + fault hooks (tests inject precise single-point +# faults through the REAL implementation path) ─────────────────────── +FAULT_HOOK: Callable[[str], None] | None = None + + +def _fault(stage: str) -> None: + if FAULT_HOOK is not None: + FAULT_HOOK(stage) + + +def _begin(conn: sqlite3.Connection) -> None: + conn.execute("BEGIN IMMEDIATE") + + +def _insert_event(conn: sqlite3.Connection, row: tuple) -> None: + conn.execute(INSERT_EVENT_SQL, row) + + +def _commit(conn: sqlite3.Connection) -> None: + conn.commit() + + +def _rollback(conn: sqlite3.Connection) -> None: + conn.rollback() + + +def _utcnow() -> str: + return datetime.now(timezone.utc).isoformat() + + +def _is_locked(exc: sqlite3.Error) -> bool: + """Classify retryable SQLite lock contention by extended error code, + falling back to the message heuristic only for interpreters without + sqlite_errorcode.""" + code = getattr(exc, "sqlite_errorcode", None) + if code is not None: + return code in (sqlite3.SQLITE_BUSY, sqlite3.SQLITE_LOCKED) + text = str(exc).lower() + return "locked" in text or "busy" in text + + +class _Transaction: + """One bounded BEGIN IMMEDIATE transaction. + + Owns begin/commit/rollback; on any exception before commit the + transaction rolls back (no event, no historical mutation). `_fault` + stages fire inside the body so fault injection is unchanged. The + translation ladder lives in the store methods' outer try. + """ + + def __init__(self, conn: sqlite3.Connection) -> None: + self._conn = conn + + def __enter__(self) -> "_Transaction": + _fault("before_begin") + _begin(self._conn) + try: + _fault("after_begin") + except BaseException: + # A fault after BEGIN but before the body must not leak an open + # write transaction: roll back before re-raising. + try: + _rollback(self._conn) + except sqlite3.Error: + pass + raise + return self + + def __exit__(self, exc_type, exc, tb) -> bool: + if exc_type is None: + try: + _commit(self._conn) + except BaseException: + # Commit failure: the seam raises before the database commits; + # roll back so no transaction leaks into the next operation. + try: + _rollback(self._conn) + except sqlite3.Error: + pass + raise + else: + try: + _rollback(self._conn) + except sqlite3.Error: + pass + return False # propagate + + +def _event_row_tuple(e: dict[str, Any]) -> tuple: + return ( + e["package_id"], e["revision"], e["event_id"], e["action_id"], + e["action"], e["action_sha256"], e["state_before"], e["state_after"], + e["previous_manifest_sha256"], e["resulting_manifest_sha256"], + e["created_at"], e["action_json"], e["manifest_json"], + ) + + +def _decode_json_blob(raw: Any, *, what: str, package_id: str) -> dict[str, Any]: + if not isinstance(raw, (bytes, str)): + raise ManifestInvalidError( + f"{what} for {package_id} has unexpected type {type(raw).__name__}" + ) + try: + if isinstance(raw, str): + raw = raw.encode("utf-8") + decoded = json.loads(raw.decode("utf-8")) + except (UnicodeEncodeError, UnicodeDecodeError, json.JSONDecodeError, + TypeError, ValueError, RecursionError) as exc: + raise ManifestInvalidError( + f"{what} corrupt for {package_id}: {exc}" + ) from exc + if not isinstance(decoded, dict): + raise ManifestInvalidError( + f"{what} for {package_id} is not a JSON object" + ) + return decoded + + +def _find_action( + conn: sqlite3.Connection, package_id: str, action_id: str +) -> dict[str, Any] | None: + row = conn.execute(ACTION_LOOKUP_SQL, (package_id, action_id)).fetchone() + return dict(row) if row is not None else None + + +def _decode_and_check_row( + event: dict[str, Any], *, package_id: str +) -> dict[str, Any]: + """Decode a stored manifest and run the bounded current-row consistency + check (identity/state/digest + schema). Used by load() and the idempotent + replay paths so every path that returns a stored manifest applies the same + verification.""" + manifest = _decode_json_blob(event["manifest_json"], what="stored manifest", + package_id=package_id) + try: + violations = check_current_row_consistency( + package_id=package_id, + event=event, + manifest=manifest, + manifest_json_bytes=bytes(event["manifest_json"]), + ) + except (TypeError, ValueError, UnicodeError, RecursionError) as exc: + raise ManifestInvalidError( + f"consistency check failed for {package_id}: {exc}", + package_id=package_id, + ) from exc + if violations: + raise ManifestInvalidError( + f"consistency violation for {package_id}: {violations[0]}", + package_id=package_id, + ) + return manifest + + +def _verify_new_references( + next_m: dict[str, Any], + current_m: dict[str, Any], + artifacts: "ArtifactStore", + package_id: str, +) -> None: + """Verify every newly referenced artifact blob (transaction step 10).""" + old_inputs = {(i or {}).get("input_id") for i in (current_m.get("inputs") or [])} + for entry in next_m.get("inputs") or []: + if entry.get("input_id") not in old_inputs: + digest = entry.get("content_sha256") + if digest and not artifacts.verify(digest): + raise ArtifactVerificationError( + f"input {entry.get('input_id')!r} content blob {digest} missing/corrupt", + package_id=package_id, + ) + old_arts = {(a or {}).get("artifact_id") for a in (current_m.get("artifacts") or [])} + for art in next_m.get("artifacts") or []: + if art.get("artifact_id") not in old_arts: + digest = art.get("sha256") + if digest and not artifacts.verify(digest): + raise ArtifactVerificationError( + f"artifact {art.get('artifact_id')!r} blob {digest} missing/corrupt", + package_id=package_id, + ) + if (current_m.get("summary") is None) and isinstance(next_m.get("summary"), dict): + digest = next_m["summary"].get("digest") + if digest and not artifacts.verify(digest): + raise ArtifactVerificationError( + f"summary body blob {digest} missing/corrupt", + package_id=package_id, + ) + + +class SqliteManifestStore: + """Canonical transactional store over the frozen SQLite model.""" + + def __init__(self, root: "str | Path", *, artifact_store: "ArtifactStore | None" = None) -> None: + from ..adapters.artifact_store import ArtifactStore # lazy: avoids import cycle + + self._root = validate_store_root(root) + self._conn = open_database(self._root, read_only=False) + try: + self._artifacts = ( + artifact_store if artifact_store is not None else ArtifactStore(self._root) + ) + except BaseException: + close_database(self._conn) + raise + + # ── create ───────────────────────────────────────────────────────── + def create( + self, package_id: str, intent_raw: str, created_at: str | None = None + ) -> dict[str, Any]: + """Create a package at revision 0 (exactly one event). + + Duplicate create raises DuplicatePackageError unless it is an exact + idempotent replay of the original creation action (same deterministic + action_id + same semantic action hash). + """ + validate_package_id(package_id) + if not isinstance(intent_raw, str): + raise InvalidPayloadError("intent_raw must be a string") + if len(intent_raw) > MAX_INTENT_CHARS: + raise InvalidPayloadError( + f"intent_raw exceeds {MAX_INTENT_CHARS} chars" + ) + if contains_control_chars(intent_raw): + raise InvalidPayloadError( + "intent_raw must not contain control characters" + ) + created_at = created_at or _utcnow() + action_id = f"act_create_{package_id}" + event_id = f"evt_{package_id}_0" + + action_bytes = canonical_action_bytes( + protocol_version=PROTOCOL_VERSION, + action=CREATE_PACKAGE_ACTION, + package_id=package_id, + action_id=action_id, + basis={}, + payload={"intent": intent_raw}, + ) + action_hash = sha256_hex(action_bytes) + + conn = self._conn + try: + with _Transaction(conn): + existing = _find_action(conn, package_id, action_id) + if existing is not None: + if existing["action_sha256"] == action_hash: + return _decode_and_check_row(existing, package_id=package_id) + # The package exists (its create was committed with a + # different intent). Per the create contract, attempting + # to create an existing package returns the stable + # PACKAGE_EXISTS error — exact idempotent replay is the + # only accepted repeat. + raise DuplicatePackageError( + f"package {package_id} already exists", package_id=package_id + ) + latest = latest_event_row(conn, package_id) + if latest is not None: + raise DuplicatePackageError( + f"package {package_id} already exists", package_id=package_id + ) + manifest = new_manifest(package_id, intent_raw, created_at) + violations, manifest_bytes = validate_manifest_canonical(manifest) + if violations: + raise ManifestInvalidError( + f"new manifest invalid: {violations[0]}", package_id=package_id + ) + event = { + "package_id": package_id, + "revision": 0, + "event_id": event_id, + "action_id": action_id, + "action": CREATE_PACKAGE_ACTION, + "action_sha256": action_hash, + "state_before": None, + "state_after": manifest["state"], + "previous_manifest_sha256": None, + "resulting_manifest_sha256": sha256_hex(manifest_bytes), + "created_at": created_at, + "action_json": action_bytes, + "manifest_json": manifest_bytes, + } + violations = check_event_invariants( + package_id=package_id, event=event, prev_event=None, + manifest=manifest, decode_blobs=False, + ) + if violations: + raise ManifestInvalidError( + f"chain invariant violation on create: {violations[0]}", + package_id=package_id, + ) + _fault("before_insert") + _insert_event(conn, _event_row_tuple(event)) + _fault("after_insert") + return manifest + except MethodFactoryError: + raise + except sqlite3.OperationalError as exc: + if _is_locked(exc): + raise ConcurrencyError( + f"database is locked: {exc}", package_id=package_id + ) from exc + raise StorageError( + f"create failed for {package_id}: {exc}", package_id=package_id + ) from exc + except (sqlite3.Error, OSError, ValueError, TypeError, UnicodeError, + RecursionError) as exc: + raise StorageError( + f"create failed for {package_id}: {exc}", package_id=package_id + ) from exc + + # ── apply ────────────────────────────────────────────────────────── + def apply(self, envelope: dict[str, Any]) -> dict[str, Any]: + """Apply one action envelope in one bounded transaction. + + `envelope` is the parsed Action Envelope dict (see parse_envelope / + envelope_from_dict). Idempotency lookup (step 3) happens BEFORE the + stale-revision comparison (step 6): a retry of an already-committed + action replays the previous result even with an older + expected_revision; reusing an action_id with a different semantic + hash always returns ACTION_ID_CONFLICT. + """ + if not isinstance(envelope, dict): + raise InvalidPayloadError("envelope must be a JSON object") + env = envelope_from_dict(envelope) # INVALID_ENVELOPE on schema failure + # package_id grammar is enforced authoritatively by envelope_from_dict + # (InvalidEnvelopeError); no redundant re-validation here. + package_id = env.package_id + + action_bytes = canonical_action_bytes( + protocol_version=PROTOCOL_VERSION, + action=env.action, + package_id=package_id, + action_id=env.action_id, + basis=env.basis, + payload=env.payload, + ) + action_hash = sha256_hex(action_bytes) + created_at = _utcnow() + + conn = self._conn + try: + with _Transaction(conn): + existing = _find_action(conn, package_id, env.action_id) + if existing is not None: + if existing["action_sha256"] == action_hash: + # Idempotent replay BEFORE stale check: return the + # previously committed result (consistency-verified), + # insert nothing. + return _decode_and_check_row(existing, package_id=package_id) + raise ActionIdConflictError( + f"action_id {env.action_id!r} reused with different content " + f"for {package_id}", + package_id=package_id, + ) + latest = latest_event_row(conn, package_id) + if latest is None: + raise PackageNotFoundError( + f"package {package_id} does not exist", package_id=package_id + ) + if env.expected_revision != latest["revision"]: + raise StaleActionError( + f"expected revision {env.expected_revision}, " + f"current {latest['revision']}", + package_id=package_id, + state=latest["state_after"], + expected_revision=env.expected_revision, + actual_revision=latest["revision"], + ) + current_manifest = _decode_and_check_row(dict(latest), package_id=package_id) + _fault("after_state_load") + + new_revision = latest["revision"] + 1 + event_id = f"evt_{package_id}_{new_revision}" + next_m, blobs = next_manifest( + current_manifest, env, event_id=event_id, created_at=created_at + ) + _fault("after_transition") + + # Single canonicalization: validate AND obtain the exact + # canonical bytes that will be stored (perf: no second pass). + violations, manifest_bytes = validate_manifest_canonical(next_m) + if violations: + raise ManifestInvalidError( + f"resulting manifest invalid: {violations[0]}", package_id=package_id + ) + _fault("after_manifest_validate") + + for path, content in blobs: + self._artifacts.put(package_id, path, content) + _verify_new_references(next_m, current_manifest, self._artifacts, package_id) + _fault("after_artifact_verify") + + event = { + "package_id": package_id, + "revision": new_revision, + "event_id": event_id, + "action_id": env.action_id, + "action": env.action, + "action_sha256": action_hash, + "state_before": latest["state_after"], + "state_after": next_m["state"], + "previous_manifest_sha256": latest["resulting_manifest_sha256"], + "resulting_manifest_sha256": sha256_hex(manifest_bytes), + "created_at": created_at, + "action_json": action_bytes, + "manifest_json": manifest_bytes, + } + violations = check_event_invariants( + package_id=package_id, + event=event, + prev_event=dict(latest), + manifest=next_m, + decode_blobs=False, # self-produced bytes; digest-bound + ) + if violations: + raise ManifestInvalidError( + f"chain invariant violation on apply: {violations[0]}", + package_id=package_id, + ) + _fault("before_insert") + _insert_event(conn, _event_row_tuple(event)) + _fault("after_insert") + return next_m + except MethodFactoryError: + raise + except sqlite3.OperationalError as exc: + if _is_locked(exc): + raise ConcurrencyError( + f"database is locked: {exc}", package_id=package_id + ) from exc + raise StorageError( + f"apply failed for {package_id}: {exc}", package_id=package_id + ) from exc + except (sqlite3.Error, OSError, ValueError, TypeError, UnicodeError, + RecursionError) as exc: + raise StorageError( + f"apply failed for {package_id}: {exc}", package_id=package_id + ) from exc + + # ── load ─────────────────────────────────────────────────────────── + def load(self, package_id: str) -> dict[str, Any]: + """Return the complete current manifest via the indexed latest-event + query only (never a history scan). Raises PackageNotFoundError for a + missing package; validates current-row consistency (identity/state/ + digest binding + schema) so obviously mismatched rows are never + returned.""" + validate_package_id(package_id) + # latest_event_row already translates sqlite3.Error -> StorageError. + row = latest_event_row(self._conn, package_id) + if row is None: + raise PackageNotFoundError( + f"package {package_id} does not exist", package_id=package_id + ) + return _decode_and_check_row(dict(row), package_id=package_id) + + # ── read_events (ordered audit/export primitive) ─────────────────── + def read_events(self, package_id: str) -> list[dict[str, Any]]: + """Return every event for a package in revision order, with the + stored action/manifest BLOBs decoded to JSON objects.""" + validate_package_id(package_id) + try: + rows = self._conn.execute(EVENTS_ALL_SQL, (package_id,)).fetchall() + except sqlite3.Error as exc: + raise StorageError( + f"read_events failed for {package_id}: {exc}", package_id=package_id + ) from exc + events: list[dict[str, Any]] = [] + for row in rows: + ev = dict(row) + ev["action_json"] = _decode_json_blob( + ev["action_json"], what="stored action", package_id=package_id) + ev["manifest_json"] = _decode_json_blob( + ev["manifest_json"], what="stored manifest", package_id=package_id) + events.append(ev) + return events + + # ── authoritative chain validator (delegates to the single kernel) ── + def validate_chain( + self, package_id: str, *, verify_artifacts: bool = False + ) -> dict[str, Any]: + """Run the authoritative revision-chain validator for one package. + + Raises ChainViolationError on the first invariant violation. + """ + validate_package_id(package_id) + return _validate_chain( + self._conn, + package_id, + verify_artifacts=verify_artifacts, + artifact_store=self._artifacts, + ) + + # ── query-plan evidence ──────────────────────────────────────────── + def explain_latest_plan(self, package_id: str) -> list[tuple]: + """EXPLAIN QUERY PLAN for the hot-path latest-event lookup.""" + return explain_latest_event_plan(self._conn, package_id) + + def close(self) -> None: + close_database(self._conn) From b5adb3e2d7d40097e7a707920bc5e69b6fa1a14f Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Fri, 7 Aug 2026 18:28:14 +0700 Subject: [PATCH 21/41] test(storage): transactional slice evidence - engine/store/chain suites + packaging guard 80 new tests (323 total) proving the transactional contract: - test_engine_apply.py (16): every action's manifest mutation + blob requirements, gate failures, illegal transitions, lineage, malformed current-manifest translation, mutator/transition-table structural guard. - test_transactional_store.py (31): create (revision-zero, duplicate, exact replay, different-intent duplicate), load (missing, complete, mismatched-row rejection), full lifecycle through apply, stale revision, illegal transition, gate failure, idempotent-replay-before-stale ordering, ACTION_ID_CONFLICT, exactly-one-event-per-revision, rollback at all 8 injected pre-commit boundaries (with same-store reuse proving no leaked transaction), commit-failure-then-retry, tampered-current-manifest typed, missing-artifact typed, thread concurrency (separate connections), BEGIN IMMEDIATE busy-timeout -> CONCURRENCY, separate-process concurrency (multiprocessing fork), hot-path query plan, lock errorcode classification. - test_chain_validator.py (26): valid chain; every revision-zero/lineage/ stored-bytes/grammar/artifact invariant corrupted independently; schema tamper detection; manifest-internal lineage binding; non-dict entry guard. - test_envelope.py: action_id grammar at the boundary. - test_manifest.py: validate_manifest non-dict 2-tuple contract regression. - test_packaging.py: guard sharpened - methodfactory.engine is now the NEW pure transition package; JSONL-era modules asserted absent, no legacy API. --- methodfactory/tests/test_chain_validator.py | 523 +++++++++++ methodfactory/tests/test_engine_apply.py | 322 +++++++ methodfactory/tests/test_envelope.py | 9 + methodfactory/tests/test_manifest.py | 12 + methodfactory/tests/test_packaging.py | 18 +- .../tests/test_transactional_store.py | 859 ++++++++++++++++++ 6 files changed, 1739 insertions(+), 4 deletions(-) create mode 100644 methodfactory/tests/test_chain_validator.py create mode 100644 methodfactory/tests/test_engine_apply.py create mode 100644 methodfactory/tests/test_transactional_store.py diff --git a/methodfactory/tests/test_chain_validator.py b/methodfactory/tests/test_chain_validator.py new file mode 100644 index 0000000..e9d754a --- /dev/null +++ b/methodfactory/tests/test_chain_validator.py @@ -0,0 +1,523 @@ +"""Authoritative revision-chain validator tests (ADR-0012 §F). + +A valid chain validates clean; every individual invariant, corrupted +independently, raises ChainViolationError. Tampering bypasses the append-only +triggers via a raw connection (DROP TRIGGER -> UPDATE), matching how a +tampered store file would appear; the validator runs on the already-open +store connection so schema re-verification does not mask the test. + +Each manifest-field tamper recomputes resulting_manifest_sha256 from the +tampered bytes so ONLY the intended invariant fires (the digest binding is +tested separately). +""" + +from __future__ import annotations + +import json +import sqlite3 +import tempfile +import unittest +from pathlib import Path + +from methodfactory.storage.errors import ChainViolationError +from methodfactory.storage.paths import DB_FILENAME +from methodfactory.storage.serialization import canonical_bytes, sha256_hex +from methodfactory.storage.store import SqliteManifestStore + + +def _record_input(action_id="act_in_1", expected_revision=0, input_id="in_1", + content="hello"): + return { + "protocol_version": "0.1", "action_id": action_id, + "package_id": "pkg_demo_001", "expected_revision": expected_revision, + "action": "record_input", "basis": {}, + "payload": {"input_id": input_id, "kind": "text", "content": content, + "source": "operator", "disposition": "incorporated"}, + } + + +def _set_objective(action_id="act_obj_1", expected_revision=1): + return { + "protocol_version": "0.1", "action_id": action_id, + "package_id": "pkg_demo_001", "expected_revision": expected_revision, + "action": "set_objective", "basis": {}, + "payload": {"statement": "Build a skill", "desired_outcomes": []}, + } + + +def _prepare_summary(action_id="act_prep_1", expected_revision=2): + return { + "protocol_version": "0.1", "action_id": action_id, + "package_id": "pkg_demo_001", "expected_revision": expected_revision, + "action": "prepare_summary", "basis": {}, "payload": {}, + } + + +def _confirm_summary(digest, action_id="act_conf_1", expected_revision=3): + return { + "protocol_version": "0.1", "action_id": action_id, + "package_id": "pkg_demo_001", "expected_revision": expected_revision, + "action": "confirm_summary", "basis": {"summary_sha256": digest}, + "payload": {"operator_id": "vincent"}, + } + + +def _record_draft(action_id="act_art_1", expected_revision=4): + return { + "protocol_version": "0.1", "action_id": action_id, + "package_id": "pkg_demo_001", "expected_revision": expected_revision, + "action": "record_draft_artifact", "basis": {}, + "payload": {"artifact_id": "art_1", "kind": "skill", + "logical_path": "skills/x/SKILL.md", "content": "body"}, + } + + +def _full_chain(store): + store.create("pkg_demo_001", "Build a skill", + created_at="2026-08-07T00:00:00+00:00") + store.apply(_record_input(expected_revision=0)) + store.apply(_set_objective(expected_revision=1)) + m3 = store.apply(_prepare_summary(expected_revision=2)) + store.apply(_confirm_summary(m3["summary"]["digest"], expected_revision=3)) + store.apply(_record_draft(expected_revision=4)) + + +def _raw(root: Path): + c = sqlite3.connect(str(root / DB_FILENAME)) + c.execute("DROP TRIGGER IF EXISTS events_no_update") + c.execute("DROP TRIGGER IF EXISTS events_no_delete") + return c + + +def _tamper_manifest(root: Path, package_id: str, revision: int, transform) -> None: + """Decode a stored manifest, transform it, and rewrite bytes + digest.""" + c = _raw(root) + row = c.execute( + "SELECT manifest_json FROM events WHERE package_id=? AND revision=?", + (package_id, revision), + ).fetchone() + m = json.loads(row[0]) + transform(m) + data = canonical_bytes(m) + c.execute( + "UPDATE events SET manifest_json=?, resulting_manifest_sha256=? " + "WHERE package_id=? AND revision=?", + (data, sha256_hex(data), package_id, revision), + ) + c.commit() + c.close() + + +def _tamper_fields(root: Path, package_id: str, revision: int, fields: dict) -> None: + c = _raw(root) + sets = ", ".join(f"{k}=?" for k in fields) + c.execute( + f"UPDATE events SET {sets} WHERE package_id=? AND revision=?", + (*fields.values(), package_id, revision), + ) + c.commit() + c.close() + + +def _delete_event(root: Path, package_id: str, revision: int) -> None: + c = _raw(root) + c.execute("DELETE FROM events WHERE package_id=? AND revision=?", + (package_id, revision)) + c.commit() + c.close() + + +def _tamper_blob(root: Path, package_id: str, revision: int, column: str, data: bytes) -> None: + c = _raw(root) + c.execute( + f"UPDATE events SET {column}=? WHERE package_id=? AND revision=?", + (data, package_id, revision), + ) + c.commit() + c.close() + + +class ChainValidatorTests(unittest.TestCase): + def test_valid_chain_passes(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + result = store.validate_chain("pkg_demo_001") + self.assertEqual(result["package_id"], "pkg_demo_001") + self.assertEqual(result["events"], 6) # create + 5 applies + self.assertTrue(result["valid"]) + finally: + store.close() + + def test_missing_package_raises(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + with self.assertRaises(ChainViolationError): + store.validate_chain("pkg_missing_999") + finally: + store.close() + + +class RevisionZeroInvariantTests(unittest.TestCase): + def test_action_must_be_create_package(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_fields(Path(td), "pkg_demo_001", 0, {"action": "record_input"}) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("revision 0 action", str(ctx.exception)) + finally: + store.close() + + def test_state_before_must_be_null(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_fields(Path(td), "pkg_demo_001", 0, {"state_before": "INTAKE"}) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("state_before", str(ctx.exception)) + finally: + store.close() + + def test_previous_manifest_hash_must_be_null(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_fields(Path(td), "pkg_demo_001", 0, + {"previous_manifest_sha256": "0" * 64}) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("previous_manifest_sha256", str(ctx.exception)) + finally: + store.close() + + def test_manifest_package_id_mismatch(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_manifest(Path(td), "pkg_demo_001", 0, + lambda m: m.update(package_id="pkg_evil_001")) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("package_id", str(ctx.exception)) + finally: + store.close() + + def test_manifest_revision_mismatch(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_manifest(Path(td), "pkg_demo_001", 0, + lambda m: m.update(revision=5)) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("revision", str(ctx.exception)) + finally: + store.close() + + def test_manifest_state_mismatch(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_manifest(Path(td), "pkg_demo_001", 0, + lambda m: m.update(state="CANCELLED")) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("state", str(ctx.exception)) + finally: + store.close() + + def test_manifest_digest_mismatch(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_fields(Path(td), "pkg_demo_001", 0, + {"resulting_manifest_sha256": "0" * 64}) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("manifest_json", str(ctx.exception)) + finally: + store.close() + + def test_action_digest_mismatch(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_fields(Path(td), "pkg_demo_001", 0, + {"action_sha256": "0" * 64}) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("action_json", str(ctx.exception)) + finally: + store.close() + + +class LineageInvariantTests(unittest.TestCase): + def test_state_before_matches_previous_state_after(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_fields(Path(td), "pkg_demo_001", 1, + {"state_before": "CANCELLED"}) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("state_before", str(ctx.exception)) + finally: + store.close() + + def test_previous_hash_matches_previous_resulting(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_fields(Path(td), "pkg_demo_001", 1, + {"previous_manifest_sha256": "0" * 64}) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("previous_manifest_sha256", str(ctx.exception)) + finally: + store.close() + + def test_manifest_revision_matches_indexed(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_manifest(Path(td), "pkg_demo_001", 1, + lambda m: m.update(revision=99)) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("revision", str(ctx.exception)) + finally: + store.close() + + def test_missing_predecessor_detected(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _delete_event(Path(td), "pkg_demo_001", 1) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("predecessor", str(ctx.exception)) + finally: + store.close() + + +class StoredBytesInvariantTests(unittest.TestCase): + def test_malformed_manifest_bytes(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_blob(Path(td), "pkg_demo_001", 0, + "manifest_json", b"{not json") + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("manifest_json", str(ctx.exception)) + finally: + store.close() + + def test_malformed_action_bytes(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_blob(Path(td), "pkg_demo_001", 0, + "action_json", b"\xff\xfe\x00 invalid") + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("action_json", str(ctx.exception)) + finally: + store.close() + + +class GrammarInvariantTests(unittest.TestCase): + def test_bad_event_id_rejected(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_fields(Path(td), "pkg_demo_001", 1, {"event_id": "bad id"}) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("event_id", str(ctx.exception)) + finally: + store.close() + + def test_bad_action_id_rejected(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_fields(Path(td), "pkg_demo_001", 1, {"action_id": "bad id"}) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("action_id", str(ctx.exception)) + finally: + store.close() + + def test_unknown_action_rejected(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_fields(Path(td), "pkg_demo_001", 1, {"action": "frobnicate"}) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("action", str(ctx.exception)) + finally: + store.close() + + +class ManifestLineageBindingTests(unittest.TestCase): + """Local review (bug-2): the manifest-internal lineage claims + (previous_manifest_sha256, transition.last_event_id/last_action_id) are + chain facts the engine writes; the validator must bind them to the + indexed row even when the digest is recomputed consistently.""" + + def test_manifest_previous_hash_bound_to_row(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_manifest(Path(td), "pkg_demo_001", 1, + lambda m: m.update(previous_manifest_sha256="1" * 64)) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("previous_manifest_sha256", str(ctx.exception)) + finally: + store.close() + + def test_manifest_transition_event_id_bound_to_row(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_manifest( + Path(td), "pkg_demo_001", 1, + lambda m: m.update(transition={ + "last_event_id": "evt_evil_99", + "last_action_id": m["transition"]["last_action_id"], + }), + ) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("last_event_id", str(ctx.exception)) + finally: + store.close() + + def test_manifest_transition_action_id_bound_to_row(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_manifest( + Path(td), "pkg_demo_001", 1, + lambda m: m.update(transition={ + "last_event_id": m["transition"]["last_event_id"], + "last_action_id": "act_evil_99", + }), + ) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("last_action_id", str(ctx.exception)) + finally: + store.close() + + +class SchemaTamperTests(unittest.TestCase): + """Local review (sec-1): the authoritative validator re-validates the + manifest schema, so digest-consistent schema-level tamper is detected.""" + + def test_schema_violation_detected(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_manifest( + Path(td), "pkg_demo_001", 1, + lambda m: m.__setitem__("inputs", [{ + **m["inputs"][0], + "content_size": 10**12, + "content_sha256": "0" * 64, + }]), + ) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("schema", str(ctx.exception)) + finally: + store.close() + + def test_load_rejects_schema_invalid_current_manifest(self): + """load() runs the bounded current-row consistency check, which now + includes schema validation: a digest-consistent schema violation in + the latest manifest is rejected (ManifestInvalidError), not returned.""" + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_manifest( + Path(td), "pkg_demo_001", 5, + lambda m: m.__setitem__("artifacts", [{ + **m["artifacts"][0], "byte_count": -5, + }]), + ) + from methodfactory.storage.errors import ManifestInvalidError + with self.assertRaises(ManifestInvalidError): + store.load("pkg_demo_001") + finally: + store.close() + + def test_verify_artifacts_non_dict_entry_no_crash(self): + """Local review (bug-1): a structurally invalid inputs/artifacts entry + must be reported as a violation, never crash with AttributeError.""" + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_manifest( + Path(td), "pkg_demo_001", 1, + lambda m: m.__setitem__("inputs", ["garbage-not-a-dict"]), + ) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001", verify_artifacts=True) + self.assertIn("not an object", str(ctx.exception)) + finally: + store.close() + + +class ArtifactVerificationTests(unittest.TestCase): + def test_missing_referenced_artifact(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + m5 = store.load("pkg_demo_001") + digest = m5["artifacts"][0]["sha256"] + (Path(td) / "blobs" / digest).unlink() + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001", verify_artifacts=True) + self.assertIn("artifact", str(ctx.exception)) + # without artifact verification the chain still validates + result = store.validate_chain("pkg_demo_001") + self.assertTrue(result["valid"]) + finally: + store.close() + + +if __name__ == "__main__": + unittest.main() diff --git a/methodfactory/tests/test_engine_apply.py b/methodfactory/tests/test_engine_apply.py new file mode 100644 index 0000000..1a74d95 --- /dev/null +++ b/methodfactory/tests/test_engine_apply.py @@ -0,0 +1,322 @@ +"""Engine transition unit tests — pure manifest transformation rules. + +Covers the deterministic apply rule (legality -> gate -> mutation -> +revision/lineage) without any storage. Each action's manifest mutation, +blob requirements, illegal transitions, and gate failures are asserted +directly. +""" + +from __future__ import annotations + +import unittest + +from methodfactory.domain.errors import ( + GateUnsatisfiedError, + IllegalTransitionError, + InvalidPayloadError, + StaleActionError, +) +from methodfactory.engine.apply import next_manifest +from methodfactory.manifest.render import render_summary +from methodfactory.manifest.schema import new_manifest +from methodfactory.protocol.envelope import envelope_from_dict +from methodfactory.storage.serialization import digest_bytes, digest_json + + +def _env(action, *, package_id="pkg_demo_001", action_id="act_1", + expected_revision=0, basis=None, payload=None): + return envelope_from_dict({ + "protocol_version": "0.1", + "action_id": action_id, + "package_id": package_id, + "expected_revision": expected_revision, + "action": action, + "basis": basis or {}, + "payload": payload or {}, + }) + + +def _intake(revision=0, **over): + m = new_manifest("pkg_demo_001", "Build a skill", "2026-08-07T00:00:00+00:00") + m["revision"] = revision + m.update(over) + return m + + +def _summary_pending(revision=3): + m = _intake(revision=revision) + m["state"] = "SUMMARY_PENDING" + m["objective"] = {"statement": "Build a skill", "desired_outcomes": []} + m["summary"] = { + "digest": "0" * 64, + "size": 1, + "preview": "p", + "presented_at": "2026-08-07T00:00:00+00:00", + "confirmation": {"status": "pending", "confirmed_at": None, + "operator_id": None, "confirmed_summary_sha256": None}, + } + return m + + +class RecordInputTests(unittest.TestCase): + def test_appends_input_with_digest_size_path(self): + m, blobs = next_manifest( + _intake(), _env("record_input", payload={ + "input_id": "in_1", "kind": "text", "content": "hello", + "source": "operator", "disposition": "incorporated"}), + event_id="evt_1", created_at="2026-08-07T01:00:00+00:00", + ) + self.assertEqual(m["revision"], 1) + self.assertEqual(m["state"], "INTAKE") + self.assertEqual(len(m["inputs"]), 1) + entry = m["inputs"][0] + self.assertEqual(entry["input_id"], "in_1") + self.assertEqual(entry["content_sha256"], digest_bytes(b"hello")) + self.assertEqual(entry["content_size"], 5) + self.assertEqual(entry["content_path"], "inputs/in_1.txt") + self.assertEqual(blobs, [("inputs/in_1.txt", "hello")]) + + def test_excluded_input_keeps_reason(self): + m, _ = next_manifest( + _intake(), _env("record_input", payload={ + "input_id": "in_1", "kind": "url", "content": "https://x", + "source": "operator", "disposition": "excluded", + "exclusion_reason": "duplicate"}), + event_id="evt_1", created_at="2026-08-07T01:00:00+00:00", + ) + self.assertEqual(m["inputs"][0]["exclusion_reason"], "duplicate") + + def test_duplicate_input_id_rejected_by_gate(self): + cur = _intake() + cur["inputs"] = [{ + "input_id": "in_1", "kind": "text", "source": "operator", + "disposition": "incorporated", "exclusion_reason": None, + "content_sha256": "0" * 64, "content_size": 1, + "content_path": "inputs/in_1.txt"}] + with self.assertRaises(InvalidPayloadError): + next_manifest( + cur, _env("record_input", payload={ + "input_id": "in_1", "kind": "text", "content": "x", + "source": "operator", "disposition": "incorporated"}), + event_id="evt_1", created_at="2026-08-07T01:00:00+00:00", + ) + + +class SetObjectiveTests(unittest.TestCase): + def test_sets_objective(self): + m, blobs = next_manifest( + _intake(), _env("set_objective", payload={ + "statement": "Build a skill", "desired_outcomes": ["a", "b"]}), + event_id="evt_1", created_at="2026-08-07T01:00:00+00:00", + ) + self.assertEqual(m["objective"]["statement"], "Build a skill") + self.assertEqual(m["objective"]["desired_outcomes"], ["a", "b"]) + self.assertEqual(blobs, []) + + def test_empty_statement_rejected(self): + with self.assertRaises(InvalidPayloadError): + next_manifest( + _intake(), _env("set_objective", payload={ + "statement": " ", "desired_outcomes": []}), + event_id="evt_1", created_at="2026-08-07T01:00:00+00:00", + ) + + +class PrepareSummaryTests(unittest.TestCase): + def test_produces_content_addressed_summary(self): + cur = _intake(revision=2) + cur["objective"] = {"statement": "Build a skill", "desired_outcomes": []} + m, blobs = next_manifest( + cur, _env("prepare_summary"), + event_id="evt_3", created_at="2026-08-07T03:00:00+00:00", + ) + self.assertEqual(m["state"], "SUMMARY_PENDING") + summary = m["summary"] + body = render_summary(cur) + self.assertEqual(summary["digest"], digest_bytes(body.encode("utf-8"))) + self.assertEqual(summary["size"], len(body.encode("utf-8"))) + self.assertTrue(summary["preview"]) + self.assertEqual(summary["presented_at"], "2026-08-07T03:00:00+00:00") + self.assertEqual(summary["confirmation"]["status"], "pending") + self.assertEqual(blobs, [("summaries/r3.txt", body)]) + + def test_missing_objective_rejected(self): + with self.assertRaises(GateUnsatisfiedError): + next_manifest( + _intake(), _env("prepare_summary"), + event_id="evt_1", created_at="2026-08-07T03:00:00+00:00", + ) + + +class ConfirmSummaryTests(unittest.TestCase): + def test_confirms_with_default_operator(self): + cur = _summary_pending() + m, _ = next_manifest( + cur, _env("confirm_summary", basis={"summary_sha256": "0" * 64}, + payload={}), + event_id="evt_4", created_at="2026-08-07T04:00:00+00:00", + ) + self.assertEqual(m["state"], "AUTHORING_AUTHORIZED") + conf = m["summary"]["confirmation"] + self.assertEqual(conf["status"], "confirmed") + self.assertEqual(conf["operator_id"], "operator") + self.assertEqual(conf["confirmed_summary_sha256"], "0" * 64) + self.assertEqual(conf["confirmed_at"], "2026-08-07T04:00:00+00:00") + + def test_wrong_digest_is_stale(self): + with self.assertRaises(StaleActionError): + next_manifest( + _summary_pending(), _env("confirm_summary", + basis={"summary_sha256": "1" * 64}, + payload={}), + event_id="evt_4", created_at="2026-08-07T04:00:00+00:00", + ) + + def test_missing_summary_rejected(self): + with self.assertRaises(GateUnsatisfiedError): + next_manifest( + _intake(revision=1, state="SUMMARY_PENDING"), + _env("confirm_summary", basis={"summary_sha256": "0" * 64}, + payload={}), + event_id="evt_4", created_at="2026-08-07T04:00:00+00:00", + ) + + +class ReviseIntakeTests(unittest.TestCase): + def test_clears_summary_and_artifacts(self): + cur = _summary_pending() + cur["artifacts"] = [{ + "artifact_id": "art_1", "kind": "skill", + "logical_path": "skills/x/SKILL.md", "sha256": "0" * 64, + "byte_count": 1, "status": "draft"}] + cur["inputs"] = [{ + "input_id": "in_1", "kind": "text", "source": "operator", + "disposition": "incorporated", "exclusion_reason": None, + "content_sha256": "0" * 64, "content_size": 1, + "content_path": "inputs/in_1.txt"}] + m, _ = next_manifest( + cur, _env("revise_intake"), + event_id="evt_5", created_at="2026-08-07T05:00:00+00:00", + ) + self.assertEqual(m["state"], "INTAKE") + self.assertIsNone(m["summary"]) + self.assertEqual(m["artifacts"], []) + # intake material (inputs + objective) is preserved + self.assertEqual(len(m["inputs"]), 1) + self.assertEqual(m["inputs"][0]["input_id"], "in_1") + self.assertEqual(m["objective"]["statement"], "Build a skill") + + +class RecordDraftArtifactTests(unittest.TestCase): + def test_appends_artifact(self): + cur = _summary_pending() + cur["state"] = "AUTHORING_AUTHORIZED" + cur["summary"]["confirmation"] = { + "status": "confirmed", "confirmed_at": "2026-08-07T04:00:00+00:00", + "operator_id": "operator", "confirmed_summary_sha256": "0" * 64} + m, blobs = next_manifest( + cur, _env("record_draft_artifact", payload={ + "artifact_id": "art_1", "kind": "skill", + "logical_path": "skills/x/SKILL.md", "content": "body"}), + event_id="evt_6", created_at="2026-08-07T06:00:00+00:00", + ) + self.assertEqual(m["state"], "DRAFT_READY") + art = m["artifacts"][0] + self.assertEqual(art["sha256"], digest_bytes(b"body")) + self.assertEqual(art["byte_count"], 4) + self.assertEqual(art["status"], "draft") + self.assertEqual(blobs, [("skills/x/SKILL.md", "body")]) + + def test_unconfirmed_authoring_rejected(self): + cur = _summary_pending() + cur["state"] = "AUTHORING_AUTHORIZED" + with self.assertRaises(GateUnsatisfiedError): + next_manifest( + cur, _env("record_draft_artifact", payload={ + "artifact_id": "art_1", "kind": "skill", + "logical_path": "skills/x/SKILL.md", "content": "body"}), + event_id="evt_6", created_at="2026-08-07T06:00:00+00:00", + ) + + +class CancelTests(unittest.TestCase): + def test_cancel_transitions_only(self): + cur = _intake(revision=1) + cur["inputs"] = [{ + "input_id": "in_1", "kind": "text", "source": "operator", + "disposition": "incorporated", "exclusion_reason": None, + "content_sha256": "0" * 64, "content_size": 1, + "content_path": "inputs/in_1.txt"}] + m, blobs = next_manifest( + cur, _env("cancel", payload={"reason": "no longer needed"}), + event_id="evt_2", created_at="2026-08-07T02:00:00+00:00", + ) + self.assertEqual(m["state"], "CANCELLED") + self.assertEqual(len(m["inputs"]), 1) + self.assertEqual(blobs, []) + + +class CommonLineageTests(unittest.TestCase): + def test_revision_state_lineage(self): + cur = _intake(revision=7) + m, _ = next_manifest( + cur, _env("record_input", payload={ + "input_id": "in_9", "kind": "text", "content": "x", + "source": "operator", "disposition": "incorporated"}), + event_id="evt_8", created_at="2026-08-07T08:00:00+00:00", + ) + self.assertEqual(m["revision"], 8) + self.assertEqual(m["updated_at"], "2026-08-07T08:00:00+00:00") + self.assertEqual(m["previous_manifest_sha256"], digest_json(cur)) + self.assertEqual(m["transition"]["last_event_id"], "evt_8") + self.assertEqual(m["transition"]["last_action_id"], "act_1") + + def test_illegal_transition(self): + with self.assertRaises(IllegalTransitionError): + next_manifest( + _summary_pending(), _env("record_input", payload={ + "input_id": "in_1", "kind": "text", "content": "x", + "source": "operator", "disposition": "incorporated"}), + event_id="evt_9", created_at="2026-08-07T09:00:00+00:00", + ) + + def test_malformed_revision_rejected(self): + """Local review (q-3): a non-int/bool revision is rejected typed, not + coerced or leaked.""" + for bad in (None, "abc", [1], 1.5, True): + with self.subTest(revision=bad): + with self.assertRaises(InvalidPayloadError): + next_manifest( + _intake(revision=bad), _env("record_input", payload={ + "input_id": "in_1", "kind": "text", "content": "x", + "source": "operator", "disposition": "incorporated"}), + event_id="evt_9", created_at="2026-08-07T09:00:00+00:00", + ) + + def test_malformed_manifest_render_translated(self): + """Local review (bug-3): a structurally malformed current manifest + cannot leak a raw KeyError from render_summary — it is translated.""" + cur = _intake(revision=2) + cur["objective"] = {"statement": "Build a skill", "desired_outcomes": []} + cur["inputs"] = [{"input_id": "in_1", "kind": "text"}] # missing fields + with self.assertRaises(InvalidPayloadError): + next_manifest( + cur, _env("prepare_summary"), + event_id="evt_3", created_at="2026-08-07T03:00:00+00:00", + ) + + +class MutatorCoverageTests(unittest.TestCase): + def test_every_legal_action_has_a_mutator(self): + """Local review (q-4): the mutator registry is structurally linked to + the Action vocabulary so a future transition-table addition cannot + silently lack an implementation.""" + from methodfactory.domain.transitions import Action + from methodfactory.engine import apply as apply_mod + + registered = set(apply_mod._ACTION_MUTATORS) + self.assertEqual(registered, set(Action)) + + +if __name__ == "__main__": + unittest.main() diff --git a/methodfactory/tests/test_envelope.py b/methodfactory/tests/test_envelope.py index 667bc29..758bbe6 100644 --- a/methodfactory/tests/test_envelope.py +++ b/methodfactory/tests/test_envelope.py @@ -125,6 +125,15 @@ def test_overlong_action_id_rejected(self): with self.assertRaises(InvalidEnvelopeError): parse_envelope(json.dumps(envelope(action_id="a" * 65))) + def test_action_id_grammar_enforced_at_boundary(self): + """Local review (sec-3): action_id must obey the identifier grammar + ([A-Za-z0-9_-]{1,128}) AT the envelope boundary — rejected as + INVALID_ENVELOPE, never deep in the transaction.""" + for bad in ("bad id", "a/b", "../x", "a\x00b", "a;b"): + with self.subTest(action_id=bad): + with self.assertRaises(InvalidEnvelopeError): + parse_envelope(json.dumps(envelope(action_id=bad))) + def test_confirm_requires_basis_summary_sha256(self): with self.assertRaises(InvalidEnvelopeError): parse_envelope(json.dumps(envelope(action="confirm_summary", payload={}))) diff --git a/methodfactory/tests/test_manifest.py b/methodfactory/tests/test_manifest.py index 4aa518c..84bfb07 100644 --- a/methodfactory/tests/test_manifest.py +++ b/methodfactory/tests/test_manifest.py @@ -121,6 +121,18 @@ def test_revision_increments_once(self): self.assertEqual(validate_manifest(m1), []) self.assertEqual(validate_manifest(m2), []) + def test_non_dict_input_collected_not_crashed(self): + """validate_manifest must collect (never raise) for non-dict input, + and validate_manifest_canonical must honor its 2-tuple contract.""" + for bad in (None, "x", 5, [1, 2, 3]): + with self.subTest(value=bad): + errors = validate_manifest(bad) # type: ignore[arg-type] + self.assertTrue(any("must be a JSON object" in e for e in errors)) + from methodfactory.manifest.schema import validate_manifest_canonical + errors, canonical = validate_manifest_canonical(None) # type: ignore[arg-type] + self.assertIsNone(canonical) + self.assertTrue(any("must be a JSON object" in e for e in errors)) + if __name__ == "__main__": unittest.main() diff --git a/methodfactory/tests/test_packaging.py b/methodfactory/tests/test_packaging.py index 14de58f..6fe56de 100644 --- a/methodfactory/tests/test_packaging.py +++ b/methodfactory/tests/test_packaging.py @@ -39,12 +39,22 @@ def test_import_surface(self): def test_old_jsonl_engine_is_absent(self): # The JSONL-era store/engine were removed in the persistence reset and - # must not be importable (ADR-0012 §8 discard list). + # must not be importable (ADR-0012 §8 discard list). The package name + # `methodfactory.engine` is now the NEW pure transition-logic package + # (no persistence, no JSONL-era API), so the guard asserts the + # JSONL-era module and legacy submodules are absent and the new engine + # exposes no legacy surface. import importlib - for mod in ("methodfactory.engine", "methodfactory.manifest.store"): - with self.assertRaises(ImportError): - importlib.import_module(mod) + with self.assertRaises(ImportError): + importlib.import_module("methodfactory.manifest.store") + with self.assertRaises(ImportError): + importlib.import_module("methodfactory.engine.jsonl") + import methodfactory.engine as engine + + self.assertFalse(hasattr(engine, "Engine")) + self.assertFalse(hasattr(engine, "JsonlStore")) + self.assertTrue(callable(engine.apply.next_manifest)) if __name__ == "__main__": diff --git a/methodfactory/tests/test_transactional_store.py b/methodfactory/tests/test_transactional_store.py new file mode 100644 index 0000000..fa728aa --- /dev/null +++ b/methodfactory/tests/test_transactional_store.py @@ -0,0 +1,859 @@ +"""Transactional store tests — create/load/apply, idempotency, concurrency, +rollback fault injection, and query-plan evidence (ADR-0012 §6/§8). + +Every mutation runs the single BEGIN IMMEDIATE transaction; failures roll +back with no event inserted, no historical row mutated, and no blob deleted. +Concurrency is proven with separate connections AND separate processes. +""" + +from __future__ import annotations + +import multiprocessing as mp +import sqlite3 +import tempfile +import threading +import unittest +from pathlib import Path +from unittest import mock + +from methodfactory.domain.errors import ( + ConcurrencyError, + DuplicatePackageError, + GateUnsatisfiedError, + IllegalTransitionError, + InvalidPayloadError, + MethodFactoryError, + StaleActionError, +) +from methodfactory.storage import store as store_mod +from methodfactory.storage.errors import ( + ActionIdConflictError, + ArtifactVerificationError, + ManifestInvalidError, + PackageNotFoundError, + StorageError, +) +from methodfactory.storage.paths import DB_FILENAME +from methodfactory.storage.store import SqliteManifestStore + + +def _envelope(action, *, action_id, expected_revision, package_id="pkg_demo_001", + basis=None, payload=None): + return { + "protocol_version": "0.1", + "action_id": action_id, + "package_id": package_id, + "expected_revision": expected_revision, + "action": action, + "basis": basis or {}, + "payload": payload or {}, + } + + +def _record_input(action_id="act_in_1", expected_revision=0, input_id="in_1", + content="hello", package_id="pkg_demo_001"): + return _envelope( + "record_input", action_id=action_id, expected_revision=expected_revision, + package_id=package_id, + payload={"input_id": input_id, "kind": "text", "content": content, + "source": "operator", "disposition": "incorporated"}, + ) + + +def _set_objective(action_id="act_obj_1", expected_revision=1): + return _envelope( + "set_objective", action_id=action_id, expected_revision=expected_revision, + payload={"statement": "Build a skill", "desired_outcomes": ["ship it"]}, + ) + + +def _prepare_summary(action_id="act_prep_1", expected_revision=2): + return _envelope("prepare_summary", action_id=action_id, + expected_revision=expected_revision) + + +def _confirm_summary(digest, action_id="act_conf_1", expected_revision=3): + return _envelope("confirm_summary", action_id=action_id, + expected_revision=expected_revision, + basis={"summary_sha256": digest}, payload={"operator_id": "vincent"}) + + +def _record_draft(action_id="act_art_1", expected_revision=4): + return _envelope( + "record_draft_artifact", action_id=action_id, + expected_revision=expected_revision, + payload={"artifact_id": "art_1", "kind": "skill", + "logical_path": "skills/x/SKILL.md", "content": "body"}, + ) + + +def _cancel(action_id="act_cancel_1", expected_revision=5): + return _envelope("cancel", action_id=action_id, + expected_revision=expected_revision, + payload={"reason": "done"}) + + +def _full_lifecycle(store, package_id="pkg_demo_001"): + m0 = store.create(package_id, "Build a skill", created_at="2026-08-07T00:00:00+00:00") + m1 = store.apply(_record_input(expected_revision=0)) + m2 = store.apply(_set_objective(expected_revision=1)) + m3 = store.apply(_prepare_summary(expected_revision=2)) + digest = m3["summary"]["digest"] + m4 = store.apply(_confirm_summary(digest, expected_revision=3)) + m5 = store.apply(_record_draft(expected_revision=4)) + m6 = store.apply(_cancel(expected_revision=5)) + return [m0, m1, m2, m3, m4, m5, m6] + + +def _event_count(root) -> int: + with sqlite3.connect(str(root / DB_FILENAME)) as c: + return c.execute("SELECT COUNT(*) FROM events").fetchone()[0] + + +def _event_revisions(root) -> list[int]: + with sqlite3.connect(str(root / DB_FILENAME)) as c: + rows = c.execute("SELECT revision FROM events ORDER BY revision").fetchall() + return [r[0] for r in rows] + + +def _latest_revision(root) -> int | None: + with sqlite3.connect(str(root / DB_FILENAME)) as c: + row = c.execute("SELECT MAX(revision) FROM events").fetchone() + return row[0] + + +def _latest_row(root) -> dict: + with sqlite3.connect(str(root / DB_FILENAME)) as c: + c.row_factory = sqlite3.Row + row = c.execute( + "SELECT * FROM events ORDER BY revision DESC LIMIT 1" + ).fetchone() + return dict(row) + + +class CreateTests(unittest.TestCase): + def test_create_revision_zero(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + m = store.create("pkg_demo_001", "Build a skill", + created_at="2026-08-07T00:00:00+00:00") + self.assertEqual(m["revision"], 0) + self.assertEqual(m["state"], "INTAKE") + self.assertEqual(m["package_id"], "pkg_demo_001") + self.assertEqual(m["intent"]["raw"], "Build a skill") + self.assertIsNone(m["previous_manifest_sha256"]) + rows = store.read_events("pkg_demo_001") + self.assertEqual(len(rows), 1) + self.assertEqual(rows[0]["revision"], 0) + self.assertEqual(rows[0]["action"], "create_package") + self.assertIsNone(rows[0]["state_before"]) + self.assertIsNone(rows[0]["previous_manifest_sha256"]) + finally: + store.close() + + def test_duplicate_create_rejected(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + store.create("pkg_demo_001", "Build a skill") + with self.assertRaises(DuplicatePackageError) as ctx: + store.create("pkg_demo_001", "Different intent") + self.assertEqual(ctx.exception.code, "PACKAGE_EXISTS") + self.assertEqual(_event_count(Path(td)), 1) + finally: + store.close() + + def test_exact_create_replay_returns_original(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + m1 = store.create("pkg_demo_001", "Build a skill", + created_at="2026-08-07T00:00:00+00:00") + m2 = store.create("pkg_demo_001", "Build a skill", + created_at="2026-08-07T00:00:00+00:00") + self.assertEqual(m1, m2) + self.assertEqual(_event_count(Path(td)), 1) # no second insert + finally: + store.close() + + def test_create_different_intent_is_duplicate(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + store.create("pkg_demo_001", "Build a skill") + # Creating an existing package with different intent is NOT a + # replay -> the stable PACKAGE_EXISTS error (create contract). + with self.assertRaises(DuplicatePackageError) as ctx: + store.create("pkg_demo_001", "Build a DIFFERENT skill") + self.assertEqual(ctx.exception.code, "PACKAGE_EXISTS") + finally: + store.close() + + def test_create_invalid_intent(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + for bad in (None, 5, "x" * 65537, "bad\x01intent"): + with self.subTest(intent=bad): + with self.assertRaises(InvalidPayloadError): + store.create("pkg_demo_001", bad) # type: ignore[arg-type] + self.assertEqual(_event_count(Path(td)), 0) + finally: + store.close() + + +class LoadTests(unittest.TestCase): + def test_load_missing_package_raises(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + with self.assertRaises(PackageNotFoundError) as ctx: + store.load("pkg_missing_999") + self.assertEqual(ctx.exception.code, "PACKAGE_NOT_FOUND") + finally: + store.close() + + def test_load_returns_complete_current_manifest(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + manifests = _full_lifecycle(store) + loaded = store.load("pkg_demo_001") + self.assertEqual(loaded, manifests[-1]) + self.assertEqual(loaded["revision"], 6) + self.assertEqual(loaded["state"], "CANCELLED") + finally: + store.close() + + def test_load_rejects_mismatched_row(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_lifecycle(store) + # Tamper the latest row's manifest_json state field and + # recompute the digest so ONLY the state binding fires. + import json as _json + + from methodfactory.storage.serialization import canonical_bytes, sha256_hex + c = sqlite3.connect(str(Path(td) / DB_FILENAME)) + c.execute("DROP TRIGGER IF EXISTS events_no_update") + row = c.execute( + "SELECT manifest_json FROM events WHERE package_id='pkg_demo_001' " + "ORDER BY revision DESC LIMIT 1" + ).fetchone() + m = _json.loads(row[0]) + m["state"] = "INTAKE" + data = canonical_bytes(m) + c.execute( + "UPDATE events SET manifest_json=?, resulting_manifest_sha256=? " + "WHERE package_id='pkg_demo_001' AND revision=6", + (data, sha256_hex(data)), + ) + c.commit() + c.close() + with self.assertRaises(ManifestInvalidError): + store.load("pkg_demo_001") + finally: + store.close() + + +class ApplyLifecycleTests(unittest.TestCase): + def test_full_lifecycle(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + [m0, m1, m2, m3, m4, m5, m6] = _full_lifecycle(store) + self.assertEqual([m["revision"] for m in (m0, m1, m2, m3, m4, m5, m6)], + [0, 1, 2, 3, 4, 5, 6]) + self.assertEqual([m["state"] for m in (m0, m1, m2, m3, m4, m5, m6)], + ["INTAKE", "INTAKE", "INTAKE", "SUMMARY_PENDING", + "AUTHORING_AUTHORIZED", "DRAFT_READY", "CANCELLED"]) + self.assertEqual(len(m1["inputs"]), 1) + self.assertEqual(m2["objective"]["statement"], "Build a skill") + self.assertIsNotNone(m3["summary"]["digest"]) + self.assertEqual(m4["summary"]["confirmation"]["status"], "confirmed") + self.assertEqual(len(m5["artifacts"]), 1) + # exactly one event per revision, contiguous + self.assertEqual(_event_revisions(Path(td)), [0, 1, 2, 3, 4, 5, 6]) + store.validate_chain("pkg_demo_001") + finally: + store.close() + + def test_revise_intake_path(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + store.create("pkg_demo_001", "Build a skill") + store.apply(_record_input(expected_revision=0)) + store.apply(_set_objective(expected_revision=1)) + m3 = store.apply(_prepare_summary(expected_revision=2)) + digest = m3["summary"]["digest"] + m4 = store.apply(_confirm_summary(digest, expected_revision=3)) + self.assertEqual(m4["state"], "AUTHORING_AUTHORIZED") + m5 = store.apply(_envelope( + "revise_intake", action_id="act_rev_1", expected_revision=4)) + self.assertEqual(m5["state"], "INTAKE") + self.assertIsNone(m5["summary"]) + # intake material preserved (inputs + objective survive) + self.assertEqual(len(m5["inputs"]), 1) + self.assertEqual(m5["inputs"][0]["input_id"], "in_1") + self.assertEqual(m5["objective"]["statement"], "Build a skill") + store.validate_chain("pkg_demo_001") + finally: + store.close() + + def test_stale_revision_rejected(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + store.create("pkg_demo_001", "Build a skill") + store.apply(_record_input(expected_revision=0)) + with self.assertRaises(StaleActionError) as ctx: + store.apply(_record_input(action_id="act_in_2", input_id="in_2", + expected_revision=0)) + self.assertEqual(ctx.exception.code, "STALE_ACTION") + self.assertEqual(ctx.exception.expected_revision, 0) + self.assertEqual(ctx.exception.actual_revision, 1) + finally: + store.close() + + def test_illegal_transition_rejected(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + store.create("pkg_demo_001", "Build a skill") + store.apply(_record_input(expected_revision=0)) + store.apply(_set_objective(expected_revision=1)) + store.apply(_prepare_summary(expected_revision=2)) + # record_input is not legal from SUMMARY_PENDING + with self.assertRaises(IllegalTransitionError) as ctx: + store.apply(_record_input(action_id="act_in_9", input_id="in_9", + expected_revision=3)) + self.assertEqual(ctx.exception.code, "ILLEGAL_TRANSITION") + finally: + store.close() + + def test_gate_failure_rejected(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + store.create("pkg_demo_001", "Build a skill") + store.apply(_record_input(expected_revision=0)) + # prepare_summary requires an objective + with self.assertRaises(GateUnsatisfiedError) as ctx: + store.apply(_prepare_summary(action_id="act_prep_9", + expected_revision=1)) + self.assertEqual(ctx.exception.code, "GATE_UNSATISFIED") + finally: + store.close() + + def test_apply_missing_package(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + with self.assertRaises(PackageNotFoundError): + store.apply(_record_input(action_id="act_x", expected_revision=0, + package_id="pkg_missing_999")) + finally: + store.close() + + def test_apply_invalid_envelope(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + from methodfactory.domain.errors import InvalidEnvelopeError + with self.assertRaises(InvalidEnvelopeError): + store.apply({"protocol_version": "0.1", "action_id": "act_1", + "package_id": "pkg_demo_001", + "expected_revision": 0, "action": "nope", + "basis": {}, "payload": {}}) + with self.assertRaises(InvalidPayloadError): + store.apply("not a dict") # type: ignore[arg-type] + finally: + store.close() + + def test_apply_tampered_current_manifest_typed(self): + """A tampered current manifest (digest-consistent but schema-invalid, + e.g. summary is a string) is rejected typed before the transition — + never a raw AttributeError from the gates (type-guarded).""" + import json as _json + + from methodfactory.storage.serialization import canonical_bytes, sha256_hex + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + store.create("pkg_demo_001", "Build a skill") + store.apply(_record_input(expected_revision=0)) + store.apply(_set_objective(expected_revision=1)) + store.apply(_prepare_summary(expected_revision=2)) + # tamper the latest manifest: state SUMMARY_PENDING with a + # non-dict summary; recompute the digest so ONLY the schema + # violation is present + c = sqlite3.connect(str(Path(td) / DB_FILENAME)) + c.execute("DROP TRIGGER IF EXISTS events_no_update") + row = c.execute( + "SELECT manifest_json FROM events WHERE package_id='pkg_demo_001' " + "ORDER BY revision DESC LIMIT 1" + ).fetchone() + m = _json.loads(row[0]) + m["summary"] = "SOME STRING" + data = canonical_bytes(m) + c.execute( + "UPDATE events SET manifest_json=?, resulting_manifest_sha256=? " + "WHERE package_id='pkg_demo_001' AND revision=3", + (data, sha256_hex(data)), + ) + c.commit() + c.close() + with self.assertRaises(MethodFactoryError) as ctx: + store.apply(_confirm_summary("0" * 64, action_id="act_conf_tamper", + expected_revision=3)) + self.assertNotIsInstance(ctx.exception, AttributeError) + self.assertIn(ctx.exception.code, + ("MANIFEST_INVALID", "GATE_UNSATISFIED")) + finally: + store.close() + + +class IdempotencyOrderingTests(unittest.TestCase): + def test_replay_before_stale_check(self): + """Same action_id + same hash with an OLDER expected_revision must + replay the committed result, NOT fail stale (lookup happens before + the revision comparison).""" + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + store.create("pkg_demo_001", "Build a skill") + m1 = store.apply(_record_input(expected_revision=0)) + # Advance the package, then retry act_in_1 with the ORIGINAL + # expected_revision (0) — must replay m1, not raise stale. + store.apply(_set_objective(expected_revision=1)) + store.apply(_prepare_summary(expected_revision=2)) + replay = store.apply(_record_input( + action_id="act_in_1", input_id="in_1", content="hello", + expected_revision=0, + )) + self.assertEqual(replay, m1) + # exactly one event per action_id; 4 events total + # (create + 3 applies: record_input, set_objective, prepare_summary) + self.assertEqual(_event_revisions(Path(td)), [0, 1, 2, 3]) + finally: + store.close() + + def test_same_action_id_different_hash_conflicts(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + store.create("pkg_demo_001", "Build a skill") + store.apply(_record_input(action_id="act_x", input_id="in_1", + content="hello", expected_revision=0)) + with self.assertRaises(ActionIdConflictError) as ctx: + store.apply(_record_input(action_id="act_x", input_id="in_1", + content="DIFFERENT", expected_revision=0)) + self.assertEqual(ctx.exception.code, "ACTION_ID_CONFLICT") + self.assertEqual(_event_count(Path(td)), 2) # no extra insert + finally: + store.close() + + def test_conflict_wins_over_stale(self): + """Reusing an action_id with different content returns CONFLICT even + when the expected_revision is also stale (lookup ordering).""" + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + store.create("pkg_demo_001", "Build a skill") + store.apply(_record_input(action_id="act_x", input_id="in_1", + content="hello", expected_revision=0)) + with self.assertRaises(ActionIdConflictError): + store.apply(_record_input(action_id="act_x", input_id="in_1", + content="DIFFERENT", expected_revision=5)) + finally: + store.close() + + def test_exactly_one_event_per_successful_revision(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_lifecycle(store) + # re-apply an already committed action (replay) -> no insert + m3 = store.load("pkg_demo_001") + # replay the record_input from rev 1 + store.apply(_record_input(action_id="act_in_1", input_id="in_1", + content="hello", expected_revision=0)) + self.assertEqual(_event_revisions(Path(td)), [0, 1, 2, 3, 4, 5, 6]) + finally: + store.close() + + +class RollbackFaultTests(unittest.TestCase): + STAGES = [ + "before_begin", + "after_begin", + "after_state_load", + "after_transition", + "after_manifest_validate", + "after_artifact_verify", + "before_insert", + "after_insert", + ] + + def _hook(self, stage): + def hook(s): + if s == stage: + raise StorageError(f"fault at {stage}") + return hook + + def test_rollback_at_each_precommit_boundary(self): + for stage in self.STAGES: + with self.subTest(stage=stage): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + store.create("pkg_demo_001", "Build a skill") + old = store_mod.FAULT_HOOK + store_mod.FAULT_HOOK = self._hook(stage) + try: + with self.assertRaises(StorageError): + store.apply(_record_input(expected_revision=0)) + finally: + store_mod.FAULT_HOOK = old + # no new event; current revision unchanged; historical + # rows intact (verified via a SEPARATE connection) + self.assertEqual(_event_count(Path(td)), 1) + self.assertEqual(_latest_revision(Path(td)), 0) + row = _latest_row(Path(td)) + self.assertEqual(row["action"], "create_package") + self.assertEqual(row["state_after"], "INTAKE") + # the SAME store must remain usable: no leaked + # transaction (a subsequent valid action commits) + m = store.apply(_record_input( + action_id=f"act_after_{stage}", input_id="in_after", + expected_revision=0)) + self.assertEqual(m["revision"], 1) + self.assertEqual(_event_revisions(Path(td)), [0, 1]) + finally: + store.close() + + def test_commit_failure_then_retry_succeeds(self): + """A commit failure BEFORE the database commits is classified: the + transaction rolls back, no event is visible, and a retry commits + cleanly (the failure was pre-commit, not an ambiguous outcome).""" + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + store.create("pkg_demo_001", "Build a skill") + with mock.patch.object(store_mod, "_commit", + side_effect=StorageError("commit fail")): + with self.assertRaises(StorageError): + store.apply(_record_input(expected_revision=0)) + self.assertEqual(_event_count(Path(td)), 1) + self.assertEqual(_latest_revision(Path(td)), 0) + # retry after the known-failed commit + m = store.apply(_record_input(expected_revision=0)) + self.assertEqual(m["revision"], 1) + self.assertEqual(_event_revisions(Path(td)), [0, 1]) + finally: + store.close() + + def test_precommit_failure_keeps_blobs(self): + """Blobs written inside the transaction before a later fault are NOT + deleted on rollback (content-addressed and immutable).""" + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + store.create("pkg_demo_001", "Build a skill") + old = store_mod.FAULT_HOOK + store_mod.FAULT_HOOK = self._hook("before_insert") + try: + with self.assertRaises(StorageError): + store.apply(_record_input(expected_revision=0)) + finally: + store_mod.FAULT_HOOK = old + blobs = list((Path(td) / "blobs").glob("*")) + self.assertTrue(blobs, "prewritten content blob must remain") + finally: + store.close() + + def test_no_raw_exception_escapes(self): + """A native failure of a contract-listed family (type error here, + simulating an unexpected internal TypeError) is translated to typed + StorageError — no raw exception escapes the transactional API.""" + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + store.create("pkg_demo_001", "Build a skill") + old = store_mod.FAULT_HOOK + store_mod.FAULT_HOOK = lambda s: (_ for _ in ()).throw( + TypeError("unexpected native failure") + ) + try: + with self.assertRaises(StorageError) as ctx: + store.apply(_record_input(expected_revision=0)) + self.assertEqual(ctx.exception.code, "STORAGE_ERROR") + finally: + store_mod.FAULT_HOOK = old + finally: + store.close() + + +class MissingArtifactTests(unittest.TestCase): + def test_missing_artifact_blob_raises_typed(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + store.create("pkg_demo_001", "Build a skill") + store.apply(_record_input(expected_revision=0)) + store.apply(_set_objective(expected_revision=1)) + m3 = store.apply(_prepare_summary(expected_revision=2)) + store.apply(_confirm_summary(m3["summary"]["digest"], expected_revision=3)) + store.apply(_record_draft(expected_revision=4)) + blob_dir = Path(td) / "blobs" + m5 = store.load("pkg_demo_001") + digest = m5["artifacts"][0]["sha256"] + (blob_dir / digest).unlink() + # chain validator in artifact-verification mode surfaces the + # missing blob as a typed CHAIN_VIOLATION + from methodfactory.storage.errors import ChainViolationError + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001", verify_artifacts=True) + self.assertIn("missing/corrupt", str(ctx.exception)) + finally: + store.close() + + def test_apply_verifies_newly_referenced_blobs(self): + """Transaction step 10: a newly referenced blob that fails + verification (patched verify -> False) raises ArtifactVerificationError + and the transaction rolls back.""" + from unittest import mock + + from methodfactory.adapters import artifact_store as art_mod + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + store.create("pkg_demo_001", "Build a skill") + with mock.patch.object( + art_mod.ArtifactStore, "verify", return_value=False + ): + with self.assertRaises(ArtifactVerificationError) as ctx: + store.apply(_record_input(expected_revision=0)) + self.assertEqual(ctx.exception.code, "ARTIFACT_VERIFICATION") + self.assertEqual(_event_count(Path(td)), 1) # rolled back + self.assertEqual(_latest_revision(Path(td)), 0) + finally: + store.close() + + +class ConcurrencyTests(unittest.TestCase): + def test_two_writers_distinct_actions_one_wins(self): + """Two writers from the same revision cannot both commit distinct + next revisions: one commits revision 1, the loser returns the stable + stale error; no duplicate revision; no partial event. Each writer + opens its OWN store/connection in its own thread (SQLite connections + are thread-bound; separate connections are the real contention + evidence).""" + with tempfile.TemporaryDirectory() as td: + root = Path(td) + setup = SqliteManifestStore(root) + setup.create("pkg_demo_001", "Build a skill") + setup.close() + barrier = threading.Barrier(2) + results: dict[str, tuple] = {} + + def writer(name, envelope): + s = SqliteManifestStore(root) + try: + barrier.wait() + m = s.apply(envelope) + results[name] = ("ok", m["revision"]) + except Exception as exc: # noqa: BLE001 + results[name] = ("err", type(exc).__name__, + getattr(exc, "code", None)) + finally: + s.close() + + t1 = threading.Thread(target=writer, args=( + "a", _record_input(action_id="act_a", input_id="in_a"))) + t2 = threading.Thread(target=writer, args=( + "b", _record_input(action_id="act_b", input_id="in_b"))) + t1.start(); t2.start(); t1.join(); t2.join() + + oks = [v for v in results.values() if v[0] == "ok"] + errs = [v for v in results.values() if v[0] == "err"] + self.assertEqual(len(oks), 1, results) + self.assertEqual(oks[0][1], 1) + self.assertEqual(len(errs), 1, results) + self.assertEqual(errs[0][1], "StaleActionError") + self.assertEqual(errs[0][2], "STALE_ACTION") + self.assertEqual(_event_revisions(root), [0, 1]) + self.assertEqual(_event_count(root), 2) + + def test_same_action_concurrent_retries_converge(self): + """Same action_id + same hash concurrently: exactly one event; both + writers return the same manifest (one commits, one replays).""" + with tempfile.TemporaryDirectory() as td: + root = Path(td) + setup = SqliteManifestStore(root) + setup.create("pkg_demo_001", "Build a skill") + setup.close() + barrier = threading.Barrier(2) + results: list = [] + + def writer(): + s = SqliteManifestStore(root) + try: + barrier.wait() + results.append(("ok", s.apply( + _record_input(action_id="act_same", input_id="in_1")))) + except Exception as exc: # noqa: BLE001 + results.append(("err", type(exc).__name__, + getattr(exc, "code", None))) + finally: + s.close() + + t1 = threading.Thread(target=writer) + t2 = threading.Thread(target=writer) + t1.start(); t2.start(); t1.join(); t2.join() + + self.assertEqual(len(results), 2) + self.assertTrue(all(r[0] == "ok" for r in results), results) + self.assertEqual(results[0][1], results[1][1]) + self.assertEqual(_event_count(root), 2) # create + one apply + self.assertEqual(_event_revisions(root), [0, 1]) + + def test_conflicting_same_action_id_cannot_both_succeed(self): + with tempfile.TemporaryDirectory() as td: + root = Path(td) + setup = SqliteManifestStore(root) + setup.create("pkg_demo_001", "Build a skill") + setup.close() + barrier = threading.Barrier(2) + results: list = [] + + def writer(content): + s = SqliteManifestStore(root) + try: + barrier.wait() + results.append(("ok", s.apply( + _record_input(action_id="act_conflict", input_id="in_1", + content=content)))) + except Exception as exc: # noqa: BLE001 + results.append(("err", type(exc).__name__, + getattr(exc, "code", None))) + finally: + s.close() + + t1 = threading.Thread(target=writer, args=("hello",)) + t2 = threading.Thread(target=writer, args=("world",)) + t1.start(); t2.start(); t1.join(); t2.join() + + oks = [r for r in results if r[0] == "ok"] + errs = [r for r in results if r[0] == "err"] + self.assertEqual(len(oks), 1, results) + self.assertEqual(len(errs), 1, results) + self.assertEqual(errs[0][1], "ActionIdConflictError") + self.assertEqual(errs[0][2], "ACTION_ID_CONFLICT") + self.assertEqual(_event_count(root), 2) + + def test_begin_immediate_bounded_by_busy_timeout(self): + """A held write lock makes BEGIN IMMEDIATE wait the configured busy + timeout then surface a typed ConcurrencyError — never a raw sqlite + exception.""" + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + blocker = sqlite3.connect(str(Path(td) / DB_FILENAME)) + try: + store.create("pkg_demo_001", "Build a skill") + blocker.execute("BEGIN IMMEDIATE") + with self.assertRaises(ConcurrencyError) as ctx: + store.apply(_record_input(expected_revision=0)) + self.assertEqual(ctx.exception.code, "CONCURRENCY") + self.assertEqual(_event_count(Path(td)), 1) + finally: + blocker.rollback() + blocker.close() + store.close() + + +def _proc_apply(root_str: str, envelope: dict, barrier, q): + s = SqliteManifestStore(root_str) + try: + m = s.apply(envelope) + q.put(("ok", m["revision"])) + except Exception as exc: # noqa: BLE001 + q.put(("err", type(exc).__name__, getattr(exc, "code", None))) + finally: + s.close() + + +class SeparateProcessConcurrencyTests(unittest.TestCase): + def test_two_processes_from_same_revision(self): + """Real separate-process writers: one commits revision 1, the loser + returns the stable stale error; no duplicate revision; no partial + event (multiprocessing fork context).""" + ctx = mp.get_context("fork") + with tempfile.TemporaryDirectory() as td: + root = Path(td) + store = SqliteManifestStore(root) + store.create("pkg_demo_001", "Build a skill") + store.close() + + barrier = ctx.Barrier(2) + q = ctx.Queue() + p1 = ctx.Process(target=_proc_apply, args=( + str(root), _record_input(action_id="act_p1", input_id="in_p1"), + barrier, q)) + p2 = ctx.Process(target=_proc_apply, args=( + str(root), _record_input(action_id="act_p2", input_id="in_p2"), + barrier, q)) + p1.start(); p2.start(); p1.join(30); p2.join(30) + self.assertEqual(p1.exitcode, 0) + self.assertEqual(p2.exitcode, 0) + + results = [q.get(timeout=5) for _ in range(2)] + oks = [r for r in results if r[0] == "ok"] + errs = [r for r in results if r[0] == "err"] + self.assertEqual(len(oks), 1, results) + self.assertEqual(oks[0][1], 1) + self.assertEqual(len(errs), 1, results) + self.assertEqual(errs[0][1], "StaleActionError") + self.assertEqual(errs[0][2], "STALE_ACTION") + self.assertEqual(_event_revisions(root), [0, 1]) + self.assertEqual(_event_count(root), 2) + + +class QueryPlanTests(unittest.TestCase): + def test_hot_path_plan_uses_primary_key_after_events(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_lifecycle(store) + plan = store.explain_latest_plan("pkg_demo_001") + text = " ".join(" ".join(str(c) for c in row) for row in plan) + self.assertIn("SEARCH events USING", text) + self.assertNotIn("SCAN events", text) + finally: + store.close() + + +class LockClassificationTests(unittest.TestCase): + def test_locked_classified_by_errorcode(self): + """Local review (sec-4/q-8): lock classification uses the SQLite + extended error code, not message substrings.""" + busy = sqlite3.OperationalError("database is locked") + busy.sqlite_errorcode = sqlite3.SQLITE_BUSY + self.assertTrue(store_mod._is_locked(busy)) + locked = sqlite3.OperationalError("database table is locked: events") + locked.sqlite_errorcode = sqlite3.SQLITE_LOCKED + self.assertTrue(store_mod._is_locked(locked)) + # an unrelated operational error is NOT lock-classified + other = sqlite3.OperationalError("no such table: events") + other.sqlite_errorcode = sqlite3.SQLITE_ERROR + self.assertFalse(store_mod._is_locked(other)) + # pre-3.11 fallback: message heuristic when no errorcode attribute + legacy = sqlite3.OperationalError("database is locked") + self.assertTrue(store_mod._is_locked(legacy)) + + +if __name__ == "__main__": + unittest.main() From e8b7879f6f338fe4b92afe1ea6cbff1345676d93 Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Fri, 7 Aug 2026 21:27:32 +0700 Subject: [PATCH 22/41] fix(storage): closure A1-A3 - semantic create identity, canonical stored JSON, action binding (senior review 4882624484) A1 (create idempotency covers the complete requested outcome): - created_at is now SEMANTIC: normalized to UTC ISO-8601 (naive/offset-less rejected), included in the canonical create_package payload, part of action_sha256. A repeat with an EXPLICIT different instant is PACKAGE_EXISTS; a retry that OMITS created_at replays using the stored creation time. - _normalize_created_at collapses every spelling of the same instant (astimezone(UTC)); documented as a deliberate pre-release store-format break (rev-0 action_json payload now carries created_at) in docs/public-surface.md. A2 (stored action/manifest JSON must themselves be canonical): - _check_canonical_form is the SINGLE implementation of the canonical-JSON invariant: canonical_bytes(decoded) == stored bytes AND sha256(canonical) == stored digest. A syntactically valid but non-canonical representation with a recomputed raw-byte hash FAILS. Used by both the validator decode path and the hot-path current-row check (which now canonicalizes ONCE, reusing the schema validator's canonical pass). No second serializer created. A3 (bind decoded action JSON back to the indexed event): - the canonical semantic action must have the exact frozen field set and bind protocol_version, package_id, action_id, action to the indexed event; revision 0 requires the canonical create_package structure (basis empty, payload.intent string, payload.created_at == row created_at). Canonical actions for a different package / action_id / action (recomputed hashes) are all rejected. Upgrade policy for future protocol_version bumps documented (fail-closed, migration required). Residuals (documented in docs/public-surface.md 'Accepted residuals'): rev>0 payload content not yet cross-bound to manifest consequence digests; manifest created_at/updated_at not bound to row timestamps; validate_chain audit path still double-canonicalizes; test tamper-pattern duplication (nit). --- docs/public-surface.md | 40 ++++ methodfactory/storage/chain.py | 208 +++++++++++++++--- methodfactory/storage/store.py | 71 +++++- methodfactory/tests/test_chain_validator.py | 186 ++++++++++++++++ .../tests/test_public_error_boundary.py | 97 ++++---- methodfactory/tests/test_sqlite_open.py | 43 ++-- methodfactory/tests/test_temp_hygiene.py | 66 ++++++ .../tests/test_transactional_store.py | 123 +++++++++++ 8 files changed, 726 insertions(+), 108 deletions(-) create mode 100644 methodfactory/tests/test_temp_hygiene.py diff --git a/docs/public-surface.md b/docs/public-surface.md index 42e1b5d..686f426 100644 --- a/docs/public-surface.md +++ b/docs/public-surface.md @@ -30,6 +30,18 @@ exception contract and are not part of the supported surface. | `close_database(conn)` | `sqlite3.Connection` | `None` | `StorageError` (`STORAGE_ERROR`) | `sqlite3.Error` | | `SqliteManifestStore(root, *, artifact_store=None)` | `str`/`Path` root, optional `ArtifactStore` | store object | `InvalidStoreRootError` (`INVALID_STORE_ROOT`), `StorageError` (`STORAGE_ERROR`) | `TypeError`/`ValueError` (root), `OSError`, `sqlite3.Error` | | `SqliteManifestStore.create(package_id, intent_raw, created_at=None)` | str + str + optional str | complete revision-0 manifest `dict` | `DuplicatePackageError` (`PACKAGE_EXISTS`, non-replay duplicate), `InvalidPayloadError` (`INVALID_PAYLOAD`), `InvalidPackageIdError` (`INVALID_PACKAGE_ID`), `ManifestInvalidError` (`MANIFEST_INVALID`), `ConcurrencyError` (`CONCURRENCY`), `StorageError` (`STORAGE_ERROR`) | `TypeError`/`ValueError`/`UnicodeError`/`RecursionError`, `OSError`, `sqlite3.Error` (incl. locked -> `CONCURRENCY`) | + +> **Create identity (senior review 4882624484, A1):** `created_at` is SEMANTIC. +> It is normalized to UTC ISO-8601 (naive/offset-less timestamps rejected) and +> included in the canonical `create_package` action payload, so it is part of +> `action_sha256`. Exact replay requires the same normalized instant; a repeat +> with an explicitly DIFFERENT instant is `PACKAGE_EXISTS`; a retry that omits +> `created_at` replays using the stored creation time. This is a deliberate +> pre-release breaking change to the SQLite store format (revision-0 +> `action_json` payload now carries `created_at`); no released stores exist +> (PR #1 Draft, v2.0.0a1), so no migration is required — any pre-A1 test store +> must be recreated. The authoritative chain validator binds +> `payload.created_at` to the indexed row `created_at`. | `SqliteManifestStore.apply(envelope)` | `dict` (parsed Action Envelope) | complete resulting manifest `dict` | `InvalidEnvelopeError` (`INVALID_ENVELOPE`), `InvalidPayloadError` (`INVALID_PAYLOAD`), `PackageNotFoundError` (`PACKAGE_NOT_FOUND`), `StaleActionError` (`STALE_ACTION`), `IllegalTransitionError` (`ILLEGAL_TRANSITION`), `GateUnsatisfiedError` (`GATE_UNSATISFIED`), `ActionIdConflictError` (`ACTION_ID_CONFLICT`), `SerializationError` (`SERIALIZATION`), `ArtifactVerificationError` (`ARTIFACT_VERIFICATION`), `ManifestInvalidError` (`MANIFEST_INVALID`), `ConcurrencyError` (`CONCURRENCY`), `StorageError` (`STORAGE_ERROR`) | `TypeError`/`ValueError`/`UnicodeError`/`RecursionError`, `OSError`, `sqlite3.Error` (incl. locked -> `CONCURRENCY`) | | `SqliteManifestStore.load(package_id)` | str | complete current manifest `dict` | `PackageNotFoundError` (`PACKAGE_NOT_FOUND`), `ManifestInvalidError` (`MANIFEST_INVALID`), `StorageError` (`STORAGE_ERROR`) | `TypeError`/`ValueError`/`UnicodeError`/`RecursionError`, `sqlite3.Error` | | `SqliteManifestStore.read_events(package_id)` | str | ordered `list[dict]` (decoded action/manifest) | `ManifestInvalidError` (`MANIFEST_INVALID`), `StorageError` (`STORAGE_ERROR`) | `sqlite3.Error`, decode/JSON errors | @@ -80,3 +92,31 @@ Validation failure handling is deliberate and split by surface: aborts the operation). Both surfaces are stable; neither leaks raw native exceptions. + +## Accepted residuals (closure A, senior review 4882624484) + +Recorded here so the acceptance is repository-durable (verified by the +code-review verify lane on 2026-08-07; each is genuine in the code): + +1. **Action payload → manifest consequence cross-binding (A3)**: for + revisions > 0 the semantic-action binding anchors the frozen field set, + `protocol_version`, `package_id`, `action_id`, `action`, and basis/payload + object types, but NOT payload content against manifest consequence digests + (e.g. `record_input` content vs `inputs[].content_sha256`). A store-writer + who recomputes `action_sha256` could rewrite action payload content and + still pass `validate_chain`; the manifest's own digests are independently + verified against the blob store. Not in the reviewer's explicit A3 + "at minimum" list; recommended for the next invariant slice. +2. **Manifest created_at/updated_at ↔ row binding**: `_bind_manifest_fields` + binds package_id/revision/state and rev>0 lineage, but not the manifest's + own `created_at`/`updated_at` to row timestamps (correct binding for + `created_at` requires threading the revision-0 row timestamp through the + kernel walk). Residual; recommended with the above. +3. **`validate_chain` double canonicalization (audit path)**: with + `check_schema=True`, each event's manifest is canonicalized once by the A2 + decode and again inside the schema validator. Bounded to the explicit audit + path (not load/apply); the perf-1 single-pass fix covered + `check_current_row_consistency` only. +4. **Test tamper-pattern duplication (nit)**: `test_transactional_store.py` + and `test_chain_validator.py` each carry an inline drop-triggers/UPDATE + tamper pattern rather than a shared helper; cosmetic. diff --git a/methodfactory/storage/chain.py b/methodfactory/storage/chain.py index c63611c..4af3dad 100644 --- a/methodfactory/storage/chain.py +++ b/methodfactory/storage/chain.py @@ -29,7 +29,11 @@ - event_id / action_id obey the identifier grammar (uniqueness is schema- enforced by the UNIQUE constraints on event_id and (package_id, action_id)); - action is the frozen vocabulary (or `create_package` at revision 0); -- stored JSON BLOBs are valid UTF-8 JSON objects; +- stored JSON BLOBs are valid UTF-8 JSON objects AND are stored in the + canonical serialized form (canonical_bytes(decoded) == stored bytes), with + digests bound to those canonical bytes; +- the decoded semantic action object is bound back to the indexed event + (protocol_version, package_id, action_id, action; frozen field set); - when artifact verification is requested: every referenced input content blob, the summary body blob, and every artifact blob exists and verifies against its recorded digest. @@ -43,9 +47,10 @@ from ..domain.transitions import ACTION_VOCABULARY from ..engine.apply import CREATE_PACKAGE_ACTION +from ..protocol.envelope import PROTOCOL_VERSION from .errors import ChainViolationError, StorageError from .paths import validate_identifier -from .serialization import sha256_hex +from .serialization import canonical_bytes, sha256_hex EVENTS_BY_REVISION_SQL = """ SELECT package_id, revision, event_id, action_id, action, action_sha256, @@ -56,23 +61,81 @@ ORDER BY revision ASC """ +# The frozen semantic-action field set (single canonical serializer output; +# closure review 4882624484-A3). Any deviation is a chain violation. +SEMANTIC_ACTION_FIELDS = frozenset( + {"protocol_version", "action", "package_id", "action_id", "basis", "payload"} +) -def _decode_json_object(data: bytes, what: str, violations: list[str]) -> dict | None: - """Decode a stored JSON BLOB; report a violation (not raise) on failure.""" + +def _check_canonical_form( + decoded: dict[str, Any], + stored_bytes: Any, + expected_hash: str | None, + what: str, + violations: list[str], + canonical: bytes | None = None, +) -> bytes | None: + """Canonical-form + digest binding (senior review 4882624484, A2). + + The SINGLE implementation of the canonical-JSON invariant used by both the + validator decode path and the hot-path current-row check: + 1. canonicalize the decoded object with the single canonical serializer; + 2. require canonical_bytes(decoded) == stored bytes (a non-canonical + representation fails even when its raw-byte hash is recomputed); + 3. when expected_hash is given, require sha256(canonical) == expected_hash. + + Callers that already canonicalized (e.g. the schema validator's single + pass) pass `canonical` to avoid a second serialization. + """ + if canonical is None: + try: + canonical = canonical_bytes(decoded) + except (TypeError, RecursionError, UnicodeEncodeError, ValueError) as exc: + violations.append(f"{what} cannot be canonicalized: {exc}") + return None try: - text = data.decode("utf-8") - except (UnicodeDecodeError, AttributeError) as exc: - violations.append(f"{what} is not valid UTF-8: {exc}") + stored = bytes(stored_bytes) + except (TypeError, ValueError): + violations.append(f"{what} is not bytes") return None + if canonical != stored: + violations.append(f"{what} is not stored in canonical JSON form") + if expected_hash is not None and sha256_hex(canonical) != expected_hash: + violations.append(f"{what} does not hash to its stored digest") + return canonical + + +def _decode_canonical_json_object( + data: Any, what: str, violations: list[str] +) -> tuple[dict | None, bytes | None]: + """Decode a stored JSON BLOB and canonicalize it (senior review + 4882624484, A2). + + Returns ``(decoded_object, canonical_bytes)`` or ``(None, None)`` after + reporting a violation for: invalid UTF-8, invalid JSON, a non-object + value, or a NON-CANONICAL representation. Uses the single canonical + serialization primitive — no second serializer is created. + """ + if not isinstance(data, (bytes, bytearray)): + violations.append(f"{what} is not bytes") + return None, None + raw = bytes(data) + try: + text = raw.decode("utf-8") + except UnicodeDecodeError as exc: + violations.append(f"{what} is not valid UTF-8: {exc}") + return None, None try: value = json.loads(text) except (json.JSONDecodeError, RecursionError) as exc: violations.append(f"{what} is not valid JSON: {exc}") - return None + return None, None if not isinstance(value, dict): violations.append(f"{what} is not a JSON object") - return None - return value + return None, None + canonical = _check_canonical_form(value, raw, None, what, violations) + return value, canonical def _bind_manifest_fields( @@ -133,33 +196,107 @@ def check_event_invariants( event_id = event.get("event_id") action_id = event.get("action_id") - # ── Stored BLOB validity + digest binding ────────────────────────── + # ── Stored BLOB validity + canonical form + digest binding ───────── + # (closure review 4882624484-A2): a syntactically valid but non-canonical JSON + # representation with a recomputed raw-byte hash must FAIL — the digest is + # bound to the CANONICAL bytes of the decoded object. manifest_from_bytes = None + manifest_canonical = None + action_obj = None + action_canonical = None if decode_blobs: - manifest_from_bytes = _decode_json_object( - event.get("manifest_json"), f"manifest_json({package_id}, rev {revision})", violations + manifest_from_bytes, manifest_canonical = _decode_canonical_json_object( + event.get("manifest_json"), + f"manifest_json({package_id}, rev {revision})", violations, ) - _decode_json_object( - event.get("action_json"), f"action_json({package_id}, rev {revision})", violations + action_obj, action_canonical = _decode_canonical_json_object( + event.get("action_json"), + f"action_json({package_id}, rev {revision})", violations, ) if manifest is None and manifest_from_bytes is not None: # Caller (validator) did not pre-decode; use the kernel's decode so # field checks still run. manifest = manifest_from_bytes - try: - if sha256_hex(bytes(event.get("manifest_json"))) != resulting_hash: + + manifest_raw = None + if isinstance(event.get("manifest_json"), (bytes, bytearray)): + manifest_raw = bytes(event["manifest_json"]) + elif decode_blobs: + # Non-bytes stored types are already reported by the decoder above. + pass + else: + try: + manifest_raw = bytes(event.get("manifest_json")) + except (TypeError, ValueError): + violations.append(f"manifest_json({package_id}, rev {revision}) is not bytes") + if manifest_raw is not None: + digest_input = manifest_canonical if manifest_canonical is not None else manifest_raw + if sha256_hex(digest_input) != resulting_hash: violations.append( f"manifest_json({package_id}, rev {revision}) does not hash to resulting_manifest_sha256" ) - except (TypeError, ValueError): - violations.append(f"manifest_json({package_id}, rev {revision}) is not bytes") - try: - if sha256_hex(bytes(event.get("action_json"))) != action_hash: + action_raw = None + if isinstance(event.get("action_json"), (bytes, bytearray)): + action_raw = bytes(event["action_json"]) + elif decode_blobs: + pass + else: + try: + action_raw = bytes(event.get("action_json")) + except (TypeError, ValueError): + violations.append(f"action_json({package_id}, rev {revision}) is not bytes") + if action_raw is not None: + digest_input = action_canonical if action_canonical is not None else action_raw + if sha256_hex(digest_input) != action_hash: violations.append( f"action_json({package_id}, rev {revision}) does not hash to action_sha256" ) - except (TypeError, ValueError): - violations.append(f"action_json({package_id}, rev {revision}) is not bytes") + + # ── Semantic-action binding (senior review 4882624484, A3) ───────── + # Bind the decoded action object back to the indexed event: exact frozen + # field set, protocol_version, package_id, action_id, action; revision 0 + # requires the canonical create_package structure (including the semantic + # creation timestamp bound to the row). + # NOTE (upgrade policy): protocol_version is bound by EXACT equality with + # the current constant. When PROTOCOL_VERSION increments, historical rows + # written under the old version must be migrated (recompute canonical + # action bytes + hashes) before validate_chain — see ADR-0012 §G / the + # migration slice. Fail-closed is deliberate. + if action_obj is not None: + awhat = f"action_json({package_id}, rev {revision})" + if set(action_obj.keys()) != SEMANTIC_ACTION_FIELDS: + violations.append( + f"{awhat} is not a canonical semantic action (field set mismatch)" + ) + else: + if action_obj.get("protocol_version") != PROTOCOL_VERSION: + violations.append(f"{awhat} protocol_version != {PROTOCOL_VERSION!r}") + if action_obj.get("package_id") != package_id: + violations.append(f"{awhat} package_id != indexed {package_id!r}") + if action_obj.get("action_id") != action_id: + violations.append(f"{awhat} action_id != indexed {action_id!r}") + if action_obj.get("action") != action_name: + violations.append(f"{awhat} action != indexed action {action_name!r}") + if not isinstance(action_obj.get("basis"), dict) or not isinstance( + action_obj.get("payload"), dict + ): + violations.append(f"{awhat} basis/payload must be objects") + if revision == 0: + if action_obj.get("action") != CREATE_PACKAGE_ACTION: + violations.append(f"{awhat} action must be create_package") + if action_obj.get("basis") != {}: + violations.append(f"{awhat} basis must be empty") + payload = action_obj.get("payload") + if not isinstance(payload, dict) or not isinstance( + payload.get("intent"), str + ): + violations.append( + f"{awhat} payload must contain intent string" + ) + if not isinstance(payload, dict) or payload.get("created_at") != event.get("created_at"): + violations.append( + f"{awhat} payload.created_at != indexed created_at" + ) # ── Revision-zero contract ───────────────────────────────────────── if revision == 0: @@ -283,18 +420,23 @@ def check_current_row_consistency( manifest, package_id=package_id, revision=event.get("revision"), state_after=event.get("state_after"), violations=violations, ) - try: - if sha256_hex(bytes(manifest_json_bytes)) != event.get("resulting_manifest_sha256"): - violations.append( - "manifest_json bytes do not hash to resulting_manifest_sha256" - ) - except (TypeError, ValueError): - violations.append("manifest_json is not bytes") - if check_schema: - from ..manifest.schema import validate_manifest as _vm + # Canonical-form + digest binding (senior review 4882624484, A2) with a + # SINGLE canonicalization: the schema validator's canonical pass feeds the + # A2 compare/hash, so the hot path does not serialize the manifest twice. + from ..manifest.schema import validate_manifest_canonical as _vmc - for schema_error in _vm(manifest): + schema_errors, manifest_canonical = _vmc(manifest) + if check_schema: + for schema_error in schema_errors: violations.append(f"manifest schema violation: {schema_error}") + _check_canonical_form( + manifest, + manifest_json_bytes, + event.get("resulting_manifest_sha256"), + f"manifest_json({package_id})", + violations, + canonical=manifest_canonical, + ) return violations diff --git a/methodfactory/storage/store.py b/methodfactory/storage/store.py index 1a404ee..0242243 100644 --- a/methodfactory/storage/store.py +++ b/methodfactory/storage/store.py @@ -100,7 +100,7 @@ ACTION_LOOKUP_SQL = """ SELECT package_id, revision, action_sha256, state_after, - resulting_manifest_sha256, manifest_json + resulting_manifest_sha256, created_at, manifest_json FROM events WHERE package_id = ? AND action_id = ? """ @@ -145,6 +145,36 @@ def _utcnow() -> str: return datetime.now(timezone.utc).isoformat() +def _normalize_created_at(value: str | None) -> str: + """Normalize a caller-supplied creation timestamp to canonical UTC ISO-8601. + + Semantic identity (senior review 4882624484, A1): ``fromisoformat`` then + ``astimezone(UTC)`` collapses every spelling of the SAME INSTANT to one + canonical text form (``Z``, ``+00:00``, any offset), so retries from any + timezone replay identically. Naive/date-only timestamps are rejected: the + manifest contract requires an explicit UTC offset (no ambiguous local + wall-clock time in the identity). + """ + if value is None: + return _utcnow() + if not isinstance(value, str): + raise InvalidPayloadError( + f"created_at must be an ISO-8601 string, got {type(value).__name__}" + ) + try: + dt = datetime.fromisoformat(value) + except ValueError as exc: + raise InvalidPayloadError( + f"created_at is not valid ISO-8601: {value!r}" + ) from exc + if dt.tzinfo is None: + raise InvalidPayloadError( + "created_at must include a UTC offset (naive timestamps are " + "ambiguous in the creation identity)" + ) + return dt.astimezone(timezone.utc).isoformat() + + def _is_locked(exc: sqlite3.Error) -> bool: """Classify retryable SQLite lock contention by extended error code, falling back to the message heuristic only for interpreters without @@ -325,9 +355,15 @@ def create( ) -> dict[str, Any]: """Create a package at revision 0 (exactly one event). + `created_at` is SEMANTIC (closure review 4882624484-A1): the normalized creation + timestamp is part of the canonical create_package semantic action and + therefore of the resulting action_sha256. A repeat with the same + package/intent but a DIFFERENT timestamp is NOT an exact replay — it + raises DuplicatePackageError. Omitted -> internal UTC now. + Duplicate create raises DuplicatePackageError unless it is an exact idempotent replay of the original creation action (same deterministic - action_id + same semantic action hash). + action_id + same semantic action hash including the timestamp). """ validate_package_id(package_id) if not isinstance(intent_raw, str): @@ -340,7 +376,8 @@ def create( raise InvalidPayloadError( "intent_raw must not contain control characters" ) - created_at = created_at or _utcnow() + created_at_provided = created_at is not None + created_at = _normalize_created_at(created_at) action_id = f"act_create_{package_id}" event_id = f"evt_{package_id}_0" @@ -350,7 +387,7 @@ def create( package_id=package_id, action_id=action_id, basis={}, - payload={"intent": intent_raw}, + payload={"intent": intent_raw, "created_at": created_at}, ) action_hash = sha256_hex(action_bytes) @@ -359,13 +396,29 @@ def create( with _Transaction(conn): existing = _find_action(conn, package_id, action_id) if existing is not None: + if not created_at_provided: + # Omitted timestamp on a retry: idempotent replay uses + # the STORED creation time (A1) so a caller that did + # not pin a timestamp can replay the original create. + action_bytes = canonical_action_bytes( + protocol_version=PROTOCOL_VERSION, + action=CREATE_PACKAGE_ACTION, + package_id=package_id, + action_id=action_id, + basis={}, + payload={ + "intent": intent_raw, + "created_at": existing["created_at"], + }, + ) + action_hash = sha256_hex(action_bytes) if existing["action_sha256"] == action_hash: return _decode_and_check_row(existing, package_id=package_id) - # The package exists (its create was committed with a - # different intent). Per the create contract, attempting - # to create an existing package returns the stable - # PACKAGE_EXISTS error — exact idempotent replay is the - # only accepted repeat. + # The package exists and the requested creation differs + # (different intent or an EXPLICIT different timestamp). + # Per the create contract, attempting to create an existing + # package returns the stable PACKAGE_EXISTS error — exact + # idempotent replay is the only accepted repeat. raise DuplicatePackageError( f"package {package_id} already exists", package_id=package_id ) diff --git a/methodfactory/tests/test_chain_validator.py b/methodfactory/tests/test_chain_validator.py index e9d754a..47f379e 100644 --- a/methodfactory/tests/test_chain_validator.py +++ b/methodfactory/tests/test_chain_validator.py @@ -137,6 +137,30 @@ def _tamper_blob(root: Path, package_id: str, revision: int, column: str, data: c.close() +def _tamper_action_raw(root: Path, package_id: str, revision: int, raw: bytes) -> None: + """Write raw action_json bytes and recompute action_sha256 over THOSE raw + bytes, so only canonical-form (A2) or action-binding (A3) invariants can + fail.""" + c = _raw(root) + c.execute( + "UPDATE events SET action_json=?, action_sha256=? " + "WHERE package_id=? AND revision=?", + (raw, sha256_hex(raw), package_id, revision), + ) + c.commit() + c.close() + + +def _tamper_action_canonical( + root: Path, package_id: str, revision: int, semantic: dict +) -> None: + """Write a CANONICAL semantic action (possibly for a different package / + action_id / action) and recompute action_sha256 over those canonical + bytes, so only the action-binding (A3) invariants can fail.""" + data = canonical_bytes(semantic) + _tamper_action_raw(root, package_id, revision, data) + + class ChainValidatorTests(unittest.TestCase): def test_valid_chain_passes(self): with tempfile.TemporaryDirectory() as td: @@ -159,6 +183,19 @@ def test_missing_package_raises(self): finally: store.close() + def test_semantic_action_fields_pinned_to_serializer(self): + """The kernel's frozen semantic-action field set must never drift from + the canonical serializer's output (single source of truth).""" + from methodfactory.storage.chain import SEMANTIC_ACTION_FIELDS + from methodfactory.storage.serialization import semantic_action + + produced = set(semantic_action( + protocol_version="0.1", action="record_input", + package_id="pkg_demo_001", action_id="act_1", + basis={}, payload={}, + ).keys()) + self.assertEqual(produced, SEMANTIC_ACTION_FIELDS) + class RevisionZeroInvariantTests(unittest.TestCase): def test_action_must_be_create_package(self): @@ -345,6 +382,155 @@ def test_malformed_action_bytes(self): store.close() +class CanonicalFormInvariantTests(unittest.TestCase): + """Closure review A2: stored action/manifest JSON must themselves be + canonical. A syntactically valid but NON-canonical representation with a + recomputed raw-byte digest must fail canonical-form verification.""" + + def _non_canonical_bytes(self, obj: dict) -> bytes: + # Semantically equal but NOT the canonical serialization: unsorted + # keys + different separators. canonical_bytes(obj) != this output. + text = json.dumps(obj, sort_keys=False, separators=(", ", ": "), + ensure_ascii=False) + self.assertNotEqual( + text.encode("utf-8"), + canonical_bytes(obj), + "fixture must actually be non-canonical", + ) + return text.encode("utf-8") + + def test_non_canonical_action_json_rejected(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + # read a valid stored action, re-serialize non-canonically, + # recompute the hash over the raw tampered bytes + c = _raw(Path(td)) + row = c.execute( + "SELECT action_json FROM events " + "WHERE package_id='pkg_demo_001' AND revision=1" + ).fetchone() + action = json.loads(row[0]) + c.close() + raw = self._non_canonical_bytes(action) + _tamper_action_raw(Path(td), "pkg_demo_001", 1, raw) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("canonical", str(ctx.exception)) + finally: + store.close() + + def test_non_canonical_manifest_json_rejected(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + c = _raw(Path(td)) + row = c.execute( + "SELECT manifest_json FROM events " + "WHERE package_id='pkg_demo_001' AND revision=1" + ).fetchone() + manifest = json.loads(row[0]) + c.close() + raw = self._non_canonical_bytes(manifest) + _tamper_blob(Path(td), "pkg_demo_001", 1, "manifest_json", raw) + c = _raw(Path(td)) + c.execute( + "UPDATE events SET resulting_manifest_sha256=? " + "WHERE package_id='pkg_demo_001' AND revision=1", + (sha256_hex(raw),), + ) + c.commit() + c.close() + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("canonical", str(ctx.exception)) + finally: + store.close() + + +class ActionBindingInvariantTests(unittest.TestCase): + """Closure review A3: the decoded semantic action object is bound back to + the indexed event. Canonical actions for a DIFFERENT package / action_id / + action (with recomputed hashes) must all be rejected.""" + + def test_action_for_different_package_rejected(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + c = _raw(Path(td)) + row = c.execute( + "SELECT action_json FROM events " + "WHERE package_id='pkg_demo_001' AND revision=1" + ).fetchone() + action = json.loads(row[0]) + c.close() + action["package_id"] = "pkg_evil_001" + _tamper_action_canonical(Path(td), "pkg_demo_001", 1, action) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("package_id != indexed", str(ctx.exception)) + finally: + store.close() + + def test_action_id_inside_action_json_rejected(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + c = _raw(Path(td)) + row = c.execute( + "SELECT action_json FROM events " + "WHERE package_id='pkg_demo_001' AND revision=1" + ).fetchone() + action = json.loads(row[0]) + c.close() + action["action_id"] = "act_evil_99" + _tamper_action_canonical(Path(td), "pkg_demo_001", 1, action) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("action_id != indexed", str(ctx.exception)) + finally: + store.close() + + def test_action_name_inside_action_json_rejected(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + c = _raw(Path(td)) + row = c.execute( + "SELECT action_json FROM events " + "WHERE package_id='pkg_demo_001' AND revision=1" + ).fetchone() + action = json.loads(row[0]) + c.close() + action["action"] = "cancel" + _tamper_action_canonical(Path(td), "pkg_demo_001", 1, action) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("action != indexed", str(ctx.exception)) + finally: + store.close() + + def test_non_semantic_field_set_rejected(self): + """A canonical JSON object that is NOT a semantic action (extra or + missing fields) must fail the frozen field-set check.""" + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + not_semantic = {"action": "record_input", "extra": 1} + _tamper_action_canonical(Path(td), "pkg_demo_001", 1, not_semantic) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("field set", str(ctx.exception)) + finally: + store.close() + + class GrammarInvariantTests(unittest.TestCase): def test_bad_event_id_rejected(self): with tempfile.TemporaryDirectory() as td: diff --git a/methodfactory/tests/test_public_error_boundary.py b/methodfactory/tests/test_public_error_boundary.py index e4f89d1..0328a2f 100644 --- a/methodfactory/tests/test_public_error_boundary.py +++ b/methodfactory/tests/test_public_error_boundary.py @@ -30,10 +30,15 @@ class LatestEventBoundaryTests(unittest.TestCase): def _db_with_manifest(self, raw: bytes): """Create a valid store, then write a specific manifest_json blob into - the events table (bypassing the store) to test the read boundary.""" + the events table (bypassing the store) to test the read boundary. + + The temp store is removed when the test completes (closure review 4882624484-A4: + no leaked host temp stores).""" + import shutil import sqlite3 root = Path(tempfile.mkdtemp()) + self.addCleanup(shutil.rmtree, root, ignore_errors=True) conn = open_database(root, read_only=False) close_database(conn) c = sqlite3.connect(str(root / DB_FILENAME)) @@ -94,30 +99,31 @@ def test_non_bytes_stored_type_translated(self): AttributeError (Finding 3).""" import sqlite3 - root = Path(tempfile.mkdtemp()) - conn = open_database(root, read_only=False) - close_database(conn) - c = sqlite3.connect(str(root / DB_FILENAME)) - c.execute( - "INSERT INTO events (package_id, revision, event_id, action_id, action, " - "action_sha256, state_before, state_after, previous_manifest_sha256, " - "resulting_manifest_sha256, created_at, action_json, manifest_json) " - "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", - ( - "pkg_int_001", 0, "evt_int", "act_int", "create_package", "0" * 64, - None, "INTAKE", None, "0" * 64, "2026-08-07T00:00:00+00:00", - b'{"action":"create_package"}', 12345, - ), - ) - c.commit() - c.close() - conn = open_database(root, read_only=True) - try: - with self.assertRaises(MethodFactoryError) as ctx: - latest_event(conn, "pkg_int_001") - self.assertEqual(ctx.exception.code, "MANIFEST_INVALID") - finally: + with tempfile.TemporaryDirectory() as td: + root = Path(td) + conn = open_database(root, read_only=False) close_database(conn) + c = sqlite3.connect(str(root / DB_FILENAME)) + c.execute( + "INSERT INTO events (package_id, revision, event_id, action_id, action, " + "action_sha256, state_before, state_after, previous_manifest_sha256, " + "resulting_manifest_sha256, created_at, action_json, manifest_json) " + "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", + ( + "pkg_int_001", 0, "evt_int", "act_int", "create_package", "0" * 64, + None, "INTAKE", None, "0" * 64, "2026-08-07T00:00:00+00:00", + b'{"action":"create_package"}', 12345, + ), + ) + c.commit() + c.close() + conn = open_database(root, read_only=True) + try: + with self.assertRaises(MethodFactoryError) as ctx: + latest_event(conn, "pkg_int_001") + self.assertEqual(ctx.exception.code, "MANIFEST_INVALID") + finally: + close_database(conn) def test_non_object_json_translated(self): """A manifest_json BLOB containing a JSON ARRAY (not an object) is @@ -135,28 +141,29 @@ def test_str_manifest_json_accepted(self): """A TEXT-typed manifest_json is accepted and decoded (Finding 3).""" import sqlite3 - root = Path(tempfile.mkdtemp()) - conn = open_database(root, read_only=False) - close_database(conn) - c = sqlite3.connect(str(root / DB_FILENAME)) - c.execute( - "INSERT INTO events (package_id, revision, event_id, action_id, action, " - "action_sha256, state_before, state_after, previous_manifest_sha256, " - "resulting_manifest_sha256, created_at, action_json, manifest_json) " - "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", - ( - "pkg_str_001", 0, "evt_str", "act_str", "create_package", "0" * 64, - None, "INTAKE", None, "0" * 64, "2026-08-07T00:00:00+00:00", - b'{"action":"create_package"}', '{"schema_version":"0.1"}', - ), - ) - c.commit() - c.close() - conn = open_database(root, read_only=True) - try: - self.assertEqual(latest_event(conn, "pkg_str_001"), {"schema_version": "0.1"}) - finally: + with tempfile.TemporaryDirectory() as td: + root = Path(td) + conn = open_database(root, read_only=False) close_database(conn) + c = sqlite3.connect(str(root / DB_FILENAME)) + c.execute( + "INSERT INTO events (package_id, revision, event_id, action_id, action, " + "action_sha256, state_before, state_after, previous_manifest_sha256, " + "resulting_manifest_sha256, created_at, action_json, manifest_json) " + "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", + ( + "pkg_str_001", 0, "evt_str", "act_str", "create_package", "0" * 64, + None, "INTAKE", None, "0" * 64, "2026-08-07T00:00:00+00:00", + b'{"action":"create_package"}', '{"schema_version":"0.1"}', + ), + ) + c.commit() + c.close() + conn = open_database(root, read_only=True) + try: + self.assertEqual(latest_event(conn, "pkg_str_001"), {"schema_version": "0.1"}) + finally: + close_database(conn) class ArtifactBoundaryTests(unittest.TestCase): diff --git a/methodfactory/tests/test_sqlite_open.py b/methodfactory/tests/test_sqlite_open.py index 42cf64d..9f39844 100644 --- a/methodfactory/tests/test_sqlite_open.py +++ b/methodfactory/tests/test_sqlite_open.py @@ -219,27 +219,28 @@ def test_metadata_drift_rejected(self): class UriPathTests(unittest.TestCase): def test_readonly_uri_with_significant_paths(self): """Finding 1 item 5: paths with spaces, Unicode, ?, #, % must open - correctly and create no sibling/alternate file.""" - base = tempfile.mkdtemp() - for name in ( - "store with space", - "สโตร์", - "store?with#special%chars", - ): - root = Path(base) / name - root.mkdir(parents=True) - conn0 = open_database(root, read_only=False) - close_database(conn0) - before = sorted(p.name for p in root.iterdir()) - conn = open_database(root, read_only=True) - close_database(conn) - after = sorted(p.name for p in root.iterdir()) - self.assertEqual(before, after, f"ro open created files in {name}") - # No sibling file created anywhere under the base dir beyond the 3 - # intended store roots. - expected_roots = {"store with space", "สโตร์", "store?with#special%chars"} - actual_roots = {p.name for p in Path(base).iterdir()} - self.assertEqual(actual_roots, expected_roots) + correctly and create no sibling/alternate file. The base temp dir is + removed on completion (closure review 4882624484-A4: no leaked temp stores).""" + with tempfile.TemporaryDirectory() as base: + for name in ( + "store with space", + "สโตร์", + "store?with#special%chars", + ): + root = Path(base) / name + root.mkdir(parents=True) + conn0 = open_database(root, read_only=False) + close_database(conn0) + before = sorted(p.name for p in root.iterdir()) + conn = open_database(root, read_only=True) + close_database(conn) + after = sorted(p.name for p in root.iterdir()) + self.assertEqual(before, after, f"ro open created files in {name}") + # No sibling file created anywhere under the base dir beyond the 3 + # intended store roots. + expected_roots = {"store with space", "สโตร์", "store?with#special%chars"} + actual_roots = {p.name for p in Path(base).iterdir()} + self.assertEqual(actual_roots, expected_roots) def test_ro_missing_no_create(self): with tempfile.TemporaryDirectory() as td: diff --git a/methodfactory/tests/test_temp_hygiene.py b/methodfactory/tests/test_temp_hygiene.py new file mode 100644 index 0000000..7bd281d --- /dev/null +++ b/methodfactory/tests/test_temp_hygiene.py @@ -0,0 +1,66 @@ +"""Temp-leak hygiene proof (senior review 4882624484, A4). + +The affected test suites (test_sqlite_open, test_public_error_boundary) +previously created unmanaged temporary stores via ``tempfile.mkdtemp()`` that +leaked as ``/tmp/tmp*/methodfactory.sqlite3`` on the host. All unmanaged temp +stores are now deterministic-cleanup (``TemporaryDirectory`` / +``addCleanup``). + +This proof is BEHAVIORAL and ISOLATED: it runs the corrected affected suites +in a SUBPROCESS whose TMPDIR points at a dedicated empty directory, then +asserts that NO Method Factory SQLite store survives anywhere under that +isolated root. A dedicated temp root avoids host-wide snapshot races (parallel +CI shards) and process-global state coupling; scanning the isolated root with +``rglob`` also covers nested store directories. +""" + +from __future__ import annotations + +import os +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +from methodfactory.storage.paths import DB_FILENAME + +# The suites that previously used unmanaged tempfile.mkdtemp() stores. +AFFECTED_SUITES = [ + "methodfactory.tests.test_sqlite_open", + "methodfactory.tests.test_public_error_boundary", +] + + +class TempLeakHygieneProof(unittest.TestCase): + def test_affected_suites_leave_no_mf_stores(self): + with tempfile.TemporaryDirectory() as isolated: + # The nested suites inherit a dedicated, empty temp root. + env = dict(os.environ) + env["TMPDIR"] = isolated + env["TEMP"] = isolated + env["TMP"] = isolated + + result = subprocess.run( + [sys.executable, "-m", "unittest", *AFFECTED_SUITES], + env=env, + capture_output=True, + text=True, + timeout=300, + ) + self.assertEqual( + result.returncode, + 0, + f"affected suites failed:\n{result.stdout}\n{result.stderr}", + ) + + leftovers = list(Path(isolated).rglob(DB_FILENAME)) + self.assertEqual( + leftovers, + [], + f"corrected suites leaked MF temp stores: {[str(p) for p in leftovers]}", + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/methodfactory/tests/test_transactional_store.py b/methodfactory/tests/test_transactional_store.py index fa728aa..715d5f1 100644 --- a/methodfactory/tests/test_transactional_store.py +++ b/methodfactory/tests/test_transactional_store.py @@ -177,6 +177,129 @@ def test_exact_create_replay_returns_original(self): finally: store.close() + def test_create_normalized_timestamp_replays(self): + """Closure review A1: semantically equal timestamps normalize to the + same canonical text, so equivalent spellings replay exactly.""" + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + m1 = store.create("pkg_demo_001", "Build a skill", + created_at="2026-08-07T00:00:00Z") + m2 = store.create("pkg_demo_001", "Build a skill", + created_at="2026-08-07T00:00:00+00:00") + self.assertEqual(m1, m2) + self.assertEqual(_event_count(Path(td)), 1) + finally: + store.close() + + def test_create_invalid_created_at_rejected(self): + """Closure review A1: an unparseable timestamp is rejected typed at + the public boundary (it can never become part of the identity).""" + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + for bad in ("not-a-date", "2026-13-40T99:99:99", 12345): + with self.subTest(created_at=bad): + with self.assertRaises(InvalidPayloadError): + store.create("pkg_demo_001", "Build a skill", + created_at=bad) # type: ignore[arg-type] + self.assertEqual(_event_count(Path(td)), 0) + finally: + store.close() + + def test_create_naive_timestamp_rejected(self): + """A naive (offset-less) timestamp is ambiguous in the creation + identity and must be rejected (A1 normalization requires UTC).""" + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + with self.assertRaises(InvalidPayloadError): + store.create("pkg_demo_001", "Build a skill", + created_at="2026-08-07T00:00:00") + self.assertEqual(_event_count(Path(td)), 0) + finally: + store.close() + + def test_create_cross_timezone_same_instant_replays(self): + """A1 normalization collapses every spelling of the SAME INSTANT to + UTC, so retries from different timezones replay identically.""" + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + m1 = store.create("pkg_demo_001", "Build a skill", + created_at="2026-08-07T07:00:00+07:00") + m2 = store.create("pkg_demo_001", "Build a skill", + created_at="2026-08-07T00:00:00+00:00") + self.assertEqual(m1, m2) + self.assertEqual(m1["created_at"], "2026-08-07T00:00:00+00:00") + self.assertEqual(_event_count(Path(td)), 1) + finally: + store.close() + + def test_create_omitted_timestamp_replays(self): + """A retry that omits created_at must replay the original create + (using the stored creation time), not fail stale on a fresh now().""" + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + m1 = store.create("pkg_demo_001", "Build a skill") + m2 = store.create("pkg_demo_001", "Build a skill") + self.assertEqual(m1, m2) + self.assertEqual(_event_count(Path(td)), 1) + finally: + store.close() + + def test_create_explicit_different_timestamp_still_conflicts(self): + """An EXPLICIT different timestamp (a different instant) is NOT a + replay even when intent matches (A1 contract).""" + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + store.create("pkg_demo_001", "Build a skill", + created_at="2026-08-07T00:00:00+00:00") + with self.assertRaises(DuplicatePackageError): + store.create("pkg_demo_001", "Build a skill", + created_at="2026-08-07T01:00:00+00:00") + self.assertEqual(_event_count(Path(td)), 1) + finally: + store.close() + + def test_create_semantic_identity_chain_binding(self): + """Closure review A1/A3: the stored revision-0 action_json carries + the semantic created_at and the chain validator binds it to the row — + tampering payload.created_at with a recomputed hash is rejected.""" + import json as _json + + from methodfactory.storage.serialization import canonical_bytes, sha256_hex + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + store.create("pkg_demo_001", "Build a skill", + created_at="2026-08-07T00:00:00+00:00") + c = sqlite3.connect(str(Path(td) / DB_FILENAME)) + c.execute("DROP TRIGGER IF EXISTS events_no_update") + c.execute("DROP TRIGGER IF EXISTS events_no_delete") + row = c.execute( + "SELECT action_json, action_sha256 FROM events " + "WHERE package_id='pkg_demo_001' AND revision=0" + ).fetchone() + action = _json.loads(row[0]) + action["payload"]["created_at"] = "2026-08-07T09:00:00+00:00" + data = canonical_bytes(action) + c.execute( + "UPDATE events SET action_json=?, action_sha256=? " + "WHERE package_id='pkg_demo_001' AND revision=0", + (data, sha256_hex(data)), + ) + c.commit() + c.close() + from methodfactory.storage.errors import ChainViolationError + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain("pkg_demo_001") + self.assertIn("created_at", str(ctx.exception)) + finally: + store.close() + def test_create_different_intent_is_duplicate(self): with tempfile.TemporaryDirectory() as td: store = SqliteManifestStore(td) From 7213e3d71d1c63666ee1890c338989789f61cabe Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Sat, 8 Aug 2026 02:17:46 +0700 Subject: [PATCH 23/41] feat(storage): invariant closure - deterministic replay + timestamp contracts (senior review 4885538290) Closes the two accepted residuals from closure A (public-surface.md #1, #2) without reopening the persistence architecture. - Deterministic action -> resulting-manifest validation: validate_chain reconstructs the normal internal ActionEnvelope from the stored canonical action_json (injecting expected_revision = revision - 1) and replays the SINGLE deterministic transition engine (engine.apply.next_manifest) over the predecessor manifest with the indexed event_id and row created_at. Canonical equality with the stored resulting manifest proves every consequence-bearing field (input digest/size/path, objective, summary digest/size/preview/presented_at/confirmation, artifact digest/byte_count/ path, state/revision/lineage, updated_at) with one rule - no per-action consequence validators. - Side-effect free: replay discards the engine's blobs_to_write; nothing is persisted during validation; committed blobs verified only by the optional artifact-verification mode. - Gate behavior: replay re-evaluates the same legality and gate rules from persisted evidence (all gates self-contained; no external runtime state). - Stored-action reconstruction fail-closed via existing envelope_from_dict: unsupported protocol version, unknown action, malformed basis/payload, invalid IDs, or invalid payload structure rejected even though persisted. - Timestamp contract: rev-0 created_at == updated_at == row created_at == payload.created_at (create path binds it); rev>0 updated_at == row created_at and created_at == revision-0 row created_at (threaded through the walk). presented_at/confirmed_at derived from the event timestamp and proven by replay - not bound indiscriminately. - Audit-path only: replay enabled in validate_chain; the apply hot path passes replay=False (unchanged, bounded); load() stays indexed latest-row. - No recursive validator dependencies: the validator calls the pure engine with decoded, validated persisted evidence, never store.apply. - 18 new tests (positive, adversarial with recomputed hashes, reconstruction fail-closed, timestamp independent corruption, side-effect-free replay). Full suite: 356 tests green (was 338). --- docs/adr/ADR-0012-persistence-architecture.md | 68 +++ docs/public-surface.md | 31 +- methodfactory/storage/chain.py | 234 ++++++++- methodfactory/storage/store.py | 1 + methodfactory/tests/test_invariant_replay.py | 463 ++++++++++++++++++ 5 files changed, 788 insertions(+), 9 deletions(-) create mode 100644 methodfactory/tests/test_invariant_replay.py diff --git a/docs/adr/ADR-0012-persistence-architecture.md b/docs/adr/ADR-0012-persistence-architecture.md index f45ae11..f457bc1 100644 --- a/docs/adr/ADR-0012-persistence-architecture.md +++ b/docs/adr/ADR-0012-persistence-architecture.md @@ -475,3 +475,71 @@ Before any evidence capture, the local worktree must be clean: the `.gitignore` ### L. Architecture CI honesty (item 12) As of this amendment, architecture CI is **unproven**: run `31127787460` was cancelled without executing steps. It is neither failed nor passed. CI is considered evidence only after a run executes successfully on the exact branch SHA. The Phase 2 submission runs CI on the exact final head SHA and reports the run URL and conclusion. + +--- + +## Amendment - invariant closure (senior review 4885538290, 2026-08-08) + +Pre-migration invariant closure (Lane 1). Closes the two accepted residuals +from closure A (`public-surface.md` #1 and #2) WITHOUT reopening the +persistence architecture, changing the state representation, or adding a new +event format. + +### Deterministic action → resulting-manifest validation + +The authoritative full-chain validator (`storage.chain.validate_chain` - the +`mf validate --full` / migration / release-evidence path) now proves that the +recorded resulting manifest is exactly the deterministic result Method +Factory would have produced: + +```text +replay(predecessor_manifest, stored_action, event_id, event_created_at) + == stored resulting manifest (canonical bytes equal) +``` + +- Stored-action reconstruction reuses the normal envelope validator + (`envelope_from_dict`) on the canonical stored `action_json` with + `expected_revision = revision - 1` injected (the only field the + semantic-action form omits by contract). Fail-closed: unsupported protocol + version, unknown action, malformed basis/payload, invalid IDs, or invalid + payload structure are rejected even though the bytes were persisted. +- The transition is the SINGLE existing deterministic engine + (`engine.apply.next_manifest`). No per-action consequence validators are + created; there is still exactly one transition implementation. +- Replay is side-effect free: the engine's `blobs_to_write` are discarded and + nothing is persisted during validation. Blob metadata is proven by + canonical equality; committed-blob integrity is verified only by the + optional artifact-verification mode. +- Replay re-evaluates the same legality and gate rules from persisted + evidence (all gates are self-contained: predecessor manifest + stored + action; no external runtime state). A historical event that could not + legally have been produced fails full-chain validation. +- No recursive validator dependencies: the validator calls the pure engine + with decoded, validated persisted evidence; it never calls the + transactional mutation path (`store.apply`). +- The replay check lives ONLY on the explicit full-chain/audit path. + `load()` remains indexed latest-row; transaction apply remains bounded and + does not replay history. + +### Timestamp contract + +Derived from the deterministic transition implementation, not a blanket rule: + +| Revision | Contract | +|---|---| +| 0 | manifest `created_at == updated_at == row created_at == action payload.created_at` | +| > 0 | manifest `updated_at == row created_at`; manifest `created_at == revision-0 row created_at` (threaded through the walk) | + +`summary.presented_at` (`prepare_summary`) and +`summary.confirmation.confirmed_at` (`confirm_summary`) are derived from the +event timestamp by the transition and are therefore proven by deterministic +replay - they are not bound indiscriminately. Independent corruption tests +cover each binding. + +### Threat-model note (unchanged) + +These checks detect internally inconsistent history, writer defects, malformed +migration output, and partial/coherent-enough tampering that violates +deterministic semantics. They are internal-consistency evidence, not +cryptographic authenticity: an attacker capable of coherently rewriting and +rehashing the entire database remains out of scope. diff --git a/docs/public-surface.md b/docs/public-surface.md index 686f426..852cb1c 100644 --- a/docs/public-surface.md +++ b/docs/public-surface.md @@ -107,16 +107,43 @@ code-review verify lane on 2026-08-07; each is genuine in the code): still pass `validate_chain`; the manifest's own digests are independently verified against the blob store. Not in the reviewer's explicit A3 "at minimum" list; recommended for the next invariant slice. + + > **CLOSED by the invariant closure (senior review 4885538290, Lane 1).** + > `validate_chain` now reconstructs the normal internal ActionEnvelope + > from the stored canonical `action_json` and replays the SINGLE + > deterministic transition engine (`engine.apply.next_manifest`) over the + > predecessor manifest with the indexed `event_id` and row `created_at`; + > canonical equality with the stored resulting manifest proves every + > consequence-bearing field (input digest/size/path, objective, + > summary digest/size/preview/presented_at/confirmation incl. + > confirmed_at/operator_id/confirmed digest, artifact digest/byte_count/ + > path, state/revision/lineage, updated_at) with one rule — no per-action + > consequence validators. Adversarial tests recompute all immediate + > hashes so only the replay invariant can fail. + 2. **Manifest created_at/updated_at ↔ row binding**: `_bind_manifest_fields` binds package_id/revision/state and rev>0 lineage, but not the manifest's own `created_at`/`updated_at` to row timestamps (correct binding for `created_at` requires threading the revision-0 row timestamp through the kernel walk). Residual; recommended with the above. + + > **CLOSED by the invariant closure (senior review 4885538290, Lane 1).** + > Revision 0: `created_at == updated_at == row created_at == action + > payload.created_at`. Revision > 0: `updated_at == row created_at` and + > `created_at == revision-0 row created_at` (threaded through the walk). + > Action-specific timestamps (`presented_at`, `confirmed_at`) are derived + > from the event timestamp by the transition and proven by replay — not + > bound indiscriminately. + 3. **`validate_chain` double canonicalization (audit path)**: with `check_schema=True`, each event's manifest is canonicalized once by the A2 decode and again inside the schema validator. Bounded to the explicit audit path (not load/apply); the perf-1 single-pass fix covered - `check_current_row_consistency` only. + `check_current_row_consistency` only. **Retained as an accepted + performance residual** (senior review 4885538290 §8: the audit-path double + canonicalization may remain unless measurements show a meaningful problem). + 4. **Test tamper-pattern duplication (nit)**: `test_transactional_store.py` and `test_chain_validator.py` each carry an inline drop-triggers/UPDATE - tamper pattern rather than a shared helper; cosmetic. + tamper pattern rather than a shared helper; cosmetic. **Retained** (the + replay module keeps a self-contained local helper). diff --git a/methodfactory/storage/chain.py b/methodfactory/storage/chain.py index 4af3dad..8837dee 100644 --- a/methodfactory/storage/chain.py +++ b/methodfactory/storage/chain.py @@ -14,6 +14,8 @@ - resulting manifest package_id == indexed package_id; - resulting manifest revision == 0; - resulting manifest state == indexed `state_after`; +- resulting manifest `created_at` == `updated_at` == indexed `created_at` + (timestamp contract, invariant-closure review 4885538290); - stored action JSON hashes to `action_sha256`; - stored manifest JSON hashes to `resulting_manifest_sha256`. @@ -22,6 +24,19 @@ - `state_before(N) == state_after(N-1)`; - `previous_manifest_sha256(N) == resulting_manifest_sha256(N-1)`; - manifest package_id / revision / state match the indexed row; +- manifest `updated_at` == indexed `created_at`; manifest `created_at` == + the revision-0 row `created_at` (threaded through the walk); +- **deterministic replay (invariant closure, review 4885538290):** + reconstructing the normal ActionEnvelope from the stored canonical action + and applying the SINGLE deterministic transition engine + (`engine.apply.next_manifest`) to the predecessor manifest with the + indexed event_id and event created_at MUST produce exactly the stored + resulting manifest (canonical bytes equal). This proves the action's + consequence-bearing fields — input content digest/size/path, objective + statement/outcomes, summary digest/size/preview/presented_at/confirmation + (incl. confirmed_at/operator_id/confirmed digest), artifact + content digest/byte_count/path, state/revision/lineage — without building + per-action consequence validators; - stored action JSON hashes to `action_sha256`; - stored manifest JSON hashes to `resulting_manifest_sha256`. @@ -37,6 +52,13 @@ - when artifact verification is requested: every referenced input content blob, the summary body blob, and every artifact blob exists and verifies against its recorded digest. + +Replay is **audit-path only**: `validate_chain` enables it; the transactional +apply hot path does not (it just produced the manifest via the same engine, +so replay would be redundant work). Replay is side-effect free: the engine's +`(manifest, blobs_to_write)` return is used for the manifest only — blobs are +discarded, nothing is persisted, and committed blobs are verified only by the +optional artifact-verification mode. """ from __future__ import annotations @@ -45,9 +67,10 @@ import sqlite3 from typing import Any +from ..domain.errors import MethodFactoryError from ..domain.transitions import ACTION_VOCABULARY -from ..engine.apply import CREATE_PACKAGE_ACTION -from ..protocol.envelope import PROTOCOL_VERSION +from ..engine.apply import CREATE_PACKAGE_ACTION, next_manifest +from ..protocol.envelope import PROTOCOL_VERSION, envelope_from_dict from .errors import ChainViolationError, StorageError from .paths import validate_identifier from .serialization import canonical_bytes, sha256_hex @@ -161,6 +184,142 @@ def _bind_manifest_fields( ) +def _bind_timestamp_fields( + manifest: dict[str, Any], + event: dict[str, Any], + *, + creation_timestamp: str | None, + violations: list[str], +) -> None: + """Timestamp contract (invariant-closure review 4885538290). + + Derived from the deterministic transition implementation, not from a + blanket "every timestamp must match the row" rule: + + - Revision 0: the create manifest is born with + ``created_at == updated_at == event.created_at`` (``new_manifest``), + and the A1 semantic action binds ``payload.created_at == event.created_at``, + so all four agree. + - Revision > 0: ``next_manifest`` sets ``updated_at = created_at`` (the + event row timestamp threaded into the engine), and ``created_at`` is + carried unchanged from revision 0, so ``updated_at == event.created_at`` + and ``created_at == revision-0 created_at``. + + Action-specific timestamps (`summary.presented_at` on `prepare_summary`, + `summary.confirmation.confirmed_at` on `confirm_summary`) are derived from + the event timestamp by the engine and are therefore proven by the + deterministic replay check (canonical equality) — they are not bound + indiscriminately here. + """ + revision = event.get("revision") + row_created = event.get("created_at") + created = manifest.get("created_at") + updated = manifest.get("updated_at") + if revision == 0: + if created != row_created: + violations.append( + f"manifest created_at != indexed created_at at revision 0" + ) + if updated != row_created: + violations.append( + f"manifest updated_at != indexed created_at at revision 0" + ) + else: + if updated != row_created: + violations.append( + f"manifest updated_at != indexed created_at at revision {revision}" + ) + if creation_timestamp is not None and created != creation_timestamp: + violations.append( + f"manifest created_at != revision-0 created_at at revision {revision}" + ) + + +def _replay_violations( + *, + package_id: str, + revision: Any, + event: dict[str, Any], + action_obj: dict[str, Any] | None, + prev_manifest: dict[str, Any] | None, + manifest_canonical: bytes | None, + violations: list[str], +) -> None: + """Deterministic action -> resulting-manifest validation (invariant + closure, review 4885538290). + + Reconstructs the normal internal ActionEnvelope from the stored canonical + action (injecting `expected_revision = revision - 1`, the only field the + semantic-action form omits by contract), then applies the SINGLE + deterministic transition engine to the predecessor manifest with the + indexed event_id and created_at. The canonical bytes of the replayed + manifest must equal the stored resulting manifest — this proves every + consequence-bearing field (input digest/size/path, objective, + summary digest/size/preview/presented_at/confirmation, artifact + digest/byte_count/path, state/revision/lineage, updated_at) with one + rule instead of seven per-action validators. + + Side-effect free: the engine returns `(next_manifest, blobs_to_write)`; + blobs are discarded here and nothing is persisted during audit replay. + Blob METADATA is proven by canonical equality; committed-blob integrity + is verified by the optional artifact-verification mode. + + Fail-closed reconstruction: an unsupported protocol version, unknown + action, malformed basis/payload, invalid IDs, or invalid payload + structure all fail via the existing envelope validator — the stored bytes + are never trusted merely because they were persisted. + """ + if revision == 0: + return # create is validated by its own revision-0 contract, not replay + if action_obj is None or manifest_canonical is None: + return # decode failure already reported by the caller + if prev_manifest is None: + violations.append( + f"deterministic replay unavailable at revision {revision}: " + "missing predecessor manifest" + ) + return + + try: + reconstructed = dict(action_obj) + reconstructed["expected_revision"] = revision - 1 + envelope = envelope_from_dict(reconstructed) + except (MethodFactoryError, TypeError, ValueError, UnicodeError, RecursionError) as exc: + violations.append( + f"stored action cannot be reconstructed at revision {revision}: {exc}" + ) + return + + try: + replayed, blobs = next_manifest( + prev_manifest, + envelope, + event_id=event.get("event_id"), + created_at=event.get("created_at"), + ) + except MethodFactoryError as exc: + violations.append( + f"stored action is not a legal deterministic transition at revision {revision}: {exc}" + ) + return + del blobs # side-effect free: never persist blobs during audit replay + + try: + replayed_canonical = canonical_bytes(replayed) + except (TypeError, RecursionError, UnicodeEncodeError, ValueError) as exc: + violations.append( + f"replayed manifest cannot be canonicalized at revision {revision}: {exc}" + ) + return + if replayed_canonical != manifest_canonical: + violations.append( + f"stored resulting manifest is not the deterministic result of the " + f"stored action at revision {revision}: replayed digest " + f"{sha256_hex(replayed_canonical)} != stored " + f"{sha256_hex(manifest_canonical)}" + ) + + def check_event_invariants( *, package_id: str, @@ -171,6 +330,9 @@ def check_event_invariants( artifact_store: Any = None, check_schema: bool = False, decode_blobs: bool = True, + prev_manifest: dict[str, Any] | None = None, + replay: bool = False, + creation_timestamp: str | None = None, ) -> list[str]: """Return every invariant violation for one event (empty = valid). @@ -186,6 +348,16 @@ def check_event_invariants( - `decode_blobs`: decode the stored BLOBs to verify JSON validity. True for the on-disk validator; the transactional apply passes False because the bytes it stores are self-produced and digest-bound. + - `replay`: run the deterministic action -> resulting-manifest replay + check (audit/full-validation mode only; the apply hot path passes + False because it just produced the manifest via the same engine). + Requires `prev_manifest` (decoded predecessor) for revision > 0. + - `creation_timestamp`: the revision-0 row `created_at`, threaded through + the walk so every revision > 0 can bind `manifest.created_at` to it and + `manifest.updated_at` to its own row `created_at`. The create path + passes the creation timestamp; the apply path passes None (unchanged + hot path — full timestamp binding is enforced by the authoritative + validator). """ violations: list[str] = [] revision = event.get("revision") @@ -334,6 +506,12 @@ def check_event_invariants( manifest, package_id=package_id, revision=revision, state_after=state_after, violations=violations, ) + # Timestamp contract (invariant-closure review 4885538290): revision-0 + # created_at/updated_at and revision>0 updated_at/created_at row + # bindings derived from the deterministic transition implementation. + _bind_timestamp_fields( + manifest, event, creation_timestamp=creation_timestamp, violations=violations, + ) # Manifest-internal lineage claims must match the indexed row (the # engine writes these as chain facts; the validator cross-checks them). # Revision 0 is exempt: the create manifest's transition fields are @@ -360,6 +538,23 @@ def check_event_invariants( for schema_error in _vm(manifest): violations.append(f"manifest schema violation: {schema_error}") + # ── Deterministic replay (invariant-closure review 4885538290) ─────── + # Audit-path only: the stored action, replayed through the SINGLE + # deterministic transition engine over the predecessor manifest, must + # reproduce the stored resulting manifest exactly. Proves every + # consequence-bearing field with one rule. Side-effect free (blobs + # returned by the engine are discarded). + if replay: + _replay_violations( + package_id=package_id, + revision=revision, + event=event, + action_obj=action_obj, + prev_manifest=prev_manifest, + manifest_canonical=manifest_canonical, + violations=violations, + ) + # ── Grammar / vocabulary ─────────────────────────────────────────── for field, value in (("event_id", event_id), ("action_id", action_id)): if not isinstance(value, str) or not value: @@ -452,9 +647,14 @@ def validate_chain( Walks every event in revision order (lazily, one row at a time), applies the single invariant kernel per event with BLOB decoding + schema - validation enabled, and raises ChainViolationError (code CHAIN_VIOLATION) - with ALL violations of the failing event on the FIRST invalid event. - Returns a summary on success. + validation + deterministic replay enabled, and raises ChainViolationError + (code CHAIN_VIOLATION) with ALL violations of the failing event on the + FIRST invalid event. Returns a summary on success. + + This is the explicit full-chain/audit validation path: it replays every + revision > 0 through the deterministic transition engine (side-effect + free) and enforces the timestamp contract. The hot paths (load/apply) do + not replay history. """ try: cursor = conn.execute(EVENTS_BY_REVISION_SQL, (package_id,)) @@ -465,18 +665,26 @@ def validate_chain( raise ChainViolationError(f"package {package_id} has no events") # Lazy walk: one event at a time, so memory stays bounded to a single - # event even for long chains (perf, audit path). + # event even for long chains (perf, audit path). prev_manifest is the + # decoded predecessor manifest needed for deterministic replay; + # creation_timestamp is threaded from the revision-0 row for the + # timestamp contract. prev_event: dict[str, Any] | None = None + prev_manifest: dict[str, Any] | None = None + creation_timestamp: str | None = None event_count = 0 while row is not None: event = dict(row) event_count += 1 # The kernel decodes and reports malformed BLOBs as violations, and - # re-validates the manifest schema (audit mode). + # re-validates the manifest schema + deterministic replay (audit mode). violations = check_event_invariants( package_id=package_id, event=event, prev_event=prev_event, + prev_manifest=prev_manifest, + replay=True, + creation_timestamp=creation_timestamp, manifest=None, verify_artifacts=verify_artifacts, artifact_store=artifact_store, @@ -488,7 +696,19 @@ def validate_chain( f"chain violation for {package_id} rev {event.get('revision')}: " + "; ".join(violations) ) + if event.get("revision") == 0: + creation_timestamp = event.get("created_at") prev_event = event + # Decode the just-validated manifest for the NEXT event's replay + # (bounded to one predecessor; a decode failure here is impossible for + # a validated event and would fail the next replay closed). + scratch: list[str] = [] + pm, _ = _decode_canonical_json_object( + event.get("manifest_json"), + f"manifest_json({package_id}, rev {event.get('revision')})", + scratch, + ) + prev_manifest = pm row = cursor.fetchone() return {"package_id": package_id, "events": event_count, "valid": True} diff --git a/methodfactory/storage/store.py b/methodfactory/storage/store.py index 0242243..927667f 100644 --- a/methodfactory/storage/store.py +++ b/methodfactory/storage/store.py @@ -451,6 +451,7 @@ def create( violations = check_event_invariants( package_id=package_id, event=event, prev_event=None, manifest=manifest, decode_blobs=False, + creation_timestamp=created_at, ) if violations: raise ManifestInvalidError( diff --git a/methodfactory/tests/test_invariant_replay.py b/methodfactory/tests/test_invariant_replay.py new file mode 100644 index 0000000..03f8e48 --- /dev/null +++ b/methodfactory/tests/test_invariant_replay.py @@ -0,0 +1,463 @@ +"""Deterministic replay + timestamp-contract tests (invariant closure, +senior review 4885538290, Lane 1). + +The authoritative validator (validate_chain) must PROVE that the stored +resulting manifest is exactly the deterministic result Method Factory would +have produced from the stored action and predecessor manifest, using the +SINGLE deterministic transition engine (engine.apply.next_manifest) — not a +second per-action validator, and not the transactional mutation path. + +Adversarial tampering recomputes every immediate hash (action_sha256 / +resulting_manifest_sha256) so the new deterministic replay invariant is the +ONLY reason the test fails — exactly the coherent-looking-rewrite scenario +the closure targets. Valid history must still pass. +""" + +from __future__ import annotations + +import json +import sqlite3 +import tempfile +import unittest +from pathlib import Path + +from methodfactory.storage.errors import ChainViolationError +from methodfactory.storage.paths import DB_FILENAME +from methodfactory.storage.serialization import canonical_bytes, sha256_hex +from methodfactory.storage.store import SqliteManifestStore + +PKG = "pkg_demo_001" + + +# ── fixtures (mirror the chain-validator patterns; local helpers are an +# accepted cosmetic nit, kept local so this module is self-contained) ── + + +def _record_input(action_id="act_in_1", expected_revision=0, input_id="in_1", + content="hello"): + return { + "protocol_version": "0.1", "action_id": action_id, + "package_id": PKG, "expected_revision": expected_revision, + "action": "record_input", "basis": {}, + "payload": {"input_id": input_id, "kind": "text", "content": content, + "source": "operator", "disposition": "incorporated"}, + } + + +def _set_objective(action_id="act_obj_1", expected_revision=1): + return { + "protocol_version": "0.1", "action_id": action_id, + "package_id": PKG, "expected_revision": expected_revision, + "action": "set_objective", "basis": {}, + "payload": {"statement": "Build a skill", "desired_outcomes": []}, + } + + +def _prepare_summary(action_id="act_prep_1", expected_revision=2): + return { + "protocol_version": "0.1", "action_id": action_id, + "package_id": PKG, "expected_revision": expected_revision, + "action": "prepare_summary", "basis": {}, "payload": {}, + } + + +def _confirm_summary(digest, action_id="act_conf_1", expected_revision=3): + return { + "protocol_version": "0.1", "action_id": action_id, + "package_id": PKG, "expected_revision": expected_revision, + "action": "confirm_summary", "basis": {"summary_sha256": digest}, + "payload": {"operator_id": "vincent"}, + } + + +def _record_draft(action_id="act_art_1", expected_revision=4): + return { + "protocol_version": "0.1", "action_id": action_id, + "package_id": PKG, "expected_revision": expected_revision, + "action": "record_draft_artifact", "basis": {}, + "payload": {"artifact_id": "art_1", "kind": "skill", + "logical_path": "skills/x/SKILL.md", "content": "body"}, + } + + +def _full_chain(store): + store.create(PKG, "Build a skill", + created_at="2026-08-07T00:00:00+00:00") + store.apply(_record_input(expected_revision=0)) + store.apply(_set_objective(expected_revision=1)) + m3 = store.apply(_prepare_summary(expected_revision=2)) + store.apply(_confirm_summary(m3["summary"]["digest"], expected_revision=3)) + store.apply(_record_draft(expected_revision=4)) + + +def _raw(root: Path): + c = sqlite3.connect(str(root / DB_FILENAME)) + c.execute("DROP TRIGGER IF EXISTS events_no_update") + c.execute("DROP TRIGGER IF EXISTS events_no_delete") + return c + + +def _tamper_action(root: Path, revision: int, semantic: dict, + *, action: str | None = None) -> None: + """Write a CANONICAL semantic action (with the SAME frozen field set) and + recompute action_sha256 over those canonical bytes, so only the replay + invariant (or an explicit A3 binding) can fail. Optionally updates the + indexed action column for coherent rewrites.""" + data = canonical_bytes(semantic) + c = _raw(root) + if action is None: + c.execute( + "UPDATE events SET action_json=?, action_sha256=? " + "WHERE package_id=? AND revision=?", + (data, sha256_hex(data), PKG, revision), + ) + else: + c.execute( + "UPDATE events SET action_json=?, action_sha256=?, action=? " + "WHERE package_id=? AND revision=?", + (data, sha256_hex(data), action, PKG, revision), + ) + c.commit() + c.close() + + +def _tamper_manifest(root: Path, revision: int, transform) -> None: + """Decode a stored manifest, transform it, rewrite bytes + digest so only + the intended invariant can fail.""" + c = _raw(root) + row = c.execute( + "SELECT manifest_json FROM events WHERE package_id=? AND revision=?", + (PKG, revision), + ).fetchone() + m = json.loads(row[0]) + transform(m) + data = canonical_bytes(m) + c.execute( + "UPDATE events SET manifest_json=?, resulting_manifest_sha256=? " + "WHERE package_id=? AND revision=?", + (data, sha256_hex(data), PKG, revision), + ) + c.commit() + c.close() + + +def _tamper_fields(root: Path, revision: int, fields: dict) -> None: + c = _raw(root) + sets = ", ".join(f"{k}=?" for k in fields) + c.execute( + f"UPDATE events SET {sets} WHERE package_id=? AND revision=?", + (*fields.values(), PKG, revision), + ) + c.commit() + c.close() + + +def _semantic_of(root: Path, revision: int) -> dict: + c = _raw(root) + row = c.execute( + "SELECT action_json FROM events WHERE package_id=? AND revision=?", + (PKG, revision), + ).fetchone() + c.close() + return json.loads(row[0]) + + +class ReplayPositiveTests(unittest.TestCase): + def test_valid_full_chain_replays_cleanly(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + result = store.validate_chain(PKG) + self.assertTrue(result["valid"]) + self.assertEqual(result["events"], 6) + finally: + store.close() + + def test_cancel_chain_replays_cleanly(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + store.create(PKG, "Build a skill", + created_at="2026-08-07T00:00:00+00:00") + store.apply({ + "protocol_version": "0.1", "action_id": "act_cancel_1", + "package_id": PKG, "expected_revision": 0, + "action": "cancel", "basis": {}, "payload": {}, + }) + result = store.validate_chain(PKG) + self.assertTrue(result["valid"]) + self.assertEqual(store.load(PKG)["state"], "CANCELLED") + finally: + store.close() + + +class ReplayAdversarialTests(unittest.TestCase): + """Coherent-looking rewrites: every immediate hash is recomputed so the + deterministic replay invariant is the ONLY reason validation fails.""" + + def test_record_input_content_change_rejected(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + action = _semantic_of(Path(td), 1) + action["payload"]["content"] = "HELLO" + _tamper_action(Path(td), 1, action) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain(PKG) + self.assertIn("deterministic", str(ctx.exception)) + finally: + store.close() + + def test_set_objective_statement_change_rejected(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + action = _semantic_of(Path(td), 2) + action["payload"]["statement"] = "A totally different goal" + _tamper_action(Path(td), 2, action) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain(PKG) + self.assertIn("deterministic", str(ctx.exception)) + finally: + store.close() + + def test_artifact_content_change_rejected(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + action = _semantic_of(Path(td), 5) + action["payload"]["content"] = "EVIL CONTENT" + _tamper_action(Path(td), 5, action) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain(PKG) + self.assertIn("deterministic", str(ctx.exception)) + finally: + store.close() + + def test_confirm_operator_id_change_rejected(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + action = _semantic_of(Path(td), 4) + action["payload"]["operator_id"] = "mallory" + _tamper_action(Path(td), 4, action) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain(PKG) + self.assertIn("deterministic", str(ctx.exception)) + finally: + store.close() + + def test_confirm_summary_basis_change_rejected(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + action = _semantic_of(Path(td), 4) + action["basis"]["summary_sha256"] = "1" * 64 + _tamper_action(Path(td), 4, action) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain(PKG) + # The gate re-evaluated during replay binds the basis to the + # stored summary digest; a stale basis fails the transition. + self.assertIn("deterministic", str(ctx.exception)) + finally: + store.close() + + def test_event_timestamp_change_with_manifest_inconsistent_rejected(self): + """Event row created_at changed; manifest updated_at left at the + original — the row binding and/or replay must catch it.""" + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_fields(Path(td), 3, {"created_at": "2099-01-01T00:00:00+00:00"}) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain(PKG) + text = str(ctx.exception) + self.assertTrue( + "updated_at" in text or "deterministic" in text, + f"expected timestamp/replay violation, got: {text}", + ) + finally: + store.close() + + def test_manifest_updated_at_change_with_hash_recomputed_rejected(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_manifest(Path(td), 2, lambda m: m.update( + updated_at="2099-01-01T00:00:00+00:00")) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain(PKG) + self.assertIn("updated_at", str(ctx.exception)) + finally: + store.close() + + def test_manifest_created_at_change_with_hash_recomputed_rejected(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_manifest(Path(td), 1, lambda m: m.update( + created_at="2099-01-01T00:00:00+00:00")) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain(PKG) + self.assertIn("created_at", str(ctx.exception)) + finally: + store.close() + + def test_action_and_manifest_changed_inconsistently_rejected(self): + """Action payload and resulting manifest BOTH rewritten (hashes + recomputed) but inconsistent with each other — replay must catch the + mismatch.""" + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + action = _semantic_of(Path(td), 2) + action["payload"]["statement"] = "Action says THIS" + _tamper_action(Path(td), 2, action) + _tamper_manifest(Path(td), 2, lambda m: m["objective"].update( + statement="Manifest says THAT")) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain(PKG) + self.assertIn("deterministic", str(ctx.exception)) + finally: + store.close() + + def test_illegal_transition_encoded_coherently_rejected(self): + """A canonical, rehashed action for a transition that is ILLEGAL from + the predecessor state (row action column updated coherently) must fail + replay even though every hash and binding passes.""" + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + # Rev 2's predecessor is INTAKE; record_draft_artifact is not + # legal from INTAKE. Rewrite the action AND the indexed action + # column coherently, with a schema-valid payload. + semantic = { + "protocol_version": "0.1", + "action": "record_draft_artifact", + "package_id": PKG, + "action_id": "act_obj_1", + "basis": {}, + "payload": {"artifact_id": "art_evil", "kind": "skill", + "logical_path": "skills/x/SKILL.md", "content": "x"}, + } + _tamper_action(Path(td), 2, semantic, action="record_draft_artifact") + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain(PKG) + self.assertIn("deterministic", str(ctx.exception)) + finally: + store.close() + + +class ReconstructionFailClosedTests(unittest.TestCase): + """Stored-action reconstruction reuses the normal envelope validator; a + persisted action that would not parse today must fail closed.""" + + def test_malformed_payload_for_action_fails_reconstruction(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + action = _semantic_of(Path(td), 1) + del action["payload"]["kind"] # record_input requires kind + _tamper_action(Path(td), 1, action) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain(PKG) + self.assertIn("cannot be reconstructed", str(ctx.exception)) + finally: + store.close() + + def test_unsupported_protocol_version_fails_closed(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + action = _semantic_of(Path(td), 1) + action["protocol_version"] = "9.9" + _tamper_action(Path(td), 1, action) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain(PKG) + self.assertIn("protocol_version", str(ctx.exception)) + finally: + store.close() + + +class TimestampIndependentTests(unittest.TestCase): + """Independent corruption tests for the timestamp contract (steering §4): + action-specific timestamps are derived from the event timestamp by the + transition, so digest-consistent tamper is caught by replay.""" + + def test_rev0_created_at_binding(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_manifest(Path(td), 0, lambda m: m.update( + created_at="2099-01-01T00:00:00+00:00")) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain(PKG) + self.assertIn("created_at", str(ctx.exception)) + finally: + store.close() + + def test_summary_presented_at_tamper_rejected(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_manifest(Path(td), 3, lambda m: m["summary"].update( + presented_at="2099-01-01T00:00:00+00:00")) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain(PKG) + self.assertIn("deterministic", str(ctx.exception)) + finally: + store.close() + + def test_summary_confirmed_at_tamper_rejected(self): + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + _tamper_manifest(Path(td), 4, lambda m: m["summary"]["confirmation"].update( + confirmed_at="2099-01-01T00:00:00+00:00")) + with self.assertRaises(ChainViolationError) as ctx: + store.validate_chain(PKG) + self.assertIn("deterministic", str(ctx.exception)) + finally: + store.close() + + +class SideEffectFreeReplayTests(unittest.TestCase): + def test_validate_chain_writes_no_blobs(self): + """Audit replay is side-effect free: validate_chain (even with + artifact verification) must not create, modify, or delete blobs.""" + with tempfile.TemporaryDirectory() as td: + store = SqliteManifestStore(td) + try: + _full_chain(store) + blobs_dir = Path(td) / "blobs" + + def snapshot() -> dict[str, bytes]: + return {p.name: p.read_bytes() for p in blobs_dir.iterdir()} if blobs_dir.exists() else {} + + before = snapshot() + result = store.validate_chain(PKG, verify_artifacts=True) + self.assertTrue(result["valid"]) + after = snapshot() + self.assertEqual(before, after) + finally: + store.close() + + +if __name__ == "__main__": + unittest.main() From 42ff7d91a91264242d6a1e1560acdaef7ace4f4b Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Sat, 8 Aug 2026 03:51:28 +0700 Subject: [PATCH 24/41] docs: freeze v0.1.2 migration and export compatibility contract Documentation-only ADR-0012 amendment (contract-correction gate 4886392385). Freezes the exact public v0.1.2 (fb5641c / v0.1.2-integrity) compatibility contract a future migration/export implementation must honor: - legacy rev0 vs rev>0 action-hash semantics (rev>0 = full semantic action minus expected_revision); - exact serializer relationship: legacy ensure_ascii=True + default allow_nan vs current ensure_ascii=False + allow_nan=False (difference is ASCII-escape/UTF-8, not float conversion); - public-valid/current-valid compatibility matrix incl. intent.raw, intent.clarified (non-reconstructable -> MIGRATION_INCOMPATIBLE), and separate record_input.content / record_draft_artifact.content boundaries; - frozen journal/cache semantics (journal canonical; cache may lag if it matches any committed snapshot; never derive from a newer-looking cache); - lock files are transient coordination state, excluded from source identity; - timestamp normalization via current next_manifest (created_at = legacy at; engine owns updated_at/presented_at/confirmed_at); - current-engine-as-transformer architecture (no second state machine); - ID preservation (legacy event_id/action_id preserved when current-valid and globally unique; rev0 act_create_package; duplicate event_id -> fail); - source-stability-before-publication ordering; - artifact publication boundary (immutable orphan-safe blobs may precede DB); - receipt/database success semantics (receipt alone is NOT success); - method-factory-events-v1 exact field set + byte contract; - legacy-v012-jsonl: hash-canonicalization vs journal-line serialization distinction; deterministic shape reconstruction, not historical bytes; - semantic receipt identity (no raw SQLite bytes as public contract); - small actionable error taxonomy incl. DESTINATION_EXISTS. No implementation. No source/test/CLI changes. --- docs/adr/ADR-0012-persistence-architecture.md | 263 ++++++++++++++++++ 1 file changed, 263 insertions(+) diff --git a/docs/adr/ADR-0012-persistence-architecture.md b/docs/adr/ADR-0012-persistence-architecture.md index f457bc1..b394e50 100644 --- a/docs/adr/ADR-0012-persistence-architecture.md +++ b/docs/adr/ADR-0012-persistence-architecture.md @@ -543,3 +543,266 @@ migration output, and partial/coherent-enough tampering that violates deterministic semantics. They are internal-consistency evidence, not cryptographic authenticity: an attacker capable of coherently rewriting and rehashing the entire database remains out of scope. + +--- + +## Amendment - migration/export contract correction (design-freeze review 4886392385, 2026-08-08) + +Documentation-only contract correction. Freezes the exact public v0.1.2 +compatibility contract that a future migration/export implementation must +honor. Does not implement migration/export and does not alter the accepted +SQLite persistence architecture. + +### 1. Legacy action-hash semantics (binding) + +Public v0.1.2 (`v0.1.2-integrity`, commit `fb5641c`) used two action-hash +rules: + +- **Revision 0** (`create_package`): + `sha256(legacy_canonical_json({"action": "create_package", "package_id": }))` + - a special reduced create hash. +- **Revision > 0**: + `sha256(legacy_canonical_json(envelope_as_dict minus "expected_revision"))`, + i.e. the six fields + `{protocol_version, action_id, package_id, action, basis, payload}`. + +The migration reader must reconstruct each rev>0 semantic action from +snapshot + blob + predecessor evidence, then **require** the stored legacy +`action_sha256` to equal the legacy canonical hash of the unique candidate. +A recovered action is accepted only when the evidence determines exactly one +candidate. No hash inversion, no invented payload. + +### 2. Canonical-serializer relationship (exact wording) + +Legacy canonical hashing used: + +- `sort_keys=True`; +- `separators=(",", ":")`; +- `ensure_ascii=True`; +- Python's **default** `allow_nan` behavior. + +Current canonical hashing uses: + +- `sort_keys=True`; +- `separators=(",", ":")`; +- `ensure_ascii=False`; +- `allow_nan=False`. + +The relevant public v0.1.2 action/manifest schemas do not contain arbitrary +floating-point semantic fields, so the observed migration compatibility +difference is the **ASCII-escape/UTF-8 representation**, not a floating-point +conversion rule. Do not overclaim equivalence of serializer options. + +Legacy digests are therefore not preserved as current canonical digests; +migration recomputes current hashes from imported content (ADR-0012 §4). + +### 3. Public-valid / current-valid compatibility contract + +Where a public-valid v0.1.2 value is now current-invalid, migration fails +closed with `MIGRATION_INCOMPATIBLE` (package_id, revision, action_id, +reason). Current validation is never weakened; historical IDs are never +renamed; no truncation, whitespace rewriting, control-character stripping, or +silent normalization. + +| Surface | Public v0.1.2 | Current | Classification | +|---|---|---|---| +| `package_id` | `^pkg_[A-Za-z0-9_-]{1,63}$` | identical | A (identical) | +| `action_id` | non-empty str ≤64, no grammar | ≤64 + `validate_identifier` + control-char | C if legacy used invalid chars, else A | +| `input_id` / `artifact_id` / `kind` | non-empty str, no grammar | `validate_identifier` + control-char | C if legacy used invalid chars, else A | +| `operator_id` | any str (or None) | `validate_identifier` | C if legacy used invalid chars, else A | +| `logical_path` | legacy: lstrip `/`, block `{..,/,\}`, ≤255 | strict relative, no backslash/`%`/control/empty-segment/`.`/`..`, ≤255 | C for `a//b`, leading/trailing `/`, `%2f`, backslash; else A | +| `intent.raw` | any string, no length/control boundary | `MAX_INTENT_CHARS = 65,536` + control-char validation | C if legacy outside boundary | +| `intent.clarified` | engine does not populate it (remains `None`) | must be string or null | Non-null legacy `clarified` is **non-reconstructable historical state** → fail closed (see §5) | +| `record_input.content` | any string, no length limit | `MAX_CONTENT_CHARS` (characters) + persisted UTF-8 blob byte limit (`MAX_ARTIFACT_BYTES`) | C if legacy exceeds either applicable boundary | +| `record_draft_artifact.content` | any string, no length limit | `MAX_CONTENT_CHARS` (characters) + artifact/blob byte limit (`MAX_ARTIFACT_BYTES`) | C if legacy exceeds either applicable boundary | +| `statement` | any string | `MAX_STATEMENT_CHARS` (16 KiB) | C if legacy exceeds | +| `desired_outcomes` | list of str, no count/length limit | `MAX_OUTCOMES` (100) + ≤16 KiB each | C if legacy exceeds | +| `reason` (exclusion/cancel) | str or None, no length limit | `MAX_REASON_CHARS` (1 KiB) + control-char | C if legacy exceeds | +| control characters | not validated in legacy envelope | validated on ids/kinds/reasons/statements/outcomes/preview | C where legacy carried control chars in a now-checked field | +| Unicode / lone surrogate | legacy accepted any str | current rejects lone surrogates at several boundaries | A for well-formed Unicode (hash differs - recomputed); C only if legacy persisted a lone surrogate | +| `event_id` | `evt_` (36 chars) | `validate_identifier` + global UNIQUE | A for legacy format | + +`record_input.content` and `record_draft_artifact.content` are frozen as +**separate** compatibility rows because their current semantic character +limits and persisted blob byte limits are the same constants but apply to +different storage boundaries. + +### 4. Frozen v0.1.2 journal/cache semantics + +- `events/.events.jsonl` is the **canonical** public v0.1.2 + history source. +- `packages/.json` is a **latest-manifest cache**. + +The frozen migration reader preserves the public crash-tolerance semantics: + +- cache absent while reconstructable journal snapshots exist is acceptable; +- cache need not equal the latest journal snapshot; +- a lagging cache is valid if its digest matches **any** committed journal + snapshot (matching public `_validate_cache_if_present()` behavior); +- a cache that matches no committed journal snapshot is invalid; +- migration derives canonical package history from the **journal**, never from + a newer-looking cache. + +Do not tighten this into `cache == last event`; that would reject legitimate +public v0.1.2 crash states. + +### 5. Non-reconstructable historical state + +Migration compatibility is for histories that can be validated and +semantically reconstructed - not arbitrary hand-rehashed/tampered structures +merely tolerated by the old loader. A non-null legacy `intent.clarified` +(which the public engine never produces) is non-reconstructable and fails +closed. Arbitrary unrecoverable `cancel.reason` → `MIGRATION_INCOMPATIBLE`. + +### 6. Timestamp normalization (advancing-clock evidence) + +Public v0.1.2 calls `now()` separately for `summary.presented_at`, +`summary.confirmation.confirmed_at`, manifest `updated_at`, and event `at`. +Advancing-clock archaeology at `fb5641c` proved: event `at` is the latest +timestamp in each event; `updated_at` is 1 tick earlier; `presented_at` / +`confirmed_at` are 2-3 ticks earlier; rev-0 has all equal. + +Frozen normalization rule: + +- rev-0 current timestamp = legacy event `at` (creation timestamp); +- rev>0 current row `created_at` = legacy event `at`; +- current `next_manifest(..., created_at=legacy_event.at)` deterministically + sets modern `updated_at`, `presented_at`, `confirmed_at`. + +Original distinct v0.1.2 internal timestamps remain in the untouched legacy +source/evidence and are **not** copied into current fields whose +deterministic contract differs. + +### 7. Current-engine-as-transformer architecture + +Migration must not hand-author rev>0 modern manifests. The architectural path: + +``` +legacy validation +→ exact action reconstruction +→ current-valid ActionEnvelope +→ CURRENT next_manifest +→ modern manifest + blobs +→ semantic equivalence check +→ SQLite insertion +``` + +No second current state machine. Fields excluded from equivalence because they +intentionally change representation: canonical-serializer-dependent hashes, +summary inline-content representation, normalized timestamps, current lineage +hashes. All other semantics must match the legacy snapshot exactly. + +### 8. ID preservation rule + +Preserve legacy `event_id` exactly if current-valid and globally unique; +preserve legacy `action_id` exactly if current-valid; preserve rev-0 +`act_create_package`. Duplicate legacy `event_id` across packages → +`MIGRATION_INCOMPATIBLE`. The current `store.create()` ID-generation +convention is not retroactively imposed on migration rows. No silent +historical-ID renaming. + +### 9. Source-stability-before-publication + +``` +initial source identity/hash +→ legacy validation +→ temporary modern store construction +→ complete modern validation +→ FINAL source identity/hash +→ require exact equality +→ ONLY THEN destination publication +``` + +No final SQLite database becomes visible before the source-stability proof +succeeds. The residual post-check TOCTOU window is documented honestly; legacy +locks are not revived to eliminate it. + +### 10. Artifact publication boundary (honest) + +Current immutable blobs may become visible before canonical DB publication. +They are content-addressed, immutable, verified, and orphan-safe. Migration is +therefore **not** claimed to make every filesystem write atomically invisible; +only canonical SQLite DB publication is atomic. No automatic orphan deletion +during migration. + +### 11. Receipt/database success semantics + +Preferred publication order: (1) publish/fsync final receipt; (2) publish/ +fsync final database; (3) final read-only verification. + +**A receipt by itself is NOT successful migration.** Migration is successful +only when: + +- final database exists; +- matching final receipt exists; +- their identities correspond to the same migration; +- final read-only database validation succeeds. + +Crash state "receipt present + DB absent" is incomplete/ambiguous migration +evidence, not success. Fail closed with explicit operator recovery +instructions. No recovery daemon or transaction journal to make two files +atomically appear together. + +### 12. `method-factory-events-v1` (supported export) + +Unique per-line field set (no duplicate keys): + +``` +format, format_version, package_id, revision, event_id, action_id, action, +state_before, state_after, action_sha256, previous_manifest_sha256, +resulting_manifest_sha256, created_at, semantic_action, manifest +``` + +One event per line; current canonical UTF-8 JSON (sorted keys, compact +separators, `ensure_ascii=False`); exactly one LF after each line; exactly one +final newline. Explicit SQL ordering: `ORDER BY package_id, revision`. + +### 13. `legacy-v012-jsonl` - hash canonicalization vs journal-line serialization + +These are two different public serializations and must not be conflated. + +- Public v0.1.2 **hash canonicalization**: + `json.dumps(value, sort_keys=True, separators=(",",":"), ensure_ascii=True)`. +- Public v0.1.2 **journal-line serialization**: + `json.dumps(event, sort_keys=True) + "\n"` - Python's default JSON spacing + and `ensure_ascii=True`. + +`legacy-v012-jsonl` is a deterministic stream of **reconstructed public +v0.1.2 event objects** serialized using the public v0.1.2 **event-line writer +semantics**. It reconstructs the public event SHAPE and byte serializer. + +It does NOT promise: + +- reproduction of an original historical machine's exact journal; +- reproduction of original internal timestamp distinctions lost during + migration; +- reconstruction of the entire legacy directory/filesystem layout. + +For multi-package output, deterministic ordering is `package_id`, then +`revision`. This is one evidence stream, whereas the old canonical store used +one journal file per package. + +### 14. Semantic receipt identity posture + +Receipt identity is semantic: exact legacy source identity/inventory; exact +source commit/tag identifier; package count; event count; destination schema +version; resulting package/event count; migration tool/version identity; full +validation result. Raw SQLite file bytes are **not** frozen as the public +semantic compatibility identity. + +### 15. Error taxonomy + +Small actionable taxonomy, no redundant subclasses per command: + +| Code | Condition | +|---|---| +| `LEGACY_STORE_DETECTED` (existing) | detect-migration-required on normal open | +| `LEGACY_SOURCE_INVALID` | unrecognized/unsupported legacy source | +| `LEGACY_CHAIN_INVALID` | invalid legacy chain | +| `MIGRATION_INCOMPATIBLE` | unreconstructable historical semantics, or public-valid value now current-invalid | +| `SOURCE_CHANGED` | source changed during migration (fail before publication) | +| `MIGRATION_PUBLISH_FAILED` | atomic publication failure | +| `DESTINATION_EXISTS` | final destination already exists (dedicated stable identity; a CLI caller must distinguish "cannot overwrite" from generic storage failure) | + +Reuse current storage/path/manifest errors where their semantics are already +exact. No alias proliferation. From b9e46c110f82b6cb8b8505ab16f21dceb47e3d32 Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Sat, 8 Aug 2026 04:17:56 +0700 Subject: [PATCH 25/41] docs: close migration compatibility contract MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documentation-only ADR-0012 closure (final documentation gate). 1. Freeze legacy .lock / semantic source-identity semantics: events/.lock is transient v0.1.2 coordination state, NOT canonical historical package evidence; excluded from receipt source hashes, semantic source identity, and before/after source-equivalence hashes. Frozen semantic source set = events/*.events.jsonl, existing packages/*.json caches, artifacts/blobs/*. Migration must not revive lock acquisition/stale-lock recovery/PID interpretation/lock repair/automatic lock deletion. If a legacy .lock is present at migration time, refuse to start with existing CONCURRENCY error semantics, report the lock path, do not mutate/delete it, leave operator investigation outside migration. 2. Correct the current action-hash contract (§G): the canonical semantic request is exactly {protocol_version, action, package_id, action_id, basis, payload}, expressed via the single semantic_action primitive; expected_revision remains the only excluded envelope field. Makes §G internally consistent with the migration/export amendment. 3. Correct character-vs-byte units in the compatibility table: MAX_STATEMENT_CHARS = 16,384 characters; desired-outcome entry limit = 16,384 characters; MAX_REASON_CHARS = 1,024 characters. Byte-style KiB wording removed (byte limits retained only where *_BYTES applies). No implementation. No source/test/CLI changes. --- docs/adr/ADR-0012-persistence-architecture.md | 51 ++++++++++++++++--- 1 file changed, 45 insertions(+), 6 deletions(-) diff --git a/docs/adr/ADR-0012-persistence-architecture.md b/docs/adr/ADR-0012-persistence-architecture.md index b394e50..8dd1555 100644 --- a/docs/adr/ADR-0012-persistence-architecture.md +++ b/docs/adr/ADR-0012-persistence-architecture.md @@ -426,22 +426,30 @@ One authoritative validator (owned by the storage layer, exercised on every tran ### G. Canonical action hash semantics (item 7) -`action_sha256` covers the complete normalized semantic request used for idempotency: +`action_sha256` covers the complete normalized semantic request used for +idempotency. It is the hash of the canonical semantic-action object produced +by the single `semantic_action` construction primitive +(`methodfactory/storage/serialization.semantic_action`), which is exactly: ```python -action_sha256 = sha256_hex(canonical_json({ +{ + "protocol_version": protocol_version, "action": action, "package_id": package_id, "action_id": action_id, "basis": basis, "payload": payload, -})) +} ``` - It includes every field that could change the requested outcome. - It excludes **only** `expected_revision` (optimistic-concurrency/transport metadata, not part of the requested outcome). - Same `action_id` + same hash → idempotent replay. Same `action_id` + different hash → `ACTION_ID_CONFLICT`. Never infer idempotency from `action_id` alone. +This definition is internally consistent with the migration/export amendment +(rev>0 legacy action hashes covered the same six fields under the legacy +serializer). + ### H. Artifact write boundary and orphan safety (item 8) - Blob writes occur before the SQLite transaction, are content-addressed, immutable, and verified to exist before the event referencing them is inserted. @@ -615,9 +623,9 @@ silent normalization. | `intent.clarified` | engine does not populate it (remains `None`) | must be string or null | Non-null legacy `clarified` is **non-reconstructable historical state** → fail closed (see §5) | | `record_input.content` | any string, no length limit | `MAX_CONTENT_CHARS` (characters) + persisted UTF-8 blob byte limit (`MAX_ARTIFACT_BYTES`) | C if legacy exceeds either applicable boundary | | `record_draft_artifact.content` | any string, no length limit | `MAX_CONTENT_CHARS` (characters) + artifact/blob byte limit (`MAX_ARTIFACT_BYTES`) | C if legacy exceeds either applicable boundary | -| `statement` | any string | `MAX_STATEMENT_CHARS` (16 KiB) | C if legacy exceeds | -| `desired_outcomes` | list of str, no count/length limit | `MAX_OUTCOMES` (100) + ≤16 KiB each | C if legacy exceeds | -| `reason` (exclusion/cancel) | str or None, no length limit | `MAX_REASON_CHARS` (1 KiB) + control-char | C if legacy exceeds | +| `statement` | any string | `MAX_STATEMENT_CHARS` = 16,384 characters | C if legacy exceeds | +| `desired_outcomes` | list of str, no count/length limit | `MAX_OUTCOMES` (100) + entry limit 16,384 characters each | C if legacy exceeds | +| `reason` (exclusion/cancel) | str or None, no length limit | `MAX_REASON_CHARS` = 1,024 characters + control-char | C if legacy exceeds | | control characters | not validated in legacy envelope | validated on ids/kinds/reasons/statements/outcomes/preview | C where legacy carried control chars in a now-checked field | | Unicode / lone surrogate | legacy accepted any str | current rejects lone surrogates at several boundaries | A for well-formed Unicode (hash differs - recomputed); C only if legacy persisted a lone surrogate | | `event_id` | `evt_` (36 chars) | `validate_identifier` + global UNIQUE | A for legacy format | @@ -646,6 +654,37 @@ The frozen migration reader preserves the public crash-tolerance semantics: Do not tighten this into `cache == last event`; that would reject legitimate public v0.1.2 crash states. +### 4a. Legacy `.lock` and semantic source-identity semantics + +`events/.lock` is **transient v0.1.2 coordination state** and is +**NOT** part of canonical historical package evidence. It must be excluded +from: + +- migration receipt source hashes; +- semantic source identity; +- before/after source-equivalence hashes. + +The frozen **semantic source set** is the existing canonical/preserved files: + +- `events/*.events.jsonl`; +- existing `packages/*.json` cache files; +- `artifacts/blobs/*`. + +Migration must not revive: + +- lock acquisition; +- stale-lock recovery; +- PID interpretation; +- lock repair; +- automatic lock deletion. + +**Conservative operational behavior when a legacy `.lock` file is present at +migration time (preferred):** refuse to start migration using the existing +`CONCURRENCY`/concurrency error semantics, report the lock path, do not +mutate or delete it, and leave operator investigation/removal outside +migration. The lock file itself remains outside semantic source identity +either way. + ### 5. Non-reconstructable historical state Migration compatibility is for histories that can be validated and From c4ddcf0fdae7ae27e2eb3802ab9fc597dc31e9b6 Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Sat, 8 Aug 2026 05:10:11 +0700 Subject: [PATCH 26/41] feat(migration): frozen v0.1.2 reader, atomic SQLite migration, deterministic exports, bounded CLI - migrations/v012_jsonl.py: frozen read-only reader for exact public fb5641c (tag v0.1.2-integrity); legacy validation, cache semantics, artifact verification, semantic source inventory; no CAS/lock/repair/ append mechanics. - migrations/migrate.py: atomic migrate_store() - legacy validation, semantic-action reconstruction keyed by legacy action_sha256, current next_manifest transformation, equivalence verification, source-stability proof, temp-DB build + full validation, durable receipt+DB publication, final read-only verification, fault seams. - migrations/export.py: deterministic method-factory-events-v1 and legacy-v012-jsonl exports (read-only, consistent read). - cli.py: restore bounded mf migrate-store + mf export; lifecycle commands remain unavailable; --version unchanged. - storage errors: add LEGACY_SOURCE_INVALID, LEGACY_CHAIN_INVALID, MIGRATION_INCOMPATIBLE, SOURCE_CHANGED, MIGRATION_PUBLISH_FAILED, DESTINATION_EXISTS. ADR-0012 amendment; no merge/tag/release/main/forensic mutation. --- methodfactory/cli.py | 76 ++- methodfactory/migrations/__init__.py | 1 + methodfactory/migrations/export.py | 260 ++++++++ methodfactory/migrations/migrate.py | 787 +++++++++++++++++++++++++ methodfactory/migrations/v012_jsonl.py | 397 +++++++++++++ methodfactory/storage/__init__.py | 19 +- methodfactory/storage/errors.py | 37 ++ 7 files changed, 1562 insertions(+), 15 deletions(-) create mode 100644 methodfactory/migrations/__init__.py create mode 100644 methodfactory/migrations/export.py create mode 100644 methodfactory/migrations/migrate.py create mode 100644 methodfactory/migrations/v012_jsonl.py diff --git a/methodfactory/cli.py b/methodfactory/cli.py index 5ce3187..d19ddcf 100644 --- a/methodfactory/cli.py +++ b/methodfactory/cli.py @@ -1,8 +1,14 @@ """`mf` CLI — thin adapter over the engine (ADR-0001). -Phase 2: version + availability notice only. The full command surface -(create/apply/status/summary/validate, migrate-store, export) returns after the -SQLite store and lifecycle are implemented (ADR-0012 Phase 2 stop gate). +Bounded command surface during the persistence reset: + + mf --version + mf migrate-store --source [--dest ] + mf export --store [--output ] --format + +Only migration/export commands are restored in this phase (ADR-0012 +amendment). Lifecycle commands (create/apply/status/summary/validate +mutation/review/trial/ship/triage) remain unavailable. """ from __future__ import annotations @@ -11,28 +17,74 @@ import sys from . import __version__ +from .domain.errors import MethodFactoryError AVAILABILITY = ( "Method Factory storage is under architecture reset (ADR-0012); " - "commands return in a later phase." + "lifecycle commands return in a later phase." ) +def _fail(err: MethodFactoryError) -> int: + print(err.as_dict(), file=sys.stderr) + return 1 + + +def _cmd_migrate_store(args) -> int: + from .migrations.migrate import migrate_store + + try: + receipt = migrate_store(args.source, dest=args.dest) + except MethodFactoryError as exc: + return _fail(exc) + import json + + print(json.dumps(receipt, sort_keys=True, indent=2)) + return 0 + + +def _cmd_export(args) -> int: + from .migrations.export import export_events + + try: + count = export_events(args.store, args.output, fmt=args.format) + except MethodFactoryError as exc: + return _fail(exc) + if args.output is None: + # events already written to stdout; report count on stderr + print(f"exported {count} events", file=sys.stderr) + else: + print(f"exported {count} events to {args.output}") + return 0 + + def main(argv: list[str] | None = None) -> int: parser = argparse.ArgumentParser(prog="mf", description="Method Factory CLI") parser.add_argument( "--version", action="version", version=f"methodfactory {__version__}" ) - parser.add_argument( - "args", nargs="*", - help="command + arguments (unavailable in this phase: persistence reset in progress)", + sub = parser.add_subparsers(dest="command") + + p_migrate = sub.add_parser("migrate-store", help="migrate a v0.1.2 store to SQLite") + p_migrate.add_argument("--source", required=True, help="legacy store root") + p_migrate.add_argument("--dest", default=None, help="destination SQLite path") + p_migrate.set_defaults(func=_cmd_migrate_store) + + p_export = sub.add_parser("export", help="deterministic event export") + p_export.add_argument("--store", required=True, help="SQLite store root") + p_export.add_argument("--output", default=None, help="output path (default stdout)") + p_export.add_argument( + "--format", + default="method-factory-events-v1", + choices=["method-factory-events-v1", "legacy-v012-jsonl"], ) + p_export.set_defaults(func=_cmd_export) + args = parser.parse_args(argv) - if args.args: - print(AVAILABILITY, file=sys.stderr) - return 2 - parser.print_help() - return 0 + if args.command is None: + parser.print_help() + return 0 + return args.func(args) if __name__ == "__main__": diff --git a/methodfactory/migrations/__init__.py b/methodfactory/migrations/__init__.py new file mode 100644 index 0000000..ac8e280 --- /dev/null +++ b/methodfactory/migrations/__init__.py @@ -0,0 +1 @@ +"""Migration package — public v0.1.2 → SQLite (ADR-0012 amendment).""" diff --git a/methodfactory/migrations/export.py b/methodfactory/migrations/export.py new file mode 100644 index 0000000..a64c16a --- /dev/null +++ b/methodfactory/migrations/export.py @@ -0,0 +1,260 @@ +"""Deterministic event exports (ADR-0012 amendment §12-§13). + +Two formats: + +1. `method-factory-events-v1` — supported evidence export. One JSON object + per line with the frozen field set, current canonical UTF-8 JSON, + `ORDER BY package_id, revision`. + +2. `legacy-v012-jsonl` — evidence/compatibility export reconstructing the + PUBLIC v0.1.2 event SHAPE using public v0.1.2 semantics: + - inline summary content; + - legacy canonical manifest hashes (ensure_ascii=True, compact); + - legacy predecessor hashes; + - legacy rev>0 action hashes using legacy canonical hash serialization; + - legacy special rev0 action hash; + - journal-line serialization `json.dumps(event, sort_keys=True) + "\\n"` + (Python default spacing, ASCII escaping). + +Both are read-only and deterministic: same DB + same exporter version -> +byte-identical output. Export never mutates the store; it uses a read-only +SQLite connection with a consistent read transaction. +""" + +from __future__ import annotations + +import json +import os +import sqlite3 +from pathlib import Path +from typing import Any + +from ..storage.errors import StorageError +from ..storage.serialization import canonical_bytes, sha256_hex +from ..storage.sqlite import ( + APPLICATION_ID, + USER_VERSION, + close_database, + open_database, +) +from .v012_jsonl import ( + legacy_digest_json, + legacy_line_json, +) + +EVENTS_V1_FORMAT = "method-factory-events-v1" +EVENTS_V1_VERSION = 1 + +LEGACY_JSONL_FORMAT = "legacy-v012-jsonl" +LEGACY_JSONL_VERSION = 1 + +EXPORT_SQL = """ +SELECT package_id, revision, event_id, action_id, action, action_sha256, + state_before, state_after, previous_manifest_sha256, + resulting_manifest_sha256, created_at, action_json, manifest_json +FROM events +ORDER BY package_id, revision +""" + + +def _read_rows(conn: sqlite3.Connection) -> list[dict]: + rows = conn.execute(EXPORT_SQL).fetchall() + return [dict(r) for r in rows] + + +def _current_event_object(row: dict) -> dict: + """Frozen `method-factory-events-v1` per-line object.""" + return { + "format": EVENTS_V1_FORMAT, + "format_version": EVENTS_V1_VERSION, + "package_id": row["package_id"], + "revision": row["revision"], + "event_id": row["event_id"], + "action_id": row["action_id"], + "action": row["action"], + "state_before": row["state_before"], + "state_after": row["state_after"], + "action_sha256": row["action_sha256"], + "previous_manifest_sha256": row["previous_manifest_sha256"], + "resulting_manifest_sha256": row["resulting_manifest_sha256"], + "created_at": row["created_at"], + "semantic_action": _decode_json(row["action_json"]), + "manifest": _decode_json(row["manifest_json"]), + } + + +def _decode_json(raw: Any) -> dict: + if isinstance(raw, str): + raw = raw.encode("utf-8") + data = json.loads(raw.decode("utf-8")) + if not isinstance(data, dict): + raise StorageError("stored JSON is not an object") + return data + + +# ── legacy-v012-jsonl reconstruction ───────────────────────────────── +def _legacy_event_object(row: dict, prev_legacy_hash: str | None) -> dict: + """Reconstruct the public v0.1.2 EVENT OBJECT shape. + + Uses legacy canonical hashing and inline summary content. + """ + semantic = _decode_json(row["action_json"]) + manifest = _decode_json(row["manifest_json"]) + + # Reconstruct inline summary content: content-addressed digest == legacy + # canonical_sha256. The summary body is stored as a blob; we need its + # bytes. We recompute the summary body via the deterministic renderer + # (byte-identical to public v0.1.2) when summary present. + legacy_manifest = _to_legacy_manifest(manifest) + + # Legacy manifest hashes (ensure_ascii=True). + resulting = legacy_digest_json(legacy_manifest) + # Legacy predecessor hash: the legacy canonical hash of the PREVIOUS + # exported line's reconstructed manifest (rows process in package_id, + # revision order). This keeps the exported chain fully consistent in the + # LEGACY hash space, even when the current-era stored previous hash + # differs (non-ASCII content; ensure_ascii divergence). + prev = prev_legacy_hash + + # Legacy action hash: rev0 special reduced; rev>0 legacy canonical of + # semantic action (six fields). + if row["revision"] == 0: + action_hash = _legacy_rev0_hash(row["package_id"]) + else: + action_hash = _legacy_hash_semantic(semantic) + + return { + "event_id": row["event_id"], + "action": row["action"], + "action_id": row["action_id"], + "revision": row["revision"], + "state_before": row["state_before"], + "state_after": row["state_after"], + "resulting_manifest_sha256": resulting, + "previous_manifest_sha256": prev, + "action_sha256": action_hash, + "at": row["created_at"], + "manifest_snapshot": legacy_manifest, + } + + +def _to_legacy_manifest(manifest: dict) -> dict: + """Convert current manifest to public v0.1.2 manifest shape. + + - summary inline content: regenerate via the deterministic renderer. + - summary canonical_sha256 = digest of inline content (== current digest). + - drop content-addressed digest/size/preview; add content + canonical. + """ + import copy + + m = copy.deepcopy(manifest) + summary = m.get("summary") + if isinstance(summary, dict): + body = _render_summary(m) + m["summary"] = { + "content": body, + "canonical_sha256": summary.get("digest") or _legacy_digest_text(body), + "presented_at": summary.get("presented_at"), + "confirmation": summary.get("confirmation"), + } + return m + + +def _render_summary(manifest: dict) -> str: + from ..manifest.render import render_summary + + return render_summary(manifest) + + +def _legacy_digest_text(content: str) -> str: + from .v012_jsonl import legacy_digest_text + + return legacy_digest_text(content) + + +def _legacy_rev0_hash(package_id: str) -> str: + from .v012_jsonl import legacy_canonical_json + + import hashlib + + return hashlib.sha256( + legacy_canonical_json({"action": "create_package", "package_id": package_id}) + ).hexdigest() + + +def _legacy_hash_semantic(semantic: dict) -> str: + from .v012_jsonl import legacy_canonical_json + + import hashlib + + return hashlib.sha256(legacy_canonical_json(semantic)).hexdigest() + + +# ── public API ──────────────────────────────────────────────────────── +def export_events( + store_root: str | Path, + output: str | Path | None, + *, + fmt: str = EVENTS_V1_FORMAT, +) -> int: + """Deterministically export events. Returns number of events written. + + `output` None -> write to stdout. Otherwise atomic temp+rename to the + output path (fail closed if destination exists). + """ + if fmt not in (EVENTS_V1_FORMAT, LEGACY_JSONL_FORMAT): + raise StorageError(f"unsupported export format {fmt!r}") + + conn = open_database(store_root, read_only=True) + try: + # Verify identity before export. + app = int(conn.execute("PRAGMA application_id").fetchone()[0]) + ver = int(conn.execute("PRAGMA user_version").fetchone()[0]) + if app != APPLICATION_ID or ver != USER_VERSION: + raise StorageError( + f"database identity mismatch (app {app}, version {ver})" + ) + rows = _read_rows(conn) + finally: + close_database(conn) + + lines = [] + prev_legacy: dict[str, str | None] = {} + for row in rows: + if fmt == EVENTS_V1_FORMAT: + obj = _current_event_object(row) + else: + obj = _legacy_event_object( + row, prev_legacy.get(row["package_id"]) + ) + prev_legacy[row["package_id"]] = obj["resulting_manifest_sha256"] + lines.append(json.dumps(obj, sort_keys=True, separators=(",", ":"), + ensure_ascii=False)) + + payload = ("\n".join(lines) + "\n").encode("utf-8") if lines else b"" + + if output is None: + import sys + + sys.stdout.buffer.write(payload) + return len(rows) + + out = Path(output) + if out.exists(): + raise StorageError(f"export destination exists: {out}") + tmp = out.with_name(out.name + ".tmp") + with open(tmp, "wb") as fh: + fh.write(payload) + fh.flush() + os.fsync(fh.fileno()) + os.replace(tmp, out) + _fsync_dir(out.parent) + return len(rows) + + +def _fsync_dir(path: Path) -> None: + dir_fd = os.open(path, os.O_RDONLY) + try: + os.fsync(dir_fd) + finally: + os.close(dir_fd) diff --git a/methodfactory/migrations/migrate.py b/methodfactory/migrations/migrate.py new file mode 100644 index 0000000..48a3a4e --- /dev/null +++ b/methodfactory/migrations/migrate.py @@ -0,0 +1,787 @@ +"""Atomic v0.1.2 → current SQLite migration (ADR-0012 amendment). + +Frozen algorithm (ADR-0012 amendment §7-§15): + +1. validate CLI arguments; +2. resolve canonical source root; positive public v0.1.2 layout detection; +3. inventory/hash immutable source set (BEFORE); +4. open legacy source read-only (frozen reader); +5. full legacy validation (v0.1.2 semantics; fail closed on ambiguity); +6. per package (sorted package IDs, journal line order): + a. rev 0: preserve legacy event_id + action_id "act_create_package"; + build current create semantic action; + b. rev>0: reconstruct semantic action; require stored legacy + action_sha256 == legacy hash of the unique candidate; + if no unique candidate -> MIGRATION_INCOMPATIBLE; + c. build current ActionEnvelope; + d. run CURRENT next_manifest(predecessor, envelope, + event_id=legacy event_id, created_at=legacy event.at); + e. semantic equivalence check vs legacy snapshot (exclusions only); +7. calculate destination + temporary destination (same directory); +8. fail if final destination exists (DESTINATION_EXISTS); +9. build new SQLite store at temp destination; +10. insert deterministic transformed events/blobs (immutable blobs + pre-written, orphan-safe); +11. PRAGMA integrity_check; +12. full current chain validation (validate_chain, verify_artifacts=True); +13. close/sync SQLite cleanly; +14. generate migration receipt data; +15. durably write temp receipt; +16. FINAL source inventory/hash (AFTER) — require exact equality with step 3; +17. ONLY THEN enter publication: + a. durable final receipt publication (os.replace temp -> final; dir fsync); + b. durable atomic final DB publication (os.replace temp DB -> final; + dir fsync); +18. final read-only verification. + +A receipt alone is NOT success. Success requires: final DB exists; matching +final receipt exists; same migration identity; final read-only validation +succeeds. Crash states fail closed with explicit operator instructions. +""" + +from __future__ import annotations + +import json +import os +import sqlite3 +import uuid +from pathlib import Path +from typing import Any, Callable + +from ..adapters.artifact_store import ArtifactStore +from ..domain.errors import ( + ActionIdReuseError, + ConcurrencyError, + GateUnsatisfiedError, + IllegalTransitionError, + InvalidEnvelopeError, + InvalidPayloadError, + ManifestInvalidError, + MethodFactoryError, + StaleActionError, +) +from ..engine.apply import CREATE_PACKAGE_ACTION, next_manifest +from ..manifest.schema import validate_manifest_canonical +from ..protocol.envelope import PROTOCOL_VERSION, envelope_from_dict +from ..storage.errors import ( + ChainViolationError, + DestinationExistsError, + LegacyChainInvalidError, + LegacySourceInvalidError, + MigrationIncompatibleError, + MigrationPublishFailedError, + SourceChangedError, + StorageError, +) +from ..storage.paths import DB_FILENAME, validate_package_id +from ..storage.serialization import canonical_bytes, sha256_hex +from ..storage.sqlite import ( + APPLICATION_ID, + USER_VERSION, + close_database, + open_database, +) +from .v012_jsonl import ( + LEGACY_ACTION_CREATE, + LEGACY_COMMIT, + LEGACY_TAG, + LegacySource, + legacy_canonical_json, + legacy_digest_json, + legacy_line_json, +) + +# Receipt format identity. +RECEIPT_FORMAT = "method-factory-migration-receipt" +RECEIPT_VERSION = "v1" + +# ── Fault seams (tests inject precise single-point faults through the REAL +# implementation path, matching the store.py FAULT_HOOK pattern) ────── +FAULT_HOOK: "Callable[[str], None] | None" = None + + +def _fault(stage: str) -> None: + if FAULT_HOOK is not None: + FAULT_HOOK(stage) + +INSERT_EVENT_SQL = """ +INSERT INTO events ( + package_id, revision, event_id, action_id, action, action_sha256, + state_before, state_after, previous_manifest_sha256, + resulting_manifest_sha256, created_at, action_json, manifest_json +) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) +""" + + +def _legacy_hash_semantic(semantic: dict) -> str: + """Legacy rev>0 action hash: sha256(legacy_canonical_json(semantic)).""" + import hashlib + + return hashlib.sha256(legacy_canonical_json(semantic)).hexdigest() + + +def _legacy_rev0_hash(package_id: str) -> str: + import hashlib + + return hashlib.sha256( + legacy_canonical_json( + {"action": "create_package", "package_id": package_id} + ) + ).hexdigest() + + +# ── semantic-action reconstruction ──────────────────────────────────── +def _reconstruct_rev0(source: LegacySource, pkg, ev) -> dict: + """Build the current create semantic action from a legacy rev-0 event. + + Preserves legacy event_id and action_id "act_create_package". + """ + snap = ev.manifest_snapshot + intent = snap.get("intent") or {} + intent_raw = intent.get("raw") + if not isinstance(intent_raw, str): + raise MigrationIncompatibleError( + f"revision 0 intent is not reconstructable for {pkg.package_id}" + ) + created_at = ev.at + return { + "protocol_version": PROTOCOL_VERSION, + "action": CREATE_PACKAGE_ACTION, + "package_id": pkg.package_id, + "action_id": LEGACY_ACTION_CREATE, + "basis": {}, + "payload": {"intent": intent_raw, "created_at": created_at}, + } + + +def _candidate_hashes(source: LegacySource, pkg, ev, candidates: list[dict]) -> list[dict]: + """Return candidates whose legacy hash matches the stored legacy hash.""" + matches = [] + for cand in candidates: + if _legacy_hash_semantic(cand) == ev.action_sha256: + matches.append(cand) + return matches + + +def _reconstruct_record_input(source: LegacySource, pkg, ev) -> dict: + snap = ev.manifest_snapshot + entry = snap["inputs"][-1] + blob = source.artifact_bytes(entry["content_sha256"]) + try: + content = blob.decode("utf-8") + except UnicodeDecodeError as exc: + raise MigrationIncompatibleError( + f"record_input content for {pkg.package_id} rev {ev.revision} " + f"is not valid UTF-8" + ) from exc + # Candidate: exclusion_reason omitted vs null (finite). + candidates = [] + base = { + "input_id": entry["input_id"], + "kind": entry["kind"], + "content": content, + "source": entry["source"], + "disposition": entry["disposition"], + } + omitted = _semantic(pkg, ev, dict(base)) + candidates.append(omitted) + nulled = dict(base) + nulled["exclusion_reason"] = None + candidates.append(_semantic(pkg, ev, nulled)) + if entry.get("exclusion_reason") is not None: + explicit = dict(base) + explicit["exclusion_reason"] = entry["exclusion_reason"] + candidates.append(_semantic(pkg, ev, explicit)) + matches = _candidate_hashes(source, pkg, ev, candidates) + if len(matches) != 1: + raise MigrationIncompatibleError( + f"record_input semantic action not uniquely reconstructable for " + f"{pkg.package_id} rev {ev.revision}" + ) + return matches[0] + + +def _reconstruct_set_objective(source: LegacySource, pkg, ev) -> dict: + snap = ev.manifest_snapshot + obj = snap["objective"] + candidates = [] + omitted = _semantic(pkg, ev, {"statement": obj["statement"]}) + candidates.append(omitted) + explicit = _semantic( + pkg, ev, + {"statement": obj["statement"], + "desired_outcomes": obj.get("desired_outcomes", [])}, + ) + candidates.append(explicit) + matches = _candidate_hashes(source, pkg, ev, candidates) + if len(matches) != 1: + raise MigrationIncompatibleError( + f"set_objective semantic action not uniquely reconstructable for " + f"{pkg.package_id} rev {ev.revision}" + ) + return matches[0] + + +def _reconstruct_prepare_summary(source: LegacySource, pkg, ev) -> dict: + candidates = [_semantic(pkg, ev, {})] + matches = _candidate_hashes(source, pkg, ev, candidates) + if len(matches) != 1: + raise MigrationIncompatibleError( + f"prepare_summary semantic action not uniquely reconstructable for " + f"{pkg.package_id} rev {ev.revision}" + ) + return matches[0] + + +def _reconstruct_confirm_summary(source: LegacySource, pkg, ev) -> dict: + snap = ev.manifest_snapshot + summary = snap.get("summary") or {} + conf = summary.get("confirmation") or {} + confirmed_sha = conf.get("confirmed_summary_sha256") or summary.get( + "canonical_sha256" + ) + basis = {"summary_sha256": confirmed_sha} + op_value = conf.get("operator_id") + candidates = [] + if op_value == "operator": + # operator default may derive from omitted / null / "" / "operator". + for op in (None, "", "operator"): + candidates.append(_semantic(pkg, ev, {"operator_id": op}, basis=basis)) + omitted = _semantic(pkg, ev, {}, basis=basis) + candidates.append(omitted) + else: + candidates.append(_semantic(pkg, ev, {"operator_id": op_value}, basis=basis)) + matches = _candidate_hashes(source, pkg, ev, candidates) + if len(matches) != 1: + raise MigrationIncompatibleError( + f"confirm_summary semantic action not uniquely reconstructable for " + f"{pkg.package_id} rev {ev.revision}" + ) + return matches[0] + + +def _reconstruct_revise_intake(source: LegacySource, pkg, ev) -> dict: + candidates = [_semantic(pkg, ev, {})] + matches = _candidate_hashes(source, pkg, ev, candidates) + if len(matches) != 1: + raise MigrationIncompatibleError( + f"revise_intake semantic action not uniquely reconstructable for " + f"{pkg.package_id} rev {ev.revision}" + ) + return matches[0] + + +def _reconstruct_record_draft_artifact(source: LegacySource, pkg, ev) -> dict: + snap = ev.manifest_snapshot + art = snap["artifacts"][-1] + blob = source.artifact_bytes(art["sha256"]) + try: + content = blob.decode("utf-8") + except UnicodeDecodeError as exc: + raise MigrationIncompatibleError( + f"artifact content for {pkg.package_id} rev {ev.revision} " + f"is not valid UTF-8" + ) from exc + payload = { + "artifact_id": art["artifact_id"], + "kind": art["kind"], + "logical_path": art["logical_path"], + "content": content, + } + candidate = _semantic(pkg, ev, payload) + matches = _candidate_hashes(source, pkg, ev, [candidate]) + if len(matches) != 1: + raise MigrationIncompatibleError( + f"record_draft_artifact semantic action not uniquely " + f"reconstructable for {pkg.package_id} rev {ev.revision}" + ) + return matches[0] + + +def _reconstruct_cancel(source: LegacySource, pkg, ev) -> dict: + # Arbitrary reason is not recoverable from the snapshot. Finite + # candidates: omitted / null / empty string. If none matches, fail closed. + candidates = [ + _semantic(pkg, ev, {}), + _semantic(pkg, ev, {"reason": None}), + _semantic(pkg, ev, {"reason": ""}), + ] + matches = _candidate_hashes(source, pkg, ev, candidates) + if len(matches) != 1: + raise MigrationIncompatibleError( + f"cancel semantic action is not reconstructable for " + f"{pkg.package_id} rev {ev.revision}: reason not recoverable from " + "persisted public evidence" + ) + return matches[0] + + +def _semantic(pkg, ev, payload, basis=None) -> dict: + return { + "protocol_version": PROTOCOL_VERSION, + "action": ev.action, + "package_id": pkg.package_id, + "action_id": ev.action_id, + "basis": basis or {}, + "payload": payload, + } + + +def _reconstruct_action(source: LegacySource, pkg, ev) -> dict: + """Reconstruct the exact six-field historical semantic action. + + Returns the current-valid semantic action object (protocol_version is + current; other fields preserved) after unique hash verification. + """ + action = ev.action + if action == "record_input": + return _reconstruct_record_input(source, pkg, ev) + if action == "set_objective": + return _reconstruct_set_objective(source, pkg, ev) + if action == "prepare_summary": + return _reconstruct_prepare_summary(source, pkg, ev) + if action == "confirm_summary": + return _reconstruct_confirm_summary(source, pkg, ev) + if action == "revise_intake": + return _reconstruct_revise_intake(source, pkg, ev) + if action == "record_draft_artifact": + return _reconstruct_record_draft_artifact(source, pkg, ev) + if action == "cancel": + return _reconstruct_cancel(source, pkg, ev) + raise MigrationIncompatibleError( + f"unknown legacy action {action!r} for {pkg.package_id} rev {ev.revision}" + ) + + +# ── current-envelope construction ───────────────────────────────────── +def _to_envelope(semantic: dict, expected_revision: int): + env_dict = dict(semantic) + env_dict["expected_revision"] = expected_revision + return envelope_from_dict(env_dict) + + +# ── migration driver ────────────────────────────────────────────────── +def migrate_store( + source_root: str | Path, + dest: str | Path | None = None, +) -> dict[str, Any]: + """Run the frozen migration. Returns the receipt dict.""" + source = LegacySource(source_root) + source.validate() + + # Destination default: /methodfactory.sqlite3 + src = Path(source_root) + final_dest = Path(dest) if dest is not None else (src / DB_FILENAME) + final_dest = final_dest.resolve() + if final_dest.exists(): + raise DestinationExistsError(f"migration destination exists: {final_dest}") + + # Lock refusal: any applicable legacy .lock file -> CONCURRENCY. + locks = sorted(source.events_dir.glob("*.lock")) + if locks: + raise ConcurrencyError( + f"legacy lock file(s) present; refusing to start migration: " + + ", ".join(str(p) for p in locks) + ) + + before = source.source_inventory() + _fault("after_source_inventory_before") + + # Build temp destination (same directory as final for atomic rename). + # NOTE: the current API treats a store path as a ROOT DIRECTORY (it + # appends DB_FILENAME and creates blobs/). The temp build therefore uses + # a temp root directory; publication moves the DB FILE to the final path. + final_root = final_dest.parent + try: + final_root.mkdir(parents=True, exist_ok=True) + os.chmod(final_root, 0o700) + except OSError as exc: + raise StorageError( + f"cannot create destination parent directory: {exc}" + ) from exc + temp_root = final_root / f".{final_dest.name}.tmp.{uuid.uuid4().hex}" + + # Build the modern store. Blobs publish to the FINAL artifact store root + # (orphan-safe on failure; ADR-0012 §11), while the DB builds at temp_root. + artifacts = ArtifactStore(final_root) + try: + _fault("before_build_store") + _build_store(temp_root, source, artifacts) + _fault("after_build_store") + + # Full validation before publication. + _validate_temp_store(temp_root, artifacts) + _fault("after_validate_temp_store") + + # Source stability proof BEFORE publication. + after = source.source_inventory() + _fault("after_source_inventory_after") + if after != before: + raise SourceChangedError( + "legacy source changed during migration; destination not published" + ) + except BaseException: + # Failure before publication: remove the temp DB root. Blobs already + # published to the final artifact root are orphan-safe (ADR-0012 §11). + import shutil + + if temp_root.exists(): + shutil.rmtree(temp_root, ignore_errors=True) + raise + + # Publication: receipt first, then DB (ADR-0012 §11). + receipt = _build_receipt(source, before, temp_root) + temp_receipt = final_dest.with_name(final_dest.name + ".receipt.json.tmp") + final_receipt = final_dest.with_name(final_dest.name + ".receipt.json") + + try: + _fault("before_receipt_write") + _write_durable(temp_receipt, receipt) + _fault("before_receipt_replace") + _atomic_replace(temp_receipt, final_receipt) + _fault("before_dir_fsync_receipt") + _fsync_dir(final_root) + _fault("before_db_replace") + _atomic_replace(temp_root / DB_FILENAME, final_dest) + _fault("before_dir_fsync_db") + _fsync_dir(final_root) + _fault("after_publication") + except BaseException as exc: + # Clean up temp artifacts; never leave a partial publication claimed. + import shutil + + if temp_root.exists(): + shutil.rmtree(temp_root, ignore_errors=True) + for p in (temp_receipt,): + try: + if p.exists(): + p.unlink() + except OSError: + pass + if isinstance(exc, MethodFactoryError): + raise + raise MigrationPublishFailedError( + f"migration publication failed: {exc}" + ) from exc + + # Success-path hygiene: the temp root dir is now empty (DB file moved out). + import shutil + + if temp_root.exists(): + shutil.rmtree(temp_root, ignore_errors=True) + + # Final read-only verification. + _fault("before_final_verify") + _verify_final(final_dest) + _fault("after_final_verify") + + return receipt + + +def _build_store(temp_root: Path, source: LegacySource, artifacts: ArtifactStore) -> None: + conn = open_database(temp_root, read_only=False) + try: + for package_id in sorted(source.packages.keys()): + pkg = source.packages[package_id] + _import_package(conn, source, pkg, artifacts) + conn.commit() + except BaseException: + conn.rollback() + raise + finally: + close_database(conn) + + +def _import_package( + conn: sqlite3.Connection, + source: LegacySource, + pkg, + artifacts: ArtifactStore, +) -> None: + previous_manifest: dict | None = None + for ev in pkg.events: + try: + manifest = _import_event( + conn, source, pkg, ev, previous_manifest, artifacts + ) + except MigrationIncompatibleError: + raise + except ( + # Current-boundary rejections of legacy-valid historical values + # are MIGRATION_INCOMPATIBLE (ADR-0012 §7), never leaked as + # engine/validator errors. + InvalidEnvelopeError, + InvalidPayloadError, + ManifestInvalidError, + IllegalTransitionError, + GateUnsatisfiedError, + StaleActionError, + ActionIdReuseError, + ChainViolationError, + ) as exc: + raise MigrationIncompatibleError( + f"legacy value for {pkg.package_id} rev {ev.revision} is " + f"current-invalid: {exc}" + ) from exc + except sqlite3.IntegrityError as exc: + # Duplicate event_id (global uniqueness) or other row-integrity + # violation -> MIGRATION_INCOMPATIBLE (ADR-0012 §7 event-ID + # global uniqueness), never a raw sqlite error. + raise MigrationIncompatibleError( + f"legacy value for {pkg.package_id} rev {ev.revision} " + f"violates destination row integrity: {exc}" + ) from exc + previous_manifest = manifest + + +def _import_event( + conn: sqlite3.Connection, + source: LegacySource, + pkg, + ev, + previous_manifest: dict | None, + artifacts: ArtifactStore, +) -> dict: + if ev.revision == 0: + semantic = _reconstruct_rev0(source, pkg, ev) + # Build the modern rev-0 manifest deterministically (create_package + # is NOT an envelope action; migration inserts the row directly, + # preserving legacy event_id/action_id). + from ..manifest.schema import new_manifest + + created_at = semantic["payload"]["created_at"] + manifest = new_manifest( + pkg.package_id, semantic["payload"]["intent"], created_at + ) + manifest_bytes = canonical_bytes(manifest) + blobs: list = [] + else: + semantic = _reconstruct_action(source, pkg, ev) + env = _to_envelope(semantic, expected_revision=ev.revision - 1) + next_m, blobs = next_manifest( + previous_manifest, + env, + event_id=ev.event_id, + created_at=ev.at, + ) + manifest = next_m + manifest_bytes = canonical_bytes(manifest) + + # Publish blobs via current immutable store (orphan-safe). + for path, content in blobs: + artifacts.put(pkg.package_id, path, content) + + # Verify semantic equivalence vs legacy snapshot (excluded: hashes, + # summary inline body, normalized timestamps, lineage hashes). + _verify_equivalence(ev, manifest) + + action_bytes = canonical_bytes(semantic) + action_hash = sha256_hex(action_bytes) + resulting_hash = sha256_hex(manifest_bytes) + prev_hash = ( + None + if ev.revision == 0 + else sha256_hex(canonical_bytes(previous_manifest)) + ) + + row = ( + pkg.package_id, ev.revision, ev.event_id, semantic["action_id"], + semantic["action"], action_hash, + ev.state_before, ev.state_after, prev_hash, resulting_hash, + ev.at, action_bytes, manifest_bytes, + ) + conn.execute(INSERT_EVENT_SQL, row) + return manifest + + +def _verify_equivalence(ev, manifest: dict) -> None: + """Compare current manifest to legacy snapshot on surviving semantics.""" + snap = ev.manifest_snapshot + # identity / state / revision + if manifest.get("package_id") != snap.get("package_id"): + raise MigrationIncompatibleError("package_id mismatch after transform") + if manifest.get("revision") != ev.revision: + raise MigrationIncompatibleError("revision mismatch after transform") + if manifest.get("state") != snap.get("state"): + raise MigrationIncompatibleError("state mismatch after transform") + # intent + if manifest.get("intent") != snap.get("intent"): + raise MigrationIncompatibleError("intent mismatch after transform") + # inputs (content bytes verified by digest) + mi = manifest.get("inputs", []) + si = snap.get("inputs", []) + if len(mi) != len(si): + raise MigrationIncompatibleError("inputs count mismatch after transform") + for a, b in zip(mi, si): + for field in ("input_id", "kind", "source", "disposition", + "exclusion_reason", "content_sha256", "content_size", + "content_path"): + if a.get(field) != b.get(field): + raise MigrationIncompatibleError( + f"input field {field} mismatch after transform" + ) + # objective + if manifest.get("objective") != snap.get("objective"): + raise MigrationIncompatibleError("objective mismatch after transform") + # summary: compare semantic confirmation + digest-vs-canonical_sha256 + ms = manifest.get("summary") + ss = snap.get("summary") + if (ms is None) != (ss is None): + raise MigrationIncompatibleError("summary presence mismatch after transform") + if ms is not None and ss is not None: + if ms.get("digest") != ss.get("canonical_sha256"): + raise MigrationIncompatibleError("summary digest mismatch after transform") + if ms.get("size") != len((ss.get("content") or "").encode("utf-8")): + raise MigrationIncompatibleError("summary size mismatch after transform") + mc = ms.get("confirmation") or {} + sc = ss.get("confirmation") or {} + # Normalized timestamps (confirmed_at/presented_at) are excluded by + # ADR-0012 §9: current engine derives them from created_at=legacy + # event.at, not the legacy engine's distinct intermediate clocks. + for field in ("status", "operator_id", "confirmed_summary_sha256"): + if mc.get(field) != sc.get(field): + raise MigrationIncompatibleError( + f"summary confirmation {field} mismatch after transform" + ) + # artifacts + ma = manifest.get("artifacts", []) + sa = snap.get("artifacts", []) + if len(ma) != len(sa): + raise MigrationIncompatibleError("artifacts count mismatch after transform") + for a, b in zip(ma, sa): + for field in ("artifact_id", "kind", "logical_path", "sha256", + "byte_count", "status"): + if a.get(field) != b.get(field): + raise MigrationIncompatibleError( + f"artifact field {field} mismatch after transform" + ) + # transition IDs (preserved historical). Rev-0 has {None, None} by + # contract in BOTH eras (no prior action); rev>0 must match the legacy + # event's preserved IDs. + mt = manifest.get("transition") or {} + st = snap.get("transition") or {} + if ev.revision == 0: + if mt.get("last_event_id") is not None or mt.get("last_action_id") is not None: + raise MigrationIncompatibleError( + "revision 0 transition must be null in the modern manifest" + ) + else: + if mt.get("last_event_id") != ev.event_id: + raise MigrationIncompatibleError("transition last_event_id mismatch") + if mt.get("last_action_id") != ev.action_id: + raise MigrationIncompatibleError("transition last_action_id mismatch") + if st.get("last_event_id") != ev.event_id: + raise MigrationIncompatibleError( + "legacy snapshot transition last_event_id inconsistent" + ) + + +def _validate_temp_store(temp_root: Path, artifacts: ArtifactStore) -> None: + conn = open_database(temp_root, read_only=False) + try: + # integrity check + row = conn.execute("PRAGMA integrity_check").fetchone() + if row is None or row[0] != "ok": + raise StorageError("temporary SQLite integrity_check failed") + counts = conn.execute( + "SELECT COUNT(*), COUNT(DISTINCT package_id) FROM events" + ).fetchone() + finally: + close_database(conn) + # authoritative full chain validation via a store wrapper + from ..storage.store import SqliteManifestStore + + store = SqliteManifestStore(temp_root, artifact_store=artifacts) + try: + for package_id in sorted( + r[0] for r in store._conn.execute( + "SELECT DISTINCT package_id FROM events" + ).fetchall() + ): + try: + store.validate_chain(package_id, verify_artifacts=True) + except ChainViolationError as exc: + # Current-manifest rejection of a migrated legacy value is + # MIGRATION_INCOMPATIBLE (ADR-0012 §7), never a raw chain + # error from the validator. + raise MigrationIncompatibleError( + f"migrated chain for {package_id} is current-invalid: {exc}" + ) from exc + finally: + store.close() + + +def _verify_final(final_dest: Path) -> None: + """Read-only verification of the FINAL DB FILE (not a store root). + + Opens the exact file path read-only; never creates or mutates. + """ + import sqlite3 + from urllib.parse import quote + + db = final_dest.resolve() + uri = f"file:{quote(str(db), safe='/')}?mode=ro" + conn = None + try: + conn = sqlite3.connect(uri, uri=True, timeout=5.0) + row = conn.execute("PRAGMA integrity_check").fetchone() + if row is None or row[0] != "ok": + raise MigrationPublishFailedError( + "final database failed read-only integrity_check" + ) + app = int(conn.execute("PRAGMA application_id").fetchone()[0]) + ver = int(conn.execute("PRAGMA user_version").fetchone()[0]) + if app != APPLICATION_ID or ver != USER_VERSION: + raise MigrationPublishFailedError( + "final database identity mismatch after publication" + ) + except sqlite3.Error as exc: + raise MigrationPublishFailedError( + f"final database read-only verification failed: {exc}" + ) from exc + finally: + if conn is not None: + conn.close() + + +def _build_receipt(source: LegacySource, before: dict, temp_db: Path) -> dict: + src_event_count = sum(len(p.events) for p in source.packages.values()) + dst_event_count = src_event_count # 1:1 revision mapping + return { + "receipt_format": RECEIPT_FORMAT, + "receipt_version": RECEIPT_VERSION, + "legacy_source_format": "v0.1.2-integrity", + "legacy_source_commit": LEGACY_COMMIT, + "legacy_source_tag": LEGACY_TAG, + "semantic_source_inventory": before, + "source_package_count": len(source.packages), + "source_event_count": src_event_count, + "destination_schema_application_id": APPLICATION_ID, + "destination_schema_version": USER_VERSION, + "destination_package_count": len(source.packages), + "destination_event_count": dst_event_count, + "migration_implementation": "methodfactory.migrations", + "validation_verdict": "PASS", + } + + +def _write_durable(path: Path, data: dict) -> None: + tmp = path.with_name(path.name + ".tmp") + with open(tmp, "w", encoding="utf-8") as fh: + fh.write(json.dumps(data, sort_keys=True, separators=(",", ":"), + ensure_ascii=False)) + fh.flush() + os.fsync(fh.fileno()) + os.replace(tmp, path) + + +def _atomic_replace(src: Path, dst: Path) -> None: + os.replace(src, dst) + + +def _fsync_dir(path: Path) -> None: + dir_fd = os.open(path, os.O_RDONLY) + try: + os.fsync(dir_fd) + finally: + os.close(dir_fd) diff --git a/methodfactory/migrations/v012_jsonl.py b/methodfactory/migrations/v012_jsonl.py new file mode 100644 index 0000000..6075614 --- /dev/null +++ b/methodfactory/migrations/v012_jsonl.py @@ -0,0 +1,397 @@ +"""Frozen public v0.1.2 legacy reader for migration (ADR-0012 amendment). + +Deliberately frozen compatibility component for exact public: + + fb5641cc1a3f1f54b96bba3af88ec5a1b010f4e5 (tag v0.1.2-integrity) + +This module reproduces ONLY the public validation semantics required to +prove a legacy source and expose immutable normalized records to the +migration layer. It is NOT a storage engine: + +- read-only; never repairs, truncates, reconciles, or rewrites; +- never acquires locks, never performs CAS, tail repair, tail classification, + stale-lock recovery, or PID ownership; +- never mutates artifacts or source files; +- fails closed on ambiguous or unsafe evidence. + +Canonical source of history: events/.events.jsonl +Latest-manifest cache: packages/.json +Artifact blobs: artifacts/blobs/ + +Cache semantics (frozen ADR): absent cache is acceptable where the journal is +reconstructable; a lagging cache is acceptable if its digest matches ANY +committed journal snapshot; a cache matching no committed snapshot fails; +the journal always owns history. + +Legacy `.lock` files (events/.lock) are transient coordination +state. If present, migration must refuse to start (the caller raises the +existing CONCURRENCY error) — the reader never deletes, repairs, inspects PID, +or acquires them, and never includes them in semantic source identity. + +Legacy serializers (public v0.1.2): +- hash canonicalization: json.dumps(value, sort_keys=True, + separators=(",",":"), ensure_ascii=True).encode("utf-8") +- journal-line serialization: json.dumps(event, sort_keys=True) + "\\n" + (Python default spacing and ASCII escaping) +""" + +from __future__ import annotations + +import hashlib +import json +import re +from pathlib import Path +from typing import Any + +from ..storage.errors import ( + LegacyChainInvalidError, + LegacySourceInvalidError, +) + +# Exact public v0.1.2 identity (ADR-0012 amendment §1). +LEGACY_COMMIT = "fb5641cc1a3f1f54b96bba3af88ec5a1b010f4e5" +LEGACY_TAG = "v0.1.2-integrity" +LEGACY_SCHEMA_VERSION = "0.1" +LEGACY_ACTION_CREATE = "act_create_package" + +PACKAGE_ID_RE = re.compile(r"^pkg_[A-Za-z0-9_-]{1,63}$") +SHA256_RE = re.compile(r"^[0-9a-f]{64}$") + + +def legacy_canonical_json(value: Any) -> bytes: + """Public v0.1.2 HASH canonicalization (ensure_ascii=True, compact).""" + return json.dumps( + value, sort_keys=True, separators=(",", ":"), ensure_ascii=True + ).encode("utf-8") + + +def legacy_digest_json(value: Any) -> str: + return hashlib.sha256(legacy_canonical_json(value)).hexdigest() + + +def legacy_line_json(event: dict) -> str: + """Public v0.1.2 JOURNAL-LINE serialization (default spacing, ASCII).""" + return json.dumps(event, sort_keys=True) + + +def legacy_digest_bytes(content: bytes) -> str: + return hashlib.sha256(content).hexdigest() + + +def legacy_digest_text(content: str) -> str: + return legacy_digest_bytes(content.encode("utf-8")) + + +class LegacyEvent: + """Immutable normalized record from a public v0.1.2 journal line.""" + + __slots__ = ( + "package_id", "event_id", "action", "action_id", "revision", + "state_before", "state_after", "resulting_manifest_sha256", + "previous_manifest_sha256", "action_sha256", "at", "manifest_snapshot", + "line_number", + ) + + def __init__(self, **kw: Any) -> None: + for k in self.__slots__: + setattr(self, k, kw.get(k)) + + +class LegacyPackage: + """A validated public v0.1.2 package: ordered events + cache status.""" + + def __init__(self, package_id: str, events: list[LegacyEvent]) -> None: + self.package_id = package_id + self.events = events + self.cache_manifest: dict | None = None # parsed packages/.json + self.cache_status: str | None = None # "absent" | "valid" | "invalid" + + +class LegacySource: + """Frozen reader over one public v0.1.2 store root. + + Exposes validated packages in deterministic order (sorted package IDs) + plus the semantic source inventory for source-immutability proofs. + """ + + def __init__(self, root: Path | str) -> None: + self.root = Path(root) + self.packages_dir = self.root / "packages" + self.events_dir = self.root / "events" + self.artifacts_dir = self.root / "artifacts" + self._validate_layout() + self.packages: dict[str, LegacyPackage] = {} + self._read_packages() + + # ── layout / safety ──────────────────────────────────────────────── + def _validate_layout(self) -> None: + """Require the exact public v0.1.2 layout and safe paths.""" + for d in (self.packages_dir, self.events_dir, self.artifacts_dir): + if not d.is_dir(): + raise LegacySourceInvalidError( + f"legacy store missing required directory {d}" + ) + if d.is_symlink(): + raise LegacySourceInvalidError( + f"legacy store directory must not be a symlink: {d}" + ) + # Source root itself must not be a symlink (avoid escaping). + if self.root.is_symlink(): + raise LegacySourceInvalidError( + f"legacy store root must not be a symlink: {self.root}" + ) + + def _check_within(self, p: Path) -> None: + """Reject paths that escape the explicit source root.""" + try: + resolved = p.resolve() + except OSError as exc: # pragma: no cover - defensive + raise LegacySourceInvalidError(f"cannot resolve {p}: {exc}") from exc + root_resolved = self.root.resolve() + if root_resolved not in resolved.parents and resolved != root_resolved: + raise LegacySourceInvalidError( + f"legacy source path escapes store root: {p}" + ) + + # ── package discovery ────────────────────────────────────────────── + def _read_packages(self) -> None: + for events_path in sorted(self.events_dir.glob("*.events.jsonl")): + # Filename grammar: .events.jsonl + name = events_path.name + if not name.endswith(".events.jsonl"): + continue + package_id = name[: -len(".events.jsonl")] + if not PACKAGE_ID_RE.match(package_id): + raise LegacySourceInvalidError( + f"legacy event filename has invalid package_id: {name}" + ) + self._check_within(events_path) + if events_path.is_symlink(): + raise LegacySourceInvalidError( + f"legacy event file must not be a symlink: {events_path}" + ) + events = self._read_journal(package_id, events_path) + pkg = LegacyPackage(package_id, events) + self._read_cache(pkg) + self.packages[package_id] = pkg + + def _read_journal(self, package_id: str, path: Path) -> list[LegacyEvent]: + try: + text = path.read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError) as exc: + raise LegacySourceInvalidError( + f"cannot read legacy journal {path}: {exc}" + ) from exc + events: list[LegacyEvent] = [] + for i, line in enumerate(text.splitlines(), start=1): + if not line.strip(): + continue # tolerate blank lines (public reader ignored them) + try: + obj = json.loads(line) + except json.JSONDecodeError as exc: + raise LegacyChainInvalidError( + f"legacy journal {package_id} line {i} is not valid JSON: {exc}" + ) from exc + if not isinstance(obj, dict): + raise LegacyChainInvalidError( + f"legacy journal {package_id} line {i} is not a JSON object" + ) + event = self._normalize_event(package_id, obj, i) + events.append(event) + return events + + def _normalize_event(self, package_id: str, obj: dict, line: int) -> LegacyEvent: + required = ( + "event_id", "action", "action_id", "revision", "state_before", + "state_after", "resulting_manifest_sha256", + "previous_manifest_sha256", "action_sha256", "at", + "manifest_snapshot", + ) + missing = [f for f in required if f not in obj] + if missing: + raise LegacyChainInvalidError( + f"legacy journal {package_id} line {line} missing fields: {missing}" + ) + snapshot = obj["manifest_snapshot"] + if not isinstance(snapshot, dict): + raise LegacyChainInvalidError( + f"legacy journal {package_id} line {line} manifest_snapshot not object" + ) + ev = LegacyEvent( + package_id=package_id, + event_id=obj["event_id"], + action=obj["action"], + action_id=obj["action_id"], + revision=obj["revision"], + state_before=obj["state_before"], + state_after=obj["state_after"], + resulting_manifest_sha256=obj["resulting_manifest_sha256"], + previous_manifest_sha256=obj["previous_manifest_sha256"], + action_sha256=obj["action_sha256"], + at=obj["at"], + manifest_snapshot=snapshot, + line_number=line, + ) + return ev + + # ── cache handling ───────────────────────────────────────────────── + def _cache_path(self, package_id: str) -> Path: + return self.packages_dir / f"{package_id}.json" + + def _read_cache(self, pkg: LegacyPackage) -> None: + path = self._cache_path(pkg.package_id) + if not path.exists(): + pkg.cache_status = "absent" + return + self._check_within(path) + if path.is_symlink(): + raise LegacySourceInvalidError( + f"legacy cache file must not be a symlink: {path}" + ) + try: + data = json.loads(path.read_text(encoding="utf-8")) + except (OSError, UnicodeDecodeError, json.JSONDecodeError) as exc: + raise LegacySourceInvalidError( + f"legacy cache corrupt for {pkg.package_id}: {exc}" + ) from exc + if not isinstance(data, dict): + raise LegacySourceInvalidError( + f"legacy cache not an object for {pkg.package_id}" + ) + pkg.cache_manifest = data + # Cache is valid iff its digest matches ANY committed journal snapshot. + known = { + e.resulting_manifest_sha256 + for e in pkg.events + if isinstance(e.resulting_manifest_sha256, str) + } + digest = legacy_digest_json(data) + if digest in known: + pkg.cache_status = "valid" + else: + pkg.cache_status = "invalid" + + # ── artifact helpers ─────────────────────────────────────────────── + def artifact_bytes(self, digest: str) -> bytes: + """Read + verify a legacy blob (never follows symlinks out of root).""" + if not isinstance(digest, str) or not SHA256_RE.match(digest): + raise LegacySourceInvalidError(f"invalid artifact digest {digest!r}") + path = self.artifacts_dir / "blobs" / digest + self._check_within(path) + if path.is_symlink(): + raise LegacySourceInvalidError( + f"legacy artifact blob must not be a symlink: {path}" + ) + try: + data = path.read_bytes() + except OSError as exc: + raise LegacySourceInvalidError( + f"cannot read legacy artifact blob {digest}: {exc}" + ) from exc + if legacy_digest_bytes(data) != digest: + raise LegacyChainInvalidError( + f"legacy artifact blob {digest} content does not match digest" + ) + return data + + # ── validation ───────────────────────────────────────────────────── + def validate(self) -> None: + """Validate every package chain using public v0.1.2 semantics. + + Raises LegacyChainInvalidError on the first violation. Source remains + byte-identical (read-only). + """ + if not self.packages: + raise LegacySourceInvalidError( + "legacy store contains no event journals" + ) + for pkg in self.packages.values(): + self._validate_package(pkg) + + def _validate_package(self, pkg: LegacyPackage) -> None: + previous: dict | None = None + previous_digest: str | None = None + for index, ev in enumerate(pkg.events): + if ev.revision != index: + raise LegacyChainInvalidError( + f"legacy chain break for {pkg.package_id} at event index " + f"{index}: revision {ev.revision!r}" + ) + if ev.state_before != (None if previous is None else previous["state"]): + raise LegacyChainInvalidError( + f"legacy chain break for {pkg.package_id} at event index " + f"{index}: state_before does not match prior state_after" + ) + snap = ev.manifest_snapshot + if snap.get("state") != ev.state_after: + raise LegacyChainInvalidError( + f"legacy chain break for {pkg.package_id} at event index " + f"{index}: state_after does not match manifest_snapshot.state" + ) + if snap.get("revision") != ev.revision: + raise LegacyChainInvalidError( + f"legacy chain break for {pkg.package_id} at event index " + f"{index}: snapshot revision mismatch" + ) + expected_prev = None if previous is None else previous_digest + if snap.get("previous_manifest_sha256") != expected_prev: + raise LegacyChainInvalidError( + f"legacy chain break for {pkg.package_id} at event index " + f"{index}: previous_manifest_sha256 chain break" + ) + digest = legacy_digest_json(snap) + if ev.resulting_manifest_sha256 != digest: + raise LegacyChainInvalidError( + f"legacy chain break for {pkg.package_id} at event index " + f"{index}: resulting_manifest_sha256 does not match snapshot" + ) + # Validate referenced blobs exist + match digest (public semantics). + for item in snap.get("inputs", []): + d = item.get("content_sha256") + if d: + self.artifact_bytes(d) + for art in snap.get("artifacts", []): + d = art.get("sha256") + if d: + self.artifact_bytes(d) + previous = {"state": snap["state"], "digest": digest} + previous_digest = digest + if pkg.cache_status == "invalid": + raise LegacyChainInvalidError( + f"legacy cache for {pkg.package_id} matches no committed " + "journal snapshot" + ) + + # ── semantic source inventory (immutability proof) ───────────────── + def source_inventory(self) -> dict[str, dict[str, Any]]: + """Deterministic inventory over the frozen semantic source set. + + Excludes `.lock` files (transient coordination state). Keys are + relative paths; values are {sha256, size, type}. + """ + inventory: dict[str, dict[str, Any]] = {} + for pattern, base in ( + ("*.events.jsonl", self.events_dir), + ("*.json", self.packages_dir), + ): + for p in sorted(base.glob(pattern)): + if p.is_symlink(): + continue # already rejected during read + rel = str(p.relative_to(self.root)) + inventory[rel] = self._file_entry(p) + blobs = self.artifacts_dir / "blobs" + if blobs.is_dir(): + for p in sorted(blobs.iterdir()): + if p.is_file() and not p.is_symlink(): + rel = str(p.relative_to(self.root)) + inventory[rel] = self._file_entry(p) + return inventory + + @staticmethod + def _file_entry(p: Path) -> dict[str, Any]: + data = p.read_bytes() + return { + "sha256": legacy_digest_bytes(data), + "size": len(data), + "type": "file", + } diff --git a/methodfactory/storage/__init__.py b/methodfactory/storage/__init__.py index 0f612b5..bbebd0a 100644 --- a/methodfactory/storage/__init__.py +++ b/methodfactory/storage/__init__.py @@ -1,8 +1,9 @@ """Storage layer — protocol, canonical primitives, limits, and SQLite schema. -Phase 2 (ADR-0012 commits 2–4). The transactional store, migration, and export -are implemented in later commits; this package carries the storage-independent -contracts and the SQLite schema creation/identity/append-only guards. +Phase 2 (ADR-0012 commits 2–4) established the storage-independent contracts +and SQLite schema creation/identity/append-only guards. The transactional +store, deterministic migration, and exports are implemented in +`storage.store`, `migrations.migrate`, and `migrations.export`. """ from .errors import ( @@ -13,13 +14,19 @@ DatabaseEmptyError, DatabaseIdMismatchError, DatabaseNotFoundError, + DestinationExistsError, InvalidPackageIdError, InvalidStoreRootError, + LegacyChainInvalidError, + LegacySourceInvalidError, LegacyStoreDetectedError, ManifestInvalidError, + MigrationIncompatibleError, + MigrationPublishFailedError, PackageNotFoundError, SchemaViolationError, SerializationError, + SourceChangedError, StorageError, UnsupportedSchemaError, ) @@ -59,8 +66,11 @@ "DatabaseEmptyError", "DatabaseIdMismatchError", "DatabaseNotFoundError", + "DestinationExistsError", "InvalidPackageIdError", "InvalidStoreRootError", + "LegacyChainInvalidError", + "LegacySourceInvalidError", "LegacyStoreDetectedError", "ManifestInvalidError", "ManifestStore", @@ -76,9 +86,12 @@ "MAX_REASON_CHARS", "MAX_STATEMENT_CHARS", "MethodFactoryError", + "MigrationIncompatibleError", + "MigrationPublishFailedError", "PackageNotFoundError", "SchemaViolationError", "SerializationError", + "SourceChangedError", "SqliteManifestStore", "StorageError", "UnsupportedSchemaError", diff --git a/methodfactory/storage/errors.py b/methodfactory/storage/errors.py index 236e568..166620e 100644 --- a/methodfactory/storage/errors.py +++ b/methodfactory/storage/errors.py @@ -142,3 +142,40 @@ class ChainViolationError(StorageError): or referenced-artifact integrity).""" code = "CHAIN_VIOLATION" + + +class LegacySourceInvalidError(StorageError): + """The legacy v0.1.2 source is unrecognized or unsupported (migration).""" + + code = "LEGACY_SOURCE_INVALID" + + +class LegacyChainInvalidError(StorageError): + """The legacy v0.1.2 chain is invalid (migration fails closed).""" + + code = "LEGACY_CHAIN_INVALID" + + +class MigrationIncompatibleError(StorageError): + """A legacy semantic action cannot be reconstructed or a public-valid value + is now current-invalid (migration fails closed; no weakening).""" + + code = "MIGRATION_INCOMPATIBLE" + + +class SourceChangedError(StorageError): + """The legacy source changed during migration; publication is refused.""" + + code = "SOURCE_CHANGED" + + +class MigrationPublishFailedError(StorageError): + """Atomic migration publication failed (rename/fsync/receipt).""" + + code = "MIGRATION_PUBLISH_FAILED" + + +class DestinationExistsError(StorageError): + """The migration destination already exists; no overwrite is performed.""" + + code = "DESTINATION_EXISTS" From 34ba9d1c5ac9a5164ae6cecb39ac6040a3e36ab5 Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Sat, 8 Aug 2026 05:10:17 +0700 Subject: [PATCH 27/41] test(migration): fb5641c-origin fixtures + 44 migration/export tests; docs update - tests/_fixtures.py: fixture generation via exact public fb5641c code (disposable worktree); standard/unicode/optional/revise/all-families/ cancel-arbitrary/over-limit/current-invalid workflows. - tests/test_migrations.py: positive migration, source preservation, semantic reconstruction, action families, optional candidates, fail-closed matrix, cache/lock semantics, fault injection, receipt identity, deterministic exports, serializer distinction, no-import/ no-8a surfaces, CLI boundary. - docs: public-surface migration/export table + error taxonomy; architecture-reset-status migration gate entry. Full suite: 400 tests green (356 baseline + 44 new). --- docs/architecture-reset-status.md | 42 +- docs/public-surface.md | 28 + methodfactory/tests/_fixtures.py | 545 ++++++++++++++++ methodfactory/tests/test_migrations.py | 870 +++++++++++++++++++++++++ 4 files changed, 1479 insertions(+), 6 deletions(-) create mode 100644 methodfactory/tests/_fixtures.py create mode 100644 methodfactory/tests/test_migrations.py diff --git a/docs/architecture-reset-status.md b/docs/architecture-reset-status.md index 1c1a504..15dfb75 100644 --- a/docs/architecture-reset-status.md +++ b/docs/architecture-reset-status.md @@ -1,6 +1,6 @@ -# Architecture Reset — Project State (2026-08-07, Phase 2) +# Architecture Reset — Project State (2026-08-08, Phase 3: migration/export) -**Status:** SQLite architecture approved in principle; senior review `4878235332` on PR #1 completed the architecture review and directed the Phase 2 implementation order (commits 1–4) with a design-convergence stop gate. This document tracks the clean `feat/sqlite-persistence-reset` branch. +**Status:** SQLite architecture approved in principle; senior review `4878235332` on PR #1 completed the architecture review and directed the Phase 2 implementation order (commits 1–4) with a design-convergence stop gate. The Phase 2 stop gate was accepted, and the migration/export implementation gate (ADR-0012 amendment, frozen at `42ff7d9` + `b9e46c1`) is now implemented and under mandatory review. This document tracks the clean `feat/sqlite-persistence-reset` branch. ## Branch topology @@ -13,21 +13,51 @@ fb5641c remote main (published v0.1.2-integrity base) ├── ADR-0012 + architecture contracts ← commits 1 (docs) — done ├── package foundation (rename, CI, ignores) ← commit 2 ├── storage protocol + canonical primitives ← commit 3 - ├── SQLite schema creation + identity + append-only guards ← commit 4 (Phase 2 stop gate) - └── (later, after gate) transactional apply, migration, exports, lifecycle + ├── SQLite schema creation + identity + append-only guards ← commit 4 (Phase 2 stop gate — accepted) + ├── transactional apply + deterministic replay + chain validator ← Phase 3 foundation + └── migration + deterministic exports + CLI (this gate) ← committed for senior review ``` -## Identities (verified 2026-08-07) +## Identities (verified 2026-08-08) | Item | Value | |---|---| | Remote main | `fb5641cc1a3f1f54b96bba3af88ec5a1b010f4e5` (untouched) | | Forensic branch | `review/jsonl-overhaul-8a7e916` = `8a7e9167d6ff77b3ccd32722683c9b42e4390687` | | Clean branch | `feat/sqlite-persistence-reset` (merge-base with origin/main = `fb5641c`; does **not** descend from 8a7e916) | -| PR #1 | https://github.com/RedEyeNinja-BKK/Method-Factory/pull/1 — **actual GitHub Draft** (reviewer converted it), DO NOT MERGE | +| PR #1 | https://github.com/RedEyeNinja-BKK/Method-Factory/pull/1 — **actual GitHub Draft**, DO NOT MERGE | | Git bundle | `method-factory-8a7e916.bundle` (SHA-256 `92c0bb1026190f9fd5e1f61bb4cd5fc16a08605aecf77f81eb5bc93b3b504f63`; `git bundle verify` OK) | | Local archival | `persistence-reset` branch preserved locally (pre-revision ADR draft, not published) | +## Migration/export implementation gate (2026-08-08) + +Bounded implementation authorized by Vincent at canonical head `b9e46c1` on +`feat/sqlite-persistence-reset`. Scope: public v0.1.2 → SQLite migration, +deterministic supported export, deterministic legacy-v0.1.2 evidence export, +and the minimal CLI/error/test/documentation surface for those capabilities. + +Implemented (pending senior review): + +- `methodfactory/migrations/v012_jsonl.py` — frozen read-only v0.1.2 reader + (exact `fb5641c` semantics; no CAS/lock/repair/append mechanics). +- `methodfactory/migrations/migrate.py` — atomic migration: legacy validation, + semantic-action reconstruction by legacy hash, current-engine transformation + (`next_manifest` only), equivalence verification, source-stability proof, + temp-DB build + validation, durable receipt + DB publication, final + read-only verification, fault seams. +- `methodfactory/migrations/export.py` — `method-factory-events-v1` and + `legacy-v012-jsonl` deterministic exports (read-only, consistent read). +- `methodfactory/cli.py` — bounded surface restored: `mf migrate-store` and + `mf export`; lifecycle commands remain unavailable; `mf --version` unchanged. +- Six new frozen migration error codes (see `docs/public-surface.md`). +- `methodfactory/tests/_fixtures.py` + `test_migrations.py` — fb5641c-origin + fixtures and 44 focused tests (full suite 400 tests green). + +Not authorized / NOT implemented in this gate: merge, PR-ready, tag, release, +`main`/forensic mutation, force-push, deployment, lifecycle expansion, +backup/restore, generic import, garbage collection, JSONL as canonical store, +or 8a7e916 repair/CAS/locking mechanics. + ## Senior review 4878235332 (2026-08-07) — accepted - SQLite reset remains **APPROVED IN PRINCIPLE**; corrected evidence package closes the prior evidence gap. diff --git a/docs/public-surface.md b/docs/public-surface.md index 852cb1c..aacc040 100644 --- a/docs/public-surface.md +++ b/docs/public-surface.md @@ -48,6 +48,34 @@ exception contract and are not part of the supported surface. | `SqliteManifestStore.validate_chain(package_id, *, verify_artifacts=False)` | str + bool | `{package_id, events, valid}` | `ChainViolationError` (`CHAIN_VIOLATION`), `StorageError` (`STORAGE_ERROR`) | `sqlite3.Error`, decode/JSON errors | | `SqliteManifestStore.explain_latest_plan(package_id)` | str | `list[tuple]` (query plan) | `StorageError` (`STORAGE_ERROR`) | `sqlite3.Error` | | `SqliteManifestStore.close()` | — | `None` | `StorageError` (`STORAGE_ERROR`) | `sqlite3.Error` | +| `migrate_store(source_root, dest=None)` | legacy store root str/Path + optional dest SQLite path | receipt `dict` (see ADR-0012 amendment §15) | `LegacySourceInvalidError` (`LEGACY_SOURCE_INVALID`), `LegacyChainInvalidError` (`LEGACY_CHAIN_INVALID`), `MigrationIncompatibleError` (`MIGRATION_INCOMPATIBLE`), `SourceChangedError` (`SOURCE_CHANGED`), `MigrationPublishFailedError` (`MIGRATION_PUBLISH_FAILED`), `DestinationExistsError` (`DESTINATION_EXISTS`), `ConcurrencyError` (`CONCURRENCY`, legacy `.lock` present), plus re-used public errors for current-boundary rejections | `OSError`, `sqlite3.Error`, `json.JSONDecodeError`, `UnicodeError`, `TypeError`/`ValueError`/`RecursionError` | +| `export_events(store_root, output, *, fmt=EVENTS_V1_FORMAT)` | store root + optional output path + format (`method-factory-events-v1` \| `legacy-v012-jsonl`) | event count `int` (events written to stdout or file) | `StorageError` (`STORAGE_ERROR`), `DatabaseNotFoundError` (`DATABASE_NOT_FOUND`), `LegacyStoreDetectedError` (`LEGACY_STORE_DETECTED`), `DatabaseEmptyError` (`DATABASE_EMPTY`), `DatabaseIdMismatchError` (`DATABASE_ID_MISMATCH`), `UnsupportedSchemaError` (`UNSUPPORTED_SCHEMA`) | `sqlite3.Error`, `OSError`, `UnicodeDecodeError`, `json.JSONDecodeError` | +| `LegacySource(root)` | str/Path legacy v0.1.2 store root | validated read-only source object | `LegacySourceInvalidError` (`LEGACY_SOURCE_INVALID`), `LegacyChainInvalidError` (`LEGACY_CHAIN_INVALID`) | `OSError`, `json.JSONDecodeError`, `UnicodeDecodeError` | + +> **Migration boundary rule (ADR-0012 amendment §17):** every public +> migration failure stays within the Method Factory typed error boundary. +> Legacy-valid values that the CURRENT public boundary rejects +> (identifier grammar, logical-path grammar, intent/input/artifact/objective/ +> reason limits, control characters, duplicate event IDs, unrecoverable +> `cancel.reason`) surface as `MIGRATION_INCOMPATIBLE` — never as raw +> envelope/engine/validator/sqlite errors. The frozen legacy reader +> (`migrations.v012_jsonl`) is read-only: it never repairs, truncates, +> reconciles, rewrites, acquires locks, or performs CAS/tail-repair. + +## Migration error codes (frozen, ADR-0012 amendment §17) + +| Code | Meaning | +|---|---| +| `LEGACY_SOURCE_INVALID` | recognized v0.1.2 layout missing/unsafe (symlink escape, missing dir, corrupt cache, invalid filename grammar) | +| `LEGACY_CHAIN_INVALID` | legacy history violates public v0.1.2 validation (revision sequence, state continuity, hash chain, snapshot digest, blob digest, cache mismatch) | +| `MIGRATION_INCOMPATIBLE` | legacy-valid value is not reconstructable or fails the current public boundary (see rule above) | +| `SOURCE_CHANGED` | semantic source identity changed between the BEFORE and AFTER inventory; nothing published | +| `MIGRATION_PUBLISH_FAILED` | durable publication (receipt/DB/fsync/final verification) failed | +| `DESTINATION_EXISTS` | final destination already exists; migration refuses to overwrite | + +> Legacy `.lock` presence reuses the existing `CONCURRENCY` semantics: the +> migration refuses to start, reports the lock path, and never deletes, +> repairs, inspects PID, acquires, or includes it in source identity. ## Internal primitives (documented native contract, NOT public) diff --git a/methodfactory/tests/_fixtures.py b/methodfactory/tests/_fixtures.py new file mode 100644 index 0000000..24fe216 --- /dev/null +++ b/methodfactory/tests/_fixtures.py @@ -0,0 +1,545 @@ +"""Migration test fixtures originating from exact public v0.1.2 (fb5641c). + +The canonical positive fixtures are generated by running the ACTUAL public +v0.1.2 code from a disposable checkout of `fb5641c`. This module locates or +builds such a checkout on demand and generates fixtures deterministically. + +Fixture provenance is recorded (generating commit, command, source inventory). +""" + +from __future__ import annotations + +import datetime +import json +import shutil +import subprocess +import sys +from pathlib import Path +from typing import Callable + +LEGACY_COMMIT = "fb5641cc1a3f1f54b96bba3af88ec5a1b010f4e5" +LEGACY_TAG = "v0.1.2-integrity" + +# Location of the canonical dev checkout (used to locate the fb5641c commit). +REPO_ROOT = Path(__file__).resolve().parents[2] + + +def _workflow_source(workflow: Callable) -> str: + """Return the exact source of a workflow function for embedding.""" + import inspect + + return inspect.getsource(workflow) + + +class AdvClock: + """Deterministic advancing clock (distinct timestamp per call).""" + + def __init__(self, start: str = "2026-08-07T00:00:00+00:00") -> None: + self.t = datetime.datetime.fromisoformat(start) + self._delta = datetime.timedelta(minutes=1) + + def __call__(self) -> str: + s = self.t.isoformat() + self.t += self._delta + return s + + +def _find_fb5641c() -> Path | None: + """Return a path to a worktree at fb5641c, or None if unavailable.""" + try: + result = subprocess.run( + ["git", "-C", str(REPO_ROOT), "cat-file", "-t", LEGACY_COMMIT], + capture_output=True, + text=True, + ) + if result.returncode != 0: + return None + except Exception: # pragma: no cover + return None + return REPO_ROOT + + +def build_legacy_checkout(dest: Path) -> Path | None: + """Create a disposable worktree of fb5641c. Returns the checkout root.""" + repo = _find_fb5641c() + if repo is None: + return None + # If the path is already a registered worktree, remove it cleanly first. + subprocess.run( + ["git", "-C", str(repo), "worktree", "remove", "--force", str(dest)], + capture_output=True, + text=True, + ) + if dest.exists(): + shutil.rmtree(dest, ignore_errors=True) + result = subprocess.run( + ["git", "-C", str(repo), "worktree", "add", "--detach", str(dest), LEGACY_COMMIT], + capture_output=True, + text=True, + ) + if result.returncode != 0: + return None + return dest + + +def generate_fixture( + root: Path, + *, + workflow: Callable, + clock: Callable[[], str] | None = None, +) -> Path: + """Generate a v0.1.2 fixture at `root` using the public code. + + `workflow(engine, apply, root)` drives the scenario; returns nothing. + """ + checkout = Path("/tmp/mf-v012-fixturegen") + if not build_legacy_checkout(checkout): + raise RuntimeError( + "cannot build fb5641c checkout; fixture generation requires the " + "exact public code" + ) + + code = f""" +import json, sys +sys.path.insert(0, {str(checkout)!r}) +from pathlib import Path +from core.manifest.store import ManifestStore +from core.adapters.artifact_store import ArtifactStore +from core.engine import PipelineEngine +from core.manifest.hashing import utcnow + +root = Path({str(root)!r}) +import shutil; shutil.rmtree(root, ignore_errors=True) +store = ManifestStore(root) +engine = PipelineEngine(store, ArtifactStore(root / "artifacts")) + +# workflow source (exact public-era code running against fb5641c) +{_workflow_source(workflow)} + +{workflow.__name__}(engine, root) +""" + # We embed the workflow by exec'ing its source; simpler: pass a script path. + script = Path("/tmp/mf-v012-gen-script.py") + script.write_text(code, encoding="utf-8") + py = sys.executable + subprocess.run([py, str(script)], check=True, capture_output=True) + return root + + +def standard_workflow(engine, root: Path) -> None: + """Full valid v0.1.2 workflow: input -> objective -> summary -> confirm + -> artifact -> cancel.""" + import datetime + + class Clock: + def __init__(self, start: str = "2026-08-07T00:00:00+00:00") -> None: + self.t = datetime.datetime.fromisoformat(start) + self._delta = datetime.timedelta(minutes=1) + + def __call__(self) -> str: + s = self.t.isoformat() + self.t += self._delta + return s + + clock = Clock() + engine._now = clock # override the engine clock deterministically + + def apply(envdict): + return engine.apply_json(json.dumps(envdict)) + + engine.create_package("pkg_demo_001", "Build a skill") + apply({ + "protocol_version": "0.1", "action_id": "act_in_1", + "package_id": "pkg_demo_001", "expected_revision": 0, + "action": "record_input", "basis": {}, + "payload": {"input_id": "in_1", "kind": "text", "content": "hello", + "source": "operator", "disposition": "incorporated"}, + }) + apply({ + "protocol_version": "0.1", "action_id": "act_obj_1", + "package_id": "pkg_demo_001", "expected_revision": 1, + "action": "set_objective", "basis": {}, + "payload": {"statement": "Build a skill", "desired_outcomes": ["ship it"]}, + }) + m2 = apply({ + "protocol_version": "0.1", "action_id": "act_prep_1", + "package_id": "pkg_demo_001", "expected_revision": 2, + "action": "prepare_summary", "basis": {}, "payload": {}, + }) + apply({ + "protocol_version": "0.1", "action_id": "act_conf_1", + "package_id": "pkg_demo_001", "expected_revision": 3, + "action": "confirm_summary", + "basis": {"summary_sha256": m2.manifest["summary"]["canonical_sha256"]}, + "payload": {"operator_id": "vincent"}, + }) + apply({ + "protocol_version": "0.1", "action_id": "act_art_1", + "package_id": "pkg_demo_001", "expected_revision": 4, + "action": "record_draft_artifact", "basis": {}, + "payload": {"artifact_id": "art_1", "kind": "skill", + "logical_path": "skills/x/SKILL.md", "content": "body"}, + }) + apply({ + "protocol_version": "0.1", "action_id": "act_cancel_1", + "package_id": "pkg_demo_001", "expected_revision": 5, + "action": "cancel", "basis": {}, "payload": {}, + }) + + +def non_ascii_workflow(engine, root: Path) -> None: + """Workflow with non-ASCII semantic content (ensure_ascii divergence).""" + import datetime + + class Clock: + def __init__(self, start: str = "2026-08-07T00:00:00+00:00") -> None: + self.t = datetime.datetime.fromisoformat(start) + self._delta = datetime.timedelta(minutes=1) + + def __call__(self) -> str: + s = self.t.isoformat() + self.t += self._delta + return s + + engine._now = Clock() + + def apply(envdict): + return engine.apply_json(json.dumps(envdict)) + + engine.create_package("pkg_unicode_001", "สร้างทักษะ – 日本語で") + apply({ + "protocol_version": "0.1", "action_id": "act_in_1", + "package_id": "pkg_unicode_001", "expected_revision": 0, + "action": "record_input", "basis": {}, + "payload": {"input_id": "in_1", "kind": "text", "content": "สวัสดี こんにちは", + "source": "operator", "disposition": "incorporated"}, + }) + apply({ + "protocol_version": "0.1", "action_id": "act_obj_1", + "package_id": "pkg_unicode_001", "expected_revision": 1, + "action": "set_objective", "basis": {}, + "payload": {"statement": "เป้าหมาย 目標", "desired_outcomes": ["ส่งมอบ", "納品"]}, + }) + + +def optional_omitted_workflow(engine, root: Path) -> None: + """Workflow exercising omitted-vs-explicit optional fields: + record_input without exclusion_reason; set_objective without + desired_outcomes; confirm_summary with default operator.""" + import datetime + + class Clock: + def __init__(self, start: str = "2026-08-07T00:00:00+00:00") -> None: + self.t = datetime.datetime.fromisoformat(start) + self._delta = datetime.timedelta(minutes=1) + + def __call__(self) -> str: + s = self.t.isoformat() + self.t += self._delta + return s + + engine._now = Clock() + + def apply(envdict): + return engine.apply_json(json.dumps(envdict)) + + engine.create_package("pkg_opt_001", "optional fields") + # record_input WITHOUT exclusion_reason (omitted) + apply({ + "protocol_version": "0.1", "action_id": "act_in_1", + "package_id": "pkg_opt_001", "expected_revision": 0, + "action": "record_input", "basis": {}, + "payload": {"input_id": "in_1", "kind": "text", "content": "hello", + "source": "operator", "disposition": "incorporated"}, + }) + # set_objective WITHOUT desired_outcomes (omitted) + apply({ + "protocol_version": "0.1", "action_id": "act_obj_1", + "package_id": "pkg_opt_001", "expected_revision": 1, + "action": "set_objective", "basis": {}, + "payload": {"statement": "Only statement"}, + }) + m2 = apply({ + "protocol_version": "0.1", "action_id": "act_prep_1", + "package_id": "pkg_opt_001", "expected_revision": 2, + "action": "prepare_summary", "basis": {}, "payload": {}, + }) + # confirm_summary WITHOUT operator_id (default "operator") + apply({ + "protocol_version": "0.1", "action_id": "act_conf_1", + "package_id": "pkg_opt_001", "expected_revision": 3, + "action": "confirm_summary", + "basis": {"summary_sha256": m2.manifest["summary"]["canonical_sha256"]}, + "payload": {}, + }) + # cancel WITHOUT reason (omitted) + apply({ + "protocol_version": "0.1", "action_id": "act_cancel_1", + "package_id": "pkg_opt_001", "expected_revision": 4, + "action": "cancel", "basis": {}, "payload": {}, + }) + + +def cancel_arbitrary_reason_workflow(engine, root: Path) -> None: + """Cancel with an arbitrary non-empty reason (not reconstructable).""" + import datetime + + class Clock: + def __init__(self, start: str = "2026-08-07T00:00:00+00:00") -> None: + self.t = datetime.datetime.fromisoformat(start) + self._delta = datetime.timedelta(minutes=1) + + def __call__(self) -> str: + s = self.t.isoformat() + self.t += self._delta + return s + + engine._now = Clock() + + def apply(envdict): + return engine.apply_json(json.dumps(envdict)) + + engine.create_package("pkg_cancel_001", "cancel with reason") + apply({ + "protocol_version": "0.1", "action_id": "act_in_1", + "package_id": "pkg_cancel_001", "expected_revision": 0, + "action": "record_input", "basis": {}, + "payload": {"input_id": "in_1", "kind": "text", "content": "hello", + "source": "operator", "disposition": "incorporated"}, + }) + apply({ + "protocol_version": "0.1", "action_id": "act_cancel_1", + "package_id": "pkg_cancel_001", "expected_revision": 1, + "action": "cancel", "basis": {}, + "payload": {"reason": "operator decided to abandon this work"}, + }) + + +def revise_intake_workflow(engine, root: Path) -> None: + """Workflow exercising revise_intake (summary invalidated).""" + import datetime + + class Clock: + def __init__(self, start: str = "2026-08-07T00:00:00+00:00") -> None: + self.t = datetime.datetime.fromisoformat(start) + self._delta = datetime.timedelta(minutes=1) + + def __call__(self) -> str: + s = self.t.isoformat() + self.t += self._delta + return s + + engine._now = Clock() + + def apply(envdict): + return engine.apply_json(json.dumps(envdict)) + + engine.create_package("pkg_revise_001", "revise intake") + apply({ + "protocol_version": "0.1", "action_id": "act_in_1", + "package_id": "pkg_revise_001", "expected_revision": 0, + "action": "record_input", "basis": {}, + "payload": {"input_id": "in_1", "kind": "text", "content": "hello", + "source": "operator", "disposition": "incorporated"}, + }) + apply({ + "protocol_version": "0.1", "action_id": "act_obj_1", + "package_id": "pkg_revise_001", "expected_revision": 1, + "action": "set_objective", "basis": {}, + "payload": {"statement": "Build a skill"}, + }) + apply({ + "protocol_version": "0.1", "action_id": "act_prep_1", + "package_id": "pkg_revise_001", "expected_revision": 2, + "action": "prepare_summary", "basis": {}, "payload": {}, + }) + apply({ + "protocol_version": "0.1", "action_id": "act_revise_1", + "package_id": "pkg_revise_001", "expected_revision": 3, + "action": "revise_intake", "basis": {}, "payload": {}, + }) + + +def all_action_families_workflow(engine, root: Path) -> None: + """Every action family in one package: input, objective, summary, confirm, + revise, artifact, cancel.""" + import datetime + + class Clock: + def __init__(self, start: str = "2026-08-07T00:00:00+00:00") -> None: + self.t = datetime.datetime.fromisoformat(start) + self._delta = datetime.timedelta(minutes=1) + + def __call__(self) -> str: + s = self.t.isoformat() + self.t += self._delta + return s + + engine._now = Clock() + + def apply(envdict): + return engine.apply_json(json.dumps(envdict)) + + engine.create_package("pkg_all_001", "all action families") + apply({ + "protocol_version": "0.1", "action_id": "act_in_1", + "package_id": "pkg_all_001", "expected_revision": 0, + "action": "record_input", "basis": {}, + "payload": {"input_id": "in_1", "kind": "text", "content": "hello", + "source": "operator", "disposition": "incorporated", + "exclusion_reason": None}, + }) + apply({ + "protocol_version": "0.1", "action_id": "act_obj_1", + "package_id": "pkg_all_001", "expected_revision": 1, + "action": "set_objective", "basis": {}, + "payload": {"statement": "Build a skill", "desired_outcomes": ["ship it"]}, + }) + m2 = apply({ + "protocol_version": "0.1", "action_id": "act_prep_1", + "package_id": "pkg_all_001", "expected_revision": 2, + "action": "prepare_summary", "basis": {}, "payload": {}, + }) + apply({ + "protocol_version": "0.1", "action_id": "act_conf_1", + "package_id": "pkg_all_001", "expected_revision": 3, + "action": "confirm_summary", + "basis": {"summary_sha256": m2.manifest["summary"]["canonical_sha256"]}, + "payload": {"operator_id": "vincent"}, + }) + apply({ + "protocol_version": "0.1", "action_id": "act_revise_1", + "package_id": "pkg_all_001", "expected_revision": 4, + "action": "revise_intake", "basis": {}, "payload": {}, + }) + # After revise: intake again; record another input (exclusion_reason), + # objective, summary, confirm, then artifact (legal in AUTHORING_AUTHORIZED). + apply({ + "protocol_version": "0.1", "action_id": "act_in_2", + "package_id": "pkg_all_001", "expected_revision": 5, + "action": "record_input", "basis": {}, + "payload": {"input_id": "in_2", "kind": "text", "content": "world", + "source": "operator", "disposition": "excluded", + "exclusion_reason": "duplicate"}, + }) + apply({ + "protocol_version": "0.1", "action_id": "act_obj_2", + "package_id": "pkg_all_001", "expected_revision": 6, + "action": "set_objective", "basis": {}, + "payload": {"statement": "Build a skill", "desired_outcomes": ["ship it"]}, + }) + m3 = apply({ + "protocol_version": "0.1", "action_id": "act_prep_2", + "package_id": "pkg_all_001", "expected_revision": 7, + "action": "prepare_summary", "basis": {}, "payload": {}, + }) + apply({ + "protocol_version": "0.1", "action_id": "act_conf_2", + "package_id": "pkg_all_001", "expected_revision": 8, + "action": "confirm_summary", + "basis": {"summary_sha256": m3.manifest["summary"]["canonical_sha256"]}, + "payload": {"operator_id": "vincent"}, + }) + apply({ + "protocol_version": "0.1", "action_id": "act_art_1", + "package_id": "pkg_all_001", "expected_revision": 9, + "action": "record_draft_artifact", "basis": {}, + "payload": {"artifact_id": "art_1", "kind": "skill", + "logical_path": "skills/x/SKILL.md", "content": "body"}, + }) + apply({ + "protocol_version": "0.1", "action_id": "act_cancel_1", + "package_id": "pkg_all_001", "expected_revision": 10, + "action": "cancel", "basis": {}, "payload": {}, + }) + + +def overlimit_intent_workflow(engine, root: Path) -> None: + """Intent exceeding the CURRENT MAX_INTENT_CHARS (public-v0.1.2-valid, + current-invalid under the frozen compatibility matrix).""" + import datetime + + class Clock: + def __init__(self, start: str = "2026-08-07T00:00:00+00:00") -> None: + self.t = datetime.datetime.fromisoformat(start) + self._delta = datetime.timedelta(minutes=1) + + def __call__(self) -> str: + s = self.t.isoformat() + self.t += self._delta + return s + + engine._now = Clock() + + def apply(envdict): + return engine.apply_json(json.dumps(envdict)) + + # 70_000 chars: > current 65_536 intent limit; legacy had no intent limit. + big = "x" * 70_000 + engine.create_package("pkg_bigintent_001", big) + apply({ + "protocol_version": "0.1", "action_id": "act_in_1", + "package_id": "pkg_bigintent_001", "expected_revision": 0, + "action": "record_input", "basis": {}, + "payload": {"input_id": "in_1", "kind": "text", "content": "hello", + "source": "operator", "disposition": "incorporated"}, + }) + + +def overlimit_input_workflow(engine, root: Path) -> None: + """Input content exceeding the CURRENT MAX_CONTENT_CHARS (legacy-valid).""" + import datetime + + class Clock: + def __init__(self, start: str = "2026-08-07T00:00:00+00:00") -> None: + self.t = datetime.datetime.fromisoformat(start) + self._delta = datetime.timedelta(minutes=1) + + def __call__(self) -> str: + s = self.t.isoformat() + self.t += self._delta + return s + + engine._now = Clock() + + def apply(envdict): + return engine.apply_json(json.dumps(envdict)) + + engine.create_package("pkg_biginput_001", "big input") + big = "y" * (1_048_576 + 5) + apply({ + "protocol_version": "0.1", "action_id": "act_in_1", + "package_id": "pkg_biginput_001", "expected_revision": 0, + "action": "record_input", "basis": {}, + "payload": {"input_id": "in_1", "kind": "text", "content": big, + "source": "operator", "disposition": "incorporated"}, + }) + + +def current_invalid_identifier_workflow(engine, root: Path) -> None: + """input_id that is public-v0.1.2-valid but current-invalid (space).""" + import datetime + + class Clock: + def __init__(self, start: str = "2026-08-07T00:00:00+00:00") -> None: + self.t = datetime.datetime.fromisoformat(start) + self._delta = datetime.timedelta(minutes=1) + + def __call__(self) -> str: + s = self.t.isoformat() + self.t += self._delta + return s + + engine._now = Clock() + + def apply(envdict): + return engine.apply_json(json.dumps(envdict)) + + engine.create_package("pkg_badid_001", "bad identifier") + apply({ + "protocol_version": "0.1", "action_id": "act_in_1", + "package_id": "pkg_badid_001", "expected_revision": 0, + "action": "record_input", "basis": {}, + "payload": {"input_id": "in 1", "kind": "text", "content": "hello", + "source": "operator", "disposition": "incorporated"}, + }) diff --git a/methodfactory/tests/test_migrations.py b/methodfactory/tests/test_migrations.py new file mode 100644 index 0000000..cc6b955 --- /dev/null +++ b/methodfactory/tests/test_migrations.py @@ -0,0 +1,870 @@ +"""Migration + deterministic export tests (ADR-0012 amendment). + +Focused coverage required by the migration/export implementation gate: + +- positive migration end-to-end; +- source byte preservation; +- unique semantic reconstruction by legacy action hash; +- every action family; +- every optional-field candidate family; +- unrecoverable cancel.reason rejection; +- advancing-clock normalization; +- rev0 ID preservation; +- duplicate event-ID rejection; +- cache crash-tolerance semantics; +- `.lock` refusal without mutation; +- compatibility-matrix fail-closed cases; +- current-engine deterministic replay for migrated rows; +- summary body content-addressing; +- artifact integrity; +- source-changed detection; +- destination-exists fail closed; +- publication fault matrix; +- migration receipt identity/success predicate; +- deterministic method-factory-events-v1; +- deterministic legacy-v012-jsonl; +- exact legacy hash-vs-line serializer distinction; +- export consistent-read behavior / no mutation; +- no import surface; +- no 8a repair/lock/CAS mechanics; +- temp/runtime artifact hygiene. + +Fixtures originate from exact public fb5641c code (see _fixtures.py). +""" + +from __future__ import annotations + +import hashlib +import json +import os +import shutil +import sqlite3 +import subprocess +import sys +import tempfile +import unittest +from pathlib import Path + +from methodfactory.domain.errors import ConcurrencyError +from methodfactory.migrations.export import ( + EVENTS_V1_FORMAT, + LEGACY_JSONL_FORMAT, + export_events, +) +import methodfactory.migrations.migrate as migrate_module +from methodfactory.migrations.migrate import migrate_store +from methodfactory.migrations.v012_jsonl import ( + LegacySource, + legacy_canonical_json, + legacy_digest_json, + legacy_line_json, +) +from methodfactory.storage.errors import ( + DestinationExistsError, + LegacyChainInvalidError, + LegacySourceInvalidError, + MigrationIncompatibleError, + MigrationPublishFailedError, + SourceChangedError, + StorageError, +) +from methodfactory.storage.serialization import digest_bytes, sha256_hex + +from ._fixtures import ( + all_action_families_workflow, + cancel_arbitrary_reason_workflow, + current_invalid_identifier_workflow, + generate_fixture, + non_ascii_workflow, + optional_omitted_workflow, + overlimit_input_workflow, + overlimit_intent_workflow, + revise_intake_workflow, + standard_workflow, +) + +FIXTURE_DIR = Path(tempfile.gettempdir()) / "mf-migration-test-fixtures" + + +def _generate(name: str, workflow) -> Path: + """Generate (once per suite) a fb5641c-origin fixture into FIXTURE_DIR.""" + FIXTURE_DIR.mkdir(parents=True, exist_ok=True) + root = FIXTURE_DIR / name + if root.exists(): + shutil.rmtree(root, ignore_errors=True) + return Path(generate_fixture(str(root), workflow=workflow)) + + +def _inventory(root: Path) -> dict[str, tuple[str, int]]: + """Deterministic semantic-source inventory (bytes + size).""" + inv: dict[str, tuple[str, int]] = {} + for base, pat in ( + (root / "events", "*.events.jsonl"), + (root / "packages", "*.json"), + ): + if not base.is_dir(): + continue + for p in sorted(base.glob(pat)): + data = p.read_bytes() + inv[str(p.relative_to(root))] = (hashlib.sha256(data).hexdigest(), len(data)) + blobs = root / "artifacts" / "blobs" + if blobs.is_dir(): + for p in sorted(blobs.iterdir()): + data = p.read_bytes() + inv[str(p.relative_to(root))] = (hashlib.sha256(data).hexdigest(), len(data)) + return inv + + +def _open_db(path: Path) -> sqlite3.Connection: + conn = sqlite3.connect(str(path)) + conn.row_factory = sqlite3.Row + return conn + + +class MigrationPositiveTests(unittest.TestCase): + """End-to-end migration of a valid legacy store.""" + + @classmethod + def setUpClass(cls): + cls.src = _generate("standard", standard_workflow) + + def test_migrate_end_to_end(self): + with tempfile.TemporaryDirectory() as td: + dest = Path(td) / "methodfactory.sqlite3" + receipt = migrate_store(self.src, dest) + self.assertEqual(receipt["source_package_count"], 1) + self.assertEqual(receipt["source_event_count"], 7) + self.assertEqual(receipt["validation_verdict"], "PASS") + # final files exist + self.assertTrue(dest.is_file()) + self.assertTrue(dest.with_name(dest.name + ".receipt.json").is_file()) + # no temp dirs remain + leftovers = [p for p in Path(td).iterdir() if ".tmp." in p.name] + self.assertEqual(leftovers, []) + + def test_source_bytes_preserved(self): + before = _inventory(self.src) + with tempfile.TemporaryDirectory() as td: + migrate_store(self.src, Path(td) / "methodfactory.sqlite3") + after = _inventory(self.src) + self.assertEqual(before, after) + + def test_migrated_chain_validates(self): + from methodfactory.storage.store import SqliteManifestStore + + with tempfile.TemporaryDirectory() as td: + dest = Path(td) / "methodfactory.sqlite3" + migrate_store(self.src, dest) + store = SqliteManifestStore(Path(td)) + try: + result = store.validate_chain("pkg_demo_001", verify_artifacts=True) + self.assertTrue(result["valid"]) + self.assertEqual(result["events"], 7) + finally: + store.close() + + def test_rev0_ids_preserved(self): + with tempfile.TemporaryDirectory() as td: + migrate_store(self.src, Path(td) / "methodfactory.sqlite3") + conn = _open_db(Path(td) / "methodfactory.sqlite3") + row = conn.execute( + "SELECT event_id, action_id, action FROM events WHERE revision=0" + ).fetchone() + conn.close() + self.assertEqual(row["action_id"], "act_create_package") + self.assertEqual(row["action"], "create_package") + self.assertTrue(row["event_id"].startswith("evt_")) + + def test_summary_content_addressed(self): + with tempfile.TemporaryDirectory() as td: + migrate_store(self.src, Path(td) / "methodfactory.sqlite3") + conn = _open_db(Path(td) / "methodfactory.sqlite3") + row = conn.execute( + "SELECT manifest_json FROM events WHERE revision=4" + ).fetchone() + conn.close() + manifest = json.loads(row["manifest_json"]) + summary = manifest["summary"] + body = summary.get("content") + self.assertIsNone(body) # content-addressed: body NOT inline + self.assertTrue(summary["digest"].startswith("56c4b55c")) + self.assertIsNotNone(summary.get("size")) + # blob exists and matches digest + blob = Path(td) / "blobs" / summary["digest"] + self.assertTrue(blob.is_file()) + self.assertEqual( + hashlib.sha256(blob.read_bytes()).hexdigest(), summary["digest"] + ) + + def test_artifact_blobs_verified(self): + with tempfile.TemporaryDirectory() as td: + migrate_store(self.src, Path(td) / "methodfactory.sqlite3") + conn = _open_db(Path(td) / "methodfactory.sqlite3") + row = conn.execute( + "SELECT manifest_json FROM events WHERE revision=5" + ).fetchone() + conn.close() + manifest = json.loads(row["manifest_json"]) + art = manifest["artifacts"][-1] + self.assertEqual(art["sha256"], digest_bytes(b"body")) + blob = Path(td) / "blobs" / art["sha256"] + self.assertEqual(blob.read_bytes(), b"body") + + def test_timestamp_normalization_advancing_clock(self): + """ADR-0012 §9: current engine derives summary timestamps from + created_at=legacy event.at, NOT the legacy engine's intermediate + clock values.""" + with tempfile.TemporaryDirectory() as td: + migrate_store(self.src, Path(td) / "methodfactory.sqlite3") + conn = _open_db(Path(td) / "methodfactory.sqlite3") + # prepare_summary is rev3 (at 00:07); confirm_summary is rev4 (00:10) + row3 = conn.execute( + "SELECT created_at, manifest_json FROM events WHERE revision=3" + ).fetchone() + row4 = conn.execute( + "SELECT created_at, manifest_json FROM events WHERE revision=4" + ).fetchone() + conn.close() + self.assertEqual(row3["created_at"], "2026-08-07T00:07:00+00:00") + self.assertEqual(row4["created_at"], "2026-08-07T00:10:00+00:00") + m3 = json.loads(row3["manifest_json"]) + m4 = json.loads(row4["manifest_json"]) + # normalized: presented_at == created_at of the preparing event + self.assertEqual(m3["summary"]["presented_at"], "2026-08-07T00:07:00+00:00") + self.assertEqual( + m4["summary"]["confirmation"]["confirmed_at"], "2026-08-07T00:10:00+00:00" + ) + # legacy engine used 00:05 (presented) / 00:08 (confirmed) — NOT copied + self.assertNotEqual(m3["summary"]["presented_at"], "2026-08-07T00:05:00+00:00") + self.assertNotEqual( + m4["summary"]["confirmation"]["confirmed_at"], "2026-08-07T00:08:00+00:00" + ) + + def test_every_action_family(self): + src = _generate("all_families", all_action_families_workflow) + with tempfile.TemporaryDirectory() as td: + receipt = migrate_store(src, Path(td) / "methodfactory.sqlite3") + self.assertEqual(receipt["source_event_count"], 12) + # all seven action families present in reconstructed actions + conn = _open_db(Path(td) / "methodfactory.sqlite3") + actions = {r["action"] for r in conn.execute("SELECT DISTINCT action FROM events")} + conn.close() + self.assertEqual( + actions, + {"create_package", "record_input", "set_objective", + "prepare_summary", "confirm_summary", "revise_intake", + "record_draft_artifact", "cancel"}, + ) + + def test_optional_field_candidates(self): + src = _generate("optional_omitted", optional_omitted_workflow) + with tempfile.TemporaryDirectory() as td: + receipt = migrate_store(src, Path(td) / "methodfactory.sqlite3") + self.assertEqual(receipt["source_event_count"], 6) + conn = _open_db(Path(td) / "methodfactory.sqlite3") + # rev1: record_input omitted exclusion_reason -> None in manifest + row = conn.execute( + "SELECT manifest_json FROM events WHERE revision=1" + ).fetchone() + manifest = json.loads(row["manifest_json"]) + self.assertEqual(manifest["inputs"][-1]["exclusion_reason"], None) + # rev2: set_objective omitted desired_outcomes -> [] + row = conn.execute( + "SELECT manifest_json FROM events WHERE revision=2" + ).fetchone() + manifest = json.loads(row["manifest_json"]) + self.assertEqual(manifest["objective"], { + "statement": "Only statement", "desired_outcomes": []}) + # rev4: confirm_summary default operator -> "operator" + row = conn.execute( + "SELECT manifest_json FROM events WHERE revision=4" + ).fetchone() + manifest = json.loads(row["manifest_json"]) + self.assertEqual( + manifest["summary"]["confirmation"]["operator_id"], "operator") + conn.close() + + def test_non_ascii_content(self): + src = _generate("non_ascii", non_ascii_workflow) + with tempfile.TemporaryDirectory() as td: + receipt = migrate_store(src, Path(td) / "methodfactory.sqlite3") + self.assertEqual(receipt["source_event_count"], 3) + conn = _open_db(Path(td) / "methodfactory.sqlite3") + row = conn.execute( + "SELECT manifest_json FROM events WHERE revision=1" + ).fetchone() + manifest = json.loads(row["manifest_json"]) + # "สวัสดี こんにちは" = 34 UTF-8 bytes + self.assertEqual(manifest["inputs"][-1]["content_size"], 34) + self.assertEqual( + manifest["inputs"][-1]["content_sha256"], + "d68784da4b0bdf1348af3886af7d7784313a629d6baaaefcc798328c8de8d06b", + ) + conn.close() + + def test_revise_intake(self): + src = _generate("revise", revise_intake_workflow) + with tempfile.TemporaryDirectory() as td: + receipt = migrate_store(src, Path(td) / "methodfactory.sqlite3") + self.assertEqual(receipt["source_event_count"], 5) + conn = _open_db(Path(td) / "methodfactory.sqlite3") + row = conn.execute( + "SELECT manifest_json FROM events WHERE revision=4" + ).fetchone() + manifest = json.loads(row["manifest_json"]) + self.assertIsNone(manifest["summary"]) + conn.close() + + def test_receipt_identity(self): + with tempfile.TemporaryDirectory() as td: + dest = Path(td) / "methodfactory.sqlite3" + migrate_store(self.src, dest) + receipt = json.loads( + dest.with_name(dest.name + ".receipt.json").read_text() + ) + self.assertEqual(receipt["receipt_format"], "method-factory-migration-receipt") + self.assertEqual(receipt["receipt_version"], "v1") + self.assertEqual(receipt["legacy_source_format"], "v0.1.2-integrity") + self.assertEqual( + receipt["legacy_source_commit"], + "fb5641cc1a3f1f54b96bba3af88ec5a1b010f4e5", + ) + self.assertEqual(receipt["source_package_count"], 1) + self.assertEqual(receipt["source_event_count"], 7) + self.assertEqual(receipt["destination_event_count"], 7) + self.assertEqual(receipt["validation_verdict"], "PASS") + # receipt inventory == current source inventory + self.assertEqual( + {k: (v["sha256"], v["size"]) for k, v in + receipt["semantic_source_inventory"].items()}, + _inventory(self.src), + ) + + +class MigrationFailClosedTests(unittest.TestCase): + """Cases that must fail typed and never publish.""" + + @classmethod + def setUpClass(cls): + cls.src = _generate("standard", standard_workflow) + + def test_unrecoverable_cancel_reason(self): + src = _generate("cancel_arbitrary", cancel_arbitrary_reason_workflow) + with tempfile.TemporaryDirectory() as td: + dest = Path(td) / "methodfactory.sqlite3" + with self.assertRaises(MigrationIncompatibleError): + migrate_store(src, dest) + self.assertFalse(dest.exists()) + self.assertFalse(dest.with_name(dest.name + ".receipt.json").exists()) + + def test_overlimit_intent(self): + src = _generate("bigintent", overlimit_intent_workflow) + with tempfile.TemporaryDirectory() as td: + with self.assertRaises(MigrationIncompatibleError): + migrate_store(src, Path(td) / "methodfactory.sqlite3") + + def test_overlimit_input(self): + src = _generate("biginput", overlimit_input_workflow) + with tempfile.TemporaryDirectory() as td: + with self.assertRaises(MigrationIncompatibleError): + migrate_store(src, Path(td) / "methodfactory.sqlite3") + + def test_current_invalid_identifier(self): + src = _generate("badid", current_invalid_identifier_workflow) + with tempfile.TemporaryDirectory() as td: + with self.assertRaises(MigrationIncompatibleError): + migrate_store(src, Path(td) / "methodfactory.sqlite3") + + def test_destination_exists(self): + with tempfile.TemporaryDirectory() as td: + dest = Path(td) / "methodfactory.sqlite3" + dest.write_bytes(b"already here") + with self.assertRaises(DestinationExistsError): + migrate_store(self.src, dest) + self.assertEqual(dest.read_bytes(), b"already here") + + def test_duplicate_event_id_across_packages(self): + """A second package reusing event_ids from the first must fail + MIGRATION_INCOMPATIBLE (global event-ID uniqueness), never leak a + raw sqlite error.""" + # Build a two-package legacy store by cloning the standard fixture + # journal and rewriting package_id in snapshot+hashes is complex; use + # a hand-crafted second journal that shares event ids by copying the + # first package's journal lines and only changing the manifest + # package_id (chain hashes still verify because we recompute them). + with tempfile.TemporaryDirectory() as td: + src = Path(td) / "src" + shutil.copytree(self.src, src) + events = [json.loads(l) for l in + (src / "events/pkg_demo_001.events.jsonl").read_text().splitlines()] + # Build pkg_demo_002 journal: same event_ids, same actions, but + # snapshot package_id rewritten to pkg_demo_002; recompute all + # manifest/action hashes using legacy canonicalization. + out_lines = [] + prev_hash = None + for ev in events: + snap = json.loads(json.dumps(ev["manifest_snapshot"])) + snap["package_id"] = "pkg_demo_002" + snap["previous_manifest_sha256"] = prev_hash + resulting = legacy_digest_json(snap) + # rev0 special action hash; rev>0 legacy canonical of semantic + if ev["revision"] == 0: + action_hash = hashlib.sha256( + legacy_canonical_json( + {"action": "create_package", "package_id": "pkg_demo_002"} + ) + ).hexdigest() + else: + semantic = { + "protocol_version": "0.1", + "action_id": ev["action_id"], + "package_id": "pkg_demo_002", + "action": ev["action"], + "basis": {}, + "payload": {}, + } + action_hash = hashlib.sha256( + legacy_canonical_json(semantic) + ).hexdigest() + newev = { + "event_id": ev["event_id"], + "action": ev["action"], + "action_id": ev["action_id"], + "revision": ev["revision"], + "state_before": ev["state_before"], + "state_after": ev["state_after"], + "resulting_manifest_sha256": resulting, + "previous_manifest_sha256": prev_hash, + "action_sha256": action_hash, + "at": ev["at"], + "manifest_snapshot": snap, + } + out_lines.append(json.dumps(newev, sort_keys=True)) + prev_hash = resulting + (src / "events/pkg_demo_002.events.jsonl").write_text( + "\n".join(out_lines) + "\n", encoding="utf-8") + # The second package shares ALL event_ids with the first. + dest = Path(td) / "methodfactory.sqlite3" + with self.assertRaises(MigrationIncompatibleError): + migrate_store(src, dest) + self.assertFalse(dest.exists()) + + +class LegacySourceTests(unittest.TestCase): + """Frozen reader semantics: cache, lock, source identity.""" + + @classmethod + def setUpClass(cls): + cls.src = _generate("standard", standard_workflow) + + def _copy(self, td: str) -> Path: + src = Path(td) / "src" + shutil.copytree(self.src, src) + return src + + def test_cache_absent_ok(self): + with tempfile.TemporaryDirectory() as td: + src = self._copy(td) + (src / "packages/pkg_demo_001.json").unlink() + ls = LegacySource(src) + ls.validate() + self.assertEqual(ls.packages["pkg_demo_001"].cache_status, "absent") + migrate_store(src, Path(td) / "methodfactory.sqlite3") + + def test_cache_lagging_valid(self): + with tempfile.TemporaryDirectory() as td: + src = self._copy(td) + # Replace cache with an earlier committed snapshot (rev2). + events = [json.loads(l) for l in + (src / "events/pkg_demo_001.events.jsonl").read_text().splitlines()] + earlier = events[2]["manifest_snapshot"] + (src / "packages/pkg_demo_001.json").write_text( + json.dumps(earlier, sort_keys=True), encoding="utf-8") + ls = LegacySource(src) + ls.validate() + self.assertEqual(ls.packages["pkg_demo_001"].cache_status, "valid") + migrate_store(src, Path(td) / "methodfactory.sqlite3") + + def test_cache_invalid_fails(self): + with tempfile.TemporaryDirectory() as td: + src = self._copy(td) + (src / "packages/pkg_demo_001.json").write_text( + json.dumps({"not": "a committed snapshot"}), encoding="utf-8") + ls = LegacySource(src) + with self.assertRaises(LegacyChainInvalidError): + ls.validate() + with self.assertRaises(LegacyChainInvalidError): + migrate_store(src, Path(td) / "methodfactory.sqlite3") + + def test_lock_refuses_without_mutation(self): + with tempfile.TemporaryDirectory() as td: + src = self._copy(td) + lock = src / "events/pkg_demo_001.lock" + lock.write_text("pid 123\n", encoding="utf-8") + before = _inventory(src) + with self.assertRaises(ConcurrencyError): + migrate_store(src, Path(td) / "methodfactory.sqlite3") + # lock untouched; source bytes identical; lock NOT in inventory + self.assertTrue(lock.exists()) + self.assertEqual(lock.read_text(), "pid 123\n") + self.assertEqual(_inventory(src), before) + self.assertNotIn("events/pkg_demo_001.lock", _inventory(src)) + + def test_corrupt_hash_fails(self): + with tempfile.TemporaryDirectory() as td: + src = self._copy(td) + journal = src / "events/pkg_demo_001.events.jsonl" + text = journal.read_text().splitlines() + line = json.loads(text[1]) + line["resulting_manifest_sha256"] = "0" * 64 + text[1] = json.dumps(line, sort_keys=True) + journal.write_text("\n".join(text) + "\n", encoding="utf-8") + ls = LegacySource(src) + with self.assertRaises(LegacyChainInvalidError): + ls.validate() + + def test_missing_blob_fails(self): + with tempfile.TemporaryDirectory() as td: + src = self._copy(td) + blob = next((src / "artifacts/blobs").iterdir()) + blob.unlink() + ls = LegacySource(src) + with self.assertRaises((LegacyChainInvalidError, LegacySourceInvalidError)): + ls.validate() + + def test_source_inventory_excludes_lock(self): + with tempfile.TemporaryDirectory() as td: + src = self._copy(td) + (src / "events/pkg_demo_001.lock").write_text("pid\n", encoding="utf-8") + inv = _inventory(src) + self.assertNotIn("events/pkg_demo_001.lock", inv) + + +class PublicationFaultTests(unittest.TestCase): + """Fault injection across publication stages; always fail closed.""" + + @classmethod + def setUpClass(cls): + cls.src = _generate("standard", standard_workflow) + + def _run_fault_at(self, stage: str, expect_publication: bool = False): + """Inject a raise at `stage`; assert typed failure + fail-closed state. + + `expect_publication=False` (fault before DB rename): no final DB may + exist. The receipt may legitimately exist for stages after receipt + publication (receipt publishes before DB per ADR-0012 §11) — but the + call must RAISE, never report success. `expect_publication=True` + (fault after DB rename): the DB file may exist but success is never + reported. + """ + old = migrate_module.FAULT_HOOK + try: + def hook(s): + if s == stage: + raise StorageError(f"fault at {stage}") + migrate_module.FAULT_HOOK = hook + with tempfile.TemporaryDirectory() as td: + dest = Path(td) / "methodfactory.sqlite3" + try: + migrate_store(self.src, dest) + self.fail(f"expected failure at {stage}") + except MethodFactoryError: + pass # typed failure; never a success return + if not expect_publication: + self.assertFalse(dest.exists()) + finally: + migrate_module.FAULT_HOOK = old + + def test_fault_matrix(self): + # Faults BEFORE the DB rename: nothing published. + pre_stages = [ + "after_source_inventory_before", + "before_build_store", + "after_build_store", + "after_validate_temp_store", + "after_source_inventory_after", + "before_receipt_write", + "before_receipt_replace", + "before_dir_fsync_receipt", + "before_db_replace", + ] + for stage in pre_stages: + with self.subTest(stage=stage): + self._run_fault_at(stage, expect_publication=False) + # Faults AFTER the DB rename: DB exists but success is never reported. + post_stages = [ + "before_dir_fsync_db", + "before_final_verify", + ] + for stage in post_stages: + with self.subTest(stage=stage): + self._run_fault_at(stage, expect_publication=True) + + def test_source_changed_detected(self): + """Mutate the source between the two inventory passes -> SOURCE_CHANGED, + no publication.""" + old = migrate_module.FAULT_HOOK + try: + def hook(s): + if s == "after_source_inventory_before": + # mutate a source file after the BEFORE inventory + journal = self.src / "events/pkg_demo_001.events.jsonl" + with open(journal, "a", encoding="utf-8") as fh: + fh.write("\n") + migrate_module.FAULT_HOOK = hook + with tempfile.TemporaryDirectory() as td: + dest = Path(td) / "methodfactory.sqlite3" + with self.assertRaises(SourceChangedError): + migrate_store(self.src, dest) + self.assertFalse(dest.exists()) + self.assertFalse( + dest.with_name(dest.name + ".receipt.json").exists() + ) + finally: + migrate_module.FAULT_HOOK = old + + def test_final_verify_fault_after_publication(self): + """A fault at after_publication (post-DB rename) is not success.""" + old = migrate_module.FAULT_HOOK + try: + def hook(s): + if s == "after_publication": + raise StorageError("fault after publication") + migrate_module.FAULT_HOOK = hook + with tempfile.TemporaryDirectory() as td: + dest = Path(td) / "methodfactory.sqlite3" + with self.assertRaises(MethodFactoryError): + migrate_store(self.src, dest) + finally: + migrate_module.FAULT_HOOK = old + + +class ExportDeterminismTests(unittest.TestCase): + """Deterministic exports (method-factory-events-v1 + legacy-v012-jsonl).""" + + @classmethod + def setUpClass(cls): + src = _generate("standard", standard_workflow) + cls.td = tempfile.TemporaryDirectory() + cls.root = Path(cls.td.name) + migrate_store(src, cls.root / "methodfactory.sqlite3") + + @classmethod + def tearDownClass(cls): + cls.td.cleanup() + + def test_v1_field_set_and_deterministic(self): + a = Path(self.td.name) / "a.jsonl" + b = Path(self.td.name) / "b.jsonl" + n1 = export_events(self.root, a, fmt=EVENTS_V1_FORMAT) + n2 = export_events(self.root, b, fmt=EVENTS_V1_FORMAT) + self.assertEqual(n1, 7) + self.assertEqual(a.read_bytes(), b.read_bytes()) + # exact final newline + text = a.read_text(encoding="utf-8") + self.assertTrue(text.endswith("\n")) + self.assertFalse(text.endswith("\n\n")) + line = json.loads(text.splitlines()[0]) + self.assertEqual( + set(line.keys()), + {"format", "format_version", "package_id", "revision", "event_id", + "action_id", "action", "state_before", "state_after", + "action_sha256", "previous_manifest_sha256", + "resulting_manifest_sha256", "created_at", "semantic_action", + "manifest"}, + ) + self.assertEqual(line["format"], "method-factory-events-v1") + self.assertEqual(line["format_version"], 1) + # ordering by package_id, revision + rows = [json.loads(l) for l in text.splitlines()] + self.assertEqual([r["revision"] for r in rows], list(range(7))) + + def test_legacy_v012_deterministic_and_chain(self): + a = Path(self.td.name) / "l1.jsonl" + b = Path(self.td.name) / "l2.jsonl" + n1 = export_events(self.root, a, fmt=LEGACY_JSONL_FORMAT) + n2 = export_events(self.root, b, fmt=LEGACY_JSONL_FORMAT) + self.assertEqual(n1, 7) + self.assertEqual(a.read_bytes(), b.read_bytes()) + rows = [json.loads(l) for l in a.read_text().splitlines()] + # legacy chain: prev == previous resulting in legacy hash space + prev = None + for r in rows: + self.assertEqual(r["previous_manifest_sha256"], prev) + prev = r["resulting_manifest_sha256"] + # legacy event shape (no semantic_action/manifest/created_at split) + self.assertNotIn("semantic_action", rows[0]) + self.assertNotIn("manifest", rows[0]) + self.assertIn("manifest_snapshot", rows[0]) + self.assertIn("at", rows[0]) + + def test_legacy_rev0_action_hash(self): + out = Path(self.td.name) / "l.jsonl" + export_events(self.root, out, fmt=LEGACY_JSONL_FORMAT) + line = json.loads(out.read_text().splitlines()[0]) + self.assertEqual( + line["action_sha256"], + "b3f006065ea54589c09f528898abbab0496455df3e39c1b7c7fd70c524750aab", + ) + + def test_legacy_hash_vs_line_serializer_distinction(self): + """Public v0.1.2 hash serializer (ensure_ascii compact) differs from + the journal-line serializer (Python default spacing, ASCII escape).""" + ev = {"at": "2026-08-07T00:00:00+00:00", "b": "é"} + canonical = legacy_digest_json # hash form: ensure_ascii compact + line = legacy_line_json(ev) # line form: default spacing + ASCII + self.assertIn("\\u00e9", line) + self.assertIn('": "', line) # default spacing after colon + # hash form has no spacing and escapes unicode too + from methodfactory.migrations.v012_jsonl import legacy_canonical_json + self.assertIn(b"\\u00e9", legacy_canonical_json(ev)) + self.assertNotIn(b'"b": "', legacy_canonical_json(ev)) + + def test_export_no_mutation(self): + before = _inventory(self.root) + out = Path(self.td.name) / "n.jsonl" + export_events(self.root, out, fmt=EVENTS_V1_FORMAT) + after = _inventory(self.root) + self.assertEqual(before, after) + # export to stdout does not touch the store either + before2 = _inventory(self.root) + self.assertEqual(before2, after) + + def test_export_consistent_read(self): + """Export succeeds and returns exact row count under a read-only open.""" + out = Path(self.td.name) / "c.jsonl" + n = export_events(self.root, out, fmt=EVENTS_V1_FORMAT) + self.assertEqual(n, 7) + self.assertEqual(len(out.read_text().splitlines()), 7) + + def test_export_destination_exists_fails(self): + out = Path(self.td.name) / "exists.jsonl" + out.write_text("x", encoding="utf-8") + with self.assertRaises(StorageError): + export_events(self.root, out, fmt=EVENTS_V1_FORMAT) + self.assertEqual(out.read_text(), "x") + + +class SurfaceBoundaryTests(unittest.TestCase): + """No import surface; no 8a repair/lock/CAS mechanics; CLI bounded.""" + + def test_no_import_surface(self): + import pkgutil + + import methodfactory.migrations as m + names = [m.name for m in pkgutil.iter_modules(m.__path__)] + self.assertNotIn("import", names) + self.assertNotIn("import_store", names) + # no import module/function in the package + for modname in names: + self.assertNotIn("import", modname) + + def test_no_8a_mechanics_in_reader(self): + """The frozen reader must not port CAS/lock/repair mechanics. + + Checks code identifiers (not docstring prose): no CAS primitive, no + lock acquisition, no tail-repair, no PID ownership, no append, no + manifest-cache reconciliation as a mutation mechanism. + """ + import methodfactory.migrations.v012_jsonl as v + + src = Path(v.__file__).read_text(encoding="utf-8") + # strip docstrings/prose: only inspect executable code + code = src.split('"""')[0] + "".join(src.split('"""')[2:]) + for banned in ("compare_and_swap", "_read_last_event", "os.link", + "fcntl", "threading", "pid", "tail_repair", + "chmod", "unlink", "mkdir", "write_text", "open(", + "acquire"): + self.assertNotIn(banned, code) + + def test_no_relaxed_serializers_in_migrate(self): + import methodfactory.migrations.migrate as mg + + src = Path(mg.__file__).read_text(encoding="utf-8") + # migration must use current canonical serialization for the modern + # rows (ensure_ascii=False), not the legacy hash form + self.assertIn("ensure_ascii=False", src) + + def test_temp_hygiene_after_failure(self): + """A failed migration leaves no temp dirs or receipt behind.""" + src = _generate("cancel_arbitrary", cancel_arbitrary_reason_workflow) + with tempfile.TemporaryDirectory() as td: + dest = Path(td) / "methodfactory.sqlite3" + try: + migrate_store(src, dest) + except MigrationIncompatibleError: + pass + leftovers = [p for p in Path(td).iterdir() if ".tmp." in p.name] + self.assertEqual(leftovers, []) + + +class CliBoundaryTests(unittest.TestCase): + """`mf --version`, migrate-store, export through the real CLI.""" + + def test_version_unchanged(self): + result = subprocess.run( + [sys.executable, "-m", "methodfactory.cli", "--version"], + capture_output=True, text=True, + ) + self.assertEqual(result.returncode, 0) + self.assertIn("methodfactory 2.0.0a1", result.stdout) + + def test_no_lifecycle_commands(self): + result = subprocess.run( + [sys.executable, "-m", "methodfactory.cli", "--help"], + capture_output=True, text=True, + ) + for banned in ("create", "apply", "status", "summary", "review", + "trial", "ship", "triage"): + self.assertNotIn(f" {banned} ", result.stdout) + + def test_cli_migrate_store(self): + src = _generate("standard", standard_workflow) + with tempfile.TemporaryDirectory() as td: + dest = Path(td) / "methodfactory.sqlite3" + result = subprocess.run( + [sys.executable, "-m", "methodfactory.cli", + "migrate-store", "--source", str(src), "--dest", str(dest)], + capture_output=True, text=True, + ) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertTrue(dest.is_file()) + self.assertIn("validation_verdict", result.stdout) + + def test_cli_export(self): + src = _generate("standard", standard_workflow) + with tempfile.TemporaryDirectory() as td: + dest = Path(td) / "methodfactory.sqlite3" + subprocess.run( + [sys.executable, "-m", "methodfactory.cli", + "migrate-store", "--source", str(src), "--dest", str(dest)], + capture_output=True, text=True, check=True, + ) + out = Path(td) / "out.jsonl" + result = subprocess.run( + [sys.executable, "-m", "methodfactory.cli", + "export", "--store", str(Path(td)), + "--output", str(out), "--format", "method-factory-events-v1"], + capture_output=True, text=True, + ) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual(len(out.read_text().splitlines()), 7) + + def test_cli_errors_typed(self): + result = subprocess.run( + [sys.executable, "-m", "methodfactory.cli", + "migrate-store", "--source", "/does/not/exist", + "--dest", "/tmp/x.sqlite3"], + capture_output=True, text=True, + ) + self.assertEqual(result.returncode, 1) + self.assertIn("LEGACY_SOURCE_INVALID", result.stderr) + + +# Imported at module level for fault tests +from methodfactory.domain.errors import MethodFactoryError +from methodfactory.storage.errors import StorageError +from methodfactory.migrations.v012_jsonl import legacy_canonical_json + +if __name__ == "__main__": + unittest.main() From 54ac31353dfdeb1d12b908bfe241bd4d787df468 Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Sat, 8 Aug 2026 06:00:15 +0700 Subject: [PATCH 28/41] fix(migration): close code-review findings - fail-closed boundary, atomic publication, legacy export chain, empty-journal guard Mandatory code-review family (4 finders + verify + dedupe + sanity) over b9e46c1..34ba9d1 returned 29 verified findings (12 major / 15 minor / 2 nit), incl. two empirically-proven defects. This commit closes the blocking set: - bug-o1: legacy-v012-jsonl export left manifest_snapshot previous_manifest_sha256 in CURRENT hash space; exported journals failed frozen-reader re-validation. Snapshot previous hash is now threaded in legacy hash space; golden re-validation tests added (standard + non-ASCII). - bug-o2: empty/truncated journal passed validation with false PASS. Require revision-0 create event; fail LEGACY_CHAIN_INVALID. - bug-1/q-9: publication failure after receipt replace left a durable PASS receipt without DB. Final receipt is now removed when the DB was not replaced; MigrationPublishFailedError carries explicit operator instructions; receipt-without-DB test added. - bug-2/sec-5/bug-7: destination no-clobber was check-then-act. Both migrate and export now publish via O_EXCL temp + atomic no-clobber link; raced destinations abort, never overwrite. - bug-3/sec-1/q-2: migration chmod'd the legacy source root in the default destination case. ArtifactStore gains chmod_existing=False; only newly-created roots get private mode; mode-untouched test added. - bug-4: malformed legacy snapshots leaked raw KeyError/TypeError during reconstruction. Frozen reader now validates snapshot container shape with typed LegacyChainInvalidError. - bug-5/q-1: legacy export serialized with compact ensure_ascii=False; now uses the frozen v0.1.2 journal-line serializer (default spacing, ASCII escaping). - bug-8: success predicate enforced - _verify_final parses and binds the published receipt to the final DB. - q-3/q-6/bug-9: dedupe legacy hash helpers into v012_jsonl; remove dead code and unused imports. - q-10: SqliteManifestStore.list_package_ids() replaces private _conn access; readonly_uri() promoted from _readonly_uri. - q-13: CLI catches the documented native boundary set. - perf-1: per-package blob verification dedupe by digest. - sec-2/sec-3: symlinked source files fail loudly in inventory. - q-7/q-8: fixture provenance recorded; per-process paths; empty-journal, within-package duplicate event_id, logical-path grammar, control characters, source-symlink, dest-permission, and no-clobber tests. Full suite: 410 tests green (356 baseline + 54 migration/export). --- methodfactory/adapters/artifact_store.py | 15 +- methodfactory/cli.py | 34 ++++- methodfactory/migrations/export.py | 97 ++++++++---- methodfactory/migrations/migrate.py | 169 ++++++++++++++------- methodfactory/migrations/v012_jsonl.py | 131 +++++++++++++++- methodfactory/storage/sqlite.py | 12 +- methodfactory/storage/store.py | 7 + methodfactory/tests/_fixtures.py | 52 ++++--- methodfactory/tests/test_migrations.py | 185 +++++++++++++++++++++++ 9 files changed, 579 insertions(+), 123 deletions(-) diff --git a/methodfactory/adapters/artifact_store.py b/methodfactory/adapters/artifact_store.py index f7b8a1b..a10db80 100644 --- a/methodfactory/adapters/artifact_store.py +++ b/methodfactory/adapters/artifact_store.py @@ -92,7 +92,16 @@ def _close_fd(fd: int) -> None: class ArtifactStore: - def __init__(self, root: Path | str) -> None: + def __init__(self, root: Path | str, *, chmod_existing: bool = True) -> None: + """Artifact store over content-addressed blobs. + + ``chmod_existing=False`` (migration only): when the root already + exists, do not chmod it. The migration publishes blobs into the final + destination root, which (with the default destination) is the legacy + source root itself; mutating its permissions would violate source + immutability (ADR-0012 §12). The blobs subdirectory is always + created private. + """ if not isinstance(root, (str, os.PathLike)): raise InvalidPayloadError( f"artifact store root must be a path, got {type(root).__name__}" @@ -103,10 +112,12 @@ def __init__(self, root: Path | str) -> None: # umask-inherited group/world-writable blobs dir would let other # local users unlink blobs or plant symlinks in the digest # namespace, defeating the immutable-store guarantees. + root_existed = self.root.is_dir() os.makedirs(self.root, mode=0o700, exist_ok=True) self.blobs = self.root / "blobs" os.makedirs(self.blobs, mode=0o700, exist_ok=True) - os.chmod(self.root, 0o700) + if chmod_existing or not root_existed: + os.chmod(self.root, 0o700) os.chmod(self.blobs, 0o700) except OSError as exc: raise InvalidPayloadError( diff --git a/methodfactory/cli.py b/methodfactory/cli.py index d19ddcf..ecd3137 100644 --- a/methodfactory/cli.py +++ b/methodfactory/cli.py @@ -14,14 +14,25 @@ from __future__ import annotations import argparse +import json +import sqlite3 import sys from . import __version__ from .domain.errors import MethodFactoryError -AVAILABILITY = ( - "Method Factory storage is under architecture reset (ADR-0012); " - "lifecycle commands return in a later phase." +# Native exceptions the public boundary promises to translate (docs +# public-surface.md). The CLI is the last line: anything that escapes the +# typed boundary still surfaces as a stable STORAGE_ERROR, never a raw +# traceback of an unhandled native exception. +_BOUNDARY_NATIVES = ( + OSError, + sqlite3.Error, + json.JSONDecodeError, + UnicodeError, + TypeError, + ValueError, + RecursionError, ) @@ -30,6 +41,18 @@ def _fail(err: MethodFactoryError) -> int: return 1 +def _fail_native(exc: BaseException) -> int: + print( + json.dumps( + {"code": "STORAGE_ERROR", + "message": f"unexpected {type(exc).__name__}: {exc}"}, + sort_keys=True, + ), + file=sys.stderr, + ) + return 1 + + def _cmd_migrate_store(args) -> int: from .migrations.migrate import migrate_store @@ -37,7 +60,8 @@ def _cmd_migrate_store(args) -> int: receipt = migrate_store(args.source, dest=args.dest) except MethodFactoryError as exc: return _fail(exc) - import json + except _BOUNDARY_NATIVES as exc: + return _fail_native(exc) print(json.dumps(receipt, sort_keys=True, indent=2)) return 0 @@ -50,6 +74,8 @@ def _cmd_export(args) -> int: count = export_events(args.store, args.output, fmt=args.format) except MethodFactoryError as exc: return _fail(exc) + except _BOUNDARY_NATIVES as exc: + return _fail_native(exc) if args.output is None: # events already written to stdout; report count on stderr print(f"exported {count} events", file=sys.stderr) diff --git a/methodfactory/migrations/export.py b/methodfactory/migrations/export.py index a64c16a..28b4128 100644 --- a/methodfactory/migrations/export.py +++ b/methodfactory/migrations/export.py @@ -10,12 +10,21 @@ PUBLIC v0.1.2 event SHAPE using public v0.1.2 semantics: - inline summary content; - legacy canonical manifest hashes (ensure_ascii=True, compact); - - legacy predecessor hashes; + - legacy predecessor hashes (event-level AND snapshot-level, both in the + legacy hash space, so the exported journal re-validates under the + frozen v0.1.2 reader); - legacy rev>0 action hashes using legacy canonical hash serialization; - legacy special rev0 action hash; - journal-line serialization `json.dumps(event, sort_keys=True) + "\\n"` (Python default spacing, ASCII escaping). + LIMITATION (honest): the export reconstructs the public v0.1.2 event + SHAPE and line serializer, not byte identity with the original journal: + summary timestamps are the normalized current-era values (ADR-0012 §9), + and lineage/action hashes are recomputed over the reconstructed shapes. + It is an evidence stream, not a byte-for-byte replay of the source + journal. + Both are read-only and deterministic: same DB + same exporter version -> byte-identical output. Export never mutates the store; it uses a read-only SQLite connection with a consistent read transaction. @@ -30,7 +39,6 @@ from typing import Any from ..storage.errors import StorageError -from ..storage.serialization import canonical_bytes, sha256_hex from ..storage.sqlite import ( APPLICATION_ID, USER_VERSION, @@ -39,7 +47,9 @@ ) from .v012_jsonl import ( legacy_digest_json, + legacy_hash_semantic, legacy_line_json, + legacy_rev0_hash, ) EVENTS_V1_FORMAT = "method-factory-events-v1" @@ -105,7 +115,7 @@ def _legacy_event_object(row: dict, prev_legacy_hash: str | None) -> dict: # canonical_sha256. The summary body is stored as a blob; we need its # bytes. We recompute the summary body via the deterministic renderer # (byte-identical to public v0.1.2) when summary present. - legacy_manifest = _to_legacy_manifest(manifest) + legacy_manifest = _to_legacy_manifest(manifest, prev_legacy_hash) # Legacy manifest hashes (ensure_ascii=True). resulting = legacy_digest_json(legacy_manifest) @@ -113,15 +123,18 @@ def _legacy_event_object(row: dict, prev_legacy_hash: str | None) -> dict: # exported line's reconstructed manifest (rows process in package_id, # revision order). This keeps the exported chain fully consistent in the # LEGACY hash space, even when the current-era stored previous hash - # differs (non-ASCII content; ensure_ascii divergence). + # differs (non-ASCII content; ensure_ascii divergence). The snapshot's + # previous_manifest_sha256 is set to the same value (see + # _to_legacy_manifest) so the exported journal re-validates under the + # frozen v0.1.2 reader. prev = prev_legacy_hash # Legacy action hash: rev0 special reduced; rev>0 legacy canonical of # semantic action (six fields). if row["revision"] == 0: - action_hash = _legacy_rev0_hash(row["package_id"]) + action_hash = legacy_rev0_hash(row["package_id"]) else: - action_hash = _legacy_hash_semantic(semantic) + action_hash = legacy_hash_semantic(semantic) return { "event_id": row["event_id"], @@ -138,16 +151,20 @@ def _legacy_event_object(row: dict, prev_legacy_hash: str | None) -> dict: } -def _to_legacy_manifest(manifest: dict) -> dict: +def _to_legacy_manifest(manifest: dict, prev_legacy_hash: str | None = None) -> dict: """Convert current manifest to public v0.1.2 manifest shape. - summary inline content: regenerate via the deterministic renderer. - summary canonical_sha256 = digest of inline content (== current digest). - drop content-addressed digest/size/preview; add content + canonical. + - previous_manifest_sha256: recomputed in the LEGACY hash space when the + predecessor hash is threaded in (rev>0). Revision 0 stays None. """ import copy m = copy.deepcopy(manifest) + if prev_legacy_hash is not None: + m["previous_manifest_sha256"] = prev_legacy_hash summary = m.get("summary") if isinstance(summary, dict): body = _render_summary(m) @@ -172,24 +189,6 @@ def _legacy_digest_text(content: str) -> str: return legacy_digest_text(content) -def _legacy_rev0_hash(package_id: str) -> str: - from .v012_jsonl import legacy_canonical_json - - import hashlib - - return hashlib.sha256( - legacy_canonical_json({"action": "create_package", "package_id": package_id}) - ).hexdigest() - - -def _legacy_hash_semantic(semantic: dict) -> str: - from .v012_jsonl import legacy_canonical_json - - import hashlib - - return hashlib.sha256(legacy_canonical_json(semantic)).hexdigest() - - # ── public API ──────────────────────────────────────────────────────── def export_events( store_root: str | Path, @@ -223,13 +222,19 @@ def export_events( for row in rows: if fmt == EVENTS_V1_FORMAT: obj = _current_event_object(row) + line = json.dumps(obj, sort_keys=True, separators=(",", ":"), + ensure_ascii=False) else: + # legacy-v012-jsonl: PUBLIC v0.1.2 journal-line serializer + # (Python default spacing + ASCII escaping), NOT the compact + # current serializer. The HASH serializer remains legacy + # canonical (ensure_ascii compact) for action/manifest digests. obj = _legacy_event_object( row, prev_legacy.get(row["package_id"]) ) prev_legacy[row["package_id"]] = obj["resulting_manifest_sha256"] - lines.append(json.dumps(obj, sort_keys=True, separators=(",", ":"), - ensure_ascii=False)) + line = legacy_line_json(obj) + lines.append(line) payload = ("\n".join(lines) + "\n").encode("utf-8") if lines else b"" @@ -242,12 +247,38 @@ def export_events( out = Path(output) if out.exists(): raise StorageError(f"export destination exists: {out}") - tmp = out.with_name(out.name + ".tmp") - with open(tmp, "wb") as fh: - fh.write(payload) - fh.flush() - os.fsync(fh.fileno()) - os.replace(tmp, out) + # Unique same-directory temp (O_EXCL) then atomic no-clobber publication: + # a raced destination is never overwritten (CWE-377 / no-clobber). + import uuid + + tmp = out.with_name(f".{out.name}.tmp.{uuid.uuid4().hex}") + try: + fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + except OSError as exc: + raise StorageError(f"cannot create export temp file: {exc}") from exc + try: + with os.fdopen(fd, "wb") as fh: + fh.write(payload) + fh.flush() + os.fsync(fh.fileno()) + try: + os.link(tmp, out) + except FileExistsError: + raise StorageError( + f"export destination appeared during export: {out}" + ) from None + except OSError as exc: + raise StorageError(f"cannot publish export: {exc}") from exc + try: + tmp.unlink() + except OSError as exc: + raise StorageError(f"cannot remove export temp: {exc}") from exc + except BaseException: + try: + tmp.unlink() + except OSError: + pass + raise _fsync_dir(out.parent) return len(rows) diff --git a/methodfactory/migrations/migrate.py b/methodfactory/migrations/migrate.py index 48a3a4e..377662e 100644 --- a/methodfactory/migrations/migrate.py +++ b/methodfactory/migrations/migrate.py @@ -61,34 +61,30 @@ StaleActionError, ) from ..engine.apply import CREATE_PACKAGE_ACTION, next_manifest -from ..manifest.schema import validate_manifest_canonical from ..protocol.envelope import PROTOCOL_VERSION, envelope_from_dict from ..storage.errors import ( ChainViolationError, DestinationExistsError, - LegacyChainInvalidError, - LegacySourceInvalidError, MigrationIncompatibleError, MigrationPublishFailedError, SourceChangedError, StorageError, ) -from ..storage.paths import DB_FILENAME, validate_package_id +from ..storage.paths import DB_FILENAME from ..storage.serialization import canonical_bytes, sha256_hex from ..storage.sqlite import ( APPLICATION_ID, USER_VERSION, close_database, open_database, + readonly_uri, ) from .v012_jsonl import ( LEGACY_ACTION_CREATE, LEGACY_COMMIT, LEGACY_TAG, LegacySource, - legacy_canonical_json, - legacy_digest_json, - legacy_line_json, + legacy_hash_semantic, ) # Receipt format identity. @@ -113,23 +109,6 @@ def _fault(stage: str) -> None: """ -def _legacy_hash_semantic(semantic: dict) -> str: - """Legacy rev>0 action hash: sha256(legacy_canonical_json(semantic)).""" - import hashlib - - return hashlib.sha256(legacy_canonical_json(semantic)).hexdigest() - - -def _legacy_rev0_hash(package_id: str) -> str: - import hashlib - - return hashlib.sha256( - legacy_canonical_json( - {"action": "create_package", "package_id": package_id} - ) - ).hexdigest() - - # ── semantic-action reconstruction ──────────────────────────────────── def _reconstruct_rev0(source: LegacySource, pkg, ev) -> dict: """Build the current create semantic action from a legacy rev-0 event. @@ -158,7 +137,7 @@ def _candidate_hashes(source: LegacySource, pkg, ev, candidates: list[dict]) -> """Return candidates whose legacy hash matches the stored legacy hash.""" matches = [] for cand in candidates: - if _legacy_hash_semantic(cand) == ev.action_sha256: + if legacy_hash_semantic(cand) == ev.action_sha256: matches.append(cand) return matches @@ -391,10 +370,18 @@ def migrate_store( # NOTE: the current API treats a store path as a ROOT DIRECTORY (it # appends DB_FILENAME and creates blobs/). The temp build therefore uses # a temp root directory; publication moves the DB FILE to the final path. + # + # Never chmod an EXISTING destination root: when --dest is omitted the + # default root IS the legacy source root, and mutating its permissions + # would violate source immutability (ADR-0012 §12). Only newly-created + # roots get the private mode. final_root = final_dest.parent + root_existed = final_root.is_dir() try: final_root.mkdir(parents=True, exist_ok=True) - os.chmod(final_root, 0o700) + # chmod only when this call actually created the root directory. + if not root_existed: + os.chmod(final_root, 0o700) except OSError as exc: raise StorageError( f"cannot create destination parent directory: {exc}" @@ -403,7 +390,10 @@ def migrate_store( # Build the modern store. Blobs publish to the FINAL artifact store root # (orphan-safe on failure; ADR-0012 §11), while the DB builds at temp_root. - artifacts = ArtifactStore(final_root) + # chmod_existing=False: the default destination root IS the legacy source + # root; its permissions must not be mutated (ADR-0012 §12 source + # immutability). + artifacts = ArtifactStore(final_root, chmod_existing=False) try: _fault("before_build_store") _build_store(temp_root, source, artifacts) @@ -430,19 +420,23 @@ def migrate_store( raise # Publication: receipt first, then DB (ADR-0012 §11). - receipt = _build_receipt(source, before, temp_root) - temp_receipt = final_dest.with_name(final_dest.name + ".receipt.json.tmp") + receipt = _build_receipt(source, before) + temp_receipt = final_dest.with_name( + f".{final_dest.name}.receipt.tmp.{uuid.uuid4().hex}" + ) final_receipt = final_dest.with_name(final_dest.name + ".receipt.json") + db_replaced = False try: _fault("before_receipt_write") _write_durable(temp_receipt, receipt) _fault("before_receipt_replace") - _atomic_replace(temp_receipt, final_receipt) + _publish_noclobber(temp_receipt, final_receipt) _fault("before_dir_fsync_receipt") _fsync_dir(final_root) _fault("before_db_replace") - _atomic_replace(temp_root / DB_FILENAME, final_dest) + _publish_noclobber(temp_root / DB_FILENAME, final_dest) + db_replaced = True _fault("before_dir_fsync_db") _fsync_dir(final_root) _fault("after_publication") @@ -458,10 +452,24 @@ def migrate_store( p.unlink() except OSError: pass + # A durable PASS receipt without its database is a misleading crash + # state: remove the final receipt unless the DB itself was already + # published (DB-with-receipt is complete-but-unverified; the raise + # below prevents any success claim). + if not db_replaced: + try: + if final_receipt.exists(): + final_receipt.unlink() + except OSError: + pass if isinstance(exc, MethodFactoryError): raise raise MigrationPublishFailedError( - f"migration publication failed: {exc}" + "migration publication failed; the destination may be incomplete. " + f"Operator instructions: if {final_receipt.name} exists without " + f"{final_dest.name}, remove the receipt and re-run. If both exist, " + f"run `mf export`/validation to confirm the DB, then re-run after " + f"removing the destination. Cause: {exc}" ) from exc # Success-path hygiene: the temp root dir is now empty (DB file moved out). @@ -470,9 +478,9 @@ def migrate_store( if temp_root.exists(): shutil.rmtree(temp_root, ignore_errors=True) - # Final read-only verification. + # Final read-only verification (binds the published receipt to the DB). _fault("before_final_verify") - _verify_final(final_dest) + _verify_final(final_dest, receipt=receipt) _fault("after_final_verify") return receipt @@ -683,9 +691,6 @@ def _validate_temp_store(temp_root: Path, artifacts: ArtifactStore) -> None: row = conn.execute("PRAGMA integrity_check").fetchone() if row is None or row[0] != "ok": raise StorageError("temporary SQLite integrity_check failed") - counts = conn.execute( - "SELECT COUNT(*), COUNT(DISTINCT package_id) FROM events" - ).fetchone() finally: close_database(conn) # authoritative full chain validation via a store wrapper @@ -693,11 +698,7 @@ def _validate_temp_store(temp_root: Path, artifacts: ArtifactStore) -> None: store = SqliteManifestStore(temp_root, artifact_store=artifacts) try: - for package_id in sorted( - r[0] for r in store._conn.execute( - "SELECT DISTINCT package_id FROM events" - ).fetchall() - ): + for package_id in store.list_package_ids(): try: store.validate_chain(package_id, verify_artifacts=True) except ChainViolationError as exc: @@ -711,16 +712,18 @@ def _validate_temp_store(temp_root: Path, artifacts: ArtifactStore) -> None: store.close() -def _verify_final(final_dest: Path) -> None: +def _verify_final(final_dest: Path, receipt: dict | None = None) -> None: """Read-only verification of the FINAL DB FILE (not a store root). - Opens the exact file path read-only; never creates or mutates. + Opens the exact file path read-only; never creates or mutates. When a + receipt dict is provided (the one just published), its semantic identity + is validated against the final DB so a mismatched or stale receipt can + never be reported as success (ADR-0012 §11 success predicate). """ import sqlite3 - from urllib.parse import quote db = final_dest.resolve() - uri = f"file:{quote(str(db), safe='/')}?mode=ro" + uri = readonly_uri(db) conn = None try: conn = sqlite3.connect(uri, uri=True, timeout=5.0) @@ -735,6 +738,8 @@ def _verify_final(final_dest: Path) -> None: raise MigrationPublishFailedError( "final database identity mismatch after publication" ) + if receipt is not None: + _verify_receipt_against_db(final_dest, receipt, conn) except sqlite3.Error as exc: raise MigrationPublishFailedError( f"final database read-only verification failed: {exc}" @@ -744,7 +749,41 @@ def _verify_final(final_dest: Path) -> None: conn.close() -def _build_receipt(source: LegacySource, before: dict, temp_db: Path) -> dict: +def _verify_receipt_against_db( + final_dest: Path, receipt: dict, conn: sqlite3.Connection +) -> None: + """Bind the published receipt to the published DB (success predicate).""" + pkg_count = int( + conn.execute("SELECT COUNT(DISTINCT package_id) FROM events").fetchone()[0] + ) + ev_count = int(conn.execute("SELECT COUNT(*) FROM events").fetchone()[0]) + if receipt.get("destination_package_count") != pkg_count: + raise MigrationPublishFailedError( + "published receipt package count does not match final database" + ) + if receipt.get("destination_event_count") != ev_count: + raise MigrationPublishFailedError( + "published receipt event count does not match final database" + ) + if receipt.get("validation_verdict") != "PASS": + raise MigrationPublishFailedError( + "published receipt does not carry a PASS verdict" + ) + # The receipt must be the exact file that was just published. + receipt_path = final_dest.with_name(final_dest.name + ".receipt.json") + try: + on_disk = json.loads(receipt_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError, UnicodeDecodeError) as exc: + raise MigrationPublishFailedError( + f"cannot re-read published receipt {receipt_path}: {exc}" + ) from exc + if on_disk != receipt: + raise MigrationPublishFailedError( + "published receipt does not match the migration result" + ) + + +def _build_receipt(source: LegacySource, before: dict) -> dict: src_event_count = sum(len(p.events) for p in source.packages.values()) dst_event_count = src_event_count # 1:1 revision mapping return { @@ -766,17 +805,43 @@ def _build_receipt(source: LegacySource, before: dict, temp_db: Path) -> dict: def _write_durable(path: Path, data: dict) -> None: - tmp = path.with_name(path.name + ".tmp") - with open(tmp, "w", encoding="utf-8") as fh: + """Write `data` to `path` durably (fsync file before rename/link). + + The temp name is unique (caller supplies it); content is JSON + canonical, UTF-8, one object. + """ + with open(path, "w", encoding="utf-8") as fh: fh.write(json.dumps(data, sort_keys=True, separators=(",", ":"), ensure_ascii=False)) fh.flush() os.fsync(fh.fileno()) - os.replace(tmp, path) -def _atomic_replace(src: Path, dst: Path) -> None: - os.replace(src, dst) +def _publish_noclobber(src: Path, dst: Path) -> None: + """Atomically publish src to dst WITHOUT overwriting an existing dst. + + Uses the same no-clobber hard-link primitive as the artifact store + (ADR-0007): os.link fails if dst exists, so a raced destination is never + replaced. On FileExistsError the publication is aborted with a typed + error (the caller fails closed). + """ + try: + os.link(src, dst) + except FileExistsError: + raise DestinationExistsError( + f"publication destination appeared during migration: {dst}" + ) from None + except OSError as exc: + raise MigrationPublishFailedError( + f"cannot publish {dst}: {exc}" + ) from exc + # Remove the temp source after successful publication. + try: + src.unlink() + except OSError as exc: + raise MigrationPublishFailedError( + f"cannot remove temp file {src}: {exc}" + ) from exc def _fsync_dir(path: Path) -> None: diff --git a/methodfactory/migrations/v012_jsonl.py b/methodfactory/migrations/v012_jsonl.py index 6075614..634173d 100644 --- a/methodfactory/migrations/v012_jsonl.py +++ b/methodfactory/migrations/v012_jsonl.py @@ -82,6 +82,21 @@ def legacy_digest_text(content: str) -> str: return legacy_digest_bytes(content.encode("utf-8")) +def legacy_rev0_hash(package_id: str) -> str: + """Public v0.1.2 special revision-0 action hash: + sha256(legacy_canonical_json({"action": "create_package", + "package_id": package_id})).""" + return legacy_digest_json( + {"action": "create_package", "package_id": package_id} + ) + + +def legacy_hash_semantic(semantic: dict) -> str: + """Public v0.1.2 rev>0 action hash: sha256 of the six-field semantic + action in legacy hash canonicalization.""" + return legacy_digest_json(semantic) + + class LegacyEvent: """Immutable normalized record from a public v0.1.2 journal line.""" @@ -309,8 +324,23 @@ def validate(self) -> None: self._validate_package(pkg) def _validate_package(self, pkg: LegacyPackage) -> None: + # A discovered package must have a revision-0 create event. An empty + # or truncated journal (no revision 0) is not a valid v0.1.2 chain + # and must fail closed — never silently drop the package. + if not pkg.events: + raise LegacyChainInvalidError( + f"legacy journal for {pkg.package_id} is empty; " + "revision 0 create event missing" + ) + first = pkg.events[0] + if first.revision != 0: + raise LegacyChainInvalidError( + f"legacy chain break for {pkg.package_id}: first event " + f"revision {first.revision!r}, expected 0" + ) previous: dict | None = None previous_digest: str | None = None + verified_blobs: set[str] = set() for index, ev in enumerate(pkg.events): if ev.revision != index: raise LegacyChainInvalidError( @@ -323,6 +353,7 @@ def _validate_package(self, pkg: LegacyPackage) -> None: f"{index}: state_before does not match prior state_after" ) snap = ev.manifest_snapshot + self._validate_snapshot_shape(pkg.package_id, index, snap) if snap.get("state") != ev.state_after: raise LegacyChainInvalidError( f"legacy chain break for {pkg.package_id} at event index " @@ -345,15 +376,19 @@ def _validate_package(self, pkg: LegacyPackage) -> None: f"legacy chain break for {pkg.package_id} at event index " f"{index}: resulting_manifest_sha256 does not match snapshot" ) - # Validate referenced blobs exist + match digest (public semantics). + # Validate referenced blobs exist + match digest (public + # semantics). Each digest is verified once per package (the same + # blob is referenced by many snapshots). for item in snap.get("inputs", []): d = item.get("content_sha256") - if d: + if d and d not in verified_blobs: self.artifact_bytes(d) + verified_blobs.add(d) for art in snap.get("artifacts", []): d = art.get("sha256") - if d: + if d and d not in verified_blobs: self.artifact_bytes(d) + verified_blobs.add(d) previous = {"state": snap["state"], "digest": digest} previous_digest = digest if pkg.cache_status == "invalid": @@ -362,12 +397,83 @@ def _validate_package(self, pkg: LegacyPackage) -> None: "journal snapshot" ) + def _validate_snapshot_shape(self, package_id: str, index: int, snap: dict) -> None: + """Validate the public v0.1.2 manifest-snapshot container shape. + + Reconstruction (`migrate._reconstruct_*`) reads typed container keys + from the snapshot; a chain-valid-but-malformed snapshot must fail + here as a typed LegacyChainInvalidError, never leak a raw + KeyError/TypeError during migration. + """ + for key in ("package_id", "revision", "state", "schema_version", + "created_at", "updated_at", "intent", "inputs", + "objective", "artifacts", "transition"): + if key not in snap: + raise LegacyChainInvalidError( + f"legacy snapshot for {package_id} at event index {index} " + f"is missing {key!r}" + ) + if not isinstance(snap["intent"], dict) or not isinstance( + snap["intent"].get("raw"), str + ): + raise LegacyChainInvalidError( + f"legacy snapshot for {package_id} at event index {index} " + "has invalid intent" + ) + if not isinstance(snap["inputs"], list): + raise LegacyChainInvalidError( + f"legacy snapshot for {package_id} at event index {index} " + "inputs must be a list" + ) + if not isinstance(snap["objective"], dict): + raise LegacyChainInvalidError( + f"legacy snapshot for {package_id} at event index {index} " + "objective must be an object" + ) + if not isinstance(snap["artifacts"], list): + raise LegacyChainInvalidError( + f"legacy snapshot for {package_id} at event index {index} " + "artifacts must be a list" + ) + if not isinstance(snap.get("summary"), (dict, type(None))): + raise LegacyChainInvalidError( + f"legacy snapshot for {package_id} at event index {index} " + "summary must be an object or null" + ) + if not isinstance(snap["transition"], dict): + raise LegacyChainInvalidError( + f"legacy snapshot for {package_id} at event index {index} " + "transition must be an object" + ) + for item in snap["inputs"]: + if not isinstance(item, dict) or not isinstance( + item.get("input_id"), str + ): + raise LegacyChainInvalidError( + f"legacy snapshot for {package_id} at event index {index} " + "has a malformed input entry" + ) + for art in snap["artifacts"]: + if not isinstance(art, dict) or not isinstance( + art.get("artifact_id"), str + ): + raise LegacyChainInvalidError( + f"legacy snapshot for {package_id} at event index {index} " + "has a malformed artifact entry" + ) + # ── semantic source inventory (immutability proof) ───────────────── def source_inventory(self) -> dict[str, dict[str, Any]]: """Deterministic inventory over the frozen semantic source set. Excludes `.lock` files (transient coordination state). Keys are relative paths; values are {sha256, size, type}. + + Symlinks FAIL LOUDLY: a symlinked event/cache/blob is rejected at + read time already; the inventory must not silently drop or follow + them (a dropped entry would attest an incomplete source set, and a + followed entry could escape the source root). TOCTOU residual + (check-then-read) is documented honestly in ADR-0012 §12. """ inventory: dict[str, dict[str, Any]] = {} for pattern, base in ( @@ -375,20 +481,33 @@ def source_inventory(self) -> dict[str, dict[str, Any]]: ("*.json", self.packages_dir), ): for p in sorted(base.glob(pattern)): - if p.is_symlink(): - continue # already rejected during read + self._assert_not_symlink(p) rel = str(p.relative_to(self.root)) inventory[rel] = self._file_entry(p) blobs = self.artifacts_dir / "blobs" if blobs.is_dir(): for p in sorted(blobs.iterdir()): - if p.is_file() and not p.is_symlink(): + if p.is_file(): + self._assert_not_symlink(p) rel = str(p.relative_to(self.root)) inventory[rel] = self._file_entry(p) return inventory + @staticmethod + def _assert_not_symlink(p: Path) -> None: + if p.is_symlink(): + raise LegacySourceInvalidError( + f"legacy source path must not be a symlink: {p}" + ) + @staticmethod def _file_entry(p: Path) -> dict[str, Any]: + # Reject symlinks defensively even if a path changed between the + # directory walk and this read (the read follows the final link). + if p.is_symlink(): + raise LegacySourceInvalidError( + f"legacy source path must not be a symlink: {p}" + ) data = p.read_bytes() return { "sha256": legacy_digest_bytes(data), diff --git a/methodfactory/storage/sqlite.py b/methodfactory/storage/sqlite.py index c8a1d9f..260b3c9 100644 --- a/methodfactory/storage/sqlite.py +++ b/methodfactory/storage/sqlite.py @@ -220,9 +220,13 @@ def detect_presence(root: Path | str) -> StorePresence: return StorePresence.NO_STORE -def _readonly_uri(db: Path) -> str: +def readonly_uri(db: Path) -> str: """Build a read-only SQLite URI that correctly escapes path-significant - characters (spaces, Unicode, ?, #, %) — Finding 1 item 5.""" + characters (spaces, Unicode, ?, #, %) — Finding 1 item 5. + + Public helper: reused by the migration final verification and any + caller that must open an exact DB file read-only without creating it. + """ # quote() with safe='' percent-encodes everything including ? # %; # sqlite3 URI parsing then unquotes the path component. The 'file:' scheme # requires an absolute path with forward slashes for the authority form. @@ -232,6 +236,10 @@ def _readonly_uri(db: Path) -> str: return f"file:{quote(path_part, safe='/')}?mode=ro" +def _readonly_uri(db: Path) -> str: + return readonly_uri(db) + + def _connect(db: Path, read_only: bool, timeout: float = 5.0) -> sqlite3.Connection: if read_only: conn = sqlite3.connect(_readonly_uri(db), uri=True, timeout=timeout) diff --git a/methodfactory/storage/store.py b/methodfactory/storage/store.py index 927667f..bd1a4ec 100644 --- a/methodfactory/storage/store.py +++ b/methodfactory/storage/store.py @@ -665,5 +665,12 @@ def explain_latest_plan(self, package_id: str) -> list[tuple]: """EXPLAIN QUERY PLAN for the hot-path latest-event lookup.""" return explain_latest_event_plan(self._conn, package_id) + def list_package_ids(self) -> list[str]: + """Return distinct package ids in deterministic (id) order.""" + rows = self._conn.execute( + "SELECT DISTINCT package_id FROM events ORDER BY package_id" + ).fetchall() + return [r[0] for r in rows] + def close(self) -> None: close_database(self._conn) diff --git a/methodfactory/tests/_fixtures.py b/methodfactory/tests/_fixtures.py index 24fe216..701301b 100644 --- a/methodfactory/tests/_fixtures.py +++ b/methodfactory/tests/_fixtures.py @@ -4,16 +4,21 @@ v0.1.2 code from a disposable checkout of `fb5641c`. This module locates or builds such a checkout on demand and generates fixtures deterministically. -Fixture provenance is recorded (generating commit, command, source inventory). +Provenance: every fixture is generated by executing the workflow function +source (embedded verbatim) against a fresh worktree at exact commit +`fb5641cc1a3f1f54b96bba3af88ec5a1b010f4e5` (tag `v0.1.2-integrity`). The +generated journal hashes are therefore exact public-code outputs; event_ids +are UUID-random, so tests assert structure/semantics rather than exact +bytes. A `_PROVENANCE.md` is written into each generated fixture root. """ from __future__ import annotations -import datetime import json import shutil import subprocess import sys +import tempfile from pathlib import Path from typing import Callable @@ -23,6 +28,10 @@ # Location of the canonical dev checkout (used to locate the fb5641c commit). REPO_ROOT = Path(__file__).resolve().parents[2] +# Per-process checkout + script paths (parallel-runner safe via tempfile). +_FIXTURE_DIR = Path(tempfile.gettempdir()) / f"mf-v012-fixtures-{__import__('os').getpid()}" +_CHECKOUT = Path(tempfile.gettempdir()) / f"mf-v012-fixturegen-{__import__('os').getpid()}" + def _workflow_source(workflow: Callable) -> str: """Return the exact source of a workflow function for embedding.""" @@ -31,21 +40,8 @@ def _workflow_source(workflow: Callable) -> str: return inspect.getsource(workflow) -class AdvClock: - """Deterministic advancing clock (distinct timestamp per call).""" - - def __init__(self, start: str = "2026-08-07T00:00:00+00:00") -> None: - self.t = datetime.datetime.fromisoformat(start) - self._delta = datetime.timedelta(minutes=1) - - def __call__(self) -> str: - s = self.t.isoformat() - self.t += self._delta - return s - - def _find_fb5641c() -> Path | None: - """Return a path to a worktree at fb5641c, or None if unavailable.""" + """Return the repo if it contains fb5641c, else None.""" try: result = subprocess.run( ["git", "-C", str(REPO_ROOT), "cat-file", "-t", LEGACY_COMMIT], @@ -86,14 +82,15 @@ def generate_fixture( root: Path, *, workflow: Callable, - clock: Callable[[], str] | None = None, ) -> Path: """Generate a v0.1.2 fixture at `root` using the public code. - `workflow(engine, apply, root)` drives the scenario; returns nothing. + `workflow(engine, root)` drives the scenario; returns nothing. The + workflow function source is embedded into the generated script verbatim + (public-era code running against the fb5641c checkout). """ - checkout = Path("/tmp/mf-v012-fixturegen") - if not build_legacy_checkout(checkout): + root = Path(root) + if not build_legacy_checkout(_CHECKOUT): raise RuntimeError( "cannot build fb5641c checkout; fixture generation requires the " "exact public code" @@ -101,15 +98,15 @@ def generate_fixture( code = f""" import json, sys -sys.path.insert(0, {str(checkout)!r}) +sys.path.insert(0, {str(_CHECKOUT)!r}) from pathlib import Path from core.manifest.store import ManifestStore from core.adapters.artifact_store import ArtifactStore from core.engine import PipelineEngine -from core.manifest.hashing import utcnow root = Path({str(root)!r}) import shutil; shutil.rmtree(root, ignore_errors=True) +root.mkdir(parents=True, exist_ok=True) store = ManifestStore(root) engine = PipelineEngine(store, ArtifactStore(root / "artifacts")) @@ -118,11 +115,18 @@ def generate_fixture( {workflow.__name__}(engine, root) """ - # We embed the workflow by exec'ing its source; simpler: pass a script path. - script = Path("/tmp/mf-v012-gen-script.py") + _FIXTURE_DIR.mkdir(parents=True, exist_ok=True) + script = _FIXTURE_DIR / "gen-script.py" script.write_text(code, encoding="utf-8") py = sys.executable subprocess.run([py, str(script)], check=True, capture_output=True) + # Provenance record inside the fixture. + (root / "_PROVENANCE.md").write_text( + "Generated by executing the workflow against public v0.1.2 code at\n" + f"commit {LEGACY_COMMIT} (tag {LEGACY_TAG}), checkout {_CHECKOUT}.\n" + f"Workflow: {workflow.__name__}\n", + encoding="utf-8", + ) return root diff --git a/methodfactory/tests/test_migrations.py b/methodfactory/tests/test_migrations.py index cc6b955..f049bcd 100644 --- a/methodfactory/tests/test_migrations.py +++ b/methodfactory/tests/test_migrations.py @@ -115,6 +115,36 @@ def _inventory(root: Path) -> dict[str, tuple[str, int]]: return inv +def _rewrite_journal(src: Path, mutator) -> None: + """Load a legacy journal, apply `mutator(events)`, then recompute the + full legacy chain (previous/resulting hashes) so the journal still + passes the frozen v0.1.2 reader — unless the mutation itself makes a + hash mismatch the intended failure. + + Removes the cache file: after a journal mutation the cached snapshot is + stale by definition; these tests target the migration transformation + boundary, not cache semantics (cache cases have their own tests). + """ + journal = src / "events/pkg_demo_001.events.jsonl" + events = [json.loads(l) for l in journal.read_text().splitlines()] + mutator(events) + prev_hash = None + for ev in events: + snap = ev["manifest_snapshot"] + snap["previous_manifest_sha256"] = prev_hash + ev["previous_manifest_sha256"] = prev_hash + resulting = legacy_digest_json(snap) + ev["resulting_manifest_sha256"] = resulting + prev_hash = resulting + journal.write_text( + "\n".join(json.dumps(e, sort_keys=True) for e in events) + "\n", + encoding="utf-8", + ) + cache = src / "packages/pkg_demo_001.json" + if cache.exists(): + cache.unlink() + + def _open_db(path: Path) -> sqlite3.Connection: conn = sqlite3.connect(str(path)) conn.row_factory = sqlite3.Row @@ -375,6 +405,125 @@ def test_current_invalid_identifier(self): with self.assertRaises(MigrationIncompatibleError): migrate_store(src, Path(td) / "methodfactory.sqlite3") + def test_empty_journal_fails_closed(self): + """An empty/truncated journal (no revision 0) must fail + LEGACY_CHAIN_INVALID — never silently drop the package.""" + with tempfile.TemporaryDirectory() as td: + src = Path(td) / "src" + shutil.copytree(self.src, src) + (src / "events/pkg_demo_001.events.jsonl").write_text("", encoding="utf-8") + (src / "packages/pkg_demo_001.json").unlink() # absent cache + with self.assertRaises(LegacyChainInvalidError): + migrate_store(src, Path(td) / "methodfactory.sqlite3") + self.assertFalse((Path(td) / "methodfactory.sqlite3").exists()) + + def test_within_package_duplicate_event_id(self): + """Two events in the SAME package with the same event_id must fail.""" + with tempfile.TemporaryDirectory() as td: + src = Path(td) / "src" + shutil.copytree(self.src, src) + def mut(evs): + # Duplicate event_id of rev0 onto rev1 (event_id is part of + # the snapshot transition, so the chain is recomputed below). + evs[1]["event_id"] = evs[0]["event_id"] + evs[1]["manifest_snapshot"]["transition"]["last_event_id"] = evs[0]["event_id"] + _rewrite_journal(src, mut) + with self.assertRaises(MigrationIncompatibleError): + migrate_store(src, Path(td) / "methodfactory.sqlite3") + + def test_invalid_logical_path(self): + """A legacy-valid logical path that fails the current strict path + grammar (absolute path) must be MIGRATION_INCOMPATIBLE.""" + with tempfile.TemporaryDirectory() as td: + src = Path(td) / "src" + shutil.copytree(self.src, src) + def mut(evs): + # rev5 is record_draft_artifact with logical_path + # skills/x/SKILL.md; rewrite it to an absolute path. + evs[5]["manifest_snapshot"]["artifacts"][0]["logical_path"] = "/etc/passwd" + _rewrite_journal(src, mut) + with self.assertRaises(MigrationIncompatibleError): + migrate_store(src, Path(td) / "methodfactory.sqlite3") + + def test_control_characters_rejected(self): + """Control characters in a legacy-valid identifier fail current + validation with MIGRATION_INCOMPATIBLE.""" + with tempfile.TemporaryDirectory() as td: + src = Path(td) / "src" + shutil.copytree(self.src, src) + def mut(evs): + evs[1]["manifest_snapshot"]["inputs"][0]["input_id"] = "in\x01" + _rewrite_journal(src, mut) + with self.assertRaises(MigrationIncompatibleError): + migrate_store(src, Path(td) / "methodfactory.sqlite3") + + def test_source_symlink_rejected(self): + """A symlinked artifact blob fails the inventory (immutability proof + attests only real in-root files).""" + with tempfile.TemporaryDirectory() as td: + src = Path(td) / "src" + shutil.copytree(self.src, src) + blob = next((src / "artifacts/blobs").iterdir()) + blob.unlink() + blob.symlink_to(src / "events/pkg_demo_001.events.jsonl") + with self.assertRaises(LegacySourceInvalidError): + migrate_store(src, Path(td) / "methodfactory.sqlite3") + + def test_dest_root_permissions_untouched(self): + """Default destination root is the legacy source root; its mode must + NOT be chmod'd by migration (ADR-0012 §12 source immutability).""" + with tempfile.TemporaryDirectory() as td: + src = Path(td) / "src" + shutil.copytree(self.src, src) + os.chmod(src, 0o755) + before_mode = src.stat().st_mode + # dest defaults to /methodfactory.sqlite3 + migrate_store(src) + after_mode = src.stat().st_mode + self.assertEqual(before_mode, after_mode) + self.assertTrue((src / "methodfactory.sqlite3").is_file()) + + def test_receipt_without_db_is_not_success(self): + """A fault after receipt publication but before DB publication leaves + no PASS receipt behind and raises (never success).""" + old = migrate_module.FAULT_HOOK + try: + def hook(s): + if s == "before_db_replace": + raise StorageError("fault before db replace") + migrate_module.FAULT_HOOK = hook + with tempfile.TemporaryDirectory() as td: + dest = Path(td) / "methodfactory.sqlite3" + with self.assertRaises(MethodFactoryError): + migrate_store(self.src, dest) + self.assertFalse(dest.exists()) + self.assertFalse( + dest.with_name(dest.name + ".receipt.json").exists() + ) + finally: + migrate_module.FAULT_HOOK = old + + def test_export_no_clobber_raced(self): + """Export must not overwrite a destination that already exists.""" + src = _generate("standard", standard_workflow) + with tempfile.TemporaryDirectory() as td: + dest = Path(td) / "methodfactory.sqlite3" + migrate_store(src, dest) + out = Path(td) / "out.jsonl" + export_events(Path(td), out, fmt=EVENTS_V1_FORMAT) + self.assertEqual(len(out.read_text().splitlines()), 7) + # Existing destination: fail closed, content untouched. + with self.assertRaises(StorageError): + export_events(Path(td), out, fmt=EVENTS_V1_FORMAT) + self.assertEqual(len(out.read_text().splitlines()), 7) + # A *different* path that appears after the exists-check is + # protected by the no-clobber link. + raced = Path(td) / "raced.jsonl" + raced.write_text("existing", encoding="utf-8") + with self.assertRaises(StorageError): + export_events(Path(td), raced, fmt=EVENTS_V1_FORMAT) + self.assertEqual(raced.read_text(), "existing") + def test_destination_exists(self): with tempfile.TemporaryDirectory() as td: dest = Path(td) / "methodfactory.sqlite3" @@ -720,6 +869,42 @@ def test_legacy_hash_vs_line_serializer_distinction(self): self.assertIn(b"\\u00e9", legacy_canonical_json(ev)) self.assertNotIn(b'"b": "', legacy_canonical_json(ev)) + def test_legacy_export_revalidates_under_frozen_reader(self): + """bug-o1 regression: the exported legacy journal must pass the + frozen v0.1.2 reader's own validation (chain + hashes), proving the + snapshot previous_manifest_sha256 is in LEGACY hash space.""" + out = Path(self.td.name) / "revalidate.jsonl" + export_events(self.root, out, fmt=LEGACY_JSONL_FORMAT) + # Reconstruct a legacy store layout from the export + migrated blobs. + legacy_root = Path(self.td.name) / "legacy-revalidate" + (legacy_root / "events").mkdir(parents=True, exist_ok=True) + (legacy_root / "packages").mkdir(parents=True, exist_ok=True) + (legacy_root / "artifacts" / "blobs").mkdir(parents=True, exist_ok=True) + shutil.copy(out, legacy_root / "events/pkg_demo_001.events.jsonl") + for blob in (self.root / "blobs").iterdir(): + shutil.copy(blob, legacy_root / "artifacts" / "blobs" / blob.name) + ls = LegacySource(legacy_root) + ls.validate() # must not raise + + def test_legacy_export_revalidates_non_ascii(self): + """Same regression for a store with non-ASCII content (ensure_ascii + divergence affects every legacy hash).""" + src = _generate("non_ascii", non_ascii_workflow) + with tempfile.TemporaryDirectory() as td: + root = Path(td) + migrate_store(src, root / "methodfactory.sqlite3") + out = root / "legacy.jsonl" + export_events(root, out, fmt=LEGACY_JSONL_FORMAT) + legacy_root = root / "legacy-revalidate" + (legacy_root / "events").mkdir(parents=True, exist_ok=True) + (legacy_root / "packages").mkdir(parents=True, exist_ok=True) + (legacy_root / "artifacts" / "blobs").mkdir(parents=True, exist_ok=True) + shutil.copy(out, legacy_root / "events/pkg_unicode_001.events.jsonl") + for blob in (root / "blobs").iterdir(): + shutil.copy(blob, legacy_root / "artifacts" / "blobs" / blob.name) + ls = LegacySource(legacy_root) + ls.validate() # must not raise + def test_export_no_mutation(self): before = _inventory(self.root) out = Path(self.td.name) / "n.jsonl" From 3a8d15890b56d2e871e04204f4de23d5cc733083 Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Sat, 8 Aug 2026 06:37:31 +0700 Subject: [PATCH 29/41] fix(migration): close round-2 review - root-mode invariant, snapshot shape completeness, workflow-generated boundary fixtures MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second code-review pass (34ba9d1..54ac313) returned 17 findings (3 major / 10 minor / 4 nit). This commit closes the blocking set and the actionable minors: - turn-1 (major): migration no longer publishes an unreopenable store when an existing destination root is not 0700. Root mode is verified before publication and fails closed with operator instructions; the legacy source root is never chmod'd (ADR-0012 §12 + §D reconciled). - bug-1/sec-1 (major): _validate_snapshot_shape extended to require every field reconstruction subscripts directly (objective.statement, input kind/source/disposition/content_sha256, artifact kind/logical_path/sha256, summary confirmation fields, transition fields) - malformed snapshots now raise typed LegacyChainInvalidError, never raw KeyError/TypeError. CLI _BOUNDARY_NATIVES gains LookupError + AttributeError. - q-1 (major): current-invalid logical-path and control-char fixtures are now WORKFLOW-GENERATED via exact fb5641c code (invalid_logical_path_ workflow, control_char_identifier_workflow) so the failures reach the CURRENT boundary and the tests assert 'current-invalid' in the message. - bug-3: post-link temp-unlink failure is non-fatal (published-but-unclean); _publish_noclobber and export publication no longer delete the receipt or report failure after a successful link. - bug-4: list_package_ids wraps sqlite3.Error into StorageError. - sec-2/q-6: receipt temp written via os.open O_EXCL 0600. - q-2: export link-race branch exercised directly (monkeypatched os.link). - q-3/q-4: stale docstrings updated (no-clobber link publication). - q-5: readonly_uri promoted; _readonly_uri shim removed. - q-7: hash helpers renamed legacy_action_hash_rev0/_semantic. - q-8: _legacy_digest_text wrapper removed. - perf-4: fixture checkout reuse when already at fb5641c. - turn-2: test_source_changed_detected uses a per-test copy. Full suite: 412 tests green. --- methodfactory/cli.py | 2 + methodfactory/migrations/export.py | 23 +++-- methodfactory/migrations/migrate.py | 68 +++++++++++---- methodfactory/migrations/v012_jsonl.py | 69 ++++++++++++--- methodfactory/storage/sqlite.py | 6 +- methodfactory/storage/store.py | 9 +- methodfactory/tests/_fixtures.py | 99 +++++++++++++++++++++- methodfactory/tests/test_migrations.py | 113 +++++++++++++++++++------ 8 files changed, 312 insertions(+), 77 deletions(-) diff --git a/methodfactory/cli.py b/methodfactory/cli.py index ecd3137..00a8393 100644 --- a/methodfactory/cli.py +++ b/methodfactory/cli.py @@ -30,6 +30,8 @@ sqlite3.Error, json.JSONDecodeError, UnicodeError, + LookupError, # KeyError / IndexError residual leaks + AttributeError, TypeError, ValueError, RecursionError, diff --git a/methodfactory/migrations/export.py b/methodfactory/migrations/export.py index 28b4128..46dd44f 100644 --- a/methodfactory/migrations/export.py +++ b/methodfactory/migrations/export.py @@ -47,9 +47,10 @@ ) from .v012_jsonl import ( legacy_digest_json, - legacy_hash_semantic, + legacy_digest_text, + legacy_action_hash_semantic, legacy_line_json, - legacy_rev0_hash, + legacy_action_hash_rev0, ) EVENTS_V1_FORMAT = "method-factory-events-v1" @@ -132,9 +133,9 @@ def _legacy_event_object(row: dict, prev_legacy_hash: str | None) -> dict: # Legacy action hash: rev0 special reduced; rev>0 legacy canonical of # semantic action (six fields). if row["revision"] == 0: - action_hash = legacy_rev0_hash(row["package_id"]) + action_hash = legacy_action_hash_rev0(row["package_id"]) else: - action_hash = legacy_hash_semantic(semantic) + action_hash = legacy_action_hash_semantic(semantic) return { "event_id": row["event_id"], @@ -170,7 +171,7 @@ def _to_legacy_manifest(manifest: dict, prev_legacy_hash: str | None = None) -> body = _render_summary(m) m["summary"] = { "content": body, - "canonical_sha256": summary.get("digest") or _legacy_digest_text(body), + "canonical_sha256": summary.get("digest") or legacy_digest_text(body), "presented_at": summary.get("presented_at"), "confirmation": summary.get("confirmation"), } @@ -183,12 +184,6 @@ def _render_summary(manifest: dict) -> str: return render_summary(manifest) -def _legacy_digest_text(content: str) -> str: - from .v012_jsonl import legacy_digest_text - - return legacy_digest_text(content) - - # ── public API ──────────────────────────────────────────────────────── def export_events( store_root: str | Path, @@ -269,10 +264,12 @@ def export_events( ) from None except OSError as exc: raise StorageError(f"cannot publish export: {exc}") from exc + # Best-effort temp cleanup after successful publication; an orphan + # temp is harmless and must not falsely fail the export. try: tmp.unlink() - except OSError as exc: - raise StorageError(f"cannot remove export temp: {exc}") from exc + except OSError: + pass except BaseException: try: tmp.unlink() diff --git a/methodfactory/migrations/migrate.py b/methodfactory/migrations/migrate.py index 377662e..4dda133 100644 --- a/methodfactory/migrations/migrate.py +++ b/methodfactory/migrations/migrate.py @@ -29,10 +29,11 @@ 15. durably write temp receipt; 16. FINAL source inventory/hash (AFTER) — require exact equality with step 3; 17. ONLY THEN enter publication: - a. durable final receipt publication (os.replace temp -> final; dir fsync); - b. durable atomic final DB publication (os.replace temp DB -> final; + a. durable final receipt publication (no-clobber os.link temp -> final; dir fsync); -18. final read-only verification. + b. durable atomic final DB publication (no-clobber os.link temp DB -> + final; dir fsync); +18. final read-only verification (binds the published receipt to the DB). A receipt alone is NOT success. Success requires: final DB exists; matching final receipt exists; same migration identity; final read-only validation @@ -84,7 +85,7 @@ LEGACY_COMMIT, LEGACY_TAG, LegacySource, - legacy_hash_semantic, + legacy_action_hash_semantic, ) # Receipt format identity. @@ -137,7 +138,7 @@ def _candidate_hashes(source: LegacySource, pkg, ev, candidates: list[dict]) -> """Return candidates whose legacy hash matches the stored legacy hash.""" matches = [] for cand in candidates: - if legacy_hash_semantic(cand) == ev.action_sha256: + if legacy_action_hash_semantic(cand) == ev.action_sha256: matches.append(cand) return matches @@ -374,7 +375,11 @@ def migrate_store( # Never chmod an EXISTING destination root: when --dest is omitted the # default root IS the legacy source root, and mutating its permissions # would violate source immutability (ADR-0012 §12). Only newly-created - # roots get the private mode. + # roots get the private mode. An EXISTING root must already satisfy the + # current store-root invariant (0700, ADR-0012 §D): otherwise the + # published store could not be reopened by the standard public API, so + # migration fails closed with operator instructions rather than + # publishing an unreopenable store. final_root = final_dest.parent root_existed = final_root.is_dir() try: @@ -386,6 +391,20 @@ def migrate_store( raise StorageError( f"cannot create destination parent directory: {exc}" ) from exc + if root_existed: + try: + root_mode = final_root.stat().st_mode & 0o777 + except OSError as exc: + raise StorageError( + f"cannot stat destination root {final_root}: {exc}" + ) from exc + if root_mode != 0o700: + raise StorageError( + f"destination root {final_root} has mode {root_mode:03o}, " + "expected 0700 (ADR-0012 §D store-root invariant). Fix the " + "root permissions or pass an explicit --dest under a private " + "root; the legacy source root is never chmod'd by migration." + ) temp_root = final_root / f".{final_dest.name}.tmp.{uuid.uuid4().hex}" # Build the modern store. Blobs publish to the FINAL artifact store root @@ -805,16 +824,26 @@ def _build_receipt(source: LegacySource, before: dict) -> dict: def _write_durable(path: Path, data: dict) -> None: - """Write `data` to `path` durably (fsync file before rename/link). + """Write `data` durably to the caller-supplied unique temp path (fsync). - The temp name is unique (caller supplies it); content is JSON + The caller publishes via `_publish_noclobber`; this function only + ensures the temp content is on disk before publication. Content is JSON canonical, UTF-8, one object. """ - with open(path, "w", encoding="utf-8") as fh: - fh.write(json.dumps(data, sort_keys=True, separators=(",", ":"), - ensure_ascii=False)) - fh.flush() - os.fsync(fh.fileno()) + # O_EXCL: the temp path must never pre-exist or follow a raced symlink. + fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + try: + with os.fdopen(fd, "w", encoding="utf-8") as fh: + fh.write(json.dumps(data, sort_keys=True, separators=(",", ":"), + ensure_ascii=False)) + fh.flush() + os.fsync(fh.fileno()) + except BaseException: + try: + path.unlink() + except OSError: + pass + raise def _publish_noclobber(src: Path, dst: Path) -> None: @@ -824,6 +853,11 @@ def _publish_noclobber(src: Path, dst: Path) -> None: (ADR-0007): os.link fails if dst exists, so a raced destination is never replaced. On FileExistsError the publication is aborted with a typed error (the caller fails closed). + + After a SUCCESSFUL link the destination is published; the temp source + unlink is best-effort. A temp-unlink failure is NOT fatal: an orphan + `.tmp.*` file is harmless and a re-raise here would falsely report the + publication as failed while the destination is already complete. """ try: os.link(src, dst) @@ -835,13 +869,11 @@ def _publish_noclobber(src: Path, dst: Path) -> None: raise MigrationPublishFailedError( f"cannot publish {dst}: {exc}" ) from exc - # Remove the temp source after successful publication. + # Best-effort cleanup of the temp source after successful publication. try: src.unlink() - except OSError as exc: - raise MigrationPublishFailedError( - f"cannot remove temp file {src}: {exc}" - ) from exc + except OSError: + pass # published-but-unclean; orphan temp is harmless def _fsync_dir(path: Path) -> None: diff --git a/methodfactory/migrations/v012_jsonl.py b/methodfactory/migrations/v012_jsonl.py index 634173d..570864c 100644 --- a/methodfactory/migrations/v012_jsonl.py +++ b/methodfactory/migrations/v012_jsonl.py @@ -82,7 +82,7 @@ def legacy_digest_text(content: str) -> str: return legacy_digest_bytes(content.encode("utf-8")) -def legacy_rev0_hash(package_id: str) -> str: +def legacy_action_hash_rev0(package_id: str) -> str: """Public v0.1.2 special revision-0 action hash: sha256(legacy_canonical_json({"action": "create_package", "package_id": package_id})).""" @@ -91,7 +91,7 @@ def legacy_rev0_hash(package_id: str) -> str: ) -def legacy_hash_semantic(semantic: dict) -> str: +def legacy_action_hash_semantic(semantic: dict) -> str: """Public v0.1.2 rev>0 action hash: sha256 of the six-field semantic action in legacy hash canonicalization.""" return legacy_digest_json(semantic) @@ -445,21 +445,70 @@ def _validate_snapshot_shape(self, package_id: str, index: int, snap: dict) -> N f"legacy snapshot for {package_id} at event index {index} " "transition must be an object" ) + # Objective fields read by reconstruction (set_objective). + if not isinstance(snap["objective"].get("statement"), str): + raise LegacyChainInvalidError( + f"legacy snapshot for {package_id} at event index {index} " + "objective.statement must be a string" + ) + # Input fields read by reconstruction (record_input): every entry + # needs input_id, kind, source, disposition, and content_sha256. for item in snap["inputs"]: - if not isinstance(item, dict) or not isinstance( - item.get("input_id"), str - ): + if not isinstance(item, dict): raise LegacyChainInvalidError( f"legacy snapshot for {package_id} at event index {index} " - "has a malformed input entry" + "has a non-object input entry" ) + for field in ("input_id", "kind", "source", "disposition", + "content_sha256"): + if not isinstance(item.get(field), str): + raise LegacyChainInvalidError( + f"legacy snapshot for {package_id} at event index " + f"{index} input entry is missing string field " + f"{field!r}" + ) + # Artifact fields read by reconstruction (record_draft_artifact): + # artifact_id, kind, logical_path, sha256. for art in snap["artifacts"]: - if not isinstance(art, dict) or not isinstance( - art.get("artifact_id"), str - ): + if not isinstance(art, dict): + raise LegacyChainInvalidError( + f"legacy snapshot for {package_id} at event index {index} " + "has a non-object artifact entry" + ) + for field in ("artifact_id", "kind", "logical_path", "sha256"): + if not isinstance(art.get(field), str): + raise LegacyChainInvalidError( + f"legacy snapshot for {package_id} at event index " + f"{index} artifact entry is missing string field " + f"{field!r}" + ) + # Summary fields read by reconstruction (confirm_summary). + summary = snap.get("summary") + if isinstance(summary, dict): + if not isinstance(summary.get("canonical_sha256"), str): + raise LegacyChainInvalidError( + f"legacy snapshot for {package_id} at event index {index} " + "summary.canonical_sha256 must be a string" + ) + conf = summary.get("confirmation") + if not isinstance(conf, dict): + raise LegacyChainInvalidError( + f"legacy snapshot for {package_id} at event index {index} " + "summary.confirmation must be an object" + ) + for field in ("status", "confirmed_at", "operator_id", + "confirmed_summary_sha256"): + if field not in conf: + raise LegacyChainInvalidError( + f"legacy snapshot for {package_id} at event index " + f"{index} summary.confirmation is missing {field!r}" + ) + # Transition fields read by equivalence checks. + for field in ("last_event_id", "last_action_id"): + if field not in snap["transition"]: raise LegacyChainInvalidError( f"legacy snapshot for {package_id} at event index {index} " - "has a malformed artifact entry" + f"transition is missing {field!r}" ) # ── semantic source inventory (immutability proof) ───────────────── diff --git a/methodfactory/storage/sqlite.py b/methodfactory/storage/sqlite.py index 260b3c9..c15a5f4 100644 --- a/methodfactory/storage/sqlite.py +++ b/methodfactory/storage/sqlite.py @@ -236,13 +236,9 @@ def readonly_uri(db: Path) -> str: return f"file:{quote(path_part, safe='/')}?mode=ro" -def _readonly_uri(db: Path) -> str: - return readonly_uri(db) - - def _connect(db: Path, read_only: bool, timeout: float = 5.0) -> sqlite3.Connection: if read_only: - conn = sqlite3.connect(_readonly_uri(db), uri=True, timeout=timeout) + conn = sqlite3.connect(readonly_uri(db), uri=True, timeout=timeout) else: conn = sqlite3.connect(str(db), timeout=timeout) conn.row_factory = sqlite3.Row diff --git a/methodfactory/storage/store.py b/methodfactory/storage/store.py index bd1a4ec..f5229bb 100644 --- a/methodfactory/storage/store.py +++ b/methodfactory/storage/store.py @@ -667,9 +667,12 @@ def explain_latest_plan(self, package_id: str) -> list[tuple]: def list_package_ids(self) -> list[str]: """Return distinct package ids in deterministic (id) order.""" - rows = self._conn.execute( - "SELECT DISTINCT package_id FROM events ORDER BY package_id" - ).fetchall() + try: + rows = self._conn.execute( + "SELECT DISTINCT package_id FROM events ORDER BY package_id" + ).fetchall() + except sqlite3.Error as exc: + raise StorageError(f"cannot list package ids: {exc}") from exc return [r[0] for r in rows] def close(self) -> None: diff --git a/methodfactory/tests/_fixtures.py b/methodfactory/tests/_fixtures.py index 701301b..7009385 100644 --- a/methodfactory/tests/_fixtures.py +++ b/methodfactory/tests/_fixtures.py @@ -56,10 +56,22 @@ def _find_fb5641c() -> Path | None: def build_legacy_checkout(dest: Path) -> Path | None: - """Create a disposable worktree of fb5641c. Returns the checkout root.""" + """Create a disposable worktree of fb5641c. Returns the checkout root. + + Skips rebuild when the per-process checkout already exists at the exact + commit (parallel-runner safe; avoids two git subprocesses per fixture). + """ repo = _find_fb5641c() if repo is None: return None + # Fast path: checkout exists and is at the frozen commit. + if dest.is_dir(): + probe = subprocess.run( + ["git", "-C", str(dest), "rev-parse", "HEAD"], + capture_output=True, text=True, + ) + if probe.returncode == 0 and probe.stdout.strip() == LEGACY_COMMIT: + return dest # If the path is already a registered worktree, remove it cleanly first. subprocess.run( ["git", "-C", str(repo), "worktree", "remove", "--force", str(dest)], @@ -547,3 +559,88 @@ def apply(envdict): "payload": {"input_id": "in 1", "kind": "text", "content": "hello", "source": "operator", "disposition": "incorporated"}, }) + + +def invalid_logical_path_workflow(engine, root: Path) -> None: + """Artifact logical_path that is public-v0.1.2-valid but + current-invalid (absolute path).""" + import datetime + + class Clock: + def __init__(self, start: str = "2026-08-07T00:00:00+00:00") -> None: + self.t = datetime.datetime.fromisoformat(start) + self._delta = datetime.timedelta(minutes=1) + + def __call__(self) -> str: + s = self.t.isoformat() + self.t += self._delta + return s + + engine._now = Clock() + + def apply(envdict): + return engine.apply_json(json.dumps(envdict)) + + engine.create_package("pkg_badpath_001", "bad logical path") + apply({ + "protocol_version": "0.1", "action_id": "act_in_1", + "package_id": "pkg_badpath_001", "expected_revision": 0, + "action": "record_input", "basis": {}, + "payload": {"input_id": "in_1", "kind": "text", "content": "hello", + "source": "operator", "disposition": "incorporated"}, + }) + apply({ + "protocol_version": "0.1", "action_id": "act_obj_1", + "package_id": "pkg_badpath_001", "expected_revision": 1, + "action": "set_objective", "basis": {}, + "payload": {"statement": "Build a skill"}, + }) + m2 = apply({ + "protocol_version": "0.1", "action_id": "act_prep_1", + "package_id": "pkg_badpath_001", "expected_revision": 2, + "action": "prepare_summary", "basis": {}, "payload": {}, + }) + apply({ + "protocol_version": "0.1", "action_id": "act_conf_1", + "package_id": "pkg_badpath_001", "expected_revision": 3, + "action": "confirm_summary", + "basis": {"summary_sha256": m2.manifest["summary"]["canonical_sha256"]}, + "payload": {"operator_id": "vincent"}, + }) + apply({ + "protocol_version": "0.1", "action_id": "act_art_1", + "package_id": "pkg_badpath_001", "expected_revision": 4, + "action": "record_draft_artifact", "basis": {}, + "payload": {"artifact_id": "art_1", "kind": "skill", + "logical_path": "/etc/passwd", "content": "body"}, + }) + + +def control_char_identifier_workflow(engine, root: Path) -> None: + """input_id containing a control character (public-valid, + current-invalid under the compatibility matrix).""" + import datetime + + class Clock: + def __init__(self, start: str = "2026-08-07T00:00:00+00:00") -> None: + self.t = datetime.datetime.fromisoformat(start) + self._delta = datetime.timedelta(minutes=1) + + def __call__(self) -> str: + s = self.t.isoformat() + self.t += self._delta + return s + + engine._now = Clock() + + def apply(envdict): + return engine.apply_json(json.dumps(envdict)) + + engine.create_package("pkg_ctrlchar_001", "control char identifier") + apply({ + "protocol_version": "0.1", "action_id": "act_in_1", + "package_id": "pkg_ctrlchar_001", "expected_revision": 0, + "action": "record_input", "basis": {}, + "payload": {"input_id": "in\x01", "kind": "text", "content": "hello", + "source": "operator", "disposition": "incorporated"}, + }) diff --git a/methodfactory/tests/test_migrations.py b/methodfactory/tests/test_migrations.py index f049bcd..9caf21c 100644 --- a/methodfactory/tests/test_migrations.py +++ b/methodfactory/tests/test_migrations.py @@ -73,8 +73,10 @@ from ._fixtures import ( all_action_families_workflow, cancel_arbitrary_reason_workflow, + control_char_identifier_workflow, current_invalid_identifier_workflow, generate_fixture, + invalid_logical_path_workflow, non_ascii_workflow, optional_omitted_workflow, overlimit_input_workflow, @@ -121,6 +123,11 @@ def _rewrite_journal(src: Path, mutator) -> None: passes the frozen v0.1.2 reader — unless the mutation itself makes a hash mismatch the intended failure. + NOTE: mutations of fields that feed the legacy semantic-action hash + (input_id, logical_path, kind, content) are NOT recomputable here + (blob content is external); those cases use workflow-generated + fixtures (current_invalid_*_workflow) instead of hand-editing. + Removes the cache file: after a journal mutation the cached snapshot is stale by definition; these tests target the migration transformation boundary, not cache semantics (cache cases have their own tests). @@ -433,29 +440,26 @@ def mut(evs): def test_invalid_logical_path(self): """A legacy-valid logical path that fails the current strict path - grammar (absolute path) must be MIGRATION_INCOMPATIBLE.""" + grammar (absolute path) must be MIGRATION_INCOMPATIBLE — proven by a + workflow-generated legacy fixture, not hand-editing (the mutation + must survive reconstruction and be rejected by the CURRENT + boundary).""" + src = _generate("badpath", invalid_logical_path_workflow) with tempfile.TemporaryDirectory() as td: - src = Path(td) / "src" - shutil.copytree(self.src, src) - def mut(evs): - # rev5 is record_draft_artifact with logical_path - # skills/x/SKILL.md; rewrite it to an absolute path. - evs[5]["manifest_snapshot"]["artifacts"][0]["logical_path"] = "/etc/passwd" - _rewrite_journal(src, mut) - with self.assertRaises(MigrationIncompatibleError): + with self.assertRaises(MigrationIncompatibleError) as ctx: migrate_store(src, Path(td) / "methodfactory.sqlite3") + # the failure must be a CURRENT-boundary rejection, not a + # reconstruction ambiguity + self.assertIn("current-invalid", str(ctx.exception)) def test_control_characters_rejected(self): """Control characters in a legacy-valid identifier fail current - validation with MIGRATION_INCOMPATIBLE.""" + validation with MIGRATION_INCOMPATIBLE (workflow-generated).""" + src = _generate("ctrlchar", control_char_identifier_workflow) with tempfile.TemporaryDirectory() as td: - src = Path(td) / "src" - shutil.copytree(self.src, src) - def mut(evs): - evs[1]["manifest_snapshot"]["inputs"][0]["input_id"] = "in\x01" - _rewrite_journal(src, mut) - with self.assertRaises(MigrationIncompatibleError): + with self.assertRaises(MigrationIncompatibleError) as ctx: migrate_store(src, Path(td) / "methodfactory.sqlite3") + self.assertIn("current-invalid", str(ctx.exception)) def test_source_symlink_rejected(self): """A symlinked artifact blob fails the inventory (immutability proof @@ -471,17 +475,37 @@ def test_source_symlink_rejected(self): def test_dest_root_permissions_untouched(self): """Default destination root is the legacy source root; its mode must - NOT be chmod'd by migration (ADR-0012 §12 source immutability).""" + NOT be chmod'd by migration (ADR-0012 §12 source immutability). A + legacy root created by the public code is 0700, so the migrated + store must be reopenable.""" with tempfile.TemporaryDirectory() as td: src = Path(td) / "src" shutil.copytree(self.src, src) - os.chmod(src, 0o755) + os.chmod(src, 0o700) before_mode = src.stat().st_mode # dest defaults to /methodfactory.sqlite3 migrate_store(src) after_mode = src.stat().st_mode self.assertEqual(before_mode, after_mode) self.assertTrue((src / "methodfactory.sqlite3").is_file()) + # the published store must reopen via the standard API + from methodfactory.migrations.export import export_events + n = export_events(src, None, fmt="method-factory-events-v1") + self.assertEqual(n, 7) + + def test_dest_root_mode_invariant_fails_closed(self): + """An existing destination root that is NOT 0700 must fail closed + with a typed error and publish nothing — never chmod the legacy + source, never publish an unreopenable store.""" + with tempfile.TemporaryDirectory() as td: + src = Path(td) / "src" + shutil.copytree(self.src, src) + os.chmod(src, 0o755) + before_mode = src.stat().st_mode + with self.assertRaises(StorageError): + migrate_store(src) # default dest = source root + self.assertEqual(src.stat().st_mode, before_mode) + self.assertFalse((src / "methodfactory.sqlite3").exists()) def test_receipt_without_db_is_not_success(self): """A fault after receipt publication but before DB publication leaves @@ -524,6 +548,37 @@ def test_export_no_clobber_raced(self): export_events(Path(td), raced, fmt=EVENTS_V1_FORMAT) self.assertEqual(raced.read_text(), "existing") + def test_export_link_race_branch(self): + """The FileExistsError branch of the no-clobber publication is + exercised directly: a destination appearing between the exists-check + and os.link must abort typed, leaving no temp and no overwrite.""" + import methodfactory.migrations.export as exp + + src = _generate("standard", standard_workflow) + with tempfile.TemporaryDirectory() as td: + root = Path(td) + migrate_store(src, root / "methodfactory.sqlite3") + raced = root / "raced.jsonl" + original_link = exp.os.link + original_exists = Path.exists + + def fake_link(src_path, dst_path): + # simulate the race: destination appears right before link + raced.write_text("existing", encoding="utf-8") + raise FileExistsError("simulated race") + + exp.os.link = fake_link + try: + with self.assertRaises(StorageError) as ctx: + export_events(root, raced, fmt=EVENTS_V1_FORMAT) + self.assertIn("appeared during export", str(ctx.exception)) + finally: + exp.os.link = original_link + self.assertEqual(raced.read_text(), "existing") + # no temp leftovers + leftovers = [p for p in root.iterdir() if ".tmp." in p.name] + self.assertEqual(leftovers, []) + def test_destination_exists(self): with tempfile.TemporaryDirectory() as td: dest = Path(td) / "methodfactory.sqlite3" @@ -751,20 +806,24 @@ def test_fault_matrix(self): def test_source_changed_detected(self): """Mutate the source between the two inventory passes -> SOURCE_CHANGED, - no publication.""" + no publication. Uses a per-test COPY so the shared class fixture is + never polluted.""" old = migrate_module.FAULT_HOOK try: - def hook(s): - if s == "after_source_inventory_before": - # mutate a source file after the BEFORE inventory - journal = self.src / "events/pkg_demo_001.events.jsonl" - with open(journal, "a", encoding="utf-8") as fh: - fh.write("\n") - migrate_module.FAULT_HOOK = hook with tempfile.TemporaryDirectory() as td: + src = Path(td) / "src" + shutil.copytree(self.src, src) + + def hook(s): + if s == "after_source_inventory_before": + # mutate a source file after the BEFORE inventory + journal = src / "events/pkg_demo_001.events.jsonl" + with open(journal, "a", encoding="utf-8") as fh: + fh.write("\n") + migrate_module.FAULT_HOOK = hook dest = Path(td) / "methodfactory.sqlite3" with self.assertRaises(SourceChangedError): - migrate_store(self.src, dest) + migrate_store(src, dest) self.assertFalse(dest.exists()) self.assertFalse( dest.with_name(dest.name + ".receipt.json").exists() From 9b718d602fcaf9edfef5cf59a700d2973d8697ad Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Sat, 8 Aug 2026 07:22:46 +0700 Subject: [PATCH 30/41] fix(migration): close round-3 review - typed snapshot container guards, no raw IndexError/AttributeError Third code-review pass (54ac313..3a8d158) returned 2 blocking majors: F1 (empty inputs/artifacts containers leaked raw IndexError from reconstruction) and F2 (truthy non-string summary.content leaked raw AttributeError from the equivalence byte comparison). Both now fail typed: - v012_jsonl._validate_snapshot_shape is action-aware: record_input requires >=1 input entry; record_draft_artifact requires >=1 artifact entry; summary.content must be str or null. - Regression tests for all four repro cases (empty inputs, empty artifacts, content=42, content=dict) assert LegacyChainInvalidError. - migrate_store root creation uses atomic mkdir(mode=0700) with FileExistsError -> pre-existing (no TOCTOU chmod of a racer's dir). - test fixture docstring corrected; dead variable removed. Full suite: 415 tests green. --- methodfactory/migrations/migrate.py | 11 +++++-- methodfactory/migrations/v012_jsonl.py | 31 +++++++++++++++--- methodfactory/tests/test_migrations.py | 44 ++++++++++++++++++++++++-- 3 files changed, 78 insertions(+), 8 deletions(-) diff --git a/methodfactory/migrations/migrate.py b/methodfactory/migrations/migrate.py index 4dda133..2c36789 100644 --- a/methodfactory/migrations/migrate.py +++ b/methodfactory/migrations/migrate.py @@ -383,8 +383,15 @@ def migrate_store( final_root = final_dest.parent root_existed = final_root.is_dir() try: - final_root.mkdir(parents=True, exist_ok=True) - # chmod only when this call actually created the root directory. + if root_existed: + final_root.mkdir(parents=True, exist_ok=True) # no-op; parents may be missing + else: + # Atomically claim the leaf; a racer's leaf is treated as + # pre-existing (never chmod a directory we did not create). + try: + final_root.mkdir(parents=True, mode=0o700) + except FileExistsError: + root_existed = True if not root_existed: os.chmod(final_root, 0o700) except OSError as exc: diff --git a/methodfactory/migrations/v012_jsonl.py b/methodfactory/migrations/v012_jsonl.py index 570864c..b03251b 100644 --- a/methodfactory/migrations/v012_jsonl.py +++ b/methodfactory/migrations/v012_jsonl.py @@ -353,7 +353,7 @@ def _validate_package(self, pkg: LegacyPackage) -> None: f"{index}: state_before does not match prior state_after" ) snap = ev.manifest_snapshot - self._validate_snapshot_shape(pkg.package_id, index, snap) + self._validate_snapshot_shape(pkg.package_id, index, ev.action, snap) if snap.get("state") != ev.state_after: raise LegacyChainInvalidError( f"legacy chain break for {pkg.package_id} at event index " @@ -397,13 +397,14 @@ def _validate_package(self, pkg: LegacyPackage) -> None: "journal snapshot" ) - def _validate_snapshot_shape(self, package_id: str, index: int, snap: dict) -> None: + def _validate_snapshot_shape(self, package_id: str, index: int, action: str, + snap: dict) -> None: """Validate the public v0.1.2 manifest-snapshot container shape. Reconstruction (`migrate._reconstruct_*`) reads typed container keys from the snapshot; a chain-valid-but-malformed snapshot must fail here as a typed LegacyChainInvalidError, never leak a raw - KeyError/TypeError during migration. + KeyError/TypeError/IndexError/AttributeError during migration. """ for key in ("package_id", "revision", "state", "schema_version", "created_at", "updated_at", "intent", "inputs", @@ -445,6 +446,20 @@ def _validate_snapshot_shape(self, package_id: str, index: int, snap: dict) -> N f"legacy snapshot for {package_id} at event index {index} " "transition must be an object" ) + # Action-aware container requirements: reconstruction subscripts + # snap["inputs"][-1] for record_input and snap["artifacts"][-1] for + # record_draft_artifact. An empty container for those actions would + # leak a raw IndexError; fail typed instead. + if action == "record_input" and not snap["inputs"]: + raise LegacyChainInvalidError( + f"legacy snapshot for {package_id} at event index {index} " + "record_input requires at least one input entry" + ) + if action == "record_draft_artifact" and not snap["artifacts"]: + raise LegacyChainInvalidError( + f"legacy snapshot for {package_id} at event index {index} " + "record_draft_artifact requires at least one artifact entry" + ) # Objective fields read by reconstruction (set_objective). if not isinstance(snap["objective"].get("statement"), str): raise LegacyChainInvalidError( @@ -482,7 +497,8 @@ def _validate_snapshot_shape(self, package_id: str, index: int, snap: dict) -> N f"{index} artifact entry is missing string field " f"{field!r}" ) - # Summary fields read by reconstruction (confirm_summary). + # Summary fields read by reconstruction (confirm_summary) and by the + # equivalence check (summary body byte comparison). summary = snap.get("summary") if isinstance(summary, dict): if not isinstance(summary.get("canonical_sha256"), str): @@ -490,6 +506,13 @@ def _validate_snapshot_shape(self, package_id: str, index: int, snap: dict) -> N f"legacy snapshot for {package_id} at event index {index} " "summary.canonical_sha256 must be a string" ) + if summary.get("content") is not None and not isinstance( + summary.get("content"), str + ): + raise LegacyChainInvalidError( + f"legacy snapshot for {package_id} at event index {index} " + "summary.content must be a string or null" + ) conf = summary.get("confirmation") if not isinstance(conf, dict): raise LegacyChainInvalidError( diff --git a/methodfactory/tests/test_migrations.py b/methodfactory/tests/test_migrations.py index 9caf21c..8e1d7ec 100644 --- a/methodfactory/tests/test_migrations.py +++ b/methodfactory/tests/test_migrations.py @@ -89,7 +89,12 @@ def _generate(name: str, workflow) -> Path: - """Generate (once per suite) a fb5641c-origin fixture into FIXTURE_DIR.""" + """Generate a fb5641c-origin fixture into FIXTURE_DIR. + + NOTE: regenerates on every call (fresh event_ids). The standard fixture + is regenerated by several classes; this is intentional for test + independence but is not "once per suite" — see _fixtures.py. + """ FIXTURE_DIR.mkdir(parents=True, exist_ok=True) root = FIXTURE_DIR / name if root.exists(): @@ -473,6 +478,42 @@ def test_source_symlink_rejected(self): with self.assertRaises(LegacySourceInvalidError): migrate_store(src, Path(td) / "methodfactory.sqlite3") + def test_empty_inputs_container_fails_typed(self): + """A record_input snapshot with empty inputs must fail typed + (LegacyChainInvalidError), never leak a raw IndexError.""" + with tempfile.TemporaryDirectory() as td: + src = Path(td) / "src" + shutil.copytree(self.src, src) + def mut(evs): + evs[1]["manifest_snapshot"]["inputs"] = [] + _rewrite_journal(src, mut) + with self.assertRaises(LegacyChainInvalidError): + migrate_store(src, Path(td) / "methodfactory.sqlite3") + + def test_empty_artifacts_container_fails_typed(self): + """A record_draft_artifact snapshot with empty artifacts must fail + typed, never leak a raw IndexError.""" + with tempfile.TemporaryDirectory() as td: + src = Path(td) / "src" + shutil.copytree(self.src, src) + def mut(evs): + evs[5]["manifest_snapshot"]["artifacts"] = [] + _rewrite_journal(src, mut) + with self.assertRaises(LegacyChainInvalidError): + migrate_store(src, Path(td) / "methodfactory.sqlite3") + + def test_summary_content_non_string_fails_typed(self): + """A truthy non-string summary.content must fail typed, never leak a + raw AttributeError from .encode().""" + with tempfile.TemporaryDirectory() as td: + src = Path(td) / "src" + shutil.copytree(self.src, src) + def mut(evs): + evs[3]["manifest_snapshot"]["summary"]["content"] = 42 + _rewrite_journal(src, mut) + with self.assertRaises(LegacyChainInvalidError): + migrate_store(src, Path(td) / "methodfactory.sqlite3") + def test_dest_root_permissions_untouched(self): """Default destination root is the legacy source root; its mode must NOT be chmod'd by migration (ADR-0012 §12 source immutability). A @@ -560,7 +601,6 @@ def test_export_link_race_branch(self): migrate_store(src, root / "methodfactory.sqlite3") raced = root / "raced.jsonl" original_link = exp.os.link - original_exists = Path.exists def fake_link(src_path, dst_path): # simulate the race: destination appears right before link From ec62f173dc807fca93de5d2ce7ed9882e38eb497 Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Sat, 8 Aug 2026 07:44:13 +0700 Subject: [PATCH 31/41] fix(migration): close round-4 review - surrogate encoding, dest-root symlink/file gate, mkdir coverage, strict CLI JSON Fourth code-review pass (3a8d158..9b718d6) returned 2 blocking majors and actionable minors: - bug-1 (major): lone-surrogate summary.content leaked raw UnicodeEncodeError from the equivalence byte comparison. The snapshot guard now encodes content at the validation boundary and fails typed LegacyChainInvalidError. - q-1 (major): the root-creation TOCTOU branch had zero coverage. Added tests for fresh-root 0700 creation, non-directory root, symlinked root, and surrogate content. - cluster-1 (minor): destination root gate now uses lstat + S_ISDIR/ S_ISLNK so a regular file or symlink at the root fails typed instead of producing misleading artifact-store errors; redundant chmod removed (mkdir(mode=0o700) already yields 0700 under any umask). - sec-2 (minor): symlinked destination root rejected before resolve() (resolve would follow the link past the lstat gate). - q-2/q-3 (minor): test docs + comment corrected. - q-4 (minor): mutation tests assert the expected action at the index before mutating (silent coverage-loss guard). - cli-1 (nit): typed CLI errors now print strict JSON (json.dumps), not Python repr. Full suite: 420 tests green. --- methodfactory/cli.py | 2 +- methodfactory/migrations/migrate.py | 30 ++++++++++-- methodfactory/migrations/v012_jsonl.py | 11 +++++ methodfactory/tests/test_migrations.py | 65 ++++++++++++++++++++++++++ 4 files changed, 103 insertions(+), 5 deletions(-) diff --git a/methodfactory/cli.py b/methodfactory/cli.py index 00a8393..a34d113 100644 --- a/methodfactory/cli.py +++ b/methodfactory/cli.py @@ -39,7 +39,7 @@ def _fail(err: MethodFactoryError) -> int: - print(err.as_dict(), file=sys.stderr) + print(json.dumps(err.as_dict(), sort_keys=True), file=sys.stderr) return 1 diff --git a/methodfactory/migrations/migrate.py b/methodfactory/migrations/migrate.py index 2c36789..fdfb95a 100644 --- a/methodfactory/migrations/migrate.py +++ b/methodfactory/migrations/migrate.py @@ -352,6 +352,15 @@ def migrate_store( # Destination default: /methodfactory.sqlite3 src = Path(source_root) final_dest = Path(dest) if dest is not None else (src / DB_FILENAME) + # Reject a symlinked destination ROOT before resolving: resolve() would + # follow the link and the lstat gate below would never see it. + import stat as _stat + + dest_parent = final_dest.parent + if dest_parent.is_symlink(): + raise StorageError( + f"destination root {dest_parent} must not be a symlink" + ) final_dest = final_dest.resolve() if final_dest.exists(): raise DestinationExistsError(f"migration destination exists: {final_dest}") @@ -384,7 +393,9 @@ def migrate_store( root_existed = final_root.is_dir() try: if root_existed: - final_root.mkdir(parents=True, exist_ok=True) # no-op; parents may be missing + # Leaf already exists; exist_ok=True no-ops (kept for the + # remove-between-check-and-mkdir race). + final_root.mkdir(parents=True, exist_ok=True) else: # Atomically claim the leaf; a racer's leaf is treated as # pre-existing (never chmod a directory we did not create). @@ -392,19 +403,30 @@ def migrate_store( final_root.mkdir(parents=True, mode=0o700) except FileExistsError: root_existed = True - if not root_existed: - os.chmod(final_root, 0o700) except OSError as exc: raise StorageError( f"cannot create destination parent directory: {exc}" ) from exc + # Reject a non-directory or symlinked destination root. is_dir() follows + # symlinks; the mode gate below must not bless a file or a link. if root_existed: try: - root_mode = final_root.stat().st_mode & 0o777 + st = final_root.lstat() except OSError as exc: raise StorageError( f"cannot stat destination root {final_root}: {exc}" ) from exc + import stat as _stat + + if _stat.S_ISLNK(st.st_mode): + raise StorageError( + f"destination root {final_root} must not be a symlink" + ) + if not _stat.S_ISDIR(st.st_mode): + raise StorageError( + f"destination root {final_root} exists but is not a directory" + ) + root_mode = st.st_mode & 0o777 if root_mode != 0o700: raise StorageError( f"destination root {final_root} has mode {root_mode:03o}, " diff --git a/methodfactory/migrations/v012_jsonl.py b/methodfactory/migrations/v012_jsonl.py index b03251b..884db81 100644 --- a/methodfactory/migrations/v012_jsonl.py +++ b/methodfactory/migrations/v012_jsonl.py @@ -513,6 +513,17 @@ def _validate_snapshot_shape(self, package_id: str, index: int, action: str, f"legacy snapshot for {package_id} at event index {index} " "summary.content must be a string or null" ) + if isinstance(summary.get("content"), str): + try: + summary["content"].encode("utf-8") + except UnicodeEncodeError as exc: + # A lone surrogate passes isinstance(str) but cannot be + # encoded; the equivalence check would leak a raw + # UnicodeEncodeError. Fail typed at the boundary. + raise LegacyChainInvalidError( + f"legacy snapshot for {package_id} at event index " + f"{index} summary.content is not valid UTF-8" + ) from exc conf = summary.get("confirmation") if not isinstance(conf, dict): raise LegacyChainInvalidError( diff --git a/methodfactory/tests/test_migrations.py b/methodfactory/tests/test_migrations.py index 8e1d7ec..7a3da7f 100644 --- a/methodfactory/tests/test_migrations.py +++ b/methodfactory/tests/test_migrations.py @@ -437,6 +437,8 @@ def test_within_package_duplicate_event_id(self): def mut(evs): # Duplicate event_id of rev0 onto rev1 (event_id is part of # the snapshot transition, so the chain is recomputed below). + assert evs[0]["action"] == "create_package" + assert evs[1]["action"] == "record_input" evs[1]["event_id"] = evs[0]["event_id"] evs[1]["manifest_snapshot"]["transition"]["last_event_id"] = evs[0]["event_id"] _rewrite_journal(src, mut) @@ -485,6 +487,7 @@ def test_empty_inputs_container_fails_typed(self): src = Path(td) / "src" shutil.copytree(self.src, src) def mut(evs): + assert evs[1]["action"] == "record_input" evs[1]["manifest_snapshot"]["inputs"] = [] _rewrite_journal(src, mut) with self.assertRaises(LegacyChainInvalidError): @@ -497,6 +500,7 @@ def test_empty_artifacts_container_fails_typed(self): src = Path(td) / "src" shutil.copytree(self.src, src) def mut(evs): + assert evs[5]["action"] == "record_draft_artifact" evs[5]["manifest_snapshot"]["artifacts"] = [] _rewrite_journal(src, mut) with self.assertRaises(LegacyChainInvalidError): @@ -509,11 +513,72 @@ def test_summary_content_non_string_fails_typed(self): src = Path(td) / "src" shutil.copytree(self.src, src) def mut(evs): + assert evs[3]["action"] == "prepare_summary" evs[3]["manifest_snapshot"]["summary"]["content"] = 42 _rewrite_journal(src, mut) with self.assertRaises(LegacyChainInvalidError): migrate_store(src, Path(td) / "methodfactory.sqlite3") + def test_summary_content_dict_fails_typed(self): + """A dict summary.content (non-str, non-null) fails typed.""" + with tempfile.TemporaryDirectory() as td: + src = Path(td) / "src" + shutil.copytree(self.src, src) + def mut(evs): + assert evs[3]["action"] == "prepare_summary" + evs[3]["manifest_snapshot"]["summary"]["content"] = {"a": 1} + _rewrite_journal(src, mut) + with self.assertRaises(LegacyChainInvalidError): + migrate_store(src, Path(td) / "methodfactory.sqlite3") + + def test_summary_content_lone_surrogate_fails_typed(self): + """A lone surrogate in summary.content passes isinstance(str) but + cannot be UTF-8 encoded; must fail typed LegacyChainInvalidError, + never leak raw UnicodeEncodeError.""" + with tempfile.TemporaryDirectory() as td: + src = Path(td) / "src" + shutil.copytree(self.src, src) + def mut(evs): + assert evs[3]["action"] == "prepare_summary" + evs[3]["manifest_snapshot"]["summary"]["content"] = "bad\ud800" + _rewrite_journal(src, mut) + with self.assertRaises(LegacyChainInvalidError): + migrate_store(src, Path(td) / "methodfactory.sqlite3") + + def test_fresh_dest_root_created_0700(self): + """--dest under a fresh directory: root created with mode 0700, + migration succeeds.""" + with tempfile.TemporaryDirectory() as td: + dest = Path(td) / "newroot" / "methodfactory.sqlite3" + migrate_store(self.src, dest) + self.assertTrue(dest.is_file()) + self.assertEqual(dest.parent.stat().st_mode & 0o777, 0o700) + + def test_dest_root_file_fails_closed(self): + """A destination root that is a regular FILE fails closed with a + typed error (never a misleading artifact-store error).""" + with tempfile.TemporaryDirectory() as td: + root = Path(td) / "rootfile" + root.write_text("not a dir", encoding="utf-8") + os.chmod(root, 0o700) + dest = root / "methodfactory.sqlite3" + with self.assertRaises(StorageError): + migrate_store(self.src, dest) + + def test_dest_root_symlink_fails_closed(self): + """A symlinked destination root fails closed (mirrors source-side + symlink policy); nothing is published through the link.""" + with tempfile.TemporaryDirectory() as td: + target = Path(td) / "target" + target.mkdir() + os.chmod(target, 0o700) + link = Path(td) / "link" + link.symlink_to(target) + dest = link / "methodfactory.sqlite3" + with self.assertRaises(StorageError): + migrate_store(self.src, dest) + self.assertFalse((target / "methodfactory.sqlite3").exists()) + def test_dest_root_permissions_untouched(self): """Default destination root is the legacy source root; its mode must NOT be chmod'd by migration (ADR-0012 §12 source immutability). A From a0450b1ac75d46a9698c941e94bb6894c33096c3 Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Sat, 8 Aug 2026 08:17:00 +0700 Subject: [PATCH 32/41] fix(migration): close round-5 review - typed UTF-8 boundary for all journal fields, umask-immune roots Fifth code-review pass (9b718d6..ec62f17) found 1 blocking major and actionable minors: - F-1 (major): the lone-surrogate guard covered only summary.content; sibling fields (intent.raw, created_at/at, event_id, state, ...) still leaked raw UnicodeEncodeError from canonical_bytes / SQLite TEXT binds. _import_package now translates UnicodeEncodeError to MigrationIncompatibleError ('not valid UTF-8') so every journal-sourced field fails typed at the migration boundary. Empirically verified for intent/event_id/at/state surrogates. - F-3 (minor): fresh-root 0700 was umask-dependent (mkdir(mode) is masked). Restored chmod 0700 in the fresh branch only (never chmod a pre-existing/source root); temp store root pre-created private so a hostile umask cannot block SQLite. Verified under umask 0o177. - F-4 (minor): module-level 'import stat'; removed dead nested import. - F-6/F-7 (minor/nit): summary.content non-string test merged into a subTest loop; content bound once in the guard. - F-2 (minor, accepted): leaf-only destination symlink policy documented (mirrors source-side reader); ancestor symlinks accepted by design. Full suite: 419 tests green. --- methodfactory/migrations/migrate.py | 38 +++++++++++++++++++++----- methodfactory/migrations/v012_jsonl.py | 9 +++--- methodfactory/tests/test_migrations.py | 36 +++++++++--------------- 3 files changed, 49 insertions(+), 34 deletions(-) diff --git a/methodfactory/migrations/migrate.py b/methodfactory/migrations/migrate.py index fdfb95a..29503e8 100644 --- a/methodfactory/migrations/migrate.py +++ b/methodfactory/migrations/migrate.py @@ -45,6 +45,7 @@ import json import os import sqlite3 +import stat import uuid from pathlib import Path from typing import Any, Callable @@ -353,9 +354,9 @@ def migrate_store( src = Path(source_root) final_dest = Path(dest) if dest is not None else (src / DB_FILENAME) # Reject a symlinked destination ROOT before resolving: resolve() would - # follow the link and the lstat gate below would never see it. - import stat as _stat - + # follow the link and the lstat gate below would never see it. (Leaf-only + # policy, mirroring the source-side reader; ancestor symlinks are + # documented as accepted.) dest_parent = final_dest.parent if dest_parent.is_symlink(): raise StorageError( @@ -403,6 +404,11 @@ def migrate_store( final_root.mkdir(parents=True, mode=0o700) except FileExistsError: root_existed = True + if not root_existed: + # mkdir(mode=0o700) is umask-masked; chmod the leaf we + # created so the 0700 guarantee holds under ANY umask. Only + # ever chmod a directory this call actually created. + os.chmod(final_root, 0o700) except OSError as exc: raise StorageError( f"cannot create destination parent directory: {exc}" @@ -416,13 +422,11 @@ def migrate_store( raise StorageError( f"cannot stat destination root {final_root}: {exc}" ) from exc - import stat as _stat - - if _stat.S_ISLNK(st.st_mode): + if stat.S_ISLNK(st.st_mode): raise StorageError( f"destination root {final_root} must not be a symlink" ) - if not _stat.S_ISDIR(st.st_mode): + if not stat.S_ISDIR(st.st_mode): raise StorageError( f"destination root {final_root} exists but is not a directory" ) @@ -435,6 +439,16 @@ def migrate_store( "root; the legacy source root is never chmod'd by migration." ) temp_root = final_root / f".{final_dest.name}.tmp.{uuid.uuid4().hex}" + # Pre-create the temp root private (umask-immune). open_database's own + # root.mkdir() is umask-masked; a hostile umask would otherwise leave + # the temp dir without owner-execute and SQLite could not open it. + try: + os.mkdir(temp_root, mode=0o700) + os.chmod(temp_root, 0o700) + except OSError as exc: + raise StorageError( + f"cannot create temporary store root {temp_root}: {exc}" + ) from exc # Build the modern store. Blobs publish to the FINAL artifact store root # (orphan-safe on failure; ADR-0012 §11), while the DB builds at temp_root. @@ -579,6 +593,16 @@ def _import_package( f"legacy value for {pkg.package_id} rev {ev.revision} is " f"current-invalid: {exc}" ) from exc + except UnicodeEncodeError as exc: + # A journal-sourced field (intent, created_at, event_id, state, + # summary body, ...) that cannot be UTF-8 encoded (lone + # surrogate) must fail typed at the migration boundary, never + # leak a raw UnicodeEncodeError from canonicalization or the + # SQLite TEXT bind. + raise MigrationIncompatibleError( + f"legacy value for {pkg.package_id} rev {ev.revision} is " + f"not valid UTF-8: {exc}" + ) from exc except sqlite3.IntegrityError as exc: # Duplicate event_id (global uniqueness) or other row-integrity # violation -> MIGRATION_INCOMPATIBLE (ADR-0012 §7 event-ID diff --git a/methodfactory/migrations/v012_jsonl.py b/methodfactory/migrations/v012_jsonl.py index 884db81..9753c28 100644 --- a/methodfactory/migrations/v012_jsonl.py +++ b/methodfactory/migrations/v012_jsonl.py @@ -501,21 +501,22 @@ def _validate_snapshot_shape(self, package_id: str, index: int, action: str, # equivalence check (summary body byte comparison). summary = snap.get("summary") if isinstance(summary, dict): + summary_content = summary.get("content") if not isinstance(summary.get("canonical_sha256"), str): raise LegacyChainInvalidError( f"legacy snapshot for {package_id} at event index {index} " "summary.canonical_sha256 must be a string" ) - if summary.get("content") is not None and not isinstance( - summary.get("content"), str + if summary_content is not None and not isinstance( + summary_content, str ): raise LegacyChainInvalidError( f"legacy snapshot for {package_id} at event index {index} " "summary.content must be a string or null" ) - if isinstance(summary.get("content"), str): + if isinstance(summary_content, str): try: - summary["content"].encode("utf-8") + summary_content.encode("utf-8") except UnicodeEncodeError as exc: # A lone surrogate passes isinstance(str) but cannot be # encoded; the equivalence check would leak a raw diff --git a/methodfactory/tests/test_migrations.py b/methodfactory/tests/test_migrations.py index 7a3da7f..9afed33 100644 --- a/methodfactory/tests/test_migrations.py +++ b/methodfactory/tests/test_migrations.py @@ -507,29 +507,19 @@ def mut(evs): migrate_store(src, Path(td) / "methodfactory.sqlite3") def test_summary_content_non_string_fails_typed(self): - """A truthy non-string summary.content must fail typed, never leak a - raw AttributeError from .encode().""" - with tempfile.TemporaryDirectory() as td: - src = Path(td) / "src" - shutil.copytree(self.src, src) - def mut(evs): - assert evs[3]["action"] == "prepare_summary" - evs[3]["manifest_snapshot"]["summary"]["content"] = 42 - _rewrite_journal(src, mut) - with self.assertRaises(LegacyChainInvalidError): - migrate_store(src, Path(td) / "methodfactory.sqlite3") - - def test_summary_content_dict_fails_typed(self): - """A dict summary.content (non-str, non-null) fails typed.""" - with tempfile.TemporaryDirectory() as td: - src = Path(td) / "src" - shutil.copytree(self.src, src) - def mut(evs): - assert evs[3]["action"] == "prepare_summary" - evs[3]["manifest_snapshot"]["summary"]["content"] = {"a": 1} - _rewrite_journal(src, mut) - with self.assertRaises(LegacyChainInvalidError): - migrate_store(src, Path(td) / "methodfactory.sqlite3") + """Any non-str, non-null summary.content must fail typed, never leak + a raw AttributeError from .encode().""" + for bad in (42, {"a": 1}, True): + with self.subTest(bad=bad): + with tempfile.TemporaryDirectory() as td: + src = Path(td) / "src" + shutil.copytree(self.src, src) + def mut(evs): + assert evs[3]["action"] == "prepare_summary" + evs[3]["manifest_snapshot"]["summary"]["content"] = bad + _rewrite_journal(src, mut) + with self.assertRaises(LegacyChainInvalidError): + migrate_store(src, Path(td) / "methodfactory.sqlite3") def test_summary_content_lone_surrogate_fails_typed(self): """A lone surrogate in summary.content passes isinstance(str) but From 7abdf5391395a325c46a1d769b7c4963b4c29b75 Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Sat, 8 Aug 2026 08:58:04 +0700 Subject: [PATCH 33/41] fix(migration): close round-6 review - fd-pinned dest root, unconditional lstat gate, temp-root cleanup Sixth code-review pass (ec62f17..a0450b1) found 1 new blocking major M-1 (mkdir->chmod TOCTOU on freshly-created destination roots: chmod follows a swapped symlink) plus minors. Closed: - M-1 (major): destination root leaf is now claimed with fd-pinned fchmod (os.open O_DIRECTORY|O_NOFOLLOW + os.fchmod 0700), never a path-based chmod; the lstat/S_ISLNK/S_ISDIR/mode gate runs UNCONDITIONALLY for existing AND created roots. - m-2 (minor): ArtifactStore init moved inside the failure-cleanup try so its failure removes temp_root; temp-root mkdir fault seam added; test asserts no orphan after temp-root failure. - m-3 (minor): temp-root chmod failure message distinguishes create vs mode; created dir removed on failure. - m-4/m-5 (minor): frozen-algorithm docstring updated for the root claim/gate steps; ancestor-symlink comment reworded (intentional). - m-6 (minor): mutation tests locate events semantically (next(action==...)) instead of assert-guarded positional indices. - q-5 (nit): unused test imports removed. Full suite: 420 tests green. --- methodfactory/migrations/migrate.py | 124 +++++++++++++++---------- methodfactory/tests/test_migrations.py | 52 ++++++++--- 2 files changed, 114 insertions(+), 62 deletions(-) diff --git a/methodfactory/migrations/migrate.py b/methodfactory/migrations/migrate.py index 29503e8..fea55f7 100644 --- a/methodfactory/migrations/migrate.py +++ b/methodfactory/migrations/migrate.py @@ -18,22 +18,15 @@ event_id=legacy event_id, created_at=legacy event.at); e. semantic equivalence check vs legacy snapshot (exclusions only); 7. calculate destination + temporary destination (same directory); -8. fail if final destination exists (DESTINATION_EXISTS); -9. build new SQLite store at temp destination; -10. insert deterministic transformed events/blobs (immutable blobs - pre-written, orphan-safe); -11. PRAGMA integrity_check; -12. full current chain validation (validate_chain, verify_artifacts=True); -13. close/sync SQLite cleanly; -14. generate migration receipt data; -15. durably write temp receipt; -16. FINAL source inventory/hash (AFTER) — require exact equality with step 3; -17. ONLY THEN enter publication: +15. generate migration receipt data; +16. durably write temp receipt; +17. FINAL source inventory/hash (AFTER) — require exact equality with step 3; +18. ONLY THEN enter publication: a. durable final receipt publication (no-clobber os.link temp -> final; dir fsync); b. durable atomic final DB publication (no-clobber os.link temp DB -> final; dir fsync); -18. final read-only verification (binds the published receipt to the DB). +19. final read-only verification (binds the published receipt to the DB). A receipt alone is NOT success. Success requires: final DB exists; matching final receipt exists; same migration identity; final read-only validation @@ -354,9 +347,9 @@ def migrate_store( src = Path(source_root) final_dest = Path(dest) if dest is not None else (src / DB_FILENAME) # Reject a symlinked destination ROOT before resolving: resolve() would - # follow the link and the lstat gate below would never see it. (Leaf-only - # policy, mirroring the source-side reader; ancestor symlinks are - # documented as accepted.) + # follow the link and the lstat gate below would never see it. Only the + # destination root leaf is gated (mirroring the source-side reader); + # ancestor symlinks are intentionally not rejected here. dest_parent = final_dest.parent if dest_parent.is_symlink(): raise StorageError( @@ -392,6 +385,11 @@ def migrate_store( # publishing an unreopenable store. final_root = final_dest.parent root_existed = final_root.is_dir() + # Atomically claim the destination root leaf and pin its inode. + # An unconditional lstat/S_ISLNK/S_ISDIR/mode gate runs for BOTH + # existing and freshly-created roots: never trust a path that could + # have been swapped between mkdir and chmod (chmod follows symlinks). + root_fd: int | None = None try: if root_existed: # Leaf already exists; exist_ok=True no-ops (kept for the @@ -405,47 +403,76 @@ def migrate_store( except FileExistsError: root_existed = True if not root_existed: - # mkdir(mode=0o700) is umask-masked; chmod the leaf we - # created so the 0700 guarantee holds under ANY umask. Only - # ever chmod a directory this call actually created. - os.chmod(final_root, 0o700) + # mkdir(mode=0o700) is umask-masked; pin the inode we + # created and fchmod it so the 0700 guarantee holds under + # ANY umask WITHOUT a path-based chmod that could follow a + # swapped symlink. + root_fd = os.open( + final_root, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW + ) + os.fchmod(root_fd, 0o700) except OSError as exc: + if root_fd is not None: + os.close(root_fd) raise StorageError( f"cannot create destination parent directory: {exc}" ) from exc # Reject a non-directory or symlinked destination root. is_dir() follows # symlinks; the mode gate below must not bless a file or a link. - if root_existed: - try: - st = final_root.lstat() - except OSError as exc: - raise StorageError( - f"cannot stat destination root {final_root}: {exc}" - ) from exc - if stat.S_ISLNK(st.st_mode): - raise StorageError( - f"destination root {final_root} must not be a symlink" - ) - if not stat.S_ISDIR(st.st_mode): - raise StorageError( - f"destination root {final_root} exists but is not a directory" - ) - root_mode = st.st_mode & 0o777 - if root_mode != 0o700: - raise StorageError( - f"destination root {final_root} has mode {root_mode:03o}, " - "expected 0700 (ADR-0012 §D store-root invariant). Fix the " - "root permissions or pass an explicit --dest under a private " - "root; the legacy source root is never chmod'd by migration." - ) + try: + st = final_root.lstat() + except OSError as exc: + if root_fd is not None: + os.close(root_fd) + raise StorageError( + f"cannot stat destination root {final_root}: {exc}" + ) from exc + if stat.S_ISLNK(st.st_mode): + if root_fd is not None: + os.close(root_fd) + raise StorageError( + f"destination root {final_root} must not be a symlink" + ) + if not stat.S_ISDIR(st.st_mode): + if root_fd is not None: + os.close(root_fd) + raise StorageError( + f"destination root {final_root} exists but is not a directory" + ) + root_mode = st.st_mode & 0o777 + if root_mode != 0o700: + if root_fd is not None: + os.close(root_fd) + raise StorageError( + f"destination root {final_root} has mode {root_mode:03o}, " + "expected 0700 (ADR-0012 §D store-root invariant). Fix the " + "root permissions or pass an explicit --dest under a private " + "root; the legacy source root is never chmod'd by migration." + ) + if root_fd is not None: + os.close(root_fd) temp_root = final_root / f".{final_dest.name}.tmp.{uuid.uuid4().hex}" - # Pre-create the temp root private (umask-immune). open_database's own - # root.mkdir() is umask-masked; a hostile umask would otherwise leave - # the temp dir without owner-execute and SQLite could not open it. + # Pre-create the temp root private (umask-immune) with fd-pinned fchmod. + # open_database's own root.mkdir() is umask-masked; a hostile umask + # would otherwise leave the temp dir without owner-execute and SQLite + # could not open it. try: os.mkdir(temp_root, mode=0o700) - os.chmod(temp_root, 0o700) + _fault("after_temp_root_mkdir") + tfd = os.open( + temp_root, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW + ) + try: + os.fchmod(tfd, 0o700) + finally: + os.close(tfd) except OSError as exc: + # Failure before use: remove the created temp dir so no orphan + # accumulates under the destination root. + try: + temp_root.rmdir() + except OSError: + pass raise StorageError( f"cannot create temporary store root {temp_root}: {exc}" ) from exc @@ -454,9 +481,10 @@ def migrate_store( # (orphan-safe on failure; ADR-0012 §11), while the DB builds at temp_root. # chmod_existing=False: the default destination root IS the legacy source # root; its permissions must not be mutated (ADR-0012 §12 source - # immutability). - artifacts = ArtifactStore(final_root, chmod_existing=False) + # immutability). ArtifactStore init is INSIDE the try so its failure also + # cleans up temp_root (no orphan accumulates). try: + artifacts = ArtifactStore(final_root, chmod_existing=False) _fault("before_build_store") _build_store(temp_root, source, artifacts) _fault("after_build_store") diff --git a/methodfactory/tests/test_migrations.py b/methodfactory/tests/test_migrations.py index 9afed33..865d51a 100644 --- a/methodfactory/tests/test_migrations.py +++ b/methodfactory/tests/test_migrations.py @@ -64,11 +64,10 @@ LegacyChainInvalidError, LegacySourceInvalidError, MigrationIncompatibleError, - MigrationPublishFailedError, SourceChangedError, StorageError, ) -from methodfactory.storage.serialization import digest_bytes, sha256_hex +from methodfactory.storage.serialization import digest_bytes from ._fixtures import ( all_action_families_workflow, @@ -437,10 +436,10 @@ def test_within_package_duplicate_event_id(self): def mut(evs): # Duplicate event_id of rev0 onto rev1 (event_id is part of # the snapshot transition, so the chain is recomputed below). - assert evs[0]["action"] == "create_package" - assert evs[1]["action"] == "record_input" - evs[1]["event_id"] = evs[0]["event_id"] - evs[1]["manifest_snapshot"]["transition"]["last_event_id"] = evs[0]["event_id"] + ev0 = next(e for e in evs if e["action"] == "create_package") + ev1 = next(e for e in evs if e["action"] == "record_input") + ev1["event_id"] = ev0["event_id"] + ev1["manifest_snapshot"]["transition"]["last_event_id"] = ev0["event_id"] _rewrite_journal(src, mut) with self.assertRaises(MigrationIncompatibleError): migrate_store(src, Path(td) / "methodfactory.sqlite3") @@ -487,8 +486,8 @@ def test_empty_inputs_container_fails_typed(self): src = Path(td) / "src" shutil.copytree(self.src, src) def mut(evs): - assert evs[1]["action"] == "record_input" - evs[1]["manifest_snapshot"]["inputs"] = [] + ev = next(e for e in evs if e["action"] == "record_input") + ev["manifest_snapshot"]["inputs"] = [] _rewrite_journal(src, mut) with self.assertRaises(LegacyChainInvalidError): migrate_store(src, Path(td) / "methodfactory.sqlite3") @@ -500,8 +499,10 @@ def test_empty_artifacts_container_fails_typed(self): src = Path(td) / "src" shutil.copytree(self.src, src) def mut(evs): - assert evs[5]["action"] == "record_draft_artifact" - evs[5]["manifest_snapshot"]["artifacts"] = [] + ev = next( + e for e in evs if e["action"] == "record_draft_artifact" + ) + ev["manifest_snapshot"]["artifacts"] = [] _rewrite_journal(src, mut) with self.assertRaises(LegacyChainInvalidError): migrate_store(src, Path(td) / "methodfactory.sqlite3") @@ -515,8 +516,10 @@ def test_summary_content_non_string_fails_typed(self): src = Path(td) / "src" shutil.copytree(self.src, src) def mut(evs): - assert evs[3]["action"] == "prepare_summary" - evs[3]["manifest_snapshot"]["summary"]["content"] = bad + ev = next( + e for e in evs if e["action"] == "prepare_summary" + ) + ev["manifest_snapshot"]["summary"]["content"] = bad _rewrite_journal(src, mut) with self.assertRaises(LegacyChainInvalidError): migrate_store(src, Path(td) / "methodfactory.sqlite3") @@ -529,8 +532,10 @@ def test_summary_content_lone_surrogate_fails_typed(self): src = Path(td) / "src" shutil.copytree(self.src, src) def mut(evs): - assert evs[3]["action"] == "prepare_summary" - evs[3]["manifest_snapshot"]["summary"]["content"] = "bad\ud800" + ev = next( + e for e in evs if e["action"] == "prepare_summary" + ) + ev["manifest_snapshot"]["summary"]["content"] = "bad\ud800" _rewrite_journal(src, mut) with self.assertRaises(LegacyChainInvalidError): migrate_store(src, Path(td) / "methodfactory.sqlite3") @@ -569,6 +574,25 @@ def test_dest_root_symlink_fails_closed(self): migrate_store(self.src, dest) self.assertFalse((target / "methodfactory.sqlite3").exists()) + def test_temp_root_failure_leaves_no_orphan(self): + """A fault after temp-root mkdir (before use) must leave no orphan + directory under the destination root.""" + old = migrate_module.FAULT_HOOK + try: + def hook(s): + if s == "after_temp_root_mkdir": + raise OSError("fault at temp root") + migrate_module.FAULT_HOOK = hook + with tempfile.TemporaryDirectory() as td: + dest = Path(td) / "methodfactory.sqlite3" + with self.assertRaises(StorageError): + migrate_store(self.src, dest) + leftovers = [p for p in Path(td).iterdir() if ".tmp." in p.name] + self.assertEqual(leftovers, []) + self.assertFalse(dest.exists()) + finally: + migrate_module.FAULT_HOOK = old + def test_dest_root_permissions_untouched(self): """Default destination root is the legacy source root; its mode must NOT be chmod'd by migration (ADR-0012 §12 source immutability). A From 6a365cd598eba38afd84eeeaa50716da9e9bf9d5 Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Sat, 8 Aug 2026 09:36:47 +0700 Subject: [PATCH 34/41] chore(migration): post-review hygiene - docstring steps restored, fd-close helper Round-7 review approved 7abdf53 for senior review (0 critical / 0 major). This commit applies the non-blocking hygiene items from that pass: - Frozen-algorithm docstring: restore steps 8-14 (destination-exists, root claim/gate, lock refusal, temp-root, build/validate) so the numbered list is contiguous 1-17. - Root-fd close sites consolidated into _close_fd() (suppresses a close failure on exception paths so it cannot mask the in-flight error). Full suite: 420 tests green. --- methodfactory/migrations/migrate.py | 48 +++++++++++++++++++---------- 1 file changed, 31 insertions(+), 17 deletions(-) diff --git a/methodfactory/migrations/migrate.py b/methodfactory/migrations/migrate.py index fea55f7..72a1933 100644 --- a/methodfactory/migrations/migrate.py +++ b/methodfactory/migrations/migrate.py @@ -18,15 +18,25 @@ event_id=legacy event_id, created_at=legacy event.at); e. semantic equivalence check vs legacy snapshot (exclusions only); 7. calculate destination + temporary destination (same directory); -15. generate migration receipt data; -16. durably write temp receipt; -17. FINAL source inventory/hash (AFTER) — require exact equality with step 3; -18. ONLY THEN enter publication: +8. fail if final destination exists (DESTINATION_EXISTS); +9. atomically claim the destination root (mkdir 0700 + fd-pinned fchmod; + unconditional lstat gate rejects symlink/non-dir/non-0700 roots); + refuse on legacy `.lock` presence (CONCURRENCY); +10. pre-create the temp store root private (0700, umask-immune); +11. build new SQLite store at temp destination; +12. insert deterministic transformed events/blobs (immutable blobs + pre-written, orphan-safe); PRAGMA integrity_check; full current chain + validation (validate_chain, verify_artifacts=True); close/sync SQLite + cleanly; +13. generate migration receipt data; +14. durably write temp receipt; +15. FINAL source inventory/hash (AFTER) — require exact equality with step 3; +16. ONLY THEN enter publication: a. durable final receipt publication (no-clobber os.link temp -> final; dir fsync); b. durable atomic final DB publication (no-clobber os.link temp DB -> final; dir fsync); -19. final read-only verification (binds the published receipt to the DB). +17. final read-only verification (binds the published receipt to the DB). A receipt alone is NOT success. Success requires: final DB exists; matching final receipt exists; same migration identity; final read-only validation @@ -412,8 +422,7 @@ def migrate_store( ) os.fchmod(root_fd, 0o700) except OSError as exc: - if root_fd is not None: - os.close(root_fd) + _close_fd(root_fd) raise StorageError( f"cannot create destination parent directory: {exc}" ) from exc @@ -422,35 +431,30 @@ def migrate_store( try: st = final_root.lstat() except OSError as exc: - if root_fd is not None: - os.close(root_fd) + _close_fd(root_fd) raise StorageError( f"cannot stat destination root {final_root}: {exc}" ) from exc if stat.S_ISLNK(st.st_mode): - if root_fd is not None: - os.close(root_fd) + _close_fd(root_fd) raise StorageError( f"destination root {final_root} must not be a symlink" ) if not stat.S_ISDIR(st.st_mode): - if root_fd is not None: - os.close(root_fd) + _close_fd(root_fd) raise StorageError( f"destination root {final_root} exists but is not a directory" ) root_mode = st.st_mode & 0o777 if root_mode != 0o700: - if root_fd is not None: - os.close(root_fd) + _close_fd(root_fd) raise StorageError( f"destination root {final_root} has mode {root_mode:03o}, " "expected 0700 (ADR-0012 §D store-root invariant). Fix the " "root permissions or pass an explicit --dest under a private " "root; the legacy source root is never chmod'd by migration." ) - if root_fd is not None: - os.close(root_fd) + _close_fd(root_fd) temp_root = final_root / f".{final_dest.name}.tmp.{uuid.uuid4().hex}" # Pre-create the temp root private (umask-immune) with fd-pinned fchmod. # open_database's own root.mkdir() is umask-masked; a hostile umask @@ -963,3 +967,13 @@ def _fsync_dir(path: Path) -> None: os.fsync(dir_fd) finally: os.close(dir_fd) + + +def _close_fd(fd: int | None) -> None: + """Close a possibly-None fd, suppressing a close error on exception + paths (a close failure must not mask the in-flight failure).""" + if fd is not None: + try: + os.close(fd) + except OSError: + pass From 775630eebdfb7c4b8357a4d1976505109b4b085b Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Sat, 8 Aug 2026 09:52:39 +0700 Subject: [PATCH 35/41] ci(release-gate): fetch full history for fb5641c fixture generation Migration tests generate fixtures from the exact public v0.1.2 commit fb5641c (tag v0.1.2-integrity) via a git worktree. actions/checkout defaults to depth 1, so fb5641c is absent from the PR-branch checkout and every fixture-dependent test failed with 'cannot build fb5641c checkout'. fetch-depth: 0 makes the frozen legacy commit available. PR-event CI on 6a365cd failed exactly here (both 3.11 and 3.12, step 'Canonical unit tests'); reproduced locally with a depth-1 clone. --- .github/workflows/release-gate.yml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.github/workflows/release-gate.yml b/.github/workflows/release-gate.yml index 8a293da..e666727 100644 --- a/.github/workflows/release-gate.yml +++ b/.github/workflows/release-gate.yml @@ -19,6 +19,11 @@ jobs: python-version: ["3.11", "3.12"] steps: - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 (pinned SHA) + with: + # Full history required: migration tests generate fixtures from the + # exact public v0.1.2 commit fb5641c (tag v0.1.2-integrity), which + # is not present in a depth-1 checkout of this branch. + fetch-depth: 0 - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 (pinned SHA) with: python-version: ${{ matrix.python-version }} From c70a6f314b2329dc3e134546ad20b41519be98ab Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Sat, 8 Aug 2026 10:03:39 +0700 Subject: [PATCH 36/41] docs(architecture-reset): mark migration/export gate complete at 775630e, stopped for senior review --- docs/architecture-reset-status.md | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/docs/architecture-reset-status.md b/docs/architecture-reset-status.md index 15dfb75..23b9815 100644 --- a/docs/architecture-reset-status.md +++ b/docs/architecture-reset-status.md @@ -1,6 +1,6 @@ # Architecture Reset — Project State (2026-08-08, Phase 3: migration/export) -**Status:** SQLite architecture approved in principle; senior review `4878235332` on PR #1 completed the architecture review and directed the Phase 2 implementation order (commits 1–4) with a design-convergence stop gate. The Phase 2 stop gate was accepted, and the migration/export implementation gate (ADR-0012 amendment, frozen at `42ff7d9` + `b9e46c1`) is now implemented and under mandatory review. This document tracks the clean `feat/sqlite-persistence-reset` branch. +**Status:** SQLite architecture approved in principle; senior review `4878235332` on PR #1 completed the architecture review and directed the Phase 2 implementation order (commits 1–4) with a design-convergence stop gate. The Phase 2 stop gate was accepted, and the migration/export implementation gate (ADR-0012 amendment, frozen at `42ff7d9` + `b9e46c1`) is **COMPLETE at `775630e`** — implemented, 7-pass code-reviewed (final verdict: 0 critical / 0 major), 420 tests green, literal-head CI green, evidence comment posted — and is now **STOPPED for independent senior review**. This document tracks the clean `feat/sqlite-persistence-reset` branch. ## Branch topology @@ -36,22 +36,37 @@ Bounded implementation authorized by Vincent at canonical head `b9e46c1` on deterministic supported export, deterministic legacy-v0.1.2 evidence export, and the minimal CLI/error/test/documentation surface for those capabilities. -Implemented (pending senior review): +**Gate status: COMPLETE — STOPPED for independent senior review.** + +- Final head: `775630eebdfb7c4b8357a4d1976505109b4b085b` (10 commits, + fast-forward from `b9e46c1`; no force-push). +- Local suite: **420 tests green** (356 baseline + 64 migration/export). +- Code review: 7 passes of the mandatory family (bug/security/performance/ + quality → verify → dedupe → sanity); final verdict 0 critical / 0 major. +- Literal-head CI: run `31236027530` on `775630e` **SUCCESS** — 3.11 job + `93048557007` and 3.12 job `93048556968`, both `Ran 420 tests … OK`, + clean-worktree + artifact-scan steps success. +- PR #1 evidence comment: id `5224200252` (32-point gate checklist). +- CI fix: `release-gate.yml` now uses `fetch-depth: 0` so migration fixture + generation can reach the frozen public commit `fb5641c` (prior run + `31235850649` failed on a shallow checkout). + +Implemented: - `methodfactory/migrations/v012_jsonl.py` — frozen read-only v0.1.2 reader (exact `fb5641c` semantics; no CAS/lock/repair/append mechanics). - `methodfactory/migrations/migrate.py` — atomic migration: legacy validation, semantic-action reconstruction by legacy hash, current-engine transformation (`next_manifest` only), equivalence verification, source-stability proof, - temp-DB build + validation, durable receipt + DB publication, final - read-only verification, fault seams. + temp-DB build + validation, durable receipt + DB publication (no-clobber + `os.link`), final read-only verification, fault seams. - `methodfactory/migrations/export.py` — `method-factory-events-v1` and `legacy-v012-jsonl` deterministic exports (read-only, consistent read). - `methodfactory/cli.py` — bounded surface restored: `mf migrate-store` and `mf export`; lifecycle commands remain unavailable; `mf --version` unchanged. - Six new frozen migration error codes (see `docs/public-surface.md`). - `methodfactory/tests/_fixtures.py` + `test_migrations.py` — fb5641c-origin - fixtures and 44 focused tests (full suite 400 tests green). + fixtures and 64 focused tests. Not authorized / NOT implemented in this gate: merge, PR-ready, tag, release, `main`/forensic mutation, force-push, deployment, lifecycle expansion, From a9fafbfd80dfe0d431930b0be772a703db3fd55f Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Sat, 8 Aug 2026 12:51:46 +0700 Subject: [PATCH 37/41] chore(release): prepare 2.0.0rc1 candidate RC1 candidate preparation gate (bounded release-preparation only; no product-semantic changes): - version identity: 2.0.0a1 -> 2.0.0rc1 in pyproject.toml and methodfactory/__init__.py; packaging + CLI version tests updated to pin 2.0.0rc1 (no split-brain versioning). - release metadata cleanup: Development Status 3-Alpha -> 4-Beta; project description drops stale 'Phase 2 foundation' wording. - candidate-head documentation: architecture-reset-status distinguishes implementation head 775630e, documentation/reporting head c70a6f3, and the new RC1 candidate head; RC1 candidate wording is 'pending independent senior acceptance and operator integration gate' - not released. - release-gate CI strengthening: build wheel+sdist from exact candidate source (git archive to a disposable dir), isolated venv wheel install, import provenance + 2.0.0rc1 version proof, CLI surface guard, and installed migration/export smoke (migrate-store, authoritative validation, both exports, legacy revalidation, source-unchanged) on both Python 3.11 and 3.12. No deploy/publish/tag/release steps. Full suite: 420 tests green. --- .github/workflows/release-gate.yml | 115 +++++++++++++++++++++++++ docs/architecture-reset-status.md | 54 +++++++++++- methodfactory/__init__.py | 2 +- methodfactory/tests/test_migrations.py | 4 +- methodfactory/tests/test_packaging.py | 6 +- pyproject.toml | 6 +- 6 files changed, 174 insertions(+), 13 deletions(-) diff --git a/.github/workflows/release-gate.yml b/.github/workflows/release-gate.yml index e666727..4faff7c 100644 --- a/.github/workflows/release-gate.yml +++ b/.github/workflows/release-gate.yml @@ -38,6 +38,121 @@ jobs: run: | mf --version python -m methodfactory --version + test "$(mf --version)" = "methodfactory 2.0.0rc1" + test "$(python -m methodfactory --version)" = "methodfactory 2.0.0rc1" + + # RC1 packaging proof — build the actual distributable wheel/sdist from + # the EXACT candidate source into a disposable directory OUTSIDE the + # repo, so the tracked-worktree and artifact-scan steps below stay + # clean. No new build dependency: pip wheel/download use the declared + # setuptools build backend. + - name: Build distributable package (wheel + sdist) + run: | + set -euo pipefail + export MF_DIST="$RUNNER_TEMP/mf-dist" + mkdir -p "$MF_DIST" "$RUNNER_TEMP/mf-src" + git archive HEAD | tar -x -C "$RUNNER_TEMP/mf-src" + cd "$RUNNER_TEMP/mf-src" + python -m pip wheel . --no-deps -w "$MF_DIST" + python -m pip download . --no-deps --no-binary :all: -d "$MF_DIST" + ls -la "$MF_DIST" + python - <<'EOF' + import hashlib, os, pathlib + dist = pathlib.Path(os.environ["MF_DIST"]) + for p in sorted(dist.iterdir()): + print(p.name, hashlib.sha256(p.read_bytes()).hexdigest()) + EOF + + - name: Isolated wheel install + version proof + run: | + set -euo pipefail + export MF_VENV="$RUNNER_TEMP/mf-venv" + export MF_DIST="$RUNNER_TEMP/mf-dist" + python -m venv "$MF_VENV" + "$MF_VENV/bin/python" -m pip install --no-deps "$MF_DIST"/*.whl + # import provenance: must resolve from the venv site-packages, not + # the development checkout. + "$MF_VENV/bin/python" - <<'EOF' + import methodfactory, os, pathlib + print("import file:", methodfactory.__file__) + venv = pathlib.Path(os.environ["MF_VENV"]).resolve() + assert pathlib.Path(methodfactory.__file__).resolve().is_relative_to(venv), \ + "methodfactory resolved outside the isolated venv" + assert methodfactory.__version__ == "2.0.0rc1", methodfactory.__version__ + EOF + test "$("$MF_VENV/bin/mf" --version)" = "methodfactory 2.0.0rc1" + test "$("$MF_VENV/bin/python" -m methodfactory --version)" = "methodfactory 2.0.0rc1" + # CLI surface: only migrate-store + export may be present. + "$MF_VENV/bin/mf" --help + for banned in create apply status summary review trial ship triage; do + if "$MF_VENV/bin/mf" --help | grep -qE "^\s+$banned\b"; then + echo "::error::forbidden lifecycle command exposed: $banned"; exit 1 + fi + done + + # Packaged functional smoke: canonical exact-fb5641c fixture -> installed + # mf migrate-store -> authoritative validation -> installed mf export + # (both formats) -> legacy evidence revalidated by the frozen reader; + # source must remain unchanged. + - name: Packaged migration/export smoke (isolated env) + run: | + set -euo pipefail + export MF_SMOKE="$RUNNER_TEMP/mf-smoke" + export MF_STORE="$RUNNER_TEMP/mf-store" + export MF_LEGACY_REVAL="$RUNNER_TEMP/mf-legacy-revalidate" + export MF_VENV="$RUNNER_TEMP/mf-venv" + python - <<'EOF' + # Generate the canonical fixture through the sanctioned mechanism + # (exact public fb5641c code via the repo's disposable worktree). + import os, pathlib, shutil + from methodfactory.tests._fixtures import generate_fixture, standard_workflow + root = pathlib.Path(os.environ["MF_SMOKE"]) + shutil.rmtree(root, ignore_errors=True) + generate_fixture(str(root), workflow=standard_workflow) + EOF + mkdir -p "$MF_STORE" + # source immutability baseline + sha256sum "$MF_SMOKE"/events/*.events.jsonl > "$MF_STORE/source.before" + # installed migrate-store + "$MF_VENV/bin/mf" migrate-store --source "$MF_SMOKE" --dest "$MF_STORE/methodfactory.sqlite3" + # authoritative validation via installed primitives + "$MF_VENV/bin/python" - <<'EOF' + import os + from methodfactory.storage.store import SqliteManifestStore + store = SqliteManifestStore(os.environ["MF_STORE"]) + try: + for pkg in store.list_package_ids(): + result = store.validate_chain(pkg, verify_artifacts=True) + assert result["valid"], result + print("validated", pkg, result) + finally: + store.close() + EOF + # installed exports (both formats) + "$MF_VENV/bin/mf" export --store "$MF_STORE" --output "$MF_STORE/events-v1.jsonl" --format method-factory-events-v1 + "$MF_VENV/bin/mf" export --store "$MF_STORE" --output "$MF_STORE/legacy.jsonl" --format legacy-v012-jsonl + test -s "$MF_STORE/events-v1.jsonl" + test -s "$MF_STORE/legacy.jsonl" + # legacy evidence export revalidates under the frozen legacy reader + "$MF_VENV/bin/python" - <<'EOF' + import os, pathlib, shutil + from methodfactory.migrations.v012_jsonl import LegacySource + lr = pathlib.Path(os.environ["MF_LEGACY_REVAL"]) + store = pathlib.Path(os.environ["MF_STORE"]) + shutil.rmtree(lr, ignore_errors=True) + (lr / "events").mkdir(parents=True) + (lr / "packages").mkdir() + (lr / "artifacts" / "blobs").mkdir(parents=True) + shutil.copy(store / "legacy.jsonl", lr / "events" / "pkg_demo_001.events.jsonl") + for blob in (store / "blobs").iterdir(): + shutil.copy(blob, lr / "artifacts" / "blobs" / blob.name) + LegacySource(lr).validate() + print("legacy evidence export revalidated OK") + EOF + # source unchanged after migration/export + sha256sum "$MF_SMOKE"/events/*.events.jsonl > "$MF_STORE/source.after" + diff "$MF_STORE/source.before" "$MF_STORE/source.after" + echo "source unchanged OK" # Honest terminology (Finding 4 item 2): this proves a CLEAN TRACKED # WORKTREE (no uncommitted/untracked tracked-tree changes). Generated diff --git a/docs/architecture-reset-status.md b/docs/architecture-reset-status.md index 23b9815..a682a70 100644 --- a/docs/architecture-reset-status.md +++ b/docs/architecture-reset-status.md @@ -1,6 +1,6 @@ -# Architecture Reset — Project State (2026-08-08, Phase 3: migration/export) +# Architecture Reset — Project State (2026-08-08, RC1 candidate preparation) -**Status:** SQLite architecture approved in principle; senior review `4878235332` on PR #1 completed the architecture review and directed the Phase 2 implementation order (commits 1–4) with a design-convergence stop gate. The Phase 2 stop gate was accepted, and the migration/export implementation gate (ADR-0012 amendment, frozen at `42ff7d9` + `b9e46c1`) is **COMPLETE at `775630e`** — implemented, 7-pass code-reviewed (final verdict: 0 critical / 0 major), 420 tests green, literal-head CI green, evidence comment posted — and is now **STOPPED for independent senior review**. This document tracks the clean `feat/sqlite-persistence-reset` branch. +**Status:** SQLite architecture approved in principle; senior review `4878235332` on PR #1 completed the architecture review and directed the Phase 2 implementation order (commits 1–4) with a design-convergence stop gate. The Phase 2 stop gate was accepted; the migration/export implementation gate (ADR-0012 amendment, frozen at `42ff7d9` + `b9e46c1`) closed at `775630e`; the documentation/reporting head is `c70a6f3`. **The current branch head is the RC1 candidate — pending independent senior acceptance and operator integration gate.** This document tracks the clean `feat/sqlite-persistence-reset` branch. ## Branch topology @@ -29,6 +29,17 @@ fb5641c remote main (published v0.1.2-integrity base) | Git bundle | `method-factory-8a7e916.bundle` (SHA-256 `92c0bb1026190f9fd5e1f61bb4cd5fc16a08605aecf77f81eb5bc93b3b504f63`; `git bundle verify` OK) | | Local archival | `persistence-reset` branch preserved locally (pre-revision ADR draft, not published) | +## Head lineage (implementation vs documentation vs RC1 candidate) + +| Head | Role | +|---|---| +| `775630eebdfb7c4b8357a4d1976505109b4b085b` | **Migration/export implementation head** — product implementation closed here (10 commits from `b9e46c1`); 7-pass code-reviewed (0 critical / 0 major); 420 tests green; literal-head CI green; PR evidence comment id `5224200252`. | +| `c70a6f314b2329dc3e134546ad20b41519be98ab` | **Documentation/reporting head** — architecture-reset status marked the migration/export gate complete/stopped for senior review (docs-only commit). | +| *(this branch head)* | **RC1 candidate** — prepared by the RC1 Candidate Preparation Gate: version identity `2.0.0rc1`, release metadata cleanup, clean package build + isolated install proof, release-gate CI strengthening, RC1 evidence. RC1 candidate — **pending independent senior acceptance and operator integration gate.** Not released. | + +Do not call the RC1 candidate "released". The eventual Git release/tag identity +is `v2.0.0-rc.1`; the tag is NOT created in this gate. + ## Migration/export implementation gate (2026-08-08) Bounded implementation authorized by Vincent at canonical head `b9e46c1` on @@ -36,9 +47,11 @@ Bounded implementation authorized by Vincent at canonical head `b9e46c1` on deterministic supported export, deterministic legacy-v0.1.2 evidence export, and the minimal CLI/error/test/documentation surface for those capabilities. -**Gate status: COMPLETE — STOPPED for independent senior review.** +**Gate status: COMPLETE — implementation closed at `775630e`, STOPPED for +independent senior review (historical record; superseded by the RC1 +candidate preparation).** -- Final head: `775630eebdfb7c4b8357a4d1976505109b4b085b` (10 commits, +- Implementation head: `775630eebdfb7c4b8357a4d1976505109b4b085b` (10 commits, fast-forward from `b9e46c1`; no force-push). - Local suite: **420 tests green** (356 baseline + 64 migration/export). - Code review: 7 passes of the mandatory family (bug/security/performance/ @@ -73,6 +86,39 @@ Not authorized / NOT implemented in this gate: merge, PR-ready, tag, release, backup/restore, generic import, garbage collection, JSONL as canonical store, or 8a7e916 repair/CAS/locking mechanics. +## RC1 candidate preparation gate (2026-08-08) + +Bounded release-preparation gate authorized by Vincent. Scope limited to: +release-version identity (`2.0.0rc1`), stale release-status/metadata cleanup, +clean-install/package proof, release-gate CI strengthening, RC1 evidence. +No product-semantic changes were authorized; the product implementation +remains frozen pending final senior review. + +- **RC1 candidate head:** *(recorded after the RC preparation commit exists — + see Head lineage table above.)* +- Version identity: `2.0.0rc1` across `pyproject.toml`, `methodfactory/ + __init__.py`, packaging tests, and CLI version tests. Eventual Git tag + identity `v2.0.0-rc.1` — NOT created in this gate. +- Release metadata: `Development Status :: 3 - Alpha` → `4 - Beta`; stale + "Phase 2 foundation" wording removed from the project description. +- Clean package build: wheel (+ sdist if supported) built from the exact + candidate source in a disposable directory; filenames + SHA-256 recorded; + no artifacts committed or left in the worktree. +- Fresh-environment install: wheel installed into a disposable venv with no + editable checkout; import provenance, `__version__`, `mf --version`, + `python -m methodfactory --version`, and CLI surface verified. +- Packaged functional smoke: canonical `fb5641c` fixture → installed + `mf migrate-store` → authoritative validation → installed `mf export` + (both formats) → legacy evidence export revalidated by the frozen + legacy reader; source unchanged. +- Release-gate CI: builds the distributable wheel, installs into an isolated + environment, proves `mf --version` = `2.0.0rc1`, runs installed + migration/export smoke on 3.11 and 3.12; tracked-worktree and artifact-scan + steps retained. +- RC1 candidate — **pending independent senior acceptance and operator + integration gate.** Not released. No tag, no GitHub Release, no PyPI + publication, no deployment. + ## Senior review 4878235332 (2026-08-07) — accepted - SQLite reset remains **APPROVED IN PRINCIPLE**; corrected evidence package closes the prior evidence gap. diff --git a/methodfactory/__init__.py b/methodfactory/__init__.py index 9be5a12..0fba063 100644 --- a/methodfactory/__init__.py +++ b/methodfactory/__init__.py @@ -5,7 +5,7 @@ ensure_ascii=True variant is removed/legacy-scoped in manifest/hashing.py). """ -__version__ = "2.0.0a1" +__version__ = "2.0.0rc1" # Reusable foundation (ADR-0012 §8 port list). Canonical serialization is the # single storage-layer implementation; manifest.hashing re-exports it. diff --git a/methodfactory/tests/test_migrations.py b/methodfactory/tests/test_migrations.py index 865d51a..b3ac1e3 100644 --- a/methodfactory/tests/test_migrations.py +++ b/methodfactory/tests/test_migrations.py @@ -1164,13 +1164,13 @@ def test_temp_hygiene_after_failure(self): class CliBoundaryTests(unittest.TestCase): """`mf --version`, migrate-store, export through the real CLI.""" - def test_version_unchanged(self): + def test_version_reports_rc1(self): result = subprocess.run( [sys.executable, "-m", "methodfactory.cli", "--version"], capture_output=True, text=True, ) self.assertEqual(result.returncode, 0) - self.assertIn("methodfactory 2.0.0a1", result.stdout) + self.assertIn("methodfactory 2.0.0rc1", result.stdout) def test_no_lifecycle_commands(self): result = subprocess.run( diff --git a/methodfactory/tests/test_packaging.py b/methodfactory/tests/test_packaging.py index 6fe56de..5af1ac6 100644 --- a/methodfactory/tests/test_packaging.py +++ b/methodfactory/tests/test_packaging.py @@ -9,15 +9,15 @@ class PackagingTests(unittest.TestCase): - def test_version_is_2_0_0a1(self): - self.assertEqual(methodfactory.__version__, "2.0.0a1") + def test_version_is_2_0_0rc1(self): + self.assertEqual(methodfactory.__version__, "2.0.0rc1") def test_distribution_metadata_version_matches(self): try: dist = importlib.metadata.version("methodfactory") except importlib.metadata.PackageNotFoundError: self.skipTest("methodfactory not installed (not an editable install)") - self.assertEqual(dist, "2.0.0a1") + self.assertEqual(dist, "2.0.0rc1") def test_mf_entry_point_registered(self): eps = importlib.metadata.entry_points(group="console_scripts") diff --git a/pyproject.toml b/pyproject.toml index a9907b2..d95014b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,14 +4,14 @@ build-backend = "setuptools.build_meta" [project] name = "methodfactory" -version = "2.0.0a1" -description = "Method Factory — prompt+code pipeline generator (SQLite canonical store, Phase 2 foundation)" +version = "2.0.0rc1" +description = "Method Factory — deterministic prompt+code pipeline generator with SQLite canonical persistence" readme = "README.md" requires-python = ">=3.11" license = { text = "MIT" } keywords = ["agents", "skills", "pipeline", "state-machine", "persistence"] classifiers = [ - "Development Status :: 3 - Alpha", + "Development Status :: 4 - Beta", "Intended Audience :: Developers", "License :: OSI Approved :: MIT License", "Programming Language :: Python :: 3", From a6aeb8ca9f0ac571ef06e3d8d5e8a1b032379e2a Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Sat, 8 Aug 2026 12:56:58 +0700 Subject: [PATCH 38/41] ci(release-gate): fix isolated-venv import provenance (CWD shadowing) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The packaged-proof steps ran venv python heredocs with CWD = the repo checkout. 'python -' puts CWD on sys.path, so 'import methodfactory' resolved from the checkout directory instead of the installed wheel site-packages, failing the provenance assertion on both 3.11 and 3.12. Fix: cd out of the repo () before all installed-venv invocations, so imports provably resolve from the wheel installation. Fixture generation still runs from the checkout (sanctioned mechanism). Store root chmod 0700 added to the smoke (migration's ADR-0012 §D root invariant is fail-closed). CI-only correction; no product code changed. --- .github/workflows/release-gate.yml | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/.github/workflows/release-gate.yml b/.github/workflows/release-gate.yml index 4faff7c..ad44172 100644 --- a/.github/workflows/release-gate.yml +++ b/.github/workflows/release-gate.yml @@ -70,6 +70,10 @@ jobs: export MF_DIST="$RUNNER_TEMP/mf-dist" python -m venv "$MF_VENV" "$MF_VENV/bin/python" -m pip install --no-deps "$MF_DIST"/*.whl + # Run OUTSIDE the repo: a heredoc through `python -` puts CWD on + # sys.path, which would shadow the installed package with the + # checkout copy and break the import-provenance proof. + cd "$RUNNER_TEMP" # import provenance: must resolve from the venv site-packages, not # the development checkout. "$MF_VENV/bin/python" - <<'EOF' @@ -104,6 +108,8 @@ jobs: python - <<'EOF' # Generate the canonical fixture through the sanctioned mechanism # (exact public fb5641c code via the repo's disposable worktree). + # Runs from the checkout on purpose: the editable install provides + # methodfactory.tests._fixtures. import os, pathlib, shutil from methodfactory.tests._fixtures import generate_fixture, standard_workflow root = pathlib.Path(os.environ["MF_SMOKE"]) @@ -111,8 +117,12 @@ jobs: generate_fixture(str(root), workflow=standard_workflow) EOF mkdir -p "$MF_STORE" + chmod 700 "$MF_STORE" # source immutability baseline sha256sum "$MF_SMOKE"/events/*.events.jsonl > "$MF_STORE/source.before" + # Run the installed tooling OUTSIDE the repo (CWD on sys.path would + # shadow the installed package with the checkout copy). + cd "$RUNNER_TEMP" # installed migrate-store "$MF_VENV/bin/mf" migrate-store --source "$MF_SMOKE" --dest "$MF_STORE/methodfactory.sqlite3" # authoritative validation via installed primitives From a037ecb240ab379ea19b8b611a0aab4d435b57a9 Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Sat, 8 Aug 2026 13:02:58 +0700 Subject: [PATCH 39/41] docs(architecture-reset): record RC1 candidate head a6aeb8c and gate evidence Records the exact RC1 candidate SHA (a6aeb8c, prepared by the RC1 Candidate Preparation Gate) in the head-lineage table, distinguishes it from the migration/export implementation head 775630e and the prior documentation/reporting head c70a6f3, and documents the gate evidence: version 2.0.0rc1 surfaces, metadata cleanup, clean wheel+sdist build, fresh-venv install proof, packaged migration/export smoke, and literal -head CI run 31242827310 (SUCCESS on 3.11 + 3.12). RC1 candidate - pending independent senior acceptance and operator integration gate. Not released. Docs-only commit. --- docs/architecture-reset-status.md | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/docs/architecture-reset-status.md b/docs/architecture-reset-status.md index a682a70..d6d278b 100644 --- a/docs/architecture-reset-status.md +++ b/docs/architecture-reset-status.md @@ -35,7 +35,8 @@ fb5641c remote main (published v0.1.2-integrity base) |---|---| | `775630eebdfb7c4b8357a4d1976505109b4b085b` | **Migration/export implementation head** — product implementation closed here (10 commits from `b9e46c1`); 7-pass code-reviewed (0 critical / 0 major); 420 tests green; literal-head CI green; PR evidence comment id `5224200252`. | | `c70a6f314b2329dc3e134546ad20b41519be98ab` | **Documentation/reporting head** — architecture-reset status marked the migration/export gate complete/stopped for senior review (docs-only commit). | -| *(this branch head)* | **RC1 candidate** — prepared by the RC1 Candidate Preparation Gate: version identity `2.0.0rc1`, release metadata cleanup, clean package build + isolated install proof, release-gate CI strengthening, RC1 evidence. RC1 candidate — **pending independent senior acceptance and operator integration gate.** Not released. | +| `a6aeb8ca9f0ac571ef06e3d8d5e8a1b032379e2a` | **RC1 candidate head** — prepared by the RC1 Candidate Preparation Gate: version identity `2.0.0rc1`, release metadata cleanup, clean package build + isolated install proof, release-gate CI strengthening (build + isolated install + packaged smoke steps), RC1 evidence. Literal-head CI run `31242827310` **SUCCESS** on this exact SHA (3.11 job `93066497291`, 3.12 job `93066497309`; all 10 steps green). RC1 candidate — **pending independent senior acceptance and operator integration gate.** Not released. | +| *(final branch head)* | **Reporting head** — this document records the RC1 candidate head above and the gate evidence (docs-only commit). | Do not call the RC1 candidate "released". The eventual Git release/tag identity is `v2.0.0-rc.1`; the tag is NOT created in this gate. @@ -94,27 +95,30 @@ clean-install/package proof, release-gate CI strengthening, RC1 evidence. No product-semantic changes were authorized; the product implementation remains frozen pending final senior review. -- **RC1 candidate head:** *(recorded after the RC preparation commit exists — - see Head lineage table above.)* +- **RC1 candidate head:** `a6aeb8ca9f0ac571ef06e3d8d5e8a1b032379e2a` (see Head + lineage table above). - Version identity: `2.0.0rc1` across `pyproject.toml`, `methodfactory/ __init__.py`, packaging tests, and CLI version tests. Eventual Git tag identity `v2.0.0-rc.1` — NOT created in this gate. - Release metadata: `Development Status :: 3 - Alpha` → `4 - Beta`; stale "Phase 2 foundation" wording removed from the project description. -- Clean package build: wheel (+ sdist if supported) built from the exact - candidate source in a disposable directory; filenames + SHA-256 recorded; - no artifacts committed or left in the worktree. +- Clean package build: wheel + sdist built from the exact candidate source + (`a6aeb8c` via `git archive`) in a disposable directory; recorded locally + — wheel `methodfactory-2.0.0rc1-py3-none-any.whl`, sdist + `methodfactory-2.0.0rc1.tar.gz`; no artifacts committed or left in the + worktree. - Fresh-environment install: wheel installed into a disposable venv with no - editable checkout; import provenance, `__version__`, `mf --version`, - `python -m methodfactory --version`, and CLI surface verified. + editable checkout; import provenance (venv site-packages), `__version__`, + `mf --version`, `python -m methodfactory --version`, and CLI surface + verified. - Packaged functional smoke: canonical `fb5641c` fixture → installed `mf migrate-store` → authoritative validation → installed `mf export` (both formats) → legacy evidence export revalidated by the frozen legacy reader; source unchanged. -- Release-gate CI: builds the distributable wheel, installs into an isolated - environment, proves `mf --version` = `2.0.0rc1`, runs installed - migration/export smoke on 3.11 and 3.12; tracked-worktree and artifact-scan - steps retained. +- Release-gate CI: run `31242827310` on the exact candidate head `a6aeb8c` + **SUCCESS** on 3.11 and 3.12 — canonical tests, wheel+sdist build, + isolated wheel install + version proof, packaged migration/export smoke, + clean-worktree, artifact-scan all green. - RC1 candidate — **pending independent senior acceptance and operator integration gate.** Not released. No tag, no GitHub Release, no PyPI publication, no deployment. From 93ec1afe3cf9829b15c238e61e00aeeb182ece9f Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Sat, 8 Aug 2026 13:03:53 +0700 Subject: [PATCH 40/41] docs(architecture-reset): label a6aeb8c as CI-proven RC preparation head The final branch head of the RC1 gate is the candidate; a6aeb8c is the CI-proven RC preparation head (run 31242827310, SUCCESS). Prevents mislabeling an intermediate SHA as the final candidate once the reporting commit exists. --- docs/architecture-reset-status.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/architecture-reset-status.md b/docs/architecture-reset-status.md index d6d278b..333ccff 100644 --- a/docs/architecture-reset-status.md +++ b/docs/architecture-reset-status.md @@ -35,8 +35,9 @@ fb5641c remote main (published v0.1.2-integrity base) |---|---| | `775630eebdfb7c4b8357a4d1976505109b4b085b` | **Migration/export implementation head** — product implementation closed here (10 commits from `b9e46c1`); 7-pass code-reviewed (0 critical / 0 major); 420 tests green; literal-head CI green; PR evidence comment id `5224200252`. | | `c70a6f314b2329dc3e134546ad20b41519be98ab` | **Documentation/reporting head** — architecture-reset status marked the migration/export gate complete/stopped for senior review (docs-only commit). | -| `a6aeb8ca9f0ac571ef06e3d8d5e8a1b032379e2a` | **RC1 candidate head** — prepared by the RC1 Candidate Preparation Gate: version identity `2.0.0rc1`, release metadata cleanup, clean package build + isolated install proof, release-gate CI strengthening (build + isolated install + packaged smoke steps), RC1 evidence. Literal-head CI run `31242827310` **SUCCESS** on this exact SHA (3.11 job `93066497291`, 3.12 job `93066497309`; all 10 steps green). RC1 candidate — **pending independent senior acceptance and operator integration gate.** Not released. | -| *(final branch head)* | **Reporting head** — this document records the RC1 candidate head above and the gate evidence (docs-only commit). | +| `a9fafbfd80dfe0d431930b0be772a703db3fd55f` | **RC1 preparation commit** — version identity `2.0.0rc1`, release metadata cleanup, candidate-head documentation, release-gate CI strengthening. | +| `a6aeb8ca9f0ac571ef06e3d8d5e8a1b032379e2a` | **RC preparation head (CI-proven)** — CI-only correction (isolated-venv import provenance; CWD shadowing fix). Literal-head CI run `31242827310` **SUCCESS** on this exact SHA (3.11 job `93066497291`, 3.12 job `93066497309`; all 10 steps green). | +| *(final branch head of this gate)* | **RC1 candidate head** — the exact final branch head produced by the RC1 Candidate Preparation Gate (this reporting commit and its parent RC preparation commits). RC1 candidate — **pending independent senior acceptance and operator integration gate.** Not released. | Do not call the RC1 candidate "released". The eventual Git release/tag identity is `v2.0.0-rc.1`; the tag is NOT created in this gate. @@ -95,8 +96,9 @@ clean-install/package proof, release-gate CI strengthening, RC1 evidence. No product-semantic changes were authorized; the product implementation remains frozen pending final senior review. -- **RC1 candidate head:** `a6aeb8ca9f0ac571ef06e3d8d5e8a1b032379e2a` (see Head - lineage table above). +- **RC1 candidate head:** the exact final branch head of this gate (RC + preparation commits `a9fafbf` + `a6aeb8c` + this reporting commit; the + CI-proven RC preparation head is `a6aeb8c`, see Head lineage table above). - Version identity: `2.0.0rc1` across `pyproject.toml`, `methodfactory/ __init__.py`, packaging tests, and CLI version tests. Eventual Git tag identity `v2.0.0-rc.1` — NOT created in this gate. From ac090e879beb737603527ad756a0206a752ff8a3 Mon Sep 17 00:00:00 2001 From: RedEyeNinja-BKK <232920946+RedEyeNinja-BKK@users.noreply.github.com> Date: Sat, 8 Aug 2026 14:25:22 +0700 Subject: [PATCH 41/41] chore(release): close RC1 evidence identity RC1 evidence & documentation closure gate (bounded; no product changes): - CI checkout identity: actions/checkout now checks out the actual PR head SHA (ref: github.event.pull_request.head.sha for pull_request, github.sha otherwise) instead of GitHub's synthetic PR merge ref; fetch-depth: 0 retained for fb5641c fixture generation. New 'Assert checkout identity' step prints actual git rev-parse HEAD vs expected SHA and fails CI on mismatch - the post-closure run is literal-commit proof. - Wheel/sdist claim: CI step renamed to 'Build distributable wheel'; removed the ineffective 'pip download --no-binary :all:' sdist line; asserts exactly one methodfactory-2.0.0rc1-*.whl before install. sdist is local supplemental evidence only. - Docs: prior runs 31242827310/31243084558 correctly described as PR synthetic-merge-tree runs (candidate-tree-equivalent, NOT literal-head); 31243084558 checked out synthetic merge 706d6bd (tree identical to 93ec1af, zero changed files); Phase 2 stop-gate entry marked accepted; progression table added; action-hash summary corrected to the exact six-field set {protocol_version, action, package_id, action_id, basis, payload} with expected_revision the sole excluded field; sdist wording corrected; final candidate wording avoids hard-coding the unknown SHA. Full suite: 420 tests green. --- .github/workflows/release-gate.yml | 39 ++++++++++++---- docs/architecture-reset-status.md | 75 +++++++++++++++++++++++------- 2 files changed, 90 insertions(+), 24 deletions(-) diff --git a/.github/workflows/release-gate.yml b/.github/workflows/release-gate.yml index ad44172..7aad20d 100644 --- a/.github/workflows/release-gate.yml +++ b/.github/workflows/release-gate.yml @@ -18,12 +18,33 @@ jobs: matrix: python-version: ["3.11", "3.12"] steps: + # Explicitly check out the ACTUAL pull-request HEAD SHA, not GitHub's + # synthetic merge ref (actions/checkout defaults to the merge ref on + # pull_request). This makes the CI evidence literal-commit proof: the + # tested tree is exactly the branch head, not a candidate-tree- + # equivalent merge. - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0 (pinned SHA) with: + ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }} # Full history required: migration tests generate fixtures from the # exact public v0.1.2 commit fb5641c (tag v0.1.2-integrity), which # is not present in a depth-1 checkout of this branch. fetch-depth: 0 + + # Literal-commit identity assertion: the checked-out HEAD MUST equal the + # event's expected SHA. A mismatch fails CI. This is the proof that a + # successful run tested the exact commit, not a synthetic merge tree. + - name: Assert checkout identity + run: | + ACTUAL="$(git rev-parse HEAD)" + EXPECTED="${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}" + echo "actual HEAD: $ACTUAL" + echo "expected SHA: $EXPECTED" + if [ "$ACTUAL" != "$EXPECTED" ]; then + echo "::error::checkout identity mismatch (actual != expected)"; exit 1 + fi + echo "checkout identity OK: tested tree is the exact expected commit" + - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 (pinned SHA) with: python-version: ${{ matrix.python-version }} @@ -41,12 +62,13 @@ jobs: test "$(mf --version)" = "methodfactory 2.0.0rc1" test "$(python -m methodfactory --version)" = "methodfactory 2.0.0rc1" - # RC1 packaging proof — build the actual distributable wheel/sdist from - # the EXACT candidate source into a disposable directory OUTSIDE the - # repo, so the tracked-worktree and artifact-scan steps below stay - # clean. No new build dependency: pip wheel/download use the declared - # setuptools build backend. - - name: Build distributable package (wheel + sdist) + # RC1 packaging proof — build the distributable WHEEL from the EXACT + # candidate source into a disposable directory OUTSIDE the repo, so the + # tracked-worktree and artifact-scan steps below stay clean. No new + # build dependency: pip wheel uses the declared setuptools backend. + # (sdist is NOT claimed by CI; any sdist evidence is local-only + # supplemental unless independently CI-produced.) + - name: Build distributable wheel run: | set -euo pipefail export MF_DIST="$RUNNER_TEMP/mf-dist" @@ -54,12 +76,13 @@ jobs: git archive HEAD | tar -x -C "$RUNNER_TEMP/mf-src" cd "$RUNNER_TEMP/mf-src" python -m pip wheel . --no-deps -w "$MF_DIST" - python -m pip download . --no-deps --no-binary :all: -d "$MF_DIST" ls -la "$MF_DIST" python - <<'EOF' import hashlib, os, pathlib dist = pathlib.Path(os.environ["MF_DIST"]) - for p in sorted(dist.iterdir()): + wheels = sorted(dist.glob("methodfactory-2.0.0rc1-*.whl")) + assert len(wheels) == 1, f"expected exactly one rc1 wheel, got {[p.name for p in dist.iterdir()]}" + for p in wheels: print(p.name, hashlib.sha256(p.read_bytes()).hexdigest()) EOF diff --git a/docs/architecture-reset-status.md b/docs/architecture-reset-status.md index 333ccff..3d873b8 100644 --- a/docs/architecture-reset-status.md +++ b/docs/architecture-reset-status.md @@ -33,11 +33,11 @@ fb5641c remote main (published v0.1.2-integrity base) | Head | Role | |---|---| -| `775630eebdfb7c4b8357a4d1976505109b4b085b` | **Migration/export implementation head** — product implementation closed here (10 commits from `b9e46c1`); 7-pass code-reviewed (0 critical / 0 major); 420 tests green; literal-head CI green; PR evidence comment id `5224200252`. | +| `775630eebdfb7c4b8357a4d1976505109b4b085b` | **Migration/export implementation head** — product implementation closed here (10 commits from `b9e46c1`); 7-pass code-reviewed (0 critical / 0 major); 420 tests green; PR-event CI green on the candidate-equivalent tree (pre-literal-checkout workflow); PR evidence comment id `5224200252`. | | `c70a6f314b2329dc3e134546ad20b41519be98ab` | **Documentation/reporting head** — architecture-reset status marked the migration/export gate complete/stopped for senior review (docs-only commit). | | `a9fafbfd80dfe0d431930b0be772a703db3fd55f` | **RC1 preparation commit** — version identity `2.0.0rc1`, release metadata cleanup, candidate-head documentation, release-gate CI strengthening. | -| `a6aeb8ca9f0ac571ef06e3d8d5e8a1b032379e2a` | **RC preparation head (CI-proven)** — CI-only correction (isolated-venv import provenance; CWD shadowing fix). Literal-head CI run `31242827310` **SUCCESS** on this exact SHA (3.11 job `93066497291`, 3.12 job `93066497309`; all 10 steps green). | -| *(final branch head of this gate)* | **RC1 candidate head** — the exact final branch head produced by the RC1 Candidate Preparation Gate (this reporting commit and its parent RC preparation commits). RC1 candidate — **pending independent senior acceptance and operator integration gate.** Not released. | +| `a6aeb8ca9f0ac571ef06e3d8d5e8a1b032379e2a` | **RC preparation head** — CI-only correction (isolated-venv import provenance; CWD shadowing fix). PR-event run `31242827310` **SUCCESS** on the candidate-equivalent tree (3.11 job `93066497291`, 3.12 job `93066497309`). NOTE: that run used actions/checkout's default PR synthetic-merge ref — candidate-tree-equivalent, NOT literal-commit identity proof. | +| *(current branch head)* | **RC1 candidate** — pending independent senior acceptance. The exact final SHA of this evidence-closure commit is recorded in the post-CI evidence report / PR comment, not hard-coded into this document (avoids a self-referential commit). RC1 candidate — **pending independent senior acceptance and operator integration gate.** Not released. | Do not call the RC1 candidate "released". The eventual Git release/tag identity is `v2.0.0-rc.1`; the tag is NOT created in this gate. @@ -58,7 +58,7 @@ candidate preparation).** - Local suite: **420 tests green** (356 baseline + 64 migration/export). - Code review: 7 passes of the mandatory family (bug/security/performance/ quality → verify → dedupe → sanity); final verdict 0 critical / 0 major. -- Literal-head CI: run `31236027530` on `775630e` **SUCCESS** — 3.11 job +- PR-event CI: run `31236027530` on `775630e` **SUCCESS** — 3.11 job `93048557007` and 3.12 job `93048556968`, both `Ran 420 tests … OK`, clean-worktree + artifact-scan steps success. - PR #1 evidence comment: id `5224200252` (32-point gate checklist). @@ -104,10 +104,11 @@ remains frozen pending final senior review. identity `v2.0.0-rc.1` — NOT created in this gate. - Release metadata: `Development Status :: 3 - Alpha` → `4 - Beta`; stale "Phase 2 foundation" wording removed from the project description. -- Clean package build: wheel + sdist built from the exact candidate source - (`a6aeb8c` via `git archive`) in a disposable directory; recorded locally - — wheel `methodfactory-2.0.0rc1-py3-none-any.whl`, sdist - `methodfactory-2.0.0rc1.tar.gz`; no artifacts committed or left in the +- Clean package build: wheel + sdist built locally from the exact candidate + source (`a6aeb8c` via `git archive`) in a disposable directory — wheel + `methodfactory-2.0.0rc1-py3-none-any.whl`, sdist + `methodfactory-2.0.0rc1.tar.gz`. The WHEEL is CI-backed; the sdist is + LOCAL supplemental evidence only. No artifacts committed or left in the worktree. - Fresh-environment install: wheel installed into a disposable venv with no editable checkout; import provenance (venv site-packages), `__version__`, @@ -117,14 +118,47 @@ remains frozen pending final senior review. `mf migrate-store` → authoritative validation → installed `mf export` (both formats) → legacy evidence export revalidated by the frozen legacy reader; source unchanged. -- Release-gate CI: run `31242827310` on the exact candidate head `a6aeb8c` - **SUCCESS** on 3.11 and 3.12 — canonical tests, wheel+sdist build, +- Release-gate CI: run `31242827310` on the candidate-equivalent tree of + `a6aeb8c` **SUCCESS** on 3.11 and 3.12 — canonical tests, wheel build, isolated wheel install + version proof, packaged migration/export smoke, - clean-worktree, artifact-scan all green. + clean-worktree, artifact-scan all green. (That run and run `31243084558` + were PR synthetic-merge checkouts — candidate-tree-equivalent, not + literal-commit identity proof; see the evidence closure section below.) +- Clean package build: WHEEL `methodfactory-2.0.0rc1-py3-none-any.whl` is + CI-backed. The sdist `methodfactory-2.0.0rc1.tar.gz` is LOCAL supplemental + evidence only (not independently CI-produced; RC1's required distributable + proof is the wheel). - RC1 candidate — **pending independent senior acceptance and operator integration gate.** Not released. No tag, no GitHub Release, no PyPI publication, no deployment. +## RC1 evidence & documentation closure gate (2026-08-08) + +Bounded evidence/documentation closure authorized by Vincent. The RC1 +product tree is frozen; this gate corrects evidence honesty only (CI +checkout identity, wheel/sdist claim accuracy, stale status wording). + +- **Checkout identity fix:** `actions/checkout` now explicitly checks out the + actual PR head SHA (`ref: ${{ github.event_name == 'pull_request' && + github.event.pull_request.head.sha || github.sha }}`) instead of GitHub's + synthetic merge ref, with `fetch-depth: 0` retained for `fb5641c` fixture + generation. A new `Assert checkout identity` step prints actual + `git rev-parse HEAD` and the expected SHA and fails CI on mismatch. The + post-closure run is the literal-commit proof. +- **Prior-run terminology correction:** runs `31242827310` and `31243084558` + were PR-event runs that checked out GitHub's synthetic PR merge ref — NOT + literal-head. They were candidate-tree-equivalent where proven. Run + `31243084558` checked out synthetic merge `706d6bd04417baac12fe73a78a7378656c21f9d9` + (`Merge 93ec1af… into fb5641c…`); independent GitHub comparison proved the + tree of `706d6bd` equals the tree of `93ec1af` (tree `62c2fdb2c9`, + zero changed files). Those runs must not be called literal-head. +- **Wheel/sdist claim:** CI builds and proves the WHEEL only. The ineffective + `pip download . --no-binary :all:` sdist line was removed; the wheel step + asserts exactly one `methodfactory-2.0.0rc1-*.whl` exists before install. + Any sdist evidence is local-only supplemental. +- Final candidate wording: the exact final SHA is recorded in the + post-CI evidence report / PR comment, not hard-coded into this commit. + ## Senior review 4878235332 (2026-08-07) — accepted - SQLite reset remains **APPROVED IN PRINCIPLE**; corrected evidence package closes the prior evidence gap. @@ -135,7 +169,14 @@ remains frozen pending final senior review. 3. `refactor: introduce storage protocol and canonical serialization primitives` 4. `feat: add SQLite schema creation, identity checks, and append-only guards` 5–8. (later) transactional apply; migration + exports; test evidence; docs alignment. -- **Phase 2 stop gate:** after commits 1–4, return head SHA, ADR diff summary, DDL + triggers, database-open state table, action-hash definition, successful CI run on the exact SHA, local unit results, `EXPLAIN QUERY PLAN`, clean `git status --short`, and confirmation of no merge/tag/release/`main` change. Do not proceed to the full lifecycle until the senior reviewer accepts this gate. +- **Phase 2 stop gate (historical — ACCEPTED):** after commits 1–4, the gate required returning head SHA, ADR diff summary, DDL + triggers, database-open state table, action-hash definition, successful CI run on the exact SHA, local unit results, `EXPLAIN QUERY PLAN`, clean `git status --short`, and confirmation of no merge/tag/release/`main` change. The senior reviewer accepted this gate; it is no longer pending. + +**Progression (current):** +1. ✅ Foundation accepted (Phase 2 stop gate, senior review `4878235332`). +2. ✅ Transactional persistence accepted (invariant closure, senior review `4885538290`). +3. ✅ Migration/export implemented + 7-pass reviewed (implementation head `775630e`; verdict 0 critical / 0 major) — accepted pending this final evidence closure. +4. ⏳ RC1 candidate (`2.0.0rc1`) pending independent senior acceptance. +5. ⏳ Merge / tag / release — operator-gated; NOT authorized by any gate so far. ## ADR-0012 amendment (commit 1, done) @@ -147,7 +188,7 @@ The amendment closes all 12 review items: 4. Physical DB contract — `methodfactory.sqlite3` under store root; `application_id` `0x4D465354`; `user_version` 1; full state table (missing/zero-byte/wrong-ID/future-version/corrupt/legacy-only/sqlite-only/neither/both); read-only URI; no accidental creation. 5. Append-only executable — UPDATE/DELETE rejection triggers in binding DDL. 6. Revision/chain invariants frozen — rev 0 create + `state_before IS NULL`; predecessor required; state/digest match; manifest columns agree; one authoritative validator. -7. `action_sha256` defined — hash of canonical `{action, package_id, action_id, basis, payload}`; excludes only `expected_revision`. +7. `action_sha256` defined — hash of the canonical semantic action over the exact six-field set `{protocol_version, action, package_id, action_id, basis, payload}`; `expected_revision` is the SOLE excluded envelope field. (ADR-0012 remains authoritative.) 8. Artifact boundary — blobs before txn, content-addressed, verified before insert, orphan-safe; no auto-delete during mutation; GC proves global unreachability. 9. Migration — v0.1.2 layout; explicit source/dest; fail-closed existing dest; same-filesystem atomic rename; no overwrite; receipt durable and part of success. 10. Evidence checksum — archive-root-relative `SHA256SUMS`; `cd && sha256sum -c SHA256SUMS` exits zero. @@ -163,6 +204,8 @@ The amendment closes all 12 review items: 1. ✅ ADR-0012 reviewed from the pushed branch (senior review 4878235332). 2. ✅ Operator GO to proceed with foundation + SQLite implementation (Phase 2 authorization). -3. ⏳ Phase 2 stop gate: commits 1–4 + CI evidence + PR comment; senior reviewer acceptance. -4. Cumulative release-candidate review (`fb5641c..rc`, 7 lanes, ADR-0012 §11). -5. Operator approval to merge / tag / release (not currently authorized). +3. ✅ Phase 2 stop gate: commits 1–4 + CI evidence + PR comment; senior reviewer acceptance (accepted). +4. ✅ Transactional persistence + invariant closure accepted (senior review 4885538290). +5. ✅ Migration/export implementation accepted pending final evidence closure (implementation head 775630e; 7-pass review, 0 critical / 0 major). +6. ⏳ RC1 candidate (`2.0.0rc1`) pending independent senior acceptance (evidence closure complete). +7. ⏳ Operator approval to merge / tag / release (not currently authorized).