Skip to content

feat(storage): transactional agent authority - #1393

Merged
yzxoi merged 15 commits into
devfrom
codex/transactional-agent-storage
Sep 15, 2026
Merged

yzxoi merged 15 commits into
devfrom
codex/transactional-agent-storage

Conversation

@EricSanchezok

@EricSanchezok EricSanchezok commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Agent authority now commits through an explicit Runtime-owned storage handle instead of independent JSON files. Session, message, inbox, workflow, approval and registry mutations can commit their records, derived indexes, receipts and notifications together. SQLite remains the default; PostgreSQL 16–18 implements the same contract with explicit namespace ownership.

Existing installations upgrade through an immutable byte-for-byte backup, resumable import checkpoints, registered owner migrations, relationship validation and a single activation boundary. Malformed historical Session records retain their evidence and block execution; global corruption stops activation. There is no live JSON mirror. Pack, merge and move use checksummed logical exports and preserve files and Git snapshots separately, including skipped conflicting Session aggregates.

The change also covers bounded streaming writes, stale-writer fencing, worker shutdown, interrupted Session import/fork cleanup, plugin installation recovery, target-switch recovery, patched macOS SQLite packaging, maintenance commands, generated contracts, architecture documentation and operational recovery guidance. Runtime failover and simultaneous replicas sharing a namespace remain separate architecture work.

Validation completed locally:

  • bun run quality:quick — all 17 gates passed, including typecheck, generated documentation, dead-code and workflow checks.
  • bun test test/storage with real PostgreSQL 16 — 61 tests passed, including the SQLite contract, owner-loss recovery and three released JSON upgrade fixtures.
  • Real CLI cancellation: four direct/process-group SIGINT and SIGTERM cases retained terminal evidence and accounting.
  • Targeted fault tests for Session/Note/Blueprint atomicity, staged import recovery, interrupted target activation, worker loss and plugin installation recovery passed.
  • Workspace package suites passed through the serialized Turbo run and complete reruns of Workbench and Product Runtime after fixture corrections; Product Runtime passed 1,447 tests.
  • Benchmark Python contracts: 248 passed; 19 opt-in integrations were skipped locally. Docker, native harness and streaming integrations run separately in CI.
  • The 30 MiB / 30,720-checkpoint completed and failed Rollout tests passed locally; completed, cancelled and failed outcomes passed in CI.
  • bun packages/product-runtime/script/build.ts --single --skip-install succeeded.
  • bun test test/cli/artifact.test.ts against the compiled product — all six isolated, outside-repository execution tests passed.

CI includes a mandatory PostgreSQL 16/17/18 matrix in addition to the repository's existing Linux, Windows, coverage and installed-artifact checks. Required check results and the complete job matrix are attached to this pull request.

EricSanchezok and others added 4 commits September 14, 2026 21:25
Replace JSON authority with Runtime-owned SQLite and PostgreSQL storage,
atomic domain mutations, durable receipts and notifications, and recoverable
legacy upgrades and logical data transfers. Preserve artifact ownership,
unknown domain data, rollback evidence and installed runtime portability.

Co-authored-by: synergy-agent <299070056+synergy-agent@users.noreply.github.com>
Activate verified authority after a successful full offline migration and
initialize the isolated Library migration fixture with its own storage handle.

Co-authored-by: synergy-agent <299070056+synergy-agent@users.noreply.github.com>
Use durable revisions instead of millisecond timestamps when clearing search
invalidation, and exercise offline migration activation through the real CLI.

Co-authored-by: synergy-agent <299070056+synergy-agent@users.noreply.github.com>
Keep bundled workflow loading independent of SQLite initialization and verify
its real worker and storage lifecycle in a compiled isolated fixture. Align
inbox assertions with atomic transcript publication and queue consumption.

Co-authored-by: synergy-agent <299070056+synergy-agent@users.noreply.github.com>
@oryn-mini

oryn-mini Bot commented Sep 14, 2026

Copy link
Copy Markdown

🧹 Oryn Mini · ✅ Task complete

Code review · Attempt 1 · Cycle 1

Waiting for your decision: Merge once the pending required checks finish, or hold until the acknowledged follow-ups are tracked as linked issues?
Validation has not run. No repair patch was published by this attempt.

View the Actions run

Run details · operational receipt
{
  "version": 1,
  "workKey": "review:4ea7be348cc337c012711ac0",
  "sourceKey": "4ea7be348cc337c012711ac0",
  "intent": "review",
  "phase": "completed",
  "attempt": 1,
  "cycle": 1,
  "leaseID": "cc5475e2-f65e-440f-b38f-d442f2c1d6de",
  "verdict": "no_action",
  "triage": {
    "category": "feature",
    "priority": "P1",
    "confidence": "high",
    "reproduction": "source_proven",
    "proof": "sufficient",
    "decision": "keep_open",
    "closeReason": "none",
    "implementationCandidate": false,
    "complexity": "large",
    "evidence": [
      {
        "path": "packages/harness/src/session/part-write-buffer.ts",
        "line": 57,
        "sha": "7cbef7dfcc31f7fef5e4b2a0150b812608aea4c0",
        "detail": "Success handler deletes the retained failure, so a later write for the same key self-heals instead of wedging the session (review blocker fixed)."
      },
      {
        "path": "packages/harness/src/session/part-write-buffer.ts",
        "line": 99,
        "sha": "7cbef7dfcc31f7fef5e4b2a0150b812608aea4c0",
        "detail": "flushWhere retries retained failures at every drain boundary; keys with a newer buffered entry flush that entry and its success clears the failure."
      },
      {
        "path": "packages/harness/test/session/part-write-buffer.test.ts",
        "line": 83,
        "sha": "7cbef7dfcc31f7fef5e4b2a0150b812608aea4c0",
        "detail": "Regression test: a transient part failure does not block later drains of the same session."
      },
      {
        "path": "packages/harness/src/storage/sqlite-worker.ts",
        "line": 9,
        "sha": "7cbef7dfcc31f7fef5e4b2a0150b812608aea4c0",
        "detail": "Worker sets umask 077 before opening the database so the db and WAL sidecars stay owner-only (review finding S-1 fixed)."
      },
      {
        "path": "packages/harness/src/storage/sqlite-driver.ts",
        "line": 101,
        "sha": "7cbef7dfcc31f7fef5e4b2a0150b812608aea4c0",
        "detail": "Driver chmods db, -wal and -shm to 0600 on open, including sidecars left by older engines."
      },
      {
        "path": "packages/harness/test/storage/transactional-store.test.ts",
        "line": 173,
        "sha": "7cbef7dfcc31f7fef5e4b2a0150b812608aea4c0",
        "detail": "Regression test asserts the -wal sidecar mode is 0600."
      },
      {
        "path": "packages/harness/src/storage/sqlite-driver.ts",
        "line": 132,
        "sha": "7cbef7dfcc31f7fef5e4b2a0150b812608aea4c0",
        "detail": "Maintenance requests get a 600s deadline instead of the 30s shared deadline, so integrity_check no longer kills the worker (review finding P-2 fixed)."
      },
      {
        "path": "packages/harness/src/storage/transactional-store.ts",
        "line": 620,
        "sha": "7cbef7dfcc31f7fef5e4b2a0150b812608aea4c0",
        "detail": "PRAGMA integrity_check is issued with { maintenance: true } to use the extended deadline."
      },
      {
        "path": "packages/harness/src/storage/portable.ts",
        "line": 46,
        "sha": "7cbef7dfcc31f7fef5e4b2a0150b812608aea4c0",
        "detail": "authorityRecordRoots (plugin approvals/audit/lock/runtime-state/incompatible/intents, permissions, permission-rules, registry) defined with the consent-suppression rationale."
      },
      {
        "path": "packages/product-runtime/src/cli/data/transfer.ts",
        "line": 109,
        "sha": "7cbef7dfcc31f7fef5e4b2a0150b812608aea4c0",
        "detail": "Untrusted data merge refuses authority-adjacent record roots (review finding S-2 fixed)."
      },
      {
        "path": "packages/product-runtime/src/cli/data/move.ts",
        "line": 252,
        "sha": "7cbef7dfcc31f7fef5e4b2a0150b812608aea4c0",
        "detail": "Same-home relocation opts in via { trusted: true }, the only transfer allowed to carry authority records."
      },
      {
        "path": "packages/harness/src/storage/bootstrap.ts",
        "line": 256,
        "sha": "7cbef7dfcc31f7fef5e4b2a0150b812608aea4c0",
        "detail": "Bootstrap convenience import also refuses authority roots because the archive may originate from another home."
      },
      {
        "path": "packages/product-runtime/test/cli/data-transfer.test.ts",
        "line": 70,
        "sha": "7cbef7dfcc31f7fef5e4b2a0150b812608aea4c0",
        "detail": "Regression test: merge refuses authority records from another home while trusted relocation keeps them."
      },
      {
        "path": "packages/harness/src/session/message-v2.ts",
        "line": 1420,
        "sha": "7cbef7dfcc31f7fef5e4b2a0150b812608aea4c0",
        "detail": "messageOrderSnapshot restored on Storage.transaction — the read-only snapshot rewrite was reverted (d32db644c) after the Cortex read-after-write regression, keeping the other fixes."
      },
      {
        "path": "docs/decisions/implemented/architecture/2026-09-14-transactional-agent-authority.md",
        "line": 3,
        "sha": "7cbef7dfcc31f7fef5e4b2a0150b812608aea4c0",
        "detail": "Implemented decision record documents the problem, decision, alternatives and consequences, linking agent-storage architecture and upgrade migration docs."
      }
    ],
    "risks": [
      "This review is source-level; behavioral assurance relies on author-reported local runs and the partially completed CI matrix — PostgreSQL 18, Quality, Typecheck and several shards were still pending at snapshot.",
      "Activation is one-way: downgrade needs a restored Home and loses post-snapshot changes; recovery depends on the immutable backup discipline being followed.",
      "Message-order cache-hit reads still take the write transaction after the deliberate revert, so reads can queue behind the single SQLite writer under concurrent streaming.",
      "storage_nodes rows are never pruned, so the index grows with historical writes on long-lived homes until the pruning follow-up lands."
    ],
    "decisionNote": {
      "question": "Merge once the pending required checks finish, or hold until the acknowledged follow-ups are tracked as linked issues?",
      "options": [
        "Merge as soon as the pending required checks finish; file the follow-up issues in parallel.",
        "Hold merge until follow-up issues for storage_nodes pruning, retry consolidation, metadata() grammar and the read-path optimization exist and are linked.",
        "Request a second independent review of the post-review fix commits before merging."
      ],
      "recommendation": "Merge once the pending required checks finish — the approval is pinned to head and covers the fixes — and open the four follow-up issues in the same pass so the acknowledged work stays visible."
    }
  },
  "run": "https://github.com/SII-Holos/synergy/actions/runs/34965022267",
  "summary": "Waiting for your decision: Merge once the pending required checks finish, or hold until the acknowledged follow-ups are tracked as linked issues?\nValidation has not run. No repair patch was published by this attempt."
}

@oryn-mini

oryn-mini Bot commented Sep 14, 2026

Copy link
Copy Markdown

🧹 Oryn Mini · 🧭 Your input is needed

🟠 P1 · High · 💡 Feature · High confidence

Why this PR

This PR replaces independent JSON file writes with a Runtime-owned transactional storage handle, so session, message, inbox, workflow, approval and registry mutations commit atomically on SQLite (default) or PostgreSQL 16–18, with immutable backups, resumable import and recovery paths for existing installs. It matters because cross-record invariants and plugin-consent records previously could tear or lose updates, and a single transient streaming-write error could wedge a session until restart. All five issues raised during review are fixed and source-verified at head; the message-order read-path optimization was reverted after a real Cortex…

Full explanation

This PR replaces independent JSON file writes with a Runtime-owned transactional storage handle, so session, message, inbox, workflow, approval and registry mutations commit atomically on SQLite (default) or PostgreSQL 16–18, with immutable backups, resumable import and recovery paths for existing installs. It matters because cross-record invariants and plugin-consent records previously could tear or lose updates, and a single transient streaming-write error could wedge a session until restart. All five issues raised during review are fixed and source-verified at head; the message-order read-path optimization was reverted after a real Cortex read-after-write regression and remains a tracked follow-up.


What changed

  • Before: each record wrote its own JSON file, so a session mutation could commit without its indexes, receipts or notifications; after: one TransactionalStore transaction commits records, derived indexes, receipts and the notification outbox together.
  • Before: JSON layouts migrated ad hoc; after: bootstrap seals an immutable byte-for-byte backup, imports with resumable checkpoints and owner ledgers, validates relationships, then activates SQL at a single boundary; global corruption stops activation.
  • Before: one transient part-write failure was retained forever and wedged the session until restart; after: a later successful write clears it and every drain boundary retries retained failures, with regression tests for same-key recovery and later turns.
  • Before: WAL sidecars inherited world-readable modes; after: the storage worker runs under umask 077 and the driver re-tightens db/-wal/-shm to 0600 on open, including sidecars left by older engines.
  • Before: untrusted data merge could import plugin approvals, permissions and registry records and silently satisfy a later consent prompt; after: authorityRecordRoots are refused on untrusted merge and convenience import; only same-home data move carries them (trusted).
  • Before: PRAGMA integrity_check ran under the 30s IPC deadline and could kill the worker on large homes; after: maintenance statements carry a 600s deadline through the IPC contract.

How it fits together

The Runtime Handle owns a transactional store committing session aggregates, receipts and notifications atomically on SQLite (subprocess IPC) or PostgreSQL; JSON installs upgrade via backup and import; the data CLI moves logical exports.

flowchart TD
  accTitle: Transactional agent storage at a glance
  accDescr: The Runtime Handle owns a transactional store committing session aggregates, receipts and notifications atomically on SQLite (subprocess IPC) or PostgreSQL; JSON installs upgrade via backup and import; the data CLI moves logical exports.
  n_0["Runtime Storage Handle"]
  n_1["Transactional logical-record store"]
  n_2["SQLite IPC worker subprocess"]
  n_3["PostgreSQL driver"]
  n_4["Session aggregate mutations"]
  n_5["Durable notification outbox"]
  n_6["JSON import and backups"]
  n_7["data pack/merge/move CLI"]
  n_0 -->|"owns"| n_1
  n_1 -->|"SQLite contract"| n_2
  n_1 -->|"same contract"| n_3
  n_4 -->|"atomic commit"| n_1
  n_1 -->|"publish after commit"| n_5
  n_6 -->|"upgrade path"| n_1
  n_7 -->|"logical exports"| n_1
Loading

Review findings

🔵 Low · Message-order cache-hit reads still take the write transactionpackages/harness/src/session/message-v2.ts:1420

The P-1 read-path optimization (read-only snapshot for the cache probe) was deliberately reverted after it broke Cortex read-after-write visibility for finalized subagent turns (4 failing tests, root-caused and reproduced by the specialist review). messageOrderSnapshot again wraps the probe in Storage.transaction, so reads can queue behind the single SQLite writer under concurrent streaming. Correct trade-off for now; the optimization should only re-land once read visibility is proven.

🔵 Low · storage_nodes index grows without pruningpackages/harness/src/storage/transactional-store.ts:192

Node rows are insert-only (ON CONFLICT DO NOTHING) with no delete path, so the index grows with historical writes on long-lived homes; recursive scans and verification walk the full tree. Acknowledged non-blocking follow-up from the review thread.

🔵 Low · Maintainability follow-ups acknowledged by reviewpackages/harness/src/storage/transactional-store.ts

The specialist review flagged duplicated retry/backoff policy across the store and driver layers and the metadata() key grammar as an undocumented coupling point for future indexed domains. Both were accepted as non-blocking; they should be tracked as issues so they are not lost after merge.

More findings (1)

🔵 Low · defer() intentionally fails fast on a retained failurepackages/harness/src/session/part-write-buffer.ts:17

After a transient part-write failure, defer() rethrows the retained error until a drain boundary retries it (flushWhere) or a successful writeNow clears it. This fail-fast is deliberate — it prevents silently buffering onto a broken store — and is covered by the regression tests. Future readers should not 'fix' it back into silent buffering.


Before merge

  • Wait for the pending required checks — PostgreSQL 18, Quality, Typecheck, Smoke Test, coverage shards, native-harness and long-rollout jobs were still queued or in progress at snapshot; the PR is merge-blocked until they finish.
  • The approving review is pinned to head 7cbef7d, so it covers the post-review fix commits; no re-review needed unless new commits land.
  • Rollout: activation is one-way; downgrade requires a restored Home and loses post-snapshot changes — ship docs/migrations/transactional-agent-storage.md guidance with the release notes.
  • Open linked issues for the acknowledged non-blocking follow-ups: storage_nodes pruning, retry-layer consolidation, metadata() key-grammar documentation, and the deferred message-order read-path optimization.
  • Skim the new docs/architecture/agent-storage.md and the decision record — they now define the durable storage boundary future changes must follow.

Maintainer decision: Merge once the pending required checks finish, or hold until the acknowledged follow-ups are tracked as linked issues?

  • Merge as soon as the pending required checks finish; file the follow-up issues in parallel.
  • Hold merge until follow-up issues for storage_nodes pruning, retry consolidation, metadata() grammar and the read-path optimization exist and are linked.
  • Request a second independent review of the post-review fix commits before merging.

Recommendation: Merge once the pending required checks finish — the approval is pinned to head and covers the fixes — and open the four follow-up issues in the same pass so the acknowledged work stays visible.

Things to watch

  • This review is source-level; behavioral assurance relies on author-reported local runs and the partially completed CI matrix — PostgreSQL 18, Quality, Typecheck and several shards were still pending at snapshot.
  • Activation is one-way: downgrade needs a restored Home and loses post-snapshot changes; recovery depends on the immutable backup discipline being followed.
  • Message-order cache-hit reads still take the write transaction after the deliberate revert, so reads can queue behind the single SQLite writer under concurrent streaming.
  • storage_nodes rows are never pruned, so the index grows with historical writes on long-lived homes until the pruning follow-up lands.

Verification

🔎 Evidence supports the conclusion. Confirmed in source; not reproduced in a live run.

No host validation commands were run for this report.

Source evidence (15)

Discussion and CI reflect the snapshot read for this review. Current checks and approval are verified separately before merging.

🧹 Oryn Mini · Source 7cbef7d · Built with Synergy core

@oryn-mini oryn-mini Bot added oryn:kind: 💡 feature A proposed capability or behavior change. oryn:priority: 🟠 P1 High-priority impact supported by the review. oryn:proof: 🔎 supported Evidence supports this conclusion; see the report for source vs live proof. oryn:status: 🧭 needs input A maintainer decision or additional information is required. labels Sep 14, 2026
Co-authored-by: synergy-agent <299070056+synergy-agent@users.noreply.github.com>
@oryn-mini oryn-mini Bot added oryn:proof: 🧩 partial Available evidence does not yet establish the claim. and removed oryn:proof: 🔎 supported Evidence supports this conclusion; see the report for source vs live proof. labels Sep 14, 2026
EricSanchezok and others added 3 commits September 14, 2026 22:27
Regenerate the core configuration schema and update lifecycle, Push, observability and benchmark probes to use explicit storage ownership and committed records.

Co-authored-by: synergy-agent <299070056+synergy-agent@users.noreply.github.com>
Give the POSIX SQLite worker its own process group so SIGINT and SIGTERM cannot close authority before the Runtime drains cancellation evidence. Verify real group signals, terminal accounting and owner-loss recovery.

Co-authored-by: synergy-agent <299070056+synergy-agent@users.noreply.github.com>
Exercise real bootstrap, recovery, inspection and target switching through CLI commands. Keep fresh Runtime composition and coverage attribution consistent across test runners.

Co-authored-by: synergy-agent <299070056+synergy-agent@users.noreply.github.com>
@oryn-mini oryn-mini Bot added oryn:proof: 🔎 supported Evidence supports this conclusion; see the report for source vs live proof. and removed oryn:proof: 🧩 partial Available evidence does not yet establish the claim. labels Sep 14, 2026
@yzxoi

yzxoi commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Review Summary: Request Changes (1 blocker)

This is a high-quality, unusually well-engineered refactor. Atomicity/crash-safety wiring verified sound across the session domain (derived indexes commit with canonical writes, durable outbox notifications never precede commit, correct two-phase journal saga, clean reserve-publish-discard staging, no JSON dual-write leftovers, deriveSemantics contract untouched). SQL is fully parameterized, path traversal validation is complete, and the fault tests use real subprocess kills rather than mocks.

Four issues should be addressed before merge — one blocking, three strongly recommended (all small, localized fixes):


🔴 Blocker: PartWriteBuffer sticky failure wedges a session permanently (part-write-buffer.ts)

  • execute() records a failure in this.failures (line 61) and no code path ever clears it (zero failures.delete/.clear() in the file).
  • defer() (17-18) and writeNow() (74-75) rethrow the retained error before executing — so a later terminal write for the same key can never self-heal.
  • flushWhere() (93-94) re-reports every retained matching failure on every drain; flushPartWrites(sessionID) filters by session, so one transient part failure poisons every subsequent turn of that session until Runtime restart — even after the database recovers.
  • The buffer lives for the store's lifetime (Storage.state() in session/index.ts:1461), so restart is the only recovery.

Reachability is real: SQLite enforces per-operation deadlines, so one transient BUSY/deadline during a timed streaming flush is enough.

Suggested fix: clear the retained failure when a subsequent writeNow for the same key succeeds (failures.delete(key) in the success handler), plus a regression test that a transient part failure does not block later turns.

🟠 WAL sidecars are world-readable (permissions regression vs JSON era)

PRAGMA journal_mode = WAL (sqlite-worker.ts:24) creates -wal/-shm under the process umask (typically 0644); the main DB is chmod'd 0600 (sqlite-driver.ts:90) but the sidecars never are. On a multi-user host, recently committed session content and plugin approvals leave the 0600 discipline the legacy JSON layer guaranteed (atomic-file.ts:33). Fix: chmod 0600 on the sidecars after WAL setup (or a restrictive umask in the worker before open).

🟠 data merge accepts auth-adjacent record roots

The merge/bootstrap accept-filters (product-runtime/src/cli/data/transfer.ts:18-25, 93-103, harness/src/storage/bootstrap.ts:241-254) exclude only local/derived sets. A crafted archive can pre-place plugin-approvals/records/<id> with a broad grant, permissions, registry, plugin-lock, etc.; buildApprovalReview then finds a sufficient existing approval and suppresses the consent prompt for a later plugin install. Pre-existing in kind (old merge copied plugin-approvals.json too), but this PR owns the boundary now — please extend the reject lists to the auth-adjacent roots.

🟠 Read path takes a write transaction (message-v2.ts:1452-1462)

messageOrderSnapshot wraps the cache probe in Storage.transaction() — even a cache hit pays a full BEGIN/COMMIT round-trip through the subprocess writer slot. readInfoList (model context assembly) calls it per read; under active streaming + concurrent readers this queues behind the single writer and surfaces as StorageBusyError from defer(). Storage.snapshot (read-only) exists for exactly this; use it for the probe and escalate to a transaction only when a rebuild is needed.

🟠 30s IPC deadline kills the worker during integrity_check (sqlite-driver.ts:112-116)

verify() runs PRAGMA integrity_check as one request under the 30s per-request deadline. On a multi-GB home this can exceed the deadline → worker killed → driver permanently closed. Fails safe, but data storage verify self-destructs on exactly the datasets that need verification most. Suggest a deadline-exempt path for maintenance statements or chunked checking.


Non-blocking follow-ups (tracked for later)

  • storage_nodes rows are never deleted — index grows linearly with historical writes (transactional-store.ts:186-194)
  • Retry policy split between store and driver layers with duplicated backoff formula (transactional-store.ts:527 / postgres-driver.ts:81)
  • metadata() key grammar is an undocumented coupling point for every future indexed domain (transactional-store.ts:55-81)
  • Orphan SessionMutation import (session/index.ts:60); unreachable untrusted branch in pending.ts:82-88; "__storage-worker-runner" magic string duplicated across three entry points
  • Merge copies archive symlinks verbatim (cli/cmd/data/shared.ts:259-275)

Verified clean

  • All storage SQL parameterized; namespace/limit/key-segment validation complete; no data interpolation into statements
  • Import integrity: strict schemas, checksummed archives re-hashed end-to-end, identity cross-checks before activation, target-must-be-empty rule
  • Plugin approval persistence strengthened vs dev (atomic transactions + checksummed recovery snapshots)
  • deriveSemantics/isSystemPart contract untouched; PostgreSQL URL hygiene clean

Validation performed locally

quality:quick 17/17 gates; test/storage 52 pass / 1 skip (needs real PG); test/session 779 pass; test/migration 42 pass; product-runtime orchestrated run, CLI/workflows/note/browser-runtime/library/workbench/runtime-local suites all green; CI matrix including PostgreSQL 16/17/18 all passing.

Review performed by parallel specialist review (security, core storage maintainability, session-domain integration) plus independent verification of every finding cited above. Fixes incoming.

yzxoi and others added 4 commits September 15, 2026 15:10
…the writer

A retained part failure now clears when a later write for the same key
succeeds and retries at every drain boundary, so one transient storage
error cannot wedge a session until Runtime restart. Message order reads
use a read-only snapshot and escalate to a transaction only when the
stored index needs rebuilding.

Co-authored-by: synergy-agent <299070056+synergy-agent@users.noreply.github.com>
…m the request deadline

The worker enforces an owner-only umask so the database and its WAL
sidecars never appear world-readable, and the driver re-tightens them
to 0600 on open to repair sidecars from older engines. Integrity
verification runs under an extended deadline instead of killing the
worker mid-maintenance on large datasets.

Co-authored-by: synergy-agent <299070056+synergy-agent@users.noreply.github.com>
…rchive

Grants, consent and trust decisions (plugin approvals, permissions,
registry and related plugin state) never cross homes through an
untrusted data merge or the bootstrap convenience import: an imported
approval would silently satisfy the consent prompt for a later plugin
install. Same-home relocation (data move) opts in explicitly.

Co-authored-by: synergy-agent <299070056+synergy-agent@users.noreply.github.com>
…ity transfer boundaries

Co-authored-by: synergy-agent <299070056+synergy-agent@users.noreply.github.com>
@yzxoi

yzxoi commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

All five findings from the review above are fixed and pushed (aaa5fd0..01a0cf7):

  • PartWriteBuffer sticky failure — a retained failure now clears when a later write for the same key succeeds and retries at every drain boundary (failures.delete in the success handler + retry-on-drain in flushWhere); writeNow no longer rethrows a stale error before executing. Regression tests cover same-key recovery and "transient failure does not block later turns of the session".
  • WAL sidecars world-readable — the worker now enforces an owner-only umask before opening the database, and the driver re-tightens ``, -wal, `-shm` to 0600 on open (also repairing sidecars left by older engines). Regression test asserts 0600 on all three files.
  • Auth-adjacent record roots importable via data merge — a shared authorityRecordRoots set (plugin approvals/audit/lock/runtime-state/incompatible/install-intents, permissions, permission-rules, registry) is refused by untrusted data merge and the bootstrap convenience import; data move (same-home relocation) opts in via { trusted: true }. Regression test covers both directions.
  • Read path on the write transactionmessageOrderSnapshot removed; loadMessageOrder reads through a read-only snapshot and escalates to its own transaction only when the stored index needs rebuilding. Cache-hit reads no longer touch the single writer slot.
  • 30s deadline kills worker during integrity_check — maintenance statements now carry an extended deadline (600s) through the IPC contract (SqlQueryOptions.maintenance); verify() passes it for PRAGMA integrity_check.

Docs updated (agent-storage architecture, decision record, migration guide). Full local validation: quality:quick 17/17, test/storage 64 pass (incl. new permission test), test/session 781 pass, product-runtime data-transfer 3 pass (incl. new authority rejection test), isolated CLI maintenance suites pass.

Non-blocking follow-ups from the review (storage_nodes pruning, retry-layer consolidation, metadata() key-grammar documentation) remain tracked in the review comment above.

The read-only snapshot rewrite of loadMessageOrder broke Cortex task
output: subagent assistant messages were read back empty after their
finalizing turn (summary extraction, final_response capture, usage
reads and durable output all lost). Root cause was isolated by reverting
only message-v2.ts at bf7813c while keeping the part-write-buffer
recovery; the original messageOrderSnapshot semantics are restored until
the read path can move off the writer without changing read-after-write
visibility.

Co-authored-by: synergy-agent <299070056+synergy-agent@users.noreply.github.com>
@yzxoi

yzxoi commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Follow-up: CI regression fixed by reverting the message-order read-path change (d32db64).

The push at bf7813c broke 4 Cortex tests in two shards (channels / product-platform): subagent assistant messages were read back empty after their finalizing turn — summary extraction found no messages, final_response captured empty text, usage reads missed the budget trigger, and durable task output came back without its result body.

Root cause isolation (all reproduced locally):

  • baseline aaa5fd039: Cortex manager suite 55/55 pass
  • bf7813ce8 (buffer recovery + order-read rewrite): 4 fail
  • revert only message-v2.ts while keeping the part-write-buffer fix: 55/55 pass

The read-only snapshot rewrite of loadMessageOrder changed read-after-write visibility for the just-finalized turn, so the P-1 optimization is reverted wholesale. The original messageOrderSnapshot semantics are restored until the read path can move off the single SQLite writer without changing read visibility (tracked with the other non-blocking follow-ups).

Kept intact: the PartWriteBuffer sticky-failure recovery (B-1), WAL sidecar 0600 permissions (S-1), authority-record rejection on untrusted merge (S-2), and the maintenance deadline exemption (P-2). Session suite re-verified after the revert: 781 pass / 0 fail.

yzxoi and others added 2 commits September 15, 2026 18:45
Keep transaction-owned parts out of streaming retries, quarantine malformed
legacy session aggregates before owner migrations, and bundle the verified
SQLite engine in macOS Harness archives. Cover rollback, released upgrades,
and packed-artifact initialization with regression tests.

Co-authored-by: synergy-agent <299070056+synergy-agent@users.noreply.github.com>
@yzxoi
yzxoi enabled auto-merge (squash) September 15, 2026 11:46
@yzxoi
yzxoi merged commit 53ed8f1 into dev Sep 15, 2026
44 checks passed
@yzxoi
yzxoi deleted the codex/transactional-agent-storage branch September 15, 2026 12:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

oryn:kind: 💡 feature A proposed capability or behavior change. oryn:priority: 🟠 P1 High-priority impact supported by the review. oryn:proof: 🔎 supported Evidence supports this conclusion; see the report for source vs live proof. oryn:status: 🧭 needs input A maintainer decision or additional information is required.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants