agent-operations-control-plane

A small, working reference implementation of the control layer that sits around a long-running agent run: explicit run states, an approval gate before consequential tool calls, bounded retries, pause and resume, fail-closed recovery after a crash, and an append-only audit log.

It has no runtime dependencies. Tests use Node's built-in node:test runner. The package supports Node 20 and newer and ships ESM JavaScript plus TypeScript declarations.

Install

Until a registry release is approved, install the tarball attached to a GitHub release:

npm install ./agent-operations-control-plane-0.1.0.tgz

After a future npm release, the equivalent command will be npm install agent-operations-control-plane.

To build that same tarball from a checkout, run npm ci && npm pack; prepack rebuilds the publishable library. The supported consumer surface is the package root. Internal file paths are intentionally not exported:

import {
  AuditLog,
  InMemoryAuditSink,
  InMemoryCheckpointStore,
  RunController,
  ToolRegistry,
  type ToolDefinition,
  type Workflow,
} from 'agent-operations-control-plane';

const echo: ToolDefinition = {
  name: 'echo',
  description: 'Return its JSON input',
  requiresApproval: false,
  async execute(input) {
    return input;
  },
};

const tools = new ToolRegistry().register(echo);
const workflow: Workflow = {
  id: 'echo-workflow',
  version: '1',
  description: 'A minimal package-consumer workflow',
  steps: [{
    id: 'echo-once',
    tool: 'echo',
    buildInput: (context) => context,
    applyResult: (_context, output) => ({ result: output }),
  }],
};
const auditSink = new InMemoryAuditSink();
const run = RunController.create({
  runId: 'example-run',
  initialContext: { message: 'hello' },
  workflow,
  tools,
  audit: new AuditLog([auditSink]),
  checkpoints: new InMemoryCheckpointStore(),
});

await run.start();

See examples/ for approval, retry, redaction, interruption, and recovery flows. Those examples import only from the public package entry point, so they double as consumer usage examples.

Reference implementation disclaimer

This repository is a fictional, clean-room reference implementation written to illustrate a set of control-plane patterns. It is not derived from, and does not describe, any production system, employer, or client. It contains no proprietary code, data, names, metrics, or configuration. The workflow in examples/ is invented and all tools are in-memory fakes. Treat it as teaching material and a starting point, not as a hardened library.

What it demonstrates

Concern	Where
Explicit run states and a transition table	src/state-machine.ts
Tool-call logging (start, success, failure, attempt number, input, output)	src/run-controller.ts, src/audit-log.ts
Approval gates before consequential actions	ToolDefinition.requiresApproval, RunController.approve / reject
Retries with a bounded attempt count that survives restarts	src/retry.ts, RunSnapshot.attempts
Pause and resume	RunController.requestPause / resume
Fail-closed step lifecycle and uncertain-step handling	RunController.executeStep, acknowledgeUncertainStep
Recovery after interruption with workflow-drift detection	RunController.recover, src/workflow.ts, src/checkpoint-store.ts
Validation of everything read back from storage	src/snapshot.ts
Redaction of audit data and persisted context	src/redaction.ts, RedactionPolicy
Append-only audit events	src/audit-log.ts
Deterministic tests, including adversarial ones	test/*.test.ts

Architecture

Workflow (ordered steps, each step = one tool call; validated + hashed at create/recover)
   |
   v
RunController --- reads/writes ---> RunSnapshot (state, stepIndex, context, approvals,
   |                                     |        attempts, inFlight, uncertainStep, workflowHash)
   |  records every action               |  persisted after every change (through RedactionPolicy)
   v                                     v
AuditLog --> AuditSink(s)          CheckpointStore
 (RedactionPolicy)                 (memory, one 0600 JSON file per run; validated on load)
   |
   v
ToolRegistry --> ToolDefinition.execute(input, {runId, stepId, attempt})

Run states

created ------> running <-----> awaiting_approval
   |               |  ^              |
   |               |  |              v
   |               |  +---------- paused
   |               v
   |          interrupted ----> running (via resume)
   |               |
   |               +----------> failed (operator fails an uncertain step)
   |
   +---> cancelled        running ---> completed | failed

The full table is TRANSITIONS in src/state-machine.ts. Every transition the controller makes goes through transition(from, to), which throws InvalidTransitionError for anything not in the table. completed, failed, and cancelled are terminal.

How a run executes

1. RunController.create validates the run id and the workflow (unique, non-empty step ids; every tool registered; positive integer retry limits), computes the workflow hash, writes a run_created event and the first checkpoint. Nothing is persisted if validation fails.
2. start() moves the run to running and enters the drive loop.
3. For each step the loop checks, in order: is a pause pending, are we out of steps, does this tool need approval that has not been granted. Any of those stops the loop and moves the run to paused, completed, or awaiting_approval.
4. Otherwise the step executes under runWithRetry. Before each attempt the attempt counter and an inFlight marker are persisted and a tool_call_started event is written, so a crash during the call is visible afterwards.
5. On success the step's applyResult merges the tool output into the run context, the step index advances, the inFlight marker is cleared, and a checkpoint_saved event is written. All of that is one logical update: if any part fails, the in-memory snapshot is rolled back.
6. On final tool failure the run moves to failed with lastError set.

Fail-closed step lifecycle

Anything that goes wrong around a tool call, as opposed to inside it, is handled by phase:

• Before execution (tool lookup, buildInput, the audit write or checkpoint write that precedes the call): the tool has not run. The run moves to failed with a step_failed event and lastError. The attempt is not counted.
• After execution (tool_call_succeeded audit write, applyResult, the checkpoint write that commits the step, or an external interrupt() that landed while the call was in flight): the tool has run and may have produced side effects that were never recorded. The run moves to interrupted with an uncertainStep marker, and resume() refuses to continue until an operator calls acknowledgeUncertainStep(operator, 'retry' | 'fail', reason). Retry limits are deliberately not consulted here: a reconciliation bug must not turn into three charges.
• Storage down entirely: if even the fail-closed checkpoint cannot be written, the controller still moves itself out of running in memory and throws UnrecoverablePersistError. Discard the controller and recover once storage is healthy.

Recovery

RunController.recover(runId, deps) loads the last checkpoint, validates its structure (types, bounds, timestamps, state/marker consistency, JSON-safety of the context), checks that it belongs to the given workflow id and that the workflow's content hash matches what the run was created with, and cross-checks approvals, attempts, and markers against the step list. Any mismatch throws (CheckpointValidationError, WorkflowDriftError) before a controller exists.

If the stored state is running, the process must have died mid-run. The run is moved to interrupted and a run_recovered event is written. If a tool call was in flight at the time (the inFlight marker is set), the step is additionally marked uncertain, exactly as for an after-execution failure: an operator must acknowledge it before resume() will re-run the step. Attempt counts live in the snapshot, so the retry budget is honoured across the restart.

The workflow hash covers the workflow id, version, and the ordered list of (step id, tool, maxAttempts). Function bodies cannot be hashed reliably, so a behavioural change inside buildInput or applyResult should be signalled by bumping Workflow.version.

Tools are resolved by name from a ToolRegistry at execution time. The registry is not part of the snapshot, so a recovered run can be given new tool instances (fresh clients, rotated credentials) without touching persisted state. The registry is validated against the workflow at recover time, so a missing tool is caught before anything runs.

Security and data handling

• Sinks and stores hold plaintext. Audit events carry full tool inputs, outputs, error messages, approver names, and rejection reasons; checkpoints carry the full run context. The bundled JsonlFileAuditSink and FileCheckpointStore write that to disk as-is (file mode 0600, directory mode 0700). Anything sensitive must be removed with a RedactionPolicy before it reaches them: redactFields([...]) strips named keys at any depth from both audit data and persisted context; allowlistAuditFields([...]) keeps only listed top-level audit fields. Context redaction applies to what is persisted, not to the in-memory context, so a recovered run only sees the redacted version and steps must be able to continue from it.
• Run ids are opaque and validated. They must match ^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$. The file store additionally resolves every path against its root, rejects anything that would land elsewhere, and refuses to read or write through symlinks.
• Nothing read from storage is trusted. Malformed JSON, unknown fields, wrong types, out-of-range values, non-canonical timestamps, prototype-polluting keys, and markers inconsistent with the state are all rejected with a clear error.
• No authentication or authorization. approve, reject, cancel, and acknowledgeUncertainStep take an operator name as a plain string and record it; they do not verify it. Wrap the controller in whatever access control your environment provides.
• No integrity protection. Checkpoints and audit files are validated for shape, not signed. Someone with write access to the store can alter run state or rewrite history. Append-only is a property of the sink interface and the file open mode, not a tamper-evident log.

Design decisions

• State is data, not behaviour. Everything needed to resume lives in one plain RunSnapshot object. The controller holds no other mutable state except an in-process pause flag. That is what makes recovery a matter of loading and validating JSON.
• The transition table is the spec. Allowed transitions are a literal object, tested exhaustively (every listed pair passes, every unlisted pair throws). Adding a state means editing one table and one test.
• Approvals are keyed by step id and stored in the snapshot. Step ids are validated unique at create and recover time, so one approval can only ever cover one step. A gated step will not execute until approvals[stepId] === 'approved', even after a restart. Rejection fails the run rather than skipping the step, because silently skipping a consequential action is worse than stopping.
• Retries are immediate and bounded; there is no backoff. runWithRetry re-invokes the tool back to back until it succeeds or maxAttempts is reached, all within one start() or resume() call. The attempt counter is checkpointed before each try, so a run recovered mid-retry still honours the bound. If you need delays between attempts, set maxAttempts: 1 and have an external scheduler call resume() later, or add the delay inside the tool.
• Infrastructure failures are not tool failures. An error from the audit log, the checkpoint store, buildInput, or applyResult is never retried and never leaves the run in running. It becomes failed if the tool had not run yet, or interrupted with an uncertain-step marker if it had.
• Audit sinks are append-only by interface. AuditSink has one method: append. The in-memory sink deep-freezes each event; the file sink only ever opens the file for appending. Sequence numbers are assigned by the AuditLog, not the sink. A sink that throws stops the run rather than leaving a gap in the trail.
• Pause is cooperative. While a tool call is in flight the controller cannot safely stop it, so a pause requested during running takes effect at the next step boundary. A pause requested while awaiting_approval takes effect immediately.
• Injected clock. Timestamps come from a Clock interface so tests and the example produce identical output every time.

Local commands

npm install
npm run typecheck   # tsc --noEmit (also aliased as npm run lint)
npm run build       # builds the publishable ESM library and declarations in dist/
npm test            # build, then run all 95 tests from a separate test output directory
npm run test:consumer # pack, install into a clean temp project, and import the public API
npm run example     # build, then run the fictional refund workflow
npm run pack:check  # inspect the publishable tarball manifest
npm run ci          # complete local CI gate

Requires Node 20 or newer.

Sample output

From npm run example, scenario A (approval gate, a flaky notification that succeeds on the second attempt, and a redaction policy that strips the customer's address from the audit trail and the checkpoint). Timestamps come from a fixed clock.

Scenario A: approval gate, bounded retry, redacted audit trail
  after start: state=awaiting_approval, waiting on step=issue
  after approve: state=completed
  final context (in memory): {"orderId":"ord-1001","reason":"damaged","customerEmail":"customer@example.test","totalCents":12000,"refundCents":12000,"policy":"full_refund","refundId":"rf-ord-1001-12000","messageId":"msg-rf-ord-1001-12000"}
  persisted context: {"orderId":"ord-1001","reason":"damaged","customerEmail":"[redacted]","totalCents":12000,"refundCents":12000,"policy":"full_refund","refundId":"rf-ord-1001-12000","messageId":"msg-rf-ord-1001-12000"}
  audit trail:
  #01 2026-01-01T09:00:01.000Z run_created              workflowId=refund-request-v1 workflowHash=d35385e777927399561d43ee7e118d93639732c99d79f8dfa0c31b3ea2a760a1 stepCount=4
  #02 2026-01-01T09:00:02.000Z state_changed            from=created to=running via=start
  #03 2026-01-01T09:00:03.000Z tool_call_started        stepId=lookup tool=lookup_order attempt=1 input={"orderId":"ord-1001"}
  #04 2026-01-01T09:00:04.000Z tool_call_succeeded      stepId=lookup tool=lookup_order attempt=1 output={"orderId":"ord-1001","customerEmail":"[redacted]","totalCents":12000,"status":"delivered"}
  #05 2026-01-01T09:00:05.000Z checkpoint_saved         stepIndex=1 completedStep=lookup
  #06 2026-01-01T09:00:06.000Z tool_call_started        stepId=calculate tool=calculate_refund attempt=1 input={"totalCents":12000,"reason":"damaged"}
  #07 2026-01-01T09:00:07.000Z tool_call_succeeded      stepId=calculate tool=calculate_refund attempt=1 output={"refundCents":12000,"policy":"full_refund"}
  #08 2026-01-01T09:00:08.000Z checkpoint_saved         stepIndex=2 completedStep=calculate
  #09 2026-01-01T09:00:09.000Z approval_requested       stepId=issue tool=issue_refund input={"orderId":"ord-1001","refundCents":12000}
  #10 2026-01-01T09:00:10.000Z state_changed            from=running to=awaiting_approval stepId=issue
  #11 2026-01-01T09:00:11.000Z approval_granted         stepId=issue approver=ops-reviewer
  #12 2026-01-01T09:00:12.000Z state_changed            from=awaiting_approval to=running via=resume
  #13 2026-01-01T09:00:13.000Z tool_call_started        stepId=issue tool=issue_refund attempt=1 input={"orderId":"ord-1001","refundCents":12000}
  #14 2026-01-01T09:00:14.000Z tool_call_succeeded      stepId=issue tool=issue_refund attempt=1 output={"refundId":"rf-ord-1001-12000"}
  #15 2026-01-01T09:00:15.000Z checkpoint_saved         stepIndex=3 completedStep=issue
  #16 2026-01-01T09:00:16.000Z tool_call_started        stepId=notify tool=notify_customer attempt=1 input={"recipient":"[redacted]","refundId":"rf-ord-1001-12000","refundCents":12000}
  #17 2026-01-01T09:00:17.000Z tool_call_failed         stepId=notify tool=notify_customer attempt=1 error=notification provider timeout (attempt 1)
  #18 2026-01-01T09:00:18.000Z retry_scheduled          stepId=notify nextAttempt=2 maxAttempts=3
  #19 2026-01-01T09:00:19.000Z tool_call_started        stepId=notify tool=notify_customer attempt=2 input={"recipient":"[redacted]","refundId":"rf-ord-1001-12000","refundCents":12000}
  #20 2026-01-01T09:00:20.000Z tool_call_succeeded      stepId=notify tool=notify_customer attempt=2 output={"messageId":"msg-rf-ord-1001-12000"}
  #21 2026-01-01T09:00:21.000Z checkpoint_saved         stepIndex=4 completedStep=notify
  #22 2026-01-01T09:00:22.000Z state_changed            from=running to=completed stepsExecuted=4

Scenario B in the same script abandons a run while a tool call is in flight, then recovers it from the checkpoint with a fresh controller. Because a call was in flight, the step is marked uncertain and resume() is refused until an operator acknowledges it. The relevant part of its trail:

  checkpoint left in state=running, stepIndex=3, inFlight={"stepId":"notify","attempt":1,"startedAt":"2026-01-01T10:00:16.000Z"}
  after recover: state=interrupted, uncertainStep=notify
  resume refused: Run run-b has an unresolved uncertain step notify (attempt 1): process ended while the tool call was in flight; its side effects are unknown. Call acknowledgeUncertainStep before resuming.
  after resume: state=completed
  ...
  #16 2026-01-01T10:00:16.000Z tool_call_started        stepId=notify tool=notify_customer attempt=1 ...
  #17 2026-01-01T10:00:17.000Z run_recovered            recoveredFrom=running stepIndex=3 inFlight={"stepId":"notify","attempt":1,"startedAt":"2026-01-01T10:00:16.000Z"}
  #18 2026-01-01T10:00:18.000Z uncertain_step_recorded  stepId=notify attempt=1 reason=process ended while the tool call was in flight; its side effects are unknown
  #19 2026-01-01T10:00:19.000Z state_changed            from=running to=interrupted reason=checkpoint was left in running state uncertainStep=notify
  #20 2026-01-01T10:00:20.000Z uncertain_step_resolved  stepId=notify attempt=1 decision=retry operator=ops-reviewer reason=notification provider dedupes on refundId
  #21 2026-01-01T10:00:21.000Z state_changed            from=interrupted to=running via=resume
  #22 2026-01-01T10:00:22.000Z tool_call_started        stepId=notify tool=notify_customer attempt=2 ...
  #23 2026-01-01T10:00:23.000Z tool_call_succeeded      stepId=notify tool=notify_customer attempt=2 output={"messageId":"msg-rf-ord-1001-9600"}

Test run summary from npm test:

# tests 95
# suites 18
# pass 95
# fail 0

test/hardening.test.ts holds the adversarial cases: duplicate and empty step ids, unknown tools, bad retry limits, unsafe run ids (../escaped, .., separators, over-long), symlinks inside the checkpoint directory, malformed and tampered checkpoint files, workflow drift between create and recover, buildInput / applyResult failures, audit sink and checkpoint store failures on either side of a tool call, an interrupt that lands while a call is in flight, and redaction of audit data and persisted context. test/consumer-install.mjs separately proves that the packed artifact installs and exposes its documented runtime entry point and declarations.

Release history is in CHANGELOG.md; the reviewed tag and artifact process is in RELEASING.md. Tag automation creates a GitHub release and attaches an npm tarball, but deliberately does not publish to npm.

Limitations

• Single process. Audit sequence numbers and the "already executing" guard are per controller instance. Running the same run from two processes at once is not prevented; a real deployment needs a lease or lock in the checkpoint store.
• Re-executed steps are not automatically idempotent. After an operator chooses retry for an uncertain step, the tool runs again. Tools with side effects must be idempotent (for example by keying on runId and stepId) or check for prior completion themselves. The control plane gives them the identifiers to do so and forces a human decision, but it cannot verify what the tool did.
• Plaintext persistence, no encryption. Audit files and checkpoints are written as readable JSON with restrictive file modes. Redaction is opt-in and key-based; it does not detect secrets it was not told about.
• No authentication, authorization, or integrity protection. See "Security and data handling" above.
• Linear workflows only. Steps run in order with no branching, fan-out, or conditional skipping.
• Approval is a single decision per step. There is no multi-party approval, expiry, or delegation.
• No scheduling. Retries within a step happen immediately. Nothing here decides when a paused or interrupted run should be resumed; it only records that it can be and makes it safe to do so.
• Workflow hash covers structure, not code. Changes inside step functions are invisible to drift detection unless Workflow.version is bumped.
• In-memory example tools. The example demonstrates control flow, not integration with real services.

License

MIT. See LICENSE.