|
| 1 | +# Workflow Resumability: Model and Direction |
| 2 | + |
| 3 | +This note describes how a `Workflow` node preserves and restores execution state |
| 4 | +across a human-in-the-loop pause, how that compares to peer agent frameworks, |
| 5 | +and the direction we are moving in. It complements `checkpoint-resume.md`, which |
| 6 | +covers the interrupt/resume lifecycle for a single node. |
| 7 | + |
| 8 | +The first thing to be clear about: a `Workflow` reconstructs its progress from |
| 9 | +the session event stream on every run, so it resumes whether or not resumability |
| 10 | +is configured. The `is_resumable` flag does not switch resume on or off. What it |
| 11 | +switches on is **durability** — persisting loadable checkpoints and letting an |
| 12 | +invocation be continued across separate runner calls. |
| 13 | + |
| 14 | +## Current model |
| 15 | + |
| 16 | +### Resume is always on: reconstruction by event replay |
| 17 | + |
| 18 | +On every run, the workflow scans the current invocation's session events and |
| 19 | +rebuilds its in-memory loop state (which nodes completed, their outputs, which |
| 20 | +are still waiting on interrupts). Completed nodes are fast-forwarded from their |
| 21 | +recovered output rather than re-executed; the interrupted node re-runs with the |
| 22 | +supplied responses. |
| 23 | + |
| 24 | +This path (`_run_impl` -> `ReplayManager.scan_workflow_events`) has no |
| 25 | +`is_resumable` guard — the scan matches events purely by invocation id. The loop |
| 26 | +state is not persisted and there is no separate workflow checkpoint to load; the |
| 27 | +session event log is the source of truth. So within an invocation, a workflow is |
| 28 | +inherently replay-resumable, flag or no flag. This is exactly what deanchen |
| 29 | +means by "still resumable even when resumability is not set." |
| 30 | + |
| 31 | +### What `is_resumable` actually adds: durability |
| 32 | + |
| 33 | +`ResumabilityConfig.is_resumable` is a durability switch. When it is set: |
| 34 | + |
| 35 | +1. **Cross-call resume.** The runner will continue an existing invocation |
| 36 | + without a fresh user message and set up a "resumed invocation" context, |
| 37 | + rehydrating the recorded agent state and end-of-agent markers from history. |
| 38 | + With the flag off, the runner requires a new message and starts a fresh |
| 39 | + invocation instead. |
| 40 | +2. **Checkpoint markers in the log.** Composite agents (`LlmAgent`, |
| 41 | + `SequentialAgent`, `LoopAgent`, `ParallelAgent`, and `LlmAgent`s wrapped as |
| 42 | + workflow nodes) write `agent_state` / `end_of_agent` events into the session |
| 43 | + only when the invocation is resumable. These make progress loadable across a |
| 44 | + process boundary. |
| 45 | +3. **Function-response routing.** Routing an incoming function response back to |
| 46 | + its originating invocation is enabled only when resumable. |
| 47 | + |
| 48 | +The config's own definition is durability-shaped: pause an invocation on a |
| 49 | +long-running call, and resume it from the last event if it paused or failed |
| 50 | +midway, best-effort and at-least-once, with in-memory state lost. So the |
| 51 | +accurate statement is: the flag decides whether progress is persisted as |
| 52 | +loadable checkpoints and whether an invocation can be resumed across runner |
| 53 | +calls — not whether the workflow can resume. Resumability here is really |
| 54 | +durability. |
| 55 | + |
| 56 | +### The `Workflow` node emits no checkpoint of its own |
| 57 | + |
| 58 | +Today the `Workflow` node does not persist a node-status checkpoint (a `nodes` |
| 59 | +payload of statuses/outputs). It relies solely on event replay. The `nodes` |
| 60 | +shape exists only as an input to graph visualization, not as something the |
| 61 | +runtime writes during a run. The only checkpoint events on the workflow path |
| 62 | +come from wrapped composite agents emitting their own `agent_state`, and those |
| 63 | +are gated on the flag as above. |
| 64 | + |
| 65 | +## How peer frameworks do it |
| 66 | + |
| 67 | +Every mainstream agent framework persists a **state snapshot with a position |
| 68 | +cursor** and, on resume, **loads that snapshot** — none reconstruct by replaying |
| 69 | +the entire history. |
| 70 | + |
| 71 | +Framework | Durable unit | Position cursor | Resume |
| 72 | +------------------------ | ------------------------------------------------------------------- | ---------------------------------------------------------------------------------- | ------ |
| 73 | +LangGraph (graph) | `StateSnapshot` per super-step in a pluggable checkpointer | `next` nodes + parent-pointer chain + per-task pending writes | re-invoke same thread id; load latest checkpoint, re-run only the interrupted node |
| 74 | +pydantic-graph (graph) | `NodeSnapshot{state, node, status}` via a state-persistence backend | the snapshot's `node` = next node to run; `status` created/pending/running/success | `iter_from_persistence()` loads the next `created` snapshot |
| 75 | +OpenAI Agents SDK (loop) | serialized `RunState` blob | run cursor inside the state; correlate by tool-call id | deserialize the state, apply approvals, resume the run |
| 76 | +Pydantic AI (loop) | message history + deferred-tool results | implicit in the transcript; correlate by tool-call id | new run over the prior message history |
| 77 | + |
| 78 | +ADK's durable unit today is the event log itself, and resume is by replay over |
| 79 | +it. Two patterns from the peers are worth copying, both consistent across them: |
| 80 | + |
| 81 | +- **A snapshot is the source of truth for resume.** The runtime writes a |
| 82 | + snapshot as it advances and reads the latest one to continue. Resume cost is |
| 83 | + bounded by the snapshot size, not by history length. |
| 84 | +- **Resume re-runs the paused unit from its start** (LangGraph and |
| 85 | + pydantic-graph both re-execute the whole node, not a saved program counter), |
| 86 | + which keeps the durable state small and pushes an idempotency contract onto |
| 87 | + the node author — the same at-least-once contract ADK already documents. |
| 88 | + |
| 89 | +## Direction: persist a workflow checkpoint as the durable source of truth |
| 90 | + |
| 91 | +Even with durability on, the `Workflow` node reloads by replaying the event |
| 92 | +history rather than loading a compact checkpoint. The direction — peer-aligned, |
| 93 | +and the one ADK's own composite agents already follow — is to persist a workflow |
| 94 | +checkpoint and load the latest one on resume: |
| 95 | + |
| 96 | +- As the workflow advances, persist node statuses and outputs as a checkpoint |
| 97 | + (an `agent_state` payload), the way composite agents already persist theirs. |
| 98 | +- On resume, seed the loop state from the most recent checkpoint, then |
| 99 | + continue: re-run only the interrupted node and dispatch newly-ready |
| 100 | + successors. |
| 101 | +- This makes resume cost independent of history length and unifies the |
| 102 | + `Workflow` node with composite agents and with LangGraph / pydantic-graph / |
| 103 | + the OpenAI SDK. |
| 104 | + |
| 105 | +This only applies when durability is on. Without `is_resumable` there is nothing |
| 106 | +to persist, and the workflow continues to resume within an invocation by replay |
| 107 | +as it does today. |
| 108 | + |
| 109 | +## Open considerations |
| 110 | + |
| 111 | +- **Payload completeness.** A workflow checkpoint must carry (or be able to |
| 112 | + recover) each completed node's output, run id, and branch — the equivalent |
| 113 | + of LangGraph's per-task pending writes — to fully replace event replay. |
| 114 | +- **Partial interrupt resolution.** A node with several interrupts may receive |
| 115 | + only some responses on a resume. The re-run-vs-wait behavior differs between |
| 116 | + an orchestrating node (re-run to dispatch the resolved branch) and a leaf |
| 117 | + node (wait for all), keyed on `rerun_on_resume`. This is decided in the |
| 118 | + shared replay-interception logic and should be settled before a load path |
| 119 | + relies on it. |
| 120 | +- **Versioning.** Long-lived paused runs can outlive a code change. A version |
| 121 | + marker on the checkpoint lets a resume route to a compatible code path (the |
| 122 | + OpenAI SDK makes this an explicit recommendation). |
| 123 | +- **Serialization.** Keep payloads JSON-serializable (Pydantic `model_dump`), |
| 124 | + so any persistence backend works and no code objects are serialized; node |
| 125 | + objects are rebound from the in-memory graph definition on resume. |
0 commit comments