Preserve canonical model transcripts end to end

DESIGN.md

@@ -157,18 +157,27 @@ schema.
 - **Durable replay**: The runtime persists canonical transcript deltas as
   runlog records so providers, replay tooling, and future backends can
   reconstruct the exact message order generically.
+- **Planner-transparent provenance**: Each model call produces an isolated
+  canonical response before planner code observes completion. Streams expose
+  only closed typed presentation events and carry the canonical response
+  separately through gateways. The runtime identifies tool turns from unchanged
+  model-facing calls and terminal turns from the canonical provider message
+  returned by its response helpers. It commits the complete selected response
+  once after atomic admission and before effects. Planners never manage
+  transcript handles or provider replay metadata, and uncertain ownership fails
+  instead of selecting by call order or visible text.
 - **History compression**: Agent designs may declare compression defaults with
   `CompressAtTurns`, `CompressAtMaxInputTokens`, `KeepMaxTurns`, and
   `KeepMaxInputTokens`. The runtime evaluates token budgets with the configured
   model client's exact `model.TokenCounter`, so tokenization stays
   deployment/model-specific while the design records the agent's default policy.
   Exact retention always keeps whole recent turns; it never truncates
   tool_use/tool_result pairs to satisfy a token budget.
-- **Bookkeeping control plane**: `Bookkeeping()` tool results stay durable for
-  hooks, streams, and run logs, but they are not replayed into future
-  planner-facing transcript/tool-output state. A bookkeeping-only turn must
-  therefore resolve in the same turn via a terminal outcome or an await/pause
-  handshake.
+- **Bookkeeping control plane**: `Bookkeeping()` calls and results remain in the
+  provider transcript so signed model-authored parts replay without modification.
+  Successful bookkeeping results are omitted only from compact `ToolOutputs` and
+  do not force another planner turn. A bookkeeping-only turn must therefore
+  resolve in the same turn via a terminal outcome or an await/pause handshake.
 - **Forced finalization control plane**: when runtime caps or deadlines force
   finalization, planners may return terminal bookkeeping tools instead of a
   prose final answer. The runtime executes only `Bookkeeping()` + `TerminalRun()`

README.md

@@ -198,13 +198,8 @@ func (p *Planner) PlanStart(ctx context.Context, in *planner.PlanInput) (*planne
 		return &planner.PlanResult{ToolCalls: summary.ToolCalls}, nil
 	}
 	return &planner.PlanResult{
-		FinalResponse: &planner.FinalResponse{
-			Message: &model.Message{
-				Role:  model.ConversationRoleAssistant,
-				Parts: []model.Part{model.TextPart{Text: summary.Text}},
-			},
-		},
-		Streamed: true,
+		FinalResponse: summary.FinalResponse(),
+		Streamed:      true,
 	}, nil
 }
 ```
@@ -469,14 +464,14 @@ Tool("commit_report", "Commit final report", func() {
 })
 ```
 
-Bookkeeping tools do not consume the normal `MaxToolCalls` budget. Their events are still durable and streamed. Successful results stay hidden from future planner turns; retryable failures stay visible with their retry hints so the planner can repair the failed call.
+Bookkeeping tools do not consume the normal `MaxToolCalls` budget. Their events are still durable and streamed, and their provider transcript blocks remain intact. Successful results stay out of compact future `ToolOutputs`; retryable failures stay visible there with their retry hints so the planner can repair the failed call.
 
 Retry-owned tool restrictions filter budgeted work tools only. Bookkeeping tools
 remain available during correction turns, including terminal run tools that close
 the run, while caller `WithRestrictToTool` policy remains run-scoped and applies
 to every tool.
 
-The workflow runtime evaluates one admitted planner result as one step: it executes tool and await work, records durable and planner-facing outputs through one canonical path, then applies one transition policy to resume, finish, or finalize. A terminal payload may only accompany hidden, non-terminal bookkeeping side effects; budgeted tools, retryable bookkeeping failures, terminal tools, and awaits must be separate planner decisions.
+The workflow runtime evaluates one admitted planner result as one step: it executes tool and await work, records durable and planner-facing outputs through one canonical path, then applies one transition policy to resume, finish, or finalize. A terminal payload may only accompany non-resuming, non-terminal bookkeeping side effects; budgeted tools, retryable bookkeeping failures, terminal tools, and awaits must be separate planner decisions. Bookkeeping calls remain in the provider transcript so signed responses are never edited.
 
 ---
 
@@ -515,9 +510,23 @@ rt := runtime.New(
 
 For model streaming inside planners, choose one style per planner call:
 
-- `PlannerContext.PlannerModelClient(id)` is recommended. It owns assistant/thinking/usage event emission and returns a `planner.StreamSummary`.
+- `PlannerContext.PlannerModelClient(id)` is recommended for the selected, single model call. It owns assistant/thinking/usage event emission and returns a `planner.StreamSummary`.
 - `PlannerContext.ModelClient(id)` gives you a raw `model.Client`. Pair it with `planner.ConsumeStream` or drain the stream yourself when you need lower-level control.
 
+The runtime captures each model response before planner code sees it. When a
+planner probes through the raw client, goa-ai matches returned model-facing tool
+calls to the exact response that produced them and replays only that transcript.
+Every stream exposes closed typed presentation events, then makes its canonical
+response available separately after clean EOF. Model gateways carry that
+response independently from planner-facing chunks, and terminal helpers return
+the selected provider message without exposing transcript identity. Future
+session turns retain provider-authored thinking without inferring ownership from
+visible text.
+Planners keep the existing obligation to preserve model tool-call identities
+and, when compiling synthetic tools, `ModelName`/`ModelPayload`; they never
+manage transcript identities. The workflow commits the selected response once
+after atomic admission and before effects. Usage includes all attempts.
+
 ---
 
 ## MCP and Registries

codegen/agent/templates/agents_quickstart.go.tpl

@@ -478,7 +478,7 @@ if err := rt.RegisterToolset(reg); err != nil { panic(err) }
 * **Interrupts (Human-in-the-Loop):** If your policy allows it, you can pause and resume agent runs with `rt.PauseRun()` and `rt.ResumeRun()`.
 * **Policies & Caps:** The `RunPolicy` in your design (max tool calls, time budgets) is automatically enforced by the runtime.
 * **Persistence & Observability:** The `runtime.New` function accepts `runtime.Options` to configure production-grade components like a Temporal engine, MongoDB for memory, and telemetry hooks.
-* **Temporal DataConverter (required):** When you use the Temporal engine, configure the Temporal client with `temporal.NewAgentDataConverter(...)` to enforce goa‑ai's boundary contract: tool results and artifacts cross workflow boundaries as canonical JSON bytes (`api.ToolEvent` / `api.ToolArtifact`), and `planner.ToolResult` is rejected if it ever tries to cross a Temporal boundary.
+* **Temporal DataConverter:** The Temporal engine automatically installs its strict data converter when `ClientOptions.DataConverter` is unset. Explicit custom converters remain the caller's responsibility.
 * **Registries & Discovery:** When you declare registries and `FromRegistry(...)` toolsets in your DSL, Goa-AI generates typed registry HTTP clients under `gen/<svc>/registry/<name>/` plus per-toolset specs helpers (with `DiscoverAndPopulate`, `Specs`, and `RegistryToolsetID`) so you can discover tools at runtime and register executors using `runtime.ToolsetRegistration`.
 
 ```go
@@ -490,25 +490,18 @@ rt := runtime.New(runtime.Options{
 })
 ```
 
-Example: constructing a Temporal engine with the required DataConverter:
+Example: constructing a Temporal worker engine:
 
 ```go
 import (
     "goa.design/goa-ai/runtime/agent/engine/temporal"
     "go.temporal.io/sdk/client"
-
-    // Your generated tool specs aggregate.
-    // The generated package exposes: func Spec(tools.Ident) (*tools.ToolSpec, bool)
-    specs "<module>/gen/<service>/agents/<agent>/specs"
 )
 
-eng, err := temporal.New(temporal.Options{
+eng, err := temporal.NewWorker(temporal.Options{
     ClientOptions: &client.Options{
         HostPort:      "127.0.0.1:7233",
         Namespace:     "default",
-        // Required: enforce goa-ai's workflow boundary contract.
-        // Tool results/artifacts cross boundaries as canonical JSON bytes (api.ToolEvent/api.ToolArtifact).
-        DataConverter: temporal.NewAgentDataConverter(specs.Spec),
     },
     WorkerOptions: temporal.WorkerOptions{
         TaskQueue: "<service>_<agent>_workflow",

codegen/agent/templates/cmd_main.go.tpl

@@ -68,6 +68,7 @@ func (c *exampleCompletionClient) Complete(_ context.Context, req *model.Request
 				},
 			},
 		},
+		StopReason: "stop",
 	}, nil
 }
 
@@ -82,31 +83,48 @@ func (c *exampleCompletionClient) Stream(_ context.Context, req *model.Request)
 	}, nil
 }
 
-// Recv advances the example stream through a single preview delta and the final
-// canonical completion payload.
+// Recv advances the example stream through preview, completion, and stop.
 func (s *exampleCompletionStreamer) Recv() (model.Chunk, error) {
 	switch s.step {
 	case 0:
 		s.step++
 		end := min(len(s.payload), 24)
-		return model.Chunk{
-			Type: model.ChunkTypeCompletionDelta,
-			CompletionDelta: &model.CompletionDelta{
+		return model.CompletionDeltaChunk{
+			Delta: model.CompletionDelta{
 				Name:  s.name,
 				Delta: string(s.payload[:end]),
 			},
 		}, nil
 	case 1:
 		s.step++
-		return model.Chunk{
-			Type: model.ChunkTypeCompletion,
-			Completion: &model.Completion{
-				Name:    s.name,
-				Payload: rawjson.Message(append([]byte(nil), s.payload...)),
-			},
-		}, nil
+		completion := model.Completion{
+			Name:    s.name,
+			Payload: rawjson.Message(append([]byte(nil), s.payload...)),
+		}
+		return model.CompletionChunk{Completion: completion}, nil
+	case 2:
+		s.step++
+		return model.StopChunk{Reason: "completed"}, nil
 	default:
-		return model.Chunk{}, io.EOF
+		return nil, io.EOF
+	}
+}
+
+// Response returns the canonical example response after clean EOF.
+func (s *exampleCompletionStreamer) Response() *model.Response {
+	if s.step < 3 {
+		return nil
+	}
+	return &model.Response{
+		Content: []model.Message{
+			{
+				Role: model.ConversationRoleAssistant,
+				Parts: []model.Part{
+					model.TextPart{Text: string(s.payload)},
+				},
+			},
+		},
+		StopReason: "completed",
 	}
 }
 
@@ -209,8 +227,8 @@ func main() {
 			if err != nil {
 				log.Fatalf("completion stream failed: %v", err)
 			}
-			if chunk.Type == model.ChunkTypeCompletionDelta && chunk.CompletionDelta != nil {
-				fmt.Printf("Completion delta %s: %s\n", completions.{{ .GoName }}, chunk.CompletionDelta.Delta)
+			if delta, ok := chunk.(model.CompletionDeltaChunk); ok {
+				fmt.Printf("Completion delta %s: %s\n", completions.{{ .GoName }}, delta.Delta.Delta)
 			}
 			value, ok, err := completions.Decode{{ .GoName }}Chunk(chunk)
 			if err != nil {

docs/dsl.md

@@ -243,7 +243,7 @@ on agent/tool contracts.
 | `ResultReminder(text)`                        | Inside `Tool`                          | Static system reminder injected after tool result                                                   |
 | `Confirmation(dsl)`                           | Inside `Tool`                          | Declares that tool execution must be explicitly approved out-of-band                                |
 | `TerminalRun()`                               | Inside `Tool`                          | Marks tool as terminal: run completes immediately after execution                                   |
-| `Bookkeeping()`                               | Inside `Tool`                          | Marks tool as control-plane bookkeeping: no `MaxToolCalls` budget, hidden from future planner turns by default |
+| `Bookkeeping()`                               | Inside `Tool`                          | Marks control-plane work that consumes no `MaxToolCalls` budget and does not force another planner turn |
 
 
 ### Tool payload defaults (Feature)
@@ -1066,12 +1066,14 @@ future reasoning input.
 Runtime contract:
 
 - bookkeeping calls do not consume the run-level `MaxToolCalls` retrieval budget,
-- bookkeeping calls are never dropped when a batch is trimmed to fit the
-  remaining budget,
+- model-authored call batches are admitted or rejected atomically, with
+  bookkeeping calls excluded from the batch's budget cost,
 - bookkeeping results still publish durable run events for hooks, streams, and
   run-log consumers,
-- successful bookkeeping results stay hidden from the model-visible transcript
-  and future `ToolOutputs`.
+- all bookkeeping calls and results remain in the provider transcript so signed
+  responses replay unchanged,
+- successful bookkeeping results stay out of compact future `ToolOutputs` and
+  do not force another planner turn.
 
 This means a bookkeeping-only planner turn is only valid when the same turn
 already resolves without another reasoning resume (a `TerminalRun` tool, a

docs/overview.md

@@ -435,18 +435,19 @@ The hook bus (`runtime/agent/hooks`) is the internal pub/sub backbone for runtim
 
 Stream sinks bridge hook events to client‑facing formats via `stream.Subscriber`.
 
-### Transcript Ledger
-
-The transcript ledger (`runtime/agent/transcript`) maintains a provider‑precise record of the
-conversation needed to rebuild model payloads exactly:
-
-- **Provider Fidelity** — Preserves ordering and shape required by providers (thinking → tool_use →
-  tool_result)
-- **Stateless API** — Pure methods safe for workflow replay
-- **Provider‑Agnostic Storage** — Converts to/from provider formats at edges
-
-Use the ledger when you need deterministic conversation replay or provider‑specific payload
-reconstruction.
+### Canonical Transcripts
+
+Ordered `model.Message` parts are the single provider-neutral transcript
+representation. Model adapters produce complete canonical responses, the
+runtime captures them before planner code observes completion, and workflow
+state persists the selected messages directly. Thinking, citations, tool use,
+provider metadata, and part ordering therefore cross durable boundaries without
+being rebuilt through a second ledger shape. The durable codec is a strict
+tagged union: each part carries its `kind`, unknown fields fail decoding, and
+tool inputs remain raw JSON bytes so replay never changes numbers or object
+representation. Canonical message copies are deep and exhaustive across the
+sealed part union, so planner, reminder, and workflow ownership boundaries
+cannot share mutable provider state.
 
 ## DSL Reference
 
@@ -479,7 +480,7 @@ policies, and MCP servers within Goa service designs.
 | `BoundedResult()` | Mark result as bounded view over larger data |
 | `ResultReminder(text)` | Static system reminder injected after tool result |
 | `TerminalRun()` | Tool completes the run immediately after execution (no follow-up planner turn) |
-| `Bookkeeping()` | Control-plane tool: no `MaxToolCalls` budget and hidden from future planner turns by default |
+| `Bookkeeping()` | Control-plane tool: no `MaxToolCalls` budget and no automatic resume; provider transcript remains exact |
 
 ### Toolset Definition
 
@@ -630,7 +631,9 @@ out, err := client.Run(ctx, "session-1", messages,
     runtime.WithLabels(map[string]string{"env": "prod"}),
     runtime.WithTaskQueue("priority"),
     runtime.WithRunTimeBudget(5*time.Minute),
-    runtime.WithAllowedTags([]string{"safe"}),
+    runtime.WithTagPolicyClauses([]runtime.TagPolicyClause{{
+        AllowedAny: []string{"safe"},
+    }}),
 )
 
 // Asynchronous run
@@ -934,8 +937,6 @@ The `sessionID` argument is required and must be a non-empty, non-whitespace str
 | `WithRunFinalizerGrace(duration)`       | Reserve time for final message |
 | `WithRunInterruptsAllowed(bool)`        | Enable human-in-the-loop     |
 | `WithRestrictToTool(tools.Ident)`       | Limit available tools        |
-| `WithAllowedTags([]string)`             | Filter by tags               |
-| `WithDeniedTags([]string)`              | Exclude by tags              |
 | `WithTagPolicyClauses([]TagPolicyClause)` | Compose explicit tag clauses |
 | `WithTiming(Timing)`                    | Set multiple timing overrides |
 
@@ -944,10 +945,6 @@ repair turns only. They do not block validated terminal bookkeeping tools during
 forced finalization; caller `WithRestrictToTool` policy remains run-scoped and
 still applies.
 
-Prefer `WithTagPolicyClauses` for new code when you need multiple allow/deny
-rules; `WithAllowedTags` and `WithDeniedTags` are convenience shorthands for a
-single clause.
-
 `WithTiming(Timing)` sets semantic run/planner/tool budgets. It does not expose
 engine-level queue-wait or heartbeat tuning.