Feature: Research-grade run logging and replay artifacts

[CompleteDotTech]

Background

#10 adds the first practical logging layer: MCP tool/http traces, smart-poll events, hashes/previews, and explicit agent decision annotations. For model evaluation and long-run debugging, the next step should specify the full research-grade logging contract in enough detail that logs are comparable across agents, game versions, machines, and future STS2MCP releases.

This issue is about the level of detail and artifact quality, not about prescribing one experimental methodology. The output should let different researchers or agent authors run their own analyses without guessing what a field means or whether two logs are comparable.

Goals

Produce run artifacts that are:

• complete enough to reconstruct the agent-visible trajectory;
• stable enough to compare across STS2MCP versions;
• compact enough to keep for many runs;
• explicit about redaction, truncation, hashes, and missing data;
• machine-readable without scraping markdown;
• useful for post-run review, replay, benchmarking, failure analysis, and model comparison.

Proposed artifact set

Each run should produce a directory or bundle with these files:

• manifest.json: run-level metadata and schema versions.
• events.jsonl: ordered event stream.
• states.jsonl: canonical state snapshots observed by the agent, preferably JSON.
• actions.jsonl: normalized requested actions and resolved outcomes.
• decisions.jsonl: agent-authored decision annotations.
• errors.jsonl: exceptions, HTTP failures, invalid actions, timeouts, and recovery notes.
• optional responses/: large raw payloads addressed by content hash when events.jsonl stores previews only.
• optional summary.json: derived post-run stats for quick indexing.

The exact filenames can change, but the separation of manifest, event stream, state snapshots, actions, decisions, errors, and large payload storage should be explicit.

Manifest detail

manifest.json should include at minimum:

• logging schema version;
• STS2MCP version and git commit when available;
• MCP bridge version or commit when available;
• STS2 game version;
• mod configuration relevant to API behavior;
• platform, OS, Python version, .NET/runtime version where available;
• start/end timestamps in UTC;
• monotonic duration;
• run id and optional externally supplied experiment id;
• profile id/name if safely available;
• character, ascension, seed, daily/custom run flags where safely available;
• singleplayer vs multiplayer mode;
• local player id for multiplayer, if safely available or hashed;
• logger settings: redaction rules, preview length, full-payload mode, compression mode;
• file list with byte sizes and SHA-256 hashes.

Event envelope

Every JSONL record should use a shared envelope:

{
  "schema_version": "...",
  "run_id": "...",
  "sequence": 123,
  "timestamp": "2026-05-10T00:00:00.000Z",
  "monotonic_ms": 12345.678,
  "event_type": "tool_call_result",
  "tool_call_id": "...",
  "parent_id": "...",
  "payload": {}
}

Required properties:

• sequence is strictly increasing within the run.
• timestamp is UTC ISO-8601 with millisecond precision or better.
• monotonic_ms is measured from logger start and never decreases.
• event_type is from a documented enum.
• payload shape is documented per event type.
• missing/unknown values use null plus an optional missing_reason, not ambiguous empty strings.

Event taxonomy

Document and emit, as applicable:

• session lifecycle: session_start, session_end, logger_flush, logger_error;
• tool lifecycle: tool_call_start, tool_call_result, tool_call_error;
• HTTP/API lifecycle: http_request, http_response, http_error;
• state lifecycle: state_observed, state_poll, state_changed, state_repeated, state_unavailable;
• action lifecycle: action_requested, action_accepted, action_rejected, action_resolved, action_timeout;
• decision annotations: agent_decision, agent_reflection, agent_plan_update;
• run lifecycle: run_started, act_started, floor_entered, combat_started, combat_turn_started, combat_ended, reward_started, run_ended;
• multiplayer lifecycle: mp_vote_started, mp_vote_updated, mp_vote_resolved, mp_player_ready_changed;
• recovery: retry, poll_timeout, manual_action_required, desync_suspected.

The first implementation does not need every event, but the enum should be stable and versioned before people build analysis tools around it.

State snapshot requirements

State records should include:

• state type and game mode;
• floor/act/room where available;
• full agent-visible JSON state, or a hash-addressed raw payload if too large;
• normalized hashes for current state and previous state;
• diff metadata when the state changed;
• whether the state was returned to the agent or only observed internally while polling;
• markdown response hash if the agent saw markdown instead of JSON;
• known redactions/truncations.

A replay/analyzer should be able to answer: what exactly did the agent know before each action?

Action and outcome requirements

For each action/tool that can mutate game state, log:

• tool name and arguments after redaction;
• pre-action state hash;
• HTTP request body;
• HTTP status/result payload;
• post-action observed state hash, if polled;
• elapsed time;
• whether the action changed state;
• normalized outcome: success, invalid, rejected, no-op, timeout, unknown;
• error message and recovery suggestion when available.

This should support queries such as: which model attempted invalid card indices, how often did smart polling save a call, which actions caused repeated states, and which failures were game/API issues versus agent mistakes?

Decision annotation requirements

Decision logs should support structured fields, not only free text:

• concise summary;
• intended next action/tool;
• target entity/card/node when applicable;
• alternatives considered;
• confidence or score if the agent supplies one;
• tags such as combat, map, reward, risk, lethal, defensive;
• related state hash;
• related tool call id;
• optional free-form rationale.

Do not require agents to expose hidden chain-of-thought. A short rationale or policy note is enough, and clients should be able to omit it.

Privacy and retention requirements

Logging should make privacy behavior explicit:

• recursive redaction of sensitive keys;
• configurable key denylist;
• optional hashing of player/profile identifiers;
• clear distinction between preview-only and full-payload modes;
• optional compression;
• bounded file sizes or rotation policy;
• documented cleanup/retention defaults.

Analysis affordances

The artifact format should make these analyses straightforward:

• replay an agent-visible transcript;
• count tool calls by type and state;
• compute token-cost proxies from response sizes;
• detect repeated/non-actionable state calls;
• measure time spent waiting on game transitions;
• compare action invalid-rate by agent/model;
• correlate decisions with outcomes;
• summarize run result by act/floor/combat/reward;
• identify schema or game-version incompatibilities.

Acceptance criteria

• A versioned logging schema is documented.
• A sample run artifact is included or generated by a deterministic smoke script.
• Logs include enough metadata to compare runs across machines and STS2MCP versions.
• Every MCP tool call can be linked to HTTP requests/responses and state polling events.
• Mutating actions can be linked to pre/post state hashes when available.
• Decision annotations can be linked to state/action context.
• Redaction, truncation, hashing, and full-payload modes are documented and visible in manifest.json.
• A lightweight validator can check sequence monotonicity, required fields, JSONL parseability, file hashes, and schema version compatibility.

Relationship to #10 and #90

#10 requested practical run logging and smart polling. #90 starts that work in the MCP bridge. This issue tracks the deeper artifact contract needed for long-term benchmarking, replay, and comparative model analysis.