Agent sessions fail with "Fatal error in message reader: Command failed with exit code 1" — actual error not surfaced via API

[bcarmeli]

Issue draft for skillberry-ai/runspace-agent

File at: https://github.com/skillberry-ai/runspace-agent/issues/new

Title:

Agent sessions fail with "Fatal error in message reader: Command failed with exit code 1" — actual error not surfaced via API

Body:

---

Summary

We're hitting a high failure rate (≈ 89%) on claude-code sessions when calling the runspace server with very large context/ directories. The server logs only:

[Runspace service] stderr: Fatal error in message reader: Command failed with exit code 1 (exit code: 1)
[Runspace service] stderr: Error output: Check stderr output for details

…but Check stderr output for details is not actionable for clients calling the runspace API: the agent's actual stderr is on the server's compute node, not in the response. GET /sessions/<id> returns a SessionDetail with an error field, but it doesn't appear to carry the agent's stderr tail in this failure mode.

Environment

• runspace-agent==0.2.0 (from PyPI)
• mode=local
• agent_type=claude-code
• agent_max_turns=50
• container_memory=4g, container_cpus=2 (nominal in local mode)
• Context dir size on each call: several hundred MB to ~1 GB (multi-iteration optimization run; each iter accumulates context/current_run/<prior_iters>/ materializations of fold-eval trajectories)
• ANTHROPIC_BASE_URL = an internal LiteLLM proxy
• ANTHROPIC_MODEL = claude-opus-4-6 (verified working with claude -p directly outside runspace)

Symptoms

1. POST /run returns 200 OK with a session_id. Server-side stdout shows the session was registered.
2. The agent process spawns. Server-side stderr shows:

   Warning: no stdin data received in 3s, proceeding without it.
   

3. ~10–30 s later, the server emits:

   Fatal error in message reader: Command failed with exit code 1 (exit code: 1)
   Error output: Check stderr output for details
   

4. GET /sessions/<id> from a client times out at our 10 s httpx.get read timeout (the server seems blocked on something around this same window). After the agent fully crashes, subsequent GETs eventually respond.
5. We re-run with up to 3 attempts. ~89% of attempts fail this way; the remainder occasionally succeed on the same code path. Reproducible across multiple machines.
6. Direct claude -p "echo hello" with the same ANTHROPIC_AUTH_TOKEN / ANTHROPIC_BASE_URL / ANTHROPIC_MODEL env vars succeeds, so it's not a credentials or model-name problem.

Hypothesis (we'd appreciate guidance)

Our trajectories materialized into context/current_run/ are now substantially larger than they were a few weeks ago (hundreds of MB to ~1 GB). We suspect the claude-code agent is choking on serializing or processing that context — either OOM-ing inside the Node runtime, or the SDK's IPC (message reader) failing under the volume.

If that's the right diagnosis, useful upstream fixes might be:

• Configurable --max-old-space-size on the agent's Node process
• A documented large_context operating mode
• Streaming the context rather than ingesting it eagerly

But honestly, we can't tell from here whether that's the right diagnosis, because the server doesn't expose the agent's actual stderr.

Asks

(1) Surface the agent's stderr in SessionDetail.error. The runspace server already captures the agent's full stderr (you log Fatal error in message reader: ... from it). Including the last N lines of that stderr in the SessionDetail.error field — instead of "Check stderr output for details" — would let API clients diagnose without needing access to the server's working directory. This is the single most useful thing for debugging from outside the host.

(2) Document expected GET /sessions/<id> latency during agent startup. When the agent is being spawned (and crashing), the server takes >10 s to respond to status queries. Clients that poll with shorter timeouts give up before they ever see the failure. A note in the README on recommended client polling timeouts would help; a server-side fast path that responds with the current status even mid-startup would be even better.

(3) Optional, but useful: suggest a large_context operating mode — environment variables (NODE_OPTIONS=--max-old-space-size=... or similar) we should set when the context is large, so the agent doesn't crash silently.

Reproduction steps

If POST /run is called with a context_dir whose total size approaches ~1 GB (e.g., aggregated multi-iteration trajectories), the agent crashes with exit code 1 within ~30 seconds. Reducing the context size makes the failure rate drop. We don't have a minimal repro yet — happy to construct one if useful, but the simple version is "increase context_dir size until you see this fail."

Side note

We've added a client-side workaround on our end: after a session failure, we re-fetch GET /sessions/<id> with a longer (60 s) read timeout and log the error field if present. This is purely diagnostic — it gives us the runspace-server-known error after the fact, but doesn't address the underlying crash. If the server starts populating error with the agent's actual stderr (ask #1 above), our workaround becomes unnecessary.

Happy to share full skill_maker logs / a minimal reproducer / our context-dir layout if that helps.