Skip to content

Hot-reloading the plugin strands the ops engine (surface has no reload callback) #54

Description

@mabry1985

Symptom

Every POST /api/plugins/spacetraders/update (and any Settings save that reloads the graph) leaves the ops engine stopped from the live module's perspective, earning nothing until a manual st_autopilot_start or a full app restart. Hit 3× while shipping fixes on 2026-07-05.

Mechanism (traced through protoAgent)

POST /{id}/update_apply_settings_changes_reload_langgraph_agent() (logs "LangGraph agent reloaded") → _reload_plugin_surfaces(new_config), which calls each surface's reload callback:

for h in STATE.plugin_surface_handles:
    reload_cb = h.get("reload")
    if not callable(reload_cb):
        continue          # <-- spacetraders registered reload=None → SKIPPED

We register the surface as register_surface(_fleet_surface_start, stop=_fleet_surface_stop, name="spacetraders-fleet")no reload=. So on a reload:

  • register() re-runs (verifiers/goal-hook/surface re-registered), but
  • neither _fleet_surface_start (where the v2.9 resume lives) nor _fleet_surface_stop fires, and
  • _reload_plugin_surfaces skips us.

The reload also _purge_plugin_modules(...) + re-imports, so the fresh fleet module has _ENGINE = None → status reports "stopped", while the pre-reload supervisor task is left running (no surface-stop) — orphaned.

Confirmed in the live log

Each update logged LangGraph agent reloaded with no following fleet-engine surface ready, no RESUMED autopilot, no fleet engine stopped. The v2.9 resume (surface-start) has therefore never once fired on a reload — only on a full boot.

Candidate fix + the catch

Register a reload callback mirroring surface-start's resume (if fleet.autopilot_intended(): fleet.start_ops(...)). Clean for config reloads (no purge, same module — start_ops's idempotency guard makes it safe).

Catch for CODE updates: the module purge means the resume would start_ops on the fresh module (_ENGINE=None) while the orphaned old-module supervisor may still be running → split-brain / double-trading. So a naive reload callback could make code-update rollouts worse than today.

Options:

  1. reload callback for the config-reload case only; keep recommending a full restart for code updates (host already returns restart_recommended: true for surface-bearing plugins).
  2. Host-side: on a surface-bearing plugin update, STOP the old surface (call stop) before purge and re-run start on the fresh module — a clean lifecycle cycle instead of a bare code swap. (protoAgent-side; pairs with #1823.)

Preferring (2) as the robust fix; (1) is a partial plugin-side mitigation. Needs a design call before shipping — flagging rather than hot-patching.

Operational workaround (now)

After a plugin hot-update, st_autopilot_start (or restart the agent) to resume the engine.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions