Skip to content

fix(#5991,#6013): reset reports an unconfirmed rebuild as a failure; status reports a corrupt graph as failed - #6035

Merged
cajasmota merged 2 commits into
mainfrom
worktree-agent-a2bec7560033992df
Jul 28, 2026
Merged

fix(#5991,#6013): reset reports an unconfirmed rebuild as a failure; status reports a corrupt graph as failed#6035
cajasmota merged 2 commits into
mainfrom
worktree-agent-a2bec7560033992df

Conversation

@cajasmota

Copy link
Copy Markdown
Owner

Two release blockers in one class: the tool reported success or health when it wasn't true. grafel is consumed by AI agents that treat its output as ground truth, so a silently-wrong success is worse than a loud failure.

#5991grafel reset exited 0 on a rebuild that never ran

The reported mechanism was a rebuild cooldown. It was not. Split mode is the default (daemon.SplitModeEnabled(), internal/daemon/server.go:298 — true unless explicitly falsy). In split mode Service.Rebuild (service.go:657-683) writes a KindRebuild request file and returns nil with the zero RebuildReply unless WaitForCompletion is set, which reset did not set. The CLI printed Rebuilding group 'X'..., received {repos: nil, elapsed: 0, entities: 0, err: nil} instantly, took finishRebuild's len(repos)==0 branch, built an empty summaryParts, and printed nothing.

Both the .grafel/ wipe and the rebuild live engine-side in RebuildFunc, so when the engine never drains that request — engine down, crash-resume backoff, dead-lettered after maxRebuildAttempts — nothing happens at all. The CLI was structurally incapable of distinguishing that from success: byte-identical output, and the exit code carried no information in either case.

Fixed by opting reset into the existing #5790 WaitForCompletion contract on both call sites, so the daemon blocks on the engine's terminal ack and returns nil only on a real StatusOK. Failures are wrapped: reset: group "X" was NOT wiped or rebuilt (the previous graph is still on disk): <reason>.

The first fix did not reach the path users are on

Review reproduced #5991 on the fixed branch, which is the most useful thing in this PR.

The dashboard's progress handler (internal/dashboard/handlers_progress.go:246-262) matches only on RunToken and has no repo scoping, so it closes the whole group stream on the first per-repo terminal event. The CLI then fell into its !ok arm, waited 5 s, and returned the zero outcome with err == nil — so wrapResetFailure(wipe, group, nil) was a no-op and the fix never executed:

--- FAIL: TestReset_SSEStreamClosesEarly_DoesNotFalselySucceed (5.03s)
    reset returned (err=<nil>) before the daemon confirmed anything
    Rebuilding group 'mygroup'...
    repo-a: done  (7 entities)

And the first fix made that ordering normal: previously the RPC returned in milliseconds so resultCh was always ready first; WaitForCompletion deliberately blocks it for the whole rebuild, so for any group taking >5 s past its first repo, SSE-close-beats-RPC is the expected interleaving. The dashboard is embedded and started by default, so dashPort > 0 and the SSE branch is production — the poll path, which the original tests exercised, runs only when the dashboard is down.

runBrokerProgress now takes waitForCompletion. When a completion guarantee was requested the RPC is the only authority: sseCh = nil and continue, so the loop keeps selecting on resultCh and the 10 s heartbeat for as long as the rebuild takes. Callers that did not ask for one (rebuild, the wizard's indexGroupWithProgress) keep the historical 5 s give-up, pinned by TestRebuild_SSEGiveUpUnchanged so the guard binds in both directions.

--wait-timeout (BEHAVIOUR CHANGE)

Blocking on an engine ack means reset --quiet could otherwise hang for the daemon's full 2 h (rebuildWaitTimeout, and net/rpc over the UDS sets no client deadline). A CI invocation that used to return instantly could hang.

A flag rather than a default cap: a default cap would false-fail a legitimately long monorepo reset — the same confident-but-wrong report this issue is about, pointed the other way. Default stays unbounded; --wait-timeout 30m bounds the CLI's wait only. Applied via boundOutcome at the single channel every renderer consumes, so it cannot be live on one path and dead on another, and --quiet runs its RPC on a goroutine so the bound reaches it too. A malformed value is rejected, not ignored.

The give-up message is deliberately weaker than a confirmed failure, because the CLI stopped waiting rather than learning the outcome:

reset: group "X" — no completion confirmed within 30m: the daemon did not answer
in time; the rebuild may or may not have completed (check `grafel status`), and
--wait-timeout only bounds this command, it does not cancel the daemon

#6013grafel status reported a corrupt graph as healthy

Two swallows on the same call, not the one reported: an empty recover() at status_stats.go:149-155, and if err == nil && doc != nil { … } dropping the returned error with no else branch. Both verified reachable with real fixtures — a truncated graph.fb panics the flatbuffers accessors, an all-zero one returns a format-version error.

RepoStatus.GraphLoadError deliberately reuses the string representation internal/mcp already uses (LoadedRepo.loadErr) rather than inventing a type. Rendered as ⚠ graph FAILED to load: … — counts above are INCOMPLETE, with graphArtifactPresent keeping "no graph yet" distinct from "failed". The recover stays, so status remains non-fatal and still reports the other repos. The group TOTAL: line is now marked ⚠ INCOMPLETE — N repos could not be read.

Error text is scrubbed to basenames — a corrupt segment-set manifest leaked an absolute path twice in one message. Diagnosis survives: graph: resolve segment-set graph.3: graph: parse manifest in graph.3: invalid character 'n' ….

Verification

21 mutations across both rounds, all caught. Independently re-verified: restoring the 5 s SSE give-up fails both TestReset_SSEStreamClosesEarly_* at 5.01 s.

One test-quality problem found and fixed mid-review: a mutant originally hung the package rather than failing, because the wait-timeout tests called runResetCmd synchronously. A mutation that manifests only as a hang is not one the suite catches. runResetBounded (goroutine + bounded select) makes those fail in 10 s with an attributable message.

go build ./... && go vet ./... && gofmt -l . clean; internal/cli — the only package touched — passes -count=1 and -race, raw exit codes via rtk proxy. Revertability checked by execution: commit 1 alone builds and passes, and reverting commit 1 leaves commit 2 building and passing.

Known and disclosed

  • grafel rebuild retains the same class of bug — fire-and-forget, empty reply, 5 s give-up, exit 0. Deliberately out of scope and pinned by two scope tests so it is an explicit decision, not an accident. Tracked as grafel rebuild can silently no-op in split mode (same shape as #5991) #6032.
  • finishRebuild prints nothing on a successful split-mode reset (empty summaryParts), so success still reads as silence. Untouched.
  • indexGroupWithProgress (wizard / group add) keeps the 5 s give-up; group add --index gets its honesty from indexGroup's own WaitForCompletion path (Split mode: 'group add --index' reports indexed:true off a fire-and-forget Rebuild ack (rebuild not actually done) #5790), not this one.
  • The DialProgress-failure branch in runPollProgress has a wrapResetFailure call no test binds — it needs DialProgress to fail against a live socket. Genuinely rare, unlike the SSE branch.
  • boundOutcome leaks the RPC goroutine on expiry; net/rpc offers no cancellation. Documented in code, reachable only when --wait-timeout is set and expires.
  • case doc == nil in status is defensive and untested — not constructible through LoadGraphFromDir today.

Independently adversarially reviewed.

Closes #5991
Closes #6013

cajasmota and others added 2 commits July 29, 2026 04:55
Split mode is the default (daemon.SplitModeEnabled is true unless
GRAFEL_SPLIT_MODE is explicitly falsy). There Service.Rebuild is
fire-and-forget: it writes a KindRebuild request file for the group and
returns nil immediately with the ZERO RebuildReply. Both the `.grafel/`
wipe and the rebuild live engine-side inside RebuildFunc, so if the
engine never drains that request (engine down, crash-resume backoff,
dead-letter) nothing at all happens — yet the CLI saw err==nil plus an
empty reply, printed nothing after "Rebuilding group 'X'...", and
exited 0. The exit code carried no information.

Two changes are needed, because the CLI has two progress renderers and
the daemon-side contract alone does not reach the one users are on.

1. reset opts into the #5790 WaitForCompletion contract (the same one
   `group add --index` uses), so the daemon blocks on the engine's
   terminal ack and err==nil means the wipe+rebuild really happened.

2. runBrokerProgress no longer treats an SSE close as evidence the
   rebuild finished. The daemon's embedded dashboard is started by
   default, so `Status` reports a port and the CLI takes the BROKER
   branch; the poll fallback only runs when the dashboard is down. The
   dashboard's progress handler matches on RunToken with no repo
   scoping, so it closes the WHOLE group stream on the FIRST per-repo
   terminal event. runBrokerProgress then waited 5s for the RPC and
   returned the ZERO outcome with err == nil — so reset printed
   "repo-a: done" and exited 0 with the rebuild unconfirmed. Change (1)
   made that the NORMAL interleaving rather than a rarity, since the
   RPC now deliberately blocks for the whole rebuild. When the caller
   asked for a completion guarantee the RPC is now the only authority:
   the dead SSE channel is nil'd and the loop keeps waiting (heartbeats
   included). Callers that did not ask for one — `rebuild` and the
   wizard's indexGroupWithProgress — keep the 5s give-up, pinned by a
   test.

A non-completion is wrapped to name what did not happen:

  reset: group "X" was NOT wiped or rebuilt (the previous graph is
  still on disk): <daemon reason>

BEHAVIOUR CHANGE: reset now BLOCKS until the daemon confirms, where it
previously returned in milliseconds. The daemon's own bound is 2h
(rebuildWaitTimeout == rebuildRPCTimeout), with faster exits only for a
never-live engine (30s) or a live-then-stale one (2m), and net/rpc sets
no client deadline — so a wedged-but-heartbeating engine could hold a
`reset --quiet`, which prints nothing at all, for two hours. The
instant return WAS the bug, but scripted callers need an escape hatch:
`reset --wait-timeout <dur>` bounds the CLI's wait only. It defaults to
unbounded so a legitimately long monorepo reset is never false-failed,
and its error is deliberately weaker than a confirmed failure — the CLI
gave up waiting, it did not learn the outcome:

  reset: group "X" — no completion confirmed within 30m: the daemon did
  not answer in time; the rebuild may or may not have completed

Scoped to wipe (i.e. `reset`) — plain `rebuild` keeps its existing
fire-and-forget semantics, its 5s SSE give-up and its unwrapped error,
all pinned by tests. runPollProgress's `wipe` parameter was accepted
and ignored (`_ bool`); it is now bound.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0111KEDUVWEWm9G5RRnJqXft
ComputeStatusSummary swallowed BOTH failure modes of its per-repo
LoadGraphFromDir call: a `defer recover()` with an empty body ("Silently
ignore panics during graph loading") and an `if err == nil && doc != nil`
that dropped the returned error on the floor. Either way the repo line
was still printed with Files/IndexedRef/IndexedSHA blank and the
endpoint/flow/orphan contributions missing — a plausible, smaller-than-
reality repo that is indistinguishable from a healthy one. Both modes are
reachable with real fixtures: a truncated graph.fb panics the flatbuffers
accessors ("slice bounds out of range"), an all-zero one returns a
format-version error.

RepoStatus gains GraphLoadError, the same string-valued "last load
failed, here is why" representation internal/mcp already uses for
LoadedRepo.loadErr (surfaced as the dashboard's LoadError and tools.go's
"unavailable" list) — see #6010 for the nil-Doc half of the same state
and #5969 for the malformed-graph class. PrintStatusSummary renders it
as an explicit "graph FAILED to load" warning that also says the counts
on the line above are incomplete rather than zero.

Three states stay distinguishable:

  healthy       GraphLoadError == "" and the counts are real
  no graph yet  GraphLoadError == "", "indexed (never)" — NOT a failure
  failed        GraphLoadError != "", warning line

graphArtifactPresent separates the last two: LoadGraphFromDir's "neither
graph.fb nor graph.json found" on a never-indexed repo stays silent,
while a graph that exists and cannot be read is reported.

The aggregated TOTAL line is marked "⚠ INCOMPLETE — N repos could not be
read" whenever any repo failed to load. Every such repo contributed
nothing to the flow/endpoint aggregates, so marking only the per-repo
lines would leave the headline number — the one most people read —
silently wrong.

GraphLoadError is scrubbed of filesystem paths before it is stored. It
is printed to the terminal and is exactly the string a user pastes into
a bug report, and a corrupt segment-set manifest surfaces as
"graph: resolve segment-set <abs>: graph: parse manifest in <abs>: …",
which discloses the repository layout. Scrubbing to basenames here,
rather than relying on which underlying error happens to fire, keeps
that true for load failures we have not seen yet; the diagnosis (which
generation, which file, what went wrong) survives.

The recover stays, so status remains non-fatal: one corrupt graph does
not stop the other repos or groups from being reported.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0111KEDUVWEWm9G5RRnJqXft
@cajasmota
cajasmota merged commit 54610b5 into main Jul 28, 2026
1 of 2 checks passed
@cajasmota
cajasmota deleted the worktree-agent-a2bec7560033992df branch August 1, 2026 06:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant