fix(#5991,#6013): reset reports an unconfirmed rebuild as a failure; status reports a corrupt graph as failed - #6035
Merged
Conversation
Split mode is the default (daemon.SplitModeEnabled is true unless GRAFEL_SPLIT_MODE is explicitly falsy). There Service.Rebuild is fire-and-forget: it writes a KindRebuild request file for the group and returns nil immediately with the ZERO RebuildReply. Both the `.grafel/` wipe and the rebuild live engine-side inside RebuildFunc, so if the engine never drains that request (engine down, crash-resume backoff, dead-letter) nothing at all happens — yet the CLI saw err==nil plus an empty reply, printed nothing after "Rebuilding group 'X'...", and exited 0. The exit code carried no information. Two changes are needed, because the CLI has two progress renderers and the daemon-side contract alone does not reach the one users are on. 1. reset opts into the #5790 WaitForCompletion contract (the same one `group add --index` uses), so the daemon blocks on the engine's terminal ack and err==nil means the wipe+rebuild really happened. 2. runBrokerProgress no longer treats an SSE close as evidence the rebuild finished. The daemon's embedded dashboard is started by default, so `Status` reports a port and the CLI takes the BROKER branch; the poll fallback only runs when the dashboard is down. The dashboard's progress handler matches on RunToken with no repo scoping, so it closes the WHOLE group stream on the FIRST per-repo terminal event. runBrokerProgress then waited 5s for the RPC and returned the ZERO outcome with err == nil — so reset printed "repo-a: done" and exited 0 with the rebuild unconfirmed. Change (1) made that the NORMAL interleaving rather than a rarity, since the RPC now deliberately blocks for the whole rebuild. When the caller asked for a completion guarantee the RPC is now the only authority: the dead SSE channel is nil'd and the loop keeps waiting (heartbeats included). Callers that did not ask for one — `rebuild` and the wizard's indexGroupWithProgress — keep the 5s give-up, pinned by a test. A non-completion is wrapped to name what did not happen: reset: group "X" was NOT wiped or rebuilt (the previous graph is still on disk): <daemon reason> BEHAVIOUR CHANGE: reset now BLOCKS until the daemon confirms, where it previously returned in milliseconds. The daemon's own bound is 2h (rebuildWaitTimeout == rebuildRPCTimeout), with faster exits only for a never-live engine (30s) or a live-then-stale one (2m), and net/rpc sets no client deadline — so a wedged-but-heartbeating engine could hold a `reset --quiet`, which prints nothing at all, for two hours. The instant return WAS the bug, but scripted callers need an escape hatch: `reset --wait-timeout <dur>` bounds the CLI's wait only. It defaults to unbounded so a legitimately long monorepo reset is never false-failed, and its error is deliberately weaker than a confirmed failure — the CLI gave up waiting, it did not learn the outcome: reset: group "X" — no completion confirmed within 30m: the daemon did not answer in time; the rebuild may or may not have completed Scoped to wipe (i.e. `reset`) — plain `rebuild` keeps its existing fire-and-forget semantics, its 5s SSE give-up and its unwrapped error, all pinned by tests. runPollProgress's `wipe` parameter was accepted and ignored (`_ bool`); it is now bound. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0111KEDUVWEWm9G5RRnJqXft
ComputeStatusSummary swallowed BOTH failure modes of its per-repo
LoadGraphFromDir call: a `defer recover()` with an empty body ("Silently
ignore panics during graph loading") and an `if err == nil && doc != nil`
that dropped the returned error on the floor. Either way the repo line
was still printed with Files/IndexedRef/IndexedSHA blank and the
endpoint/flow/orphan contributions missing — a plausible, smaller-than-
reality repo that is indistinguishable from a healthy one. Both modes are
reachable with real fixtures: a truncated graph.fb panics the flatbuffers
accessors ("slice bounds out of range"), an all-zero one returns a
format-version error.
RepoStatus gains GraphLoadError, the same string-valued "last load
failed, here is why" representation internal/mcp already uses for
LoadedRepo.loadErr (surfaced as the dashboard's LoadError and tools.go's
"unavailable" list) — see #6010 for the nil-Doc half of the same state
and #5969 for the malformed-graph class. PrintStatusSummary renders it
as an explicit "graph FAILED to load" warning that also says the counts
on the line above are incomplete rather than zero.
Three states stay distinguishable:
healthy GraphLoadError == "" and the counts are real
no graph yet GraphLoadError == "", "indexed (never)" — NOT a failure
failed GraphLoadError != "", warning line
graphArtifactPresent separates the last two: LoadGraphFromDir's "neither
graph.fb nor graph.json found" on a never-indexed repo stays silent,
while a graph that exists and cannot be read is reported.
The aggregated TOTAL line is marked "⚠ INCOMPLETE — N repos could not be
read" whenever any repo failed to load. Every such repo contributed
nothing to the flow/endpoint aggregates, so marking only the per-repo
lines would leave the headline number — the one most people read —
silently wrong.
GraphLoadError is scrubbed of filesystem paths before it is stored. It
is printed to the terminal and is exactly the string a user pastes into
a bug report, and a corrupt segment-set manifest surfaces as
"graph: resolve segment-set <abs>: graph: parse manifest in <abs>: …",
which discloses the repository layout. Scrubbing to basenames here,
rather than relying on which underlying error happens to fire, keeps
that true for load failures we have not seen yet; the diagnosis (which
generation, which file, what went wrong) survives.
The recover stays, so status remains non-fatal: one corrupt graph does
not stop the other repos or groups from being reported.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0111KEDUVWEWm9G5RRnJqXft
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two release blockers in one class: the tool reported success or health when it wasn't true. grafel is consumed by AI agents that treat its output as ground truth, so a silently-wrong success is worse than a loud failure.
#5991 —
grafel resetexited 0 on a rebuild that never ranThe reported mechanism was a rebuild cooldown. It was not. Split mode is the default (
daemon.SplitModeEnabled(),internal/daemon/server.go:298— true unless explicitly falsy). In split modeService.Rebuild(service.go:657-683) writes aKindRebuildrequest file and returnsnilwith the zeroRebuildReplyunlessWaitForCompletionis set, whichresetdid not set. The CLI printedRebuilding group 'X'..., received{repos: nil, elapsed: 0, entities: 0, err: nil}instantly, tookfinishRebuild'slen(repos)==0branch, built an emptysummaryParts, and printed nothing.Both the
.grafel/wipe and the rebuild live engine-side inRebuildFunc, so when the engine never drains that request — engine down, crash-resume backoff, dead-lettered aftermaxRebuildAttempts— nothing happens at all. The CLI was structurally incapable of distinguishing that from success: byte-identical output, and the exit code carried no information in either case.Fixed by opting
resetinto the existing #5790WaitForCompletioncontract on both call sites, so the daemon blocks on the engine's terminal ack and returnsnilonly on a realStatusOK. Failures are wrapped:reset: group "X" was NOT wiped or rebuilt (the previous graph is still on disk): <reason>.The first fix did not reach the path users are on
Review reproduced #5991 on the fixed branch, which is the most useful thing in this PR.
The dashboard's progress handler (
internal/dashboard/handlers_progress.go:246-262) matches only onRunTokenand has no repo scoping, so it closes the whole group stream on the first per-repo terminal event. The CLI then fell into its!okarm, waited 5 s, and returned the zero outcome witherr == nil— sowrapResetFailure(wipe, group, nil)was a no-op and the fix never executed:And the first fix made that ordering normal: previously the RPC returned in milliseconds so
resultChwas always ready first;WaitForCompletiondeliberately blocks it for the whole rebuild, so for any group taking >5 s past its first repo, SSE-close-beats-RPC is the expected interleaving. The dashboard is embedded and started by default, sodashPort > 0and the SSE branch is production — the poll path, which the original tests exercised, runs only when the dashboard is down.runBrokerProgressnow takeswaitForCompletion. When a completion guarantee was requested the RPC is the only authority:sseCh = nilandcontinue, so the loop keeps selecting onresultChand the 10 s heartbeat for as long as the rebuild takes. Callers that did not ask for one (rebuild, the wizard'sindexGroupWithProgress) keep the historical 5 s give-up, pinned byTestRebuild_SSEGiveUpUnchangedso the guard binds in both directions.--wait-timeout(BEHAVIOUR CHANGE)Blocking on an engine ack means
reset --quietcould otherwise hang for the daemon's full 2 h (rebuildWaitTimeout, andnet/rpcover the UDS sets no client deadline). A CI invocation that used to return instantly could hang.A flag rather than a default cap: a default cap would false-fail a legitimately long monorepo reset — the same confident-but-wrong report this issue is about, pointed the other way. Default stays unbounded;
--wait-timeout 30mbounds the CLI's wait only. Applied viaboundOutcomeat the single channel every renderer consumes, so it cannot be live on one path and dead on another, and--quietruns its RPC on a goroutine so the bound reaches it too. A malformed value is rejected, not ignored.The give-up message is deliberately weaker than a confirmed failure, because the CLI stopped waiting rather than learning the outcome:
#6013 —
grafel statusreported a corrupt graph as healthyTwo swallows on the same call, not the one reported: an empty
recover()atstatus_stats.go:149-155, andif err == nil && doc != nil { … }dropping the returned error with no else branch. Both verified reachable with real fixtures — a truncatedgraph.fbpanics the flatbuffers accessors, an all-zero one returns a format-version error.RepoStatus.GraphLoadErrordeliberately reuses the string representationinternal/mcpalready uses (LoadedRepo.loadErr) rather than inventing a type. Rendered as⚠ graph FAILED to load: … — counts above are INCOMPLETE, withgraphArtifactPresentkeeping "no graph yet" distinct from "failed". Therecoverstays, sostatusremains non-fatal and still reports the other repos. The groupTOTAL:line is now marked⚠ INCOMPLETE — N repos could not be read.Error text is scrubbed to basenames — a corrupt segment-set manifest leaked an absolute path twice in one message. Diagnosis survives:
graph: resolve segment-set graph.3: graph: parse manifest in graph.3: invalid character 'n' ….Verification
21 mutations across both rounds, all caught. Independently re-verified: restoring the 5 s SSE give-up fails both
TestReset_SSEStreamClosesEarly_*at 5.01 s.One test-quality problem found and fixed mid-review: a mutant originally hung the package rather than failing, because the wait-timeout tests called
runResetCmdsynchronously. A mutation that manifests only as a hang is not one the suite catches.runResetBounded(goroutine + bounded select) makes those fail in 10 s with an attributable message.go build ./... && go vet ./... && gofmt -l .clean;internal/cli— the only package touched — passes-count=1and-race, raw exit codes viartk proxy. Revertability checked by execution: commit 1 alone builds and passes, and reverting commit 1 leaves commit 2 building and passing.Known and disclosed
grafel rebuildretains the same class of bug — fire-and-forget, empty reply, 5 s give-up, exit 0. Deliberately out of scope and pinned by two scope tests so it is an explicit decision, not an accident. Tracked as grafel rebuild can silently no-op in split mode (same shape as #5991) #6032.finishRebuildprints nothing on a successful split-mode reset (emptysummaryParts), so success still reads as silence. Untouched.indexGroupWithProgress(wizard /group add) keeps the 5 s give-up;group add --indexgets its honesty fromindexGroup's ownWaitForCompletionpath (Split mode: 'group add --index' reports indexed:true off a fire-and-forget Rebuild ack (rebuild not actually done) #5790), not this one.DialProgress-failure branch inrunPollProgresshas awrapResetFailurecall no test binds — it needsDialProgressto fail against a live socket. Genuinely rare, unlike the SSE branch.boundOutcomeleaks the RPC goroutine on expiry;net/rpcoffers no cancellation. Documented in code, reachable only when--wait-timeoutis set and expires.case doc == nilin status is defensive and untested — not constructible throughLoadGraphFromDirtoday.Independently adversarially reviewed.
Closes #5991
Closes #6013