fix(#6014,#6015): real rebuild in-flight across the process boundary; dashboard stops serving a stale graph.json

[cajasmota]

Two release blockers in the same class as #5991/#6013: the tool reported a state that was not true.

#6014 — in_flight=0 always in split mode

The issue proposed moving the counter to the branch split mode takes. There is no such branch. Two separate defects produced the symptom.

1. The work moved processes. Service.rebuildInFlight / groupsActiveCount are incremented below Service.Rebuild's split-mode early return, and in split mode the rebuild runs in the engine. Serve cannot derive the number locally, so the fix has to be cross-process — anything else would be a fabricated plausible-looking number.

The obvious shortcut was checked and rejected: statusfile.EngineInFlight mirrors indexstate.InFlight, whose only writer is indexstate.Set(len(s.inflight)) in the scheduler — and cmd/grafel/daemon.go:1541-1543 already documents that the group rebuild "indexes every repo DIRECTLY … it never calls scheduler.Enqueue, so s.inflight stays empty". It would have read 0 for the entire 10–12 minute rebuild while looking correct.

The real engine-side truth is rebuildWorker.active, the per-group single-flight guard. engineRebuildInFlight is incremented in lockstep with it, published by the existing liveness heartbeat, and read back via RebuildInFlightFromStatusFile() — the same route #5954's stage gate and #6002's GroupAlgoInFlight already use, sharing their freshness rule (stale heartbeat ⇒ 0, never a dead engine's last-known count).

2. The RPC itself was uncounted. s.inFlight — the literal in_flight= field — was also below the split branch. Harmless for fire-and-forget, but a lie for WaitForCompletion, which blocks the handler for the entire rebuild. Moved to the top of the handler.

The first fix was correct on the wire and dead on the screen

Review caught this on the live daemon: the count crossed the process boundary correctly and the renderer threw it away.

StatusReply.RebuildInFlight has one consumer, internal/cli/status.go:205, nested inside if st.RSSBudgetMB > 0. reply.RSSBudgetMB is set at service.go:399 — inside if s.scheduler != nil. In split mode serve's scheduler is nil, so that whole block never prints. grafel status against the running daemon showed in_flight=0 and no rebuild: line at all.

That is the same defect class as the bug being fixed, one layer up — and the code asserted otherwise, claiming the count reached grafel status.

Hoisting the conditional alone would not have been testable: the block lives inside runStatus, which requires a live client.Dial, so every conditional there was unreachable from a test. That is how a dead print gate survived. The detail block is now extracted into printDaemonDetail (following the existing printAnnotationStatus precedent) and the rebuild line hoisted out of the RSSBudgetMB gate there. Independently verified: re-nesting it fails TestPrintDaemonDetail_RebuildLinePrintsWithNoRSSBudget.

Both false comments corrected.

#6015 — dashboard served a stale graph.json forever

As filed, plus a layout the issue does not mention: a gate hardcoded to graph.fb finds nothing on a segment-set repo, concludes "no graph", and serves the stale JSON anyway. Freshness is measured against graph.CurrentGraphDescriptor — manifest.json for a segment set (its atomic commit point, written after every segment flush and before the current pointer flip), the resolved .fb otherwise.

On equal mtimes: cmd/grafel/index.go:828-830 does now := time.Now(); os.Chtimes(fbPath, now, now); os.Chtimes(outPath, now, now). An equal mtime is deliberately engineered, not filesystem-resolution luck, so !Before would have been actively wrong. Comparison is After; ties regenerate.

Known cost, documented and pinned rather than described: because those mtimes are forced equal, the JSON fast path is now dead for single-file-gen repos that opted into --export-json — every graph request re-marshals. Blast radius is narrow (FB-only is the default; this affects --export-json, grafel quality, xrepo_verify) and specifically not the 427k-entity corpus. Segment sets keep the fast path, because writeSegments returns the gen dir while the gate reads the earlier-written manifest.json. That asymmetry is pinned by TestRepoGraphBytes_SegmentSetKeepsJSONFastPathWhenManifestOlder, so the comment is a checked claim.

The content-identity fix was considered and deferred: descriptions.CurrentSourceKey computes a key for the current graph, but nothing stores a comparable key on graph.json, so using it properly means stamping a generation identity into the JSON — a new sidecar contract, well outside this fix.

Behaviour change, decided deliberately: an unreadable .fb now errors instead of falling back to graph.json. That is right for a silently-stale snapshot, but a bare error is wrong for format-version skew, which is transient-by-reindex. graphUnreadableError runs graph.ReindexRequiredReason (segment-set aware) so the panel names the fixing command. Two tests: version skew must say "reindex required"; any other load failure must not — dressing every failure as a reindex prompt would send users to reindex something a reindex cannot fix.

Verification

#6014: 6/6 mutations killed (including the re-nested gate). #6015: 11/12. The original sweep was re-run after the refactor and all still die.

The sole survivor is dropping ji.IsDir() — benign, since os.ReadFile on a directory fails and falls through anyway. It is kept with a comment saying explicitly that it is belt-and-braces and that no test binds it, so nobody later mistakes it for load-bearing.

One earlier escape, disclosed and closed: the RebuildInFlightFromStatusFile freshness guard was shadowed by StageGateFromStatusFile()'s own check, so it could be deleted entirely with the RPC test green. TestRebuildInFlightFromStatusFile_FreshVsStale binds the accessor directly.

Also verified during review and requiring no change: no counter leak (the Add(-1) defer covers panic, early return, backoff, retry and dead-letter, all inside runGroup); genuine lockstep with w.active; manifest.json really is the segment-set commit point, so the gate cannot report fresh mid-write.

go build ./... && go vet ./... && gofmt -l . clean. internal/cli, internal/daemon, internal/dashboard, internal/statusfile all pass post-rebase with raw exit codes via rtk proxy.

Known and disclosed

• The heartbeat is a plain 5 s ticker with no state-change coalescing, so a running rebuild reports 0 for up to 5 s after it starts. Rebuilds are multi-minute so this is acceptable; documented at RebuildInFlightFromStatusFile, noting the right tightening is a notify on rebuildWorker.submit rather than a shorter tick. Not done here — it touches the heartbeat's construction seam.
• StatusReply.GroupAlgoRunning (group names) is still empty in split mode; the sidecar carries counts only, as group-algo child: 29.5h elapsed for 116min CPU (~6.5% duty) — the pass is blocked, not slow #6002 left it. The same is now true of rebuilds.
• A monolith calling s.rebuild directly still reports via the serve-local atomics, so the two modes report through different mechanisms rather than fully converging.

Independently adversarially reviewed.

Closes #6014
Closes #6015