Non-deterministic hang under parallel reconcile — DAG scheduler never converges; `--concurrency 1` avoids it

[dewet22]

Thanks for flate — it's become the gate our Flux repo's CI trusts before Renovate automerges chart bumps.

I've been hitting a non-deterministic hang and managed to capture goroutine dumps that point fairly clearly at the DAG scheduler, so I wanted to write it up in case it's useful.

What happens

flate build all / flate test all hangs indefinitely — no output after the load phase, runs until the CI job timeout — on roughly half of all runs at the default --concurrency 48. The other half render normally in a few seconds, correct output. Which of my two clusters hangs varies run to run: the same commit rendered one clean in ~7s and hung the other past 150s on the next push.

At --concurrency 1 I've not seen it hang once — 8/8 clean runs, and the output is byte-identical to the good parallel runs (same document counts). So it looks like a race that only shows up once ≥2 nodes are dispatched concurrently.

What the dumps show

I wrapped flate in timeout -s QUIT to force a goroutine dump on hang, on two separate hung runs. Two things stood out.

1. The scheduler is parked waiting for a convergence that never comes. Goroutine 1 sits in sync.Cond.Wait:

goroutine 1 [runnable]:
  sync.(*Cond).Wait(...)
  flate/pkg/schedule.(*Scheduler).Run              schedule/schedule.go:241
  flate/pkg/orchestrator.(*Orchestrator).runDAG    orchestrator/dagrun.go:126
  flate/pkg/orchestrator.(*Orchestrator).render    orchestrator/run.go:188

2. Meanwhile it's spawning dispatch workers without bound. The highest goroutine id in a 90-second dump was 3,701,368 (~41k goroutines/sec). None are in a helm/values/template frame, so it isn't a render compute loop — they're all in this reconcile→require→fire cycle:

goroutine <N> [runnable]:
  flate/pkg/store.(*Store).fireUnderLock            store/events.go:208
  flate/pkg/store.(*Store).SetCondition             store/status.go:197
  flate/pkg/store.(*Store).UpdateStatus             store/status.go:171
  flate/pkg/controllers/base.(*Controller).Require  base/base.go:471
  flate/pkg/controllers/base.RequireRefresh[...]    base/base.go:513
  flate/pkg/controllers/helmrelease.(*Controller).reconcile   helmrelease/controller.go:157
  flate/pkg/controllers/base.RunWithStatusOutcome[...]         base/base.go:610
  flate/pkg/controllers/base.DispatchNode[...]                base/base.go:680
  flate/pkg/controllers/helmrelease.(*Controller).ReconcileNode  helmrelease/controller.go:110
  flate/pkg/orchestrator.dagDispatcher.Dispatch     orchestrator/dagrun.go:27
  flate/pkg/schedule.(*Scheduler).Run.func2         schedule/schedule.go:200
  flate/pkg/task.(*Service).Go.func1                task/task.go:78
  created by flate/pkg/task.(*Service).Go in goroutine 1

My read (a guess — I haven't traced the source)

It looks like reconcile-time Require/RequireRefresh writes a store condition and fireUnderLock notifies waiters, which re-dispatches DAG nodes, which reconcile and Require again — a refresh loop that doesn't reach a fixed point under concurrent dispatch, so the scheduler's convergence Cond is never satisfied. Serialising makes the condition updates settle in dependency order, which would explain why --concurrency 1 always converges. That's inference from the stacks though, not something I've confirmed in the code — you'll know the fixpoint accounting far better than I do.

I did notice a fair amount of prior work in this area (missed-wake / quiescence in #333, listener races in #274 and #440, rerun-at-drain in #703/#704), all merged before 0.4.10 — so this may be a remaining edge of the same class rather than anything new in the design.

Environment

• flate 0.4.10 (latest), static linux/arm64 release binary, run in alpine:3 under the GitLab Kubernetes executor.
• A Flux GitOps tree, ~40–90 HelmReleases across a dependency DAG (dependsOn and postBuild substitution relationships).
• Repro: default --concurrency, re-run a few times to hit it; --concurrency 1 to avoid it.
• GOMAXPROCS 128 (Ampere Altra node) — the high core count makes the goroutine churn vivid, but I don't think it's required to trigger the underlying race.

Happy to attach both full goroutine dumps or try patches / a --engine variant if that helps narrow it down. Thanks again.

[buroa]

Thanks for this issue. I will definitely look into this. Races are hard but they are fun to find, so give me some time on this one :-)

[buroa]

Thanks for the goroutine dumps - they made this much faster to pin down.

Tracing it against the code, the causal loop isn't quite the status-write path in the stacks. Require/RequireRefresh writes a Pending status, and SetCondition dedups an identical condition while OnStatusWake ignores non-terminal statuses - so a Pending write can't re-dispatch a node. That path in the dump is just where a worker happened to be parked.

The actual non-convergence comes through the object-arrival path. AddRendered has no content dedup (unlike AddObject, which skips an equal re-add), so every re-emit of a rendered doc fires EventObjectAdded → OnArrival, which unconditionally sets the scheduler's dirty bit and re-arms the drain/rerun accounting. The quiescence logic assumes a no-op re-render produces no arrival, so dirty gets re-armed after every drain and inFlight never settles - the convergence Cond is never released. At --concurrency 1 emissions serialize, the store reaches a byte-stable state, and the existing dedups absorb the re-emits, which is why serial always converges.

Interestingly the ResourceSet controller already worked around this exact class by not replaying emit on a dedup hit - that's the template for the fix. Plan: give AddRendered the same content-dedup AddObject has so an identical rendered doc fires no arrival, plus a regression test that reproduces the parallel livelock.

Will follow up when the fix is up.

[buroa]

GC Issue Triage

• verdict: reproduced
• priority: p1
• next_action: fix
• body_hash: sha256:f6db3767c9d305bf2e66c51ecbe98ba2b7d3cc65bdbd17d8af8e0f08977990be
• gate: not human approved
• artifact: plans/github/issues/home-operations/flate/828/triage/sha256:f6db3767c9d305bf2e66c51ecbe98ba2b7d3cc65bdbd17d8af8e0f08977990be/triage-report.md

Analysis

Summary

Confirmed as a real, concurrency-gated livelock in the DAG reconcile
scheduler. Under --concurrency >= 2 the scheduler's runnable frontier is
refilled as fast as it drains, so Scheduler.Run never reaches its structural
fixpoint and parks in sync.Cond.Wait (pkg/schedule/schedule.go:241) while
dispatch workers spawn without bound — exactly the goroutine dumps attached to
the issue. At --concurrency 1 it always converges.

Building on the maintainer's 2026-07-22 analysis: the goroutine-dump frame
through Require -> UpdateStatus(Pending) -> fireUnderLock is indeed a red
herring, and I can confirm that independently. The refinement this triage adds
is that the object-arrival fix as stated (give AddRendered the content-dedup
AddObject has) is good hygiene but is off the critical path for the in-flight
storm in these dumps. The scheduler is a faithful amplifier: it re-dispatches
a node only when that node's stored object is re-emitted with content the store
deems changed. The actual defect is therefore a schedulable node whose render
output is non-deterministic across re-runs under concurrent dispatch — most
likely a data race in the HR values/render path. --concurrency 1 serializes
renders to a byte-stable snapshot, which is why serial always converges.

Evidence

Dump provenance. Every frame in both dumps maps line-for-line to HEAD, and
git log --oneline v0.4.10..HEAD -- pkg/schedule pkg/store pkg/controllers/base pkg/orchestrator pkg/task is empty, so the reported 0.4.10 code path is present
verbatim at HEAD (0.4.12). Parked side: schedule.go:241 (s.cond.Wait()) <-
dagrun.go:126 (sched.Run) <- run.go:188. Worker side: store/events.go:208
(fireUnderLock) <- status.go:197/:171 <- base.go:471
(UpdateStatus(StatusPending, pendingMsg)) <- base.go:513 (RequireRefresh) <-
helmrelease/controller.go:157 <- base.go:610/:680 <-
helmrelease/controller.go:110 <- dagrun.go:27 <- schedule.go:200 <-
task/task.go:78.

The status-write path re-dispatches nothing (red herring). Require writes
Pending unconditionally (base.go:470-472), but SetCondition no-ops an
unchanged condition (status.go:188-200, conditionEqual ignores
LastTransitionTime), and the sole EventStatusUpdated listener,
OnStatusWake, drops non-terminal status (schedule.go:402-405). So a Pending
write wakes no node; that frame is just the deepest common frame the profiler
sampled in the hot reconcile path.

What actually re-dispatches a node. A terminal node is re-run only by
OnArrival(id) when its stored content changes (schedule.go:383-387);
rerunRequested on a running node is set only by OnArrival on that same id
(schedule.go:388-389). Both require EventObjectAdded for a schedulable
id. But schedulable kinds are emitted through AddObject/AddObjects, which
dedup on content via reflect.DeepEqual (store.go:243,251,291) — an identical
re-emit fires no event. AddRendered (no dedup, store.go:380) is reached only
by the two non-dispatchable-leaf call sites (emit/emit.go:136 default branch;
helmrelease/controller.go:518 !isFluxSourceKind branch — see the
ShouldDispatchAsObject/IsLeafReconcilable routing at emit/emit.go:100-104, 128-137). Those leaves (Deployments, Services, raw CRs) are not scheduler nodes
and nothing parks on them, so OnArrival(leaf) only sets s.dirty=true
(schedule.go:371) — and dirty is read solely at the quiescent fixpoint
inFlight == 0 (schedule.go:209-222), i.e. it is dead during the in-flight
storm the dumps show. Hence de-duping AddRendered removes spurious churn (and
matches the guard the ResourceSet controller already applies at
resourceset/controller.go:124-141) but does not stop an in-flight
schedulable-node re-dispatch storm.

Empirical (read-only; full log in repro-evidence.log). Deterministic
topologies converge identically at every concurrency: a 60-node and a 96-node
Kustomization DAG (postBuild substitution, dependsOn, id collisions,
overlapping peers at one spec.path, a selector ResourceSet, raw children)
produced byte-identical status-event counts (372, 596) at c=1 and c=48/96 across
45 runs, 0 hangs. The livelock mechanism itself is real but
concurrency-independent: a non-converging re-emission cycle, and a self-feeding
selector ResourceSet (fingerprint grows each pass so the dedup guard misses),
both livelock at c=1 and c=48 alike, and reproduce the report's exact scheduler
signature (cond.Wait at schedule.go:241; worker at schedule.go:200 down the
dagDispatcher.Dispatch -> ReconcileNode -> DispatchNode -> reconcile chain).
This proves the engine has no re-dispatch bound, and localizes #828's
concurrency-dependence to the trigger, not the engine.

Note: -race could not be run (the sandbox has no C compiler; the detector needs
cgo) and the HR/helm render path cannot run offline (charts need fetched
artifacts), so the specific non-deterministic emit was not confirmed locally.

Recommendation

fix. The maintainer has already posted a confirmed root cause and stated a fix
is in progress; this triage recommends the fix target the emit determinism
rather than the scheduler:

1. Pin the unstable emit with a -race reproduction of a parallel HR render
   (requires a cgo-capable toolchain and fetched chart artifacts), or a
   byte-diff of a schedulable node's emitted object across two concurrent runs.
   This directly identifies which node's render is unstable.
2. Eliminate the non-determinism at the source. The leading candidate
   (medium confidence, unverified pending step 1): the shared values.Cache
   canonical is folded by reference into hr.Values on the no-TargetPath
share path (values/values.go ExpandValueReferences/deepMergeShared)
   and then handed to helm's in-place-mutating render with no defensive copy
   (helm/template.go:73,97, finalValues := hrValues); hr.Clone() deep-copies
   before that re-aliasing, so it does not protect the canonical. Concurrent
   sibling HRs that share the same valuesFrom source then read a canonical
   another HR is mutating. Fix by owning hr.Values before helm coalesces it
   (deep-copy at template.go:73) and/or by not aliasing a cache canonical into
   a mutable value graph.
3. Keep the AddRendered dedup as hygiene (it mirrors the RS-controller
   guard and stops spurious dirty/broadcast churn) — just don't rely on it as
   the Non-deterministic hang under parallel reconcile — DAG scheduler never converges; --concurrency 1 avoids it #828 fix.
4. Optional defense-in-depth: the scheduler currently has no per-node
   re-dispatch bound, so any non-converging emit hangs rather than failing. A
   large-threshold re-dispatch cap in complete() that terminalizes a runaway
   node would convert a future determinism regression into a loud failure
   instead of a silent hang. This bounds the blast radius; it does not replace
   the determinism fix.

Priority p1: it breaks flate build/test on ~half of runs at the default
--concurrency for a realistically sized Flux tree (the CI-gate use case), but a
reliable workaround exists (--concurrency 1) and it is a hang, not data
corruption or a security exposure.

Residual risk / open question: the specific non-deterministic emit (step 2) is a
hypothesis until the -race reproduction in step 1 confirms it. What is proven
here is the necessary-and-sufficient condition (a content-unstable schedulable
emit under concurrent dispatch), the refutation of the status-write path, and
that de-duping AddRendered alone does not address the in-flight storm.

[buroa]

GC Issue Fix

• state: success
• summary: Run complete (scoped). Implementation committed at eb77002, code review approved (0 required changes), draft PR fix(schedule): bound per-node re-dispatch and surface a loud CapErr (#828) #859 opened. This ships only the approved Exit-B safety net (Change A): a per-node re-dispatch cap that bounds the scheduler's amplifier so a non-converging emit fails loudly instead of hanging silently. It converts a silent hang into a loud, attributable failure - it is not a full fix and the risk is not "low and contained". GC-828-R3 (serial/no-regression) is covered and independently verified; GC-828-R1, GC-828-R2, GC-828-R4 (every-run convergence, byte-identical parallel output, and the red->green -race repro) remain an explicitly-approved Change B follow-up, blocked on a cgo + fetched-chart -race environment. Non-deterministic hang under parallel reconcile — DAG scheduler never converges; --concurrency 1 avoids it #828 must stay open.
• pr: fix(schedule): bound per-node re-dispatch and surface a loud CapErr (#828) #859
• artifact: plans/github/issues/home-operations/flate/828/fix/fl-oku