ci(validate-flux-render): work around flate's concurrent-dispatch livelock - #1795
Merged
Conversation
…elock Root-caused the recurring "flate + kubeconform (kubenuc)" hang (3 occurrences, 2026-08-06): a confirmed, still-open upstream bug in flate itself (home-operations/flate#828), not anything specific to this repo. Under flate's default concurrency (runtime.NumCPU()*4), a race in the DAG scheduler can make a HelmRelease's rendered output non-deterministic across re-runs, which re-arms the scheduler's dispatch loop and never converges - a livelock. kubenuc is this repo's largest/most complex cluster tree (26 chart sources, 29 apps), making it statistically most likely to hit a non-deterministic, DAG-size-scaling race - not something unique to its content. The issue reporter's documented workaround (--concurrency 1) got 8/8 clean runs vs hangs on ~half of parallel runs, byte-identical output to the good parallel runs. Add it to both flate test and flate build. This prevents the race rather than just bounding its blast radius like the timeout-minutes added in #1794, which stays as defense-in-depth in case this doesn't fully eliminate it. The upstream fix in flight (home-operations/flate#859) only adds a re-dispatch cap that converts a hang into a loud failure - it's not merged, and even merged it wouldn't restore correctness, just fail faster. Revisit --concurrency 1 once a flate release actually fixes the underlying render non-determinism. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Daniele De Lorenzi <2905124+dark-vex@users.noreply.github.com>
This was referenced Aug 6, 2026
dark-vex
added a commit
that referenced
this pull request
Aug 7, 2026
ci(validate-flux-render): work around flate's concurrent-dispatch livelock
dark-vex
added a commit
that referenced
this pull request
Aug 7, 2026
oc-ampere had zero live e2e coverage (only offline flate/kubeconform schema checks) despite being a real production Flux Operator cluster. Ports the same top-level-resource loop (Mechanism 1) and fail-closed per-app loop (Mechanism 2) already merged for kubenuc/k8s-vms-daniele (#1790-#1795), extended with a new fourth resolution tier: a dirname()-based fallback that parses the root apps/kustomization.yaml for apps with no per-app deploy.yaml of their own. 2 of 4 real apps deploy for real: flux-operator (real-path fallback; verified via `helm template` with empty values that its chart installs only a Deployment/RBAC/CRDs with no hooks and stays idle until a FluxInstance CR exists — flux-instance.yaml itself stays skip-listed in this job, which bootstraps Flux via plain `flux install` instead) and ngx-webhook (new tier-4 resolver; no credentials, no outbound calls, NodePort-only). system-upgrade-controller and teleport-agent are excluded for the same live-external-side-effect reasons as k3s-rabbit's. Production oc-ampere's compute is genuinely ARM64 (terraform/oci/k8s-armchair, VM.Standard.A1.Flex); this e2e run validates manifest/reconciliation correctness on the (undocumented, presumed x86_64) self-hosted runner, not ARM64 image availability — stated plainly in the workflow's header comment. The comment-only touch to apps/ngx-webhook/manifests/deploy.yml exercises the tier-4 resolver on this workflow's first real CI run. Signed-off-by: Daniele De Lorenzi <2905124+dark-vex@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Root-caused the recurring "flate + kubeconform (kubenuc)" hang mitigated by #1794's
timeout-minutes: 5— this is the actual fix, not just a blast-radius bound.Root cause: a confirmed, still-open bug in
flateitself — home-operations/flate#828. Under flate's default concurrency (runtime.NumCPU()*4), a race in the DAG scheduler can make a HelmRelease's rendered output non-deterministic across re-runs, which re-arms the scheduler's dispatch loop and it never converges — a livelock, not a deadlock. Confirmed by both the issue reporter and an independent triage in the thread (goroutine dumps, byte-diff testing, and a written root-cause chain through the scheduler/store code).This is not specific to this repo's kubenuc content — kubenuc is just this repo's largest/most complex cluster tree (26 chart sources, 29 apps, vs. 12/1/21 chart sources and 18/3/4 apps for the other three), and the race is non-deterministic with probability scaling by DAG size, not something about kubenuc's manifests specifically.
Fix
Add
--concurrency 1to bothflate testandflate buildinvocations. The upstream issue reporter's documented workaround: 8/8 clean runs at concurrency 1 vs. hangs on ~half of parallel runs at the default, byte-identical output to the good parallel runs — serializing dispatch avoids the race entirely.The upstream fix in flight (PR #859) only adds a re-dispatch cap that converts a silent hang into a loud failure — it's still unmerged, and even merged it wouldn't restore correctness, just fail faster instead of hanging.
timeout-minutes: 5from #1794 stays as defense-in-depth in case--concurrency 1doesn't fully eliminate the race, or a future flate version reintroduces it.Test plan
validate-flux-render.yml, so all 4 clusters run on this PR directly — confirm all 4 still pass and check whether serializing meaningfully increases runtime (expected: still a few seconds per the upstream reporter's data for trees of this size)🤖 Generated with Claude Code