fix(#6045): one Go soft memory limit per installation, split 30/70 serve/engine - #6046
Merged
Conversation
…rve/engine `daemon: applied Go soft memory limit (#3648)` was logged twice on every daemon start — once by `grafel serve`, once by `grafel engine --foreground`. Each process called debug.SetMemoryLimit with the FULL resolved limit, so on the default ceiling the real ceiling for the installation was 5120MB while `grafel status` advertised 2560MB. Every measurement taken against that budget (epic #5954 in particular) was against a number that was silently 2x. The resolved limit is now the budget for the WHOLE installation and each process applies only its plane's share. ## Channel: derived per-plane, not passed down The engine child is spawned by defaultEngineChildCommand (supervise.go) with `cmd.Env = os.Environ()` and a load-bearing comment forbidding env synthesis: force-appending GRAFEL_DAEMON_ROOT there once flipped the child to a different store layout and silently dropped serve-written reindex requests. Rather than add a second thing to that env, both processes derive their share from the plane they are ACTUALLY running: run(planeMonolith) -> memPlaneMonolith (whole budget) run(planeServeOnly) -> memPlaneServe (serve share) RunEngine -> memPlaneEngine (engine share) Both processes already see the same env, the same settings.json and the same host RAM, so both resolve the identical TOTAL independently and slice it. No channel to keep in sync, nothing new in the child env, and a standalone `grafel engine` started by hand still stays inside the installation budget. ## Ratio: 30/70, not 50/50 (MemLimitServeShare) The engine is the write plane — scheduler, watcher, extraction, fbwriter — and is where the allocation happens; a legitimate large reindex peaks at ~1-1.5GB Go heap per job (the measurement memLimitCeilingMB is already justified by). An even split of the 2560MB default would hand the engine 1280MB, BELOW its normal working set, making the runtime GC continuously against a limit it cannot honour. serve's large data structure is the graph_cache mmap, which is file-backed and not Go heap, so GOMEMLIMIT does not account for it at all. What serve actually allocates is MCP request/response marshalling and dashboard JSON: small, bursty, short-lived. 0.30 gives serve 768MB and the engine 1792MB on the default ceiling — clears the measured per-job peak with headroom, and the pair's total is exactly the advertised figure (the engine absorbs the rounding, so serve+engine == total for every input). Monolith (GRAFEL_SPLIT_MODE=0) is one process and keeps the WHOLE budget; the escape hatch is unchanged and is not halved. ## Reporting `grafel status` / `grafel doctor` now print the effective total plus both shares: go soft mem limit: 2560MB (768MB serve + 1792MB engine) (fraction-of-RAM (capped); ...) and the log names the plane and the total: applied Go soft memory limit (#3648, split #6045) limit_mb=1792 total_mb=2560 plane=engine ## RSS-budget admission control Checked — it does NOT have the same defect, despite looking like it does. The budget is armed in startEnginePlane, and split-mode serve runs with planeServeOnly, which skips the engine plane entirely, so exactly one process ever logs `scheduler: RSS-budget admission control enabled`. A regression test pins that (monolith must arm it, split-mode serve must not) so a future change that starts the engine plane inside serve cannot reintroduce the doubling there. Out of scope, per the issue: idle footprint / retention. On macOS Go returns pages with MADV_FREE_REUSABLE and they stay counted in phys_footprint; a 2396MB engine here fell to 1282MB under real memory pressure with no work done. That is measurement artifact, not retention. Known and deliberately unchanged: when an explicit GOMEMLIMIT env var is set, the Go runtime applied it before main() in BOTH planes and we cannot retroactively split it. That is the operator's explicit choice; the log now names the plane so the doubling is at least visible. Tests (strict TDD, all mutation-verified): - share arithmetic: sum-to-total, engine majority, disabled stays disabled - applyMemoryLimit per plane, asserting the runtime limit AND the log fields - RunServe split-mode: real in-process serve applies 3000 of 10000, not 10000 - RunServe monolith: real in-process daemon keeps the whole 10000 - RunEngine: real engine subprocess applies 7000 of 10000, not 10000 - status/doctor line: total plus both shares, and they must sum to the total - RSS budget armed by exactly one plane Mutants confirmed caught: split reverted to per-process-full; status line reporting one share; monolith halved (both via shareOf and via the plane mapping); log line reverted to its old fields; each call site reverted to the monolith plane; serve arming the engine plane. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0111KEDUVWEWm9G5RRnJqXft
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
daemon: applied Go soft memory limitwas logged twice on every daemon start — once bygrafel serve, once bygrafel engine --foreground. Each process setGOMEMLIMITto the full resolved budget independently, so the real ceiling for the installation was 2x whatgrafel statusadvertised.Reported by @auxmedrano with an unusually clean repro — idle daemon, empty index,
in_flight=0, 0.0% CPU — and confirmed independently here:Two lines, ~200 ms apart, one per process. On a 16 GB machine the reporter measured 11.4 GB of 12.2 GB swap in use.
This also undermines epic #5954's own measurements: that work reasoned about bounding memory against an advertised 2560 MB budget which was in fact 5120 MB.
Why the budget is derived, not passed down
The obvious channel — synthesising the child's environment in
defaultEngineChildCommand(internal/daemon/supervise.go:81) — is explicitly forbidden there, by a comment recording a past incident: force-appendingGRAFEL_DAEMON_ROOTonce flipped the child to~/.grafel/statewhile serve used~/.grafel/store, silently dropping serve-written reindex requests.So both processes instead derive their share from the plane they are running:
run(planeMonolith)→memPlaneMonolith(whole budget)run(planeServeOnly)→memPlaneServeRunEngine→memPlaneEngineBoth already see the same environment, the same
settings.jsonand the same host RAM, so each resolves the identical total independently and slices it. Nothing new in the child environment, nothing to keep in sync, and a hand-startedgrafel enginestays inside the budget too.The split is 30/70, not 50/50
MemLimitServeShare = 0.30, a named constant with the reasoning recorded beside it.An even split would be actively harmful. The engine is the write plane and a large reindex peaks at ~1–1.5 GB heap per job — the same measurement that justifies
memLimitCeilingMB. A 50/50 split of the 2560 MB default gives the engine 1280 MB, below its normal working set, so the runtime would GC continuously against a limit it cannot honour.serve's largest structure is the graph_cache mmap, which is file-backed and therefore not Go heap —
GOMEMLIMITdoes not account for it at all. serve's real heap is MCP and dashboard marshalling.So: 768 MB serve + 1792 MB engine, summing exactly to 2560 (the engine absorbs rounding). Monolith mode keeps the whole budget.
The RSS-budget admission control is not the same defect
It looks like it, since
scheduler: RSS-budget admission control enabled budget_mb=2048also appears per-process in the reporter's log. But it is armed insidestartEnginePlane, and split-mode serve runsplaneServeOnly, which skips the engine plane entirely (internal/daemon/server.go:660). Exactly one process arms it.Pinned in both directions by
TestRSSBudgetAdmissionControl_IsEnginePlaneOnly: monolith must log the marker (proving the fixture can exhibit it), split-mode serve must not.What the log and status now print
From a binary built from this tree:
The bug was visible in the log, so the fix had to change what the log says.
Verification
8 mutations, all killed: split reverted to per-process-full; status printing one share; monolith halved; the log line reverted to its old fields;
RunEnginemapped to the monolith plane;run()mapped to monolith; monolith mapped to the serve plane; serve also arming the engine plane. Independently re-verified — collapsing the split failsTestSplitMemLimitMB_EngineGetsMajority.Two findings from the author worth recording, both instances of patterns this project keeps hitting:
TestRSSBudgetAdmissionControl_IsEnginePlaneOnlywas vacuous on first run: the fixture neededMaxRSSBudgetMB: 2048and a non-nilSchedulerIndexbefore it could exhibit the marker at all. Caught and fixed before commit.math.MaxInt64inpinProcessMemLimit.go build ./... && go vet ./... && gofmt -l .clean;internal/daemon/...,internal/cliandcmd/grafelgreen with raw exit codes viartk proxy.Known and disclosed
GOMEMLIMITenv var is still per-process. The Go runtime applies it beforemain()in both planes and it cannot be retroactively split. That is the operator's explicit choice; the log now names the plane so the doubling is at least visible.GOMEMLIMITand the scheduler's RSS budget govern different things — total installation memory is serve + engine + admitted subprocess RSS. Out of scope here, but it belongs in any epic(index): reduce indexing time + RSS + worktree-derived incremental indexing #5954 measurement.MADV_FREE_REUSABLEand they remain counted inphys_footprint. Measured here: an engine reading 2396 MB fell to 1282 MB under real memory pressure with no work done, and to 264 MB on its own after 12 h idle. The 2x ceiling and the swap pressure are real; the footprint figure is partly not.Independently adversarially reviewed.
Closes #6045