engine idle floor is ~1.28GB after a reindex (RSS shows 2.4GB; ~1.1GB is reclaimable MADV_FREE)

[cajasmota]

Summary

After a full reindex the engine settles at ~2.5 GB RSS and stays there indefinitely, with no work in flight. sched: returned idle heap to OS fires and RSS does not move.

On a 16 GB machine shared with the user's real work — the stated driver for #5954 — this permanent floor matters more than the transient peak the epic has been optimising.

Measured

Full from-scratch reindex of the corpus group (427,301 entities / 1,852,625 relationships), engine at bbf22392f. Clean run: swap flat at 582 MB throughout, available memory never below 6359 MB, so these are not pressure artifacts.

Engine RSS sampled every 2 s:

t=   0s  engine=  693MB     <- freshly restarted
t= 121s  engine= 1806MB
t= 241s  engine= 2190MB
t= 601s  engine= 2494MB
t= 720s  engine= 2495MB
...
t=1921s  engine= 2495MB     <- flat for 22 minutes

sched: returned idle heap to OS idle_for=34s freeosmemory_took=96ms fired at t=914s. RSS after it: unchanged at 2495 MB. Confirmed again 20 minutes after the run ended — 2496 MB, 0.5% CPU, in_flight=0.

For contrast, the same engine before this reindex was 365 MB, and 693 MB immediately after a restart. So the ~2.5 GB is acquired by the reindex and then never released.

Where the time and memory actually go

Same run, phase breakdown:

phase	duration	peak RSS
extraction (index children)	111 s	3830 MB
engine in-process (resolve refs / links / analytics)	~760 s	2495 MB
community detection (group-algo)	6 s	1697 MB
peak summed across all processes		4767 MB

Two things worth noting for anyone reasoning about this:

• Extraction and community detection together are 117 s of a ~880 s pipeline. The dominant wall-time cost is now engine in-process work, which emits almost no log lines — the 14-minute window after group-algo contains exactly one message.
• The 4767 MB peak is transient and serialised (~2 min, during extraction). The 2.5 GB floor is permanent. For the concurrent-worktree-agent workload, the floor is the binding constraint.

Probable cause, unverified

FreeOSMemory on darwin releases pages via MADV_FREE, which does not reduce RSS until the kernel actually reclaims under pressure. If so, the memory is available to the system and the RSS number overstates the harm — but it would still show up as pressure on a machine that is already tight, and it means the log line is claiming something the system does not reflect.

The alternative is that the engine is genuinely retaining live heap after a rebuild (index structures, the prior graph, or analytics scratch never dropped). These have very different fixes, so the first task is to distinguish them — compare runtime.MemStats live heap (next_gc / (1 + GOGC/100)) against RSS at idle, rather than trusting either number alone.

Note heap_inuse is not live heap; that mistake has been made in this epic before.

Why this is filed separately from the peak work

#5954's slices so far (#6009, #6011, #6017, #6025) all target transient peak during the pipeline, and they are measured wins. None of them touch what the engine holds at rest. This is a different axis and needs its own measurement before any fix is designed.

Refs #5954

[cajasmota]

Answered: mostly MADV_FREE, and my framing above was wrong

Decisive test — apply real memory pressure and see whether the kernel reclaims the pages without the engine doing anything. Allocated 5 GB in 256 MB chunks, touching every page:

baseline:        engine=2396MB  avail=6886MB
  +1.0GB         engine=2396MB
  +2.0GB         engine=2353MB
  +3.0GB         engine=2169MB
  +4.0GB         engine=1628MB
  +5.0GB         engine=1281MB
PEAK PRESSURE:   engine=1282MB
after 5s hold:   engine=1289MB
released:        engine=1290MB

reclaimed under demand	~1114 MB — was MADV_FREE, available to the system all along
genuine resident floor	~1282 MB — did not return after pressure was released

So sched: returned idle heap to OS was honest. Go really did return those pages; macOS simply does not decrement RSS until something demands the memory. The title of this issue and the "never releases it" framing overstate the harm by roughly 1.1 GB. Correcting rather than quietly editing, because the wrong version was the basis for prioritising this.

What is still real

A ~1.28 GB floor per engine, post-reindex, at idle. That is meaningful for the concurrent-worktree-agent workload but it is not the emergency the 2.5 GB figure implied, and it does not obviously outrank the transient peak any more.

The broader consequence

Every RSS number in #5954 is measured the same way and carries the same overstatement. That includes the 4767 MB whole-pipeline peak measured on bbf22392f, and by implication the 5849 MB baseline it was compared against. Neither is wrong as an upper bound, but neither is a footprint, and the delta between them is only trustworthy if both include a similar proportion of reclaimable pages — which nobody has checked.

Before any further slice is justified on peak-RSS grounds, the peak should be re-derived from a metric that distinguishes live heap from returned-but-not-reclaimed pages. Live heap is next_gc / (1 + GOGC/100); heap_inuse is not live heap (see internal/graph/groupalgo/phase.go:11).

Not yet answered

What the remaining ~1.28 GB actually is. GRAFEL_MEMTRACE_DIR was never successfully set on the engine — the supervisor spawns grafel engine --foreground inheriting serve's environment, so enabling it means restarting serve, which drops live MCP sessions. Getting heap attribution without a serve restart is its own small problem and is the blocker on answering it.

[cajasmota]

Closing — the premise does not survive a longer observation window

Same engine, same graph, no reindex in between, 12h27m after the run that produced the 2.4 GB reading:

engine pid=40238  rss=264MB  cpu=3.5%  state=S  up=12:29:08
serve  pid=97353  rss=19.4MB          in_flight=0
group: 427,303 entities · 1,852,630 relationships   (intact)

	right after reindex	after 12h idle
engine	2396 MB	264 MB
serve	212 MB	19 MB

So the memory is released. It needed either real demand (the pressure test above reclaimed ~1114 MB in seconds) or simply time. There is no permanent floor, and sched: returned idle heap to OS was accurate throughout.

What actually happened: I sampled RSS a few minutes after a reindex, saw a flat line across a 22-minute window, and concluded "never releases it". A flat RSS line over 22 minutes is not evidence of a permanent floor — it is evidence that nothing asked for the memory during those 22 minutes. Two separate corrections were needed to get here, which is the real lesson: on macOS, an RSS plateau observed while the machine has spare memory says nothing about whether the memory is retained.

Closing as invalid.

What survives, and is worth keeping

1. RSS is the wrong metric for this codebase's memory work, and it is the metric epic(index): reduce indexing time + RSS + worktree-derived incremental indexing #5954 has used throughout. Every peak figure in that epic — including the 4767 MB whole-pipeline peak and the 5849 MB baseline it is compared against — is an upper bound that includes returned-but-unreclaimed pages, in unknown and probably differing proportions. The allocation-based benchmarks (TotalAlloc, B/op) do not have this problem and remain trustworthy. Re-deriving the pipeline peak from live heap (next_gc / (1 + GOGC/100); note heap_inuse is not live heap) is the prerequisite for justifying any further peak-focused slice.

2. The engine cannot be instrumented without restarting serve. The child inherits serve's environment unchanged, so GRAFEL_MEMTRACE_DIR cannot be set on the engine alone, and restarting serve drops live MCP sessions. A marker-file fallback (~/.grafel/.memtrace-dir, read at engine start when the env var is absent) works and is the obvious fix, but it is a hack as prototyped.

3. memtrace.Record should carry HeapReleased, HeapIdle and Sys. It currently records HeapAlloc/HeapInuse/HeapSys/NextGC but not the three fields that distinguish "returned to the OS" from "still live" — which is precisely the question this issue turned out to be about. That is a small, permanent improvement worth making on its own merits.

Filing (2) and (3) separately since they are real and this issue is not.