[Bug] ResourceBasedTuner memory metric counts reclaimable page cache (cgroup memory.current), freezing admission after cache-heavy bursts

[odedkedemdreeze]

What are you really trying to do?

Run activity workers under ResourceBasedTuner in a memory-limited container (Kubernetes, cgroup v2, no swap). Activities are memory-heavy and touch a lot of files (in our case terraform-based recovery: a ~700 MB provider plugin is mmap'd per run), so we gate admission on memory to avoid OOM.

Describe the bug

ResourceBasedTuner's memory measurement is effectively cgroup memory.current, which includes reclaimable page cache. Page cache does not fall when an activity finishes and poses no OOM risk (the kernel evicts it under pressure), but the tuner treats it as "used memory." After a cache-heavy burst, the reading stays pinned above target on an otherwise-idle pod, so can_reserve stays false and admission freezes for minutes — even for tiny activities that need almost no memory. Real (anonymous) memory is low the whole time.

This is a measurement bug, not a policy bug: the PID controller is fed the wrong signal.

Root cause (source-traced)

crates/sdk-core/src/worker/tuner/resource_based.rs, RealSysInfoInner::refresh() (cgroup branch, ~L508 on main):

if let Some(cgroup_limits) = lock.cgroup_limits() {
    self.total_mem.store(cgroup_limits.total_memory, Ordering::Release);
    self.cur_mem_usage.store(
        cgroup_limits.total_memory - cgroup_limits.free_memory,   // == memory.current
        Ordering::Release,
    );
    ...

cgroup_limits comes from sysinfo (0.38). Its cgroup-v2 path (src/unix/linux/system.rs) computes:

read_u64("/sys/fs/cgroup/memory.current"),                    // mem_cur
read_table_key("/sys/fs/cgroup/memory.stat", "anon", ' '),    // mem_rss
...
limits.free_memory = limits.total_memory.saturating_sub(mem_cur);
rss: mem_rss,

So total_memory − free_memory == memory.current (page cache included). Notably, sysinfo already parses memory.stat's anon into CGroupLimits.rss, but RealSysInfoInner ignores it and uses total − free instead. used_mem_percent = cur_mem_usage / total_mem then feeds the memory PID controller and the emitted usage gauge, so both track page cache.

Minimal reproduction

Container: cgroup v2, memory limit 2 GiB, no swap (standard EKS AL2023, kernel 6.12).

1. Start a worker with ResourceBasedTuner (e.g. target_memory_usage = 0.7) and let it admit activities.
2. Populate page cache — e.g. cat a ~1.4 GiB file to /dev/null (or run any file-mmap-heavy activity), then let it exit.
3. Observe: /sys/fs/cgroup/memory.current is now pinned near the limit while /sys/fs/cgroup/memory.stat anon is low. The tuner's used_mem_percent tracks memory.current (not anon), stays above target_memory_usage, and no new slots are admitted until the cache is evicted — which on an idle pod may not happen for a long time.

Direct measurement from a live pod (uid 1001, 2 GiB limit), reading a 674 MiB file into cache then dropping it with posix_fadvise(DONTNEED):

                        memory.current   anon    file(cache)
baseline                    621 MiB       573      24
after reading provider     1279 MiB       581     674   <- tuner sees +658 MiB "used", admission freezes
after fadvise DONTNEED       627 MiB       581      21   <- real usage was flat the whole time

anon (the true non-reclaimable footprint) never moved; only reclaimable file cache did — yet that's what gated admission.

Expected behavior

Admission should track memory that actually risks OOM — i.e. not reclaimable page cache. On a no-swap cgroup, that's anonymous memory (≈ memory.stat anon); more generally the working set (memory.current − inactive_file), which is exactly what the kubelet uses for its own memory-pressure/eviction decisions. Under this signal, the idle pod in the repro would show low usage and keep admitting.

Proposed fix (seeking maintainer direction before a PR)

The one-line minimal change is to use the anonymous figure sysinfo already provides:

self.cur_mem_usage.store(cgroup_limits.rss, Ordering::Release);   // memory.stat anon

But since this changes the metric for every language SDK, I'd like your steer on the preferred shape:

1. Anonymous only (cgroup_limits.rss) — simplest; correct on no-swap; but ignores swap and active file cache.
2. Working set (memory.current − inactive_file) — matches kubelet's eviction metric; the most defensible general default; would need inactive_file surfaced (sysinfo exposes memory.stat, or read it directly).
3. Configurable metric on ResourceBasedTunerOptions (e.g. memory_metric: Total | WorkingSet | Anonymous, default preserving current behavior) — no behavior change for existing users, opt-in for the corrected signal.

Happy to open a PR with a tuner test once you indicate which direction you'd accept (and whether it should be gated behind a config option for backwards compatibility).

Environment

• sdk-core main (verified at 9f83b7e307dc032b31ff3bd3811ef3438106f77a; lines unchanged on current main)
• sysinfo 0.38
• Linux cgroup v2, kernel 6.12 (Amazon Linux 2023), no swap, container memory limit via memory.max
• Observed via the Python SDK (temporalio 1.30.0) but the code path is language-agnostic sdk-core.

Related

Companion to the deadlock bug/fix in temporalio/sdk-python#1642 / temporalio/sdk-python#1643 (same workload). That fixed the CustomSlotSupplier GIL↔mutex deadlock; this issue is the separate memory-metric problem in the built-in ResourceBasedTuner.