bw serve leaks ~10 KB per HTTP request regardless of endpoint

[ggiesen]

Summary

bw serve retains roughly 10 KB of process memory per HTTP request handled, independent of which endpoint is called (/status, /sync, /object/item/... all behave the same). Over hours/days of normal operation this accumulates to tens of MB to multiple GB, eventually triggering V8 heap exhaustion (Ineffective mark-compacts near heap limit ... Allocation failed - JavaScript heap out of memory) and process death.

The leak is in the request-handling path, not in vault decryption or any sync-specific machinery — /status, which doesn't touch vault items, leaks at the same per-request rate as /object/item/....

Observed in production

Four bw serve deployments in our cluster all exhibit the same pattern. Memory growth measured via container_memory_working_set_bytes (cAdvisor).

A canonical 30-hour cycle on one deployment — smooth, monotonic, 110 MB → 330 MB before OOM-induced restart:

The same pattern, scaled up: a deployment that ran for 9.5 days without a memory limit so the only ceiling was V8's internal heap. 130 MB → 1.1 GB, perfectly linear, no plateau:

A deployment with a wrapper script that detects bw process death and restarts in-process — same per-request leak, capped at V8's default ~256 MB old-gen ceiling, ~22 hour cycles repeating for 10 days:

Diagnostic experiment

To isolate the cause we set HEALTH_CHECK_INTERVAL=600 (sync once every 10 minutes instead of every 30 seconds) on one deployment. Sync rate dropped 20×; total leak rate only dropped ~35%.

Config	API call rate (status + sync)	Leak rate	MB per request
30 s sync wrapper interval (probes + wrapper)	~720 req/h	7.73 MB/h	0.0107
600 s sync wrapper interval (probes + wrapper)	~492 req/h	5.05 MB/h	0.0103

Per-request leak is essentially identical at very different sync frequencies. The leak is per-HTTP-request, not specific to /sync. If sync was the cause, dropping sync 20× would have reduced leak rate ~95%, not ~35%.

Liveness/readiness probes (which only call /status) account for most of the residual API rate — and the residual leak rate scales with them, confirming /status leaks at the same rate as endpoints that touch vault data.

Predicted minimal reproduction

We have not directly run this minimal docker-based repro — the per-request leak rate is inferred from production cAdvisor measurements correlated with API call rate. Based on those measurements, an upstream maintainer should be able to reproduce in minutes:

# Auth + serve
bw login --apikey   # (BW_CLIENTID, BW_CLIENTSECRET in env)
bw unlock --raw > /tmp/session
export BW_SESSION=$(cat /tmp/session)
bw serve --port 8087 &
SERVE_PID=$!

# Hit /status in a tight loop — no vault items needed
while true; do curl -s http://localhost:8087/status > /dev/null; done &

# Watch RSS climb
while true; do echo "$(date +%T) $(ps -o rss= -p $SERVE_PID) KB"; sleep 60; done

Predicted: at ~1 req/s, RSS grows ~36 MB/h, hitting V8's default ~256 MB old-gen ceiling within a few hours.

Versions tested

• bw 2025.12.1 (nexe-built single binary in our deployments), confirmed leak.
• Latest npm release 2026.4.1 not directly tested — would be useful to confirm whether the leak still reproduces on current.

Environment

• Linux x86_64, single-binary bw (nexe-bundled Node).
• No proxy, no TLS — request handling is direct HTTP.
• Vault size is independent of leak rate in our observations: deployments serving small vaults (~150 items) leak at the same per-request rate as ones serving thousands of items. Consistent with /status leaking at the same rate as /object/item/....

Suggested investigation

Since /status leaks at the same rate as endpoints that touch vault data, the leak is unlikely to be in:

• Vault decryption / cache management
• Sync handlers
• Item-specific endpoint logic

More likely candidates:

• Per-request middleware or logging that retains references
• HTTP response buffer / framework state not being released
• An accumulating in-memory log or telemetry buffer
• Closure-capture of request/response objects in promise chains

A heap profile (--heap-prof) taken before and after a few thousand /status requests should pinpoint the retainer.

Existing related issues

• getting heap out of memory error while running ansible script from jenkins cli#261 (now in the archived bitwarden/cli repo) — closed for repository migration. Symptoms match: heap OOM under sustained bw serve API traffic from Ansible.
• CLI bw serve is killed after few minutes on arm64  #8762 — "CLI `bw serve` is killed after few minutes on arm64". Closed; symptoms align (OOM under load).

Filing fresh against this repo since both prior reports are closed and neither investigation appears to have continued.

Workarounds (documented for downstream consumers, not blocking on this issue)

• NODE_OPTIONS=--max-old-space-size=512 (or higher) defers the OOM but doesn't prevent it
• A wrapper script that detects bw process death and restarts in-process keeps the listening socket present (~7 s outage per restart)
• Client-side retry on ConnectionError / Timeout covers the restart window transparently

[bitwarden-bot]

Thank you for reporting this issue! We've added this to our internal tracking system.
ID: PM-36882
Link (for internal use): https://bitwarden.atlassian.net/browse/PM-36882

[ggiesen]

Source-code analysis

The stack is Koa + @koa/router + koa-bodyparser + koa-json (apps/cli/src/commands/serve.command.ts). Despite /status looking trivial, StatusCommand.run (apps/cli/src/commands/status.command.ts) does meaningful per-request work — most notably:

• userAutoUnlockKeyService.setUserKeyInMemoryIfAutoUserKeySet(userId) at libs/common/src/platform/services/user-auto-unlock-key.service.ts, which unconditionally calls keyService.getUserKeyFromStorage(...) and then keyService.setUserKey(autoUserKey, userId) on every request. The doc comment notes this is a "half measure improvement" awaiting a state-provider migration. Worth checking that re-setUserKey-ing the same key with the same userId is fully idempotent and doesn't push to any subject/observer chain.
• Multiple firstValueFrom(observable$) subscriptions to BehaviorSubject-backed services (account, environment, auth). Should be safe, but worth a heap-profile diff before/after a few thousand /status requests to confirm subscriptions are released.
• The Koa middleware pipeline — origin check → bodyparser → koaJson → router. If any middleware retains ctx via a closure that outlives the response (fire-and-forget promise, etc.), the entire request/response is kept alive.

A --heap-prof profile across a few thousand /status requests, with a focus on:

1. retainers of Koa.Context instances
2. growth in any Subject/BehaviorSubject observer arrays
3. retention rooted in KeyService or auto-unlock paths

…should narrow the source quickly. The leak occurs even when /status is the only endpoint hit, so anything specific to vault decryption, sync, or item endpoints can be ruled out.

[pamperer562580892423]

Just one remark from another user:

Versions tested
bw 2025.12.1 (nexe-built single binary in our deployments), confirmed leak.
Latest npm release 2026.4.1 not directly tested — would be useful to confirm whether the leak still reproduces on current.

CLI 2025.12.1 is not supported any longer, so I would recommend to update your report with an up-to-date CLI version.

 

(--> https://bitwarden.com/help/bitwarden-software-release-support/#bitwarden-clients)

[ggiesen]

Thanks @pamperer562580892423 - fair point on the support stance. Quick context on why we're testing against 2025.12.1: we deploy via ghcr.io/charlesthomas/bitwarden-cli (a community wrapper image, since Bitwarden doesn't publish an official bw serve container). That image's :2026.x.y tags ship the 2025.12.1 bw binary unintentionally - there's a build-arg name mismatch in the upstream Dockerfile/workflow, tracked in charlesthomas/bitwarden-cli#9. So even pulling the latest tagged image doesn't get us a current CLI.

Two paths forward on our side:

1. Wait on charlesthomas/bitwarden-cli#9 to land, then re-test on a 2026.x image and confirm whether the leak persists.
2. Build our own image with npm install -g @bitwarden/cli@latest to bypass the wrapper entirely.

Either way, the leak pattern here (~10 KB retained per HTTP request, observable on the trivial /status endpoint regardless of vault contents) is structural enough that we'd expect it to reproduce on current. If anyone can reproduce it on bw 2026.4.1 directly via the minimal repro in the issue body, that would be the fastest signal - both for confirming this isn't fixed and for ruling out the older-CLI-version objection.

Will update once we've re-tested.

[mschroeder-fzj]

I can confirm that the issue still persists with version 2026.4.1 and any other 2026.x version.

I am running the CLI client with bw serve in a Kubernetes environment.
Additionally /sync is also leaking memory which can verified by checking the used VmRSS of the bw serve --hostname 0.0.0.0 process:

• Run bw serve in a container and access its shell
• Get the correct PID (should be 7):

  sh -c '
  for p in /proc/[0-9]*; do
    cmd=$(tr "\0" " " < $p/cmdline 2>/dev/null)
    case "$cmd" in
      *bw*)
        echo "---"
        echo "PID: ${p##*/}"
        echo "$cmd"
      ;;
    esac
  done
  '

• Run the sync API call:

  PID=7
  sh -c '
    for i in $(seq 1 30); do
      wget -q -O /dev/null http://127.0.0.1:8087/sync?force=true
    done
    cat /proc/$PID/status | grep VmRSS
    cat /proc/$PID/smaps_rollup
  '

Interesstingly bw sync doesn't seem to have that issue, which suggest the issue is in the http API.

[cbbit]

Hi there,

This report has been escalated for further investigation. If you have more information that can help us, please add it below.

Thanks!

[ggiesen]

Thanks @cbbit for the escalation. Following up with the current-version data we promised earlier in the thread.

The leak persists in bw 2026.4.1

All four of our bw serve deployments now run bw 2026.4.1. Memory grows monotonically in every one of them. Independently confirms @mschroeder-fzj's observation earlier in this thread.

Cleanest single curve - our ESO bw serve deployment, probe-driven traffic only (~2 requests/min via liveness probe), 42 hours of monotonic growth from ~120 MB to ~410 MB before V8 hits its heap ceiling:

The per-call signature persists

Running two bw serve instances at different HEALTH_CHECK_INTERVALs (which drives the wrapper's /sync rate, in turn driving the total /sync + /status call rate):

Wrapper sync interval	API calls/hour (approx)	Observed slope	Per-request retention
30s	~720 req/h	~17 MB/h	~0.024 MB/req
600s	~492 req/h	~12 MB/h	~0.024 MB/req

Per-request retention is essentially constant (~0.024 MB/req) regardless of which deployment. Visualized on a single panel, both deployments since rollout to bw 2026.4.1, same image, same memory limit, same time window:

The proportional reduction at lower call rate matches the same comparison on bw 2025.12.1 from the original issue body (600s/30s ratio was 0.65 then, 0.71 now). Same mechanism, observable in both versions.

What this updates

The structural finding from our earlier source-code analysis - leak in the per-HTTP-request retention path, not in any endpoint-specific or vault-decryption code - holds on current. No new code pointers to add; just confirming the analysis applies to bw 2026.4.1.

Existing workarounds (NODE_OPTIONS=--max-old-space-size, in-process restart wrapper, client-side retry on ConnectionError) still effective at hiding the impact, but don't address the underlying retention.

Happy to capture a --heap-prof profile if useful for engineering - we have these deployments running continuously and can isolate a /status load window if helpful.