v0.12.4 consumes 8GB RAM on startup after building with or without jemalloc on production grade server

[jcstein]

Celestia Node version

v0.12.4

OS

ubuntu, production grade server

Install tools

No response

Others

No response

Steps to reproduce it

Run light node

Expected result

Syncs using under 500 MB RAM as per documentation

Actual result

Uses 8GB RAM on startup, with and without jemalloc

Relevant log output

No response

Notes

Originally reported in celestia-da rollkit/celestia-da#35 (comment)

[jcstein]

cc @tuxcanfly and @mycodecrafting for visibility on the issue

[mycodecrafting]

some added info.

OS: Ubuntu 20.04.6 LTS
Arch type: x86_64

Build and install following guide here: https://docs.celestia.org/nodes/celestia-node

Go version: go1.21.1 linux/amd64

celestia built version

Semantic version: v0.12.4
Commit: 8e5a71746bc9bd9300ad3e4853797e64fdd1d018
Build Date: Tue 16 Jan 2024 08:26:39 PM UTC
System version: amd64/linux
Golang version: go1.21.1

[tuxcanfly]

Attaching heap snapshots from @mycodecrafting collected at 1m intervals.

heap-snapshots.tar.gz

[jcstein]

from @mycodecrafting: Ubuntu server and mem usage on celestia-node has grown to 18.6G overnight

[Wondertan]

Does this run over an clean node or synced node?

[Wondertan]

After debugging this closely with @mycodecrafting, my conclusion is that the issue is a false alarm:

• The memory is not being leaked but cached(TIL).
• The runtime is smart enough to avoid unnecessary syscalls for memory allocations as it sees an abundance of RAM.
  • This is also dependent on the OS capabilities
• It is based on reading through following issues(#33376 (runtime: GC: heap idle is not released to linux golang/go#33376), #22439 (runtime: memory not being returned to OS golang/go#22439))

If we ever find an OOM and an ever-growing heap, then we can revisit the conclusion with more data points,
but until then, there is nothing to worry about as the node works and behaves correctly.