Calculation of kubeReserved.memory

[cyrus-mc]

Was recently putting together some dashboards and became confused by what I was seeing for reported memory usage vs what I thought was running on the box.

Long story short looking at container_memory_working_set_bytes{id="/kubepods.slice"} for all POD(s) and comparing that to container_memory_working_set_bytes{id="/"} (which is basically the system memory usage) I noticed a large difference. That difference can only be used up by the runtime or system components.

When I dug into the usage of memory for containerd (using memory.current - inactive_file) I noted that containerd was using 5G of memory. This seemed oddly strange to me that the runtime would be using up that much memory, given that the calculation for kubeReserved.memory on this specific EC2 instance is only 850M.

I am sure I am missing something here in how these values are calculate because when calculating the containerd memory usage, combined with the POD memory usage the value is greater than what kubectl top nodes gives for that specific node.

[etungsten]

Hi @cyrus-mc, can you share the entire /sys/fs/cgroup/memory/runtime.slice/containerd.service/memory.stat for us to take a look at?

I assume the reported memory usage on the dashboard is higher than expected. What was the difference between container_memory_working_set_bytes{id="/kubepods.slice"} and container_memory_working_set_bytes{id="/"}?

[cyrus-mc]

@etungsten Thanks for the reply.

So there are a couple of issues that I am noting here and not sure it has anything to specifically do with Bottlerocket as I see this same type of discrepancy on CoreOS (my previous base OS for worker nodes; moving to Bottlerocket). Will stick to the main point of this discussion.

This deep dive all started because running kubectl top nodes the memory being reported seemed higher than what I expected it to be (on a t3.small for example, used to run kiam server, the memory reported was 134%).

Metric container_memory_working_set_bytes{id="/"} returns a value very close to what kubectl top nodes returns (which for this specific node I am looking at is 5G). I then ran sum(container_memory_working_set_bytes{id="/kubepods.slice"}) which returned about 1.1G.

So based on the above I was seeing a 4G difference between the total system working set bytes and the working set bytes used by all PODs on the system. From there I checked out what the working set was for the system.slice.

• system.slice:

  • memory.usage_in_bytes = 256614400
  • total_inactive_file = 33996800 (from memory.stat file)

According to how cadvisor calculates workingSetBytes that means system.slice working set is 222621696 bytes (222) megs, so system components aren't using up the majority of that 4G.

• runtime.slice

  • memory.usage_in_bytes = 12386115584
  • total_inactive_file = 7732719616

That means a workingSet of 4.6G. Breaking that down further

• runtime.slice/kubelet.service

  • memory.usage_in_bytes = 206360576
  • total_inactive_file = 10502144

WorkingSet of 195858432 (200M)

• runtime.slice/containerd.service

  • memory.usage_in_bytes = 12177403904
  • total_inactive_file = 7722487808

Full memory.stat file

cache 10739638272
rss 170172416
rss_huge 0
shmem 2027520
mapped_file 282771456
dirty 0
writeback 135168
swap 0
pgpgin 1814966769
pgpgout 1812303130
pgfault 2064747102
pgmajfault 198
inactive_anon 171012096
active_anon 1757184
inactive_file 7722622976
active_file 3014918144
unevictable 0
hierarchical_memory_limit 9223372036854771712
hierarchical_memsw_limit 9223372036854771712
total_cache 10739638272
total_rss 170172416
total_rss_huge 0
total_shmem 2027520
total_mapped_file 282636288
total_dirty 0
total_writeback 135168
total_swap 0
total_pgpgin 1814966769
total_pgpgout 1812303130
total_pgfault 2064747102
total_pgmajfault 198
total_inactive_anon 171012096
total_active_anon 1757184
total_inactive_file 7722622976
total_active_file 3014918144
total_unevictable 0

WorkingSet of 4454916096 (4.4G)