Improved finding docker container id on cgroup v2 enabled systems

[tobiasstadler]

What does this PR do?

Try to find the docker container id in /proc/self/mountinfo on group v2 enabled system. Works on Ubuntu 21.10, Fedora 35 and Docker Desktop for Mac (4.3.0)

Checklist

• This is an enhancement of existing features, or a new feature in existing plugins
  • I have updated CHANGELOG.asciidoc
  • I have added tests that prove my fix is effective or that my feature works
  • Added an API method or config option? Document in which version this will be introduced
  • I have made corresponding changes to the documentation

[github-actions]

👋 @tobiasstadler Thanks a lot for your contribution!

It may take some time before we review a PR, so even if you don’t see activity for some time, it does not mean that we have forgotten about it.

Every once in a while we go through a process of prioritization, after which we are focussing on the tasks that were planned for the upcoming milestone. The prioritization status is typically reflected through the PR labels. It could be pending triage, a candidate for a future milestone, or have a target milestone set to it.

[apmmachine]

❕ Build Aborted

The PR is not allowed to run in the CI yet

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-06-20T11:39:24.365+0000

• Duration: 4 min 46 sec

Steps errors

Expand to view the steps failures

Load a resource file from a library

• Took 0 min 0 sec . View more details here
• Description: approval-list/elastic/apm-agent-java.yml

Error signal

• Took 0 min 0 sec . View more details here
• Description: githubApiCall: The REST API call https://api.github.com/orgs/elastic/members/tobiasstadler return the message : java.lang.Exception: httpRequest: Failure connecting to the service https://api.github.com/orgs/elastic/members/tobiasstadler : httpRequest: Failure connecting to the service https://api.github.com/orgs/elastic/members/tobiasstadler : Code: 404Error: {"message":"User does not exist or is not a member of the organization","documentation_url":"https://docs.github.com/rest/reference/orgs#check-organization-membership-for-a-user"}

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• run benchmark tests : Run the benchmark tests.

• run jdk compatibility tests : Run the JDK Compatibility tests.

• run integration tests : Run the Agent Integration tests.

• run end-to-end tests : Run the APM-ITs.

• run windows tests : Build & tests on windows.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[SylvainJuge]

Hi @tobiasstadler , thanks for opening this PR.

As this is a cross-agent topic we will have to handle this for all agents. There is already an issue opened for that : elastic/apm#523

Do you have a reference or examples on the file format ?

The tricky part here is that it makes it work only for Docker, while it allows to help run integration tests for the agent (which already helps a lot), it won't cover other types of containers.

[tobiasstadler]

The file format of /proc/self/mountinfo is described in https://man7.org/linux/man-pages/man5/proc.5.html.

It seems like for containerd it is (always) ${SOME_PATH}/${CONTAINER_ID}/(hostname|host|resolv.conf) (see https://github.com/containerd/containerd/blob/b93119501ff5251b1e79fcb034d78ad76793b021/pkg/cri/server/helpers_linux.go#L86).

e.g. on Docker for Mac:
docker run ubuntu cat /etc/self/mountinfo

473 387 0:143 / / rw,relatime master:181 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/HOY5TBZRCYUBQ4CYBGZQ6TSNEM:/var/lib/docker/overlay2/l/WUKBWU4V4V5WWC4QR6Q6SPWP4T,upperdir=/var/lib/docker/overlay2/af64ca21098b0841825b5175774e5827348cf7e00b8434ecc44ac4f4896815f4/diff,workdir=/var/lib/docker/overlay2/af64ca21098b0841825b5175774e5827348cf7e00b8434ecc44ac4f4896815f4/work
474 473 0:146 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
475 473 0:147 / /dev rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
476 475 0:148 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
477 473 0:149 / /sys ro,nosuid,nodev,noexec,relatime - sysfs sysfs ro
486 477 0:30 / /sys/fs/cgroup ro,nosuid,nodev,noexec,relatime - cgroup2 cgroup rw
487 475 0:145 / /dev/mqueue rw,nosuid,nodev,noexec,relatime - mqueue mqueue rw
488 475 0:150 / /dev/shm rw,nosuid,nodev,noexec,relatime - tmpfs shm rw,size=65536k
489 473 254:1 /docker/containers/8c6dd990bf77f52f1c2e0400ae757c1f77fd64393599588c62a19d37d68565e6/resolv.conf /etc/resolv.conf rw,relatime - ext4 /dev/vda1 rw
490 473 254:1 /docker/containers/8c6dd990bf77f52f1c2e0400ae757c1f77fd64393599588c62a19d37d68565e6/hostname /etc/hostname rw,relatime - ext4 /dev/vda1 rw
491 473 254:1 /docker/containers/8c6dd990bf77f52f1c2e0400ae757c1f77fd64393599588c62a19d37d68565e6/hosts /etc/hosts rw,relatime - ext4 /dev/vda1 rw
388 474 0:146 /bus /proc/bus ro,nosuid,nodev,noexec,relatime - proc proc rw
389 474 0:146 /fs /proc/fs ro,nosuid,nodev,noexec,relatime - proc proc rw
390 474 0:146 /irq /proc/irq ro,nosuid,nodev,noexec,relatime - proc proc rw
391 474 0:146 /sys /proc/sys ro,nosuid,nodev,noexec,relatime - proc proc rw
392 474 0:146 /sysrq-trigger /proc/sysrq-trigger ro,nosuid,nodev,noexec,relatime - proc proc rw
393 474 0:151 / /proc/acpi ro,relatime - tmpfs tmpfs ro
394 474 0:147 /null /proc/kcore rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
395 474 0:147 /null /proc/keys rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
396 474 0:147 /null /proc/timer_list rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
397 474 0:147 /null /proc/sched_debug rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
398 477 0:152 / /sys/firmware ro,relatime - tmpfs tmpfs ro

e.g. on Minikube with containerd:
kubectl run --image ubuntu -i --tty u

1352 1261 0:187 / / rw,relatime master:413 - overlay overlay rw,lowerdir=/mnt/sda1/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/44/fs,upperdir=/mnt/sda1/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/45/fs,workdir=/mnt/sda1/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/45/work
1353 1352 0:188 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
1354 1352 0:189 / /dev rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
1355 1354 0:190 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
1356 1354 0:177 / /dev/mqueue rw,nosuid,nodev,noexec,relatime - mqueue mqueue rw
1357 1352 0:182 / /sys ro,nosuid,nodev,noexec,relatime - sysfs sysfs ro
1358 1357 0:191 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime - tmpfs tmpfs rw,mode=755
1359 1358 0:22 /kubepods/besteffort/pod72f9a42e-50d3-47ee-8ec7-0bccfb6ea7ae/52371e0edd6309da3d4eaee6d303aa3401abbef92c34c2b1ddba7b28d82d6cb0 /sys/fs/cgroup/systemd ro,nosuid,nodev,noexec,relatime master:8 - cgroup cgroup rw,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd
1360 1358 0:24 /kubepods/besteffort/pod72f9a42e-50d3-47ee-8ec7-0bccfb6ea7ae/52371e0edd6309da3d4eaee6d303aa3401abbef92c34c2b1ddba7b28d82d6cb0 /sys/fs/cgroup/perf_event ro,nosuid,nodev,noexec,relatime master:11 - cgroup cgroup rw,perf_event
1361 1358 0:25 /kubepods/besteffort/pod72f9a42e-50d3-47ee-8ec7-0bccfb6ea7ae/52371e0edd6309da3d4eaee6d303aa3401abbef92c34c2b1ddba7b28d82d6cb0 /sys/fs/cgroup/pids ro,nosuid,nodev,noexec,relatime master:12 - cgroup cgroup rw,pids
1362 1358 0:26 /kubepods/besteffort/pod72f9a42e-50d3-47ee-8ec7-0bccfb6ea7ae/52371e0edd6309da3d4eaee6d303aa3401abbef92c34c2b1ddba7b28d82d6cb0 /sys/fs/cgroup/blkio ro,nosuid,nodev,noexec,relatime master:13 - cgroup cgroup rw,blkio
1363 1358 0:27 /kubepods/besteffort/pod72f9a42e-50d3-47ee-8ec7-0bccfb6ea7ae/52371e0edd6309da3d4eaee6d303aa3401abbef92c34c2b1ddba7b28d82d6cb0 /sys/fs/cgroup/net_cls,net_prio ro,nosuid,nodev,noexec,relatime master:14 - cgroup cgroup rw,net_cls,net_prio
1364 1358 0:28 /kubepods/besteffort/pod72f9a42e-50d3-47ee-8ec7-0bccfb6ea7ae/52371e0edd6309da3d4eaee6d303aa3401abbef92c34c2b1ddba7b28d82d6cb0 /sys/fs/cgroup/cpu,cpuacct ro,nosuid,nodev,noexec,relatime master:15 - cgroup cgroup rw,cpu,cpuacct
1365 1358 0:29 /kubepods/besteffort/pod72f9a42e-50d3-47ee-8ec7-0bccfb6ea7ae/52371e0edd6309da3d4eaee6d303aa3401abbef92c34c2b1ddba7b28d82d6cb0 /sys/fs/cgroup/cpuset ro,nosuid,nodev,noexec,relatime master:16 - cgroup cgroup rw,cpuset
1366 1358 0:30 /kubepods/besteffort/pod72f9a42e-50d3-47ee-8ec7-0bccfb6ea7ae/52371e0edd6309da3d4eaee6d303aa3401abbef92c34c2b1ddba7b28d82d6cb0 /sys/fs/cgroup/hugetlb ro,nosuid,nodev,noexec,relatime master:17 - cgroup cgroup rw,hugetlb
1367 1358 0:31 /kubepods/besteffort/pod72f9a42e-50d3-47ee-8ec7-0bccfb6ea7ae/52371e0edd6309da3d4eaee6d303aa3401abbef92c34c2b1ddba7b28d82d6cb0 /sys/fs/cgroup/freezer ro,nosuid,nodev,noexec,relatime master:18 - cgroup cgroup rw,freezer
1368 1358 0:32 /kubepods/besteffort/pod72f9a42e-50d3-47ee-8ec7-0bccfb6ea7ae/52371e0edd6309da3d4eaee6d303aa3401abbef92c34c2b1ddba7b28d82d6cb0 /sys/fs/cgroup/devices ro,nosuid,nodev,noexec,relatime master:19 - cgroup cgroup rw,devices
1369 1358 0:33 /kubepods/besteffort/pod72f9a42e-50d3-47ee-8ec7-0bccfb6ea7ae/52371e0edd6309da3d4eaee6d303aa3401abbef92c34c2b1ddba7b28d82d6cb0 /sys/fs/cgroup/memory ro,nosuid,nodev,noexec,relatime master:20 - cgroup cgroup rw,memory
1370 1352 8:1 /var/lib/kubelet/pods/72f9a42e-50d3-47ee-8ec7-0bccfb6ea7ae/etc-hosts /etc/hosts rw,relatime - ext4 /dev/sda1 rw
1371 1354 8:1 /var/lib/kubelet/pods/72f9a42e-50d3-47ee-8ec7-0bccfb6ea7ae/containers/u/241e0542 /dev/termination-log rw,relatime - ext4 /dev/sda1 rw
1372 1352 8:1 /var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/4af99a0ad37d0582b3ef55d27b20c7946c35c97ec18f5c9cc1fbf414feec6070/hostname /etc/hostname rw,relatime - ext4 /dev/sda1 rw
1373 1352 8:1 /var/lib/containerd/io.containerd.grpc.v1.cri/sandboxes/4af99a0ad37d0582b3ef55d27b20c7946c35c97ec18f5c9cc1fbf414feec6070/resolv.conf /etc/resolv.conf rw,relatime - ext4 /dev/sda1 rw
1374 1354 0:175 / /dev/shm rw,nosuid,nodev,noexec,relatime - tmpfs shm rw,size=65536k
1375 1352 0:174 / /run/secrets/kubernetes.io/serviceaccount ro,relatime - tmpfs tmpfs rw,size=5952364k
1262 1354 0:190 /0 /dev/console rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
1263 1353 0:188 /asound /proc/asound ro,nosuid,nodev,noexec,relatime - proc proc rw
1264 1353 0:188 /bus /proc/bus ro,nosuid,nodev,noexec,relatime - proc proc rw
1265 1353 0:188 /fs /proc/fs ro,nosuid,nodev,noexec,relatime - proc proc rw
1266 1353 0:188 /irq /proc/irq ro,nosuid,nodev,noexec,relatime - proc proc rw
1267 1353 0:188 /sys /proc/sys ro,nosuid,nodev,noexec,relatime - proc proc rw
1268 1353 0:188 /sysrq-trigger /proc/sysrq-trigger ro,nosuid,nodev,noexec,relatime - proc proc rw
1269 1353 0:192 / /proc/acpi ro,relatime - tmpfs tmpfs ro
1270 1353 0:189 /null /proc/kcore rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
1271 1353 0:189 /null /proc/keys rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
1272 1353 0:189 /null /proc/timer_list rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
1273 1353 0:193 / /proc/scsi ro,relatime - tmpfs tmpfs ro
1274 1357 0:194 / /sys/firmware ro,relatime - tmpfs tmpfs ro

[tobiasstadler]

In my opinion this is a step in the right direction, but definitely not the last.

[SylvainJuge]

Superseded by #3199 , with a similar heuristic, some refactors and even a testcase in json to help testing across agents.