How to correct configuration capabilities fo prometheus-node-exporter? maybe need add rules for seLinux?

[jidckii]

Recently, I wondered why I do not have metrics for file systems in the cluster?

It turned out that node-exporter did not have enough privileges + the exception rules needed to be corrected.

node_scrape_collector_success{collector="filesystem"} 0
in logs:
err="open /host/proc/1/mounts: permission denied"

I added privileged: true , but statistics for PVC did not appear, only for rootfs.

## promtheus-node-exporter sub-chart configurable values
## Please see https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-node-exporter
##
prometheus-node-exporter:
  tolerations:
    - key: ""
      operator: Exists
      effect: NoSchedule
    - key: ""
      operator: Exists
      effect: NoExecute
  resources:
    limits:
      cpu: 500m
      memory: 256Mi
    requests:
      cpu: 10m
      memory: 50Mi
  containerSecurityContext:
    allowPrivilegeEscalation: true
    privileged: true
    capabilities:
      add:
      - CAP_SYS_RESOURCE
      - CAP_SYS_ADMIN
  extraArgs:
    - --log.level=debug
    - --collector.filesystem.mount-points-exclude=^/(dev|proc|boot|run/credentials/.+|run/k3s/containerd/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/)
    - --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs|tmpds|tmpfs|shm|fuse.lxcfs)$

Error in in logs:

ts=2023-09-07T09:46:38.109Z caller=filesystem_linux.go:96 level=debug collector=filesystem msg="Error on statfs() system call" rootfs=/host/root/var/lib/kubelet/pods/af246936-a625-4dd1-a64c-e07ce1876fef/volumes/kubernetes.io~csi/pvc-94e4c6a3-03ad-4ce9-9d9b-f955d591c4a5/mount err="permission denied"

  containerSecurityContext:
    allowPrivilegeEscalation: true
    privileged: true
    runAsNonRoot: false
    runAsUser: 0

I had to run it as root, then the data showed up. But it doesn't seem very safe.
I'm not strong in seLinux policies, but it seems that the barrel can be in this?

[mysticaltech]

@jidckii Sorry just saw that now, next time just tag me directly. Here's how to fix this: #697 (comment)

[khash]

I'm sure I'm missing something major here, but the error with prometheus node exporter is in the prometheus pod's logs which are not an audit file and therefore audit2allow just returns with "Nothing to do".

The logs are in container logs on the host (/var/log/containers) and a typical error line looks like this:

2025-01-19T13:00:50.549378407Z stderr F ts=2025-01-19T13:00:50.548Z caller=collector.go:169 level=error msg="collector failed" name=filesystem duration_seconds=0.019186539 err="open /host/proc/1/mounts: permission denied"

This perhaps is my misunderstanding of the policy system, so apologies in advance if that's the case.

[khash]

This is the policy file I could come up with and it's still failing:

node_exporter.te
module node_exporter 1.0;

require {
    type container_t;
    type proc_t;
    type sysfs_t;
    type root_t;
    type etc_t;
    type device_t;
    type var_run_t;
    type container_file_t;
    class file { getattr open read };
    class dir { getattr open read search };
    class filesystem getattr;
    class lnk_file read;
}

#============= container_t ==============
# Allow container to access /host directory
allow container_t container_file_t:dir { getattr open read search };
allow container_t container_file_t:file { getattr open read };
allow container_t container_file_t:lnk_file read;

# Allow access through /host/proc
allow container_t container_file_t:dir { getattr open read search };
allow container_t proc_t:file { getattr open read };
allow container_t proc_t:dir { getattr open read search };
allow container_t proc_t:lnk_file read;

# Allow access through /host/sys
allow container_t container_file_t:dir { getattr open read search };
allow container_t sysfs_t:dir { getattr open read search };
allow container_t sysfs_t:file { getattr open read };
allow container_t sysfs_t:lnk_file read;

# Allow access through /host/root
allow container_t container_file_t:dir { getattr open read search };
allow container_t root_t:dir { getattr open read search };
allow container_t root_t:file { getattr open read };
allow container_t root_t:lnk_file read;
allow container_t etc_t:dir { getattr open read search };
allow container_t etc_t:file { getattr open read };
allow container_t etc_t:lnk_file read;
allow container_t device_t:dir { getattr open read search };
allow container_t var_run_t:dir { getattr open read search };
allow container_t var_run_t:file { getattr open read };

# Allow traversal through /host prefix
allow container_t container_file_t:dir search;