Prometheus node exporter shows btrfs filesystem error even no error found

[dusatvoj]

Host operating system: output of uname -a

# uname -a
Linux gitaly 6.1.0-28-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.119-1 (2024-11-22) x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

# prometheus-node-exporter --version
node_exporter, version 1.5.0 (branch: debian/sid, revision: 1.5.0-1+b6)
  build user:       [email protected]
  build date:       20230409-10:11:09
  go version:       go1.19.8
  platform:         linux/amd64

node_exporter command line flags

N/A

node_exporter log output

after restart ...

Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.953Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=debian/sid, revision=1.5.0-1+b6)"
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.955Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.8, platform=linux/amd64, [email protected], date=20230409-10:11:09)"
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.955Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run|sys|mnt|media|var/lib/docker/.+|var/lib/containers/storage/.+)($|/)
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.956Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.956Z caller=netdev_common.go:61 level=info collector=netdev msg="Parsed flag --collector.netdev.device-exclude" flag=^lo$
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=systemd_linux.go:133 level=info collector=systemd msg="Parsed flag --collector.systemd.unit-include" flag=.+
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=systemd_linux.go:135 level=info collector=systemd msg="Parsed flag --collector.systemd.unit-exclude" flag=.+\.(automount|device|mount|scope|slice|target)
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:110 level=info msg="Enabled collectors"
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=arp
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=bcache
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=bonding
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=btrfs
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=conntrack
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=cpu
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=cpufreq
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=diskstats
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=dmi
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=edac
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=entropy
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=fibrechannel
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=filefd
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=filesystem
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=hwmon
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=infiniband
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=ipvs
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=loadavg
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=mdadm
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=meminfo
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=netclass
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=netdev
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=netstat
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=nfs
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=nfsd
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=nvme
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=os
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=powersupplyclass
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=pressure
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=rapl
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=schedstat
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=selinux
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=sockstat
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=softnet
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=stat
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=systemd
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=tapestats
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=textfile
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=thermal_zone
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=time
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=timex
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=udp_queues
Jan 02 11:57:34 gitaly prometheus-node-exporter[4022759]: ts=2025-01-02T10:57:34.957Z caller=node_exporter.go:117 level=info collector=uname

Are you running node_exporter in Docker?

Nope

What did you do that produced an error?

Nothing just added alerts and found out filesystem error on one subvolume.

But the btrfs shows me:

# btrfs scrub status /var/opt/gitlab/git-data/repositories
UUID:             fdfe8ec6-547f-40ed-9643-275a4ed96714
Scrub started:    Wed Jan  1 20:49:36 2025
Status:           finished
Duration:         0:15:28
Total to scrub:   1.57TiB
Rate:             1.74GiB/s
Error summary:    no errors found

What did you expect to see?

No alert for filesystem error when btrfs doesn't report filesystem error (haven't found any).

What did you see instead?

filesystem error = 1