MK8S-11 Extend Threshhold for NodeSystemSaturation

Author: eg-ayoub

CHANGELOG.md

@@ -15,6 +15,11 @@
   and Fluent Bit Helm chart version to [0.50.0](https://github.com/fluent/helm-charts/releases/tag/fluent-bit-0.50.0)
   (PR[#4627](https://github.com/scality/metalk8s/pull/4627))
 
+### Bug Fixes
+
+- Fix a Bug where NodeSystemSaturation alert triggers too early after only 15 minutes of high load
+  (PR[#4641](https://github.com/scality/metalk8s/pull/4641))
+
 ## Release 130.0.4 (in development)
 
 ## Release 130.0.3

charts/kube-prometheus-stack.yaml

@@ -9,6 +9,10 @@ commonLabels:
   app.kubernetes.io/managed-by: 'metalk8s'
   metalk8s.scality.com/monitor: ''
 
+customRules:
+  NodeSystemSaturation:
+    for: "30m"
+
 # NOTE: cannot use this at the moment because kube-state-metrics uses it incorrectly
 # global:
 #   imageRegistry: '__var__(repo.registry_endpoint)'

salt/metalk8s/addons/prometheus-operator/deployed/chart.sls

@@ -74765,7 +74765,7 @@ spec:
       expr: |-
         node_load1{job="node-exporter"}
         / count without (cpu, mode) (node_cpu_seconds_total{job="node-exporter", mode="idle"}) > 2
-      for: 15m
+      for: 30m
       labels:
         severity: warning
     - alert: NodeMemoryMajorPagesFaults