-
Notifications
You must be signed in to change notification settings - Fork 3.8k
Description
Is your feature request related to a problem? Please describe.
In some environments and test setups, a hardcoded readiness probe can affect the scaling time of the Grafana Loki Canary feature. This usually occurs due to low resource availability in demo clusters or during stress tests. In production environments, the readiness probe may also affect the log collector, especially in cases of misconfiguration or when running on older/legacy Kubernetes environments.
In my case, the following message frequently appears:
"Readiness probe failed: Get "http://<MY_PRIVATE_IP>:3500/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers)".
To address this, I would like to add a customizable readiness probe to the Loki Canary DaemonSet.
Describe the solution you'd like
Change the actual hardcoded readiness probe in Loki Canary Helm.
Describe alternatives you've considered
Change the actual hardcoded:
initialDelaySeconds: 15
timeoutSeconds: 1
to
initialDelaySeconds: 30
timeoutSeconds: 5
Additional context
Readiness Probe Failure Report
Pod Info
- Name:
loki-canary-h9wsj.18684f0903c85a51
- Namespace:
monitoring
- Kind:
Pod
- Container:
loki-canary
Event Details
-
Type:
Warning
-
Reason:
Unhealthy
-
Message: Readiness probe failed: Get "http://<MY_PRIVATE_IP>:3500/metrics
":context deadline exceeded (Client.Timeout exceeded while awaiting headers) -
Source:
kubelet <MY_CLUSTER_AND_NODE_ID>
Timestamps
- First Seen:
2025-09-24T16:35:54-03:00
(16h ago) - Last Seen:
2025-09-25T09:14:34-03:00
(9m 11s ago) - Created:
2025-09-25T09:14:34-03:00
- Count:
15
occurrences