Skip to content

Increase Loki Canary readiness probe customization #19302

@strovertz

Description

@strovertz

Is your feature request related to a problem? Please describe.

In some environments and test setups, a hardcoded readiness probe can affect the scaling time of the Grafana Loki Canary feature. This usually occurs due to low resource availability in demo clusters or during stress tests. In production environments, the readiness probe may also affect the log collector, especially in cases of misconfiguration or when running on older/legacy Kubernetes environments.

In my case, the following message frequently appears:
"Readiness probe failed: Get "http://<MY_PRIVATE_IP>:3500/metrics": context deadline exceeded (Client.Timeout exceeded while awaiting headers)".

To address this, I would like to add a customizable readiness probe to the Loki Canary DaemonSet.

Describe the solution you'd like
Change the actual hardcoded readiness probe in Loki Canary Helm.

Describe alternatives you've considered
Change the actual hardcoded:

initialDelaySeconds: 15
timeoutSeconds: 1

to

initialDelaySeconds: 30
timeoutSeconds: 5

Additional context

Readiness Probe Failure Report

Pod Info

  • Name: loki-canary-h9wsj.18684f0903c85a51
  • Namespace: monitoring
  • Kind: Pod
  • Container: loki-canary

Event Details

  • Type: Warning

  • Reason: Unhealthy

  • Message: Readiness probe failed: Get "http://<MY_PRIVATE_IP>:3500/metrics
    ":context deadline exceeded (Client.Timeout exceeded while awaiting headers)

  • Source: kubelet <MY_CLUSTER_AND_NODE_ID>

Timestamps

  • First Seen: 2025-09-24T16:35:54-03:00 (16h ago)
  • Last Seen: 2025-09-25T09:14:34-03:00 (9m 11s ago)
  • Created: 2025-09-25T09:14:34-03:00
  • Count: 15 occurrences

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions