Skip to content

Latest commit

 

History

History
93 lines (56 loc) · 3.09 KB

README.md

File metadata and controls

93 lines (56 loc) · 3.09 KB
Raw

k3d Examples

Agent Environment

The k3d example uses k3d and tanka to produce a Kubernetes environment that implements a full Grafana Agent environment for testing.

Requirements

Getting Started

Build latest agent images with make agent-image agentctl-image in the project root directory if there are local changes to test.

Run the following to create your cluster:

# Create a new k3d cluster
./scripts/create.bash

# Import images into k3d if they are not available on docker hub
k3d image import -c agent-k3d grafana/agent:latest
k3d image import -c agent-k3d grafana/agentctl:latest

tk apply ./environment

# Navigate to grafana.k3d.localhost:30080 in your browser to view dashboards

# Delete the k3d cluster when you're done with it
k3d cluster delete agent-k3d

Smoke Test Environment

The smoke test environment is used to validate samples end to end.

Running

Smoke Test environment is invoked via /scripts/smoke_test.bash

This tool will spin up cluster of Grafana Agent, Cortex, Avalanche and Crow instances. The tool will then periodically kill instances and check for any failed alerts. At the end of the duration (default 3h) it will end the testing.

What to look for?

These alerts are viewable here.

Prometheus alerts are triggered:

  • If any Crow instances are not running or Crow samples are not being propagated correctly.
  • If any Grafana Agents are not running or Grafana Agent limits are outside their norm.

NOTE: The alerts might be in pending until the system settles down.

An alert firing will look similar to the below.

If at the end of the test any issues are found they will look similar to the below.

How to trigger an alert?

Changing the avalanche setting for label_count to 1000, located here. This will ensure the GrafanaAgentMemHigh alert exceeds the limit.

Architecture

By default, a k3d cluster will be created running the following instances

  • agent-single - single instance
  • agent-cluster - 3 Grafana Agents in clustered configuration
  • crow-cluster - serves the agent cluster
  • crow-single - serves the single agent
  • cortex
  • avalanche - selection of avalanche instances serving traffic

Crow instance will check to see if the metrics that were scraped shows up in the prometheus endpoint and then will emit metrics on the success of those metrics. This success/failure result will trigger an alert if it is incorrect.

Flow

Avalanche

Avalanche is used to add some additional load on the system and general testing.