This module defines the resources required for monitoring the state of system. This includes:
- Grafana for visualization and dashboards.
- Prometheus for metric monitoring.
- Loki for log storage and disovery.
- Promtail for log scraping.
- First, initialize terraform:
terraform init- Validate that the terraform resources are valid. If your terraform is valid the validate command will respond with "Success! The configuration is valid."
terraform validate- Run a terraform plan. The terraform plan will give you a summary of all the changes terraform will perform prior to deploying any change.
terraform plan- Deploy the changes by applying the terraform plan. You will be asked to confirm the changes and must respond with "yes".
terraform applyInstructions for installing individual monitoring services can be found in their specific README files.
Charts and visualizations for viewing system usage and performance are available in the Grafana Dashboard. For developer access to the Grafana dashboard, first enable port forwarding:
or run this on the command line:
kubectl port-forward --namespace monitoring svc/kube-prometheus-stack-grafana 8080:3009This will allow you to access the Grafana dashboard at http://localhost:8080
If you don't already have your Grafana credentials, they can be obtained with:
# Username
kubectl get secret kube-prometheus-stack-grafana -n monitoring -o jsonpath="{.data.admin-user}" | base64 --decode
# Password
kubectl get secret kube-prometheus-stack-grafana -n monitoring -o jsonpath="{.data.admin-password}" | base64 --decode
And use those credentials to login at: localhost:{port_from_above}
Loki and Prometheus have automatically been added as data sources to Grafana.
The Prometheus server UI can be accessed by port-forwarding to a localhost port:
kubectl port-forward --namespace monitoring svc/kube-prometheus-stack-prometheus 9090:9090You could also port-forward the kube-prometheus-stack-prometheus service using the Rancher Desktop UI. This will allow you to access the Prometheus dashboard at http://localhost:9090
There is no login needed. Observe/monitor the cluster status by clicking on 'Status' on the top ribbon and exploring the various options on that tab.
To visualize basic Loki monitoring logs:
In Grafana click "Explore" on the left-side menu. For Label Filters enter "namespace = monitoring". Click "Run Query" and you should see logs from the Loki instance:
The promtail agent will be installed as a DaemonSet and configured with the default settings to scrape logs from any pod in the cluster. The logs will be sent to and stored in Loki.