Red Hat Observability Service

This project holds the configuration files for our internal Red Hat Observability Service based on Observatorium.

Requirements

• jsonnet-bundler

Installing jsonnet dependencies

To install all dependencies:

# Installs pinned versions from `jsonnetfile.lock.json` file.

jb install

To update a dependency, normally the process would be:

# Updates `kube-thanos` to master and sets the new hash in `jsonnetfile.lock.json`.
jb update https://github.com/thanos-io/kube-thanos/jsonnet/kube-thanos@main

# Update all dependancies to master and sets the new hashes in `jsonnetfile.lock.json`.
jb update

Currently jb update does not seem to work properly - see jsonnet-bundler/jsonnet-bundler#142. As a workaround, try following this workflow: https://github.com/openshift/cluster-monitoring-operator/blob/master/Documentation/development.md#updating-individual-vendored-jsonnet-code

Grafana dashboards

All dashboards are generated in resources/observability/grafana (legacy: manifests/production/grafana) with:

make grafana

Staging: Update the commit hash ref in https://gitlab.cee.redhat.com/service/app-interface/blob/master/data/services/observability/cicd/saas/saas-grafana.yaml

Production: Update the commit hash ref in https://gitlab.cee.redhat.com/service/app-interface/blob/master/data/services/observability/cicd/saas/saas-grafana.yaml

Prometheus Rules

Use synchronize.sh to create a MR against app-interface to update dashboards.

Components - Deployments, ServiceMonitors, ConfigMaps etc...

All components manifests are generated in resources/services (legacy: manifests/production/) with:

make manifests

Staging: update the commit hash ref in https://gitlab.cee.redhat.com/service/app-interface/blob/master/data/services/telemeter/cicd/saas.yaml

Production: update the commit hash ref in https://gitlab.cee.redhat.com/service/app-interface/blob/master/data/services/telemeter/cicd/saas.yaml

CI Jobs

Jobs runs are posted in:

#sd-app-sre-info for grafana dashboards

and

#team-monitoring-info for everything else.

Troubleshooting

1. Enable port forwarding for a user - example
2. Add a pod name to the allowed list for port forwarding - example