Tracing

Tracing allows to collect fine grained details about policy evaluations. It can be a useful tool for debugging issues inside of your Kubewarden deployment and policies.

We will use Jaeger -- used to receive, store and visualize trace events.

Prerequisites

Cert-Manager and OpenTelemetry are required. Follow these instructions to install cert-manager. Once that is complete you will need to install the OpenTelemetry operator

Install Jaeger

Adapted from here

We first need to add the helm repository that contains the Jaeger Operator charts.

https://jaegertracing.github.io/helm-charts

NOTE: Due to this issue - the jaeger-operator must be installed with version 2.28.0

You can add them with the UI as we did in previous steps or with kubectl.

Install with kubectl:

helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
helm install --namespace jaeger --create-namespace jaeger-operator jaegertracing/jaeger-operator

Once the Jaeger Operator is installed you need to create a Jaeger resource, we will use the default AllInOne strategy:

kubectl apply -f - <<EOF
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: all-in-one
  namespace: jaeger
spec:
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
EOF

This strategy is meant to be used only for development, testing, and demo purposes

Update kubewarden-controller with Jaeger endpoint

As we did in a previous step, you will need to edit your kubewarden-controller resource to add the Jaeger endpoint all-in-one-collector.jaeger.svc.cluster.local:14250.

For instance:

policyServer:
  telemetry:
    enabled: True
    tracing:
      jaeger:
        endpoint: "all-in-one-collector.jaeger.svc.cluster.local:14250"

Note: Regardless if your cluster is downstream or "local" the endpoint needs cluster.local to connect properly.

The all-in-one-collector is the service we installed under the jaeger namespace.

Update OpenTelemetryCollectors

The last step is to update our OpenTelemetryCollector sidecar with the tls property to insecure: true.

1. Navigate to More Resources -> opentelemtry.io -> OpenTelemetryCollectors
2. Choose to edit the kubewarden sidecar
3. Add the tls.insecure: true property to exporters.jaeger

For instance:

exporters:
  jaeger:
    endpoint: all-in-one-collector.jaeger.svc:14250
    tls:
      insecure: true

4. Redeploy your Jaeger and OpenTelemetry resources to apply the new configuration

You should now be able to view any failed requests for any given policy's detail page. You can also view them from the Jaeger UI which will be at this endpoint: <cluster-ip>/api/v1/namespaces/jaeger/services/http:all-in-one-query:16686/proxy/ (granted you installed Jaeger into a namespace titled jaeger).