OpenTelemetry Acceptance Tests (OATs)

OpenTelemetry Acceptance Tests (OATs), or OATs for short, is a test framework for OpenTelemetry.

• Declarative tests written in YAML
• Supported signals: traces, logs, metrics
• Full round-trip testing: from the application to the observability stack
  • Data is stored in the LGTM stack (Loki, Grafana, Tempo, Prometheus, OpenTelemetry Collector)
  • Data is queried using LogQL, PromQL, and TraceQL
  • All data is sent to the observability stack via OTLP - so OATs can also be used with other observability stacks
• End-to-end testing
  • Docker Compose with the docker-otel-lgtm image
  • Kubernetes with the docker-otel-lgtm and k3d

Under the hood, OATs uses Ginkgo and Gomega to run the tests.

Getting Started

You can use the test cases in prom_client_java as a reference. The GitHub action uses a script to run the tests.

1. Create a folder oats-tests for the following files
2. Create Dockerfile to build the application you want to test

   FROM eclipse-temurin:21-jre
   COPY target/example-exporter-opentelemetry.jar ./app.jar
   ENTRYPOINT [ "java", "-jar", "./app.jar" ]

3. Create docker-compose.yaml to start the application and any dependencies

   version: '3.4'
   
   services:
     java:
       build:
         dockerfile: Dockerfile
       environment:
         OTEL_SERVICE_NAME: "rolldice"
         OTEL_EXPORTER_OTLP_ENDPOINT: http://lgtm:4318
         OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
         OTEL_METRIC_EXPORT_INTERVAL: "5000"  # so we don't have to wait 60s for metrics

4. Create oats.yaml with the test cases

   # OATs is an acceptance testing framework for OpenTelemetry - https://github.com/grafana/oats
   docker-compose:
     files:
       - ./docker-compose.yaml
   expected:
     metrics:
       - promql: 'uptime_seconds_total{}'
         value: '>= 0'

5. cd /path/to/oats/yaml
6. go install github.com/onsi/ginkgo/v2/ginkgo
7. TESTCASE_BASE_PATH=/path/to/oats-tests ginkgo -v

Test Case Syntax

You can use any file name that matches oats*.yaml (e.g. oats-test.yaml), that doesn't end in -template.yaml. oats-template.yaml is reserved for template files, which are used in the include section.

The syntax is a bit similar to https://github.com/kubeshop/tracetest

This is an example:

include:
  - ../oats-template.yaml
docker-compose:
  file: ../docker-compose.yaml
input:
  - url: http://localhost:8080/stock
interval: 500ms # interval between requests to the input URL
expected:
  traces:
    - traceql: '{ name =~ "SELECT .*product"}'
      spans:
        - name: 'regex:SELECT .*'
          attributes:
            db.system: h2
  logs:
    - logql: '{exporter = "OTLP"}'
      contains: 
        - 'hello LGTM'
  metrics:
    - promql: 'db_client_connections_max{pool_name="HikariPool-1"}'
      value: "== 10"
  dashboards: # Grafana dashboards
    - path: ../jdbc-dashboard.json
      panels:
        - title: Connection pool waiting requests
          value: "== 0"
        - title: Connection pool utilization
          value: "> 0"

Query traces

Each entry in the traces array is a test case for traces.

expected:
  traces:
    - traceql: '{ name =~ "SELECT .*product"}'
      spans:
        - name: 'regex:SELECT .*' # regex match
          attributes:
            db.system: h2
          allow-duplicates: true # allow multiple spans with the same attributes

Query logs

Each entry in the logs array is a test case for logs.

expected:
  logs:
    - logql: '{service_name="rolldice"} |~ `Anonymous player is rolling the dice.*`'
      equals: 'Anonymous player is rolling the dice'
      attributes:
        service_name: rolldice
      attribute-regexp:  
        container_id: ".*"
      no-extra-attributes: true # fail if there are extra attributes
    - logql: '{service_name="rolldice"} |~ `Anonymous player is rolling the dice.*`'
      regexp: 'Anonymous player is .*'

Query metrics

expected:
  metrics:
    - promql: 'db_client_connections_max{pool_name="HikariPool-1"}'
      value: "== 10"
  dashboards: # Useful if you populate Grafana dashboards from JSON
    - path: ../jdbc-dashboard.json 
      panels:
        - title: Connection pool waiting requests
          value: "== 0"
        - title: Connection pool utilization
          value: "> 0"

Docker Compose

Describes the docker-compose file(s) to use for the test. The files typically define the instrumented application you want to test and optionally some dependencies, e.g. a database server to send requests to. You don't need (and shouldn't have) to define the observability stack (e.g. Prometheus, Grafana, etc.), because this is provided by the test framework (and may test different versions of the observability stack, e.g. OTel Collector and Grafana Alloy).

This docker-compose file is relative to the oats.yaml file.

Kubernetes

A local Kubernetes cluster can be used to test the application in a Kubernetes environment rather than in docker-compose. This is useful to test the application in a more realistic environment - and when you want to test Kubernetes specific features.

Describes the Kubernetes manifest(s) to use for the test.

kubernetes:
  dir: k8s
  app-service: dice
  app-docker-file: Dockerfile
  app-docker-context: ..
  app-docker-tag: dice:1.1-SNAPSHOT
  app-docker-port: 8080

Debugging

If you want to run a single test case, you can use the --focus option:

TESTCASE_BASE_PATH=/path/to/project ginkgo -v --focus="jdbc"

You can increase the timeout, which is useful if you want to inspect the telemetry data manually in Grafana at http://localhost:3000

TESTCASE_TIMEOUT=1h TESTCASE_BASE_PATH=/path/to/project ginkgo -v

You can keep the container running without executing the tests - which is useful to debug in Grafana manually:

TESTCASE_MANUAL_DEBUG=true TESTCASE_BASE_PATH=/path/to/project ginkgo -v