[WIP] feat: Armada OpenTelemetry Proof of Concept

[dejanzele]

What type of PR is this?

Enhancement

What this PR does / why we need it:

Add support for OpenTelemetry in Armada

How to test?

SigNoz is an open-source alternative to DataDog.

1. Run SigNoz by cloning https://github.com/SigNoz/signoz, navigate to deploy/docker and run docker compose up -d
2. As SigNoz exposes the UI on localhost:8080, edit the scheduler in _local/scheduler/config.yaml to run the HTTP server on a different port

# _local/scheduler/config.yaml
http:
  port: 8085

3. Run the Armada dependencies described in Local Development with Goreman
4. Apply migrations by running ./scripts/localdev-init.sh
5. Use the Fake Executor profile described in Local Development with Fake Executor

For generating load, you can use the following submit-jobs.sh script from https://gist.github.com/dejanzele/f6580f0249517ccaf40530d62909f490, and be sure to copy the benchmark-200-jobs.yaml file in examples/benchmark-200-jobs.yaml.

Usage:

submit-jobs.sh -c <count> -j <job-set>