Skip to content

Latest commit

 

History

History

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 

Monitoring Prometheus with OpenTelemetry Collector

This simple example demonstrates monitoring prometheus sources with the OpenTelemetry collector, using the prometheus receiver and sending the data to New Relic via OTLP. A simple prometheus data generator is configured to generate dummy metrics scraped by prometheus receiver.

Requirements

Running the example

  1. Update the NEW_RELIC_API_KEY value in secrets.yaml to your New Relic license key.

    # ...omitted for brevity
    stringData:
      # New Relic API key to authenticate the export requests.
      # docs: https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/#license-key
      NEW_RELIC_API_KEY: <INSERT_API_KEY>
    • Note, be careful to avoid inadvertent secret sharing when modifying secrets.yaml. To ignore changes to this file from git, run git update-index --skip-worktree k8s/secrets.yaml.

    • If your account is based in the EU, update the NEW_RELIC_OTLP_ENDPOINT value in collector.yaml the endpoint to: https://otlp.eu01.nr-data.net

    # ...omitted for brevity
    env:
     # The default US endpoint is set here. You can change the endpoint and port based on your requirements if needed.
     # docs: https://docs.newrelic.com/docs/more-integrations/open-source-telemetry-integrations/opentelemetry/best-practices/opentelemetry-otlp/#configure-endpoint-port-protocol
     - name: NEW_RELIC_OTLP_ENDPOINT
       value: https://otlp.eu01.nr-data.net
  2. Run the application with the following command.

    kubectl apply -f k8s/
    • When finished, cleanup resources with the following command. This is also useful to reset if modifying configuration.
    kubectl delete -f k8s/

Viewing your data

To review your statsd data in New Relic, navigate to "New Relic -> Query Your Data". To list the metrics reported, query for:

FROM Metric SELECT uniques(metricName) WHERE otel.library.name = 'otelcol/prometheusreceiver' LIMIT MAX

See get started with querying for additional details on querying data in New Relic.

Additional notes

This example monitors a simple prometheus data generator instance defined in prometheus-data-generator.yaml. To use in production, you'll need to modify the .receivers.prometheus.config.scrape_configs value in collector.yaml ConfigMap to point at your prometheus sources.

The prometheus receiver includes service.name and service.instance.id resource attributes derived from job name and target configured in .receivers.prometheus.config.scraep_configs. As documented here, New Relic considers any data with service.name as a service despite the fact that not all prometheus data sources are services. As a result, you can find a prometheus_data_generator entity under "New Relic -> All Entities -> Services - OpenTelemetry", although the panels will not contain data because the scraped metrics do not represent APM data.