This simple example demonstrates monitoring prometheus sources with the OpenTelemetry collector, using the prometheus receiver and sending the data to New Relic via OTLP. A simple prometheus data generator is configured to generate dummy metrics scraped by prometheus receiver.
- You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. Docker desktop includes a standalone Kubernetes server and client which is useful for local testing.
- A New Relic account
- A New Relic license key
-
Update the
NEW_RELIC_API_KEY
value in secrets.yaml to your New Relic license key.# ...omitted for brevity stringData: # New Relic API key to authenticate the export requests. # docs: https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/#license-key NEW_RELIC_API_KEY: <INSERT_API_KEY>
-
Note, be careful to avoid inadvertent secret sharing when modifying
secrets.yaml
. To ignore changes to this file from git, rungit update-index --skip-worktree k8s/secrets.yaml
. -
If your account is based in the EU, update the
NEW_RELIC_OTLP_ENDPOINT
value in collector.yaml the endpoint to: https://otlp.eu01.nr-data.net
# ...omitted for brevity env: # The default US endpoint is set here. You can change the endpoint and port based on your requirements if needed. # docs: https://docs.newrelic.com/docs/more-integrations/open-source-telemetry-integrations/opentelemetry/best-practices/opentelemetry-otlp/#configure-endpoint-port-protocol - name: NEW_RELIC_OTLP_ENDPOINT value: https://otlp.eu01.nr-data.net
-
-
Run the application with the following command.
kubectl apply -f k8s/
- When finished, cleanup resources with the following command. This is also useful to reset if modifying configuration.
kubectl delete -f k8s/
To review your statsd data in New Relic, navigate to "New Relic -> Query Your Data". To list the metrics reported, query for:
FROM Metric SELECT uniques(metricName) WHERE otel.library.name = 'otelcol/prometheusreceiver' LIMIT MAX
See get started with querying for additional details on querying data in New Relic.
This example monitors a simple prometheus data generator instance defined in prometheus-data-generator.yaml. To use in production, you'll need to modify the .receivers.prometheus.config.scrape_configs
value in collector.yaml ConfigMap to point at your prometheus sources.
The prometheus receiver includes service.name
and service.instance.id
resource attributes derived from job name and target configured in .receivers.prometheus.config.scraep_configs
. As documented here, New Relic considers any data with service.name
as a service despite the fact that not all prometheus data sources are services. As a result, you can find a prometheus_data_generator
entity under "New Relic -> All Entities -> Services - OpenTelemetry", although the panels will not contain data because the scraped metrics do not represent APM data.