Skip to content

Commit d2234ca

Browse files
jszwedkoclaude
andauthored
Add OpenMetrics-based Datadog Agent integration (#25795)
* Add OpenMetrics-based Datadog Agent integration Add a `datadog_agent` integration that scrapes the Agent's `/telemetry` OpenMetrics endpoint, collecting the telemetry analogous to what the expvar-based Agent Metrics integration collected. Mark Agent Metrics as deprecated in favor of it, and move the `datadog-agent` app_id to the new integration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Add changelog entry Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fix changelog entry punctuation Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Drop telemetry.enabled requirement from docs The Agent always serves /telemetry; telemetry.enabled only enables additional telemetry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Apply Docs review suggestions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep agent_metrics app_id; use datadog-agent-openmetrics for the new integration app_ids can't be reused, so agent_metrics keeps datadog-agent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Add migration guide from Agent Metrics Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Apply Docs style conventions to the migration guide Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Apply Docs review suggestions to the migration guide Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 027825a commit d2234ca

29 files changed

Lines changed: 1700 additions & 0 deletions

‎.github/workflows/config/labeler.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -357,6 +357,10 @@ integration/crio:
357357
- changed-files:
358358
- any-glob-to-any-file:
359359
- crio/**/*
360+
integration/datadog_agent:
361+
- changed-files:
362+
- any-glob-to-any-file:
363+
- datadog_agent/**/*
360364
integration/datadog_checks_tests_helper:
361365
- changed-files:
362366
- any-glob-to-any-file:

‎.github/workflows/test-all.yml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -965,6 +965,26 @@ jobs:
965965
minimum-base-package: ${{ inputs.minimum-base-package }}
966966
pytest-args: ${{ inputs.pytest-args }}
967967
secrets: inherit
968+
j439d33b:
969+
uses: ./.github/workflows/test-target.yml
970+
with:
971+
job-name: Datadog Agent
972+
target: datadog_agent
973+
platform: linux
974+
runner: '["ubuntu-22.04"]'
975+
repo: "${{ inputs.repo }}"
976+
context: ${{ inputs.context }}
977+
python-version: "${{ inputs.python-version }}"
978+
latest: ${{ inputs.latest }}
979+
agent-image: "${{ inputs.agent-image }}"
980+
agent-image-py2: "${{ inputs.agent-image-py2 }}"
981+
agent-image-windows: "${{ inputs.agent-image-windows }}"
982+
agent-image-windows-py2: "${{ inputs.agent-image-windows-py2 }}"
983+
test-py2: ${{ inputs.test-py2 }}
984+
test-py3: ${{ inputs.test-py3 }}
985+
minimum-base-package: ${{ inputs.minimum-base-package }}
986+
pytest-args: ${{ inputs.pytest-args }}
987+
secrets: inherit
968988
j2d21154:
969989
uses: ./.github/workflows/test-target.yml
970990
with:

‎agent_metrics/README.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
# Agent Metrics Integration
22

3+
<div class="alert alert-warning">
4+
This integration is deprecated. Use the <a href="https://docs.datadoghq.com/integrations/datadog_agent/">Datadog Agent</a> integration instead. The Datadog Agent integration collects the Agent's internal telemetry from its OpenMetrics endpoint with typed metrics, more granular tags, and a larger set of available metrics.
5+
</div>
6+
37
## Overview
48

59
Get internal metrics from the Datadog Agent to create visualizations and monitors in Datadog.

‎code-coverage.datadog.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -206,6 +206,9 @@ services:
206206
- id: crio
207207
paths:
208208
- crio/datadog_checks/crio/
209+
- id: datadog_agent
210+
paths:
211+
- datadog_agent/datadog_checks/datadog_agent/
209212
- id: datadog_checks_base
210213
paths:
211214
- datadog_checks_base/datadog_checks/base/

‎datadog_agent/CHANGELOG.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
# CHANGELOG - Datadog Agent
2+
3+
<!-- towncrier release notes start -->
4+

‎datadog_agent/README.md‎

Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
# Agent Check: Datadog Agent
2+
3+
## Overview
4+
5+
This check monitors the [Datadog Agent][1] itself by scraping the internal telemetry it exposes in the OpenMetrics format. Use it to track the health and throughput of the Agent's core components, such as the forwarder, DogStatsD, the aggregator, the check scheduler, and the Logs Agent.
6+
7+
This integration replaces the deprecated [Agent Metrics][2] integration, which collects similar data through the `go_expvar` check. Compared to the expvar endpoint, the telemetry endpoint provides typed metrics, more granular tags, and a larger set of available metrics.
8+
9+
**Note:** The Agent's internal telemetry might change between minor Agent versions. Such changes might not be mentioned in the Agent's changelog.
10+
11+
## Setup
12+
13+
### Installation
14+
15+
The Datadog Agent check is included in the [Datadog Agent][3] package. No additional installation is needed.
16+
17+
Some default metrics require Agent v7.86.0 or later. Earlier Agent versions report only part of the default set.
18+
19+
### Configuration
20+
21+
The Agent always serves its telemetry at `/telemetry` on the port set by `expvar_port` in [`datadog.yaml`][4], which defaults to `5000`. This check does not require `telemetry.enabled: true`. That setting only enables additional telemetry.
22+
23+
1. Edit the `datadog_agent.d/conf.yaml` file in the `conf.d/` folder at the root of your [Agent's configuration directory][4]. See the [sample datadog_agent.d/conf.yaml][5] for all available configuration options. For example:
24+
25+
```yaml
26+
init_config:
27+
28+
instances:
29+
- openmetrics_endpoint: http://localhost:5000/telemetry
30+
```
31+
32+
To collect telemetry beyond the default set, list the metrics to add with the `extra_metrics` option. You can view all available metrics by querying the `/telemetry` endpoint directly.
33+
34+
2. [Restart the Agent][6].
35+
36+
### Validation
37+
38+
[Run the Agent's status subcommand][7] and look for `datadog_agent` under the Checks section.
39+
40+
### Migrating from Agent Metrics
41+
42+
The Datadog Agent integration collects similar data to the [Agent Metrics][2] integration, but most metric names, types, and tags differ. You must update any dashboards and monitors that query metrics from the Agent Metrics integration.
43+
44+
1. Configure the Datadog Agent integration as described in [Configuration](#configuration).
45+
2. Optionally, keep `go_expvar.d/agent_stats.yaml` in place while you update your dashboards and monitors, so that you can compare the old and new metrics. Three metrics have the same name in both integrations: `datadog.agent.aggregator.dogstatsd_contexts`, `datadog.agent.forwarder.transactions.retry_queue_size`, and `datadog.agent.scheduler.checks_entered`. While both checks run, each check submits these metrics.
46+
3. Update your dashboards and monitors using the [metric mapping](#metric-mapping).
47+
4. Remove `go_expvar.d/agent_stats.yaml` and [restart the Agent][6].
48+
49+
#### Behavior changes
50+
51+
- **Counters are submitted as counts.** Agent Metrics submitted counters as per-second rates. The Datadog Agent integration submits them as counts with a `.count` suffix. For example, to get the equivalent of `datadog.agent.forwarder.transactions.success`, query `datadog.agent.forwarder.transactions.success.count` with the `.as_rate()` function.
52+
- **Related metrics are combined and distinguished by tags.** For example, `datadog.agent.dogstatsd.processed.count`, tagged by `message_type` and `state`, replaces the separate DogStatsD packet and parse error metrics.
53+
- **Some metrics are now under the `datadog.agent.` namespace.** Agent Metrics submitted `scheduler.queues.size`, `scheduler.queues.interval`, `batch_strategy.idle_time`, and `batch_strategy.in_use_time` without a namespace.
54+
- **Go runtime metrics are not collected by default.** Agent Metrics also collected Go memory statistics, such as `datadog.agent.memstats.heap_alloc`. The telemetry endpoint exposes them as `go_memstats_*` and `go_gc_*` metrics. To collect them, use the `extra_metrics` option. For example:
55+
56+
```yaml
57+
instances:
58+
- openmetrics_endpoint: http://localhost:5000/telemetry
59+
extra_metrics:
60+
- go_memstats_heap_alloc_bytes: go.memstats.heap_alloc
61+
- go_memstats_heap_inuse_bytes: go.memstats.heap_inuse
62+
```
63+
64+
#### Metric mapping
65+
66+
Unless noted otherwise, all metric names in the table are prefixed with `datadog.agent.`.
67+
68+
| Agent Metrics metric | Datadog Agent metric |
69+
| --- | --- |
70+
| `forwarder.transactions.success` | `forwarder.transactions.success.count` |
71+
| `forwarder.transactions.errors` | `forwarder.transactions.errors.count` |
72+
| `forwarder.transactions.errors_by_type.dns_errors` | `forwarder.transactions.errors.count` with `error_type:dns_lookup_failure` |
73+
| `forwarder.transactions.errors_by_type.tls_errors` | `forwarder.transactions.errors.count` with `error_type:tls_handshake_failure` |
74+
| `forwarder.transactions.errors_by_type.connection_errors` | `forwarder.transactions.errors.count` with `error_type:connection_failure` |
75+
| `forwarder.transactions.errors_by_type.wrote_request_errors` | `forwarder.transactions.errors.count` with `error_type:writing_failure` |
76+
| `forwarder.transactions.errors_by_type.sent_request_errors` | `forwarder.transactions.errors.count` with `error_type:invalid_request` |
77+
| `forwarder.transactions.http_errors` | `forwarder.transactions.http_errors.count`, tagged by `code` |
78+
| `forwarder.transactions.retry_queue_size` | `forwarder.transactions.retry_queue_size`, tagged by `domain` |
79+
| `forwarder.transactions.dropped_on_input` | No longer reported by the Agent. See `forwarder.transactions.dropped.count`. |
80+
| `dogstatsd.metric_packets` | `dogstatsd.processed.count` with `message_type:metrics` and `state:ok` |
81+
| `dogstatsd.metric_parse_errors` | `dogstatsd.processed.count` with `message_type:metrics` and `state:error` |
82+
| `dogstatsd.event_packets` | `dogstatsd.processed.count` with `message_type:events` and `state:ok` |
83+
| `dogstatsd.event_parse_errors` | `dogstatsd.processed.count` with `message_type:events` and `state:error` |
84+
| `dogstatsd.service_check_packets` | `dogstatsd.processed.count` with `message_type:service_checks` and `state:ok` |
85+
| `dogstatsd.service_check_parse_errors` | `dogstatsd.processed.count` with `message_type:service_checks` and `state:error` |
86+
| `dogstatsd_udp.packets` | `dogstatsd_udp.packets.count` with `state:ok` |
87+
| `dogstatsd_udp.packet_reading_errors` | `dogstatsd_udp.packets.count` with `state:error` |
88+
| `dogstatsd_uds.packets` | `dogstatsd_uds.packets.count` with `state:ok` |
89+
| `dogstatsd_uds.packet_reading_errors` | `dogstatsd_uds.packets.count` with `state:error` |
90+
| `dogstatsd_uds.origin_detection_errors` | `dogstatsd_uds.origin_detection_errors.count` |
91+
| `aggregator.dogstatsd_contexts` | `aggregator.dogstatsd_contexts`, tagged by `shard` |
92+
| `aggregator.series_flushed` | `aggregator.flush.count` with `data_type:series` |
93+
| `aggregator.service_check_flushed` | `aggregator.flush.count` with `data_type:service_checks` |
94+
| `aggregator.events_flushed` | `aggregator.flush.count` with `data_type:events` |
95+
| `aggregator.checks_metric_sample` | `aggregator.processed.count` with `data_type:metrics` |
96+
| `aggregator.dogstatsd_metric_sample` | `aggregator.processed.count` with `data_type:dogstatsd_metrics` |
97+
| `aggregator.service_check` | `aggregator.processed.count` with `data_type:service_checks` |
98+
| `aggregator.event` | `aggregator.processed.count` with `data_type:events` |
99+
| `aggregator.number_of_flush` | `aggregator.number_of_flush.count` |
100+
| `aggregator.flush.main_flush_time.last_flush` | `aggregator.flush_time` with `flush_type:main` |
101+
| `aggregator.flush.checks_metric_sample_flush_time.last_flush` | `aggregator.flush_time` with `flush_type:checks_metric_sample` |
102+
| `aggregator.flush.service_check_flush_time.last_flush` | `aggregator.flush_time` with `flush_type:service_check` |
103+
| `aggregator.flush.event_flush_time.last_flush` | `aggregator.flush_time` with `flush_type:event` |
104+
| `aggregator.flush.metric_sketch_flush_time.last_flush` | `aggregator.flush_time` with `flush_type:metric_sketch` |
105+
| `aggregator.flush_count.series.last_flush` | `aggregator.flush_count` with `flush_type:series` |
106+
| `aggregator.flush_count.service_checks.last_flush` | `aggregator.flush_count` with `flush_type:service_checks` |
107+
| `aggregator.flush_count.events.last_flush` | `aggregator.flush_count` with `flush_type:events` |
108+
| `aggregator.flush_count.sketches.last_flush` | `aggregator.flush_count` with `flush_type:sketches` |
109+
| `aggregator.hostname_update` | No longer reported by the Agent. |
110+
| `scheduler.checks_entered` | `scheduler.checks_entered`, tagged by `check_name` |
111+
| `scheduler.queues_count` | `scheduler.queues_count.count` |
112+
| `scheduler.queues.size` (no namespace) | `scheduler.queues.size`, tagged by `interval` |
113+
| `scheduler.queues.interval` (no namespace) | No equivalent. Use the `interval` tag of `scheduler.queues.size`. |
114+
| `splitter.*` | No longer reported by the Agent. |
115+
| `logs_agent.logs_decoded` | `logs_agent.logs_decoded.count` |
116+
| `logs_agent.logs_processed` | `logs_agent.logs_processed.count` |
117+
| `logs_agent.logs_sent` | `logs_agent.logs_sent.count` |
118+
| `logs_agent.bytes_sent` | `logs_agent.bytes_sent.count` |
119+
| `logs_agent.encoded_bytes_sent` | `logs_agent.encoded_bytes_sent.count` |
120+
| `logs_agent.destination_errors` | `logs_agent.destination_errors.count` |
121+
| `logs_agent.is_running` | Not collected. |
122+
| `batch_strategy.idle_time` (no namespace) | `batch_strategy.idle_time.count`, tagged by `sender` (for example, `sender:logs_0_reliable_0` instead of `sender:0`) |
123+
| `batch_strategy.in_use_time` (no namespace) | `batch_strategy.in_use_time.count`, tagged by `sender` |
124+
| `memstats.*` | Not collected by default. See the `extra_metrics` example in [Behavior changes](#behavior-changes). |
125+
126+
## Data Collected
127+
128+
### Metrics
129+
130+
See [metadata.csv][8] for a list of metrics provided by this integration.
131+
132+
### Events
133+
134+
The Datadog Agent integration does not include any events.
135+
136+
### Service Checks
137+
138+
See [service_checks.json][9] for a list of service checks provided by this integration.
139+
140+
## Troubleshooting
141+
142+
Need help? Contact [Datadog support][10].
143+
144+
[1]: https://docs.datadoghq.com/agent/
145+
[2]: https://docs.datadoghq.com/integrations/agent_metrics/
146+
[3]: /account/settings/agent/latest
147+
[4]: https://docs.datadoghq.com/agent/configuration/agent-configuration-files/
148+
[5]: https://github.com/DataDog/integrations-core/blob/master/datadog_agent/datadog_checks/datadog_agent/data/conf.yaml.example
149+
[6]: https://docs.datadoghq.com/agent/configuration/agent-commands/#start-stop-and-restart-the-agent
150+
[7]: https://docs.datadoghq.com/agent/configuration/agent-commands/#agent-status-and-information
151+
[8]: https://github.com/DataDog/integrations-core/blob/master/datadog_agent/metadata.csv
152+
[9]: https://github.com/DataDog/integrations-core/blob/master/datadog_agent/assets/service_checks.json
153+
[10]: https://docs.datadoghq.com/help/
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
name: Datadog Agent
2+
files:
3+
- name: datadog_agent.yaml
4+
options:
5+
- template: init_config
6+
options:
7+
- template: init_config/openmetrics
8+
- template: instances
9+
options:
10+
- template: instances/openmetrics
11+
overrides:
12+
openmetrics_endpoint.value.example: http://localhost:5000/telemetry
13+
openmetrics_endpoint.description: |
14+
The Datadog Agent's OpenMetrics telemetry endpoint. It is served on `expvar_port`, which
15+
defaults to 5000.
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
provides:
2+
- id: datadog-agent-metrics
3+
always_on: true
4+
data_type: metrics
5+
direction: inbound
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
[
2+
{
3+
"agent_version": "7.86.0",
4+
"integration": "Datadog Agent",
5+
"check": "datadog.agent.openmetrics.health",
6+
"statuses": [
7+
"ok",
8+
"critical"
9+
],
10+
"groups": [
11+
"host",
12+
"endpoint"
13+
],
14+
"name": "Datadog Agent OpenMetrics endpoint health",
15+
"description": "Returns `CRITICAL` if the check is unable to connect to the Datadog Agent's telemetry endpoint, otherwise returns `OK`."
16+
}
17+
]
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Initial Release.

0 commit comments

Comments
 (0)