|
| 1 | +# Agent Check: Datadog Agent |
| 2 | + |
| 3 | +## Overview |
| 4 | + |
| 5 | +This check monitors the [Datadog Agent][1] itself by scraping the internal telemetry it exposes in the OpenMetrics format. Use it to track the health and throughput of the Agent's core components, such as the forwarder, DogStatsD, the aggregator, the check scheduler, and the Logs Agent. |
| 6 | + |
| 7 | +This integration replaces the deprecated [Agent Metrics][2] integration, which collects similar data through the `go_expvar` check. Compared to the expvar endpoint, the telemetry endpoint provides typed metrics, more granular tags, and a larger set of available metrics. |
| 8 | + |
| 9 | +**Note:** The Agent's internal telemetry might change between minor Agent versions. Such changes might not be mentioned in the Agent's changelog. |
| 10 | + |
| 11 | +## Setup |
| 12 | + |
| 13 | +### Installation |
| 14 | + |
| 15 | +The Datadog Agent check is included in the [Datadog Agent][3] package. No additional installation is needed. |
| 16 | + |
| 17 | +Some default metrics require Agent v7.86.0 or later. Earlier Agent versions report only part of the default set. |
| 18 | + |
| 19 | +### Configuration |
| 20 | + |
| 21 | +The Agent always serves its telemetry at `/telemetry` on the port set by `expvar_port` in [`datadog.yaml`][4], which defaults to `5000`. This check does not require `telemetry.enabled: true`. That setting only enables additional telemetry. |
| 22 | + |
| 23 | +1. Edit the `datadog_agent.d/conf.yaml` file in the `conf.d/` folder at the root of your [Agent's configuration directory][4]. See the [sample datadog_agent.d/conf.yaml][5] for all available configuration options. For example: |
| 24 | + |
| 25 | + ```yaml |
| 26 | + init_config: |
| 27 | + |
| 28 | + instances: |
| 29 | + - openmetrics_endpoint: http://localhost:5000/telemetry |
| 30 | + ``` |
| 31 | +
|
| 32 | + To collect telemetry beyond the default set, list the metrics to add with the `extra_metrics` option. You can view all available metrics by querying the `/telemetry` endpoint directly. |
| 33 | + |
| 34 | +2. [Restart the Agent][6]. |
| 35 | + |
| 36 | +### Validation |
| 37 | + |
| 38 | +[Run the Agent's status subcommand][7] and look for `datadog_agent` under the Checks section. |
| 39 | + |
| 40 | +### Migrating from Agent Metrics |
| 41 | + |
| 42 | +The Datadog Agent integration collects similar data to the [Agent Metrics][2] integration, but most metric names, types, and tags differ. You must update any dashboards and monitors that query metrics from the Agent Metrics integration. |
| 43 | + |
| 44 | +1. Configure the Datadog Agent integration as described in [Configuration](#configuration). |
| 45 | +2. Optionally, keep `go_expvar.d/agent_stats.yaml` in place while you update your dashboards and monitors, so that you can compare the old and new metrics. Three metrics have the same name in both integrations: `datadog.agent.aggregator.dogstatsd_contexts`, `datadog.agent.forwarder.transactions.retry_queue_size`, and `datadog.agent.scheduler.checks_entered`. While both checks run, each check submits these metrics. |
| 46 | +3. Update your dashboards and monitors using the [metric mapping](#metric-mapping). |
| 47 | +4. Remove `go_expvar.d/agent_stats.yaml` and [restart the Agent][6]. |
| 48 | + |
| 49 | +#### Behavior changes |
| 50 | + |
| 51 | +- **Counters are submitted as counts.** Agent Metrics submitted counters as per-second rates. The Datadog Agent integration submits them as counts with a `.count` suffix. For example, to get the equivalent of `datadog.agent.forwarder.transactions.success`, query `datadog.agent.forwarder.transactions.success.count` with the `.as_rate()` function. |
| 52 | +- **Related metrics are combined and distinguished by tags.** For example, `datadog.agent.dogstatsd.processed.count`, tagged by `message_type` and `state`, replaces the separate DogStatsD packet and parse error metrics. |
| 53 | +- **Some metrics are now under the `datadog.agent.` namespace.** Agent Metrics submitted `scheduler.queues.size`, `scheduler.queues.interval`, `batch_strategy.idle_time`, and `batch_strategy.in_use_time` without a namespace. |
| 54 | +- **Go runtime metrics are not collected by default.** Agent Metrics also collected Go memory statistics, such as `datadog.agent.memstats.heap_alloc`. The telemetry endpoint exposes them as `go_memstats_*` and `go_gc_*` metrics. To collect them, use the `extra_metrics` option. For example: |
| 55 | + |
| 56 | + ```yaml |
| 57 | + instances: |
| 58 | + - openmetrics_endpoint: http://localhost:5000/telemetry |
| 59 | + extra_metrics: |
| 60 | + - go_memstats_heap_alloc_bytes: go.memstats.heap_alloc |
| 61 | + - go_memstats_heap_inuse_bytes: go.memstats.heap_inuse |
| 62 | + ``` |
| 63 | + |
| 64 | +#### Metric mapping |
| 65 | + |
| 66 | +Unless noted otherwise, all metric names in the table are prefixed with `datadog.agent.`. |
| 67 | + |
| 68 | +| Agent Metrics metric | Datadog Agent metric | |
| 69 | +| --- | --- | |
| 70 | +| `forwarder.transactions.success` | `forwarder.transactions.success.count` | |
| 71 | +| `forwarder.transactions.errors` | `forwarder.transactions.errors.count` | |
| 72 | +| `forwarder.transactions.errors_by_type.dns_errors` | `forwarder.transactions.errors.count` with `error_type:dns_lookup_failure` | |
| 73 | +| `forwarder.transactions.errors_by_type.tls_errors` | `forwarder.transactions.errors.count` with `error_type:tls_handshake_failure` | |
| 74 | +| `forwarder.transactions.errors_by_type.connection_errors` | `forwarder.transactions.errors.count` with `error_type:connection_failure` | |
| 75 | +| `forwarder.transactions.errors_by_type.wrote_request_errors` | `forwarder.transactions.errors.count` with `error_type:writing_failure` | |
| 76 | +| `forwarder.transactions.errors_by_type.sent_request_errors` | `forwarder.transactions.errors.count` with `error_type:invalid_request` | |
| 77 | +| `forwarder.transactions.http_errors` | `forwarder.transactions.http_errors.count`, tagged by `code` | |
| 78 | +| `forwarder.transactions.retry_queue_size` | `forwarder.transactions.retry_queue_size`, tagged by `domain` | |
| 79 | +| `forwarder.transactions.dropped_on_input` | No longer reported by the Agent. See `forwarder.transactions.dropped.count`. | |
| 80 | +| `dogstatsd.metric_packets` | `dogstatsd.processed.count` with `message_type:metrics` and `state:ok` | |
| 81 | +| `dogstatsd.metric_parse_errors` | `dogstatsd.processed.count` with `message_type:metrics` and `state:error` | |
| 82 | +| `dogstatsd.event_packets` | `dogstatsd.processed.count` with `message_type:events` and `state:ok` | |
| 83 | +| `dogstatsd.event_parse_errors` | `dogstatsd.processed.count` with `message_type:events` and `state:error` | |
| 84 | +| `dogstatsd.service_check_packets` | `dogstatsd.processed.count` with `message_type:service_checks` and `state:ok` | |
| 85 | +| `dogstatsd.service_check_parse_errors` | `dogstatsd.processed.count` with `message_type:service_checks` and `state:error` | |
| 86 | +| `dogstatsd_udp.packets` | `dogstatsd_udp.packets.count` with `state:ok` | |
| 87 | +| `dogstatsd_udp.packet_reading_errors` | `dogstatsd_udp.packets.count` with `state:error` | |
| 88 | +| `dogstatsd_uds.packets` | `dogstatsd_uds.packets.count` with `state:ok` | |
| 89 | +| `dogstatsd_uds.packet_reading_errors` | `dogstatsd_uds.packets.count` with `state:error` | |
| 90 | +| `dogstatsd_uds.origin_detection_errors` | `dogstatsd_uds.origin_detection_errors.count` | |
| 91 | +| `aggregator.dogstatsd_contexts` | `aggregator.dogstatsd_contexts`, tagged by `shard` | |
| 92 | +| `aggregator.series_flushed` | `aggregator.flush.count` with `data_type:series` | |
| 93 | +| `aggregator.service_check_flushed` | `aggregator.flush.count` with `data_type:service_checks` | |
| 94 | +| `aggregator.events_flushed` | `aggregator.flush.count` with `data_type:events` | |
| 95 | +| `aggregator.checks_metric_sample` | `aggregator.processed.count` with `data_type:metrics` | |
| 96 | +| `aggregator.dogstatsd_metric_sample` | `aggregator.processed.count` with `data_type:dogstatsd_metrics` | |
| 97 | +| `aggregator.service_check` | `aggregator.processed.count` with `data_type:service_checks` | |
| 98 | +| `aggregator.event` | `aggregator.processed.count` with `data_type:events` | |
| 99 | +| `aggregator.number_of_flush` | `aggregator.number_of_flush.count` | |
| 100 | +| `aggregator.flush.main_flush_time.last_flush` | `aggregator.flush_time` with `flush_type:main` | |
| 101 | +| `aggregator.flush.checks_metric_sample_flush_time.last_flush` | `aggregator.flush_time` with `flush_type:checks_metric_sample` | |
| 102 | +| `aggregator.flush.service_check_flush_time.last_flush` | `aggregator.flush_time` with `flush_type:service_check` | |
| 103 | +| `aggregator.flush.event_flush_time.last_flush` | `aggregator.flush_time` with `flush_type:event` | |
| 104 | +| `aggregator.flush.metric_sketch_flush_time.last_flush` | `aggregator.flush_time` with `flush_type:metric_sketch` | |
| 105 | +| `aggregator.flush_count.series.last_flush` | `aggregator.flush_count` with `flush_type:series` | |
| 106 | +| `aggregator.flush_count.service_checks.last_flush` | `aggregator.flush_count` with `flush_type:service_checks` | |
| 107 | +| `aggregator.flush_count.events.last_flush` | `aggregator.flush_count` with `flush_type:events` | |
| 108 | +| `aggregator.flush_count.sketches.last_flush` | `aggregator.flush_count` with `flush_type:sketches` | |
| 109 | +| `aggregator.hostname_update` | No longer reported by the Agent. | |
| 110 | +| `scheduler.checks_entered` | `scheduler.checks_entered`, tagged by `check_name` | |
| 111 | +| `scheduler.queues_count` | `scheduler.queues_count.count` | |
| 112 | +| `scheduler.queues.size` (no namespace) | `scheduler.queues.size`, tagged by `interval` | |
| 113 | +| `scheduler.queues.interval` (no namespace) | No equivalent. Use the `interval` tag of `scheduler.queues.size`. | |
| 114 | +| `splitter.*` | No longer reported by the Agent. | |
| 115 | +| `logs_agent.logs_decoded` | `logs_agent.logs_decoded.count` | |
| 116 | +| `logs_agent.logs_processed` | `logs_agent.logs_processed.count` | |
| 117 | +| `logs_agent.logs_sent` | `logs_agent.logs_sent.count` | |
| 118 | +| `logs_agent.bytes_sent` | `logs_agent.bytes_sent.count` | |
| 119 | +| `logs_agent.encoded_bytes_sent` | `logs_agent.encoded_bytes_sent.count` | |
| 120 | +| `logs_agent.destination_errors` | `logs_agent.destination_errors.count` | |
| 121 | +| `logs_agent.is_running` | Not collected. | |
| 122 | +| `batch_strategy.idle_time` (no namespace) | `batch_strategy.idle_time.count`, tagged by `sender` (for example, `sender:logs_0_reliable_0` instead of `sender:0`) | |
| 123 | +| `batch_strategy.in_use_time` (no namespace) | `batch_strategy.in_use_time.count`, tagged by `sender` | |
| 124 | +| `memstats.*` | Not collected by default. See the `extra_metrics` example in [Behavior changes](#behavior-changes). | |
| 125 | + |
| 126 | +## Data Collected |
| 127 | + |
| 128 | +### Metrics |
| 129 | + |
| 130 | +See [metadata.csv][8] for a list of metrics provided by this integration. |
| 131 | + |
| 132 | +### Events |
| 133 | + |
| 134 | +The Datadog Agent integration does not include any events. |
| 135 | + |
| 136 | +### Service Checks |
| 137 | + |
| 138 | +See [service_checks.json][9] for a list of service checks provided by this integration. |
| 139 | + |
| 140 | +## Troubleshooting |
| 141 | + |
| 142 | +Need help? Contact [Datadog support][10]. |
| 143 | + |
| 144 | +[1]: https://docs.datadoghq.com/agent/ |
| 145 | +[2]: https://docs.datadoghq.com/integrations/agent_metrics/ |
| 146 | +[3]: /account/settings/agent/latest |
| 147 | +[4]: https://docs.datadoghq.com/agent/configuration/agent-configuration-files/ |
| 148 | +[5]: https://github.com/DataDog/integrations-core/blob/master/datadog_agent/datadog_checks/datadog_agent/data/conf.yaml.example |
| 149 | +[6]: https://docs.datadoghq.com/agent/configuration/agent-commands/#start-stop-and-restart-the-agent |
| 150 | +[7]: https://docs.datadoghq.com/agent/configuration/agent-commands/#agent-status-and-information |
| 151 | +[8]: https://github.com/DataDog/integrations-core/blob/master/datadog_agent/metadata.csv |
| 152 | +[9]: https://github.com/DataDog/integrations-core/blob/master/datadog_agent/assets/service_checks.json |
| 153 | +[10]: https://docs.datadoghq.com/help/ |
0 commit comments