DEP-204 Add docs for datadog/otel destinations for monitoring - #38323
Conversation
There was a problem hiding this comment.
Pull request overview
Documents expanded Self-Managed monitoring destinations and the planned default observability behavior.
Changes:
- Adds OTLP, remote-write, and Google Cloud Monitoring guidance.
- Replaces legacy Datadog SQL-exporter instructions.
- Updates installation, upgrade, and navigation documentation.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
upgrade-on-gcp.md |
Documents the observability default during GCP upgrades. |
upgrade-on-azure.md |
Documents the observability default during Azure upgrades. |
upgrade-on-aws.md |
Documents the observability default during AWS upgrades. |
install-on-gcp.md |
Updates GCP installation guidance. |
install-on-azure.md |
Updates Azure installation guidance. |
install-on-aws.md |
Updates AWS installation guidance. |
opentelemetry.md |
Adds external telemetry destination guidance. |
grafana.md |
Cross-links external monitoring destinations. |
datadog.md |
Replaces the legacy Datadog workflow. |
self-managed/_index.md |
Expands Self-Managed monitoring navigation. |
monitor/_index.md |
Expands monitoring navigation. |
Suppressed comments (1)
doc/user/content/manage/monitor/self-managed/grafana.md:312
- “Push a copy” is inaccurate for the linked Prometheus remote-write configuration, which redirects the single Thanos destination and leaves the bundled dashboards empty. Describe both additive exports and replacement explicitly.
Those two endpoints are pull-based: your tooling queries the stack. To have the
stack push a copy of its metrics to an external backend instead, see
[Datadog](/manage/monitor/self-managed/datadog/) or [OpenTelemetry and remote
write](/manage/monitor/self-managed/opentelemetry/).
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| The [monitoring stack](/manage/monitor/self-managed/grafana/) collects metrics | ||
| into a Grafana Alloy gateway and remote-writes them to the bundled Thanos. The | ||
| gateway can also push a copy to backends outside the cluster, so a central | ||
| observability platform sees Materialize without anyone querying the stack. | ||
|
|
||
| Each destination is independent, and each gets its own copy of the metrics with | ||
| its own filter. Full-fidelity local storage in Thanos and a smaller, cheaper | ||
| slice to a metered SaaS backend are the same install, not a tradeoff. |
| | **Google Cloud Monitoring** | `enable_google_cloud_metrics` | Additive, GCP only. Authenticates with Workload Identity. | | ||
| | **Prometheus remote write** (Mimir, Amazon Managed Prometheus, Grafana Cloud, another Thanos) | `additional_values` | **Not** additive. This is the destination the bundled Thanos occupies. | | ||
|
|
||
| All of these need **TF v12.0.0** or later. |
| monitoring stack alongside your Materialize deployment. Starting in **TF | ||
| v12.0.0**, `enable_observability` defaults to `true`, so the stack is installed |
| The gateway also fans metrics out to destinations outside the cluster, in | ||
| addition to Thanos. See [Datadog](/manage/monitor/self-managed/datadog/) and | ||
| [OpenTelemetry and remote | ||
| write](/manage/monitor/self-managed/opentelemetry/). |
| - The monitoring stack installed, with `enable_observability = true`. See | ||
| [Grafana](/manage/monitor/self-managed/grafana/). Datadog export requires **TF | ||
| v12.0.0** or later. |
| {{< warning >}} | ||
| Repoint remote write only when the external store is replacing Thanos, not when | ||
| you want a second copy. For a second copy, use the OTLP destination above. | ||
| {{< /warning >}} |
|
|
||
| This guide walks you through the steps required to monitor the performance and | ||
| overall health of your Materialize region using [Datadog](https://www.datadoghq.com/). | ||
| The [monitoring stack](/manage/monitor/self-managed/grafana/) collects metrics |
There was a problem hiding this comment.
I'd try to make this document standalone, so that a user can use just the datadog page! @Alphadelta14 lmk if this makes sense.
--
This guide walks you through the steps required to monitor the performance and
overall health of your Materialize region using Datadog. Materialize Self-Managed utilizes remote writes to push logs and metrics into Datadog.
How it works
$I'd re-incorporate the material from the introduction on the existing Grafana page! I'd do this by creating shared content that can be used in both pages
Instructions
Before you begin
Ensure you have:
-A Materialize deployment created with the Materialize Terraform modules.
-Terraform ⧉ installed.
- kubectl ⧉ installed and configured to connect to your cluster.
Step 1: Enable Observability
$ Same content as what we have in the existing grafana page
Step 2: Configure the datadog destination
Same as existing content
Step 3: Confirm metrics are arriving
Step 4: Build alerts
Same as existing content
How to control which metrics Datadog receives
same as existing content
How to forward logs
same as existing content
Instructions when using Helm
same as existing content
| identifier: "otlp-sm" | ||
| --- | ||
|
|
||
| The [monitoring stack](/manage/monitor/self-managed/grafana/) collects metrics |
There was a problem hiding this comment.
Similar to the datadog notes, I would modify this slightly:
--
This guide walks you through the steps required to monitor the performance and
overall health of your Materialize region using any OpenTelemetry compatible destination, such as HoneyComb or Grafana Cloud.
$ Include your existing table here
How it works
$I'd re-incorporate the material from the introduction on the existing Grafana page! I'd do this by creating shared content that can be used in both pages
Instructions
Before you begin
Ensure you have:
-A Materialize deployment created with the Materialize Terraform modules.
-Terraform ⧉ installed.
- kubectl ⧉ installed and configured to connect to your cluster.
Step 1: Enable Observability
$ Same content as what we have in the existing grafana page
Step 2: Choose which metrics to deliver
Same as existing content
Step 3: Export to an OLTP endpoint
Step 4: Confirm metrics are being delivered
Same as existing content
Step 5: Configure alerts
How to export to a Prometheus remote-write store
same as existing content
Instructions when using Helm
same as existing content
| ⧉](https://materializeinc.github.io/materialize-monitoring/metrics/storing/) for | ||
| the Secret's keys, the per-`authType` blocks, and the SigV4 setup. | ||
|
|
||
| ## Export to Google Cloud Monitoring |
There was a problem hiding this comment.
I'd move GCM to its own page!
|
|
||
| Logs are considerably higher volume than metrics. Turn this on deliberately. | ||
|
|
||
| ## Export to a Prometheus remote-write store |
There was a problem hiding this comment.
I'd consider moving this to its own page. It feels like this is a different path entirely than the other paths on the page. do you agree @Alphadelta14?
| stack needs, so you do not have to configure scrape targets, data sources, or | ||
| dashboards yourself. | ||
|
|
||
| The gateway also fans metrics out to destinations outside the cluster, in |
There was a problem hiding this comment.
Very minor, but I'd keep this out of the Grafana page. Let's use the Grafana page just for grafana, and then keep the high level concepts for whta the MZ terraform modules do in shared content
| Prometheus-and-Grafana pair that collected metrics only. **TF v10.1.0** then | ||
| added durable state for Grafana and a load balancer to reach it on. If you are | ||
| upgrading from before v10.0.0, read [Upgrading from the previous | ||
| added durable state for Grafana and a load balancer to reach it on, and **TF |
There was a problem hiding this comment.
I'd just make this: "If you're upgrading from a previous version of Terraform, read Upgrading from the previous stack first"
|
@Alphadelta14 could you take a look at the latest changes and lmk if everything is good? If yes let's merge in once TF v12 is out |
The Grafana page's upgrade heading was renamed, which changed its generated anchor and left four links pointing at the old `#upgrading-from-the-previous-stack`. htmltest reported all four as broken, failing the `lint-docs` job. Name the heading "How to upgrade from previous versions of the Materialize Terraform Modules" and repoint the links at it: the in-page reference on the Grafana page, plus the cross-page references from the AWS, Azure, and GCP upgrade guides. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
78c218f to
0c5bcdb
Compare
Motivation
This documents the last piece of other monitoring destinations: Datadog, OpenTelemetry (OTLP, Honeycomb, GCM), RemoteWrite (Thanos, AMP) that were connected in MaterializeInc/materialize-terraform-self-managed#379.
Description
This replaces our datadog docs, updates install instructions, and adds another page for other OTel dests.
Since enable_observability is enable by default in MaterializeInc/materialize-terraform-self-managed#377, this also mentions that.
Verification