diff --git a/doc/user/content/console/monitoring.md b/doc/user/content/console/monitoring.md index a4c05bbb0844c..da9cdd9e4ae2a 100644 --- a/doc/user/content/console/monitoring.md +++ b/doc/user/content/console/monitoring.md @@ -20,6 +20,6 @@ The **Monitoring** section contains the following screens: | Feature | Description | |---------|-------------| | **Environment Overview** | Review the health of your environment. | -| **Query History** | Access your query history. | +| **Query History** | Access your query history. Query history is sampled. In self-managed deployments, the sample rate is configurable. See [Query History](/self-managed-deployments/query-history/). | | **Sources** | Review your sources. You can select a source to go to its [Database object explorer page](/console/data/). | | **Sinks** | Review your sinks. You can select a sink to go to its [Database object explorer page](/console/data/). | diff --git a/doc/user/content/reference/system-catalog/mz_internal.md b/doc/user/content/reference/system-catalog/mz_internal.md index 65dcf69958410..7a709f89b9f87 100644 --- a/doc/user/content/reference/system-catalog/mz_internal.md +++ b/doc/user/content/reference/system-catalog/mz_internal.md @@ -37,9 +37,11 @@ The `mz_object_global_ids` table maps Materialize catalog item IDs to global IDs {{< public-preview />}} {{< warning >}} -Do not rely on all statements being logged in this view. Materialize -controls the maximum rate at which statements are sampled, and may change -this rate at any time. +Do not rely on all statements being logged in this view. The maximum rate at +which statements are sampled is capped by the +`statement_logging_max_sample_rate` system parameter. In Materialize Cloud, +Materialize controls this cap and may change it at any time. In self-managed +deployments, it is set by the operator. {{< /warning >}} {{< warning >}} @@ -54,8 +56,11 @@ Entries in this log may be sampled. The sampling rate is controlled by the configuration parameter `statement_logging_sample_rate`, which may be set to any value between 0 and 1. For example, to disable statement logging entirely for a session, execute `SET -statement_logging_sample_rate TO 0`. Materialize may apply a lower -sampling rate than the one set in this parameter. +statement_logging_sample_rate TO 0`. The effective rate is the lower of this +parameter and the `statement_logging_max_sample_rate` system parameter, so a +lower cap may apply than the one set here. In self-managed deployments, see +[Query History](/self-managed-deployments/query-history/) for how to configure +the cap. The view can be accessed by Materialize _superusers_ or users that have been granted the [`mz_monitor` role](/security/appendix/appendix-built-in-roles/#system-catalog-roles). @@ -1389,8 +1394,9 @@ logging an execution is controlled by the `statement_logging_sample_rate` configuration parameter. A value of 0 means to log nothing; a value of 0.8 means to log approximately 80% of statement executions. If `statement_logging_sample_rate` is higher -than `statement_logging_max_sample_rate` (which is set by Materialize -and cannot be changed by users), the latter is used instead. +than `statement_logging_max_sample_rate`, the latter is used instead. In +Materialize Cloud, that cap is set by Materialize. In self-managed deployments, +it is set by the operator. | Field | Type | Meaning | |-------------------------|------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| diff --git a/doc/user/content/self-managed-deployments/configuration-system-parameters.md b/doc/user/content/self-managed-deployments/configuration-system-parameters.md index 1c2e5969153c7..2673d4e5e697e 100644 --- a/doc/user/content/self-managed-deployments/configuration-system-parameters.md +++ b/doc/user/content/self-managed-deployments/configuration-system-parameters.md @@ -211,6 +211,8 @@ The following are some commonly configured system parameters: | `max_clusters` | Maximum number of clusters in the region | | `max_sources` | Maximum number of sources in the region | | `max_sinks` | Maximum number of sinks in the region | +| `statement_logging_max_sample_rate` | Cap on the fraction of statements recorded in [query history](/self-managed-deployments/query-history/). Setting it here overrides the Helm chart value. | +| `statement_logging_target_data_rate` | Sustained bytes per second that statement logging may write. Bounds query history growth on busy instances. | For a complete list of available system parameters and their descriptions, see the [configuration parameters](/sql/alter-system-set/#key-configuration-parameters) @@ -311,6 +313,7 @@ kubectl logs -l app=environmentd -n materialize-environment | grep -i "system.*p ## See also +- [Query History](/self-managed-deployments/query-history/) - [Materialize Operator Configuration](/installation/configuration/) - [Materialize CRD Field Descriptions](/installation/appendix-materialize-crd-field-descriptions/) - [Troubleshooting](/installation/troubleshooting/) diff --git a/doc/user/content/self-managed-deployments/query-history.md b/doc/user/content/self-managed-deployments/query-history.md new file mode 100644 index 0000000000000..f0d2e785419aa --- /dev/null +++ b/doc/user/content/self-managed-deployments/query-history.md @@ -0,0 +1,192 @@ +--- +title: "Query History" +description: "How query history and statement logging are configured in self-managed Materialize deployments" +menu: + main: + parent: "sm-deployments" + weight: 72 +--- + +The Materialize Console includes a **Query History** view, under its +[**Monitoring**](/console/monitoring/) section, that lists the SQL statements +recently issued to your Materialize instance along with their duration, status, +and the cluster that ran them. Query history is available in self-managed +deployments and is enabled by default. + +Query history is backed by *statement logging*: Materialize records a randomly +sampled fraction of statement executions into the system catalog, most visibly +[`mz_recent_activity_log`](/reference/system-catalog/mz_internal/#mz_recent_activity_log), +which covers the last 24 hours. Sampling means the view is a representative +sample of your workload rather than a complete audit log. For a complete record +of DDL, use +[`mz_audit_events`](/reference/system-catalog/mz_catalog/#mz_audit_events) +instead. + +To see query history in the Console, connect as a Materialize *superuser* or as +a user granted the [`mz_monitor` +role](/security/appendix/appendix-built-in-roles/#system-catalog-roles). + +## Defaults + +The Materialize operator Helm chart sets +`operator.args.statementLoggingMaxSampleRate` to `0.99`, and leaves +`operator.args.statementLoggingTargetDataRate` unset so that `environmentd`'s +own default of 2071 bytes per second applies. Statement logging is therefore on +by default, sampling nearly every statement up to the byte rate below. + +The rate that actually applies to a statement is the smaller of two parameters: + +| Parameter | Scope | Meaning | +|-----------|-------|---------| +| `statement_logging_sample_rate` | Session | The rate a session asks for. A session may `SET` this to any value between 0 and 1, but only values below the cap have any effect. | +| `statement_logging_max_sample_rate` | System | A cap on the above. The chart value sets this. | + +Because the system parameter is a cap, lowering it reduces logging for every +session regardless of what individual sessions request. + +Sampling is not the only limit, and it is not the one that bounds volume. +Materialize also throttles statement logging to a sustained byte rate, +`statement_logging_target_data_rate`, and drops sampled executions that would +exceed it. So a sample rate of `1.0` does not guarantee every statement is +recorded, and on a busy instance the byte rate is what determines how much +history you actually accumulate. + +{{< note >}} +The chart also enables `enableInternalStatementLogging`, so statements issued by +the `mz_system` user are logged. Internal activity, including the Console's own +catalog queries, is sampled alongside your workload and counts toward the cost +described below. +{{< /note >}} + +## Tune statement logging + +Two parameters bound how much query history you collect: + +| Parameter | Chart value | Bounds | +|-----------|-------------|--------| +| `statement_logging_max_sample_rate` | `operator.args.statementLoggingMaxSampleRate` (`0.99`) | The fraction of executions considered for logging. | +| `statement_logging_target_data_rate` | `operator.args.statementLoggingTargetDataRate` (unset, so 2071) | The sustained bytes per second written. Must be greater than 0. | + +The two are not interchangeable. Lower the data rate to hold storage growth +down. Lowering the sample rate instead makes query history less representative +without lowering the ceiling on what statement logging stores, because the byte +rate is already the binding limit. Each can be set either through the Helm chart +or as a system parameter, and those two paths interact, so read the note on +precedence below before picking one. + +### Using the Helm chart + +Set either value when installing or upgrading the operator. For example, to +halve how fast query history grows: + +```shell +helm upgrade my-materialize-operator materialize/materialize-operator \ + --set operator.args.statementLoggingTargetDataRate=1035 +``` + +Or, in your `values.yaml`: + +```yaml +operator: + args: + statementLoggingMaxSampleRate: 0.99 + statementLoggingTargetDataRate: 1035 +``` + +Setting `statementLoggingMaxSampleRate` to `0` disables statement logging +entirely. Already-logged statements remain visible until they age out of the +24-hour window. Setting either value to `null` inherits `environmentd`'s own +default, `0.99` for the sample rate and 2071 bytes per second for the data rate. +The data rate must be greater than 0. + +The operator passes these values to `environmentd` as the *defaults* for +`statement_logging_max_sample_rate` and +`statement_logging_target_data_rate`, so they only take effect when +`environmentd` restarts. Upgrading the operator does not by itself roll out your Materialize +instances: you also need to request a rollout, as described in [modifying the +custom +resource](/self-managed-deployments/#modifying-the-custom-resource). For the +full list of chart values, see [Materialize Operator +Configuration](/self-managed-deployments/operator-configuration/). + +### Using system parameters + +Because the chart values are only defaults, you can override either at runtime +without a rollout, through the `system-params.json` ConfigMap described in +[Configuring System +Parameters](/self-managed-deployments/configuration-system-parameters/): + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: mz-system-params + namespace: materialize-environment +data: + system-params.json: | + { + "statement_logging_target_data_rate": 1035 + } +``` + +Or with [`ALTER SYSTEM SET`](/sql/alter-system-set/), connected as the +`mz_system` user: + +```mzsql +ALTER SYSTEM SET statement_logging_target_data_rate = 1035; +``` + +{{< note >}} +A value set through the ConfigMap or `ALTER SYSTEM SET` is stored in the catalog +and takes precedence over the Helm chart value, which is only a default. While +such an override is in place, editing the corresponding chart value has no +effect. + +To go back to the chart-provided value, first remove the parameter from the +ConfigMap, then run [`ALTER SYSTEM +RESET`](/sql/alter-system-reset/) for it. Removing it from the ConfigMap alone is +not enough, because the last synced value remains in the catalog. Resetting it +while it is still in the ConfigMap is also not enough, because the sync loop +reapplies it. +{{< /note >}} + +To check the values currently in effect: + +```mzsql +SHOW statement_logging_max_sample_rate; +SHOW statement_logging_target_data_rate; +``` + +## Cost of statement logging + +Statement logging has two distinct costs, and each is governed by a different +parameter: + +- **CPU on `environmentd`**, governed by the sample rate. Every logged execution + is prepared and written by the control plane, so the overhead scales with your + statement throughput, not with your data volume. Instances serving many short + queries pay the most. + +- **Storage**, governed by the target data rate. Logged statements, including + their SQL text, consume space in your blob storage and metadata backend. + Although `mz_recent_activity_log` only surfaces the last 24 hours, the + underlying statement history collections are never truncated, so their + footprint grows for the lifetime of the instance. + +That second point is the one to plan around: query history is not a fixed-size +buffer, and its growth rate is set by `statement_logging_target_data_rate`. On +an instance with limited storage, lower that parameter. Reach for the sample +rate only when you want to reduce `environmentd` CPU overhead, and expect less +representative history in exchange. + +## See also + +- [Console monitoring](/console/monitoring/) +- [`mz_internal` statement logging + relations](/reference/system-catalog/mz_internal/#mz_recent_activity_log) +- [`ALTER SYSTEM SET`](/sql/alter-system-set/) +- [`ALTER SYSTEM RESET`](/sql/alter-system-reset/) +- [Configuring System + Parameters](/self-managed-deployments/configuration-system-parameters/) +- [Materialize Operator + Configuration](/self-managed-deployments/operator-configuration/) diff --git a/doc/user/content/transform-data/troubleshooting.md b/doc/user/content/transform-data/troubleshooting.md index 42b5c31618e0c..f1d8b32b04b34 100644 --- a/doc/user/content/transform-data/troubleshooting.md +++ b/doc/user/content/transform-data/troubleshooting.md @@ -291,7 +291,7 @@ attribute performance information to each LIR operator. ## How do I troubleshoot slow queries? Materialize stores a (sampled) log of the SQL statements that are issued against -your Materialize region in the last **three days**, along with various metadata +your Materialize region in the last **24 hours**, along with various metadata about these statements. You can access this log via the **"Query history"** tab in the [Materialize console](/console/). You can filter and sort statements by type, duration, and other dimensions. @@ -300,10 +300,11 @@ This data is also available via the [mz_internal.mz_recent_activity_log](/reference/system-catalog/mz_internal/#mz_recent_activity_log) catalog table. -It's important to note that the default (and max) sample rate for most -Materialize organizations is 99%, which means that not all statements will be -captured in the log. The sampling rate is not user-configurable, and may change -at any time. +It's important to note that statements are sampled, so not all of them will be +captured in the log. In Materialize Cloud, the default and maximum sample rate +for most organizations is 99%, and Materialize may change it at any time. In +self-managed deployments, the maximum sample rate is set by the operator. See +[Query History](/self-managed-deployments/query-history/). If you're looking for a complete audit history, use the [mz_audit_events](/reference/system-catalog/mz_catalog/#mz_audit_events) catalog table, which records all DDL commands issued against your Materialize