docs: track CNS-136's revised statement logging defaults

CNS-136 changed the chart sample rate default from 0.1 to 0.99, matching
Cloud, and clarified that the sample rate is not what bounds how much
query history stores.

Update the documented defaults and invert the tuning guidance to match:
statement_logging_target_data_rate is the parameter that bounds storage
growth, so that is the one to lower on instances with limited storage.
Lowering the sample rate only reduces environmentd CPU overhead and
makes the history less representative, without lowering the ceiling.

CNS-139

Author: jubrad

doc/user/content/self-managed-deployments/query-history.md

@@ -26,16 +26,14 @@ To see query history in the Console, connect as a Materialize *superuser* or as
 a user granted the [`mz_monitor`
 role](/security/appendix/appendix-built-in-roles/#system-catalog-roles).
 
-## Default sample rate
+## Defaults
 
-The Materialize operator Helm chart sets the
-`operator.args.statementLoggingMaxSampleRate` value to `0.1`, so roughly 10% of
-statement executions are sampled.
-
-This value is lower than `environmentd`'s own default of `0.99`. Statement
-logging costs CPU on `environmentd` and storage for the retained history, and
-that overhead is most noticeable on the small instances typical of self-managed
-deployments, so the chart samples a fraction of statements instead.
+The Materialize operator Helm chart sets
+`operator.args.statementLoggingMaxSampleRate` to `0.99`, and leaves
+`operator.args.statementLoggingTargetDataRate` unset so that `environmentd`'s
+own default of 2071 bytes per second applies. Both are the values Materialize
+Cloud runs at, so query history in a self-managed deployment behaves the same as
+in Cloud out of the box.
 
 The rate that actually applies to a statement is the smaller of two parameters:
 
@@ -47,17 +45,16 @@ The rate that actually applies to a statement is the smaller of two parameters:
 Because the system parameter is a cap, lowering it reduces logging for every
 session regardless of what individual sessions request.
 
-Sampling is not the only limit. Materialize also throttles statement logging to
-a sustained byte rate, `statement_logging_target_data_rate`, which defaults to
-2071 bytes per second. Sampled executions beyond that budget are dropped, so a
-sample rate of `1.0` does not guarantee that every statement is recorded. On
-busy instances this byte rate, rather than the sample rate, is what actually
-bounds how much history you collect.
+Sampling is not the only limit, and it is not the one that bounds volume.
+Materialize also throttles statement logging to a sustained byte rate,
+`statement_logging_target_data_rate`, and drops sampled executions that would
+exceed it. So a sample rate of `1.0` does not guarantee every statement is
+recorded, and on a busy instance the byte rate is what determines how much
+history you actually accumulate.
 
 {{< note >}}
-Unlike Materialize Cloud, self-managed deployments log statements issued by the
-`mz_system` user by default, because the chart enables
-`enableInternalStatementLogging`. Internal activity, including the Console's own
+The chart also enables `enableInternalStatementLogging`, so statements issued by
+the `mz_system` user are logged. Internal activity, including the Console's own
 catalog queries, is sampled alongside your workload and counts toward the cost
 described below.
 {{< /note >}}
@@ -68,43 +65,45 @@ Two parameters bound how much query history you collect:
 
 | Parameter | Chart value | Bounds |
 |-----------|-------------|--------|
-| `statement_logging_max_sample_rate` | `operator.args.statementLoggingMaxSampleRate` (`0.1`) | The fraction of executions considered for logging. |
-| `statement_logging_target_data_rate` | `operator.args.statementLoggingTargetDataRate` (unset) | The sustained bytes per second written, 2071 by default. |
+| `statement_logging_max_sample_rate` | `operator.args.statementLoggingMaxSampleRate` (`0.99`) | The fraction of executions considered for logging. |
+| `statement_logging_target_data_rate` | `operator.args.statementLoggingTargetDataRate` (unset, so 2071) | The sustained bytes per second written. Must be greater than 0. |
 
-Raise the sample rate for more representative history. Lower the data rate to
-hold storage growth down. Each can be set either through the Helm chart or as a
-system parameter, and those two paths interact, so read the note on precedence
-below before picking one.
+The two are not interchangeable. Lower the data rate to hold storage growth
+down. Lowering the sample rate instead makes query history less representative
+without lowering the ceiling on what statement logging stores, because the byte
+rate is already the binding limit. Each can be set either through the Helm chart
+or as a system parameter, and those two paths interact, so read the note on
+precedence below before picking one.
 
 ### Using the Helm chart
 
-Set `operator.args.statementLoggingMaxSampleRate` when installing or upgrading
-the operator:
+Set either value when installing or upgrading the operator. For example, to
+halve how fast query history grows:
 
 ```shell
 helm upgrade my-materialize-operator materialize/materialize-operator \
-  --set operator.args.statementLoggingMaxSampleRate=0.5
+  --set operator.args.statementLoggingTargetDataRate=1035
 ```
 
 Or, in your `values.yaml`:
 
 ```yaml
 operator:
   args:
-    statementLoggingMaxSampleRate: 0.5
-    statementLoggingTargetDataRate: 4096
+    statementLoggingMaxSampleRate: 0.99
+    statementLoggingTargetDataRate: 1035
 ```
 
 Setting `statementLoggingMaxSampleRate` to `0` disables statement logging
-entirely. Already-logged
-statements remain visible until they age out of the 24-hour window. Setting the
-value to `null` inherits `environmentd`'s default of `0.99`.
-`statementLoggingTargetDataRate` is unset by default, which inherits
-`environmentd`'s default of 2071 bytes per second.
-
-The operator passes this value to `environmentd` as the *default* for
-`statement_logging_max_sample_rate`, so it only takes effect when `environmentd`
-restarts. Upgrading the operator does not by itself roll out your Materialize
+entirely. Already-logged statements remain visible until they age out of the
+24-hour window. Setting either value to `null` inherits `environmentd`'s own
+default, `0.99` for the sample rate and 2071 bytes per second for the data rate.
+The data rate must be greater than 0.
+
+The operator passes these values to `environmentd` as the *defaults* for
+`statement_logging_max_sample_rate` and
+`statement_logging_target_data_rate`, so they only take effect when
+`environmentd` restarts. Upgrading the operator does not by itself roll out your Materialize
 instances: you also need to request a rollout, as described in [modifying the
 custom
 resource](/self-managed-deployments/#modifying-the-custom-resource). For the
@@ -113,9 +112,9 @@ Configuration](/self-managed-deployments/operator-configuration/).
 
 ### Using system parameters
 
-Because the chart value is only a default, you can override it at runtime
-without a rollout, either through the `system-params.json` ConfigMap described
-in [Configuring System
+Because the chart values are only defaults, you can override either at runtime
+without a rollout, through the `system-params.json` ConfigMap described in
+[Configuring System
 Parameters](/self-managed-deployments/configuration-system-parameters/):
 
 ```yaml
@@ -127,22 +126,22 @@ metadata:
 data:
   system-params.json: |
     {
-      "statement_logging_max_sample_rate": 0.5
+      "statement_logging_target_data_rate": 1035
     }
 ```
 
 Or with [`ALTER SYSTEM SET`](/sql/alter-system-set/), connected as the
 `mz_system` user:
 
 ```mzsql
-ALTER SYSTEM SET statement_logging_max_sample_rate = 0.5;
+ALTER SYSTEM SET statement_logging_target_data_rate = 1035;
 ```
 
 {{< note >}}
 A value set through the ConfigMap or `ALTER SYSTEM SET` is stored in the catalog
 and takes precedence over the Helm chart value, which is only a default. While
-such an override is in place, editing
-`operator.args.statementLoggingMaxSampleRate` has no effect.
+such an override is in place, editing the corresponding chart value has no
+effect.
 
 To go back to the chart-provided value, first remove the parameter from the
 ConfigMap, then run [`ALTER SYSTEM
@@ -152,32 +151,34 @@ while it is still in the ConfigMap is also not enough, because the sync loop
 reapplies it.
 {{< /note >}}
 
-To check the rate currently in effect:
+To check the values currently in effect:
 
 ```mzsql
 SHOW statement_logging_max_sample_rate;
+SHOW statement_logging_target_data_rate;
 ```
 
-## Cost of raising the sample rate
+## Cost of statement logging
 
-Raising the sample rate gives more complete query history at the cost of:
+Statement logging has two distinct costs, and each is governed by a different
+parameter:
 
-- **CPU on `environmentd`**: every logged execution is prepared and written by
-  the control plane, so the overhead scales with your statement throughput, not
-  with your data volume. Instances serving many short queries pay the most.
+- **CPU on `environmentd`**, governed by the sample rate. Every logged execution
+  is prepared and written by the control plane, so the overhead scales with your
+  statement throughput, not with your data volume. Instances serving many short
+  queries pay the most.
 
-- **Storage**: logged statements, including their SQL text, are retained in the
-  system catalog and consume space in your blob storage and metadata backend.
+- **Storage**, governed by the target data rate. Logged statements, including
+  their SQL text, consume space in your blob storage and metadata backend.
   Although `mz_recent_activity_log` only surfaces the last 24 hours, the
-  underlying statement history collections are not truncated, so their footprint
-  grows for the lifetime of the instance.
-
-Workloads dominated by a high rate of small, fast queries are the ones where a
-high sample rate is most likely to be felt. If you raise the rate, do it
-incrementally and watch `environmentd` CPU utilization and your storage
-footprint. Because the byte rate is what bounds sustained growth, lowering
-`statement_logging_target_data_rate` is the more direct control if your concern
-is storage rather than sampling fidelity.
+  underlying statement history collections are never truncated, so their
+  footprint grows for the lifetime of the instance.
+
+That second point is the one to plan around: query history is not a fixed-size
+buffer, and its growth rate is set by `statement_logging_target_data_rate`. On
+an instance with limited storage, lower that parameter. Reach for the sample
+rate only when you want to reduce `environmentd` CPU overhead, and expect less
+representative history in exchange.
 
 ## See also