From f2ecd7b129754168194860675a4a752d313a82cc Mon Sep 17 00:00:00 2001 From: "David Muto (pseudomuto)" Date: Mon, 29 Jun 2026 16:34:10 -0400 Subject: [PATCH] [rfc]: Payload Compression Add an RFC proposing how the Temporal Proxy will optionally compress workflow payloads before forwarding them upstream. Centralizing compression at the proxy lets operators enable it in one place, transparently to workflow developers, instead of building it into every Worker's Data Converter or a separate codec server. It composes with the proxy's encryption (always compressing before encrypting) and can help payloads stay under Temporal's per-payload and gRPC request size limits. The proposal covers the three supported algorithms (LZ4, ZSTD, Snappy) and why others are excluded, the per-payload metadata scheme that drives decompression, global and per-namespace configuration including per-namespace opt-out, and the observability metrics. Structure mirrors the encryption RFC (#49), with a reference to the overview RFC up top. --- rfc/compression.md | 290 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 290 insertions(+) create mode 100644 rfc/compression.md diff --git a/rfc/compression.md b/rfc/compression.md new file mode 100644 index 0000000..48df05e --- /dev/null +++ b/rfc/compression.md @@ -0,0 +1,290 @@ +# Compression of Workflow Data + +This RFC outlines a proposal for how the Temporal Proxy will optionally compress customer data before forwarding it +upstream. It expands on the **Payload compression** capability introduced in the +[Temporal Proxy overview RFC](./01-overview.md). + +Questions and feedback can be directed to your Temporal account rep, the proxy team, or the community Slack channel. + +- **Slack:** [#proxy](https://temporalio.slack.com/archives/C0BD5MZF6UD) +- **Email:** + +## Background + +Workflow payloads cost bandwidth on the way to the upstream cluster and storage once they land in Workflow history, +visibility, and archival. Large, text-heavy payloads (JSON, XML, logs, serialized documents) are often highly +compressible, so compressing them before they leave the Worker pool can meaningfully cut both costs. + +Today, a customer who wants this must build compression into every Worker's Data Converter or stand up their own codec +server, then keep that consistent across many teams. By centralizing compression at the proxy, operators turn it on in +one place, transparently to workflow developers, and pair it cleanly with the proxy's encryption if enabled. + +Compression is designed as an **opt-in** feature. It is useful on its own and composes with encryption when both are +enabled. + +> [!TIP] +> +> **Staying under Temporal's [size limits](https://docs.temporal.io/cloud/limits).** Temporal Cloud caps each payload at +> 2 MB, and every gRPC request at 4 MB across all payloads plus command metadata, so a Workflow can exceed the request +> limit even when each individual payload is under 2 MB. Because the proxy compresses before forwarding upstream, the +> cluster sees the compressed bytes, so compression can bring a payload, or a request carrying many of them, back under +> these limits. + +## Goals and non-goals + +Goals for the initial release: + +- **Transparent:** no special Worker or client configuration; nothing changes for workflow developers. +- **Effective:** meaningful size reduction on compressible payloads, with a choice of algorithms to trade speed against + ratio. +- **Composable:** works cleanly alongside encryption, always compressing before encrypting. +- **Reliable under change:** the proxy must never produce something it cannot later decompress. Changing the configured + algorithm, or turning compression off, must never strand data already on the wire. +- **Observable:** operators can see whether compression is actually helping for their traffic. + +Non-goals for the initial release: + +- **Automatic content-type detection or auto-skip** of already-compressed or incompressible data. Operators use the + ratio metric and per-namespace configuration instead. +- **Per-payload algorithm selection.** The configured algorithm applies to all eligible payloads. + +## At a glance + +- **Scheme:** per-payload. The proxy compresses the payload's `Data` and records the algorithm in the payload's + `Metadata` so it can be decompressed correctly on the return trip. +- **Algorithms:** LZ4 (throughput), ZSTD (ratio, tunable level), and Snappy (fast, simple). All ship in every build. +- **Ordering:** compression runs before encryption, because encrypted bytes generally do not compress. +- **Granularity:** a global setting, plus optional per-namespace overrides. +- **Default algorithm:** none. When compression is enabled, an algorithm must be chosen explicitly. +- **Minimum size:** payloads below a configurable threshold are forwarded uncompressed, since the framing overhead + outweighs any savings. +- **Decompression:** always attempted on any payload whose metadata marks it compressed, regardless of whether + compression is currently enabled. + +> [!NOTE] +> +> Decoding is driven entirely by the payload, not the current configuration. Because every build understands all +> supported algorithms, changing the configured algorithm or disabling compression never prevents an older payload from +> being decompressed. + +## How it works + +Compression is implemented as an interceptor that walks the request/response objects and acts on [Payload] values, +similar to how encryption works. When enabled, the proxy compresses eligible payloads before forwarding them upstream +and decompresses them before returning results downstream. All of this is transparent; Workers and clients need no +special configuration. + +A compressed payload has its `Data` field set to the compressed bytes of the original data, and its `Metadata` extended +with a marker recording which algorithm was used. The payload's existing encoding metadata is preserved untouched, so +the original value is fully reconstructed on the way back. + +```mermaid +flowchart LR + PT["payload Data (plaintext)"] --> COMP["compress (LZ4 / ZSTD / Snappy)"] + COMP --> CT["compressed Data"] + CT --> P[[Payload]] + MARK["metadata: compression/algorithm"] --> P +``` + +> [!NOTE] +> +> [Memo] and [Header] field values are already Payload objects, so they are compressed and decompressed in the same way +> without any special handling. In practice many of these are small enough to fall below the minimum-size threshold and +> are forwarded uncompressed. + +[Memo]: https://github.com/temporalio/api/blob/8e0453c3a17693d7abb78853c3ccccf0c632e782/temporal/api/common/v1/message.proto#L53 +[Header]: https://github.com/temporalio/api/blob/8e0453c3a17693d7abb78853c3ccccf0c632e782/temporal/api/common/v1/message.proto#L59 +[Payload]: https://github.com/temporalio/api/blob/8e0453c3a17693d7abb78853c3ccccf0c632e782/temporal/api/common/v1/message.proto#L32 + +### Why compress before encrypt? + +Encryption turns data into high-entropy, effectively random bytes, which do not compress. Compressing first is the only +ordering that yields any savings; reversing it would spend CPU on both steps for no benefit. When both features are +enabled the pipeline is therefore `compress -> encrypt` on the way upstream, and `decrypt -> decompress` on the way +back. The two interceptors are independent: either can be enabled without the other. + +### Choosing an algorithm + +The proxy proposes three algorithms. We deliberately keep the list short; adding more later is straightforward. + +| Algorithm | Speed | Compression ratio | Notes | +| ---------- | ---------------------------- | ---------------------------------------- | ------------------------------------------------------------------------ | +| **LZ4** | Fastest | Lowest of the three | Symmetric; very fast to decompress as well as compress. | +| **Snappy** | Comparable to LZ4 | Similar to LZ4, usually a touch behind | No tunable level; chosen for simplicity and codec-server parity. | +| **ZSTD** | Slower than LZ4, but tunable | Highest of the three, rises with `level` | At low levels approaches LZ4 speed; higher levels trade speed for ratio. | + +There is **no implicit default**. When an operator enables compression they must name an algorithm, so payload format is +never changed based on a default we happened to pick. As a rough guide, reach for ZSTD when storage and bandwidth +savings matter most (level 3 is a sensible starting point) and LZ4 or Snappy when proxy CPU or added latency is the +primary concern. Exact throughput and ratio depend heavily on your payloads, so treat the table as a relative guide; the +[compression-ratio metric](#observability) tells you whether the choice is paying off for your traffic, and the +[zstd benchmarks](https://github.com/facebook/zstd#benchmarks) give published comparisons on a standard corpus. + +> [!NOTE] +> +> Only the algorithm is recorded in payload metadata, not the level. Decompression is self-describing given the +> algorithm; the level affects encoding effort only, so old payloads decode correctly even after the configured level +> changes. + +### Algorithms considered but excluded + +The supported list is intentionally small; because each payload records its own algorithm, more can be added later +without a format change. + +- **gzip / DEFLATE.** ZSTD compresses better and faster across DEFLATE's useful range, so it strictly improves on gzip + for this use case. gzip's main draw, in the author's opinion, is universal interoperability, which does not apply here + since the proxy compresses and decompresses its own payloads. There is no external consumer of the compressed bytes to + remain compatible with. +- **Brotli.** Strong ratios on text, which fits this workload, but slower at the high quality levels where it wins, and + ZSTD already covers the high-ratio end. Straightforward to add later if text ratio becomes a priority. +- **bzip2, xz / LZMA.** Archival-grade ratios at speeds far too slow for an inline, per-request transform on the hot + path. + +## Global settings and per-namespace overrides + +Compression is configured with a global block that sets the proxy-wide behavior, plus optional per-namespace +`overrides`. A namespace without its own entry, including one created after the proxy started, uses the global settings. + +Per-namespace overrides are not strictly necessary, but there are a few real reasons to reach for them: + +- **Safe rollout.** Try a setting on a low-risk namespace (for example, staging or test) before applying it everywhere. +- **Mixed payload profiles.** One namespace may carry large compressible documents while another carries small or + already-compressed blobs where compression only costs CPU. Per-namespace settings let you compress the former and skip + the latter. +- **Cost and latency tuning.** A latency-sensitive namespace can use LZ4 while a storage-heavy one uses a higher ZSTD + level. +- **Opt-out.** Set `enabled: false` for a single namespace whose payloads do not benefit (already compressed or + incompressible) while compression stays on everywhere else. Unlike encryption, compression has no fail-closed + constraint, so disabling it for one namespace is safe. + +When compression is turned off (`enabled: false`), the decode path still runs: the proxy keeps decompressing any payload +whose metadata marks it compressed. Disabling compression is therefore safe and never strands data already on the wire. +Likewise, switching the configured algorithm only affects new payloads; existing ones carry their own algorithm marker +and continue to decode. + +## Compression + +Compression happens automatically based on the proxy configuration. For each payload being forwarded upstream, the proxy +first resolves the applicable settings (a namespace override merged over the global block, otherwise the global block +alone), then compresses only when compression is enabled for that namespace and the payload is at least `minBytes`. + +```mermaid +flowchart TD + PT["payload (forwarding upstream)"] --> RES["resolve settings (namespace override, else global)"] + RES --> EN{"enabled for this namespace?"} + EN -->|"no"| PASS["forward unchanged"] + EN -->|"yes"| SZ{"Data size >= minBytes?"} + SZ -->|"no"| PASS2["forward unchanged"] + SZ -->|"yes"| COMP["compress with configured algorithm"] + COMP --> MARK["record compression/algorithm in metadata"] + MARK --> UP["forward upstream"] +``` + +## Decompression + +Decompression is likewise transparent and requires no Worker or client configuration. The proxy only acts on payloads +whose metadata marks them compressed, so plaintext payloads and payloads compressed by some other system pass through +untouched. Crucially, the decode path ignores the `enabled` flag entirely; it is driven only by what the payload +records. + +```mermaid +flowchart TD + IN["payload (returning downstream)"] --> MARKED{"metadata marks it compressed?"} + MARKED -->|"no"| PASS["pass through unchanged"] + MARKED -->|"yes"| DEC["decompress with the recorded algorithm"] + DEC --> STRIP["remove compression marker from metadata"] + STRIP --> OUT["original payload -> return to caller"] +``` + +## Configuration reference + +Compression is configured under the top-level `compression` block. A fully annotated example: + +```yaml +compression: + # Whether to compress NEW payloads. Optional; defaults to false. + # Decompression is attempted on any compressed payload regardless of this + # flag, so turning it off does NOT strand data already on the wire. + enabled: true + + # Algorithm used to compress new payloads. REQUIRED when `enabled: true`. + # One of: lz4 | zstd | snappy. There is no default. + algorithm: zstd + + # Compression effort. ZSTD only; ignored by lz4 and snappy. Optional. + # Higher is smaller and slower. Validated against the supported range. + level: 3 + + # Payloads whose Data is smaller than this (in bytes) are forwarded + # uncompressed, since framing overhead would outweigh any savings. + # Optional; defaults to a small value. + minBytes: 256 + + # Per-namespace overrides, keyed by namespace. Each value may set any of + # `enabled`, `algorithm`, `level`, and `minBytes`, and is merged field-by-field + # over the global block for that namespace. A namespace not listed here uses + # the global settings. Optional. + overrides: + staging: + algorithm: lz4 + archive: + algorithm: zstd + level: 9 + precompressed: + enabled: false # opt this namespace out while compression stays on globally +``` + +### Field reference + +| Field | Type | Required | Description | +| ----------- | -------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------- | +| `enabled` | `bool` | No (default `false`) | Whether to compress new payloads. Decompression is attempted regardless. | +| `algorithm` | `string` | Yes, when `enabled: true` | One of `lz4`, `zstd`, `snappy`. An unrecognized value is rejected at startup. No default. | +| `level` | `int` | No | ZSTD compression effort; ignored by `lz4` and `snappy`. Validated against the supported range. | +| `minBytes` | `int` | No (small default) | Payloads with `Data` smaller than this are forwarded uncompressed. Must be `>= 0`. | +| `overrides` | `map` | No | Per-namespace settings (`enabled`, `algorithm`, `level`, `minBytes`) merged field-by-field over the global block. | + +## Observability + +### Metrics + +| Metric | Type | Labels | +| ----------------------- | --------- | ------------------------------------------------------------- | +| `compression_ops_total` | Counter | operation (compress/decompress), algorithm, namespace, result | +| `payload_size_bytes` | Histogram | stage (pre_compress/post_compress/post_encrypt), namespace | + +### Examples of using these metrics + +#### Compression ratio + +Tracks how effective compression is, expressed as original size over compressed size (so 2.0 means the data halved; +higher is better). A ratio at or near 1.0 for a namespace means its payloads barely compress (already compressed, +encrypted, or otherwise incompressible), so compression is spending CPU for little or no gain; that namespace is a +candidate for a faster algorithm or for disabling compression via an override. + +**PromQL:** +`avg(rate(payload_size_bytes_sum{stage="pre_compress"}[5m])) / avg(rate(payload_size_bytes_sum{stage="post_compress"}[5m]))` +by namespace. + +#### Compression error rate + +Compression or decompression errors cascade into request failures. Splitting by operation distinguishes an encode +problem from a decode problem (for example, a payload compressed by an algorithm an older build did not understand). + +**PromQL:** `rate(compression_ops_total{result="error"}[5m]) / rate(compression_ops_total[5m])` by operation. + +## Operational notes + +- **Configuration is validated at startup.** The proxy refuses to start on problems it can detect up front: a missing + `algorithm` while `enabled: true`, an unrecognized algorithm, a `level` outside the supported range, or a negative + `minBytes`. The goal is to surface foreseeable mistakes as a clear startup failure rather than a runtime error + mid-request. +- **Decoding is never gated by `enabled`.** Turning compression off stops new payloads from being compressed but does + not stop decompression, so disabling it never loses access to data already on the wire. +- **Restart on config change.** The proxy reads compression configuration at startup; changing the algorithm, level, + threshold, or overrides requires a restart. +- **Compression spends proxy CPU.** For typical payload volumes this is negligible, especially with LZ4, Snappy, or low + ZSTD levels, but it is real. Already-compressed or pre-encrypted data wastes CPU; the compression-ratio metric makes + that visible. +- **Compression runs before encryption.** When both are enabled the upstream pipeline is `compress -> encrypt` and the + downstream pipeline is `decrypt -> decompress`.