[compress]: Add payload compression algorithms#59
Open
pseudomuto wants to merge 1 commit into
Open
Conversation
The payload compression RFC (#50) settles on LZ4, ZSTD, and Snappy as the algorithms the proxy will support. This adds the package implementing them so later work (metadata tagging, config, wiring into the proxy) has a concrete compressor to build on. Each algorithm is a small type (LZ4, Zstd, Snappy) exposing the same shape: an Encoding method returning the content-encoding identifier that will drive decompression, plus Compress/Decompress over whole in-memory buffers. All treat their input as read-only and return freshly allocated output, which keeps them safe for concurrent use.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #59 +/- ##
==========================================
+ Coverage 85.12% 85.50% +0.37%
==========================================
Files 33 36 +3
Lines 1042 1090 +48
==========================================
+ Hits 887 932 +45
- Misses 131 132 +1
- Partials 24 26 +2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
This PR introduces a new pkg/compress package that provides in-memory buffer compression/decompression primitives (LZ4, ZSTD, Snappy) intended to be wired into the Temporal proxy’s payload pipeline in later work.
Changes:
- Added
LZ4,Zstd, andSnappycompressor implementations with a shared “Encoding + Compress/Decompress” shape. - Added a common compressor test suite plus per-algorithm unit tests.
- Added module dependencies for LZ4, ZSTD, and Snappy implementations.
Reviewed changes
Copilot reviewed 9 out of 10 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/compress/doc.go | Package documentation for the compressor primitives. |
| pkg/compress/lz4.go | Adds LZ4 compressor implementation. |
| pkg/compress/lz4_test.go | Adds LZ4 tests (suite + encoding test). |
| pkg/compress/snappy.go | Adds Snappy compressor implementation. |
| pkg/compress/snappy_test.go | Adds Snappy tests (suite + encoding test). |
| pkg/compress/zstd.go | Adds ZSTD compressor implementation with configurable level. |
| pkg/compress/zstd_test.go | Adds ZSTD tests, including constructor level coverage. |
| pkg/compress/compressor_test.go | Adds shared behavioral test suite for compressors (round-trip, invalid input, concurrency, etc.). |
| go.mod | Adds compression library dependencies. |
| go.sum | Adds checksums for the new dependencies. |
Comment on lines
+18
to
+21
| // Encoding returns the content-encoding identifier for Snappy-framed data. | ||
| func (c *Snappy) Encoding() string { | ||
| return "application/x-snappy-framed" | ||
| } |
Comment on lines
+29
to
+30
| // Decompress returns the decoded form of Snappy-encoded data, or an error if | ||
| // data is not a valid Snappy frame. It does not modify data. |
| func TestSnappyEncoding(t *testing.T) { | ||
| t.Parallel() | ||
|
|
||
| require.Equal(t, "application/x-snappy-framed", compress.NewSnappy().Encoding()) |
Comment on lines
+37
to
+40
| // Encoding returns the content-encoding identifier for Zstandard data. | ||
| func (c *Zstd) Encoding() string { | ||
| return "application/zstd" | ||
| } |
Comment on lines
+48
to
+49
| require.Equal(t, "application/zstd", c.Encoding()) | ||
| } |
Comment on lines
+21
to
+24
| // Encoding returns the content-encoding identifier for LZ4-framed data. | ||
| func (c *LZ4) Encoding() string { | ||
| return "application/x-lz4" | ||
| } |
| func TestLZ4Encoding(t *testing.T) { | ||
| t.Parallel() | ||
|
|
||
| require.Equal(t, "application/x-lz4", compress.NewLZ4().Encoding()) |
| func NewZstd(level int) (*Zstd, error) { | ||
| enc, err := zstd.NewWriter(nil, zstd.WithEncoderLevel(zstd.EncoderLevelFromZstd(level))) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("failed to create Zstd encoder level: %d, %w", level, err) |
Comment on lines
+4
to
+7
| // Each compressor ([Snappy], [Zstd], [LZ4]) exposes the same shape: an Encoding | ||
| // method returning the associated HTTP/gRPC content-encoding identifier, and | ||
| // Compress/Decompress methods that operate on whole in-memory buffers. Both | ||
| // treat their input as read-only and always return a newly allocated result. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The payload compression RFC (#50) settles on LZ4, ZSTD, and Snappy as the algorithms the proxy will support. This adds the package implementing them so later work (metadata tagging, config, wiring into the proxy) has a concrete compressor to build on.
Each algorithm is a small type (LZ4, Zstd, Snappy) exposing the same shape: an Encoding method returning the content-encoding identifier that will drive decompression, plus Compress/Decompress over whole in-memory buffers. All treat their input as read-only and return freshly allocated output, which keeps them safe for concurrent use.