Skip to content

perf: Implement Adaptive Compression (zstd with file-type heuristic & multi-threading)Β #124

Description

@kavix

🎯 Goal

Upgrade Eko's blob compression from standard gzip to zstd with adaptive compression levels and file-type detection.

πŸ’‘ Why

  • zstd offers 3x faster decompression speed than gzip while achieving higher compression ratios (4–6x on text).
  • Already compressed files (e.g. .png, .zip, .mp4, .gz) waste CPU when compressed again.
  • Text files (.go, .js, .json, .md) compress extremely well.

πŸ› οΈ Implementation Steps

  1. Integrate github.com/klauspost/compress/zstd in internal/objects/store.go.
  2. Add a file extension heuristic check: skip compression entirely for binary media/archive extensions (.png, .jpg, .zip, .tar, .pdf).
  3. Use multi-threaded zstd encoding for blobs larger than 1MB.
  4. Benchmark save/restore speeds and disk size across text vs binary test suites.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions