π― Goal
Upgrade Eko's blob compression from standard gzip to zstd with adaptive compression levels and file-type detection.
π‘ Why
- zstd offers 3x faster decompression speed than gzip while achieving higher compression ratios (4β6x on text).
- Already compressed files (e.g.
.png, .zip, .mp4, .gz) waste CPU when compressed again.
- Text files (
.go, .js, .json, .md) compress extremely well.
π οΈ Implementation Steps
- Integrate
github.com/klauspost/compress/zstd in internal/objects/store.go.
- Add a file extension heuristic check: skip compression entirely for binary media/archive extensions (
.png, .jpg, .zip, .tar, .pdf).
- Use multi-threaded zstd encoding for blobs larger than 1MB.
- Benchmark save/restore speeds and disk size across text vs binary test suites.
π― Goal
Upgrade Eko's blob compression from standard
gzipto zstd with adaptive compression levels and file-type detection.π‘ Why
.png,.zip,.mp4,.gz) waste CPU when compressed again..go,.js,.json,.md) compress extremely well.π οΈ Implementation Steps
github.com/klauspost/compress/zstdininternal/objects/store.go..png,.jpg,.zip,.tar,.pdf).