Skip to content

Conversation

@andresilva
Copy link
Collaborator

@andresilva andresilva commented Dec 20, 2025

This PR update the storage crate to use CRC32C for checksumming. For 4 KB blocks, CRC-32C (Castagnoli) has higher guaranteed hamming distance than the classic CRC-32 (IEEE). Concretely, CRC-32C guarantees detection of any combination of up to 5 bit errors within a 4 KB block, whereas CRC-32 (IEEE) only guarantees detection of up to ~3-4 bit errors at that size. Both detect all burst errors up to 32 bits, but CRC-32C has fewer undetected multi-bit patterns beyond that.

I ran the full storage benchmark suite against main, crc_fast with CRC32 algorithm and crc_fast with CRC32C algorithm (this PR). Benchmarks were done on a AMD Ryzen 9 9950X3D.

main vs fast-crc32
main vs fast-crc32c (this PR)
fast-crc32 vs fast-crc32c

Closes #2589.

@andresilva andresilva changed the title [storage] create util for crc32 [storage] move to crc32c Dec 20, 2025
@andresilva andresilva added the breaking-format This PR modifies codec and/or storage formats. label Dec 20, 2025
@cloudflare-workers-and-pages
Copy link

cloudflare-workers-and-pages bot commented Dec 20, 2025

Deploying monorepo with  Cloudflare Pages  Cloudflare Pages

Latest commit: 4c340cb
Status: ✅  Deploy successful!
Preview URL: https://7c8d067e.monorepo-eu0.pages.dev
Branch Preview URL: https://storage-faster-crc.monorepo-eu0.pages.dev

View logs

@andresilva
Copy link
Collaborator Author

Created some benchmarks for the crates directly: https://andresilva.github.io/crc32-bench/report/

Benchmark code is here https://github.com/andresilva/crc32-bench.

@andresilva andresilva moved this to Ready for Review in Tracker Dec 22, 2025
@andresilva andresilva self-assigned this Dec 22, 2025
@andresilva
Copy link
Collaborator Author

andresilva commented Dec 23, 2025

crc-fast did a new release that makes it faster at small inputs (https://github.com/awesomized/crc-fast-rust/releases/tag/1.9.0):

@cloudflare-workers-and-pages
Copy link

cloudflare-workers-and-pages bot commented Dec 29, 2025

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
commonware-mcp 4c340cb Jan 01 2026, 05:00 PM

@andresilva andresilva closed this Jan 8, 2026
@github-project-automation github-project-automation bot moved this from Ready for Review to Done in Tracker Jan 8, 2026
@andresilva
Copy link
Collaborator Author

Replaced by #2740.

@codecov
Copy link

codecov bot commented Jan 8, 2026

Codecov Report

❌ Patch coverage is 96.15385% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.62%. Comparing base (c5c573e) to head (4c340cb).
⚠️ Report is 34 commits behind head on main.

Files with missing lines Patch % Lines
storage/src/crc32.rs 93.87% 3 Missing ⚠️
@@           Coverage Diff           @@
##             main    #2591   +/-   ##
=======================================
  Coverage   92.62%   92.62%           
=======================================
  Files         357      358    +1     
  Lines      102956   103007   +51     
=======================================
+ Hits        95366    95415   +49     
- Misses       7590     7592    +2     
Files with missing lines Coverage Δ
storage/src/freezer/mod.rs 99.70% <100.00%> (+<0.01%) ⬆️
storage/src/freezer/storage.rs 89.56% <100.00%> (ø)
storage/src/journal/contiguous/fixed.rs 97.40% <100.00%> (ø)
storage/src/journal/segmented/variable.rs 93.78% <100.00%> (ø)
storage/src/lib.rs 0.00% <ø> (ø)
storage/src/metadata/storage.rs 89.57% <100.00%> (ø)
storage/src/ordinal/mod.rs 99.64% <100.00%> (ø)
storage/src/ordinal/storage.rs 84.03% <100.00%> (ø)
storage/src/crc32.rs 93.87% <93.87%> (ø)

... and 3 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c5c573e...4c340cb. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking-format This PR modifies codec and/or storage formats.

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

[storage] Move to CRC-32C

3 participants