Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add 'not compressed' bit to LZ4 fragmented compressor so it can skip compression/decompression of small items #2564

Open
mykaul opened this issue Dec 3, 2024 · 1 comment

Comments

@mykaul
Copy link

mykaul commented Dec 3, 2024

Here's an example of a very inefficient compression - 29 bytes (incl. 4 bytes length field (length = 25), before compressed content), translate to 4 bytes of the compressed length = 21 bytes of compressed data, which end up decompressed into ... 21 bytes of data. We've gained practically nothing here.

We could probably use another bit (just like last_chunk_flag - https://github.com/scylladb/seastar/blob/665fed057d6ce08de4a78659403203bcce79b6cb/src/rpc/lz4_fragmented_compressor.cc#L61C27-L61C42 ) in order to let the receiver know we did NOT compress this blob, as it was worthless.
Removing another bit from the compressed content 4 bytes field (leaving 30 bits 'only') seem OK to me, but of course will need to be negotiated between sides.

Image

@mykaul
Copy link
Author

mykaul commented Dec 3, 2024

CC @michoecho

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant