feat: implement lz4 compression #204
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* feat: configuring per-stream compressibility * doc: explain `is_compressible` config * feat: compression as an enum * fix: remove global `enable_compression` * refactor: `compress` ~> `lz4_compress`
de-sh
added a commit
that referenced
this pull request
May 15, 2023
* feat: implement lz4 compression (#204) * feat: implement lz4 compression * fix: compression should be opt-in * feat: allow configuring compressibility per-stream (#209) * feat: configuring per-stream compressibility * doc: explain `is_compressible` config * feat: compression as an enum * fix: remove global `enable_compression` * refactor: `compress` ~> `lz4_compress` * fix: broken merge * feat: allow configuring per-stream persistability (#207) * feat: allow configuring non-persistent streams * doc: add example config for non-persistent stream * fix: serializer test * fix: `is_persistable` ~> `persistance` * fix: normal ~> slow even if non-persistable data * fix: logs are always persisted * Merge branch 'next' into non-persist * fix: use compressed stream topic as is from config (#230) * feat!: per-stream persistence to disk (#229) * Guard log config in mac * Extract some methods into persistence struct * Finish storage module with optional persistence * feat: use `storage` instead of `disk` (#223) * fix: ensure test run successfully * feat: `use storage` not `disk` * feat: restructure persistence config and use * fix: serializer tests * refactor: remove old `disk` * feat: trace lost segment with debug log * style: clippy suggestion * fix: load previously persisted files * fix: comment out unused code * refactor: simplify serializer test * feat: add test to ensure files removed post flush (#228) * fix: add partial read file presence check * feat: completely per-stream persistance * fix: serializer metrics * doc: explain persist/compress configs * refactor: stream topic * fix: ensure all backlogs are cleared on catchup * refactor: unnecessary result * fix: default to `MAX_BUFFER_SIZE` * fix: add missing import * fix: append stream name to path in persistence * refactor: `StorageHandler` * fix: finish merge * doc: persistence config change * fix: remove unncessarily included config variable * fix: metrics updation * refactor: `StorageHandler.next()` returns `Option` --------- Co-authored-by: tekjar <[email protected]> * style: clippy suggestion --------- Co-authored-by: Devdutt Shenoi <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #
Add lz4 compression as a feature of uplink, decompression is already supported by beamd, other schemes can also be implemented once there are suitable de-compressor implementations on the beamd side.
Changes
Uplink now pushes data onto the same topic with an appended
/lz4to indicate the payload is compressed with the lz4 scheme.Compression is an opt-in feature per-stream and can be configured as follows(underlying usage of enums is done to support multiple schemes in the future, currently only Lz4 compression is supported):
Why?
Allows for much less network bandwidth use
Trials Performed
Pushing simulator device_shadow onto stage compressed with lz4 and checking it's presence within clickhouse proving that the compressed payload was decompressed and written into the database by beamd.
uplink
clickhouse