Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

storage: book keep dirty_ratio in disk_log_impl #24649

Open
wants to merge 4 commits into
base: dev
Choose a base branch
from

Conversation

WillemKauf
Copy link
Contributor

@WillemKauf WillemKauf commented Dec 23, 2024

TODO: Consider eviction, ducktape testing

The dirty ratio of a log is defined as the ratio between the number of bytes in "dirty" segments and the total number of bytes in closed segments.

Dirty segments are closed segments which have not yet been cleanly compacted- i.e, duplicates for keys in this segment could be found in the prefix of the log up to this segment.

Add book-keeping to disk_log_impl in order to cache both _dirty_segment_bytes as well as _closed_segment_bytes, which allows us to calculate the dirty ratio, and add observability for it in storage::probe.

In the future, this could be used in combination with a compaction configuration a la min.cleanable.dirty.ratio to schedule compaction.

Backports Required

  • none - not a bug fix
  • none - this is a backport
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v24.3.x
  • v24.2.x
  • v24.1.x

Release Notes

Improvements

  • Adds the observable metrics dirty_segment_bytes and closed_segment_bytes to the storage layer.

@WillemKauf WillemKauf force-pushed the dirty_ratio_compaction branch from c2a0252 to 018f7a2 Compare December 23, 2024 21:37
void update_dirty_segment_bytes(uint64_t bytes);

// Update the number of bytes in closed segments.
// Unlike _dirty_segment_bytes, this can only increase.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is currently true, but maybe it should be decremented during the case of retention enforcement/segment eviction.

@vbotbuildovich
Copy link
Collaborator

CI test results

test results on build#60092
test_id test_kind job_url test_status passed
gtest_raft_rpunit.gtest_raft_rpunit unit https://buildkite.com/redpanda/redpanda/builds/60092#0193f576-772e-4d29-8e01-3035f1e9661c FLAKY 1/2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants