Skip to content

Conversation

@grobinson-grafana
Copy link
Contributor

@grobinson-grafana grobinson-grafana commented Dec 8, 2025

What this PR does / why we need it:

This pull request adds a new abstraction downscalePermittedFunc which returns true if a downscale is permitted. It adds a function newChainedDownscalePermittedFunc that allows multiple funcs to be chained, and a newOffsetCommittedDownscaleFunc that checks if all records have been committed up to the end offset.

This prevents consumers from shutting down before processing all records in a partition as this would lead to data loss.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

@grobinson-grafana grobinson-grafana requested a review from a team as a code owner December 8, 2025 23:07
@grobinson-grafana
Copy link
Contributor Author

Tested with a new topic (partition has no records):

level=info ts=2025-12-08T22:45:55.966684Z caller=downscale.go:38 component=dataobj-consumer msg="no records produced for partition"

Tested with a topic that has uncommitted records:

level=info ts=2025-12-08T22:46:26.342909Z caller=downscale.go:59 component=dataobj-consumer msg="there are uncommitted offsets" last_committed_offset=-1 end_offset=2 delta=2

Tested with a topic that has all records committed:

level=info ts=2025-12-08T22:49:20.018399Z caller=downscale.go:50 component=dataobj-consumer msg="all offsets have been committed" last_committed_offset=3 end_offset=4

End offset is the offset of the next record (the last produced record + 1).

@grobinson-grafana grobinson-grafana force-pushed the grobinson/add-downscale-permitted-func branch 3 times, most recently from e7b1e99 to 7585547 Compare December 8, 2025 23:15
@grobinson-grafana grobinson-grafana force-pushed the grobinson/add-downscale-permitted-func branch 3 times, most recently from 143ce5d to 1e28454 Compare December 8, 2025 23:24
@grobinson-grafana grobinson-grafana changed the title feat: add DownscalePermittedFunc to check downscale is permitted feat: add downscalePermittedFunc to check downscale is permitted Dec 8, 2025
This commit adds a new abstraction downscalePermittedFunc which
returns true if a downscale is permitted. It adds a function
newChainedDownscalePermittedFunc that allows multiple funcs to
be chained, and a newOffsetCommittedDownscaleFunc that checks
if all records have been committed up to the end offset.

This prevents consumers from shutting down before processing all
records in a partition as this would lead to data loss.
@grobinson-grafana grobinson-grafana force-pushed the grobinson/add-downscale-permitted-func branch from 1e28454 to af7ae46 Compare December 8, 2025 23:30
@grobinson-grafana grobinson-grafana merged commit c0c27b3 into main Dec 9, 2025
70 checks passed
@grobinson-grafana grobinson-grafana deleted the grobinson/add-downscale-permitted-func branch December 9, 2025 22:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants