feat: re-enable time-partitioning #1267
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
WalkthroughThis pull request refactors how time partitioning is represented and processed in event structures. The nullable Changes
Sequence Diagram(s)sequenceDiagram
participant Client as Client
participant HTTP as HTTP Ingest Handler
participant EventProc as Event Processor
participant TimeGen as Utc::now
Client->>HTTP: Submit event log
HTTP->>EventProc: Call push_logs_unchecked (with is_time_partitioned = false)
EventProc->>TimeGen: If is_time_partitioned is true, request current timestamp
TimeGen-->>EventProc: Return current timestamp
EventProc-->>HTTP: Processed event returned
HTTP-->>Client: Response sent
sequenceDiagram
participant Client as Client
participant Parser as Parseable Module
participant StreamMgr as Stream Manager
Client->>Parser: Request stream creation/update
Parser->>Parser: Validate time_partition_limit and custom_partition
Parser->>StreamMgr: Call create_stream with time_partition_in_days (if applicable)
StreamMgr-->>Parser: Stream creation/update response
Parser-->>Client: Return outcome of stream creation/update
Possibly related PRs
Suggested labels
Suggested reviewers
Poem
📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
⏰ Context from checks skipped due to timeout of 90000ms (10)
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
src/parseable/streams.rs (1)
147-148: Timestamp source changed from parsed_timestamp to current timeThe code now uses
Utc::now()instead of the providedparsed_timestampwhen creating the time range for theDiskWriter. This change is part of re-enabling time-partitioning and shifts from event time to ingestion time for determining partition boundaries.Consider adding a comment explaining the reasoning behind using the current time rather than the parsed timestamp to make the intent clearer to future developers.
+ // Use current time for partitioning to ensure consistent partition boundaries + // based on ingestion time rather than event timestamps let range = TimeRange::granularity_range(Utc::now(), OBJECT_STORE_DATA_GRANULARITY);
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
src/parseable/streams.rs(1 hunks)
🧰 Additional context used
🧬 Code Definitions (1)
src/parseable/streams.rs (1)
src/utils/time.rs (1)
granularity_range(267-282)
⏰ Context from checks skipped due to timeout of 90000ms (10)
- GitHub Check: Build Default aarch64-unknown-linux-gnu
- GitHub Check: Build Default x86_64-apple-darwin
- GitHub Check: Build Default x86_64-pc-windows-msvc
- GitHub Check: Build Default aarch64-apple-darwin
- GitHub Check: Build Kafka x86_64-unknown-linux-gnu
- GitHub Check: Build Default x86_64-unknown-linux-gnu
- GitHub Check: Build Kafka aarch64-apple-darwin
- GitHub Check: Quest Smoke and Load Tests for Distributed deployments
- GitHub Check: coverage
- GitHub Check: Quest Smoke and Load Tests for Standalone deployments
de-sh
commented
Mar 24, 2025
Signed-off-by: Devdutt Shenoi <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #XXXX.
Description
This PR has:
Summary by CodeRabbit