Skip to content

Conversation

wjyao0316
Copy link
Contributor

Description

This commit add 5 minutes delay to partition creation on live event in DimensionalTimeSliceCrawler.

In general, newly generated events become queryable after 30 ~ 120 second. Delay 5 minutes give enough time for the newly generated events to become queryable to largely reduce the possibility of losing events due to eventual consistency in vender API side.

Issues Resolved

Resolves #[N/A]

Check List

  • New functionality includes testing.
  • New functionality has a documentation issue. Please link to it in this PR.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

I also manually run the test locally and verify the partitions covering all time range.

Logs I added

2025-09-25T10:33:15,046 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Creating worker partitions from 1758801600 to 1758805200
2025-09-25T10:33:15,048 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Creating worker partitions from 1758805200 to 1758808800
2025-09-25T10:33:15,048 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Creating worker partitions from 1758808800 to 1758812400
2025-09-25T10:33:15,049 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Creating worker partitions from 1758812400 to 1758816000
2025-09-25T10:33:15,050 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Creating worker partitions from 1758816000 to 1758819600
2025-09-25T10:33:15,051 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Creating worker partitions from 1758819600 to 1758821295
2025-09-25T10:34:15,056 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Creating worker partitions from 1758821295 to 1758821355
2025-09-25T10:35:15,068 [pool-7-thread-1] INFO  org.opensearch.dataprepper.plugins.source.source_crawler.base.DimensionalTimeSliceCrawler - Creating worker partitions from 1758821355 to 1758821415

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

… on live event

This commit add 5 minutes delay to partition creation on live event in
DimensionalTimeSliceCrawler.

In general, newly generated events become queryable after 30 ~ 120 second.
Delay 5 minutes give enough time for the newly generated events to
become queryable to largely reduce the possibility of losing events due
to eventual consistency in vender API side.

Signed-off-by: Wenjie Yao <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants