Skip to content

Conversation

@wandna-amazon
Copy link

@wandna-amazon wandna-amazon commented Nov 28, 2025

add missing partition metrics to LeaderOnlyTokenCrawler

aligns LeaderOnlyTokenCrawler metrics with TokenPaginationCrawler as part of the connectors metrics audit (see details)

changes

• added WorkerPartitionWaitTime metric - tracks time partitions wait before processing
• added WorkerPartitionProcessLatency metric - measures actual partition processing time
• updated executePartition() method to record both metrics
• added comprehensive test coverage following existing patterns

benefits

• consistent metrics across both crawler implementations
• better visibility into retry partition performance
• improved observability for partition processing bottlenecks

test results

./gradlew :data-prepper-plugins:saas-source-plugins:source-crawler:test --tests "*testExecutePartitionMetrics*"

OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.8/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 7s
51 actionable tasks: 4 executed, 47 up-to-date

BUILD SUCCESSFUL in 7s 51 actionable tasks: 4 executed, 47 up-to-date

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant