[BUG] EventProcessorClient not consuming existing messages in the EventHub

[rajaramrajagopal-rogers]

Describe the bug
Created EventProcessorClient exactly as per the code mentioned in this page - https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-java-get-started-send to consume the messages in the EventHub. We have thousands of message already present in the EvenHub, but the EventProcessorClient is consuming only new event messages, not consuming the existing messages that are not consumed before. We tried with EventHub with 3 partitions and 3 consumer pods with same EventProcessorClient code.

Exception or Stack Trace
Add the exception log and stack trace if available

To Reproduce
Steps to reproduce the behavior:

• create a event hub with 3 partitions
• create a separate consumer group
• post at least 3000 event messages to eventhub distributed among all 3 partitions
• create EventProcessorClient as per instructions from the documentation page - https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-java-get-started-send?tabs=passwordless%2Croles-azure-portal#receive-events
• deploy the code to 3 pods/instances. Now we have 3 identical consumer pods ready to consume messages.
• If it is working fine the messages will be printed in logs but it is not consuming the existing messages.

Code Snippet
Exact replica of code in this sample except the eventhub and blob container details - https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/eventhubs/azure-messaging-eventhubs-checkpointstore-blob/src/samples/java/com/azure/messaging/eventhubs/checkpointstore/blob/EventProcessorBlobCheckpointStoreSample.java

also the lines 67 and 70 were commented out as per the Microsoft Support Engineer. Whole java files and logs are attached in the Support request ID - 2412200040008002

Java Code updated as per Microsoft Support Engineer suggestions which used for testing - EventHubReceiver.txt

Expected behavior
The existing event messages from eventhub should be consumed by EvenProcessorClient as per the partition ownership without overlap in consuming message. Each Pod should read from corresponding ownership claimed partition only.

Screenshots
If applicable, add screenshots to help explain your problem.

Setup (please complete the following information):

• OS: Windows 10 Enterprise

• IDE: IntelliJ

• Library/Libraries:
  implementation 'com.azure.spring:spring-cloud-azure-starter-eventhubs:5.18.0'
  implementation 'com.azure:azure-messaging-eventhubs:5.19.2'
  implementation 'com.azure:azure-identity:1.14.2'
  implementation 'com.azure:azure-messaging-eventhubs-checkpointstore-blob:1.20.2'

• Java version: 21.0.4

• App Server/Environment: Tomcat

• Frameworks: Spring Boot

If you suspect a dependency version mismatch (e.g. you see NoClassDefFoundError, NoSuchMethodError or similar), please check out Troubleshoot dependency version conflict article first. If it doesn't provide solution for the problem, please provide:

• verbose dependency tree (mvn dependency:tree -Dverbose)
• exception message, full stack trace, and any available logs

Additional context
Add any other context about the problem here. PFA logs from all 3 pods dev07-ov-buyflow-offline-service-rogers-5d6895697f-ghb94-logs.txt
dev07-ov-buyflow-offline-service-rogers-5fcc4bb589-w8wn2-logs.txt
dev07-ov-buyflow-offline-service-rogers-5fcc4bb589-r67mk-logs.txt
dev07-ov-buyflow-offline-service-rogers-5fcc4bb589-jfq95-logs.txt
dev07-ov-buyflow-offline-service-rogers-5d6895697f-n4wj2-logs.txt
dev07-ov-buyflow-offline-service-rogers-5d6895697f-hd48r-logs.txt

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Bug Description Added
• Repro Steps Added
• Setup information Added

[github-actions]

@anuchandy @conniey @lmolkova

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[conniey]

Looking at your code sample, it does not appear initialPartitionEventPosition(Map initialPartitionEventPosition) is set when creating the EventProcessorClient from the builder.

Does your checkpoint store have existing checkpoints?

If an existing checkpoint cannot be found, the default starting position, the end of the Event Hubs stream, is used.

[github-actions]

Hi @rajaramrajagopal-rogers. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text "/unresolve" to remove the "issue-addressed" label and continue the conversation.