-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AIO version of EventHubConsumerClient
has significant memory leak
#36398
Comments
Thank you for your feedback. Tagging and routing to the team member best able to assist. |
Hi @smoke - Thanks for opening an issue! Are you using websocket, i.e. passing in TransportType.AmqpOverWebsocket? Or are you using the sample as is without any modifications? |
Hi @smoke. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue. |
Sample as is, with added few lines to send the events through |
@smoke Would you also be able to provide debug level logging to see if the memory jumps correlate with underlying errors?
Pass in |
Hi @smoke. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue. |
This memory leak is also impacting my application (forwarding large JSON audit events from Microsoft Entra to Azure Event Hub). I reproduced the memory leak issue in a scratch environment. To do so, I wrote 200MB of dummy JSON data to a new event hub and downloaded it using the async reference implementation that Smoke linked in the issue. Debug logs from the receiver are attached. Unfortunately, the Sync implementation doesn't work in my environment due to a different issue with our dual-stack TCP outgoing proxy. Please let me know if I can provide any additional context or logging! |
Please find the log file with debug logger debug.log.tar.gz Here a graph of the memory usage that clearly shows the memory leak: Notice that to have realistic scenario I have used the following async def on_event(partition_context, event):
# Put your code here.
# If the operation is i/o intensive, async will have better performance.
print("Received event from partition: {}.".format(partition_context.partition_id))
### changed code starts ###
print('Simulating work') # noqa: T201
await asyncio.sleep(30000)
### changed code finishes ###
await partition_context.update_checkpoint(event) Update: I have tested with code that does not have any additional code and there is no memory leak, apparently the problem is when there is some async code inside the Here the pristine code log |
Hi @smoke, A few questions:
|
|
Hi @smoke we were able to reproduce your issue and are looking into a fix, will update when we have more information. |
Describe the bug
AIO version of EventHubConsumerClient has significant memory leak, when async code is used inside
on_event
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Long running events processing to not leak memory.
Screenshots
![image](https://private-user-images.githubusercontent.com/196165/347021999-ad8b69a3-f8c8-43a7-bb45-25338e080d8d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjMxMDc1OTUsIm5iZiI6MTcyMzEwNzI5NSwicGF0aCI6Ii8xOTYxNjUvMzQ3MDIxOTk5LWFkOGI2OWEzLWY4YzgtNDNhNy1iYjQ1LTI1MzM4ZTA4MGQ4ZC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODA4JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgwOFQwODU0NTVaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT03NTE2Mjk3N2UxNzMzODgxMzgwOWZhMmUwMjkxZWZhZTc3ZjJhNTY4Y2E5OTdlN2NlYzNmYjE4MWI0Mjk2MzAzJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.nyb9ey6rSqu9qVyACNaCJ_zTv7mj5p7NojvHNZYi7ow)
AIO version memory usage over 12h:
(note the spike downs are POD restarts due to OOM killer)
Sync version memory usage over 12h:
![image](https://private-user-images.githubusercontent.com/196165/347021546-1cb7bd8b-6dfe-47cb-9062-a2eeff1cba91.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjMxMDc1OTUsIm5iZiI6MTcyMzEwNzI5NSwicGF0aCI6Ii8xOTYxNjUvMzQ3MDIxNTQ2LTFjYjdiZDhiLTZkZmUtNDdjYi05MDYyLWEyZWVmZjFjYmE5MS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwODA4JTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDgwOFQwODU0NTVaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1lNTVkY2U4OTI4ZDJjMjNjYmY4YWU2ZmZlYmEwMTE5MDI1MjJkMTA2ZGRhZjE1YzMzOTBhOGI0YzhhZTViNjFmJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.nECz81ro9B8v16atJneYyESHPYSgrbeNRqow5RhnP-w)
Additional context
For example the sync implementation has no issues at all https://github.dev/Azure/azure-sdk-for-python/tree/main/sdk/eventhub/azure-eventhub/samples/sync_samples/recv.py
The text was updated successfully, but these errors were encountered: