-
Notifications
You must be signed in to change notification settings - Fork 108
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Deleting SQS Message when S3 file not found #30
Comments
Also have this issue, and it's very annoying. The exception is thrown inside the library, so there's no way to handle this except for cloning this library. |
You can always set up a dead letter queue to identify and isolate the wrong SQS messages. |
The issue to me is that the deletion of the S3 payload happens before the SQS message is deleted. If something goes wrong like a cosmic ray strike between the S3 payload deletion and the SQS message deletion, then we have a dangling pointer in the SQS message. When the SQS message redrives, it fails with a "S3 payload missing!" and the message goes to the DLQ. I'd much rather you deleted the SQS message first and then make a best-effort delete attempt on the S3 payload second. Maybe we could set a generous TTL on the S3 payload so that it gets automatically cleaned up in the unlikely (but possible) event that the SQS message was deleted, but the S3 payload was not. |
Yes, I wish the delete order was at least configurable so one can avoid dangling pointer issue. That caused me to spend 1-2 hours of debugging just to find out nothing was actually wrong. |
We found an issue when receiving a message from SQS with S3 pointer but the file's has been deleted from S3 this library only throws an exception. However, we would like to delete the SQS message when this issue happen.
Getting SQS message/handler will help client to decide how to process the message in this case.
The text was updated successfully, but these errors were encountered: