Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deleting SQS Message when S3 file not found #30

Open
PaulinaSuwardi opened this issue Apr 26, 2018 · 4 comments
Open

Deleting SQS Message when S3 file not found #30

PaulinaSuwardi opened this issue Apr 26, 2018 · 4 comments

Comments

@PaulinaSuwardi
Copy link

We found an issue when receiving a message from SQS with S3 pointer but the file's has been deleted from S3 this library only throws an exception. However, we would like to delete the SQS message when this issue happen.
Getting SQS message/handler will help client to decide how to process the message in this case.

@stanpalatnik
Copy link

Also have this issue, and it's very annoying. The exception is thrown inside the library, so there's no way to handle this except for cloning this library.

@Damiox
Copy link
Contributor

Damiox commented Oct 25, 2018

You can always set up a dead letter queue to identify and isolate the wrong SQS messages.
I just created Pull Request #33 in case setting up a dead letter queue doesn't solve your issue.

@woldieJDI
Copy link

The issue to me is that the deletion of the S3 payload happens before the SQS message is deleted. If something goes wrong like a cosmic ray strike between the S3 payload deletion and the SQS message deletion, then we have a dangling pointer in the SQS message. When the SQS message redrives, it fails with a "S3 payload missing!" and the message goes to the DLQ.

I'd much rather you deleted the SQS message first and then make a best-effort delete attempt on the S3 payload second. Maybe we could set a generous TTL on the S3 payload so that it gets automatically cleaned up in the unlikely (but possible) event that the SQS message was deleted, but the S3 payload was not.

@onsah
Copy link

onsah commented Jul 1, 2022

Yes, I wish the delete order was at least configurable so one can avoid dangling pointer issue. That caused me to spend 1-2 hours of debugging just to find out nothing was actually wrong.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants