Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

async ingest should use SNS #861

Open
donaldgray opened this issue Jun 3, 2024 · 0 comments
Open

async ingest should use SNS #861

donaldgray opened this issue Jun 3, 2024 · 0 comments
Labels
API https://dlcs-book.readthedocs.io/en/latest/ engine sustainability Tech Stack

Comments

@donaldgray
Copy link
Member

To maintain backwards compatibility with Deliverator, when carrying out an async ingest the API contacts Engine by directly putting a message on a queue. While this mechanism works it is preferrable to use SNS and fanout to SQS. This will enable us to:

  • Have multiple listeners to ingest requests, this currently isn't used but could be in future.
  • Control which queues messages are fanned out to (e.g. could have different queues for different customers).

API has logic to workout which queue to add to - this could be separate SNS topics, or single SNS with an attribute fanning out to different SQS, or somewhere in between (e.g. an image, timebased and file queue with a 'priority' attribute).

@donaldgray donaldgray added sustainability Tech Stack API https://dlcs-book.readthedocs.io/en/latest/ engine labels Jun 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API https://dlcs-book.readthedocs.io/en/latest/ engine sustainability Tech Stack
Projects
None yet
Development

No branches or pull requests

1 participant