Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Persist all new incoming Kafka records to an s3 bucket #1266

Open
dakota002 opened this issue Aug 21, 2023 · 1 comment · May be fixed by nasa-gcn/gcn-monitor#30
Open

Persist all new incoming Kafka records to an s3 bucket #1266

dakota002 opened this issue Aug 21, 2023 · 1 comment · May be fixed by nasa-gcn/gcn-monitor#30
Assignees
@dakota002

Comments

@dakota002
Copy link
Contributor

Set up a bucket and utilize kafka connect to store records until we set up a long-term db solution

https://docs.confluent.io/kafka-connectors/s3-sink/current/overview.html
@jracusin
Copy link
Contributor

jracusin commented Apr 1, 2024

Acceptance criteria

  • setup S3 bucket
  • save all notices in all formats including GCN Classic (voevent, text, binary) and JSON records distributed over Kafka

@dakota002 dakota002 mentioned this issue Aug 22, 2024
Closed
@dakota002 dakota002 self-assigned this Aug 22, 2024
@dakota002 dakota002 linked a pull request Sep 3, 2024 that will close this issue
Open
@dakota002 dakota002 linked a pull request Sep 3, 2024 that will close this issue
updates the consumer loop to put messages in the s3 bucket nasa-gcn/gcn-monitor#30
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dakota002 dakota002

Labels
None yet
Projects
GCN
Status: In Review
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

updates the consumer loop to put messages in the s3 bucket dakota002/gcn-monitor
2 participants
@jracusin @dakota002