feat: Add SQS message processing components #270
Conversation
Add SQS utilities for delta production: - SqsBatchProcessor: Batch parsing, validation, and filtering - SqsMessageOperations: SQS receive/delete operations and queue attributes - SqsMessageParser: Parse and sort SQS messages by timestamp - SqsParsedMessage: Data class for parsed opt-out messages - SqsWindowReader: Reads messages for complete 5-minute time windows
| JsonObject body = new JsonObject(message.body()); | ||
| String identityHash = body.getString("identity_hash"); | ||
| String advertisingId = body.getString("advertising_id"); | ||
| String traceId = body.getString("trace_id"); |
There was a problem hiding this comment.
I think we also need the uid_trace_id for helping the debug
There was a problem hiding this comment.
We store uid_trace_id into trace_id on the sqs message
There was a problem hiding this comment.
Why are we writing uid_trace_id into the trace_id field on the SQS messages? will it be confusing?
Should the optout service have its own trace_id, with uid_trace_id acting as the end-to-end identifier across the whole flow (tc-portal → operator → optout)?
There was a problem hiding this comment.
I am not sure if we need an opt-out-specific ID. The sqs messageID allows us to correlate the enqueue / dequeue operations already, and no harm in reusing the tcportal/operator trace right? Maybe we just rename the field uid_trace_id?
Ian-Nara
force-pushed
the
ian-UID2-6345-sqs-components
branch
from
d9e2641 to
6e9fa46
Compare
Ian-Nara
force-pushed
the
ian-UID2-6345-sqs-components
branch
from
6e9fa46 to
bbda3f2
Compare
Add new test files for SqsMessageOperations and SqsWindowReader
…p reason, not boolean)
Add SQS utilities for delta production: