Skip to content

Commit 4190cd2

Browse files
feat(eap-items): Point eap_items_dlq_replay at eap_items_2 tables (#8575)
Repoints the existing `eap_items_dlq_replay` storage from `eap_items_1_local` / `eap_items_1_dist` to `eap_items_2_local` / `eap_items_2_dist`. The main `eap-items-consumer` in us, de and s4s2 now writes `eap_items_2`, so DLQ replays should land there too. The schema and `UUIDColumnProcessor` columns are synced with `eap_items_2`, which adds `conversation_id`, `session_id` and `ai_conversation_id`. They had drifted from the primary storage. The storage name, the input topic (`snuba-dead-letter-items`) and the lack of a `dlq_topic` don't change. Because the name doesn't change, `eap_items_dlq_grace_period_min` still has no entry for it, so replayed rows aren't DLQ'd again. This applies everywhere the replay consumer runs with the default storage. Any replay consumer that's running when this deploys starts writing `eap_items_2` on its next restart. No ops change is needed for regions that use the default storage. This replaces #8570, which added a separate `eap_items_2_dlq_replay` storage instead. <!-- junior-request-attribution:start --> via **Ryan Broughan**. <!-- junior-request-attribution:end --> <!-- junior-session-footer:start --> <!-- junior-conversation-id:slack%3AC0CAC22GMKM%3A1792468188.049119 --> -- [View Junior Session](https://junior-prod.sentry.dev/conversations/slack%3AC0CAC22GMKM%3A1792468188.049119) [[Sentry]](https://sentry.sentry.io/explore/conversations/slack%3AC0CAC22GMKM%3A1792468188.049119/?project=4510944073809921) <!-- junior-session-footer:end --> Co-authored-by: sentry-junior[bot] <264270552+sentry-junior[bot]@users.noreply.github.com> Co-authored-by: Ryan Broughan <ryan.broughan@sentry.io>
1 parent d76765c commit 4190cd2

1 file changed

Lines changed: 7 additions & 4 deletions

File tree

‎snuba/datasets/configuration/events_analytics_platform/storages/eap_items_dlq_replay.yaml‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,9 @@ schema:
1616
{ name: item_type, type: UInt, args: { size: 8 } },
1717
{ name: timestamp, type: DateTime },
1818
{ name: trace_id, type: UUID },
19+
{ name: conversation_id, type: UUID },
20+
{ name: session_id, type: UUID },
21+
{ name: ai_conversation_id, type: String },
1922
{ name: item_id, type: UInt, args: { size: 128 } },
2023
{ name: sampling_weight, type: UInt, args: { size: 64 } },
2124
{ name: sampling_factor, type: Float, args: { size: 64 } },
@@ -112,15 +115,15 @@ schema:
112115
{ name: attributes_array_float, type: Map, args: { key: { type: String }, value: { type: Array, args: { inner_type: { type: Float, args: { size: 64 } } } } } },
113116
{ name: attributes_array_bool, type: Map, args: { key: { type: String }, value: { type: Array, args: { inner_type: { type: Bool } } } } },
114117
]
115-
local_table_name: eap_items_1_local
116-
dist_table_name: eap_items_1_dist
118+
local_table_name: eap_items_2_local
119+
dist_table_name: eap_items_2_dist
117120
partition_format: [retention_days, date]
118121

119122
query_processors:
120123
- processor: UniqInSelectAndHavingProcessor
121124
- processor: UUIDColumnProcessor
122125
args:
123-
columns: [trace_id]
126+
columns: [trace_id, conversation_id, session_id]
124127
- processor: HexIntColumnProcessor
125128
args:
126129
columns: [item_id]
@@ -144,7 +147,7 @@ mandatory_condition_checkers:
144147
field_name: organization_id
145148

146149
# Consumes from the eap_items DLQ topic and writes to the same ClickHouse
147-
# tables as the primary `eap_items` storage. Intentionally has no
150+
# tables as the primary `eap_items_2` storage. Intentionally has no
148151
# `dlq_topic`: messages that fail processing here will fail the consumer
149152
# loudly rather than getting silently re-DLQ'd back into the same topic.
150153
stream_loader:

0 commit comments

Comments
 (0)