Hello
We are configuring MSK Connect and attempting to filter data based on the EventId using the MSK Aiven connector. The setup works when pushing data from a single topic to a single index. However, it does not work when we are trying to push data from a single topic to 2 OpenSearch indices. We have tried using both the DataStream and RegexRoute options of the Aiven transformation, but have not had success.
Our goal is to load data containing EventId 4688 into the first index, and data without EventId 4688 (including all other EventIds) into the second index.
However, the connector is creating and in OpenSearch, we are not able to see a single index from this connector.
I am attaching the connector configuration for reference.
Configuration
transforms=Filter4688,RouteByEventId4688,FilterOthers,RouteByEventIdOthers
transforms.Filter4688.type=io.aiven.kafka.connect.transforms.FilterByFieldValue$Value
transforms.Filter4688.field.value.matches=True
transforms.Filter4688.field.name=EventId
transforms.Filter4688.field.value.pattern=^4688$
transforms.RouteByEventId4688.type=org.apache.kafka.connect.transforms.RegexRouter
transforms.RouteByEventId4688.regex=.*
transforms.RouteByEventId4688.replacement=topic-json-poc-4688
transforms.FilterOthers.type=io.aiven.kafka.connect.transforms.FilterByFieldValue$Value
transforms.FilterOthers.field.value.matches=False
transforms.FilterOthers.field.name=EventId
transforms.FilterOthers.field.value.pattern=^4688$
transforms.RouteByEventIdOthers.type=org.apache.kafka.connect.transforms.RegexRouter
transforms.RouteByEventIdOthers.regex=.*
transforms.RouteByEventIdOthers.replacement=topic-json-poc-other
If anyone can help please share your input with us.
Hello
We are configuring MSK Connect and attempting to filter data based on the EventId using the MSK Aiven connector. The setup works when pushing data from a single topic to a single index. However, it does not work when we are trying to push data from a single topic to 2 OpenSearch indices. We have tried using both the DataStream and RegexRoute options of the Aiven transformation, but have not had success.
Our goal is to load data containing EventId 4688 into the first index, and data without EventId 4688 (including all other EventIds) into the second index.
However, the connector is creating and in OpenSearch, we are not able to see a single index from this connector.
I am attaching the connector configuration for reference.
Configuration
transforms=Filter4688,RouteByEventId4688,FilterOthers,RouteByEventIdOthers
transforms.Filter4688.type=io.aiven.kafka.connect.transforms.FilterByFieldValue$Value
transforms.Filter4688.field.value.matches=True
transforms.Filter4688.field.name=EventId
transforms.Filter4688.field.value.pattern=^4688$
transforms.RouteByEventId4688.type=org.apache.kafka.connect.transforms.RegexRouter
transforms.RouteByEventId4688.regex=.*
transforms.RouteByEventId4688.replacement=topic-json-poc-4688
transforms.FilterOthers.type=io.aiven.kafka.connect.transforms.FilterByFieldValue$Value
transforms.FilterOthers.field.value.matches=False
transforms.FilterOthers.field.name=EventId
transforms.FilterOthers.field.value.pattern=^4688$
transforms.RouteByEventIdOthers.type=org.apache.kafka.connect.transforms.RegexRouter
transforms.RouteByEventIdOthers.regex=.*
transforms.RouteByEventIdOthers.replacement=topic-json-poc-other
If anyone can help please share your input with us.