How to Implement a DLQ with Brighter and Kafka

[romtur]

Hello Brighter team,

I am using the Brighter library for messaging with Kafka as the message broker. I want to implement a dead letter queue (DLQ) to handle messages that fail processing after a certain number of attempts.

I have searched through the documentation and existing discussions but couldn't find concrete information on whether Brighter natively supports DLQ with Kafka. Therefore, I have a few questions:

1. Is there built-in support for a dead letter queue in Brighter when using Kafka as the message broker?
2. If there isn't built-in support, could you provide guidance or a recommended approach on how to implement a DLQ with Brighter and Kafka?
3. Are there any best practices or patterns within Brighter that I should follow to ensure reliable and efficient handling of failed messages?

Thank you in advance for your assistance and for the excellent work on Brighter!

[jakoss]

For now you can create pipeline and your own custom implementation of DLQ for Kafka. I've opened feature request for official implementation here: #3660

Also, for now DLQ support in Brighter seems to be not fully functional: #3667