[feature] Consuming messages from redis channels

[semboko]

I have a process in my microservice, which emits messages and publish them into a redis channel. These messages are consumed by the other microservice, but at the same time they have to be sent via websockets to the browser application. Currently, I have to send the messages from the emitter into two places: the redis channel and centrifugo server via http.

Would be great to have an option to directly consume messages from redis channels or to somehow write such an extension.

[FZambia]

Hello @semboko !

BTW, could you provide more details which data structure in Redis you are using so that other microservices consume messages eventually. Is it PUB/SUB or queue?

Actually, in early versions Centrifugo integrated with Redis LIST to consume published messages and re-broadcast them to channels. But there were some issues with it: if the queue grow then subscribers could get non-actual real-time messages. For the real-time system (even soft real-time) this seems unacceptable. Synchronous API gives more control in this perspective.

Also, having such a queue does not allow handling errors easily. So eventually we removed this integration.

But, if thoughts above seem OK for your use case - then probably you can add one more microservice to your system which acts like your other microservices consuming Redis - but it's only task is to re-publish messages to Centrifugo using Centrifugo's HTTP/GRPC API. What do you think?

[semboko]

Hi, @FZambia! I use pub/sub redis channels. Your considerations about growing queue can actually be relevant to my system too. I need to think about that, thank you. Maybe my current configuration is just ok, since it works.

Adding another microservice is also an option. There are tools such as Benthos, which can do exactly this job - routing messages. But I am worrying about adding an additional layer of complexity and additional hub of communication. It also affects the predictability of the entire system. The native streaming to centrifugo seems to be a more reliable solution.

[FZambia]

So do I understand right that you want to publish into Centrifugo channel by publishing a message to Redis PUB/SUB channel? I.e. not a Redis-based queue?

I can't say I strongly against the idea as it definitely may have some advantages, but at the same time it breaks semantics of publishing and exposes Redis Engine internals to the public API.

Actually it's already possible to publish over Redis PUB/SUB by sending a message to the correct channel in a correct format which Centrifugo expects (Protobuf-based at the moment) – if Centrifugo node is subscribed to that channel publication will be received and broadcasted to active subscribers. But that's something we are not exposing publicly.

But I am cautious to proceed with the idea without real-world measurements and understanding the benefit – because the cost of adding this may overweight the benefit in long-term. This may also result into a situation when people decided to use the mechanism and then realize they want request response from Centrifugo or publishing with history - and it will be harder to adapt the publisher code to achieve this.

[FZambia]

Somewhat related post in our blog: Asynchronous message streaming to Centrifugo with Benthos