native topic mirroring

[easel]

I'm in the process of migrating from Kafka to Redpanda. During the transition, I'll need to run kafka mirrormaker, which is generally pretty inconvenient to configure, inconvenient to run, and not that fast. A native "rpk subscribe" command would be super cool.

I have two use cases:

1. Mirroring topics from Apache Kafka. The subscriber would need to be built on librdfkafka or similiar in order to be able to connect as a real kafka consumer. This lowers the barrier to entry for trying out redpanda, perhaps as a wasm transform engine, read mirror for analytics or for permanent migration.

2. Mirroring topics from RedPanda. I have a primary and secondary datacenters, and need to mirror some topics between them so I can consume them for analytics. In this scenario both datacenters are running redpanda, so a native subscription could be optimized to take advantage of this. It would be pretty cool if this happened automatically if it notices that the source broker is running redpanda.

An example of a similar capability is the clickhouse kafka engine: https://clickhouse.tech/docs/en/engines/table-engines/integrations/kafka/

You basically just create tables from topics and clickhouse manages librdfkafka for you and pulls the data down.

Conceivably, similar connectors for other services like kinesis or pulsar would be interesting too.

[emaxerrno]

@BenPope - would be interesting for us to brainstorm how to use the client you are writing and expose some kind of sync process that does core-to-core mapping (works w/ kafka, optimized for redpanda)

[emaxerrno]

Following up on here after more discussions with @easel - the goals for mm2-style workload should be about throughput saturation with bandwidth constraints and not latency.

we can do something similar to our dirty index tracking for our catch up workload that @mmaslankaprv did for raft.