Allow topics to be defined in code

[vuldin]

I want to define my topics and associated details (how many partitions, etc.) in a config file that can be checked in to git and applied automatically after deployment.

It occured to me that maybe redpanda could provide this capability. One way this could be done is by adding new variables to the redpanda cluster operator config file found here:
https://raw.githubusercontent.com/vectorizedio/redpanda/dev/src/go/k8s/config/samples/one_node_cluster.yaml

The variables could be similar to the state config file from kafka-gitops:

 topics:
   test-topic:
     partitions: 6
     replication: 3

More details on their state.yaml here: https://devshawn.github.io/kafka-gitops/#/confluent-cloud?id=desired-state-file

I'm just interested in topics for now, but kafka-gitops has other capabilities that may or may not make sense to be a variable that can be sent to the redpanda operator.

Related slack discussion: https://vectorizedcommunity.slack.com/archives/C01AJDUT88N/p1638209789373700?thread_ts=1638209789.373700&cid=C01AJDUT88N

[emaxerrno]

Cc @vsaraswat i think this is a good idea.

We already have the admin api exposed. So we’ll need to think through the dev Experience here.

I think adding it to rpk would be good

@twmb and @mmaslankaprv

[vsaraswat]

+1 on facilitating gitops. @twmb any thoughts on where this fits within the rpk hierarchy? Maybe loading a config under rpk topic?

[twmb]

It makes sense to be under rpk topic, but it's not so clear exactly where. rpk topic create could take a bulk option, but we also would need to ensure the final state is as described (delete & recreate if the partitions need to be reduced, add partitions if necessary). I'm also not sure if we should be reproducing the job of terraform, and kafka-gitops describes itself as "It allows you to define topics and services through the use of a desired state file, much like Terraform".

[vuldin]

I'm also not sure if we should be reproducing the job of terraform, and kafka-gitops describes itself as "It allows you to define topics and services through the use of a desired state file, much like Terraform".

I think the author of kafka-gitops mentions terraform only because they wanted a tool that handles kafka configuration similar to how infrastructure-as-code tools (like terraform) handle infrastructure. As far as I know terraform doesn't handle kafka topics and other similar configuration, that's why kafka-gitops, julie, and other similar tools were created.

[emaxerrno]

makes sense. I basically see this as extending redpanda w/ a controller endpoint because you need a "reconciliation loop"

effectively what the user describes is the end state of the cluster not necessarily what the cluster has. so something (prolly the controller) needs to reconcile and work across network partitions, etc, where reconciliation may take a long time but eventually correct final state is guaranteed.

[vuldin]

Just to add some more context, other kafka implementations like Strimzi, Red Hat AMQ, and IBM Event Streams allow defining topic details via code through a kubernetes custom resource: https://github.com/strimzi/strimzi-kafka-operator/blob/f2ce3ea17c28846ad61640df55f46231a4cea17f/api/src/test/resources/io/strimzi/api/kafka/model/KafkaTopic.yaml

And quarkus takes a different approach allowing topic details to be defined in your app's application.properties (below defines a topic test with 2 partitions:

quarkus.kafka.devservices.topic-partitions.test=2

[twmb]

FWIW this is #1709

[zimbatm]

There is also a Terraform provider to configure the topics: https://github.com/Mongey/terraform-provider-kafka

[zimbatm]

Note that this approach doesn't always work because the deployer might not have access to Redpanda that's running inside of the k8s cluster.