Issue with consumer group example

[zzhangsg]

Description

Context
From the example provided in this repository, a single consumer is created within the consumer group to process messages, and it gets assigned to all partitions of a Kafka topic. However, Kafka's design encourages leveraging partitioning to enable parallelism, which seems to favor multiple consumers within the same group for optimal resource utilization.

To align with Kafka's design and achieve partition-level parallelism, we are exploring a setup where N consumers are initialized within the same consumer group. This involves creating N clients and passing them to the NewConsumerGroupFromClient function with the same consumer group ID so that each consumer gets one partition.

Questions

1. Is this model valid?

• Can we create multiple consumers within the same group by initializing multiple clients and assigning them to the same consumer group ID?

2. Rebalancing and Partition Assignment:

• How does the partition rebalancing handling work when using this model?
• If the number of partitions increases or decreases, how are the changes reflected across the multiple consumers within the group?

Versions

Sarama	Kafka	Go

Configuration

Logs

logs: CLICK ME

Additional Context

[zzhangsg]

Hi @dnwe / @puellanivis, been browsing the past issues and saw that you are active contributors to this repo, could you please help share some insights when you get a chance? TIA!