4. Kafka Topic & Partition

Kafka Topics

• Kafka Will have default topic called __consumer_offsets

• Topic is Kafka component where producers are connected
• Publisher Publish a message in Kafka topic
• Kafka in Topic is multi subscriber i.e., Topic can have more than 1 consumers
• In Kafka Topic are logical entity, which means, when we say message is published to Topic, it means message published to PARTITIONS inside Topic

---

Kafka Topic Partition

• Topics is divided into multiple parts called PARTITIONS
• PARTITIONS can be considered as a linear Data structure same as ARRAY
• Message are actually stored in Topic Patition
• Every partition will have partition number
• Every partition has increasing index (similar to array index where it starts from 0 and goes on) called OFFSET
• New messages are always pushed at rear end
• Data which published is immutable (can't be changed) after publish
• We can have 1 r many partition in a single topic
• In Multi Broker Kafka Cluster , Partitions are distributed across whole cluster i.e., For Ex: P0(Partition) will be on B1(Broker1), P2 will be in B2, P1 & P3 will be in B#

---

Message Publishing

• 1Topic & 1Partition
  • Message will published
• 1Topic & 2Partition
  • Message will get randomly published by default ,without any pattern followed ( But we can control it i.e., we can send a message to only one Partition )
• 1Topic & 2Partition - redirecting message to particular partition
  • Partitioner method should contain following arguments
    • key_bytes
    • all_partition
    • available_partition