Kafka Deep Dive: Topics, Partitions, Brokers, & Consumer Groups

In Apache Kafka, a Topic is a logical category or feed name to which records are published. Topics are divided into one or more Partitions:

• Physical Storage: A partition is an ordered, immutable sequence of records continuously appended to structured log files on disk (stored in ≈ 1GB segment files such as 00000000000000000000.log and indexed by .index and .timeindex).
• Unit of Parallelism: A single partition cannot be split across multiple broker nodes. Therefore, the number of partitions in a topic represents the maximum theoretical concurrency for write ingestion and consumer processing.
• Partition Keys & Deterministic Hashing: When a producer publishes a record with a key (user_12345), Kafka computes MurmurHash2(key) % num_partitions to assign the partition. All records sharing the same key are guaranteed to land on the identical partition in strict chronological order.

To guarantee high availability and prevent data loss upon hardware failure, Kafka partitions are replicated across multiple broker nodes according to the Replication Factor (typically 3 in production):

• Partition Leader: One broker is elected as the Leader for each partition. All producer writes and consumer reads route directly through the Leader.
• Follower Replicas: The remaining replicas actively pull records from the Leader over the network to replicate the log sequentially.
• In-Sync Replicas (ISR): The subset of follower replicas that are actively caught up with the Leader's log end offset. If a follower lags behind (controlled by replica.lag.time.max.ms = 30000), the Leader ejects it from the ISR pool.
• Producer Durability (acks=all):
  • acks=0: Producer does not wait for any broker acknowledgment (< 1ms latency, high data loss risk).
  • acks=1: Producer waits only for the local Leader to write to disk (2 - 5ms).
  • acks=all (or -1) with min.insync.replicas=2: The Leader writes to disk AND confirms replication to at least one ISR follower before returning success to the producer (10 - 20ms), guaranteeing zero data loss even if the leader crashes.

A Consumer Group represents a coordinated fleet of consumers pooling together to ingest data from a topic:

The Fundamental Cardinality Rule:

Within any single Consumer Group, each topic partition is assigned to AT MOST ONE active consumer instance.

• Scenario 1 (Partitions = Consumers): A topic has 6 partitions and the group has 6 consumers. Each consumer processes exactly 1 partition (100\% efficiency).
• Scenario 2 (Consumers < Partitions): A topic has 6 partitions and the group has 3 consumers. Each consumer is assigned 2 partitions.
• Scenario 3 (Consumers > Partitions): A topic has 6 partitions and the group has 10 consumers. 6 consumers process data, while 4 consumers sit completely idle as hot standbys.
• Independent Groups: Multiple distinct consumer groups (e.g., order-fulfillment and fraud-analytics) each receive their own independent stream of all messages from all partitions without competing with each other.

When a consumer joins or leaves a group (or crashes due to exceeding max.poll.interval.ms), Kafka triggers a Group Rebalance to reassign partition ownership:

• Eager Rebalancing (Legacy): All consumers stop processing, drop all partition assignments, rejoin the group coordinator, and receive new assignments. This causes a "stop-the-world" latency spike across the entire processing pipeline.
• Cooperative Sticky Rebalancing (Modern Kafka): Consumers only give up partitions that must be migrated to rebalance the load. Surviving consumers continue processing unaffected partitions seamlessly without global pipeline pauses.
• Heartbeat Thread: Consumers run a background heartbeat thread (heartbeat.interval.ms = 3000) sending liveness pings to the broker Group Coordinator. If no heartbeat is received within session.timeout.ms = 45000, the broker assumes the consumer is dead and initiates a rebalance.