Kafka Offsets & Event Replay
Leverage log immutability: Commit offset internals (__consumer_offsets), auto vs manual commits, time-travel offset rewinds, and disaster recovery replay.
01.1. What is an Offset in Kafka?
An Offset is a monotonically increasing 64-bit integer assigned to each record as it is appended to a partition:
- Immutability: Once written to disk, a message's offset within its partition never changes.
- Log End Offset (LEO): The offset of the next record to be written to the partition leader.
- High Watermark (HW): The highest offset replicated across all In-Sync Replicas (ISR). Consumers can only read records up to the High Watermark to prevent reading uncommitted data that could be lost during leader failover.
- Tracking Read Progress: Unlike traditional message brokers that physically delete messages upon consumption, Kafka retains the data. Consumers merely advance an internal cursor pointing to the next offset to read.
Kafka Offset Reset & Historical Event Replay Workflow
Kafka Offset Reset & Historical Event Replay Workflow
Rewinding consumer group offset pointers to backfill or recover from software defects.
Unlock Topic #114: Kafka Offsets & Event Replay
You are viewing a preview. The full in-depth engineering deep dive, interactive simulators, architecture flowcharts, and self-assessment quizzes for this topic are available with Pro or Lifetime Access.
Failure modes, high-throughput bottlenecks, and real FAANG implementation decisions.
Interactive system topology diagrams, live parameter simulators, and downloadable SVG charts.
Staff-level multiple-choice quiz questions with instant feedback and answer explanations.
Firebase Google authentication automatically syncs your completed topics and quiz scores.
How clear and staff-actionable was this system breakdown?