Kafka Consumer Failure & Offset Behavior Demonstration

Before the consumer malfunctions offset behavior：

1. Initial Situation Recap:

1.1 Consumer Group: my-consumer-group-1

1.2 my-consumer-1 is responsible for:

• my-partition-0, having read up to offset=1 (value: 10)

• my-partition-1, having read up to offset=1 (value: A)

1.3 my-consumer-2 is responsible for:

• my-partition-2, having read up to offset=1 (value: aa)

2. Fault Scenario:

my-consumer-1 fails and can no longer consume messages.

Kafka will trigger a rebalance, and my-consumer-2 will take over the partitions originally handled by my-consumer-1 (i.e., partition-0 and partition-1).

3. Key question: What is the offset read by a new consumer?

Kafka's offset management has two approaches:

Case 1: Using automatic offset submission (default scenario)

• If my-consumer-1 successfully submits offset=1, the new consumer will start consuming from offset=1 (i.e., skip 10 and A, and begin consuming 20 and B).

• If my-consumer-1 has not yet submitted the offset (e.g., it has read 10 and A but not submitted the offset), the new consumer may start reading from offset=0 (i.e., re-read 10 and A again), depending on the settings of enable.auto.commit and auto.offset.reset.

Case 2: Using manual offset submission

• If the application has used commitSync() to submit offset=1, the new consumer will start from offset=1.

• If the manual submission was unsuccessful, it will re-read the uncommitted offset (i.e., offset=0).

4. Conclusion is as follows

Assuming my-consumer-1 has read offset=1 (10 and A) but has not yet committed the offset, then:

• After a rebalance, the new consumer will take over partition-0 and partition-1.

• The new offsets will be:

  • partition-0 → offset=0 (re-read 10)

  • partition-1 → offset=0 (re-read A)

However, if my-consumer-1 successfully committed the offset before crashing, then the new consumer will start from offset=1.

After the consumer malfunctions offset behavior：