Kafka Consumer Crash Recovery: Resuming Processing Explained

Understanding Kafka Consumer Crash Recovery

When a Kafka consumer crashes, it raises important questions about where processing resumes. This topic is crucial for software engineers, especially in high-level design (HLD) interviews at FAANG companies.

Core Concept

The essential flow during a crash is:

1. CRASH - The consumer fails unexpectedly.
2. REASSIGN - Partitions are reassigned to other consumers in the group.
3. RESUME FROM CHECKPOINT - Processing resumes from the last committed offset.

Consumer Crash Scenario

Consider a Kafka topic with partitions:

• P0 | P1 | P2 | P3
  • Consumer A is processing records from P1.
  • If Consumer A crashes while processing a record, Kafka detects the failure and triggers a rebalance.

Partition Reassignment

Initially, the partition ownership is:

• Consumer A → P0, P1
• Consumer B → P2, P3

After the crash, the rebalance occurs:

• Consumer B → P0, P1, P2, P3

Where Does Processing Resume?

The key question is: where does processing resume? It depends on the committed offsets:

• Each consumer maintains a committed offset that acts as a recovery checkpoint.
• After a crash, the new owner of the partitions (e.g., Consumer B) may not start from the last processed record.

Important Considerations

1. Offset Management: The consumer's configuration determines how offsets are managed.
2. At-Least-Once Processing: If a record was processed but not committed before the crash, it may be processed again, leading to duplicates.
3. Potential Skipped Processing: If the offset is committed before processing, the consumer may skip that record after recovery.

Production Recovery Pattern

The recovery process generally follows these steps:

1. Consumer Crash
2. Group Rebalance
3. Partition Reassignment
4. Recover from Offset
5. Reprocess if Needed
6. Idempotent Side Effects

This ensures that the system can recover gracefully while maintaining data integrity.