Kafka Replay Without Breaking Production: Safe Architecture

Introduction

In modern software systems, especially those handling large-scale data, replaying events from a message broker like Kafka can be crucial. However, doing this without disrupting live production traffic is a challenge. This guide discusses a safe architecture for replaying Kafka events.

The Problem with Direct Replay

When attempting to replay events, a common mistake is to reset the offsets of the production consumer group. This leads to:

• Mixing old events with live events.
• Operational complications and potential data inconsistencies.

Dangerous Design Example

Production Consumer Group
          ↓
      Reset Offset
          ↓
  10M Historical Events
          ↓
   Production Database

This approach can cause issues as it processes both old and new events simultaneously.

Safe Architecture for Replay

To avoid these pitfalls, it's essential to separate the live processing from the replay processing. Here’s how:

Use a Separate Replay Group

                 Kafka
                /     \
               ↓       ↓
        Production    Replay
           Group       Group
             ↓           ↓
        Live Flow    Replay Flow

• Independent Offsets: Ensure that the production offsets are not the same as the replay offsets. This allows the replay consumer to process historical records without affecting the production group's progress.

Avoid Direct Writes to Production

If the replay writes directly to the production systems, it can lead to unintended side effects. Instead, consider the following pattern:

Replay Group
     ↓
Replay Processor
     ↓
Staging / Derived Store
     ↓
Validation
     ↓
Controlled Cutover

• Staging: This is not always necessary but can help validate the replay before affecting production.

Control the Replay Rate

Replaying events at maximum speed can overwhelm systems. Implement a rate limiter:

Replay Consumer
      ↓
Rate Limiter
      ↓
Database

• Gradually increase the replay rate (e.g., from 1K/sec to 10K/sec) to protect live workloads.

Key Considerations

• Side Effects: Be aware that replaying events can trigger business actions like payments or notifications again. This can lead to duplicates if not handled properly.

Conclusion

By isolating replay processing and carefully managing how events are replayed, you can safely handle Kafka event replays without disrupting production traffic.