Understanding Kafka Dead Letter Topics and Poison Messages

What is a Dead Letter Topic (DLT)?

A Dead Letter Topic (DLT) is a mechanism in Kafka used to handle messages that fail to process successfully after multiple retry attempts. Instead of endlessly retrying, these messages are sent to a DLT for further investigation.

What is a Poison Message?

A poison message is a message that continually fails to process. This can occur due to:

• Invalid payloads
• Schema incompatibility
• Unrecoverable business validation errors
• Unexpected data formats
• Application-specific processing failures

Example of a Poison Message

Imagine a message flow:

A → B → C → D
B ❌
↓
Retry
↓
B ❌
↓
Retry
↓
B ❌

Here, message B keeps failing, becoming a poison message.

The DLT Architecture

A typical flow with DLT looks like this:

MAIN TOPIC
    ↓
CONSUMER
    ↓
RETRY 1
    ↓
RETRY 2
    ↓
RETRY 3
    ↓
DLT
    ↓
INVESTIGATE
    ↓
REPROCESS

After exhausting retries, the message goes to the DLT for further investigation.

What to Include in the DLT?

When storing messages in a DLT, it’s important to include diagnostic information:

• Original Event
• Original Topic
• Partition
• Offset
• Failure Reason
• Retry Count
• Timestamp

For example:

topic = orders
partition = 3
offset = 184920
attempts = 3
error = InvalidCustomerState

Important Considerations

• DLT is not a solution: Just sending a message to the DLT does not mean the problem is solved. It requires monitoring, investigation, and controlled reprocessing.
• Reprocessing Risks: When reprocessing messages, especially those involving external systems (like payment APIs), ensure idempotency to avoid duplicate operations.

Production DLT Checklist

A robust DLT design should consider:

• Bounded retries
• Useful failure metadata
• Monitoring and alerting
• Retention policies
• Access control
• Controlled reprocessing
• Idempotency
• Throttling during replay
• Corrected consumer version

By implementing these practices, you can effectively manage failed messages in Kafka and maintain system reliability.