Session Type
Breakout Session
Name
Processing Exception Handling and Dead Letter Queue in Kafka Streams
Date
Wednesday, May 21, 2025
Time
10:00 AM - 10:45 AM
Location Name
Breakout Room 4
Description
A major concern when starting with Kafka Streams is how to handle (un)expected errors. Generally, you want to track these errors, identify the records that caused the failures, and possibly reprocess them. To achieve this, you often need to implement a custom try-catch mechanism and send these errors to a dedicated topic. Does this challenge sound familiar? Welcome aboard! At Michelin, we face it too.
For our own needs, we embedded this kind of error-handling mechanism in a home-made solution, but this solution has its limitations. Thus, we proposed two Kafka Improvement Proposals to enhance the Kafka Streams exception handling experience.
KIP-1033 introduces a new processing exception handler, complementing existing deserialization and production exception handlers. Now, any exceptions that occur during processing are caught and transmitted to the handler, allowing you to define your error-handling logic.
Complementary to this, KIP-1034 adds native support for routing failed records to a dead-letter queue topic of your choice.
By the end of this talk, you will walk away with the latest updates these KIPs bring, helping you build more robust Kafka Streams applications against processing errors with less effort.
Speakers


Level
Introductory
Target Audience
Architect, Data Engineer/Scientist, Developer
Tags
Apache Kafka, Kafka Streams, Stream Processing