Session Type
Breakout Session
Name
Async Processing: Get any Kafka Streams app into the throughput Olympics
Date
Wednesday, May 21, 2025
Time
1:00 PM - 1:45 PM
Location Name
Breakout Room 6
Description

Since the very dawn of Kafka Streams it has been haunted by the possibility of high latencies in a topology. Custom logic with heavy processing, RPCs, and remote state stores were more or less considered out of the question, and those who bravely tried anyways ran into endless problems. At the same time, even simple applications were getting in trouble as companies grew and so did their workloads, yet apps could only scale up to the partition count. Are slow and/or heavy applications just not a good fit for Kafka Streams? The answer of course is no! With Kafka Streams anything is possible. In this talk we'll introduce Responsive's Async Processor, a lightweight wrapper that turns your slowest applications into medal winners. With just a few lines of code you can easily convert any Kafka Streams app to async. By injecting an async thread pool to hand over the actual processing work, records can be executed in parallel, even within a partition all without sacrificing any of the usual correctness guarantees like same-key ordering or exactly-one semantics. This feature is production-ready and available now through the Responsive SDK, but we'll end by discussing our vision for native async processing in open source Kafka Streams. We'll also go over some neat features we added to Kafka Streams like the ProcessorWrapper, which is used by the async framework but has the potential for so much more. Join us to hear all the gory details of async processing and how it reimagines the limits of Kafka Streams itself. Bring your slowest application and find out if it can make the team!

A. Sophie Blee-Goldman
Level
Intermediate
Target Audience
Architect, Developer
Industry
Technology
Tags
Kafka Streams, Stream Processing