Session Type
Breakout Session
Name
Streaming Aggregations at Scale using Kafka and Kafka Streams
Date
Wednesday, March 19, 2025
Time
4:00 PM - 4:45 PM
Location Name
Scarlet 2
Description

Windowed aggregation on streams is a powerful concept for analytical data processing, especially in cases where waiting hours or even minutes for data to be available is inconceivable. While most people think of aggregations as an analytical requirement, they also help trim down data to size and can be critical to scaling systems without leading to ballooning costs. We had a similar use case in ShareChat where we had hundreds of thousands of counter increments (updates)/sec for everything from the number of views on a post to revenue numbers. Our databases could not keep up with the write volume, and there were frequent hot-spotting issues. Furthermore, the data was often inconsistent due to multiple writers, and taking locks further added to our misery. To solve this issue, we used Kafka and Kafka Streams to build an aggregation framework that can handle hundreds of thousands of counter increments per second. Streaming aggregation helped us batch updates and helped reduce the write throughput in our DBs. Further, it helped solve our hot-spotting issues and eliminated the need to take locks. This talk discusses the challenges of building the platform, managing an in-house Kafka setup, and the lessons learned from tuning Kafka Streams. Furthermore, we discuss how we optimised the solution to scale to even 1M updates/sec with zero hiccups or manual involvement. Today, the offering forms an integral part of our core streaming platform, and the talk will be helpful for developers who have similar requirements for streaming aggregations or want to know more about event-driven architectures using Kafka and Kafka Streams.

Shubham Dhal Shivam Yadav
Level
Intermediate
Target Audience
Architect, Developer
Industry
Advertising/Media, Technology
Tags
Apache Kafka, Event-Driven Systems, Kafka Streams, Stream Processing, Tales from the trenches