Session Type
Breakout Session
Name
Mastering real-time anomaly detection with open source tools
Date
Tuesday, May 20, 2025
Time
2:00 PM - 2:45 PM
Location Name
Breakout Room 2
Description
Detecting problems as they happen is essential in today’s fast-moving world. This talk shows how to build a simple, powerful system for real-time anomaly detection. We’ll use Apache Kafka for streaming data, Apache Flink for processing it, and AI to find unusual patterns. Whether it’s spotting fraud, monitoring systems, or tracking IoT devices, this solution is flexible and reliable. First, we’ll explain how Kafka helps collect and manage fast-moving data. Then, we’ll show how Flink processes this data in real time to detect events as they happen. We’ll also explore how to add AI to the pipeline, using pre-trained models to find anomalies with high accuracy. Finally, we’ll look at how Apache Iceberg can store past data for analysis and model improvements. Combining real-time detection with historical data makes the system smarter and more effective over time. This talk includes clear examples and practical steps to help you build your own pipeline. It’s perfect for anyone who wants to learn how to use open-source tools to spot problems in real-time data streams.
Olena Kutsenko
Level
Intermediate
Target Audience
Data Engineer/Scientist, Developer
Tags
Apache Flink, Apache Iceberg, Apache Kafka