Real-time systems don’t just process data — they accumulate it. As datasets grow, teams must balance low-latency “hot” data with cost-efficient “cold” storage, without sacrificing correctness or performance.

In this talk, we’ll share a practical architecture combining Apache Flink for streaming, Apache Spark for historical processing, Apache Iceberg as the cold data lake layer, StarRocks as a unified OLAP and federation engine, and Airflow for orchestration. We’ll explore how to tier data effectively, age it safely, handle late events, deduplicate across batch and streaming boundaries, and maintain strong query performance across hot and cold layers. We’ll focus on real-world challenges, trade-offs, and architectural patterns for building scalable streaming-first data platforms.