At HelloFresh, we process millions of events daily through Kafka, powering everything from recipe recommendations to delivery logistics. But our data teams faced a persistent challenge: making this streaming data queryable for analytics, AI/ML, and business intelligence without drowning in operational complexity.
This talk walks through our technical journey of building a self-service "Shareable Datasets" platform that automatically transforms Kafka topics into queryable Apache Iceberg tables in S3.
The Problem We Solved
Our legacy approach used Kafka Connect to sink data as Parquet files to S3. While functional, it created significant friction: partition management required manual intervention or expensive Glue Crawlers, schema evolution was painful, and small file accumulation degraded query performance. Data producers couldn't easily debug their own data, and downstream consumers lacked confidence in data freshness and quality.
Our Architecture
We evaluated multiple approaches—Glue Crawlers, custom Lambda functions, Airflow-based partition management, Flink/Spark—before committing to a Kafka Connect to Iceberg architecture. Key technical decisions included:
Kafka Connect with Iceberg Sink: Direct streaming from Kafka to Iceberg tables, handling schema evolution natively
AWS Glue Catalog: Primary metadata catalog with Unity Catalog federation for Databricks/Photon query engines
Automated Data Contracts: Schema validation, freshness checks, and row count monitoring triggered automatically when tables are created
Standardization Pipeline: Automated deduplication, timestamp normalization, and array flattening for analytics-ready data
Technical Depth
I'll cover the specific challenges we solved: handling at-least-once delivery duplicates, flattening nested Protobuf structures, managing schema evolution across 600+ topics, and implementing cost-effective table compaction. We'll examine our decision to use Kafka Connect over Apache Flink, the trade-offs involved, and how we handle backwards compatibility for legacy consumers.
What Audience Learn
Attendees will leave with:
A practical blueprint for migrating from Parquet-based sinks to Iceberg
Evaluation criteria for choosing between Kafka Connect, Flink,Spark and custom solutions.
Patterns for automated data quality enforcement on streaming data
Lessons learned from operating Iceberg at scale with 600+ Kafka topics
This is not a theoretical overview—it's a production case study from a company serving 7+ million customers globally, with concrete numbers on implementation effort and operational improvements.

By attending this session, your contact information may be shared with the sponsor for relevant follow up for this event only.