Name
Ending the Streaming Data Quality Nightmare: Building Real-Time Data Intelligence into Your Data Pipelines
Date
Wednesday, October 29, 2025
Time
9:00 AM - 9:45 AM
Location Name
Meet-up Hub
Description

The days of treating data quality as an afterthought are over. While streaming-first architectures have made real-time data more accessible than ever, most organizations still push data validation to downstream systems, creating a cascade of quality issues, broken pipelines, and emergency fixes.

What if your streaming data could validate itself? What if you could guarantee semantic correctness at ingestion, eliminating quality problems before they propagate throughout your entire data ecosystem?

In this session, Scott Haines will show you how to build intelligent streaming data pipelines that validate data meaning and structure in real-time, freeing your data engineering teams from the endless cycle of reactive fixes.

You'll discover:
* Why Protobuf is becoming essential for modern data engineering and how it enables semantic validation at scale
* How to implement real-time data quality checks using the open-source Protovalidate library
* Proven architectural patterns for embedding validation guardrails directly into Kafka streams
* Advanced techniques for semantic validation in zero-ETL Unity Catalog environments with Managed Iceberg

Walk away with a practical toolkit for building self-validating data pipelines that catch semantic errors at the source, dramatically reducing downstream quality issues and those dreaded 3 AM debugging calls.

Scott Haines