Session Type
Breakout Session
Name
Serverless Streaming Pipelines: Integrating Delta Live Tables, Kafka, and Serverless Architectures
Date
Wednesday, March 19, 2025
Time
3:00 PM - 3:45 PM
Location Name
Scarlet 2
Description

Real-time data processing is at the heart of modern applications, demanding scalability, reliability, and efficiency. This talk showcases how Delta Live Tables (DLT), Apache Kafka, and serverless architectures converge to create dynamic, end-to-end streaming pipelines without the need for version management. We’ll delve into: Advanced Use Cases: Learn how industries are leveraging DLT and Kafka to build resilient, real-time streaming workflows for applications. Versionless Pipelines: Understand how Databricks' automatic runtime upgrades for Delta Live Tables simplify maintenance while supporting platform enhancements. Optimization Strategies: Explore cost-effective, scalable design principles that serverless architectures bring to streaming workflows. Technical Deep-Dive: Discover how DLT handles checkpointing and data quality enforcement in streaming workflows, ensuring data reliability and fault tolerance. Interactive Demo: Witness a hands-on deployment of a zero-maintenance, serverless pipeline integrating DLT and Kafka for real-world data challenges. Attendees will be equipped with actionable insights and best practices to modernize their streaming workflows, reduce operational complexity, and unlock the full potential of real-time analytics.

Vikas Reddy Aravabhumi Shasidhar Eranti
Level
Intermediate
Target Audience
Architect, Data Engineer/Scientist, Developer
Industry
Energy/Utilities, Retail/E-Commerce, Transportation
Tags
Apache Kafka, Integration, Stream Processing