Session Type
Breakout Session
Name
Unifying Kafka and Relational Databases for Event Streaming Applications
Date
Wednesday, March 19, 2025
Time
1:00 PM - 1:45 PM
Location Name
Scarlet 3
Description

Kafka and relational databases have long been part of event-driven architectures and streaming applications. However, Kafka topics and database tables have historically been separate abstractions with independent storage and transaction mechanisms. Making them work together seamlessly can be challenging, especially because queuing has been viewed as an anti-pattern in a stock database. This talk will describe how to close this gap by providing a customized queuing abstraction inside the database that can be accessed via both SQL and Kafka’s Java APIs. Since topics are directly supported by the database engine, applications can easily leverage ACID properties of local database transactions allowing exactly-once event processing. Patterns such as Transactional Outbox (writing a data value and sending an event) or any atomicity required across many discrete database and streaming operations can be supported out of the box. In addition, the full power of SQL queries can be used to view records in topics and also to join records in topics with rows in database tables. In this talk we cover the synergy between Kafka's Java APIs, SQL, and the transactional capabilities of the Oracle Database. We describe the implementation, which uses a transactional event queue (TxEventQ) to implement a Kafka topic and a modified Kafka client that provides a single, unified JDBC connection to the database for event processing and traditional database access.

Mahesh Girkar Shailesh Dwivedi
Level
Intermediate
Target Audience
Architect, Developer, Executive (Technical)
Tags
Apache Kafka, Event-Driven Systems, Integration, Stream Processing