Session Type
Breakout Session
Name
Empowering Developers with a Centralized Kafka Library
Date
Tuesday, May 20, 2025
Time
3:30 PM - 4:15 PM
Location Name
Breakout Room 5
Description

This presentation details how our platform enablement team developed a centralized Kafka library, empowering developers to build applications with ease. Faced with inconsistent Kafka processing approaches across teams, we created a common library, inspired by the multi-threaded consumer approach described in this Confluent blog post: https://www.confluent.io/blog/kafka-consumer-multi-threaded-messaging (re-implemented in Kotlin). We'll share our challenges, successes, and the current state of this library, now used in over 20 services. Initially, varying team approaches to Kafka processing led to inconsistencies and duplicated effort. Our team recognized the need for standardization. Internal library simplifies Kafka development, promotes best practices, and centralizes key functionalities. It wraps the kafka-clients library, offering simple interfaces for building Kafka consumers and producers that integrate with our Confluent clusters, schema registry, and Avro serialization. Core feature is its multi-threaded consumer implementation, enabling efficient consumption from multiple partitions. We'll share the technical hurdles we encountered during development, discussing our design decisions, multi-threading challenges, and lessons learned. Crucially, the library supports dead-letter queues and message redelivery. It also supports cross-cluster consumers, essential for GDPR compliance, allowing production to multiple Confluent clusters in different regions. We'll cover our versioning strategy and package overlap issues, explaining how we created thin, relocated, and uber JAR versions. Interesting feature is runtime consumer control. By producing events to an internal topic, we can start/stop consumers in live applications without redeployment. The library has simplified Kafka development, promoted consistency, reduced the learning curve, and centralized core functionality. This presentation is ideal for Kafka developers seeking to build internal Kafka libraries.

Ademir Spahic
Level
Intermediate
Target Audience
Architect, Developer
Industry
Healthcare
Tags
Apache Kafka, Architecture, Event-Driven Systems, Microservices