Duplicate events in streaming pipelines compromise data integrity, skew analytics, and inflate operational costs. Streaming systems, such as Apache Kafka, Apache Flink, Google Pub/Sub, and RabbitMQ, inherently face duplicate data challenges due to network retries, client failures, and the complexities of exactly-once processing. Duplicate data is a pervasive problem for all data-intensive systems, but practical guidance for solving these problems remains elusive.

This session consolidates the problem space into the go-to deduplication playbook. I will guide you through core concepts and practical strategies via a series of 11 structured episodes. We will cover deduplication challenges, event fingerprinting, storage-level options, idempotency, processing guarantees, consumer-side state management, inbox/outbox, checkpointing, two-phase commit, probabilistic structures, and multi-layer dedupe. Additionally, the series demonstrates how to apply one or more of these strategies across different streaming architectures.

You will master proven techniques for handling duplicate events in real-time systems through real-world scenarios, practical insights, and clear visualizations. This is a flexible toolkit for developers and architects building deduplicated, high-throughput streaming systems.