Dream11, the world’s largest fantasy sports platform, manages unparalleled scale, with RPM surpassing 300 million during flagship events like IPL 2024. With Kafka producers forming the backbone of real-time data pipelines, Dream11 faced a significant challenge: soaring cross-availability-zone (AZ) network costs due to the indiscriminate partitioning strategy of regular producer partitioners. To address this, Dream11 engineering developed the RackAwareStickyPartitioner, a custom solution for Kafka producers that achieved a 70% reduction in cross-AZ network costs. By intelligently routing producer batches to Kafka partitions within the same AZ, this innovation minimized cross-AZ traffic while preserving high throughput. A 10-day controlled experiment demonstrated a dramatic cost reduction in “DataTransfer-Regional-Bytes” by over 30%. This optimization is tailored for high-throughput scenarios, with careful consideration required for low-volume applications to avoid partition skew. Join this session to explore how Dream11 engineered a cost-efficient solution for Kafka producers at scale, sharing insights on architecture, challenges, and real-world impact.