At Salesforce, we orchestrate one of the industry's largest data streaming footprints: over 500 Kafka clusters processing 3 trillion events per day and consuming 60 PB of data monthly. At this extreme scale, our Kafka Connect clusters often operate with consumer groups exceeding 1,000 members.

With group sizes this large, rebalances of consumer groups using standard dynamic membership pose a serious availability risk. When hundreds of tasks stop, rejoin, and sync simultaneously, this inevitable disruption can cause massive latency spikes and significant throughput degradation, making routine maintenance a high-risk operation. Even with Incremental Cooperative Rebalancing, a worker restart can trigger an extended rebalance for the underlying Task-level consumer group.

To solve this, we extended Static Membership to the Kafka Connect task level. By assigning persistent identities (group.instance.id) to ephemeral tasks, we prevent unnecessary rebalances during transient failures and rolling restarts. However, bringing KIP-345 principles to the Connect framework wasn't seamless. This session provides a candid technical deep dive into that journey. We will detail the implementation strategies used to generate unique static IDs for tasks and share the critical bugs we discovered and resolved in the process. We will discuss how we hardened our logic to ensure seamless rolling upgrades and persistent cluster stability. 

Attendees will leave with a deeper understanding of how Kafka Connect behaves when pushed to its architectural limits. By examining real failure modes that only appear at extreme scale, this session shows how small design decisions such as task identity and group membership can have an outsized impact on stability and operations. Whether you run tens of connectors or thousands, you will gain practical insight into the scaling challenges that emerge as systems grow and how to think about Kafka Connect resiliency before those limits are reached.