Kafka Cluster of Cluster is a new solution Uber developed for Canary isolation. The strategy is to minimize the impact of failures caused by new deployments. The goal is to limit the "blast radius" of a bad deployment (of brokers, producers, or consumers) to a small subset of traffic, allowing for detection and mitigation before a catastrophic failure occurs.

Key Changes Proposed:

• Canary Brokers: A new broker metadata field called Pod will be introduced to identify a subset of brokers as "canary brokers". New cluster deployments should always start with these canary brokers to test the build with live, but limited, traffic.
• Canary Partitions: A new configuration, percent.canary.partition, will allow a specific percentage of a topic's partitions to be designated as "canary partitions". These partitions are placed exclusively on the canary brokers.
• Producers in a canary environment can implement a custom Partitioner to ensure their traffic is sent only to canary partitions.
• Consumers in a canary environment can implement a custom ConsumerPartitionAssignor to ensure they consume messages only from canary partitions.