At Atlassian, we have numerous inter-connected services whose shards are deployed on data centers across the globe. These services together participate in a complex stream-based ‘data migration’ workflow. To enable this, within the Platform org, we built a scalable, resilient and globally consistent Orchestrator. This Orchestrator leverages Kafka’s State store, Kafka streams and Kafka Connect. It provides a “service mesh” equivalent for Kafka-integrated services, enabling seamless coordination and communication between different “steps” of the workflow. The architecture allows different shards and nodes of services to enter and exit the service mesh, specify tenants, allocate usage-based quota to callers and so on. Central to our solution are 5 things: 1️⃣ A State store based context management that’s globally consistent 2️⃣ An SDK that services can readily integrate. This SDK seamlessly abstracts out Kafka for application developers. 3️⃣ A dynamic registry of services which not only catalogs the available services but also maintains an up-to-date map of service deployments across data centers, their health and their usage. 4️⃣ The orchestrator's intelligent routing algorithms that enable an application developer to seamlessly run a workflow that automatically resolves the most appropriate ‘data shard’ for each ‘step’ of the workflow, based on the application's requirements and the callee service’s constraints 5️⃣ A Kafka Connect based “message relay” service which handles cross data center message movement and which optionally provides exactly-once guarantees. Join us to explore the inner workings of this Orchestrator, how it leverages Kafka’s (possibly less popular) capabilities to address modern distributed stream-based data migration applications. We'll discuss real-world use cases from Jira, Confluence and other popular products of Atlassian and share our insights which can help you push the boundaries of what's possible with Kafka.