With more than 100 million players worldwide and 5,000 game titles, PlayStation generates over 15,000 telemetry events per second — data that must be processed reliably, securely, and in real time. Historically, these gameplay events were handled by service-based Kafka consumers that became difficult to scale, extend, and operate under load.
To overcome these challenges, our team at Sony Interactive Entertainment built the Gameplay Pipeline — one of the first fully productionized Apache Flink jobs within PlayStation. This system consumes Kafka topics at massive scale and transforms raw telemetry into actionable, privacy-aware insights that power analytics and user-facing experiences. A key example is providing parents with accurate daily playtime summaries for their children, reinforcing healthy play habits and trust.
Technically, the pipeline implements exactly-once semantics with Flink checkpointing, Kafka offset management, and Cassandra persistence for long-term storage. It ensures idempotent processing, asynchronous enrichment and filtering, and precise session-splitting across local day boundaries — including time-zone and daylight-saving transitions. Additional production-grade features include batching and retry patterns, DLQ handling, and Datadog monitoring integrated with Flink metrics. With over 80% automated test coverage and fault-tolerant design, the system has become a blueprint for scalable, maintainable streaming across PlayStation.
Attendees will learn why we replaced microservice-based Kafka consumers with Flink’s stateful, checkpoint-driven architecture, how we managed state consistency and scaling at PlayStation scale, and what patterns proved critical for production readiness. The talk emphasizes real-world lessons from deploying Flink on top of Confluent Kafka, building operational resilience, and empowering new gameplay analytics use cases.
This work was also featured as a keynote at Flink Forward Barcelona 2025, and continues to shape PlayStation’s journey toward a unified, real-time data platform.
