As organizations race to deploy Large Language Models, a critical infrastructure challenge emerges: how do you run a 70B+ parameter model when it exceeds your available hardware? The standard answer—buy bigger GPUs—isn't always feasible due to cost, availability, or deployment constraints. This talk presents an alternative: pipeline-parallel inference using Apache Kafka as the coordination layer, turning existing distributed infrastructure into an LLM serving platform.

The core architecture partitions transformer layers across multiple nodes, where each node holds a fraction of the model weights. Inference flows as a streaming pipeline: Node 1 processes the first N layers and publishes the resulting hidden state tensor to a Kafka topic. Node 2 consumes this message, processes the next N layers, and publishes downstream—continuing until the final node produces the output token. This pattern transforms Kafka from a data streaming platform into a distributed compute orchestrator for AI workloads.

This is experimental work exploring the boundaries of what's possible. CPU-only inference introduces significant latency—seconds per token rather than milliseconds—making it impractical for real-time applications. However, the architecture becomes compelling when nodes are equipped with consumer-grade GPUs. A cluster can collectively run models that would otherwise require expensive datacenter hardware. The Kafka coordination overhead becomes negligible compared to GPU compute time.

Why Kafka specifically? The benefits extend beyond message delivery: built-in backpressure handles heterogeneous node speeds, topic partitioning enables multi-tenant model serving, consumer groups provide horizontal scaling, and existing observability tools monitor inference pipelines without additional instrumentation.