Buoyant Releases Kinference Control Plane for vLLM on Kubernetes

Buoyant has released Kinference, an Apache 2.0-licensed open-source control plane for vLLM deployments on Kubernetes. The Rust-based project sits ahead of vLLM workers and uses declarative policies to route requests, apply token-rate limits, balance traffic using KV-cache data, and provide per-tenant metrics. The current release is v0.1.0-alpha.7 and is configured through five Kubernetes custom resources.

Kinference includes a traffic gateway and an inference handler with a gRPC interface. Its policies can consider tenants, models, GPU-pool state and semantic tags. Examples include routing simpler prompts to smaller models, sending bursts to external providers, keeping PII-related requests local, admitting batches only on lightly loaded workers, and exposing several models under one alias. The system predicts time to first token using cache overlap and queue length.

In Buoyant’s benchmark, three vLLM 0.31.0 pods each used one L40S GPU and served Qwen3.8-27B-FP8. Compared with random Kubernetes Service routing, median TTFT on shared-prefix traffic fell from 6.88 seconds to 0.31 seconds, while peak goodput increased from 14.0 to 103.0 requests per minute. The results are self-reported; traffic without shared prefixes showed no meaningful median TTFT improvement. Kinference remains alpha, with rate limiting and other planned capabilities not yet shipped.