llm-d 1.0 adds cache-aware routing for distributed AI inference
Google, Red Hat and IBM have released llm-d 1.0, the first stable version of an open-source, Kubernetes-native stack for distributed large-language-model inference. It coordinates vLLM and SGLang servers into a shared pool for self-hosted agent and reinforcement-learning workloads. The project is a CNCF Sandbox initiative, uses the Apache 2.0 license and has more than 500 contributors from over 25 organizations.