Google releases open-source TPU Sync for KV cache transfers
Google Cloud has released TPU Sync, an Apache-2.0 C++ library for moving KV caches and model weights across Cloud TPU memory tiers. It treats TPU HBM, pinned host DRAM, shared storage and the data-center network as a single hierarchy, using asynchronous DMA while preserving the tensors’ TPU tile layout. Python wheels are available for PyTorch and JAX.
The library is designed for inference KV-cache movement and for broadcasting weights from trainers to samplers in reinforcement learning. It integrates with vLLM-TorchTPU, SGLang-TorchTPU, llm-d on GKE and native JAX. Storage backends, reinforcement-learning weight transfer and some cache-management functions remain experimental. The README does not yet recommend the project for general use.
In Google’s internal July 2026 tests on TPU v7x, Qwen3 235B KV-transfer latency fell from several seconds to about 230 milliseconds. Google reported more than 90% lower transfer overhead on 235B and 480B mixture-of-experts models, with average time to first token down 60% and 90%, respectively, at low concurrency. Prefix-caching tests showed up to 90% lower average time to first token. Google noted that results will vary in real-world deployments.
Written by Sakhanda Wire AI from reporting by MarkTechPost. Facts, quotes and full context are in the original article below.