Sakhanda Wire
NVDA $154.51 -1.04% MSFT $506.49 -2.22% GOOGL $281.27 -0.86% META $621.51 -0.78% AMZN $256.90 -0.33%
← Back to the news

Google releases open-source TPU Sync for KV cache transfers

Google Cloud has released TPU Sync, an Apache-2.0 C++ library for moving KV caches and model weights across Cloud TPU memory tiers. It treats TPU HBM, pinned host DRAM, shared storage and the data-center network as a single hierarchy, using asynchronous DMA while preserving the tensors’ TPU tile layout. Python wheels are available for PyTorch and JAX.

The library is designed for inference KV-cache movement and for broadcasting weights from trainers to samplers in reinforcement learning. It integrates with vLLM-TorchTPU, SGLang-TorchTPU, llm-d on GKE and native JAX. Storage backends, reinforcement-learning weight transfer and some cache-management functions remain experimental. The README does not yet recommend the project for general use.

In Google’s internal July 2026 tests on TPU v7x, Qwen3 235B KV-transfer latency fell from several seconds to about 230 milliseconds. Google reported more than 90% lower transfer overhead on 235B and 480B mixture-of-experts models, with average time to first token down 60% and 90%, respectively, at low concurrency. Prefix-caching tests showed up to 90% lower average time to first token. Google noted that results will vary in real-world deployments.

Written by Sakhanda Wire AI from reporting by MarkTechPost. Facts, quotes and full context are in the original article below.

Originally published by MarkTechPost on

Read the original on MarkTechPost ↗

The reporting and the image belong to MarkTechPost and are credited and linked above; the text on this page is Sakhanda Wire's own summary.

← Back to the news

More stories

All the latest news