GPU stack reuses LLM kernels for faster embedding service
Perplexity has described the serving infrastructure behind pplx-embed and ranking models used in Perplexity Search, Computer and the API Platform. The company says GPU embedding inference is largely standardized on mature Hopper and Blackwell hardware, making runtime software and request handling the main areas for improvement.
The system uses one engine for three workloads: batch embedding during index creation or re-indexing, low-latency online embedding for queries, and document scoring after vector search. Rather than creating a separate embedding engine, Perplexity reuses prefill and decode kernels from its large-language-model stack.
Three services divide the work. Ivy, a Rust HTTP gateway, parses requests, tokenizes inputs, splits large batches and balances chunks across replicas. Tulip is a Rust gRPC interface that schedules and batches requests on a first-come, first-served basis. ROSE performs inference and manages model definitions, kernels and CUDA graphs. Whole-model CUDA graphs reduce launch overhead, while LazyTensors allow CPU preparation for the next batch to overlap with GPU processing. The team says latency generally follows token count, and a sub-billion-parameter model reaches saturation at about 512 tokens.
Written by Sakhanda Wire AI from reporting by MarkTechPost. Facts, quotes and full context are in the original article below.