Sakhanda Wire
NVDA $153.73 -1.54% MSFT $506.30 -2.26% GOOGL $280.09 -1.28% META $619.28 -1.14% AMZN $256.62 -0.44%
← Back to the news

Perplexity Open-Sources Lily Inference Engine for Apple Silicon

Perplexity has released Lily as open source, presenting it as the local inference engine used by Hybrid Compute in Perplexity Computer. The runtime is written in Rust and uses custom Metal kernels, without PyTorch or MLX in its execution path. A public demo in the pplx-garden repository provides greedy generation through a minimal OpenAI-compatible HTTP API.

Lily is designed specifically for Qwen3.6-35B-A3B on Apple silicon. Its 4-bit checkpoint occupies 19.4 GB, making a Mac with at least 32 GB of unified memory the practical minimum. The model activates roughly 3B parameters per token and combines mixture-of-experts routing with full-attention and Gated DeltaNet layers. The implementation fuses dequantization with matrix multiplication and keeps routing operations on the GPU.

In tests on a 40-core, 128 GB M5 Max, Lily averaged 4,156 prefill tokens per second and 170.0 decode tokens per second, compared with 3,388 and 126.4 for MLX-LM. That represents 1.23x higher prefill throughput and 1.35x higher decode throughput. A teacher-forced evaluation found perplexity 0.04% higher, with the same top-ranked token in 96.35% of positions.

Written by Sakhanda Wire AI from reporting by MarkTechPost. Facts, quotes and full context are in the original article below.

Originally published by MarkTechPost on

Read the original on MarkTechPost ↗

The reporting and the image belong to MarkTechPost and are credited and linked above; the text on this page is Sakhanda Wire's own summary.

← Back to the news

More stories

All the latest news