TokenGate Speeds Long-Context LLM Prefill by Up to 11.3x
Researchers from Stanford and UC Berkeley introduced TokenGate, a drop-in module that adds learned token pruning to pretrained large language models. The system lets an LLM process all input tokens through lower layers, then uses a small MLP to score them. Only the highest-scoring document tokens continue through upper layers; the question portion is retained. The researchers combined the gate with LoRA (r=16) on Qwen3-1.7B and Qwen3.5-2B.