New distillation method cuts memory and cost for large language models
Multiverse Computing researchers have presented a knowledge-distillation method designed to reduce the memory and computing requirements of compressing large language models. The approach separates teacher-model processing from student training by calculating the teacher’s 100 most likely tokens for each position once and storing them for reuse.
The accompanying fused, chunked Kullback–Leibler loss processes sequences in sections rather than creating a full vocabulary-by-sequence matrix. It also avoids retaining the student’s complete logits grid, recomputing each section during backpropagation instead. According to the authors, this makes peak memory scale more gradually with context length.
In a single-H200 test using Llama 3.1 8B Instruct as teacher and a 3.2B Llama model as student at 8K context, the fused method used 58.3 GB of memory, compared with 102.8 GB for online distillation. In a 32K-token GPT-OSS 20B experiment, the setup was reduced from four GPU nodes to one, while step time dropped from 57.0 to 12.23 seconds. The team says the resulting smaller model retained most of the teacher’s accuracy on BoolQ and HellaSwag and remained within about nine points on MMLU. The implementation has been open-sourced.
Written by Sakhanda Wire AI from reporting by Hugging Face. Facts, quotes and full context are in the original article below.