Sakhanda Wire
NVDA $154.51 -1.04% MSFT $506.49 -2.22% GOOGL $281.27 -0.86% META $621.51 -0.78% AMZN $256.90 -0.33%
← Back to the news

Hugging Face adds GGUF support to Transformers

Hugging Face has added support for running GGUF models in Transformers, allowing users to load quantized checkpoints through the familiar Python and PyTorch APIs. The feature is initially focused on local inference on Apple Silicon and the Qwen3.5 architecture. Users can select a GGUF file from the Hub and pass its repository and filename to from_pretrained.

GGUF packages model weights and metadata in one file and offers quantization levels that reduce memory use at the cost of some precision. For Unsloth's Qwen3.5-4B, the Q4_K_M file is 2.74 GB, compared with 8.42 GB for BF16. The project suggests starting with Q4_K_M and using Q5_K_M or Q6_K when more memory is available.

Transformers reuses ggml kernels through the kernels library and is reported to perform close to llama.cpp in tests on three checkpoints, although the measurements used different conditions. The integration also supports transformers serve with an OpenAI-compatible API, model evaluation, custom generation code and dequantization for continued training. The article says llama.cpp remains the recommended engine when efficient local inference is the main priority.

Written by Sakhanda Wire AI from reporting by Hugging Face. Facts, quotes and full context are in the original article below.

Originally published by Hugging Face on

Read the original on Hugging Face ↗

The reporting and the image belong to Hugging Face and are credited and linked above; the text on this page is Sakhanda Wire's own summary.

← Back to the news

More stories

All the latest news