LLM formats differ in storage, quantization and hardware support
GGUF, GPTQ, AWQ, EXL2 and EXL3 address the same basic challenge: reducing the memory needed to run large language models. The guide distinguishes containers, which describe how tensors are stored, from quantization methods, which reduce the number of bits used for weights. Safetensors and PyTorch files are containers, while GPTQ, AWQ and llama.cpp quantization are methods. Memory estimates cover weights only; KV-cache and runtime requirements add further overhead.
GGUF is the native format of llama.cpp and can include weights, tokenizer data, special tokens and a chat template in one file. Its quantization families range from roughly 1.56 to 6.56 bits per weight, with imatrix calibration available for improving low-bit results. GPTQ and AWQ are calibrated methods commonly used with GPUs, while bitsandbytes NF4 quantizes models during loading and remains widely used for QLoRA fine-tuning.
EXL2 targets ExLlamaV2 and consumer NVIDIA GPUs, allowing mixed bitrates such as 4.65 bits per weight. EXL3, built on QTIP, supports very low bitrates and additional features including quantized KV caches and multimodal inference, but requires CUDA 12.4 or later. On Apple Silicon, both GGUF through llama.cpp and MLX are established options. Runtime compatibility remains central when selecting a format.
Written by Sakhanda Wire AI from reporting by MarkTechPost. Facts, quotes and full context are in the original article below.