Guide details how to run NVIDIA Dory 1.0 4B with llama.cpp

A coding tutorial examines Dory 1.0 4B, a compact reasoning model developed by NVIDIA’s Nemotron Labs. Its middle layers form a shared block that can be executed one, two, three or four times for each token. Increasing the loop count raises computation per token without increasing the parameter count, allowing one checkpoint to operate at four compute levels without retraining.

The guide builds the llama.cpp branch specified by NVIDIA’s model card and downloads a 3.7 GB GGUF model file. It reads the model architecture directly from that file to estimate parameter work and key-value cache requirements for each loop setting. The included implementation uses an input group of 16 layer entries, a shared recurrent group of 40, and an output group of 16; the file’s default loop count is four.

The tutorial then compares those estimates with measurements from the running server. It records decoding speed for every loop count and tests whether using a smaller cache for only the final loop affects predictions. A paired GSM8K evaluation examines how loop count relates to accuracy, response length and the time needed for each solved question.