StateBench measures how models track changing information in long documents

Researchers have introduced StateBench, a benchmark for evaluating how language models follow changing information about entities across long documents. The tasks reflect situations such as variables being reassigned in code, characters moving in a story, or objects changing hands. They range from finding an unchanging or most recent value to following cumulative updates and dependencies between variables.

The benchmark has four difficulty levels: r-trivial, aperiodic, soluble, and general finite. Tests use comparable assignment sequences while changing only the type of update. Evaluations of pretrained models found strong performance on the simplest tasks, declining results on aperiodic tasks as inputs grew, and near-chance performance on general finite problems. Format also affected results.

Four frontier models—Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol, and GPT-6 Luna—performed well on the two easier levels, but their accuracy on soluble and general finite tasks fell sharply with longer inputs, reaching 13% or less on the longest general finite tests. Models trained directly on StateBench showed a different pattern: Gated DeltaNet and hybrid systems generalized to much longer sequences, while transformers did not learn the hardest level. The researchers said questions remain about memory limitations, precision, and tracking new entities introduced over time.