ThinkingBox tests whether AI agents leave correct database states
ThinkingBox is a benchmark from Microsoft and Hugging Face that evaluates AI agents by examining the backend records and side effects they leave behind, rather than relying on their final wording or valid tool calls. It runs agents in isolated MCP tool sessions and checks whether each workflow reaches its required terminal state.
The benchmark covers 507 stateful business tasks in retail, auto insurance, travel, neobanking and consulting. Each task is attempted 20 times from a clean backend. In the reported single-attempt results, Claude Opus 5.5 led overall with a 67.16% pass@1 score, while Kimi-K3 was the strongest open-weight model at 57.37%. Kimi-K3 solved at least one attempt on 476 tasks, but only 68 tasks passed all 20 trials. Claude Opus 5.5 passed all attempts on 241 tasks.
The authors say that 79.9% of failures involved tool usage, while 10.3% involved incorrect state updates. A common-set analysis found that 67.24% of failed trials ended cleanly despite executable checks identifying errors. ThinkingBox is available through Hugging Face, and the benchmark can be run through OpenEnv.
Written by Sakhanda Wire AI from reporting by Hugging Face. Facts, quotes and full context are in the original article below.