XYBench tests whether AI assistants solve the problem behind the question
Researchers are releasing XYBench, a benchmark containing 8,115 real-world questions designed to test whether AI assistants can recognize mistaken assumptions and address users’ underlying goals. The dataset covers programming and everyday activities, including cooking and home repair, and received the Best Resource Paper Award at EMNLP 2026.
The benchmark was developed with researchers from Carnegie Mellon, NVIDIA, and Johns Hopkins. It combines 1,557 questions from Stack Overflow, 184 from Stack Exchange, 102 manually curated examples, and 6,272 questions generated from WikiHow guides. Each example distinguishes the stated request from the intended request inferred from context.
Tests of Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro found that the models more often addressed the literal request than the underlying goal. On Stack Overflow, they primarily focused on the underlying goal in 35–41% of responses, compared with 47–57% focused on the stated request. Researchers also found that models were better at selecting useful advice than producing it, indicating a gap between recognizing and applying inferred intent. XYBench currently covers English-language, mainly procedural questions.
Written by Sakhanda Wire AI from reporting by Hugging Face. Facts, quotes and full context are in the original article below.