New benchmark confirms AI models still perform poorly at visual perception
Key Points
- Moonshot AI has introduced PerceptionBench, a test that evaluates the visual perception of multimodal models independently of logical reasoning and external knowledge.
- The test assesses basic visual abilities based on real-world errors. All leading models, including GPT-5.6 Sol, Kimi K3, and Claude Fable 5, showed significant weaknesses in the evaluation.
- The authors conclude that many supposed logical errors are actually caused by flawed visual perception. This confirms earlier studies showing that even the best models fail at basic visual tasks.
Moonshot AI's PerceptionBench tests how well multimodal AI models can actually "see," separate from logical reasoning. No frontier model reaches 60 percent accuracy, and GPT-5.6 Sol leads by a narrow margin. Many supposed reasoning errors actually happen as early as the image-reading stage.
The team behind the Chinese AI assistant Kimi has introduced PerceptionBench, a benchmark that isolates and tests the visual perception of multimodal language models. Unlike standard testing methods, PerceptionBench breaks vision down into ten atomic sub-skills instead of lumping perception, knowledge, and reasoning into a single task. Every question can be answered just by looking at the image, with no reasoning or outside knowledge required.
Categories built from real errors, not theory
The authors explain their approach by pointing out that existing benchmarks each capture only a narrow slice of perception errors. The 42 open-source benchmarks they analyzed show little overlap in their error profiles, so each one covers a different subset of visual weaknesses. No single test or small group of tests was enough to capture visual perception as a whole.
Rather than defining categories up front, the authors built their taxonomy from actual model errors and traced each one back to the earliest failed step in existing benchmarks. The result is ten "skill domains": Visual Relation, Counting, Attributes, Depth & 3D, Localization, Comparison, Fine-grained Recognition, Context Integration, OCR, and Hallucination.
From an internal pool of over 17,000 verified questions, Moonshot AI is publishing 3,000 tasks. Sixty percent were derived from attributed model errors, while 40 percent were reformulated using augmented images. The tasks seem trivial on the surface: figuring out where a symbol sits on a clock face, counting flowers inside a red box, or deciding which of two pencil cups shows a gray-pink combo versus a solid pink with a cartoon design.
No model breaks 60 percent
Among the 16 frontier models tested, the highest overall accuracy is 59.7 percent, scored by GPT-5.6 Sol. Kimi K3 follows at 58.5 percent, Claude Fable 5 at 57.2 percent, and Gemini 3.1 Pro at 56.2 percent. GPT-5.5 comes in at 55.8 percent. Open-source models like Qwen3.5-397B-A17B (47.5 percent) and GLM-4.6V (32.5 percent) trail far behind.
The category-level results are more telling than the overall ranking. Models with nearly identical aggregate scores diverge sharply in individual categories. On average, "hallucination" is the weakest skill across the board. GPT-5.6 Sol scores only 26.9 percent there despite taking the top spot overall, while the weaker Gemini 3.5 Flash ranks among the best with 50.6 percent. This sub-test checks whether models invent objects that don't exist when the correct answer is simply "zero."
Many "reasoning errors" are actually perception failures
The authors argue that many multimodal model failures typically chalked up to "reasoning errors" actually happen at the perception level. When a model botches a multi-step task, the first step, correctly reading the image, has often already gone wrong.
PerceptionBench breaks those questions into perception-only sub-questions, making it possible to pinpoint which specific visual ability is failing. The dataset and evaluation code are available on GitHub at MoonshotAI/PerceptionBench.
A well-known problem with little progress
Moonshot AI builds the open-source Kimi K3, which has closed the gap to Claude Fable 5 and GPT-5.6 Sol to within a few points on general benchmarks. K3 still lags well behind in specialized areas like offensive cybersecurity and complex math. On visual perception, K3 now performs on par with its Western rivals.
The same research team already released WorldVQA, a benchmark that separates object recognition from reasoning. The best model there, Gemini 3 Pro, fell short of 50 percent at 47.4 percent, and all models systematically overestimated their own confidence.
A separate study by Chinese institutions, with Moonshot AI's involvement, used the BabyVision benchmark to show that frontier models fail at basic visual tasks tied to early childhood development, such as tracing lines or counting hidden blocks. Gemini 3 Pro scored 49.7 percent on those tasks, while humans hit 94.1 percent. The researchers attribute this gap to a verbalization bottleneck where visual information gets translated into language and loses fidelity.
AI News Without the Hype – Curated by Humans
Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.