Sakhanda Wire
NVDA $154.93 -0.77% MSFT $505.37 -2.44% GOOGL $279.95 -1.33% META $618.13 -1.32% AMZN $257.42 -0.13%
← Back to the news

New benchmark finds major visual weaknesses in leading AI models

Moonshot AI has introduced PerceptionBench, an evaluation designed to measure visual perception in multimodal language models without relying on reasoning or outside knowledge. Its tasks test ten abilities identified from real model mistakes: visual relations, counting, attributes, depth and 3D understanding, localization, comparison, fine-grained recognition, context integration, OCR, and hallucination.

The publicly released set contains 3,000 questions selected from an internal collection of more than 17,000 verified items. Sixty percent came from documented model errors, while the rest were reformulated with modified images. Examples include locating a symbol on a clock face, counting flowers within a colored box, and distinguishing between similar pencil cups.

None of the 16 frontier models tested reached 60 percent accuracy. GPT-5.6 Sol ranked first at 59.7 percent, followed by Kimi K3 at 58.5 percent, Claude Fable 5 at 57.2 percent, and Gemini 3.1 Pro at 56.2 percent. GPT-5.5 scored 55.8 percent. Qwen3.5-397B-A17B reached 47.5 percent and GLM-4.6V 32.5 percent. The authors say many apparent reasoning failures begin with incorrect image interpretation. Hallucination was the weakest category on average.

Written by Sakhanda Wire AI from reporting by The Decoder. Facts, quotes and full context are in the original article below.

Originally published by The Decoder on

Read the original on The Decoder ↗

The reporting and the image belong to The Decoder and are credited and linked above; the text on this page is Sakhanda Wire's own summary.

← Back to the news

More stories

All the latest news