Sakhanda Wire
NVDA $154.00 -1.37% MSFT $506.28 -2.26% GOOGL $280.87 -1.00% META $620.34 -0.97% AMZN $256.88 -0.34%
← Back to the news

New method targets reliability gaps in AI agent evaluations

A new method for evaluating and stabilizing AI agents focuses on consistency, not only average accuracy. In AppWorld, a ReAct agent using GPT-4.1 succeeded on 77.4% of five repeated runs on average, but completed all five runs successfully for just 53.0% of tasks. The resulting 24.4-percentage-point gap shows that some tasks are solved inconsistently even when nothing changes in the request.

The Consistency Analyzer, added to the ALTK-Evolve system, examines one recorded trajectory and resamples each decision point with five completions by default. It identifies choices vulnerable to small changes, without requiring ground-truth labels, model internals or a new end-to-end run. Those points are converted into guidelines that are supplied to the agent during inference.

On 168 AppWorld test_normal tasks, guidelines raised Pass^5 from 53.0% to 69.0% and Mean@5 from 77.4% to 81.0%, reducing the gap to 12.0 points. On related tasks, Pass^5 improved by 13.0 points. The authors distinguish Pass^5, where every repeated run must succeed, from Pass@5, where only one attempt needs to work.

Written by Sakhanda Wire AI from reporting by Hugging Face. Facts, quotes and full context are in the original article below.

Originally published by Hugging Face on

Read the original on Hugging Face ↗

The reporting and the image belong to Hugging Face and are credited and linked above; the text on this page is Sakhanda Wire's own summary.

← Back to the news

More stories

All the latest news