Sakhanda Wire
NVDA $152.56 -2.29% MSFT $505.79 -2.36% GOOGL $279.70 -1.42% META $618.14 -1.32% AMZN $256.03 -0.67%
← Back to the news

Agents reproduced 2,226 ICML papers in a 19-day challenge

More than 1,200 community members joined a 19-day effort to reproduce research presented at ICML 2026. From July 15 to August 2, participants used coding agents including Claude Code, Codex, Cursor and Pi to examine papers claim by claim. They published 6,816 Trackio logbooks covering 2,226 papers, or about 34% of the conference, and launched 2,962 cloud jobs.

An automated judge assessed 35,908 claims as verified, falsified, toy-scale or inconclusive. At least one claim was independently verified in 1,103 papers, including 266 papers whose claims were all confirmed. Another 496 papers contained at least one falsified or contested claim, while 242 papers produced opposing verdicts from different reproduction teams. Missing artifacts were the most common reason that no conclusion could be reached.

The organizers said the results show that agents can expand the scale of research checking but still need human direction. Participants had to identify incorrect assumptions, investigate conflicting results and make judgments that experiments could not fully automate. Authors confirmed findings involving several papers, and two arXiv corrections were underway when the article was published.

Written by Sakhanda Wire AI from reporting by Hugging Face. Facts, quotes and full context are in the original article below.

Originally published by Hugging Face on

Read the original on Hugging Face ↗

The reporting and the image belong to Hugging Face and are credited and linked above; the text on this page is Sakhanda Wire's own summary.

← Back to the news

More stories

All the latest news