CWE-bench v2 expands public testing of cybersecurity models
CWE-bench v2 introduces a public benchmark of 100 tasks for evaluating AI agents on software security work. Each task uses a real open-source repository, a short instruction and a goal: identify and fix a weakness while preserving existing functionality. Scoring is programmatic and deterministic, allowing results to be reproduced without additional cost. The tasks cover 91 CWE categories, and a separate held-out set is used to track performance on unseen problems.
Collinear AI says every task was checked for realistic instructions, adequate verification and compatibility with the original software. The checks were also reviewed manually, and the public set, grader and execution harness are open for feedback and contributions. Results are available through leaderboards for open-weight and API-only models.
Among 11 evaluated systems, Gemini 4 Argon recorded 67.8% pass@1 and 81.0% pass@4, narrowly ahead of Claude Opus 5.5 at 67.3% and 78.0%. At least one model solved 95 tasks, while the leading model failed 19% of tasks on every attempt. Harbor can run the benchmark from its dataset repository, although access conditions must first be accepted.
Written by Sakhanda Wire AI from reporting by Hugging Face. Facts, quotes and full context are in the original article below.