Open and closed AI models show narrowing gap in cyber capabilities

The Vals CyberOffense benchmark found that AI agents can independently attack simulated corporate networks, starting with only a public web address. Running from isolated Kali Linux machines, the agents move through web applications and Windows servers toward administrator access, while an external system records and scores every action. GPT-6 Sol Daybreak achieved the strongest result, reaching 80.8 percent of the objectives. The open-weight Deepseek V4.1 Flash 1017 reached 72.8 percent, at about $3 per run compared with roughly $22 for GPT-6 Sol Daybreak.

In the more difficult Lighthouse scenario, GPT-6 Sol Daybreak completed 27 of 31 objectives in under three hours, including taking administrator control of one company and using stolen passwords against another. Deepseek completed 21 objectives. Vals said longer runs were important: Deepseek made little progress during its first 25 million tokens and most of its gains after 100 million, meaning short tests may understate open models’ capabilities.

The findings are consistent with a July analysis by the UK AI Security Institute, which estimated that the lead of closed frontier models had narrowed to four to seven months from six to ten months during most of 2025. Vals said the same systems could also help defenders identify and fix vulnerabilities, while warning that attacks on real networks may increase.