CloudOpsBench identifies reasoning as the main AI cloud operations bottleneck

CloudOpsBench, a new open-source benchmark, evaluates how AI agents perform common DevOps work in cloud environments. Version 1.0 contains 45 tasks on AWS, covering activities such as repairing services, deleting resources, limiting access rights, deploying updates, and shutting down projects. The benchmark was developed by experienced DevOps engineers.

The strongest reported results came from Claude Fable 5.1, which completed 60 percent of the tasks, and GPT-6 Astra, which completed 53 percent. No model solved the full set. The team led by Ang Chen at the University of Michigan said the central limitation was generally reasoning rather than missing technical knowledge.

Around 80 percent of failures were linked to incorrect judgments about a task’s scope, incomplete checks of relevant services, or agents stopping before the work was finished. The remaining roughly 20 percent involved incorrect actions, some of which could cause irreversible infrastructure changes. Tasks run in both the real AWS cloud and an emulator. Planned expansions include GCP, Azure, and OCI. The GitHub project uses the Apache 2.0 license and includes a detailed failure analysis.