CUGA outperforms Hermes and Deep Agents in AppWorld comparison

An IBM Research team compared three agent harnesses using the same open-weight model, GLM-5.3-flash, on 90 AppWorld tasks covering 30 scenarios. The benchmark spans nine applications and 457 APIs, including email, payments and file storage. CUGA completed 73 tasks, or 81.1%, while Hermes completed 42 tasks, or 46.7%, and Deep Agents completed 27, or 30.0%. CUGA also passed all tasks in 20 scenarios, compared with eight for Hermes and two for Deep Agents.

The researchers changed the harness rather than the model, using the Harbor evaluation framework. CUGA’s setup included task decomposition, focused tool selection, verification before execution and automatic reflection. Hermes used Harbor’s default configuration with AppWorld tools over MCP, while Deep Agents used LangChain middleware and additional tool-management features.

The authors said the results are not a general ranking: Hermes and Deep Agents were not designed specifically for multi-application API work, and the configurations were not identical. The sample covered one benchmark and did not test policy enforcement. They presented CUGA as an enterprise-oriented, open-source harness that can run within private, sovereign or air-gapped environments and enforce approvals and tool guards at runtime.