Sakhanda Wire
NVDA $153.78 -1.51% MSFT $504.71 -2.56% GOOGL $279.10 -1.63% META $617.85 -1.37% AMZN $256.64 -0.43%
← Back to the news

Optima lets users benchmark AI models on their own workflows

Artificial Analysis has launched Optima, a platform for creating AI benchmarks tailored to specific use cases. Users can supply their own datasets, workflows, agent traces or a description of a scenario with sample inputs and outputs. The platform can then generate proposed test cases and evaluation criteria for review before testing models.

Optima compares models across response quality, cost per completed task and time per task. It supports rubric-based scoring against defined criteria and pairwise comparisons based on user preferences. For agent applications, the broader task cost can reveal differences that token prices alone may miss, including repeated attempts, failures and extra cleanup.

Artificial Analysis says early users tested finance and accounting agents, legal writing style and proprietary image data. The platform charges model token costs without markup. Rubric evaluations cost $0.125 per criterion per model, while pairwise evaluations cost $0.375 per comparison. The company notes that benchmark reliability still depends on representative test cases, clear criteria and sound implementation; a lower-cost or faster result may not necessarily deliver greater business value.

Written by Sakhanda Wire AI from reporting by The Decoder. Facts, quotes and full context are in the original article below.

Originally published by The Decoder on

Read the original on The Decoder ↗

The reporting and the image belong to The Decoder and are credited and linked above; the text on this page is Sakhanda Wire's own summary.

← Back to the news

More stories

All the latest news