Sakhanda Wire
NVDA $152.94 -2.05% MSFT $505.64 -2.38% GOOGL $280.11 -1.27% META $619.47 -1.11% AMZN $256.54 -0.47%
← Back to the news

Anthropic Adds Plugin Evaluation Workflow to Claude Code

Anthropic has introduced a plugin evaluation workflow for Claude Code that tests plugins with realistic prompts, grades the resulting work, and compares it with a run where the plugin is absent. The workflow is available in Claude Code v2.1.269 and later for plugins containing a plugin.json or .claude-plugin/plugin.json manifest, as well as skills-directory plugins. Model calls used for evaluations and judging are billed to the relevant plan or API account.

An evaluation suite is stored in the plugin’s evals/ directory. Cases contain a prompt.md file and graders, with settings such as model, timeout, maximum turns, tags, and permitted tools. Six grader types are supported: regex, tool_used, tool_order, file_exists, llm, and baseline. The first four analyze transcripts or files without additional charges; the last two use judge models and add to costs.

By default, each case runs with and without the plugin. The difference, Δ, measures the plugin’s contribution, while a tool_used: Skill grader can indicate whether the skill triggered. Anthropic says a near-zero Δ combined with a failed skill grader can reveal behavior that manifest validation misses. Results include per-grader verdicts and judge votes. CI can enforce thresholds and spending limits with --threshold and --max-cost-usd; --trust-plugin is required for untrusted non-interactive checkouts.

Written by Sakhanda Wire AI from reporting by MarkTechPost. Facts, quotes and full context are in the original article below.

Originally published by MarkTechPost on

Read the original on MarkTechPost ↗

The reporting and the image belong to MarkTechPost and are credited and linked above; the text on this page is Sakhanda Wire's own summary.

← Back to the news

More stories

All the latest news