Anthropic Adds Plugin Evaluation Workflow to Claude Code
Anthropic has introduced a plugin evaluation workflow for Claude Code that tests plugins with realistic prompts, grades the resulting work, and compares it with a run where the plugin is absent. The workflow is available in Claude Code v2.1.269 and later for plugins containing a plugin.json or .claude-plugin/plugin.json manifest, as well as skills-directory plugins. Model calls used for evaluations and judging are billed to the relevant plan or API account.
An evaluation suite is stored in the plugin’s evals/ directory. Cases contain a prompt.md file and graders, with settings such as model, timeout, maximum turns, tags, and permitted tools. Six grader types are supported: regex, tool_used, tool_order, file_exists, llm, and baseline. The first four analyze transcripts or files without additional charges; the last two use judge models and add to costs.
By default, each case runs with and without the plugin. The difference, Δ, measures the plugin’s contribution, while a tool_used: Skill grader can indicate whether the skill triggered. Anthropic says a near-zero Δ combined with a failed skill grader can reveal behavior that manifest validation misses. Results include per-grader verdicts and judge votes. CI can enforce thresholds and spending limits with --threshold and --max-cost-usd; --trust-plugin is required for untrusted non-interactive checkouts.
Written by Sakhanda Wire AI from reporting by MarkTechPost. Facts, quotes and full context are in the original article below.