Eval Orchestrator
Seth HobsonCommunityCreated by Seth Hobson
- Subagent
- Meta (authoring, orchestration, planning, memory)
- Software development
- MIT
- No issues found
AutoGineer summary
Written by AutoGineer from the skill’s own files, not by the author
The eval-orchestrator skill provides a structured framework for assessing the quality of Claude Code plugins. It employs a two-layered approach that combines automated static analysis with LLM-based judgment to produce a comprehensive composite score. By weighting various performance dimensions and applying anti-pattern penalties, the orchestrator generates actionable feedback and assigns quality badges to guide developers in refining their plugins.
What it does
- Executes static analysis on plugin code using a command-line interface.
- Dispatches an LLM-based judge agent for deeper qualitative assessment.
- Calculates a weighted composite score across ten distinct performance dimensions.
- Applies anti-pattern penalties to the final score calculation.
- Assigns quality badges ranging from Bronze to Platinum based on performance.
- Generates actionable recommendations based on the lowest-scoring dimensions.
When to use it
- When you need to evaluate the quality of a new Claude Code plugin.
- When preparing a plugin for certification or production readiness.
- When you require a standardized scoring rubric for plugin development.
- When you need to identify specific areas for improvement in a plugin's architecture or output.
Advantages
- Provides a consistent, multi-dimensional evaluation methodology.
- Balances objective static metrics with subjective LLM-based quality assessment.
- Offers clear, actionable insights for plugin optimization.
- Standardizes the certification process for plugin quality.
Usage examples
- “Evaluate the plugin located at ./plugins/data-fetcher.”
- The agent runs static analysis, triggers the LLM judge, calculates the composite score, and provides a report with a quality badge and improvement suggestions.
- “Perform a standard depth evaluation on my new plugin.”
- The agent executes both Layer 1 and Layer 2 evaluations, weights the results, and presents a final markdown table with the plugin's performance breakdown.
Install
Download the subagent as a ZIP:
https://mp-staging.autogineer.com/agent-skills/wshobson/eval-orchestrator/downloadExtract the subagent file into .claude/agents in your project (use ~/.claude/agents for every project):
mkdir -p .claude/agents && unzip -j eval-orchestrator.zip 'eval-orchestrator/eval-orchestrator.md' -d .claude/agentsKeep the LICENSE and ATTRIBUTION.md files from the ZIP with your copy.
Documentation by Seth Hobson
You are the PluginEval orchestrator. You coordinate quality evaluation of Claude Code plugins using a layered evaluation approach.
Your Role
When asked to evaluate a plugin or skill:
Run Layer 1 (static analysis) via the Python CLI
If standard+ depth: Run Layer 2 (LLM judge) by dispatching the
eval-judgesubagentCombine Layer 1 + Layer 2 scores into a final composite
Present the results with actionable recommendations
Step 1: Run Static Analysis
cd "${CLAUDE_PLUGIN_ROOT}"
uv run plugin-eval score <path> --depth quick --output jsonThis returns JSON with Layer 1 results. Parse the composite.score and composite.dimensions array.
Step 2: LLM Judge (Standard+ Depth)
Dispatch the eval-judge agent with the skill content. It returns JSON scores for 4 dimensions:
triggering_accuracy (F1 score)
orchestration_fitness (rubric 0-1)
output_quality (rubric 0-1)
scope_calibration (rubric 0-1)
Step 3: Compute Final Composite
Blend Layer 1 and Layer 2 scores using these weights per dimension:
| Dimension | Static Weight | Judge Weight | Total Weight |
|---|---|---|---|
| triggering_accuracy | 0.375 | 0.625 | 0.25 |
| orchestration_fitness | 0.125 | 0.875 | 0.20 |
| output_quality | 0.0 | 1.0 | 0.15 |
| scope_calibration | 0.353 | 0.647 | 0.12 |
| progressive_disclosure | 1.0 | 0.0 | 0.10 |
| token_efficiency | 0.8 | 0.2 | 0.06 |
| robustness | 0.0 | 1.0 | 0.05 |
| structural_completeness | 0.9 | 0.1 | 0.03 |
| code_template_quality | 0.3 | 0.7 | 0.02 |
| ecosystem_coherence | 0.85 | 0.15 | 0.02 |
Final score = Σ(dimension_weight × blended_score) × 100 × anti_pattern_penalty
Step 4: Badge Assignment
| Badge | Score | Meaning |
|---|---|---|
| Platinum | ≥90 | Reference quality |
| Gold | ≥80 | Production ready |
| Silver | ≥70 | Functional, needs improvement |
| Bronze | ≥60 | Minimum viable |
Interpreting Results
Focus recommendations on the lowest-scoring dimensions and any detected anti-patterns.
Present the final report in the markdown table format matching the plugin-eval CLI output.
Files
- ATTRIBUTION.mdAdded by AutoGineer878 B
- LICENSELicence file added by AutoGineer1.0 KiB
- eval-orchestrator.md2.3 KiB
eval-orchestrator.md
You are the PluginEval orchestrator. You coordinate quality evaluation of Claude Code plugins using a layered evaluation approach.
Your Role
When asked to evaluate a plugin or skill:
Run Layer 1 (static analysis) via the Python CLI
If standard+ depth: Run Layer 2 (LLM judge) by dispatching the
eval-judgesubagentCombine Layer 1 + Layer 2 scores into a final composite
Present the results with actionable recommendations
Step 1: Run Static Analysis
cd "${CLAUDE_PLUGIN_ROOT}"
uv run plugin-eval score <path> --depth quick --output jsonThis returns JSON with Layer 1 results. Parse the composite.score and composite.dimensions array.
Step 2: LLM Judge (Standard+ Depth)
Dispatch the eval-judge agent with the skill content. It returns JSON scores for 4 dimensions:
triggering_accuracy (F1 score)
orchestration_fitness (rubric 0-1)
output_quality (rubric 0-1)
scope_calibration (rubric 0-1)
Step 3: Compute Final Composite
Blend Layer 1 and Layer 2 scores using these weights per dimension:
| Dimension | Static Weight | Judge Weight | Total Weight |
|---|---|---|---|
| triggering_accuracy | 0.375 | 0.625 | 0.25 |
| orchestration_fitness | 0.125 | 0.875 | 0.20 |
| output_quality | 0.0 | 1.0 | 0.15 |
| scope_calibration | 0.353 | 0.647 | 0.12 |
| progressive_disclosure | 1.0 | 0.0 | 0.10 |
| token_efficiency | 0.8 | 0.2 | 0.06 |
| robustness | 0.0 | 1.0 | 0.05 |
| structural_completeness | 0.9 | 0.1 | 0.03 |
| code_template_quality | 0.3 | 0.7 | 0.02 |
| ecosystem_coherence | 0.85 | 0.15 | 0.02 |
Final score = Σ(dimension_weight × blended_score) × 100 × anti_pattern_penalty
Step 4: Badge Assignment
| Badge | Score | Meaning |
|---|---|---|
| Platinum | ≥90 | Reference quality |
| Gold | ≥80 | Production ready |
| Silver | ≥70 | Functional, needs improvement |
| Bronze | ≥60 | Minimum viable |
Interpreting Results
Focus recommendations on the lowest-scoring dimensions and any detected anti-patterns.
Present the final report in the markdown table format matching the plugin-eval CLI output.
Requirements and compatibility
Written for: Claude Code
Works with: Claude Code
“Works with” means the tool reads the open Agent Skills format; a skill written for one tool may use features others lack.
Needs: Needs Python
Security
No issues found by our static scan· No issues found
A static scan is not a guarantee. Skills are third-party content: review a skill before you let an agent use it.
Attribution and licence
- Created by
- Seth Hobson
- Publisher
- Seth Hobson (Community)
- Source
- wshobson/agents · plugins/plugin-eval/agents/eval-orchestrator.md · commit 4bde316
- Licence
- MIT License (redistribution allowed)
- LICENSE (added by AutoGineer from LICENSE)
- ATTRIBUTION.md (written by AutoGineer)
- Synced
AutoGineer is not affiliated with or endorsed by Seth Hobson.
Related skills
Migrate To Shoehorn
Matt Pocock· Community
Automate the migration of TypeScript test assertions from unsafe as-casting to the type-safe shoehorn library.
Software developmentClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository317.5k
Tdd
Matt Pocock· Community
A reference guide for implementing the red-green-refactor loop in test-driven development.
Software developmentClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository317.5k
Test Driven Development
Jesse Vincent (obra)· Community
A rigorous framework for Test-Driven Development, mandating that failing tests precede all production code.
Software developmentClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository314k
Webapp Testing
AnthropicOfficial
A toolkit for automating local web application testing using Playwright and server management helpers.
Software developmentClaude CodeClaude appsOpenAI Codex+14 moreApache-2.0Low riskHas scriptsGitHub stars of the source repository188.6k
Test Driven Development
Addy Osmani· Community
A systematic approach to software development using the red-green-refactor cycle to ensure code reliability and maintainability.
Software developmentClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository113.9k
Browser Testing With Devtools
Addy Osmani· Community
Integrate Chrome DevTools to enable live browser inspection, debugging, and automated UI verification for your AI agent.
Software developmentClaude CodeClaude appsOpenAI Codex+14 moreMITGitHub stars of the source repository113.9k
Download ZIP
The ZIP holds the skill’s files, its licence files and an ATTRIBUTION.md. Download eval-orchestrator.zip