eval-skills

AI Agent Skill unit testing framework — a framework-agnostic toolkit for discovering, scaffolding, selecting, evaluating, and reporting on AI skills.

This skill fills the L1 (Skill Unit Test) gap that LangSmith / DeepEval leave open: while those platforms focus on agent-level and trajectory-level evaluation (L2-L3), eval-skills targets the individual skill level, ensuring each building block meets quality standards before it ever enters an agent pipeline.

When to Use This Skill

• - Before deploying a new skill to production — run eval to verify it meets your quality gate.
• When choosing between multiple candidate skills — run select to rank them on the same benchmark.
• When a skill is upgraded — run report diff to detect regressions.
• In CI/CD — use --exit-on-fail to block merges that degrade skill quality.
• When bootstrapping a new skill — run create to generate a ready-to-fill skeleton.

Capabilities

1. Find Skills

Search for existing skills by keyword, tag, or adapter type.

CODEBLOCK0

Option	Description	Default
INLINECODE5	Keyword search (matches name, description, tags)	—
INLINECODE6

Filter by tags (intersection: skill must have ALL specified tags) | — |
| -a, --adapter <type> | Filter by adapter type (http, subprocess, mcp) | — |
| --min-completion <rate> | Minimum historical completion rate (0.0 ~ 1.0) | — |
| --skills-dir <dir> | Directory to scan for skill.json files | ./skills |
| --limit <n> | Maximum number of results | 20 |

Results are ranked by search relevance (when --query is provided) or by historical completion rate (descending).

2. Create Skills

Generate a skill skeleton from a template to bootstrap development.

CODEBLOCK1

Option	Description	Default
INLINECODE18	Required. Skill name	—
INLINECODE19

Template type: http_request, python_script, mcp_tool | http_request |
| --output-dir <dir> | Output directory | ./skills |
| --description <text> | Human-readable description embedded in skill.json | — |

Generated file structure:

CODEBLOCK2

3. Evaluate Skills

Run benchmark evaluations against one or more skills. This is the core command.

CODEBLOCK3

Option	Description	Default
INLINECODE28	Required. Skill file(s) or directory(ies)	—
INLINECODE29

Built-in benchmark ID or path to benchmark.json | coding-easy |
| --tasks <file> | Custom tasks JSON file (replaces benchmark) | — |
| --concurrency <n> | Number of parallel task executions | 4 |
| --timeout <ms> | Per-task timeout in milliseconds | 30000 |
| --retries <n> | Retry count on task failure (with incremental backoff) | 0 |
| --runs <n> | Repeat evaluation N times for consistency scoring | 1 |
| --evaluator <type> | Default scorer type (see Scorer Types below) | exact |
| --format <formats...> | Output formats: json, markdown, html | json markdown |
| --output-dir <dir> | Report output directory | ./reports |
| --exit-on-fail | Exit with code 1 if any skill falls below threshold | disabled |
| --min-completion <rate> | Threshold for --exit-on-fail | 0.7 |
| --dry-run | Validate configuration only; do not execute tasks | disabled |
| --benchmarks-dir <dir> | Directory containing built-in benchmarks | ./benchmarks |
| --store <path> | SQLite database path for persistent result storage | ./eval-skills.db |
| -c, --config <path> | Path to eval-skills.config.yaml | auto-detected |

Evaluation flow:

1. 1. Load skills from --skills paths (supports both single skill.json and directories)
2. Load benchmark tasks from --benchmark or INLINECODE64
3. Build the cartesian product: INLINECODE65
4. Execute all task items concurrently (controlled by --concurrency, with timeout and retry)
5. Score each result using the appropriate scorer
6. Aggregate into SkillCompletionReport per skill
7. Write reports to INLINECODE68

4. Select Skills

Filter and rank skills based on evaluation reports using a multi-dimensional strategy.

CODEBLOCK4

Option	Description	Default
INLINECODE69	Required. Candidate skills directory or JSON file	—
INLINECODE70

Evaluation reports JSON file | — |
| --strategy <file> | SelectStrategy YAML/JSON file | built-in default |
| --min-completion <rate> | Override minimum completion rate filter | — |
| --top-k <n> | Return only the top K results | all |
| --output <file> | Write selected skills to file | stdout |

Selection pipeline: Filter (by completion rate, error rate, latency, adapter type, required tags) -> Score -> Rank (by compositeScore, completionRate, latency, or tokenCost) -> TopK

Example strategy.yaml:

CODEBLOCK5

5. Run Pipeline

Execute the full end-to-end pipeline: Find -> Eval -> Select -> Report in a single command.

CODEBLOCK6

This command automates the entire process:

1. 1. Find — scans --skills-dir and optionally filters by INLINECODE77
2. Eval — evaluates all candidate skills against INLINECODE78
3. Select — filters and ranks results using --min-completion, --top-k, and optional INLINECODE81
4. Report — generates output files in all requested --formats

6. Generate & Compare Reports

Convert report format

CODEBLOCK7

Supported output formats: markdown, html.

Diff two reports (regression detection)

CODEBLOCK8

Generates a side-by-side delta table per skill showing changes in completion rate, error rate, P95 latency, and composite score with directional arrows.

7. Initialize Project

CODEBLOCK9

Creates the project scaffold:

• - eval-skills.config.yaml — global configuration
• INLINECODE86 — directory for skill definitions
• INLINECODE87 — directory for benchmark files
• INLINECODE88 — directory for evaluation output

8. Manage Configuration

CODEBLOCK10

Configuration is resolved in priority order:

1. 1. CLI flags (highest priority)
2. INLINECODE89 in current directory
3. INLINECODE90
4. Built-in defaults (concurrency: 4, timeoutMs: 30000, outputDir: ./reports)

Scorer Types

Each task in a benchmark specifies an evaluator type. The scorer compares the skill's actual output against the expected output.

Type	Aliases	Description	Score Range
INLINECODE94	INLINECODE95	Strict equality comparison. Supports caseSensitive option.	0 or 1
INLINECODE97

— | Checks for the presence of all specified keywords in the output. Partial credit: matched_keywords / total_keywords. | 0.0 ~ 1.0 |
| json_schema | schema | Validates output against a JSON Schema (using Ajv). | 0 or 1 |
| llm_judge | — | Sends the output + expected rubric to an LLM (configurable model) for quality rating. | 0.0 ~ 1.0 |
| custom | — | Loads a custom scorer from expectedOutput.customScorerPath. | 0.0 ~ 1.0 |

Evaluation Metrics

Every evaluation produces a SkillCompletionReport with these metrics:

Metric	Description	Formula
Completion Rate	Fraction of tasks that passed	INLINECODE105
Partial Score

Mean score across all tasks | mean(task_scores) |
| Error Rate | Fraction of tasks that errored or timed out | (error_count + timeout_count) / total_count |
| Consistency Score | Stability across multiple runs (requires --runs >= 2) | 1 - stddev(per_run_completion_rates) |
| P50 / P95 / P99 Latency | Response time percentiles | Sorted percentile of latencyMs |
| Composite Score | Weighted overall quality score | 0.5 * CR + 0.2 * (1 - latP95_norm) + 0.3 * (1 - ER) |

Built-in Benchmarks

ID	Domain	Tasks	Scoring	Description
INLINECODE112	coding	20	mean / exact_match	Math expressions, string reversal, palindrome detection
INLINECODE113

tool-use | 5 | mean / contains | Metadata completeness, description quality, structure checks | | web-search-basic | web | 8 | mean / contains + schema | Factual queries, keyword verification, structured output validation | | gaia-v1 | general | — | mean | Placeholder for GAIA benchmark Level 1 tasks | | toolbench-lite | tool-use | — | mean | Placeholder for ToolBench single-tool scenarios |

Custom Benchmark

Create a benchmark.json file:

CODEBLOCK11

CODEBLOCK12

Adapter Types

Skills communicate through adapters. The adapter type is specified in skill.json via adapterType.

Adapter	Protocol	How it works	Key config
INLINECODE120	REST POST	Sends POST { skillId, version, input } to skill.entrypoint. Supports Bearer / API-Key auth via env vars.	INLINECODE123, authType, INLINECODE125
INLINECODE126

JSON-RPC 2.0 over stdin/stdout | Spawns skill.entrypoint (e.g. python3 skill.py), writes JSON-RPC request to stdin, reads response from stdout. | command, args |
| mcp | MCP Protocol | (Phase 2) Native Model Context Protocol integration via @modelcontextprotocol/sdk. | — |

Workflow Examples

Evaluating a Single Skill

CODEBLOCK13

Comparing Multiple Candidate Skills

CODEBLOCK14

Full Pipeline (One Command)

CODEBLOCK15

CI/CD Quality Gate

CODEBLOCK16

Regression Detection

CODEBLOCK17

Best Practices

1. 1. Always use --runs 3 or more when evaluating for production decisions. Single-run results can be noisy; the consistency score captures stability across runs.

1. 2. Use --exit-on-fail in CI/CD pipelines to enforce quality gates. Set --min-completion to your acceptable threshold (recommended: 0.8 for production skills).

1. 3. Create domain-specific custom benchmarks rather than relying solely on built-in ones. Your custom benchmark should reflect real-world inputs your skill will encounter.

1. 4. Use report diff after every skill upgrade to catch regressions early. Compare the new evaluation against a saved baseline report.

1. 5. Use --dry-run before long evaluations to validate your configuration (skill paths, benchmark resolution, task count) without actually executing tasks.

1. 6. Persist results with --store to track skill quality over time. The SQLite store enables historical trend queries.

1. 7. Start with --concurrency 1 when debugging a failing skill, then increase for production benchmarking.

1. 8. Tag your benchmark tasks to enable per-category analysis (e.g., filter by math, string, edge-case).

Skill JSON Schema

Every skill must provide a skill.json that conforms to this structure:

CODEBLOCK18

Validation rules:

• - id: lowercase alphanumeric with _ or -, non-empty
• INLINECODE147: semver format (X.Y.Z)
• INLINECODE149: one of http, subprocess, mcp, langchain, INLINECODE154
• INLINECODE155: non-empty string (URL for http, command for subprocess)

Global Options

These options are available on all commands:

Option	Description
INLINECODE156	Path to configuration file
INLINECODE157

JSON output format (CI-friendly) |
| --no-color | Disable colored output |
| -v, --verbose | Verbose logging |
| --version | Show version |
| -h, --help | Show help |