AB Test Eval — Automated Component Benchmarking via Subagents

Evaluate any OpenClaw component (skill, script, hook, cron job) by spawning parallel subagents and comparing arms. Supports multiple eval modes, auto-grading, and regression tracking.

Step 1: Choose the Eval Mode

Pick the mode that matches the user's intent:

Mode	Question	Arms
baseline	Does the skill help at all?	with-skill vs without-skill
regression

Did changes break anything? | skill-v2 vs skill-v1 |
| model-swap | Works on another model? | model-A vs model-B |
| prompt-variant | Which description works better? | variant-A vs variant-B |
| trigger-accuracy | Dispatches correctly? | should-trigger vs should-not |
| adversarial | Robust against bad inputs? | clean vs perturbed |
| script-test | Script produces correct output? | script-A vs script-B |
| hook-dryrun | Hook responds correctly? | with-hook vs without |
| cron-dryrun | Cron payload does the right thing? | cron-run vs baseline |
| integration | Full stack works together? | full vs missing-component |

Default to baseline if unclear.

Step 2: Prepare Directory Structure

Create the eval workspace as a sibling to the skill directory:

CODEBLOCK0

Create directories with mkdir -p. Use descriptive arm names (e.g. with_skill, without_skill, new_version, old_version).

Step 3: Define or Generate Evals

If evals already exist

Read <skill-dir>/evals/evals.json and present the cases to the user for confirmation before running. Do not auto-run without sign-off.

If evals are missing

Generate them by reading the skill's SKILL.md and creating 4-6 realistic eval cases:

1. 1. Happy path — clear request the skill should nail
2. Ambiguous request — could go multiple ways
3. Edge case — unusual params or corner case
4. Negative case — similar but should NOT trigger this skill
5. Multi-step case — complex multi-tool request
6. Adversarial case (if mode=adversarial) — misleading / typo / injected junk

Write to <skill-dir>/evals/evals.json:

CODEBLOCK1

Then show them to the user: "Here are the test cases I plan to run. Do these look right, or do you want to add more?"

Wait for approval before spawning subagents.

Step 4: Efficiency Controls — Dry-Run Preview & Smoke Test

Before spawning expensive subagents, offer the user two efficiency controls (especially useful when eval count > 3 or arms > 2).

--dry Preview

Generate a preview report that lists exactly what will run, without spawning any subagents:

CODEBLOCK2

Present this to the user and ask: "This looks like X evals across Y arms. Should I proceed, or do you want to trim the list?"

--smoke Smoke Test

If the user wants a quick confidence check, run only the first eval end-to-end (all arms + grading). This verifies the pipeline works before committing to the full run.

After a successful smoke test, ask: "Smoke test passed. Should I run the remaining N evals now?"

Step 5: Write Assertions

While waiting for user approval (or while subagents run), draft assertions in eval_metadata.json for each eval.

Save to <workspace>/iteration-N/<eval-name>/eval_metadata.json:

CODEBLOCK3

Assertions use text and expected fields. These are the basis for grading.

Step 6: Spawn Subagents in Parallel

For each eval, spawn all arms in the same turn. Launch as many as the environment allows concurrently.

Baseline mode

• - withskill: load SKILL.md, execute prompt, save outputs
• withoutskill: same prompt, no skill, save outputs

Regression mode

• - newskill: load updated INLINECODE15
• oldskill: load a snapshot of the previous version (make a cp -r snapshot before editing)

Model-swap mode

• - model-a: run with skill + model A override
• model-b: run with skill + model B override

Prompt-variant mode

• - variant-a: load skill variant A's INLINECODE17
• variant-b: load skill variant B's INLINECODE18

Trigger-accuracy mode

Each prompt gets ONE subagent tasked as the dispatcher:

"You are the dispatcher. Given this user prompt, would you load <skill-path>/SKILL.md before responding? Answer yes/no and explain why."

Save yes/no explanations, then grade TP/FP/TN/FN.

Adversarial mode

• - clean: normal prompt + skill
• perturbed: prompt with typos / injected irrelevance / misleading framing + skill

Script-test mode

• - Run the bundled script with controlled inputs and assert on stdout, exit code, and generated files.
• Arms can be: current-script vs previous-script, or script-with-skill-guidance vs naive-approach.
• Assertions focus on correctness, idempotency, and edge-case handling.

Hook-dryrun mode

• - Simulate a hook event by spawning a subagent and telling it: "Pretend you are an OpenClaw agent receiving a <hook-type> event with this payload. Given this hook's SKILL.md or config, what would you do?"
• Do NOT modify actual system hook registrations. This is a read-only simulation.

Cron-dryrun mode

• - Extract the cron job's payload (task command or script path from jobs.json or cron config).
• Run the payload in an isolated subagent or exec dry-run context.
• Assert on expected side effects, file outputs, or command sequence.
• Also verify the cron expression is valid and produces expected schedule times.

Integration mode

• - Test the full stack: user prompt → skill dispatch → script execution → hook response.
• Arms: full-stack vs missing-script vs missing-hook vs skill-only.

Task template for standard arms:

CODEBLOCK4

Step 7: Capture Timing from Notifications

When each subagent completes, its notification includes total_tokens and duration_ms. This is the only chance to capture it.

Save to <arm>/timing.json:

CODEBLOCK5

Process each notification as it arrives rather than batching.

Step 8: Auto-Grade with LLM-as-Judge

Spawn a grading subagent per eval to compare all arms against the assertions:

CODEBLOCK6

Each grading.json schema:

CODEBLOCK7

For trigger-accuracy runs, save a separate trigger_grading.json with tp, fp, tn, fn tallies at the eval level.

Step 9: Aggregate and Generate Report

Write benchmark.json:

CODEBLOCK8

Append a compact line to history.jsonl for regression tracking.

Then write benchmark.md with:

• - Executive summary (delta, winner, biggest weaknesses)
• Per-eval breakdown table
• Notable failures with quotes
• Recommendations for improving the skill

Present the summary to the user directly in chat.

Step 10: Iterate Based on Feedback

1. 1. Discuss results with the user
2. Improve the skill based on failed assertions
3. Rerun into INLINECODE36
4. Compare history.jsonl entries for trend
5. Repeat until satisfied

Mode-Specific Notes

• - Regression: Snapshot old skill before editing (cp -r). Use previous version as baseline.
• Model-swap: Use sessions_spawn with model override.
• Prompt-variant: Create two temp skill copies with different descriptions.
• Trigger-accuracy: Generate 10 queries (5 should-trigger, 5 near-miss should-not). Grade precision/recall/F1.
• Adversarial: Perturbations include typos, irrelevant context injection, misleading framing. Report degradation score = cleanavg - perturbedavg.
• Script-test: Run via exec for deterministic results unless script invokes LLM. Check happy-path AND error handling.
• Hook-dryrun: Simulate event via subagent with exact payload JSON. Do NOT modify actual hook registrations.
• Cron-dryrun: Validate cron expression and list next N execution times. If payload sends messages, use dry-run constraint.
• Integration: For missing-component arms, tell subagent: "You do NOT have access to ."

Hard Constraints

• - Do not auto-run evals without user sign-off — present evals and wait for approval before spawning
• Respect --dry and --smoke — offer preview / smoke-test paths to improve UX and reduce wasted tokens