AgentBench for OpenClaw

Benchmark your OpenClaw agent's general capabilities across 40 real-world tasks spanning 7 domains.

Commands

When the user says any of these, follow the corresponding instructions:

• - /benchmark — Run the full benchmark suite (all 40 tasks)
• /benchmark --fast — Run only easy+medium tasks (19 tasks)
• /benchmark --suite <name> — Run one domain only
• /benchmark --task <id> — Run a single task
• /benchmark --strict — Tag results as externally verified scoring
• /benchmark-list — List all tasks grouped by domain
• /benchmark-results — Show results from previous runs
• /benchmark-compare — Compare two runs side-by-side

Flags are combinable: INLINECODE8

Running a Benchmark

Step 1: Discover Tasks

Read task.yaml files from the tasks/ directory in this skill:

CODEBLOCK0

Each task.yaml contains: name, id, suite, difficulty, mode, usermessage, inputfiles, expectedoutputs, expectedmetrics, scoring weights.

Filter by --suite or --task if specified. If --fast is set and --task is not, filter to only tasks where difficulty is "easy" or "medium".

Profile is "fast" if --fast was specified, otherwise "full".

List discovered tasks with count and suites.

Step 2: Set Up Run Directory

Generate a run ID from the current timestamp: INLINECODE15

Read suite_version from skill.json in this skill directory.

Create the results directory:
CODEBLOCK1

Announce: INLINECODE18

Step 3: Execute Each Task

For each task:

1. 1. Set up workspace:

- Create /tmp/agentbench-task-{task-id}/ as workspace - Copy input files from tasks/{suite}/{task}/inputs/ to the workspace (if inputs/ exists) - If the task directory contains a setup.sh: run bash tasks/{suite}/{task}/setup.sh {workspace-path} - For file-unchanged validators: compute checksums of specified files after setup, before task execution

1. 2. Announce: INLINECODE24

1. 3. Record start time (milliseconds): INLINECODE25

1. 4. Execute the task yourself directly:

- Read the task's user_message and execute it as if a real user sent you the request - Work ONLY within the workspace directory - If input files are listed, read them from the workspace - Execute naturally — use the appropriate tools (read, write, edit, exec, websearch, webfetch, etc.) - Create any output files in the workspace directory - When done, write a brief execution-trace.md to the workspace: - What you understood the task to be - What approach you took - What files you created or modified - Any difficulties or decisions you made

1. 5. Record end time and compute duration

1. 6. Collect metrics:

- total_time_ms: end - start - tool_calls_total: count how many tool calls you made during this task - errors: count any tool call failures - planning_ratio: estimate the fraction of time spent reading/thinking vs producing output (approximate is fine)

1. 7. Layer 0 — Automated Structural Checks (compute directly):

After task execution, check the workspace. For each entry in expected_outputs: - file-exists: Check if file exists. 30 points if found, 0 if not. - content-contains: Read file, check each required section keyword (case-insensitive). Points proportional to matches found. Pool: 40 points. - word-count-range: Count words. In range = 30 points. Within 2x range = 15 points. Outside = 0. - git-log-contains: Check git log for expected strings. 30 points if all found, proportional otherwise. - directory-structure: Check all paths exist. 30 points if all present, proportional for partial. - command-output-contains: Run command, check output contains all strings. 30 points if match, 0 if not. - file-unchanged: Compare checksum against pre-execution checksum. 30 points if unchanged, 0 if modified. - link-consistency: Scan files for link syntax consistency. 30 points if consistent, 15 if mostly consistent (>70% one style), 0 if mixed. - Normalize total to 0-100.

1. 8. Layer 1 — Metrics Analysis (compute directly):

If task has expected_metrics: - Tool calls within expected range: 40 points - Tool calls within 2x range: 20 points - Outside 2x range: 0 points - Planning ratio within expected range: 30 points - Planning ratio outside but within 2x: 15 points - Way off: 0 points - Zero errors: 30 points - 1-2 errors: 15 points - 3+ errors: 0 points - Normalize to 0-100. If no metrics available, score as 50. - Token estimate is tracked for reporting but NOT scored.

1. 9. Layer 2 — Behavioral Analysis (self-evaluate honestly, 0-100):

Score based on HOW you executed:

Instruction Adherence (30 points):
- 30: Followed all instructions precisely
- 20: Mostly followed, minor deviations
- 10: Significant deviations
- 0: Ignored or misunderstood

Tool Appropriateness (25 points) — rule-based first:
- Penalty: -10 for each use of exec cat instead of read to read files
- Penalty: -10 for each use of exec echo/printf instead of write to create files
- Penalty: -5 for each use of exec sed/awk instead of edit for file edits
- Start at 25, apply penalties, floor at 0

Approach Quality (25 points) — check read-before-write:
- 25: Read all inputs before producing output
- 15: Read most inputs, minor gaps
- 5: Started producing output without reading context
- 0: No clear approach

Error Recovery (20 points):
- 20: Clean recovery or no errors occurred
- 10: Partial recovery
- 0: Failed to recover

1. 10. Layer 3 — Output Quality (self-evaluate honestly, 0-100):

Score the deliverable:

Completeness (25): All requirements met? Gaps?
Accuracy (25): Content correct? Calculations right?
Formatting (25): Well-structured? Correct file format?
Polish (25): Would a user be satisfied?

1. 11. Compute composite score:

    score = (L0 × 0.20) + (L1 × 0.35) + (L2 × 0.20) + (L3 × 0.25)
    

Use weights from task.yaml if specified, otherwise these defaults.

1. 12. Save task result to agentbench-results/{run-id}/{task-id}/:

- scores.json: All layer scores, composite, breakdown, notes - metrics.json: Timing, tool calls, errors, planning ratio - Copy output files

1. 13. Display: INLINECODE50

Step 4: Generate Report

After all tasks:

1. 1. Compute domain averages (group by suite, average composite scores)
2. Compute overall score (average of domain scores — equal domain weighting)
3. Compute aggregate metrics

Generate three files in agentbench-results/{run-id}/:

results.json — Machine-readable with this structure:
CODEBLOCK3

If --strict was used, set scoring_method to "externally-verified".

Integrity signature: After building results.json (without signature field), compute:

SIG=$(echo -n "$CONTENT" | openssl dgst -sha256 -hmac "agentbench-v1-{run_id}-{suite_version}-integrity" | awk '{print $2}')

Add as "signature" field to results.json.

report.md — Markdown summary: Overall Score, Metrics, Domain Breakdown, Task Details, Top Failures, Recommendations.

report.html — Self-contained HTML dashboard (inline CSS/JS, no external deps):

• - Score display with color (green 80+, yellow 60-79, red <60)
• Domain cards with score bars
• Task detail table (sortable, expandable)
• Top failures section
• Dark mode via prefers-color-scheme
• Footer: "Generated by AgentBench v1.0.0 (OpenClaw) | Suite v{suite_version} | Profile: {profile}"

Step 5: Present Results

1. 1. Display overall score
2. Show domain breakdown
3. Tell user where results are saved
4. Mention they can submit to https://www.agentbench.app/submit

Step 6: Clean Up

Run teardown.sh if present. Remove temp workspace directories unless --keep-workspace was specified.

Listing Tasks (/benchmark-list)

Read all task.yaml files, group by suite, display as:
CODEBLOCK5

Viewing Results (/benchmark-results)

List all directories in agentbench-results/, show run ID, date, overall score, profile, and task count for each.

Comparing Runs (/benchmark-compare)

Show two runs side-by-side: overall scores, domain scores, and per-task deltas. Warn if profiles differ.

Key Differences from Claude Code Version

• - No hooks — metrics are self-tracked (timing, tool call counting)
• No subagents — you execute tasks directly in sequence
• Same tasks, same scoring, same output format — results are cross-platform comparable
• Same integrity signature — submissions work on the same leaderboard

Important Notes

• - Be honest in self-evaluation (L2/L3). Inflated scores are obvious on the leaderboard.
• The objective layers (L0 + L1) carry 55% of the weight — they can't be faked.
• Token estimates are informational only, not scored.
• Any link syntax is accepted in skill graph tasks — consistency is what's scored.