agent-autoresearch

Any agent can run this. The experiment is always: change something → measure it → keep what works.

---

The Core Idea

Karpathy's insight: give an agent a fixed time budget, let it modify one file, measure if things got better, keep or discard, repeat.

Applied to agents: your workspace is train.py. Your SOUL.md, scripts, and skills are the experiment substrate.

CODEBLOCK0

You are not just optimizing content. You are optimizing the agent itself.

---

What Can Be Mutated

The agent can propose changes to any file it owns:

Category	Examples
Behavior	New response patterns, different tone, new check routines
Workflow

New scripts, automations, cron jobs, notification flows |
| Memory | Updated MEMORY.md entries, new daily conventions |
| Identity | Revised SOUL.md directives, new operational rules |
| Skills | New skill installations, skill configurations |
| Quality | New validation logic, error handling patterns |

The agent cannot mutate: safety rules, constitution, security boundaries, or files it doesn't own.

---

Project Structure

CODEBLOCK1

---

🚀 Quick Start

CODEBLOCK2

CODEBLOCK3

---

Baseline Metrics

Track what matters for the agent's mission. Examples:

Mission	Metric	How to Measure
Task completion	INLINECODE1	% tasks completed vs assigned
Response quality

output_quality_score | Human rating 1-10 or diff-based |
| Speed | avg_response_time_s | Seconds per response |
| Self-improvement | learnings_logged | Entries added to MEMORY.md per week |
| Autonomy | escalations_to_human | Times human was unnecessarily interrupted |

Establish baseline with ≥ 10 measurements before running experiments.

---

Verdict Logic

CODEBLOCK4

For quality/rating metrics (higher is better): above thresholds apply.
For cost/latency metrics (lower is better): flip the sign in calculation.

---

Key Rules

• - ❌ One mutation at a time — test one change per experiment
• ❌ No baseline — need ≥10 measurements before experimenting
• ❌ Vibes verdicts — use actual measurements
• ❌ Mutate safety/constitution files — never
• ❌ Kill streak ≥ 3 → pause and wait for human review
• ❌ Infinite MODIFY — max one extension
• ❌ Revert a KEEP — only a newer KEEP overrides

---

Commands

Command	What
INLINECODE6	Check current state
INLINECODE7

Establish baseline | | python3 analyze.py experiments/active.md --auto | Compute verdict | | python3 evolve.py experiments/active.md | Execute KEEP verdict | | python3 evolve.py experiments/active.md --kill | Execute KILL verdict |

---

Security

• - Agents can only mutate files within their own workspace
• Safety rules and constitution are always excluded from mutation
• External API calls require human approval
• Destructive operations (rm, git reset --hard) require explicit confirmation