ITIL Ops — IT Service Management for AI Agents

Structured incident, problem, and change management adapted from ITIL 4 for autonomous agent operations.

Core Concepts

Severity Levels

Incident vs Problem vs Change

• - Incident: Something broke. Restore service ASAP. (reactive)
• Problem: Pattern of incidents. Find and fix root cause. (proactive)
• Change: Planned modification. Assess risk before executing. (controlled)

Incident Management

Detection Sources

Scan these in order of criticality:

1. 1. Service crashes — journalctl --user -u SERVICE --since "12 hours ago" for watchdog timeouts, SIGABRT, SIGSEGV, core dumps
2. Cron failures — consecutive error count > 2 in job state files
3. Health endpoints — HTTP health checks returning non-200
4. Resource pressure — disk > 80%, RAM > 80%, swap active
5. Data integrity — schema validation failures, broken files, load errors

Detection Script

Run scripts/itil-review.sh to scan all sources. It outputs:

• - ITIL_CLEAR if nothing found (reply HEARTBEAT_OK)
• Formatted report with incidents and problems if issues detected

Incident Lifecycle

CODEBLOCK0

Auto-Classification Rules

CODEBLOCK1

Creating Incident Tickets

When incidents are found, create coordination tasks:

CODEBLOCK2

Problem Management

Pattern Detection

An incident becomes a problem when:

• - Same error occurs 3+ times in 24h
• Same incident type recurs across 2+ review cycles
• Multiple related incidents share a common root cause

Root Cause Analysis (RCA)

When a problem is identified:

1. 1. Gather evidence — journal logs, error messages, state files, recent changes
2. Timeline — reconstruct the sequence of events
3. 5 Whys — ask why iteratively until you reach the actual root cause
4. Fix classification:

Problem Ticket Format

CODEBLOCK3

Known Error Database

Track resolved problems in state file (itil-state.json):

CODEBLOCK4

Change Management

Pre-Change Checklist

Before modifying services, configs, or infrastructure:

1. 1. What's changing? — specific files, services, configs
2. Why? — linked incident/problem ticket
3. Risk? — what could go wrong
4. Rollback plan? — how to undo if it breaks
5. Test? — how to verify it worked
6. Notify? — does the human need to know

Change Categories

Post-Change Verification

After any change:

1. 1. Check service status — INLINECODE4
2. Watch logs for 60s — INLINECODE5
3. Run health check — INLINECODE6
4. Verify no new errors in first 5 minutes

Event Management

Log Monitoring Patterns

CODEBLOCK5

Health Check Endpoints

Check services with curl:
CODEBLOCK6

Configure endpoints in the review script for your environment.

Continual Improvement

Review Cadence

KPIs to Track

• - MTTR (Mean Time to Resolve) — how fast do we fix incidents?
• Incident recurrence rate — are the same things breaking?
• False positive rate — are we alerting on non-issues?
• Known error resolution — are problems getting permanent fixes?

State Tracking

The review script maintains itil-state.json with:

• - Last review timestamp and results
• Incident/problem counts per review
• System metrics (disk, RAM, restart count)
• Cross-review pattern detection data

Cron Setup

Recommended Schedule

CODEBLOCK7

File Structure

CODEBLOCK8

Integration Points

• - Coordination tasks — skcapstone coord create for incident/problem tickets
• Memory snapshots — skmemory_snapshot to record resolutions for future reference
• Heartbeat — integrate with existing heartbeat to run lightweight checks
• Cron — scheduled reviews via OpenClaw cron system
• Alerting — Telegram/Discord delivery for P1/P2 issues