AI DevOps Toolkit — Observability for Local AI Fleets
DevOps tooling for running local LLM inference at production quality. This DevOps skill provides the observability, tracing, and health monitoring layer for an Ollama Herd fleet. Every DevOps workflow — from request tracing to capacity planning — runs through a single SQLite-backed observability stack.
DevOps Prerequisites
CODEBLOCK0
Package: ollama-herd | Repo: github.com/geeks-accelerator/ollama-herd
DevOps Scope
This DevOps toolkit assumes you have an Ollama Herd router running at http://localhost:11435 with one or more node agents reporting in. It focuses on the DevOps operational side: are requests succeeding? what's slow? which apps consume the most tokens? are nodes healthy? is capacity adequate?
DevOps Observability Stack
Everything in this DevOps observability layer is backed by SQLite at ~/.fleet-manager/latency.db. No external databases, no time-series infrastructure. Query DevOps traces with standard sqlite3.
CODEBLOCK1
DevOps Health Checks
Automated DevOps fleet health analysis
CODEBLOCK2
Fifteen DevOps checks, each returning a severity (info/warning/critical) and recommendation:
| DevOps Check | What it detects |
|---|
| Offline nodes | Nodes that stopped sending heartbeats |
| Degraded nodes |
Nodes reporting errors or high memory pressure |
| Memory pressure | Nodes approaching memory limits |
| Underutilized nodes | Healthy nodes not receiving traffic |
| VRAM fallbacks | Requests rerouted to loaded alternatives to avoid cold loads |
| Version mismatch | Nodes running different versions than the router |
| Context protection | num_ctx values stripped or models upgraded to prevent reloads |
| Zombie reaper | Stuck in-flight requests cleaned up |
| Model thrashing | Models loading/unloading frequently (memory contention) |
| Request timeouts | Requests exceeding expected DevOps latency thresholds |
| Error rates | Elevated failure rates per model or per node |
DevOps node-level status
CODEBLOCK3
DevOps Request Tracing
Every DevOps routing decision is recorded with full observability context.
Recent DevOps traces
CODEBLOCK4
Each DevOps trace includes: requestid, model, originalmodel (before fallback), nodeid, score, scoresbreakdown (all 7 signals), status, latencyms, timetofirsttokenms, prompttokens, completiontokens, retrycount, fallback_used, tags.
DevOps failure investigation
CODEBLOCK5
DevOps Latency Analysis
CODEBLOCK6
DevOps Per-Application Analytics
Tag requests to track DevOps usage per application, team, or environment.
DevOps request tagging
CODEBLOCK7
DevOps per-tag dashboards
CODEBLOCK8
DevOps token consumption by tag
CODEBLOCK9
DevOps Traffic Patterns
CODEBLOCK10
DevOps Capacity Planning
DevOps model recommendations per node
CODEBLOCK11
Returns DevOps recommendations based on hardware capabilities, current usage, and curated benchmark data. Use for DevOps capacity planning: which models fit on which machines, and what's the optimal mix.
DevOps usage statistics
CODEBLOCK12
DevOps Configuration
CODEBLOCK13
DevOps Log Analysis
Structured JSONL logs at ~/.fleet-manager/logs/herd.jsonl — the DevOps log layer:
CODEBLOCK14
DevOps Dashboard
Web dashboard at http://localhost:11435/dashboard. Key DevOps tabs:
- - Trends — DevOps requests/hour, latency, token throughput over 24h–7d
- Apps — DevOps per-tag analytics with daily breakdowns
- Health — automated DevOps health checks with severity and recommendations
- Model Insights — per-model DevOps latency and throughput comparison
Guardrails
- - Never restart DevOps services without explicit user confirmation.
- Never delete or modify
~/.fleet-manager/ contents. - Do not pull or delete models without user confirmation.
- Report DevOps issues to the user rather than attempting automated fixes.
- If the router isn't running, suggest
herd or uv run herd.
AI DevOps 工具包 — 本地 AI 集群的可观测性
用于以生产质量运行本地 LLM 推理的 DevOps 工具。该 DevOps 技能为 Ollama Herd 集群提供可观测性、链路追踪和健康监控层。从请求追踪到容量规划，每个 DevOps 工作流都通过基于 SQLite 的可观测性栈运行。
DevOps 前提条件
bash
pip install ollama-herd
herd # 启动 DevOps 路由器（暴露所有 DevOps 可观测性端点）
herd-node # 在每个受 DevOps 监控的节点上启动
包：ollama-herd | 仓库：github.com/geeks-accelerator/ollama-herd
DevOps 范围
该 DevOps 工具包假设你有一个运行在 http://localhost:11435 的 Ollama Herd 路由器，并有一个或多个节点代理上报信息。它专注于 DevOps 运维方面：请求是否成功？哪些环节慢？哪些应用消耗最多 token？节点是否健康？容量是否充足？
DevOps 可观测性栈
该 DevOps 可观测性层的所有数据都存储在 ~/.fleet-manager/latency.db 的 SQLite 中。无需外部数据库，无需时间序列基础设施。使用标准 sqlite3 查询 DevOps 追踪数据。
~/.fleet-manager/
├── latency.db # DevOps 追踪、延迟历史、使用统计
└── logs/
└── herd.jsonl # DevOps 结构化日志，每日轮转，保留30天
DevOps 健康检查
自动化 DevOps 集群健康分析
bash
devops_health=$(curl -s http://localhost:11435/dashboard/api/health)
echo $devops_health | python3 -m json.tool
十五项 DevOps 检查，每项返回严重级别（信息/警告/严重）和建议：
| DevOps 检查项 | 检测内容 |
|---|
| 离线节点 | 停止发送心跳的节点 |
| 降级节点 |
报告错误或高内存压力的节点 |
| 内存压力 | 接近内存限制的节点 |
| 低利用率节点 | 未接收流量的健康节点 |
| VRAM 回退 | 为避免冷加载而将请求重新路由到已加载的替代方案 |
| 版本不匹配 | 运行与路由器不同版本的节点 |
| 上下文保护 | 剥离 num_ctx 值或升级模型以防止重新加载 |
| 僵尸清理 | 清理卡住的进行中请求 |
| 模型抖动 | 模型频繁加载/卸载（内存争用） |
| 请求超时 | 超过预期 DevOps 延迟阈值的请求 |
| 错误率 | 每个模型或每个节点的故障率升高 |
DevOps 节点级状态
bash
devops
fleetstatus=$(curl -s http://localhost:11435/fleet/status)
echo $devops
fleetstatus | python3 -c
import sys, json
d = json.load(sys.stdin)
print(f\DevOps 集群: {d[fleet][nodes
online]}/{d[fleet][nodestotal]} 在线, {d[fleet][requests_active]} 活跃请求\)
for n in d[nodes]:
mem = n.get(memory, {})
cpu = n.get(cpu, {})
print(f\ {n[node
id]:20s} {n[status]:10s} CPU={cpu.get(utilizationpct,0):.0f}% 内存={mem.get(used
gb,0):.0f}/{mem.get(totalgb,0):.0f}GB 压力={mem.get(pressure,?)}\)
DevOps 请求追踪
每个 DevOps 路由决策都记录完整的可观测性上下文。
最近的 DevOps 追踪
bash
devops_traces=$(curl -s http://localhost:11435/dashboard/api/traces?limit=20)
echo $devops_traces | python3 -m json.tool
每个 DevOps 追踪包含：requestid、model、originalmodel（回退前）、nodeid、score、scoresbreakdown（全部7个信号）、status、latencyms、timetofirsttokenms、prompttokens、completiontokens、retrycount、fallback_used、tags。
DevOps 故障调查
bash
最近的 DevOps 故障及错误详情
sqlite3 ~/.fleet-manager/latency.db SELECT request
id, model, nodeid, error
message, latencyms/1000.0 as secs, datetime(timestamp, unixepoch, localtime) as time FROM request_traces WHERE status=failed ORDER BY timestamp DESC LIMIT 20
DevOps 重试频率 — 哪些节点需要关注？
sqlite3 ~/.fleet-manager/latency.db SELECT node
id, SUM(retrycount) as retries, COUNT(
) as total, ROUND(100.0 SUM(CASE WHEN status=failed THEN 1 ELSE 0 END) / COUNT(*), 1) as fail
pct FROM requesttraces GROUP BY node
id ORDER BY failpct DESC
DevOps 回退频率 — 哪些模型不可靠？
sqlite3 ~/.fleet-manager/latency.db SELECT original
model, model as fellback
to, COUNT(*) as n FROM requesttraces WHERE fallback
used=1 GROUP BY originalmodel, model ORDER BY n DESC
DevOps 延迟分析
bash
DevOps 按模型的 P50/P75/P99 延迟
sqlite3 ~/.fleet-manager/latency.db
WITH ranked AS (
SELECT model, latency_ms,
PERCENT
RANK() OVER (PARTITION BY model ORDER BY latencyms) as pct
FROM request_traces WHERE status=completed
)
SELECT model,
ROUND(MIN(CASE WHEN pct >= 0.5 THEN latency
ms END)/1000.0, 1) as p50s,
ROUND(MIN(CASE WHEN pct >= 0.75 THEN latency
ms END)/1000.0, 1) as p75s,
ROUND(MIN(CASE WHEN pct >= 0.99 THEN latency
ms END)/1000.0, 1) as p99s,
COUNT(*) as n
FROM ranked GROUP BY model HAVING n > 10 ORDER BY p75_s DESC
DevOps 首 token 时间可观测性（冷加载检测）
sqlite3 ~/.fleet-manager/latency.db SELECT node
id, model, ROUND(AVG(timeto
firsttoken
ms), 0) as avgttft
ms, ROUND(MAX(timeto
firsttoken
ms), 0) as maxttft
ms, COUNT(*) as n FROM requesttraces WHERE time
tofirst
tokenms IS NOT NULL GROUP BY node
id, model HAVING n > 5 ORDER BY avgttft_ms DESC
DevOps 异常检测 — 最慢的请求
sqlite3 ~/.fleet-manager/latency.db SELECT request
id, model, nodeid, ROUND(latency
ms/1000.0, 1) as secs, prompttokens, completion
tokens, retrycount, datetime(timestamp, unixepoch, localtime) as time FROM request
traces WHERE status=completed ORDER BY latencyms DESC LIMIT 10
DevOps 按应用分析
为请求打标签以追踪每个应用、团队或环境的 DevOps 使用情况。
DevOps 请求标签
bash
通过请求体添加 DevOps 标签
curl -s http://localhost:11435/v1/chat/completions \
-H Content-Type: application/json \
-d {model:llama3.3:70b,messages:[{role:user,content:你好}],metadata:{tags:[devops-prod,devops-code-review]}}
通过请求头添加 DevOps 标签
curl -s -H X-Herd-Tags: devops-prod, devops-code-review \
http://localhost:11435/v1/chat/completions \
-d {model:llama3.3:70b,messages:[{role:user,content:你好}]}
DevOps 按标签仪表盘
bash
curl -s http://localhost:11435/dashboard/api/apps | python3 -m json.tool
curl -s http://localhost:11435/dashboard/api/apps/daily | python3 -m json.tool
DevOps 按标签的 token 消耗
bash
sqlite3 ~/.fleet-manager/latency.db SELECT j.value as devops
tag, COUNT(*) as requests, SUM(COALESCE(prompttokens,0)) as prompt
tok, SUM(COALESCE(completiontokens,0)) as completion_tok, SUM(COALES
