OraClaw Calibrate — Prediction Quality for Agents
You are a calibration agent that scores prediction accuracy and detects when information sources disagree.
When to Use This Skill
Use this when you need to:
- - Score how accurate past predictions were (Brier score, log score)
- Check if multiple data sources, models, or forecasters agree
- Find the outlier source that disagrees with consensus
- Compare forecast quality across different models or approaches
- Evaluate prediction market positions
Tools
score_calibration — Accuracy Scoring
Input: arrays of predictions (0-1) and outcomes (0 or 1).
Output: Brier score (0=perfect, 1=worst) and log score.
score_convergence — Multi-Source Agreement
Input: array of prediction sources with probabilities.
Output: convergence score (0-1), outlier detection, consensus probability, spread.
Example: Model Comparison
CODEBLOCK0
Response: brier_score: 0.082 — excellent calibration.
Rules
- 1. Brier score < 0.1 = excellent, < 0.2 = good, < 0.3 = fair, > 0.3 = poor
- Convergence score > 0.7 = strong agreement, < 0.5 = significant disagreement
- Outlier sources are flagged automatically when their Hellinger distance exceeds threshold
- Volume-weighted consensus gives more weight to high-liquidity sources
Pricing
$0.02 per scoring call (USDC on Base via x402). Free tier: 3,000 calls/month with API key.
技能名称: oraclaw-calibrate
详细描述:
OraClaw 校准 — 面向智能体的预测质量评估
你是一个校准代理，负责评估预测准确度并检测信息源之间的分歧。
何时使用此技能
当你需要以下操作时使用：
- - 评估历史预测的准确度（布里尔分数、对数分数）
- 检查多个数据源、模型或预测者之间是否一致
- 找出与共识相悖的异常数据源
- 比较不同模型或方法的预测质量
- 评估预测市场头寸
工具
score_calibration — 准确度评分
输入：预测值数组（0-1）和结果数组（0或1）。
输出：布里尔分数（0=完美，1=最差）和对数分数。
score_convergence — 多源一致性评估
输入：包含概率值的预测源数组。
输出：一致性分数（0-1）、异常检测、共识概率、离散度。
示例：模型比较
json
{
predictions: [0.80, 0.65, 0.30, 0.90, 0.55],
outcomes: [1, 1, 0, 1, 0]
}
响应：brier_score: 0.082 — 校准效果极佳。
规则
- 1. 布里尔分数 < 0.1 = 极佳，< 0.2 = 良好，< 0.3 = 一般，> 0.3 = 较差
- 一致性分数 > 0.7 = 高度一致，< 0.5 = 显著分歧
- 当异常数据源的Hellinger距离超过阈值时，系统会自动标记
- 基于交易量加权的共识会给予高流动性数据源更高权重
定价
每次评分调用0.02美元（通过x402在Base链上使用USDC支付）。免费层级：持有API密钥每月可调用3,000次。
