Skill: Semantic Consistency Auditor
ID: 212
Name: semantic-consistency-auditor
Description: Introduces BERTScore and COMET algorithms to evaluate the semantic consistency between AI-generated clinical notes and expert gold standards from the "semantic entailment" level.
When to Use
- - Use this skill when the task needs Use semantic consistency auditor for academic writing workflows that need structured execution, explicit assumptions, and clear output boundaries.
- Use this skill for academic writing tasks that require explicit assumptions, bounded scope, and a reproducible output format.
- Use this skill when you need a documented fallback path for missing inputs, execution errors, or partial evidence.
Key Features
- - Scope-focused workflow aligned to: Use semantic consistency auditor for academic writing workflows that need structured execution, explicit assumptions, and clear output boundaries.
- Packaged executable path(s):
scripts/main.py. - Reference material available in
references/ for task-specific guidance. - Structured execution path designed to keep outputs consistent and reviewable.
Dependencies
See ## Prerequisites above for related details.
- -
Python: 3.10+. Repository baseline for current packaged skills. - INLINECODE5 :
unspecified. Declared in requirements.txt. - INLINECODE8 :
unspecified. Declared in requirements.txt. - INLINECODE11 :
unspecified. Declared in requirements.txt. - INLINECODE14 :
unspecified. Declared in requirements.txt. - INLINECODE17 :
unspecified. Declared in requirements.txt. - INLINECODE20 :
unspecified. Declared in requirements.txt.
Example Usage
See ## Usage above for related details.
CODEBLOCK0
Example run plan:
- 1. Confirm the user input, output path, and any required config values.
- Edit the in-file
CONFIG block or documented parameters if the script uses fixed settings. - Run
python scripts/main.py with the validated inputs. - Review the generated output and return the final artifact with any assumptions called out.
Implementation Details
See ## Workflow above for related details.
- - Execution model: validate the request, choose the packaged workflow, and produce a bounded deliverable.
- Input controls: confirm the source files, scope limits, output format, and acceptance criteria before running any script.
- Primary implementation surface:
scripts/main.py. - Reference guidance:
references/ contains supporting rules, prompts, or checklists. - Parameters to clarify first: input path, output path, scope filters, thresholds, and any domain-specific constraints.
- Output discipline: keep results reproducible, identify assumptions explicitly, and avoid undocumented side effects.
Quick Check
Use this command to verify that the packaged script entry point can be parsed before deeper execution.
CODEBLOCK1
Audit-Ready Commands
Use these concrete commands for validation. They are intentionally self-contained and avoid placeholder paths.
CODEBLOCK2
Workflow
- 1. Confirm the user objective, required inputs, and non-negotiable constraints before doing detailed work.
- Validate that the request matches the documented scope and stop early if the task would require unsupported assumptions.
- Use the packaged script path or the documented reasoning path with only the inputs that are actually available.
- Return a structured result that separates assumptions, deliverables, risks, and unresolved items.
- If execution fails or inputs are incomplete, switch to the fallback path and state exactly what blocked full completion.
Overview
Semantic Consistency Auditor is a medical AI evaluation tool used to assess the semantic consistency between AI-generated clinical notes and expert-written gold standards from a semantic level. This tool is not limited to traditional string matching or bag-of-words models, but uses deep learning models to understand semantic entailment relationships, capable of identifying expressions with different wording but similar meaning.
Algorithms
1. BERTScore
BERTScore uses pre-trained BERT model contextual embeddings to calculate similarity between candidate text and reference text:
- - Precision: How much semantics in the candidate text is covered by the reference text
- Recall: How much semantics in the reference text is covered by the candidate text
- F1 Score: Harmonic mean of Precision and Recall
2. COMET (Cross-lingual Optimized Metric for Evaluation of Translation)
COMET is a neural network-based evaluation metric originally used for machine translation evaluation, applicable to semantic entailment tasks:
- - Uses XLM-RoBERTa encoder to capture deep semantics
- Outputs semantic consistency scores between 0-1
- Gives high scores to semantically equivalent but differently expressed text
Installation
CODEBLOCK3
Configuration
Configure in ~/.openclaw/skills/semantic-consistency-auditor/config.yaml:
CODEBLOCK4
Usage
Command Line
CODEBLOCK5
Python API
CODEBLOCK6
Input Format
Single Case (Command Line)
Pass text directly through --ai-generated and --gold-standard parameters.
Batch Evaluation File (JSON)
CODEBLOCK7
Output Format
Summary Mode
CODEBLOCK8
Detailed Mode
CODEBLOCK9
Error Handling
- - If required inputs are missing, state exactly which fields are missing and request only the minimum additional information.
- If the task goes outside the documented scope, stop instead of guessing or silently widening the assignment.
- If
scripts/main.py fails, report the failure point, summarize what still can be completed safely, and provide a manual fallback. - Do not fabricate files, citations, data, search results, or execution outcomes.
Performance Notes
- - BERTScore: First run will download model (approximately 400MB-1GB)
- COMET: First run will download model (approximately 500MB-1.5GB)
- GPU Acceleration: Significantly improves evaluation speed in CUDA environment
- Batch Processing: Recommended for batch evaluation to fully utilize GPU parallel capability
References
- 1. Zhang et al. "BERTScore: Evaluating Text Generation with BERT" ICLR 2020
- Rei et al. "COMET: A Neural Framework for MT Evaluation" EMNLP 2020
- Medical Record Standardization Evaluation Guidelines (National Health Commission)
Changelog
- - v1.0.0 (2026-02-06): Initial version, supports dual-algorithm evaluation with BERTScore and COMET
Prerequisites
CODEBLOCK10
Evaluation Criteria
Success Metrics
- - [ ] Successfully executes main functionality
- [ ] Output meets quality standards
- [ ] Handles edge cases gracefully
- [ ] Performance is acceptable
Test Cases
- 1. Basic Functionality: Standard input → Expected output
- Edge Case: Invalid input → Graceful error handling
- Performance: Large dataset → Acceptable processing time
Output Requirements
Every final response should make these items explicit when they are relevant:
- - Objective or requested deliverable
- Inputs used and assumptions introduced
- Workflow or decision path
- Core result, recommendation, or artifact
- Constraints, risks, caveats, or validation needs
- Unresolved items and next-step checks
Input Validation
This skill accepts requests that match the documented purpose of semantic-consistency-auditor and include enough context to complete the workflow safely.
Do not continue the workflow when the request is out of scope, missing a critical input, or would require unsupported assumptions. Instead respond:
INLINECODE34 only handles its documented workflow. Please provide the missing required inputs or switch to a more suitable skill.
References
Response Template
Use the following fixed structure for non-trivial requests:
- 1. Objective
- Inputs Received
- Assumptions
- Workflow
- Deliverable
- Risks and Limits
- Next Checks
If the request is simple, you may compress the structure, but still keep assumptions and limits explicit when they affect correctness.
技能：语义一致性审核器
ID: 212
名称: semantic-consistency-auditor
描述: 引入BERTScore和COMET算法，从“语义蕴含”层面评估AI生成的临床笔记与专家金标准之间的语义一致性。
使用时机
- - 当任务需要使用语义一致性审核器进行学术写作工作流，且需要结构化执行、明确假设和清晰输出边界时，使用此技能。
- 当学术写作任务需要明确假设、限定范围和可复现的输出格式时，使用此技能。
- 当需要针对缺失输入、执行错误或部分证据提供有文档记录的备用路径时，使用此技能。
主要特性
- - 聚焦范围的工作流，对齐于：使用语义一致性审核器进行需要结构化执行、明确假设和清晰输出边界的学术写作工作流。
- 打包的可执行路径：scripts/main.py。
- 参考资料位于 references/ 目录，提供任务特定指导。
- 结构化执行路径，旨在保持输出一致性和可审查性。
依赖项
相关详情请参见上文的## 先决条件。
- - Python: 3.10+。当前打包技能的仓库基线。
- bert_score: 未指定。在 requirements.txt 中声明。
- comet: 未指定。在 requirements.txt 中声明。
- dataclasses: 未指定。在 requirements.txt 中声明。
- numpy: 未指定。在 requirements.txt 中声明。
- torch: 未指定。在 requirements.txt 中声明。
- yaml: 未指定。在 requirements.txt 中声明。
使用示例
相关详情请参见上文的## 用法。
bash
cd 20260318/scientific-skills/Academic Writing/semantic-consistency-auditor
python -m py_compile scripts/main.py
python scripts/main.py --help
示例运行计划：
- 1. 确认用户输入、输出路径以及任何必需的配置值。
- 如果脚本使用固定设置，编辑文件内的 CONFIG 块或文档化参数。
- 使用验证后的输入运行 python scripts/main.py。
- 审查生成的输出，并返回最终产物，同时注明所有假设。
实现细节
相关详情请参见上文的## 工作流。
- - 执行模型：验证请求，选择打包的工作流，并生成有边界的可交付成果。
- 输入控制：在运行任何脚本前，确认源文件、范围限制、输出格式和验收标准。
- 主要实现面：scripts/main.py。
- 参考指南：references/ 包含支持性规则、提示或检查清单。
- 需优先明确的参数：输入路径、输出路径、范围过滤器、阈值以及任何领域特定约束。
- 输出纪律：保持结果可复现，明确标识假设，避免未文档化的副作用。
快速检查
使用此命令验证打包的脚本入口点在深入执行前可被解析。
bash
python -m py_compile scripts/main.py
审计就绪命令
使用这些具体命令进行验证。它们有意保持自包含，避免使用占位符路径。
bash
python -m py_compile scripts/main.py
python scripts/main.py --help
工作流
- 1. 在进行详细工作前，确认用户目标、所需输入和不可协商的约束条件。
- 验证请求是否与文档化范围匹配，如果任务需要不支持的假设，则尽早停止。
- 使用打包的脚本路径或文档化的推理路径，仅使用实际可用的输入。
- 返回结构化结果，将假设、可交付成果、风险和未解决事项分开。
- 如果执行失败或输入不完整，切换到备用路径，并明确说明阻止完整执行的具体原因。
概述
语义一致性审核器是一种医学AI评估工具，用于从语义层面评估AI生成的临床笔记与专家编写的金标准之间的语义一致性。该工具不局限于传统的字符串匹配或词袋模型，而是使用深度学习模型理解语义蕴含关系，能够识别措辞不同但含义相似的表达。
算法
1. BERTScore
BERTScore使用预训练的BERT模型上下文嵌入来计算候选文本与参考文本之间的相似度：
- - 精确率：候选文本中的语义被参考文本覆盖的程度
- 召回率：参考文本中的语义被候选文本覆盖的程度
- F1分数：精确率和召回率的调和平均值
2. COMET（跨语言翻译评估优化指标）
COMET是一种基于神经网络的评估指标，最初用于机器翻译评估，适用于语义蕴含任务：
- - 使用XLM-RoBERTa编码器捕获深层语义
- 输出0-1之间的语义一致性分数
- 对语义等价但表达不同的文本给予高分
安装
text
创建虚拟环境（推荐）
python -m venv venv
source venv/bin/activate # Linux/Mac
或 venv\Scripts\activate # Windows
安装依赖
pip install bertscore comet-ml transformers torch
配置
在 ~/.openclaw/skills/semantic-consistency-auditor/config.yaml 中配置：
yaml
BERTScore配置
bertscore:
model: microsoft/deberta-xlarge-mnli # 中文可使用 bert-base-chinese
lang: zh # 语言代码：zh, en等
rescale
withbaseline: true
device: auto # auto, cpu, cuda
COMET配置
comet:
model: Unbabel/wmt22-comet-da # COMET模型
batch_size: 8
device: auto
评估阈值
thresholds:
bertscore_f1: 0.85
comet_score: 0.75
semantic_consistency: 0.80 # 综合分数阈值
用法
命令行
text
评估单个病例对
python scripts/main.py \
--ai-generated 患者发热3天，最高体温39°C，伴有咳嗽。 \
--gold-standard 患者主诉发热3天，最高体温39°C，伴有咳嗽症状。 \
--output results.json
从JSON文件批量评估
python scripts/main.py \
--input-file batch_cases.json \
--output results.json \
--format detailed
使用特定模型
python scripts/main.py \
--ai-generated ... \
--gold-standard ... \
--bert-model bert-base-chinese \
--comet-model Unbabel/wmt20-comet-da
Python API
python
from semanticconsistencyauditor import SemanticConsistencyAuditor
初始化评估器
auditor = SemanticConsistencyAuditor(
bert_model=microsoft/deberta-xlarge-mnli,
comet_model=Unbabel/wmt22-comet-da,
lang=zh
)
评估单个病例
result = auditor.evaluate(
ai_text=患者发热3天...,
gold_text=患者主诉发热3天...
)
print(fBERTScore F1: {result[bertscore][f1]:.4f})
print(fCOMET分数: {result[comet][score]:.4f})
print(f一致性: {result[consistency]:.4f})
print(f通过: {result[passed]})
批量评估
results = auditor.evaluate_batch([
{ai: ..., gold: ...},
{ai: ..., gold: ...}
])
输入格式
单个病例（命令行）
通过 --ai-generated 和 --gold-standard 参数直接传递文本。
批量评估文件（JSON）
json
[
{
case_id: CASE001,
ai_generated: 患者发热3天，最高体温39°C，伴有咳嗽。,
gold_standard: 患者主诉发热3天，最高体温39°C，伴有咳嗽症状。,
metadata: {
department: 呼吸科,
disease_type: 上呼吸道感染
}
},
{
case_id: CASE002,
ai_generated: ...,
gold_standard: ...
}
]
输出格式
摘要模式
json
{
overall: {
total_cases: 100,
passed_cases: 85,
pass_rate: 0.85,
avgbertscoref1: 0.8923,
avgcometscore: 0.8234,
avg_consistency: 0.8579
},
thresholds: {
bertscore_f1: 0.85,
comet_score: 0.75,
semantic_consistency: 0.80
}
}
详细模式
json
