Rag Evaluator

AI-powered RAG (Retrieval-Augmented Generation) evaluation toolkit. Configure, benchmark, compare, and optimize your RAG pipelines from the command line. Track prompts, evaluations, fine-tuning experiments, costs, and usage — all with persistent local logging and full export capabilities.

Commands

Run rag-evaluator <command> [args] to use.

Each domain command (configure, benchmark, compare, etc.) works in two modes:

• - Without arguments: displays the most recent 20 entries from that category
• With arguments: logs the input with a timestamp and saves to the category log file

Data Storage

All data is stored locally in ~/.local/share/rag-evaluator/:

• - Each command creates its own log file (e.g., configure.log, benchmark.log)
• A unified history.log tracks all activity across commands
• Entries are stored in timestamp|value pipe-delimited format
• Export supports JSON, CSV, and plain text formats

Requirements

• - Bash 4+ with set -euo pipefail strict mode
• Standard Unix utilities: date, wc, du, tail, grep, sed, INLINECODE32
• No external dependencies or API keys required

When to Use

1. 1. Evaluating RAG pipeline quality — log evaluation scores, compare retrieval strategies, and track improvements over time
2. Benchmarking different configurations — run benchmarks across embedding models, chunk sizes, or retrieval methods and compare results side by side
3. Tracking costs and usage — monitor API costs and token usage across experiments to stay within budget
4. Managing prompt engineering — log prompt variations, test them against your pipeline, and analyze which templates perform best
5. Generating reports for stakeholders — export evaluation data as JSON/CSV for dashboards, or generate text reports summarizing RAG performance

Examples

CODEBLOCK0

Output

All commands output to stdout. Redirect to a file if needed:

CODEBLOCK1

Configuration

Set DATA_DIR by modifying the script, or use the default: ~/.local/share/rag-evaluator/