RAG Retrieval Regression Gate

Test whether a RAG or enterprise knowledge assistant still retrieves the right evidence before the generated answer hides a retrieval failure.

Three-Step Workflow

1. 1. Build a reviewed golden_queries.json with expected source document IDs, critical cases, no-answer cases and tags. Export the ranked document IDs returned by the current baseline and a candidate configuration.
2. Run the local evaluator or import the n8n workflow. It calculates Hit@K, Recall@K, MRR, nDCG and no-answer correctness for both runs, then identifies per-query regressions.
3. Block release when thresholds fail or a critical query regresses. Send the generated repair queue to the knowledge owner, then rerun the same dataset after changing one retrieval variable.

Quick Start

CODEBLOCK0

Outputs:

• - rag_eval_report.json: baseline, candidate, deltas and per-query evidence.
• INLINECODE2: missing expected sources and retrieved alternatives, sorted by severity.
• INLINECODE3: clear PASS/BLOCK decision and threshold checks.

Rules

• - Use source IDs, not generated prose, as the deterministic retrieval truth.
• Have a domain owner approve expected documents and critical cases.
• Keep no-answer cases so a retriever is not rewarded for returning irrelevant context.
• Change one retrieval variable per candidate run when diagnosing regressions.
• Treat model or AI judge scores as optional supporting evidence, not the only gate.
• Require human approval before changing production indexes, filters, prompts or knowledge sources.

Read references/usage.md for schemas and recovery. Read references/metrics.md before changing thresholds.