Thanh Quang
add RAG evaluation and guarded fusion retrieval
1239f44
|
Raw History Blame Contribute Delete
7.43 kB

A newer version of the Gradio SDK is available: 6.32.0

Upgrade

RAG evaluation

This package has two deliberately separate evaluation paths.

Deterministic retrieval evaluation

  • evaluation_dataset.py validates JSONL records and converts the legacy CSV.
  • retrieval_evaluator.py measures Recall@1/3/5/10, MRR, nDCG, and p50/p95 retrieval latency without calling an LLM.
  • run_retrieval.py rebuilds and evaluates isolated collections for rc_500_50, rc_800_100, rc_1000_150, and rc_1500_200.
  • external_benchmarks.py downloads and normalizes XQuAD-VI or BEIR SciFact.
  • run_external_retrieval.py indexes each public corpus independently and evaluates document-level qrels across the same four chunking strategies.
  • pdf_corpus.py freezes an explicit PDF corpus, prepares a human annotation queue, and applies only completed reviews.

Convert the benchmark:

uv run python -m src.evaluation.evaluation_dataset \
  'data/[Data]-Benchmark-Rag.csv' \
  --output data/benchmark_rag.jsonl \
  --report data/benchmark_rag_annotation_report.json

Annotate the records listed in the report with verified source_file and gold_pages, or exact gold_chunk_ids. Records lacking verified gold source metadata remain visible in per-case output but are excluded from aggregate ranking metrics.

Evaluate an existing collection:

uv run python -m src.evaluation.retrieval_evaluator \
  data/benchmark_rag.jsonl \
  --collection doculearn_rag \
  --k 10 \
  --output-dir evaluation-results/retrieval/current

Evaluate all chunking strategies:

uv run python -m src.evaluation.run_retrieval \
  data/benchmark_rag.jsonl \
  --k 10 \
  --output-dir evaluation-results/retrieval/chunking

Use --filters-json with either command to apply production-compatible metadata filters.

Product PDF corpus and annotation

The production PDF discovery function only reads top-level files in data/. Evaluation can instead use an explicit immutable manifest without changing production ingestion:

uv run python -m src.evaluation.pdf_corpus build-manifest \
  --pdf-dir 'data/[Data]-Documents-PDFs' \
  --output data/evaluation/pdf_corpus_manifest.json

The manifest stores relative paths, SHA-256 hashes, byte sizes, and page counts. Loading it fails if a PDF has moved or changed.

Generate the annotation queue:

uv run python -m src.evaluation.pdf_corpus build-queue \
  data/benchmark_rag.jsonl \
  --manifest data/evaluation/pdf_corpus_manifest.json \
  --output evaluation-results/annotation/benchmark_rag_queue.jsonl \
  --top-n 5

Candidate pages use deterministic lexical overlap from the question and reference answer. They are navigation aids, not gold labels. Review every record and edit its review object:

  • status: "approved" requires a verified source_file and gold_pages;
  • status: "unanswerable" requires both source fields to remain empty;
  • status: "pending" blocks dataset export.

Apply a completed queue:

uv run python -m src.evaluation.pdf_corpus apply-queue \
  data/benchmark_rag.jsonl \
  --manifest data/evaluation/pdf_corpus_manifest.json \
  --queue evaluation-results/annotation/benchmark_rag_queue.jsonl \
  --output data/benchmark_rag_annotated.jsonl \
  --report data/benchmark_rag_annotation_apply_report.json

Run the annotated dataset against exactly the frozen PDFs:

uv run python -m src.evaluation.run_retrieval \
  data/benchmark_rag_annotated.jsonl \
  --corpus-manifest data/evaluation/pdf_corpus_manifest.json \
  --k 10 \
  --output-dir evaluation-results/retrieval/product-pdf

After choosing a chunking strategy, test expanded dense retrieval with BM25-style lexical reranking independently of production:

uv run python -m src.evaluation.run_reranking \
  data/benchmark_rag_annotated.jsonl \
  --collection doculearn_product_eval__rc_800_100 \
  --candidate-k 50 \
  --k 10 \
  --dense-weight 0.25 \
  --max-chunks-per-page 1 \
  --output-dir evaluation-results/retrieval/product-pdf-rerank \
  --baseline-cases \
    evaluation-results/retrieval/product-pdf-baseline/rc_800_100/cases.jsonl \
  --error-analysis-output \
    evaluation-results/retrieval/product-pdf-rerank/error_analysis.json

This retrieves a larger dense candidate pool, combines normalized dense and lexical scores, and limits duplicate page occupancy. It does not call an LLM or change src.rag.retrieve; promotion to production requires a separate decision after comparing quality, latency, and per-question regressions.

Public benchmark baselines

Prepare the Vietnamese XQuAD benchmark:

uv run python -m src.evaluation.external_benchmarks xquad-vi \
  --output-dir data/evaluation/external/xquad-vi

Prepare BEIR SciFact:

uv run python -m src.evaluation.external_benchmarks beir-scifact \
  --output-dir data/evaluation/external/beir-scifact

Each directory contains corpus.jsonl, queries.jsonl, and manifest.json. External query records use gold_document_ids because BEIR qrels can identify multiple relevant documents. This is deliberately separate from the PDF dataset schema, whose gold evidence is expressed as pages or exact chunks.

Run all four chunking strategies without an LLM:

uv run python -m src.evaluation.run_external_retrieval \
  data/evaluation/external/xquad-vi \
  --output-dir evaluation-results/retrieval/xquad-vi

uv run python -m src.evaluation.run_external_retrieval \
  data/evaluation/external/beir-scifact \
  --output-dir evaluation-results/retrieval/beir-scifact

Validate the same expanded dense plus lexical reranker on one selected chunking strategy before considering production promotion:

uv run python -m src.evaluation.run_external_reranking \
  data/evaluation/external/xquad-vi \
  --output-dir evaluation-results/retrieval/xquad-vi-rerank \
  --strategy rc_800_100 \
  --candidate-k 50 \
  --k 10 \
  --dense-weight 0.25 \
  --max-chunks-per-page 1

uv run python -m src.evaluation.run_external_reranking \
  data/evaluation/external/beir-scifact \
  --output-dir evaluation-results/retrieval/beir-scifact-rerank \
  --strategy rc_800_100 \
  --candidate-k 50 \
  --k 10 \
  --dense-weight 0.25 \
  --max-chunks-per-page 1

Compare each summary.json and cases.jsonl with the matching dense baseline. Promotion decisions should consider aggregate quality, p50/p95 latency, and per-query regressions rather than relying on a single metric.

Use --limit 10 for a quick integration smoke test. Omit it for a reportable baseline. Ranking is measured at document level while the indexed documents are still split by each registered chunking strategy. Repeated chunks from the same document do not receive duplicate relevance credit. External baselines use exact in-memory cosine search with the configured production embedding model, avoiding large temporary local Qdrant collections.

Public benchmarks test language and retrieval-engine behavior. They do not replace the document-grounded benchmark for the PDFs used by DocuLearn-RAG.

Ragas answer evaluation

ragas_evaluator.py converts generated DocuLearn-RAG answers into a Ragas dataset. run_chunking.py runs the full retrieval-plus-generation pipeline and writes mean answer-quality metrics.

This path uses the configured local embedding model and Gemini as judge. It requires GEMINI_API_KEY, may incur API usage, and must not be confused with the LLM-free retrieval baseline.