Spaces:
Sleeping
Download src/evaluation/README.md from Jky71/doculearn-rag: direct link, hf CLI and curl.
- Browser
- Download file 7.43 kB
-
https://huggingface.co/spaces/Jky71/doculearn-rag/resolve/main/src/evaluation/README.md
- Command line
-
hf download hf://spaces/Jky71/doculearn-rag/src/evaluation/README.md
-
curl -L -o README.md https://huggingface.co/spaces/Jky71/doculearn-rag/resolve/main/src/evaluation/README.md
A newer version of the Gradio SDK is available: 6.32.0
RAG evaluation
This package has two deliberately separate evaluation paths.
Deterministic retrieval evaluation
evaluation_dataset.pyvalidates JSONL records and converts the legacy CSV.retrieval_evaluator.pymeasures Recall@1/3/5/10, MRR, nDCG, and p50/p95 retrieval latency without calling an LLM.run_retrieval.pyrebuilds and evaluates isolated collections forrc_500_50,rc_800_100,rc_1000_150, andrc_1500_200.external_benchmarks.pydownloads and normalizes XQuAD-VI or BEIR SciFact.run_external_retrieval.pyindexes each public corpus independently and evaluates document-level qrels across the same four chunking strategies.pdf_corpus.pyfreezes an explicit PDF corpus, prepares a human annotation queue, and applies only completed reviews.
Convert the benchmark:
uv run python -m src.evaluation.evaluation_dataset \
'data/[Data]-Benchmark-Rag.csv' \
--output data/benchmark_rag.jsonl \
--report data/benchmark_rag_annotation_report.json
Annotate the records listed in the report with verified source_file and
gold_pages, or exact gold_chunk_ids. Records lacking verified gold source
metadata remain visible in per-case output but are excluded from aggregate
ranking metrics.
Evaluate an existing collection:
uv run python -m src.evaluation.retrieval_evaluator \
data/benchmark_rag.jsonl \
--collection doculearn_rag \
--k 10 \
--output-dir evaluation-results/retrieval/current
Evaluate all chunking strategies:
uv run python -m src.evaluation.run_retrieval \
data/benchmark_rag.jsonl \
--k 10 \
--output-dir evaluation-results/retrieval/chunking
Use --filters-json with either command to apply production-compatible
metadata filters.
Product PDF corpus and annotation
The production PDF discovery function only reads top-level files in data/.
Evaluation can instead use an explicit immutable manifest without changing
production ingestion:
uv run python -m src.evaluation.pdf_corpus build-manifest \
--pdf-dir 'data/[Data]-Documents-PDFs' \
--output data/evaluation/pdf_corpus_manifest.json
The manifest stores relative paths, SHA-256 hashes, byte sizes, and page counts. Loading it fails if a PDF has moved or changed.
Generate the annotation queue:
uv run python -m src.evaluation.pdf_corpus build-queue \
data/benchmark_rag.jsonl \
--manifest data/evaluation/pdf_corpus_manifest.json \
--output evaluation-results/annotation/benchmark_rag_queue.jsonl \
--top-n 5
Candidate pages use deterministic lexical overlap from the question and
reference answer. They are navigation aids, not gold labels. Review every
record and edit its review object:
status: "approved"requires a verifiedsource_fileandgold_pages;status: "unanswerable"requires both source fields to remain empty;status: "pending"blocks dataset export.
Apply a completed queue:
uv run python -m src.evaluation.pdf_corpus apply-queue \
data/benchmark_rag.jsonl \
--manifest data/evaluation/pdf_corpus_manifest.json \
--queue evaluation-results/annotation/benchmark_rag_queue.jsonl \
--output data/benchmark_rag_annotated.jsonl \
--report data/benchmark_rag_annotation_apply_report.json
Run the annotated dataset against exactly the frozen PDFs:
uv run python -m src.evaluation.run_retrieval \
data/benchmark_rag_annotated.jsonl \
--corpus-manifest data/evaluation/pdf_corpus_manifest.json \
--k 10 \
--output-dir evaluation-results/retrieval/product-pdf
After choosing a chunking strategy, test expanded dense retrieval with BM25-style lexical reranking independently of production:
uv run python -m src.evaluation.run_reranking \
data/benchmark_rag_annotated.jsonl \
--collection doculearn_product_eval__rc_800_100 \
--candidate-k 50 \
--k 10 \
--dense-weight 0.25 \
--max-chunks-per-page 1 \
--output-dir evaluation-results/retrieval/product-pdf-rerank \
--baseline-cases \
evaluation-results/retrieval/product-pdf-baseline/rc_800_100/cases.jsonl \
--error-analysis-output \
evaluation-results/retrieval/product-pdf-rerank/error_analysis.json
This retrieves a larger dense candidate pool, combines normalized dense and
lexical scores, and limits duplicate page occupancy. It does not call an LLM
or change src.rag.retrieve; promotion to production requires a separate
decision after comparing quality, latency, and per-question regressions.
Public benchmark baselines
Prepare the Vietnamese XQuAD benchmark:
uv run python -m src.evaluation.external_benchmarks xquad-vi \
--output-dir data/evaluation/external/xquad-vi
Prepare BEIR SciFact:
uv run python -m src.evaluation.external_benchmarks beir-scifact \
--output-dir data/evaluation/external/beir-scifact
Each directory contains corpus.jsonl, queries.jsonl, and manifest.json.
External query records use gold_document_ids because BEIR qrels can identify
multiple relevant documents. This is deliberately separate from the PDF
dataset schema, whose gold evidence is expressed as pages or exact chunks.
Run all four chunking strategies without an LLM:
uv run python -m src.evaluation.run_external_retrieval \
data/evaluation/external/xquad-vi \
--output-dir evaluation-results/retrieval/xquad-vi
uv run python -m src.evaluation.run_external_retrieval \
data/evaluation/external/beir-scifact \
--output-dir evaluation-results/retrieval/beir-scifact
Validate the same expanded dense plus lexical reranker on one selected chunking strategy before considering production promotion:
uv run python -m src.evaluation.run_external_reranking \
data/evaluation/external/xquad-vi \
--output-dir evaluation-results/retrieval/xquad-vi-rerank \
--strategy rc_800_100 \
--candidate-k 50 \
--k 10 \
--dense-weight 0.25 \
--max-chunks-per-page 1
uv run python -m src.evaluation.run_external_reranking \
data/evaluation/external/beir-scifact \
--output-dir evaluation-results/retrieval/beir-scifact-rerank \
--strategy rc_800_100 \
--candidate-k 50 \
--k 10 \
--dense-weight 0.25 \
--max-chunks-per-page 1
Compare each summary.json and cases.jsonl with the matching dense baseline.
Promotion decisions should consider aggregate quality, p50/p95 latency, and
per-query regressions rather than relying on a single metric.
Use --limit 10 for a quick integration smoke test. Omit it for a reportable
baseline. Ranking is measured at document level while the indexed documents
are still split by each registered chunking strategy. Repeated chunks from the
same document do not receive duplicate relevance credit. External baselines
use exact in-memory cosine search with the configured production embedding
model, avoiding large temporary local Qdrant collections.
Public benchmarks test language and retrieval-engine behavior. They do not replace the document-grounded benchmark for the PDFs used by DocuLearn-RAG.
Ragas answer evaluation
ragas_evaluator.py converts generated DocuLearn-RAG answers into a Ragas
dataset. run_chunking.py runs the full retrieval-plus-generation pipeline
and writes mean answer-quality metrics.
This path uses the configured local embedding model and Gemini as judge. It
requires GEMINI_API_KEY, may incur API usage, and must not be confused with
the LLM-free retrieval baseline.