Spaces:
Sleeping
Sleeping
|
Download src/evaluation/README.md from Jky71/doculearn-rag: direct link, hf CLI and curl.
- Browser
- Download file 7.43 kB
-
https://huggingface.co/spaces/Jky71/doculearn-rag/resolve/main/src/evaluation/README.md
- Command line
-
hf download hf://spaces/Jky71/doculearn-rag/src/evaluation/README.md
-
curl -L -o README.md https://huggingface.co/spaces/Jky71/doculearn-rag/resolve/main/src/evaluation/README.md
7.43 kB
| # RAG evaluation | |
| This package has two deliberately separate evaluation paths. | |
| ## Deterministic retrieval evaluation | |
| - `evaluation_dataset.py` validates JSONL records and converts the legacy CSV. | |
| - `retrieval_evaluator.py` measures Recall@1/3/5/10, MRR, nDCG, and p50/p95 | |
| retrieval latency without calling an LLM. | |
| - `run_retrieval.py` rebuilds and evaluates isolated collections for | |
| `rc_500_50`, `rc_800_100`, `rc_1000_150`, and `rc_1500_200`. | |
| - `external_benchmarks.py` downloads and normalizes XQuAD-VI or BEIR SciFact. | |
| - `run_external_retrieval.py` indexes each public corpus independently and | |
| evaluates document-level qrels across the same four chunking strategies. | |
| - `pdf_corpus.py` freezes an explicit PDF corpus, prepares a human annotation | |
| queue, and applies only completed reviews. | |
| Convert the benchmark: | |
| ```bash | |
| uv run python -m src.evaluation.evaluation_dataset \ | |
| 'data/[Data]-Benchmark-Rag.csv' \ | |
| --output data/benchmark_rag.jsonl \ | |
| --report data/benchmark_rag_annotation_report.json | |
| ``` | |
| Annotate the records listed in the report with verified `source_file` and | |
| `gold_pages`, or exact `gold_chunk_ids`. Records lacking verified gold source | |
| metadata remain visible in per-case output but are excluded from aggregate | |
| ranking metrics. | |
| Evaluate an existing collection: | |
| ```bash | |
| uv run python -m src.evaluation.retrieval_evaluator \ | |
| data/benchmark_rag.jsonl \ | |
| --collection doculearn_rag \ | |
| --k 10 \ | |
| --output-dir evaluation-results/retrieval/current | |
| ``` | |
| Evaluate all chunking strategies: | |
| ```bash | |
| uv run python -m src.evaluation.run_retrieval \ | |
| data/benchmark_rag.jsonl \ | |
| --k 10 \ | |
| --output-dir evaluation-results/retrieval/chunking | |
| ``` | |
| Use `--filters-json` with either command to apply production-compatible | |
| metadata filters. | |
| ## Product PDF corpus and annotation | |
| The production PDF discovery function only reads top-level files in `data/`. | |
| Evaluation can instead use an explicit immutable manifest without changing | |
| production ingestion: | |
| ```bash | |
| uv run python -m src.evaluation.pdf_corpus build-manifest \ | |
| --pdf-dir 'data/[Data]-Documents-PDFs' \ | |
| --output data/evaluation/pdf_corpus_manifest.json | |
| ``` | |
| The manifest stores relative paths, SHA-256 hashes, byte sizes, and page counts. | |
| Loading it fails if a PDF has moved or changed. | |
| Generate the annotation queue: | |
| ```bash | |
| uv run python -m src.evaluation.pdf_corpus build-queue \ | |
| data/benchmark_rag.jsonl \ | |
| --manifest data/evaluation/pdf_corpus_manifest.json \ | |
| --output evaluation-results/annotation/benchmark_rag_queue.jsonl \ | |
| --top-n 5 | |
| ``` | |
| Candidate pages use deterministic lexical overlap from the question and | |
| reference answer. They are navigation aids, not gold labels. Review every | |
| record and edit its `review` object: | |
| - `status: "approved"` requires a verified `source_file` and `gold_pages`; | |
| - `status: "unanswerable"` requires both source fields to remain empty; | |
| - `status: "pending"` blocks dataset export. | |
| Apply a completed queue: | |
| ```bash | |
| uv run python -m src.evaluation.pdf_corpus apply-queue \ | |
| data/benchmark_rag.jsonl \ | |
| --manifest data/evaluation/pdf_corpus_manifest.json \ | |
| --queue evaluation-results/annotation/benchmark_rag_queue.jsonl \ | |
| --output data/benchmark_rag_annotated.jsonl \ | |
| --report data/benchmark_rag_annotation_apply_report.json | |
| ``` | |
| Run the annotated dataset against exactly the frozen PDFs: | |
| ```bash | |
| uv run python -m src.evaluation.run_retrieval \ | |
| data/benchmark_rag_annotated.jsonl \ | |
| --corpus-manifest data/evaluation/pdf_corpus_manifest.json \ | |
| --k 10 \ | |
| --output-dir evaluation-results/retrieval/product-pdf | |
| ``` | |
| After choosing a chunking strategy, test expanded dense retrieval with | |
| BM25-style lexical reranking independently of production: | |
| ```bash | |
| uv run python -m src.evaluation.run_reranking \ | |
| data/benchmark_rag_annotated.jsonl \ | |
| --collection doculearn_product_eval__rc_800_100 \ | |
| --candidate-k 50 \ | |
| --k 10 \ | |
| --dense-weight 0.25 \ | |
| --max-chunks-per-page 1 \ | |
| --output-dir evaluation-results/retrieval/product-pdf-rerank \ | |
| --baseline-cases \ | |
| evaluation-results/retrieval/product-pdf-baseline/rc_800_100/cases.jsonl \ | |
| --error-analysis-output \ | |
| evaluation-results/retrieval/product-pdf-rerank/error_analysis.json | |
| ``` | |
| This retrieves a larger dense candidate pool, combines normalized dense and | |
| lexical scores, and limits duplicate page occupancy. It does not call an LLM | |
| or change `src.rag.retrieve`; promotion to production requires a separate | |
| decision after comparing quality, latency, and per-question regressions. | |
| ## Public benchmark baselines | |
| Prepare the Vietnamese XQuAD benchmark: | |
| ```bash | |
| uv run python -m src.evaluation.external_benchmarks xquad-vi \ | |
| --output-dir data/evaluation/external/xquad-vi | |
| ``` | |
| Prepare BEIR SciFact: | |
| ```bash | |
| uv run python -m src.evaluation.external_benchmarks beir-scifact \ | |
| --output-dir data/evaluation/external/beir-scifact | |
| ``` | |
| Each directory contains `corpus.jsonl`, `queries.jsonl`, and `manifest.json`. | |
| External query records use `gold_document_ids` because BEIR qrels can identify | |
| multiple relevant documents. This is deliberately separate from the PDF | |
| dataset schema, whose gold evidence is expressed as pages or exact chunks. | |
| Run all four chunking strategies without an LLM: | |
| ```bash | |
| uv run python -m src.evaluation.run_external_retrieval \ | |
| data/evaluation/external/xquad-vi \ | |
| --output-dir evaluation-results/retrieval/xquad-vi | |
| uv run python -m src.evaluation.run_external_retrieval \ | |
| data/evaluation/external/beir-scifact \ | |
| --output-dir evaluation-results/retrieval/beir-scifact | |
| ``` | |
| Validate the same expanded dense plus lexical reranker on one selected | |
| chunking strategy before considering production promotion: | |
| ```bash | |
| uv run python -m src.evaluation.run_external_reranking \ | |
| data/evaluation/external/xquad-vi \ | |
| --output-dir evaluation-results/retrieval/xquad-vi-rerank \ | |
| --strategy rc_800_100 \ | |
| --candidate-k 50 \ | |
| --k 10 \ | |
| --dense-weight 0.25 \ | |
| --max-chunks-per-page 1 | |
| uv run python -m src.evaluation.run_external_reranking \ | |
| data/evaluation/external/beir-scifact \ | |
| --output-dir evaluation-results/retrieval/beir-scifact-rerank \ | |
| --strategy rc_800_100 \ | |
| --candidate-k 50 \ | |
| --k 10 \ | |
| --dense-weight 0.25 \ | |
| --max-chunks-per-page 1 | |
| ``` | |
| Compare each `summary.json` and `cases.jsonl` with the matching dense baseline. | |
| Promotion decisions should consider aggregate quality, p50/p95 latency, and | |
| per-query regressions rather than relying on a single metric. | |
| Use `--limit 10` for a quick integration smoke test. Omit it for a reportable | |
| baseline. Ranking is measured at document level while the indexed documents | |
| are still split by each registered chunking strategy. Repeated chunks from the | |
| same document do not receive duplicate relevance credit. External baselines | |
| use exact in-memory cosine search with the configured production embedding | |
| model, avoiding large temporary local Qdrant collections. | |
| Public benchmarks test language and retrieval-engine behavior. They do not | |
| replace the document-grounded benchmark for the PDFs used by DocuLearn-RAG. | |
| ## Ragas answer evaluation | |
| `ragas_evaluator.py` converts generated DocuLearn-RAG answers into a Ragas | |
| dataset. `run_chunking.py` runs the full retrieval-plus-generation pipeline | |
| and writes mean answer-quality metrics. | |
| This path uses the configured local embedding model and Gemini as judge. It | |
| requires `GEMINI_API_KEY`, may incur API usage, and must not be confused with | |
| the LLM-free retrieval baseline. | |