Spaces:
Build error
Build error
|
Download docs/explanations/evaluation.md from AIenthusSP/RAG-Documind-AI: direct link, hf CLI and curl.
- Browser
- Download file 5.27 kB
-
https://huggingface.co/spaces/AIenthusSP/RAG-Documind-AI/resolve/main/docs/explanations/evaluation.md
- Command line
-
hf download hf://spaces/AIenthusSP/RAG-Documind-AI/docs/explanations/evaluation.md
-
curl -L -o evaluation.md https://huggingface.co/spaces/AIenthusSP/RAG-Documind-AI/resolve/main/docs/explanations/evaluation.md
5.27 kB
π Deep Dive: Promptfoo RAG Evaluation
π§ Why Promptfoo?
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β PROMPTFOO BENEFITS β
β β
β β
Declarative YAML β Easy test case management β
β β
LLM-as-Judge β GPT evaluates answer quality β
β β
CI/CD ready β GitHub Actions integration β
β β
Multi-provider β Test across different models β
β β
Free & open β No vendor lock-in β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
π Evaluation Architecture
Golden Test Cases (YAML)
β
Promptfoo Config
β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β For each test: β
β 1. Inject vars into prompt template β
β 2. Call LLM provider (Gemini) β
β 3. Run assertions (contains, llm-rubric, relevance) β
β 4. Record pass/fail, latency, cost β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
JSON Results β Markdown Report
π― Key Files
| File | Purpose |
|---|---|
eval/promptfooconfig.yaml |
Main config with prompts & tests |
eval/golden_tests.yaml |
Expected Q&A pairs |
eval/runner.py |
Python wrapper for automation |
eval/results/ |
Output directory for results |
π Test Categories
1. Factual Accuracy
- description: "Factual retrieval"
vars:
question: "What is RAG?"
context: "RAG combines retrieval with generation..."
assert:
- type: contains
value: "retrieval"
- type: llm-rubric
value: "Answer correctly explains RAG"
2. Faithfulness (No Hallucination)
- description: "Should not hallucinate"
assert:
- type: not-contains
value: "PostgreSQL"
- type: llm-rubric
value: "Answer only uses facts from context"
3. Relevance
- description: "Answer should be relevant"
assert:
- type: relevance
threshold: 0.7
4. Edge Cases
- description: "Handle empty context"
vars:
question: "What is X?"
context: ""
assert:
- type: llm-rubric
value: "Admits information is not available"
π Assertion Types
| Type | Purpose |
|---|---|
contains |
Response includes text |
not-contains |
Response excludes text |
contains-any |
Response includes any of list |
llm-rubric |
GPT judges quality |
relevance |
Semantic similarity score |
cost |
Max cost per query |
latency |
Max response time |
π Running Evaluations
# Install Promptfoo (one-time)
npm install -g promptfoo
# Run evaluation
cd eval
npx promptfoo eval
# View results in browser
npx promptfoo view
# Run with Python wrapper
python eval/runner.py
π Sample Output
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β RAG Evaluation Results β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β Total Tests: 10 β
β Passed: 9 β
β Failed: 1 β
β Pass Rate: 90% β
β Avg Latency: 1,234ms β
β Total Cost: $0.0089 β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
π CI/CD Integration
# .github/workflows/eval.yml
- name: Run RAG Evaluation
run: |
npx promptfoo eval -c eval/promptfooconfig.yaml
env:
GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}
π Summary
| Component | Purpose |
|---|---|
promptfooconfig.yaml |
Test definitions |
golden_tests.yaml |
Expected behaviors |
runner.py |
Python automation |
llm-rubric |
AI-powered evaluation |
| CI/CD | Automated quality checks |