RAG-Documind-AI / docs /explanations /evaluation.md
AIenthusSP's picture
feat: add Promptfoo evaluation framework
d9d0a60
|
Raw History Blame Contribute Delete
5.27 kB

πŸ” Deep Dive: Promptfoo RAG Evaluation

🧠 Why Promptfoo?

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                    PROMPTFOO BENEFITS                            β”‚
β”‚                                                                  β”‚
β”‚  βœ… Declarative YAML β†’ Easy test case management               β”‚
β”‚  βœ… LLM-as-Judge β†’ GPT evaluates answer quality                 β”‚
β”‚  βœ… CI/CD ready β†’ GitHub Actions integration                    β”‚
β”‚  βœ… Multi-provider β†’ Test across different models               β”‚
β”‚  βœ… Free & open β†’ No vendor lock-in                             β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

πŸ“ Evaluation Architecture

Golden Test Cases (YAML)
    ↓
Promptfoo Config
    ↓
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  For each test:                                                  β”‚
β”‚  1. Inject vars into prompt template                            β”‚
β”‚  2. Call LLM provider (Gemini)                                  β”‚
β”‚  3. Run assertions (contains, llm-rubric, relevance)           β”‚
β”‚  4. Record pass/fail, latency, cost                            β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
    ↓
JSON Results β†’ Markdown Report

🎯 Key Files

File Purpose
eval/promptfooconfig.yaml Main config with prompts & tests
eval/golden_tests.yaml Expected Q&A pairs
eval/runner.py Python wrapper for automation
eval/results/ Output directory for results

πŸ“‹ Test Categories

1. Factual Accuracy

- description: "Factual retrieval"
  vars:
    question: "What is RAG?"
    context: "RAG combines retrieval with generation..."
  assert:
    - type: contains
      value: "retrieval"
    - type: llm-rubric
      value: "Answer correctly explains RAG"

2. Faithfulness (No Hallucination)

- description: "Should not hallucinate"
  assert:
    - type: not-contains
      value: "PostgreSQL"
    - type: llm-rubric
      value: "Answer only uses facts from context"

3. Relevance

- description: "Answer should be relevant"
  assert:
    - type: relevance
      threshold: 0.7

4. Edge Cases

- description: "Handle empty context"
  vars:
    question: "What is X?"
    context: ""
  assert:
    - type: llm-rubric
      value: "Admits information is not available"

πŸ”„ Assertion Types

Type Purpose
contains Response includes text
not-contains Response excludes text
contains-any Response includes any of list
llm-rubric GPT judges quality
relevance Semantic similarity score
cost Max cost per query
latency Max response time

πŸš€ Running Evaluations

# Install Promptfoo (one-time)
npm install -g promptfoo

# Run evaluation
cd eval
npx promptfoo eval

# View results in browser
npx promptfoo view

# Run with Python wrapper
python eval/runner.py

πŸ“Š Sample Output

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ RAG Evaluation Results                                        β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Total Tests: 10                                               β”‚
β”‚ Passed: 9                                                     β”‚
β”‚ Failed: 1                                                     β”‚
β”‚ Pass Rate: 90%                                                β”‚
β”‚ Avg Latency: 1,234ms                                          β”‚
β”‚ Total Cost: $0.0089                                           β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

πŸ”— CI/CD Integration

# .github/workflows/eval.yml
- name: Run RAG Evaluation
  run: |
    npx promptfoo eval -c eval/promptfooconfig.yaml
  env:
    GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}

πŸ“‹ Summary

Component Purpose
promptfooconfig.yaml Test definitions
golden_tests.yaml Expected behaviors
runner.py Python automation
llm-rubric AI-powered evaluation
CI/CD Automated quality checks