benchmarks/evals/semantic_search_quality/README.md

# Semantic Search Quality Evaluation

This evaluation measures the quality of semantic search queries generated by Forge using an LLM-as-judge approach.

## Quick Start

### Test Your Queries Manually

Before running the full eval, test your queries to get immediate feedback:

```bash
# Interactive mode
npx tsx benchmarks/evals/semantic_search_quality/test_queries.ts

# Non-interactive mode
npx tsx benchmarks/evals/semantic_search_quality/test_queries.ts \
  "retry mechanism with exponential backoff" \
  "I need to find the implementation code for retry logic"
```

The query tester provides:
- Detailed scores for embedding and reranking queries
- Specific strengths and weaknesses
- Actionable suggestions for improvement
- Differentiation analysis

### Run the Full Evaluation

```bash
# Run the entire semantic search quality eval
npm run eval semantic_search_quality

# Run with specific parallelism
npm run eval semantic_search_quality -- --parallelism 10

# Run with specific model
npm run eval semantic_search_quality -- --model "anthropic/claude-sonnet-4.5"
```

## Purpose

The semantic search feature uses two types of queries:
1. **Embedding Query (`query`)**: Converted to vector embeddings for semantic similarity search
2. **Reranking Query (`use_case`)**: Used by the reranker to filter results based on intent

This eval verifies that:
- Queries are well-formed and contain sufficient context
- The two queries are different and serve distinct purposes
- Results match the user's intent (implementation vs docs vs tests, etc.)
- Correct file types are returned based on intent
- Unwanted file types are avoided

## How It Works

### 1. Test Execution
The eval runs various semantic search tasks covering different intents:
- **Implementation**: Find actual code, not documentation
- **Flow Understanding**: Understand how systems work
- **Architecture**: Understand structural design
- **Tests**: Find test examples and fixtures
- **Documentation**: Find docs and guides
- **Debugging**: Investigate issues
- **Modification**: Find code to modify
- **Configuration**: Find config files and schemas

### 2. LLM Judge Evaluation
After each task, an LLM judge (Gemini 3 Pro) evaluates:

#### Query Quality (40 points)
- **Embedding Query (15pts)**: Domain terms, technical context, behavior description
- **Reranking Query (15pts)**: Intent clarity, context about WHY, specificity
- **Query Differentiation (10pts)**: Queries serve different purposes

#### Result Relevance (60 points)
- **Intent Matching (25pts)**: Results match the stated intent
- **File Type Accuracy (20pts)**: Correct file types returned
- **Avoidance Compliance (15pts)**: Unwanted file types avoided

#### Pass Criteria
- Total score >= 70/100
- No critical issues (e.g., identical queries, severe intent mismatch)

### 3. Structured Output
The judge uses structured output (Zod schema) to provide:
- Detailed scores for each dimension
- Specific feedback on what's good/bad
- List of critical issues
- Overall pass/fail determination

## Environment Variables

Required:
- `GOOGLE_APPLICATION_CREDENTIALS`: Path to Google Cloud service account JSON credentials file

Or authenticate using `gcloud auth application-default login`.

The LLM judge uses Vertex AI to run Gemini 3 Pro. It supports Google's Application Default Credentials through the `google-auth-library`. The most common authentication method is to set the path to a JSON credentials file in the `GOOGLE_APPLICATION_CREDENTIALS` environment variable.

## Test Cases

The eval includes diverse test cases:

### Implementation Queries
```yaml
- task: "Find the retry mechanism with exponential backoff implementation"
  intent: "implementation"
  expected_file_types: "rust,typescript,javascript"
  should_avoid: "markdown,txt,documentation"
```

### Flow Understanding
```yaml
- task: "How does the authentication flow work from request to response?"
  intent: "flow_understanding"
  expected_file_types: "rust,typescript"
  should_avoid: "test,markdown"
```

### Documentation
```yaml
- task: "Find documentation on how to configure semantic search"
  intent: "documentation"
  expected_file_types: "markdown,txt"
  should_avoid: "rust,typescript,javascript"
```

## Interpreting Results

### High Scores (>80)
Queries are excellent - domain-specific, well-differentiated, and would return highly relevant results.

### Medium Scores (60-80)
Queries are functional but have room for improvement. Check feedback for specific suggestions.

### Low Scores (<60)
Queries have significant issues:
- Too generic or vague
- Embedding and reranking queries too similar
- Intent mismatch
- Would return wrong file types

## Common Issues

1. **Identical Queries**: Embedding and reranking queries are the same or too similar
   - **Fix**: Make reranking query focus on intent, not just rephrasing

2. **Generic Queries**: Lack domain-specific context
   - **Fix**: Add technical terms, describe behavior, include framework names

3. **Intent Mismatch**: Queries would return docs when implementation is needed
   - **Fix**: Be explicit in reranking query about what type of code is needed

4. **Poor Differentiation**: Both queries try to do the same thing
   - **Fix**: Embedding = WHAT code does, Reranking = WHY you need it

## Architecture

```
task.yml
├── Runs Forge with semantic search task
├── Captures debug output (context.json)
└── Validates with llm_judge.ts
    ├── Extracts sem_search calls
    ├── Calls Gemini 3 Pro with structured output
    └── Returns evaluation with scores
```

## Extending the Eval

To add new test cases:

1. Add to `task.yml` under `sources`:
```yaml
- task: "Your task description"
  intent: "implementation|flow_understanding|tests|documentation|debugging|modification|architecture|configuration"
  expected_file_types: "comma,separated,extensions"
  should_avoid: "comma,separated,extensions"
```

2. Choose appropriate intent:
   - `implementation`: Actual code implementation
   - `flow_understanding`: Understanding how things work
   - `architecture`: Structural/design understanding
   - `tests`: Test files and fixtures
   - `documentation`: Docs, README, guides
   - `debugging`: Finding bugs/issues
   - `modification`: Code to modify/extend
   - `configuration`: Config files, schemas

## Future Improvements

- [ ] Add actual result verification (not just query quality)
- [ ] Measure embedding quality directly
- [ ] Test reranking effectiveness with real results
- [ ] Add precision/recall metrics
- [ ] Compare against human-labeled ground truth
- [ ] Test edge cases (very short queries, multi-lingual code, etc.)