benchmarks/evals/semantic_search_quality/README.md

Semantic Search Quality Evaluation

This evaluation measures the quality of semantic search queries generated by Forge using an LLM-as-judge approach.

Quick Start

Test Your Queries Manually

Before running the full eval, test your queries to get immediate feedback:

# Interactive mode
npx tsx benchmarks/evals/semantic_search_quality/test_queries.ts

# Non-interactive mode
npx tsx benchmarks/evals/semantic_search_quality/test_queries.ts \
  "retry mechanism with exponential backoff" \
  "I need to find the implementation code for retry logic"

The query tester provides:

• Detailed scores for embedding and reranking queries
• Specific strengths and weaknesses
• Actionable suggestions for improvement
• Differentiation analysis

Run the Full Evaluation

# Run the entire semantic search quality eval
npm run eval semantic_search_quality

# Run with specific parallelism
npm run eval semantic_search_quality -- --parallelism 10

# Run with specific model
npm run eval semantic_search_quality -- --model "anthropic/claude-sonnet-4.5"

Purpose

The semantic search feature uses two types of queries:

1. Embedding Query (query): Converted to vector embeddings for semantic similarity search
2. Reranking Query (use_case): Used by the reranker to filter results based on intent

This eval verifies that:

• Queries are well-formed and contain sufficient context
• The two queries are different and serve distinct purposes
• Results match the user's intent (implementation vs docs vs tests, etc.)
• Correct file types are returned based on intent
• Unwanted file types are avoided

How It Works

1. Test Execution

The eval runs various semantic search tasks covering different intents:

• Implementation: Find actual code, not documentation
• Flow Understanding: Understand how systems work
• Architecture: Understand structural design
• Tests: Find test examples and fixtures
• Documentation: Find docs and guides
• Debugging: Investigate issues
• Modification: Find code to modify
• Configuration: Find config files and schemas

2. LLM Judge Evaluation

After each task, an LLM judge (Gemini 3 Pro) evaluates:

Query Quality (40 points)

• Embedding Query (15pts): Domain terms, technical context, behavior description
• Reranking Query (15pts): Intent clarity, context about WHY, specificity
• Query Differentiation (10pts): Queries serve different purposes

Result Relevance (60 points)

• Intent Matching (25pts): Results match the stated intent
• File Type Accuracy (20pts): Correct file types returned
• Avoidance Compliance (15pts): Unwanted file types avoided

Pass Criteria

• Total score >= 70/100
• No critical issues (e.g., identical queries, severe intent mismatch)

3. Structured Output

The judge uses structured output (Zod schema) to provide:

• Detailed scores for each dimension
• Specific feedback on what's good/bad
• List of critical issues
• Overall pass/fail determination

Environment Variables

Required:

• GOOGLE_APPLICATION_CREDENTIALS: Path to Google Cloud service account JSON credentials file

Or authenticate using gcloud auth application-default login.

The LLM judge uses Vertex AI to run Gemini 3 Pro. It supports Google's Application Default Credentials through the google-auth-library. The most common authentication method is to set the path to a JSON credentials file in the GOOGLE_APPLICATION_CREDENTIALS environment variable.

Test Cases

The eval includes diverse test cases:

Implementation Queries

- task: "Find the retry mechanism with exponential backoff implementation"
  intent: "implementation"
  expected_file_types: "rust,typescript,javascript"
  should_avoid: "markdown,txt,documentation"

Flow Understanding

- task: "How does the authentication flow work from request to response?"
  intent: "flow_understanding"
  expected_file_types: "rust,typescript"
  should_avoid: "test,markdown"

Documentation

- task: "Find documentation on how to configure semantic search"
  intent: "documentation"
  expected_file_types: "markdown,txt"
  should_avoid: "rust,typescript,javascript"

Interpreting Results

High Scores (>80)

Queries are excellent - domain-specific, well-differentiated, and would return highly relevant results.

Medium Scores (60-80)

Queries are functional but have room for improvement. Check feedback for specific suggestions.

Low Scores (<60)

Queries have significant issues:

• Too generic or vague
• Embedding and reranking queries too similar
• Intent mismatch
• Would return wrong file types

Common Issues

1. Identical Queries: Embedding and reranking queries are the same or too similar

   • Fix: Make reranking query focus on intent, not just rephrasing

2. Generic Queries: Lack domain-specific context

   • Fix: Add technical terms, describe behavior, include framework names

3. Intent Mismatch: Queries would return docs when implementation is needed

   • Fix: Be explicit in reranking query about what type of code is needed

4. Poor Differentiation: Both queries try to do the same thing

   • Fix: Embedding = WHAT code does, Reranking = WHY you need it

Architecture

task.yml
├── Runs Forge with semantic search task
├── Captures debug output (context.json)
└── Validates with llm_judge.ts
    ├── Extracts sem_search calls
    ├── Calls Gemini 3 Pro with structured output
    └── Returns evaluation with scores

Extending the Eval

To add new test cases:

1. Add to task.yml under sources:

- task: "Your task description"
  intent: "implementation|flow_understanding|tests|documentation|debugging|modification|architecture|configuration"
  expected_file_types: "comma,separated,extensions"
  should_avoid: "comma,separated,extensions"

2. Choose appropriate intent:
   • implementation: Actual code implementation
   • flow_understanding: Understanding how things work
   • architecture: Structural/design understanding
   • tests: Test files and fixtures
   • documentation: Docs, README, guides
   • debugging: Finding bugs/issues
   • modification: Code to modify/extend
   • configuration: Config files, schemas

Future Improvements

• Add actual result verification (not just query quality)
• Measure embedding quality directly
• Test reranking effectiveness with real results
• Add precision/recall metrics
• Compare against human-labeled ground truth
• Test edge cases (very short queries, multi-lingual code, etc.)