File size: 1,916 Bytes
1244914 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 | # Semantic Search Tool Selection Evaluation
This evaluation validates that the Forge agent correctly identifies and uses the `sem_search` (Codebase Search) tool when presented with conceptual, functionality-based code location queries.
## What We're Testing
The agent's ability to recognize when a user query requires semantic understanding rather than exact pattern matching. Specifically, we test whether the agent:
1. **Correctly invokes `sem_search`** for conceptual queries about code functionality
2. **Understands the distinction** between semantic search (conceptual) vs regex search (exact patterns)
3. **Recognizes appropriate use cases** such as:
- Finding code by its purpose/behavior (e.g., "retry logic with exponential backoff")
- Locating implementation patterns (e.g., "authentication token validation")
- Discovering architectural components (e.g., "message transformation between AI providers")
- Identifying system behaviors (e.g., "rate limiting for API requests")
## Test Scenarios
The evaluation uses real-world queries that describe:
- **Message transformation** between different AI provider formats
- **Retry mechanisms** with exponential backoff patterns
- **Authentication flows** including token validation and refresh
- **Error handling** for network failures
- **Caching strategies** for API responses
- **Streaming response** handling from LLM APIs
- **File validation** including upload and size checks
- **Rate limiting** implementations for API requests
- **Tool registration** and availability management
- **Context management** including conversation history truncation
## Expected Behavior
For all test queries, the agent should invoke the `sem_search` tool (displayed as "Codebase Search" in logs) rather than falling back to regex-based `search` tool, demonstrating understanding that these are conceptual queries requiring semantic code comprehension.
|