| # Semantic Search Tool Selection Evaluation |
|
|
| This evaluation validates that the Forge agent correctly identifies and uses the `sem_search` (Codebase Search) tool when presented with conceptual, functionality-based code location queries. |
|
|
| ## What We're Testing |
|
|
| The agent's ability to recognize when a user query requires semantic understanding rather than exact pattern matching. Specifically, we test whether the agent: |
|
|
| 1. **Correctly invokes `sem_search`** for conceptual queries about code functionality |
| 2. **Understands the distinction** between semantic search (conceptual) vs regex search (exact patterns) |
| 3. **Recognizes appropriate use cases** such as: |
| - Finding code by its purpose/behavior (e.g., "retry logic with exponential backoff") |
| - Locating implementation patterns (e.g., "authentication token validation") |
| - Discovering architectural components (e.g., "message transformation between AI providers") |
| - Identifying system behaviors (e.g., "rate limiting for API requests") |
| |
| ## Test Scenarios |
| |
| The evaluation uses real-world queries that describe: |
| - **Message transformation** between different AI provider formats |
| - **Retry mechanisms** with exponential backoff patterns |
| - **Authentication flows** including token validation and refresh |
| - **Error handling** for network failures |
| - **Caching strategies** for API responses |
| - **Streaming response** handling from LLM APIs |
| - **File validation** including upload and size checks |
| - **Rate limiting** implementations for API requests |
| - **Tool registration** and availability management |
| - **Context management** including conversation history truncation |
| |
| ## Expected Behavior |
| |
| For all test queries, the agent should invoke the `sem_search` tool (displayed as "Codebase Search" in logs) rather than falling back to regex-based `search` tool, demonstrating understanding that these are conceptual queries requiring semantic code comprehension. |
| |
