nniback / tests /TEST_CASES.md
Kyrlos11's picture
Upload 842 files
22087d0 verified
|
Raw History Blame Contribute Delete
5.35 kB
# Agent Test Cases
Comprehensive test case list for validating agent reliability.
---
## 1. Core Agent Lifecycle
### 1.1 Basic Operations
- [ ] Start simple agent run ("Hello, what can you do?")
- [ ] Start agent run on existing thread
- [ ] Start agent run on new thread
- [ ] Verify status transitions: `queued` β†’ `running` β†’ `completed`
- [ ] Verify SSE stream events arrive in real-time
### 1.2 Stop/Cancel
- [ ] Stop agent run via API while running
- [ ] Stop agent run via UI during tool execution
- [ ] Stop during LLM streaming
- [ ] Verify final status is `stopped`
### 1.3 Auto-Continue
- [ ] Trigger auto-continue via tool calls
- [ ] Trigger auto-continue via length limit
- [ ] Hit max auto-continue limit (25)
- [ ] Verify `agent_terminated` stops auto-continue
---
## 2. Limits and Billing
### 2.1 Concurrent Runs
- [ ] Start run at concurrent limit
- [ ] Attempt second run β†’ `AGENT_RUN_LIMIT_EXCEEDED`
- [ ] Complete first, verify second can start
### 2.2 Credits
- [ ] Run with sufficient credits β†’ success
- [ ] Run with zero credits β†’ `INSUFFICIENT_CREDITS`
### 2.3 Model Access
- [ ] Use allowed model β†’ success
- [ ] Use disallowed model β†’ `MODEL_ACCESS_DENIED`
---
## 3. Sandbox Tools
### 3.1 Shell (`sb_shell_tool`)
- [ ] Execute `echo "hello"`
- [ ] Execute `ls -la`
- [ ] Execute failing command (non-zero exit)
- [ ] Execute Python script
- [ ] Long-running command timeout
### 3.2 Files (`sb_files_tool`)
- [ ] Create file
- [ ] Read file
- [ ] Update file
- [ ] Delete file
- [ ] Create directory
- [ ] File not found error
### 3.3 Vision (`sb_vision_tool`)
- [ ] Analyze screenshot
- [ ] Extract text (OCR)
- [ ] Handle corrupted image
### 3.4 Expose (`sb_expose_tool`)
- [ ] Expose port 8080
- [ ] Verify URL accessible
### 3.5 Presentation (`sb_presentation_tool`)
- [ ] Generate presentation
- [ ] Export to PDF
### 3.6 Git Sync (`sb_git_sync`)
- [ ] Clone public repo
- [ ] Clone private repo
- [ ] Push changes
- [ ] Handle auth failure
### 3.7 Upload (`sb_upload_file_tool`)
- [ ] Upload small file (<1MB)
- [ ] Upload at limit (50MB)
- [ ] Exceed limit β†’ error
- [ ] Verify signed URL returned
---
## 4. Search Tools
### 4.1 Web Search
- [ ] Single query
- [ ] Batch queries (array)
- [ ] With date filter
- [ ] Handle timeout
### 4.2 Image/People/Company/Paper Search
- [ ] Basic search
- [ ] Handle no results
---
## 5. Browser Tool
### 5.1 Navigation
- [ ] Navigate to URL
- [ ] Handle invalid URL
- [ ] Handle timeout
### 5.2 Actions
- [ ] Click button
- [ ] Fill form
- [ ] Select dropdown
- [ ] Scroll page
- [ ] Element not found error
### 5.3 Extraction
- [ ] Extract text
- [ ] Extract structured data
- [ ] Take screenshot
---
## 6. Agent Builder Tools
### 6.1 Configuration
- [ ] Get current config
- [ ] Update name/description
- [ ] Enable/disable tools
### 6.2 MCP Integration
- [ ] Search MCP servers
- [ ] Test connection
- [ ] Handle connection failure
### 6.3 Credential Profiles
- [ ] Create profile
- [ ] Configure for agent
- [ ] Handle invalid profile
### 6.4 Triggers
- [ ] Create scheduled trigger
- [ ] Toggle on/off
- [ ] Delete trigger
---
## 7. Knowledge Base
- [ ] Semantic search in workspace
- [ ] Sync global KB
- [ ] Upload to global KB
- [ ] Verify 50MB limit
---
## 8. Error Handling
### 8.1 LLM Errors
- [ ] Handle timeout
- [ ] Handle rate limit
- [ ] Handle API error (500)
- [ ] Verify retry (1 retry)
### 8.2 Sandbox Errors
- [ ] Handle not found
- [ ] Handle 502/503/504 (retry)
- [ ] Handle starting state
### 8.3 Database Errors
- [ ] Handle connection timeout
- [ ] Verify retry with reconnect
### 8.4 Tool Errors
- [ ] Handle execution failure
- [ ] Verify agent recovers
---
## 9. Streaming
- [ ] Stream connects successfully
- [ ] Chunks arrive in order
- [ ] Reconnection on disconnect
- [ ] Catch-up missed messages
- [ ] Verify all event types: `content`, `tool_call`, `tool_result`, `status`
---
## 10. Concurrency
### 10.1 Slot Manager
- [ ] Reserve slot
- [ ] Release on completion
- [ ] Release on error
- [ ] Rapid start/stop cycles
### 10.2 Webhooks
- [ ] Process once
- [ ] Reject duplicate
- [ ] Handle race condition
---
## 11. Graceful Shutdown
- [ ] Active runs stopped
- [ ] Status updated to `stopped`
- [ ] Redis stop signals set
---
## 12. Edge Cases
### 12.1 Input
- [ ] Empty prompt
- [ ] Very long prompt (>100k tokens)
- [ ] Special characters
- [ ] Non-English language
### 12.2 Output
- [ ] Very long response (length limit)
- [ ] Response with code blocks
### 12.3 State
- [ ] No tools enabled
- [ ] All tools enabled
- [ ] Invalid model name
- [ ] Deleted project/thread
---
## 13. Security
- [ ] User can only access own runs
- [ ] Admin endpoints require admin role
- [ ] SQL injection prevented
- [ ] Path traversal prevented
---
## 14. Performance
- [ ] TTFT < 3s
- [ ] Prep stages < 100ms
- [ ] 10 concurrent runs
- [ ] 50 concurrent runs
- [ ] No memory leaks
---
## Test Priority
1. **Critical**: Core Lifecycle, Limits, Error Handling
2. **High**: Sandbox Tools, Browser, Streaming
3. **Medium**: Search, Agent Builder, KB
4. **Low**: Edge Cases, Performance
## Monitoring
Watch logs for these prefixes:
- `[SLOT]` - Concurrent run management
- `[LLM]` - LLM calls and timing
- `[REDIS]` - Redis operations
- `[SANDBOX]` - Sandbox lifecycle