|
Download tests/TEST_CASES.md from Kyrlos11/nniback: direct link, hf CLI and curl.
- Browser
- Download file 5.35 kB
-
https://huggingface.co/spaces/Kyrlos11/nniback/resolve/main/tests/TEST_CASES.md
- Command line
-
hf download hf://spaces/Kyrlos11/nniback/tests/TEST_CASES.md
-
curl -L -o TEST_CASES.md https://huggingface.co/spaces/Kyrlos11/nniback/resolve/main/tests/TEST_CASES.md
5.35 kB
Agent Test Cases
Comprehensive test case list for validating agent reliability.
1. Core Agent Lifecycle
1.1 Basic Operations
- Start simple agent run ("Hello, what can you do?")
- Start agent run on existing thread
- Start agent run on new thread
- Verify status transitions:
queuedβrunningβcompleted - Verify SSE stream events arrive in real-time
1.2 Stop/Cancel
- Stop agent run via API while running
- Stop agent run via UI during tool execution
- Stop during LLM streaming
- Verify final status is
stopped
1.3 Auto-Continue
- Trigger auto-continue via tool calls
- Trigger auto-continue via length limit
- Hit max auto-continue limit (25)
- Verify
agent_terminatedstops auto-continue
2. Limits and Billing
2.1 Concurrent Runs
- Start run at concurrent limit
- Attempt second run β
AGENT_RUN_LIMIT_EXCEEDED - Complete first, verify second can start
2.2 Credits
- Run with sufficient credits β success
- Run with zero credits β
INSUFFICIENT_CREDITS
2.3 Model Access
- Use allowed model β success
- Use disallowed model β
MODEL_ACCESS_DENIED
3. Sandbox Tools
3.1 Shell (sb_shell_tool)
- Execute
echo "hello" - Execute
ls -la - Execute failing command (non-zero exit)
- Execute Python script
- Long-running command timeout
3.2 Files (sb_files_tool)
- Create file
- Read file
- Update file
- Delete file
- Create directory
- File not found error
3.3 Vision (sb_vision_tool)
- Analyze screenshot
- Extract text (OCR)
- Handle corrupted image
3.4 Expose (sb_expose_tool)
- Expose port 8080
- Verify URL accessible
3.5 Presentation (sb_presentation_tool)
- Generate presentation
- Export to PDF
3.6 Git Sync (sb_git_sync)
- Clone public repo
- Clone private repo
- Push changes
- Handle auth failure
3.7 Upload (sb_upload_file_tool)
- Upload small file (<1MB)
- Upload at limit (50MB)
- Exceed limit β error
- Verify signed URL returned
4. Search Tools
4.1 Web Search
- Single query
- Batch queries (array)
- With date filter
- Handle timeout
4.2 Image/People/Company/Paper Search
- Basic search
- Handle no results
5. Browser Tool
5.1 Navigation
- Navigate to URL
- Handle invalid URL
- Handle timeout
5.2 Actions
- Click button
- Fill form
- Select dropdown
- Scroll page
- Element not found error
5.3 Extraction
- Extract text
- Extract structured data
- Take screenshot
6. Agent Builder Tools
6.1 Configuration
- Get current config
- Update name/description
- Enable/disable tools
6.2 MCP Integration
- Search MCP servers
- Test connection
- Handle connection failure
6.3 Credential Profiles
- Create profile
- Configure for agent
- Handle invalid profile
6.4 Triggers
- Create scheduled trigger
- Toggle on/off
- Delete trigger
7. Knowledge Base
- Semantic search in workspace
- Sync global KB
- Upload to global KB
- Verify 50MB limit
8. Error Handling
8.1 LLM Errors
- Handle timeout
- Handle rate limit
- Handle API error (500)
- Verify retry (1 retry)
8.2 Sandbox Errors
- Handle not found
- Handle 502/503/504 (retry)
- Handle starting state
8.3 Database Errors
- Handle connection timeout
- Verify retry with reconnect
8.4 Tool Errors
- Handle execution failure
- Verify agent recovers
9. Streaming
- Stream connects successfully
- Chunks arrive in order
- Reconnection on disconnect
- Catch-up missed messages
- Verify all event types:
content,tool_call,tool_result,status
10. Concurrency
10.1 Slot Manager
- Reserve slot
- Release on completion
- Release on error
- Rapid start/stop cycles
10.2 Webhooks
- Process once
- Reject duplicate
- Handle race condition
11. Graceful Shutdown
- Active runs stopped
- Status updated to
stopped - Redis stop signals set
12. Edge Cases
12.1 Input
- Empty prompt
- Very long prompt (>100k tokens)
- Special characters
- Non-English language
12.2 Output
- Very long response (length limit)
- Response with code blocks
12.3 State
- No tools enabled
- All tools enabled
- Invalid model name
- Deleted project/thread
13. Security
- User can only access own runs
- Admin endpoints require admin role
- SQL injection prevented
- Path traversal prevented
14. Performance
- TTFT < 3s
- Prep stages < 100ms
- 10 concurrent runs
- 50 concurrent runs
- No memory leaks
Test Priority
- Critical: Core Lifecycle, Limits, Error Handling
- High: Sandbox Tools, Browser, Streaming
- Medium: Search, Agent Builder, KB
- Low: Edge Cases, Performance
Monitoring
Watch logs for these prefixes:
[SLOT]- Concurrent run management[LLM]- LLM calls and timing[REDIS]- Redis operations[SANDBOX]- Sandbox lifecycle