nniback / tests /TEST_CASES.md
Kyrlos11's picture
Upload 842 files
22087d0 verified
|
Raw History Blame Contribute Delete
5.35 kB

Agent Test Cases

Comprehensive test case list for validating agent reliability.


1. Core Agent Lifecycle

1.1 Basic Operations

  • Start simple agent run ("Hello, what can you do?")
  • Start agent run on existing thread
  • Start agent run on new thread
  • Verify status transitions: queued β†’ running β†’ completed
  • Verify SSE stream events arrive in real-time

1.2 Stop/Cancel

  • Stop agent run via API while running
  • Stop agent run via UI during tool execution
  • Stop during LLM streaming
  • Verify final status is stopped

1.3 Auto-Continue

  • Trigger auto-continue via tool calls
  • Trigger auto-continue via length limit
  • Hit max auto-continue limit (25)
  • Verify agent_terminated stops auto-continue

2. Limits and Billing

2.1 Concurrent Runs

  • Start run at concurrent limit
  • Attempt second run β†’ AGENT_RUN_LIMIT_EXCEEDED
  • Complete first, verify second can start

2.2 Credits

  • Run with sufficient credits β†’ success
  • Run with zero credits β†’ INSUFFICIENT_CREDITS

2.3 Model Access

  • Use allowed model β†’ success
  • Use disallowed model β†’ MODEL_ACCESS_DENIED

3. Sandbox Tools

3.1 Shell (sb_shell_tool)

  • Execute echo "hello"
  • Execute ls -la
  • Execute failing command (non-zero exit)
  • Execute Python script
  • Long-running command timeout

3.2 Files (sb_files_tool)

  • Create file
  • Read file
  • Update file
  • Delete file
  • Create directory
  • File not found error

3.3 Vision (sb_vision_tool)

  • Analyze screenshot
  • Extract text (OCR)
  • Handle corrupted image

3.4 Expose (sb_expose_tool)

  • Expose port 8080
  • Verify URL accessible

3.5 Presentation (sb_presentation_tool)

  • Generate presentation
  • Export to PDF

3.6 Git Sync (sb_git_sync)

  • Clone public repo
  • Clone private repo
  • Push changes
  • Handle auth failure

3.7 Upload (sb_upload_file_tool)

  • Upload small file (<1MB)
  • Upload at limit (50MB)
  • Exceed limit β†’ error
  • Verify signed URL returned

4. Search Tools

4.1 Web Search

  • Single query
  • Batch queries (array)
  • With date filter
  • Handle timeout

4.2 Image/People/Company/Paper Search

  • Basic search
  • Handle no results

5. Browser Tool

5.1 Navigation

  • Navigate to URL
  • Handle invalid URL
  • Handle timeout

5.2 Actions

  • Click button
  • Fill form
  • Select dropdown
  • Scroll page
  • Element not found error

5.3 Extraction

  • Extract text
  • Extract structured data
  • Take screenshot

6. Agent Builder Tools

6.1 Configuration

  • Get current config
  • Update name/description
  • Enable/disable tools

6.2 MCP Integration

  • Search MCP servers
  • Test connection
  • Handle connection failure

6.3 Credential Profiles

  • Create profile
  • Configure for agent
  • Handle invalid profile

6.4 Triggers

  • Create scheduled trigger
  • Toggle on/off
  • Delete trigger

7. Knowledge Base

  • Semantic search in workspace
  • Sync global KB
  • Upload to global KB
  • Verify 50MB limit

8. Error Handling

8.1 LLM Errors

  • Handle timeout
  • Handle rate limit
  • Handle API error (500)
  • Verify retry (1 retry)

8.2 Sandbox Errors

  • Handle not found
  • Handle 502/503/504 (retry)
  • Handle starting state

8.3 Database Errors

  • Handle connection timeout
  • Verify retry with reconnect

8.4 Tool Errors

  • Handle execution failure
  • Verify agent recovers

9. Streaming

  • Stream connects successfully
  • Chunks arrive in order
  • Reconnection on disconnect
  • Catch-up missed messages
  • Verify all event types: content, tool_call, tool_result, status

10. Concurrency

10.1 Slot Manager

  • Reserve slot
  • Release on completion
  • Release on error
  • Rapid start/stop cycles

10.2 Webhooks

  • Process once
  • Reject duplicate
  • Handle race condition

11. Graceful Shutdown

  • Active runs stopped
  • Status updated to stopped
  • Redis stop signals set

12. Edge Cases

12.1 Input

  • Empty prompt
  • Very long prompt (>100k tokens)
  • Special characters
  • Non-English language

12.2 Output

  • Very long response (length limit)
  • Response with code blocks

12.3 State

  • No tools enabled
  • All tools enabled
  • Invalid model name
  • Deleted project/thread

13. Security

  • User can only access own runs
  • Admin endpoints require admin role
  • SQL injection prevented
  • Path traversal prevented

14. Performance

  • TTFT < 3s
  • Prep stages < 100ms
  • 10 concurrent runs
  • 50 concurrent runs
  • No memory leaks

Test Priority

  1. Critical: Core Lifecycle, Limits, Error Handling
  2. High: Sandbox Tools, Browser, Streaming
  3. Medium: Search, Agent Builder, KB
  4. Low: Edge Cases, Performance

Monitoring

Watch logs for these prefixes:

  • [SLOT] - Concurrent run management
  • [LLM] - LLM calls and timing
  • [REDIS] - Redis operations
  • [SANDBOX] - Sandbox lifecycle