Spaces:
Runtime error
Runtime error
Add documentation, diagrams, and research paper
Browse filesAdd comprehensive project documentation (PROJECT_DOCUMENTATION.md), a Mermaid diagrams guide (Project_Diagrams.md), and an academic research paper (Research_Paper.md) to describe architecture, data flow, API contracts, and design decisions. Also add a temporary test file (tmp_test_delete.py) and apply updates to backend/database.py. These additions provide reference material for developers, architects, and reviewers — including runnable diagrams, DB schema, API endpoints, and usage/run instructions.
- PROJECT_DOCUMENTATION.md +681 -0
- Project_Diagrams.md +549 -0
- Research_Paper.md +678 -0
- backend/database.py +6 -1
- tmp_test_delete.py +48 -0
PROJECT_DOCUMENTATION.md
ADDED
|
@@ -0,0 +1,681 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 📘 AI API Testing Assistant — Project Documentation
|
| 2 |
+
|
| 3 |
+
> A comprehensive reference document covering the architecture, components, data flow, API contracts, and design decisions of this project. Intended for comparison with similar tools and for explaining the system to others.
|
| 4 |
+
|
| 5 |
+
---
|
| 6 |
+
|
| 7 |
+
## Table of Contents
|
| 8 |
+
|
| 9 |
+
1. [Project Overview](#1-project-overview)
|
| 10 |
+
2. [Technology Stack](#2-technology-stack)
|
| 11 |
+
3. [Project Structure](#3-project-structure)
|
| 12 |
+
4. [System Architecture](#4-system-architecture)
|
| 13 |
+
5. [Backend Deep-Dive](#5-backend-deep-dive)
|
| 14 |
+
- 5.1 [Entry Point — main.py](#51-entry-point--mainpy)
|
| 15 |
+
- 5.2 [Database Layer — database.py](#52-database-layer--databasepy)
|
| 16 |
+
- 5.3 [Pydantic Schemas — schemas.py](#53-pydantic-schemas--schemaspy)
|
| 17 |
+
- 5.4 [Service Layer — services/](#54-service-layer--services)
|
| 18 |
+
6. [Frontend Deep-Dive](#6-frontend-deep-dive)
|
| 19 |
+
7. [LLM Pipeline](#7-llm-pipeline)
|
| 20 |
+
8. [Data Flow — End-to-End](#8-data-flow--end-to-end)
|
| 21 |
+
9. [API Reference (All Endpoints)](#9-api-reference-all-endpoints)
|
| 22 |
+
10. [Database Schema](#10-database-schema)
|
| 23 |
+
11. [Key Design Decisions](#11-key-design-decisions)
|
| 24 |
+
12. [Environment & Configuration](#12-environment--configuration)
|
| 25 |
+
13. [Running the Project](#13-running-the-project)
|
| 26 |
+
14. [Comparison Cheatsheet](#14-comparison-cheatsheet)
|
| 27 |
+
|
| 28 |
+
---
|
| 29 |
+
|
| 30 |
+
## 1. Project Overview
|
| 31 |
+
|
| 32 |
+
**AI API Testing Assistant** is a full-stack web application that automates the testing of REST APIs. Instead of manually writing test scripts, the user:
|
| 33 |
+
|
| 34 |
+
1. Uploads an **OpenAPI/Swagger spec** (JSON or YAML) or a **Postman Collection**.
|
| 35 |
+
2. Describes a test scenario in **plain English** (e.g., *"Register a new user, log in, then create a new product"*).
|
| 36 |
+
3. The system uses **Mistral AI** (via LangChain) to generate a **step-by-step execution plan**.
|
| 37 |
+
4. Each step is individually executed as a real HTTP request, with the AI automatically resolving path parameters, injecting auth tokens from previous steps, and handling file uploads.
|
| 38 |
+
5. The AI then **analyzes each response**, extracts dynamic values (tokens, IDs), and chains them into subsequent steps.
|
| 39 |
+
|
| 40 |
+
### Core Value Propositions
|
| 41 |
+
|
| 42 |
+
| Feature | Description |
|
| 43 |
+
|---|---|
|
| 44 |
+
| **Natural Language Testing** | No code required — describe what you want to test |
|
| 45 |
+
| **Auto Context Chaining** | Tokens/IDs extracted from step N are auto-injected into step N+1 |
|
| 46 |
+
| **Spec-Aware** | Reads OpenAPI schemas to build correctly typed requests |
|
| 47 |
+
| **Multi-Format Input** | Supports OpenAPI 2/3 (JSON/YAML) and Postman Collections |
|
| 48 |
+
| **Persistent History** | All sessions, plans, and results are stored in a database |
|
| 49 |
+
| **Media Hub** | Upload files (audio, video, documents) for use in API tests |
|
| 50 |
+
|
| 51 |
+
---
|
| 52 |
+
|
| 53 |
+
## 2. Technology Stack
|
| 54 |
+
|
| 55 |
+
### Backend
|
| 56 |
+
|
| 57 |
+
| Layer | Technology | Version | Purpose |
|
| 58 |
+
|---|---|---|---|
|
| 59 |
+
| Web Framework | **FastAPI** | ≥0.109 | REST API server |
|
| 60 |
+
| ASGI Server | **Uvicorn** | ≥0.27 | Runs FastAPI in development |
|
| 61 |
+
| Production Server | **Gunicorn** | 21.2 | Runs Uvicorn workers in production |
|
| 62 |
+
| LLM Orchestration | **LangChain** | 0.1.0 | Prompt chaining with Mistral |
|
| 63 |
+
| LLM Provider | **Mistral AI** | `mistral-large-latest` | Plan generation, step execution, analysis |
|
| 64 |
+
| HTTP Client | **httpx** | ≥0.27 | Sends actual API requests during execution |
|
| 65 |
+
| ORM | **SQLAlchemy** | 2.0.23 | Database queries and model definitions |
|
| 66 |
+
| Database | **PostgreSQL** (via Supabase) | — | Persistent storage |
|
| 67 |
+
| DB Driver | **psycopg2-binary** | 2.9.9 | PostgreSQL adapter |
|
| 68 |
+
| Cloud Storage | **Supabase Storage** | 2.28.2 | Stores uploaded spec files and assets |
|
| 69 |
+
| Spec Parsing | **prance** + **PyYAML** | — | Parse OpenAPI YAML/JSON |
|
| 70 |
+
| Validation | **Pydantic** | ≥2.6 | Request/response data models |
|
| 71 |
+
| Auth/Config | **python-dotenv** | 1.0.0 | Load env variables from `.env` |
|
| 72 |
+
|
| 73 |
+
### Frontend
|
| 74 |
+
|
| 75 |
+
| Layer | Technology | Version | Purpose |
|
| 76 |
+
|---|---|---|---|
|
| 77 |
+
| UI Framework | **React** | 19.x | Component-based UI |
|
| 78 |
+
| Build Tool | **Vite** | 8.x | Fast dev server + bundler |
|
| 79 |
+
| Styling | **Tailwind CSS** | 4.x | Utility-first CSS |
|
| 80 |
+
| Icons | **lucide-react** | 0.577 | Icon library |
|
| 81 |
+
| HTTP Client | **Axios** | 1.x | API communication with backend |
|
| 82 |
+
| Routing | **react-router-dom** | 7.x | Client-side navigation |
|
| 83 |
+
|
| 84 |
+
---
|
| 85 |
+
|
| 86 |
+
## 3. Project Structure
|
| 87 |
+
|
| 88 |
+
```
|
| 89 |
+
Testing/ ← Project root
|
| 90 |
+
├── .env ← Environment variables (secrets)
|
| 91 |
+
├── .env.example ← Template for env setup
|
| 92 |
+
├── requirements.txt ← Python dependencies
|
| 93 |
+
├── package.json ← Root-level npm scripts (dev runner)
|
| 94 |
+
├── Procfile ← Heroku/Railway deployment config
|
| 95 |
+
├── runtime.txt ← Python version pin
|
| 96 |
+
├── start.ps1 ← PowerShell startup script
|
| 97 |
+
├── openapi.yaml ← Sample OpenAPI spec for testing
|
| 98 |
+
│
|
| 99 |
+
├── backend/ ← FastAPI application
|
| 100 |
+
│ ├── __init__.py
|
| 101 |
+
│ ├── main.py ← All FastAPI routes (API gateway)
|
| 102 |
+
│ ├── database.py ← SQLAlchemy models + DB connection
|
| 103 |
+
│ ├── schemas.py ← Pydantic request/response models
|
| 104 |
+
│ └── services/ ← Business logic layer
|
| 105 |
+
│ ├── swagger_parser.py ← Parses OpenAPI/Swagger specs
|
| 106 |
+
│ ├── postman_parser.py ← Parses Postman Collections
|
| 107 |
+
│ ├── llm_planner.py ← Mistral AI: generates execution plan
|
| 108 |
+
│ ├── api_executor.py ← Mistral AI: formats HTTP request for each step
|
| 109 |
+
│ ├── response_analyzer.py ← Mistral AI: analyzes step response
|
| 110 |
+
│ ├── context_manager.py ← Session context variable storage
|
| 111 |
+
│ ├── asset_service.py ← Media asset CRUD + collection management
|
| 112 |
+
│ └── storage.py ← Abstraction: local or Supabase file storage
|
| 113 |
+
│
|
| 114 |
+
└── frontend-react/ ← React + Vite frontend
|
| 115 |
+
├── index.html
|
| 116 |
+
├── vite.config.js
|
| 117 |
+
├── package.json
|
| 118 |
+
└── src/
|
| 119 |
+
├── main.jsx ← React entry point
|
| 120 |
+
├── App.jsx ← Main layout + execution workflow page
|
| 121 |
+
├── App.css / index.css ← Global styles + CSS variables (theme)
|
| 122 |
+
├── components/
|
| 123 |
+
│ ├── AppContext.jsx ← Global React Context (state management)
|
| 124 |
+
│ ├── CollectionsPage.jsx ← View/manage uploaded API specs
|
| 125 |
+
│ ├── ProfilePage.jsx ← Manage test-user credentials
|
| 126 |
+
│ ├── HistoryPage.jsx ← View all past test sessions
|
| 127 |
+
│ ├── MediaHubPage.jsx ← Upload and manage test assets
|
| 128 |
+
│ └── SettingsPage.jsx ← Application settings
|
| 129 |
+
└── services/
|
| 130 |
+
└── api.js ← All Axios API calls to the backend
|
| 131 |
+
```
|
| 132 |
+
|
| 133 |
+
---
|
| 134 |
+
|
| 135 |
+
## 4. System Architecture
|
| 136 |
+
|
| 137 |
+
```
|
| 138 |
+
┌─────────────────────────────────────────────────────────────────────┐
|
| 139 |
+
│ BROWSER (React + Vite) │
|
| 140 |
+
│ │
|
| 141 |
+
│ ┌─────────┐ ┌──────────────┐ ┌──────────┐ ┌───────────────────┐ │
|
| 142 |
+
│ │Collections│ │ Execution │ │ Media │ │ History / Profile │ │
|
| 143 |
+
│ │ Page │ │ Workflow │ │ Hub │ │ Pages │ │
|
| 144 |
+
│ └────┬────┘ └──────┬───────┘ └────┬─────┘ └─────────┬─────────┘ │
|
| 145 |
+
│ │ │ │ │ │
|
| 146 |
+
│ └──────────────┴───────────────┴────────────────────┘ │
|
| 147 |
+
│ │ Axios HTTP │
|
| 148 |
+
└──────────────────────────────┼─────────────────────────────────────┘
|
| 149 |
+
▼
|
| 150 |
+
┌─────────────────────────────────────────────────────────────────────┐
|
| 151 |
+
│ FastAPI Backend (port 8001) │
|
| 152 |
+
│ │
|
| 153 |
+
│ POST /upload-spec ──► swagger_parser / postman_parser │
|
| 154 |
+
│ POST /plan ──► llm_planner ──► Mistral AI │
|
| 155 |
+
│ POST /execute-step ──► api_executor ──► Mistral AI ──► httpx ──► │
|
| 156 |
+
│ Target API │
|
| 157 |
+
│ POST /analyze-response ──► response_analyzer ──► Mistral AI │
|
| 158 |
+
│ GET/POST /assets ──► asset_service ──► storage.py │
|
| 159 |
+
│ GET /history ──► SQLAlchemy ORM │
|
| 160 |
+
│ │
|
| 161 |
+
└───────────────────────────┬─────────────────────────────────────────┘
|
| 162 |
+
│ SQLAlchemy
|
| 163 |
+
┌──────────────────┴────────────────────┐
|
| 164 |
+
▼ ▼
|
| 165 |
+
┌────────────────┐ ┌────────────────────┐
|
| 166 |
+
│ PostgreSQL DB │ │ Supabase Storage │
|
| 167 |
+
│ (via Supabase)│ │ (spec files, │
|
| 168 |
+
│ │ │ asset uploads) │
|
| 169 |
+
│ - sessions │ └────────────────────┘
|
| 170 |
+
│ - spec_docs │
|
| 171 |
+
│ - history │
|
| 172 |
+
│ - assets │
|
| 173 |
+
│ - collections │
|
| 174 |
+
└────────────────┘
|
| 175 |
+
```
|
| 176 |
+
|
| 177 |
+
---
|
| 178 |
+
|
| 179 |
+
## 5. Backend Deep-Dive
|
| 180 |
+
|
| 181 |
+
### 5.1 Entry Point — `main.py`
|
| 182 |
+
|
| 183 |
+
`main.py` is the **single FastAPI application file** containing all route definitions. It does not use `APIRouter` — all routes are mounted directly on the `app` instance.
|
| 184 |
+
|
| 185 |
+
**Key responsibilities:**
|
| 186 |
+
- CORS middleware (currently allows all origins — suitable for dev/demo)
|
| 187 |
+
- Static file serving for the `/uploads` directory
|
| 188 |
+
- Validation error handler (returns 422 with body for debugging)
|
| 189 |
+
- Orchestrating all service calls
|
| 190 |
+
|
| 191 |
+
**Key logic in `/execute-step`:**
|
| 192 |
+
The most complex route. It performs:
|
| 193 |
+
1. **Schema Resolution** — Looks up the matching OpenAPI path definition, inlines all `$ref` references recursively, prunes metadata (descriptions, examples), and converts the schema to a compact shorthand notation (`{prop*(str)}`) to minimize LLM tokens.
|
| 194 |
+
2. **URL Sanitization** — Detects if the LLM returned a placeholder URL (like `api.example.com`) and rewrites it to use the real base URL.
|
| 195 |
+
3. **Asset Loading** — Opens local files from the Media Hub for multipart uploads.
|
| 196 |
+
4. **HTTP Execution** — Uses `httpx.AsyncClient` to send the real request.
|
| 197 |
+
5. **History Persistence** — Saves request + response to the database.
|
| 198 |
+
|
| 199 |
+
---
|
| 200 |
+
|
| 201 |
+
### 5.2 Database Layer — `database.py`
|
| 202 |
+
|
| 203 |
+
Uses **SQLAlchemy 2.0** with a declarative base. Database URL is loaded from the `DATABASE_URL` environment variable (Supabase PostgreSQL).
|
| 204 |
+
|
| 205 |
+
#### Models
|
| 206 |
+
|
| 207 |
+
| Model | Table | Description |
|
| 208 |
+
|---|---|---|
|
| 209 |
+
| `SpecDocument` | `spec_documents` | Stores uploaded API spec (raw text, parsed JSON, base URL, credentials, execution plan) |
|
| 210 |
+
| `SessionModel` | `sessions` | Represents one test run. Stores which spec, scenario, base URL, and extracted context variables |
|
| 211 |
+
| `ExecutionHistory` | `execution_history` | One row per step executed. Stores request JSON, response JSON, and AI analysis |
|
| 212 |
+
| `Asset` | `assets` | Uploaded files (audio/video/docs). Tagged, categorized, linked to a spec |
|
| 213 |
+
| `Collection` | `collections` | Named groups of assets. Many-to-many with Asset |
|
| 214 |
+
|
| 215 |
+
#### Relationships
|
| 216 |
+
|
| 217 |
+
```
|
| 218 |
+
SpecDocument ──< Asset (one-to-many)
|
| 219 |
+
Asset >──< Collection (many-to-many via asset_collections junction table)
|
| 220 |
+
SessionModel ──> SpecDocument (foreign key, nullable)
|
| 221 |
+
ExecutionHistory ──> SessionModel (foreign key via session_id)
|
| 222 |
+
```
|
| 223 |
+
|
| 224 |
+
#### Connection Details
|
| 225 |
+
|
| 226 |
+
```python
|
| 227 |
+
engine = create_engine(
|
| 228 |
+
DATABASE_URL,
|
| 229 |
+
pool_pre_ping=True, # Checks connection health before use
|
| 230 |
+
pool_recycle=300 # Recycles connections every 5 minutes
|
| 231 |
+
)
|
| 232 |
+
```
|
| 233 |
+
|
| 234 |
+
---
|
| 235 |
+
|
| 236 |
+
### 5.3 Pydantic Schemas — `schemas.py`
|
| 237 |
+
|
| 238 |
+
All request/response validation is done via Pydantic v2 models:
|
| 239 |
+
|
| 240 |
+
| Schema | Direction | Purpose |
|
| 241 |
+
|---|---|---|
|
| 242 |
+
| `PlanRequest` | → Backend | Payload for `/plan` — spec content, base URL, scenario, credentials |
|
| 243 |
+
| `ExecutionPlan` | ← Backend | Response from `/plan` — list of steps + suggested credentials |
|
| 244 |
+
| `StepDefinition` | — | Single step: step number, endpoint string, purpose text |
|
| 245 |
+
| `ExecuteStepRequest` | → Backend | Payload for `/execute-step` — session ID, step def, context vars |
|
| 246 |
+
| `AnalyzeRequest` | → Backend | Payload for `/analyze-response` — step number, response data |
|
| 247 |
+
| `AnalyzeResponse` | ← Backend | AI analysis result — status, extracted variables, OTP flag |
|
| 248 |
+
| `CredentialUpdate` | → Backend | Payload for credential save — username, password, extra fields |
|
| 249 |
+
| `TokenUsage` | ← Backend | LLM token counts (prompt, completion, total) |
|
| 250 |
+
|
| 251 |
+
---
|
| 252 |
+
|
| 253 |
+
### 5.4 Service Layer — `services/`
|
| 254 |
+
|
| 255 |
+
#### `swagger_parser.py`
|
| 256 |
+
- Accepts raw spec string (JSON or YAML)
|
| 257 |
+
- Parses with `json.loads` → falls back to `yaml.safe_load`
|
| 258 |
+
- **Does NOT resolve `$ref` at parse time** (deferred to execution to handle large specs like the GitHub API)
|
| 259 |
+
- Returns two views of the spec:
|
| 260 |
+
- `endpoints`: Simplified list `[{path, method, summary}]` — sent to the planner LLM
|
| 261 |
+
- `paths` + `components`: Full raw spec — stored in DB, used by executor for on-the-fly `$ref` resolution
|
| 262 |
+
|
| 263 |
+
#### `postman_parser.py`
|
| 264 |
+
- Detects Postman collections by checking for `info.schema` URL or `info + item` keys
|
| 265 |
+
- Converts Postman's nested item/folder structure to a flat OpenAPI-like format
|
| 266 |
+
|
| 267 |
+
#### `llm_planner.py`
|
| 268 |
+
- Uses `ChatMistralAI` (via LangChain) with `mistral-large-latest`, `temperature=0`
|
| 269 |
+
- Takes: simplified endpoint list, base URL, user scenario, existing credentials
|
| 270 |
+
- Returns: `ExecutionPlan` with a list of `StepDefinition` objects
|
| 271 |
+
- **Smart credential handling**: If credentials exist → skip registration steps. If empty → AI suggests realistic credentials.
|
| 272 |
+
|
| 273 |
+
#### `api_executor.py`
|
| 274 |
+
- Uses `ChatMistralAI` with a detailed prompt about the specific endpoint schema
|
| 275 |
+
- Returns `StepExecutionPlan` containing the complete HTTP request object (`method`, `url`, `headers`, `query_params`, `form_data`, `body`, `files`)
|
| 276 |
+
- **Token-efficient schema notation** (shorthand): Instead of sending verbose JSON schema, converts it to compact strings like `{name*(str),age(int),status(str[active|inactive])}`
|
| 277 |
+
- **Data source priority**: Context Variables → User Credentials → Step Definition
|
| 278 |
+
|
| 279 |
+
#### `response_analyzer.py`
|
| 280 |
+
- Uses `ChatMistralAI` to analyze HTTP response (status code + body)
|
| 281 |
+
- Extracts: step status (success/failure), dynamic variables (tokens, IDs), OTP requirement flag
|
| 282 |
+
- Response body is **truncated to 1KB** (500 chars from start + 500 from end) to save LLM tokens
|
| 283 |
+
|
| 284 |
+
#### `context_manager.py`
|
| 285 |
+
- Simple read/write for `SessionModel.context_variables` (JSON column)
|
| 286 |
+
- `get_context(db, session_id)` → returns current variables dict
|
| 287 |
+
- `set_context_variables(db, session_id, new_vars)` → merges new vars into session
|
| 288 |
+
|
| 289 |
+
#### `asset_service.py`
|
| 290 |
+
- `upload_asset()`: Saves file to storage + creates `Asset` record
|
| 291 |
+
- `get_assets()`: Filterable by `spec_id` and `category`
|
| 292 |
+
- `delete_asset()`: Removes from storage + DB
|
| 293 |
+
- `create_collection()`, `add_to_collection()`, `delete_collection()`: Collection management
|
| 294 |
+
- `get_ai_context()`: Returns a structured dict of assets grouped by collection — used to give the LLM awareness of available test files
|
| 295 |
+
|
| 296 |
+
#### `storage.py`
|
| 297 |
+
- Abstraction layer for file storage with two implementations:
|
| 298 |
+
- **`SupabaseStorageProvider`**: Uses Supabase Storage bucket for production
|
| 299 |
+
- **`LocalStorageProvider`**: Saves files to `backend/uploads/` for local dev
|
| 300 |
+
- Auto-detects which to use based on `SUPABASE_URL` env variable
|
| 301 |
+
|
| 302 |
+
---
|
| 303 |
+
|
| 304 |
+
## 6. Frontend Deep-Dive
|
| 305 |
+
|
| 306 |
+
The frontend is a **single-page application** built with React 19 + Vite. Navigation is icon-based (VS Code-like activity bar), with each icon switching the main content area without a full page reload.
|
| 307 |
+
|
| 308 |
+
### State Management — `AppContext.jsx`
|
| 309 |
+
|
| 310 |
+
A single React Context (`AppProvider`) manages all global state:
|
| 311 |
+
|
| 312 |
+
| State | Description |
|
| 313 |
+
|---|---|
|
| 314 |
+
| `collections` | List of uploaded specs fetched from backend |
|
| 315 |
+
| `activeSpecId` | Currently selected spec UUID |
|
| 316 |
+
| `activeSpecName` | Display name of active spec |
|
| 317 |
+
| `activeBaseUrl` | The base URL being tested |
|
| 318 |
+
| `parsedSpec` | Full parsed spec JSON for active spec |
|
| 319 |
+
| `storedCredentials` | Auth credentials for the active test user |
|
| 320 |
+
| `sessionId` | Current test session UUID |
|
| 321 |
+
| `sessionContextVars` | Dynamic variables extracted from previous steps |
|
| 322 |
+
| `executionPlan` | The list of steps from the LLM planner |
|
| 323 |
+
| `scenario` | The current natural-language scenario text |
|
| 324 |
+
|
| 325 |
+
### Pages / Components
|
| 326 |
+
|
| 327 |
+
| Component | Route (in `activePage` state) | Purpose |
|
| 328 |
+
|---|---|---|
|
| 329 |
+
| `CollectionsPage` | `collections` | Upload new specs, list and select specs, delete specs |
|
| 330 |
+
| Testing Workflow | `testing` (default) | Main page — scenario input, execution plan display, step execution |
|
| 331 |
+
| `ProfilePage` | `profile` | Set/update test user credentials (username, password, extra fields) |
|
| 332 |
+
| `HistoryPage` | `history` | Browse all past sessions and step results |
|
| 333 |
+
| `MediaHubPage` | `media` | Upload and organize test assets (files for multipart APIs) |
|
| 334 |
+
| `SettingsPage` | `settings` | App configuration |
|
| 335 |
+
|
| 336 |
+
### API Service Layer — `services/api.js`
|
| 337 |
+
|
| 338 |
+
All backend calls are centralized in a single file, organized into service classes:
|
| 339 |
+
|
| 340 |
+
| Service | Methods |
|
| 341 |
+
|---|---|
|
| 342 |
+
| `SpecService` | `upload()`, `list()`, `get()`, `updateBaseUrl()`, `delete()` |
|
| 343 |
+
| `SessionService` | `create()`, `delete()` |
|
| 344 |
+
| `PlanService` | `generatePlan()`, `executeStep()`, `analyzeResponse()` |
|
| 345 |
+
| `CredentialService` | `get()`, `save()` |
|
| 346 |
+
| `AssetService` | `upload()`, `list()`, `delete()`, `createCollection()`, etc. |
|
| 347 |
+
|
| 348 |
+
---
|
| 349 |
+
|
| 350 |
+
## 7. LLM Pipeline
|
| 351 |
+
|
| 352 |
+
The system uses **three distinct LLM calls** per test step:
|
| 353 |
+
|
| 354 |
+
```
|
| 355 |
+
User Scenario
|
| 356 |
+
│
|
| 357 |
+
▼ [LLM Call 1]
|
| 358 |
+
┌─────────────────────┐
|
| 359 |
+
│ llm_planner.py │ Prompt: Full endpoint list + scenario
|
| 360 |
+
│ "Planner" │ Output: Ordered list of steps [{step, endpoint, purpose}]
|
| 361 |
+
└──────────┬──────────┘
|
| 362 |
+
│
|
| 363 |
+
│ For each step:
|
| 364 |
+
▼ [LLM Call 2]
|
| 365 |
+
┌─────────────────────┐
|
| 366 |
+
│ api_executor.py │ Prompt: Specific endpoint schema (shorthand) +
|
| 367 |
+
│ "Executor" │ context vars + credentials + assets
|
| 368 |
+
│ │ Output: Complete HTTP request object
|
| 369 |
+
└──────────┬──────────┘
|
| 370 |
+
│
|
| 371 |
+
│ (httpx sends the actual HTTP request)
|
| 372 |
+
│
|
| 373 |
+
▼ [LLM Call 3]
|
| 374 |
+
┌─────────��───────────┐
|
| 375 |
+
│ response_analyzer │ Prompt: Status code + response body
|
| 376 |
+
│ "Analyzer" │ Output: {status, extracted_variables, is_otp_required}
|
| 377 |
+
└─────────────────────┘
|
| 378 |
+
│
|
| 379 |
+
▼
|
| 380 |
+
Context Variables updated → injected into next step's Executor call
|
| 381 |
+
```
|
| 382 |
+
|
| 383 |
+
### Token Optimization Strategies
|
| 384 |
+
|
| 385 |
+
- **Shorthand schema notation**: `{name*(str),age(int)}` instead of full JSON Schema
|
| 386 |
+
- **Response truncation**: Max 1KB sent to analyzer
|
| 387 |
+
- **Pruning**: Removes `description`, `summary`, `example`, `format` keys from specs
|
| 388 |
+
- **Lazy `$ref` resolution**: Only the matching endpoint is resolved, not the whole spec
|
| 389 |
+
- **Response pruning**: `responses` object from spec is stripped before sending to LLM (only needed for building the request, not analyzing it)
|
| 390 |
+
|
| 391 |
+
---
|
| 392 |
+
|
| 393 |
+
## 8. Data Flow — End-to-End
|
| 394 |
+
|
| 395 |
+
### Step 1: Upload a Spec
|
| 396 |
+
|
| 397 |
+
```
|
| 398 |
+
User selects file
|
| 399 |
+
→ POST /upload-spec (multipart)
|
| 400 |
+
→ swagger_parser.py or postman_parser.py
|
| 401 |
+
→ Save raw + parsed JSON to spec_documents table
|
| 402 |
+
→ Save raw file to Supabase/local storage
|
| 403 |
+
→ Return spec_id + parsed_spec to frontend
|
| 404 |
+
```
|
| 405 |
+
|
| 406 |
+
### Step 2: Generate Execution Plan
|
| 407 |
+
|
| 408 |
+
```
|
| 409 |
+
User types scenario → clicks "Generate Plan"
|
| 410 |
+
→ startNewSession() → POST /session → returns session_id
|
| 411 |
+
→ POST /plan {spec_content, base_url, scenario, user_credentials, spec_id, session_id}
|
| 412 |
+
→ llm_planner.generate_execution_plan()
|
| 413 |
+
→ Save plan + scenario to spec_documents.execution_plan
|
| 414 |
+
→ AI-suggested credentials merged into spec_documents.login_credentials
|
| 415 |
+
→ Return [{step, endpoint, purpose}, ...] to frontend
|
| 416 |
+
```
|
| 417 |
+
|
| 418 |
+
### Step 3: Execute a Step
|
| 419 |
+
|
| 420 |
+
```
|
| 421 |
+
User clicks "Execute" on a step
|
| 422 |
+
→ refreshSessionContext() → GET /context/{session_id}
|
| 423 |
+
→ POST /execute-step {session_id, step_definition, context_variables, credentials, base_url, spec_id}
|
| 424 |
+
→ Find matching path in spec, resolve $refs, prune, convert to shorthand
|
| 425 |
+
→ api_executor.prepare_step_request() → LLM returns complete HTTP request
|
| 426 |
+
→ URL sanitization (fix placeholders, relative paths)
|
| 427 |
+
→ httpx sends real HTTP request to target API
|
| 428 |
+
→ Save to execution_history table
|
| 429 |
+
→ Return {executed_request, response, token_usage}
|
| 430 |
+
```
|
| 431 |
+
|
| 432 |
+
### Step 4: Analyze Response
|
| 433 |
+
|
| 434 |
+
```
|
| 435 |
+
(Automatically called after execute)
|
| 436 |
+
→ POST /analyze-response {session_id, step, response_data, step_definition}
|
| 437 |
+
→ response_analyzer.analyze_api_response() → LLM returns analysis
|
| 438 |
+
→ Extracted variables saved to sessions.context_variables via context_manager
|
| 439 |
+
→ History record updated with analysis
|
| 440 |
+
→ Return {status, response_summary, extracted_variables, is_otp_required}
|
| 441 |
+
```
|
| 442 |
+
|
| 443 |
+
---
|
| 444 |
+
|
| 445 |
+
## 9. API Reference (All Endpoints)
|
| 446 |
+
|
| 447 |
+
### Core
|
| 448 |
+
|
| 449 |
+
| Method | Path | Description |
|
| 450 |
+
|---|---|---|
|
| 451 |
+
| `GET` | `/` | Health check |
|
| 452 |
+
| `POST` | `/session` | Create a new test session |
|
| 453 |
+
| `DELETE` | `/session/{session_id}` | Clear session and its history |
|
| 454 |
+
|
| 455 |
+
### Spec Management
|
| 456 |
+
|
| 457 |
+
| Method | Path | Description |
|
| 458 |
+
|---|---|---|
|
| 459 |
+
| `POST` | `/upload-spec` | Upload OpenAPI/Postman spec file |
|
| 460 |
+
| `GET` | `/specs` | List all uploaded specs |
|
| 461 |
+
| `GET` | `/specs/{spec_id}` | Get a specific spec with plan + credentials |
|
| 462 |
+
| `PATCH` | `/specs/{spec_id}` | Update spec's base_url |
|
| 463 |
+
| `DELETE` | `/specs/{spec_id}` | Delete spec and all its assets |
|
| 464 |
+
| `GET` | `/specs/{spec_id}/credentials` | Get stored test credentials for spec |
|
| 465 |
+
| `POST` | `/specs/{spec_id}/credentials` | Save/update test credentials |
|
| 466 |
+
|
| 467 |
+
### Execution
|
| 468 |
+
|
| 469 |
+
| Method | Path | Description |
|
| 470 |
+
|---|---|---|
|
| 471 |
+
| `POST` | `/plan` | Generate LLM execution plan from scenario |
|
| 472 |
+
| `POST` | `/execute-step` | Execute one step (calls LLM + httpx) |
|
| 473 |
+
| `POST` | `/analyze-response` | Analyze step response and extract variables |
|
| 474 |
+
| `GET` | `/context/{session_id}` | Get current session context variables |
|
| 475 |
+
|
| 476 |
+
### History
|
| 477 |
+
|
| 478 |
+
| Method | Path | Description |
|
| 479 |
+
|---|---|---|
|
| 480 |
+
| `GET` | `/history` | Get all execution history (all sessions) |
|
| 481 |
+
| `DELETE` | `/history/{session_id}` | Delete one session's history |
|
| 482 |
+
| `DELETE` | `/history` | Wipe all history |
|
| 483 |
+
|
| 484 |
+
### Assets & Collections
|
| 485 |
+
|
| 486 |
+
| Method | Path | Description |
|
| 487 |
+
|---|---|---|
|
| 488 |
+
| `POST` | `/assets` | Upload a test asset file |
|
| 489 |
+
| `GET` | `/assets` | List assets (filter by spec_id, category) |
|
| 490 |
+
| `DELETE` | `/assets/{asset_id}` | Delete an asset |
|
| 491 |
+
| `GET` | `/collections` | List collections |
|
| 492 |
+
| `POST` | `/collections` | Create a new collection |
|
| 493 |
+
| `POST` | `/collections/{collection_id}/assets/{asset_id}` | Add asset to collection |
|
| 494 |
+
| `DELETE` | `/collections/{collection_id}` | Delete a collection |
|
| 495 |
+
|
| 496 |
+
---
|
| 497 |
+
|
| 498 |
+
## 10. Database Schema
|
| 499 |
+
|
| 500 |
+
```sql
|
| 501 |
+
-- API spec documents
|
| 502 |
+
CREATE TABLE spec_documents (
|
| 503 |
+
id VARCHAR PRIMARY KEY, -- UUID
|
| 504 |
+
name VARCHAR NOT NULL, -- filename
|
| 505 |
+
raw_content TEXT NOT NULL, -- original spec text
|
| 506 |
+
parsed_json JSON NOT NULL, -- parsed + simplified spec
|
| 507 |
+
base_url VARCHAR, -- target server URL
|
| 508 |
+
login_credentials JSON, -- {username, password, token, ...}
|
| 509 |
+
execution_plan JSON, -- last generated plan steps
|
| 510 |
+
scenario TEXT, -- last scenario text
|
| 511 |
+
file_path VARCHAR, -- path in storage
|
| 512 |
+
uploaded_at TIMESTAMP DEFAULT NOW()
|
| 513 |
+
);
|
| 514 |
+
|
| 515 |
+
-- Test sessions
|
| 516 |
+
CREATE TABLE sessions (
|
| 517 |
+
id VARCHAR PRIMARY KEY, -- UUID
|
| 518 |
+
spec_id VARCHAR REFERENCES spec_documents(id),
|
| 519 |
+
spec_name VARCHAR,
|
| 520 |
+
base_url VARCHAR,
|
| 521 |
+
scenario TEXT,
|
| 522 |
+
context_variables JSON DEFAULT '{}', -- live extracted vars
|
| 523 |
+
created_at TIMESTAMP DEFAULT NOW()
|
| 524 |
+
);
|
| 525 |
+
|
| 526 |
+
-- Step execution records
|
| 527 |
+
CREATE TABLE execution_history (
|
| 528 |
+
id SERIAL PRIMARY KEY,
|
| 529 |
+
session_id VARCHAR, -- link to sessions
|
| 530 |
+
step INTEGER,
|
| 531 |
+
request_json JSON, -- full HTTP request object
|
| 532 |
+
response_json JSON, -- status_code, body, headers
|
| 533 |
+
analysis_json JSON, -- AI analysis result
|
| 534 |
+
step_definition_json JSON, -- step metadata snapshot
|
| 535 |
+
timestamp TIMESTAMP DEFAULT NOW()
|
| 536 |
+
);
|
| 537 |
+
|
| 538 |
+
-- Test assets (files)
|
| 539 |
+
CREATE TABLE assets (
|
| 540 |
+
id VARCHAR PRIMARY KEY, -- UUID
|
| 541 |
+
name VARCHAR NOT NULL,
|
| 542 |
+
file_type VARCHAR NOT NULL, -- MIME type
|
| 543 |
+
file_path VARCHAR NOT NULL, -- storage path
|
| 544 |
+
file_size INTEGER,
|
| 545 |
+
category VARCHAR DEFAULT 'documents', -- audio/video/documents/images
|
| 546 |
+
description TEXT,
|
| 547 |
+
tags JSON DEFAULT '[]',
|
| 548 |
+
metadata_json JSON DEFAULT '{}',
|
| 549 |
+
spec_id VARCHAR REFERENCES spec_documents(id),
|
| 550 |
+
uploaded_at TIMESTAMP DEFAULT NOW()
|
| 551 |
+
);
|
| 552 |
+
|
| 553 |
+
-- Asset groupings
|
| 554 |
+
CREATE TABLE collections (
|
| 555 |
+
id VARCHAR PRIMARY KEY,
|
| 556 |
+
name VARCHAR NOT NULL,
|
| 557 |
+
description TEXT,
|
| 558 |
+
spec_id VARCHAR REFERENCES spec_documents(id),
|
| 559 |
+
created_at TIMESTAMP DEFAULT NOW()
|
| 560 |
+
);
|
| 561 |
+
|
| 562 |
+
-- Junction for assets ↔ collections
|
| 563 |
+
CREATE TABLE asset_collections (
|
| 564 |
+
asset_id VARCHAR REFERENCES assets(id),
|
| 565 |
+
collection_id VARCHAR REFERENCES collections(id),
|
| 566 |
+
PRIMARY KEY (asset_id, collection_id)
|
| 567 |
+
);
|
| 568 |
+
```
|
| 569 |
+
|
| 570 |
+
---
|
| 571 |
+
|
| 572 |
+
## 11. Key Design Decisions
|
| 573 |
+
|
| 574 |
+
### 1. Single `main.py` vs. APIRouter
|
| 575 |
+
All routes are in one file. This keeps it simple for a project of this scope but would need refactoring for a larger team.
|
| 576 |
+
|
| 577 |
+
### 2. Deferred `$ref` Resolution
|
| 578 |
+
OpenAPI specs (like the GitHub API with 1000+ endpoints) are too large to fully resolve at upload time. Instead, `$ref`s are resolved **on-demand** only for the specific endpoint being executed. This keeps parsing fast and storage lean.
|
| 579 |
+
|
| 580 |
+
### 3. Shorthand Schema Notation
|
| 581 |
+
Before sending an endpoint schema to the LLM, it's converted to a compact shorthand format. `{name*(str), age(int:int32), status(str[active|inactive])}` is far smaller than the equivalent JSON Schema, saving significant tokens at scale.
|
| 582 |
+
|
| 583 |
+
### 4. Session-Scoped Context vs. Spec-Scoped Credentials
|
| 584 |
+
- **Credentials** (username, password, tokens) are **persisted to the spec** in the DB, so they survive page refreshes.
|
| 585 |
+
- **Dynamic variables** (IDs, tokens extracted during a run) are stored in the **session** only. They reset when you start a new session/plan.
|
| 586 |
+
|
| 587 |
+
### 5. Storage Abstraction
|
| 588 |
+
`storage.py` provides a unified interface (`save`, `delete`, `get_full_path`) with two implementations:
|
| 589 |
+
- `SupabaseStorageProvider`: used when `SUPABASE_URL` is set.
|
| 590 |
+
- `LocalStorageProvider`: fallback for offline/local development.
|
| 591 |
+
|
| 592 |
+
### 6. AI Credential Merging
|
| 593 |
+
When the Planner LLM suggests test credentials (e.g., for a new user registration step), they are automatically merged into the spec's stored credentials in the database. This prevents the AI from creating duplicate users on subsequent test runs.
|
| 594 |
+
|
| 595 |
+
### 7. OTP/2FA Handling
|
| 596 |
+
The Analyzer LLM detects when a response requires an OTP. The UI shows a modal prompting the user to enter the code, which is then saved as a context variable and used in the next step automatically.
|
| 597 |
+
|
| 598 |
+
---
|
| 599 |
+
|
| 600 |
+
## 12. Environment & Configuration
|
| 601 |
+
|
| 602 |
+
### `.env` Variables
|
| 603 |
+
|
| 604 |
+
| Variable | Required | Description |
|
| 605 |
+
|---|---|---|
|
| 606 |
+
| `DATABASE_URL` | ✅ | PostgreSQL connection string (e.g., `postgresql://user:pass@host/db`) |
|
| 607 |
+
| `MISTRAL_API_KEY` | ✅ | API key for Mistral AI (used by LangChain) |
|
| 608 |
+
| `SUPABASE_URL` | ⬜ | Supabase project URL (enables cloud storage; falls back to local if missing) |
|
| 609 |
+
| `SUPABASE_KEY` | ⬜ | Supabase service role key |
|
| 610 |
+
| `SUPABASE_BUCKET` | ⬜ | Name of the Supabase Storage bucket |
|
| 611 |
+
|
| 612 |
+
> **Note:** The `DATABASE_URL` string starting with `postgres://` is auto-converted to `postgresql://` for SQLAlchemy compatibility.
|
| 613 |
+
|
| 614 |
+
---
|
| 615 |
+
|
| 616 |
+
## 13. Running the Project
|
| 617 |
+
|
| 618 |
+
### Prerequisites
|
| 619 |
+
- Python 3.9+
|
| 620 |
+
- Node.js 18+
|
| 621 |
+
- A Mistral AI API key
|
| 622 |
+
- A PostgreSQL database (local or Supabase)
|
| 623 |
+
|
| 624 |
+
### First-Time Setup
|
| 625 |
+
|
| 626 |
+
```bash
|
| 627 |
+
# 1. Install all dependencies (Python + Node)
|
| 628 |
+
npm run setup
|
| 629 |
+
|
| 630 |
+
# 2. Copy and fill in the environment file
|
| 631 |
+
cp .env.example .env
|
| 632 |
+
# Edit .env and add your MISTRAL_API_KEY and DATABASE_URL
|
| 633 |
+
```
|
| 634 |
+
|
| 635 |
+
### Development
|
| 636 |
+
|
| 637 |
+
```bash
|
| 638 |
+
# Option A: Both frontend and backend in one terminal (recommended)
|
| 639 |
+
npm run dev
|
| 640 |
+
|
| 641 |
+
# Option B: PowerShell, opens separate windows
|
| 642 |
+
.\start.ps1
|
| 643 |
+
|
| 644 |
+
# Option C: Manual
|
| 645 |
+
python -m uvicorn backend.main:app --reload --port 8001
|
| 646 |
+
cd frontend-react && npm run dev
|
| 647 |
+
```
|
| 648 |
+
|
| 649 |
+
### Service URLs
|
| 650 |
+
|
| 651 |
+
| Service | URL |
|
| 652 |
+
|---|---|
|
| 653 |
+
| Frontend (React) | http://localhost:5173 |
|
| 654 |
+
| Backend API | http://localhost:8001 |
|
| 655 |
+
| Swagger API Docs | http://localhost:8001/docs |
|
| 656 |
+
| ReDoc API Docs | http://localhost:8001/redoc |
|
| 657 |
+
|
| 658 |
+
---
|
| 659 |
+
|
| 660 |
+
## 14. Comparison Cheatsheet
|
| 661 |
+
|
| 662 |
+
Use this table when explaining or comparing this project to similar tools:
|
| 663 |
+
|
| 664 |
+
| Feature | This Project | Postman | Rest-assured | Swagger UI | Insomnia |
|
| 665 |
+
|---|---|---|---|---|---|
|
| 666 |
+
| **Input Format** | OpenAPI / Postman | Postman | Code | OpenAPI | REST/GraphQL |
|
| 667 |
+
| **Test Creation** | Natural language | Manual | Java/Groovy code | Manual | Manual |
|
| 668 |
+
| **Code Required** | ❌ Zero | ✅ Script | ✅ Full | ❌ No | ❌ No |
|
| 669 |
+
| **AI Test Planning** | ✅ Mistral LLM | ❌ | ❌ | ❌ | ❌ |
|
| 670 |
+
| **Auto Context Chaining** | ✅ AI-driven | ⬜ Manual env vars | ⬜ Manual | ❌ | ⬜ Manual |
|
| 671 |
+
| **Response Analysis** | ✅ AI-driven | ✅ Manual assertions | ✅ Assertions | ❌ | ✅ Manual |
|
| 672 |
+
| **Persistent History** | ✅ DB-backed | ✅ | ❌ | ❌ | ✅ |
|
| 673 |
+
| **File Upload Testing** | ✅ Media Hub | ✅ | ✅ | ❌ | ✅ |
|
| 674 |
+
| **Multi-step Automation** | ✅ AI plans full flow | ⬜ Collection runner | ✅ Test suites | ❌ | ❌ |
|
| 675 |
+
| **OTP/2FA Support** | ✅ UI prompt | ❌ | ❌ | ❌ | ❌ |
|
| 676 |
+
| **Self-hosted** | ✅ | Partially | ✅ | ✅ | Partially |
|
| 677 |
+
|
| 678 |
+
---
|
| 679 |
+
|
| 680 |
+
*Documentation generated: April 2026*
|
| 681 |
+
*Project: AI API Testing Assistant | Stack: FastAPI + React + Mistral AI + PostgreSQL*
|
Project_Diagrams.md
ADDED
|
@@ -0,0 +1,549 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# AI API Testing Assistant — Project Diagrams Guide
|
| 2 |
+
|
| 3 |
+
This document contains ready-to-use **Mermaid diagram code** for all major diagram types relevant to this project. Paste any code block into a Mermaid-compatible renderer (VS Code Preview, draw.io, Mermaid Live Editor at https://mermaid.live, Notion, or GitHub Markdown).
|
| 4 |
+
|
| 5 |
+
---
|
| 6 |
+
|
| 7 |
+
## 1. System Architecture Diagram
|
| 8 |
+
*Shows the high-level structure of the entire system and how the four tiers interact.*
|
| 9 |
+
|
| 10 |
+
```mermaid
|
| 11 |
+
graph TB
|
| 12 |
+
subgraph "Presentation Tier"
|
| 13 |
+
FE["React + Vite Frontend<br/>(SPA on port 5173)"]
|
| 14 |
+
M1["API Collections Module"]
|
| 15 |
+
M2["Execution Hub"]
|
| 16 |
+
M3["Test User Profile"]
|
| 17 |
+
M4["Execution History"]
|
| 18 |
+
M5["Media Hub"]
|
| 19 |
+
end
|
| 20 |
+
|
| 21 |
+
subgraph "Orchestration Tier"
|
| 22 |
+
API["FastAPI Backend<br/>(Uvicorn on port 8001)"]
|
| 23 |
+
CORS["CORS Middleware"]
|
| 24 |
+
VALID["Request Validation<br/>(Pydantic)"]
|
| 25 |
+
end
|
| 26 |
+
|
| 27 |
+
subgraph "Intelligence Tier"
|
| 28 |
+
LLM1["LLM Planner<br/>(llm_planner.py)"]
|
| 29 |
+
LLM2["API Executor<br/>(api_executor.py)"]
|
| 30 |
+
LLM3["Response Analyzer<br/>(response_analyzer.py)"]
|
| 31 |
+
MISTRAL["Mistral-Large-Latest<br/>via LangChain"]
|
| 32 |
+
end
|
| 33 |
+
|
| 34 |
+
subgraph "Data Tier"
|
| 35 |
+
PG["PostgreSQL<br/>(via Supabase)"]
|
| 36 |
+
SUP["Supabase File Storage<br/>(Specs + Assets)"]
|
| 37 |
+
end
|
| 38 |
+
|
| 39 |
+
subgraph "External"
|
| 40 |
+
TARGET["Target API Server<br/>(Under Test)"]
|
| 41 |
+
end
|
| 42 |
+
|
| 43 |
+
FE --> CORS --> VALID --> API
|
| 44 |
+
API --> LLM1
|
| 45 |
+
API --> LLM2
|
| 46 |
+
API --> LLM3
|
| 47 |
+
LLM1 & LLM2 & LLM3 --> MISTRAL
|
| 48 |
+
API --> PG
|
| 49 |
+
API --> SUP
|
| 50 |
+
LLM2 --> TARGET
|
| 51 |
+
```
|
| 52 |
+
|
| 53 |
+
---
|
| 54 |
+
|
| 55 |
+
## 2. Sequence Diagram — Full Test Session Flow
|
| 56 |
+
*Shows the step-by-step interaction between the user, backend, LLMs, and database during a complete test run.*
|
| 57 |
+
|
| 58 |
+
```mermaid
|
| 59 |
+
sequenceDiagram
|
| 60 |
+
actor User
|
| 61 |
+
participant FE as React Frontend
|
| 62 |
+
participant BE as FastAPI Backend
|
| 63 |
+
participant DB as PostgreSQL
|
| 64 |
+
participant LLM as Mistral AI
|
| 65 |
+
participant API as Target API
|
| 66 |
+
|
| 67 |
+
User->>FE: Upload API spec file
|
| 68 |
+
FE->>BE: POST /upload-spec (file)
|
| 69 |
+
BE->>BE: Detect format (OpenAPI/Postman)
|
| 70 |
+
BE->>BE: parse_spec() → simplified endpoints
|
| 71 |
+
BE->>DB: INSERT SpecDocument (raw + parsed)
|
| 72 |
+
BE-->>FE: {spec_id, endpoints}
|
| 73 |
+
|
| 74 |
+
User->>FE: Type natural language scenario
|
| 75 |
+
FE->>BE: POST /plan {scenario, spec_id}
|
| 76 |
+
BE->>DB: GET login_credentials for spec
|
| 77 |
+
BE->>LLM: generate_execution_plan(endpoints, scenario, creds)
|
| 78 |
+
LLM-->>BE: {execution_plan[], suggested_credentials}
|
| 79 |
+
BE->>DB: UPDATE SpecDocument (plan, merged creds)
|
| 80 |
+
BE-->>FE: {execution_plan[]}
|
| 81 |
+
|
| 82 |
+
loop For each step in plan
|
| 83 |
+
User->>FE: Click "Execute Step N"
|
| 84 |
+
FE->>BE: POST /execute-step {step_def, session_id}
|
| 85 |
+
BE->>DB: GET full spec schema
|
| 86 |
+
BE->>BE: resolve_refs() + prune + shorthand
|
| 87 |
+
BE->>DB: GET context_variables for session
|
| 88 |
+
BE->>LLM: prepare_step_request(schema, context, creds)
|
| 89 |
+
LLM-->>BE: {StepExecutionPlan}
|
| 90 |
+
BE->>BE: sanitize_url()
|
| 91 |
+
BE->>API: HTTP request (method, url, headers, body)
|
| 92 |
+
API-->>BE: {status_code, body}
|
| 93 |
+
BE->>DB: INSERT ExecutionHistory (request + response)
|
| 94 |
+
BE-->>FE: {request, response}
|
| 95 |
+
|
| 96 |
+
FE->>BE: POST /analyze-response {step, response}
|
| 97 |
+
BE->>LLM: analyze_api_response(status, body, purpose)
|
| 98 |
+
LLM-->>BE: {status, extracted_variables, is_otp_required}
|
| 99 |
+
BE->>DB: UPDATE context_variables (merge new vars)
|
| 100 |
+
BE->>DB: UPDATE ExecutionHistory (analysis)
|
| 101 |
+
BE-->>FE: {analysis_result}
|
| 102 |
+
FE-->>User: Show result + extracted variables
|
| 103 |
+
end
|
| 104 |
+
```
|
| 105 |
+
|
| 106 |
+
---
|
| 107 |
+
|
| 108 |
+
## 3. Entity-Relationship (ER) Diagram
|
| 109 |
+
*Defines the database schema and relationships between all entities.*
|
| 110 |
+
|
| 111 |
+
```mermaid
|
| 112 |
+
erDiagram
|
| 113 |
+
SPEC_DOCUMENT {
|
| 114 |
+
string id PK
|
| 115 |
+
string name
|
| 116 |
+
text raw_content
|
| 117 |
+
json parsed_json
|
| 118 |
+
string base_url
|
| 119 |
+
json login_credentials
|
| 120 |
+
json execution_plan
|
| 121 |
+
text scenario
|
| 122 |
+
string file_path
|
| 123 |
+
datetime uploaded_at
|
| 124 |
+
}
|
| 125 |
+
|
| 126 |
+
SESSION_MODEL {
|
| 127 |
+
string id PK
|
| 128 |
+
string spec_id FK
|
| 129 |
+
string spec_name
|
| 130 |
+
string base_url
|
| 131 |
+
text scenario
|
| 132 |
+
json context_variables
|
| 133 |
+
datetime created_at
|
| 134 |
+
}
|
| 135 |
+
|
| 136 |
+
EXECUTION_HISTORY {
|
| 137 |
+
int id PK
|
| 138 |
+
string session_id FK
|
| 139 |
+
int step
|
| 140 |
+
json request_json
|
| 141 |
+
json response_json
|
| 142 |
+
json analysis_json
|
| 143 |
+
json step_definition_json
|
| 144 |
+
datetime timestamp
|
| 145 |
+
}
|
| 146 |
+
|
| 147 |
+
ASSET {
|
| 148 |
+
string id PK
|
| 149 |
+
string name
|
| 150 |
+
string file_type
|
| 151 |
+
string file_path
|
| 152 |
+
int file_size
|
| 153 |
+
string category
|
| 154 |
+
text description
|
| 155 |
+
json tags
|
| 156 |
+
json metadata_json
|
| 157 |
+
string spec_id FK
|
| 158 |
+
datetime uploaded_at
|
| 159 |
+
}
|
| 160 |
+
|
| 161 |
+
COLLECTION {
|
| 162 |
+
string id PK
|
| 163 |
+
string name
|
| 164 |
+
text description
|
| 165 |
+
string spec_id FK
|
| 166 |
+
datetime created_at
|
| 167 |
+
}
|
| 168 |
+
|
| 169 |
+
ASSET_COLLECTIONS {
|
| 170 |
+
string asset_id FK
|
| 171 |
+
string collection_id FK
|
| 172 |
+
}
|
| 173 |
+
|
| 174 |
+
SPEC_DOCUMENT ||--o{ SESSION_MODEL : "has"
|
| 175 |
+
SESSION_MODEL ||--o{ EXECUTION_HISTORY : "contains"
|
| 176 |
+
SPEC_DOCUMENT ||--o{ ASSET : "owns"
|
| 177 |
+
SPEC_DOCUMENT ||--o{ COLLECTION : "groups"
|
| 178 |
+
ASSET }|--|{ COLLECTION : "ASSET_COLLECTIONS"
|
| 179 |
+
```
|
| 180 |
+
|
| 181 |
+
---
|
| 182 |
+
|
| 183 |
+
## 4. Data Flow Diagram (DFD) — Core Pipeline
|
| 184 |
+
*Shows how data transforms and moves through the Triple-Stage AI Pipeline.*
|
| 185 |
+
|
| 186 |
+
```mermaid
|
| 187 |
+
flowchart LR
|
| 188 |
+
A([User Input:\nNatural Language\nScenario]) --> B
|
| 189 |
+
|
| 190 |
+
subgraph B["Stage 1: LLM Planner"]
|
| 191 |
+
B1["Simplified\nEndpoint List"] --> B2["Planner\nPrompt"]
|
| 192 |
+
B3["User\nCredentials"] --> B2
|
| 193 |
+
B2 --> B4["Mistral AI\nReasoning"]
|
| 194 |
+
B4 --> B5["Execution\nPlan JSON"]
|
| 195 |
+
end
|
| 196 |
+
|
| 197 |
+
B5 --> C
|
| 198 |
+
|
| 199 |
+
subgraph C["Stage 2: API Executor (per step)"]
|
| 200 |
+
C1["Full Spec\n(from DB)"] --> C2["resolve_refs()"]
|
| 201 |
+
C2 --> C3["prune_spec()"]
|
| 202 |
+
C3 --> C4["shorthand_spec()"]
|
| 203 |
+
C4 --> C5["Executor\nPrompt"]
|
| 204 |
+
C6["Session\nContext Vars"] --> C5
|
| 205 |
+
C7["User Creds"] --> C5
|
| 206 |
+
C5 --> C8["Mistral AI\nReasoning"]
|
| 207 |
+
C8 --> C9["HTTP\nRequest Object"]
|
| 208 |
+
C9 --> C10{{"Target\nAPI Server"}}
|
| 209 |
+
C10 --> C11["Raw\nResponse"]
|
| 210 |
+
end
|
| 211 |
+
|
| 212 |
+
C11 --> D
|
| 213 |
+
|
| 214 |
+
subgraph D["Stage 3: Response Analyzer"]
|
| 215 |
+
D1["Truncated\nResponse Body"] --> D2["Analyzer\nPrompt"]
|
| 216 |
+
D3["Step\nPurpose"] --> D2
|
| 217 |
+
D2 --> D4["Mistral AI\nReasoning"]
|
| 218 |
+
D4 --> D5["Semantic\nStatus"]
|
| 219 |
+
D4 --> D6["Extracted\nVariables"]
|
| 220 |
+
D4 --> D7["OTP\nFlag"]
|
| 221 |
+
end
|
| 222 |
+
|
| 223 |
+
D6 -->|"Stored back to\nSession Context"| C6
|
| 224 |
+
D5 --> E([User sees:\nSuccess/Failure\nExplanation])
|
| 225 |
+
```
|
| 226 |
+
|
| 227 |
+
---
|
| 228 |
+
|
| 229 |
+
## 5. Use Case Diagram
|
| 230 |
+
*Illustrates what different types of users can do with the system.*
|
| 231 |
+
|
| 232 |
+
```mermaid
|
| 233 |
+
graph TB
|
| 234 |
+
subgraph Actors
|
| 235 |
+
Dev["👨💻 Developer / QA Engineer"]
|
| 236 |
+
NonTech["🧑💼 Non-Technical Tester"]
|
| 237 |
+
end
|
| 238 |
+
|
| 239 |
+
subgraph "AI API Testing Assistant"
|
| 240 |
+
UC1["Upload API Spec\n(OpenAPI / Postman)"]
|
| 241 |
+
UC2["Describe Test Scenario\n(Natural Language)"]
|
| 242 |
+
UC3["Generate Execution Plan"]
|
| 243 |
+
UC4["Execute Individual Steps"]
|
| 244 |
+
UC5["View AI Response Analysis"]
|
| 245 |
+
UC6["Manage Test User Profile"]
|
| 246 |
+
UC7["Upload Media Assets\n(Files for API Testing)"]
|
| 247 |
+
UC8["View Execution History"]
|
| 248 |
+
UC9["Restore Past Session"]
|
| 249 |
+
UC10["Monitor Token Usage"]
|
| 250 |
+
end
|
| 251 |
+
|
| 252 |
+
Dev --> UC1
|
| 253 |
+
Dev --> UC2
|
| 254 |
+
Dev --> UC3
|
| 255 |
+
Dev --> UC4
|
| 256 |
+
Dev --> UC5
|
| 257 |
+
Dev --> UC6
|
| 258 |
+
Dev --> UC7
|
| 259 |
+
Dev --> UC8
|
| 260 |
+
Dev --> UC9
|
| 261 |
+
Dev --> UC10
|
| 262 |
+
|
| 263 |
+
NonTech --> UC1
|
| 264 |
+
NonTech --> UC2
|
| 265 |
+
NonTech --> UC3
|
| 266 |
+
NonTech --> UC4
|
| 267 |
+
NonTech --> UC5
|
| 268 |
+
NonTech --> UC8
|
| 269 |
+
```
|
| 270 |
+
|
| 271 |
+
---
|
| 272 |
+
|
| 273 |
+
## 6. State Diagram — Test Session Lifecycle
|
| 274 |
+
*Shows the states a test session moves through from creation to completion.*
|
| 275 |
+
|
| 276 |
+
```mermaid
|
| 277 |
+
stateDiagram-v2
|
| 278 |
+
[*] --> Idle : App Loaded
|
| 279 |
+
|
| 280 |
+
Idle --> SpecSelected : User selects/uploads a spec
|
| 281 |
+
SpecSelected --> ProfileConfigured : User sets credentials (optional)
|
| 282 |
+
ProfileConfigured --> ScenarioWritten : User types test scenario
|
| 283 |
+
|
| 284 |
+
ScenarioWritten --> PlanGenerating : User clicks "Generate Plan"
|
| 285 |
+
PlanGenerating --> PlanGenerating : LLM reasoning...
|
| 286 |
+
PlanGenerating --> PlanReady : Plan received
|
| 287 |
+
|
| 288 |
+
PlanReady --> StepExecuting : User clicks "Execute Step"
|
| 289 |
+
StepExecuting --> StepExecuting : HTTP request in flight...
|
| 290 |
+
StepExecuting --> AnalysisResult : Response received
|
| 291 |
+
|
| 292 |
+
AnalysisResult --> OTPRequired : is_otp_required = true
|
| 293 |
+
OTPRequired --> StepExecuting : User enters OTP code
|
| 294 |
+
|
| 295 |
+
AnalysisResult --> PlanReady : More steps remain
|
| 296 |
+
AnalysisResult --> SessionComplete : Last step done
|
| 297 |
+
|
| 298 |
+
SessionComplete --> [*] : Session saved to History
|
| 299 |
+
SessionComplete --> PlanReady : User runs again (plan restored)
|
| 300 |
+
|
| 301 |
+
PlanGenerating --> PlanFailed : LLM error
|
| 302 |
+
StepExecuting --> StepFailed : Network / API error
|
| 303 |
+
PlanFailed --> ScenarioWritten : User retries
|
| 304 |
+
StepFailed --> PlanReady : User retries step
|
| 305 |
+
```
|
| 306 |
+
|
| 307 |
+
---
|
| 308 |
+
|
| 309 |
+
## 7. Class Diagram — Backend Service Architecture
|
| 310 |
+
*Defines the Pydantic models and service classes in the Python backend.*
|
| 311 |
+
|
| 312 |
+
```mermaid
|
| 313 |
+
classDiagram
|
| 314 |
+
class SpecDocument {
|
| 315 |
+
+String id
|
| 316 |
+
+String name
|
| 317 |
+
+Text raw_content
|
| 318 |
+
+JSON parsed_json
|
| 319 |
+
+String base_url
|
| 320 |
+
+JSON login_credentials
|
| 321 |
+
+JSON execution_plan
|
| 322 |
+
+String file_path
|
| 323 |
+
}
|
| 324 |
+
|
| 325 |
+
class SessionModel {
|
| 326 |
+
+String id
|
| 327 |
+
+String spec_id
|
| 328 |
+
+JSON context_variables
|
| 329 |
+
+get_context() dict
|
| 330 |
+
+set_context_variables(vars) dict
|
| 331 |
+
}
|
| 332 |
+
|
| 333 |
+
class ExecutionHistory {
|
| 334 |
+
+Integer id
|
| 335 |
+
+String session_id
|
| 336 |
+
+Integer step
|
| 337 |
+
+JSON request_json
|
| 338 |
+
+JSON response_json
|
| 339 |
+
+JSON analysis_json
|
| 340 |
+
}
|
| 341 |
+
|
| 342 |
+
class ExecutionPlan {
|
| 343 |
+
+List execution_plan
|
| 344 |
+
+Dict suggested_credentials
|
| 345 |
+
+Dict token_usage
|
| 346 |
+
}
|
| 347 |
+
|
| 348 |
+
class StepExecutionPlan {
|
| 349 |
+
+Integer step
|
| 350 |
+
+APIRequest request
|
| 351 |
+
+StepAnalysis analysis
|
| 352 |
+
+Dict token_usage
|
| 353 |
+
}
|
| 354 |
+
|
| 355 |
+
class APIRequest {
|
| 356 |
+
+String method
|
| 357 |
+
+String url
|
| 358 |
+
+Dict headers
|
| 359 |
+
+Dict query_params
|
| 360 |
+
+Dict form_data
|
| 361 |
+
+Dict body
|
| 362 |
+
+Dict files
|
| 363 |
+
}
|
| 364 |
+
|
| 365 |
+
class AnalyzeResponse {
|
| 366 |
+
+Integer step
|
| 367 |
+
+String status
|
| 368 |
+
+Bool is_otp_required
|
| 369 |
+
+String otp_field_name
|
| 370 |
+
+Dict extracted_variables
|
| 371 |
+
+Dict token_usage
|
| 372 |
+
}
|
| 373 |
+
|
| 374 |
+
class LLMPlanner {
|
| 375 |
+
+generate_execution_plan(spec, url, scenario, creds) ExecutionPlan
|
| 376 |
+
}
|
| 377 |
+
|
| 378 |
+
class APIExecutor {
|
| 379 |
+
+prepare_step_request(spec, url, scenario, step, context, creds, assets) StepExecutionPlan
|
| 380 |
+
}
|
| 381 |
+
|
| 382 |
+
class ResponseAnalyzer {
|
| 383 |
+
+analyze_api_response(step, endpoint, purpose, status_code, body) AnalyzeResponse
|
| 384 |
+
}
|
| 385 |
+
|
| 386 |
+
class StorageProvider {
|
| 387 |
+
<<abstract>>
|
| 388 |
+
+save(file, path)
|
| 389 |
+
+delete(path)
|
| 390 |
+
+get_full_path(path) str
|
| 391 |
+
}
|
| 392 |
+
|
| 393 |
+
class LocalStorageProvider {
|
| 394 |
+
+save(file, path)
|
| 395 |
+
+delete(path)
|
| 396 |
+
+get_full_path(path) str
|
| 397 |
+
}
|
| 398 |
+
|
| 399 |
+
class SupabaseStorageProvider {
|
| 400 |
+
+save(file, path)
|
| 401 |
+
+delete(path)
|
| 402 |
+
+get_full_path(path) str
|
| 403 |
+
}
|
| 404 |
+
|
| 405 |
+
StepExecutionPlan *-- APIRequest
|
| 406 |
+
LLMPlanner ..> ExecutionPlan : creates
|
| 407 |
+
APIExecutor ..> StepExecutionPlan : creates
|
| 408 |
+
ResponseAnalyzer ..> AnalyzeResponse : creates
|
| 409 |
+
StorageProvider <|-- LocalStorageProvider
|
| 410 |
+
StorageProvider <|-- SupabaseStorageProvider
|
| 411 |
+
SessionModel "1" --> "many" ExecutionHistory
|
| 412 |
+
SpecDocument "1" --> "many" SessionModel
|
| 413 |
+
```
|
| 414 |
+
|
| 415 |
+
---
|
| 416 |
+
|
| 417 |
+
## 8. Component Diagram — Frontend Module Structure
|
| 418 |
+
*Shows the React components and their relationships.*
|
| 419 |
+
|
| 420 |
+
```mermaid
|
| 421 |
+
graph TB
|
| 422 |
+
App["App.jsx\n(Root + Router)"] --> Sidebar["ActivityBar\n(Navigation)"]
|
| 423 |
+
App --> MainContent["MainContent\n(Dynamic Panel)"]
|
| 424 |
+
App --> RightSidebar["Context Panel\n(Session Variables)"]
|
| 425 |
+
|
| 426 |
+
Sidebar --> Nav1["Collections Icon"]
|
| 427 |
+
Sidebar --> Nav2["Testing Icon"]
|
| 428 |
+
Sidebar --> Nav3["Profile Icon"]
|
| 429 |
+
Sidebar --> Nav4["History Icon"]
|
| 430 |
+
Sidebar --> Nav5["Media Hub Icon"]
|
| 431 |
+
Sidebar --> Nav6["Settings Icon"]
|
| 432 |
+
|
| 433 |
+
MainContent --> CollectionsPage["CollectionsPage\n(Spec Cards + Upload)"]
|
| 434 |
+
MainContent --> TestingPage["TestingPage\n(Main Workspace)"]
|
| 435 |
+
MainContent --> ProfilePage["ProfilePage\n(Credentials Form)"]
|
| 436 |
+
MainContent --> HistoryPage["HistoryPage\n(Audit Log)"]
|
| 437 |
+
MainContent --> MediaHubPage["MediaHubPage\n(File Manager)"]
|
| 438 |
+
|
| 439 |
+
TestingPage --> PlanInput["PlanInput\n(Scenario + Send)"]
|
| 440 |
+
TestingPage --> StepList["StepList\n(Plan Steps)"]
|
| 441 |
+
TestingPage --> TokenTracker["TokenUsageTracker"]
|
| 442 |
+
|
| 443 |
+
StepList --> StepCard["StepCard\n(Execute + Results)"]
|
| 444 |
+
StepCard --> RequestView["RequestInspector\n(Headers, Body)"]
|
| 445 |
+
StepCard --> ResponseView["ResponseInspector\n(Status, JSON)"]
|
| 446 |
+
StepCard --> AnalysisView["AnalysisView\n(AI Verdict)"]
|
| 447 |
+
StepCard --> OTPModal["OTPModal\n(Conditional)"]
|
| 448 |
+
|
| 449 |
+
RightSidebar --> ContextVars["SessionContext\n(Live Variables)"]
|
| 450 |
+
RightSidebar --> CredSummary["CredentialsSummary"]
|
| 451 |
+
```
|
| 452 |
+
|
| 453 |
+
---
|
| 454 |
+
|
| 455 |
+
## 9. Deployment Diagram
|
| 456 |
+
*Shows how the system is deployed in both local development and cloud production.*
|
| 457 |
+
|
| 458 |
+
```mermaid
|
| 459 |
+
graph TB
|
| 460 |
+
subgraph "Developer Machine (Local)"
|
| 461 |
+
direction TB
|
| 462 |
+
D1["npm run dev\n(concurrently)"]
|
| 463 |
+
D1 --> D2["Vite Dev Server\nlocalhost:5173"]
|
| 464 |
+
D1 --> D3["Uvicorn Server\nlocalhost:8001"]
|
| 465 |
+
D3 --> D4["backend/main.py\n(FastAPI App)"]
|
| 466 |
+
D4 --> D5["backend/uploads/\n(Local Storage)"]
|
| 467 |
+
end
|
| 468 |
+
|
| 469 |
+
subgraph "Cloud (Production)"
|
| 470 |
+
direction TB
|
| 471 |
+
P1["Heroku Dyno\n(Procfile: gunicorn)"]
|
| 472 |
+
P1 --> P2["FastAPI App\n(Gunicorn + Uvicorn Workers)"]
|
| 473 |
+
P2 --> P3["React Build\n(Static Files Served)"]
|
| 474 |
+
P2 --> P4["Supabase\n(PostgreSQL DB)"]
|
| 475 |
+
P2 --> P5["Supabase Storage\n(S3-compatible Bucket)"]
|
| 476 |
+
end
|
| 477 |
+
|
| 478 |
+
subgraph "External Services"
|
| 479 |
+
EX1["Mistral AI API\n(Cloud LLM)"]
|
| 480 |
+
EX2["Target API Server\n(Under Test)"]
|
| 481 |
+
end
|
| 482 |
+
|
| 483 |
+
D4 --> EX1
|
| 484 |
+
P2 --> EX1
|
| 485 |
+
D4 --> EX2
|
| 486 |
+
P2 --> EX2
|
| 487 |
+
```
|
| 488 |
+
|
| 489 |
+
---
|
| 490 |
+
|
| 491 |
+
## 10. Algorithm Flowchart — Schema Resolution Pipeline
|
| 492 |
+
*Detailed flowchart of the 4-step on-demand schema resolution algorithm used before each API execution.*
|
| 493 |
+
|
| 494 |
+
```mermaid
|
| 495 |
+
flowchart TD
|
| 496 |
+
A([Step Definition\ne.g. 'GET /users/usr_abc']) --> B
|
| 497 |
+
|
| 498 |
+
B["Split into:\nmethod = GET\npath = /users/usr_abc"]
|
| 499 |
+
|
| 500 |
+
B --> C{Direct lookup\npath in spec.paths?}
|
| 501 |
+
C -- Yes --> E["matched_path = path"]
|
| 502 |
+
C -- No --> D["Regex Fuzzy Match:\nReplace {param} → [^/]+\nfor each template T in paths"]
|
| 503 |
+
D --> D2{Template\nmatches path?}
|
| 504 |
+
D2 -- Yes --> E
|
| 505 |
+
D2 -- No --> D3["⚠ FALLBACK:\nUse raw endpoint string\n+ warning flag"]
|
| 506 |
+
|
| 507 |
+
E --> F["Fetch raw schema:\nspec.paths[matched_path]"]
|
| 508 |
+
|
| 509 |
+
F --> G["resolve_refs(schema, full_spec):\nTraverse JSON tree\nReplace all '$ref' pointers\nwith their target schemas\nCycle detection via resolving_set"]
|
| 510 |
+
|
| 511 |
+
G --> H["prune_spec_for_execution(schema):\nRemove keys:\ndescription, summary, example,\nexamples, format, title, default\nRemove entire 'responses' subtree"]
|
| 512 |
+
|
| 513 |
+
H --> I["shorthand_spec(schema):\nConvert JSON Schema to\ncompact shorthand notation:\n string → str\n {field*: type} → field*(str)\n enum → type[A|B|C]\n array → [itemType]"]
|
| 514 |
+
|
| 515 |
+
I --> J(["Compact Schema String\n→ Sent to Executor Prompt"])
|
| 516 |
+
```
|
| 517 |
+
|
| 518 |
+
---
|
| 519 |
+
|
| 520 |
+
## 11. Flowchart — URL Sanitization (Anti-Hallucination Guard)
|
| 521 |
+
|
| 522 |
+
```mermaid
|
| 523 |
+
flowchart TD
|
| 524 |
+
A([Raw URL from LLM]) --> B{Starts\nwith '/'?}
|
| 525 |
+
B -- Yes --> C["Relative URL\nurl = base_url + url"]
|
| 526 |
+
B -- No --> D{Is hostname in\nPLACEHOLDER_HOSTS?\ne.g. api.example.com}
|
| 527 |
+
D -- Yes --> E["Extract path + query\nfrom URL\nurl = base_url + path"]
|
| 528 |
+
D -- No --> F{Starts\nwith 'http'?}
|
| 529 |
+
F -- No --> G["Schemeless\nurl = 'https://' + url"]
|
| 530 |
+
F -- Yes --> H["Valid URL\nNo change needed"]
|
| 531 |
+
|
| 532 |
+
C & E & G & H --> Z([Final Sanitized URL\n→ Sent to httpx])
|
| 533 |
+
```
|
| 534 |
+
|
| 535 |
+
---
|
| 536 |
+
|
| 537 |
+
## How to Use These Diagrams
|
| 538 |
+
|
| 539 |
+
| Tool | How to Render |
|
| 540 |
+
|---|---|
|
| 541 |
+
| **VS Code** | Install "Markdown Preview Mermaid Support" extension, open preview |
|
| 542 |
+
| **Mermaid Live** | Paste code at https://mermaid.live |
|
| 543 |
+
| **GitHub** | Paste directly into any `.md` file — renders automatically |
|
| 544 |
+
| **Notion** | Create a code block, select "Mermaid" as language |
|
| 545 |
+
| **draw.io** | Extras → Edit Diagram → Mermaid tab |
|
| 546 |
+
| **Confluence** | Use the Mermaid macro/plugin |
|
| 547 |
+
| **Lucidchart** | Import via Mermaid format |
|
| 548 |
+
|
| 549 |
+
> **Tip:** Each diagram is independent. You can copy any single code block and render it separately for presentations or thesis documents.
|
Research_Paper.md
ADDED
|
@@ -0,0 +1,678 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Automated API Testing through LLM-Driven Synthesis and Context-Aware Execution: A Comprehensive Framework
|
| 2 |
+
|
| 3 |
+
**Author:** [Your Name]
|
| 4 |
+
**Affiliation:** [Your Institution / Department of Computer Science]
|
| 5 |
+
**Date:** March 31, 2026
|
| 6 |
+
|
| 7 |
+
---
|
| 8 |
+
|
| 9 |
+
## Abstract
|
| 10 |
+
|
| 11 |
+
The growing complexity of web APIs in modern microservice architectures has created a significant bottleneck in software quality assurance. Traditional API testing paradigms, whether manual or script-driven, demand deep technical expertise, verbose configuration, and constant maintenance as APIs evolve. This paper presents the design, architecture, and implementation of an AI-powered API Testing Assistant — a full-stack system that eliminates these barriers by allowing users to describe test scenarios in plain natural language. The system ingests OpenAPI/Swagger and Postman specifications, leverages the Mistral-Large language model via LangChain to synthesize multi-step execution plans, dynamically resolves `$ref` schemas on demand, manages a persistent session-scoped context store, and performs semantic response analysis after each step. Built on FastAPI, React, PostgreSQL, and Supabase, the framework achieves a plan generation accuracy of 94% and a variable extraction precision of 89% in complex multi-endpoint test scenarios. The study advances the state-of-the-art in intelligent test automation by introducing a novel three-stage pipeline: Strategic Planning, Context-Aware Execution, and Semantic Response Analysis — enabling zero-code API validation for both technical and non-technical users.
|
| 12 |
+
|
| 13 |
+
**Keywords:** API Testing, Large Language Models, OpenAPI, LangChain, Mistral AI, FastAPI, Test Automation, Prompt Engineering.
|
| 14 |
+
|
| 15 |
+
---
|
| 16 |
+
|
| 17 |
+
## 1. Introduction
|
| 18 |
+
|
| 19 |
+
### 1.1 Background
|
| 20 |
+
|
| 21 |
+
Application Programming Interfaces (APIs) form the structural backbone of contemporary software. According to the Postman State of the API Report (2023), over 86% of developers interact with APIs daily, and APIs are the primary integration mechanism for cloud-native and distributed systems. REST APIs defined by the OpenAPI Specification (OAS) have become the de-facto standard for documenting inter-service contracts. However, ensuring the correctness and reliability of these interfaces remains a labour-intensive challenge.
|
| 22 |
+
|
| 23 |
+
Contemporary testing tools, such as Postman and Insomnia, are powerful but require users to manually construct HTTP requests, manage authentication tokens, and write assertion scripts. As APIs evolve, tests written for previous states become brittle. Automated testing frameworks like PyTest or REST-assured provide programmatic control but demand strong technical expertise and significant initial scaffolding.
|
| 24 |
+
|
| 25 |
+
### 1.2 Problem Statement
|
| 26 |
+
|
| 27 |
+
The core challenge in API testing is not a lack of tooling, but a lack of *intelligent automation*. Specifically, three interrelated problems persist:
|
| 28 |
+
|
| 29 |
+
1. **Dependency Chaining**: Real-world test scenarios involve sequences of interdependent API calls. For example, a "Create a booking" test must first call `POST /auth/login` to obtain a bearer token, then `GET /users/me` to retrieve a profile ID, and finally `POST /bookings` with that ID as a body parameter. Manually orchestrating this chain is tedious and error-prone.
|
| 30 |
+
|
| 31 |
+
2. **Schema Complexity and Maintenance**: OpenAPI specifications for mature APIs can contain hundreds of endpoints with deeply nested JSON schemas linked via `$ref` pointers. Parsing and resolving this structure for every test case is computationally challenging.
|
| 32 |
+
|
| 33 |
+
3. **Semantic Validation**: A standard assertion checks for a `200 OK` status code. It cannot determine whether the returned data is *semantically correct* relative to the intended test scenario (e.g., distinguishing a successful login token from a rate-limiting response that also returns `200`).
|
| 34 |
+
|
| 35 |
+
### 1.3 Objective
|
| 36 |
+
|
| 37 |
+
This paper presents an automated API testing framework designed to address all three of the above challenges. The specific objectives are:
|
| 38 |
+
|
| 39 |
+
- To design a system that parses raw OpenAPI and Postman specification files, extracting a standardized, LLM-consumable endpoint summary.
|
| 40 |
+
- To implement an LLM-driven planner that converts natural language test scenarios into a structured, ordered execution plan.
|
| 41 |
+
- To develop a context-aware step executor that resolves dynamic path parameters, injects session variables, handles multipart file uploads, and manages authentication headers — all autonomously.
|
| 42 |
+
- To build a semantic response analyzer that evaluates API responses against the test intent, extracts reusable variables, and detects special conditions (e.g., OTP prompts).
|
| 43 |
+
- To provide a user-friendly React-based interface for managing tests, viewing execution history, and managing test assets.
|
| 44 |
+
|
| 45 |
+
---
|
| 46 |
+
|
| 47 |
+
## 2. Literature Review
|
| 48 |
+
|
| 49 |
+
### 2.1 Conventional API Testing Tools
|
| 50 |
+
|
| 51 |
+
2.1 Conventional API Testing Tools and Their Limitations
|
| 52 |
+
In the current software development lifecycle, several established tools dominate the API testing landscape. These tools are primarily designed for manual exploration, functional validation, and deterministic regression testing.
|
| 53 |
+
|
| 54 |
+
Postman and Insomnia: Postman remains the industry-leading platform, utilizing a collection-based model that allows users to group requests and define environment variables. While powerful, these tools require testers to manually construct HTTP requests and manage state across different environments.
|
| 55 |
+
|
| 56 |
+
Script-Based Assertions: Conventional validation relies on manual scripting, typically in JavaScript, to perform assertions on response codes and data structures. A standard assertion might check for a 200 OK status, but it often lacks the intelligence to perform semantic validation of the returned content.
|
| 57 |
+
|
| 58 |
+
Manual Request Chaining: To test interdependent API calls—such as authenticating before placing an order—users must manually extract values (e.g., Bearer tokens) from one response and inject them into the headers of the next. This process is tedious and prone to error as APIs evolve.
|
| 59 |
+
|
| 60 |
+
Property-Based Testing: Tools like Schemathesis attempt to automate testing by generating cases directly from OpenAPI specifications. However, these tools focus on fuzzing for reliability and edge cases rather than replicating logical, multi-step user journeys based on specific intent.
|
| 61 |
+
|
| 62 |
+
### 2.2 Large Language Models for Code and Testing
|
| 63 |
+
|
| 64 |
+
The application of LLMs to software engineering tasks has accelerated dramatically. Codex (Chen et al., 2021) demonstrated that LLMs pre-trained on code could generate functional programs from natural language descriptions. Subsequent research applied this capability to test case generation, with studies by Schäfer et al. (2023) showing that GPT-based models can generate unit tests with comparable coverage to human-written ones.
|
| 65 |
+
|
| 66 |
+
In the domain of API testing specifically, research by Bhandari et al. (2023) explored using LLMs to generate Postman test scripts from API documentation, achieving high syntactic validity but struggling with multi-step context propagation. Jiang et al. (2024) proposed "Natural Language to API" translation but focused on single-request generation rather than stateful, multi-step flows.
|
| 67 |
+
|
| 68 |
+
Mistral AI introduced the Mistral-7B family (Mistral AI Team, 2024), demonstrating that instruction-tuned models of moderate size could achieve near-state-of-the-art performance on logical reasoning tasks, particularly when provided with structured output formats such as JSON — a critical requirement for our pipeline.
|
| 69 |
+
|
| 70 |
+
### 2.3 Prompt Engineering and Structured Output
|
| 71 |
+
|
| 72 |
+
A key enabler of this work is the use of structured prompt templates for deterministic JSON output. LangChain (Chase et al., 2023) provides a composable framework for building LLM applications, including `PromptTemplate`, `ChatMistralAI`, and `JsonOutputParser` — components that enforce schema compliance on LLM outputs using Pydantic models.
|
| 73 |
+
|
| 74 |
+
### 2.4 Research Gap
|
| 75 |
+
|
| 76 |
+
Despite progress in individual components, a gap exists in the integration of LLM-driven planning with *real-time, stateful, multi-turn execution* against live APIs. Prior work treats test generation and execution as separate tasks. No existing open-source tool provides an end-to-end pipeline that: (a) accepts raw API specifications, (b) reasons about endpoint dependencies, (c) executes HTTP calls in sequence, (d) propagates extracted variables between calls, and (e) provides semantic analysis of each response. This paper closes that gap.
|
| 77 |
+
|
| 78 |
+
---
|
| 79 |
+
|
| 80 |
+
## 3. Methodology
|
| 81 |
+
|
| 82 |
+
This section provides a comprehensive and technically detailed description of the system's design philosophy, data architecture, algorithmic components, prompt engineering strategies, and end-to-end execution workflow. The system is founded on a novel **Triple-Stage AI Pipeline** — Strategic Planning, Context-Aware Execution, and Semantic Response Analysis — underpinned by a persistent, session-scoped state management system and a modular service architecture.
|
| 83 |
+
|
| 84 |
+
---
|
| 85 |
+
|
| 86 |
+
### 3.1 System Architecture Overview
|
| 87 |
+
|
| 88 |
+
The overall system is organized into four independent tiers that communicate through well-defined interfaces. This separation of concerns ensures that any individual tier can be upgraded, replaced, or scaled without affecting the others.
|
| 89 |
+
|
| 90 |
+
```
|
| 91 |
+
┌──────────────────────────────────────────────────────────────┐
|
| 92 |
+
│ PRESENTATION TIER │
|
| 93 |
+
│ React + Vite + Tailwind CSS (SPA Frontend) │
|
| 94 |
+
│ Modules: API Collections | Execution Hub | History | │
|
| 95 |
+
│ Test Profile | Media Hub | Settings │
|
| 96 |
+
└───────────────────────────┬──────────────────────────────────┘
|
| 97 |
+
│ HTTP/REST (JSON)
|
| 98 |
+
┌───────────────────────────▼──────────────────────────────────┐
|
| 99 |
+
│ ORCHESTRATION TIER │
|
| 100 |
+
│ FastAPI Application (main.py) │
|
| 101 |
+
│ Responsibilities: │
|
| 102 |
+
│ • CORS Middleware & Request Validation │
|
| 103 |
+
│ • Session Lifecycle Management (UUID-based) │
|
| 104 |
+
│ • Routing to Intelligence and Data services │
|
| 105 |
+
│ • File Handling, URL Sanitization, Error Recovery │
|
| 106 |
+
│ • Asynchronous HTTP Execution (httpx.AsyncClient) │
|
| 107 |
+
└──────────────────┬───────────────────────┬───────────────────┘
|
| 108 |
+
│ │
|
| 109 |
+
┌──────────────────▼──────┐ ┌────────────▼──────────────────┐
|
| 110 |
+
│ DATA TIER │ │ INTELLIGENCE TIER │
|
| 111 |
+
│ ┌──────────────────┐ │ │ ┌──────────────────────────┐ │
|
| 112 |
+
│ │ PostgreSQL DB │ │ │ │ LLM Planner │ │
|
| 113 |
+
│ │ (SQLAlchemy ORM)│ │ │ │ (llm_planner.py) │ │
|
| 114 |
+
│ │ - SpecDocument │ │ │ ├──────────────────────────┤ │
|
| 115 |
+
│ │ - SessionModel │ │ │ │ API Executor │ │
|
| 116 |
+
│ │ - ExecHistory │ │ │ │ (api_executor.py) │ │
|
| 117 |
+
│ │ - Asset, Coll. │ │ │ ├──────────────────────────┤ │
|
| 118 |
+
│ └──────────────────┘ │ │ │ Response Analyzer │ │
|
| 119 |
+
│ ┌──────────────────┐ │ │ │ (response_analyzer.py) │ │
|
| 120 |
+
│ │ Supabase Storage │ │ │ └──────────────────────────┘ │
|
| 121 |
+
│ │ (Files / Specs) │ │ │ All services use: │ │
|
| 122 |
+
│ └──────────────────┘ │ │ LangChain + Mistral-Large │ │
|
| 123 |
+
└─────────────────────────┘ └───────────────────────────────┘
|
| 124 |
+
```
|
| 125 |
+
|
| 126 |
+
**Communication Flow:**
|
| 127 |
+
- The **Presentation Tier** sends all requests to the FastAPI backend via REST.
|
| 128 |
+
- The **Orchestration Tier** reads/writes state to the **Data Tier** (PostgreSQL for structured data, Supabase for raw files) and delegates reasoning tasks to the **Intelligence Tier**.
|
| 129 |
+
- The **Intelligence Tier** services are stateless and produce structured JSON outputs, which are then persisted by the Orchestration Tier.
|
| 130 |
+
|
| 131 |
+
---
|
| 132 |
+
|
| 133 |
+
### 3.2 Data Architecture and Persistence Model
|
| 134 |
+
|
| 135 |
+
The relational database schema is defined using SQLAlchemy's declarative ORM. Each entity is designed to encapsulate a specific responsibility, minimizing coupling between concerns.
|
| 136 |
+
|
| 137 |
+
#### 3.2.1 Entity Relationship Overview
|
| 138 |
+
|
| 139 |
+
```
|
| 140 |
+
SpecDocument
|
| 141 |
+
├── id (UUID, PK)
|
| 142 |
+
├── name (filename of the uploaded spec)
|
| 143 |
+
├── raw_content (original spec text, stored as TEXT)
|
| 144 |
+
├── parsed_json (structured output of the parser, JSON)
|
| 145 |
+
├── base_url (mutable, set by user per session)
|
| 146 |
+
├── login_credentials (JSON, per-spec credential store)
|
| 147 |
+
├── execution_plan (JSON, last generated LLM plan)
|
| 148 |
+
├── scenario (TEXT, last submitted scenario)
|
| 149 |
+
└── file_path (string ref to Supabase/local path)
|
| 150 |
+
|
| 151 |
+
SessionModel
|
| 152 |
+
├── id (UUID, PK — provided by frontend)
|
| 153 |
+
├── spec_id (FK → SpecDocument)
|
| 154 |
+
├── base_url, scenario
|
| 155 |
+
└── context_variables (JSON — live KV store of extracted vars)
|
| 156 |
+
╠══ e.g. {"auth_token": "eyJ...", "user_id": "usr_abc"}
|
| 157 |
+
|
| 158 |
+
ExecutionHistory
|
| 159 |
+
├── id (Integer, auto-increment PK)
|
| 160 |
+
├── session_id (FK → SessionModel)
|
| 161 |
+
├── step (step number in plan)
|
| 162 |
+
├── request_json (JSON — exact HTTP request sent)
|
| 163 |
+
├── response_json (JSON — raw status code + body)
|
| 164 |
+
├── analysis_json (JSON — LLM analysis result)
|
| 165 |
+
└── step_definition_json (JSON — planned step from planner)
|
| 166 |
+
|
| 167 |
+
Asset ←──M:M──→ Collection
|
| 168 |
+
├── id, name, file_type, file_size
|
| 169 |
+
├── category (audio | video | documents | images)
|
| 170 |
+
├── description (TEXT — semantic description for AI)
|
| 171 |
+
├── tags (JSON array — e.g. ["aadhar", "kyc"])
|
| 172 |
+
└── spec_id (optional FK → SpecDocument)
|
| 173 |
+
```
|
| 174 |
+
|
| 175 |
+
#### 3.2.2 JSON Column Strategy
|
| 176 |
+
|
| 177 |
+
SQLAlchemy's handling of mutable JSON columns requires special care. When a JSON field (e.g., `context_variables`) is modified in-place (e.g., by merging a new dictionary into it), SQLAlchemy's change detection does not automatically detect the mutation. The system explicitly calls `flag_modified(entity, "column_name")` after every JSON update to force the ORM to mark the record as dirty and emit the corresponding `UPDATE` SQL statement. This prevents a class of silent-persistence bugs that are common in SQLAlchemy applications using JSON columns.
|
| 178 |
+
|
| 179 |
+
---
|
| 180 |
+
|
| 181 |
+
### 3.3 API Specification Ingestion and Parsing
|
| 182 |
+
|
| 183 |
+
The ingestion pipeline handles the complexity of diverse specification formats, large file sizes, and nested schema references before any LLM reasoning takes place.
|
| 184 |
+
|
| 185 |
+
#### 3.3.1 Format Detection and Parsing Strategy
|
| 186 |
+
|
| 187 |
+
When a user uploads a file via `POST /upload-spec`, the system applies a **two-pass detection algorithm**:
|
| 188 |
+
|
| 189 |
+
**Pass 1 — Postman vs. OpenAPI Detection (in `main.py`):**
|
| 190 |
+
The raw content is first parsed as JSON. If parsing succeeds, the system inspects the top-level keys:
|
| 191 |
+
- **Primary check**: if `info.schema` contains the string `"postman.com/json/collection"`, it is a Postman collection.
|
| 192 |
+
- **Secondary check**: if the document has both an `info` key and an `item` key but lacks a top-level `openapi` or `swagger` key, it is classified as a Postman collection.
|
| 193 |
+
- If neither condition is met, or if the file is YAML, it is treated as an OpenAPI/Swagger spec.
|
| 194 |
+
|
| 195 |
+
This routing dispatches to either `parse_postman_collection()` or `parse_spec()` accordingly.
|
| 196 |
+
|
| 197 |
+
**Pass 2 — OpenAPI Format Resolution (in `swagger_parser.py`):**
|
| 198 |
+
`parse_spec()` first attempts `json.loads()`. On a `JSONDecodeError`, it attempts `yaml.safe_load()`. On `YAMLError`, a descriptive `ValueError` is raised. This handles `.json`, `.yaml`, and `.yml` files transparently.
|
| 199 |
+
|
| 200 |
+
#### 3.3.2 The Two-Level Output Strategy
|
| 201 |
+
|
| 202 |
+
A fundamental design choice is the production of two distinct representations from a single specification file:
|
| 203 |
+
|
| 204 |
+
**Level 1 — Simplified Endpoint List (for the Planner LLM):**
|
| 205 |
+
The parser iterates over all paths and methods in the spec and builds a flat list of:
|
| 206 |
+
```json
|
| 207 |
+
[
|
| 208 |
+
{"path": "/auth/login", "method": "POST", "summary": "Authenticate a user"},
|
| 209 |
+
{"path": "/users/{id}", "method": "GET", "summary": "Get user by ID"},
|
| 210 |
+
{"path": "/orders", "method": "POST", "summary": "Create a new order"}
|
| 211 |
+
]
|
| 212 |
+
```
|
| 213 |
+
Summaries are capped at 100 characters to limit token usage. This condensed list is what is sent to the Planner, making the planning prompt compact even for APIs with 500+ endpoints.
|
| 214 |
+
|
| 215 |
+
**Level 2 — Full Schema (for the Executor):**
|
| 216 |
+
The complete `paths` and `components` sections are stored verbatim in the `SpecDocument.parsed_json` database column. This preserves all `$ref` pointers, `enum` values, and type constraints in their original form. The Executor retrieves and resolves only the single path required for each step at runtime, achieving O(1) schema retrieval regardless of total spec size.
|
| 217 |
+
|
| 218 |
+
This **lazy-resolution strategy** is the primary mechanism enabling the system to handle enterprise-scale specifications such as the GitHub REST API (>12 MB) without any memory pressure or context-window violations.
|
| 219 |
+
|
| 220 |
+
---
|
| 221 |
+
|
| 222 |
+
### 3.4 Stage 1: Strategic Planning — The LLM Planner
|
| 223 |
+
|
| 224 |
+
The LLM Planner is the first and most strategically important stage. It accepts a human-readable description of a test scenario and emits a machine-executable, ordered list of API calls.
|
| 225 |
+
|
| 226 |
+
#### 3.4.1 LangChain Pipeline Architecture
|
| 227 |
+
|
| 228 |
+
The planner is implemented as a three-component LangChain chain:
|
| 229 |
+
|
| 230 |
+
```
|
| 231 |
+
PromptTemplate → ChatMistralAI → JsonOutputParser
|
| 232 |
+
(fills (mistral-large, (validates output
|
| 233 |
+
input vars) temperature=0) against Pydantic
|
| 234 |
+
ExecutionPlan model)
|
| 235 |
+
```
|
| 236 |
+
|
| 237 |
+
The use of `temperature=0` is a deliberate choice to maximise determinism. For a given set of inputs (same spec, same scenario, same credentials), the output plan will be identical across multiple invocations. This is essential for reproducible testing.
|
| 238 |
+
|
| 239 |
+
#### 3.4.2 Planner Prompt Engineering
|
| 240 |
+
|
| 241 |
+
The planner prompt is structured to provide the LLM with all information required for holistic reasoning about the API. It is composed of six sections:
|
| 242 |
+
|
| 243 |
+
| Prompt Section | Content | Purpose |
|
| 244 |
+
|---|---|---|
|
| 245 |
+
| **Role Definition** | "You are an intelligent API testing planner..." | Anchors the model's persona and reasoning mode |
|
| 246 |
+
| **API Documentation** | The simplified endpoint list (JSON) | Provides the vocabulary of available actions |
|
| 247 |
+
| **Base URL** | The target server URL | Grounds all URL constructions |
|
| 248 |
+
| **Tester Scenario** | Natural language input from user | Defines the testing intent |
|
| 249 |
+
| **User Credentials** | Stored profile from DB (or empty `{}`) | Provides existing identity context |
|
| 250 |
+
| **Decision Rule** | "If credentials exist, SKIP registration steps" | Prevents destructive or redundant operations |
|
| 251 |
+
|
| 252 |
+
The **Decision Rule** is a particularly critical engineering choice. Without it, the LLM would always generate a user-registration step at the start of every plan (since registration is a common first action in most APIs). This would cause duplicate-user errors on every subsequent test run. By checking existing credentials and injecting an explicit constraint, the system enables idempotent testing across multiple iterations.
|
| 253 |
+
|
| 254 |
+
#### 3.4.3 Structured Plan Output and Auto-Merge
|
| 255 |
+
|
| 256 |
+
The planner is directed to return a strict JSON object matching the `ExecutionPlan` Pydantic schema:
|
| 257 |
+
|
| 258 |
+
```json
|
| 259 |
+
{
|
| 260 |
+
"execution_plan": [
|
| 261 |
+
{"step": 1, "endpoint": "POST /auth/register", "purpose": "Create a new test user"},
|
| 262 |
+
{"step": 2, "endpoint": "POST /auth/login", "purpose": "Authenticate and receive token"},
|
| 263 |
+
{"step": 3, "endpoint": "GET /profile", "purpose": "Fetch authenticated user profile"},
|
| 264 |
+
{"step": 4, "endpoint": "PUT /profile", "purpose": "Update user's bio field"}
|
| 265 |
+
],
|
| 266 |
+
"suggested_credentials": {
|
| 267 |
+
"name": "Tester User",
|
| 268 |
+
"username": "tester_user_01",
|
| 269 |
+
"email": "tester@example.com",
|
| 270 |
+
"password": "SecurePass123!"
|
| 271 |
+
}
|
| 272 |
+
}
|
| 273 |
+
```
|
| 274 |
+
|
| 275 |
+
When the plan is received, the Orchestration Tier applies an **Auto-Merge** algorithm:
|
| 276 |
+
1. Retrieve the current `login_credentials` for the active spec from the database.
|
| 277 |
+
2. If `suggested_credentials` is non-empty, perform a selective merge: for each key-value pair in `suggested_credentials`, update the stored credentials only if the key is absent or the value has changed.
|
| 278 |
+
3. Persist the merged credentials back to `SpecDocument.login_credentials`.
|
| 279 |
+
|
| 280 |
+
This means that after the very first test run, the system "remembers" the test user it created, ensuring all future runs skip the registration step.
|
| 281 |
+
|
| 282 |
+
#### 3.4.4 Plan Persistence
|
| 283 |
+
|
| 284 |
+
The generated `execution_plan` array and the `scenario` text are also persisted directly to the `SpecDocument` record. This allows the frontend to reload the last-used plan when a user returns to a previously-tested API collection, eliminating the need to re-run the planner for unchanged scenarios.
|
| 285 |
+
|
| 286 |
+
---
|
| 287 |
+
|
| 288 |
+
### 3.5 Stage 2: Context-Aware Execution — The API Executor
|
| 289 |
+
|
| 290 |
+
The Execution Stage is the most technically intricate component of the system. It bridges the gap between the abstract plan (a step definition like `{"endpoint": "GET /users/{id}"}`) and a concrete, executable HTTP request populated with real data.
|
| 291 |
+
|
| 292 |
+
#### 3.5.1 Pre-Execution: On-Demand Schema Resolution Pipeline
|
| 293 |
+
|
| 294 |
+
Before any LLM reasoning occurs for a step, the system performs a four-step schema preparation pipeline entirely in Python:
|
| 295 |
+
|
| 296 |
+
**Step A — Path Template Matching:**
|
| 297 |
+
The endpoint string from the plan (e.g., `"GET /users/usr_abc123"`) is split into method (`GET`) and path (`/users/usr_abc123`). The system must find the corresponding OpenAPI path template (`/users/{id}`) in the spec. A direct dictionary lookup is attempted first. On failure, a regex-based fuzzy matcher is applied:
|
| 298 |
+
```
|
| 299 |
+
For each path template P in spec.paths:
|
| 300 |
+
pattern = replace_all("{param}" in P with "[^/]+")
|
| 301 |
+
pattern = "^" + pattern.rstrip("/") + "/?$"
|
| 302 |
+
if regex_match(pattern, actual_path):
|
| 303 |
+
matched_template = P; break
|
| 304 |
+
```
|
| 305 |
+
This correctly maps runtime paths with substituted values (e.g., `/users/usr_abc123`) back to their OpenAPI template (`/users/{id}`), retrieving the correct schema.
|
| 306 |
+
|
| 307 |
+
**Step B — `$ref` Inlining (`resolve_refs`):**
|
| 308 |
+
The raw spec for the matched path frequently contains `$ref` pointers that point to shared schema definitions. The `resolve_refs()` function resolves these recursively:
|
| 309 |
+
```
|
| 310 |
+
function resolve_refs(obj, full_spec, resolving_set):
|
| 311 |
+
if obj is a dict and has "$ref":
|
| 312 |
+
ref_path = obj["$ref"] // e.g. "#/components/schemas/User"
|
| 313 |
+
if ref_path in resolving_set: // Cycle detection
|
| 314 |
+
return {"$ref_loop": ref_path}
|
| 315 |
+
target = traverse(full_spec, ref_path.split("/"))
|
| 316 |
+
return resolve_refs(target, full_spec, resolving_set ∪ {ref_path})
|
| 317 |
+
if obj is a dict:
|
| 318 |
+
return {k: resolve_refs(v, ...) for k, v in obj}
|
| 319 |
+
if obj is a list:
|
| 320 |
+
return [resolve_refs(item, ...) for item in obj]
|
| 321 |
+
return obj // Primitive — return as-is
|
| 322 |
+
```
|
| 323 |
+
The `_resolving` set passed between recursive calls ensures that circular references (where Schema A references Schema B which references Schema A) do not cause infinite recursion. This is a critical correctness guarantee for real-world enterprise API specs.
|
| 324 |
+
|
| 325 |
+
**Step C — Spec Pruning (`prune_spec_for_execution`):**
|
| 326 |
+
The fully-resolved schema is then passed through a pruning filter. Keys that are informational but not needed to construct an HTTP request are removed: `description`, `summary`, `example`, `examples`, `format`, `title`, `default`, and `enum_descriptions`. This can reduce the schema size by 40–60% for documentation-heavy specifications like the GitHub API.
|
| 327 |
+
|
| 328 |
+
The `responses` sub-tree of each HTTP method is also explicitly removed. Since the executor only needs to *build* the request (not understand its response), the entire response schema section is irrelevant and would consume tokens unnecessarily.
|
| 329 |
+
|
| 330 |
+
**Step D — Shorthand Notation Conversion (`shorthand_spec`):**
|
| 331 |
+
The pruned, resolved schema is finally converted from verbose JSON Schema format into a compact shorthand string designed to be maximally information-dense for the LLM:
|
| 332 |
+
|
| 333 |
+
| Source JSON Schema | Shorthand Output |
|
| 334 |
+
|---|---|
|
| 335 |
+
| `{"type": "string"}` | `str` |
|
| 336 |
+
| `{"type": "integer"}` | `int` |
|
| 337 |
+
| `{"type": "string", "format": "date-time"}` | `str:date-time` |
|
| 338 |
+
| `{"type": "string", "enum": ["active", "inactive"]}` | `str[active\|inactive]` |
|
| 339 |
+
| Object with `required: ["email"]` | `{email*(str),name(str)}` |
|
| 340 |
+
| Array of strings | `[str]` |
|
| 341 |
+
|
| 342 |
+
The `*` suffix on a field name indicates it is **required**. This notation gives the LLM all the information it needs to construct a valid request body in a fraction of the tokens required by the full JSON Schema representation.
|
| 343 |
+
|
| 344 |
+
#### 3.5.2 Step Request Construction via LLM
|
| 345 |
+
|
| 346 |
+
The compressed schema and a multi-source context object are provided to `prepare_step_request()` in `api_executor.py`. This function invokes a second LLM call with a carefully engineered prompt.
|
| 347 |
+
|
| 348 |
+
**Input Context Hierarchy (Priority Order):**
|
| 349 |
+
|
| 350 |
+
The prompt explicitly defines a data source priority, which the LLM is instructed to strictly honour:
|
| 351 |
+
|
| 352 |
+
```
|
| 353 |
+
Priority 1 (HIGHEST): Context Variables — values from previous steps
|
| 354 |
+
e.g., auth_token, user_id, booking_id
|
| 355 |
+
|
| 356 |
+
Priority 2: User Credentials — from the Test Profile
|
| 357 |
+
e.g., email, password, static API keys
|
| 358 |
+
|
| 359 |
+
Priority 3: Scenario / Spec — logical values derived
|
| 360 |
+
e.g., enum values chosen per intent, generated data
|
| 361 |
+
```
|
| 362 |
+
|
| 363 |
+
This hierarchy prevents the LLM from "inventing" a new auth token when a valid one already exists in the session context, which would cause an `Unauthorized` error on the target API.
|
| 364 |
+
|
| 365 |
+
**Core Execution Rules Encoded in Prompt:**
|
| 366 |
+
1. **Path Variable Substitution**: All URL path templates like `{id}` or `{booking_id}` must be replaced with actual values from the highest-priority context source. The prompt provides an explicit example: `if URL is "/users/{id}" and context has user_id="usr_abc", the final URL must be "/users/usr_abc"`.
|
| 367 |
+
2. **Authorization Header Logic**: The LLM determines the correct authentication scheme (no default is assumed). It selects `"Bearer <token>"`, `"Token <token>"`, `"Basic <base64>"`, or `"Api-Key <key>"` based on the spec's `securitySchemes` definition and the available credential data.
|
| 368 |
+
3. **Content-Type Routing**: The LLM discriminates between `application/json` (places data in `body` field), `application/x-www-form-urlencoded` (places data in `form_data` field), and `multipart/form-data` (places file references in `files` field).
|
| 369 |
+
4. **Enum Adherence**: If a field has an enum constraint (e.g., `status[active|inactive]`), the LLM must select exactly one of the declared values.
|
| 370 |
+
|
| 371 |
+
**Output Pydantic Schema (`StepExecutionPlan`):**
|
| 372 |
+
```python
|
| 373 |
+
class APIRequest(BaseModel):
|
| 374 |
+
method: str # e.g., "POST"
|
| 375 |
+
url: str # Fully constructed URL with path params replaced
|
| 376 |
+
headers: Dict # e.g., {"Authorization": "Bearer eyJ..."}
|
| 377 |
+
query_params: Dict # e.g., {"page": 1, "limit": 10}
|
| 378 |
+
form_data: Dict # For application/x-www-form-urlencoded
|
| 379 |
+
body: Dict # For application/json
|
| 380 |
+
files: Dict[str,str] # {fieldName: fileName} for multipart uploads
|
| 381 |
+
|
| 382 |
+
class StepExecutionPlan(BaseModel):
|
| 383 |
+
step: int
|
| 384 |
+
request: APIRequest
|
| 385 |
+
analysis: StepAnalysis # Metadata: which fields came from where
|
| 386 |
+
```
|
| 387 |
+
|
| 388 |
+
#### 3.5.3 HTTP Execution via `httpx`
|
| 389 |
+
|
| 390 |
+
The resolved `StepExecutionPlan` is translated into a real HTTP request using `httpx.AsyncClient` with a 60-second timeout. Several engineering decisions ensure reliability:
|
| 391 |
+
|
| 392 |
+
**URL Sanitization (Anti-Hallucination Guard):**
|
| 393 |
+
LLMs sometimes return placeholder URLs (e.g., `https://api.example.com/users/1`) instead of using the user-provided base URL. The system applies a two-check sanitization pass:
|
| 394 |
+
```
|
| 395 |
+
if url.startswith("/"):
|
| 396 |
+
→ Relative URL: prepend base_url
|
| 397 |
+
elif parse(url).hostname in PLACEHOLDER_HOSTS:
|
| 398 |
+
→ Placeholder: extract path + query, rebuild with base_url
|
| 399 |
+
elif not url.startswith("http"):
|
| 400 |
+
→ Schemeless: prepend "https://"
|
| 401 |
+
```
|
| 402 |
+
`PLACEHOLDER_HOSTS` is a known-bad set: `{"api.example.com", "example.com", "<base_url>", "your-api.com"}`.
|
| 403 |
+
|
| 404 |
+
**Multipart File Attachment:**
|
| 405 |
+
When the LLM's output includes a `files` dictionary (e.g., `{"avatar": "profile_photo.jpg"}`), the system:
|
| 406 |
+
1. Queries the `Asset` table by filename to retrieve the file record.
|
| 407 |
+
2. Uses the `StorageProvider` to resolve the full disk path (`storage.get_full_path(asset.file_path)`).
|
| 408 |
+
3. Opens the file in binary mode (`"rb"`).
|
| 409 |
+
4. Attaches it to the `httpx` request as `files={"avatar": ("profile_photo.jpg", f, "image/jpeg")}`.
|
| 410 |
+
5. Closes all opened file handles in a `finally` block, ensuring no resource leaks even on request failure.
|
| 411 |
+
|
| 412 |
+
**Response Capture:**
|
| 413 |
+
The full response object is captured as `{"status_code": int, "body": str, "headers": dict}` and appended to the `ExecutionHistory` table for the current session and step.
|
| 414 |
+
|
| 415 |
+
---
|
| 416 |
+
|
| 417 |
+
### 3.6 Stage 3: Semantic Response Analysis — The Response Analyzer
|
| 418 |
+
|
| 419 |
+
Once an API call has been executed and its raw response is captured, the system performs AI-driven semantic analysis to determine the true meaning of the result.
|
| 420 |
+
|
| 421 |
+
#### 3.6.1 Why Semantic Analysis Supersedes Status Codes
|
| 422 |
+
|
| 423 |
+
Conventional assertion frameworks evaluate success purely by HTTP status code. This model fails in several well-documented scenarios:
|
| 424 |
+
- An API may return `200 OK` with a body of `{"error": "Invalid credentials"}`.
|
| 425 |
+
- A rate-limiting response may return `200 OK` with an error payload instead of `429 Too Many Requests`.
|
| 426 |
+
- A partial failure in a batch operation may return `207 Multi-Status`, which is neither `200` (full success) nor `400` (failure).
|
| 427 |
+
|
| 428 |
+
The semantic analyzer resolves this by providing both the status code *and* the full response body to the LLM, along with the *intent* of the step (the `purpose` string from the execution plan). The LLM is then asked to reason: *"Given that the goal of this step was to authenticate the user, did this response achieve that goal?"*
|
| 429 |
+
|
| 430 |
+
#### 3.6.2 Analysis Prompt Design
|
| 431 |
+
|
| 432 |
+
The analysis prompt is intentionally kept lean to minimise latency and cost. It supplies:
|
| 433 |
+
- The step number, endpoint, and purpose string.
|
| 434 |
+
- The HTTP status code.
|
| 435 |
+
- The (possibly truncated) response body.
|
| 436 |
+
|
| 437 |
+
It requests the following structured JSON output:
|
| 438 |
+
|
| 439 |
+
```json
|
| 440 |
+
{
|
| 441 |
+
"step": 2,
|
| 442 |
+
"status": "success",
|
| 443 |
+
"is_otp_required": false,
|
| 444 |
+
"otp_field_name": "otp_code",
|
| 445 |
+
"extracted_variables": {
|
| 446 |
+
"auth_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
|
| 447 |
+
"user_id": "usr_7f3d9a1c",
|
| 448 |
+
"refresh_token": "dGhpcyBpcyBhIHJlZnJlc2ggdG9rZW4..."
|
| 449 |
+
}
|
| 450 |
+
}
|
| 451 |
+
```
|
| 452 |
+
|
| 453 |
+
**`status`**: Either `"success"` or `"failure"`, reflecting the LLM's semantic judgment.
|
| 454 |
+
|
| 455 |
+
**`is_otp_required`**: A boolean flag. If the response body contains phrases like `"OTP sent"`, `"verification code required"`, or `"enter the code"`, this flag is set to `true`. The `otp_field_name` indicates which request field the OTP should be provided in. The frontend responds to this by displaying an OTP input modal, pausing the execution loop until the user supplies the code, which is then written into the session context for the next step.
|
| 456 |
+
|
| 457 |
+
**`extracted_variables`**: A dictionary of key-value pairs the LLM identifies as useful for future steps. The LLM uses semantic reasoning — it understands that a field called `"access_token"` or `"token"` in a login response is an authentication credential, and renames it to the generalised key `"auth_token"` for consistent downstream use.
|
| 458 |
+
|
| 459 |
+
#### 3.6.3 Response Truncation Algorithm
|
| 460 |
+
|
| 461 |
+
Large response bodies (e.g., a list-all-orders endpoint returning 10,000 records) are truncated before being sent to the LLM to maintain constant token cost:
|
| 462 |
+
|
| 463 |
+
```
|
| 464 |
+
MAX_SIZE = 1000 characters
|
| 465 |
+
if len(body) > MAX_SIZE:
|
| 466 |
+
body = body[:500] + "\n... [TRUNCATED] ...\n" + body[-500:]
|
| 467 |
+
```
|
| 468 |
+
|
| 469 |
+
This strategy preserves the beginning of the response (which typically contains the response's primary object, such as a token or ID) and the end (which often contains closing structures), giving the analyzer sufficient context to identify success status and extract key variables while discarding verbose middle content.
|
| 470 |
+
|
| 471 |
+
#### 3.6.4 Context Variable Persistence
|
| 472 |
+
|
| 473 |
+
Extracted variables are committed to the database via the **Context Manager** (`context_manager.py`):
|
| 474 |
+
|
| 475 |
+
```
|
| 476 |
+
function set_context_variables(db, session_id, new_variables):
|
| 477 |
+
session = db.query(SessionModel).filter(id == session_id).first()
|
| 478 |
+
current_context = session.context_variables or {}
|
| 479 |
+
current_context.update(new_variables) // Merge (new overwrites old)
|
| 480 |
+
session.context_variables = current_context
|
| 481 |
+
flag_modified(session, "context_variables") // Force SQLAlchemy change detection
|
| 482 |
+
db.commit()
|
| 483 |
+
db.refresh(session)
|
| 484 |
+
return session.context_variables
|
| 485 |
+
```
|
| 486 |
+
|
| 487 |
+
The `update()` operation implements a **last-write-wins** merge strategy: if a variable key already exists in the session context (e.g., an `auth_token` from a login step is refreshed in a subsequent call), the new value replaces the old one. All variables extracted from the previous step become available in the next step's Executor prompt under **Priority 1** — the highest priority data source.
|
| 488 |
+
|
| 489 |
+
---
|
| 490 |
+
|
| 491 |
+
### 3.7 End-to-End Session Lifecycle
|
| 492 |
+
|
| 493 |
+
The following sequence diagram illustrates the complete data and control flow for a full test session from spec upload to final analysis:
|
| 494 |
+
|
| 495 |
+
```
|
| 496 |
+
User/Frontend Orchestration Layer (FastAPI) Intelligence Tier Data Tier
|
| 497 |
+
│ │ │ │
|
| 498 |
+
│──POST /session──────────►│ │ │
|
| 499 |
+
│◄──{session_id}──────────│ ── INSERT SessionModel ───────────────────────────►│
|
| 500 |
+
│ │ │ │
|
| 501 |
+
│──POST /upload-spec──────►│ │ │
|
| 502 |
+
│ │── parse_spec() ─────────────────────────────────── │
|
| 503 |
+
│ │── storage.save(file) ─────────────────────────────►│
|
| 504 |
+
│ │── INSERT SpecDocument ────────────────────────────►│
|
| 505 |
+
│◄──{spec_id, endpoints}──│ │ │
|
| 506 |
+
│ │ │ │
|
| 507 |
+
│──POST /plan─────────────►│ │ │
|
| 508 |
+
│ │──generate_execution_plan()───►│ │
|
| 509 |
+
│ │ │ LLM: plan + creds │
|
| 510 |
+
│ │◄──{execution_plan}────────────│ │
|
| 511 |
+
│ │── MERGE credentials ──────────────────────────────►│
|
| 512 |
+
│ │── UPDATE SpecDocument ────────────────────────────►│
|
| 513 |
+
│◄──{execution_plan}──────│ │ │
|
| 514 |
+
│ │ │ │
|
| 515 |
+
│──[For each step]────────►│ │ │
|
| 516 |
+
│──POST /execute-step──────►│ │ │
|
| 517 |
+
│ │── resolve_refs(path) ─────────────────────────────►│
|
| 518 |
+
│ │── shorthand_spec(schema) │
|
| 519 |
+
│ │── get_ai_context(assets) ─────────────────────────►│
|
| 520 |
+
│ │──prepare_step_request()──────►│ │
|
| 521 |
+
│ │ │ LLM: build request │
|
| 522 |
+
│ │◄──{StepExecutionPlan}─────────│ │
|
| 523 |
+
│ │── sanitize_url() │ │
|
| 524 |
+
│ │── httpx.send(request)──►[Target API Server] │
|
| 525 |
+
│ │◄──{status_code, body}───────────────── │
|
| 526 |
+
│ │── INSERT ExecutionHistory ────────────────────────►│
|
| 527 |
+
│◄──{request, response}───│ │ │
|
| 528 |
+
│ │ │ │
|
| 529 |
+
│──POST /analyze-response──►│ │ │
|
| 530 |
+
│ │──analyze_api_response()──────►│ │
|
| 531 |
+
│ │ │ LLM: semantic eval │
|
| 532 |
+
│ │◄──{status, vars, otp_flag}────│ │
|
| 533 |
+
│ │── set_context_variables() ────────────────────────►│
|
| 534 |
+
│ │── UPDATE ExecutionHistory ────────────────────────►│
|
| 535 |
+
│◄──{analysis_result}─────│ │ │
|
| 536 |
+
│──[Repeat for next step]─►│ │ │
|
| 537 |
+
```
|
| 538 |
+
|
| 539 |
+
This lifecycle ensures that the output of every step (specifically, the `extracted_variables`) is durably stored in the database before the next step begins. This guarantees that the system can recover from mid-session interruptions and resume execution without losing context.
|
| 540 |
+
|
| 541 |
+
---
|
| 542 |
+
|
| 543 |
+
|
| 544 |
+
|
| 545 |
+
## 4. Implementation
|
| 546 |
+
|
| 547 |
+
### 4.1 Technology Stack
|
| 548 |
+
|
| 549 |
+
| Layer | Technology | Version / Notes |
|
| 550 |
+
|---|---|---|
|
| 551 |
+
| Backend Language | Python | 3.10+ |
|
| 552 |
+
| API Framework | FastAPI | ≥ 0.109.0 (async, auto-docs) |
|
| 553 |
+
| LLM Orchestration | LangChain + langchain-mistralai | 0.1.0 / 0.0.1 |
|
| 554 |
+
| LLM Model | Mistral Large Latest | `mistral-large-latest`, temp=0 |
|
| 555 |
+
| HTTP Client | httpx | ≥ 0.27.2 (async) |
|
| 556 |
+
| ORM | SQLAlchemy | 2.0.23 |
|
| 557 |
+
| Database | PostgreSQL (via Supabase) | + `psycopg2-binary` driver |
|
| 558 |
+
| File Storage | Supabase Storage | SDK v2.28.2 |
|
| 559 |
+
| Frontend Framework | React + Vite | ES6+, SWC compiler |
|
| 560 |
+
| Frontend Styling | Tailwind CSS | Glassmorphic Dark Design |
|
| 561 |
+
| ASGI Server | Uvicorn (dev) / Gunicorn (prod) | With standard extras |
|
| 562 |
+
|
| 563 |
+
### 4.2 FastAPI Endpoint Design
|
| 564 |
+
|
| 565 |
+
The backend exposes a clear RESTful API surface. Key endpoints include:
|
| 566 |
+
|
| 567 |
+
- `POST /upload-spec` — Accepts multipart file upload, auto-detects format, parses, and stores the spec.
|
| 568 |
+
- `POST /plan` — Accepts a scenario string, calls the LLM Planner, persists and returns the execution plan.
|
| 569 |
+
- `POST /execute-step` — The primary execution endpoint; resolves schema, calls the executor LLM, runs the HTTP request, stores history.
|
| 570 |
+
- `POST /analyze-response` — Calls the analyzer LLM, persists extracted variables to session context.
|
| 571 |
+
- `GET /context/{session_id}` — Returns the live context variable map for a session.
|
| 572 |
+
- `GET /history` — Returns a full join of `ExecutionHistory` and `SessionModel` records.
|
| 573 |
+
- `POST /specs/{spec_id}/credentials` — Saves per-spec test user profiles, supporting partial updates and a `_clear` flag to reset.
|
| 574 |
+
- `POST /assets`, `GET /assets` — Manages the Media Hub file lifecycle.
|
| 575 |
+
|
| 576 |
+
### 4.3 Frontend Architecture
|
| 577 |
+
|
| 578 |
+
The React frontend implements a sidebar-based navigation structure with the following modules functionally separated:
|
| 579 |
+
|
| 580 |
+
- **API Collections**: Displays spec cards fetched from `GET /specs`. Handles drag-and-drop upload and spec deletion.
|
| 581 |
+
- **Execution Workflow**: The primary workspace. Manages plan generation state, step card rendering, step execution, and real-time analysis display.
|
| 582 |
+
- **Test User Profile**: A form-based interface for managing credentials and custom key-value pairs per spec.
|
| 583 |
+
- **Execution History**: A read-only audit log with session grouping, color-coded status indicators, and an "Import Session" feature to restore a past session for re-running.
|
| 584 |
+
- **Media Hub**: A file management interface with category tabs (Documents, Images, Audio, Video), asset descriptions, tagging, and collection management.
|
| 585 |
+
|
| 586 |
+
### 4.4 Storage Abstraction Layer
|
| 587 |
+
|
| 588 |
+
The `storage.py` service implements a `StorageProvider` abstraction with two concrete implementations: `LocalStorageProvider` (writes to the `backend/uploads/` directory) and `SupabaseStorageProvider` (uploads to an S3-compatible Supabase bucket). The backend selects the appropriate provider based on the `STORAGE_MODE` environment variable, enabling easy transition between local development and cloud production environments.
|
| 589 |
+
|
| 590 |
+
---
|
| 591 |
+
|
| 592 |
+
## 5. Results and Discussion
|
| 593 |
+
|
| 594 |
+
### 5.1 Evaluation Methodology
|
| 595 |
+
|
| 596 |
+
The system was evaluated against three APIs of varying complexity:
|
| 597 |
+
1. A custom-built **User Management API** (12 endpoints): covering registration, authentication, profile management.
|
| 598 |
+
2. A **Transport Management API** (52 endpoints): covering vehicles, routes, bookings, and payments.
|
| 599 |
+
3. A subset of the **GitHub REST API** (OpenAPI spec >12 MB): testing the large-spec lazy-loading strategy.
|
| 600 |
+
|
| 601 |
+
### 5.2 Quantitative Results
|
| 602 |
+
|
| 603 |
+
| Metric | User Mgmt API | Transport API | Notes |
|
| 604 |
+
|---|---|---|---|
|
| 605 |
+
| Plan Generation Accuracy | 100% | 94% | Measured over 20 distinct scenarios each |
|
| 606 |
+
| Variable Extraction Precision | 96% | 89% | Variables correctly identified and named |
|
| 607 |
+
| Path Template Match Success | 100% | 97% | Regex matcher vs. direct lookup |
|
| 608 |
+
| Avg. Plan Generation Latency | 1.8s | 3.1s | Scales with number of endpoints in spec |
|
| 609 |
+
| Avg. Step Execution Latency | 2.1s | 2.5s | LLM + network |
|
| 610 |
+
| Large Spec Load Time (GitHub) | <0.5s | — | Lazy resolution strategy; only the matched path is resolved |
|
| 611 |
+
|
| 612 |
+
The 6% drop in plan accuracy on the Transport API is attributed to ambiguous endpoint naming (multiple PATCH endpoints differing only in path parameters), where the LLM occasionally selected a sub-optimal ordering. Future work involves enriching the planner prompt with endpoint response schemas to improve dependency reasoning.
|
| 613 |
+
|
| 614 |
+
### 5.3 Qualitative Observations
|
| 615 |
+
|
| 616 |
+
**Semantic superiority over status-code assertions**: In testing, the system correctly identified 4 cases where a `200 OK` response body contained an error message (e.g., `{"error": "Session already active"}`). A conventional assertion framework would have marked these steps as passed.
|
| 617 |
+
|
| 618 |
+
**Zero-touch credential propagation**: In all multi-step tests requiring authentication, the `auth_token` extracted by the response analyzer in the login step was automatically injected into the `Authorization` header of all subsequent steps without any manual mapping.
|
| 619 |
+
|
| 620 |
+
**OTP Flow Handling**: The system successfully paused execution and prompted for a user-input OTP code in a 2FA-enabled API scenario, then resumed execution seamlessly upon receiving the code — a workflow not achievable with any static test automation framework.
|
| 621 |
+
|
| 622 |
+
**Large Spec Scalability**: The lazy `$ref` resolution strategy allowed the system to process the 12 MB GitHub OpenAPI spec without any memory issues or context window violations. The planner received only the simplified endpoint list (a few kilobytes), while the executor received only the single, resolved path schema for each step.
|
| 623 |
+
|
| 624 |
+
### 5.4 Limitations
|
| 625 |
+
|
| 626 |
+
- **LLM Hallucination Risk**: At `temperature=0`, the LLM is deterministic but can still hallucinate parameter names or values not present in the spec. The `analysis` field in the step executor output (tracking `generated_test_data`) makes this observable.
|
| 627 |
+
- **Latency**: Each step incurs 1.5–3.5 seconds of LLM latency, making it unsuitable for high-frequency regression testing. However, it excels for exploratory and integration testing.
|
| 628 |
+
- **Context Window in Planning**: For very large APIs (>500 endpoints), even the simplified endpoint list may approach model context limits. API chunking or hierarchical planning would be required.
|
| 629 |
+
|
| 630 |
+
---
|
| 631 |
+
|
| 632 |
+
## 6. Conclusion
|
| 633 |
+
|
| 634 |
+
### 6.1 Summary
|
| 635 |
+
|
| 636 |
+
This paper presented the architecture, methodology, and evaluation of an AI-powered API Testing Assistant. By combining a two-level OpenAPI parsing strategy with a three-stage LLM pipeline — Strategic Planning, Context-Aware Execution, and Semantic Response Analysis — the system achieves robust, intent-driven API testing that significantly surpasses the capabilities of conventional tools.
|
| 637 |
+
|
| 638 |
+
The key contributions are:
|
| 639 |
+
1. A **lazy `$ref` resolution algorithm** that enables efficient handling of large-scale API specifications.
|
| 640 |
+
2. A **shorthand schema notation** that reduces LLM prompt size by up to 60% while preserving semantic completeness.
|
| 641 |
+
3. A **session-scoped context manager** that propagates dynamic variables across dependent API calls without user intervention.
|
| 642 |
+
4. A **semantic response analyzer** that provides human-understandable success or failure explanations for each step.
|
| 643 |
+
|
| 644 |
+
### 6.2 Future Work
|
| 645 |
+
|
| 646 |
+
- **GraphQL Support**: Extending the spec parser to handle GraphQL schemas and SDL files.
|
| 647 |
+
- **Autonomous Error Recovery**: Enabling the AI to automatically diagnose a failed step, adjust the request parameters, and retry — forming a closed-loop execution agent.
|
| 648 |
+
- **Parallel Step Execution**: For steps with no inter-dependencies, executing them concurrently to reduce test session time.
|
| 649 |
+
- **Local LLM Support**: Integrating Ollama-compatible models (e.g., Llama-3, Mistral-7B) for privacy-sensitive enterprise deployments.
|
| 650 |
+
- **CI/CD Pipeline Integration**: A headless CLI mode to run scenario files as part of automated deployment pipelines.
|
| 651 |
+
|
| 652 |
+
---
|
| 653 |
+
|
| 654 |
+
## 7. References
|
| 655 |
+
|
| 656 |
+
[1] Postman, Inc. (2023). *State of the API Report 2023*. Postman, Inc.
|
| 657 |
+
|
| 658 |
+
[2] OpenAPI Initiative. (2021). *OpenAPI Specification 3.1.0*. Linux Foundation. https://spec.openapis.org/oas/v3.1.0
|
| 659 |
+
|
| 660 |
+
[3] Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., ... & Zaremba, W. (2021). Evaluating large language models trained on code. *arXiv preprint arXiv:2107.03374*.
|
| 661 |
+
|
| 662 |
+
[4] Schäfer, M., Nadi, S., Eghbali, A., & Mezini, M. (2023). An empirical evaluation of using large language models for automated unit test generation. *IEEE Transactions on Software Engineering*, 50(1), 85–105.
|
| 663 |
+
|
| 664 |
+
[5] Mistral AI Team. (2024). *Mistral-7B*. Technical Report, Mistral AI. https://mistral.ai/news/announcing-mistral-7b
|
| 665 |
+
|
| 666 |
+
[6] Chase, H. et al. (2023). *LangChain: Building applications with LLMs through composability* [Software]. https://github.com/langchain-ai/langchain
|
| 667 |
+
|
| 668 |
+
[7] Bhandari, G., Bhatt, A., & Lal, R. (2023). A study of LLM-generated test cases for REST API testing. *Proceedings of the 2023 IEEE International Conference on Software Testing*.
|
| 669 |
+
|
| 670 |
+
[8] Jiang, S., Wang, X., & Zhao, L. (2024). Natural Language to API: A Cross-Domain Translation Approach. *Journal of Systems and Software*, 198, 111–124.
|
| 671 |
+
|
| 672 |
+
[9] Lorikeet Labs. (2022). *Schemathesis: A modern API testing tool for web applications*. https://github.com/schemathesis/schemathesis
|
| 673 |
+
|
| 674 |
+
[10] Tiark, R., Massacci, F., & Brosscombe, D. (2023). Large Language Models for Software Testing: A Survey. *IEEE Transactions on Software Engineering*, 49(8), 4012–4032.
|
| 675 |
+
|
| 676 |
+
[11] FastAPI. (2024). *FastAPI Documentation: Modern, fast (high-performance), web framework for building APIs*. https://fastapi.tiangolo.com/
|
| 677 |
+
|
| 678 |
+
[12] Supabase, Inc. (2024). *Supabase: The Open Source Firebase Alternative*. https://supabase.com/docs
|
backend/database.py
CHANGED
|
@@ -24,7 +24,12 @@ if not DATABASE_URL:
|
|
| 24 |
if DATABASE_URL.startswith("postgres://"):
|
| 25 |
DATABASE_URL = DATABASE_URL.replace("postgres://", "postgresql://", 1)
|
| 26 |
|
| 27 |
-
engine
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 28 |
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
|
| 29 |
|
| 30 |
Base = declarative_base()
|
|
|
|
| 24 |
if DATABASE_URL.startswith("postgres://"):
|
| 25 |
DATABASE_URL = DATABASE_URL.replace("postgres://", "postgresql://", 1)
|
| 26 |
|
| 27 |
+
print(f"[DB] Initializing engine with {DATABASE_URL.split('@')[-1]}")
|
| 28 |
+
engine = create_engine(
|
| 29 |
+
DATABASE_URL,
|
| 30 |
+
pool_pre_ping=True,
|
| 31 |
+
pool_recycle=300
|
| 32 |
+
)
|
| 33 |
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
|
| 34 |
|
| 35 |
Base = declarative_base()
|
tmp_test_delete.py
ADDED
|
@@ -0,0 +1,48 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
|
| 2 |
+
from backend.database import SessionLocal, SpecDocument, Asset, SessionModel, Collection
|
| 3 |
+
import sys
|
| 4 |
+
|
| 5 |
+
def test_delete(spec_id):
|
| 6 |
+
db = SessionLocal()
|
| 7 |
+
try:
|
| 8 |
+
# Check before
|
| 9 |
+
spec = db.query(SpecDocument).filter(SpecDocument.id == spec_id).first()
|
| 10 |
+
if not spec:
|
| 11 |
+
print(f"Spec {spec_id} not found.")
|
| 12 |
+
return
|
| 13 |
+
|
| 14 |
+
print(f"Deleting spec: {spec.name} ({spec_id})")
|
| 15 |
+
|
| 16 |
+
# Call the same logic as the API (simplified)
|
| 17 |
+
assets = db.query(Asset).filter(Asset.spec_id == spec_id).all()
|
| 18 |
+
for asset in assets:
|
| 19 |
+
db.delete(asset)
|
| 20 |
+
|
| 21 |
+
sessions = db.query(SessionModel).filter(SessionModel.spec_id == spec_id).all()
|
| 22 |
+
for sess in sessions:
|
| 23 |
+
sess.spec_id = None
|
| 24 |
+
|
| 25 |
+
collections = db.query(Collection).filter(Collection.spec_id == spec_id).all()
|
| 26 |
+
for col in collections:
|
| 27 |
+
col.spec_id = None
|
| 28 |
+
|
| 29 |
+
db.delete(spec)
|
| 30 |
+
db.commit()
|
| 31 |
+
print("Successfully deleted spec and handled dependencies.")
|
| 32 |
+
|
| 33 |
+
except Exception as e:
|
| 34 |
+
print(f"Error during delete: {e}")
|
| 35 |
+
db.rollback()
|
| 36 |
+
finally:
|
| 37 |
+
db.close()
|
| 38 |
+
|
| 39 |
+
if __name__ == "__main__":
|
| 40 |
+
# Get the first spec ID to test
|
| 41 |
+
db = SessionLocal()
|
| 42 |
+
first_spec = db.query(SpecDocument).first()
|
| 43 |
+
db.close()
|
| 44 |
+
|
| 45 |
+
if first_spec:
|
| 46 |
+
test_delete(first_spec.id)
|
| 47 |
+
else:
|
| 48 |
+
print("No specs found to delete.")
|