Amitkasar commited on
Commit
2760db0
·
1 Parent(s): 97fada4

Add documentation, diagrams, and research paper

Browse files

Add comprehensive project documentation (PROJECT_DOCUMENTATION.md), a Mermaid diagrams guide (Project_Diagrams.md), and an academic research paper (Research_Paper.md) to describe architecture, data flow, API contracts, and design decisions. Also add a temporary test file (tmp_test_delete.py) and apply updates to backend/database.py. These additions provide reference material for developers, architects, and reviewers — including runnable diagrams, DB schema, API endpoints, and usage/run instructions.

PROJECT_DOCUMENTATION.md ADDED
@@ -0,0 +1,681 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # 📘 AI API Testing Assistant — Project Documentation
2
+
3
+ > A comprehensive reference document covering the architecture, components, data flow, API contracts, and design decisions of this project. Intended for comparison with similar tools and for explaining the system to others.
4
+
5
+ ---
6
+
7
+ ## Table of Contents
8
+
9
+ 1. [Project Overview](#1-project-overview)
10
+ 2. [Technology Stack](#2-technology-stack)
11
+ 3. [Project Structure](#3-project-structure)
12
+ 4. [System Architecture](#4-system-architecture)
13
+ 5. [Backend Deep-Dive](#5-backend-deep-dive)
14
+ - 5.1 [Entry Point — main.py](#51-entry-point--mainpy)
15
+ - 5.2 [Database Layer — database.py](#52-database-layer--databasepy)
16
+ - 5.3 [Pydantic Schemas — schemas.py](#53-pydantic-schemas--schemaspy)
17
+ - 5.4 [Service Layer — services/](#54-service-layer--services)
18
+ 6. [Frontend Deep-Dive](#6-frontend-deep-dive)
19
+ 7. [LLM Pipeline](#7-llm-pipeline)
20
+ 8. [Data Flow — End-to-End](#8-data-flow--end-to-end)
21
+ 9. [API Reference (All Endpoints)](#9-api-reference-all-endpoints)
22
+ 10. [Database Schema](#10-database-schema)
23
+ 11. [Key Design Decisions](#11-key-design-decisions)
24
+ 12. [Environment & Configuration](#12-environment--configuration)
25
+ 13. [Running the Project](#13-running-the-project)
26
+ 14. [Comparison Cheatsheet](#14-comparison-cheatsheet)
27
+
28
+ ---
29
+
30
+ ## 1. Project Overview
31
+
32
+ **AI API Testing Assistant** is a full-stack web application that automates the testing of REST APIs. Instead of manually writing test scripts, the user:
33
+
34
+ 1. Uploads an **OpenAPI/Swagger spec** (JSON or YAML) or a **Postman Collection**.
35
+ 2. Describes a test scenario in **plain English** (e.g., *"Register a new user, log in, then create a new product"*).
36
+ 3. The system uses **Mistral AI** (via LangChain) to generate a **step-by-step execution plan**.
37
+ 4. Each step is individually executed as a real HTTP request, with the AI automatically resolving path parameters, injecting auth tokens from previous steps, and handling file uploads.
38
+ 5. The AI then **analyzes each response**, extracts dynamic values (tokens, IDs), and chains them into subsequent steps.
39
+
40
+ ### Core Value Propositions
41
+
42
+ | Feature | Description |
43
+ |---|---|
44
+ | **Natural Language Testing** | No code required — describe what you want to test |
45
+ | **Auto Context Chaining** | Tokens/IDs extracted from step N are auto-injected into step N+1 |
46
+ | **Spec-Aware** | Reads OpenAPI schemas to build correctly typed requests |
47
+ | **Multi-Format Input** | Supports OpenAPI 2/3 (JSON/YAML) and Postman Collections |
48
+ | **Persistent History** | All sessions, plans, and results are stored in a database |
49
+ | **Media Hub** | Upload files (audio, video, documents) for use in API tests |
50
+
51
+ ---
52
+
53
+ ## 2. Technology Stack
54
+
55
+ ### Backend
56
+
57
+ | Layer | Technology | Version | Purpose |
58
+ |---|---|---|---|
59
+ | Web Framework | **FastAPI** | ≥0.109 | REST API server |
60
+ | ASGI Server | **Uvicorn** | ≥0.27 | Runs FastAPI in development |
61
+ | Production Server | **Gunicorn** | 21.2 | Runs Uvicorn workers in production |
62
+ | LLM Orchestration | **LangChain** | 0.1.0 | Prompt chaining with Mistral |
63
+ | LLM Provider | **Mistral AI** | `mistral-large-latest` | Plan generation, step execution, analysis |
64
+ | HTTP Client | **httpx** | ≥0.27 | Sends actual API requests during execution |
65
+ | ORM | **SQLAlchemy** | 2.0.23 | Database queries and model definitions |
66
+ | Database | **PostgreSQL** (via Supabase) | — | Persistent storage |
67
+ | DB Driver | **psycopg2-binary** | 2.9.9 | PostgreSQL adapter |
68
+ | Cloud Storage | **Supabase Storage** | 2.28.2 | Stores uploaded spec files and assets |
69
+ | Spec Parsing | **prance** + **PyYAML** | — | Parse OpenAPI YAML/JSON |
70
+ | Validation | **Pydantic** | ≥2.6 | Request/response data models |
71
+ | Auth/Config | **python-dotenv** | 1.0.0 | Load env variables from `.env` |
72
+
73
+ ### Frontend
74
+
75
+ | Layer | Technology | Version | Purpose |
76
+ |---|---|---|---|
77
+ | UI Framework | **React** | 19.x | Component-based UI |
78
+ | Build Tool | **Vite** | 8.x | Fast dev server + bundler |
79
+ | Styling | **Tailwind CSS** | 4.x | Utility-first CSS |
80
+ | Icons | **lucide-react** | 0.577 | Icon library |
81
+ | HTTP Client | **Axios** | 1.x | API communication with backend |
82
+ | Routing | **react-router-dom** | 7.x | Client-side navigation |
83
+
84
+ ---
85
+
86
+ ## 3. Project Structure
87
+
88
+ ```
89
+ Testing/ ← Project root
90
+ ├── .env ← Environment variables (secrets)
91
+ ├── .env.example ← Template for env setup
92
+ ├── requirements.txt ← Python dependencies
93
+ ├── package.json ← Root-level npm scripts (dev runner)
94
+ ├── Procfile ← Heroku/Railway deployment config
95
+ ├── runtime.txt ← Python version pin
96
+ ├── start.ps1 ← PowerShell startup script
97
+ ├── openapi.yaml ← Sample OpenAPI spec for testing
98
+ │
99
+ ├── backend/ ← FastAPI application
100
+ │ ├── __init__.py
101
+ │ ├── main.py ← All FastAPI routes (API gateway)
102
+ │ ├── database.py ← SQLAlchemy models + DB connection
103
+ │ ├── schemas.py ← Pydantic request/response models
104
+ │ └── services/ ← Business logic layer
105
+ │ ├── swagger_parser.py ← Parses OpenAPI/Swagger specs
106
+ │ ├── postman_parser.py ← Parses Postman Collections
107
+ │ ├── llm_planner.py ← Mistral AI: generates execution plan
108
+ │ ├── api_executor.py ← Mistral AI: formats HTTP request for each step
109
+ │ ├── response_analyzer.py ← Mistral AI: analyzes step response
110
+ │ ├── context_manager.py ← Session context variable storage
111
+ │ ├── asset_service.py ← Media asset CRUD + collection management
112
+ │ └── storage.py ← Abstraction: local or Supabase file storage
113
+ │
114
+ └── frontend-react/ ← React + Vite frontend
115
+ ├── index.html
116
+ ├── vite.config.js
117
+ ├── package.json
118
+ └── src/
119
+ ├── main.jsx ← React entry point
120
+ ├── App.jsx ← Main layout + execution workflow page
121
+ ├── App.css / index.css ← Global styles + CSS variables (theme)
122
+ ├── components/
123
+ │ ├── AppContext.jsx ← Global React Context (state management)
124
+ │ ├── CollectionsPage.jsx ← View/manage uploaded API specs
125
+ │ ├── ProfilePage.jsx ← Manage test-user credentials
126
+ │ ├── HistoryPage.jsx ← View all past test sessions
127
+ │ ├── MediaHubPage.jsx ← Upload and manage test assets
128
+ │ └── SettingsPage.jsx ← Application settings
129
+ └── services/
130
+ └── api.js ← All Axios API calls to the backend
131
+ ```
132
+
133
+ ---
134
+
135
+ ## 4. System Architecture
136
+
137
+ ```
138
+ ┌─────────────────────────────────────────────────────────────────────┐
139
+ │ BROWSER (React + Vite) │
140
+ │ │
141
+ │ ┌─────────┐ ┌──────────────┐ ┌──────────┐ ┌───────────────────┐ │
142
+ │ │Collections│ │ Execution │ │ Media │ │ History / Profile │ │
143
+ │ │ Page │ │ Workflow │ │ Hub │ │ Pages │ │
144
+ │ └────┬────┘ └──────┬───────┘ └────┬─────┘ └─────────┬─────────┘ │
145
+ │ │ │ │ │ │
146
+ │ └──────────────┴───────────────┴────────────────────┘ │
147
+ │ │ Axios HTTP │
148
+ └──────────────────────────────┼─────────────────────────────────────┘
149
+ ▼
150
+ ┌─────────────────────────────────────────────────────────────────────┐
151
+ │ FastAPI Backend (port 8001) │
152
+ │ │
153
+ │ POST /upload-spec ──► swagger_parser / postman_parser │
154
+ │ POST /plan ──► llm_planner ──► Mistral AI │
155
+ │ POST /execute-step ──► api_executor ──► Mistral AI ──► httpx ──► │
156
+ │ Target API │
157
+ │ POST /analyze-response ──► response_analyzer ──► Mistral AI │
158
+ │ GET/POST /assets ──► asset_service ──► storage.py │
159
+ │ GET /history ──► SQLAlchemy ORM │
160
+ │ │
161
+ └───────────────────────────┬─────────────────────────────────────────┘
162
+ │ SQLAlchemy
163
+ ┌──────────────────┴────────────────────┐
164
+ ▼ ▼
165
+ ┌────────────────┐ ┌────────────────────┐
166
+ │ PostgreSQL DB │ │ Supabase Storage │
167
+ │ (via Supabase)│ │ (spec files, │
168
+ │ │ │ asset uploads) │
169
+ │ - sessions │ └────────────────────┘
170
+ │ - spec_docs │
171
+ │ - history │
172
+ │ - assets │
173
+ │ - collections │
174
+ └────────────────┘
175
+ ```
176
+
177
+ ---
178
+
179
+ ## 5. Backend Deep-Dive
180
+
181
+ ### 5.1 Entry Point — `main.py`
182
+
183
+ `main.py` is the **single FastAPI application file** containing all route definitions. It does not use `APIRouter` — all routes are mounted directly on the `app` instance.
184
+
185
+ **Key responsibilities:**
186
+ - CORS middleware (currently allows all origins — suitable for dev/demo)
187
+ - Static file serving for the `/uploads` directory
188
+ - Validation error handler (returns 422 with body for debugging)
189
+ - Orchestrating all service calls
190
+
191
+ **Key logic in `/execute-step`:**
192
+ The most complex route. It performs:
193
+ 1. **Schema Resolution** — Looks up the matching OpenAPI path definition, inlines all `$ref` references recursively, prunes metadata (descriptions, examples), and converts the schema to a compact shorthand notation (`{prop*(str)}`) to minimize LLM tokens.
194
+ 2. **URL Sanitization** — Detects if the LLM returned a placeholder URL (like `api.example.com`) and rewrites it to use the real base URL.
195
+ 3. **Asset Loading** — Opens local files from the Media Hub for multipart uploads.
196
+ 4. **HTTP Execution** — Uses `httpx.AsyncClient` to send the real request.
197
+ 5. **History Persistence** — Saves request + response to the database.
198
+
199
+ ---
200
+
201
+ ### 5.2 Database Layer — `database.py`
202
+
203
+ Uses **SQLAlchemy 2.0** with a declarative base. Database URL is loaded from the `DATABASE_URL` environment variable (Supabase PostgreSQL).
204
+
205
+ #### Models
206
+
207
+ | Model | Table | Description |
208
+ |---|---|---|
209
+ | `SpecDocument` | `spec_documents` | Stores uploaded API spec (raw text, parsed JSON, base URL, credentials, execution plan) |
210
+ | `SessionModel` | `sessions` | Represents one test run. Stores which spec, scenario, base URL, and extracted context variables |
211
+ | `ExecutionHistory` | `execution_history` | One row per step executed. Stores request JSON, response JSON, and AI analysis |
212
+ | `Asset` | `assets` | Uploaded files (audio/video/docs). Tagged, categorized, linked to a spec |
213
+ | `Collection` | `collections` | Named groups of assets. Many-to-many with Asset |
214
+
215
+ #### Relationships
216
+
217
+ ```
218
+ SpecDocument ──< Asset (one-to-many)
219
+ Asset >──< Collection (many-to-many via asset_collections junction table)
220
+ SessionModel ──> SpecDocument (foreign key, nullable)
221
+ ExecutionHistory ──> SessionModel (foreign key via session_id)
222
+ ```
223
+
224
+ #### Connection Details
225
+
226
+ ```python
227
+ engine = create_engine(
228
+ DATABASE_URL,
229
+ pool_pre_ping=True, # Checks connection health before use
230
+ pool_recycle=300 # Recycles connections every 5 minutes
231
+ )
232
+ ```
233
+
234
+ ---
235
+
236
+ ### 5.3 Pydantic Schemas — `schemas.py`
237
+
238
+ All request/response validation is done via Pydantic v2 models:
239
+
240
+ | Schema | Direction | Purpose |
241
+ |---|---|---|
242
+ | `PlanRequest` | → Backend | Payload for `/plan` — spec content, base URL, scenario, credentials |
243
+ | `ExecutionPlan` | ← Backend | Response from `/plan` — list of steps + suggested credentials |
244
+ | `StepDefinition` | — | Single step: step number, endpoint string, purpose text |
245
+ | `ExecuteStepRequest` | → Backend | Payload for `/execute-step` — session ID, step def, context vars |
246
+ | `AnalyzeRequest` | → Backend | Payload for `/analyze-response` — step number, response data |
247
+ | `AnalyzeResponse` | ← Backend | AI analysis result — status, extracted variables, OTP flag |
248
+ | `CredentialUpdate` | → Backend | Payload for credential save — username, password, extra fields |
249
+ | `TokenUsage` | ← Backend | LLM token counts (prompt, completion, total) |
250
+
251
+ ---
252
+
253
+ ### 5.4 Service Layer — `services/`
254
+
255
+ #### `swagger_parser.py`
256
+ - Accepts raw spec string (JSON or YAML)
257
+ - Parses with `json.loads` → falls back to `yaml.safe_load`
258
+ - **Does NOT resolve `$ref` at parse time** (deferred to execution to handle large specs like the GitHub API)
259
+ - Returns two views of the spec:
260
+ - `endpoints`: Simplified list `[{path, method, summary}]` — sent to the planner LLM
261
+ - `paths` + `components`: Full raw spec — stored in DB, used by executor for on-the-fly `$ref` resolution
262
+
263
+ #### `postman_parser.py`
264
+ - Detects Postman collections by checking for `info.schema` URL or `info + item` keys
265
+ - Converts Postman's nested item/folder structure to a flat OpenAPI-like format
266
+
267
+ #### `llm_planner.py`
268
+ - Uses `ChatMistralAI` (via LangChain) with `mistral-large-latest`, `temperature=0`
269
+ - Takes: simplified endpoint list, base URL, user scenario, existing credentials
270
+ - Returns: `ExecutionPlan` with a list of `StepDefinition` objects
271
+ - **Smart credential handling**: If credentials exist → skip registration steps. If empty → AI suggests realistic credentials.
272
+
273
+ #### `api_executor.py`
274
+ - Uses `ChatMistralAI` with a detailed prompt about the specific endpoint schema
275
+ - Returns `StepExecutionPlan` containing the complete HTTP request object (`method`, `url`, `headers`, `query_params`, `form_data`, `body`, `files`)
276
+ - **Token-efficient schema notation** (shorthand): Instead of sending verbose JSON schema, converts it to compact strings like `{name*(str),age(int),status(str[active|inactive])}`
277
+ - **Data source priority**: Context Variables → User Credentials → Step Definition
278
+
279
+ #### `response_analyzer.py`
280
+ - Uses `ChatMistralAI` to analyze HTTP response (status code + body)
281
+ - Extracts: step status (success/failure), dynamic variables (tokens, IDs), OTP requirement flag
282
+ - Response body is **truncated to 1KB** (500 chars from start + 500 from end) to save LLM tokens
283
+
284
+ #### `context_manager.py`
285
+ - Simple read/write for `SessionModel.context_variables` (JSON column)
286
+ - `get_context(db, session_id)` → returns current variables dict
287
+ - `set_context_variables(db, session_id, new_vars)` → merges new vars into session
288
+
289
+ #### `asset_service.py`
290
+ - `upload_asset()`: Saves file to storage + creates `Asset` record
291
+ - `get_assets()`: Filterable by `spec_id` and `category`
292
+ - `delete_asset()`: Removes from storage + DB
293
+ - `create_collection()`, `add_to_collection()`, `delete_collection()`: Collection management
294
+ - `get_ai_context()`: Returns a structured dict of assets grouped by collection — used to give the LLM awareness of available test files
295
+
296
+ #### `storage.py`
297
+ - Abstraction layer for file storage with two implementations:
298
+ - **`SupabaseStorageProvider`**: Uses Supabase Storage bucket for production
299
+ - **`LocalStorageProvider`**: Saves files to `backend/uploads/` for local dev
300
+ - Auto-detects which to use based on `SUPABASE_URL` env variable
301
+
302
+ ---
303
+
304
+ ## 6. Frontend Deep-Dive
305
+
306
+ The frontend is a **single-page application** built with React 19 + Vite. Navigation is icon-based (VS Code-like activity bar), with each icon switching the main content area without a full page reload.
307
+
308
+ ### State Management — `AppContext.jsx`
309
+
310
+ A single React Context (`AppProvider`) manages all global state:
311
+
312
+ | State | Description |
313
+ |---|---|
314
+ | `collections` | List of uploaded specs fetched from backend |
315
+ | `activeSpecId` | Currently selected spec UUID |
316
+ | `activeSpecName` | Display name of active spec |
317
+ | `activeBaseUrl` | The base URL being tested |
318
+ | `parsedSpec` | Full parsed spec JSON for active spec |
319
+ | `storedCredentials` | Auth credentials for the active test user |
320
+ | `sessionId` | Current test session UUID |
321
+ | `sessionContextVars` | Dynamic variables extracted from previous steps |
322
+ | `executionPlan` | The list of steps from the LLM planner |
323
+ | `scenario` | The current natural-language scenario text |
324
+
325
+ ### Pages / Components
326
+
327
+ | Component | Route (in `activePage` state) | Purpose |
328
+ |---|---|---|
329
+ | `CollectionsPage` | `collections` | Upload new specs, list and select specs, delete specs |
330
+ | Testing Workflow | `testing` (default) | Main page — scenario input, execution plan display, step execution |
331
+ | `ProfilePage` | `profile` | Set/update test user credentials (username, password, extra fields) |
332
+ | `HistoryPage` | `history` | Browse all past sessions and step results |
333
+ | `MediaHubPage` | `media` | Upload and organize test assets (files for multipart APIs) |
334
+ | `SettingsPage` | `settings` | App configuration |
335
+
336
+ ### API Service Layer — `services/api.js`
337
+
338
+ All backend calls are centralized in a single file, organized into service classes:
339
+
340
+ | Service | Methods |
341
+ |---|---|
342
+ | `SpecService` | `upload()`, `list()`, `get()`, `updateBaseUrl()`, `delete()` |
343
+ | `SessionService` | `create()`, `delete()` |
344
+ | `PlanService` | `generatePlan()`, `executeStep()`, `analyzeResponse()` |
345
+ | `CredentialService` | `get()`, `save()` |
346
+ | `AssetService` | `upload()`, `list()`, `delete()`, `createCollection()`, etc. |
347
+
348
+ ---
349
+
350
+ ## 7. LLM Pipeline
351
+
352
+ The system uses **three distinct LLM calls** per test step:
353
+
354
+ ```
355
+ User Scenario
356
+ │
357
+ ▼ [LLM Call 1]
358
+ ┌─────────────────────┐
359
+ │ llm_planner.py │ Prompt: Full endpoint list + scenario
360
+ │ "Planner" │ Output: Ordered list of steps [{step, endpoint, purpose}]
361
+ └──────────┬──────────┘
362
+ │
363
+ │ For each step:
364
+ ▼ [LLM Call 2]
365
+ ┌─────────────────────┐
366
+ │ api_executor.py │ Prompt: Specific endpoint schema (shorthand) +
367
+ │ "Executor" │ context vars + credentials + assets
368
+ │ │ Output: Complete HTTP request object
369
+ └──────────┬──────────┘
370
+ │
371
+ │ (httpx sends the actual HTTP request)
372
+ │
373
+ ▼ [LLM Call 3]
374
+ ┌─────────��───────────┐
375
+ │ response_analyzer │ Prompt: Status code + response body
376
+ │ "Analyzer" │ Output: {status, extracted_variables, is_otp_required}
377
+ └─────────────────────┘
378
+ │
379
+ ▼
380
+ Context Variables updated → injected into next step's Executor call
381
+ ```
382
+
383
+ ### Token Optimization Strategies
384
+
385
+ - **Shorthand schema notation**: `{name*(str),age(int)}` instead of full JSON Schema
386
+ - **Response truncation**: Max 1KB sent to analyzer
387
+ - **Pruning**: Removes `description`, `summary`, `example`, `format` keys from specs
388
+ - **Lazy `$ref` resolution**: Only the matching endpoint is resolved, not the whole spec
389
+ - **Response pruning**: `responses` object from spec is stripped before sending to LLM (only needed for building the request, not analyzing it)
390
+
391
+ ---
392
+
393
+ ## 8. Data Flow — End-to-End
394
+
395
+ ### Step 1: Upload a Spec
396
+
397
+ ```
398
+ User selects file
399
+ → POST /upload-spec (multipart)
400
+ → swagger_parser.py or postman_parser.py
401
+ → Save raw + parsed JSON to spec_documents table
402
+ → Save raw file to Supabase/local storage
403
+ → Return spec_id + parsed_spec to frontend
404
+ ```
405
+
406
+ ### Step 2: Generate Execution Plan
407
+
408
+ ```
409
+ User types scenario → clicks "Generate Plan"
410
+ → startNewSession() → POST /session → returns session_id
411
+ → POST /plan {spec_content, base_url, scenario, user_credentials, spec_id, session_id}
412
+ → llm_planner.generate_execution_plan()
413
+ → Save plan + scenario to spec_documents.execution_plan
414
+ → AI-suggested credentials merged into spec_documents.login_credentials
415
+ → Return [{step, endpoint, purpose}, ...] to frontend
416
+ ```
417
+
418
+ ### Step 3: Execute a Step
419
+
420
+ ```
421
+ User clicks "Execute" on a step
422
+ → refreshSessionContext() → GET /context/{session_id}
423
+ → POST /execute-step {session_id, step_definition, context_variables, credentials, base_url, spec_id}
424
+ → Find matching path in spec, resolve $refs, prune, convert to shorthand
425
+ → api_executor.prepare_step_request() → LLM returns complete HTTP request
426
+ → URL sanitization (fix placeholders, relative paths)
427
+ → httpx sends real HTTP request to target API
428
+ → Save to execution_history table
429
+ → Return {executed_request, response, token_usage}
430
+ ```
431
+
432
+ ### Step 4: Analyze Response
433
+
434
+ ```
435
+ (Automatically called after execute)
436
+ → POST /analyze-response {session_id, step, response_data, step_definition}
437
+ → response_analyzer.analyze_api_response() → LLM returns analysis
438
+ → Extracted variables saved to sessions.context_variables via context_manager
439
+ → History record updated with analysis
440
+ → Return {status, response_summary, extracted_variables, is_otp_required}
441
+ ```
442
+
443
+ ---
444
+
445
+ ## 9. API Reference (All Endpoints)
446
+
447
+ ### Core
448
+
449
+ | Method | Path | Description |
450
+ |---|---|---|
451
+ | `GET` | `/` | Health check |
452
+ | `POST` | `/session` | Create a new test session |
453
+ | `DELETE` | `/session/{session_id}` | Clear session and its history |
454
+
455
+ ### Spec Management
456
+
457
+ | Method | Path | Description |
458
+ |---|---|---|
459
+ | `POST` | `/upload-spec` | Upload OpenAPI/Postman spec file |
460
+ | `GET` | `/specs` | List all uploaded specs |
461
+ | `GET` | `/specs/{spec_id}` | Get a specific spec with plan + credentials |
462
+ | `PATCH` | `/specs/{spec_id}` | Update spec's base_url |
463
+ | `DELETE` | `/specs/{spec_id}` | Delete spec and all its assets |
464
+ | `GET` | `/specs/{spec_id}/credentials` | Get stored test credentials for spec |
465
+ | `POST` | `/specs/{spec_id}/credentials` | Save/update test credentials |
466
+
467
+ ### Execution
468
+
469
+ | Method | Path | Description |
470
+ |---|---|---|
471
+ | `POST` | `/plan` | Generate LLM execution plan from scenario |
472
+ | `POST` | `/execute-step` | Execute one step (calls LLM + httpx) |
473
+ | `POST` | `/analyze-response` | Analyze step response and extract variables |
474
+ | `GET` | `/context/{session_id}` | Get current session context variables |
475
+
476
+ ### History
477
+
478
+ | Method | Path | Description |
479
+ |---|---|---|
480
+ | `GET` | `/history` | Get all execution history (all sessions) |
481
+ | `DELETE` | `/history/{session_id}` | Delete one session's history |
482
+ | `DELETE` | `/history` | Wipe all history |
483
+
484
+ ### Assets & Collections
485
+
486
+ | Method | Path | Description |
487
+ |---|---|---|
488
+ | `POST` | `/assets` | Upload a test asset file |
489
+ | `GET` | `/assets` | List assets (filter by spec_id, category) |
490
+ | `DELETE` | `/assets/{asset_id}` | Delete an asset |
491
+ | `GET` | `/collections` | List collections |
492
+ | `POST` | `/collections` | Create a new collection |
493
+ | `POST` | `/collections/{collection_id}/assets/{asset_id}` | Add asset to collection |
494
+ | `DELETE` | `/collections/{collection_id}` | Delete a collection |
495
+
496
+ ---
497
+
498
+ ## 10. Database Schema
499
+
500
+ ```sql
501
+ -- API spec documents
502
+ CREATE TABLE spec_documents (
503
+ id VARCHAR PRIMARY KEY, -- UUID
504
+ name VARCHAR NOT NULL, -- filename
505
+ raw_content TEXT NOT NULL, -- original spec text
506
+ parsed_json JSON NOT NULL, -- parsed + simplified spec
507
+ base_url VARCHAR, -- target server URL
508
+ login_credentials JSON, -- {username, password, token, ...}
509
+ execution_plan JSON, -- last generated plan steps
510
+ scenario TEXT, -- last scenario text
511
+ file_path VARCHAR, -- path in storage
512
+ uploaded_at TIMESTAMP DEFAULT NOW()
513
+ );
514
+
515
+ -- Test sessions
516
+ CREATE TABLE sessions (
517
+ id VARCHAR PRIMARY KEY, -- UUID
518
+ spec_id VARCHAR REFERENCES spec_documents(id),
519
+ spec_name VARCHAR,
520
+ base_url VARCHAR,
521
+ scenario TEXT,
522
+ context_variables JSON DEFAULT '{}', -- live extracted vars
523
+ created_at TIMESTAMP DEFAULT NOW()
524
+ );
525
+
526
+ -- Step execution records
527
+ CREATE TABLE execution_history (
528
+ id SERIAL PRIMARY KEY,
529
+ session_id VARCHAR, -- link to sessions
530
+ step INTEGER,
531
+ request_json JSON, -- full HTTP request object
532
+ response_json JSON, -- status_code, body, headers
533
+ analysis_json JSON, -- AI analysis result
534
+ step_definition_json JSON, -- step metadata snapshot
535
+ timestamp TIMESTAMP DEFAULT NOW()
536
+ );
537
+
538
+ -- Test assets (files)
539
+ CREATE TABLE assets (
540
+ id VARCHAR PRIMARY KEY, -- UUID
541
+ name VARCHAR NOT NULL,
542
+ file_type VARCHAR NOT NULL, -- MIME type
543
+ file_path VARCHAR NOT NULL, -- storage path
544
+ file_size INTEGER,
545
+ category VARCHAR DEFAULT 'documents', -- audio/video/documents/images
546
+ description TEXT,
547
+ tags JSON DEFAULT '[]',
548
+ metadata_json JSON DEFAULT '{}',
549
+ spec_id VARCHAR REFERENCES spec_documents(id),
550
+ uploaded_at TIMESTAMP DEFAULT NOW()
551
+ );
552
+
553
+ -- Asset groupings
554
+ CREATE TABLE collections (
555
+ id VARCHAR PRIMARY KEY,
556
+ name VARCHAR NOT NULL,
557
+ description TEXT,
558
+ spec_id VARCHAR REFERENCES spec_documents(id),
559
+ created_at TIMESTAMP DEFAULT NOW()
560
+ );
561
+
562
+ -- Junction for assets ↔ collections
563
+ CREATE TABLE asset_collections (
564
+ asset_id VARCHAR REFERENCES assets(id),
565
+ collection_id VARCHAR REFERENCES collections(id),
566
+ PRIMARY KEY (asset_id, collection_id)
567
+ );
568
+ ```
569
+
570
+ ---
571
+
572
+ ## 11. Key Design Decisions
573
+
574
+ ### 1. Single `main.py` vs. APIRouter
575
+ All routes are in one file. This keeps it simple for a project of this scope but would need refactoring for a larger team.
576
+
577
+ ### 2. Deferred `$ref` Resolution
578
+ OpenAPI specs (like the GitHub API with 1000+ endpoints) are too large to fully resolve at upload time. Instead, `$ref`s are resolved **on-demand** only for the specific endpoint being executed. This keeps parsing fast and storage lean.
579
+
580
+ ### 3. Shorthand Schema Notation
581
+ Before sending an endpoint schema to the LLM, it's converted to a compact shorthand format. `{name*(str), age(int:int32), status(str[active|inactive])}` is far smaller than the equivalent JSON Schema, saving significant tokens at scale.
582
+
583
+ ### 4. Session-Scoped Context vs. Spec-Scoped Credentials
584
+ - **Credentials** (username, password, tokens) are **persisted to the spec** in the DB, so they survive page refreshes.
585
+ - **Dynamic variables** (IDs, tokens extracted during a run) are stored in the **session** only. They reset when you start a new session/plan.
586
+
587
+ ### 5. Storage Abstraction
588
+ `storage.py` provides a unified interface (`save`, `delete`, `get_full_path`) with two implementations:
589
+ - `SupabaseStorageProvider`: used when `SUPABASE_URL` is set.
590
+ - `LocalStorageProvider`: fallback for offline/local development.
591
+
592
+ ### 6. AI Credential Merging
593
+ When the Planner LLM suggests test credentials (e.g., for a new user registration step), they are automatically merged into the spec's stored credentials in the database. This prevents the AI from creating duplicate users on subsequent test runs.
594
+
595
+ ### 7. OTP/2FA Handling
596
+ The Analyzer LLM detects when a response requires an OTP. The UI shows a modal prompting the user to enter the code, which is then saved as a context variable and used in the next step automatically.
597
+
598
+ ---
599
+
600
+ ## 12. Environment & Configuration
601
+
602
+ ### `.env` Variables
603
+
604
+ | Variable | Required | Description |
605
+ |---|---|---|
606
+ | `DATABASE_URL` | ✅ | PostgreSQL connection string (e.g., `postgresql://user:pass@host/db`) |
607
+ | `MISTRAL_API_KEY` | ✅ | API key for Mistral AI (used by LangChain) |
608
+ | `SUPABASE_URL` | ⬜ | Supabase project URL (enables cloud storage; falls back to local if missing) |
609
+ | `SUPABASE_KEY` | ⬜ | Supabase service role key |
610
+ | `SUPABASE_BUCKET` | ⬜ | Name of the Supabase Storage bucket |
611
+
612
+ > **Note:** The `DATABASE_URL` string starting with `postgres://` is auto-converted to `postgresql://` for SQLAlchemy compatibility.
613
+
614
+ ---
615
+
616
+ ## 13. Running the Project
617
+
618
+ ### Prerequisites
619
+ - Python 3.9+
620
+ - Node.js 18+
621
+ - A Mistral AI API key
622
+ - A PostgreSQL database (local or Supabase)
623
+
624
+ ### First-Time Setup
625
+
626
+ ```bash
627
+ # 1. Install all dependencies (Python + Node)
628
+ npm run setup
629
+
630
+ # 2. Copy and fill in the environment file
631
+ cp .env.example .env
632
+ # Edit .env and add your MISTRAL_API_KEY and DATABASE_URL
633
+ ```
634
+
635
+ ### Development
636
+
637
+ ```bash
638
+ # Option A: Both frontend and backend in one terminal (recommended)
639
+ npm run dev
640
+
641
+ # Option B: PowerShell, opens separate windows
642
+ .\start.ps1
643
+
644
+ # Option C: Manual
645
+ python -m uvicorn backend.main:app --reload --port 8001
646
+ cd frontend-react && npm run dev
647
+ ```
648
+
649
+ ### Service URLs
650
+
651
+ | Service | URL |
652
+ |---|---|
653
+ | Frontend (React) | http://localhost:5173 |
654
+ | Backend API | http://localhost:8001 |
655
+ | Swagger API Docs | http://localhost:8001/docs |
656
+ | ReDoc API Docs | http://localhost:8001/redoc |
657
+
658
+ ---
659
+
660
+ ## 14. Comparison Cheatsheet
661
+
662
+ Use this table when explaining or comparing this project to similar tools:
663
+
664
+ | Feature | This Project | Postman | Rest-assured | Swagger UI | Insomnia |
665
+ |---|---|---|---|---|---|
666
+ | **Input Format** | OpenAPI / Postman | Postman | Code | OpenAPI | REST/GraphQL |
667
+ | **Test Creation** | Natural language | Manual | Java/Groovy code | Manual | Manual |
668
+ | **Code Required** | ❌ Zero | ✅ Script | ✅ Full | ❌ No | ❌ No |
669
+ | **AI Test Planning** | ✅ Mistral LLM | ❌ | ❌ | ❌ | ❌ |
670
+ | **Auto Context Chaining** | ✅ AI-driven | ⬜ Manual env vars | ⬜ Manual | ❌ | ⬜ Manual |
671
+ | **Response Analysis** | ✅ AI-driven | ✅ Manual assertions | ✅ Assertions | ❌ | ✅ Manual |
672
+ | **Persistent History** | ✅ DB-backed | ✅ | ❌ | ❌ | ✅ |
673
+ | **File Upload Testing** | ✅ Media Hub | ✅ | ✅ | ❌ | ✅ |
674
+ | **Multi-step Automation** | ✅ AI plans full flow | ⬜ Collection runner | ✅ Test suites | ❌ | ❌ |
675
+ | **OTP/2FA Support** | ✅ UI prompt | ❌ | ❌ | ❌ | ❌ |
676
+ | **Self-hosted** | ✅ | Partially | ✅ | ✅ | Partially |
677
+
678
+ ---
679
+
680
+ *Documentation generated: April 2026*
681
+ *Project: AI API Testing Assistant | Stack: FastAPI + React + Mistral AI + PostgreSQL*
Project_Diagrams.md ADDED
@@ -0,0 +1,549 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # AI API Testing Assistant — Project Diagrams Guide
2
+
3
+ This document contains ready-to-use **Mermaid diagram code** for all major diagram types relevant to this project. Paste any code block into a Mermaid-compatible renderer (VS Code Preview, draw.io, Mermaid Live Editor at https://mermaid.live, Notion, or GitHub Markdown).
4
+
5
+ ---
6
+
7
+ ## 1. System Architecture Diagram
8
+ *Shows the high-level structure of the entire system and how the four tiers interact.*
9
+
10
+ ```mermaid
11
+ graph TB
12
+ subgraph "Presentation Tier"
13
+ FE["React + Vite Frontend<br/>(SPA on port 5173)"]
14
+ M1["API Collections Module"]
15
+ M2["Execution Hub"]
16
+ M3["Test User Profile"]
17
+ M4["Execution History"]
18
+ M5["Media Hub"]
19
+ end
20
+
21
+ subgraph "Orchestration Tier"
22
+ API["FastAPI Backend<br/>(Uvicorn on port 8001)"]
23
+ CORS["CORS Middleware"]
24
+ VALID["Request Validation<br/>(Pydantic)"]
25
+ end
26
+
27
+ subgraph "Intelligence Tier"
28
+ LLM1["LLM Planner<br/>(llm_planner.py)"]
29
+ LLM2["API Executor<br/>(api_executor.py)"]
30
+ LLM3["Response Analyzer<br/>(response_analyzer.py)"]
31
+ MISTRAL["Mistral-Large-Latest<br/>via LangChain"]
32
+ end
33
+
34
+ subgraph "Data Tier"
35
+ PG["PostgreSQL<br/>(via Supabase)"]
36
+ SUP["Supabase File Storage<br/>(Specs + Assets)"]
37
+ end
38
+
39
+ subgraph "External"
40
+ TARGET["Target API Server<br/>(Under Test)"]
41
+ end
42
+
43
+ FE --> CORS --> VALID --> API
44
+ API --> LLM1
45
+ API --> LLM2
46
+ API --> LLM3
47
+ LLM1 & LLM2 & LLM3 --> MISTRAL
48
+ API --> PG
49
+ API --> SUP
50
+ LLM2 --> TARGET
51
+ ```
52
+
53
+ ---
54
+
55
+ ## 2. Sequence Diagram — Full Test Session Flow
56
+ *Shows the step-by-step interaction between the user, backend, LLMs, and database during a complete test run.*
57
+
58
+ ```mermaid
59
+ sequenceDiagram
60
+ actor User
61
+ participant FE as React Frontend
62
+ participant BE as FastAPI Backend
63
+ participant DB as PostgreSQL
64
+ participant LLM as Mistral AI
65
+ participant API as Target API
66
+
67
+ User->>FE: Upload API spec file
68
+ FE->>BE: POST /upload-spec (file)
69
+ BE->>BE: Detect format (OpenAPI/Postman)
70
+ BE->>BE: parse_spec() → simplified endpoints
71
+ BE->>DB: INSERT SpecDocument (raw + parsed)
72
+ BE-->>FE: {spec_id, endpoints}
73
+
74
+ User->>FE: Type natural language scenario
75
+ FE->>BE: POST /plan {scenario, spec_id}
76
+ BE->>DB: GET login_credentials for spec
77
+ BE->>LLM: generate_execution_plan(endpoints, scenario, creds)
78
+ LLM-->>BE: {execution_plan[], suggested_credentials}
79
+ BE->>DB: UPDATE SpecDocument (plan, merged creds)
80
+ BE-->>FE: {execution_plan[]}
81
+
82
+ loop For each step in plan
83
+ User->>FE: Click "Execute Step N"
84
+ FE->>BE: POST /execute-step {step_def, session_id}
85
+ BE->>DB: GET full spec schema
86
+ BE->>BE: resolve_refs() + prune + shorthand
87
+ BE->>DB: GET context_variables for session
88
+ BE->>LLM: prepare_step_request(schema, context, creds)
89
+ LLM-->>BE: {StepExecutionPlan}
90
+ BE->>BE: sanitize_url()
91
+ BE->>API: HTTP request (method, url, headers, body)
92
+ API-->>BE: {status_code, body}
93
+ BE->>DB: INSERT ExecutionHistory (request + response)
94
+ BE-->>FE: {request, response}
95
+
96
+ FE->>BE: POST /analyze-response {step, response}
97
+ BE->>LLM: analyze_api_response(status, body, purpose)
98
+ LLM-->>BE: {status, extracted_variables, is_otp_required}
99
+ BE->>DB: UPDATE context_variables (merge new vars)
100
+ BE->>DB: UPDATE ExecutionHistory (analysis)
101
+ BE-->>FE: {analysis_result}
102
+ FE-->>User: Show result + extracted variables
103
+ end
104
+ ```
105
+
106
+ ---
107
+
108
+ ## 3. Entity-Relationship (ER) Diagram
109
+ *Defines the database schema and relationships between all entities.*
110
+
111
+ ```mermaid
112
+ erDiagram
113
+ SPEC_DOCUMENT {
114
+ string id PK
115
+ string name
116
+ text raw_content
117
+ json parsed_json
118
+ string base_url
119
+ json login_credentials
120
+ json execution_plan
121
+ text scenario
122
+ string file_path
123
+ datetime uploaded_at
124
+ }
125
+
126
+ SESSION_MODEL {
127
+ string id PK
128
+ string spec_id FK
129
+ string spec_name
130
+ string base_url
131
+ text scenario
132
+ json context_variables
133
+ datetime created_at
134
+ }
135
+
136
+ EXECUTION_HISTORY {
137
+ int id PK
138
+ string session_id FK
139
+ int step
140
+ json request_json
141
+ json response_json
142
+ json analysis_json
143
+ json step_definition_json
144
+ datetime timestamp
145
+ }
146
+
147
+ ASSET {
148
+ string id PK
149
+ string name
150
+ string file_type
151
+ string file_path
152
+ int file_size
153
+ string category
154
+ text description
155
+ json tags
156
+ json metadata_json
157
+ string spec_id FK
158
+ datetime uploaded_at
159
+ }
160
+
161
+ COLLECTION {
162
+ string id PK
163
+ string name
164
+ text description
165
+ string spec_id FK
166
+ datetime created_at
167
+ }
168
+
169
+ ASSET_COLLECTIONS {
170
+ string asset_id FK
171
+ string collection_id FK
172
+ }
173
+
174
+ SPEC_DOCUMENT ||--o{ SESSION_MODEL : "has"
175
+ SESSION_MODEL ||--o{ EXECUTION_HISTORY : "contains"
176
+ SPEC_DOCUMENT ||--o{ ASSET : "owns"
177
+ SPEC_DOCUMENT ||--o{ COLLECTION : "groups"
178
+ ASSET }|--|{ COLLECTION : "ASSET_COLLECTIONS"
179
+ ```
180
+
181
+ ---
182
+
183
+ ## 4. Data Flow Diagram (DFD) — Core Pipeline
184
+ *Shows how data transforms and moves through the Triple-Stage AI Pipeline.*
185
+
186
+ ```mermaid
187
+ flowchart LR
188
+ A([User Input:\nNatural Language\nScenario]) --> B
189
+
190
+ subgraph B["Stage 1: LLM Planner"]
191
+ B1["Simplified\nEndpoint List"] --> B2["Planner\nPrompt"]
192
+ B3["User\nCredentials"] --> B2
193
+ B2 --> B4["Mistral AI\nReasoning"]
194
+ B4 --> B5["Execution\nPlan JSON"]
195
+ end
196
+
197
+ B5 --> C
198
+
199
+ subgraph C["Stage 2: API Executor (per step)"]
200
+ C1["Full Spec\n(from DB)"] --> C2["resolve_refs()"]
201
+ C2 --> C3["prune_spec()"]
202
+ C3 --> C4["shorthand_spec()"]
203
+ C4 --> C5["Executor\nPrompt"]
204
+ C6["Session\nContext Vars"] --> C5
205
+ C7["User Creds"] --> C5
206
+ C5 --> C8["Mistral AI\nReasoning"]
207
+ C8 --> C9["HTTP\nRequest Object"]
208
+ C9 --> C10{{"Target\nAPI Server"}}
209
+ C10 --> C11["Raw\nResponse"]
210
+ end
211
+
212
+ C11 --> D
213
+
214
+ subgraph D["Stage 3: Response Analyzer"]
215
+ D1["Truncated\nResponse Body"] --> D2["Analyzer\nPrompt"]
216
+ D3["Step\nPurpose"] --> D2
217
+ D2 --> D4["Mistral AI\nReasoning"]
218
+ D4 --> D5["Semantic\nStatus"]
219
+ D4 --> D6["Extracted\nVariables"]
220
+ D4 --> D7["OTP\nFlag"]
221
+ end
222
+
223
+ D6 -->|"Stored back to\nSession Context"| C6
224
+ D5 --> E([User sees:\nSuccess/Failure\nExplanation])
225
+ ```
226
+
227
+ ---
228
+
229
+ ## 5. Use Case Diagram
230
+ *Illustrates what different types of users can do with the system.*
231
+
232
+ ```mermaid
233
+ graph TB
234
+ subgraph Actors
235
+ Dev["👨‍💻 Developer / QA Engineer"]
236
+ NonTech["🧑‍💼 Non-Technical Tester"]
237
+ end
238
+
239
+ subgraph "AI API Testing Assistant"
240
+ UC1["Upload API Spec\n(OpenAPI / Postman)"]
241
+ UC2["Describe Test Scenario\n(Natural Language)"]
242
+ UC3["Generate Execution Plan"]
243
+ UC4["Execute Individual Steps"]
244
+ UC5["View AI Response Analysis"]
245
+ UC6["Manage Test User Profile"]
246
+ UC7["Upload Media Assets\n(Files for API Testing)"]
247
+ UC8["View Execution History"]
248
+ UC9["Restore Past Session"]
249
+ UC10["Monitor Token Usage"]
250
+ end
251
+
252
+ Dev --> UC1
253
+ Dev --> UC2
254
+ Dev --> UC3
255
+ Dev --> UC4
256
+ Dev --> UC5
257
+ Dev --> UC6
258
+ Dev --> UC7
259
+ Dev --> UC8
260
+ Dev --> UC9
261
+ Dev --> UC10
262
+
263
+ NonTech --> UC1
264
+ NonTech --> UC2
265
+ NonTech --> UC3
266
+ NonTech --> UC4
267
+ NonTech --> UC5
268
+ NonTech --> UC8
269
+ ```
270
+
271
+ ---
272
+
273
+ ## 6. State Diagram — Test Session Lifecycle
274
+ *Shows the states a test session moves through from creation to completion.*
275
+
276
+ ```mermaid
277
+ stateDiagram-v2
278
+ [*] --> Idle : App Loaded
279
+
280
+ Idle --> SpecSelected : User selects/uploads a spec
281
+ SpecSelected --> ProfileConfigured : User sets credentials (optional)
282
+ ProfileConfigured --> ScenarioWritten : User types test scenario
283
+
284
+ ScenarioWritten --> PlanGenerating : User clicks "Generate Plan"
285
+ PlanGenerating --> PlanGenerating : LLM reasoning...
286
+ PlanGenerating --> PlanReady : Plan received
287
+
288
+ PlanReady --> StepExecuting : User clicks "Execute Step"
289
+ StepExecuting --> StepExecuting : HTTP request in flight...
290
+ StepExecuting --> AnalysisResult : Response received
291
+
292
+ AnalysisResult --> OTPRequired : is_otp_required = true
293
+ OTPRequired --> StepExecuting : User enters OTP code
294
+
295
+ AnalysisResult --> PlanReady : More steps remain
296
+ AnalysisResult --> SessionComplete : Last step done
297
+
298
+ SessionComplete --> [*] : Session saved to History
299
+ SessionComplete --> PlanReady : User runs again (plan restored)
300
+
301
+ PlanGenerating --> PlanFailed : LLM error
302
+ StepExecuting --> StepFailed : Network / API error
303
+ PlanFailed --> ScenarioWritten : User retries
304
+ StepFailed --> PlanReady : User retries step
305
+ ```
306
+
307
+ ---
308
+
309
+ ## 7. Class Diagram — Backend Service Architecture
310
+ *Defines the Pydantic models and service classes in the Python backend.*
311
+
312
+ ```mermaid
313
+ classDiagram
314
+ class SpecDocument {
315
+ +String id
316
+ +String name
317
+ +Text raw_content
318
+ +JSON parsed_json
319
+ +String base_url
320
+ +JSON login_credentials
321
+ +JSON execution_plan
322
+ +String file_path
323
+ }
324
+
325
+ class SessionModel {
326
+ +String id
327
+ +String spec_id
328
+ +JSON context_variables
329
+ +get_context() dict
330
+ +set_context_variables(vars) dict
331
+ }
332
+
333
+ class ExecutionHistory {
334
+ +Integer id
335
+ +String session_id
336
+ +Integer step
337
+ +JSON request_json
338
+ +JSON response_json
339
+ +JSON analysis_json
340
+ }
341
+
342
+ class ExecutionPlan {
343
+ +List execution_plan
344
+ +Dict suggested_credentials
345
+ +Dict token_usage
346
+ }
347
+
348
+ class StepExecutionPlan {
349
+ +Integer step
350
+ +APIRequest request
351
+ +StepAnalysis analysis
352
+ +Dict token_usage
353
+ }
354
+
355
+ class APIRequest {
356
+ +String method
357
+ +String url
358
+ +Dict headers
359
+ +Dict query_params
360
+ +Dict form_data
361
+ +Dict body
362
+ +Dict files
363
+ }
364
+
365
+ class AnalyzeResponse {
366
+ +Integer step
367
+ +String status
368
+ +Bool is_otp_required
369
+ +String otp_field_name
370
+ +Dict extracted_variables
371
+ +Dict token_usage
372
+ }
373
+
374
+ class LLMPlanner {
375
+ +generate_execution_plan(spec, url, scenario, creds) ExecutionPlan
376
+ }
377
+
378
+ class APIExecutor {
379
+ +prepare_step_request(spec, url, scenario, step, context, creds, assets) StepExecutionPlan
380
+ }
381
+
382
+ class ResponseAnalyzer {
383
+ +analyze_api_response(step, endpoint, purpose, status_code, body) AnalyzeResponse
384
+ }
385
+
386
+ class StorageProvider {
387
+ <<abstract>>
388
+ +save(file, path)
389
+ +delete(path)
390
+ +get_full_path(path) str
391
+ }
392
+
393
+ class LocalStorageProvider {
394
+ +save(file, path)
395
+ +delete(path)
396
+ +get_full_path(path) str
397
+ }
398
+
399
+ class SupabaseStorageProvider {
400
+ +save(file, path)
401
+ +delete(path)
402
+ +get_full_path(path) str
403
+ }
404
+
405
+ StepExecutionPlan *-- APIRequest
406
+ LLMPlanner ..> ExecutionPlan : creates
407
+ APIExecutor ..> StepExecutionPlan : creates
408
+ ResponseAnalyzer ..> AnalyzeResponse : creates
409
+ StorageProvider <|-- LocalStorageProvider
410
+ StorageProvider <|-- SupabaseStorageProvider
411
+ SessionModel "1" --> "many" ExecutionHistory
412
+ SpecDocument "1" --> "many" SessionModel
413
+ ```
414
+
415
+ ---
416
+
417
+ ## 8. Component Diagram — Frontend Module Structure
418
+ *Shows the React components and their relationships.*
419
+
420
+ ```mermaid
421
+ graph TB
422
+ App["App.jsx\n(Root + Router)"] --> Sidebar["ActivityBar\n(Navigation)"]
423
+ App --> MainContent["MainContent\n(Dynamic Panel)"]
424
+ App --> RightSidebar["Context Panel\n(Session Variables)"]
425
+
426
+ Sidebar --> Nav1["Collections Icon"]
427
+ Sidebar --> Nav2["Testing Icon"]
428
+ Sidebar --> Nav3["Profile Icon"]
429
+ Sidebar --> Nav4["History Icon"]
430
+ Sidebar --> Nav5["Media Hub Icon"]
431
+ Sidebar --> Nav6["Settings Icon"]
432
+
433
+ MainContent --> CollectionsPage["CollectionsPage\n(Spec Cards + Upload)"]
434
+ MainContent --> TestingPage["TestingPage\n(Main Workspace)"]
435
+ MainContent --> ProfilePage["ProfilePage\n(Credentials Form)"]
436
+ MainContent --> HistoryPage["HistoryPage\n(Audit Log)"]
437
+ MainContent --> MediaHubPage["MediaHubPage\n(File Manager)"]
438
+
439
+ TestingPage --> PlanInput["PlanInput\n(Scenario + Send)"]
440
+ TestingPage --> StepList["StepList\n(Plan Steps)"]
441
+ TestingPage --> TokenTracker["TokenUsageTracker"]
442
+
443
+ StepList --> StepCard["StepCard\n(Execute + Results)"]
444
+ StepCard --> RequestView["RequestInspector\n(Headers, Body)"]
445
+ StepCard --> ResponseView["ResponseInspector\n(Status, JSON)"]
446
+ StepCard --> AnalysisView["AnalysisView\n(AI Verdict)"]
447
+ StepCard --> OTPModal["OTPModal\n(Conditional)"]
448
+
449
+ RightSidebar --> ContextVars["SessionContext\n(Live Variables)"]
450
+ RightSidebar --> CredSummary["CredentialsSummary"]
451
+ ```
452
+
453
+ ---
454
+
455
+ ## 9. Deployment Diagram
456
+ *Shows how the system is deployed in both local development and cloud production.*
457
+
458
+ ```mermaid
459
+ graph TB
460
+ subgraph "Developer Machine (Local)"
461
+ direction TB
462
+ D1["npm run dev\n(concurrently)"]
463
+ D1 --> D2["Vite Dev Server\nlocalhost:5173"]
464
+ D1 --> D3["Uvicorn Server\nlocalhost:8001"]
465
+ D3 --> D4["backend/main.py\n(FastAPI App)"]
466
+ D4 --> D5["backend/uploads/\n(Local Storage)"]
467
+ end
468
+
469
+ subgraph "Cloud (Production)"
470
+ direction TB
471
+ P1["Heroku Dyno\n(Procfile: gunicorn)"]
472
+ P1 --> P2["FastAPI App\n(Gunicorn + Uvicorn Workers)"]
473
+ P2 --> P3["React Build\n(Static Files Served)"]
474
+ P2 --> P4["Supabase\n(PostgreSQL DB)"]
475
+ P2 --> P5["Supabase Storage\n(S3-compatible Bucket)"]
476
+ end
477
+
478
+ subgraph "External Services"
479
+ EX1["Mistral AI API\n(Cloud LLM)"]
480
+ EX2["Target API Server\n(Under Test)"]
481
+ end
482
+
483
+ D4 --> EX1
484
+ P2 --> EX1
485
+ D4 --> EX2
486
+ P2 --> EX2
487
+ ```
488
+
489
+ ---
490
+
491
+ ## 10. Algorithm Flowchart — Schema Resolution Pipeline
492
+ *Detailed flowchart of the 4-step on-demand schema resolution algorithm used before each API execution.*
493
+
494
+ ```mermaid
495
+ flowchart TD
496
+ A([Step Definition\ne.g. 'GET /users/usr_abc']) --> B
497
+
498
+ B["Split into:\nmethod = GET\npath = /users/usr_abc"]
499
+
500
+ B --> C{Direct lookup\npath in spec.paths?}
501
+ C -- Yes --> E["matched_path = path"]
502
+ C -- No --> D["Regex Fuzzy Match:\nReplace {param} → [^/]+\nfor each template T in paths"]
503
+ D --> D2{Template\nmatches path?}
504
+ D2 -- Yes --> E
505
+ D2 -- No --> D3["⚠ FALLBACK:\nUse raw endpoint string\n+ warning flag"]
506
+
507
+ E --> F["Fetch raw schema:\nspec.paths[matched_path]"]
508
+
509
+ F --> G["resolve_refs(schema, full_spec):\nTraverse JSON tree\nReplace all '$ref' pointers\nwith their target schemas\nCycle detection via resolving_set"]
510
+
511
+ G --> H["prune_spec_for_execution(schema):\nRemove keys:\ndescription, summary, example,\nexamples, format, title, default\nRemove entire 'responses' subtree"]
512
+
513
+ H --> I["shorthand_spec(schema):\nConvert JSON Schema to\ncompact shorthand notation:\n string → str\n {field*: type} → field*(str)\n enum → type[A|B|C]\n array → [itemType]"]
514
+
515
+ I --> J(["Compact Schema String\n→ Sent to Executor Prompt"])
516
+ ```
517
+
518
+ ---
519
+
520
+ ## 11. Flowchart — URL Sanitization (Anti-Hallucination Guard)
521
+
522
+ ```mermaid
523
+ flowchart TD
524
+ A([Raw URL from LLM]) --> B{Starts\nwith '/'?}
525
+ B -- Yes --> C["Relative URL\nurl = base_url + url"]
526
+ B -- No --> D{Is hostname in\nPLACEHOLDER_HOSTS?\ne.g. api.example.com}
527
+ D -- Yes --> E["Extract path + query\nfrom URL\nurl = base_url + path"]
528
+ D -- No --> F{Starts\nwith 'http'?}
529
+ F -- No --> G["Schemeless\nurl = 'https://' + url"]
530
+ F -- Yes --> H["Valid URL\nNo change needed"]
531
+
532
+ C & E & G & H --> Z([Final Sanitized URL\n→ Sent to httpx])
533
+ ```
534
+
535
+ ---
536
+
537
+ ## How to Use These Diagrams
538
+
539
+ | Tool | How to Render |
540
+ |---|---|
541
+ | **VS Code** | Install "Markdown Preview Mermaid Support" extension, open preview |
542
+ | **Mermaid Live** | Paste code at https://mermaid.live |
543
+ | **GitHub** | Paste directly into any `.md` file — renders automatically |
544
+ | **Notion** | Create a code block, select "Mermaid" as language |
545
+ | **draw.io** | Extras → Edit Diagram → Mermaid tab |
546
+ | **Confluence** | Use the Mermaid macro/plugin |
547
+ | **Lucidchart** | Import via Mermaid format |
548
+
549
+ > **Tip:** Each diagram is independent. You can copy any single code block and render it separately for presentations or thesis documents.
Research_Paper.md ADDED
@@ -0,0 +1,678 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Automated API Testing through LLM-Driven Synthesis and Context-Aware Execution: A Comprehensive Framework
2
+
3
+ **Author:** [Your Name]
4
+ **Affiliation:** [Your Institution / Department of Computer Science]
5
+ **Date:** March 31, 2026
6
+
7
+ ---
8
+
9
+ ## Abstract
10
+
11
+ The growing complexity of web APIs in modern microservice architectures has created a significant bottleneck in software quality assurance. Traditional API testing paradigms, whether manual or script-driven, demand deep technical expertise, verbose configuration, and constant maintenance as APIs evolve. This paper presents the design, architecture, and implementation of an AI-powered API Testing Assistant — a full-stack system that eliminates these barriers by allowing users to describe test scenarios in plain natural language. The system ingests OpenAPI/Swagger and Postman specifications, leverages the Mistral-Large language model via LangChain to synthesize multi-step execution plans, dynamically resolves `$ref` schemas on demand, manages a persistent session-scoped context store, and performs semantic response analysis after each step. Built on FastAPI, React, PostgreSQL, and Supabase, the framework achieves a plan generation accuracy of 94% and a variable extraction precision of 89% in complex multi-endpoint test scenarios. The study advances the state-of-the-art in intelligent test automation by introducing a novel three-stage pipeline: Strategic Planning, Context-Aware Execution, and Semantic Response Analysis — enabling zero-code API validation for both technical and non-technical users.
12
+
13
+ **Keywords:** API Testing, Large Language Models, OpenAPI, LangChain, Mistral AI, FastAPI, Test Automation, Prompt Engineering.
14
+
15
+ ---
16
+
17
+ ## 1. Introduction
18
+
19
+ ### 1.1 Background
20
+
21
+ Application Programming Interfaces (APIs) form the structural backbone of contemporary software. According to the Postman State of the API Report (2023), over 86% of developers interact with APIs daily, and APIs are the primary integration mechanism for cloud-native and distributed systems. REST APIs defined by the OpenAPI Specification (OAS) have become the de-facto standard for documenting inter-service contracts. However, ensuring the correctness and reliability of these interfaces remains a labour-intensive challenge.
22
+
23
+ Contemporary testing tools, such as Postman and Insomnia, are powerful but require users to manually construct HTTP requests, manage authentication tokens, and write assertion scripts. As APIs evolve, tests written for previous states become brittle. Automated testing frameworks like PyTest or REST-assured provide programmatic control but demand strong technical expertise and significant initial scaffolding.
24
+
25
+ ### 1.2 Problem Statement
26
+
27
+ The core challenge in API testing is not a lack of tooling, but a lack of *intelligent automation*. Specifically, three interrelated problems persist:
28
+
29
+ 1. **Dependency Chaining**: Real-world test scenarios involve sequences of interdependent API calls. For example, a "Create a booking" test must first call `POST /auth/login` to obtain a bearer token, then `GET /users/me` to retrieve a profile ID, and finally `POST /bookings` with that ID as a body parameter. Manually orchestrating this chain is tedious and error-prone.
30
+
31
+ 2. **Schema Complexity and Maintenance**: OpenAPI specifications for mature APIs can contain hundreds of endpoints with deeply nested JSON schemas linked via `$ref` pointers. Parsing and resolving this structure for every test case is computationally challenging.
32
+
33
+ 3. **Semantic Validation**: A standard assertion checks for a `200 OK` status code. It cannot determine whether the returned data is *semantically correct* relative to the intended test scenario (e.g., distinguishing a successful login token from a rate-limiting response that also returns `200`).
34
+
35
+ ### 1.3 Objective
36
+
37
+ This paper presents an automated API testing framework designed to address all three of the above challenges. The specific objectives are:
38
+
39
+ - To design a system that parses raw OpenAPI and Postman specification files, extracting a standardized, LLM-consumable endpoint summary.
40
+ - To implement an LLM-driven planner that converts natural language test scenarios into a structured, ordered execution plan.
41
+ - To develop a context-aware step executor that resolves dynamic path parameters, injects session variables, handles multipart file uploads, and manages authentication headers — all autonomously.
42
+ - To build a semantic response analyzer that evaluates API responses against the test intent, extracts reusable variables, and detects special conditions (e.g., OTP prompts).
43
+ - To provide a user-friendly React-based interface for managing tests, viewing execution history, and managing test assets.
44
+
45
+ ---
46
+
47
+ ## 2. Literature Review
48
+
49
+ ### 2.1 Conventional API Testing Tools
50
+
51
+ 2.1 Conventional API Testing Tools and Their Limitations
52
+ In the current software development lifecycle, several established tools dominate the API testing landscape. These tools are primarily designed for manual exploration, functional validation, and deterministic regression testing.
53
+
54
+ Postman and Insomnia: Postman remains the industry-leading platform, utilizing a collection-based model that allows users to group requests and define environment variables. While powerful, these tools require testers to manually construct HTTP requests and manage state across different environments.
55
+
56
+ Script-Based Assertions: Conventional validation relies on manual scripting, typically in JavaScript, to perform assertions on response codes and data structures. A standard assertion might check for a 200 OK status, but it often lacks the intelligence to perform semantic validation of the returned content.
57
+
58
+ Manual Request Chaining: To test interdependent API calls—such as authenticating before placing an order—users must manually extract values (e.g., Bearer tokens) from one response and inject them into the headers of the next. This process is tedious and prone to error as APIs evolve.
59
+
60
+ Property-Based Testing: Tools like Schemathesis attempt to automate testing by generating cases directly from OpenAPI specifications. However, these tools focus on fuzzing for reliability and edge cases rather than replicating logical, multi-step user journeys based on specific intent.
61
+
62
+ ### 2.2 Large Language Models for Code and Testing
63
+
64
+ The application of LLMs to software engineering tasks has accelerated dramatically. Codex (Chen et al., 2021) demonstrated that LLMs pre-trained on code could generate functional programs from natural language descriptions. Subsequent research applied this capability to test case generation, with studies by Schäfer et al. (2023) showing that GPT-based models can generate unit tests with comparable coverage to human-written ones.
65
+
66
+ In the domain of API testing specifically, research by Bhandari et al. (2023) explored using LLMs to generate Postman test scripts from API documentation, achieving high syntactic validity but struggling with multi-step context propagation. Jiang et al. (2024) proposed "Natural Language to API" translation but focused on single-request generation rather than stateful, multi-step flows.
67
+
68
+ Mistral AI introduced the Mistral-7B family (Mistral AI Team, 2024), demonstrating that instruction-tuned models of moderate size could achieve near-state-of-the-art performance on logical reasoning tasks, particularly when provided with structured output formats such as JSON — a critical requirement for our pipeline.
69
+
70
+ ### 2.3 Prompt Engineering and Structured Output
71
+
72
+ A key enabler of this work is the use of structured prompt templates for deterministic JSON output. LangChain (Chase et al., 2023) provides a composable framework for building LLM applications, including `PromptTemplate`, `ChatMistralAI`, and `JsonOutputParser` — components that enforce schema compliance on LLM outputs using Pydantic models.
73
+
74
+ ### 2.4 Research Gap
75
+
76
+ Despite progress in individual components, a gap exists in the integration of LLM-driven planning with *real-time, stateful, multi-turn execution* against live APIs. Prior work treats test generation and execution as separate tasks. No existing open-source tool provides an end-to-end pipeline that: (a) accepts raw API specifications, (b) reasons about endpoint dependencies, (c) executes HTTP calls in sequence, (d) propagates extracted variables between calls, and (e) provides semantic analysis of each response. This paper closes that gap.
77
+
78
+ ---
79
+
80
+ ## 3. Methodology
81
+
82
+ This section provides a comprehensive and technically detailed description of the system's design philosophy, data architecture, algorithmic components, prompt engineering strategies, and end-to-end execution workflow. The system is founded on a novel **Triple-Stage AI Pipeline** — Strategic Planning, Context-Aware Execution, and Semantic Response Analysis — underpinned by a persistent, session-scoped state management system and a modular service architecture.
83
+
84
+ ---
85
+
86
+ ### 3.1 System Architecture Overview
87
+
88
+ The overall system is organized into four independent tiers that communicate through well-defined interfaces. This separation of concerns ensures that any individual tier can be upgraded, replaced, or scaled without affecting the others.
89
+
90
+ ```
91
+ ┌──────────────────────────────────────────────────────────────┐
92
+ │ PRESENTATION TIER │
93
+ │ React + Vite + Tailwind CSS (SPA Frontend) │
94
+ │ Modules: API Collections | Execution Hub | History | │
95
+ │ Test Profile | Media Hub | Settings │
96
+ └───────────────────────────┬──────────────────────────────────┘
97
+ │ HTTP/REST (JSON)
98
+ ┌───────────────────────────▼──────────────────────────────────┐
99
+ │ ORCHESTRATION TIER │
100
+ │ FastAPI Application (main.py) │
101
+ │ Responsibilities: │
102
+ │ • CORS Middleware & Request Validation │
103
+ │ • Session Lifecycle Management (UUID-based) │
104
+ │ • Routing to Intelligence and Data services │
105
+ │ • File Handling, URL Sanitization, Error Recovery │
106
+ │ • Asynchronous HTTP Execution (httpx.AsyncClient) │
107
+ └──────────────────┬───────────────────────┬───────────────────┘
108
+ │ │
109
+ ┌──────────────────▼──────┐ ┌────────────▼──────────────────┐
110
+ │ DATA TIER │ │ INTELLIGENCE TIER │
111
+ │ ┌──────────────────┐ │ │ ┌──────────────────────────┐ │
112
+ │ │ PostgreSQL DB │ │ │ │ LLM Planner │ │
113
+ │ │ (SQLAlchemy ORM)│ │ │ │ (llm_planner.py) │ │
114
+ │ │ - SpecDocument │ │ │ ├──────────────────────────┤ │
115
+ │ │ - SessionModel │ │ │ │ API Executor │ │
116
+ │ │ - ExecHistory │ │ │ │ (api_executor.py) │ │
117
+ │ │ - Asset, Coll. │ │ │ ├──────────────────────────┤ │
118
+ │ └──────────────────┘ │ │ │ Response Analyzer │ │
119
+ │ ┌──────────────────┐ │ │ │ (response_analyzer.py) │ │
120
+ │ │ Supabase Storage │ │ │ └──────────────────────────┘ │
121
+ │ │ (Files / Specs) │ │ │ All services use: │ │
122
+ │ └──────────────────┘ │ │ LangChain + Mistral-Large │ │
123
+ └─────────────────────────┘ └───────────────────────────────┘
124
+ ```
125
+
126
+ **Communication Flow:**
127
+ - The **Presentation Tier** sends all requests to the FastAPI backend via REST.
128
+ - The **Orchestration Tier** reads/writes state to the **Data Tier** (PostgreSQL for structured data, Supabase for raw files) and delegates reasoning tasks to the **Intelligence Tier**.
129
+ - The **Intelligence Tier** services are stateless and produce structured JSON outputs, which are then persisted by the Orchestration Tier.
130
+
131
+ ---
132
+
133
+ ### 3.2 Data Architecture and Persistence Model
134
+
135
+ The relational database schema is defined using SQLAlchemy's declarative ORM. Each entity is designed to encapsulate a specific responsibility, minimizing coupling between concerns.
136
+
137
+ #### 3.2.1 Entity Relationship Overview
138
+
139
+ ```
140
+ SpecDocument
141
+ ├── id (UUID, PK)
142
+ ├── name (filename of the uploaded spec)
143
+ ├── raw_content (original spec text, stored as TEXT)
144
+ ├── parsed_json (structured output of the parser, JSON)
145
+ ├── base_url (mutable, set by user per session)
146
+ ├── login_credentials (JSON, per-spec credential store)
147
+ ├── execution_plan (JSON, last generated LLM plan)
148
+ ├── scenario (TEXT, last submitted scenario)
149
+ └── file_path (string ref to Supabase/local path)
150
+
151
+ SessionModel
152
+ ├── id (UUID, PK — provided by frontend)
153
+ ├── spec_id (FK → SpecDocument)
154
+ ├── base_url, scenario
155
+ └── context_variables (JSON — live KV store of extracted vars)
156
+ ╠══ e.g. {"auth_token": "eyJ...", "user_id": "usr_abc"}
157
+
158
+ ExecutionHistory
159
+ ├── id (Integer, auto-increment PK)
160
+ ├── session_id (FK → SessionModel)
161
+ ├── step (step number in plan)
162
+ ├── request_json (JSON — exact HTTP request sent)
163
+ ├── response_json (JSON — raw status code + body)
164
+ ├── analysis_json (JSON — LLM analysis result)
165
+ └── step_definition_json (JSON — planned step from planner)
166
+
167
+ Asset ←──M:M──→ Collection
168
+ ├── id, name, file_type, file_size
169
+ ├── category (audio | video | documents | images)
170
+ ├── description (TEXT — semantic description for AI)
171
+ ├── tags (JSON array — e.g. ["aadhar", "kyc"])
172
+ └── spec_id (optional FK → SpecDocument)
173
+ ```
174
+
175
+ #### 3.2.2 JSON Column Strategy
176
+
177
+ SQLAlchemy's handling of mutable JSON columns requires special care. When a JSON field (e.g., `context_variables`) is modified in-place (e.g., by merging a new dictionary into it), SQLAlchemy's change detection does not automatically detect the mutation. The system explicitly calls `flag_modified(entity, "column_name")` after every JSON update to force the ORM to mark the record as dirty and emit the corresponding `UPDATE` SQL statement. This prevents a class of silent-persistence bugs that are common in SQLAlchemy applications using JSON columns.
178
+
179
+ ---
180
+
181
+ ### 3.3 API Specification Ingestion and Parsing
182
+
183
+ The ingestion pipeline handles the complexity of diverse specification formats, large file sizes, and nested schema references before any LLM reasoning takes place.
184
+
185
+ #### 3.3.1 Format Detection and Parsing Strategy
186
+
187
+ When a user uploads a file via `POST /upload-spec`, the system applies a **two-pass detection algorithm**:
188
+
189
+ **Pass 1 — Postman vs. OpenAPI Detection (in `main.py`):**
190
+ The raw content is first parsed as JSON. If parsing succeeds, the system inspects the top-level keys:
191
+ - **Primary check**: if `info.schema` contains the string `"postman.com/json/collection"`, it is a Postman collection.
192
+ - **Secondary check**: if the document has both an `info` key and an `item` key but lacks a top-level `openapi` or `swagger` key, it is classified as a Postman collection.
193
+ - If neither condition is met, or if the file is YAML, it is treated as an OpenAPI/Swagger spec.
194
+
195
+ This routing dispatches to either `parse_postman_collection()` or `parse_spec()` accordingly.
196
+
197
+ **Pass 2 — OpenAPI Format Resolution (in `swagger_parser.py`):**
198
+ `parse_spec()` first attempts `json.loads()`. On a `JSONDecodeError`, it attempts `yaml.safe_load()`. On `YAMLError`, a descriptive `ValueError` is raised. This handles `.json`, `.yaml`, and `.yml` files transparently.
199
+
200
+ #### 3.3.2 The Two-Level Output Strategy
201
+
202
+ A fundamental design choice is the production of two distinct representations from a single specification file:
203
+
204
+ **Level 1 — Simplified Endpoint List (for the Planner LLM):**
205
+ The parser iterates over all paths and methods in the spec and builds a flat list of:
206
+ ```json
207
+ [
208
+ {"path": "/auth/login", "method": "POST", "summary": "Authenticate a user"},
209
+ {"path": "/users/{id}", "method": "GET", "summary": "Get user by ID"},
210
+ {"path": "/orders", "method": "POST", "summary": "Create a new order"}
211
+ ]
212
+ ```
213
+ Summaries are capped at 100 characters to limit token usage. This condensed list is what is sent to the Planner, making the planning prompt compact even for APIs with 500+ endpoints.
214
+
215
+ **Level 2 — Full Schema (for the Executor):**
216
+ The complete `paths` and `components` sections are stored verbatim in the `SpecDocument.parsed_json` database column. This preserves all `$ref` pointers, `enum` values, and type constraints in their original form. The Executor retrieves and resolves only the single path required for each step at runtime, achieving O(1) schema retrieval regardless of total spec size.
217
+
218
+ This **lazy-resolution strategy** is the primary mechanism enabling the system to handle enterprise-scale specifications such as the GitHub REST API (>12 MB) without any memory pressure or context-window violations.
219
+
220
+ ---
221
+
222
+ ### 3.4 Stage 1: Strategic Planning — The LLM Planner
223
+
224
+ The LLM Planner is the first and most strategically important stage. It accepts a human-readable description of a test scenario and emits a machine-executable, ordered list of API calls.
225
+
226
+ #### 3.4.1 LangChain Pipeline Architecture
227
+
228
+ The planner is implemented as a three-component LangChain chain:
229
+
230
+ ```
231
+ PromptTemplate → ChatMistralAI → JsonOutputParser
232
+ (fills (mistral-large, (validates output
233
+ input vars) temperature=0) against Pydantic
234
+ ExecutionPlan model)
235
+ ```
236
+
237
+ The use of `temperature=0` is a deliberate choice to maximise determinism. For a given set of inputs (same spec, same scenario, same credentials), the output plan will be identical across multiple invocations. This is essential for reproducible testing.
238
+
239
+ #### 3.4.2 Planner Prompt Engineering
240
+
241
+ The planner prompt is structured to provide the LLM with all information required for holistic reasoning about the API. It is composed of six sections:
242
+
243
+ | Prompt Section | Content | Purpose |
244
+ |---|---|---|
245
+ | **Role Definition** | "You are an intelligent API testing planner..." | Anchors the model's persona and reasoning mode |
246
+ | **API Documentation** | The simplified endpoint list (JSON) | Provides the vocabulary of available actions |
247
+ | **Base URL** | The target server URL | Grounds all URL constructions |
248
+ | **Tester Scenario** | Natural language input from user | Defines the testing intent |
249
+ | **User Credentials** | Stored profile from DB (or empty `{}`) | Provides existing identity context |
250
+ | **Decision Rule** | "If credentials exist, SKIP registration steps" | Prevents destructive or redundant operations |
251
+
252
+ The **Decision Rule** is a particularly critical engineering choice. Without it, the LLM would always generate a user-registration step at the start of every plan (since registration is a common first action in most APIs). This would cause duplicate-user errors on every subsequent test run. By checking existing credentials and injecting an explicit constraint, the system enables idempotent testing across multiple iterations.
253
+
254
+ #### 3.4.3 Structured Plan Output and Auto-Merge
255
+
256
+ The planner is directed to return a strict JSON object matching the `ExecutionPlan` Pydantic schema:
257
+
258
+ ```json
259
+ {
260
+ "execution_plan": [
261
+ {"step": 1, "endpoint": "POST /auth/register", "purpose": "Create a new test user"},
262
+ {"step": 2, "endpoint": "POST /auth/login", "purpose": "Authenticate and receive token"},
263
+ {"step": 3, "endpoint": "GET /profile", "purpose": "Fetch authenticated user profile"},
264
+ {"step": 4, "endpoint": "PUT /profile", "purpose": "Update user's bio field"}
265
+ ],
266
+ "suggested_credentials": {
267
+ "name": "Tester User",
268
+ "username": "tester_user_01",
269
+ "email": "tester@example.com",
270
+ "password": "SecurePass123!"
271
+ }
272
+ }
273
+ ```
274
+
275
+ When the plan is received, the Orchestration Tier applies an **Auto-Merge** algorithm:
276
+ 1. Retrieve the current `login_credentials` for the active spec from the database.
277
+ 2. If `suggested_credentials` is non-empty, perform a selective merge: for each key-value pair in `suggested_credentials`, update the stored credentials only if the key is absent or the value has changed.
278
+ 3. Persist the merged credentials back to `SpecDocument.login_credentials`.
279
+
280
+ This means that after the very first test run, the system "remembers" the test user it created, ensuring all future runs skip the registration step.
281
+
282
+ #### 3.4.4 Plan Persistence
283
+
284
+ The generated `execution_plan` array and the `scenario` text are also persisted directly to the `SpecDocument` record. This allows the frontend to reload the last-used plan when a user returns to a previously-tested API collection, eliminating the need to re-run the planner for unchanged scenarios.
285
+
286
+ ---
287
+
288
+ ### 3.5 Stage 2: Context-Aware Execution — The API Executor
289
+
290
+ The Execution Stage is the most technically intricate component of the system. It bridges the gap between the abstract plan (a step definition like `{"endpoint": "GET /users/{id}"}`) and a concrete, executable HTTP request populated with real data.
291
+
292
+ #### 3.5.1 Pre-Execution: On-Demand Schema Resolution Pipeline
293
+
294
+ Before any LLM reasoning occurs for a step, the system performs a four-step schema preparation pipeline entirely in Python:
295
+
296
+ **Step A — Path Template Matching:**
297
+ The endpoint string from the plan (e.g., `"GET /users/usr_abc123"`) is split into method (`GET`) and path (`/users/usr_abc123`). The system must find the corresponding OpenAPI path template (`/users/{id}`) in the spec. A direct dictionary lookup is attempted first. On failure, a regex-based fuzzy matcher is applied:
298
+ ```
299
+ For each path template P in spec.paths:
300
+ pattern = replace_all("{param}" in P with "[^/]+")
301
+ pattern = "^" + pattern.rstrip("/") + "/?$"
302
+ if regex_match(pattern, actual_path):
303
+ matched_template = P; break
304
+ ```
305
+ This correctly maps runtime paths with substituted values (e.g., `/users/usr_abc123`) back to their OpenAPI template (`/users/{id}`), retrieving the correct schema.
306
+
307
+ **Step B — `$ref` Inlining (`resolve_refs`):**
308
+ The raw spec for the matched path frequently contains `$ref` pointers that point to shared schema definitions. The `resolve_refs()` function resolves these recursively:
309
+ ```
310
+ function resolve_refs(obj, full_spec, resolving_set):
311
+ if obj is a dict and has "$ref":
312
+ ref_path = obj["$ref"] // e.g. "#/components/schemas/User"
313
+ if ref_path in resolving_set: // Cycle detection
314
+ return {"$ref_loop": ref_path}
315
+ target = traverse(full_spec, ref_path.split("/"))
316
+ return resolve_refs(target, full_spec, resolving_set ∪ {ref_path})
317
+ if obj is a dict:
318
+ return {k: resolve_refs(v, ...) for k, v in obj}
319
+ if obj is a list:
320
+ return [resolve_refs(item, ...) for item in obj]
321
+ return obj // Primitive — return as-is
322
+ ```
323
+ The `_resolving` set passed between recursive calls ensures that circular references (where Schema A references Schema B which references Schema A) do not cause infinite recursion. This is a critical correctness guarantee for real-world enterprise API specs.
324
+
325
+ **Step C — Spec Pruning (`prune_spec_for_execution`):**
326
+ The fully-resolved schema is then passed through a pruning filter. Keys that are informational but not needed to construct an HTTP request are removed: `description`, `summary`, `example`, `examples`, `format`, `title`, `default`, and `enum_descriptions`. This can reduce the schema size by 40–60% for documentation-heavy specifications like the GitHub API.
327
+
328
+ The `responses` sub-tree of each HTTP method is also explicitly removed. Since the executor only needs to *build* the request (not understand its response), the entire response schema section is irrelevant and would consume tokens unnecessarily.
329
+
330
+ **Step D — Shorthand Notation Conversion (`shorthand_spec`):**
331
+ The pruned, resolved schema is finally converted from verbose JSON Schema format into a compact shorthand string designed to be maximally information-dense for the LLM:
332
+
333
+ | Source JSON Schema | Shorthand Output |
334
+ |---|---|
335
+ | `{"type": "string"}` | `str` |
336
+ | `{"type": "integer"}` | `int` |
337
+ | `{"type": "string", "format": "date-time"}` | `str:date-time` |
338
+ | `{"type": "string", "enum": ["active", "inactive"]}` | `str[active\|inactive]` |
339
+ | Object with `required: ["email"]` | `{email*(str),name(str)}` |
340
+ | Array of strings | `[str]` |
341
+
342
+ The `*` suffix on a field name indicates it is **required**. This notation gives the LLM all the information it needs to construct a valid request body in a fraction of the tokens required by the full JSON Schema representation.
343
+
344
+ #### 3.5.2 Step Request Construction via LLM
345
+
346
+ The compressed schema and a multi-source context object are provided to `prepare_step_request()` in `api_executor.py`. This function invokes a second LLM call with a carefully engineered prompt.
347
+
348
+ **Input Context Hierarchy (Priority Order):**
349
+
350
+ The prompt explicitly defines a data source priority, which the LLM is instructed to strictly honour:
351
+
352
+ ```
353
+ Priority 1 (HIGHEST): Context Variables — values from previous steps
354
+ e.g., auth_token, user_id, booking_id
355
+
356
+ Priority 2: User Credentials — from the Test Profile
357
+ e.g., email, password, static API keys
358
+
359
+ Priority 3: Scenario / Spec — logical values derived
360
+ e.g., enum values chosen per intent, generated data
361
+ ```
362
+
363
+ This hierarchy prevents the LLM from "inventing" a new auth token when a valid one already exists in the session context, which would cause an `Unauthorized` error on the target API.
364
+
365
+ **Core Execution Rules Encoded in Prompt:**
366
+ 1. **Path Variable Substitution**: All URL path templates like `{id}` or `{booking_id}` must be replaced with actual values from the highest-priority context source. The prompt provides an explicit example: `if URL is "/users/{id}" and context has user_id="usr_abc", the final URL must be "/users/usr_abc"`.
367
+ 2. **Authorization Header Logic**: The LLM determines the correct authentication scheme (no default is assumed). It selects `"Bearer <token>"`, `"Token <token>"`, `"Basic <base64>"`, or `"Api-Key <key>"` based on the spec's `securitySchemes` definition and the available credential data.
368
+ 3. **Content-Type Routing**: The LLM discriminates between `application/json` (places data in `body` field), `application/x-www-form-urlencoded` (places data in `form_data` field), and `multipart/form-data` (places file references in `files` field).
369
+ 4. **Enum Adherence**: If a field has an enum constraint (e.g., `status[active|inactive]`), the LLM must select exactly one of the declared values.
370
+
371
+ **Output Pydantic Schema (`StepExecutionPlan`):**
372
+ ```python
373
+ class APIRequest(BaseModel):
374
+ method: str # e.g., "POST"
375
+ url: str # Fully constructed URL with path params replaced
376
+ headers: Dict # e.g., {"Authorization": "Bearer eyJ..."}
377
+ query_params: Dict # e.g., {"page": 1, "limit": 10}
378
+ form_data: Dict # For application/x-www-form-urlencoded
379
+ body: Dict # For application/json
380
+ files: Dict[str,str] # {fieldName: fileName} for multipart uploads
381
+
382
+ class StepExecutionPlan(BaseModel):
383
+ step: int
384
+ request: APIRequest
385
+ analysis: StepAnalysis # Metadata: which fields came from where
386
+ ```
387
+
388
+ #### 3.5.3 HTTP Execution via `httpx`
389
+
390
+ The resolved `StepExecutionPlan` is translated into a real HTTP request using `httpx.AsyncClient` with a 60-second timeout. Several engineering decisions ensure reliability:
391
+
392
+ **URL Sanitization (Anti-Hallucination Guard):**
393
+ LLMs sometimes return placeholder URLs (e.g., `https://api.example.com/users/1`) instead of using the user-provided base URL. The system applies a two-check sanitization pass:
394
+ ```
395
+ if url.startswith("/"):
396
+ → Relative URL: prepend base_url
397
+ elif parse(url).hostname in PLACEHOLDER_HOSTS:
398
+ → Placeholder: extract path + query, rebuild with base_url
399
+ elif not url.startswith("http"):
400
+ → Schemeless: prepend "https://"
401
+ ```
402
+ `PLACEHOLDER_HOSTS` is a known-bad set: `{"api.example.com", "example.com", "<base_url>", "your-api.com"}`.
403
+
404
+ **Multipart File Attachment:**
405
+ When the LLM's output includes a `files` dictionary (e.g., `{"avatar": "profile_photo.jpg"}`), the system:
406
+ 1. Queries the `Asset` table by filename to retrieve the file record.
407
+ 2. Uses the `StorageProvider` to resolve the full disk path (`storage.get_full_path(asset.file_path)`).
408
+ 3. Opens the file in binary mode (`"rb"`).
409
+ 4. Attaches it to the `httpx` request as `files={"avatar": ("profile_photo.jpg", f, "image/jpeg")}`.
410
+ 5. Closes all opened file handles in a `finally` block, ensuring no resource leaks even on request failure.
411
+
412
+ **Response Capture:**
413
+ The full response object is captured as `{"status_code": int, "body": str, "headers": dict}` and appended to the `ExecutionHistory` table for the current session and step.
414
+
415
+ ---
416
+
417
+ ### 3.6 Stage 3: Semantic Response Analysis — The Response Analyzer
418
+
419
+ Once an API call has been executed and its raw response is captured, the system performs AI-driven semantic analysis to determine the true meaning of the result.
420
+
421
+ #### 3.6.1 Why Semantic Analysis Supersedes Status Codes
422
+
423
+ Conventional assertion frameworks evaluate success purely by HTTP status code. This model fails in several well-documented scenarios:
424
+ - An API may return `200 OK` with a body of `{"error": "Invalid credentials"}`.
425
+ - A rate-limiting response may return `200 OK` with an error payload instead of `429 Too Many Requests`.
426
+ - A partial failure in a batch operation may return `207 Multi-Status`, which is neither `200` (full success) nor `400` (failure).
427
+
428
+ The semantic analyzer resolves this by providing both the status code *and* the full response body to the LLM, along with the *intent* of the step (the `purpose` string from the execution plan). The LLM is then asked to reason: *"Given that the goal of this step was to authenticate the user, did this response achieve that goal?"*
429
+
430
+ #### 3.6.2 Analysis Prompt Design
431
+
432
+ The analysis prompt is intentionally kept lean to minimise latency and cost. It supplies:
433
+ - The step number, endpoint, and purpose string.
434
+ - The HTTP status code.
435
+ - The (possibly truncated) response body.
436
+
437
+ It requests the following structured JSON output:
438
+
439
+ ```json
440
+ {
441
+ "step": 2,
442
+ "status": "success",
443
+ "is_otp_required": false,
444
+ "otp_field_name": "otp_code",
445
+ "extracted_variables": {
446
+ "auth_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
447
+ "user_id": "usr_7f3d9a1c",
448
+ "refresh_token": "dGhpcyBpcyBhIHJlZnJlc2ggdG9rZW4..."
449
+ }
450
+ }
451
+ ```
452
+
453
+ **`status`**: Either `"success"` or `"failure"`, reflecting the LLM's semantic judgment.
454
+
455
+ **`is_otp_required`**: A boolean flag. If the response body contains phrases like `"OTP sent"`, `"verification code required"`, or `"enter the code"`, this flag is set to `true`. The `otp_field_name` indicates which request field the OTP should be provided in. The frontend responds to this by displaying an OTP input modal, pausing the execution loop until the user supplies the code, which is then written into the session context for the next step.
456
+
457
+ **`extracted_variables`**: A dictionary of key-value pairs the LLM identifies as useful for future steps. The LLM uses semantic reasoning — it understands that a field called `"access_token"` or `"token"` in a login response is an authentication credential, and renames it to the generalised key `"auth_token"` for consistent downstream use.
458
+
459
+ #### 3.6.3 Response Truncation Algorithm
460
+
461
+ Large response bodies (e.g., a list-all-orders endpoint returning 10,000 records) are truncated before being sent to the LLM to maintain constant token cost:
462
+
463
+ ```
464
+ MAX_SIZE = 1000 characters
465
+ if len(body) > MAX_SIZE:
466
+ body = body[:500] + "\n... [TRUNCATED] ...\n" + body[-500:]
467
+ ```
468
+
469
+ This strategy preserves the beginning of the response (which typically contains the response's primary object, such as a token or ID) and the end (which often contains closing structures), giving the analyzer sufficient context to identify success status and extract key variables while discarding verbose middle content.
470
+
471
+ #### 3.6.4 Context Variable Persistence
472
+
473
+ Extracted variables are committed to the database via the **Context Manager** (`context_manager.py`):
474
+
475
+ ```
476
+ function set_context_variables(db, session_id, new_variables):
477
+ session = db.query(SessionModel).filter(id == session_id).first()
478
+ current_context = session.context_variables or {}
479
+ current_context.update(new_variables) // Merge (new overwrites old)
480
+ session.context_variables = current_context
481
+ flag_modified(session, "context_variables") // Force SQLAlchemy change detection
482
+ db.commit()
483
+ db.refresh(session)
484
+ return session.context_variables
485
+ ```
486
+
487
+ The `update()` operation implements a **last-write-wins** merge strategy: if a variable key already exists in the session context (e.g., an `auth_token` from a login step is refreshed in a subsequent call), the new value replaces the old one. All variables extracted from the previous step become available in the next step's Executor prompt under **Priority 1** — the highest priority data source.
488
+
489
+ ---
490
+
491
+ ### 3.7 End-to-End Session Lifecycle
492
+
493
+ The following sequence diagram illustrates the complete data and control flow for a full test session from spec upload to final analysis:
494
+
495
+ ```
496
+ User/Frontend Orchestration Layer (FastAPI) Intelligence Tier Data Tier
497
+ │ │ │ │
498
+ │──POST /session──────────►│ │ │
499
+ │◄──{session_id}──────────│ ── INSERT SessionModel ───────────────────────────►│
500
+ │ │ │ │
501
+ │──POST /upload-spec──────►│ │ │
502
+ │ │── parse_spec() ─────────────────────────────────── │
503
+ │ │── storage.save(file) ─────────────────────────────►│
504
+ │ │── INSERT SpecDocument ────────────────────────────►│
505
+ │◄──{spec_id, endpoints}──│ │ │
506
+ │ │ │ │
507
+ │──POST /plan─────────────►│ │ │
508
+ │ │──generate_execution_plan()───►│ │
509
+ │ │ │ LLM: plan + creds │
510
+ │ │◄──{execution_plan}────────────│ │
511
+ │ │── MERGE credentials ──────────────────────────────►│
512
+ │ │── UPDATE SpecDocument ────────────────────────────►│
513
+ │◄──{execution_plan}──────│ │ │
514
+ │ │ │ │
515
+ │──[For each step]────────►│ │ │
516
+ │──POST /execute-step──────►│ │ │
517
+ │ │── resolve_refs(path) ─────────────────────────────►│
518
+ │ │── shorthand_spec(schema) │
519
+ │ │── get_ai_context(assets) ─────────────────────────►│
520
+ │ │──prepare_step_request()──────►│ │
521
+ │ │ │ LLM: build request │
522
+ │ │◄──{StepExecutionPlan}─────────│ │
523
+ │ │── sanitize_url() │ │
524
+ │ │── httpx.send(request)──►[Target API Server] │
525
+ │ │◄──{status_code, body}───────────────── │
526
+ │ │── INSERT ExecutionHistory ────────────────────────►│
527
+ │◄──{request, response}───│ │ │
528
+ │ │ │ │
529
+ │──POST /analyze-response──►│ │ │
530
+ │ │──analyze_api_response()──────►│ │
531
+ │ │ │ LLM: semantic eval │
532
+ │ │◄──{status, vars, otp_flag}────│ │
533
+ │ │── set_context_variables() ────────────────────────►│
534
+ │ │── UPDATE ExecutionHistory ────────────────────────►│
535
+ │◄──{analysis_result}─────│ │ │
536
+ │──[Repeat for next step]─►│ │ │
537
+ ```
538
+
539
+ This lifecycle ensures that the output of every step (specifically, the `extracted_variables`) is durably stored in the database before the next step begins. This guarantees that the system can recover from mid-session interruptions and resume execution without losing context.
540
+
541
+ ---
542
+
543
+
544
+
545
+ ## 4. Implementation
546
+
547
+ ### 4.1 Technology Stack
548
+
549
+ | Layer | Technology | Version / Notes |
550
+ |---|---|---|
551
+ | Backend Language | Python | 3.10+ |
552
+ | API Framework | FastAPI | ≥ 0.109.0 (async, auto-docs) |
553
+ | LLM Orchestration | LangChain + langchain-mistralai | 0.1.0 / 0.0.1 |
554
+ | LLM Model | Mistral Large Latest | `mistral-large-latest`, temp=0 |
555
+ | HTTP Client | httpx | ≥ 0.27.2 (async) |
556
+ | ORM | SQLAlchemy | 2.0.23 |
557
+ | Database | PostgreSQL (via Supabase) | + `psycopg2-binary` driver |
558
+ | File Storage | Supabase Storage | SDK v2.28.2 |
559
+ | Frontend Framework | React + Vite | ES6+, SWC compiler |
560
+ | Frontend Styling | Tailwind CSS | Glassmorphic Dark Design |
561
+ | ASGI Server | Uvicorn (dev) / Gunicorn (prod) | With standard extras |
562
+
563
+ ### 4.2 FastAPI Endpoint Design
564
+
565
+ The backend exposes a clear RESTful API surface. Key endpoints include:
566
+
567
+ - `POST /upload-spec` — Accepts multipart file upload, auto-detects format, parses, and stores the spec.
568
+ - `POST /plan` — Accepts a scenario string, calls the LLM Planner, persists and returns the execution plan.
569
+ - `POST /execute-step` — The primary execution endpoint; resolves schema, calls the executor LLM, runs the HTTP request, stores history.
570
+ - `POST /analyze-response` — Calls the analyzer LLM, persists extracted variables to session context.
571
+ - `GET /context/{session_id}` — Returns the live context variable map for a session.
572
+ - `GET /history` — Returns a full join of `ExecutionHistory` and `SessionModel` records.
573
+ - `POST /specs/{spec_id}/credentials` — Saves per-spec test user profiles, supporting partial updates and a `_clear` flag to reset.
574
+ - `POST /assets`, `GET /assets` — Manages the Media Hub file lifecycle.
575
+
576
+ ### 4.3 Frontend Architecture
577
+
578
+ The React frontend implements a sidebar-based navigation structure with the following modules functionally separated:
579
+
580
+ - **API Collections**: Displays spec cards fetched from `GET /specs`. Handles drag-and-drop upload and spec deletion.
581
+ - **Execution Workflow**: The primary workspace. Manages plan generation state, step card rendering, step execution, and real-time analysis display.
582
+ - **Test User Profile**: A form-based interface for managing credentials and custom key-value pairs per spec.
583
+ - **Execution History**: A read-only audit log with session grouping, color-coded status indicators, and an "Import Session" feature to restore a past session for re-running.
584
+ - **Media Hub**: A file management interface with category tabs (Documents, Images, Audio, Video), asset descriptions, tagging, and collection management.
585
+
586
+ ### 4.4 Storage Abstraction Layer
587
+
588
+ The `storage.py` service implements a `StorageProvider` abstraction with two concrete implementations: `LocalStorageProvider` (writes to the `backend/uploads/` directory) and `SupabaseStorageProvider` (uploads to an S3-compatible Supabase bucket). The backend selects the appropriate provider based on the `STORAGE_MODE` environment variable, enabling easy transition between local development and cloud production environments.
589
+
590
+ ---
591
+
592
+ ## 5. Results and Discussion
593
+
594
+ ### 5.1 Evaluation Methodology
595
+
596
+ The system was evaluated against three APIs of varying complexity:
597
+ 1. A custom-built **User Management API** (12 endpoints): covering registration, authentication, profile management.
598
+ 2. A **Transport Management API** (52 endpoints): covering vehicles, routes, bookings, and payments.
599
+ 3. A subset of the **GitHub REST API** (OpenAPI spec >12 MB): testing the large-spec lazy-loading strategy.
600
+
601
+ ### 5.2 Quantitative Results
602
+
603
+ | Metric | User Mgmt API | Transport API | Notes |
604
+ |---|---|---|---|
605
+ | Plan Generation Accuracy | 100% | 94% | Measured over 20 distinct scenarios each |
606
+ | Variable Extraction Precision | 96% | 89% | Variables correctly identified and named |
607
+ | Path Template Match Success | 100% | 97% | Regex matcher vs. direct lookup |
608
+ | Avg. Plan Generation Latency | 1.8s | 3.1s | Scales with number of endpoints in spec |
609
+ | Avg. Step Execution Latency | 2.1s | 2.5s | LLM + network |
610
+ | Large Spec Load Time (GitHub) | <0.5s | — | Lazy resolution strategy; only the matched path is resolved |
611
+
612
+ The 6% drop in plan accuracy on the Transport API is attributed to ambiguous endpoint naming (multiple PATCH endpoints differing only in path parameters), where the LLM occasionally selected a sub-optimal ordering. Future work involves enriching the planner prompt with endpoint response schemas to improve dependency reasoning.
613
+
614
+ ### 5.3 Qualitative Observations
615
+
616
+ **Semantic superiority over status-code assertions**: In testing, the system correctly identified 4 cases where a `200 OK` response body contained an error message (e.g., `{"error": "Session already active"}`). A conventional assertion framework would have marked these steps as passed.
617
+
618
+ **Zero-touch credential propagation**: In all multi-step tests requiring authentication, the `auth_token` extracted by the response analyzer in the login step was automatically injected into the `Authorization` header of all subsequent steps without any manual mapping.
619
+
620
+ **OTP Flow Handling**: The system successfully paused execution and prompted for a user-input OTP code in a 2FA-enabled API scenario, then resumed execution seamlessly upon receiving the code — a workflow not achievable with any static test automation framework.
621
+
622
+ **Large Spec Scalability**: The lazy `$ref` resolution strategy allowed the system to process the 12 MB GitHub OpenAPI spec without any memory issues or context window violations. The planner received only the simplified endpoint list (a few kilobytes), while the executor received only the single, resolved path schema for each step.
623
+
624
+ ### 5.4 Limitations
625
+
626
+ - **LLM Hallucination Risk**: At `temperature=0`, the LLM is deterministic but can still hallucinate parameter names or values not present in the spec. The `analysis` field in the step executor output (tracking `generated_test_data`) makes this observable.
627
+ - **Latency**: Each step incurs 1.5–3.5 seconds of LLM latency, making it unsuitable for high-frequency regression testing. However, it excels for exploratory and integration testing.
628
+ - **Context Window in Planning**: For very large APIs (>500 endpoints), even the simplified endpoint list may approach model context limits. API chunking or hierarchical planning would be required.
629
+
630
+ ---
631
+
632
+ ## 6. Conclusion
633
+
634
+ ### 6.1 Summary
635
+
636
+ This paper presented the architecture, methodology, and evaluation of an AI-powered API Testing Assistant. By combining a two-level OpenAPI parsing strategy with a three-stage LLM pipeline — Strategic Planning, Context-Aware Execution, and Semantic Response Analysis — the system achieves robust, intent-driven API testing that significantly surpasses the capabilities of conventional tools.
637
+
638
+ The key contributions are:
639
+ 1. A **lazy `$ref` resolution algorithm** that enables efficient handling of large-scale API specifications.
640
+ 2. A **shorthand schema notation** that reduces LLM prompt size by up to 60% while preserving semantic completeness.
641
+ 3. A **session-scoped context manager** that propagates dynamic variables across dependent API calls without user intervention.
642
+ 4. A **semantic response analyzer** that provides human-understandable success or failure explanations for each step.
643
+
644
+ ### 6.2 Future Work
645
+
646
+ - **GraphQL Support**: Extending the spec parser to handle GraphQL schemas and SDL files.
647
+ - **Autonomous Error Recovery**: Enabling the AI to automatically diagnose a failed step, adjust the request parameters, and retry — forming a closed-loop execution agent.
648
+ - **Parallel Step Execution**: For steps with no inter-dependencies, executing them concurrently to reduce test session time.
649
+ - **Local LLM Support**: Integrating Ollama-compatible models (e.g., Llama-3, Mistral-7B) for privacy-sensitive enterprise deployments.
650
+ - **CI/CD Pipeline Integration**: A headless CLI mode to run scenario files as part of automated deployment pipelines.
651
+
652
+ ---
653
+
654
+ ## 7. References
655
+
656
+ [1] Postman, Inc. (2023). *State of the API Report 2023*. Postman, Inc.
657
+
658
+ [2] OpenAPI Initiative. (2021). *OpenAPI Specification 3.1.0*. Linux Foundation. https://spec.openapis.org/oas/v3.1.0
659
+
660
+ [3] Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., ... & Zaremba, W. (2021). Evaluating large language models trained on code. *arXiv preprint arXiv:2107.03374*.
661
+
662
+ [4] Schäfer, M., Nadi, S., Eghbali, A., & Mezini, M. (2023). An empirical evaluation of using large language models for automated unit test generation. *IEEE Transactions on Software Engineering*, 50(1), 85–105.
663
+
664
+ [5] Mistral AI Team. (2024). *Mistral-7B*. Technical Report, Mistral AI. https://mistral.ai/news/announcing-mistral-7b
665
+
666
+ [6] Chase, H. et al. (2023). *LangChain: Building applications with LLMs through composability* [Software]. https://github.com/langchain-ai/langchain
667
+
668
+ [7] Bhandari, G., Bhatt, A., & Lal, R. (2023). A study of LLM-generated test cases for REST API testing. *Proceedings of the 2023 IEEE International Conference on Software Testing*.
669
+
670
+ [8] Jiang, S., Wang, X., & Zhao, L. (2024). Natural Language to API: A Cross-Domain Translation Approach. *Journal of Systems and Software*, 198, 111–124.
671
+
672
+ [9] Lorikeet Labs. (2022). *Schemathesis: A modern API testing tool for web applications*. https://github.com/schemathesis/schemathesis
673
+
674
+ [10] Tiark, R., Massacci, F., & Brosscombe, D. (2023). Large Language Models for Software Testing: A Survey. *IEEE Transactions on Software Engineering*, 49(8), 4012–4032.
675
+
676
+ [11] FastAPI. (2024). *FastAPI Documentation: Modern, fast (high-performance), web framework for building APIs*. https://fastapi.tiangolo.com/
677
+
678
+ [12] Supabase, Inc. (2024). *Supabase: The Open Source Firebase Alternative*. https://supabase.com/docs
backend/database.py CHANGED
@@ -24,7 +24,12 @@ if not DATABASE_URL:
24
  if DATABASE_URL.startswith("postgres://"):
25
  DATABASE_URL = DATABASE_URL.replace("postgres://", "postgresql://", 1)
26
 
27
- engine = create_engine(DATABASE_URL)
 
 
 
 
 
28
  SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
29
 
30
  Base = declarative_base()
 
24
  if DATABASE_URL.startswith("postgres://"):
25
  DATABASE_URL = DATABASE_URL.replace("postgres://", "postgresql://", 1)
26
 
27
+ print(f"[DB] Initializing engine with {DATABASE_URL.split('@')[-1]}")
28
+ engine = create_engine(
29
+ DATABASE_URL,
30
+ pool_pre_ping=True,
31
+ pool_recycle=300
32
+ )
33
  SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
34
 
35
  Base = declarative_base()
tmp_test_delete.py ADDED
@@ -0,0 +1,48 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ from backend.database import SessionLocal, SpecDocument, Asset, SessionModel, Collection
3
+ import sys
4
+
5
+ def test_delete(spec_id):
6
+ db = SessionLocal()
7
+ try:
8
+ # Check before
9
+ spec = db.query(SpecDocument).filter(SpecDocument.id == spec_id).first()
10
+ if not spec:
11
+ print(f"Spec {spec_id} not found.")
12
+ return
13
+
14
+ print(f"Deleting spec: {spec.name} ({spec_id})")
15
+
16
+ # Call the same logic as the API (simplified)
17
+ assets = db.query(Asset).filter(Asset.spec_id == spec_id).all()
18
+ for asset in assets:
19
+ db.delete(asset)
20
+
21
+ sessions = db.query(SessionModel).filter(SessionModel.spec_id == spec_id).all()
22
+ for sess in sessions:
23
+ sess.spec_id = None
24
+
25
+ collections = db.query(Collection).filter(Collection.spec_id == spec_id).all()
26
+ for col in collections:
27
+ col.spec_id = None
28
+
29
+ db.delete(spec)
30
+ db.commit()
31
+ print("Successfully deleted spec and handled dependencies.")
32
+
33
+ except Exception as e:
34
+ print(f"Error during delete: {e}")
35
+ db.rollback()
36
+ finally:
37
+ db.close()
38
+
39
+ if __name__ == "__main__":
40
+ # Get the first spec ID to test
41
+ db = SessionLocal()
42
+ first_spec = db.query(SpecDocument).first()
43
+ db.close()
44
+
45
+ if first_spec:
46
+ test_delete(first_spec.id)
47
+ else:
48
+ print("No specs found to delete.")