Spaces:
Sleeping
Sleeping
| # Load testing | |
| A [Locust](https://locust.io/) load test for the Autonomous Executive Email | |
| Copilot API. It exercises the hot read paths and a representative mutating flow | |
| so you can measure latency and throughput before a deploy or a capacity change. | |
| Locust is an **optional development dependency** and is deliberately *not* in | |
| `requirements.txt` (it pulls in a web UI and async stack the runtime does not | |
| need). Install it on demand. | |
| ## Install | |
| ```bash | |
| pip install locust | |
| ``` | |
| ## Run the server under test | |
| In one terminal, start the API the usual way (see the README quickstart): | |
| ```bash | |
| uvicorn env.api:app --host 0.0.0.0 --port 8000 | |
| ``` | |
| The same app serves on container port `7860` in Docker; point `--host` at | |
| whatever address/port the server is actually listening on. | |
| ## Run the load test | |
| From the repository root: | |
| ```bash | |
| # Interactive web UI on http://localhost:8089 | |
| locust -f scripts/loadtest/locustfile.py --host http://localhost:8000 | |
| ``` | |
| Open <http://localhost:8089>, set the number of users and spawn rate, and start | |
| the run. For an unattended (headless) run with a fixed shape and duration: | |
| ```bash | |
| locust -f scripts/loadtest/locustfile.py --host http://localhost:8000 \ | |
| --headless --users 50 --spawn-rate 10 --run-time 2m | |
| ``` | |
| - `--users` — peak number of concurrent simulated clients. | |
| - `--spawn-rate` — how many users to add per second until the peak is reached. | |
| - `--run-time` — total duration (e.g. `30s`, `2m`, `1h`). | |
| Against a non-local target, set `--host` accordingly, e.g. | |
| `--host http://localhost:7860` (Docker) or a remote base URL. | |
| ## What it does | |
| Each simulated user (`EmailCopilotUser`) waits 0.5–2.0s between tasks and picks | |
| a task by weight: | |
| | Task | Endpoint(s) | Weight | Notes | | |
| |------|-------------|-------:|-------| | |
| | `health` | `GET /health` | 10 | Cheapest path; mirrors the healthcheck/probes. | | |
| | `tasks` | `GET /tasks` | 5 | Task metadata + JSON schemas. | | |
| | `state` | `GET /state` | 5 | Current environment snapshot. | | |
| | `metrics` | `GET /metrics` | 3 | Prometheus text exposition. | | |
| | `reset_then_step` | `POST /reset`, `POST /step` | 2 | Representative mutating flow. | | |
| On start, each user resets the environment so `/state` and `/step` have an | |
| episode to act on. The mutating flow resets, reads the returned observation, and | |
| classifies the first email (falling back to a `prioritize` action when no emails | |
| remain). The read paths are weighted far above the writes because that matches | |
| real traffic (probes, scrapers, dashboards) and keeps SQLite — the zero-config | |
| default backend — from being the dominant factor in the numbers. | |
| > Note: if the server has `API_AUTH_TOKEN` set, the mutating routes (`/reset`, | |
| > `/step`) require a token. Either run the load test against a server with auth | |
| > disabled, or extend the requests in `locustfile.py` to send an | |
| > `Authorization: Bearer <token>` header. Likewise, leave | |
| > `RATE_LIMIT_PER_MINUTE` at `0` (the default) so the limiter does not skew | |
| > results. | |
| ## Reading the results | |
| - **Web UI** — the *Statistics* tab shows, per named request, the request count, | |
| failure count, median / 95th / 99th percentile latency, and current RPS. The | |
| *Charts* tab plots RPS, response times, and user count over time. The | |
| *Failures* tab lists any non-2xx/transport errors with a reason. | |
| - **Headless** — Locust prints an aggregated table on exit. Add | |
| `--csv results/loadtest` to also write | |
| `results/loadtest_stats.csv`, `_stats_history.csv`, and `_failures.csv` for | |
| later analysis or CI artifacts. | |
| Focus on the **p95 / p99 latency** columns (tail latency, what users feel) and | |
| the **failure rate**, not just the mean. A rising p99 while RPS is flat usually | |
| means you have found a bottleneck (often the SQLite write path under the | |
| `/reset` + `/step` flow). | |
| ## Suggested target SLOs | |
| These are starting points for a single-instance deployment on modest hardware; | |
| tune them to your environment and re-baseline after infra changes. | |
| | Metric | Target | | |
| |--------|--------| | |
| | `GET /health` p99 latency | < 20 ms | | |
| | `GET /tasks`, `GET /state` p95 latency | < 100 ms | | |
| | `GET /metrics` p95 latency | < 150 ms | | |
| | `POST /reset` + `POST /step` p95 latency | < 400 ms | | |
| | Error rate (all endpoints) | < 0.1% | | |
| | Sustained throughput before p95 SLOs breach | record as the capacity number | | |
| A useful capacity exercise: ramp `--users` until any p95 SLO is breached or the | |
| error rate climbs above the threshold, then record the RPS just below that point | |
| as the safe sustained throughput for one instance. | |