Spaces:
Sleeping
Sleeping
File size: 4,518 Bytes
78d9fa0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 | # Load testing
A [Locust](https://locust.io/) load test for the Autonomous Executive Email
Copilot API. It exercises the hot read paths and a representative mutating flow
so you can measure latency and throughput before a deploy or a capacity change.
Locust is an **optional development dependency** and is deliberately *not* in
`requirements.txt` (it pulls in a web UI and async stack the runtime does not
need). Install it on demand.
## Install
```bash
pip install locust
```
## Run the server under test
In one terminal, start the API the usual way (see the README quickstart):
```bash
uvicorn env.api:app --host 0.0.0.0 --port 8000
```
The same app serves on container port `7860` in Docker; point `--host` at
whatever address/port the server is actually listening on.
## Run the load test
From the repository root:
```bash
# Interactive web UI on http://localhost:8089
locust -f scripts/loadtest/locustfile.py --host http://localhost:8000
```
Open <http://localhost:8089>, set the number of users and spawn rate, and start
the run. For an unattended (headless) run with a fixed shape and duration:
```bash
locust -f scripts/loadtest/locustfile.py --host http://localhost:8000 \
--headless --users 50 --spawn-rate 10 --run-time 2m
```
- `--users` — peak number of concurrent simulated clients.
- `--spawn-rate` — how many users to add per second until the peak is reached.
- `--run-time` — total duration (e.g. `30s`, `2m`, `1h`).
Against a non-local target, set `--host` accordingly, e.g.
`--host http://localhost:7860` (Docker) or a remote base URL.
## What it does
Each simulated user (`EmailCopilotUser`) waits 0.5–2.0s between tasks and picks
a task by weight:
| Task | Endpoint(s) | Weight | Notes |
|------|-------------|-------:|-------|
| `health` | `GET /health` | 10 | Cheapest path; mirrors the healthcheck/probes. |
| `tasks` | `GET /tasks` | 5 | Task metadata + JSON schemas. |
| `state` | `GET /state` | 5 | Current environment snapshot. |
| `metrics` | `GET /metrics` | 3 | Prometheus text exposition. |
| `reset_then_step` | `POST /reset`, `POST /step` | 2 | Representative mutating flow. |
On start, each user resets the environment so `/state` and `/step` have an
episode to act on. The mutating flow resets, reads the returned observation, and
classifies the first email (falling back to a `prioritize` action when no emails
remain). The read paths are weighted far above the writes because that matches
real traffic (probes, scrapers, dashboards) and keeps SQLite — the zero-config
default backend — from being the dominant factor in the numbers.
> Note: if the server has `API_AUTH_TOKEN` set, the mutating routes (`/reset`,
> `/step`) require a token. Either run the load test against a server with auth
> disabled, or extend the requests in `locustfile.py` to send an
> `Authorization: Bearer <token>` header. Likewise, leave
> `RATE_LIMIT_PER_MINUTE` at `0` (the default) so the limiter does not skew
> results.
## Reading the results
- **Web UI** — the *Statistics* tab shows, per named request, the request count,
failure count, median / 95th / 99th percentile latency, and current RPS. The
*Charts* tab plots RPS, response times, and user count over time. The
*Failures* tab lists any non-2xx/transport errors with a reason.
- **Headless** — Locust prints an aggregated table on exit. Add
`--csv results/loadtest` to also write
`results/loadtest_stats.csv`, `_stats_history.csv`, and `_failures.csv` for
later analysis or CI artifacts.
Focus on the **p95 / p99 latency** columns (tail latency, what users feel) and
the **failure rate**, not just the mean. A rising p99 while RPS is flat usually
means you have found a bottleneck (often the SQLite write path under the
`/reset` + `/step` flow).
## Suggested target SLOs
These are starting points for a single-instance deployment on modest hardware;
tune them to your environment and re-baseline after infra changes.
| Metric | Target |
|--------|--------|
| `GET /health` p99 latency | < 20 ms |
| `GET /tasks`, `GET /state` p95 latency | < 100 ms |
| `GET /metrics` p95 latency | < 150 ms |
| `POST /reset` + `POST /step` p95 latency | < 400 ms |
| Error rate (all endpoints) | < 0.1% |
| Sustained throughput before p95 SLOs breach | record as the capacity number |
A useful capacity exercise: ramp `--users` until any p95 SLO is breached or the
error rate climbs above the threshold, then record the RPS just below that point
as the safe sustained throughput for one instance.
|