api/README.md

# api/ -- FastAPI backend

Wraps `redactor/` (the same engine `scripts/redact.py` uses) behind an
upload-a-file, poll-for-progress, download-the-result HTTP API, plus a
minimal static page to try it by hand.

## Run it

```bash
pip install -r requirements.txt          # includes fastapi/uvicorn
python -m spacy download en_core_web_trf # if you haven't already
uvicorn api.main:app --reload
```

Open http://127.0.0.1:8000 for the upload page, or http://127.0.0.1:8000/docs
for interactive API docs (auto-generated by FastAPI).

## Why this is a job queue, not a synchronous request

Redacting a large document can take real time (see EVALUATION.md's
runtime numbers with `en_core_web_trf`). Holding an HTTP request open for
minutes is fragile -- browsers, proxies, and most hosts time out
long-idle connections regardless of which model is running underneath.
So the API is async by design: submit a job, get an id back immediately,
poll a status endpoint for progress, fetch the result once it's done.
The upload page just wraps this in a progress bar via `setInterval`
polling every second (see `static/index.html`).

## Endpoints

| Method | Path | What |
|---|---|---|
| POST | `/api/jobs` | Upload a `.docx` (multipart form field `file`). Returns `{job_id, status}` immediately -- the actual redaction runs on a background thread. |
| GET | `/api/jobs/{job_id}` | Poll status: `{status, units_done, units_total, progress_percent, estimated_seconds_remaining, redactions_made, error}`. `status` is one of `queued`/`running`/`done`/`failed`. |
| GET | `/api/jobs/{job_id}/download` | The redacted `.docx`, once `status == "done"` (409 otherwise). |
| GET | `/api/jobs/{job_id}/log` | The `(category, original_value, fake_value)` CSV audit trail for that job. |
| GET/POST/DELETE | `/api/config/categories[/{name}]` | List / add a regex-based category / remove a category. See README.md's "Customizing" section. |
| GET/POST/DELETE | `/api/config/structural-columns[/{index}]` | List / add / remove a personnel-table column rule. |
| GET/POST/DELETE | `/api/config/exclude-list[/{term}]` | List / add / remove an ORGANIZATION deny-list entry. |

Every `/api/config/*` mutation calls `RedactionEngine.reload_categories()`
on the shared engine instance afterward, so it takes effect on the next
job immediately -- without restarting the server or reloading the
transformer model.

## Module map

```
api/
  main.py              FastAPI app + lifespan (loads RedactionEngine ONCE at startup, not per-job)
  settings.py          env-tunable knobs: JOBS_DIR, MAX_UPLOAD_BYTES, JOB_RETENTION_HOURS
  schemas.py            Pydantic request/response models (the wire format)
  job_store.py          in-memory Job record + thread-safe JobStore (see its docstring for scaling notes)
  redaction_worker.py   runs one job to completion; the only file here that touches redactor/
  config_editor.py      reads/writes config/pii_categories.yaml + org_exclude_list.txt (ruamel.yaml, comment-preserving)
  routers/jobs.py        the job endpoints
  routers/config.py      the settings endpoints
  static/index.html      upload form + progress bar
  static/settings.html   Categories / Table layouts / Exclude list editor
  static/style.css        shared styling for both pages
  static/app.js           index.html's logic
  static/settings.js      settings.html's logic
```

## Known limitations (demo-scoped, on purpose)

- **In-memory job store.** Job state lives in a Python dict in one
  process. Restart the server and every job's status is gone (the files
  on disk under `api_jobs/` survive, but nothing tracks them anymore).
  Fine for trying this out or a single instance; swap `job_store.py` for
  something Redis/Postgres-backed if you need multi-worker or
  restart-safe job tracking -- routers only talk to `JOB_STORE`, so
  that's the one place that would need to change.
- **No auth.** Anyone who can reach the server can submit jobs and, if
  they know or guess a `job_id` (a UUID4, so not practically guessable,
  but not access-controlled either), download that job's result.
- **No cleanup job yet.** `JOB_RETENTION_HOURS` is defined in
  `settings.py` but nothing currently sweeps old `api_jobs/*` directories
  -- add a scheduled task if this runs somewhere long-lived.
- **Config edits aren't file-locked.** Two people hitting `/api/config/*`
  at the exact same moment could race on the same YAML write. Fine for
  one person customizing settings through the UI; add a lock in
  `config_editor.py` if this is ever multi-user.

## Hosting

This needs real memory -- `en_core_web_trf` is a transformer model, not
the older `en_core_web_lg`. A 512MB free-tier box isn't enough (confirmed
by an actual OOM kill on Render's free tier, not just estimated). See
**`DEPLOYMENT.md`** at the repo root for concrete, cost-compared steps for
both Hugging Face Spaces and Render's paid tiers -- the `Dockerfile` here
works unmodified on either.