| --- |
| title: CLOAK API |
| emoji: 🛡️ |
| colorFrom: blue |
| colorTo: indigo |
| sdk: docker |
| app_port: 7860 |
| pinned: false |
| --- |
| |
| # CLOAK model API (Hugging Face Space) |
|
|
| The models and engines the CLOAK server calls, in one Space with one secret. API only: there is |
| no UI and no interactive API docs (`/docs` and `/openapi.json` are switched off on purpose). |
|
|
| | Path | What it does | Server caller | |
| |---|---|---| |
| | `POST /detect` | Personal data in text: nested NER, `Wardline/CLOAK_V3.12_onnx`, ONNX INT8 on CPU | `server/src/detection/pii.ts` | |
| | `POST /file/classify` · `POST /file/analyze` | What kind of document a text or file is (26 classes) | `server/src/detection/classify.ts` | |
| | `POST /copilot/parse` · `GET /copilot/examples` | Policy Copilot: a policy described in plain English → drafts the policy forms would save | `server/src/modules/org/routes/copilot.ts` | |
| | `GET /health` | Status of all three, with no secret needed and none revealed | Settings → backend status | |
|
|
| Every path except `/` and `/health` needs `Authorization: Bearer <API_SECRET>`. |
|
|
| ## Deploy |
|
|
| 1. Push this folder to the Space repo. It is a **Docker Space** (`sdk: docker` above): HF builds |
| `Dockerfile`, which pins Python 3.11.9 and runs as uid 1000. If the Space was created as a |
| Gradio Space, the front-matter above switches it; HF rebuilds on push. |
| 2. Space secrets: |
| - `API_SECRET`: a long random value (32+ characters). CLOAK sends it as the Bearer token. |
| If unset it falls back to `HF_TOKEN`; that works, but puts the HF account token in CLOAK's |
| settings, so don't. |
| - `HF_TOKEN`: read access to the model repos. |
| - Optional: `MODEL_REPO`, `MODEL_FILE`, `DOCCLASS_*` (below), `LOG_LEVEL`. |
| 3. In the CLOAK dashboard → Settings: **Backend URL** = the Space URL |
| (`https://<space>.hf.space`), **API key** = `API_SECRET`. The Policy Copilot uses the same |
| two settings; nothing else to configure. |
| |
| The same image runs on a customer's own host, for installs that must not send text or files to |
| Hugging Face: |
|
|
| ```bash |
| docker build -t cloak-model-api . |
| docker run -p 7860:7860 -e API_SECRET=... -e HF_TOKEN=... cloak-model-api |
| ``` |
|
|
| Point CLOAK's Backend URL at it. (`POLICY_COPILOT_URL` on the server can point the copilot at a |
| different host, if it ever has to run apart from the models.) |
|
|
| ## Security |
|
|
| - One Bearer secret for every data path, compared in constant time. |
| - Bodies are bounded before they are read: `/detect` by `MAX_INPUT_CHARS`, `/copilot/*` at 2 MB |
| (text at most 2,000 characters); `/file/*` by docclass's own limits. |
| - Failures are real HTTP errors (401, 411, 413, 422, 503) with generic messages; the reason goes |
| to the Space log only. |
| - Responses carry `Cache-Control: no-store`, `nosniff`, `DENY` framing and no referrer. |
| - Nothing is stored. The admin's copilot text is never logged: only request id, outcome and timing. |
| - The HF token is passed to each download and never written to disk. |
| - The service user cannot modify the code in the image; only its model cache is writable. |
|
|
| ## `/detect` contract (must stay in sync with `server/src/detection/pii.ts`) |
|
|
| - `POST /detect` · `{"text": "..."}` |
| → `{entities: [{type, text, start, end, confidence}], nested, redacted_text, model, processing_time_ms}` |
| - Failures are never 200-with-empty, so DSPM records scan **errors**, not false "clean". |
| - `MAX_INPUT_CHARS = 80_000` here must be ≥ `DETECT_MAX_CHARS` in `pii.ts`. |
| - `GET /health` → `{status, model, engine, threads, classes, max_input_chars, auth_configured, docclass, copilot}`. |
|
|
| ## Document classification (`docclass.py`) |
|
|
| `/detect` finds personal details in text. The `/file` paths say what KIND of document a file is |
| (26 classes, including `credentials`) and suggest an action. CLOAK stores both. |
|
|
| | Path | Body | Runs | |
| |---|---|---| |
| | `POST /file/classify` | `{"text": "...", "column_headers": [...]}` | Layer 3 on text CLOAK already extracted (~5 ms) | |
| | `POST /file/analyze` | raw bytes, `Content-Type: application/octet-stream`, `X-Filename: <name>` | Layers 1+2+3: real type from bytes, archive contents, OCR for images and scanned PDFs | |
|
|
| - **Python 3.11.9 only** (the Dockerfile pins it): the model is a scikit-learn pickle built on |
| 3.11.9, and `requirements.txt` pins its exact libraries. Other versions can load it and |
| silently behave differently. |
| - The model is fetched from `DOCCLASS_REPO` (default: `MODEL_REPO`) / `DOCCLASS_FILE` (default |
| `layer3_25class.joblib`) and **refused unless its SHA-256 matches** `DOCCLASS_SHA256` (pinned in |
| `docclass.py`). Upload a new model → update the hash in the same change. |
| - If it can't load, `/file/*` returns 503 and the other paths are unaffected. |
| - Limits: 25 MB per file, archives 3 levels / 500 entries / 150 MB unpacked, at most 2 OCR jobs at |
| once (`DOCCLASS_OCR_CONCURRENCY`), first 3 pages of a scanned PDF (`DOCCLASS_PDF_OCR_PAGES`). |
| - Always load the model through `Layer3DocType`, never the raw pickle. |
|
|
| ## Policy Copilot (`copilot/`, `copilot_api.py`) |
| |
| Turns "block Aadhaar and PAN in ChatGPT for interns" into the exact body the policy form would |
| save, with the questions and warnings an admin needs before saving. Rules and vocabulary only: |
| no model, no network, a few milliseconds per request. |
| |
| - **Stateless and write-free.** The CLOAK server sends the tenant's context with each request |
| (groups, rules, detectors, block lists, connectors, policies) and saves nothing until the admin |
| confirms in the dashboard, through the normal policy routes, with the admin's own permissions. |
| Drafts are always saved switched off. |
| - **Adds to existing policies instead of duplicating them.** When a request overlaps a policy that |
| already exists (same group, channel or mail service), the admin chooses to add it there or create |
| a separate one; "add printing to the USB Lock policy" picks it directly. Only a list of changes is |
| sent (switch on, add to a list, make the action stricter), applied to the policy as it is at save |
| time. A merge never removes anything, and never changes the name, group or on/off state. |
| - **Validated twice.** Every draft passes `copilot/check.py` against the form's schema before it |
| leaves the Space; the server then accepts only save steps to the three policy-creation routes. |
| - **Platform facts are synced, not hand-copied.** Built-in rules, the 42 built-in block lists, AI |
| sites and every form's defaults come from the cloakv2 source: |
| |
| ```bash |
| node tools/sync_platform.mjs .. # from hf-space/, with cloakv2 as the parent |
| ``` |
| |
| Run it whenever a policy form, preset or block list changes, review `git diff copilot/platform`, |
| and redeploy the Space. Phrases live in `copilot/lexicon/vocab.yaml`; the loader refuses a phrase |
| that points at a rule, list or setting the snapshot doesn't have. |
| - If the copilot can't load, `/copilot/*` returns 503, the dashboard tells the admin to use the |
| form, and the other paths are unaffected. |
| |
| Try it in a terminal: `python -m copilot.cli "block usb for interns"`. |
| |
| ## Files |
| |
| ``` |
| app.py PII model, /detect, /health, shared auth and limits; mounts the other two |
| docclass.py document pipeline (type from bytes, archives, extraction, OCR, Layer 3) |
| copilot_api.py /copilot routes over copilot/ |
| copilot/ the Policy Copilot engine; platform/ holds the synced snapshot |
| tools/ sync_platform.mjs (not shipped in the image) |
| Dockerfile the Space image; .dockerignore keeps tools and docs out |
| ``` |
| |
