Load the classifier from Devseis/devseis-ai-act-classifier-v6 (pinned) instead of the bundled copy
dd15de2 verified | title: Caveat — AI Act risk check experiment | |
| emoji: ⚖️ | |
| colorFrom: red | |
| colorTo: indigo | |
| sdk: static | |
| pinned: false | |
| license: cc-by-4.0 | |
| models: | |
| - Devseis/devseis-ai-act-classifier-v6 | |
| - Devseis/eu-ai-act-classifier-experiment | |
| datasets: | |
| - Devseis/devseis-ai-act-classifier-v6-data | |
| - Devseis/eu-ai-act-risk-classification | |
| short_description: EU AI Act risk check, plus a beginner course on how it works | |
| thumbnail: https://huggingface.co/spaces/Devseis/Caveat/resolve/main/og-image.png | |
| # Caveat — two experimental AI Act classifiers, compared live | |
| Caveat is a live demo of the experiment that started at | |
| [`Devseis/eu-ai-act-classifier-experiment`](https://huggingface.co/Devseis/eu-ai-act-classifier-experiment) | |
| (versions 1–3). The current model is **Devseis AI Act Classifier v6**: | |
| [`Devseis/devseis-ai-act-classifier-v6`](https://huggingface.co/Devseis/devseis-ai-act-classifier-v6) | |
| (weights, card, evaluation) with data at | |
| [`Devseis/devseis-ai-act-classifier-v6-data`](https://huggingface.co/datasets/Devseis/devseis-ai-act-classifier-v6-data). | |
| The page loads a runtime copy from `models/devseis-ai-act-classifier-v6/`. | |
| It is not a compliance tool — it exists to show, on real live output, why | |
| neither approach should be trusted on its own. This file briefs a new | |
| session (human or AI) on what exists and how it works. | |
| **Full build history, challenges and fixes, rules and open items: [`HANDBOOK.md`](HANDBOOK.md).** | |
| ## What it is | |
| A single static page with four tabs (the URL hash selects one: `#results`, `#story`, `#learn-<n>`): | |
| - **Check**: type or pick one of 8 example use cases. Both models run live, side by side, and the | |
| page says whether they agree. Each card shows the model's name, size and where it runs. | |
| - **Results**: the v6 numbers (headline tiles, recall per tier, a comparison with earlier versions | |
| including confidence intervals), the known weak spots, and what runs when you press Check. | |
| - **The story**: a 12-chapter timeline from v1 to v6. Each chapter gives what was done, what | |
| happened, the key number and the lesson, including the mistakes. | |
| - **Learn**: a 14-lesson plain-language course for non-technical readers, with analogies, real | |
| examples from this project, interactive widgets, a quiz per lesson, a build-it-yourself checklist | |
| and a searchable 46-term glossary. Underlined terms anywhere on the page open a definition. | |
| Progress, the theme and the checklist are kept in the visitor's `localStorage`. | |
| **Privacy (keep it this way):** Microsoft Clarity loads **only after the visitor clicks "Accept analytics"** in the | |
| consent banner (choice kept 12 months; "Analytics settings" in the footer to change it; Clarity cookies are | |
| deleted unless consent is given). The font is self-hosted in `fonts/` (SIL OFL), so no request goes to Google. transformers.js and the ONNX Runtime | |
| WebAssembly engine are self-hosted in `vendor/transformers-3.0.0/` (byte-identical to npm 3.0.0; `env.backends.onnx.wasm.wasmPaths` | |
| points there), so the page makes no third-party CDN requests. | |
| `privacy.html` documents every data flow; update it whenever a third-party service is added. | |
| ## Architecture — genuinely $0/month | |
| - **Frontend**: this HF Space (`Devseis/Caveat`), `sdk: static` — plain | |
| HTML/CSS/vanilla JS, no framework, no build step. | |
| - **Part 1 (fine-tuned classifier)**: runs **entirely in the visitor's | |
| browser**. No server call. Devseis AI Act Classifier v6 (fine-tuned from | |
| `nlpaueb/legal-bert-small-uncased` on `data_v6_systems`) is loaded from its repo | |
| [`Devseis/devseis-ai-act-classifier-v6`](https://huggingface.co/Devseis/devseis-ai-act-classifier-v6), pinned to | |
| commit `7c594f6` (`MODEL_REVISION` in `index.html`), so the repo's download count shows how many browsers ran the | |
| check (the browser caches the files after the first use). [`@huggingface/transformers`](https://github.com/huggingface/transformers.js) | |
| 3.0.0 loads it with `dtype: "q8"`, | |
| which fetches `onnx/model_quantized.onnx` (~36MB, int8), and runs it with WASM. | |
| The int8 file matches the PyTorch model on 98.6% of test rows under | |
| `onnxruntime-web`. The full model card is on the Hub, at `Devseis/devseis-ai-act-classifier-v6`. | |
| Earlier versions (v1–v3, DistilBERT) are documented in | |
| `Devseis/eu-ai-act-classifier-experiment`. | |
| - **GPAI rule**: `gpai_rule.js` runs before the classifier. Descriptions of a | |
| provider training or releasing a general-purpose model get `gpai` or | |
| `gpai_systemic_risk` from the rule (compute figure > 10^25 FLOPs, or a | |
| Commission designation), and the classifier is skipped. "Built on someone | |
| else's model" gets a note, and the classifier decides the tier. | |
| - **Part 2 (reasoning check)**: a Cloudflare Worker, | |
| `devseis-aiact-reasoning-proxy` | |
| (`devseis-aiact-reasoning-proxy.umer-wasim.workers.dev`), source lives | |
| outside this repo (deployed via `wrangler`). Calls Cloudflare Workers AI | |
| (`@cf/meta/llama-3.1-8b-instruct-fp8`, free tier) fresh on every request, | |
| with the EU AI Act reference rules given as grounding context in the | |
| system prompt — no training, no fine-tuning, no stored state. | |
| **No API keys, no paid services, no per-request cost either side.** | |
| ## Why two different serving strategies | |
| `Devseis/eu-ai-act-classifier-experiment`'s free HF-hosted Inference API | |
| does not support arbitrary custom fine-tuned models | |
| (`"Model not supported by provider hf-inference"`, confirmed when this was | |
| built) — so Part 1 had to move client-side instead. This Space's org | |
| (`Devseis`) also cannot host `gradio`/`docker` Spaces on the free tier | |
| (only `static` — confirmed by a `402 Payment Required` when this was | |
| tried), which is the other reason Part 1 couldn't be a small Python | |
| backend. In-browser ONNX inference is the only genuinely free path left | |
| for a custom classifier under these constraints. | |
| ## Known limitation of Part 2 specifically | |
| The original "Part 2" reasoning check (documented in the model repo) was | |
| done by a much larger model (Claude Sonnet 5) reasoning directly, and | |
| flagged one case (an election-prediction scenario) as a genuine boundary | |
| call between tiers. The free `llama-3.1-8b-instruct-fp8` used **live, in | |
| this Space** is meaningfully smaller. In testing during development it | |
| got the core tier right on that same case but did not reliably surface | |
| the nuance — a concrete example of why "a bigger model would probably do | |
| better" is not the same as "this free model does better." Don't assume | |
| Part 2's live output here matches the quality documented in the write-up. | |
| ## Files | |
| - `index.html` — the app. | |
| - `privacy.html` — privacy notice with an analytics on/off control. | |
| - `fonts/` — self-hosted Space Grotesk (`fonts.css`, woff2 files, `OFL.txt` licence). | |
| - `vendor/transformers-3.0.0/` — self-hosted transformers.js and ONNX Runtime Web engine, with licences (see its README). | |
| - `gpai_rule.js` — the GPAI rule (tests live outside this repo, in `caveat-gpai/`). | |
| - `models/devseis-ai-act-classifier-v6/` — Part 1's model: config, tokenizer, | |
| `onnx/model_quantized.onnx`, and the model card (`README.md`). | |
| This folder is a runtime copy of `Devseis/devseis-ai-act-classifier-v6` and is licensed | |
| **CC BY-SA 4.0** (it is fine-tuned from LEGAL-BERT-SMALL; attribution in its README). Data and | |
| evaluation reports are in the model and dataset repos linked above. | |
| - `manifest.json`, `privacy.html`, `devseis-logo.png`, `icon-192.png`, | |
| `icon-512.png` — shared assets, same pattern as `Nox`. | |
| ## Deploying changes | |
| Frontend: plain static files, push to this repo like any other Space. | |
| Part 2's Worker: edit and `wrangler deploy` from its own source (not | |
| tracked in this repo). Part 1's model: retrain and re-export | |
| (the `model-comparison/` project: `train_v4.py`, then `export_v4.py`), then | |
| replace the files in `models/devseis-ai-act-classifier-v6/` (or add a new folder and | |
| change the model name in `index.html`). The `.onnx` file is stored with Git LFS. | |