Load the classifier from Devseis/devseis-ai-act-classifier-v6 (pinned) instead of the bundled copy
dd15de2 verified | # Devseis · Caveat — project handbook | |
| > Published copy of the Devseis project handbook. Folder paths in sections 1 and 6 (`~/Documents/hf/...`) refer to the | |
| > Devseis development workspace, not to files in this Space. | |
| **Read this first in any new session.** It explains what exists, how it was built, what went wrong and how it was | |
| fixed, the rules we work by, and what could come next. Last updated: 1 October 2026. | |
| **Caveat** (https://huggingface.co/spaces/Devseis/Caveat) is a free, static web page by Devseis (a Luxembourg | |
| cybersecurity, data-protection and AI-governance consultancy). A visitor describes an AI system; two models classify it | |
| into an EU AI Act risk tier side by side; the page shows honest results, the build story, and a 14-lesson course with a | |
| certificate. It is a **demo and marketing tool, not a compliance tool and not legal advice**. | |
| --- | |
| ## 1. Map: what lives where | |
| ### On the Hugging Face Hub (organisation `Devseis`) | |
| | Repo | What it is | Licence | | |
| |---|---|---| | |
| | [`spaces/Devseis/Caveat`](https://huggingface.co/spaces/Devseis/Caveat) | The live page (static Space). Loads the model from `Devseis/devseis-ai-act-classifier-v6` at commit `7c594f6` (`MODEL_REVISION`); the old copy in `models/` is no longer used. | page CC BY 4.0; model folder CC BY-SA 4.0 | | |
| | [`Devseis/devseis-ai-act-classifier-v6`](https://huggingface.co/Devseis/devseis-ai-act-classifier-v6) | **Current model**: weights, int8 ONNX, card, evaluation reports | **CC BY-SA 4.0** (required, see §5) | | |
| | [`datasets/Devseis/devseis-ai-act-classifier-v6-data`](https://huggingface.co/datasets/Devseis/devseis-ai-act-classifier-v6-data) | Current training/val/test data | CC BY 4.0 | | |
| | `Devseis/devseis-ai-act-classifier-v5` (+ `-v5-data`) | Previous version, removed 2026-10-12: its test rows are all in v6-data (`test_v6.csv`) and its change notes in `CHANGES_v5.md` there | — | | |
| | `Devseis/eu-ai-act-classifier-experiment` | Versions 1–3 (DistilBERT), kept for history, with pointer + correction note | CC BY 4.0 | | |
| | `datasets/Devseis/eu-ai-act-risk-classification` | Reference table of tiers (the original ground truth) | CC BY 4.0 | | |
| | `spaces/Devseis/README` | The organisation profile card (also local copy `devseis-org-card.md`) | — | | |
| | Collection "EU AI Act & AI Governance Toolkit — curated by Devseis" | Lists Caveat, v6 model and data, other Devseis datasets | — | | |
| ### On this Mac (`~/Documents/hf/`) | |
| | Folder / file | What it is | | |
| |---|---| | |
| | `model-comparison/` | **Training project.** `README.md` there is the original training brief (rules + steps). | | |
| | `model-comparison/data*/` | Versioned datasets: `data/` (v2), `data_v3/`, `data_v4/` (6 labels, reference only), `data_v4_systems/`, `data_v5_systems/`, `data_v6_systems/` — each with `CHANGES.md`. **Never edit these.** | | |
| | `model-comparison/relabel_audit/` | Scripts that built each data version (`apply_relabels.py`, `extend_v4/5/6.py`) and the lists of new scenarios (`new_scenarios_v*.csv`). | | |
| | `model-comparison/train_v4.py` | Train/tune/final runs for any data version (`--data --suffix --out`). | | |
| | `model-comparison/report_v4.py` | Builds `results_v*/report.md` and a previous-version comparison report from saved predictions. | | |
| | `model-comparison/export_v4.py` | ONNX export + int8 quantisation + agreement checks (picks the setting on validation). | | |
| | `model-comparison/threshold.py` | Studies the "uncertain" confidence threshold on validation. | | |
| | `model-comparison/results_v4/ v5/ v6/` | Per-seed predictions, reports, tuning file, saved seed-42 model. | | |
| | `model-comparison/export/`, `export_v6/` | Browser exports and model/data cards for v5 and v6. | | |
| | `Caveat-space/` | Local copy of the Space. **It lags the live Space** (pushes go through the API) — resync before editing: `git fetch && git reset --hard origin/main`. | | |
| | `caveat-gpai/` | GPAI rule (`gpai_rule.js`) + its tests (`node test.mjs` → must print `655/655 correct`). | | |
| | `eu-ai-act-risk-classification/` | Downloaded reference dataset; `synthetic/test.csv` is the **out-of-distribution (OOD) test set** — never train on it. | | |
| | `OJ_L_202401689_EN_TXT.pdf` | The AI Act (Official Journal). Commission draft guidelines PDFs sit next to it. | | |
| | `caveat-demo/` | Demo copies: certificate, printable summary, social preview image. | | |
| | `tools/browser-tests/` | Reusable Chrome/jsdom tests for the page (layout, consent, real model, network hosts, certificate, int8 agreement). See its README. | | |
| --- | |
| ## 2. How Caveat works (what runs when someone presses "Check") | |
| 1. **GPAI rule** (`gpai_rule.js`, runs first in the browser). Descriptions of a *provider* training/releasing a | |
| general-purpose model are decided by rule: > 10^25 FLOPs or Commission designation → `gpai_systemic_risk`, | |
| otherwise `gpai`. "Built on someone else's model" → note only; the classifier decides. Why a rule: small text | |
| models read exponents badly; the law here is a number. | |
| 2. **Trained model** — Devseis AI Act Classifier v6 (legal-bert-small, 35M params, int8 ONNX 35.5 MB) runs **in the | |
| visitor's browser** via transformers.js 3.0.0 + ONNX Runtime Web, both **self-hosted** in `vendor/`. Downloaded | |
| only when someone shows intent (focus the text box, tap an example, press Check). Shows "Uncertain" when the top | |
| probability is < 50%. | |
| 3. **Reasoning AI** — Llama 3.1 8B Instruct on Cloudflare Workers AI (free tier), via the Worker | |
| `devseis-aiact-reasoning-proxy.umer-wasim.workers.dev` (source is **not** in this workspace), given the AI Act | |
| rules as a system prompt. Not trained, never independently validated, the weakest part. | |
| 4. The page shows both verdicts, whether they agree, an expert-call box (pre-filled email to umer.wasim@devseis.com, | |
| "free 30-min AI Act readiness call"), and a printable one-page summary. | |
| Other page features: Results tab (numbers, comparison, model-vs-expert card, verified AI Act key dates, "Built by | |
| Devseis"), Story tab (12-chapter build timeline), Learn tab (14 lessons, glossary of 46 terms, quizzes, a 12-question | |
| final assessment with pass mark 9/12 that unlocks a certificate drawn in the browser, LinkedIn add/share using company | |
| ID 12603388), consent banner, social preview image. | |
| --- | |
| ## 3. How it was built — the version history | |
| | Version | Data | Model | Key result | Lesson | | |
| |---|---|---|---|---| | |
| | v1 | 401 rows from only 64 scenarios | DistilBERT | 61% accuracy; 60% of banned practices called "high risk" | Too few distinct examples → memorising | | |
| | v2–v3 (Parts 2–3) | 268 scenarios from the real legal text + Commission examples | DistilBERT | 86% → 71% with more data; real bug: script kept the *last* epoch. Fixed by selecting on **validation macro-F1** → 76.5%, 90% banned caught | Model selection matters as much as data | | |
| | audit (v3 data) | 6 wrong labels fixed (3 in test), 661 ungrammatical rows fixed | — | Published 76.5% was really 74.7% | Check the answer key | | |
| | backbone comparison (v2, v4) | same splits, 3 seeds | DistilBERT vs legal-bert-small vs SetFit | SetFit: best accuracy but called 16 banned cases "minimal"; legal-bert: much higher banned recall, half the size → chosen | One headline number hides the mistake that matters | | |
| | v4 | +37 scenarios (Annex I, Omnibus bans, Art 6(3)) | legal-bert-small | 95% banned caught, but phone/voice agents 0/18 | Coverage gap: every chatbot example typed | | |
| | v5 | +23 (voice/phone agents, consented media) | legal-bert-small | 98% banned, 0 severe errors; OOD 76%, missed face-scraping | Find the gap, add examples, re-measure | | |
| | **v6 (live)** | +39 (harmless look-alikes, Art 5(1)(e) wording, consented media) **+ a second writing style per training scenario** | legal-bert-small | **99% banned, 0 severe; 81.5% acc on v5 rows; OOD 80%**; face-scraping fixed | Write examples the way people write | | |
| v6 numbers (3 seeds, v5 test rows = 198 rows / 66 scenarios): accuracy 0.815, macro-F1 0.843 (CI 0.758–0.912), | |
| banned recall 0.990, limited 0.963, minimal 0.719, high 0.761. Full v6 test (210 / 70): 0.822 / 0.850 / 0.991. | |
| **Gains over v5 are consistent but not statistically established.** Shipped checkpoint = seed 42 (fixed in advance). | |
| --- | |
| ## 4. Challenges and how they were solved | |
| ### Data and evaluation | |
| | Challenge | Solution | | |
| |---|---| | |
| | Rows are rewordings, so row counts overstate the sample size | Always count **scenarios**; split by scenario; bootstrap CIs by scenario | | |
| | Leakage between splits | Scenario-grouped splits; char-4gram Jaccard ≥ 0.45 check against val/test/OOD for every new scenario | | |
| | Wrong labels (incl. in test) | Audit against the Act (`relabel_audit.py`); corrected numbers published, old cards corrected | | |
| | Seed-to-seed swings of up to 9 points | 3 seeds per configuration, mean ± std, paired bootstrap; never call a winner on noise | | |
| | Tempting to tune on test | Rule: choose on validation only; test once per final config; OOD never used for selection (we even kept seed 42 when another seed scored better on OOD) | | |
| | Over-flagging harmless systems | v6 look-alike scenarios (+ paired high-risk counterparts) — improved only modestly (68% → 72%) | | |
| | One template style → weak on unusual wording | v6 style rows (question, pitch, tender…) → OOD 76% → 80% | | |
| | Confidence threshold for "uncertain" | Pre-set rule (≥ 90% answered accuracy on validation) found **no** threshold: errors are mostly confident. Chosen with Umer: flag only genuinely split predictions (< 50%) | | |
| ### Machine and tooling | |
| | Challenge | Solution | | |
| |---|---| | |
| | Intel Mac: PyTorch stops at 2.2.2 | Python 3.11 venv in `model-comparison/.venv` with pinned versions (`requirements.txt`) | | |
| | Two PyTorch jobs at once → load average 120, both stall | Never run torch jobs in parallel; one training job at a time | | |
| | Mac sleeping during long runs | Keep it awake; runs resume after wake | | |
| | SetFit `predict` returns a list; safetensors refuses non-contiguous legal-bert weights; a variable-name clash | Fixed in `train_v4.py` (contiguous weights before save, etc.) | | |
| | `HF_HUB_OFFLINE=1` breaks when a repo's latest commit isn't cached | Run without offline mode when loading the published v2 model | | |
| | No git-lfs installed | Push with `huggingface_hub.create_commit(..., parent_commit=<head>)` — also prevents overwriting concurrent changes | | |
| | Pushing the local README would drop edits made via the API (e.g. `short_description`, `thumbnail`) | Always rebuild the README from the **live** version before pushing | | |
| | EUR-Lex blocks curl (202 challenge) | Open it with real Chrome via `puppeteer-core` (`npm i puppeteer-core`, use `/Applications/Google Chrome.app`) | | |
| | Headless Chrome won't render below 500 px wide | Use Puppeteer device emulation for phone tests | | |
| | jsdom page test needs stubs | `tools/browser-tests/uitest.mjs`: stubs `pipeline`, `env.backends.onnx.wasm`, `gpaiCheck`, `fetch` | | |
| ### Browser deployment | |
| | Challenge | Solution | | |
| |---|---| | |
| | HF Inference API doesn't serve custom models; org can't host gradio/docker on free tier | Static Space + in-browser inference (transformers.js + ONNX) | | |
| | int8 answers differ from PyTorch, and differ by runtime | Try 8 quantisation settings, pick by **validation** agreement; measure under **onnxruntime-web** (what browsers run), one row at a time (dynamic int8 scales per batch). v6: 98.6% in-browser | | |
| | The originally served v2 file agreed only 94.5% | Fixed by the export script's setting selection | | |
| | transformers.js in Node ignores `quantized: true` | Use `dtype: "q8"` | | |
| | Self-hosted WASM engine is served uncompressed by HF's CDN (21.7 MB vs 4.2 MB from jsDelivr) | Accepted for the privacy benefit; documented. Model is lazy-loaded to compensate | | |
| | `[hidden]` overridden by `display:flex` (stuck spinner) | Global `[hidden]{display:none !important}` | | |
| | Canvas taint for the certificate logo | Logo hosted in the Space itself (`devseis-logo.png`) | | |
| ### Legal and privacy (Devseis sells GDPR/AI Act services — the page must be exemplary) | |
| | Challenge | Solution | | |
| |---|---| | |
| | legal-bert-small is **CC BY-SA 4.0**; first card said CC BY 4.0 | Model relicensed CC BY-SA 4.0 + full LEGAL-BERT attribution, change list, no-endorsement line on every copy | | |
| | Omnibus point letters disputed ((ba)/(bb) vs (l)) | Verified in the Official Journal: Regulation (EU) 2026/1744, Art 1(7) → **(ba)/(bb)**, apply 2 Dec 2026; Art 5(1a) scope noted | | |
| | Microsoft Clarity ran without consent; Google Fonts sent IPs to Google; jsDelivr too | Clarity only after "Accept analytics" (consentv2, 12-month choice, cookies deleted otherwise, project `yqfpt9xwwk`); font and code self-hosted; privacy notice rewritten (controller, flows, legal bases, retention, transfers, CNPD) | | |
| | Certificate could look like an accredited qualification | Marked "self-paced learning module — not an accredited qualification"; no verification number; typeset signatory, no fake signature | | |
| | Marketing claims | Every number on the page traced to a saved report; limitations shown alongside | | |
| --- | |
| ## 5. Rules we work by (do not break) | |
| 1. Never edit the versioned data folders; create a new `data_v{N}_systems/` with a build script instead. | |
| 2. Never re-split or move rows between splits; new scenarios get a deliberate split; previous versions' rows stay put. | |
| 3. Choose hyperparameters/checkpoints/thresholds on **validation only**; evaluate on test once; never use the OOD set | |
| for any decision. | |
| 4. Ship a new model only if it matches or beats the live one on banned-practice recall with **no more severe errors** | |
| (banned → minimal/limited), on the previous test rows. | |
| 5. Model files derived from legal-bert-small stay **CC BY-SA 4.0** with attribution. | |
| 6. Ask Umer before public pushes unless he has approved that change; pin pushes to the current head commit. | |
| 7. Every claim on the page or a card must match a saved report; say plainly when a gain is within noise. | |
| 8. Keep the page free of third-party requests before consent; update `privacy.html` whenever a service is added. | |
| --- | |
| ## 6. How to do common tasks | |
| ```bash | |
| cd ~/Documents/hf/model-comparison | |
| # train a data version (3 seeds), reusing the tuned config copied into results_vN/tune/legalbert.json | |
| TOKENIZERS_PARALLELISM=false .venv/bin/python train_v4.py --phase final --models legalbert --data data_v6_systems --suffix v6 --out results_v6 | |
| # reports (full test + previous-version comparison) | |
| .venv/bin/python train_v4.py --phase report --models legalbert --data data_v6_systems --suffix v6 --out results_v6 | |
| # export to int8 ONNX with agreement checks | |
| .venv/bin/python export_v4.py --model-dir results_v6/models/legalbert --out export_v6 --data data_v6_systems --suffix v6 | |
| # GPAI rule tests | |
| cd ../caveat-gpai && node test.mjs # must print 655/655 correct | |
| # preview the page locally | |
| cd ../Caveat-space && python3 -m http.server 8000 # http://localhost:8000 | |
| # browser tests (serve the Space on port 8766 first) | |
| cd ../tools/browser-tests && npm install && node uitest.mjs && node net.mjs http://localhost:8766/ | |
| ``` | |
| New model version checklist: build data (script + `CHANGES.md`, similarity check) → train 3 seeds → reports → shipping | |
| rule → export (≥ ~99% agreement, check under onnxruntime-web) → threshold study → new Hub model + dataset repos (cards | |
| with attribution) → Space runtime folder + `MODEL_ID` + every number on the page (Results, Story, glossary, lessons, | |
| print sheet) → pointers on the previous repos → profile + collection → verify live (real model, network hosts, phone). | |
| --- | |
| ## 7. Open items and ideas for the future | |
| - **Umer to review** the 39 v6 scenarios (`relabel_audit/new_scenarios_v6.csv`), especially minimal-risk calls such as | |
| 1:1 face verification, car-damage claims and campaign leaflet routing. | |
| - **Re-tune the learning rate on v6 validation** (all versions since v4 reuse the v4-tuned 2e-5). | |
| - **Over-flagging** remains the biggest model weakness (minimal recall 0.72–0.74) → more look-alike scenarios. | |
| - **Real-world test set**: 30–50 real system descriptions labelled by Umer — the only true accuracy measure. | |
| - **Stronger reasoning AI** (larger free Cloudflare model, or pass the GPAI rule's result as context); its Worker source | |
| should be brought into this workspace. | |
| - **Calibration**: the model's confidence isn't a reliable warning; temperature scaling or an ensemble could help. | |
| - **French/German versions** for the Luxembourg market. | |
| - **Accessibility/performance audit** (Lighthouse). | |
| - **Review Clarity** events after a few weeks (`cta_book_call`, `print_summary`, `exam_passed`, `certificate_created`). | |
| - Keep the AI Act **key dates** in sync with any further amendments (built from the OJ texts; update `DATES` in `index.html`). | |