HANDBOOK.md

Devseis · Caveat — project handbook

Published copy of the Devseis project handbook. Folder paths in sections 1 and 6 (~/Documents/hf/...) refer to the Devseis development workspace, not to files in this Space.

Read this first in any new session. It explains what exists, how it was built, what went wrong and how it was fixed, the rules we work by, and what could come next. Last updated: 1 October 2026.

Caveat (https://huggingface.co/spaces/Devseis/Caveat) is a free, static web page by Devseis (a Luxembourg cybersecurity, data-protection and AI-governance consultancy). A visitor describes an AI system; two models classify it into an EU AI Act risk tier side by side; the page shows honest results, the build story, and a 14-lesson course with a certificate. It is a demo and marketing tool, not a compliance tool and not legal advice.

---

1. Map: what lives where

On the Hugging Face Hub (organisation Devseis)

Repo	What it is	Licence
spaces/Devseis/Caveat	The live page (static Space). Loads the model from Devseis/devseis-ai-act-classifier-v6 at commit 7c594f6 (MODEL_REVISION); the old copy in models/ is no longer used.	page CC BY 4.0; model folder CC BY-SA 4.0
Devseis/devseis-ai-act-classifier-v6	Current model: weights, int8 ONNX, card, evaluation reports	CC BY-SA 4.0 (required, see §5)
datasets/Devseis/devseis-ai-act-classifier-v6-data	Current training/val/test data	CC BY 4.0
Devseis/devseis-ai-act-classifier-v5 (+ -v5-data)	Previous version, removed 2026-10-12: its test rows are all in v6-data (test_v6.csv) and its change notes in CHANGES_v5.md there	—
Devseis/eu-ai-act-classifier-experiment	Versions 1–3 (DistilBERT), kept for history, with pointer + correction note	CC BY 4.0
datasets/Devseis/eu-ai-act-risk-classification	Reference table of tiers (the original ground truth)	CC BY 4.0
spaces/Devseis/README	The organisation profile card (also local copy devseis-org-card.md)	—
Collection "EU AI Act & AI Governance Toolkit — curated by Devseis"	Lists Caveat, v6 model and data, other Devseis datasets	—

On this Mac (~/Documents/hf/)

Folder / file	What it is
model-comparison/	Training project. README.md there is the original training brief (rules + steps).
model-comparison/data*/	Versioned datasets: data/ (v2), data_v3/, data_v4/ (6 labels, reference only), data_v4_systems/, data_v5_systems/, data_v6_systems/ — each with CHANGES.md. Never edit these.
model-comparison/relabel_audit/	Scripts that built each data version (apply_relabels.py, extend_v4/5/6.py) and the lists of new scenarios (new_scenarios_v*.csv).
model-comparison/train_v4.py	Train/tune/final runs for any data version (--data --suffix --out).
model-comparison/report_v4.py	Builds results_v*/report.md and a previous-version comparison report from saved predictions.
model-comparison/export_v4.py	ONNX export + int8 quantisation + agreement checks (picks the setting on validation).
model-comparison/threshold.py	Studies the "uncertain" confidence threshold on validation.
model-comparison/results_v4/ v5/ v6/	Per-seed predictions, reports, tuning file, saved seed-42 model.
model-comparison/export/, export_v6/	Browser exports and model/data cards for v5 and v6.
Caveat-space/	Local copy of the Space. It lags the live Space (pushes go through the API) — resync before editing: git fetch && git reset --hard origin/main.
caveat-gpai/	GPAI rule (gpai_rule.js) + its tests (node test.mjs → must print 655/655 correct).
eu-ai-act-risk-classification/	Downloaded reference dataset; synthetic/test.csv is the out-of-distribution (OOD) test set — never train on it.
OJ_L_202401689_EN_TXT.pdf	The AI Act (Official Journal). Commission draft guidelines PDFs sit next to it.
caveat-demo/	Demo copies: certificate, printable summary, social preview image.
tools/browser-tests/	Reusable Chrome/jsdom tests for the page (layout, consent, real model, network hosts, certificate, int8 agreement). See its README.

---

2. How Caveat works (what runs when someone presses "Check")

1. GPAI rule (gpai_rule.js, runs first in the browser). Descriptions of a provider training/releasing a general-purpose model are decided by rule: > 10^25 FLOPs or Commission designation → gpai_systemic_risk, otherwise gpai. "Built on someone else's model" → note only; the classifier decides. Why a rule: small text models read exponents badly; the law here is a number.
2. Trained model — Devseis AI Act Classifier v6 (legal-bert-small, 35M params, int8 ONNX 35.5 MB) runs in the visitor's browser via transformers.js 3.0.0 + ONNX Runtime Web, both self-hosted in vendor/. Downloaded only when someone shows intent (focus the text box, tap an example, press Check). Shows "Uncertain" when the top probability is < 50%.
3. Reasoning AI — Llama 3.1 8B Instruct on Cloudflare Workers AI (free tier), via the Worker devseis-aiact-reasoning-proxy.umer-wasim.workers.dev (source is not in this workspace), given the AI Act rules as a system prompt. Not trained, never independently validated, the weakest part.
4. The page shows both verdicts, whether they agree, an expert-call box (pre-filled email to umer.wasim@devseis.com, "free 30-min AI Act readiness call"), and a printable one-page summary.

Other page features: Results tab (numbers, comparison, model-vs-expert card, verified AI Act key dates, "Built by Devseis"), Story tab (12-chapter build timeline), Learn tab (14 lessons, glossary of 46 terms, quizzes, a 12-question final assessment with pass mark 9/12 that unlocks a certificate drawn in the browser, LinkedIn add/share using company ID 12603388), consent banner, social preview image.

---

3. How it was built — the version history

Version	Data	Model	Key result	Lesson
v1	401 rows from only 64 scenarios	DistilBERT	61% accuracy; 60% of banned practices called "high risk"	Too few distinct examples → memorising
v2–v3 (Parts 2–3)	268 scenarios from the real legal text + Commission examples	DistilBERT	86% → 71% with more data; real bug: script kept the last epoch. Fixed by selecting on validation macro-F1 → 76.5%, 90% banned caught	Model selection matters as much as data
audit (v3 data)	6 wrong labels fixed (3 in test), 661 ungrammatical rows fixed	—	Published 76.5% was really 74.7%	Check the answer key
backbone comparison (v2, v4)	same splits, 3 seeds	DistilBERT vs legal-bert-small vs SetFit	SetFit: best accuracy but called 16 banned cases "minimal"; legal-bert: much higher banned recall, half the size → chosen	One headline number hides the mistake that matters
v4	+37 scenarios (Annex I, Omnibus bans, Art 6(3))	legal-bert-small	95% banned caught, but phone/voice agents 0/18	Coverage gap: every chatbot example typed
v5	+23 (voice/phone agents, consented media)	legal-bert-small	98% banned, 0 severe errors; OOD 76%, missed face-scraping	Find the gap, add examples, re-measure
v6 (live)	+39 (harmless look-alikes, Art 5(1)(e) wording, consented media) + a second writing style per training scenario	legal-bert-small	99% banned, 0 severe; 81.5% acc on v5 rows; OOD 80%; face-scraping fixed	Write examples the way people write

v6 numbers (3 seeds, v5 test rows = 198 rows / 66 scenarios): accuracy 0.815, macro-F1 0.843 (CI 0.758–0.912), banned recall 0.990, limited 0.963, minimal 0.719, high 0.761. Full v6 test (210 / 70): 0.822 / 0.850 / 0.991. Gains over v5 are consistent but not statistically established. Shipped checkpoint = seed 42 (fixed in advance).

---

4. Challenges and how they were solved

Data and evaluation

Challenge	Solution
Rows are rewordings, so row counts overstate the sample size	Always count scenarios; split by scenario; bootstrap CIs by scenario
Leakage between splits	Scenario-grouped splits; char-4gram Jaccard ≥ 0.45 check against val/test/OOD for every new scenario
Wrong labels (incl. in test)	Audit against the Act (relabel_audit.py); corrected numbers published, old cards corrected
Seed-to-seed swings of up to 9 points	3 seeds per configuration, mean ± std, paired bootstrap; never call a winner on noise
Tempting to tune on test	Rule: choose on validation only; test once per final config; OOD never used for selection (we even kept seed 42 when another seed scored better on OOD)
Over-flagging harmless systems	v6 look-alike scenarios (+ paired high-risk counterparts) — improved only modestly (68% → 72%)
One template style → weak on unusual wording	v6 style rows (question, pitch, tender…) → OOD 76% → 80%
Confidence threshold for "uncertain"	Pre-set rule (≥ 90% answered accuracy on validation) found no threshold: errors are mostly confident. Chosen with Umer: flag only genuinely split predictions (< 50%)

Machine and tooling

Challenge	Solution
Intel Mac: PyTorch stops at 2.2.2	Python 3.11 venv in model-comparison/.venv with pinned versions (requirements.txt)
Two PyTorch jobs at once → load average 120, both stall	Never run torch jobs in parallel; one training job at a time
Mac sleeping during long runs	Keep it awake; runs resume after wake
SetFit predict returns a list; safetensors refuses non-contiguous legal-bert weights; a variable-name clash	Fixed in train_v4.py (contiguous weights before save, etc.)
HF_HUB_OFFLINE=1 breaks when a repo's latest commit isn't cached	Run without offline mode when loading the published v2 model
No git-lfs installed	Push with huggingface_hub.create_commit(..., parent_commit=<head>) — also prevents overwriting concurrent changes
Pushing the local README would drop edits made via the API (e.g. short_description, thumbnail)	Always rebuild the README from the live version before pushing
EUR-Lex blocks curl (202 challenge)	Open it with real Chrome via puppeteer-core (npm i puppeteer-core, use /Applications/Google Chrome.app)
Headless Chrome won't render below 500 px wide	Use Puppeteer device emulation for phone tests
jsdom page test needs stubs	tools/browser-tests/uitest.mjs: stubs pipeline, env.backends.onnx.wasm, gpaiCheck, fetch

Browser deployment

Challenge	Solution
HF Inference API doesn't serve custom models; org can't host gradio/docker on free tier	Static Space + in-browser inference (transformers.js + ONNX)
int8 answers differ from PyTorch, and differ by runtime	Try 8 quantisation settings, pick by validation agreement; measure under onnxruntime-web (what browsers run), one row at a time (dynamic int8 scales per batch). v6: 98.6% in-browser
The originally served v2 file agreed only 94.5%	Fixed by the export script's setting selection
transformers.js in Node ignores quantized: true	Use dtype: "q8"
Self-hosted WASM engine is served uncompressed by HF's CDN (21.7 MB vs 4.2 MB from jsDelivr)	Accepted for the privacy benefit; documented. Model is lazy-loaded to compensate
[hidden] overridden by display:flex (stuck spinner)	Global [hidden]{display:none !important}
Canvas taint for the certificate logo	Logo hosted in the Space itself (devseis-logo.png)

Legal and privacy (Devseis sells GDPR/AI Act services — the page must be exemplary)

Challenge	Solution
legal-bert-small is CC BY-SA 4.0; first card said CC BY 4.0	Model relicensed CC BY-SA 4.0 + full LEGAL-BERT attribution, change list, no-endorsement line on every copy
Omnibus point letters disputed ((ba)/(bb) vs (l))	Verified in the Official Journal: Regulation (EU) 2026/1744, Art 1(7) → (ba)/(bb), apply 2 Dec 2026; Art 5(1a) scope noted
Microsoft Clarity ran without consent; Google Fonts sent IPs to Google; jsDelivr too	Clarity only after "Accept analytics" (consentv2, 12-month choice, cookies deleted otherwise, project yqfpt9xwwk); font and code self-hosted; privacy notice rewritten (controller, flows, legal bases, retention, transfers, CNPD)
Certificate could look like an accredited qualification	Marked "self-paced learning module — not an accredited qualification"; no verification number; typeset signatory, no fake signature
Marketing claims	Every number on the page traced to a saved report; limitations shown alongside

---

5. Rules we work by (do not break)

1. Never edit the versioned data folders; create a new data_v{N}_systems/ with a build script instead.
2. Never re-split or move rows between splits; new scenarios get a deliberate split; previous versions' rows stay put.
3. Choose hyperparameters/checkpoints/thresholds on validation only; evaluate on test once; never use the OOD set for any decision.
4. Ship a new model only if it matches or beats the live one on banned-practice recall with no more severe errors (banned → minimal/limited), on the previous test rows.
5. Model files derived from legal-bert-small stay CC BY-SA 4.0 with attribution.
6. Ask Umer before public pushes unless he has approved that change; pin pushes to the current head commit.
7. Every claim on the page or a card must match a saved report; say plainly when a gain is within noise.
8. Keep the page free of third-party requests before consent; update privacy.html whenever a service is added.

---

6. How to do common tasks

cd ~/Documents/hf/model-comparison
# train a data version (3 seeds), reusing the tuned config copied into results_vN/tune/legalbert.json
TOKENIZERS_PARALLELISM=false .venv/bin/python train_v4.py --phase final --models legalbert --data data_v6_systems --suffix v6 --out results_v6
# reports (full test + previous-version comparison)
.venv/bin/python train_v4.py --phase report --models legalbert --data data_v6_systems --suffix v6 --out results_v6
# export to int8 ONNX with agreement checks
.venv/bin/python export_v4.py --model-dir results_v6/models/legalbert --out export_v6 --data data_v6_systems --suffix v6
# GPAI rule tests
cd ../caveat-gpai && node test.mjs          # must print 655/655 correct
# preview the page locally
cd ../Caveat-space && python3 -m http.server 8000   # http://localhost:8000
# browser tests (serve the Space on port 8766 first)
cd ../tools/browser-tests && npm install && node uitest.mjs && node net.mjs http://localhost:8766/

New model version checklist: build data (script + CHANGES.md, similarity check) → train 3 seeds → reports → shipping rule → export (≥ ~99% agreement, check under onnxruntime-web) → threshold study → new Hub model + dataset repos (cards with attribution) → Space runtime folder + MODEL_ID + every number on the page (Results, Story, glossary, lessons, print sheet) → pointers on the previous repos → profile + collection → verify live (real model, network hosts, phone).

---

7. Open items and ideas for the future

• Umer to review the 39 v6 scenarios (relabel_audit/new_scenarios_v6.csv), especially minimal-risk calls such as 1:1 face verification, car-damage claims and campaign leaflet routing.
• Re-tune the learning rate on v6 validation (all versions since v4 reuse the v4-tuned 2e-5).
• Over-flagging remains the biggest model weakness (minimal recall 0.72–0.74) → more look-alike scenarios.
• Real-world test set: 30–50 real system descriptions labelled by Umer — the only true accuracy measure.
• Stronger reasoning AI (larger free Cloudflare model, or pass the GPAI rule's result as context); its Worker source should be brought into this workspace.
• Calibration: the model's confidence isn't a reliable warning; temperature scaling or an ensemble could help.
• French/German versions for the Luxembourg market.
• Accessibility/performance audit (Lighthouse).
• Review Clarity events after a few weeks (cta_book_call, print_summary, exam_passed, certificate_created).
• Keep the AI Act key dates in sync with any further amendments (built from the OJ texts; update DATES in index.html).