README.md

metadata

title: Caveat — AI Act risk check experiment
emoji: ⚖️
colorFrom: red
colorTo: indigo
sdk: static
pinned: false
license: cc-by-4.0
models:
  - Devseis/devseis-ai-act-classifier-v6
  - Devseis/eu-ai-act-classifier-experiment
datasets:
  - Devseis/devseis-ai-act-classifier-v6-data
  - Devseis/eu-ai-act-risk-classification
short_description: EU AI Act risk check, plus a beginner course on how it works
thumbnail: https://huggingface.co/spaces/Devseis/Caveat/resolve/main/og-image.png

Caveat — two experimental AI Act classifiers, compared live

Caveat is a live demo of the experiment that started at Devseis/eu-ai-act-classifier-experiment (versions 1–3). The current model is Devseis AI Act Classifier v6: Devseis/devseis-ai-act-classifier-v6 (weights, card, evaluation) with data at Devseis/devseis-ai-act-classifier-v6-data. The page loads a runtime copy from models/devseis-ai-act-classifier-v6/. It is not a compliance tool — it exists to show, on real live output, why neither approach should be trusted on its own. This file briefs a new session (human or AI) on what exists and how it works.

Full build history, challenges and fixes, rules and open items: HANDBOOK.md.

What it is

A single static page with four tabs (the URL hash selects one: #results, #story, #learn-<n>):

• Check: type or pick one of 8 example use cases. Both models run live, side by side, and the page says whether they agree. Each card shows the model's name, size and where it runs.
• Results: the v6 numbers (headline tiles, recall per tier, a comparison with earlier versions including confidence intervals), the known weak spots, and what runs when you press Check.
• The story: a 12-chapter timeline from v1 to v6. Each chapter gives what was done, what happened, the key number and the lesson, including the mistakes.
• Learn: a 14-lesson plain-language course for non-technical readers, with analogies, real examples from this project, interactive widgets, a quiz per lesson, a build-it-yourself checklist and a searchable 46-term glossary. Underlined terms anywhere on the page open a definition. Progress, the theme and the checklist are kept in the visitor's localStorage.

Privacy (keep it this way): Microsoft Clarity loads only after the visitor clicks "Accept analytics" in the consent banner (choice kept 12 months; "Analytics settings" in the footer to change it; Clarity cookies are deleted unless consent is given). The font is self-hosted in fonts/ (SIL OFL), so no request goes to Google. transformers.js and the ONNX Runtime WebAssembly engine are self-hosted in vendor/transformers-3.0.0/ (byte-identical to npm 3.0.0; env.backends.onnx.wasm.wasmPaths points there), so the page makes no third-party CDN requests. privacy.html documents every data flow; update it whenever a third-party service is added.

Architecture — genuinely $0/month

• Frontend: this HF Space (Devseis/Caveat), sdk: static — plain HTML/CSS/vanilla JS, no framework, no build step.
• Part 1 (fine-tuned classifier): runs entirely in the visitor's browser. No server call. Devseis AI Act Classifier v6 (fine-tuned from nlpaueb/legal-bert-small-uncased on data_v6_systems) is loaded from its repo Devseis/devseis-ai-act-classifier-v6, pinned to commit 7c594f6 (MODEL_REVISION in index.html), so the repo's download count shows how many browsers ran the check (the browser caches the files after the first use). @huggingface/transformers 3.0.0 loads it with dtype: "q8", which fetches onnx/model_quantized.onnx (~36MB, int8), and runs it with WASM. The int8 file matches the PyTorch model on 98.6% of test rows under onnxruntime-web. The full model card is on the Hub, at Devseis/devseis-ai-act-classifier-v6. Earlier versions (v1–v3, DistilBERT) are documented in Devseis/eu-ai-act-classifier-experiment.
• GPAI rule: gpai_rule.js runs before the classifier. Descriptions of a provider training or releasing a general-purpose model get gpai or gpai_systemic_risk from the rule (compute figure > 10^25 FLOPs, or a Commission designation), and the classifier is skipped. "Built on someone else's model" gets a note, and the classifier decides the tier.
• Part 2 (reasoning check): a Cloudflare Worker, devseis-aiact-reasoning-proxy (devseis-aiact-reasoning-proxy.umer-wasim.workers.dev), source lives outside this repo (deployed via wrangler). Calls Cloudflare Workers AI (@cf/meta/llama-3.1-8b-instruct-fp8, free tier) fresh on every request, with the EU AI Act reference rules given as grounding context in the system prompt — no training, no fine-tuning, no stored state.

No API keys, no paid services, no per-request cost either side.

Why two different serving strategies

Devseis/eu-ai-act-classifier-experiment's free HF-hosted Inference API does not support arbitrary custom fine-tuned models ("Model not supported by provider hf-inference", confirmed when this was built) — so Part 1 had to move client-side instead. This Space's org (Devseis) also cannot host gradio/docker Spaces on the free tier (only static — confirmed by a 402 Payment Required when this was tried), which is the other reason Part 1 couldn't be a small Python backend. In-browser ONNX inference is the only genuinely free path left for a custom classifier under these constraints.

Known limitation of Part 2 specifically

The original "Part 2" reasoning check (documented in the model repo) was done by a much larger model (Claude Sonnet 5) reasoning directly, and flagged one case (an election-prediction scenario) as a genuine boundary call between tiers. The free llama-3.1-8b-instruct-fp8 used live, in this Space is meaningfully smaller. In testing during development it got the core tier right on that same case but did not reliably surface the nuance — a concrete example of why "a bigger model would probably do better" is not the same as "this free model does better." Don't assume Part 2's live output here matches the quality documented in the write-up.

Files

• index.html — the app.
• privacy.html — privacy notice with an analytics on/off control.
• fonts/ — self-hosted Space Grotesk (fonts.css, woff2 files, OFL.txt licence).
• vendor/transformers-3.0.0/ — self-hosted transformers.js and ONNX Runtime Web engine, with licences (see its README).
• gpai_rule.js — the GPAI rule (tests live outside this repo, in caveat-gpai/).
• models/devseis-ai-act-classifier-v6/ — Part 1's model: config, tokenizer, onnx/model_quantized.onnx, and the model card (README.md). This folder is a runtime copy of Devseis/devseis-ai-act-classifier-v6 and is licensed CC BY-SA 4.0 (it is fine-tuned from LEGAL-BERT-SMALL; attribution in its README). Data and evaluation reports are in the model and dataset repos linked above.
• manifest.json, privacy.html, devseis-logo.png, icon-192.png, icon-512.png — shared assets, same pattern as Nox.

Deploying changes

Frontend: plain static files, push to this repo like any other Space. Part 2's Worker: edit and wrangler deploy from its own source (not tracked in this repo). Part 1's model: retrain and re-export (the model-comparison/ project: train_v4.py, then export_v4.py), then replace the files in models/devseis-ai-act-classifier-v6/ (or add a new folder and change the model name in index.html). The .onnx file is stored with Git LFS.