| --- |
| license: apache-2.0 |
| language: |
| - en |
| library_name: pytorch |
| pipeline_tag: token-classification |
| tags: |
| - dark-patterns |
| - deceptive-design |
| - ui-analysis |
| - accessibility-tree |
| - consumer-protection |
| - regulatory-compliance |
| - not-legal-advice |
| datasets: |
| - NagaYu/sludge-ui-counterfactuals |
| metrics: |
| - f1 |
| - precision |
| - recall |
| model-index: |
| - name: sludge-small |
| results: |
| - task: |
| type: token-classification |
| name: Dark-pattern detection and element localisation on UI trees |
| dataset: |
| type: NagaYu/sludge-ui-counterfactuals |
| name: Sludge counterfactual UI corpus (synthetic) |
| split: test |
| metrics: |
| - type: f1 |
| name: micro-F1 (screen x class) |
| value: 0.9172 |
| - type: f1 |
| name: macro-F1 |
| value: 0.8937 |
| - type: accuracy |
| name: element localisation hit rate |
| value: 1.0000 |
| --- |
| |
| # Sludge — 12 M UI-tree dark-pattern detector |
|
|
| > ## This model does not determine legality. |
| > |
| > It reports **provisions that may be implicated** and the **screen elements that are the |
| > factual basis** for looking at them. Whether a provision is actually engaged depends on facts |
| > no UI tree contains — the purposes of processing, the legal basis relied on, the audience, the |
| > rest of the journey, prior consent, sector rules — and is an assessment for a qualified human. |
| > |
| > It has no feature that labels a named company's product as unlawful, and it is not built to |
| > acquire one. It is a tool for **reviewing interfaces**, not for publishing verdicts about them. |
|
|
|
|
| Sludge reads a **UI tree** — an accessibility tree, a DOM plus computed style, an AX hierarchy — |
| and returns, for each finding, a three-tuple: |
|
|
| | provision that may be implicated | the element of that provision at issue | evidence | |
| |---|---|---| |
| | Directive 2011/83/EU Art. 22 | `express_consent_not_default_option` | `cook_consent_marketing_9` | |
|
|
| …plus what to change, measured against the screen itself: |
|
|
| > Render `'Reject all'` at 200×52 px with 16 px type, matching `'Accept all'` |
| > (currently 95×29 px, 9 px). |
|
|
| - **Dataset:** [`NagaYu/sludge-ui-counterfactuals`](https://huggingface.co/datasets/NagaYu/sludge-ui-counterfactuals) |
| - **Demo:** [`NagaYu/sludge`](https://huggingface.co/spaces/NagaYu/sludge) |
|
|
| ## Why a UI tree instead of a screenshot |
|
|
| On-device the tree is already there, and it gives **exact** strings, geometry, colours, default |
| selection state, hierarchy and link targets. A screenshot pipeline has to recover all of that, |
| and gets some of it wrong. |
|
|
| That is the project's central claim, and the benchmark tests it rather than assuming it: the |
| **same rule analyser** was run on the exact tree and on a real rendered screenshot (OpenCV |
| element detection, colour and checkbox-state recovery from pixels, text through an OCR channel). |
|
|
| | same analyser, different input | micro-F1 | ms/screen | |
| |---|---|---| |
| | on the exact UI tree | 0.767 | 1.36 | |
| | on a rendered screenshot | 0.228 | 21.14 | |
|
|
| **Swapping the tree for a screenshot costs 0.539 micro-F1.** Of that, |
| 0.233 is text-recovery error alone; the rest is lost hierarchy, guessed roles, |
| and link targets that pixels simply do not contain. |
|
|
| ## Results |
|
|
| Held-out **layouts** (`test`) and unseen class **combinations** (`hard`). Split is by UI |
| template, never by sample, so no layout is shared across the boundary. |
|
|
| | system | micro-F1 (test) | macro-F1 | localisation | ms/screen | micro-F1 (hard) | |
| |---|---|---|---|---|---| |
| | Rules on the exact tree | 0.767 | — | — | 1.36 | 0.847 | |
| | Screenshot CV+OCR (same analyser) | 0.228 | — | — | 21.14 | 0.326 | |
| | **Sludge** | **0.917** (0.908–0.926) | 0.894 | **1.000** | 6.77 | 0.849 | |
|
|
| Localisation = the returned evidence intersects the ground-truth element set, over correctly |
| classified pairs. Ranges are 95% percentile bootstrap intervals over records. |
|
|
| **The two F1 columns are not comparable to each other** — `hard` screens carry more patterns |
| each, and micro-F1 rises with the positive rate. Compare systems within a column. |
|
|
| ### Size / accuracy / cost |
|
|
| | checkpoint | parameters | epochs | micro-F1 | localisation | ms/screen | |
| |---|---|---|---|---|---| |
| | `sludge-tiny` | 0.9 M | 8 | 0.863 | 1.000 | 1.48 ms | |
| | `sludge-small` | 12.0 M | 8 | 0.917 | 1.000 | 6.44 ms | |
| | `sludge-base` | 101.7 M | 4 | 0.892 | 1.000 | 33.55 ms | |
|
|
| **Not a clean scaling study** — the larger checkpoint got fewer epochs on a contended machine, |
| so a lower score at a larger size means *undertrained*, not *bigger is worse*. `sludge-small` is |
| the published default because it is the accuracy/cost optimum here and meets the |
| millisecond-per-screen requirement that `sludge-base` does not. |
|
|
| **What is in this repo:** `sludge-small` (root, plus every export format) and `sludge-tiny/`. |
| `sludge-base` weights are **not** published — it is undertrained and `sludge-small` dominates it |
| on every axis, so shipping 388 MB would be misleading rather than useful. Its row above is a |
| real measurement; reproduce it with |
| `python scripts/train.py --preset base --epochs 8`. |
|
|
| ### It reacts to the edit, not the template |
|
|
| The corpus is built from **counterfactual pairs**: a fair screen, and the same screen with |
| **one thing changed**. On held-out layouts, **24 of 27 classes** cross |
| their own decision threshold on that single edit. |
|
|
| ### It can be moved to an operating point |
|
|
| | profile | target FPR | rules | screenshot | **Sludge** | |
| |---|---|---|---|---| |
| | triage | 10% | 0.928 | 0.394 | **0.987** | |
| | review | 5% | 0.928 | 0.394 | **0.981** | |
| | audit | 1% | 0.928 | 0.076 | **0.967** | |
| | *distinct scores available* | | *7* | *8* | *1356* | |
|
|
| The rule engine returns the same recall at 1% and 10% because its scores take a single-digit |
| number of distinct values — there is no threshold between those points that separates anything. |
| That is the operational difference between a rule engine and a calibrated model, independent of |
| F1. |
|
|
| ## Exported runtimes, measured |
|
|
| Forward pass, one screen, 1 CPU thread on `arm`. Every format is exported **and |
| run**, then compared to PyTorch on whether it makes the same decisions. |
|
|
| | format | ms/screen | size | decisions matching PyTorch | |
| |---|---|---|---| |
| | PyTorch | 10.70 ms | 48.0 MB | — | |
| | ONNX | 33.24 ms | 48.3 MB | 1.0000 | |
| | ONNX int8 | 13.34 ms | 12.4 MB | 0.9898 (11/40 screens flip) | |
| | Core ML (fp16) | 2.36 ms | 24.2 MB | 1.0000 | |
| | GGUF (f16) | — | 24.0 MB | 1.0000 | |
|
|
| End to end — featurise, forward, decode — **4.91 ms** p50, |
| **5.96 ms** p95 per screen. That is the number the benchmark uses. |
|
|
| **Two things worth knowing before you pick a runtime.** |
|
|
| *The exported graphs have a **static** node axis* (fixed at `max_nodes`; the caller pads and |
| masks). A dynamic node axis looks like it works — it matches PyTorch on the screen it was traced |
| with — and then raises on any screen of a different size, because the attention reshapes bake in |
| the traced node count. So ONNX pads every screen to 128 elements whether it has 13 or 128, which |
| is why its wall time here exceeds PyTorch's. Re-export at a smaller fixed axis if your screens |
| are small; the padding is verified output-preserving either way. |
|
|
| *This model does not survive int8.* Dynamic quantisation — per-tensor, per-channel, MatMul-only, |
| QInt8 and QUInt8 alike — flips at least one screen-level decision on |
| 11 of |
| 40 sample screens, with probability |
| differences up to 0.998. An earlier version of this card quoted 0.995 agreement for int8; that |
| figure was measured over the raw per-node logit tensor, where almost every entry is confidently |
| negative and agrees, and it hid the flips. The table above now reports agreement on the |
| **screen-level decision the detector actually thresholds**. fp16 is not an alternative either: |
| onnxruntime-web's WASM build rejects float16 attention ops. Use fp32 unless you have measured |
| that quantisation is safe for your screens. |
|
|
| **On GGUF, plainly:** the file is a valid GGUF v3 container and round-trips through |
| `sludge.gguf`, but **llama.cpp cannot execute it** — this is not one of its graph architectures. |
| It is a portable weight container for the accompanying loader, not a drop-in llama.cpp model. |
|
|
| ## Usage |
|
|
| ```bash |
| pip install "sludge[all] @ git+https://github.com/<you>/sludge" |
| sludge check tree.json --profile review --jurisdiction eu |
| ``` |
|
|
| ```python |
| from huggingface_hub import snapshot_download |
| from sludge.model import SludgeDetector |
| from sludge.provisions import ProvisionMapper |
| from sludge.remediate import RemediationSuggester |
| from sludge.uitree import tree_from_html |
| |
| detector = SludgeDetector.load(snapshot_download("NagaYu/sludge-small")) |
| tree = tree_from_html(open("page.html").read()) |
| |
| detections = detector.detect(tree) # class + evidence element ids |
| hits = ProvisionMapper.load("eu").map(detections) # provisions that MAY be implicated |
| fixes = RemediationSuggester().suggest(tree, detections) # what to change, with target values |
| |
| for h in hits: |
| print(h.citation, [e["id"] for e in h.elements], h.evidence) |
| ``` |
|
|
| ## The taxonomy (intermediate representation) |
|
|
| **7 categories / 27 classes**, all from published work — nothing here |
| is coined by this project. Per-class provenance ships in `sludge/taxonomy.py` and prints with |
| `sludge taxonomy`. |
|
|
| | Category (Mathur et al. 2019) | Classes | |
| |---|---| |
| | Sneaking | `sneak_into_basket` · `hidden_costs` · `hidden_subscription` · `bait_and_switch` · `hidden_information` | |
| | Urgency | `countdown_timer` · `limited_time_message` | |
| | Misdirection | `confirmshaming` · `visual_interference` · `false_hierarchy` · `preselection` · `trick_question` · `pressured_selling` · `disguised_ad` | |
| | Social Proof | `activity_message` · `testimonials_uncertain_origin` | |
| | Scarcity | `low_stock_message` · `high_demand_message` | |
| | Obstruction | `hard_to_cancel` · `price_comparison_prevention` · `intermediate_currency` · `privacy_maze` · `dead_end` | |
| | Forced Action | `forced_enrollment` · `forced_registration` · `privacy_zuckering` · `nagging` | |
|
|
| Sources: Mathur et al. (2019) CSCW · Gray et al. (2018) CHI · Brignull, *deceptive.design* · |
| EDPB Guidelines 03/2022 · OECD (2022) DEP No. 336 · Gray et al. (2024) CHI. Where this project |
| had to place a class its source work organises differently, the class records that as an |
| aggregation decision rather than presenting it as a citation. |
|
|
| ## Provision mapping is configuration |
|
|
| Classes map onto the **constituent elements** of provisions, from swappable YAML tables shipped |
| for `eu`, `us_federal` and `us_ca`. Point `--config-dir` at your own directory and every output |
| changes with no code change. |
|
|
| The shipped `us_federal.yaml` carries the worked example: the FTC Negative Option Rule |
| (16 C.F.R. Part 425) was finalised and then vacated in its entirety by the Eighth Circuit in |
| July 2025, while ROSCA beneath it stayed in force. Here that is one `enabled: false` line plus a |
| status note. A tool with the table compiled in would still be asserting it. |
|
|
| ## Training |
|
|
| Trained **only on synthetic screens**. No screenshot or UI tree from a real application is used |
| for training. Public dark-pattern datasets are for *evaluation only*, under their own terms, via |
| `scripts/eval_real_data.py`, which refuses to run without an explicit licence acknowledgement. |
|
|
| Selection and per-class thresholds use a **dev split of two held-out training templates**, never |
| the test set. Node order is shuffled during training so the positional embedding cannot |
| fingerprint a layout. |
|
|
| ## Limitations |
|
|
| - **The corpus is synthetic**, with bounded lexical and layout variation. Read the numbers as |
| *relative* between systems, not as an estimate of field accuracy. |
| - **Flow-scope classes need a flow.** Given a single screen, `hard_to_cancel`, `privacy_maze`, |
| `nagging`, `dead_end` and `forced_registration` have no evidence available; Sludge reports |
| nothing for them rather than guessing. |
| - **A quiet report is not a clean bill of health.** It means nothing crossed the threshold the |
| chosen profile fixes. |
| - **The provision tables are a starting point**, written against operative articles, not |
| recitals or national transpositions. Verify against consolidated text before relying on them. |
| - **The multimodal-LLM comparison arm was not run here** — without an API key it contributes no |
| scores, and the figures mark its published reference point with an open marker rather than |
| inventing numbers. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @software{sludge2026, |
| title = {Sludge: deceptive design pattern detection on UI trees}, |
| year = {2026}, |
| url = {https://huggingface.co/NagaYu/sludge-small}, |
| note = {Reports provisions that may be implicated; does not determine legality.} |
| } |
| ``` |
|
|
| Licence: Apache-2.0. The provision tables are configuration, not legal advice. |
|
|
