Blame for README.md

---
datasets:
- NagaYu/signpost-label-quality
language:
- de
- en
- es
- fr
- it
- ja
- ko
- pt
- ru
- zh
library_name: generic
license: mit
metrics:
- accuracy
- f1
- precision
- recall
pipeline_tag: text-classification
tags:
- accessibility
- a11y
- aria
- wcag
- label-quality
- multilingual
- synthetic
- signpost-hashed
---

# Signpost label-quality classifier (hashed backend)

Judges whether an accessibility label actually identifies the control it is
attached to -- the question left over after `axe-core` and Xcode's Accessibility
Inspector have both passed. There *is* a name; does the name mean anything on
this screen?

Repository: `NagaYu/signpost-classifier`

| | |
|---|---|
| Code and the full evaluation | <https://github.com/NagaYu/signpost> |
| Dataset it was trained on | <https://huggingface.co/datasets/NagaYu/signpost-label-quality> |
| Try it in your browser | <https://huggingface.co/spaces/NagaYu/signpost> |

It classifies one label into eight classes: `ok` plus seven ways a label fails
(`generic`, `role_echo`, `identifier`, `duplicate`, `verbose`, `missing`,
`mismatch`), and reports the character span of the offending part.


## How to use it

```python
from signpost.context import ContextBuilder, ContextConfig
from signpost.model import QualityClassifier

clf = QualityClassifier.load("path/to/this/checkpoint")
builder = ContextBuilder(ContextConfig(level="full"))  # must match the checkpoint
bundles = builder.build_screen(screen)               # a signpost.types.ScreenTree
for bundle, prediction in zip(bundles, clf.predict(bundles)):
    print(bundle.name, prediction.error_type, round(prediction.confidence, 3))
```

or from the command line, over an axe run, an XCUITest dump or a uiautomator
dump:

```bash
signpost check tree.json --model path/to/this/checkpoint
```

The context level is pinned in the checkpoint and enforced on every call:
feeding it bundles built at another level raises rather than returning numbers
that look fine and mean nothing.

## What was measured

Held-out **origins** -- no site or app on the test side appears in training --
with 860 labels, 309 planted defects and 511 labels the strict quality gate calls
good.

| what was measured | value | on |
| --- | --- | --- |
| of the planted defects the rule baselines missed, share Signpost reported | 54.6% [48.3%, 60.8%] | 238 missed defects |
| the same, with the right error type | 53.4% | 238 missed defects |
| good labels this model reported anyway (over-flagging) | 0.59% [0.20%, 1.71%] | 511 good labels |
| the same, for the rule baselines | 0.00% | 511 good labels |
| hard set: defects every syntactic check still passes | 40.6% | n=96 |
| error-type accuracy over the eight classes | 0.824 | - |
| macro F1 over the eight classes | 0.710 | - |
| median time to judge one label | 0.054 ms | 17585 labels/s |

The operating threshold is **0.644**, chosen on the *training* split as the most
talkative cut whose measured over-flagging stayed inside a 2% budget. The test
numbers above were computed at that threshold and it was never tuned on them.

### Per class

| class | precision | recall | F1 | support |
| --- | --- | --- | --- | --- |
| ok | 0.79 | 0.99 | 0.88 | 515 |
| generic | 1.00 | 0.69 | 0.82 | 55 |
| role_echo | 0.75 | 0.19 | 0.30 | 48 |
| identifier | 0.94 | 0.57 | 0.71 | 54 |
| duplicate | 0.74 | 0.39 | 0.51 | 36 |
| verbose | 1.00 | 1.00 | 1.00 | 36 |
| missing | 1.00 | 0.96 | 0.98 | 27 |
| mismatch | 0.94 | 0.32 | 0.48 | 53 |

### Does the context help?

The same backend, the same hyperparameters, the same seed and the same screens,
trained once per context level. The only thing that varies is what the model was
allowed to look at.

| context level | what the model may see | macro F1 | over-flagging | hard set |
| --- | --- | --- | --- | --- |
| none | role and name -- what a rule checker sees | 0.611 | 2.90% | 10.4% |
| local | + value, hint, visible text, icon asset, traits | 0.612 | 2.12% | 10.4% |
| full | + siblings, neighbouring text, screen title, inferred action | 0.774 | 0.97% | 49.0% |

## Training

- Corpus: `signpost/data/seed_screens.jsonl (synthetic)`
- Split: by origin, 30% of nodes held out, stratified by language; train and
  test share no origin, checked before training.
- Defects: planted by `signpost/inject.py` at a rate of 35%, with a different
  seed on each side of the split so a test defect is not a training defect
  relocated.
- Backend: `hashed`, class weighting `balanced`, seed 0.
- **No base model.** The hashed backend is a matrix of floats over
  hashed context features, trained from scratch. There is no encoder to
  attribute, which is why the frontmatter carries no `base_model`.
- Fit: 0.7 s on Darwin arm64 python 3.13.5.
- No rewrite adapter is included. The template rewriters in `signpost.rewrite` need no model and are what `signpost check` uses by default.

## Limitations -- read these before quoting a number above

- **The corpus is synthetic.** Every screen was composed by
  `scripts/make_seed_corpus.py` from a vocabulary written for this project, and
  every defect was planted by a generator this project also wrote. A model
  evaluated here is partly being measured on how learnable that generator is.
  Real screens are messier: more abbreviations, more house style, more labels
  that are bad in ways nobody thought to plant. Treat these figures as an upper
  bound on a real codebase rather than an estimate of one.
- **Some classes it is bad at.** Weakest in this run:
  `role_echo` recall 0.19 on 48 rows; `mismatch` recall 0.32 on 53 rows.
  In the project's reference run (`python benchmarks/run.py`, seed 0) the two
  weakest are `role_echo` recall 0.21 and `mismatch` recall 0.32 -- and
  `mismatch` is the class this project exists for. A defect this model misses is
  a defect nobody is told about.
- **It is not a replacement for anything.** Not for `axe-core`, not for Xcode's
  Accessibility Inspector or Android's lint checks -- they are better than this
  model at the classes they cover, `missing` above all -- and not for review by
  people who use assistive technology. It narrows what a human has to look at.
- **It does not know what anyone will experience.** Every finding points at a
  documented guideline (WCAG 2.2, Apple's and Google's own guidance) through
  `signpost.cite`, and says nothing about how any person will perceive a screen.
- **`duplicate` is a property of a screen**, not of a label. A row taken out of
  its screen cannot be judged for that class.
- **The threshold is a policy, not a fact.** A team gating a merge queue can
  afford fewer false alarms than a team reviewing a hundred screens by hand;
  `calibrate_threshold` is how they pick their own, and `set_threshold` is how
  they impose it.

## Reproducing this checkpoint

```bash
python scripts/train.py --backend hashed --seed 0 --test-frac 0.30 --inject-rate 0.35
```

Same seed, same bytes: the hashed backend trains full-batch with no sampling
anywhere in the loop.

## Licence

`mit`. The corpus is synthetic and carries no third-party redistribution
question; `signpost/collect.py` is the path for real pages and keeps per-row
provenance.