--- datasets: - NagaYu/signpost-label-quality language: - de - en - es - fr - it - ja - ko - pt - ru - zh library_name: generic license: mit metrics: - accuracy - f1 - precision - recall pipeline_tag: text-classification tags: - accessibility - a11y - aria - wcag - label-quality - multilingual - synthetic - signpost-hashed --- # Signpost label-quality classifier (hashed backend) Judges whether an accessibility label actually identifies the control it is attached to -- the question left over after `axe-core` and Xcode's Accessibility Inspector have both passed. There *is* a name; does the name mean anything on this screen? Repository: `NagaYu/signpost-classifier` | | | |---|---| | Code and the full evaluation | | | Dataset it was trained on | | | Try it in your browser | | It classifies one label into eight classes: `ok` plus seven ways a label fails (`generic`, `role_echo`, `identifier`, `duplicate`, `verbose`, `missing`, `mismatch`), and reports the character span of the offending part. ## How to use it ```python from signpost.context import ContextBuilder, ContextConfig from signpost.model import QualityClassifier clf = QualityClassifier.load("path/to/this/checkpoint") builder = ContextBuilder(ContextConfig(level="full")) # must match the checkpoint bundles = builder.build_screen(screen) # a signpost.types.ScreenTree for bundle, prediction in zip(bundles, clf.predict(bundles)): print(bundle.name, prediction.error_type, round(prediction.confidence, 3)) ``` or from the command line, over an axe run, an XCUITest dump or a uiautomator dump: ```bash signpost check tree.json --model path/to/this/checkpoint ``` The context level is pinned in the checkpoint and enforced on every call: feeding it bundles built at another level raises rather than returning numbers that look fine and mean nothing. ## What was measured Held-out **origins** -- no site or app on the test side appears in training -- with 860 labels, 309 planted defects and 511 labels the strict quality gate calls good. | what was measured | value | on | | --- | --- | --- | | of the planted defects the rule baselines missed, share Signpost reported | 54.6% [48.3%, 60.8%] | 238 missed defects | | the same, with the right error type | 53.4% | 238 missed defects | | good labels this model reported anyway (over-flagging) | 0.59% [0.20%, 1.71%] | 511 good labels | | the same, for the rule baselines | 0.00% | 511 good labels | | hard set: defects every syntactic check still passes | 40.6% | n=96 | | error-type accuracy over the eight classes | 0.824 | - | | macro F1 over the eight classes | 0.710 | - | | median time to judge one label | 0.054 ms | 17585 labels/s | The operating threshold is **0.644**, chosen on the *training* split as the most talkative cut whose measured over-flagging stayed inside a 2% budget. The test numbers above were computed at that threshold and it was never tuned on them. ### Per class | class | precision | recall | F1 | support | | --- | --- | --- | --- | --- | | ok | 0.79 | 0.99 | 0.88 | 515 | | generic | 1.00 | 0.69 | 0.82 | 55 | | role_echo | 0.75 | 0.19 | 0.30 | 48 | | identifier | 0.94 | 0.57 | 0.71 | 54 | | duplicate | 0.74 | 0.39 | 0.51 | 36 | | verbose | 1.00 | 1.00 | 1.00 | 36 | | missing | 1.00 | 0.96 | 0.98 | 27 | | mismatch | 0.94 | 0.32 | 0.48 | 53 | ### Does the context help? The same backend, the same hyperparameters, the same seed and the same screens, trained once per context level. The only thing that varies is what the model was allowed to look at. | context level | what the model may see | macro F1 | over-flagging | hard set | | --- | --- | --- | --- | --- | | none | role and name -- what a rule checker sees | 0.611 | 2.90% | 10.4% | | local | + value, hint, visible text, icon asset, traits | 0.612 | 2.12% | 10.4% | | full | + siblings, neighbouring text, screen title, inferred action | 0.774 | 0.97% | 49.0% | ## Training - Corpus: `signpost/data/seed_screens.jsonl (synthetic)` - Split: by origin, 30% of nodes held out, stratified by language; train and test share no origin, checked before training. - Defects: planted by `signpost/inject.py` at a rate of 35%, with a different seed on each side of the split so a test defect is not a training defect relocated. - Backend: `hashed`, class weighting `balanced`, seed 0. - **No base model.** The hashed backend is a matrix of floats over hashed context features, trained from scratch. There is no encoder to attribute, which is why the frontmatter carries no `base_model`. - Fit: 0.7 s on Darwin arm64 python 3.13.5. - No rewrite adapter is included. The template rewriters in `signpost.rewrite` need no model and are what `signpost check` uses by default. ## Limitations -- read these before quoting a number above - **The corpus is synthetic.** Every screen was composed by `scripts/make_seed_corpus.py` from a vocabulary written for this project, and every defect was planted by a generator this project also wrote. A model evaluated here is partly being measured on how learnable that generator is. Real screens are messier: more abbreviations, more house style, more labels that are bad in ways nobody thought to plant. Treat these figures as an upper bound on a real codebase rather than an estimate of one. - **Some classes it is bad at.** Weakest in this run: `role_echo` recall 0.19 on 48 rows; `mismatch` recall 0.32 on 53 rows. In the project's reference run (`python benchmarks/run.py`, seed 0) the two weakest are `role_echo` recall 0.21 and `mismatch` recall 0.32 -- and `mismatch` is the class this project exists for. A defect this model misses is a defect nobody is told about. - **It is not a replacement for anything.** Not for `axe-core`, not for Xcode's Accessibility Inspector or Android's lint checks -- they are better than this model at the classes they cover, `missing` above all -- and not for review by people who use assistive technology. It narrows what a human has to look at. - **It does not know what anyone will experience.** Every finding points at a documented guideline (WCAG 2.2, Apple's and Google's own guidance) through `signpost.cite`, and says nothing about how any person will perceive a screen. - **`duplicate` is a property of a screen**, not of a label. A row taken out of its screen cannot be judged for that class. - **The threshold is a policy, not a fact.** A team gating a merge queue can afford fewer false alarms than a team reviewing a hundred screens by hand; `calibrate_threshold` is how they pick their own, and `set_threshold` is how they impose it. ## Reproducing this checkpoint ```bash python scripts/train.py --backend hashed --seed 0 --test-frac 0.30 --inject-rate 0.35 ``` Same seed, same bytes: the hashed backend trains full-batch with no sampling anywhere in the loop. ## Licence `mit`. The corpus is synthetic and carries no third-party redistribution question; `signpost/collect.py` is the path for real pages and keeps per-row provenance.