| --- |
| datasets: |
| - NagaYu/signpost-label-quality |
| language: |
| - de |
| - en |
| - es |
| - fr |
| - it |
| - ja |
| - ko |
| - pt |
| - ru |
| - zh |
| library_name: generic |
| license: mit |
| metrics: |
| - accuracy |
| - f1 |
| - precision |
| - recall |
| pipeline_tag: text-classification |
| tags: |
| - accessibility |
| - a11y |
| - aria |
| - wcag |
| - label-quality |
| - multilingual |
| - synthetic |
| - signpost-hashed |
| --- |
| |
| # Signpost label-quality classifier (hashed backend) |
|
|
| Judges whether an accessibility label actually identifies the control it is |
| attached to -- the question left over after `axe-core` and Xcode's Accessibility |
| Inspector have both passed. There *is* a name; does the name mean anything on |
| this screen? |
|
|
| Repository: `NagaYu/signpost-classifier` |
|
|
| | | | |
| |---|---| |
| | Code and the full evaluation | <https://github.com/NagaYu/signpost> | |
| | Dataset it was trained on | <https://huggingface.co/datasets/NagaYu/signpost-label-quality> | |
| | Try it in your browser | <https://huggingface.co/spaces/NagaYu/signpost> | |
|
|
| It classifies one label into eight classes: `ok` plus seven ways a label fails |
| (`generic`, `role_echo`, `identifier`, `duplicate`, `verbose`, `missing`, |
| `mismatch`), and reports the character span of the offending part. |
|
|
|
|
| ## How to use it |
|
|
| ```python |
| from signpost.context import ContextBuilder, ContextConfig |
| from signpost.model import QualityClassifier |
| |
| clf = QualityClassifier.load("path/to/this/checkpoint") |
| builder = ContextBuilder(ContextConfig(level="full")) # must match the checkpoint |
| bundles = builder.build_screen(screen) # a signpost.types.ScreenTree |
| for bundle, prediction in zip(bundles, clf.predict(bundles)): |
| print(bundle.name, prediction.error_type, round(prediction.confidence, 3)) |
| ``` |
|
|
| or from the command line, over an axe run, an XCUITest dump or a uiautomator |
| dump: |
|
|
| ```bash |
| signpost check tree.json --model path/to/this/checkpoint |
| ``` |
|
|
| The context level is pinned in the checkpoint and enforced on every call: |
| feeding it bundles built at another level raises rather than returning numbers |
| that look fine and mean nothing. |
|
|
| ## What was measured |
|
|
| Held-out **origins** -- no site or app on the test side appears in training -- |
| with 860 labels, 309 planted defects and 511 labels the strict quality gate calls |
| good. |
|
|
| | what was measured | value | on | |
| | --- | --- | --- | |
| | of the planted defects the rule baselines missed, share Signpost reported | 54.6% [48.3%, 60.8%] | 238 missed defects | |
| | the same, with the right error type | 53.4% | 238 missed defects | |
| | good labels this model reported anyway (over-flagging) | 0.59% [0.20%, 1.71%] | 511 good labels | |
| | the same, for the rule baselines | 0.00% | 511 good labels | |
| | hard set: defects every syntactic check still passes | 40.6% | n=96 | |
| | error-type accuracy over the eight classes | 0.824 | - | |
| | macro F1 over the eight classes | 0.710 | - | |
| | median time to judge one label | 0.054 ms | 17585 labels/s | |
|
|
| The operating threshold is **0.644**, chosen on the *training* split as the most |
| talkative cut whose measured over-flagging stayed inside a 2% budget. The test |
| numbers above were computed at that threshold and it was never tuned on them. |
|
|
| ### Per class |
|
|
| | class | precision | recall | F1 | support | |
| | --- | --- | --- | --- | --- | |
| | ok | 0.79 | 0.99 | 0.88 | 515 | |
| | generic | 1.00 | 0.69 | 0.82 | 55 | |
| | role_echo | 0.75 | 0.19 | 0.30 | 48 | |
| | identifier | 0.94 | 0.57 | 0.71 | 54 | |
| | duplicate | 0.74 | 0.39 | 0.51 | 36 | |
| | verbose | 1.00 | 1.00 | 1.00 | 36 | |
| | missing | 1.00 | 0.96 | 0.98 | 27 | |
| | mismatch | 0.94 | 0.32 | 0.48 | 53 | |
| |
| ### Does the context help? |
| |
| The same backend, the same hyperparameters, the same seed and the same screens, |
| trained once per context level. The only thing that varies is what the model was |
| allowed to look at. |
| |
| | context level | what the model may see | macro F1 | over-flagging | hard set | |
| | --- | --- | --- | --- | --- | |
| | none | role and name -- what a rule checker sees | 0.611 | 2.90% | 10.4% | |
| | local | + value, hint, visible text, icon asset, traits | 0.612 | 2.12% | 10.4% | |
| | full | + siblings, neighbouring text, screen title, inferred action | 0.774 | 0.97% | 49.0% | |
| |
| ## Training |
| |
| - Corpus: `signpost/data/seed_screens.jsonl (synthetic)` |
| - Split: by origin, 30% of nodes held out, stratified by language; train and |
| test share no origin, checked before training. |
| - Defects: planted by `signpost/inject.py` at a rate of 35%, with a different |
| seed on each side of the split so a test defect is not a training defect |
| relocated. |
| - Backend: `hashed`, class weighting `balanced`, seed 0. |
| - **No base model.** The hashed backend is a matrix of floats over |
| hashed context features, trained from scratch. There is no encoder to |
| attribute, which is why the frontmatter carries no `base_model`. |
| - Fit: 0.7 s on Darwin arm64 python 3.13.5. |
| - No rewrite adapter is included. The template rewriters in `signpost.rewrite` need no model and are what `signpost check` uses by default. |
|
|
| ## Limitations -- read these before quoting a number above |
|
|
| - **The corpus is synthetic.** Every screen was composed by |
| `scripts/make_seed_corpus.py` from a vocabulary written for this project, and |
| every defect was planted by a generator this project also wrote. A model |
| evaluated here is partly being measured on how learnable that generator is. |
| Real screens are messier: more abbreviations, more house style, more labels |
| that are bad in ways nobody thought to plant. Treat these figures as an upper |
| bound on a real codebase rather than an estimate of one. |
| - **Some classes it is bad at.** Weakest in this run: |
| `role_echo` recall 0.19 on 48 rows; `mismatch` recall 0.32 on 53 rows. |
| In the project's reference run (`python benchmarks/run.py`, seed 0) the two |
| weakest are `role_echo` recall 0.21 and `mismatch` recall 0.32 -- and |
| `mismatch` is the class this project exists for. A defect this model misses is |
| a defect nobody is told about. |
| - **It is not a replacement for anything.** Not for `axe-core`, not for Xcode's |
| Accessibility Inspector or Android's lint checks -- they are better than this |
| model at the classes they cover, `missing` above all -- and not for review by |
| people who use assistive technology. It narrows what a human has to look at. |
| - **It does not know what anyone will experience.** Every finding points at a |
| documented guideline (WCAG 2.2, Apple's and Google's own guidance) through |
| `signpost.cite`, and says nothing about how any person will perceive a screen. |
| - **`duplicate` is a property of a screen**, not of a label. A row taken out of |
| its screen cannot be judged for that class. |
| - **The threshold is a policy, not a fact.** A team gating a merge queue can |
| afford fewer false alarms than a team reviewing a hundred screens by hand; |
| `calibrate_threshold` is how they pick their own, and `set_threshold` is how |
| they impose it. |
|
|
| ## Reproducing this checkpoint |
|
|
| ```bash |
| python scripts/train.py --backend hashed --seed 0 --test-frac 0.30 --inject-rate 0.35 |
| ``` |
|
|
| Same seed, same bytes: the hashed backend trains full-batch with no sampling |
| anywhere in the loop. |
|
|
| ## Licence |
|
|
| `mit`. The corpus is synthetic and carries no third-party redistribution |
| question; `signpost/collect.py` is the path for real pages and keeps per-row |
| provenance. |
|
|
