projectsidewalk/sidewalk-tagger-ai-crosswalk

Project Sidewalk Tagger: Crosswalk

A multi-label classifier that predicts which accessibility tags apply to a Crosswalk label in a crop of a Google Street View image. Project Sidewalk serves it through sidewalk-ai-api to suggest tags for crowdsourced labels.

This is the DINOv2 model trained on the cleaned dataset in Towards Fine-Grained Sidewalk Accessibility Assessment with Deep Learning: Initial Benchmarks and an Open Dataset (ASSETS '24). Training and evaluation code is in sidewalk-tagger-ai.

Provenance

validated-dino-cls-b-crosswalk-tags-best.pth is a byte-for-byte copy of the file of the same name in projectsidewalk/sidewalk-tagger-ai-models at commit 65959dbc80b87e4f39385204c4b639cbcf58e1a8, which holds every model from the paper and remains available unchanged.

sha256: 2399313abc4b462fa9a708695f6febd05d0bd199e1f7f715af55ee312e618f55

Model

• Backbone: DINOv2 ViT-B/14 with 4 register tokens (dinov2_vitb14_reg4), fine-tuned.
• Head: Linear(768, 256) → ReLU → Linear(256, 10), with a sigmoid per class.
• Input: RGB crop resized to 256 px and padded to a multiple of 14, normalized with ImageNet mean and std.
• Format: a PyTorch state dict; it loads with torch.load(..., weights_only=True).

The architecture is custom rather than a transformers class. See DinoVisionTransformerClassifier and ImageTagger in sidewalk_ai_api/tagger.py for loading and preprocessing.

Classes

The output has 10 classes, in this order. NULL entries are unused placeholders that preserve the class indices from training.

Index	Tag
0	brick/cobblestone
1	broken surface
2	bumpy
3	NULL
4	NULL
5	paint fading
6	rail/tram track
7	NULL
8	uneven surface
9	very long crossing

Thresholds

Sigmoid scores aren't calibrated probabilities. The per-tag thresholds Project Sidewalk uses to suggest that a tag is present or absent are in tag_thresholds.json in sidewalk-ai-api.

Evaluation

See the paper for per-tag results on the test split of sidewalk-tagger-ai-validated. REPRODUCE_RESULTS.md in sidewalk-tagger-ai reproduces them.

Citation

@inproceedings{10.1145/3663548.3688531,
  author = {Liu, Xinlei and Wu, Kevin and Kulkarni, Minchu and Saugstad, Michael and Rapo, Peyton Anton and Freiburger, Jeremy and Hosseini, Maryam and Li, Chu and Froehlich, Jon E.},
  title = {Towards Fine-Grained Sidewalk Accessibility Assessment with Deep Learning: Initial Benchmarks and an Open Dataset},
  year = {2024},
  publisher = {Association for Computing Machinery},
  doi = {10.1145/3663548.3688531},
  booktitle = {Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility},
  articleno = {103},
  numpages = {12}
}