ViSeLA-CUHK-PEDES-Baseline-Router
ACM Multimedia 2026 · Oral Presentation & Outstanding Paper Award
Paper · ACM DOI · Project page · Code · Release v1.0.0
Vision-Grounded Semantic Learning and Alignment for Scalable Multilingual Text-Based Person Retrieval
Xinyu Zhou, Fan Li, Shasha Chen, Min Cao, Ahtamjan Ahmat, Qiaoming Zhu, and Mang Ye.
ViSeLA retrieves person images from descriptions in English, Chinese, French, and German. Semantic-aware multilingual learning routes text tokens to experts by their semantic content. Vision-grounded alignment uses shared learnable queries to align corresponding visual and textual cues. The released architecture combines a BEiT-2 base vision encoder, XLM-RoBERTa text encoding, and multimodal interaction for reranking.
Checkpoint and results
This is the official CUHK-PEDES(M) multilingual baseline-router fine-tuning checkpoint, initialized from CCLM-X2VLM. It uses four experts, four learnable queries, refiner.sim_type: avg, and routing_temp: 0.85. Its stored fields are state_dict and config.
The released checkpoint's reported retrieval results are:
| Language | R@1 (%) |
|---|---|
| English (EN) | 78.22 |
| Chinese (ZH) | 76.07 |
| French (FR) | 76.74 |
| German (DE) | 76.10 |
| Average | 76.78 |
Average mAP: 68.67%. These are the reported CUHK-PEDES(M) results for this baseline-router checkpoint, using CCLM-X2VLM initialization and average similarity. The paper's main ViSeLA configuration uses full MultiPedes pretraining and LogSumExp similarity, achieving 77.94% Avg R@1 / 70.16% Avg mAP (Table 2).
Download and use
Install the official code environment following the ViSeLA installation instructions. Download the checkpoint and its configuration with huggingface_hub:
from huggingface_hub import hf_hub_download
repo_id = "linda23153/ViSeLA-CUHK-PEDES-Baseline-Router"
local_dir = "checkpoints/ViSeLA-CUHK-PEDES-Baseline-Router"
for filename in (
"config.json",
"visela_cuhk_pedes_baseline_router.pth",
"config.yaml",
"configs/config_beit2_base.json",
"SHA256SUMS.txt",
):
path = hf_hub_download(repo_id=repo_id, filename=filename, local_dir=local_dir)
print(path)
Run these shell commands from a clone of the official code repository, with the downloaded checkpoints/ directory under that clone:
python tools/demo_retrieval.py \
--checkpoint checkpoints/ViSeLA-CUHK-PEDES-Baseline-Router/visela_cuhk_pedes_baseline_router.pth \
--config checkpoints/ViSeLA-CUHK-PEDES-Baseline-Router/config.yaml \
--text-encoder FacebookAI/xlm-roberta-base \
--top-k 3
The demo reads the bundled gallery and multilingual queries by default; use --image-dir and --query for a custom gallery. The loader resolves the bundled BEiT-2 architecture configuration when the YAML path is absent from the current working directory. This is a custom PyTorch checkpoint loaded by the official ViSeLA code. config.json records architecture and file metadata for Hub clients.
Full benchmark evaluation
Copy the downloaded config.yaml to local_baseline_router.yaml in the official code checkout. Set anno_dir and image_dir to your CUHK-PEDES annotations and images, data.code_switch_dir to the code-switch annotations, and model.saved_path to a writable output directory. Set vision_config to config/config_beit2_base.json in the code repository. Keep the exported avg similarity and routing temperature 0.85.
python main.py --config local_baseline_router.yaml \
--eval-only \
--resume checkpoints/ViSeLA-CUHK-PEDES-Baseline-Router/visela_cuhk_pedes_baseline_router.pth
Obtain the benchmark images from the original dataset provider; the linked benchmark repository supplies annotations. Downstream code uses ch as the Chinese filename/language key; this corresponds to ZH in the metrics table.
Files and provenance
| File | Contents |
|---|---|
visela_cuhk_pedes_baseline_router.pth |
Original PyTorch weights from GitHub Release v1.0.0 |
config.yaml |
Embedded checkpoint configuration exported as plain YAML, with portable paths |
config.json |
Architecture and checkpoint summary |
configs/config_beit2_base.json |
Official BEiT-2 architecture configuration |
SHA256SUMS.txt |
SHA256 checksums for the weight and three configuration files |
Weight SHA256: 3776aea5fe499d8c0365173fb73e662ae56edc2314068053f977f1506d22a079.
The YAML contains the checkpoint architecture and training hyperparameters, with portable paths and a public Hub identifier for the text encoder. Set the dataset and output paths for your installation before training or evaluation.
Verify from the downloaded model directory:
sha256sum -c SHA256SUMS.txt
Related official resources
| Resource | Repository |
|---|---|
| MultiPedes pretraining annotations | linda23153/MultiPedes |
| Multilingual downstream benchmarks | linda23153/ViSeLA-Multilingual-Benchmarks |
| HAM + LUPerson pretraining checkpoint | linda23153/ViSeLA-Pretrain-HAM-LUPerson |
| CUHK-PEDES baseline-router checkpoint | linda23153/ViSeLA-CUHK-PEDES-Baseline-Router |
License and intended use
These weights are released for academic research on multilingual text-based person retrieval and cross-modal representation learning, subject to the terms of the upstream pretrained models and pedestrian datasets. See THIRD_PARTY_NOTICES.md for the source models, datasets, and their terms.
Citation
@inproceedings{zhou2026visela,
title={Vision-Grounded Semantic Learning and Alignment for Scalable Multilingual Text-Based Person Retrieval},
author={Zhou, Xinyu and Li, Fan and Chen, Shasha and Cao, Min and Ahmat, Ahtamjan and Zhu, Qiaoming and Ye, Mang},
booktitle={Proceedings of the 34th ACM International Conference on Multimedia (MM '26)},
year={2026},
doi={10.1145/3767308.3836472},
url={https://arxiv.org/abs/2611.15328},
note={Oral Presentation, Outstanding Paper Award}
}
- Downloads last month
- 51