ViSeLA-CUHK-PEDES-Baseline-Router

ACM Multimedia 2026 · Oral Presentation & Outstanding Paper Award

Paper · ACM DOI · Project page · Code · Release v1.0.0

Vision-Grounded Semantic Learning and Alignment for Scalable Multilingual Text-Based Person Retrieval
Xinyu Zhou, Fan Li, Shasha Chen, Min Cao, Ahtamjan Ahmat, Qiaoming Zhu, and Mang Ye.

ViSeLA retrieves person images from descriptions in English, Chinese, French, and German. Semantic-aware multilingual learning routes text tokens to experts by their semantic content. Vision-grounded alignment uses shared learnable queries to align corresponding visual and textual cues. The released architecture combines a BEiT-2 base vision encoder, XLM-RoBERTa text encoding, and multimodal interaction for reranking.

Checkpoint and results

This is the official CUHK-PEDES(M) multilingual baseline-router fine-tuning checkpoint, initialized from CCLM-X2VLM. It uses four experts, four learnable queries, refiner.sim_type: avg, and routing_temp: 0.85. Its stored fields are state_dict and config.

The released checkpoint's reported retrieval results are:

Language R@1 (%)
English (EN) 78.22
Chinese (ZH) 76.07
French (FR) 76.74
German (DE) 76.10
Average 76.78

Average mAP: 68.67%. These are the reported CUHK-PEDES(M) results for this baseline-router checkpoint, using CCLM-X2VLM initialization and average similarity. The paper's main ViSeLA configuration uses full MultiPedes pretraining and LogSumExp similarity, achieving 77.94% Avg R@1 / 70.16% Avg mAP (Table 2).

Download and use

Install the official code environment following the ViSeLA installation instructions. Download the checkpoint and its configuration with huggingface_hub:

from huggingface_hub import hf_hub_download

repo_id = "linda23153/ViSeLA-CUHK-PEDES-Baseline-Router"
local_dir = "checkpoints/ViSeLA-CUHK-PEDES-Baseline-Router"
for filename in (
    "config.json",
    "visela_cuhk_pedes_baseline_router.pth",
    "config.yaml",
    "configs/config_beit2_base.json",
    "SHA256SUMS.txt",
):
    path = hf_hub_download(repo_id=repo_id, filename=filename, local_dir=local_dir)
    print(path)

Run these shell commands from a clone of the official code repository, with the downloaded checkpoints/ directory under that clone:

python tools/demo_retrieval.py \
  --checkpoint checkpoints/ViSeLA-CUHK-PEDES-Baseline-Router/visela_cuhk_pedes_baseline_router.pth \
  --config checkpoints/ViSeLA-CUHK-PEDES-Baseline-Router/config.yaml \
  --text-encoder FacebookAI/xlm-roberta-base \
  --top-k 3

The demo reads the bundled gallery and multilingual queries by default; use --image-dir and --query for a custom gallery. The loader resolves the bundled BEiT-2 architecture configuration when the YAML path is absent from the current working directory. This is a custom PyTorch checkpoint loaded by the official ViSeLA code. config.json records architecture and file metadata for Hub clients.

Full benchmark evaluation

Copy the downloaded config.yaml to local_baseline_router.yaml in the official code checkout. Set anno_dir and image_dir to your CUHK-PEDES annotations and images, data.code_switch_dir to the code-switch annotations, and model.saved_path to a writable output directory. Set vision_config to config/config_beit2_base.json in the code repository. Keep the exported avg similarity and routing temperature 0.85.

python main.py --config local_baseline_router.yaml \
  --eval-only \
  --resume checkpoints/ViSeLA-CUHK-PEDES-Baseline-Router/visela_cuhk_pedes_baseline_router.pth

Obtain the benchmark images from the original dataset provider; the linked benchmark repository supplies annotations. Downstream code uses ch as the Chinese filename/language key; this corresponds to ZH in the metrics table.

Files and provenance

File Contents
visela_cuhk_pedes_baseline_router.pth Original PyTorch weights from GitHub Release v1.0.0
config.yaml Embedded checkpoint configuration exported as plain YAML, with portable paths
config.json Architecture and checkpoint summary
configs/config_beit2_base.json Official BEiT-2 architecture configuration
SHA256SUMS.txt SHA256 checksums for the weight and three configuration files

Weight SHA256: 3776aea5fe499d8c0365173fb73e662ae56edc2314068053f977f1506d22a079.

The YAML contains the checkpoint architecture and training hyperparameters, with portable paths and a public Hub identifier for the text encoder. Set the dataset and output paths for your installation before training or evaluation.

Verify from the downloaded model directory:

sha256sum -c SHA256SUMS.txt

Related official resources

Resource Repository
MultiPedes pretraining annotations linda23153/MultiPedes
Multilingual downstream benchmarks linda23153/ViSeLA-Multilingual-Benchmarks
HAM + LUPerson pretraining checkpoint linda23153/ViSeLA-Pretrain-HAM-LUPerson
CUHK-PEDES baseline-router checkpoint linda23153/ViSeLA-CUHK-PEDES-Baseline-Router

License and intended use

These weights are released for academic research on multilingual text-based person retrieval and cross-modal representation learning, subject to the terms of the upstream pretrained models and pedestrian datasets. See THIRD_PARTY_NOTICES.md for the source models, datasets, and their terms.

Citation

@inproceedings{zhou2026visela,
  title={Vision-Grounded Semantic Learning and Alignment for Scalable Multilingual Text-Based Person Retrieval},
  author={Zhou, Xinyu and Li, Fan and Chen, Shasha and Cao, Min and Ahmat, Ahtamjan and Zhu, Qiaoming and Ye, Mang},
  booktitle={Proceedings of the 34th ACM International Conference on Multimedia (MM '26)},
  year={2026},
  doi={10.1145/3767308.3836472},
  url={https://arxiv.org/abs/2611.15328},
  note={Oral Presentation, Outstanding Paper Award}
}
Downloads last month
51
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Paper for linda23153/ViSeLA-CUHK-PEDES-Baseline-Router