Publish official ViSeLA v1.0.0 checkpoint, portable configs and model card
Browse files- README.md +128 -0
- SHA256SUMS.txt +4 -0
- config.json +24 -0
- config.yaml +86 -0
- configs/config_beit2_base.json +5 -0
- visela_cuhk_pedes_baseline_router.pth +3 -0
README.md
ADDED
|
@@ -0,0 +1,128 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: other
|
| 3 |
+
license_name: research-only-upstream-terms
|
| 4 |
+
license_link: https://github.com/Flame-Chasers/ViSeLA/blob/main/THIRD_PARTY_NOTICES.md
|
| 5 |
+
language: [en, zh, fr, de]
|
| 6 |
+
tags: [arxiv:2611.15328, person-re-identification, text-to-image-retrieval, multilingual, vision-language, pytorch, acm-mm-2026]
|
| 7 |
+
---
|
| 8 |
+
# ViSeLA-CUHK-PEDES-Baseline-Router
|
| 9 |
+
|
| 10 |
+
**ACM Multimedia 2026 · Oral Presentation & Outstanding Paper Award**
|
| 11 |
+
|
| 12 |
+
[Paper](https://arxiv.org/abs/2611.15328) · [ACM DOI](https://doi.org/10.1145/3767308.3836472) · [Project page](https://flame-chasers.github.io/ViSeLA/) · [Code](https://github.com/Flame-Chasers/ViSeLA) · [Release v1.0.0](https://github.com/Flame-Chasers/ViSeLA/releases/tag/v1.0.0)
|
| 13 |
+
|
| 14 |
+
**Vision-Grounded Semantic Learning and Alignment for Scalable Multilingual Text-Based Person Retrieval**
|
| 15 |
+
Xinyu Zhou, Fan Li, Shasha Chen, Min Cao, Ahtamjan Ahmat, Qiaoming Zhu, and Mang Ye.
|
| 16 |
+
|
| 17 |
+
ViSeLA retrieves person images from descriptions in English, Chinese, French, and German. Semantic-aware multilingual learning routes text tokens to experts by their semantic content. Vision-grounded alignment uses shared learnable queries to align corresponding visual and textual cues. The released architecture combines a BEiT-2 base vision encoder, XLM-RoBERTa text encoding, and multimodal interaction for reranking.
|
| 18 |
+
|
| 19 |
+
## Checkpoint identity and evaluation
|
| 20 |
+
|
| 21 |
+
This is the official **CUHK-PEDES(M) multilingual baseline-router fine-tuning checkpoint**, initialized from **CCLM-X2VLM**. It uses four experts, four learnable queries, `refiner.sim_type: avg`, and `routing_temp: 0.85`. Its stored fields are `state_dict` and `config`.
|
| 22 |
+
|
| 23 |
+
The released checkpoint's reported retrieval results are:
|
| 24 |
+
|
| 25 |
+
| Language | R@1 (%) |
|
| 26 |
+
| --- | ---: |
|
| 27 |
+
| English (EN) | 78.22 |
|
| 28 |
+
| Chinese (ZH) | 76.07 |
|
| 29 |
+
| French (FR) | 76.74 |
|
| 30 |
+
| German (DE) | 76.10 |
|
| 31 |
+
| **Average** | **76.78** |
|
| 32 |
+
|
| 33 |
+
**Average mAP: 68.67%.** Values describe the released checkpoint's CUHK-PEDES(M) evaluation; they are reported release results, not a new evaluation performed for this mirror.
|
| 34 |
+
|
| 35 |
+
For comparison, [Table 2 of the paper](https://arxiv.org/html/2611.15328v1#S5) reports **77.94% Avg R@1 / 70.16% Avg mAP** for ViSeLA with **full MultiPedes pretraining and `lsp_max` similarity**. The baseline-router file here uses CCLM-X2VLM initialization and `avg` similarity. Keep these training settings and results separate when citing or comparing the models. The paper's ViSeLA row without person-centric pretraining (76.83% / 68.71%) also describes a different configuration from this released baseline router.
|
| 36 |
+
|
| 37 |
+
## Download and use
|
| 38 |
+
|
| 39 |
+
Install the official code environment following the [ViSeLA installation instructions](https://github.com/Flame-Chasers/ViSeLA#quick-start). Download the checkpoint and its configuration with `huggingface_hub`:
|
| 40 |
+
|
| 41 |
+
```python
|
| 42 |
+
from huggingface_hub import hf_hub_download
|
| 43 |
+
|
| 44 |
+
repo_id = "linda23153/ViSeLA-CUHK-PEDES-Baseline-Router"
|
| 45 |
+
local_dir = "checkpoints/ViSeLA-CUHK-PEDES-Baseline-Router"
|
| 46 |
+
for filename in (
|
| 47 |
+
"config.json",
|
| 48 |
+
"visela_cuhk_pedes_baseline_router.pth",
|
| 49 |
+
"config.yaml",
|
| 50 |
+
"configs/config_beit2_base.json",
|
| 51 |
+
"SHA256SUMS.txt",
|
| 52 |
+
):
|
| 53 |
+
path = hf_hub_download(repo_id=repo_id, filename=filename, local_dir=local_dir)
|
| 54 |
+
print(path)
|
| 55 |
+
```
|
| 56 |
+
|
| 57 |
+
Run these shell commands from a clone of the official code repository, with the downloaded `checkpoints/` directory under that clone:
|
| 58 |
+
|
| 59 |
+
```bash
|
| 60 |
+
python tools/demo_retrieval.py \
|
| 61 |
+
--checkpoint checkpoints/ViSeLA-CUHK-PEDES-Baseline-Router/visela_cuhk_pedes_baseline_router.pth \
|
| 62 |
+
--config checkpoints/ViSeLA-CUHK-PEDES-Baseline-Router/config.yaml \
|
| 63 |
+
--text-encoder FacebookAI/xlm-roberta-base \
|
| 64 |
+
--top-k 3
|
| 65 |
+
```
|
| 66 |
+
|
| 67 |
+
The demo reads the bundled gallery and multilingual queries by default; use `--image-dir` and `--query` for a custom gallery. The loader resolves the bundled BEiT-2 architecture configuration when the YAML path is absent from the current working directory. This is a custom PyTorch checkpoint loaded by the official ViSeLA code. `config.json` records architecture and file metadata for Hub clients.
|
| 68 |
+
|
| 69 |
+
### Full benchmark evaluation
|
| 70 |
+
|
| 71 |
+
Copy the downloaded `config.yaml` to `local_baseline_router.yaml` in the official code checkout. Set `anno_dir` and `image_dir` to your CUHK-PEDES annotations and images, `data.code_switch_dir` to the code-switch annotations, and `model.saved_path` to a writable output directory. Set `vision_config` to `config/config_beit2_base.json` in the code repository. Keep the exported `avg` similarity and routing temperature 0.85.
|
| 72 |
+
|
| 73 |
+
```bash
|
| 74 |
+
python main.py --config local_baseline_router.yaml \
|
| 75 |
+
--eval-only \
|
| 76 |
+
--resume checkpoints/ViSeLA-CUHK-PEDES-Baseline-Router/visela_cuhk_pedes_baseline_router.pth
|
| 77 |
+
```
|
| 78 |
+
|
| 79 |
+
Obtain the benchmark images from the original dataset provider; the linked benchmark repository supplies annotations. Downstream code uses `ch` as the Chinese filename/language key; this corresponds to ZH in the metrics table.
|
| 80 |
+
|
| 81 |
+
## Files and provenance
|
| 82 |
+
|
| 83 |
+
| File | Contents |
|
| 84 |
+
| --- | --- |
|
| 85 |
+
| `visela_cuhk_pedes_baseline_router.pth` | Original GitHub v1.0.0 PyTorch weights; byte-for-byte unchanged |
|
| 86 |
+
| `config.yaml` | Embedded checkpoint configuration exported as plain YAML, with portable paths |
|
| 87 |
+
| `config.json` | Architecture and checkpoint summary |
|
| 88 |
+
| `configs/config_beit2_base.json` | Official BEiT-2 architecture configuration |
|
| 89 |
+
| `SHA256SUMS.txt` | SHA256 checksums for the weight and three configuration files |
|
| 90 |
+
|
| 91 |
+
Weight SHA256: `3776aea5fe499d8c0365173fb73e662ae56edc2314068053f977f1506d22a079`.
|
| 92 |
+
|
| 93 |
+
The YAML preserves the embedded architecture and training hyperparameters. Machine-local dataset and output paths are replaced by `/path/to/…` placeholders or relative paths; the text encoder uses its public Hub identifier. The raw weight retains its original embedded configuration. Inspect and set paths before training or benchmark evaluation.
|
| 94 |
+
|
| 95 |
+
Verify from the downloaded model directory:
|
| 96 |
+
|
| 97 |
+
```bash
|
| 98 |
+
sha256sum -c SHA256SUMS.txt
|
| 99 |
+
```
|
| 100 |
+
|
| 101 |
+
## Related official resources
|
| 102 |
+
|
| 103 |
+
| Resource | Repository |
|
| 104 |
+
| --- | --- |
|
| 105 |
+
| MultiPedes pretraining annotations | [linda23153/MultiPedes](https://huggingface.co/datasets/linda23153/MultiPedes) |
|
| 106 |
+
| Multilingual downstream benchmarks | [linda23153/ViSeLA-Multilingual-Benchmarks](https://huggingface.co/datasets/linda23153/ViSeLA-Multilingual-Benchmarks) |
|
| 107 |
+
| HAM + LUPerson pretraining checkpoint | [linda23153/ViSeLA-Pretrain-HAM-LUPerson](https://huggingface.co/linda23153/ViSeLA-Pretrain-HAM-LUPerson) |
|
| 108 |
+
| CUHK-PEDES baseline-router checkpoint | [linda23153/ViSeLA-CUHK-PEDES-Baseline-Router](https://huggingface.co/linda23153/ViSeLA-CUHK-PEDES-Baseline-Router) |
|
| 109 |
+
|
| 110 |
+
## License and intended use
|
| 111 |
+
|
| 112 |
+
The ViSeLA source code is MIT licensed. These model weights remain subject to the terms of the upstream pretrained weights and the academic research licenses of the pedestrian datasets used for training. The `research-only-upstream-terms` metadata describes these combined conditions; the code's MIT license does not grant unrestricted rights to the weights or source data. Consult the official [THIRD_PARTY_NOTICES.md](https://github.com/Flame-Chasers/ViSeLA/blob/main/THIRD_PARTY_NOTICES.md) and the original dataset and pretrained-model providers before use.
|
| 113 |
+
|
| 114 |
+
The intended use is academic research on multilingual text-based person retrieval and cross-modal representation learning. Retrieval scores describe visual–textual similarity within the evaluated gallery. They are not evidence of a person's identity. Benchmark results reflect the datasets and language distributions evaluated in the paper; new galleries and translated queries require their own evaluation. Use person images only with the permissions required by their source and applicable terms.
|
| 115 |
+
|
| 116 |
+
## Citation
|
| 117 |
+
|
| 118 |
+
```bibtex
|
| 119 |
+
@inproceedings{zhou2026visela,
|
| 120 |
+
title={Vision-Grounded Semantic Learning and Alignment for Scalable Multilingual Text-Based Person Retrieval},
|
| 121 |
+
author={Zhou, Xinyu and Li, Fan and Chen, Shasha and Cao, Min and Ahmat, Ahtamjan and Zhu, Qiaoming and Ye, Mang},
|
| 122 |
+
booktitle={Proceedings of the 34th ACM International Conference on Multimedia (MM '26)},
|
| 123 |
+
year={2026},
|
| 124 |
+
doi={10.1145/3767308.3836472},
|
| 125 |
+
url={https://arxiv.org/abs/2611.15328},
|
| 126 |
+
note={Oral Presentation, Outstanding Paper Award}
|
| 127 |
+
}
|
| 128 |
+
```
|
SHA256SUMS.txt
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
3776aea5fe499d8c0365173fb73e662ae56edc2314068053f977f1506d22a079 visela_cuhk_pedes_baseline_router.pth
|
| 2 |
+
acb9fd1da89e3270a82fe50b0acc0444fb8a6869440ea73bcdaf1e7d2d922ade config.yaml
|
| 3 |
+
86bc91f38a3e3420941d8aa6f6b571d20e57a048713f66f582d91f951756444b config.json
|
| 4 |
+
fdf6edf0d09f2631643ad58c808e533fb03bf2c39e50c6b0149ab3948eb197a5 configs/config_beit2_base.json
|
config.json
ADDED
|
@@ -0,0 +1,24 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"model_type": "visela",
|
| 3 |
+
"vision_backbone": "beit2_base_patch16_224",
|
| 4 |
+
"text_encoder": "xlm-roberta-base",
|
| 5 |
+
"text_encoder_hub_id": "FacebookAI/xlm-roberta-base",
|
| 6 |
+
"num_experts": 4,
|
| 7 |
+
"num_queries": 4,
|
| 8 |
+
"image_res": 224,
|
| 9 |
+
"patch_size": 16,
|
| 10 |
+
"hidden_size": 768,
|
| 11 |
+
"text_num_hidden_layers": 18,
|
| 12 |
+
"text_fusion_start_at": 12,
|
| 13 |
+
"expert_blocks": [
|
| 14 |
+
11
|
| 15 |
+
],
|
| 16 |
+
"sim_type": "avg",
|
| 17 |
+
"routing_temp": 0.85,
|
| 18 |
+
"checkpoint_file": "visela_cuhk_pedes_baseline_router.pth",
|
| 19 |
+
"config_file": "config.yaml",
|
| 20 |
+
"vision_config": "configs/config_beit2_base.json",
|
| 21 |
+
"checkpoint_role": "cuhk_pedes_multilingual_finetuning",
|
| 22 |
+
"framework": "pytorch",
|
| 23 |
+
"source_release": "https://github.com/Flame-Chasers/ViSeLA/releases/tag/v1.0.0"
|
| 24 |
+
}
|
config.yaml
ADDED
|
@@ -0,0 +1,86 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
device: cuda:0
|
| 2 |
+
misc:
|
| 3 |
+
seed: 1
|
| 4 |
+
experiment:
|
| 5 |
+
input_resolution:
|
| 6 |
+
- 224
|
| 7 |
+
- 224
|
| 8 |
+
text_length: 77
|
| 9 |
+
schedule:
|
| 10 |
+
lr: 5.0e-05
|
| 11 |
+
epoch: 10
|
| 12 |
+
epoch_warmup: 1
|
| 13 |
+
lr_start: 1.0e-06
|
| 14 |
+
lr_end: 5.0e-06
|
| 15 |
+
weight_decay: 0.02
|
| 16 |
+
betas:
|
| 17 |
+
- 0.9
|
| 18 |
+
- 0.98
|
| 19 |
+
eps: 1.0e-08
|
| 20 |
+
ratio_factor: 1.0
|
| 21 |
+
niter_per_ep: 1064
|
| 22 |
+
refiner:
|
| 23 |
+
enable_sdm: false
|
| 24 |
+
num_query: 4
|
| 25 |
+
num_hidden_layers: 2
|
| 26 |
+
hidden_size: 768
|
| 27 |
+
sim_type: avg
|
| 28 |
+
model:
|
| 29 |
+
resume: models/CCLM-X2VLM/cclm_x2vlm_base.th
|
| 30 |
+
saved_path: ckpts/CUHK-PEDES/baseline_router
|
| 31 |
+
use_gather: true
|
| 32 |
+
embed_dim: 256
|
| 33 |
+
temp: 0.07
|
| 34 |
+
k_test: 256
|
| 35 |
+
itm:
|
| 36 |
+
weight: 4.0
|
| 37 |
+
use_mask: false
|
| 38 |
+
mim:
|
| 39 |
+
weight: 4.0
|
| 40 |
+
use_mim: false
|
| 41 |
+
mask_ratio: 0.5
|
| 42 |
+
mlm:
|
| 43 |
+
is_mlm: true
|
| 44 |
+
mask_prob: 0.4
|
| 45 |
+
max_masks: 12
|
| 46 |
+
skipgram_prb: 0.2
|
| 47 |
+
skipgram_size: 3
|
| 48 |
+
expert_blocks:
|
| 49 |
+
- 11
|
| 50 |
+
num_of_experts: 4
|
| 51 |
+
routing_temp: 0.85
|
| 52 |
+
sdm:
|
| 53 |
+
use_sdm: false
|
| 54 |
+
id:
|
| 55 |
+
use_id: false
|
| 56 |
+
vision_config: configs/config_beit2_base.json
|
| 57 |
+
use_clip_vit: false
|
| 58 |
+
use_beit_v2: true
|
| 59 |
+
image_res: 224
|
| 60 |
+
patch_size: 16
|
| 61 |
+
text_encoder: FacebookAI/xlm-roberta-base
|
| 62 |
+
text_num_hidden_layers: 18
|
| 63 |
+
text_fusion_start_at: 12
|
| 64 |
+
use_one_cl_proj_only: false
|
| 65 |
+
log:
|
| 66 |
+
print_period: 200
|
| 67 |
+
data:
|
| 68 |
+
batch_size: 64
|
| 69 |
+
test_batch_size: 128
|
| 70 |
+
num_workers: 8
|
| 71 |
+
languages:
|
| 72 |
+
- en
|
| 73 |
+
- ch
|
| 74 |
+
- fr
|
| 75 |
+
- de
|
| 76 |
+
use_code_switch: true
|
| 77 |
+
code_switch_dir: data/CUHK/code_switch
|
| 78 |
+
code_switch_prob: 0.3
|
| 79 |
+
distributed:
|
| 80 |
+
backend: nccl
|
| 81 |
+
url: env://
|
| 82 |
+
rank: 0
|
| 83 |
+
world_size: 4
|
| 84 |
+
local_rank: 0
|
| 85 |
+
anno_dir: data/CUHK
|
| 86 |
+
image_dir: data/CUHK/imgs
|
configs/config_beit2_base.json
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"ckpt": "data/beitv2_base_patch16_224_pt1k_ft21k.pth",
|
| 3 |
+
"vision_width": 768,
|
| 4 |
+
"patch_size": 16
|
| 5 |
+
}
|
visela_cuhk_pedes_baseline_router.pth
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3776aea5fe499d8c0365173fb73e662ae56edc2314068053f977f1506d22a079
|
| 3 |
+
size 1759538597
|