linda23153 commited on
Commit
cb15977
·
verified ·
1 Parent(s): 5865aca

Publish official ViSeLA v1.0.0 checkpoint, portable configs and model card

Browse files
README.md ADDED
@@ -0,0 +1,128 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ license_name: research-only-upstream-terms
4
+ license_link: https://github.com/Flame-Chasers/ViSeLA/blob/main/THIRD_PARTY_NOTICES.md
5
+ language: [en, zh, fr, de]
6
+ tags: [arxiv:2611.15328, person-re-identification, text-to-image-retrieval, multilingual, vision-language, pytorch, acm-mm-2026]
7
+ ---
8
+ # ViSeLA-CUHK-PEDES-Baseline-Router
9
+
10
+ **ACM Multimedia 2026 · Oral Presentation & Outstanding Paper Award**
11
+
12
+ [Paper](https://arxiv.org/abs/2611.15328) · [ACM DOI](https://doi.org/10.1145/3767308.3836472) · [Project page](https://flame-chasers.github.io/ViSeLA/) · [Code](https://github.com/Flame-Chasers/ViSeLA) · [Release v1.0.0](https://github.com/Flame-Chasers/ViSeLA/releases/tag/v1.0.0)
13
+
14
+ **Vision-Grounded Semantic Learning and Alignment for Scalable Multilingual Text-Based Person Retrieval**
15
+ Xinyu Zhou, Fan Li, Shasha Chen, Min Cao, Ahtamjan Ahmat, Qiaoming Zhu, and Mang Ye.
16
+
17
+ ViSeLA retrieves person images from descriptions in English, Chinese, French, and German. Semantic-aware multilingual learning routes text tokens to experts by their semantic content. Vision-grounded alignment uses shared learnable queries to align corresponding visual and textual cues. The released architecture combines a BEiT-2 base vision encoder, XLM-RoBERTa text encoding, and multimodal interaction for reranking.
18
+
19
+ ## Checkpoint identity and evaluation
20
+
21
+ This is the official **CUHK-PEDES(M) multilingual baseline-router fine-tuning checkpoint**, initialized from **CCLM-X2VLM**. It uses four experts, four learnable queries, `refiner.sim_type: avg`, and `routing_temp: 0.85`. Its stored fields are `state_dict` and `config`.
22
+
23
+ The released checkpoint's reported retrieval results are:
24
+
25
+ | Language | R@1 (%) |
26
+ | --- | ---: |
27
+ | English (EN) | 78.22 |
28
+ | Chinese (ZH) | 76.07 |
29
+ | French (FR) | 76.74 |
30
+ | German (DE) | 76.10 |
31
+ | **Average** | **76.78** |
32
+
33
+ **Average mAP: 68.67%.** Values describe the released checkpoint's CUHK-PEDES(M) evaluation; they are reported release results, not a new evaluation performed for this mirror.
34
+
35
+ For comparison, [Table 2 of the paper](https://arxiv.org/html/2611.15328v1#S5) reports **77.94% Avg R@1 / 70.16% Avg mAP** for ViSeLA with **full MultiPedes pretraining and `lsp_max` similarity**. The baseline-router file here uses CCLM-X2VLM initialization and `avg` similarity. Keep these training settings and results separate when citing or comparing the models. The paper's ViSeLA row without person-centric pretraining (76.83% / 68.71%) also describes a different configuration from this released baseline router.
36
+
37
+ ## Download and use
38
+
39
+ Install the official code environment following the [ViSeLA installation instructions](https://github.com/Flame-Chasers/ViSeLA#quick-start). Download the checkpoint and its configuration with `huggingface_hub`:
40
+
41
+ ```python
42
+ from huggingface_hub import hf_hub_download
43
+
44
+ repo_id = "linda23153/ViSeLA-CUHK-PEDES-Baseline-Router"
45
+ local_dir = "checkpoints/ViSeLA-CUHK-PEDES-Baseline-Router"
46
+ for filename in (
47
+ "config.json",
48
+ "visela_cuhk_pedes_baseline_router.pth",
49
+ "config.yaml",
50
+ "configs/config_beit2_base.json",
51
+ "SHA256SUMS.txt",
52
+ ):
53
+ path = hf_hub_download(repo_id=repo_id, filename=filename, local_dir=local_dir)
54
+ print(path)
55
+ ```
56
+
57
+ Run these shell commands from a clone of the official code repository, with the downloaded `checkpoints/` directory under that clone:
58
+
59
+ ```bash
60
+ python tools/demo_retrieval.py \
61
+ --checkpoint checkpoints/ViSeLA-CUHK-PEDES-Baseline-Router/visela_cuhk_pedes_baseline_router.pth \
62
+ --config checkpoints/ViSeLA-CUHK-PEDES-Baseline-Router/config.yaml \
63
+ --text-encoder FacebookAI/xlm-roberta-base \
64
+ --top-k 3
65
+ ```
66
+
67
+ The demo reads the bundled gallery and multilingual queries by default; use `--image-dir` and `--query` for a custom gallery. The loader resolves the bundled BEiT-2 architecture configuration when the YAML path is absent from the current working directory. This is a custom PyTorch checkpoint loaded by the official ViSeLA code. `config.json` records architecture and file metadata for Hub clients.
68
+
69
+ ### Full benchmark evaluation
70
+
71
+ Copy the downloaded `config.yaml` to `local_baseline_router.yaml` in the official code checkout. Set `anno_dir` and `image_dir` to your CUHK-PEDES annotations and images, `data.code_switch_dir` to the code-switch annotations, and `model.saved_path` to a writable output directory. Set `vision_config` to `config/config_beit2_base.json` in the code repository. Keep the exported `avg` similarity and routing temperature 0.85.
72
+
73
+ ```bash
74
+ python main.py --config local_baseline_router.yaml \
75
+ --eval-only \
76
+ --resume checkpoints/ViSeLA-CUHK-PEDES-Baseline-Router/visela_cuhk_pedes_baseline_router.pth
77
+ ```
78
+
79
+ Obtain the benchmark images from the original dataset provider; the linked benchmark repository supplies annotations. Downstream code uses `ch` as the Chinese filename/language key; this corresponds to ZH in the metrics table.
80
+
81
+ ## Files and provenance
82
+
83
+ | File | Contents |
84
+ | --- | --- |
85
+ | `visela_cuhk_pedes_baseline_router.pth` | Original GitHub v1.0.0 PyTorch weights; byte-for-byte unchanged |
86
+ | `config.yaml` | Embedded checkpoint configuration exported as plain YAML, with portable paths |
87
+ | `config.json` | Architecture and checkpoint summary |
88
+ | `configs/config_beit2_base.json` | Official BEiT-2 architecture configuration |
89
+ | `SHA256SUMS.txt` | SHA256 checksums for the weight and three configuration files |
90
+
91
+ Weight SHA256: `3776aea5fe499d8c0365173fb73e662ae56edc2314068053f977f1506d22a079`.
92
+
93
+ The YAML preserves the embedded architecture and training hyperparameters. Machine-local dataset and output paths are replaced by `/path/to/…` placeholders or relative paths; the text encoder uses its public Hub identifier. The raw weight retains its original embedded configuration. Inspect and set paths before training or benchmark evaluation.
94
+
95
+ Verify from the downloaded model directory:
96
+
97
+ ```bash
98
+ sha256sum -c SHA256SUMS.txt
99
+ ```
100
+
101
+ ## Related official resources
102
+
103
+ | Resource | Repository |
104
+ | --- | --- |
105
+ | MultiPedes pretraining annotations | [linda23153/MultiPedes](https://huggingface.co/datasets/linda23153/MultiPedes) |
106
+ | Multilingual downstream benchmarks | [linda23153/ViSeLA-Multilingual-Benchmarks](https://huggingface.co/datasets/linda23153/ViSeLA-Multilingual-Benchmarks) |
107
+ | HAM + LUPerson pretraining checkpoint | [linda23153/ViSeLA-Pretrain-HAM-LUPerson](https://huggingface.co/linda23153/ViSeLA-Pretrain-HAM-LUPerson) |
108
+ | CUHK-PEDES baseline-router checkpoint | [linda23153/ViSeLA-CUHK-PEDES-Baseline-Router](https://huggingface.co/linda23153/ViSeLA-CUHK-PEDES-Baseline-Router) |
109
+
110
+ ## License and intended use
111
+
112
+ The ViSeLA source code is MIT licensed. These model weights remain subject to the terms of the upstream pretrained weights and the academic research licenses of the pedestrian datasets used for training. The `research-only-upstream-terms` metadata describes these combined conditions; the code's MIT license does not grant unrestricted rights to the weights or source data. Consult the official [THIRD_PARTY_NOTICES.md](https://github.com/Flame-Chasers/ViSeLA/blob/main/THIRD_PARTY_NOTICES.md) and the original dataset and pretrained-model providers before use.
113
+
114
+ The intended use is academic research on multilingual text-based person retrieval and cross-modal representation learning. Retrieval scores describe visual–textual similarity within the evaluated gallery. They are not evidence of a person's identity. Benchmark results reflect the datasets and language distributions evaluated in the paper; new galleries and translated queries require their own evaluation. Use person images only with the permissions required by their source and applicable terms.
115
+
116
+ ## Citation
117
+
118
+ ```bibtex
119
+ @inproceedings{zhou2026visela,
120
+ title={Vision-Grounded Semantic Learning and Alignment for Scalable Multilingual Text-Based Person Retrieval},
121
+ author={Zhou, Xinyu and Li, Fan and Chen, Shasha and Cao, Min and Ahmat, Ahtamjan and Zhu, Qiaoming and Ye, Mang},
122
+ booktitle={Proceedings of the 34th ACM International Conference on Multimedia (MM '26)},
123
+ year={2026},
124
+ doi={10.1145/3767308.3836472},
125
+ url={https://arxiv.org/abs/2611.15328},
126
+ note={Oral Presentation, Outstanding Paper Award}
127
+ }
128
+ ```
SHA256SUMS.txt ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ 3776aea5fe499d8c0365173fb73e662ae56edc2314068053f977f1506d22a079 visela_cuhk_pedes_baseline_router.pth
2
+ acb9fd1da89e3270a82fe50b0acc0444fb8a6869440ea73bcdaf1e7d2d922ade config.yaml
3
+ 86bc91f38a3e3420941d8aa6f6b571d20e57a048713f66f582d91f951756444b config.json
4
+ fdf6edf0d09f2631643ad58c808e533fb03bf2c39e50c6b0149ab3948eb197a5 configs/config_beit2_base.json
config.json ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model_type": "visela",
3
+ "vision_backbone": "beit2_base_patch16_224",
4
+ "text_encoder": "xlm-roberta-base",
5
+ "text_encoder_hub_id": "FacebookAI/xlm-roberta-base",
6
+ "num_experts": 4,
7
+ "num_queries": 4,
8
+ "image_res": 224,
9
+ "patch_size": 16,
10
+ "hidden_size": 768,
11
+ "text_num_hidden_layers": 18,
12
+ "text_fusion_start_at": 12,
13
+ "expert_blocks": [
14
+ 11
15
+ ],
16
+ "sim_type": "avg",
17
+ "routing_temp": 0.85,
18
+ "checkpoint_file": "visela_cuhk_pedes_baseline_router.pth",
19
+ "config_file": "config.yaml",
20
+ "vision_config": "configs/config_beit2_base.json",
21
+ "checkpoint_role": "cuhk_pedes_multilingual_finetuning",
22
+ "framework": "pytorch",
23
+ "source_release": "https://github.com/Flame-Chasers/ViSeLA/releases/tag/v1.0.0"
24
+ }
config.yaml ADDED
@@ -0,0 +1,86 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ device: cuda:0
2
+ misc:
3
+ seed: 1
4
+ experiment:
5
+ input_resolution:
6
+ - 224
7
+ - 224
8
+ text_length: 77
9
+ schedule:
10
+ lr: 5.0e-05
11
+ epoch: 10
12
+ epoch_warmup: 1
13
+ lr_start: 1.0e-06
14
+ lr_end: 5.0e-06
15
+ weight_decay: 0.02
16
+ betas:
17
+ - 0.9
18
+ - 0.98
19
+ eps: 1.0e-08
20
+ ratio_factor: 1.0
21
+ niter_per_ep: 1064
22
+ refiner:
23
+ enable_sdm: false
24
+ num_query: 4
25
+ num_hidden_layers: 2
26
+ hidden_size: 768
27
+ sim_type: avg
28
+ model:
29
+ resume: models/CCLM-X2VLM/cclm_x2vlm_base.th
30
+ saved_path: ckpts/CUHK-PEDES/baseline_router
31
+ use_gather: true
32
+ embed_dim: 256
33
+ temp: 0.07
34
+ k_test: 256
35
+ itm:
36
+ weight: 4.0
37
+ use_mask: false
38
+ mim:
39
+ weight: 4.0
40
+ use_mim: false
41
+ mask_ratio: 0.5
42
+ mlm:
43
+ is_mlm: true
44
+ mask_prob: 0.4
45
+ max_masks: 12
46
+ skipgram_prb: 0.2
47
+ skipgram_size: 3
48
+ expert_blocks:
49
+ - 11
50
+ num_of_experts: 4
51
+ routing_temp: 0.85
52
+ sdm:
53
+ use_sdm: false
54
+ id:
55
+ use_id: false
56
+ vision_config: configs/config_beit2_base.json
57
+ use_clip_vit: false
58
+ use_beit_v2: true
59
+ image_res: 224
60
+ patch_size: 16
61
+ text_encoder: FacebookAI/xlm-roberta-base
62
+ text_num_hidden_layers: 18
63
+ text_fusion_start_at: 12
64
+ use_one_cl_proj_only: false
65
+ log:
66
+ print_period: 200
67
+ data:
68
+ batch_size: 64
69
+ test_batch_size: 128
70
+ num_workers: 8
71
+ languages:
72
+ - en
73
+ - ch
74
+ - fr
75
+ - de
76
+ use_code_switch: true
77
+ code_switch_dir: data/CUHK/code_switch
78
+ code_switch_prob: 0.3
79
+ distributed:
80
+ backend: nccl
81
+ url: env://
82
+ rank: 0
83
+ world_size: 4
84
+ local_rank: 0
85
+ anno_dir: data/CUHK
86
+ image_dir: data/CUHK/imgs
configs/config_beit2_base.json ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ {
2
+ "ckpt": "data/beitv2_base_patch16_224_pt1k_ft21k.pth",
3
+ "vision_width": 768,
4
+ "patch_size": 16
5
+ }
visela_cuhk_pedes_baseline_router.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3776aea5fe499d8c0365173fb73e662ae56edc2314068053f977f1506d22a079
3
+ size 1759538597