PyTorch
LIU1712 commited on
Commit
7c42ae2
Β·
verified Β·
1 Parent(s): d626114

Upload README.md

Browse files
Files changed (1) hide show
  1. README.md +140 -0
README.md CHANGED
@@ -1,3 +1,143 @@
1
  ---
2
  license: apache-2.0
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+ library_name: pytorch
4
  ---
5
+
6
+ # DAAN: Mitigating Information Asymmetries for Point-Supervised Video Moment Retrieval (TPAMI 2026)
7
+
8
+ **Hao Liu**<sup>1</sup>, **Yupeng Hu**<sup>1</sup>, **Kun Wang**<sup>1</sup>, **Junchao Wang**<sup>1</sup>, **Zhiran Li**<sup>1</sup>, **Liqiang Nie**<sup>2</sup>
9
+
10
+ <sup>1</sup> School of Software, Shandong University, Jinan, China
11
+ <sup>2</sup> School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen, China
12
+
13
+ This repository hosts the official pretrained checkpoints for **DAAN**, a point-supervised Video Moment Retrieval (VMR) framework that mitigates Semantic-Positional Asymmetry and Instance-Duration Asymmetry through **Center-Guided Calibration (CGC)** and a **Duration-Aware Proposal Discriminator (DPD)**.
14
+
15
+ πŸ”— **Paper:** [IEEE TPAMI 2026](https://doi.org/10.1109/TPAMI.2026.3742682)
16
+ πŸ”— **GitHub Repository:** [iLearn-Lab/TPAMI26-DAAN](https://github.com/iLearn-Lab/TPAMI26-DAAN)
17
+
18
+ ---
19
+
20
+ ## Model Information
21
+
22
+ ### 1. Model Name
23
+
24
+ **DAAN** (Dual-Asymmetry Aligned Network).
25
+
26
+ ### 2. Task Type & Applicable Tasks
27
+
28
+ - **Task Type:** Point-Supervised Video Moment Retrieval (VMR) / Temporal Localization / Vision-Language Learning.
29
+ - **Applicable Tasks:** Locating the temporal interval in an untrimmed video that matches a natural language query, using only a single annotated frame per training instance instead of full temporal boundaries.
30
+
31
+ ### 3. Project Introduction
32
+
33
+ Point-supervised VMR substantially reduces annotation effort but introduces two inherent information asymmetries:
34
+
35
+ - **Semantic-Positional Asymmetry:** An annotated frame confirms the event semantics but does not reveal its relative position within the target moment.
36
+ - **Instance-Duration Asymmetry:** An annotated frame confirms the event's occurrence but provides no explicit information about its temporal duration.
37
+
38
+ DAAN addresses these challenges with two complementary components. **CGC** dynamically estimates a pseudo temporal center and recalibrates attention through local and global cross-modal cues, while **DPD** learns query-conditioned compatibility between proposal content and boundary representations to distinguish plausible temporal spans without boundary supervision.
39
+
40
+ ### 4. Training Data Source
41
+
42
+ DAAN is trained and evaluated on standard point-supervised VMR benchmarks:
43
+
44
+ - **Charades-STA** (I3D, C3D, and VGG visual features)
45
+ - **TACoS** (C3D visual features)
46
+ - **ActivityNet Captions** (C3D visual features)
47
+
48
+ ---
49
+
50
+ ## Usage & Basic Inference
51
+
52
+ This model repository provides pretrained weights and their corresponding configuration files. For the evaluation code and dataset preparation instructions, please refer to the [official GitHub repository](https://github.com/iLearn-Lab/TPAMI26-DAAN).
53
+
54
+ ### Step 1: Prepare the Environment
55
+
56
+ Clone the GitHub repository and install dependencies:
57
+
58
+ ```bash
59
+ git clone https://github.com/iLearn-Lab/TPAMI26-DAAN.git
60
+ cd TPAMI26-DAAN
61
+
62
+ python3.10 -m venv .venv
63
+ source .venv/bin/activate # Linux / Mac
64
+
65
+ pip install torch==2.3.0 torchtext==0.18.0 "numpy<2" scipy pyyaml tqdm matplotlib huggingface_hub
66
+ ```
67
+
68
+ Choose the PyTorch build appropriate for your CUDA environment.
69
+
70
+ ### Step 2: Download Model Weights & Data
71
+
72
+ Download the pretrained checkpoints from this Hugging Face repository:
73
+
74
+ ```bash
75
+ hf download iLearn-Lab/TPAMI26-DAAN --local-dir checkpoints/TPAMI26-DAAN
76
+ ```
77
+
78
+ The downloaded files are organized as follows:
79
+
80
+ ```text
81
+ checkpoints/TPAMI26-DAAN/
82
+ β”œβ”€β”€ cha-i3d/
83
+ β”‚ β”œβ”€β”€ config.yaml
84
+ β”‚ └── model_best.pt
85
+ β”œβ”€β”€ cha-c3d/
86
+ β”‚ β”œβ”€β”€ config.yaml
87
+ β”‚ └── model_best.pt
88
+ β”œβ”€β”€ cha-vgg/
89
+ β”‚ β”œβ”€β”€ config.yaml
90
+ β”‚ └── model_best.pt
91
+ └── tacos/
92
+ β”œβ”€β”€ config.yaml
93
+ └── model_best.pt
94
+ ```
95
+
96
+
97
+ Prepare video features and point annotations following [ViGA](https://github.com/r-cui/ViGA)'s dataset preparation protocol. **The benchmark datasets and extracted video features are not included in this model repository.**
98
+
99
+ Before evaluation, update the local paths in:
100
+
101
+ - `checkpoints/TPAMI26-DAAN/<checkpoint>/config.yaml`: `feature_dir` (and ensure `feature_dim` matches the extracted features).
102
+ - `src/utils/utils.py`: annotation directories for the corresponding datasets.
103
+
104
+
105
+ ### Step 3: Run Inference
106
+
107
+ Evaluate a downloaded checkpoint from the root of the GitHub repository:
108
+
109
+ ```bash
110
+ python -m src.experiment.eval --exp checkpoints/TPAMI26-DAAN/cha-c3d
111
+ ```
112
+
113
+
114
+ ---
115
+
116
+ ## Limitations & Notes
117
+
118
+ **Disclaimer:** These checkpoints are provided for academic research purposes.
119
+
120
+ - The model requires access to the original benchmark datasets and extracted video features for evaluation.
121
+ - Some configuration files currently contain local path settings and should be updated before use.
122
+
123
+ ---
124
+
125
+ ## Citation
126
+
127
+ If you find our work useful in your research, please consider citing our paper:
128
+
129
+ ```bibtex
130
+ @article{liu2026daan,
131
+ title={DAAN: Mitigating Information Asymmetries for Point-Supervised Video Moment Retrieval},
132
+ author={Liu, Hao and Hu, Yupeng and Wang, Kun and Wang, Junchao and Li, Zhiran and Nie, Liqiang},
133
+ journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
134
+ year={2026},
135
+ doi={10.1109/TPAMI.2026.3742682}
136
+ }
137
+ ```
138
+
139
+ ---
140
+
141
+ ## Contact
142
+
143
+ If you have any questions, feel free to contact me at **liuh90210@gmail.com**.