Publish portable inference-only Dispider checkpoint

README.md

@@ -1,50 +1,120 @@
 ---
 license: apache-2.0
+library_name: transformers
 pipeline_tag: video-text-to-text
+inference: false
+tags:
+  - video
+  - streaming-video
+  - multimodal
+  - qwen2
+  - cvpr-2025
 ---
 
-<p align="center">
-    <b><font size="6">Dispider</font></b> 
-<p>
+# Dispider
+
+Official checkpoint for **Dispider: Enabling Video LLMs with Active Real-Time
+Interaction via Disentangled Perception, Decision, and Reaction** (CVPR 2025).
+
+- [Code](https://github.com/Mark12Ding/Dispider)
+- [Paper](https://arxiv.org/abs/2501.03218)
+
+Dispider continuously perceives a video stream, uses a compact Decision model
+to determine when a response is needed, and invokes the larger Reaction model
+only after a trigger.
+
+## Architecture
+
+| Component | Purpose | Backbone |
+| --- | --- | --- |
+| Perception | Encode each sampled 16-frame clip | CLIP vision tower |
+| Perception-Decision | Build temporal memory | Compact Qwen2-1.5B |
+| Decision | Trigger or remain silent | Decision head on the compact model |
+| Reaction | Generate the response | Qwen2-7B |
+
+## Checkpoint layout
+
+The composite shards contain all model weights exactly once. The nested
+`perception_decision/` directory contains metadata only.
+
+```text
+Dispider/
+├── config.json
+├── dispider_checkpoint_manifest.json
+├── model-00001-of-00004.safetensors
+├── model-00002-of-00004.safetensors
+├── model-00003-of-00004.safetensors
+├── model-00004-of-00004.safetensors
+├── model.safetensors.index.json
+└── perception_decision/
+    ├── config.json
+    ├── tokenizer metadata
+    └── vision_tower/
+        ├── config.json
+        └── preprocessor_config.json
+```
+
+Download it directly:
+
+```bash
+huggingface-cli download Mar2Ding/Dispider \
+  --local-dir checkpoints/Dispider
+```
+
+Use the code repository with Python 3.10, PyTorch 2.2.0, FlashAttention
+2.5.9.post1, and Transformers 4.41.2. Load either the repository ID or a local
+snapshot through `dispider.model.load_pretrained_model`; generic
+`AutoModel.from_pretrained()` does not assemble the video components.
+The accompanying code is an inference-only runtime.
+
+## Offline inference
+
+```bash
+CUDA_VISIBLE_DEVICES=0 python inference.py \
+  --model_path Mar2Ding/Dispider \
+  --video_path /path/to/video.mp4 \
+  --prompt "What is happening in the video?"
+```
+
+## Streaming inference
+
+```bash
+CUDA_VISIBLE_DEVICES=0 python stream_inference.py /path/to/video.mp4 \
+  --model Mar2Ding/Dispider \
+  --prompt "Tell me when the person starts performing." \
+  --decision-kv-cache auto
+```
+
+The command emits timestamped JSON responses as Decision triggers Reaction.
+The optional Decision KV cache stores only stable prefixes and automatically
+falls back to full computation near the trigger threshold.
+
+## OVO-Bench example
+
+From the code repository:
+
+```bash
+python scripts/evaluate_ovobench.py \
+  --ovo-root ../OVO-Bench \
+  --model-path checkpoints/Dispider \
+  --data-root /path/to/ovo-data \
+  --output-dir outputs/ovobench \
+  --gpus 0,1,2,3 \
+  --resume
+```
+
+The evaluator validates complete coverage, rejects missing or null outputs,
+and computes the official category metrics. Generated results remain local.
+
+## Citation
 
-<div align="center">
-  
-[💻Github Repo](https://github.com/Mark12Ding/Dispider)
-
-[📖Paper](https://arxiv.org/abs/2501.03218)
-
-</div>
-
-
-## Quick Start
-First download the checkpoints at the folder. 
-
-
-**Important**: Modify the ``mm_compressor`` path in config.json to align with your local environment. The checkpoint for ``mm_compressor`` is located within a sub-folder of this repository.
-
-For detailed evaluation, please refer to [Github repo](https://github.com/Mark12Ding/Dispider).
-
-
-
-
-
-
-## ✒️ Citation
-If you find our work helpful for your research, please consider giving a star ⭐ and citation 📝.
 ```bibtex
-@article{qian2025dispider,
-        title={Dispider: Enabling Video LLMs with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction},
-        author={Qian, Rui and Ding, Shuangrui and Dong, Xiaoyi and Zhang, Pan and Zang, Yuhang and Cao, Yuhang and Lin, Dahua and Wang, Jiaqi},
-        journal={arXiv preprint arXiv:2501.03218},
-        year={2025}
-      }
-
-@article{qian2025streaming,
-        title={Streaming long video understanding with large language models},
-        author={Qian, Rui and Dong, Xiaoyi and Zhang, Pan and Zang, Yuhang and Ding, Shuangrui and Lin, Dahua and Wang, Jiaqi},
-        journal={Advances in Neural Information Processing Systems},
-        volume={37},
-        pages={119336--119360},
-        year={2025}
-      }
-```
+@inproceedings{qian2025dispider,
+  title={Dispider: Enabling Video {LLMs} with Active Real-Time Interaction via Disentangled Perception, Decision, and Reaction},
+  author={Qian, Rui and Ding, Shuangrui and Dong, Xiaoyi and Zhang, Pan and Zang, Yuhang and Cao, Yuhang and Lin, Dahua and Wang, Jiaqi},
+  booktitle={Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)},
+  pages={24045--24055},
+  month={June},
+  year={2025}
+}
+```

config.json

@@ -1,31 +1,26 @@
 {
-  "_name_or_path": "Dispider_ckpt",
+  "_name_or_path": ".",
   "architectures": [
-    "LongQwen2ForCausalLM"
+    "Reaction"
   ],
   "attention_dropout": 0.0,
   "bos_token_id": 151643,
-  "clip_hidden_size": 4096,
-  "clip_projector_type": "mlp2x_gelu",
-  "compress_hidden_size": 1536,
-  "compress_projector_lr": null,
-  "compress_projector_type": "mlp2x_gelu",
+  "checkpoint_format_version": 2,
+  "decision_hidden_size": 1536,
+  "decision_projector_type": "mlp2x_gelu",
   "eos_token_id": 151645,
-  "freeze_compress_mlp_adapter": false,
   "hidden_act": "silu",
   "hidden_size": 3584,
-  "image_aspect_ratio": "square",
   "initializer_range": 0.02,
   "intermediate_size": 18944,
   "max_position_embeddings": 32768,
-  "max_window_layers": 28,
-  "mm_compressor": "/mnt/petrelfs/dingshuangrui/LLaVA/checkpoints/long-qwen-16f-64r-4p-16-causal-qwen-streaming-final/stream_compressor",
-  "mm_use_im_patch_token": false,
-  "mm_use_im_start_end": false,
-  "model_type": "qwen2",
+  "model_type": "dispider_reaction",
   "num_attention_heads": 28,
   "num_hidden_layers": 28,
   "num_key_value_heads": 4,
+  "perception_decision": "perception_decision",
+  "perception_hidden_size": 4096,
+  "perception_projector_type": "mlp2x_gelu",
   "rms_norm_eps": 1e-06,
   "rope_theta": 1000000.0,
   "sliding_window": 131072,
@@ -34,10 +29,7 @@
   "tokenizer_padding_side": "right",
   "torch_dtype": "bfloat16",
   "transformers_version": "4.41.2",
-  "tune_clip_mlp_adapter": false,
-  "tune_compress_mlp_adapter": false,
   "use_cache": true,
-  "use_compress_proj": true,
   "use_sliding_window": false,
   "vocab_size": 152064
 }

dispider_checkpoint_manifest.json

@@ -0,0 +1,39 @@
+{
+  "embedded_perception_decision": {
+    "composite_files": [
+      "model-00001-of-00004.safetensors",
+      "model-00002-of-00004.safetensors",
+      "model-00003-of-00004.safetensors",
+      "model-00004-of-00004.safetensors"
+    ],
+    "state_dict_prefix": "model.compressor.compressor.",
+    "verification": "dtype+shape+torch.equal",
+    "verified_key_count": 735
+  },
+  "files": [
+    ".gitattributes",
+    "README.md",
+    "added_tokens.json",
+    "config.json",
+    "generation_config.json",
+    "merges.txt",
+    "model-00001-of-00004.safetensors",
+    "model-00002-of-00004.safetensors",
+    "model-00003-of-00004.safetensors",
+    "model-00004-of-00004.safetensors",
+    "model.safetensors.index.json",
+    "perception_decision/added_tokens.json",
+    "perception_decision/config.json",
+    "perception_decision/generation_config.json",
+    "perception_decision/merges.txt",
+    "perception_decision/special_tokens_map.json",
+    "perception_decision/tokenizer_config.json",
+    "perception_decision/vision_tower/config.json",
+    "perception_decision/vision_tower/preprocessor_config.json",
+    "perception_decision/vocab.json",
+    "special_tokens_map.json",
+    "tokenizer_config.json",
+    "vocab.json"
+  ],
+  "format_version": 2
+}

{stream_compressor → perception_decision}/config.json

@@ -1,31 +1,26 @@
 {
-  "_name_or_path": "/mnt/hwfile/mllm/qianrui/checkpoints/llava-qwen-2-pool-16f-64r-4p-causal-vary-CN",
+  "_name_or_path": "perception_decision",
   "architectures": [
-    "StreamGroundQwenForCausalLM"
+    "Decision"
   ],
   "attention_dropout": 0.0,
   "bos_token_id": 151643,
+  "checkpoint_format_version": 2,
   "eos_token_id": 151645,
-  "freeze_mm_mlp_adapter": false,
   "hidden_act": "silu",
   "hidden_size": 1536,
-  "image_aspect_ratio": "square",
   "initializer_range": 0.02,
   "intermediate_size": 8960,
   "max_position_embeddings": 32768,
-  "max_window_layers": 28,
-  "mm_hidden_size": 1024,
-  "mm_projector_lr": null,
-  "mm_projector_type": "pool",
-  "mm_use_im_patch_token": false,
-  "mm_use_im_start_end": false,
-  "mm_vision_select_feature": "patch",
-  "mm_vision_select_layer": -2,
-  "mm_vision_tower": "openai/clip-vit-large-patch14",
-  "model_type": "stream_ground_qwen",
+  "model_type": "dispider_decision",
   "num_attention_heads": 12,
   "num_hidden_layers": 28,
   "num_key_value_heads": 2,
+  "perception_vision_tower": "vision_tower",
+  "perception_hidden_size": 1024,
+  "perception_projector_type": "pool",
+  "perception_select_feature": "patch",
+  "perception_select_layer": -2,
   "pool_num": 4,
   "resolution": 64,
   "rms_norm_eps": 1e-06,
@@ -36,9 +31,7 @@
   "tokenizer_padding_side": "right",
   "torch_dtype": "bfloat16",
   "transformers_version": "4.41.2",
-  "tune_mm_mlp_adapter": false,
   "use_cache": true,
-  "use_mm_proj": true,
   "use_sliding_window": false,
   "vocab_size": 151936
 }

perception_decision/vision_tower/config.json

@@ -0,0 +1,171 @@
+{
+  "_name_or_path": "clip-vit-large-patch14/",
+  "architectures": [
+    "CLIPModel"
+  ],
+  "initializer_factor": 1.0,
+  "logit_scale_init_value": 2.6592,
+  "model_type": "clip",
+  "projection_dim": 768,
+  "text_config": {
+    "_name_or_path": "",
+    "add_cross_attention": false,
+    "architectures": null,
+    "attention_dropout": 0.0,
+    "bad_words_ids": null,
+    "bos_token_id": 0,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "dropout": 0.0,
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": 2,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "hidden_act": "quick_gelu",
+    "hidden_size": 768,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "initializer_factor": 1.0,
+    "initializer_range": 0.02,
+    "intermediate_size": 3072,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "layer_norm_eps": 1e-05,
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "max_position_embeddings": 77,
+    "min_length": 0,
+    "model_type": "clip_text_model",
+    "no_repeat_ngram_size": 0,
+    "num_attention_heads": 12,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_hidden_layers": 12,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": 1,
+    "prefix": null,
+    "problem_type": null,
+    "projection_dim" : 768,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "sep_token_id": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": null,
+    "torchscript": false,
+    "transformers_version": "4.16.0.dev0",
+    "use_bfloat16": false,
+    "vocab_size": 49408
+  },
+  "text_config_dict": {
+    "hidden_size": 768,
+    "intermediate_size": 3072,
+    "num_attention_heads": 12,
+    "num_hidden_layers": 12,
+    "projection_dim": 768
+  },
+  "torch_dtype": "float32",
+  "transformers_version": null,
+  "vision_config": {
+    "_name_or_path": "",
+    "add_cross_attention": false,
+    "architectures": null,
+    "attention_dropout": 0.0,
+    "bad_words_ids": null,
+    "bos_token_id": null,
+    "chunk_size_feed_forward": 0,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "diversity_penalty": 0.0,
+    "do_sample": false,
+    "dropout": 0.0,
+    "early_stopping": false,
+    "encoder_no_repeat_ngram_size": 0,
+    "eos_token_id": null,
+    "finetuning_task": null,
+    "forced_bos_token_id": null,
+    "forced_eos_token_id": null,
+    "hidden_act": "quick_gelu",
+    "hidden_size": 1024,
+    "id2label": {
+      "0": "LABEL_0",
+      "1": "LABEL_1"
+    },
+    "image_size": 224,
+    "initializer_factor": 1.0,
+    "initializer_range": 0.02,
+    "intermediate_size": 4096,
+    "is_decoder": false,
+    "is_encoder_decoder": false,
+    "label2id": {
+      "LABEL_0": 0,
+      "LABEL_1": 1
+    },
+    "layer_norm_eps": 1e-05,
+    "length_penalty": 1.0,
+    "max_length": 20,
+    "min_length": 0,
+    "model_type": "clip_vision_model",
+    "no_repeat_ngram_size": 0,
+    "num_attention_heads": 16,
+    "num_beam_groups": 1,
+    "num_beams": 1,
+    "num_hidden_layers": 24,
+    "num_return_sequences": 1,
+    "output_attentions": false,
+    "output_hidden_states": false,
+    "output_scores": false,
+    "pad_token_id": null,
+    "patch_size": 14,
+    "prefix": null,
+    "problem_type": null,
+    "projection_dim" : 768,
+    "pruned_heads": {},
+    "remove_invalid_values": false,
+    "repetition_penalty": 1.0,
+    "return_dict": true,
+    "return_dict_in_generate": false,
+    "sep_token_id": null,
+    "task_specific_params": null,
+    "temperature": 1.0,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenizer_class": null,
+    "top_k": 50,
+    "top_p": 1.0,
+    "torch_dtype": null,
+    "torchscript": false,
+    "transformers_version": "4.16.0.dev0",
+    "use_bfloat16": false
+  },
+  "vision_config_dict": {
+    "hidden_size": 1024,
+    "intermediate_size": 4096,
+    "num_attention_heads": 16,
+    "num_hidden_layers": 24,
+    "patch_size": 14,
+    "projection_dim": 768
+  }
+}

perception_decision/vision_tower/preprocessor_config.json

@@ -0,0 +1,19 @@
+{
+  "crop_size": 224,
+  "do_center_crop": true,
+  "do_normalize": true,
+  "do_resize": true,
+  "feature_extractor_type": "CLIPFeatureExtractor",
+  "image_mean": [
+    0.48145466,
+    0.4578275,
+    0.40821073
+  ],
+  "image_std": [
+    0.26862954,
+    0.26130258,
+    0.27577711
+  ],
+  "resample": 3,
+  "size": 224
+}

stream_compressor/model.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:dae10330fbf1f2b7201903d840701ce035de69530645bcbc1e9cac3f9abbd9e6
-size 4177946928

training_args.bin

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:e471f601782d00eb4fecfbdb7b2253a1151dd38b3cd18b70108b018465207fb0
-size 6776