Hyun1A/DCAT-Vicuna-Vision2Video

DCAT · Vicuna-7B · Vision → Video

Project Page  |  Paper (PDF)  |  GitHub

DCAT (Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs, NeurIPS 2026 Spotlight) transfers cross-modal alignment from a strong donor MLLM to a weaker recipient MLLM with a training-free, closed-form merge that uses only a few hundred unlabeled calibration samples. This checkpoint is the Vicuna-7B-v1.5 video MLLM (ModelCompose video-naivemc) after DCAT merging with the vision MLLM of the same family.

This repository contains only the adapter weights (adapter_model.bin, ModelCompose format) and config.json. They are loaded on top of lmsys/vicuna-7b-v1.5; the LanguageBind video encoder is downloaded from LanguageBind/LanguageBind_Video_merge on first use.

Usage

git clone https://github.com/Hyun1A/DCAT.git && cd DCAT
pip install -r requirements_vicuna.txt

python quick_start/inference.py --setting vicuna-video --model Hyun1A/DCAT-Vicuna-Vision2Video \
    --media sample.mp4 --question "What is the main content of this video?" --choices "option 1" "option 2" "option 3" "option 4"

See the GitHub repository for installation details, the inference code, and the scripts that reproduce the merge.

Citation

@inproceedings{seo2026dcat,
  title     = {Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs},
  author    = {Seo, Hoigi and Lee, Byung Hyun and Kim, Minjun and
               Mah, Dohyun and Lee, Jongho and Chun, Se Young},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026},
  url       = {https://arxiv.org/abs/2610.04580}
}