license: llama3.1
pipeline_tag: video-text-to-text
tags:
- video
- model-merging
- dcat
- llama-3.1
- llava
DCAT · LLaMA-3.1-8B · Vision → Video
Project Page | Paper (PDF) | GitHub
DCAT (Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs, NeurIPS 2026 Spotlight) transfers cross-modal alignment from a strong donor MLLM to a weaker recipient MLLM with a training-free, closed-form merge that uses only a few hundred unlabeled calibration samples. This checkpoint is a LLaVA-Video (LLaMA-3.1-8B) video MLLM after DCAT merging with a LLaVA LLaMA-3.1-8B vision MLLM.
This repository contains the full merged weights (LLaVA-NeXT format). Loading needs LLaVA-NeXT with the small patch shipped in the GitHub repository (third_party/llava_next_dcat.patch), which adds the aapool_mlp projector used by this checkpoint.
Usage
git clone https://github.com/Hyun1A/DCAT.git && cd DCAT
pip install -r requirements_llama.txt
git clone https://github.com/LLaVA-VL/LLaVA-NeXT.git third_party/LLaVA-NeXT
git -C third_party/LLaVA-NeXT checkout bce12e4
git -C third_party/LLaVA-NeXT apply ../llava_next_dcat.patch
pip install -e third_party/LLaVA-NeXT --no-deps
python quick_start/inference.py --setting llama-video --model Hyun1A/DCAT-LLaMA-Vision2Video \
--media sample.mp4 --question "What is the main content of this video?" --choices "option 1" "option 2" "option 3" "option 4"
See the GitHub repository for installation details, the inference code, and the scripts that reproduce the merge.
Citation
@inproceedings{seo2026dcat,
title = {Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs},
author = {Seo, Hoigi and Lee, Byung Hyun and Kim, Minjun and
Mah, Dohyun and Lee, Jongho and Chun, Se Young},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2026},
url = {https://arxiv.org/abs/2610.04580}
}