---
license: llama3.1
pipeline_tag: video-text-to-text
tags:
  - video
  - model-merging
  - dcat
  - llama-3.1
  - llava
---

# DCAT · LLaMA-3.1-8B · Vision → Video

<p align="center">
  <a href="https://seohoiki3215.github.io/DCAT_project_page/"><b>Project Page</b></a> &nbsp;|&nbsp;
  <a href="https://arxiv.org/abs/2610.04580"><b>Paper (PDF)</b></a> &nbsp;|&nbsp;
  <a href="https://github.com/Hyun1A/DCAT"><b>GitHub</b></a>
</p>

**DCAT** (*Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs*, NeurIPS 2026 Spotlight)
transfers cross-modal alignment from a strong donor MLLM to a weaker recipient MLLM with a training-free,
closed-form merge that uses only a few hundred unlabeled calibration samples. This checkpoint is a LLaVA-Video (LLaMA-3.1-8B) **video** MLLM after DCAT merging with a LLaVA LLaMA-3.1-8B **vision** MLLM.

This repository contains the **full merged weights** (LLaVA-NeXT format). Loading needs LLaVA-NeXT with the small patch shipped in the GitHub repository (`third_party/llava_next_dcat.patch`), which adds the `aapool_mlp` projector used by this checkpoint.


## Usage

```bash
git clone https://github.com/Hyun1A/DCAT.git && cd DCAT
pip install -r requirements_llama.txt
git clone https://github.com/LLaVA-VL/LLaVA-NeXT.git third_party/LLaVA-NeXT
git -C third_party/LLaVA-NeXT checkout bce12e4
git -C third_party/LLaVA-NeXT apply ../llava_next_dcat.patch
pip install -e third_party/LLaVA-NeXT --no-deps

python quick_start/inference.py --setting llama-video --model Hyun1A/DCAT-LLaMA-Vision2Video \
    --media sample.mp4 --question "What is the main content of this video?" --choices "option 1" "option 2" "option 3" "option 4"
```

See the [GitHub repository](https://github.com/Hyun1A/DCAT#quick-start) for installation details, the
inference code, and the scripts that reproduce the merge.

## Citation

```bibtex
@inproceedings{seo2026dcat,
  title     = {Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs},
  author    = {Seo, Hoigi and Lee, Byung Hyun and Kim, Minjun and
               Mah, Dohyun and Lee, Jongho and Chun, Se Young},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026},
  url       = {https://arxiv.org/abs/2610.04580}
}
```