| --- |
| license: llama3.1 |
| pipeline_tag: video-text-to-text |
| tags: |
| - video |
| - model-merging |
| - dcat |
| - llama-3.1 |
| - llava |
| --- |
| |
| # DCAT · LLaMA-3.1-8B · Vision → Video |
|
|
| <p align="center"> |
| <a href="https://seohoiki3215.github.io/DCAT_project_page/"><b>Project Page</b></a> | |
| <a href="https://arxiv.org/abs/2610.04580"><b>Paper (PDF)</b></a> | |
| <a href="https://github.com/Hyun1A/DCAT"><b>GitHub</b></a> |
| </p> |
|
|
| **DCAT** (*Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs*, NeurIPS 2026 Spotlight) |
| transfers cross-modal alignment from a strong donor MLLM to a weaker recipient MLLM with a training-free, |
| closed-form merge that uses only a few hundred unlabeled calibration samples. This checkpoint is a LLaVA-Video (LLaMA-3.1-8B) **video** MLLM after DCAT merging with a LLaVA LLaMA-3.1-8B **vision** MLLM. |
|
|
| This repository contains the **full merged weights** (LLaVA-NeXT format). Loading needs LLaVA-NeXT with the small patch shipped in the GitHub repository (`third_party/llava_next_dcat.patch`), which adds the `aapool_mlp` projector used by this checkpoint. |
|
|
|
|
| ## Usage |
|
|
| ```bash |
| git clone https://github.com/Hyun1A/DCAT.git && cd DCAT |
| pip install -r requirements_llama.txt |
| git clone https://github.com/LLaVA-VL/LLaVA-NeXT.git third_party/LLaVA-NeXT |
| git -C third_party/LLaVA-NeXT checkout bce12e4 |
| git -C third_party/LLaVA-NeXT apply ../llava_next_dcat.patch |
| pip install -e third_party/LLaVA-NeXT --no-deps |
| |
| python quick_start/inference.py --setting llama-video --model Hyun1A/DCAT-LLaMA-Vision2Video \ |
| --media sample.mp4 --question "What is the main content of this video?" --choices "option 1" "option 2" "option 3" "option 4" |
| ``` |
|
|
| See the [GitHub repository](https://github.com/Hyun1A/DCAT#quick-start) for installation details, the |
| inference code, and the scripts that reproduce the merge. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{seo2026dcat, |
| title = {Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs}, |
| author = {Seo, Hoigi and Lee, Byung Hyun and Kim, Minjun and |
| Mah, Dohyun and Lee, Jongho and Chun, Se Young}, |
| booktitle = {Advances in Neural Information Processing Systems (NeurIPS)}, |
| year = {2026}, |
| url = {https://arxiv.org/abs/2610.04580} |
| } |
| ``` |
|
|
