Hyun1A/DCAT-Vicuna-Vision2Audio

DCAT · Vicuna-7B · Vision → Audio

Project Page  |  Paper (PDF)  |  GitHub

DCAT (Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs, NeurIPS 2026 Spotlight) transfers cross-modal alignment from a strong donor MLLM to a weaker recipient MLLM with a training-free, closed-form merge that uses only a few hundred unlabeled calibration samples. This checkpoint is the Vicuna-7B-v1.5 audio MLLM (ModelCompose audio-naivemc) after DCAT merging with the vision MLLM of the same family.

This repository contains only the adapter weights (adapter_model.bin, ModelCompose format) and config.json. They are loaded on top of lmsys/vicuna-7b-v1.5. encoders/beats/ holds the BEATs audio encoder checkpoint used by the model (MIT license, see LICENSE_BEATs.txt).

Usage

git clone https://github.com/Hyun1A/DCAT.git && cd DCAT
pip install -r requirements_vicuna.txt

python quick_start/inference.py --setting vicuna-audio --model Hyun1A/DCAT-Vicuna-Vision2Audio \
    --media sample.wav --question "Based on the given audio, identify the source of the speaking voice." --choices Man Woman Child Robot

See the GitHub repository for installation details, the inference code, and the scripts that reproduce the merge.

Citation

@inproceedings{seo2026dcat,
  title     = {Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs},
  author    = {Seo, Hoigi and Lee, Byung Hyun and Kim, Minjun and
               Mah, Dohyun and Lee, Jongho and Chun, Se Young},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026},
  url       = {https://arxiv.org/abs/2610.04580}
}