Hyun1A/DCAT-LLaMA-Vision2Audio

DCAT · LLaMA-3.1-8B · Vision → Audio

Project Page  |  Paper (PDF)  |  GitHub

DCAT (Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs, NeurIPS 2026 Spotlight) transfers cross-modal alignment from a strong donor MLLM to a weaker recipient MLLM with a training-free, closed-form merge that uses only a few hundred unlabeled calibration samples. This checkpoint is the Ultravox v0.4.1 (LLaMA-3.1-8B-Instruct) audio MLLM after DCAT merging with a LLaVA LLaMA-3.1-8B vision MLLM.

This repository contains the full merged weights together with Ultravox's remote code. Ultravox rebuilds its language model from text_model_id (meta-llama/Meta-Llama-3.1-8B-Instruct, gated) while loading, so the merged weights are copied in from the shards afterwards; the script below does this for you.

Usage

git clone https://github.com/Hyun1A/DCAT.git && cd DCAT
pip install -r requirements_llama.txt
huggingface-cli login   # access to meta-llama/Meta-Llama-3.1-8B-Instruct, or pass --text-model /path/to/Llama-3.1-8B-Instruct

python quick_start/inference.py --setting llama-audio --model Hyun1A/DCAT-LLaMA-Vision2Audio \
    --media sample.wav --question "Based on the given audio, identify the source of the speaking voice." --choices Man Woman Child Robot

See the GitHub repository for installation details, the inference code, and the scripts that reproduce the merge.

Citation

@inproceedings{seo2026dcat,
  title     = {Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs},
  author    = {Seo, Hoigi and Lee, Byung Hyun and Kim, Minjun and
               Mah, Dohyun and Lee, Jongho and Chun, Se Young},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026},
  url       = {https://arxiv.org/abs/2610.04580}
}