---
license: llama3.1
base_model: fixie-ai/ultravox-v0_4_1-llama-3_1-8b
pipeline_tag: audio-text-to-text
tags:
  - audio
  - model-merging
  - dcat
  - llama-3.1
  - ultravox
---

# DCAT · LLaMA-3.1-8B · Vision → Audio

<p align="center">
  <a href="https://seohoiki3215.github.io/DCAT_project_page/"><b>Project Page</b></a> &nbsp;|&nbsp;
  <a href="https://arxiv.org/abs/2610.04580"><b>Paper (PDF)</b></a> &nbsp;|&nbsp;
  <a href="https://github.com/Hyun1A/DCAT"><b>GitHub</b></a>
</p>

**DCAT** (*Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs*, NeurIPS 2026 Spotlight)
transfers cross-modal alignment from a strong donor MLLM to a weaker recipient MLLM with a training-free,
closed-form merge that uses only a few hundred unlabeled calibration samples. This checkpoint is the Ultravox v0.4.1 (LLaMA-3.1-8B-Instruct) **audio** MLLM after DCAT merging with a LLaVA LLaMA-3.1-8B **vision** MLLM.

This repository contains the **full merged weights** together with Ultravox's remote code. Ultravox rebuilds its language model from `text_model_id` (`meta-llama/Meta-Llama-3.1-8B-Instruct`, gated) while loading, so the merged weights are copied in from the shards afterwards; the script below does this for you.


## Usage

```bash
git clone https://github.com/Hyun1A/DCAT.git && cd DCAT
pip install -r requirements_llama.txt
huggingface-cli login   # access to meta-llama/Meta-Llama-3.1-8B-Instruct, or pass --text-model /path/to/Llama-3.1-8B-Instruct

python quick_start/inference.py --setting llama-audio --model Hyun1A/DCAT-LLaMA-Vision2Audio \
    --media sample.wav --question "Based on the given audio, identify the source of the speaking voice." --choices Man Woman Child Robot
```

See the [GitHub repository](https://github.com/Hyun1A/DCAT#quick-start) for installation details, the
inference code, and the scripts that reproduce the merge.

## Citation

```bibtex
@inproceedings{seo2026dcat,
  title     = {Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs},
  author    = {Seo, Hoigi and Lee, Byung Hyun and Kim, Minjun and
               Mah, Dohyun and Lee, Jongho and Chun, Se Young},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026},
  url       = {https://arxiv.org/abs/2610.04580}
}
```