| --- |
| license: llama3.1 |
| base_model: fixie-ai/ultravox-v0_4_1-llama-3_1-8b |
| pipeline_tag: audio-text-to-text |
| tags: |
| - audio |
| - model-merging |
| - dcat |
| - llama-3.1 |
| - ultravox |
| --- |
| |
| # DCAT · LLaMA-3.1-8B · Vision → Audio |
|
|
| <p align="center"> |
| <a href="https://seohoiki3215.github.io/DCAT_project_page/"><b>Project Page</b></a> | |
| <a href="https://arxiv.org/abs/2610.04580"><b>Paper (PDF)</b></a> | |
| <a href="https://github.com/Hyun1A/DCAT"><b>GitHub</b></a> |
| </p> |
|
|
| **DCAT** (*Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs*, NeurIPS 2026 Spotlight) |
| transfers cross-modal alignment from a strong donor MLLM to a weaker recipient MLLM with a training-free, |
| closed-form merge that uses only a few hundred unlabeled calibration samples. This checkpoint is the Ultravox v0.4.1 (LLaMA-3.1-8B-Instruct) **audio** MLLM after DCAT merging with a LLaVA LLaMA-3.1-8B **vision** MLLM. |
|
|
| This repository contains the **full merged weights** together with Ultravox's remote code. Ultravox rebuilds its language model from `text_model_id` (`meta-llama/Meta-Llama-3.1-8B-Instruct`, gated) while loading, so the merged weights are copied in from the shards afterwards; the script below does this for you. |
|
|
|
|
| ## Usage |
|
|
| ```bash |
| git clone https://github.com/Hyun1A/DCAT.git && cd DCAT |
| pip install -r requirements_llama.txt |
| huggingface-cli login # access to meta-llama/Meta-Llama-3.1-8B-Instruct, or pass --text-model /path/to/Llama-3.1-8B-Instruct |
| |
| python quick_start/inference.py --setting llama-audio --model Hyun1A/DCAT-LLaMA-Vision2Audio \ |
| --media sample.wav --question "Based on the given audio, identify the source of the speaking voice." --choices Man Woman Child Robot |
| ``` |
|
|
| See the [GitHub repository](https://github.com/Hyun1A/DCAT#quick-start) for installation details, the |
| inference code, and the scripts that reproduce the merge. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{seo2026dcat, |
| title = {Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs}, |
| author = {Seo, Hoigi and Lee, Byung Hyun and Kim, Minjun and |
| Mah, Dohyun and Lee, Jongho and Chun, Se Young}, |
| booktitle = {Advances in Neural Information Processing Systems (NeurIPS)}, |
| year = {2026}, |
| url = {https://arxiv.org/abs/2610.04580} |
| } |
| ``` |
|
|
