| --- |
| license: cc-by-nc-4.0 |
| library_name: transformers |
| pipeline_tag: any-to-any |
| tags: |
| - audio-visual |
| - spoken-dialogue |
| - talking-head |
| - streaming |
| --- |
| |
| # JAVIS |
|
|
| JAVIS is an end-to-end, streaming spoken-dialogue agent that answers with voice and face at the same time. Built on |
| Fun-Audio-Chat-8B, it generates the reply text, CosyVoice3 speech tokens and 20-d FLOAT motion coefficients in one |
| autoregressive pass; CosyVoice3 and the FLOAT decoder render the waveform and a 512x512 talking face packet by packet. |
|
|
| [Paper](https://arxiv.org/abs/2610.22499) · [Code](https://github.com/kaistmm/JAVIS) · [Project page](https://mm.kaist.ac.kr/projects/JAVIS) |
|
|
| ## Files |
|
|
| - `model-*.safetensors`, `config.json`, tokenizer and processor files: the JAVIS model (the Fun-Audio-Chat LM with the |
| trained Joint Refined Head LoRA merged in and the Motion Decoder) |
| - `direction_Q.pt`: the FLOAT motion basis (20 x 512) used to render the predicted motion |
|
|
| ## Usage |
|
|
| ```bash |
| git clone --recursive https://github.com/kaistmm/JAVIS.git && cd JAVIS |
| huggingface-cli download junseok520/JAVIS --local-dir pretrained_models/JAVIS |
| python inference/infer.py --audio examples/user.wav --portrait examples/portrait.png --voice_ref examples/voice_ref.wav |
| ``` |
| See the code repository for installation, real-time streaming and training. |
|
|
| ## License |
|
|
| The weights are released under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) (non-commercial use, with |
| attribution; full text in `LICENSE`). The model builds on Fun-Audio-Chat-8B (Apache-2.0), but it predicts motion in the |
| [FLOAT](https://github.com/deepbrainai-research/float) latent space, renders it with the FLOAT decoder, and was trained |
| on data that includes non-commercial sources (e.g. Seamless Interaction, CC BY-NC 4.0). FLOAT itself is licensed under |
| CC BY-NC-ND 4.0 (non-commercial, research use only; see its README). The inference code on GitHub is Apache-2.0. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{ahn2026javis, |
| title = {JAVIS: Joint Audio-Visual Interaction and Streaming for Real-Time Conversational Agents}, |
| author = {Ahn, Junseok and Park, Jaehyun and Choi, Jeongsoo and Chung, Joon Son and Kim, Ji-Hoon}, |
| journal = {arXiv preprint arXiv:2610.22499}, |
| year = {2026} |
| } |
| ``` |
|
|
