---
license: cc-by-nc-4.0
library_name: transformers
pipeline_tag: any-to-any
tags:
- audio-visual
- spoken-dialogue
- talking-head
- streaming
---

# JAVIS

JAVIS is an end-to-end, streaming spoken-dialogue agent that answers with voice and face at the same time. Built on
Fun-Audio-Chat-8B, it generates the reply text, CosyVoice3 speech tokens and 20-d FLOAT motion coefficients in one
autoregressive pass; CosyVoice3 and the FLOAT decoder render the waveform and a 512x512 talking face packet by packet.

[Paper](https://arxiv.org/abs/2610.22499) · [Code](https://github.com/kaistmm/JAVIS) · [Project page](https://mm.kaist.ac.kr/projects/JAVIS)

## Files

- `model-*.safetensors`, `config.json`, tokenizer and processor files: the JAVIS model (the Fun-Audio-Chat LM with the
  trained Joint Refined Head LoRA merged in and the Motion Decoder)
- `direction_Q.pt`: the FLOAT motion basis (20 x 512) used to render the predicted motion

## Usage

```bash
git clone --recursive https://github.com/kaistmm/JAVIS.git && cd JAVIS
huggingface-cli download junseok520/JAVIS --local-dir pretrained_models/JAVIS
python inference/infer.py --audio examples/user.wav --portrait examples/portrait.png --voice_ref examples/voice_ref.wav
```
See the code repository for installation, real-time streaming and training.

## License

The weights are released under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) (non-commercial use, with
attribution; full text in `LICENSE`). The model builds on Fun-Audio-Chat-8B (Apache-2.0), but it predicts motion in the
[FLOAT](https://github.com/deepbrainai-research/float) latent space, renders it with the FLOAT decoder, and was trained
on data that includes non-commercial sources (e.g. Seamless Interaction, CC BY-NC 4.0). FLOAT itself is licensed under
CC BY-NC-ND 4.0 (non-commercial, research use only; see its README). The inference code on GitHub is Apache-2.0.

## Citation

```bibtex
@article{ahn2026javis,
  title   = {JAVIS: Joint Audio-Visual Interaction and Streaming for Real-Time Conversational Agents},
  author  = {Ahn, Junseok and Park, Jaehyun and Choi, Jeongsoo and Chung, Joon Son and Kim, Ji-Hoon},
  journal = {arXiv preprint arXiv:2610.22499},
  year    = {2026}
}
```