JAVIS

JAVIS is an end-to-end, streaming spoken-dialogue agent that answers with voice and face at the same time. Built on Fun-Audio-Chat-8B, it generates the reply text, CosyVoice3 speech tokens and 20-d FLOAT motion coefficients in one autoregressive pass; CosyVoice3 and the FLOAT decoder render the waveform and a 512x512 talking face packet by packet.

Paper · Code · Project page

Files

• model-*.safetensors, config.json, tokenizer and processor files: the JAVIS model (the Fun-Audio-Chat LM with the trained Joint Refined Head LoRA merged in and the Motion Decoder)
• direction_Q.pt: the FLOAT motion basis (20 x 512) used to render the predicted motion

Usage

git clone --recursive https://github.com/kaistmm/JAVIS.git && cd JAVIS
huggingface-cli download junseok520/JAVIS --local-dir pretrained_models/JAVIS
python inference/infer.py --audio examples/user.wav --portrait examples/portrait.png --voice_ref examples/voice_ref.wav

See the code repository for installation, real-time streaming and training.

License

The weights are released under CC BY-NC 4.0 (non-commercial use, with attribution; full text in LICENSE). The model builds on Fun-Audio-Chat-8B (Apache-2.0), but it predicts motion in the FLOAT latent space, renders it with the FLOAT decoder, and was trained on data that includes non-commercial sources (e.g. Seamless Interaction, CC BY-NC 4.0). FLOAT itself is licensed under CC BY-NC-ND 4.0 (non-commercial, research use only; see its README). The inference code on GitHub is Apache-2.0.

Citation

@article{ahn2026javis,
  title   = {JAVIS: Joint Audio-Visual Interaction and Streaming for Real-Time Conversational Agents},
  author  = {Ahn, Junseok and Park, Jaehyun and Choi, Jeongsoo and Chung, Joon Son and Kim, Ji-Hoon},
  journal = {arXiv preprint arXiv:2610.22499},
  year    = {2026}
}