license: cc-by-nc-4.0
library_name: transformers
pipeline_tag: any-to-any
tags:
- audio-visual
- spoken-dialogue
- talking-head
- streaming
JAVIS
JAVIS is an end-to-end, streaming spoken-dialogue agent that answers with voice and face at the same time. Built on Fun-Audio-Chat-8B, it generates the reply text, CosyVoice3 speech tokens and 20-d FLOAT motion coefficients in one autoregressive pass; CosyVoice3 and the FLOAT decoder render the waveform and a 512x512 talking face packet by packet.
Paper · Code · Project page
Files
model-*.safetensors,config.json, tokenizer and processor files: the JAVIS model (the Fun-Audio-Chat LM with the trained Joint Refined Head LoRA merged in and the Motion Decoder)direction_Q.pt: the FLOAT motion basis (20 x 512) used to render the predicted motion
Usage
git clone --recursive https://github.com/kaistmm/JAVIS.git && cd JAVIS
huggingface-cli download junseok520/JAVIS --local-dir pretrained_models/JAVIS
python inference/infer.py --audio examples/user.wav --portrait examples/portrait.png --voice_ref examples/voice_ref.wav
See the code repository for installation, real-time streaming and training.
License
The weights are released under CC BY-NC 4.0 (non-commercial use, with
attribution; full text in LICENSE). The model builds on Fun-Audio-Chat-8B (Apache-2.0), but it predicts motion in the
FLOAT latent space, renders it with the FLOAT decoder, and was trained
on data that includes non-commercial sources (e.g. Seamless Interaction, CC BY-NC 4.0). FLOAT itself is licensed under
CC BY-NC-ND 4.0 (non-commercial, research use only; see its README). The inference code on GitHub is Apache-2.0.
Citation
@article{ahn2026javis,
title = {JAVIS: Joint Audio-Visual Interaction and Streaming for Real-Time Conversational Agents},
author = {Ahn, Junseok and Park, Jaehyun and Choi, Jeongsoo and Chung, Joon Son and Kim, Ji-Hoon},
journal = {arXiv preprint arXiv:2610.22499},
year = {2026}
}