File size: 2,238 Bytes
7a2cd79 9534ffd 7a2cd79 9534ffd 85cb614 9534ffd 7a2cd79 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 | ---
license: cc-by-nc-4.0
library_name: transformers
pipeline_tag: any-to-any
tags:
- audio-visual
- spoken-dialogue
- talking-head
- streaming
---
# JAVIS
JAVIS is an end-to-end, streaming spoken-dialogue agent that answers with voice and face at the same time. Built on
Fun-Audio-Chat-8B, it generates the reply text, CosyVoice3 speech tokens and 20-d FLOAT motion coefficients in one
autoregressive pass; CosyVoice3 and the FLOAT decoder render the waveform and a 512x512 talking face packet by packet.
[Paper](https://arxiv.org/abs/2610.22499) · [Code](https://github.com/kaistmm/JAVIS) · [Project page](https://mm.kaist.ac.kr/projects/JAVIS)
## Files
- `model-*.safetensors`, `config.json`, tokenizer and processor files: the JAVIS model (the Fun-Audio-Chat LM with the
trained Joint Refined Head LoRA merged in and the Motion Decoder)
- `direction_Q.pt`: the FLOAT motion basis (20 x 512) used to render the predicted motion
## Usage
```bash
git clone --recursive https://github.com/kaistmm/JAVIS.git && cd JAVIS
huggingface-cli download junseok520/JAVIS --local-dir pretrained_models/JAVIS
python inference/infer.py --audio examples/user.wav --portrait examples/portrait.png --voice_ref examples/voice_ref.wav
```
See the code repository for installation, real-time streaming and training.
## License
The weights are released under [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) (non-commercial use, with
attribution; full text in `LICENSE`). The model builds on Fun-Audio-Chat-8B (Apache-2.0), but it predicts motion in the
[FLOAT](https://github.com/deepbrainai-research/float) latent space, renders it with the FLOAT decoder, and was trained
on data that includes non-commercial sources (e.g. Seamless Interaction, CC BY-NC 4.0). FLOAT itself is licensed under
CC BY-NC-ND 4.0 (non-commercial, research use only; see its README). The inference code on GitHub is Apache-2.0.
## Citation
```bibtex
@article{ahn2026javis,
title = {JAVIS: Joint Audio-Visual Interaction and Streaming for Real-Time Conversational Agents},
author = {Ahn, Junseok and Park, Jaehyun and Choi, Jeongsoo and Chung, Joon Son and Kim, Ji-Hoon},
journal = {arXiv preprint arXiv:2610.22499},
year = {2026}
}
```
|