README.md

---
license: cc-by-nc-nd-4.0
language:
- en
- ar
pipeline_tag: image-to-video
tags:
- audio-to-video
- talking-head
- portrait-animation
- lip-sync
- flow-matching
- motion-generation
- video-generation
- wav2vec2
- pytorch
- real-time
- onnx
- tensorrt
---

# [ICCV 2025] FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portrait

Official PyTorch implementation of **FLOAT** — *Flow Matching for Audio-driven Talking Portrait Video Generation* — plus the
**pre-converted ONNX / TensorRT graphs** used by the [FLOAT Fast](https://github.com/mohamed-em2m/float-fast) serving stack.

![FLOAT overview](./assets/float-abstract.png)

**FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking Portrait**<br>
[Taekyung Ki](https://taekyungki.github.io), [Dongchan Min](https://kevinmin95.github.io), [Gyeongsu Chae](https://www.aistudios.com/ko)

Project page: [deepbrainai-research.github.io/float](https://deepbrainai-research.github.io/float/) · Serving stack: [github.com/mohamed-em2m/float-fast](https://github.com/mohamed-em2m/float-fast)

**Abstract**: *With the rapid advancement of diffusion-based generative models, portrait image animation has achieved remarkable results. However, it still faces challenges in temporally consistent video generation and fast sampling due to its iterative sampling nature. This paper presents FLOAT, an audio-driven talking portrait video generation method based on flow matching generative model. Instead of a pixel-based latent space, we take advantage of a learned orthogonal motion latent space, enabling efficient generation and editing of temporally consistent motion. To achieve this, we introduce a transformer-based vector field predictor with an effective frame-wise conditioning mechanism. Additionally, our method supports speech-driven emotion enhancement, enabling a natural incorporation of expressive motions. Extensive experiments demonstrate that our method outperforms state-of-the-art audio-driven talking portrait methods in terms of visual quality, motion fidelity, and efficiency.*

**TL;DR**: FLOAT is a flow-matching based audio-driven talking portrait video generation method, which can enhance the
speech-driven emotional motion.

---

## Repository layout

```
mohamedemam/float/
├── float.pth                                   # the main model checkpoint (827 MB)
├── wav2vec2-base-960h/                         # audio encoder  (facebook/wav2vec2-base-960h)
├── wav2vec-english-speech-emotion-recognition/ # emotion encoder (r-f/wav2vec-…-recognition)
├── assets/                                     # figures used by this card
└── onnx/                                       # pre-converted ONNX graphs (see below)
    ├── fmt_step_v1.onnx            628 MB   # FMT vector-field step  (the ODE inner loop)
    ├── emotion_v1.onnx            1.27 GB   # speech-emotion classifier
    ├── wav2vec_v1.onnx             380 MB   # audio encoder + projection head
    ├── decoder_v1.onnx              94 MB   # frame decoder (StyleGAN-style synthesis)
    ├── refenc_v1.onnx               91 MB   # reference-image encoder
    ├── direction_v1.onnx            42 KB   # motion direction basis
    ├── decoder_v1.onnx.no_trt              # sidecar: the runtime must not use TensorRT here
    ├── backends.generated.yaml              # measured per-component backend plan
    └── .build_manifest.json                 # build provenance for the artifacts
```

## ONNX / TensorRT accelerated inference

The `onnx/` folder holds the **same six components**, exported to ONNX and ready to run on
TensorRT / CUDA / CPU. They are produced by
[`float-fast`](https://github.com/mohamed-em2m/float-fast)'s `scripts/convert_to_onnx.sh` and are
**drop-in replacements for the torch modules** — the ODE loop, the CFG combine and the chunk
autoregression stay in Python, only the hot inner steps move into the graph.

Every artifact is **gated before it is published**: the converter compares the exported graph against
the *deployed* torch model and refuses to pass a graph that does not reproduce it. Two rules make
that gate meaningful, and both are easy to get wrong:

- the reference is the **deployed** model, never the module that was traced;
- both sides run **the same computation** (fp32 on both sides — TF32 alone moves a 22-conv stack
  by tens of uint8 levels, which would drown the graph's own error).

Measured for the graphs in this folder:

| artifact | provider | fidelity vs the torch model |
|---|---|---|
| `fmt_step_v1.onnx` | TensorRT FP16 | cosine 0.999996 (per ODE step) |
| `decoder_v1.onnx` | CUDA (fp32) | max 1 / mean 0.002 uint8 level on a 512×512 frame |
| `emotion_v1.onnx`, `wav2vec_v1.onnx`, `refenc_v1.onnx`, `direction_v1.onnx` | TensorRT FP16 | cosine ≥ 0.99999 |

`backends.generated.yaml` is the **measured** plan for the machine the conversion ran on (an
RTX A6000): `emotion`, `audio` and `fmt` on `onnx/trt`, `refenc` compiled, `direction` and
`decoder` eager. Re-run the selector on your own box to re-measure — the choices are per-GPU:

```bash
git clone https://github.com/mohamed-em2m/float-fast && cd float-fast
uv run --frozen --no-sync python -m fast_float.onnx.select_backends \
    --onnx-dir ./onnx --out ./onnx/backends.generated.yaml --ckpt ./checkpoints/float.pth
```

Two things are deliberately **not** in this folder:

- **the TensorRT engine cache** — engines are tied to the GPU and the TensorRT/CUDA versions, so
  the target box builds its own on first use (and caches them under `onnx/trt_cache/`);
- **a runtime config you can trust blindly** — see the selector above.

### Using the graphs with FLOAT Fast

```bash
# 1. fetch the checkpoints and the ONNX graphs
huggingface-cli download mohamedemam/float --include "onnx/*" --local-dir ./hf
huggingface-cli download mohamedemam/float --include "float.pth" --local-dir ./checkpoints
huggingface-cli download mohamedemam/float --include "wav2vec2-base-960h/*" --local-dir ./checkpoints
huggingface-cli download mohamedemam/float --include "wav2vec-english-speech-emotion-recognition/*" --local-dir ./checkpoints
cp -r hf/onnx ./onnx

# 2. build the environment (uv venv, deps, ONNX stack, TensorRT 10.9)
./scripts/setup.sh

# 3. re-measure the backends for this box, then serve
uv run --frozen --no-sync python -m fast_float.onnx.select_backends \
    --onnx-dir ./onnx --out ./onnx/backends.generated.yaml --ckpt ./checkpoints/float.pth
./scripts/run_all.sh        # backend :8000 + Studio frontend :3000
```

FLOAT Fast then serves the model over HTTP:

```bash
curl -X POST http://localhost:8000/generate \
  -H 'Content-Type: application/json' \
  -d '{"audio_base64": "'"$(base64 -w0 audio.wav)"'",
       "audio_format": "wav",
       "image_base64": "'"$(base64 -w0 portrait.jpg)"'"}'
```

It also streams the frames live over a WebSocket (`/ws/stream/{request_id}`) and over WebRTC
(`/offer`), so a browser can play the portrait while the MP4 is still being written.

---

## Generation Results

| Result 1 | Result 2 |
|-------------|-------------|
| <video src="https://github.com/user-attachments/assets/8c00274d-795d-4ee9-870f-84a859f3e23f"> </video> | <video src="https://github.com/user-attachments/assets/c6e142b3-519b-4cda-b26d-e088414b478d"> </video> |

| Result 3 | Result 4 |
|---|---|
| <video src="https://github.com/user-attachments/assets/7b201a5f-a293-46cd-974f-0612062d8d94"> </video> |  <video src="https://github.com/user-attachments/assets/dd4b74dd-40b4-4864-b87d-7cbf4f0d66da"> </video> |

<br>
Our method runs faster than current diffusion-based methods with fewer sampling steps and lower memory cost. For more details, please refer to the paper.
<div align='center'>
    <image width= 80% src="./assets/fps.png" alt="FLOAT inference speed"> </image>
</div>

## Updates

- [2026.10.21] 🧩 The `onnx/` folder now hosts the **pre-converted ONNX / TensorRT graphs** for the
  FLOAT Fast serving stack (FMT step, emotion, audio, frame decoder, reference encoder, motion
  direction), each gated against the torch model before publication.
- [2025.06.26] 🎉🎉 FLOAT is accepted to **[ICCV 2025](https://iccv.thecvf.com)** 🎉🎉
- [2025.02.17] The inference code and checkpoints are released under a **[Non-commercial License](https://creativecommons.org/licenses/by-nc-nd/4.0/)**
- [2024.12.03] Selected as a [HuggingFace Daily Papers](https://huggingface.co/papers?date=2024-12-03) on December 3, 2024
- [2024.12.02] The paper is publicly available on [ArXiv](https://arxiv.org/abs/2412.01064)

---

## Getting started

### Requirements

```bash
# The original FLOAT environment
conda create -n FLOAT python=3.8.5
conda activate FLOAT
sh environments.sh

# or manual installation
pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118
pip install -r requirements.txt
```

- Tested on Linux with A100 and V100 GPUs.
- The FLOAT Fast stack needs a newer stack (Python 3.10, PyTorch 2.7.1+cu126, ONNX Runtime 1.23,
  optionally TensorRT 10.9).

### Preparing the checkpoints

1. Download the checkpoints — either from this repository:

    ```bash
    huggingface-cli download mohamedemam/float --local-dir ./checkpoints
    ```

    or from the original [Google Drive](https://drive.google.com/file/d/1rvWuM12cyvNvBQNCLmG4Fr2L1rpjQBF0/view?usp=sharing)
    (the FLOAT Fast scripts do this automatically).

2. The checkpoints should be organised as follows:

    ```bash
    ./checkpoints
    |-- checkpoints_here
    |-- float.pth                            # main model
    |-- wav2vec2-base-960h/                  # audio encoder
    |   |-- config.json
    |   |-- feature_extractor_config.json
    |   |-- model.safetensors
    |   |-- preprocessor_config.json
    |   |-- pytorch_model.bin
    |   |-- special_tokens_map.json
    |   |-- tokenizer_config.json
    |   '-- vocab.json
    '-- wav2vec-english-speech-emotion-recognition/   # emotion encoder
        |-- config.json
        |-- preprocessor_config.json
        |-- pytorch_model.bin
        '-- training_args.bin
    ```

    - The W2V based models come from
      [wav2vec2-base-960h](https://huggingface.co/facebook/wav2vec2-base-960h) and
      [wav2vec-english-speech-emotion-recognition](https://huggingface.co/r-f/wav2vec-english-speech-emotion-recognition).

### Generating a talking portrait from a single image and audio

> [!IMPORTANT]
> For the best quality, please mind the pre-processing:
>
> - FLOAT is trained on frontal head pose distributions — a non-frontal image gives suboptimal results.
> - The performance of talking-portrait methods depends on their training-time pre-processing (e.g. field
>   of view). The inference code includes automatic face cropping, which may introduce black **padding**
>   regions; it can be disabled in `generate.py` at the cost of quality.
> - Heavy background music hurts the result — extract the vocals first with
>   [ClearVoice](https://github.com/modelscope/ClearerVoice-Studio).

**1. Generating video 1 (emotion from audio)**

You can generate a video with an emotion derived from the audio without specifying `--emo`. Adjust the
intensity with `--e_cfg_scale` (default 1); for more emotion-intensive video try 5–10.

```bash
CUDA_VISIBLE_DEVICES=0 python generate.py \
    --ref_path path/to/reference/image \
    --aud_path path/to/audio \
    --seed 15 \
    --a_cfg_scale 2 \
    --e_cfg_scale 1 \
    --ckpt_path ./checkpoints/float.pth \
    --no_crop                    # [optional] skip cropping
```

**2. Generating video 2 (redirecting the emotion)**

Pass `--emo` to pick one of the seven emotions:
`['angry', 'disgust', 'fear', 'happy', 'neutral', 'sad', 'surprise']`.

```bash
CUDA_VISIBLE_DEVICES=0 python generate.py \
    --ref_path path/to/reference/image \
    --aud_path path/to/audio \
    --emo 'happy' \
    --seed 15 \
    --a_cfg_scale 2 \
    --e_cfg_scale 1 \
    --ckpt_path ./checkpoints/float.pth \
    --no_crop
```

<video src="https://github.com/user-attachments/assets/fb3826cd-231b-46f2-809b-11adebe9a1cf"> </video>

**3. Running the example**

```bash
CUDA_VISIBLE_DEVICES=0 python generate.py \
    --ref_path assets/sam_altman.webp \
    --aud_path assets/aud-sample-vs-1.wav \
    --seed 15 \
    --a_cfg_scale 2 \
    --e_cfg_scale 1 \
    --ckpt_path ./checkpoints/float.pth
```

| Before Crop | After Crop | Result |
|-------------|-----------|--------|
| ![reference portrait](assets/sam_altman.webp) | ![after face crop](assets/sam_altman_512x512.jpg) | <video src='https://github.com/user-attachments/assets/3353e4e0-00f5-429b-bc66-5db9a72186b8'> </video> |

---

## Intended use and limitations

- **Research use only.** This work is licensed under
  [CC BY-NC-ND 4.0](https://creativecommons.org/licenses/by-nc-nd/4.0/): no commercial use, and no
  derivative works.
- The model is trained on **frontal** head poses and is driven by a single reference image and a
  speech track; it does not handle large head rotations, multi-person scenes or non-speech audio.
- The emotion is either predicted from the audio or set explicitly to one of the seven basic emotions.
- The `onnx/` graphs are **community-produced** (by the FLOAT Fast project) and are provided as-is;
  they are numerically validated against the torch model, but the upstream authors do not support them.

## ❗ License ❗

This work is licensed under a
[Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License](https://creativecommons.org/licenses/by-nc-nd/4.0/).
You may not use this work for commercial purposes and may use it only for research purposes.
**For any commercial inquiries or collaboration opportunities**, please contact [daniel@deepbrain.io](mailto:daniel@deepbrain.io).

## Development

This repository is a research demonstration implementation and is provided as a one-time code drop.
For any research-related inquiries, please contact the first author [Taekyung Ki](https://github.com/TaekyungKi).
This work was done during the first author's South Korean Alternative Military Service at DeepBrain AI.
This repository includes only the inference code; the training code will not be released.

## Citation

```bibtex
@inproceedings{ki2025float,
  title={Float: Generative motion latent flow matching for audio-driven talking portrait},
  author={Ki, Taekyung and Min, Dongchan and Chae, Gyeongsu},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={14699--14710},
  year={2025}
}
```

## Related works

- [StyleLipSync: Style-based Personalized Lip-sync Video Generation](https://arxiv.org/abs/2305.00521)<br>
- [StyleTalker: One-shot Style-based Audio-driven Talking Head Video Generation](https://arxiv.org/abs/2208.10922)<br>
- [Export3D: Learning to Generate Conditional Tri-plane for 3D-aware Expression Controllable Portrait Animation](https://arxiv.org/abs/2404.00636)<br>
- [Latent Image Animator: Learning to Animate Images via Latent Space Navigation](https://arxiv.org/abs/2203.09043)<br>

## Acknowledgements

The source images and audio are collected from the internet and other baselines, such as LIA, SadTalker,
EMO, VASA-1, Hallo, LivePortrait, Loopy, and others. We appreciate their valuable contributions to this
field. We employ the Wav2Vec2.0-based speech emotion recognizer by
[Rob Field](https://huggingface.co/r-f/wav2vec-english-speech-emotion-recognition). We appreciate this
good work.