File size: 11,539 Bytes

---
license: apache-2.0
language:
  - en
  - ja
  - zh
  - multilingual
datasets:
  - amphion/Emilia-Dataset
  - nvidia/hifitts-2
  - reazon-research/reazonspeech
  - MLCommons/peoples_speech
  - facebook/voxpopuli
  - facebook/multilingual_librispeech
  - facebook/omnilingual-asr-corpus
  - ai4bharat/IndicVoices
pipeline_tag: audio-to-audio
tags:
  - audio
  - speech
  - neural-audio-codec
  - speech-tokenizer
  - variable-frame-rate
  - streaming
  - text-to-speech
---

# RubatoCodec

[![Code: GitHub](https://img.shields.io/badge/Code-GitHub-black?logo=github)](https://github.com/third-intelligence/RubatoCodec)
[![Paper: arXiv](https://img.shields.io/badge/arXiv-2610.29434-b31b1b)](https://arxiv.org/abs/2610.29434)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue)](https://www.apache.org/licenses/LICENSE-2.0)

RubatoCodec is a neural speech codec that divides speech into a fixed number of
variable-duration events. Each event carries discrete code indices and an
explicit duration. Event placement adapts to speech content, while the number of
events is determined exactly by the utterance length, so the token sequence
length a speech generator must produce is predictable. A causal decoder renders
completed events incrementally, which makes the codec suitable for streaming
text-to-speech.

The name comes from the musical term *rubato*: local timing flexes while the
overall tempo is preserved.

- **Adaptive Firing with an exact event budget.** Multi-scale latent
  dissimilarity and waveform spectral flux guide cumulative allocation on a
  50-Hz latent grid. For `T` frames the event count is exactly
  `max(1, min(T, round(0.25 · T)))`, giving 12.5 events/s. No learned boundary
  predictor, boundary labels, or rate loss is used.
- **Streaming decoding.** Events can be decoded one at a time or in chunks.
  Event-by-event decoding takes about 11 ms per event on an NVIDIA H200, well
  below the ~80 ms of audio each event represents on average.
- **Hybrid residual quantizer with depth control.** A semantically supervised
  FSQ first layer is followed by 15 DAC-style residual VQ layers, each with
  4,096 entries. Any prefix depth from 1 to 16 can be decoded, giving code-index
  bitrates of 0.15–2.4 kbit/s plus duration information.
- **Multilingual.** Trained on approximately 375,000 hours of speech, mostly
  English, Japanese, and Chinese, with 16-kHz input and 44.1-kHz output.

![RubatoCodec architecture](assets/architecture.png)

## Model details

| | |
|---|---|
| Input / output sample rate | 16 kHz / 44.1 kHz |
| Latent grid | 50 Hz |
| Event rate | 12.5 events/s (retention ratio 0.25) |
| Quantizer | 16 layers × 4,096 entries; layer 0 FSQ with semantic distillation, layers 1–15 residual VQ |
| Bitrate | 1.23 kbit/s at depth 8, 2.43 kbit/s at depth 16 (LibriSpeech, including duration information) |
| Decoder | Fully causal Transformer + ISTFT head, streaming capable |
| Parameters | 246M codec (72M encoder, 173M decoder, 1M quantizer) + frozen [w2v-BERT 2.0](https://huggingface.co/facebook/w2v-bert-2.0) (580M, encoding only) |
| License | Apache 2.0 |

## Usage

Install the package from the [GitHub repository](https://github.com/third-intelligence/RubatoCodec):

```bash
git clone https://github.com/third-intelligence/RubatoCodec
cd RubatoCodec
uv sync
```

Encode and decode:

```python
import soundfile as sf
import torch

from rubatocodec import RubatoCodec

codec = RubatoCodec.from_pretrained("third-intelligence/RubatoCodec", device="cuda")

data, sr = sf.read("speech.wav", dtype="float32")  # mono waveform
wav = torch.from_numpy(data)

enc = codec.encode(wav, sr=sr)       # {"codes": (16, K) int64, "durs": (K,) int64}
audio = codec.decode(enc)            # (1, 1, T_audio) at 44.1 kHz, full depth
audio = codec.decode(enc, depth=8)   # lower bitrate with the first 8 layers

sf.write("recon.wav", audio[0, 0].cpu().numpy(), codec.output_sr)
```

`durs` gives each event's duration in 20-ms grid frames. Loading also downloads
`facebook/w2v-bert-2.0`, which is used only for encoding.

Streaming decoding, inference settings, the code format, and training are
documented in the repository:
[inference guide](https://github.com/third-intelligence/RubatoCodec/blob/main/docs/inference.md),
[architecture](https://github.com/third-intelligence/RubatoCodec/blob/main/docs/architecture.md),
[training](https://github.com/third-intelligence/RubatoCodec/blob/main/docs/training.md).

## Evaluation

RubatoCodec is evaluated on LibriSpeech test-clean, VCTK, JVS nonpara30, JVNV,
and seed-tts-eval test-zh against 13 other codecs. At comparable bitrates, it
shows the best intelligibility preservation among the compared codecs on the
English and Japanese benchmarks and high predicted quality, while the strongest
comparators achieve higher PESQ-WB and STOI.

dWER/dCER compare ASR transcripts of the reconstructed and original audio
(Whisper-large-v3 for English and Japanese, Paraformer-zh for Mandarin). SIM is
ECAPA-TDNN speaker similarity. Bitrates include duration information.

The tables below show a subset of the compared codecs. **Bold** and
<ins>underline</ins> mark the best and second-best scores within each bitrate
band.

**LibriSpeech test-clean (English)**

| Codec | Event rate (Hz) | Bitrate (kbit/s) | UTMOSv2 ↑ | DNSMOS ↑ | PESQ-WB ↑ | STOI ↑ | dWER (%) ↓ | SIM ↑ |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| *~1.2 kbit/s* | | | | | | | | |
| FlexiCodec, τ=1, depth 8 | 12.5 | 1.24 | 3.194 | 3.284 | 2.813 | 0.940 | <ins>0.83</ins> | 0.805 |
| MiMo-Audio-Tokenizer, depth 6 | 25.0 | 1.20 | 3.183 | 3.291 | 2.677 | 0.927 | 0.86 | <ins>0.879</ins> |
| MOSS-Audio-Tokenizer, depth 10 | 12.5 | 1.25 | 3.230 | 3.217 | **3.052** | **0.949** | 0.89 | **0.885** |
| DTM-Codec, 80-Hz target | 81.5 | 1.28 | <ins>3.421</ins> | **3.334** | <ins>2.971</ins> | <ins>0.947</ins> | 1.41 | 0.845 |
| **RubatoCodec, depth 8** | 12.5 | 1.23 | **3.478** | <ins>3.327</ins> | 2.684 | 0.931 | **0.75** | 0.862 |
| *~2.4 kbit/s* | | | | | | | | |
| Qwen3-TTS-Tokenizer-12Hz | 12.5 | 2.20 | 3.230 | 3.273 | <ins>3.177</ins> | <ins>0.958</ins> | <ins>0.53</ins> | 0.931 |
| MiMo-Audio-Tokenizer, depth 13 | 25.0 | 2.42 | 3.210 | <ins>3.288</ins> | 3.094 | 0.947 | 0.67 | <ins>0.933</ins> |
| MOSS-Audio-Tokenizer, depth 20 | 12.5 | 2.50 | <ins>3.239</ins> | 3.203 | **3.536** | **0.968** | 0.62 | **0.943** |
| **RubatoCodec, depth 16** | 12.5 | 2.43 | **3.486** | **3.314** | 3.029 | 0.950 | **0.50** | 0.920 |

**JVNV (Japanese emotional speech with nonverbal vocalizations)**

On expressive speech, RubatoCodec keeps dCER below 2% at both bitrates, while
every other compared codec exceeds 3%.

| Codec | Event rate (Hz) | Bitrate (kbit/s) | UTMOSv2 ↑ | DNSMOS ↑ | PESQ-WB ↑ | STOI ↑ | dCER (%) ↓ | SIM ↑ |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| *~1.2 kbit/s* | | | | | | | | |
| FlexiCodec, τ=1, depth 8 | 12.5 | 1.24 | 1.811 | 3.128 | 2.062 | 0.862 | 5.52 | 0.771 |
| MiMo-Audio-Tokenizer, depth 6 | 25.0 | 1.20 | **2.681** | **3.173** | 2.295 | 0.872 | 7.88 | 0.860 |
| MOSS-Audio-Tokenizer, depth 10 | 12.5 | 1.25 | 2.396 | 3.089 | **3.032** | **0.917** | <ins>5.44</ins> | **0.909** |
| DTM-Codec, 80-Hz target | 46.1 | 0.77 | 2.523 | 3.154 | 2.081 | 0.862 | 14.82 | 0.783 |
| **RubatoCodec, depth 8** | 12.5 | 1.23 | <ins>2.628</ins> | <ins>3.164</ins> | <ins>2.650</ins> | <ins>0.907</ins> | **1.92** | <ins>0.903</ins> |
| *~2.4 kbit/s* | | | | | | | | |
| Qwen3-TTS-Tokenizer-12Hz | 12.5 | 2.20 | 2.638 | 3.097 | <ins>3.041</ins> | <ins>0.931</ins> | <ins>3.01</ins> | <ins>0.945</ins> |
| MiMo-Audio-Tokenizer, depth 13 | 25.0 | 2.42 | **2.708** | **3.176** | 2.949 | 0.911 | 4.21 | 0.927 |
| MOSS-Audio-Tokenizer, depth 20 | 12.5 | 2.50 | 2.383 | 3.057 | **3.496** | **0.941** | 3.96 | **0.953** |
| **RubatoCodec, depth 16** | 12.5 | 2.43 | <ins>2.651</ins> | <ins>3.138</ins> | 3.032 | 0.929 | **1.54** | 0.941 |

**Quality versus bitrate** (RubatoCodec is marked with a red star)

![English reconstruction versus bitrate](assets/english_reconstruction.png)

![Japanese and Mandarin reconstruction versus bitrate](assets/multilingual_reconstruction.png)

**Adaptive event placement.** Each rectangle in the lower strip is one event.
Longer events cover pauses and low-energy regions, while speech receives
shorter events, with the same 12.5 events/s budget in every utterance.

![Adaptive event placement on English, Japanese, and Mandarin speech](assets/codec_firing_examples.png)

The full protocol, all compared codecs and operating points, and throughput
measurements are in the
[evaluation document](https://github.com/third-intelligence/RubatoCodec/blob/main/docs/evaluation.md)
and the [paper](https://arxiv.org/abs/2610.29434).

## Training data

Approximately 375,000 hours of public and internal speech. Public sources include
[Emilia-YODAS](https://huggingface.co/datasets/amphion/Emilia-Dataset),
[HiFiTTS-2](https://huggingface.co/datasets/nvidia/hifitts-2) (44-kHz subset),
[ReazonSpeech](https://huggingface.co/datasets/reazon-research/reazonspeech),
[The People's Speech](https://huggingface.co/datasets/MLCommons/peoples_speech)
(excluding ShareAlike-licensed data),
[VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli),
[Multilingual LibriSpeech](https://huggingface.co/datasets/facebook/multilingual_librispeech)
(excluding English),
[Omnilingual ASR Corpus](https://huggingface.co/datasets/facebook/omnilingual-asr-corpus),
and [IndicVoices](https://huggingface.co/datasets/ai4bharat/IndicVoices). English, Japanese,
and Chinese account for approximately 44%, 27%, and 13% of the data, and other
languages for the remaining 16%.

## Limitations

- RubatoCodec is trained on speech only and is intended for speech. It is not
  designed for music or general audio.
- Encoding requires the whole utterance, because the event budget is defined at
  the utterance level. Only decoding is streaming.
- Signal-fidelity metrics (PESQ-WB, STOI) are below those of the strongest
  compared codecs at similar bitrates.
- Evaluation covers English, Japanese, and Mandarin, all of which appear in the
  training data. Performance on other languages has not been evaluated.
- Like other speech codecs, RubatoCodec can be used in speech generation systems
  capable of voice cloning. Do not use it to impersonate people or to create
  deceptive content without consent.

## Citation

```bibtex
@misc{arata2026rubatocodecbudgetedadaptivespeech,
      title={RubatoCodec: Budgeted Adaptive Speech Tokenization with Streaming Decoding},
      author={Chihiro Arata and Shohei Taniguchi and Gouki Minegishi and Kan Hatakeyama-Sato},
      year={2026},
      eprint={2610.29434},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2610.29434},
}
```

## Acknowledgements

RubatoCodec builds on [X-Codec-2.0](https://github.com/zhenye234/X-Codec-2.0)
and uses [w2v-BERT 2.0](https://huggingface.co/facebook/w2v-bert-2.0) as its
frozen semantic encoder. It also adapts components from
[BigCodec](https://github.com/Aria-K-Alethia/BigCodec),
[DAC](https://github.com/descriptinc/descript-audio-codec),
[Inworld TTS](https://github.com/inworld-ai/tts), and
[BigVGAN](https://github.com/NVIDIA/BigVGAN). See the repository for the full
list and their licenses.

## License

The model weights and code are released under the
[Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0). Third-party
components retain their respective licenses.