pocket-tts-fastdec-18m

A fast, causal convolutional replacement for the Mimi decoder of Kyutai's Pocket-TTS english_18m_refl_2026-10 model, distilled so that Pocket-TTS runs in real time on very small CPUs (a Raspberry Pi Zero W).

This repository contains only the new decoder (2.05 M parameters, 8.2 MB, fp32) plus a small Python package that plugs it into the stock model. The language model, flow head, tokenizer, voices and Mimi encoder are not included: they stay Kyutai's, and you load them as usual.

What was replaced, and by what

Pocket-TTS turns text into a stream of 32-dim latent frames at 12.5 Hz (flow LM), and the Mimi decoder turns those latents into 24 kHz audio.

Stock Mimi decoder (removed)	Replaced by
quantizer projection (32 → 128)	Linear(32 → 256) input layer of the new core
×16 upsampling to 200 Hz	a head predicting 16 sub-frames × 128 channels per latent frame (+ a small depthwise smoothing conv)
4-layer causal transformer at 200 Hz (d=128, attention over a 50-step context)	5 residual layers at 12.5 Hz: LayerNorm → depthwise causal conv (k=4) → LayerNorm → MLP (256→512→256)
SEANet decoder, base width 24 (channels 192/96/48/24, 249 MMAC/s for the 2m model, 435 for 7m/18m)	causal channel-pruned SEANet, same layout (strides 6/5/4, transposed convs, one residual unit per stage) with channels 128/64/32/16, hidden 32/16/8 (114 MMAC/s)

Unchanged: the flow LM and its latents, the tokenizer, the voice embeddings and the Mimi encoder (so voice cloning from audio still works). The decoder is strictly causal and carries an explicit streaming state, so chunk-by-chunk decoding gives the same audio as decoding all at once (max difference 3e-6, see tests/). The teacher's quantizer, upsampler, transformer and SEANet weights are dropped from the loaded model.

Usage

You need the stock Pocket-TTS weights (see kyutai/pocket-tts, gated, and accept its terms), PyTorch ≥ 2.5 and pocket-tts>=4.0.1.

pip install "git+https://huggingface.co/fdemelo/pocket-tts-fastdec-18m"      # installs the small `pocket_tts_fastdec` package (pulls pocket-tts)

import soundfile as sf
from pocket_tts_fastdec import load_model

tts = load_model("18m")                       # stock english_18m_refl_2026-10 + this decoder (downloaded from this repo)
state = tts.get_state_for_audio_prompt("alba")     # any Pocket-TTS catalogue voice, or a path to a wav for voice cloning
audio = tts.generate_audio(state, "Hello world, this is a distilled decoder speaking.")   # torch tensor, 24 kHz
sf.write("out.wav", audio.numpy(), tts.sample_rate)

# streaming, as in stock Pocket-TTS
for chunk in tts.generate_audio_stream(state, "Streaming works chunk by chunk."):
    ...

Command line: pocket-tts-fastdec generate --size 18m --text "Hello" --voice alba --out out.wav. A local folder with decoder.safetensors + config.json also works: load_model("18m", decoder="path/to/dir"). load_model calls TTSModel.load_model(language="english_18m_refl_2026-10") and then attach_fast_decoder(tts, decoder); use the latter to patch a model you loaded yourself.

The same decoder weights work with the English refl_2026-10 2m, 7m and 18m models (they share the latent normalisation); one repo per size is published for convenience. It was not tested with any other Pocket-TTS model.

Quality

12 Harvard sentences, voice alba, PyTorch fp32, UTMOS22: stock 4.40 → with this decoder 4.20. Averaged over six catalogue voices (72 utterances) the drop against the stock decoder is about 0.3 UTMOS; word error rate (Whisper base.en) is within noise of stock. The decoder is deliberately lossy: some voices (e.g. marius) lose more than others. Seed noise is about ±0.03 UTMOS.

Speed on a Raspberry Pi Zero W (loom/ggml engine, Q8_0 weights, 4.8 s of audio; not PyTorch numbers): 2m 3.1 s (VC4 GPU) / 3.9 s (CPU); 7m 4.7 s / 5.8 s (≈ real time); 18m 8.5–9 s (the stock 18m language model is itself too heavy for this board).

Audio samples

Stock decoder vs. this decoder, same seed, three catalogue voices (alba, marius, javert). Files are in samples/.

voice	text	stock	this decoder
alba	Harvard
alba	long text
marius	Harvard
marius	long text
javert	Harvard
javert	long text

Training procedure

• Teacher: the frozen stock model. The student never changes the language model: it learns to map the flow-LM latents (normalised with emb_mean/emb_std) to audio.
• Stage 1 (core, 20k steps): regression of the 200 Hz features of the teacher's decoder transformer from the latents, plus an STFT loss through the frozen teacher SEANet.
• Stage 2 (joint, 120k steps, cosine LR): core + pruned SEANet (initialised by channel-pruning the teacher SEANet by activation energy) trained on multi-resolution STFT + L1 + 0.3 × latent MSE, plus an LSGAN loss with feature matching from a multi-period discriminator (from step 2000).
• Data (≈ 67 h of latents, 45k utterances): (a) latents generated by the 2m Pocket-TTS language model for HiFi-TTS-2 transcripts with the 21 catalogue voices, random sampling temperature in {0.3–0.9}; (b) 14.7 h of real English speech encoded with the Mimi encoder: LibriTTS (train-clean-100/360; CC BY 4.0), HiFi-TTS (CC BY 4.0) and VCTK (CC BY 4.0). Real speech was cut at pauses into ≤ 8 s segments, kept only if clean (SNR proxy ≥ 30 dB, no clipping), balanced over pitch level and pitch range and over speakers, passed through VAD, Resemble-Enhance denoising and −20 dBFS normalisation so that diversity comes from prosody and spectrum rather than recording conditions.
• One student trained on 2m latents serves all three model sizes; adding real speech improved the 7m/18m results over LM-only latents.
• Evaluation: UTMOS22 and Whisper-base.en WER on Harvard sentences, mel/STFT distance to the stock output.

Limitations

English only; quality is below the stock decoder; trained and tested only with the refl_2026-10 English models; a different latent normalisation would need retraining.

License, attribution and responsible use

Weights: CC-BY-4.0. Derived from Kyutai's Pocket-TTS (CC-BY-4.0): attribution to Kyutai (Pocket-TTS, arXiv:2509.06926). Real-speech training corpora are all CC BY 4.0 (LibriTTS, HiFi-TTS, VCTK). Disclosure: the synthetic-latent part of the training data was generated with the 21 voice embeddings of the Pocket-TTS catalogue; two of them (cosette, jean) were derived by Kyutai from corpora with non-commercial terms (Expresso, EARS). No audio from those corpora was used, only the voice embeddings Kyutai distributes, but if that matters for your use case, please assess it. Code in this repo (src/): MIT.

Prohibited use (inherited from Kyutai): use must comply with all applicable laws and must not involve voice impersonation or cloning without explicit and lawful consent, disinformation or deception (including presenting generated audio as genuine recordings), or unlawful, harmful, abusive, discriminatory or privacy-invasive content.