You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

pocket-tts-enid-v3-edit: Indonesian + English TTS with mid-sentence speech editing

Private / internal. Kyutai pocket-tts, trained from scratch for Indonesian + English, with speech editing added.

What it does:

  • Text-to-speech in Indonesian, English and code-switched Indonesian/English, in the voice of a ~5 s audio prompt (zero-shot voice cloning). Output is 24 kHz mono. Runs faster than real time on a laptop CPU.
  • Speech editing: replace, insert or regenerate a span of words inside an existing recording, in the speaker's voice. The new words join smoothly into the audio on both sides, and the audio outside the span stays the original.
  • Long text is split into sentence chunks automatically; streaming output is available from the Python API.

One model does both. The edit-capable weights pass the plain-TTS accuracy gate against the earlier TTS-only release, so they replace it for plain TTS too.

Which model?

6l-edit-100k (default) small-f1-soup
Flow LM size 89M 64M (28% smaller, flow LM ~1.3Γ— faster)
Quality best voice similarity Indonesian about equal; English voice similarity lower
Temperature 0.3 0.5 (0.3 can rarely collapse into near-silence)
TTS config pocket_tts_enid_v3_edit_6l_tts.yaml pocket_tts_enid_small_f1_soup_tts.yaml
Edit config code/configs/6l_edit_infer.yaml code/configs_small/small_f1_soup_edit_infer.yaml

Use 6L unless size or speed matters more than English voice similarity. Both run in the same environment, and the TTS configs already set the right temperature. Benchmarks for both are under Results.

Quick start: text-to-speech

Access. This repo is private. Log in once with an account that has access (hf auth login, or set HF_TOKEN). Every hf:// path below downloads from this repo; the first run fetches ~440 MB (6L) or ~340 MB (small) of weights.

Install (Python 3.10–3.14; tested on macOS Apple Silicon CPU and Linux):

pip install pocket-tts==4.0.1

Command line:

pocket-tts generate \
  --config hf://acul3/pocket-tts-id-en-edit/pocket_tts_enid_v3_edit_6l_tts.yaml \
  --voice prompt.wav \
  --text "Selamat pagi, acara hari ini kita mulai dengan evaluasi kinerja kuartal ketiga." \
  --output-path out.wav
# small model: --config hf://acul3/pocket-tts-id-en-edit/pocket_tts_enid_small_f1_soup_tts.yaml

About 5 s for a 4 s sentence on an Apple M-series CPU, model loading included.

Python:

import librosa, soundfile as sf, torch          # pip install librosa soundfile
from pocket_tts import TTSModel

m = TTSModel.load_model(config="hf://acul3/pocket-tts-id-en-edit/pocket_tts_enid_v3_edit_6l_tts.yaml")

# voice prompt: silence-trimmed, ~5 s, at the model's sample rate (24 kHz)
wav, _ = librosa.load("prompt.wav", sr=m.sample_rate, mono=True)
wav, _ = librosa.effects.trim(wav, top_db=35)
voice = m.get_state_for_audio_prompt(torch.from_numpy(wav[None, : 5 * m.sample_rate])   # [channels, samples])

audio = m.generate_audio(voice, "Oke, next agenda-nya kita bahas timeline launching produk baru, ya.")
sf.write("out.wav", audio.numpy(), m.sample_rate)   # 1-D float tensor, 24 kHz

for chunk in m.generate_audio_stream(voice, "Kalimat yang panjang ..."):   # streaming: chunks as they are ready
    ...

Tips:

  • Voice prompt: silence-trim it and use about 5 s. The output copies the prompt's acoustics (room, mic, noise), so a clean prompt gives clean output. get_state_for_audio_prompt also accepts a file path.
  • Numbers: write them as words ("dua puluh lima", not "25").
  • Long text is split at sentence boundaries into chunks of at most max_tokens (default 50) and generated in turn (~1.3 min stories tested: WER < 1%).
  • No cfg knob needed: both models are distilled with guidance baked in. Stock pocket-tts has no cfg option; that's expected. Change the temperature with --temperature / TTSModel.load_model(..., temp=...).
  • pocket-tts 3.x can't load the small model (it fails with a pydantic extra_forbidden error on z_dim); use 4.0.1. The 6L model works on 3.3.0 too.

Quick start: speech editing

Editing needs our patched pocket-tts (the stock package can't load the 3 edit tensors) and word timestamps for the clip. It runs on CUDA or CPU (no MPS); on a laptop CPU an edit takes about 0.6 s.

1. Set up the patched repo (one checkout runs both models):

hf download acul3/pocket-tts-id-en-edit --local-dir enid --include "code/*"   # patches + configs, ~0.3 MB
git clone https://github.com/kyutai-labs/pocket-tts && cd pocket-tts && git checkout v4.0.1
git apply ../enid/code/small-edit-v4.0.1.patch
uv sync                                  # or: python -m venv .venv && pip install -e . (β‰ˆ 1.5 min)

The first edit run also downloads the stock kyutai/pocket-tts English weights (~40 s); only the Mimi codec is used.

2. Get word timestamps. If your ASR gives word timestamps, use them. Otherwise force-align a transcript with the aligner that ships in the patched repo. Make align_in.jsonl, one clip per line:

{"path": "clip.wav", "duration": 3.96, "transcript": "Kemudian yang lain yang berpengaruh terhadap Indonesia, kita juga punya angka ya."}
PYTHONPATH=. uv run python -m training.scripts.align_data align_in.jsonl aligned.jsonl \
  --model indonesian-nlp/wav2vec2-large-xlsr-indonesian --device cpu
# English clips: leave out --model (default facebook/wav2vec2-base-960h)

This adds a "words" list (punctuation stays attached to the words, which is fine). The first run downloads the 1.2 GB aligner; aligning itself takes about a second per clip.

  • Forced-aligned timestamps work better than Whisper's. Whisper's word timestamps have no gaps between words, so the --expand-gap fix (below) widens almost every edit by one word.

3. Write cases.jsonl, one edit per line: an aligned row plus id, span, new_text (and start, default 0):

{"id": "e1", "path": "clip.wav", "start": 0.0, "duration": 3.96,
 "words": [{"word": "Kemudian", "start": 0.121, "end": 0.462}, "..."],
 "span": [10, 11], "new_text": "data yang lengkap"}
  • span = [i, j] replaces words[i:j]; j == i inserts before word i.
  • If new_text is omitted, the original words are regenerated.
  • start / duration select a window of a longer file. The model sees at most 5 s before the edit and 4 s after it.

4. Run:

# 6L
PYTHONPATH=. uv run python -m training.scripts.speech_edit ../enid/code/configs/6l_edit_infer.yaml \
  hf://acul3/pocket-tts-id-en-edit/6l-edit-100k/model.safetensors cases.jsonl out/ --cfg 1.0
# small
PYTHONPATH=. uv run python -m training.scripts.speech_edit ../enid/code/configs_small/small_f1_soup_edit_infer.yaml \
  hf://acul3/pocket-tts-id-en-edit/small-f1-soup/model.safetensors cases.jsonl out/ --cfg 1.0 --temp 0.5

Always pass --cfg 1.0 for these models: the script's default (2.0) is for the 24L teacher.

Outputs in out/:

  • <id>.wave.wav: original audio outside the span (use this).
  • <id>.latent.wav: whole clip decoded from the spliced latents (uniform codec colour).
  • <id>.orig.wav: codec round-trip of the original, for reference.
  • results.jsonl: per edit, the effective span (span_eff), frame counts and the best-of candidates.

Inference fixes, on by default (each can be turned off):

Flag What it does Off
--snap moves each cut Β±1 frame to the quietest point inside the word gap --no-snap
--expand-gap 0.04 also regenerates the neighbouring word when two words run together (< 40 ms gap) --expand-gap 0
--dur-guard no EOS before --min-ratio (0.75) Γ— the length expected from the speaker's own speaking rate --no-dur-guard
--best-of 4 generates 4 candidates in one batch, keeps the one closest to the expected length --best-of 1

Other options: --temp (default 0.3), --seed, --max-voice-prompt-sec (5), --right-max-sec (4), --eos-threshold, --baseline (left context only, for comparison), --no-use-ema.

Teacher. teacher-24l-edit-75k/ (24 layers, run at cfg 2.0) is for distilling new students, not for use: the 6L student matches it on every gate metric. Its training args are in teacher-24l-edit-75k/training/args.yaml.

Legacy setup: the 6L model was trained on pocket-tts a012a26 (two commits after v3.3.0) with code/speech-edit.patch. That setup still works with code/configs/6l_edit_infer.yaml (same span lengths on our check cases).

How editing works

pocket-tts is autoregressive and causal, so it cannot look ahead. Edit mode puts the audio after the edit earlier in the sequence than the words being generated (the VoiceCraft idea):

[bos][left context ≀ 5 s][right-context marker][right context ≀ 4 s][text: left words ⟨S⟩ new words ⟨E⟩ right words] β†’ new span latents β†’ EOS
  • Left context is the normal voice prompt (the audio just before the edit).
  • Right context is the next ≀ 4 s of the original, projected like the voice prompt, after a learned marker.
  • Text is the surrounding words with learned span-start/end markers around the words to generate.
  • Splicing: the model generates only the span and stops (EOS). The new latents are spliced between the original Mimi latents and decoded. The released output keeps the original waveform outside the span, with a 40 ms crossfade at each join.
  • New parameters: 3 small vectors (edit_right_bos, edit_span_start, edit_span_end). Without them the sequence is bit-identical to stock pocket-tts.

Files

Path What Use
6l-edit-100k/model.safetensors 6L student, edit-capable, EMA (flow LM + Mimi) speech editing (patched repo)
6l-edit-100k/model_tts_only.safetensors same weights minus the 3 edit tensors plain TTS with stock pocket-tts
pocket_tts_enid_v3_edit_6l_tts.yaml stock pocket-tts config for the 6L TTS-only weights (temperature 0.3) plain TTS
small-f1-soup/model.safetensors small model, edit-capable, EMA speech editing (patched repo, v4.0.1)
small-f1-soup/model_tts_only.safetensors same minus edit tensors plain TTS with stock pocket-tts β‰₯ 4.0
pocket_tts_enid_small_f1_soup_tts.yaml stock pocket-tts config for the small TTS-only weights (temperature 0.5) plain TTS
tokenizer.json, tokenizer.model SentencePiece unigram 4000 (id + en) all configs
code/small-edit-v4.0.1.patch all our changes on top of pocket-tts v4.0.1: speech edit, z bottleneck, configs, tests editing / training (both models)
code/speech-edit.patch the same speech-edit changes on top of pocket-tts a012a26 (v3.3.0 + 2) legacy 6L setup
code/configs/6l_edit_infer.yaml, code/configs_small/small_f1_soup_edit_infer.yaml edit inference configs editing
code/configs/, code/configs_small/ (other files) teacher fine-tune, student distillation and small-model training configs (paths point to our training machine) reproduce training
teacher-24l-edit-75k/ 24L teacher (EMA) + its training checkpoint and args distil new students, further fine-tunes
teacher-24l-edit-z128-30k/model.safetensors the z128 teacher (EMA) used for the small model distil new small students
benchmarks/ accuracy-gate tables (3 replicates) and edit-benchmark scores; benchmarks/small/ for the small model β€”

Results

Normal TTS: accuracy gate, 3 replicates

Protocol: trimmed 5 s prompt, EMA; minimax-id / FLEURS-id, 100 utterances each; Whisper-large-v3 WER, WavLM-ECAPA SIM-o, UTMOS. Students at cfg 1.0, temperature 0.3.

Earlier TTS-only 6L 200k 6L edit 100k Ξ” Gate small-f1-soup
WER minimax / FLEURS (%) 1.52 / 4.49 1.44 / 4.53 βˆ’0.07 / +0.04 ≀ +0.3 / +0.5 βœ… 1.59 / 4.42
SIM minimax / FLEURS 0.577 / 0.605 0.581 / 0.599 +0.004 / βˆ’0.006 β‰₯ βˆ’0.01 βœ… 0.554 / 0.574
UTMOS minimax / FLEURS 3.197 / 2.830 3.183 / 2.810 βˆ’0.014 / βˆ’0.020 β‰₯ βˆ’0.05 βœ… 3.10 / 2.73
  • small-f1-soup passes its own pre-registered gate vs the 6L edit model. English voice similarity drops more than Indonesian: LibriSpeech test-clean SIM 0.457 vs 0.513 (English is a minority of the training mix).

  • Teacher, cfg 2.0 (accepted with the FLEURS UTMOS dip; it did not carry over to the student):

    Earlier 24L 400k Edit teacher 24L 75k
    WER minimax / FLEURS (%) 1.52 / 4.89 1.69 / 4.84
    SIM minimax / FLEURS 0.604 / 0.620 0.595 / 0.618
    UTMOS minimax / FLEURS 3.10 / 2.74 3.06 / 2.66

Speech editing: 200-case benchmark

The cases come from a held-out, speaker-disjoint set: 50 reconstruction + 50 substitution per language, spans of 1–3 words.

  • Joins: loudness (dB) and pitch (semitones) jumps across each join, compared with the original recording at the same place.
  • Intelligibility: Zapformer ASR on the edited clip.

Medians unless noted.

Stock student, left context only 6L edit (default fixes) small-f1-soup Natural
Loudness jump vs natural (left / right join) +30 dB (right) +0.4 – 1.2 dB +2.2 / +0.8 dB β€”
Pitch jump after / before the edit 5.5 / 2.3 st 1.6 / 1.55 st 1.6 / 1.43 st 1.95 / 1.6 st
Span length vs original (reconstruction) 1.33Γ— 1.00Γ— 1.00Γ— 1.0Γ—
Length off by > 30 % (outliers) β€” 16 % β€” β€”
Span speaker similarity (WavLM-ECAPA) 0.301 0.348 0.318 0.538 raw / 0.358 after a Mimi round-trip
WER of the edited clip (mean) 0.113 0.092 0.087 0.115 (codec round-trip of the original)
New words heard by ASR (mean) 83 % 89 % 89 % β€”
  • Speaker-similarity ceiling: most of the gap to natural is the Mimi codec on short spans. The original words themselves score 0.358 after a codec round-trip, and the 6L edit model reaches ~97 % of that.
  • Loudness-jump caveat: at cuts snapped into near-silence, the raw-vs-codec noise floor inflates the dB metric even though it is inaudible. Judge joins by ear.

Speed

  • 6L, plain TTS: ~10Γ— faster than real time on an Apple M-series CPU (CLI, after loading).
  • small: flow LM 1.3–1.4Γ— faster than 6L on x86 CPU; the Mimi decoder (0.16 RTF) is unchanged and dominates.
  • Edit: ~0.6 s per edit on a laptop CPU (best-of-4 batch included).

On-device

Not in this repo, but they exist; ask the maintainer:

  • Android (LiteRT), 6L edit model, TTS + editing. A 408 MB pack used in our Android app. Verified on a phone (Snapdragon 8 Elite, 4 CPU threads): outputs match the host, a full edit takes 1.7–3.8 s for 3–11 s clips.
  • small model: an ExecuTorch .pte for react-native-executorch (TTS, streaming, voice cloning on device) and a GGUF pair for llama.rn / codec.cpp (q8_0 backbone 55 MB + f16 codec 92 MB). Plain TTS only; editing is not ported.
  • The stock pocket-tts runtimes (audio.cpp, sherpa-onnx) do not support editing; load model_tts_only.safetensors for plain TTS there.

How the models were made

Data: Indonesian + English speech with word-level timestamps. The data sources are not disclosed.

  • Latents are precomputed, including a cold-encoded stitch for every cut the loader can pick, so training reads no audio.

Edit samples (40 % of training rows):

  • Spans are 1–8 words / 0.3–3 s, cut at word-gap midpoints, with β‰₯ 0.5 s of context on both sides.
  • Targets are the stored full-context latents, so a generated span splices back between the original ones.

6L stages:

  1. Feasibility check: a direct 6L fine-tune for 25k steps showed that editing is learnable. The right-join loudness jump fell from 30 dB to 5.3 dB.
  2. Teacher: fine-tuned the earlier 24L 400k teacher for 75k steps, lr 5e-5 cosine, CFG dropout 0.2.
  3. Student: depth-distilled from the teacher (cfg-coef 2.0, guidance baked in).
    • Backbone warm-started from the earlier 6L 200k; heads and edit vectors come from the teacher.
    • 100k steps, lr 1e-4 cosine, EMA 0.9999.

Small model (6 layers, d_model 1024, 16 heads, FFN 2048, z_dim 128 bottleneck flow_lm.z_proj feeding the LSD sampler and EOS heads):

  1. z128 teacher. The 24L edit teacher gets a 128-dim bottleneck by reduced-rank regression on the head inputs, then a 30k-step fine-tune: FLEURS SIM βˆ’0.010 vs the original teacher. (Kyutai's 64-dim interface cost too much voice similarity on our data: βˆ’0.024.)
  2. A128. 6L/1024 + z128 student, warm-started from the 6L edit model, distilled with guidance baked in. 50k steps.
  3. F1. PCA cut of A128's FFN 4096 β†’ 2048 (shrink_checkpoint --cov), distilled for 150k steps.
  4. Edit-boost. F1 + 40k steps with 60% edit samples (lr 5e-5).
  5. Soup. 50/50 weight average of F1-150k and the edit-boost: this release.

Hardware: 2 Γ— A100-80GB; about 1.5 GPU-days for the 6L edit model.

Limitations

  • Occasional bad draws remain in about 16 % of edits, mostly length outliers; best-of-4 and the duration guard reduce them. Re-run with another --seed if an edit sounds wrong.
  • Mimi codec colour: the edited words are Mimi-decoded while the surrounding audio is the original waveform, so on very clean studio clips the edited words can sound slightly different. Use latent.wav for uniform colour.
  • Needs word timestamps for the clip (step 2 above). Edits at run-together words are cleaner with --expand-gap.
  • small model: English voice similarity is lower than 6L, and the edited span's similarity is 0.03 lower.
Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support