seastar105/pocket-tts-korean-300m

Pocket-TTS Korean 300M

Pocket-TTS Korean 300M is a Korean zero-shot text-to-speech model compatible with the official Pocket-TTS Python package and CLI. It is a 24-layer teacher adapted from Kyutai's English Pocket-TTS 24-layer checkpoint.

This is a community model and is not an official Kyutai release.

Model details

• Architecture: Pocket-TTS FlowLM with Lagrangian Self Distillation (LSD) and the Mimi neural audio codec
• Exact parameter count: 336,067,970
  • FlowLM: 316,013,633
  • Mimi: 20,054,337
• FlowLM: 24 transformer layers, model dimension 1024, 16 attention heads
• Korean tokenizer: SentencePiece, 4,000 tokens
• Audio: mono, 24 kHz, 12.5 latent frames per second
• Published weight: step-50k EMA checkpoint.
• Bundle contents: EMA FlowLM weights and the frozen Mimi codec in one Pocket-TTS-format safetensors bundle
• Weight format: float32 safetensors
• Base checkpoint: kyutai/pocket-tts, languages/english_2026-04_24l/model.safetensors, revision 492522650173a0653b7575cdc25ae09810e5d741

Despite the rounded 300M repository name, this is the larger 24-layer teacher, not the standard 6-layer Pocket-TTS student. It therefore uses more memory and is expected to run more slowly on CPU than the approximately 100M official student models.

Quick start

Install and run the official Pocket-TTS CLI:

uvx pocket-tts generate \
    --config hf://seastar105/pocket-tts-korean-300m/korean.yaml \
    --voice ./voice_prompt.wav \
    --text "안녕하세요. 한국어 음성 합성 모델입니다."

The voice prompt should contain clean speech from a speaker who has consented to voice cloning.

Python usage:

from pocket_tts import TTSModel
import scipy.io.wavfile

model = TTSModel.load_model(
    config="hf://seastar105/pocket-tts-korean-300m/korean.yaml"
)
voice_state = model.get_state_for_audio_prompt("./voice_prompt.wav")
audio = model.generate_audio(
    voice_state,
    "안녕하세요. 한국어 음성 합성 모델입니다.",
)
scipy.io.wavfile.write(
    "korean_tts.wav",
    model.sample_rate,
    audio.detach().cpu().numpy(),
)

Pocket-TTS requires Python 3.10 or newer and PyTorch 2.5 or newer. Keep the loaded model and voice state in memory when synthesizing multiple utterances.

Training

The model was warm-started from the released English 24-layer checkpoint. The Korean SentencePiece embedding was reset while the rest of the model was retained and adapted.

Note: In our experiments, from-scratch Korean training collapsed very quickly. This released teacher was therefore warm-started from Kyutai's English 24-layer checkpoint; it is not a from-scratch model.

Note: All intermediate training checkpoints are archived in seastar105/pocket-tts-checkpoints.

• Dataset: seastar105/emilia-yodas-ko-filtered-pocket-tts, with 918,609 training utterances (2,276.88 hours) and 9,472 validation utterances (23.00 hours)
• Training: 50,000 steps, global batch size 64, on 4 NVIDIA RTX 5090 GPUs
• Optimization: AdamW, learning rate 2e-4, 1,000-step warmup followed by a constant schedule, EMA decay 0.999

The exact resolved training arguments are included in training_args.yaml. The training audio and transcripts are not redistributed in this repository.

Evaluation

All 25 EMA checkpoints were evaluated on the complete 500-item Korean zero-shot split of yuekai/CV3-Eval.

Lower CER is better; higher speaker similarity and UTMOS are better. The 2k checkpoint is an immature outlier, especially for CER. Automated scores are useful for checkpoint selection but do not replace listening tests.

Best checkpoint by metric

Model	Lowest CER ↓	Highest speaker sim. ↑	Highest UTMOS ↑
Teacher	20,000 (5.637%)	50,000 (0.9329)	38,000 (2.9502)

Protocol

• Checkpoints: EMA model.safetensors, steps 2k–50k at 2k intervals, from archive revision fd4eadb8b1eac9181029f398564ccc8a767f5d8a.
• Data: all 500 zero_shot_ko items, revision 6ea9d3650fffcbed7c6279e6d1546d01ef1d2796.
• Generation: seed 0, temperature 0.3, CFG 2.0, one decode step, EOS threshold -1.0, maximum 30 seconds, and full prompt audio.
• Intelligibility: openai/whisper-large-v3 with Korean forced, revision 06f233fe06e710322aca913c1bc4249a0d71fce1. CER removes punctuation and preserves spaces; no-space CER additionally removes spaces.
• Voice and quality: microsoft/wavlm-base-plus-sv speaker similarity and UTMOS.
• Storage: audio was deleted immediately after all metrics for each item were committed. No evaluation audio is stored in this repository.

Machine-readable aggregate results are available as CSV and JSON. The earlier five-checkpoint raw-training evaluations remain under eval/cv3_ko_raw_step*.json for provenance and are distinct from this EMA sweep.

Raw scores — 25 EMA checkpoints

Step	CER ↓	No-space CER ↓	Speaker sim. ↑	UTMOS ↑	No EOS
2,000	47.999%	53.586%	0.9294	2.8242	1
4,000	6.936%	6.959%	0.9326	2.8847	0
6,000	7.733%	7.641%	0.9315	2.9068	0
8,000	6.156%	6.162%	0.9304	2.9039	0
10,000	6.711%	6.793%	0.9321	2.8850	0
12,000	6.055%	6.013%	0.9318	2.9202	0
14,000	6.108%	6.128%	0.9277	2.9233	0
16,000	5.971%	5.910%	0.9304	2.9147	0
18,000	5.769%	5.721%	0.9314	2.9100	0
20,000	5.637%	5.532%	0.9321	2.9314	0
22,000	6.020%	5.944%	0.9322	2.9120	1
24,000	5.967%	5.870%	0.9308	2.9123	0
26,000	5.914%	5.818%	0.9295	2.9195	0
28,000	5.923%	5.967%	0.9299	2.9130	0
30,000	5.870%	5.795%	0.9308	2.9374	0
32,000	6.103%	6.053%	0.9307	2.9413	0
34,000	5.879%	5.784%	0.9313	2.9148	1
36,000	5.698%	5.618%	0.9327	2.9265	0
38,000	5.685%	5.635%	0.9317	2.9502	0
40,000	6.169%	6.059%	0.9319	2.9148	0
42,000	6.526%	6.420%	0.9317	2.9150	1
44,000	5.751%	5.641%	0.9321	2.9177	0
46,000	7.147%	7.102%	0.9312	2.9443	0
48,000	6.033%	5.956%	0.9320	2.9318	1
50,000	6.143%	6.099%	0.9329	2.9481	0

Limitations

• English reading quality is very poor; treat this model as Korean-only for practical use.
• Code-switching, numbers, abbreviations, rare names, and unusual punctuation were not systematically evaluated.
• Four intermediate EMA checkpoints each produced one 30-second no-EOS output; the step-50k EMA checkpoint produced none.
• UTMOS is an automated estimate and may be less reliable for Korean than for the data on which it was developed.
• No human listening study, demographic fairness audit, or robustness audit has been performed.
• Output quality and speaker identity depend strongly on prompt cleanliness, duration, recording conditions, and consented speaker coverage.
• Pocket-TTS processes one request at a time and is not thread-safe.

Responsible use

Only clone or imitate a voice with the speaker's explicit and lawful consent. Do not use this model for impersonation, fraud, deception, misinformation, harassment, privacy invasion, or any unlawful or harmful activity. Clearly disclose synthesized audio where listeners could reasonably mistake it for a genuine recording.

License and attribution

The model is released under CC BY 4.0, following the license of the Kyutai Pocket-TTS base weights. Credit Kyutai for Pocket-TTS and cite the original project when using this derivative model:

• Project: https://github.com/kyutai-labs/pocket-tts
• Base model: https://huggingface.co/kyutai/pocket-tts
• Paper: https://arxiv.org/abs/2509.06926

Users remain responsible for complying with the licenses and terms applicable to their prompts, generated content, and downstream uses.