PolyPlex-11B-CJKE

A full-duplex spoken dialogue model for Chinese, Japanese, Korean and English

Quickstart · The tokenizer · The processor · Lineage · License · Citation

Preview release: tokenizer and processor only. This repository currently ships the PolyPlex text tokenizer and the input processor. The model weights will be added to this repository when training is finished. Everything below runs on upstream transformers today; no fork is needed.

✨ Highlights

• A vocabulary sized by speech, not by text. Every token of a full-duplex model fills one 80 ms frame. The vocabulary was grown language by language until words stop being cut by the next word on real conversational speech, measured on Whisper word timings of YouTube speech in 110 languages.
• 223,973 tokens, smaller than Gemma 3. Helium (Moshi's vocabulary) is kept with its ids, Qwen3's tokens are merged in, and whole words and subwords are added only where speech needs them. Written-only Qwen3 words that speech never uses are dropped.
• No byte fragments in speech. Every standard character of every language is a token, so speech never falls back to bytes, which cannot be aligned to sound.
• Drop-in for 🤗 Transformers. AutoTokenizer and AutoProcessor load it with stock transformers; the processor builds the frame-aligned text, user-audio and assistant-audio streams, including the PersonaPlex persona prompt.

🚀 Quickstart

pip install "transformers>=5.14" torch

Tokenizer

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("RetentionLabs/PolyPlex-11B-CJKE")

tokenizer.tokenize("안녕하세요, 반갑습니다.")   # ['▁안녕하세요', ',', '▁반갑습니다', '.']
tokenizer.tokenize("Hello there!")               # ['▁Hello', '▁there', '!']
tokenizer.tokenize("你好，很高兴认识你")           # ['▁', '你好', '，', '很高兴', '认识', '你']
tokenizer.tokenize("こんにちは、元気ですか")       # ['▁', 'こんにちは', '、', '元', '気', 'ですか']

tokenizer.pad_token_id, tokenizer.unk_token_id   # (3, 0): PAD and EPAD of the duplex text stream

Processor

The processor loads the Mimi codec and turns text and audio into the three frame-aligned streams the model reads.

import numpy as np
from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained("RetentionLabs/PolyPlex-11B-CJKE", trust_remote_code=True)

speech = np.zeros(24_000 * 2, dtype=np.float32)          # 2 s of user speech at 24 kHz
inputs = processor(text="안녕하세요", audio=speech)
# input_ids               (1, 25)    text padded with PAD to one token per frame
# user_audio_codes        (1, 8, 25) Mimi codes of the user's speech
# assistant_audio_codes   (1, 8, 25) encoded silence, since no assistant audio was given

prompt = processor(text_prompt="You are a friendly assistant who speaks Korean.")   # persona prefix
waveform = processor.decode_audio(inputs["user_audio_codes"])                       # (1, 1, 48000)

🔤 The tokenizer

Why a duplex model needs its own vocabulary

A text LLM can cut a rare word into many pieces: the sequence only gets longer. A full-duplex speech model writes its text stream in step with audio, one token per 80 ms frame (Moshi, PersonaPlex). A word must fit in the frames between its own onset and the next word's onset. When it does not, the next word overwrites its tail and the word is cut (Moshi, Section 3.4.4, Eq. 5). Byte fragments are worse still: a fragment of a UTF-8 character matches no sound at all.

Qwen3's byte-level BPE, for example, cuts 14.5% of the tokens of spoken Korean into byte fragments, 96% of them at a word's first character. So the PolyPlex vocabulary is designed around one measurable quantity.

Word cut rate: the share of words whose tokens do not all survive in the text stream when the words are placed at their real onsets, one token per frame, the way the model is trained.

Construction

Step	What	Size
1. Base	Helium (Moshi's SentencePiece vocabulary) keeps every token and id. Qwen3's tokens are added with the leading space written as ▁ (Ġword → ▁word); a string both have is one token, Helium's. Every string is NFC.	32,000 + Qwen3
2. Tier 1: characters	Every standard character of each language (CLDR 48.2.3 exemplars, plus the full Unicode block for Indic, Southeast Asian and Georgian scripts), each also as a word-start form ▁c for letters and digits. Unconditional.	13,457 new
3. Tier 2: whole words	Frequent spoken word forms that take two or more tokens, ordered by frames saved (count × (tokens − 1)), added 250 at a time while the cut rate falls.	20,168
4. Tier 3: subwords	Subwords from a Unigram model trained on each language's multi-token spoken words, added 1,000 at a time while the cut rate falls, plus non-standard characters of the language's own scripts seen three or more times.	64,228
5. Scores	Unigram scores by EM over the spoken text of every language, with the vocabulary fixed.
6. Pruning	Qwen3 multi-character tokens the spoken EM never uses are dropped (written Chinese and English words, symbol runs), as are unused word-start forms of single Han and kana characters. Code pieces, emoji and single characters are kept for tool use.	−30,685

For each language, both orders are run (subwords first, then whole words; and whole words first, then subwords), and the order that reaches the target with fewer tokens is kept. A subword shortens every word that holds it, so subwords win for scripts where words take many tokens (Greek: 5,016 subwords instead of 21,016 tokens; Hindi: 6,000 instead of 19,500); whole words win where a few hundred to a few thousand forms suffice (Korean: 3,750 words).

Measuring the cut rate

• Speech. User-subtitled YouTube videos of YODAS2 in every language that has them, Korean included. Neighbouring subtitle segments are joined into pieces of at most 25 s.
• Word timings. Whisper large-v3, with the transcript forced (teacher forcing) and DTW over the alignment heads' cross-attention. The timings are released as RetentionLabs/polyplex-word-timings.
• Filtering. Pieces whose alignment loss exceeds 1.61 (the 95th percentile on KsponSpeech) are subtitles that do not match the speech. Pieces where over 10% of neighbouring onsets are under 20 ms apart were not aligned. A word whose span or gap is under 30 ms per character is an alignment error: it keeps its frames but counts neither as cut nor in the total.
• Approximation. Languages left with fewer than 2,000 timed words, and scripts without spaces, place words by their share of characters in the utterance duration (FLEURS recordings, or the script's characters per second).
• Targets. 1%, or the lowest reachable rate (every word one token) plus 0.2 points when that is higher. The main languages, English, Korean, Japanese and Chinese, go on to the lowest rate plus 0.2 points even below 1%.

Result

Final cut rates are measured with the final tokenizer on the evaluation part of each language, held out from selection.

Language	Tier 1	Tier 2	Tier 3	Word cut rate	Target
English	10	0	0	0.18%	0.28%
Korean	2,432	3,750	48	0.23%	0.23%
Japanese	8,817	0	3	0.00%	0.20%
Chinese (Simplified)	6,850	0	3	0.00%	0.20%
Chinese (Traditional)	6,801	0	98	0.03%	0.20%
Cantonese	6,801	0	0	0.01%	1.00%

Across all 110 measured languages the median cut rate is 0.25%, and 109 are below their target. Punjabi stays at 1.97%: it has no subtitles in OpenSubtitles and five YODAS2 shards, so its candidate text is 2,198 lines. It is left above the target rather than filled with written text.

Vocabulary	Size
Helium (Moshi, PersonaPlex)	32,000
Llama 3	128,256
Qwen3	151,669
PolyPlex	223,973
Gemma 3	262,144

Layout and behaviour

Item	Value
Model	SentencePiece-style Unigram with byte fallback (tokenizers)
Normalization	NFC, then spaces to ▁ with one ▁ in front of the text
Word starts	Marked by ▁ on the first token of every word. EPAD marks the end of a pause and is skipped when the previous word still occupies its frame (Moshi), so it cannot mark word starts.
Digits	One token per digit; no multi-digit tokens exist
Ids 0 – 31,999	Helium, unchanged: <unk> = 0 (EPAD in the duplex stream), <s> = 1, </s> = 2, <pad> = 3 (PAD), bytes 4 – 259
Ids 32,000 – 32,005	Control tokens <tool_call> </tool_call> <tool_response> </tool_response> <think> </think>
Ids 32,006 – 223,972	The rest of the base block, then one block per script, each ordered by score
LM head	224,000 rows (padded to a multiple of 128). The opening frame id is 224,000 and exists only as an input row.

Control and special tokens never come out of text: their Unigram score lies far below every text piece, so a literal <think> in text is cut as ▁< think >.

The id range of each script block is stored as blocks in tokenizer_config.json. A script-group output head (the base block plus one script's block) can be cut from it:

Block table

Block	Ids	Tokens
base	0 – 36,606	36,607
latin	36,607 – 107,775	71,169
cyrillic	107,776 – 116,037	8,262
greek	116,038 – 121,270	5,233
armenian	121,271 – 126,639	5,369
georgian	126,640 – 126,896	257
hebrew	126,897 – 130,022	3,126
arabic	130,023 – 139,702	9,680
devanagari	139,703 – 150,566	10,864
bengali	150,567 – 150,739	173
gurmukhi	150,740 – 157,581	6,842
gujarati	157,582 – 157,985	404
oriya	157,986 – 158,390	405
tamil	158,391 – 162,326	3,936
telugu	162,327 – 162,493	167
kannada	162,494 – 162,651	158
malayalam	162,652 – 163,847	1,196
sinhala	163,848 – 178,857	15,010
thai	178,858 – 180,917	2,060
lao	180,918 – 183,056	2,139
khmer	183,057 – 185,486	2,430
myanmar	185,487 – 190,240	4,754
hangul	190,241 – 200,006	9,766
han	200,007 – 221,049	21,043
kana	221,050 – 222,717	1,668
other	222,718 – 223,948	1,231
mixed	223,949 – 223,972	24

start, end = tokenizer.init_kwargs["blocks"]["hangul"]

Data used to build the vocabulary

Use	Data
Standard characters	Unicode CLDR 48.2.3 exemplar characters
Candidate words and EM scores	OpenSubtitles (first 200,000 lines per language), YODAS2 subtitles of shards not used for evaluation, FLEURS transcripts, Korean spoken corpora (AI Hub daily multi-session dialogue, colloquial Korean–English), Wikipedia only for languages without spoken text
Word cut rate	YODAS2 pieces with Whisper word timings; FLEURS read speech as a reference only
Check of the method on Korean	KsponSpeech evaluation split

🎛 The processor

processing_polyplex.py depends on upstream transformers only. Its behaviour matches the PersonaPlex processor output for output (tested on text, audio, assistant audio and persona prompts).

Input	Output
text	input_ids, attention_mask, padded with PAD to one token per audio frame
audio	user_audio_codes (batch, 8, frames)
assistant_audio	assistant_audio_codes (batch, 8, frames)
A stream left out	Filled with Mimi-encoded silence, not with a reserved id
voice_prompt, text_prompt	The persona prefix in front of the inputs (needs pyloudnorm for the voice prompt)

The persona prefix follows PersonaPlex's training format:

Stage	Text	Assistant stream	User stream
Voice prompt	PAD	the voice sample	440 Hz tone
0.5 s	PAD	silence	tone
Text prompt	<system> … <system>, one token per frame	silence	tone
0.5 s	PAD	silence	tone

🧬 Lineage

kyutai/moshiko-pytorch-bf16 ──▶ nvidia/personaplex-7b-v1 ──┐
                                                            ├──▶ RetentionLabs/PolyPlex-11B-CJKE
Qwen/Qwen3-8B ──────────────────────────────────────────────┘

PolyPlex keeps PersonaPlex's attention, its feed-forward layers, its depth decoder and the Mimi codec, and places the feed-forward layers of Qwen3-8B beside PersonaPlex's in every layer. Qwen3 carries multilingual knowledge; PersonaPlex's feed-forward layers keep the duplex behaviour of speaking, listening and turn-taking. The text vocabulary joins Helium's and Qwen3's as described above.

📚 Related work

Work	What PolyPlex takes from it
Moshi (Kyutai, 2024)	The full-duplex architecture, the frame-aligned text stream with PAD/EPAD (Section 3.4.4), the Helium vocabulary and the Mimi codec
PersonaPlex (NVIDIA, 2026)	The duplex checkpoint PolyPlex is grafted onto, and the persona prompt the processor builds
J-Moshi (Ohashi et al., Interspeech 2025)	A Japanese Moshi, pre-trained on large-scale Japanese spoken dialogue (J-CHAT) and fine-tuned on 344 hours of stereo dialogue
Human-1 (Josh Talks, 2026)	A Hindi Moshi with a Hindi tokenizer and re-initialized text parameters, trained on 26,000 hours of real telephone conversations
Qwen3 (Qwen Team, 2025)	The multilingual feed-forward layers and token strings
Pretraining LLMs with NVFP4 (NVIDIA, 2025)	Which parts stay in high precision when the backbone runs in NVFP4: embeddings, the LM head, normalizations and attention, and a share of the linear layers
An Empirical Study of Qwen3 Quantization (2025)	Qwen3 holds up under 4-bit weights but collapses when activations are quantized to 4-bit integers, which is why the backbone uses block-scaled NVFP4 and is checked layer by layer

J-Moshi and Human-1 adapt Moshi to one new language each with thousands of hours of that language's conversational speech. PolyPlex takes multilingual knowledge from a text LLM instead, so a new language needs its text and a little speech rather than a large conversational corpus.

📜 License

This repository is governed by the NVIDIA Open Model License Agreement, because PolyPlex is derived from PersonaPlex.

Attributions:

• PersonaPlex is derived from Moshiko by Kyutai, released under CC-BY-4.0. The first 32,000 tokens of the vocabulary are Moshi's Helium tokenizer.
• Token strings and, in the full model, feed-forward weights come from Qwen3-8B by the Qwen team, released under Apache-2.0.
• processing_polyplex.py is released under Apache-2.0.

📖 Citation

@misc{polyplex2026,
  title        = {PolyPlex: Data-efficient Multilingual Transfer for Duplex Spoken LMs},
  author       = {{RetentionLabs}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/RetentionLabs/PolyPlex-11B-CJKE}}
}

@article{defossez2024moshi,
  title   = {Moshi: a speech-text foundation model for real-time dialogue},
  author  = {D{\'e}fossez, Alexandre and Mazar{\'e}, Laurent and Orsini, Manu and Royer, Am{\'e}lie and
             P{\'e}rez, Patrick and J{\'e}gou, Herv{\'e} and Grave, Edouard and Zeghidour, Neil},
  journal = {arXiv preprint arXiv:2410.00037},
  year    = {2024}
}

@article{roy2026personaplex,
  title   = {PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models},
  author  = {Roy, Rajarshi and Raiman, Jonathan and Lee, Sang-gil and Ene, Teodor-Dumitru and Kirby, Robert and
             Kim, Sungwon and Kim, Jaehyeon and Catanzaro, Bryan},
  journal = {arXiv preprint arXiv:2602.06053},
  year    = {2026}
}

@article{qwen3technicalreport,
  title   = {Qwen3 Technical Report},
  author  = {{Qwen Team}},
  journal = {arXiv preprint arXiv:2505.09388},
  year    = {2025}
}