anilkeshwani/speech-integration-speech-bpe

Speech BPE models over discrete speech units

Byte pair encoding (BPE) models that merge frequent sequences of discrete speech units into larger units. There is one model per speech tokenizer, each learned on the units of MLS English (Multilingual LibriSpeech): HuBERT Large layer 22 k-means 5000, SpeechTokenizer RVQ level 0, Mimi split-RVQ level 0 and FocalCodec 50 Hz. Mimi also has models at 4096, 8192 and 32768 units for a vocabulary-size ablation.

In the speech-integration project, these models shorten the speech part of each training sample for Llama 3.2 1B. They are applied on the fly when a sample is built: the stored unit sequence is deduplicated (runs of the same unit collapsed to one) and then encoded with the model. The extended Llama vocabulary has one token per BPE unit.

Models

Compression is measured on the full MLS validation (dev) split, 3,807 utterances: the total number of deduplicated input units divided by the total number of BPE units. "Units used" is the number of distinct BPE units that occur in the encoded validation split.

Model	Trained on	Input codebook	BPE units	Training data	Mean input length	Mean BPE units	Compression	Units used
hubert_large_ll60k-layer_22-bpe16384	anilkeshwani/mls-hubert_large_ll60k-layer_22	5,000	16,384	trained on 250,000 utterances (129,889,222 units after deduplication)	522.6	303.9	1.720	16,296
speechtokenizer-rvq_0-bpe16384	anilkeshwani/mls-speechtokenizer-rvq_0	1,024	16,384	trained on 250,000 utterances (112,744,234 units after deduplication)	451.3	184.9	2.441	16,325
mimi-srvq_0-bpe16384	anilkeshwani/mls-mimi-srvq_0	2,048	16,384	trained on 500,000 utterances (86,329,893 units after deduplication)	174.4	114.1	1.529	16,027
focalcodec-bpe16384	anilkeshwani/mls-focalcodec	8,192	16,384	trained on 100,000 utterances (74,112,316 units after deduplication)	742.1	736.0	1.008	15,281
mimi-srvq_0-bpe4096	anilkeshwani/mls-mimi-srvq_0	2,048	4,096	first 2,048 merges of mimi-srvq_0-bpe16384	174.4	139.0	1.255	4,051
mimi-srvq_0-bpe8192	anilkeshwani/mls-mimi-srvq_0	2,048	8,192	first 6,144 merges of mimi-srvq_0-bpe16384	174.4	124.9	1.396	8,141
mimi-srvq_0-bpe32768	anilkeshwani/mls-mimi-srvq_0	2,048	32,768	trained on 500,000 utterances (86,329,893 units after deduplication)	174.4	104.9	1.663	29,012

FocalCodec units compress by only 1.01x, so FocalCodec BPE was dropped from the planned comparison. The model is included for completeness.

Compression at smaller vocabularies

validation_compression.json in each trained model's folder also reports the compression of the same model with its merge list truncated to smaller sizes (see "Training"):

Model	BPE units	Mean BPE units	p95	Max	Compression
hubert_large_ll60k-layer_22-bpe16384	10,000	348.0	479	579	1.502
hubert_large_ll60k-layer_22-bpe16384	15,000	310.1	433	538	1.685
hubert_large_ll60k-layer_22-bpe16384	16,384	303.9	425	530	1.720
speechtokenizer-rvq_0-bpe16384	2,048	297.8	406	587	1.515
speechtokenizer-rvq_0-bpe16384	3,072	261.8	360	560	1.724
speechtokenizer-rvq_0-bpe16384	4,096	242.9	336	540	1.857
speechtokenizer-rvq_0-bpe16384	8,192	209.6	294	507	2.153
speechtokenizer-rvq_0-bpe16384	16,384	184.9	262	457	2.441
mimi-srvq_0-bpe16384	4,096	139.0	190	219	1.255
mimi-srvq_0-bpe16384	6,144	130.2	180	212	1.340
mimi-srvq_0-bpe16384	8,192	124.9	175	209	1.396
mimi-srvq_0-bpe16384	16,384	114.1	161	201	1.529
focalcodec-bpe16384	16,384	736.0	958	993	1.008
mimi-srvq_0-bpe32768	4,096	139.0	190	219	1.255
mimi-srvq_0-bpe32768	6,144	130.2	180	212	1.340
mimi-srvq_0-bpe32768	8,192	124.9	175	209	1.396
mimi-srvq_0-bpe32768	16,384	114.1	161	201	1.529
mimi-srvq_0-bpe32768	32,768	104.9	149	193	1.663

Format

Each folder <tokenizer>-bpe<n_units>/ holds:

• tokenizer.json: a Hugging Face tokenizers BPE model with no normalizer, no pre-tokenizer, no post-processor and no special tokens.
• speech_bpe.json: metadata (source dataset, codebook size n_dsus, n_units, sampling seed, number of training utterances and units, trainer version, git commit and command, validation statistics).
• validation_compression.json: compression statistics on the MLS validation split.
• README.md: the provenance note written by the training script. Its paths are those of the training host.

Each speech unit i (0 to n_dsus - 1) is written as the Unicode character chr(0xF0000 + i), in the Supplementary Private Use Area-A, so a unit sequence is a string with one character per unit. Unit IDs are renumbered after training so that BPE unit i is the single speech unit i for i < n_dsus. Merged units follow in merge order, from n_dsus to n_units - 1. A model with fewer units is therefore a prefix of a larger model trained on the same data. A whole utterance is one word, so merges never cross an utterance (or, in interleaved CPT, a speech span).

Encoding example

from itertools import groupby

from datasets import load_dataset
from huggingface_hub import snapshot_download
from tokenizers import Tokenizer

PUA_START = 0xF0000
name = "mimi-srvq_0-bpe16384"
path = snapshot_download("anilkeshwani/speech-integration-speech-bpe", allow_patterns=f"{name}/*")
tok = Tokenizer.from_file(f"{path}/{name}/tokenizer.json")
id_to_str = {i: s for s, i in tok.get_vocab().items()}

row = load_dataset("anilkeshwani/mls-mimi-srvq_0", split="validation")[0]
dsus = [k for k, _ in groupby(row["speech_tokens"])]  # deduplicate first: the models were trained that way
units = tok.encode("".join(chr(PUA_START + d) for d in dsus), add_special_tokens=False).ids

# decode BPE units back to the deduplicated speech units
assert [ord(c) - PUA_START for u in units for c in id_to_str[u]] == dsus
print(len(row["speech_tokens"]), "raw units,", len(dsus), "deduplicated,", len(units), "BPE units")

For the first validation utterance this prints 155 raw units, 146 deduplicated, 95 BPE units.

Training

The 16384-unit models and the 32768-unit Mimi model were trained by scripts/train_speech_bpe.py of the speech-integration project:

1. Sample utterances from the MLS English train split (10,808,037 rows) without replacement with numpy.random.default_rng(42831), and sort the row indices before reading. The sample size is 500,000 for Mimi, 250,000 for HuBERT and SpeechTokenizer and 100,000 for FocalCodec (n_train in speech_bpe.json). After deduplication, the HuBERT, SpeechTokenizer and FocalCodec sequences are 2.6 to 4.3 times longer than Mimi's, so the smaller samples still hold 74M to 130M units, against 86M for Mimi.
2. Deduplicate each unit sequence (collapse runs of the same unit).
3. Map each unit to its Private Use Area character and train tokenizers.trainers.BpeTrainer (tokenizers 0.21.4) with the full codebook as the initial alphabet, vocab_size=n_units and min_frequency=2, one word per utterance.
4. Renumber the vocabulary as described under "Format", check the round trip on the full validation split and record compression.

The Mimi models with 4096 and 8192 units were made by scripts/truncate_speech_bpe.py, which keeps the first n_units - n_dsus merges of mimi-srvq_0-bpe16384. Greedy BPE picks each merge from the pair counts left by the earlier merges, independently of the target size, so this equals training at the smaller size on the same data. The 32768-unit model was trained separately on the same sample. Its first 14,336 merges and its first 16,384 unit IDs are identical to mimi-srvq_0-bpe16384, as expected.

The FocalCodec model was trained from a local copy of anilkeshwani/mls-focalcodec with the same codes.

License

The BPE models are released under the Creative Commons Attribution 4.0 International license (CC BY 4.0). They are learned from units computed on Multilingual LibriSpeech (CC BY 4.0, Pratap et al., 2020), which should be attributed.

Citation

MLS:

@inproceedings{pratap2020mls,
  title     = {{MLS}: A Large-Scale Multilingual Dataset for Speech Research},
  author    = {Pratap, Vineel and Xu, Qiantong and Sriram, Anuroop and Synnaeve, Gabriel and Collobert, Ronan},
  booktitle = {Interspeech 2020},
  pages     = {2757--2761},
  year      = {2020},
  publisher = {ISCA},
  doi       = {10.21437/Interspeech.2020-2826},
  eprint    = {2012.03411},
  archivePrefix = {arXiv}
}

Related repos

• anilkeshwani/mls-hubert_large_ll60k-layer_22
• anilkeshwani/mls-speechtokenizer-rvq_0
• anilkeshwani/mls-mimi-srvq_0
• anilkeshwani/mls-focalcodec
• anilkeshwani/speech-integration-eval-sets: includes per-utterance time maps of BPE units for LibriSpeech dev and MLS validation
• Collection: Speech units for LLM speech integration (MLS English)