File size: 1,463 Bytes
c8936ba | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | ---
base_model: google/gemma-4-E2B-it
license: gemma
language: en
tags:
- vocabulary-trimming
- tokenizer-pruning
---
# Gemma 4 E2B-it, vocabulary pruned to English (m = 1)
Homework 1 (tokenizer pruning), Modern Methods and Algorithms of Generative AI, Skoltech, Fall 2026.
`google/gemma-4-E2B-it` with its vocabulary pruned from 262,144 to 103,351 tokens (padded to 103,360 rows):
every token that occurs at least once when 12 MB of English Wikipedia (`wikimedia/wikipedia`, `20231101.en`)
is tokenized, plus all special/added tokens, the tokenizer's base alphabet, and the merge closure (route
closure: every piece formed while BPE builds a kept token). No training: the input embedding (tied with the
output head) and the per-layer embedding table are sliced with the id map (`pruning.json`: new id -> old id);
all token-id fields in the configs are remapped.
| | original | pruned |
|---|---:|---:|
| vocabulary | 262,144 | 103,351 |
| parameters | 5.10 B | 3.44 B (-32.7 %) |
| bf16 checkpoint | 9.54 GiB | 6.4 GiB |
| held-out bits per byte (English Wikipedia, 1 MB) | 1.495 | 1.526 |
| MMLU-Pro (12,032, direct answer) | 31.55 | 31.16 |
Loads with `transformers` 5.14 (`AutoModelForImageTextToText`, `AutoTokenizer`). Rows 103,351-103,359 are zero
padding: suppress them when generating. Text outside English Wikipedia's vocabulary (other scripts, some
markdown, ALL-CAPS words) is spelled with smaller pieces or bytes and is generated less reliably.
|