---
language:
- en
license: cc-by-sa-4.0
library_name: numpy
pipeline_tag: text-generation
tags:
- next-word-prediction
- word-completion
- keyboard
- on-device
- lstm
- int8
datasets:
- allenai/soda
- OpenAssistant/oasst2
- google-research-datasets/taskmaster1
- google-research-datasets/taskmaster3
- allenai/c4
---

# Chime 2

Chime is a small English model that predicts the next word and completes the word being typed. I built it for [LocalType](https://dr.eamer.dev/downloads/apps/localtype/), my Android keyboard, where it runs on the phone on every keystroke. It has 2,025,114 parameters, and the int8 model is a 2.2 MB file. [Try it in your browser](https://dr.eamer.dev/chime/).

This is Chime 2, the file LocalType has bundled since 0.5.58. It has the same shape and size as every earlier Chime. What changed is how some of its training text was chosen: a filter built from my own sent text messages picked 1,000,000 sentences out of public web text because they read like texting, and the model trained on those along with everything Chime 1.2 learned from. My messages only steered the choice. None of their words are in the training text.

| Version | In LocalType | What changed | Where |
| --- | --- | --- | --- |
| **Chime 2** | 0.5.58 onward | 1,000,000 web sentences chosen by a filter built from my messages | this repository, tag `v2.0` |
| Chime 1.2 | 0.5.56 and 0.5.57 | Tatoeba sentences added to the training text | this repository, tag `v1.2` |
| Chime 1.1 | 0.5.3 to 0.5.55 | human-written sources repeated eight times instead of four | [lukeslp/chime-sms-2m](https://huggingface.co/lukeslp/chime-sms-2m) |
| Chime 1.0 | 0.5.0 to 0.5.2 | the first bundled model | not published |

## Model details

- **Developed by:** Luke Steuber
- **Model type:** word-level language model: one CIFG-LSTM layer (670 units, projected to 96 dimensions) with tied input and output embeddings, the design of Gboard's published next-word model ([Hard et al., 2018](https://arxiv.org/abs/1811.03604))
- **Vocabulary:** 16,384 tokens: 16,054 words, 293 emoji, 31 punctuation marks and 6 special tokens. It is the October vocabulary, unchanged.
- **Context:** the sentence being typed, up to 63 tokens
- **Language:** English
- **License:** CC BY-SA 4.0 for the weights, Apache-2.0 for the code

## Uses

Chime ranks candidate words for a keyboard's suggestion strip: the next word after a space, or completions of a partly typed word. It is not a chat or text-generation model, and it does not correct spelling.

LocalType learns personal word choices separately. Chime's weights stay fixed on the phone.

## How to use

```sh
python3.12 -m venv .venv && . .venv/bin/activate
python -m pip install huggingface_hub
hf download lukeslp/chime --local-dir chime && cd chime
python -m pip install -r requirements.txt
python inference.py "see you "
```

A trailing space asks for the next word; without one, the last word is completed. From Python, `LocalTypeModel().predict("see you ", k=6)` in `inference.py` does the same. Inference needs only NumPy and runs offline.

`cifg_int8.bin` is the int8 model with its vocabulary, the file LocalType ships, and `read_bin` in `inference.py` reads both. `model.safetensors` holds the float32 weights; `config.json` lists their shapes.

## Result on unseen text

The measure is next-word top-three accuracy: how often the word that was typed next is among three suggestions, with no personal learning. The set is a fiftieth of the Mobile Text Dataset's phone-written training file (Vertanen and Kristensson, 2019), taken by a hash of each line, after dropping every line that equals a line of either model's training text. Nothing had scored these lines before, and they were scored once.

| Unseen set | Sentences | Words scored | Chime 1.2 | Chime 2 | Difference (95% interval) |
| --- | ---: | ---: | ---: | ---: | ---: |
| Phone-written forum and blog sentences | 18,554 | 216,806 | 24.85% | 26.01% | +1.16 (+1.04 to +1.28) |

Potential keystroke savings rose by 1.12 points (+1.07 to +1.17). Intervals come from a paired bootstrap over sentences.

What this figure can and cannot carry:

- It is one set, and its sentences are forum and blog prose typed on phones, not text messages.
- I had planned a second check on my own messages sent after the filter was built. It needed about a month of new messages, and I shipped without it. So the claim that the filter helps on texting rests on the development lines below, which are optimistic.
- The Mobile Text sentences have no stated license, so I use them for scoring only.
- Potential savings assumes the right word is tapped the moment it appears, so it is a ceiling. Savings in everyday typing have not been measured.

## Development measurements

These lines helped choose between recipes, so they are not held-out results. The first two rows are 5,000 lines each from the development side of my scoring sets; the last two come from the training sources' development splits.

| Development set | Lines | Chime 1.2 | Chime 2 | Difference (95% interval) |
| --- | ---: | ---: | ---: | ---: |
| My own sent text messages | 5,000 | 24.17% | 25.51% | +1.34 (+1.12 to +1.57) |
| Phone-written forum and blog sentences | 5,000 | 25.35% | 26.38% | +1.03 (+0.79 to +1.26) |
| NUS SMS | 463 | 19.28% | 19.51% | +0.23 (−0.55 to +1.00) |
| Taskmaster-1 | 567 | 42.43% | 41.29% | −1.14 (−1.97 to −0.39) |

Two things to read from this table:

- **The messages row is optimistic.** Those lines were never training text and the filter never saw them, but they come from the same conversations and months as the messages the filter was built from.
- **Chime 2 is worse on task requests.** Taskmaster-1 is people ordering food, booking rides and making appointments in writing. Chime 2 is 1.14 points behind Chime 1.2 there, and the interval excludes zero. The web sentences pull the model toward casual text and away from that kind of request.

## Numerical checks and speed

Int8 export raised perplexity from 13.516883 to 13.523262 (+0.047%) on 5,000 development lines. The NumPy code in this package, LocalType's Kotlin code and the browser demo's JavaScript each reproduce the export's 120 reference contexts and their top-ten rankings. Numerical agreement is separate from prediction quality.

I have not timed this file on a phone. It has the same shape and byte size as Chime 1.1, and int8 timing does not depend on the weight values. Chime 1.1 measured 3.09 ms (Pixel 9a) and 3.24 ms (Pixel 10) warm p95 for a next-word call with retained context; [its card](https://huggingface.co/lukeslp/chime-sms-2m) has the full table.

## Training

Seven licensed English sources:

| Source | Text used | License |
| --- | --- | --- |
| [NUS SMS Corpus](https://doi.org/10.25540/WVM0-4RNX) | text messages people volunteered, mostly in Singapore, 2003 to 2015 | CC BY 4.0 |
| [Taskmaster-1](https://github.com/google-research-datasets/Taskmaster) | the user's side of written task dialogues | CC BY 4.0 |
| [Taskmaster-3](https://github.com/google-research-datasets/Taskmaster/tree/master/TM-3-2020) | the customer's side of written movie-ticket dialogues | CC BY 4.0 |
| [OpenAssistant 2](https://huggingface.co/datasets/OpenAssistant/oasst2) | English prompts written by people | Apache-2.0 |
| [SODA](https://huggingface.co/datasets/allenai/soda) | dialogue generated by GPT-3.5 | CC BY 4.0 |
| [Tatoeba](https://tatoeba.org) | English example sentences written by volunteers | CC BY 2.0 FR |
| [C4](https://huggingface.co/datasets/allenai/c4) | 1,000,000 sentences from public web pages | ODC-BY 1.0, plus the Common Crawl terms of use |

SODA is synthetic and makes up most of the text, so the NUS SMS, Taskmaster-1 and OpenAssistant lines were repeated eight times in each pass. Tatoeba, Taskmaster-3, SODA and C4 count once. Phone numbers, URLs, email addresses and credential-like strings were filtered out, and the text was deduplicated before it was split.

### How the web sentences were chosen

The pool is 10,847,777 sentences of 3 to 25 words from two of C4's 1,024 English training files, after the same line rules as the other sources and an exact dedupe against them and against every line I score on.

The filter follows data selection by importance resampling over hashed n-grams ([Xie et al., 2023](https://arxiv.org/abs/2302.03169)). It counts hashed word n-grams in two places, 41,832 of my own sent messages and the pool itself, and gives each pool sentence a weight for how much more its wording looks like the first than the second. It then draws 1,000,000 sentences by those weights. A control run with 1,000,000 sentences drawn at random from the same pool gained almost nothing on my messages, so the choice matters and web text alone does not.

What crosses from my messages into the model is only which public sentences were drawn:

- The messages were read on my computer and are not published, uploaded or in the training text.
- The filter does not count words that C4 usually capitalizes, which keeps names in my messages from steering it.
- The 1,000,000 sentences are all public web text, written by the authors of those pages.

A list drawn this way still reflects how I write, in the way a reading list reflects a reader.

### Recipe

I trained it from scratch for 102,195 updates (2.4 passes) on one A100 in about two hours and twenty minutes, with AdamW at a learning rate of 0.002 (warmup, then cosine decay) and batches of 256 sentences of up to 64 tokens. I kept the checkpoint at step 98,000, which had the lowest development perplexity on the four original human-written sources, and quantized it to int8 with one scale per row.

| Quantity | Measured value |
| --- | --- |
| Training text before repetition | 10,199,307 lines |
| Weighted pass | 11,061,518 lines; 223,713,082 model target tokens |
| Completed run | 102,195 updates; 2.365 batcher passes; 529,090,678 target-token exposures |
| Selected checkpoint | Step 98,000; human-macro development perplexity 59.6815 |
| Final checkpoint | Step 102,195; human-macro development perplexity 59.9743 |
| Training loop | 8,346.10 seconds (2h 19m 6s) |
| Training computation | 6,961.15 seconds; 0.068116 seconds/update |
| Training-loop cost estimate | $5.80 at $2.50/hour; excludes setup and uploads; not a bill |
| Hardware / precision | One NVIDIA A100; PyTorch float32 |
| Batch / maximum sequence | 256 / 64 tokens |
| Optimizer | AdamW; weight decay 0.01; gradient norm clip 1.0 |
| Learning-rate schedule | 0.002 peak; 200-update warmup; cosine decay to 10% of peak |
| Seed | 20260925 |

The update count is the October run's, kept fixed so the files differ only in their text. With more text a pass is longer, so the same number of updates covers 2.4 passes. Human-written lines, counting the web sentences, form 31.305% of the weighted pass; SODA remains the majority. Repeated lines are not additional unique data.

## Limitations

- English only. Suggestions reflect older Singapore texting, task dialogue, assistant prompts, example sentences, synthetic dialogue and casual web text, and the fixed vocabulary will not learn new words or names.
- The web sentences were chosen to read like one person's texting. Another person's texting may gain less.
- It is worse than Chime 1.2 on task requests (Taskmaster-1, above).
- The vocabulary was learned from the corpus and includes a few offensive words. LocalType removes them before showing suggestions; do the same in your own application.
- One unseen set backs the result, and the check on newer messages was not run.
- Filtering removed text that looked sensitive, which does not prove the source text is anonymous. Web text is not checked for how it was written.

## License

The weights and vocabulary are CC BY-SA 4.0, and `inference.py` and `localtype_tokenizer.py` are Apache-2.0; see [LICENSE.md](LICENSE.md). [NOTICE.txt](NOTICE.txt) and [TATOEBA-CONTRIBUTORS.txt](TATOEBA-CONTRIBUTORS.txt) credit the training data, whose authors do not endorse this model. The int8 model's SHA-256 is `55795f14a9cb11ee419e8b33bec749199811fc6fbefa1efbb89bdd6e22c39b53`.

## Citation

```bibtex
@misc{steuber2026chime2,
  author = {Luke Steuber},
  title = {Chime 2: An On-Device English Keyboard Prediction Model},
  year = {2026},
  url = {https://huggingface.co/lukeslp/chime}
}
```

Also by me: [Drummer-540M](https://huggingface.co/lukeslp/drummer2-540m), a 542M-parameter language model trained from scratch.