Blame for README.md

---
library_name: onnx
pipeline_tag: text-generation
datasets:
  - corbt/enron-emails
  - HuggingFaceFW/fineweb-edu
  - HuggingFaceTB/smoltalk
language:
  - en
tags:
  - autocomplete
  - gru
  - transformer
  - fill-in-the-middle
  - onnxruntime-web
  - in-browser
  - offline-page
model-index:
- name: tinynext-v2
  results:
  - task:
      type: text-generation
      name: Autocomplete (typing saved at a 50% cutoff)
    dataset:
      type: wikimedia/wikipedia
      name: Wikipedia articles (held out)
    metrics:
    - type: typing_saved
      name: Typing saved (%)
      value: 4.0
    - type: suggestion_precision
      name: Suggestions that were right (%)
      value: 76
  - task:
      type: text-generation
      name: Autocomplete (typing saved at a 50% cutoff)
    dataset:
      type: OpenAssistant/oasst1
      name: OpenAssistant messages, validation split
    metrics:
    - type: typing_saved
      name: Typing saved (%)
      value: 7.3
    - type: suggestion_precision
      name: Suggestions that were right (%)
      value: 78
  - task:
      type: text-generation
      name: Autocomplete (typing saved at a 50% cutoff)
    dataset:
      type: corbt/enron-emails
      name: Enron emails from held-out senders
    metrics:
    - type: typing_saved
      name: Typing saved (%)
      value: 4.3
    - type: suggestion_precision
      name: Suggestions that were right (%)
      value: 79
---

<p align="center">
  <img src="banner.png" alt="tinynext: suggests the next few words, Tab to take them" width="880">
</p>

<p align="center">
  <img style="display: inline-block; vertical-align: middle; margin: 2px;" alt="params: 944K" src="https://img.shields.io/badge/params-944K-6e6e73?style=flat-square&labelColor=1d1d1f"> <img style="display: inline-block; vertical-align: middle; margin: 2px;" alt="size: 0.98 MB" src="https://img.shields.io/badge/size-0.98%20MB-6e6e73?style=flat-square&labelColor=1d1d1f"> <img style="display: inline-block; vertical-align: middle; margin: 2px;" alt="speed: ~4 ms" src="https://img.shields.io/badge/speed-~4%20ms-6e6e73?style=flat-square&labelColor=1d1d1f"> <img style="display: inline-block; vertical-align: middle; margin: 2px;" alt="trained: $0.35" src="https://img.shields.io/badge/trained-$0.35-6e6e73?style=flat-square&labelColor=1d1d1f"> <img style="display: inline-block; vertical-align: middle; margin: 2px;" alt="data: Enron email" src="https://img.shields.io/badge/data-Enron%20email-6e6e73?style=flat-square&labelColor=1d1d1f">
</p>

<h3 align="center">A tiny email autocomplete that runs in your browser.<br>It learned to write from Enron's emails.</h3>

<p align="center">
  <a href="https://huggingface.co/spaces/cfahlgren1/enron-autocomplete"><b>Try the demo&nbsp;→</b></a>
  &nbsp;&nbsp;·&nbsp;&nbsp;
  <a href="https://github.com/cfahlgren1/offline-page"><b>Write with it in offline.page&nbsp;→</b></a>
</p>

<br>

tinynext is the autocomplete in [offline.page](https://github.com/cfahlgren1/offline-page). It's the same model as [`cfahlgren1/enron-autocomplete`](https://huggingface.co/cfahlgren1/enron-autocomplete), copied here under its new name.

## v2: the next few words, with context

<sub>In [`v2/`](https://huggingface.co/cfahlgren1/tinynext/tree/main/v2). The files at the root of this repo are still v1.</sub>

v1 guesses one letter at a time, so its suggestions are short word endings. v2 is a **12.5M-parameter transformer that predicts word pieces**, so one step can be a whole word, and it reads up to about 1,500 characters before the cursor (and, when there is any, text after it). It runs in a Web Worker in offline.page: add `?autocomplete=v2` to try it.

**How much typing it saves.** Every model is scored by [`v2/eval.py`](v2/eval.py) on the same held-out text: type it character by character, pause every 12 characters, and count a suggestion only if the text really continues that way. Same decoding rule and 50% cutoff as the app, suggestions of 4+ characters.

| Writing | v1 | v2 | SmolLM2-135M (ceiling, ~100 MB) |
|---|---|---|---|
| Articles (Wikipedia) | 1.5% | **4.0%** | 8.1% |
| Messages (OpenAssistant) | 2.0% | **7.3%** | 16.8% |
| Email (Enron, unseen senders) | 3.0% | **4.3%** | 5.7% |

| Writing (v2) | Suggests at | Right | 2+ words suggested at | 2+ words right |
|---|---|---|---|---|
| Articles (Wikipedia) | 11% | 76% | 3% | 74% |
| Messages (OpenAssistant) | 16% | 78% | 8% | 72% |
| Email (Enron, unseen senders) | 11% | 79% | 3% | 60% |

**In the browser.**

| | |
|---|---|
| Download | **13.3 MB**: int8 weights with one scale per 64 values, expanded to float32 when the page loads |
| Matches full precision | same next word 99% of the time, same 5-word path 92%, typing saved within 0.2 points |
| Speed | **4 ms** per suggestion (14 ms p95) in a Web Worker, Chrome on an M3 Max; typing stays under one frame |

The browser computes in float32 on purpose: onnxruntime-web's int8 kernels were 8–11× slower than its float kernels for this model, and int8 activations changed the top word too often. Only the download is compressed.

**Training.** 1.5B tokens (60% [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu), 30% [SmolTalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk), 10% Enron email from training senders), half of it as fill-in-the-middle examples. 28 minutes on one A100 (about $1.50) plus 20 minutes of data prep on a CPU job (about $0.34). Validation loss 2.99. [Training curves on Trackio](https://huggingface.co/spaces/cfahlgren1/tinynext-trackio).

**Limits.**

- Text after the cursor doesn't help yet: when there is some, v2 does slightly worse than with the text before alone. Experiments are under way.
- It still leans toward office email, and it only sees keyboard characters.
- It's English only.

<details>
<summary><b>v2 files</b></summary>

| File | What it is |
|---|---|
| `v2/web/graph.onnx`, `weights.q8`, `weights.json` | The browser model: graph, int8 weights and how to expand them |
| `v2/web/tokenizer.json`, `config.json` | 8K byte-level BPE tokenizer and model shape |
| `v2/model.safetensors` | Float32 weights |
| `v2/train.py` | Data prep, training, eval, export and upload in one script |
| `v2/eval.py`, `v2/eval_set.json` | The eval harness and the fixed text it scores |
| `v2/export.py`, `v2/compress.py` | Re-export to ONNX, and the int8 download format |
| `v2/metrics/*.json` | Eval results for v1, v2 and SmolLM2-135M |
| `v2/report.json` | Training settings, loss history and export checks |

</details>

<details>
<summary><b>Train it yourself</b></summary>

```bash
# Prepare data once on a CPU job, then train on a GPU from the prepared files.
hf jobs uv run --flavor cpu-xl --secrets HF_TOKEN train.py --stage prepare
hf jobs uv run --flavor a100-large --secrets HF_TOKEN train.py --stage train
```

</details>

## v1: the original character model

### What it does

It finishes the word or short phrase you're typing. Press <kbd>Tab</kbd> to take the suggestion.

| You type | It suggests |
|---|---|
| Please accept my resig | **nation** |
| Please accept my resignation. I will be o | **ut of the office** |
| The books are fine. Please call me i | **f you have any** |
| Enron N | **orth America Corp.** |

### How sure it has to be

The model knows how sure it is of each guess. You pick a cutoff, and it only suggests when it's at least that sure. A higher cutoff means fewer suggestions, and more of them right.

<table>
  <tr>
    <td align="center" width="33%"><h2>80%</h2>of suggestions are right<br><sub>at a 50% cutoff</sub></td>
    <td align="center" width="33%"><h2>11%</h2>of the time it suggests<br><sub>at a 50% cutoff</sub></td>
    <td align="center" width="33%"><h2>~6</h2>characters per right suggestion</td>
  </tr>
</table>

| Cutoff | How often it suggests | How often it's right |
|---|---|---|
| 20% | 30% of the time | 48% |
| 35% | 18% of the time | 66% |
| **50%** *(offline.page)* | **11% of the time** | **80%** |
| 70% | 7% of the time | 90% |

The demo uses 40%, and offline.page uses 50%.

These numbers come from 5,000 places in emails from people the model never saw in training. “Right” means the suggestion exactly matched what the person typed next.

#### Limits

- It only knows office email from 2001. Outside that it has little to say.
- With no cutoff it rambles. “Dear Santa,” becomes “I am attaching a copy of the agreement to the Enron Corp.”
- It learned real names and sign-offs from the emails, so keep suggestions short.
- English only.

### Use it

The model takes one character and its memory of the text so far. It returns its guess for the next character and an updated memory. So each key press is one small step.

<details>
<summary><b>JavaScript example</b></summary>

```js
import * as ort from "onnxruntime-web";

const session = await ort.InferenceSession.create("model_quint8.onnx", {
  executionProviders: ["wasm"],
});
let state = new ort.Tensor("float32", new Float32Array(512), [1, 512]);

async function step(id) {
  const input_id = new ort.Tensor("int64", BigInt64Array.of(BigInt(id)), [1]);
  const { logits, next_state } = await session.run({ input_id, state });
  state = next_state;
  return logits.data;
}

let logits = await step(1); // 1 starts a new email
for (const char of "Let me kn") logits = await step(char.charCodeAt(0) - 29);
```

A character's id is its character code minus 29. A new line is 2 and the end of an email is 1. Only English keyboard characters are supported.

To build a suggestion, keep taking the most likely next character and multiply the probabilities together as you go. Stop at a new line, or when that running product drops below your cutoff. Show the text up to the last finished word.

</details>

<details>
<summary><b>Files</b></summary>

| File | What it is |
|---|---|
| `model_quint8.onnx` | The model to use. 0.98 MB. |
| `model.onnx` | The same model before shrinking. 3.8 MB. |
| `best.pt` | The PyTorch checkpoint. |
| `vocab.json` | The character ids. |
| `report.json` | The full training and test results. |
| `train.py` | The training script. |

I also tried a bigger version with 3.46M parameters and 3.5 MB. It wasn't better. When both models suggest equally often, 9% of the time, the small one is right about 85% of the time and the big one 82%. The big one got the same training settings and may need different ones to do well.

</details>

### Training

It read 40 million characters from 119,859 emails sent from enron.com addresses, 30 times over. Typos were added to what it read, so it keeps predicting when you make one. The run took 14 minutes on one A10G with Hugging Face Jobs and cost 35 cents.

[`train.py`](train.py) is the whole thing: it downloads the emails, trains, tests and exports the model. To train your own:

```bash
hf jobs uv run --flavor a10g-large --secrets HF_TOKEN train.py --repo your-name/your-model
```

<br>

<p align="center">
  <sub>tinynext is one of two tiny models behind <a href="https://github.com/cfahlgren1/offline-page">offline.page</a>, a writing app that works in airplane mode.<br>Its partner, <a href="https://huggingface.co/cfahlgren1/tinyfix">tinyfix</a>, fixes typos from context.</sub>
</p>