v6: modified Kneser-Ney + per-word extension rule (held-out keystrokes saved 22.2% -> 23.0%, precision 90.2% -> 90.8%)
3c53a76 verified | license: mit | |
| language: | |
| - en | |
| library_name: transformers | |
| pipeline_tag: text-generation | |
| tags: | |
| - llama | |
| - compiled | |
| - no-training | |
| - autocomplete | |
| - n-gram | |
| - code | |
| - browser | |
| datasets: | |
| - allenai/WildChat-1M | |
| - OpenAssistant/oasst2 | |
| - google-research-datasets/mbpp | |
| - neulab/conala | |
| widget: | |
| - text: You can use the | |
| # ghost-typer | |
| **Finishes your sentences. Never takes the byline.** | |
| Ghost-text autocomplete for prompts to a coding assistant. It's **compiled, not trained**: no | |
| gradient descent, no pretrained weights, no other model. Every number comes from a closed-form | |
| rule in [`build_weights.py`](build_weights.py) applied to ~126k lines of human-typed text | |
| ([sources](data/SOURCES.md)). | |
| It only speaks when it's sure. A suggestion appears only above a confidence threshold whose | |
| precision is measured on held-out prompts. | |
| - **Live demo:** https://tmickleydoyle.github.io/ghost-typer/ (runs entirely in your tab) | |
| - **Code:** https://github.com/tmickleydoyle/ghost-typer | |
| - **Transformers model (this repo):** https://huggingface.co/tmickleydoyle/ghost-typer | |
| - **Browser model:** https://huggingface.co/tmickleydoyle/ghost-typer-web | |
| ## Variants | |
| | Repo / file | Precision | Size | What it is | | |
| |---|---|---|---| | |
| | **this repo:** `model.safetensors` | **float32** (canonical) | 1.60 GB | the compiled `LlamaForCausalLM`; loads with `transformers` | | |
| | [`ghost-typer-web`](https://huggingface.co/tmickleydoyle/ghost-typer-web): `model.bin` | **uint8** (recommended for apps) | 10.7 MB | browser build; zero-dependency TypeScript runtime | | |
| | [`ghost-typer-web`](https://huggingface.co/tmickleydoyle/ghost-typer-web): `model-f32.bin` | float32 | 16.8 MB | unquantized reference of the browser build | | |
| - **Two compiles, one model family:** the browser build is a sibling compile, not a | |
| quantization of `model.safetensors`. It uses a bigger vocabulary (30,893 vs 6,000 words) | |
| because words cost almost nothing in its sparse format. | |
| - **Quantization cost:** within the browser build, uint8 vs float32 costs 0.14 points of | |
| keystrokes saved (23.00% vs 23.14%) at the same precision (90.8% vs 90.7%). | |
| - **No half-precision transformers model:** its logits are scaled by 1e-5 to keep the network | |
| exact, which float16/bf16 would break. | |
| | | Browser model (`ghost-typer-web/model.bin`) | Transformers model (`model.safetensors`) | | |
| |---|---|---| | |
| | Use it for | apps: on-device, no server | `transformers`, `pipeline`, research | | |
| | Size | **10.7 MB** (uint8-quantized) | 1.60 GB, 400M params (97% zeros) | | |
| | Vocabulary / contexts | **30,893 words** / 512,533 contexts | 6,000 words / 12,607 contexts | | |
| | Speed | **2.5 ms median, 4.3 ms p95 per keystroke** | 65 ms per next token (PyTorch CPU) | | |
| Both are the same kind of network. The browser model stores only the non-zero parts of the | |
| compiled transformer; see [how it works](#how-it-works). | |
| ## What it suggests | |
| Output of the browser model (uint8) with the default policy. Gray is the ghost text; | |
| *(quiet)* means it suggests nothing. | |
| | Typed | Ghost text | Confidence | | |
| |---|---|---| | |
| | `write a python fun` | `ction to` | 94% | | |
| | `write a react comp` | `onent` | 90% | | |
| | `how do i conv` | `ert` | 99% | | |
| | `i'm getting an err` | `or` | 99% | | |
| | `can you explain the differ` | `ence between` | 90% | | |
| | `how to install` | *(quiet)* | | | |
| ## How often it's right | |
| The model is compiled **without** 10% of the prompts. Each held-out prompt is then typed one | |
| character at a time, and after every keystroke the model either shows ghost text or stays | |
| quiet. This is measured with `python scripts/benchmark.py` on the shipped quantized file. | |
| - **Precision:** the share of shown suggestions that exactly match what the user typed next. | |
| - **Coverage:** the share of keystrokes where something is shown. | |
| - **Keystrokes saved:** the share saved by accepting every correct suggestion with Tab. | |
| | First-word threshold τ | Precision | Coverage | Keystrokes saved | Real chat prompts only (WildChat) | | |
| |---|---|---|---|---| | |
| | **0.62 (default)** | **90.8%** | 39.2% | **23.0%** | 90.0% precision, 22.2% saved | | |
| | 0.80 | 94.9% | 30.5% | 18.3% | 94.5%, 17.8% | | |
| | 0.90 (`strict`) | 96.7% | 24.4% | 15.1% | 96.4%, 14.7% | | |
| | 0.95 | 97.4% | 19.0% | 12.1% | 97.1%, 11.9% | | |
| Every row extends word by word while each next word is ≥ 80% likely (`ext` = 0.8). The | |
| previous build (Kneser-Ney, extend while the combined probability ≥ 0.71) scored 90.2% | |
| precision and 22.2% saved. Held-out sets: 1,000 WildChat prompts plus 711 OASST, MBPP and | |
| CoNaLa prompts. Both thresholds are runtime settings, so an app can trade coverage for | |
| precision without recompiling. | |
| ## Use it in your app | |
| `web/autocomplete.ts` and `tokenizer.ts` are zero-dependency TypeScript with no DOM access. They | |
| run on the main thread, in a Web Worker, or in Bun/Node. A single bundled `ghost-typer.js` is in | |
| [`ghost-typer-web`](https://huggingface.co/tmickleydoyle/ghost-typer-web). | |
| ```ts | |
| import { CompiledAutocomplete } from "./web/autocomplete.ts"; | |
| const ac = await CompiledAutocomplete.load( | |
| "https://huggingface.co/tmickleydoyle/ghost-typer-web/resolve/main/model.bin", | |
| ); | |
| input.addEventListener("input", () => { | |
| const { ghost, prob } = ac.suggest(input.value); // "" when not confident | |
| showGhostText(ghost); // Tab -> input.value += ghost | |
| }); | |
| ac.suggest(text, { ...ac.policy, tau: 0.9 }); // stricter: ~97% precision | |
| ``` | |
| Run the demo locally. It's a static page: | |
| ```bash | |
| git clone https://github.com/tmickleydoyle/ghost-typer && cd ghost-typer | |
| python3 -m http.server 8080 -d web # open http://localhost:8080 | |
| bun web/cli.ts "write a python fun" # or from the terminal | |
| ``` | |
| ## Use it with transformers | |
| ```python | |
| from transformers import pipeline | |
| gen = pipeline("text-generation", model="tmickleydoyle/ghost-typer") | |
| gen("You can use the", do_sample=False)[0]["generated_text"] # 'You can use the following code:' | |
| ``` | |
| `model.safetensors` is a plain `LlamaForCausalLM` (no `trust_remote_code`). Its logits are the | |
| compiled n-gram's log-probabilities, matching the reference to within 2e-5 nats. On | |
| transformers 4.x, also pass `add_special_tokens=True`; 4.x pipelines don't prepend BOS by | |
| default. For confidence-gated ghost text in Python, use `suggest.py`. | |
| ## Performance: compile, don't train | |
| Measured on a 10-core Apple Silicon laptop (32 GB RAM), CPU only. Raw numbers are in | |
| [`docs/benchmarks.json`](docs/benchmarks.json). | |
| | | | | |
| |---|---| | |
| | Compile the browser model (126,347 lines) | **21 s** | | |
| | Compile the 400M-parameter transformers model | **14 s** (peak memory 3.6 GB) | | |
| | GPU hours / compute cost | **0 / $0** | | |
| | Reproducible | byte-identical rebuilds (both artifacts) | | |
| | Browser model in Chrome | about 360 ms to load (local); a few ms per keystroke | | |
| ## How it works | |
| - **The network:** an exact transformer implementation of an **interpolated modified | |
| Kneser-Ney 5-gram model** (Chen & Goodman discounts per order, as in KenLM). It predicts the next token from the current token plus the 3 before it, | |
| backing off to shorter contexts. | |
| - **Attention:** 3 heads with constant queries and keys. Their RoPE phases make each head copy | |
| the token exactly 1, 2 or 3 positions back into its own slot. | |
| - **MLP:** one neuron per kept context. It fires only when all its tokens match and adds | |
| `log p(next | context) − log p(next | shorter context)` to the logits. These additions | |
| telescope to the longest known context's log-probability. | |
| - **Browser model:** stores each neuron's context and logit changes. The changes are always | |
| ≥ 0, so they're stored as uint8 with one scale per context length. That costs 0.14 points | |
| of keystrokes saved on held-out prompts. The runtime binary-searches the sorted contexts, evaluating only | |
| the neurons that match. | |
| - **Confidence policy** (`suggest.py`, mirrored by `web/autocomplete.ts`): | |
| - **Partial words:** completes the word being typed, among vocabulary words with that | |
| prefix. Its probability is scaled by the chance that a word with this prefix is in the | |
| vocabulary at all. | |
| - **When to show:** only if that probability is ≥ τ (0.62). | |
| - **Extension:** grows word by word while each next word is ≥ `ext` (0.8) likely on its | |
| own. This beat the old rule (combined probability ≥ τ) by 0.9 points of keystrokes saved. | |
| Forcing 2 or 3 words cut keystrokes saved to 6% and 2%. | |
| - **Settings were measured, not hand-tuned:** `scripts/sweep.py` compiled each candidate and | |
| kept the most keystrokes saved at ≥ 90% precision. It covered source weights, vocabulary | |
| size, pruning and context length. | |
| - **Longer contexts** (5–8 tokens) never beat 4 tokens: at best a tie, with long contexts | |
| pruned harder ([experiments](docs/experiments_lookback.json)). | |
| - **A "topic" attention head** didn't help either (`scripts/experiment_topic.py`). | |
| Details, formulas and the proof of exactness are in [docs/how-it-works.md](docs/how-it-works.md). | |
| ## Limitations | |
| - **Short memory and a lookup:** it sees 4 tokens and repeats frequent patterns. It doesn't | |
| understand code, and it doesn't write code (code was removed from the data). | |
| - **Narrow domain:** it's strongest on how people phrase coding requests, and quiet on most | |
| other text. | |
| - **Lowercase:** input is lowercased, so suggestions are lowercase. | |
| - **Data:** WildChat prompts are what people typed into ChatGPT, and some users pasted text | |
| into them. Sentences containing anything that looks like a credential (API keys, tokens, | |
| private keys) or a long random string are dropped, and a test enforces it. ~1,100 everyday | |
| phrases in `phrases/` were written by an AI coding assistant (Claude). See | |
| [data/SOURCES.md](data/SOURCES.md). | |
| ## Rebuild and verify | |
| ```bash | |
| pip install -r requirements.txt | |
| python scripts/fetch_coding_prompts.py # downloads data/*.txt (WildChat: ~3.4 GB, cached) | |
| python build_weights.py # transformers model ("hf" in compile_settings.json) | |
| python export_web.py # browser model ("web" + "policy" + "export") | |
| python scripts/stage_hf_web.py # ghost-typer-web repo folder (uint8 + float32) -> dist/hf-web | |
| bun build web/demo.ts --outfile web/demo.js --target browser | |
| python test_model.py # HF loading, exactness, browser runtime == Python == PyTorch | |
| python scripts/sweep.py # settings comparison -> docs/sweep.json | |
| python scripts/benchmark.py # -> docs/benchmarks.json | |
| ``` | |
| Tested with transformers 4.52 and 5.22, Bun 1.3, Node 24 and Chrome. | |
| ## Licenses | |
| The model, code and hand-written phrases are MIT. The files in `data/` contain excerpts from | |
| WildChat-1M (ODC-BY), OpenAssistant oasst2 (Apache-2.0), MBPP (CC-BY-4.0) and CoNaLa (MIT); see | |
| [data/SOURCES.md](data/SOURCES.md) for attribution. | |
| ## Files | |
| | File | Purpose | | |
| |---|---| | |
| | `web/index.html`, `web/demo.ts`, `web/demo.js` | the demo page | | |
| | `web/autocomplete.ts`, `tokenizer.ts`, `web/cli.ts` | browser runtime and CLI (`web/model.bin` is on GitHub and in `ghost-typer-web`) | | |
| | `config.json`, `generation_config.json`, `model.safetensors`, `tokenizer*.json` | transformers model (`model.safetensors` is on Hugging Face only) | | |
| | `build_weights.py`, `export_web.py`, `compile_settings.json`, `suggest.py` | compiler, settings and confidence policy | | |
| | `phrases/`, `data/` | source text and attribution | | |
| | `scripts/`, `docs/` | data download, settings comparison, benchmarks, experiments, write-up | | |
| | `test_model.py` | verification suite | | |
| | `model.ts`, `autocomplete.ts` | dense-model TypeScript runtime and CLI | | |