v6: modified Kneser-Ney + per-word extension rule (held-out keystrokes saved 22.2% -> 23.0%, precision 90.2% -> 90.8%)
c90e10f verified |
Download README.md from tmickleydoyle/ghost-typer-web: direct link, hf CLI and curl.
- Browser
- Download file 5.04 kB
-
https://huggingface.co/tmickleydoyle/ghost-typer-web/resolve/main/README.md
- Command line
-
hf download hf://tmickleydoyle/ghost-typer-web/README.md
-
curl -L -o README.md https://huggingface.co/tmickleydoyle/ghost-typer-web/resolve/main/README.md
5.04 kB
| license: mit | |
| language: | |
| - en | |
| base_model: tmickleydoyle/ghost-typer | |
| pipeline_tag: text-generation | |
| library_name: ghost-typer | |
| tags: | |
| - browser | |
| - quantized | |
| - uint8 | |
| - typescript | |
| - autocomplete | |
| - compiled | |
| - no-training | |
| - n-gram | |
| datasets: | |
| - allenai/WildChat-1M | |
| - OpenAssistant/oasst2 | |
| - google-research-datasets/mbpp | |
| - neulab/conala | |
| # ghost-typer-web | |
| **Finishes your sentences. Never takes the byline.** | |
| The browser build of [ghost-typer](https://huggingface.co/tmickleydoyle/ghost-typer): compiled | |
| (not trained) ghost-text autocomplete for coding prompts. It runs entirely on-device with a | |
| zero-dependency TypeScript runtime. **Demo:** https://tmickleydoyle.github.io/ghost-typer/ | |
| ## Files | |
| | File | Precision | Size | Held-out precision | Keystrokes saved | Use it for | | |
| |---|---|---|---|---|---| | |
| | **`model.bin`** | **uint8** logit changes, one scale per context length | **10.7 MB** | 90.8% | 23.0% | **apps (recommended)** | | |
| | `model-f32.bin` | float32 (unquantized) | 16.8 MB | 90.7% | 23.1% | reference, exactness checks | | |
| Both files hold the same compile: 30,893 words and 512,533 contexts. Quantization costs 0.14 | |
| points of keystrokes saved (23.00% vs 23.14%) and leaves precision unchanged (90.79% vs | |
| 90.73%), measured by typing 1,711 held-out prompts one character at a time. The runtime reads | |
| either file. | |
| **Update (v6):** modified Kneser-Ney smoothing and a per-word extension rule. Previous build: | |
| 90.2% precision, 22.2% keystrokes saved, same size. `model.bin` now carries `policy.ext`, which | |
| only the current runtime (`ghost-typer.js`, `src/`) reads. Update the runtime together with the | |
| model: an older runtime ignores `ext` and its suggestions run below 90% precision. | |
| The runtime itself is `ghost-typer.js` (bundled ESM, ~5 KB) with sources in `src/`. A demo page | |
| (`index.html`, `demo.js`) loads `./model.bin`. | |
| ## Use it | |
| Copy `ghost-typer.js`, or `src/` for TypeScript, into your app. Load the model from this repo: | |
| ```ts | |
| import { CompiledAutocomplete } from "./ghost-typer.js"; | |
| const ac = await CompiledAutocomplete.load( | |
| "https://huggingface.co/tmickleydoyle/ghost-typer-web/resolve/main/model.bin", | |
| ); | |
| input.addEventListener("input", () => { | |
| const { ghost, prob } = ac.suggest(input.value); // "" when not confident | |
| showGhostText(ghost); // Tab -> input.value += ghost | |
| }); | |
| ``` | |
| The default policy shows the word being typed if it is ≥ 62% likely (τ = 0.62), then extends | |
| word by word while each next word is ≥ 80% likely on its own (`ext` = 0.8). | |
| `ac.suggest(text, { ...ac.policy, tau: 0.9 })` gives a stricter threshold (96.7% precision, | |
| 15.1% keystrokes saved). | |
| Per keystroke it takes 2.5 ms median and 4.3 ms p95 (Bun). Loading the uint8 file takes about | |
| 360 ms in Chrome. | |
| To try the demo locally: `git clone https://huggingface.co/tmickleydoyle/ghost-typer-web`, then | |
| `python3 -m http.server -d ghost-typer-web 8080`. | |
| ## How this relates to the transformers model | |
| The [main repo](https://huggingface.co/tmickleydoyle/ghost-typer) holds the float32 | |
| `LlamaForCausalLM` (`model.safetensors`, 400M parameters, 97% zeros) and the compiler. Both | |
| artifacts are compiled from the same text by the same closed-form rules: an interpolated | |
| modified Kneser-Ney 5-gram, with one MLP neuron per context. | |
| - **What's stored:** the browser model keeps only the non-zero parts, namely each neuron's | |
| context tokens and the logit changes it writes. | |
| - **Different settings:** it's compiled with browser settings (30,893-word vocabulary vs | |
| 6,000), because extra words cost almost nothing here. Every word would add a hidden | |
| dimension to the dense model. | |
| - **Not a quantization of `model.safetensors`:** it's a sibling build of the same model | |
| family. `model.bin` is a uint8 quantization of `model-f32.bin`. | |
| - **Exactness check:** the runtime reproduces PyTorch's next-token distribution within 1e-3 | |
| nats when given the dense model's own tables. That's a test in the source repo. | |
| ## Format (`GHST` v2) | |
| ``` | |
| "GHST" | u32 version (2) | u32 header bytes | header JSON | pad to 8 | |
| sections, each 8-byte aligned (offsets in the header): | |
| vocab utf-8 pieces joined by "\n" | |
| oov_keys, oov_vals prefix -> P(out of vocabulary | prefix), u8 / 255 | |
| base f32[V] always-on unigram neuron | |
| ctx{k} u16[n_k * k] context ids, oldest first, sorted (binary search) | |
| count{k} u16[n_k] logit changes per context | |
| dids{k} u16[pairs_k] next-token ids | |
| dvals{k} u8 | f32 [pairs_k] logit changes (u8: value * scale_k from the header) | |
| ``` | |
| ## Source, data and licenses | |
| - **Compiler, tests and benchmarks:** https://github.com/tmickleydoyle/ghost-typer (also | |
| mirrored in the main HF repo). | |
| - **License:** MIT. | |
| - **Training text:** human-typed prompts from WildChat-1M (ODC-BY), OpenAssistant oasst2 | |
| (Apache-2.0), MBPP (CC-BY-4.0) and CoNaLa (MIT), plus ~1,100 everyday phrases written by an | |
| AI coding assistant (Claude). Attribution is in `data/SOURCES.md` in the main repo. | |