mindtype-tagger-tiny-v1.6

MindType's own grammar and spelling correction model, trained from scratch by Mindscend for an on-device keyboard (iOS and Android). Text never leaves the phone.

• Tiny version for older phones: 12.7M parameters (half of mindtype-tagger-v1.6), encoder-only transformer (6 layers, d=256), own 16k byte-level BPE tokenizer.
• Predicts one edit label per word (KEEP / DELETE / REPLACE_x / APPEND_x / case) in a single pass, so it is fast, tiny, and cannot invent words or rewrite correct text.
• Core ML package (8-bit weights) for iOS 18+, and ONNX (int8) for Android and servers.

What changed: the v1.6 recipe (past-tense training pairs) on the tiny encoder, trained for 22,600 steps. Tense errors went from almost never fixed (F0.5 0.11) to mostly fixed (F0.5 0.85, 61.3% exact), at nearly the same accuracy as the full-size v1.6 for half the size.

Results (MindType evaluation suite, 4,356 held-out cases)

Settings: KEEP bias 2.0 and the plausibility guard used in the MindType apps.

Set	n	F0.5	Exact	False-positive rate	Lost protected spans
casual_clean	118	–	0.983	0.017	0.000
casual_noisy	109	0.909	0.752	0.000	0.000
coedit	579	0.470	0.107	0.000	0.000
formal_clean	1000	–	0.956	0.044	0.000
formal_noisy	1000	0.861	0.627	0.000	0.000
golden	50	0.914	0.820	0.077	0.000
preserve	1000	0.830	0.635	0.000	0.000
preserve_clean	500	–	1.000	0.000	0.000
overall	4356	0.788	0.693	0.029	0.000

False-positive rate on 2,000 held-out casual messages (never used for training): 1.60% (full-size v1.6: 1.65% on the same messages).

On the *_clean sets there is nothing to fix; the false-positive rate is what counts there.

Compared with fine-tuned small LLMs and earlier versions (same suite)

Model	Size	F0.5 ↑	Exact ↑	False positives ↓	Lost names/numbers ↓
Gemma 3 270M instruct, zero-shot	270M	0.122	0.2%	99.6%	3.9%
Gemma 3 270M + LoRA r32	270M	0.571	57.4%	23.1%	2.9%
SmolLM2 360M + LoRA r32	360M	0.663	63.1%	20.1%	0.3%
mindtype-tagger-v1.1 (keep bias 2, guard)	26M	0.748	68.3%	8.65%	0.0%
mindtype-tagger-v1.2 (keep bias 2, guard)	26M	0.800	72.4%	3.1%	0.0%
mindtype-tagger-v1.5 (keep bias 2, guard)	26M	0.799	71.2%	2.4%	0.0%
mindtype-tagger-v1.6 (keep bias 2, guard)	26M	0.802	71.3%	2.8%	0.0%
mindtype-tagger-tiny-v1.6 (keep bias 2, guard)	13M	0.788	69.3%	2.9%	0.0%

Error types (separate test sets, not used for training)

Error type	n	full-size v1.6 F0.5	tiny v1.6 F0.5	tiny v1.6 exact
Real misspellings ("recieve")	500	0.957	0.958	91.6%
Dropped words ("I going home")	500	0.683	0.638	32.2%
Present tense about the past ("Yesterday I go")	248	0.885	0.850	61.3%

Dropped words are now the weakest; they're next.

Files

File	What it is
model.safetensors	PyTorch weights (fp32): encoder + per-token classification head
config.json	architecture (layers, width, heads, vocabulary and label counts)
tokenizer.json	16k byte-level BPE tokenizer (Hugging Face tokenizers format)
labels.json	the 15,000 edit labels, index → label
plausible_pairs.json	the word pairs the plausibility guard accepts
coreml/MindTypeTagger.mlpackage	Core ML model (8-bit weights, 13 MB), inputs padded to 16/32/64/96/160 tokens; 99.92% label agreement with fp32
onnx/mindtype_tagger.onnx	ONNX model (opset 17, int8 dynamic quantization, 13 MB), inputs ids, mask (int64, [1, T]), output logits; 98.8% label agreement with fp32
metrics.json, log.jsonl, run.json	evaluation results and training logs

Training

1. Pretraining: masked language modeling on FineWeb-Edu (ODC-BY).
2. Correction: 12.02M (noisy, clean) pairs (22,600 steps at batch 256): synthetic phone-typing errors, CoEdIT GEC (Apache-2.0), and casual messages written by a local open-weight model (Qwen2.5-14B-Instruct) and filtered for correctness, plus real misspellings, dropped words and tense errors (1.2% of pairs: past-tense sentences with a tense error). 30% are already-correct text, so the model learns to leave good text alone.

Hardware: one RTX 3080 Ti laptop GPU; the correction step took about 35 minutes.

How it works

Text is split into words and punctuation; each word gets a label from labels.json: $KEEP, $DELETE, $REPLACE_<word>, $APPEND_<word> (insert after), or a case change. The corrected sentence is the labels applied to the words, and the model is run again (up to 3 passes) on its own output. One sentence takes a few milliseconds on a phone CPU (Core ML) and on a laptop CPU (ONNX Runtime).

Inference settings used for the results: KEEP bias 2.0 (added to the $KEEP logit; higher = fewer, surer fixes). Names, numbers, emails and URLs are never changed.

Limitations

English only, sentence-level, experimental. Fixes what's there; it doesn't reword or change tone. About 40% of tense errors and two thirds of dropped words are still missed. It can still change correct text (2.9% of correct sentences in our suite, 1.6% of real casual messages).

License

Apache-2.0 (weights, tokenizer, Core ML and ONNX models). Training data: FineWeb-Edu (ODC-BY), CoEdIT (Apache-2.0).