dmottad's picture
Third version: new templates from independent authors; blind-test results in the card
5cf4e10 verified
|
Raw History Blame Contribute Delete
5.53 kB
---
license: apache-2.0
base_model: Qwen/Qwen3-0.6B-Base
library_name: mlx
pipeline_tag: text-generation
language:
- it
- en
- es
- fr
- de
- pt
tags:
- mlx
- proofreading
- style
- notes
- llm-wiki
---
# proofreader-0.6B-MLX-6bit
A small style proofreader for notes, built to run while you type. Given a
sentence or a paragraph, it points at the **wordy or clumsy phrase**, copied
verbatim from the text, and proposes a shorter one; when the text is fine it
says nothing. It answers with one JSON object and stops.
```
Text (English):
Due to the fact that the office is closed, we will send the documents on Monday.
{"suggestions": [{"type": "style", "original": "Due to the fact that", "replacement": "Because"}]}
```
It is the style checker of the [wordspace](https://getwordspace.com) notes
app, where spelling and grammar come from the operating system and the model
handles style only. It works in Italian, English, Spanish, French, German and
Portuguese.
Fine-tuned from [Qwen3-0.6B-Base](https://huggingface.co/Qwen/Qwen3-0.6B-Base)
with LoRA, fused, and quantized to 6 bits for MLX. See `NOTICE` for the full
list of changes.
## Use
```bash
pip install mlx-lm
hf download wordspace/proofreader-0.6B-MLX-6bit --local-dir proofreader
cd proofreader && python example.py
```
Build the two chat messages with `protocol.py` (the system prompt must stay
identical, character for character), apply the tokenizer's chat template,
and generate greedily. Read the first JSON object of the answer and keep the
`style` suggestions whose `original` appears in the text
(`protocol.style_suggestions`). The model stops at its end-of-text token,
after 18 tokens on average.
## Results
**Blind test.** 217 sentences (103 wordy, 114 clean) written for this test
by six independent authors, one per language, who saw neither the training
templates nor the earlier tests; a second reader relabelled everything and
cases they disagreed on were dropped. None shares eight consecutive words
with the training data. Compared on the same sentences, with the same prompt,
with gemma-3-4b-it (QAT, 4 bit), the model the app used for this task, and
with the previous version of this model.
| | this model | previous version | gemma-3-4b |
|---|---|---|---|
| wordy phrases found | **82/103** | 65/103 | 26/103 |
| correction judged good | **50/103** | 18/103 | 4/103 |
| · phrase in no training template (58) | **18/58** | 2/58 | 1/58 |
| clean sentences touched (the harm) | 8/114 | 5/114 | 18/114 |
"Judged good": the corrected sentence, read without knowing which model
wrote it, is grammatical, keeps the meaning and is tighter. Two separate
LLM judges, good only if both agree (they agreed on 172 of 173). Against the
previous version: found p = 0.002, judged good p < 0.001, harm p = 0.51
(exact McNemar on the same sentences); against gemma all p < 0.001 except
harm, p = 0.052. Wordy phrases found, by language: German 13/16, English
15/16, Spanish 15/17, French 14/19, Italian 14/18, Portuguese 11/17.
**Hand-written probe.** 192 sentences, 32 per language, by the authors of
the training templates: 60 with a phrase the templates know, in new
sentences; 36 with phrases the original templates did not contain; 96 clean.
| | this model | gemma-3-4b | p |
|---|---|---|---|
| wordy phrases found | **88/96** | 27/96 | < 0.001 |
| · known phrases, new sentences | 59/60 | 18/60 | < 0.001 |
| · other phrases | **29/36** | 9/36 | < 0.001 |
| clean sentences touched (the harm) | **2/96** | 8/96 | 0.070 |
| seconds per sentence (median, Apple silicon) | **0.15** | 0.64 | — |
On the eight wordy and eight clean sentences of the app's own test: 7/8
found against 3/8, no clean sentence touched by either. The 6-bit model gives the same answers
as the full-precision one within noise (blind test: 82 vs 82 found, 8 vs 8
touched; probe: 88 vs 88 found, 2 vs 3 touched).
## Limits
- Style only: wordy phrases, nominal style ("carry out an inspection of"),
redundancies ("repeat again"). It is not trained to fix spelling or
grammar and should leave them alone.
- On phrases outside what it has learned, it proposes a good correction
for fewer than one sentence in three (18/58 in the blind test). Treat
suggestions as suggestions.
- On held-out validation notes it touches 9% of the clean ones (15/173):
more in clean twins of wordy sentences (16%) and above all in **German
(7 of 29 clean German cases)**, against 0 of 35 in English. German is the
weakest language.
- Trained on short notes: sentences and paragraphs of a few sentences.
## Training
24,000 synthetic cases generated from hand-written templates in six
languages, about 185 wordy and 130 clean templates per language, most of
them written by authors who had not seen the others (so that the model
learns the task, not one writer's habits): wordy sentences with their defects marked, the same sentences
already concise (minimal pairs, so the model learns the phrasing and not
the topic), clean notes, and paragraphs mixing them; almost half have
nothing to correct. One template in five was kept for validation only. LoRA
rank 16 on all layers, AdamW, learning rate 1e-4, gradient clipping at norm
1, loss on the answer tokens only, every answer closed by the end-of-text
token so that the model stops. The checkpoint was chosen by the task on
validation (wordy phrases found and clean sentences left alone), not by the
loss. Any generated case sharing eight consecutive words with a test
sentence was discarded. No personal notes were used.