| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3-0.6B |
| base_model_relation: finetune |
| pipeline_tag: text-generation |
| language: |
| - en |
| tags: |
| - coreml |
| - apple-neural-engine |
| - on-device |
| - qwen3 |
| - social-reply |
| library_name: coreml |
| --- |
| |
| # short-reply-0.6b-coreml |
|
|
| A sub-1B model that reads one social post and drafts one short reply (≤ 12 words), packaged for Core ML. |
| Qwen3-0.6B, LoRA-tuned by distilling a 1.7B reply model on 4,615 generated posts, then merged. One 724 MB |
| package with two functions over shared weights: |
|
|
| | Function | Shape | Runs on | p50 (M5 Pro, macOS 27, from Swift) | |
| |---|---|---|---:| |
| | `prefill` | 160 tokens, fixed | Neural Engine (1,920 / 1,921 ops) | 28 ms | |
| | `decode` | 1 token, 512-slot KV cache in `MLState` | GPU | 24 ms / token | |
|
|
| A reply takes about 190 ms. Weights are 8-bit palettized (k-means, per-grouped-channel, group 32); the shared |
| embedding / `lm_head` weight stays fp16 (a palettized embedding table costs +13 ms per token on the gather). |
| Linear int8 weights run on the ANE but return garbage there; palettized weights work. |
|
|
| ## Quality |
|
|
| On 100 held-out real posts, 89 replies are identical to the PyTorch checkpoint; a blind LLM-judge A/B against the |
| project's 1.7B model scores 88 vs 93 (all of relevant / tone / grounded / ≤ 12 words), the same as the PyTorch |
| 0.6B. About 1 in 8 drafts is generic or slightly off. Replies are drafts for a person to review, not for posting |
| automatically. Training used only generated and hand-written posts; no real social posts were used for training. |
|
|
| ## Files |
|
|
| - `short_reply_0_6b.mlpackage` — multifunction Core ML package (`prefill`, `decode`), token-id inputs (embedding |
| lookup and RoPE tables are inside the graph). |
| - `tokenizer.json` — Qwen3 byte-level BPE. |
| - `config.json` — host settings: prefill length 160, cache 512, pad / stop ids, max new tokens 32, and the system |
| prompt the model was trained with. |
|
|
| ## Demo |
|
|
| Menu-bar app in [FluidInference/FluidUse](https://github.com/FluidInference/FluidUse) (`ShortReplyDemo`): |
| open a post's reply box in any app, press **9**, the draft is pasted into the box; **0** regenerates it. About |
| 190 ms per reply with the Neural Engine on the prompt and the GPU on the tokens. |
|
|
| ```bash |
| hf download FluidInference/short-reply-0.6b-coreml --local-dir ~/Models/short-reply-0.6b-coreml |
| Sources/ShortReplyDemo/demo.sh --x ~/Models/short-reply-0.6b-coreml |
| ``` |
|
|
| ## Use |
|
|
| Swift host: `ShortReplyManager` in the same repository. |
|
|
| ```swift |
| let replies = try await ShortReplyManager.load(from: modelDirectory) |
| let draft = try await replies.draft(for: "Finally passed my driving test on the third try.") |
| print(draft.reply, draft.timing.totalSeconds) |
| ``` |
|
|
| Prompt (Qwen3 chat template, thinking disabled): |
|
|
| ``` |
| <|im_start|>system |
| {config.systemPrompt}<|im_end|> |
| <|im_start|>user |
| Post: {post} |
| Reply:<|im_end|> |
| <|im_start|>assistant |
| <think> |
| |
| </think> |
| |
| ``` |
|
|
| Left-pad the token ids to 160 with `padID`, mask padded columns, run `prefill`, copy its `keys` / `values` |
| (`[28, 8, 160, 128]` fp16) into the decoder state (`k_cache_i` / `v_cache_i`, `[1, 8, 512, 128]`), then call |
| `decode` with one token at a time (`input_ids`, `attention_mask` `[1, 1, 1, 512]`, `cache_position`) until |
| `<|im_end|>` or 32 tokens. Greedy decoding reproduces the benchmarked replies. |
|
|
| ## Provenance |
|
|
| Base: [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) (Apache-2.0). Conversion: coremltools 9.0, |
| `minimum_deployment_target` macOS 15 / iOS 18. Training data: 4,615 generated posts labeled by a 1.7B reply model |
| plus 478 hand-written pairs; no real social posts. Training and conversion records are kept in the author's lab |
| repository. |
|
|
