File size: 3,807 Bytes
a668d1c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 | ---
license: apache-2.0
base_model: Qwen/Qwen2.5-Coder-0.5B-Instruct
base_model_relation: quantized
pipeline_tag: text-generation
language:
- en
tags:
- coreml
- apple-neural-engine
- on-device
- qwen2
- code
library_name: coreml
---
# qwen2.5-coder-0.5b-coreml
[Qwen2.5-Coder-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B-Instruct) converted to Core ML: it
writes Python from a plain-English request, fully on device. One 944 MB fp16 package with two functions that share
the weights:
| Function | Shape | Runs on | p50 (M5 Pro, macOS 27, from Swift) |
|---|---|---|---:|
| `prefill` | 448 tokens, fixed | Neural Engine (1,451 / 1,456 ops) | 52 ms |
| `decode` | 1 token, 1,024-slot KV cache in `MLState` | GPU | 25–28 ms / token (36–40 tok/s) |
A typical function (100–330 tokens) is written in 3–9 s. The prompt runs on the Neural Engine; the token-by-token
writing runs on the GPU, because the Neural Engine rejects the decoder's in-graph KV-cache write. By time, the
Neural Engine carries about 1% of a task.
## Quality
Greedy decoding, pass@1, same prompt and test harness for both rows:
| Benchmark | PyTorch fp32 | This Core ML package |
|---|---:|---:|
| HumanEval (164) | 90 / 164 = 54.9% | 89 / 164 = 54.3% |
| MBPP sanitized, test split (257) | — | 119 / 257 = 46.3% |
On HumanEval, 142 of 164 completions are token-for-token identical to PyTorch; one task flips from pass to fail.
Qwen reports 61.6% HumanEval for this model with its own evaluation harness; the numbers above use plain greedy
decoding (`repetition_penalty` 1.0) and a simple code extractor, applied identically to both rows.
## Files
- `qwen2_5_coder_0_5b.mlpackage` — multifunction Core ML package (`prefill`, `decode`), token-id inputs (embedding
lookup and RoPE tables are inside the graph).
- `tokenizer.json` — Qwen2.5 byte-level BPE.
- `config.json` — host settings: prefill length 448, cache 1,024, pad / stop ids, max new tokens 512, system prompt.
## Demo
`CodeWriterDemo` in [FluidInference/FluidUse](https://github.com/FluidInference/FluidUse): a list of ten short
Python tasks (from MBPP, one per Python feature: list comprehension, slicing, dict, string methods, `sorted`, …).
The model writes each one live into an editor, then `python3` runs the task's asserts. The demo list is
hand-picked for the recording; the MBPP row above is the unbiased number.
```bash
swift run -c release CodeWriterDemo # downloads this repository on first launch
```
## Use
Swift host: `CodeWriterManager` in the same repository.
```swift
let writer = try await CodeWriterManager.load(from: modelDirectory)
let result = try await writer.write(task: "Write a function to reverse each string in a list.") { text in
print(text) // the text so far, after every token
}
print(result.code, result.timing.tokensPerSecond)
```
Prompt (Qwen2.5 chat template):
```
<|im_start|>system
{config.systemPrompt}<|im_end|>
<|im_start|>user
{task}<|im_end|>
<|im_start|>assistant
```
Left-pad the token ids to 448 with `padID`, mask padded columns, run `prefill`, copy its `keys` / `values`
(`[24, 2, 448, 64]` fp16) into the decoder state (`k_cache_i` / `v_cache_i`, `[1, 2, 1024, 64]`), then call
`decode` one token at a time (`input_ids`, `attention_mask` `[1, 1, 1, 1024]`, `cache_position`) until
`<|im_end|>`. Greedy decoding reproduces the benchmarked completions.
## Provenance
Base: [Qwen/Qwen2.5-Coder-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B-Instruct) (Apache-2.0),
weights unchanged apart from fp16 storage. Conversion: coremltools 9.0, torch 2.7.0, `minimum_deployment_target`
macOS 15 / iOS 18. Benchmarks: [HumanEval](https://github.com/openai/human-eval) (MIT),
[MBPP](https://github.com/google-research/google-research/tree/master/mbpp) (CC-BY-4.0).
|