File size: 7,122 Bytes
97c675b
 
458a549
 
 
b1b3920
 
 
 
 
 
 
 
458a549
 
 
 
b1b3920
 
 
 
97c675b
b1b3920
458a549
b1b3920
458a549
b1b3920
 
 
 
 
 
 
 
 
 
 
138b57e
458a549
b1b3920
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
138b57e
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
458a549
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
b1b3920
 
 
 
 
 
 
 
 
 
 
138b57e
 
 
 
 
 
 
458a549
 
 
 
 
 
 
 
b1b3920
 
 
 
 
458a549
b1b3920
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
---
license: apache-2.0
base_model:
  - Qwen/Qwen3.5-0.8B
  - Qwen/Qwen3-ASR-0.6B-hf
tags:
  - gguf
  - llama.cpp
  - on-device
  - mobile
  - ios
  - macos
  - qwen3.5
  - automatic-speech-recognition
  - asr
  - speech
  - qwen3-asr
pipeline_tag: text-generation
language:
  - en
  - vi
---

# DarkSurf models: small language and speech models that run on phones

Language and speech-to-text models that have been measured and picked to **run on the device**, with no server: the apps in this
project (starting with DarkSurf, a privacy-first browser for macOS and iOS) download one of these files once and run it
locally with [llama.cpp](https://github.com/ggml-org/llama.cpp) on Metal. Prompts and page text never leave the device.

Only models that were measured against their full-precision original are published here. Everything else is kept
out of this repo.

## Models

| Catalog id | File | Size | Min. RAM | Use |
|---|---|---:|---:|---|
| `qwen3.5-0.8b-ud-q4kxl` | `Qwen3.5-0.8B-UD-Q4_K_XL.gguf` | 559 MB | 3 GB | Summarize, ask about a page, translate. Text only |
| `qwen3.5-2b-ud-q4kxl` | `Qwen3.5-2B-UD-Q4_K_XL.gguf` (+ optional `mmproj-F16.gguf` for images) | 1.3 GB (+ 0.67 GB) | 6 GB | Understands photos and reads text in them; better answers than the 0.8B |
| `qwen3-asr-0.6b-q8` (speech) | `Qwen3-ASR-0.6B-Q8_0.gguf` + `mmproj-Qwen3-ASR-0.6B-Q8_0.gguf` | 847 MB | 3 GB | Speech to text, 52 languages including Vietnamese |

### Qwen3.5-0.8B UD-Q4_K_XL

[Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B) (Apache-2.0) in Unsloth's Dynamic 4-bit quantization
([`unsloth/Qwen3.5-0.8B-GGUF`](https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF)), copied unchanged. SHA-256
`3177ebd67afe4438374da19e690bc1b98756f7e0fea9240e1be404336156a7b5`. Text only: no vision projector.

Measured on an Apple M4 (llama.cpp `3c809f4`, Metal) against the bf16 GGUF, wikitext-2, 100 chunks of 512 tokens:

| | Size | Perplexity | vs bf16 | Same top-1 token as bf16 | Prefill | Decode |
|---|---:|---:|---:|---:|---:|---:|
| bf16 (reference) | 1.40 GiB | 18.52 | | | 1866 tok/s | 51 tok/s |
| **UD-Q4_K_XL** | 532 MiB | 18.79 | +1.5% | 92.0% | 1736 tok/s | 103 tok/s |

The Dynamic 4-bit file loses about half as much as a plain Q4_K_M (+3.3%) for 25 MiB more.

**Limits.** This is a 0.8B model: good for short summaries, extraction and simple questions; weak at long reasoning
and at non-English text (Vietnamese translations read fluently but contain meaning errors). The minimum RAM is an
estimate, and speed and memory have **not yet been measured on an iPhone or iPad**. Perplexity was measured in English
only.

### Qwen3.5 2B UD-Q4_K_XL (text and vision)

[Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B) (Apache-2.0) in Unsloth's Dynamic 4-bit quantization
([`unsloth/Qwen3.5-2B-GGUF`](https://huggingface.co/unsloth/Qwen3.5-2B-GGUF) revision `f6d5376…`), copied unchanged,
plus the image encoder `mmproj-F16.gguf` (also Unsloth's, unchanged). The text file works on its own; the `mmproj` adds
image understanding. SHA-256 values are in `catalog.json`.

Measured on an Apple M4 with llama.cpp `3c809f4` (Metal):

| | Result |
|---|---|
| Quality vs the bf16 GGUF (wikitext-2) | perplexity +2.2% (12.35 vs 12.09), 93.4% same top-1 token, mean KL divergence 0.016 |
| Function check, 9 cases | 8 pass: reads a newspaper headline and date, describes a landscape (also in Vietnamese), answers text questions; one flower mislabeled |
| Speed | about 990 tok/s prompt, 63 tok/s decoding; the image encoder takes 0.8 to 3.2 s per photo depending on its size |
| Memory, 4096-token context | 2.3 GiB resident |

Cap the image size (`--image-max-tokens`) and **always set a small context (4096)**: the default needs 6 GiB. Not run on an
iPhone or iPad yet; no vision benchmark was run, only the nine cases above.

### Qwen3-ASR 0.6B Q8_0 (speech to text)

[Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B-hf) (Apache-2.0), converted to GGUF here with llama.cpp
`3c809f4` and quantized to Q8_0. **Two files, both required**: the language model (639 MB) and the audio encoder
`mmproj-…` (208 MB); sizes and SHA-256 are in `catalog.json`. Converting the `-hf` checkpoint needs two fixes that
llama.cpp's converter does not have yet (nested config and tensor names for the audio encoder, and the M-RoPE sections);
the converted files match the official `ggml-org` GGUF tensor for tensor.

Measured on an Apple M4 with llama.cpp `3c809f4` (Metal), on three short clips (English and Vietnamese synthetic speech,
one LibriSpeech clip):

| | Result |
|---|---|
| Word error rate | 1.8%, identical to the transformers fp32 reference (Vietnamese clip 0%) |
| Speed | 40 s of speech in about 2 s after a 0.9 s load (about 20x faster than real time) |
| Memory | 0.8 GiB peak footprint with a 4096-token context (1.4 GiB resident) |

**Always set a small context (4096).** The default 65 536-token context needs 8 GiB for the KV cache alone.

Limits: three clips are a smoke test, not an accuracy benchmark; noise, accents, long recordings and streaming are not
measured, and **it has not been run on an iPhone or iPad yet**. The language label it prints is one per recording.

## Run it

```sh
hf download haohan84/darksurf-models Qwen3.5-0.8B-UD-Q4_K_XL.gguf
llama-completion -m Qwen3.5-0.8B-UD-Q4_K_XL.gguf -c 8192 --top-k 20 --top-p 0.8 --temp 0.7 -no-cnv -e \
  -p $'<|im_start|>user\nSummarize in one sentence: ...<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n'
```

The prompt is ChatML. The empty `<think></think>` block turns thinking off (faster, no scratchpad in the answer);
without it Qwen3.5 starts with a long reasoning section. Tested with llama.cpp `3c809f4`.

Vision (text file + `mmproj`):

```sh
llama-mtmd-cli -m Qwen3.5-2B-UD-Q4_K_XL.gguf --mmproj mmproj-F16.gguf -c 4096 --temp 0 \
  --image photo.jpg -p "Describe this photo in two sentences."
```

Speech to text (needs both files):

```sh
llama-mtmd-cli -m Qwen3-ASR-0.6B-Q8_0.gguf --mmproj mmproj-Qwen3-ASR-0.6B-Q8_0.gguf -c 4096 --temp 0 \
  --audio speech.wav -p "Transcribe the audio."
# -> language English<asr_text>Mr. Quilter is the apostle of the middle classes, ...
```

In an app, link llama.cpp as an xcframework (Metal) and load the file in-process with the same prompt and sampling
settings. Keep the 8k context, and unload the model when idle.

## How the apps get models

- `catalog.json` lists the models offered (chat models in `models`, speech models with several files in `speech`), each pinned to a commit of this repo (`/resolve/<commit>/…`) with its size
  and SHA-256. `catalog.json.sig` is an Ed25519 signature; apps accept a downloaded catalog only if it verifies against
  the public key built into the app, and check every download against the catalog's size and SHA-256.
- The file is downloaded when the user asks for it, and kept on the device until deleted.

## Credits and license

Model: Qwen team, [Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B), Apache-2.0. Quantization: Unsloth. Runtime:
llama.cpp (MIT). The files here are redistributed under the original Apache-2.0 license.