---
license: apache-2.0
base_model:
  - Qwen/Qwen3.5-0.8B
  - Qwen/Qwen3-ASR-0.6B-hf
tags:
  - gguf
  - llama.cpp
  - on-device
  - mobile
  - ios
  - macos
  - qwen3.5
  - automatic-speech-recognition
  - asr
  - speech
  - qwen3-asr
pipeline_tag: text-generation
language:
  - en
  - vi
---

# DarkSurf models: small language and speech models that run on phones

Language and speech-to-text models that have been measured and picked to **run on the device**, with no server: the apps in this
project (starting with DarkSurf, a privacy-first browser for macOS and iOS) download one of these files once and run it
locally with [llama.cpp](https://github.com/ggml-org/llama.cpp) on Metal. Prompts and page text never leave the device.

Only models that were measured against their full-precision original are published here. Everything else is kept
out of this repo.

## Models

| Catalog id | File | Size | Min. RAM | Use |
|---|---|---:|---:|---|
| `qwen3.5-0.8b-ud-q4kxl` | `Qwen3.5-0.8B-UD-Q4_K_XL.gguf` | 559 MB | 3 GB | Summarize, ask about a page, translate. Text only |
| `qwen3.5-2b-ud-q4kxl` | `Qwen3.5-2B-UD-Q4_K_XL.gguf` (+ optional `mmproj-F16.gguf` for images) | 1.3 GB (+ 0.67 GB) | 6 GB | Understands photos and reads text in them; better answers than the 0.8B |
| `qwen3-asr-0.6b-q8` (speech) | `Qwen3-ASR-0.6B-Q8_0.gguf` + `mmproj-Qwen3-ASR-0.6B-Q8_0.gguf` | 847 MB | 3 GB | Speech to text, 52 languages including Vietnamese |

### Qwen3.5-0.8B UD-Q4_K_XL

[Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B) (Apache-2.0) in Unsloth's Dynamic 4-bit quantization
([`unsloth/Qwen3.5-0.8B-GGUF`](https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF)), copied unchanged. SHA-256
`3177ebd67afe4438374da19e690bc1b98756f7e0fea9240e1be404336156a7b5`. Text only: no vision projector.

Measured on an Apple M4 (llama.cpp `3c809f4`, Metal) against the bf16 GGUF, wikitext-2, 100 chunks of 512 tokens:

| | Size | Perplexity | vs bf16 | Same top-1 token as bf16 | Prefill | Decode |
|---|---:|---:|---:|---:|---:|---:|
| bf16 (reference) | 1.40 GiB | 18.52 | | | 1866 tok/s | 51 tok/s |
| **UD-Q4_K_XL** | 532 MiB | 18.79 | +1.5% | 92.0% | 1736 tok/s | 103 tok/s |

The Dynamic 4-bit file loses about half as much as a plain Q4_K_M (+3.3%) for 25 MiB more.

**Limits.** This is a 0.8B model: good for short summaries, extraction and simple questions; weak at long reasoning
and at non-English text (Vietnamese translations read fluently but contain meaning errors). The minimum RAM is an
estimate, and speed and memory have **not yet been measured on an iPhone or iPad**. Perplexity was measured in English
only.

### Qwen3.5 2B UD-Q4_K_XL (text and vision)

[Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B) (Apache-2.0) in Unsloth's Dynamic 4-bit quantization
([`unsloth/Qwen3.5-2B-GGUF`](https://huggingface.co/unsloth/Qwen3.5-2B-GGUF) revision `f6d5376…`), copied unchanged,
plus the image encoder `mmproj-F16.gguf` (also Unsloth's, unchanged). The text file works on its own; the `mmproj` adds
image understanding. SHA-256 values are in `catalog.json`.

Measured on an Apple M4 with llama.cpp `3c809f4` (Metal):

| | Result |
|---|---|
| Quality vs the bf16 GGUF (wikitext-2) | perplexity +2.2% (12.35 vs 12.09), 93.4% same top-1 token, mean KL divergence 0.016 |
| Function check, 9 cases | 8 pass: reads a newspaper headline and date, describes a landscape (also in Vietnamese), answers text questions; one flower mislabeled |
| Speed | about 990 tok/s prompt, 63 tok/s decoding; the image encoder takes 0.8 to 3.2 s per photo depending on its size |
| Memory, 4096-token context | 2.3 GiB resident |

Cap the image size (`--image-max-tokens`) and **always set a small context (4096)**: the default needs 6 GiB. Not run on an
iPhone or iPad yet; no vision benchmark was run, only the nine cases above.

### Qwen3-ASR 0.6B Q8_0 (speech to text)

[Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B-hf) (Apache-2.0), converted to GGUF here with llama.cpp
`3c809f4` and quantized to Q8_0. **Two files, both required**: the language model (639 MB) and the audio encoder
`mmproj-…` (208 MB); sizes and SHA-256 are in `catalog.json`. Converting the `-hf` checkpoint needs two fixes that
llama.cpp's converter does not have yet (nested config and tensor names for the audio encoder, and the M-RoPE sections);
the converted files match the official `ggml-org` GGUF tensor for tensor.

Measured on an Apple M4 with llama.cpp `3c809f4` (Metal), on three short clips (English and Vietnamese synthetic speech,
one LibriSpeech clip):

| | Result |
|---|---|
| Word error rate | 1.8%, identical to the transformers fp32 reference (Vietnamese clip 0%) |
| Speed | 40 s of speech in about 2 s after a 0.9 s load (about 20x faster than real time) |
| Memory | 0.8 GiB peak footprint with a 4096-token context (1.4 GiB resident) |

**Always set a small context (4096).** The default 65 536-token context needs 8 GiB for the KV cache alone.

Limits: three clips are a smoke test, not an accuracy benchmark; noise, accents, long recordings and streaming are not
measured, and **it has not been run on an iPhone or iPad yet**. The language label it prints is one per recording.

## Run it

```sh
hf download haohan84/darksurf-models Qwen3.5-0.8B-UD-Q4_K_XL.gguf
llama-completion -m Qwen3.5-0.8B-UD-Q4_K_XL.gguf -c 8192 --top-k 20 --top-p 0.8 --temp 0.7 -no-cnv -e \
  -p $'<|im_start|>user\nSummarize in one sentence: ...<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n'
```

The prompt is ChatML. The empty `<think></think>` block turns thinking off (faster, no scratchpad in the answer);
without it Qwen3.5 starts with a long reasoning section. Tested with llama.cpp `3c809f4`.

Vision (text file + `mmproj`):

```sh
llama-mtmd-cli -m Qwen3.5-2B-UD-Q4_K_XL.gguf --mmproj mmproj-F16.gguf -c 4096 --temp 0 \
  --image photo.jpg -p "Describe this photo in two sentences."
```

Speech to text (needs both files):

```sh
llama-mtmd-cli -m Qwen3-ASR-0.6B-Q8_0.gguf --mmproj mmproj-Qwen3-ASR-0.6B-Q8_0.gguf -c 4096 --temp 0 \
  --audio speech.wav -p "Transcribe the audio."
# -> language English<asr_text>Mr. Quilter is the apostle of the middle classes, ...
```

In an app, link llama.cpp as an xcframework (Metal) and load the file in-process with the same prompt and sampling
settings. Keep the 8k context, and unload the model when idle.

## How the apps get models

- `catalog.json` lists the models offered (chat models in `models`, speech models with several files in `speech`), each pinned to a commit of this repo (`/resolve/<commit>/…`) with its size
  and SHA-256. `catalog.json.sig` is an Ed25519 signature; apps accept a downloaded catalog only if it verifies against
  the public key built into the app, and check every download against the catalog's size and SHA-256.
- The file is downloaded when the user asks for it, and kept on the device until deleted.

## Credits and license

Model: Qwen team, [Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B), Apache-2.0. Quantization: Unsloth. Runtime:
llama.cpp (MIT). The files here are redistributed under the original Apache-2.0 license.