| --- |
| license: apache-2.0 |
| base_model: |
| - Qwen/Qwen3.5-0.8B |
| - Qwen/Qwen3-ASR-0.6B-hf |
| tags: |
| - gguf |
| - llama.cpp |
| - on-device |
| - mobile |
| - ios |
| - macos |
| - qwen3.5 |
| - automatic-speech-recognition |
| - asr |
| - speech |
| - qwen3-asr |
| pipeline_tag: text-generation |
| language: |
| - en |
| - vi |
| --- |
| |
| # DarkSurf models: small language and speech models that run on phones |
|
|
| Language and speech-to-text models that have been measured and picked to **run on the device**, with no server: the apps in this |
| project (starting with DarkSurf, a privacy-first browser for macOS and iOS) download one of these files once and run it |
| locally with [llama.cpp](https://github.com/ggml-org/llama.cpp) on Metal. Prompts and page text never leave the device. |
|
|
| Only models that were measured against their full-precision original are published here. Everything else is kept |
| out of this repo. |
|
|
| ## Models |
|
|
| | Catalog id | File | Size | Min. RAM | Use | |
| |---|---|---:|---:|---| |
| | `qwen3.5-0.8b-ud-q4kxl` | `Qwen3.5-0.8B-UD-Q4_K_XL.gguf` | 559 MB | 3 GB | Summarize, ask about a page, translate. Text only | |
| | `qwen3.5-2b-ud-q4kxl` | `Qwen3.5-2B-UD-Q4_K_XL.gguf` (+ optional `mmproj-F16.gguf` for images) | 1.3 GB (+ 0.67 GB) | 6 GB | Understands photos and reads text in them; better answers than the 0.8B | |
| | `qwen3-asr-0.6b-q8` (speech) | `Qwen3-ASR-0.6B-Q8_0.gguf` + `mmproj-Qwen3-ASR-0.6B-Q8_0.gguf` | 847 MB | 3 GB | Speech to text, 52 languages including Vietnamese | |
|
|
| ### Qwen3.5-0.8B UD-Q4_K_XL |
|
|
| [Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B) (Apache-2.0) in Unsloth's Dynamic 4-bit quantization |
| ([`unsloth/Qwen3.5-0.8B-GGUF`](https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF)), copied unchanged. SHA-256 |
| `3177ebd67afe4438374da19e690bc1b98756f7e0fea9240e1be404336156a7b5`. Text only: no vision projector. |
|
|
| Measured on an Apple M4 (llama.cpp `3c809f4`, Metal) against the bf16 GGUF, wikitext-2, 100 chunks of 512 tokens: |
|
|
| | | Size | Perplexity | vs bf16 | Same top-1 token as bf16 | Prefill | Decode | |
| |---|---:|---:|---:|---:|---:|---:| |
| | bf16 (reference) | 1.40 GiB | 18.52 | | | 1866 tok/s | 51 tok/s | |
| | **UD-Q4_K_XL** | 532 MiB | 18.79 | +1.5% | 92.0% | 1736 tok/s | 103 tok/s | |
|
|
| The Dynamic 4-bit file loses about half as much as a plain Q4_K_M (+3.3%) for 25 MiB more. |
|
|
| **Limits.** This is a 0.8B model: good for short summaries, extraction and simple questions; weak at long reasoning |
| and at non-English text (Vietnamese translations read fluently but contain meaning errors). The minimum RAM is an |
| estimate, and speed and memory have **not yet been measured on an iPhone or iPad**. Perplexity was measured in English |
| only. |
|
|
| ### Qwen3.5 2B UD-Q4_K_XL (text and vision) |
|
|
| [Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B) (Apache-2.0) in Unsloth's Dynamic 4-bit quantization |
| ([`unsloth/Qwen3.5-2B-GGUF`](https://huggingface.co/unsloth/Qwen3.5-2B-GGUF) revision `f6d5376…`), copied unchanged, |
| plus the image encoder `mmproj-F16.gguf` (also Unsloth's, unchanged). The text file works on its own; the `mmproj` adds |
| image understanding. SHA-256 values are in `catalog.json`. |
|
|
| Measured on an Apple M4 with llama.cpp `3c809f4` (Metal): |
|
|
| | | Result | |
| |---|---| |
| | Quality vs the bf16 GGUF (wikitext-2) | perplexity +2.2% (12.35 vs 12.09), 93.4% same top-1 token, mean KL divergence 0.016 | |
| | Function check, 9 cases | 8 pass: reads a newspaper headline and date, describes a landscape (also in Vietnamese), answers text questions; one flower mislabeled | |
| | Speed | about 990 tok/s prompt, 63 tok/s decoding; the image encoder takes 0.8 to 3.2 s per photo depending on its size | |
| | Memory, 4096-token context | 2.3 GiB resident | |
|
|
| Cap the image size (`--image-max-tokens`) and **always set a small context (4096)**: the default needs 6 GiB. Not run on an |
| iPhone or iPad yet; no vision benchmark was run, only the nine cases above. |
|
|
| ### Qwen3-ASR 0.6B Q8_0 (speech to text) |
| |
| [Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B-hf) (Apache-2.0), converted to GGUF here with llama.cpp |
| `3c809f4` and quantized to Q8_0. **Two files, both required**: the language model (639 MB) and the audio encoder |
| `mmproj-…` (208 MB); sizes and SHA-256 are in `catalog.json`. Converting the `-hf` checkpoint needs two fixes that |
| llama.cpp's converter does not have yet (nested config and tensor names for the audio encoder, and the M-RoPE sections); |
| the converted files match the official `ggml-org` GGUF tensor for tensor. |
|
|
| Measured on an Apple M4 with llama.cpp `3c809f4` (Metal), on three short clips (English and Vietnamese synthetic speech, |
| one LibriSpeech clip): |
|
|
| | | Result | |
| |---|---| |
| | Word error rate | 1.8%, identical to the transformers fp32 reference (Vietnamese clip 0%) | |
| | Speed | 40 s of speech in about 2 s after a 0.9 s load (about 20x faster than real time) | |
| | Memory | 0.8 GiB peak footprint with a 4096-token context (1.4 GiB resident) | |
|
|
| **Always set a small context (4096).** The default 65 536-token context needs 8 GiB for the KV cache alone. |
|
|
| Limits: three clips are a smoke test, not an accuracy benchmark; noise, accents, long recordings and streaming are not |
| measured, and **it has not been run on an iPhone or iPad yet**. The language label it prints is one per recording. |
|
|
| ## Run it |
|
|
| ```sh |
| hf download haohan84/darksurf-models Qwen3.5-0.8B-UD-Q4_K_XL.gguf |
| llama-completion -m Qwen3.5-0.8B-UD-Q4_K_XL.gguf -c 8192 --top-k 20 --top-p 0.8 --temp 0.7 -no-cnv -e \ |
| -p $'<|im_start|>user\nSummarize in one sentence: ...<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n' |
| ``` |
|
|
| The prompt is ChatML. The empty `<think></think>` block turns thinking off (faster, no scratchpad in the answer); |
| without it Qwen3.5 starts with a long reasoning section. Tested with llama.cpp `3c809f4`. |
|
|
| Vision (text file + `mmproj`): |
|
|
| ```sh |
| llama-mtmd-cli -m Qwen3.5-2B-UD-Q4_K_XL.gguf --mmproj mmproj-F16.gguf -c 4096 --temp 0 \ |
| --image photo.jpg -p "Describe this photo in two sentences." |
| ``` |
|
|
| Speech to text (needs both files): |
|
|
| ```sh |
| llama-mtmd-cli -m Qwen3-ASR-0.6B-Q8_0.gguf --mmproj mmproj-Qwen3-ASR-0.6B-Q8_0.gguf -c 4096 --temp 0 \ |
| --audio speech.wav -p "Transcribe the audio." |
| # -> language English<asr_text>Mr. Quilter is the apostle of the middle classes, ... |
| ``` |
|
|
| In an app, link llama.cpp as an xcframework (Metal) and load the file in-process with the same prompt and sampling |
| settings. Keep the 8k context, and unload the model when idle. |
|
|
| ## How the apps get models |
|
|
| - `catalog.json` lists the models offered (chat models in `models`, speech models with several files in `speech`), each pinned to a commit of this repo (`/resolve/<commit>/…`) with its size |
| and SHA-256. `catalog.json.sig` is an Ed25519 signature; apps accept a downloaded catalog only if it verifies against |
| the public key built into the app, and check every download against the catalog's size and SHA-256. |
| - The file is downloaded when the user asks for it, and kept on the device until deleted. |
|
|
| ## Credits and license |
|
|
| Model: Qwen team, [Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B), Apache-2.0. Quantization: Unsloth. Runtime: |
| llama.cpp (MIT). The files here are redistributed under the original Apache-2.0 license. |
|
|
