Text Generation
GGUF
English
Vietnamese
llama.cpp
on-device
mobile
ios
macos
qwen3.5
automatic-speech-recognition
asr
speech
qwen3-asr
conversational
Instructions to use haohan84/darksurf-models with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use haohan84/darksurf-models with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf haohan84/darksurf-models:UD-Q4_K_XL # Run inference directly in the terminal: llama cli -hf haohan84/darksurf-models:UD-Q4_K_XL
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf haohan84/darksurf-models:UD-Q4_K_XL # Run inference directly in the terminal: llama cli -hf haohan84/darksurf-models:UD-Q4_K_XL
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf haohan84/darksurf-models:UD-Q4_K_XL # Run inference directly in the terminal: ./build/bin/llama cli -hf haohan84/darksurf-models:UD-Q4_K_XL
Use Docker
docker model run hf.co/haohan84/darksurf-models:UD-Q4_K_XL
- Llama new
- LM Studio
- Jan
- vLLM
How to use haohan84/darksurf-models with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "haohan84/darksurf-models" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "haohan84/darksurf-models", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/haohan84/darksurf-models:UD-Q4_K_XL
- Ollama
How to use haohan84/darksurf-models with Ollama:
ollama run hf.co/haohan84/darksurf-models:UD-Q4_K_XL
- Unsloth Desktop
- Docker Model Runner
How to use haohan84/darksurf-models with Docker Model Runner:
docker model run hf.co/haohan84/darksurf-models:UD-Q4_K_XL
- Lemonade
How to use haohan84/darksurf-models with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull haohan84/darksurf-models:UD-Q4_K_XL
Run and chat with the model
lemonade run user.darksurf-models-UD-Q4_K_XL
List all available models
lemonade list
- Atomic Chat
File size: 7,122 Bytes
97c675b 458a549 b1b3920 458a549 b1b3920 97c675b b1b3920 458a549 b1b3920 458a549 b1b3920 138b57e 458a549 b1b3920 138b57e 458a549 b1b3920 138b57e 458a549 b1b3920 458a549 b1b3920 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 | ---
license: apache-2.0
base_model:
- Qwen/Qwen3.5-0.8B
- Qwen/Qwen3-ASR-0.6B-hf
tags:
- gguf
- llama.cpp
- on-device
- mobile
- ios
- macos
- qwen3.5
- automatic-speech-recognition
- asr
- speech
- qwen3-asr
pipeline_tag: text-generation
language:
- en
- vi
---
# DarkSurf models: small language and speech models that run on phones
Language and speech-to-text models that have been measured and picked to **run on the device**, with no server: the apps in this
project (starting with DarkSurf, a privacy-first browser for macOS and iOS) download one of these files once and run it
locally with [llama.cpp](https://github.com/ggml-org/llama.cpp) on Metal. Prompts and page text never leave the device.
Only models that were measured against their full-precision original are published here. Everything else is kept
out of this repo.
## Models
| Catalog id | File | Size | Min. RAM | Use |
|---|---|---:|---:|---|
| `qwen3.5-0.8b-ud-q4kxl` | `Qwen3.5-0.8B-UD-Q4_K_XL.gguf` | 559 MB | 3 GB | Summarize, ask about a page, translate. Text only |
| `qwen3.5-2b-ud-q4kxl` | `Qwen3.5-2B-UD-Q4_K_XL.gguf` (+ optional `mmproj-F16.gguf` for images) | 1.3 GB (+ 0.67 GB) | 6 GB | Understands photos and reads text in them; better answers than the 0.8B |
| `qwen3-asr-0.6b-q8` (speech) | `Qwen3-ASR-0.6B-Q8_0.gguf` + `mmproj-Qwen3-ASR-0.6B-Q8_0.gguf` | 847 MB | 3 GB | Speech to text, 52 languages including Vietnamese |
### Qwen3.5-0.8B UD-Q4_K_XL
[Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B) (Apache-2.0) in Unsloth's Dynamic 4-bit quantization
([`unsloth/Qwen3.5-0.8B-GGUF`](https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF)), copied unchanged. SHA-256
`3177ebd67afe4438374da19e690bc1b98756f7e0fea9240e1be404336156a7b5`. Text only: no vision projector.
Measured on an Apple M4 (llama.cpp `3c809f4`, Metal) against the bf16 GGUF, wikitext-2, 100 chunks of 512 tokens:
| | Size | Perplexity | vs bf16 | Same top-1 token as bf16 | Prefill | Decode |
|---|---:|---:|---:|---:|---:|---:|
| bf16 (reference) | 1.40 GiB | 18.52 | | | 1866 tok/s | 51 tok/s |
| **UD-Q4_K_XL** | 532 MiB | 18.79 | +1.5% | 92.0% | 1736 tok/s | 103 tok/s |
The Dynamic 4-bit file loses about half as much as a plain Q4_K_M (+3.3%) for 25 MiB more.
**Limits.** This is a 0.8B model: good for short summaries, extraction and simple questions; weak at long reasoning
and at non-English text (Vietnamese translations read fluently but contain meaning errors). The minimum RAM is an
estimate, and speed and memory have **not yet been measured on an iPhone or iPad**. Perplexity was measured in English
only.
### Qwen3.5 2B UD-Q4_K_XL (text and vision)
[Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B) (Apache-2.0) in Unsloth's Dynamic 4-bit quantization
([`unsloth/Qwen3.5-2B-GGUF`](https://huggingface.co/unsloth/Qwen3.5-2B-GGUF) revision `f6d5376…`), copied unchanged,
plus the image encoder `mmproj-F16.gguf` (also Unsloth's, unchanged). The text file works on its own; the `mmproj` adds
image understanding. SHA-256 values are in `catalog.json`.
Measured on an Apple M4 with llama.cpp `3c809f4` (Metal):
| | Result |
|---|---|
| Quality vs the bf16 GGUF (wikitext-2) | perplexity +2.2% (12.35 vs 12.09), 93.4% same top-1 token, mean KL divergence 0.016 |
| Function check, 9 cases | 8 pass: reads a newspaper headline and date, describes a landscape (also in Vietnamese), answers text questions; one flower mislabeled |
| Speed | about 990 tok/s prompt, 63 tok/s decoding; the image encoder takes 0.8 to 3.2 s per photo depending on its size |
| Memory, 4096-token context | 2.3 GiB resident |
Cap the image size (`--image-max-tokens`) and **always set a small context (4096)**: the default needs 6 GiB. Not run on an
iPhone or iPad yet; no vision benchmark was run, only the nine cases above.
### Qwen3-ASR 0.6B Q8_0 (speech to text)
[Qwen3-ASR-0.6B](https://huggingface.co/Qwen/Qwen3-ASR-0.6B-hf) (Apache-2.0), converted to GGUF here with llama.cpp
`3c809f4` and quantized to Q8_0. **Two files, both required**: the language model (639 MB) and the audio encoder
`mmproj-…` (208 MB); sizes and SHA-256 are in `catalog.json`. Converting the `-hf` checkpoint needs two fixes that
llama.cpp's converter does not have yet (nested config and tensor names for the audio encoder, and the M-RoPE sections);
the converted files match the official `ggml-org` GGUF tensor for tensor.
Measured on an Apple M4 with llama.cpp `3c809f4` (Metal), on three short clips (English and Vietnamese synthetic speech,
one LibriSpeech clip):
| | Result |
|---|---|
| Word error rate | 1.8%, identical to the transformers fp32 reference (Vietnamese clip 0%) |
| Speed | 40 s of speech in about 2 s after a 0.9 s load (about 20x faster than real time) |
| Memory | 0.8 GiB peak footprint with a 4096-token context (1.4 GiB resident) |
**Always set a small context (4096).** The default 65 536-token context needs 8 GiB for the KV cache alone.
Limits: three clips are a smoke test, not an accuracy benchmark; noise, accents, long recordings and streaming are not
measured, and **it has not been run on an iPhone or iPad yet**. The language label it prints is one per recording.
## Run it
```sh
hf download haohan84/darksurf-models Qwen3.5-0.8B-UD-Q4_K_XL.gguf
llama-completion -m Qwen3.5-0.8B-UD-Q4_K_XL.gguf -c 8192 --top-k 20 --top-p 0.8 --temp 0.7 -no-cnv -e \
-p $'<|im_start|>user\nSummarize in one sentence: ...<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n'
```
The prompt is ChatML. The empty `<think></think>` block turns thinking off (faster, no scratchpad in the answer);
without it Qwen3.5 starts with a long reasoning section. Tested with llama.cpp `3c809f4`.
Vision (text file + `mmproj`):
```sh
llama-mtmd-cli -m Qwen3.5-2B-UD-Q4_K_XL.gguf --mmproj mmproj-F16.gguf -c 4096 --temp 0 \
--image photo.jpg -p "Describe this photo in two sentences."
```
Speech to text (needs both files):
```sh
llama-mtmd-cli -m Qwen3-ASR-0.6B-Q8_0.gguf --mmproj mmproj-Qwen3-ASR-0.6B-Q8_0.gguf -c 4096 --temp 0 \
--audio speech.wav -p "Transcribe the audio."
# -> language English<asr_text>Mr. Quilter is the apostle of the middle classes, ...
```
In an app, link llama.cpp as an xcframework (Metal) and load the file in-process with the same prompt and sampling
settings. Keep the 8k context, and unload the model when idle.
## How the apps get models
- `catalog.json` lists the models offered (chat models in `models`, speech models with several files in `speech`), each pinned to a commit of this repo (`/resolve/<commit>/…`) with its size
and SHA-256. `catalog.json.sig` is an Ed25519 signature; apps accept a downloaded catalog only if it verifies against
the public key built into the app, and check every download against the catalog's size and SHA-256.
- The file is downloaded when the user asks for it, and kept on the device until deleted.
## Credits and license
Model: Qwen team, [Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B), Apache-2.0. Quantization: Unsloth. Runtime:
llama.cpp (MIT). The files here are redistributed under the original Apache-2.0 license.
|