Text Generation
GGUF
English
Vietnamese
llama.cpp
on-device
mobile
ios
macos
qwen3.5
automatic-speech-recognition
asr
speech
qwen3-asr
conversational
Instructions to use haohan84/darksurf-models with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use haohan84/darksurf-models with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf haohan84/darksurf-models:UD-Q4_K_XL # Run inference directly in the terminal: llama cli -hf haohan84/darksurf-models:UD-Q4_K_XL
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf haohan84/darksurf-models:UD-Q4_K_XL # Run inference directly in the terminal: llama cli -hf haohan84/darksurf-models:UD-Q4_K_XL
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf haohan84/darksurf-models:UD-Q4_K_XL # Run inference directly in the terminal: ./build/bin/llama cli -hf haohan84/darksurf-models:UD-Q4_K_XL
Use Docker
docker model run hf.co/haohan84/darksurf-models:UD-Q4_K_XL
- Llama new
- LM Studio
- Jan
- vLLM
How to use haohan84/darksurf-models with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "haohan84/darksurf-models" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "haohan84/darksurf-models", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/haohan84/darksurf-models:UD-Q4_K_XL
- Ollama
How to use haohan84/darksurf-models with Ollama:
ollama run hf.co/haohan84/darksurf-models:UD-Q4_K_XL
- Unsloth Desktop
- Docker Model Runner
How to use haohan84/darksurf-models with Docker Model Runner:
docker model run hf.co/haohan84/darksurf-models:UD-Q4_K_XL
- Lemonade
How to use haohan84/darksurf-models with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull haohan84/darksurf-models:UD-Q4_K_XL
Run and chat with the model
lemonade run user.darksurf-models-UD-Q4_K_XL
List all available models
lemonade list
- Atomic Chat
Model card: models that run on mobile devices
Browse files
README.md
CHANGED
|
@@ -1,3 +1,77 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
+
base_model: Qwen/Qwen3.5-0.8B
|
| 4 |
+
tags:
|
| 5 |
+
- gguf
|
| 6 |
+
- llama.cpp
|
| 7 |
+
- on-device
|
| 8 |
+
- mobile
|
| 9 |
+
- ios
|
| 10 |
+
- macos
|
| 11 |
+
- qwen3.5
|
| 12 |
+
pipeline_tag: text-generation
|
| 13 |
+
language:
|
| 14 |
+
- en
|
| 15 |
+
- vi
|
| 16 |
---
|
| 17 |
+
|
| 18 |
+
# DarkSurf models: small language models that run on phones
|
| 19 |
+
|
| 20 |
+
Language models that have been measured and picked to **run on the device**, with no server: the apps in this
|
| 21 |
+
project (starting with DarkSurf, a privacy-first browser for macOS and iOS) download one of these files once and run it
|
| 22 |
+
locally with [llama.cpp](https://github.com/ggml-org/llama.cpp) on Metal. Prompts and page text never leave the device.
|
| 23 |
+
|
| 24 |
+
Only models that were measured against their full-precision original are published here. Everything else is kept
|
| 25 |
+
out of this repo.
|
| 26 |
+
|
| 27 |
+
## Models
|
| 28 |
+
|
| 29 |
+
| Catalog id | File | Size | Min. RAM | Use |
|
| 30 |
+
|---|---|---:|---:|---|
|
| 31 |
+
| `qwen3.5-0.8b-ud-q4kxl` | `Qwen3.5-0.8B-UD-Q4_K_XL.gguf` | 559 MB | 3 GB | Summarize, ask about a page, translate. Text only |
|
| 32 |
+
|
| 33 |
+
### Qwen3.5-0.8B UD-Q4_K_XL
|
| 34 |
+
|
| 35 |
+
[Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B) (Apache-2.0) in Unsloth's Dynamic 4-bit quantization
|
| 36 |
+
([`unsloth/Qwen3.5-0.8B-GGUF`](https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF)), copied unchanged. SHA-256
|
| 37 |
+
`3177ebd67afe4438374da19e690bc1b98756f7e0fea9240e1be404336156a7b5`. Text only: no vision projector.
|
| 38 |
+
|
| 39 |
+
Measured on an Apple M4 (llama.cpp `3c809f4`, Metal) against the bf16 GGUF, wikitext-2, 100 chunks of 512 tokens:
|
| 40 |
+
|
| 41 |
+
| | Size | Perplexity | vs bf16 | Same top-1 token as bf16 | Prefill | Decode |
|
| 42 |
+
|---|---:|---:|---:|---:|---:|---:|
|
| 43 |
+
| bf16 (reference) | 1.40 GiB | 18.52 | | | 1866 tok/s | 51 tok/s |
|
| 44 |
+
| **UD-Q4_K_XL** | 532 MiB | 18.79 | +1.5% | 92.0% | 1736 tok/s | 103 tok/s |
|
| 45 |
+
|
| 46 |
+
The Dynamic 4-bit file loses about half as much as a plain Q4_K_M (+3.3%) for 25 MiB more.
|
| 47 |
+
|
| 48 |
+
**Limits.** This is a 0.8B model: good for short summaries, extraction and simple questions; weak at long reasoning
|
| 49 |
+
and at non-English text (Vietnamese translations read fluently but contain meaning errors). The minimum RAM is an
|
| 50 |
+
estimate, and speed and memory have **not yet been measured on an iPhone or iPad**. Perplexity was measured in English
|
| 51 |
+
only.
|
| 52 |
+
|
| 53 |
+
## Run it
|
| 54 |
+
|
| 55 |
+
```sh
|
| 56 |
+
hf download haohan84/darksurf-models Qwen3.5-0.8B-UD-Q4_K_XL.gguf
|
| 57 |
+
llama-completion -m Qwen3.5-0.8B-UD-Q4_K_XL.gguf -c 8192 --top-k 20 --top-p 0.8 --temp 0.7 -no-cnv -e \
|
| 58 |
+
-p $'<|im_start|>user\nSummarize in one sentence: ...<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n'
|
| 59 |
+
```
|
| 60 |
+
|
| 61 |
+
The prompt is ChatML. The empty `<think></think>` block turns thinking off (faster, no scratchpad in the answer);
|
| 62 |
+
without it Qwen3.5 starts with a long reasoning section. Tested with llama.cpp `3c809f4`.
|
| 63 |
+
|
| 64 |
+
In an app, link llama.cpp as an xcframework (Metal) and load the file in-process with the same prompt and sampling
|
| 65 |
+
settings. Keep the 8k context, and unload the model when idle.
|
| 66 |
+
|
| 67 |
+
## How the apps get models
|
| 68 |
+
|
| 69 |
+
- `catalog.json` lists the models offered, each pinned to a commit of this repo (`/resolve/<commit>/…`) with its size
|
| 70 |
+
and SHA-256. `catalog.json.sig` is an Ed25519 signature; apps accept a downloaded catalog only if it verifies against
|
| 71 |
+
the public key built into the app, and check every download against the catalog's size and SHA-256.
|
| 72 |
+
- The file is downloaded when the user asks for it, and kept on the device until deleted.
|
| 73 |
+
|
| 74 |
+
## Credits and license
|
| 75 |
+
|
| 76 |
+
Model: Qwen team, [Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B), Apache-2.0. Quantization: Unsloth. Runtime:
|
| 77 |
+
llama.cpp (MIT). The files here are redistributed under the original Apache-2.0 license.
|