haohan84 commited on
Commit
b1b3920
·
verified ·
1 Parent(s): 1f9f41c

Model card: models that run on mobile devices

Browse files
Files changed (1) hide show
  1. README.md +74 -0
README.md CHANGED
@@ -1,3 +1,77 @@
1
  ---
2
  license: apache-2.0
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+ base_model: Qwen/Qwen3.5-0.8B
4
+ tags:
5
+ - gguf
6
+ - llama.cpp
7
+ - on-device
8
+ - mobile
9
+ - ios
10
+ - macos
11
+ - qwen3.5
12
+ pipeline_tag: text-generation
13
+ language:
14
+ - en
15
+ - vi
16
  ---
17
+
18
+ # DarkSurf models: small language models that run on phones
19
+
20
+ Language models that have been measured and picked to **run on the device**, with no server: the apps in this
21
+ project (starting with DarkSurf, a privacy-first browser for macOS and iOS) download one of these files once and run it
22
+ locally with [llama.cpp](https://github.com/ggml-org/llama.cpp) on Metal. Prompts and page text never leave the device.
23
+
24
+ Only models that were measured against their full-precision original are published here. Everything else is kept
25
+ out of this repo.
26
+
27
+ ## Models
28
+
29
+ | Catalog id | File | Size | Min. RAM | Use |
30
+ |---|---|---:|---:|---|
31
+ | `qwen3.5-0.8b-ud-q4kxl` | `Qwen3.5-0.8B-UD-Q4_K_XL.gguf` | 559 MB | 3 GB | Summarize, ask about a page, translate. Text only |
32
+
33
+ ### Qwen3.5-0.8B UD-Q4_K_XL
34
+
35
+ [Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B) (Apache-2.0) in Unsloth's Dynamic 4-bit quantization
36
+ ([`unsloth/Qwen3.5-0.8B-GGUF`](https://huggingface.co/unsloth/Qwen3.5-0.8B-GGUF)), copied unchanged. SHA-256
37
+ `3177ebd67afe4438374da19e690bc1b98756f7e0fea9240e1be404336156a7b5`. Text only: no vision projector.
38
+
39
+ Measured on an Apple M4 (llama.cpp `3c809f4`, Metal) against the bf16 GGUF, wikitext-2, 100 chunks of 512 tokens:
40
+
41
+ | | Size | Perplexity | vs bf16 | Same top-1 token as bf16 | Prefill | Decode |
42
+ |---|---:|---:|---:|---:|---:|---:|
43
+ | bf16 (reference) | 1.40 GiB | 18.52 | | | 1866 tok/s | 51 tok/s |
44
+ | **UD-Q4_K_XL** | 532 MiB | 18.79 | +1.5% | 92.0% | 1736 tok/s | 103 tok/s |
45
+
46
+ The Dynamic 4-bit file loses about half as much as a plain Q4_K_M (+3.3%) for 25 MiB more.
47
+
48
+ **Limits.** This is a 0.8B model: good for short summaries, extraction and simple questions; weak at long reasoning
49
+ and at non-English text (Vietnamese translations read fluently but contain meaning errors). The minimum RAM is an
50
+ estimate, and speed and memory have **not yet been measured on an iPhone or iPad**. Perplexity was measured in English
51
+ only.
52
+
53
+ ## Run it
54
+
55
+ ```sh
56
+ hf download haohan84/darksurf-models Qwen3.5-0.8B-UD-Q4_K_XL.gguf
57
+ llama-completion -m Qwen3.5-0.8B-UD-Q4_K_XL.gguf -c 8192 --top-k 20 --top-p 0.8 --temp 0.7 -no-cnv -e \
58
+ -p $'<|im_start|>user\nSummarize in one sentence: ...<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n'
59
+ ```
60
+
61
+ The prompt is ChatML. The empty `<think></think>` block turns thinking off (faster, no scratchpad in the answer);
62
+ without it Qwen3.5 starts with a long reasoning section. Tested with llama.cpp `3c809f4`.
63
+
64
+ In an app, link llama.cpp as an xcframework (Metal) and load the file in-process with the same prompt and sampling
65
+ settings. Keep the 8k context, and unload the model when idle.
66
+
67
+ ## How the apps get models
68
+
69
+ - `catalog.json` lists the models offered, each pinned to a commit of this repo (`/resolve/<commit>/…`) with its size
70
+ and SHA-256. `catalog.json.sig` is an Ed25519 signature; apps accept a downloaded catalog only if it verifies against
71
+ the public key built into the app, and check every download against the catalog's size and SHA-256.
72
+ - The file is downloaded when the user asks for it, and kept on the device until deleted.
73
+
74
+ ## Credits and license
75
+
76
+ Model: Qwen team, [Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B), Apache-2.0. Quantization: Unsloth. Runtime:
77
+ llama.cpp (MIT). The files here are redistributed under the original Apache-2.0 license.