Matthew Tchouikine commited on
Commit ·
4d98655
1
Parent(s): 03c78ed
Add GGUF results and speed (llama.cpp, vLLM)
Browse files
README.md
CHANGED
|
@@ -28,26 +28,28 @@ Quark is the first in a series of small, powerful scientific reasoning models. Y
|
|
| 28 |
|
| 29 |
## Results
|
| 30 |
|
| 31 |
-
Accuracy (%), measured with [EvalScope](https://github.com/modelscope/evalscope) and vLLM
|
| 32 |
-
up to 131,072 output tokens, and following the Qwen3.5 model card's sampling
|
| 33 |
-
AIME 2025 is averaged over 16 samples per problem and GPQA Diamond over 8. HLE (text-only, 2,158
|
| 34 |
-
is graded by Qwen3.5-27B as judge.
|
| 35 |
-
|
| 36 |
-
| Benchmark | Qwen3.5-9B | **Quark** |
|
| 37 |
-
|---|---|---|
|
| 38 |
-
| GPQA Diamond | 81.6 | **87.4** |
|
| 39 |
-
| AIME 2025 | 86.9 | **91.7** |
|
| 40 |
-
| MMLU-Pro Physics | 90.7 | 91.0 |
|
| 41 |
-
| Humanity's Last Exam (HLE), text | 15.9 | **22.7** |
|
| 42 |
-
| HLE, physics subset | 8.4 | **15.8** |
|
| 43 |
-
|
| 44 |
-
|
|
|
|
| 45 |
|
| 46 |
**Compared with other open models** (scores as reported by each developer):
|
| 47 |
|
| 48 |
| Model | Size | GPQA Diamond | AIME 2025 | HLE |
|
| 49 |
|---|---|---|---|---|
|
| 50 |
| **Quark** | 9B | **87.4** | **91.7** | **22.7** |
|
|
|
|
| 51 |
| Qwen3-8B | 8B | 59.6 | 69.3 | 4.4 |
|
| 52 |
| NVIDIA Nemotron Nano 9B v2 | 9B | 64.0 | 72.1 | 6.5 |
|
| 53 |
| gpt-oss-20b | 21B | 71.5 | 91.7 | 10.9 |
|
|
@@ -57,7 +59,74 @@ Both models were measured by us with the same setup.
|
|
| 57 |
| DeepSeek-R1-0528 | 671B | 81.0 | 87.5 | 17.7 |
|
| 58 |
| Kimi K2 Thinking | 1T | 84.5 | 94.5 | 23.9 |
|
| 59 |
|
| 60 |
-
— = not reported. Sources: each model's official model card
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 61 |
|
| 62 |
## Training data
|
| 63 |
|
|
|
|
| 28 |
|
| 29 |
## Results
|
| 30 |
|
| 31 |
+
Accuracy (%), measured with [EvalScope](https://github.com/modelscope/evalscope) and vLLM (the GGUF with
|
| 32 |
+
llama.cpp): thinking mode, up to 131,072 output tokens, and following the Qwen3.5 model card's sampling
|
| 33 |
+
settings. AIME 2025 is averaged over 16 samples per problem and GPQA Diamond over 8. HLE (text-only, 2,158
|
| 34 |
+
questions) is graded by Qwen3.5-27B as judge.
|
| 35 |
+
|
| 36 |
+
| Benchmark | Qwen3.5-9B | **Quark** | Quark Q8_0 ([GGUF](#quantized-version-gguf)) |
|
| 37 |
+
|---|---|---|---|
|
| 38 |
+
| GPQA Diamond | 81.6 | **87.4** | 87.8 |
|
| 39 |
+
| AIME 2025 | 86.9 | **91.7** | 94.0 |
|
| 40 |
+
| MMLU-Pro Physics | 90.7 | 91.0 | 91.6 |
|
| 41 |
+
| Humanity's Last Exam (HLE), text | 15.9 | **22.7** | 22.3 |
|
| 42 |
+
| HLE, physics subset | 8.4 | **15.8** | 16.3 |
|
| 43 |
+
|
| 44 |
+
All were measured by us with the same setup. The 8-bit GGUF matches Quark: the differences are within
|
| 45 |
+
run-to-run variation.
|
| 46 |
|
| 47 |
**Compared with other open models** (scores as reported by each developer):
|
| 48 |
|
| 49 |
| Model | Size | GPQA Diamond | AIME 2025 | HLE |
|
| 50 |
|---|---|---|---|---|
|
| 51 |
| **Quark** | 9B | **87.4** | **91.7** | **22.7** |
|
| 52 |
+
| Quark Q8_0 (GGUF) | 9B, 8-bit | 87.8 | 94.0 | 22.3 |
|
| 53 |
| Qwen3-8B | 8B | 59.6 | 69.3 | 4.4 |
|
| 54 |
| NVIDIA Nemotron Nano 9B v2 | 9B | 64.0 | 72.1 | 6.5 |
|
| 55 |
| gpt-oss-20b | 21B | 71.5 | 91.7 | 10.9 |
|
|
|
|
| 59 |
| DeepSeek-R1-0528 | 671B | 81.0 | 87.5 | 17.7 |
|
| 60 |
| Kimi K2 Thinking | 1T | 84.5 | 94.5 | 23.9 |
|
| 61 |
|
| 62 |
+
— = not reported. Sources: each model's official model card; Quark and its GGUF measured by us (see
|
| 63 |
+
above), with the GGUF's differences from Quark within run-to-run variation.
|
| 64 |
+
|
| 65 |
+
## Quantized version (GGUF)
|
| 66 |
+
|
| 67 |
+
An 8-bit GGUF for llama.cpp, Ollama and LM Studio is available at
|
| 68 |
+
[FirstPrinciplesLabs/quark-GGUF](https://huggingface.co/FirstPrinciplesLabs/quark-GGUF), with the vision
|
| 69 |
+
encoder for image input. Its accuracy is in the [Results](#results) table: it matches Quark on all
|
| 70 |
+
benchmarks.
|
| 71 |
+
|
| 72 |
+
## Speed
|
| 73 |
+
|
| 74 |
+
Tokens per second on an NVIDIA H200 and a MacBook Pro, for the original bf16 weights and the Q8_0
|
| 75 |
+
[GGUF](#quantized-version-gguf).
|
| 76 |
+
|
| 77 |
+
### llama.cpp
|
| 78 |
+
|
| 79 |
+
Speed for a single user with llama.cpp, by how much context is already filled.
|
| 80 |
+
|
| 81 |
+
Generation (writing the answer), tokens per second:
|
| 82 |
+
|
| 83 |
+
| Hardware | Context | Q8_0 | bf16 |
|
| 84 |
+
|---|---|---|---|
|
| 85 |
+
| NVIDIA H200 | empty | 216 | 185 |
|
| 86 |
+
| NVIDIA H200 | 32K | 191 | 167 |
|
| 87 |
+
| NVIDIA H200 | 100K | 156 | 139 |
|
| 88 |
+
| MacBook Pro, M4, 16 GB | empty | 11.2 | does not fit (18.4 GB) |
|
| 89 |
+
| MacBook Pro, M4, 16 GB | 32K | 8.7 | does not fit |
|
| 90 |
+
|
| 91 |
+
Prompt reading (a 512-token prompt), tokens per second:
|
| 92 |
+
|
| 93 |
+
| Hardware | Context | Q8_0 | bf16 |
|
| 94 |
+
|---|---|---|---|
|
| 95 |
+
| NVIDIA H200 | empty | 9,106 | 16,587 |
|
| 96 |
+
| NVIDIA H200 | 32K | 7,677 | 12,200 |
|
| 97 |
+
| NVIDIA H200 | 100K | 5,544 | 7,800 |
|
| 98 |
+
| MacBook Pro, M4, 16 GB | empty | 208 | does not fit |
|
| 99 |
+
| MacBook Pro, M4, 16 GB | 32K | 139 | does not fit |
|
| 100 |
+
|
| 101 |
+
Generation is limited by memory speed, where the smaller Q8_0 file is faster. Prompt reading is limited
|
| 102 |
+
by compute, where llama.cpp runs bf16 faster on NVIDIA GPUs. For thinking-mode answers, generation
|
| 103 |
+
takes most of the time.
|
| 104 |
+
|
| 105 |
+
On Apple Silicon, speed scales with memory bandwidth (about 90% of it is used), so Macs with faster
|
| 106 |
+
memory run proportionally faster. The model needs about 10 GB of memory plus room for the context;
|
| 107 |
+
16 GB Macs can run it with shorter contexts.
|
| 108 |
+
|
| 109 |
+
### vLLM
|
| 110 |
+
|
| 111 |
+
vLLM targets NVIDIA and AMD GPUs; on Macs, use llama.cpp or tools built on it.
|
| 112 |
+
|
| 113 |
+
With vLLM on one NVIDIA H200, 512-token prompts and 4,096-token answers in thinking mode, speed of each
|
| 114 |
+
request, tokens per second:
|
| 115 |
+
|
| 116 |
+
| Concurrent requests | bf16 | Q8_0 GGUF |
|
| 117 |
+
|---|---|---|
|
| 118 |
+
| 1 | 184 | 66 |
|
| 119 |
+
| 8 | 162 | 32 |
|
| 120 |
+
| 24 | 145 | 16 |
|
| 121 |
+
| 64 | 104 | 6.4 |
|
| 122 |
+
| 128 | 71 | 3.3 |
|
| 123 |
+
|
| 124 |
+
With vLLM, use the bf16 weights: vLLM runs GGUF files only through the
|
| 125 |
+
[vllm-gguf-plugin](https://github.com/vllm-project/vllm-gguf-plugin), which is much slower (measured with
|
| 126 |
+
vLLM 0.29 and plugin version 0.0.5). The GGUF is meant for llama.cpp, where it is the fastest option for
|
| 127 |
+
a single user (see [llama.cpp](#llamacpp)). For a single user, vLLM with bf16 and llama.cpp run at about
|
| 128 |
+
the same speed (184 and 185 tokens per second); vLLM's advantage is serving many users at once, with each
|
| 129 |
+
of 128 concurrent requests still getting 71 tokens per second. Measured with `evalscope perf`.
|
| 130 |
|
| 131 |
## Training data
|
| 132 |
|