File size: 2,523 Bytes
f0ec7d4 9cb7e0a fb64230 9cb7e0a f0ec7d4 9cb7e0a fb64230 9cb7e0a fb64230 80787bd 9cb7e0a fb64230 88e3a40 fb64230 9cb7e0a fb64230 9cb7e0a fb64230 9cb7e0a 80787bd 226d8bb 80787bd fb64230 9cb7e0a 0bc930d 9cb7e0a 0bc930d 9cb7e0a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 | ---
license: apache-2.0
base_model:
- FirstPrinciplesLabs/quark
base_model_relation: quantized
pipeline_tag: text-generation
language:
- en
tags:
- gguf
- llama.cpp
- physics
- science
- reasoning
- math
---
# Quark int8
[Quark](https://huggingface.co/FirstPrinciplesLabs/quark), a 9B reasoning model for physics, science and
maths by [FirstPrinciples](https://firstprinciples.com/), quantized to int8 for
[llama.cpp](https://github.com/ggml-org/llama.cpp) and tools built on it (Ollama, LM Studio). It is also on
Ollama as [FirstPrinciples/quark-q8](https://ollama.com/FirstPrinciples/quark-q8).
| File | Format | Size | Input |
|---|---|---|---|
| `quark-Q8_0.gguf` | Q8_0 (int8) | 9.8 GB | text |
Q8_0 is llama.cpp's int8 format: weights are stored as 8-bit integers, with one scale shared by each
block of 32.
## Results
The int8 version matches the original bf16 model on GPQA Diamond, AIME 2025, MMLU-Pro Physics and
Humanity's Last Exam, and generates faster. See the Quark model card for the
[benchmark scores](https://huggingface.co/FirstPrinciplesLabs/quark#results) and
[throughput](https://huggingface.co/FirstPrinciplesLabs/quark#throughput) on an NVIDIA H200 and a
MacBook Pro (M4, 16 GB).
## Usage
### Ollama
The int8 model is on Ollama as [FirstPrinciples/quark-q8](https://ollama.com/FirstPrinciples/quark-q8),
with the recommended sampling settings and a 131,072-token context built in. Install
[Ollama](https://ollama.com/download), then run:
```bash
ollama run FirstPrinciples/quark-q8
```
A 131,072-token context needs about 15 GB of GPU memory. On a 16 GB Mac, lower it inside the chat with
`/set parameter num_ctx 32768` (longer answers are then cut off)
### llama.cpp
Install llama.cpp, which includes `llama-server`, following its
[installation guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md) (Homebrew,
winget, conda, Nix, and builds for NVIDIA GPUs).
Then start a server; it downloads the model on first run and serves an OpenAI-compatible API and a chat
page at `http://localhost:8080`:
```bash
llama-server -hf FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0 --jinja -c 131072 \
--temp 1.0 --top-k 20 --top-p 0.95 --presence-penalty 1.5
```
This sets the recommended sampling settings for thinking mode (from the
[Quark model card](https://huggingface.co/FirstPrinciplesLabs/quark#usage)). A 131,072-token context needs about
15 GB of GPU memory; on a 16 GB Mac, use `-c 32768` instead (longer answers are then cut off).
## License
Apache 2.0
|