Instructions to use FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0 with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0:Q8_0
# Run inference directly in the terminal:
llama cli -hf FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0:Q8_0

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0:Q8_0
# Run inference directly in the terminal:
llama cli -hf FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0:Q8_0

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0:Q8_0
# Run inference directly in the terminal:
./build/bin/llama cli -hf FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0:Q8_0

Use Docker

docker model run hf.co/FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0:Q8_0

Llama new
LM Studio
Jan

vLLM

How to use FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0:Q8_0

Ollama
How to use FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0 with Ollama:
```
ollama run hf.co/FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0:Q8_0
```
Unsloth Desktop

How to use FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0 with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0:Q8_0

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0:Q8_0"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0 with Docker Model Runner:
```
docker model run hf.co/FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0:Q8_0
```

Lemonade

How to use FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0:Q8_0

Run and chat with the model

lemonade run user.quark_quantized_weight_only-Q8_0-Q8_0

List all available models

lemonade list

Hermes Agent

How to use FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0 with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0:Q8_0

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0:Q8_0

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0 with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0:Q8_0

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0:Q8_0" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Matthew Tchouikine commited on 5 days ago

Commit

a8c24fc

1 Parent(s): b3cc926

Remove redundant accuracy/speed results

Browse files

Files changed (1) hide show

README.md +5 -46

README.md CHANGED Viewed

@@ -26,53 +26,12 @@ science and maths by [FirstPrinciples](https://firstprinciples.com/), for
 | `quark-Q8_0.gguf` | language model | Q8_0 (8-bit) | 9.8 GB |
 | `mmproj-quark-BF16.gguf` | vision encoder, for image input | BF16 (16-bit) | 0.9 GB |
-## Results
-Accuracy (%), measured with [EvalScope](https://github.com/modelscope/evalscope) in the same setup as the
-[Quark model card](https://huggingface.co/FirstPrinciplesLabs/quark): thinking mode, up to 131,072 output
-tokens, Qwen3.5 sampling settings. The GGUF was run with llama.cpp, the original with vLLM.
-| Benchmark | Quark (bf16) | **Quark Q8_0 GGUF** |
-|---|---|---|
-| GPQA Diamond | 87.4 | 87.8 |
-| AIME 2025 | 91.7 | 94.0 |
-| MMLU-Pro Physics | 91.0 | 91.6 |
-| Humanity's Last Exam (HLE), text | 22.7 | 22.3 |
-| HLE, physics subset | 15.8 | 16.3 |
-All differences are within run-to-run variation: the 8-bit model matches the original.
-## Speed
-Speed for a single user with llama.cpp (tokens per second), by how much context is already filled.
-Generation (writing the answer):
-| Hardware | Context | Q8_0 | bf16 |
-|---|---|---|---|
-| NVIDIA H200 | empty | 216 | 185 |
-| NVIDIA H200 | 32K | 191 | 167 |
-| NVIDIA H200 | 100K | 156 | 139 |
-| MacBook Pro, M4, 16 GB | empty | 11.2 | does not fit (18.4 GB) |
-| MacBook Pro, M4, 16 GB | 32K | 8.7 | does not fit |
-Prompt reading (a 512-token prompt):
-| Hardware | Context | Q8_0 | bf16 |
-|---|---|---|---|
-| NVIDIA H200 | empty | 9,106 | 16,587 |
-| NVIDIA H200 | 32K | 7,677 | 12,200 |
-| NVIDIA H200 | 100K | 5,544 | 7,800 |
-| MacBook Pro, M4, 16 GB | empty | 208 | does not fit |
-| MacBook Pro, M4, 16 GB | 32K | 139 | does not fit |
-Generation is limited by memory speed, where the smaller Q8_0 file is faster. Prompt reading is limited
-by compute, where llama.cpp runs bf16 faster on NVIDIA GPUs. For thinking-mode answers, generation
-takes most of the time.
-On Apple Silicon, speed scales with memory bandwidth (about 90% of it is used), so Macs with faster
-memory run proportionally faster. The model needs about 10 GB of memory plus room for the context;
-16 GB Macs can run it with shorter contexts.
 ## Usage

 | `quark-Q8_0.gguf` | language model | Q8_0 (8-bit) | 9.8 GB |
 | `mmproj-quark-BF16.gguf` | vision encoder, for image input | BF16 (16-bit) | 0.9 GB |
+## Results and speed
+The Q8_0 GGUF matches the original model on GPQA Diamond, AIME 2025, MMLU-Pro Physics and Humanity's Last
+Exam. On the Quark model card: [benchmark scores](https://huggingface.co/FirstPrinciplesLabs/quark#results)
+and [speed with llama.cpp and vLLM](https://huggingface.co/FirstPrinciplesLabs/quark#speed) (Q8_0 vs bf16,
+on an NVIDIA H200 and a MacBook Pro).
 ## Usage