Instructions to use CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0 with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0:Q8_0
# Run inference directly in the terminal:
llama cli -hf CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0:Q8_0

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0:Q8_0
# Run inference directly in the terminal:
llama cli -hf CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0:Q8_0

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0:Q8_0
# Run inference directly in the terminal:
./llama-cli -hf CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0:Q8_0

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0:Q8_0
# Run inference directly in the terminal:
./build/bin/llama-cli -hf CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0:Q8_0

Use Docker

docker model run hf.co/CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0:Q8_0

LM Studio
Jan

vLLM

How to use CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0:Q8_0

Ollama
How to use CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0 with Ollama:
```
ollama run hf.co/CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0:Q8_0
```
Unsloth Desktop

How to use CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0 with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0:Q8_0

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0:Q8_0"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0 with Docker Model Runner:
```
docker model run hf.co/CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0:Q8_0
```

Lemonade

How to use CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0:Q8_0

Run and chat with the model

lemonade run user.Qwen3.8-Flash-Next-NVFP4-Q8_0-Q8_0

List all available models

lemonade list

Hermes Agent

How to use CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0 with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0:Q8_0

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0:Q8_0

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0 with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0:Q8_0

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0:Q8_0" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3.8-Flash-Next 177B NVFP4 GGUF: run a 177B MoE from an SSD on a 16 GB GPU

Run a 177B parameter model on a consumer PC by streaming it from an NVMe SSD. This GGUF is made for SSD offloading: about 100 GiB of the 119 GiB file stays on the SSD and is read as needed, so the model runs on a 16 GB GPU with 32 GB of RAM. It runs with inferredThoughts, an open-source engine for SSD streaming of mixture-of-experts models.

GitHub: compiledthoughts/Inferred-Thoughts

Speed when streaming from SSD

RTX 5060 Ti 16 GB, 32 GB DDR5 RAM, Gen5 NVMe SSD, native Windows 11, inferred serve version 1.1, default settings.

context (`--ctx`)	prefill	decode
8,192	193 tok/s (5,700-token prompt)	8.5 to 9.1 tok/s
32,000	119 to 139 tok/s (2,800 to 3,300-token turns)	7.3 to 8.1 tok/s

On the same machine, llama.cpp decoded this model at 4.9 tok/s.

How SSD streaming works

Only 10 of 512 experts per layer run for each token, so most of the model is idle at any moment.
The engine keeps the dense weights and the busiest experts in VRAM, the next busiest in RAM, and reads the rest from the NVMe SSD on demand.
Long prompts are prefilled one layer at a time, so each expert is read from the SSD once per prompt instead of once per chunk.
The result: a model far larger than your VRAM and RAM combined runs at usable speed, without offloading whole layers to the CPU.

Run it

Requirements:

An RTX 50-series or RTX PRO Blackwell GPU (16 GB VRAM), NVIDIA driver R570+, CUDA 12.8+
32 GB of RAM and a fast local NVMe SSD (Gen4 or Gen5) with 128 GB free
Windows 11 or WSL2 (Ubuntu 24.04)

hf download CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0 Qwen3.8-Flash-Next-NVFP4-Q8_0.gguf --local-dir models
git clone https://github.com/compiledthoughts/Inferred-Thoughts
cd Inferred-Thoughts
cargo build --release --features cuda
./target/release/inferred serve -m ../models/Qwen3.8-Flash-Next-NVFP4-Q8_0.gguf --backend cuda --port 8080 --ctx 8192

The server is OpenAI-compatible at http://127.0.0.1:8080/v1, with a chat page at http://127.0.0.1:8080/. It works with coding agents such as Cline, including tool calling.

What is in this file

A GGUF conversion of nvidia/Qwen3.8-Flash-Next-NVFP4, NVIDIA's NVFP4 quantization of Qwen/Qwen3.8-Flash-Next. Routed experts stay in NVIDIA's NVFP4 and are not requantized. Everything else is Q8_0, hence the name.

This is an independent conversion. It is not made, reviewed or endorsed by NVIDIA or by Qwen.

176.944B parameters in 119.02 GiB (127,809,147,712 bytes), counted from its 1,512 tensors:

part	parameters	type	size
routed experts (512 per layer, 48 layers)	120.796B	NVFP4, with per-expert scales	63.28 GiB
dense: attention, GatedDeltaNet, shared experts, hyper-connections	4.312B	Q8_0, norms F32	4.45 GiB
token embedding	0.636B	Q8_0
hashed n-gram (PLE) table	51.200B	Q8_0, FP8 scale restored	50.66 GiB

Per token: the 4.3B dense parameters, 10 of 512 experts per layer (2.4B), and 16 rows of the n-gram table.
Not included: the MTP head (about 4B parameters) and the vision encoder, so this is a text-only model.
Tools report the file as Q8_0, because GGUF carries one type label per file (general.file_type), which cannot express a mix.

Files

file	size	sha256
`Qwen3.8-Flash-Next-NVFP4-Q8_0.gguf`	127809147712 bytes	`04124cb939a1ae968b53ce101b222a8eaf56bcb6742bae1976bede373b811a27`

License

Governed by the NVIDIA Open Model License, as the source checkpoint is, and by the Qwen Community License 1.0. Both texts are in this repository.

Downloads last month: 2,097

GGUF

Model size

177B params

Architecture

qwen4exp

Hardware compatibility

8-bit

Model tree for CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0

Base model

Qwen/Qwen3.8-Flash-Next

Quantized

nvidia/Qwen3.8-Flash-Next-NVFP4

Quantized

(8)

this model