Instructions to use StanzaAPI/qwen-moe-stream with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use StanzaAPI/qwen-moe-stream with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf StanzaAPI/qwen-moe-stream:F16
# Run inference directly in the terminal:
llama cli -hf StanzaAPI/qwen-moe-stream:F16

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf StanzaAPI/qwen-moe-stream:F16
# Run inference directly in the terminal:
llama cli -hf StanzaAPI/qwen-moe-stream:F16

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf StanzaAPI/qwen-moe-stream:F16
# Run inference directly in the terminal:
./build/bin/llama cli -hf StanzaAPI/qwen-moe-stream:F16

Use Docker

docker model run hf.co/StanzaAPI/qwen-moe-stream:F16

Llama new
LM Studio
Jan
Ollama
How to use StanzaAPI/qwen-moe-stream with Ollama:
```
ollama run hf.co/StanzaAPI/qwen-moe-stream:F16
```
Unsloth Desktop

How to use StanzaAPI/qwen-moe-stream with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf StanzaAPI/qwen-moe-stream:F16

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "StanzaAPI/qwen-moe-stream:F16"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use StanzaAPI/qwen-moe-stream with Docker Model Runner:
```
docker model run hf.co/StanzaAPI/qwen-moe-stream:F16
```

Lemonade

How to use StanzaAPI/qwen-moe-stream with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull StanzaAPI/qwen-moe-stream:F16

Run and chat with the model

lemonade run user.qwen-moe-stream-F16

List all available models

lemonade list

Hermes Agent

How to use StanzaAPI/qwen-moe-stream with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf StanzaAPI/qwen-moe-stream:F16

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default StanzaAPI/qwen-moe-stream:F16

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use StanzaAPI/qwen-moe-stream with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf StanzaAPI/qwen-moe-stream:F16

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "StanzaAPI/qwen-moe-stream:F16" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

qwen-moe-stream / README.md

nirdaddy

Upload README.md with huggingface_hub

cbe0fea verified 13 days ago

preview code

Raw History Blame Contribute Delete

4.14 kB

---
license: apache-2.0
base_model: Qwen/Qwen3-4B-Instruct-2507
library_name: llama.cpp
tags:
  - gguf
  - llama.cpp
  - moe
  - mixture-of-experts
  - qwen3
  - inference
  - streaming
---
# Qwen3-4B-Instruct-2507 — exact 8-expert MoE reformat

This repository contains a **format-level conversion** of
[Qwen/Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507)
from a dense GGUF to an 8-expert `qwen3moe` GGUF that llama.cpp can load.

It is **not a trained MoE**. The dense FFN is the sum of its hidden neurons:

```
FFN(x) = Σ_j W_down[:, j] · silu(W_gate[j] @ x) · (W_up[j] @ x)
```

The conversion partitions the 9,728 hidden neurons into 8 disjoint slices of
1,216 neurons. With all 8 experts selected and a uniform router, the result is
the same function. The exact model matches the source within F16 round-off.

## Files

| file | size | description |
|---|---:|---|
| `Qwen3-4B-MoE8-exact-f16.gguf` | 6.25 GB | exact full-fanout model; **use this one** |
| `Qwen3-4B-MoE8-ridge6-f16.gguf` | 6.25 GB | lossy top-6 variant, 75% of FFN bytes |
| `Qwen3-4B-MoE8-exact-f16.manifest.json` | 2.1 MB | tensor offsets, expert byte ranges, permutation hashes |

Checksums are in [`results/CHECKSUMS.txt`](https://github.com/DrMkdaddy/qwen-moe-stream/blob/main/results/CHECKSUMS.txt).

The non-FFN tensors (attention, embeddings, LM head) are copied bit-for-bit
from the source Q4_K_M file. The FFN experts are written F16 after
dequantization. The source is Q4_K_M, so the F16 conversion carries that
quantization's round-off.

## Run with llama.cpp

```bash
llama-cli -m Qwen3-4B-MoE8-exact-f16.gguf \
  -p "The capital of France is" -n 16 -ngl 0 -t 8 --temp 0
```

`llama.cpp` build b11050 or newer is required for the `qwen3moe` graph. The
exact model is a normal dense-equivalent FFN, so it is slower than the Q4
source: the FFN is F16 and all 8 experts run.

## Measurements

Held-out text, 512-token context, CPU, `llama-perplexity`:

| model | PPL | ratio vs dense |
|---|---:|---:|
| dense Qwen3-4B Q4_K_M | 17.6203 | 1.000× |
| MoE8 exact (all 8 experts) | **17.6157** | 0.9997× |
| MoE8 top-6 ridge router | 20.8920 | 1.186× |

Logit comparison, exact vs dense:

| metric | value |
|---|---:|
| mean KL divergence | 0.001350 |
| same top token | 96.47% |
| PPL ratio | 0.99972 |

Streaming, 4 tokens × 36 layers, real expert tensors, `madvise` after use:

| schedule | peak RSS | bytes touched |
|---|---:|---:|
| resident, no eviction | 5.93 GB | 21.5 GB |
| streamed, evict after use | **0.78 GB** | 21.5 GB |

The 0.78 GB figure is the executor alone, not a full serving stack. The exact
model's FFN is 5.38 GB; the file is 6.25 GB.

## What top-6 means

The top-6 file is a **lossy** point on the quality/IO curve. It reads 75% of
the FFN bytes and raises PPL from 17.62 to 20.89. The experts are slices of a
dense FFN, not experts trained to produce complete updates; selecting a subset
drops real terms. The exact file is the faithful one. See the project README
for the oracle ceiling and the negative result on gradient-trained routers.

## Calibration and provenance

The ridge router was fitted on 384 token positions per layer from public project READMEs
and specification documents, dumped with a llama.cpp eval callback. No model
weights were trained. No private user data was used. The conversion scripts
and full methodology live at:

- GitHub: https://github.com/DrMkdaddy/qwen-moe-stream
- Source model: https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507

## Limitations

- The exact model is a reformat, not a new model, and not a speedup. All 8
  experts run; only resident memory is improved by streaming.
- The top-6 model is lossy and should not be presented as equivalent.
- Only the FFN is MoE-ified. Attention, embeddings, and the LM head stay dense.
- Measurements are from one CPU host with `-ngl 0`; GPU offload will change
  the memory and throughput numbers.
- F16 experts from a Q4_K_M source are larger than the source file.

## License

The conversion code is MIT. The weights are derived from
Qwen/Qwen3-4B-Instruct-2507 and are distributed under **Apache-2.0**.

	---
	license: apache-2.0
	base_model: Qwen/Qwen3-4B-Instruct-2507
	library_name: llama.cpp
	tags:
	- gguf
	- llama.cpp
	- moe
	- mixture-of-experts
	- qwen3
	- inference
	- streaming
	---

	# Qwen3-4B-Instruct-2507 — exact 8-expert MoE reformat

	This repository contains a format-level conversion of
	[Qwen/Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507)
	from a dense GGUF to an 8-expert `qwen3moe` GGUF that llama.cpp can load.

	It is not a trained MoE. The dense FFN is the sum of its hidden neurons:

	```
	FFN(x) = Σ_j W_down[:, j] · silu(W_gate[j] @ x) · (W_up[j] @ x)
	```

	The conversion partitions the 9,728 hidden neurons into 8 disjoint slices of
	1,216 neurons. With all 8 experts selected and a uniform router, the result is
	the same function. The exact model matches the source within F16 round-off.

	## Files

	\| file \| size \| description \|
	\|---\|---:\|---\|
	\| `Qwen3-4B-MoE8-exact-f16.gguf` \| 6.25 GB \| exact full-fanout model; use this one \|
	\| `Qwen3-4B-MoE8-ridge6-f16.gguf` \| 6.25 GB \| lossy top-6 variant, 75% of FFN bytes \|
	\| `Qwen3-4B-MoE8-exact-f16.manifest.json` \| 2.1 MB \| tensor offsets, expert byte ranges, permutation hashes \|

	Checksums are in [`results/CHECKSUMS.txt`](https://github.com/DrMkdaddy/qwen-moe-stream/blob/main/results/CHECKSUMS.txt).

	The non-FFN tensors (attention, embeddings, LM head) are copied bit-for-bit
	from the source Q4_K_M file. The FFN experts are written F16 after
	dequantization. The source is Q4_K_M, so the F16 conversion carries that
	quantization's round-off.

	## Run with llama.cpp

	```bash
	llama-cli -m Qwen3-4B-MoE8-exact-f16.gguf \
	-p "The capital of France is" -n 16 -ngl 0 -t 8 --temp 0
	```

	`llama.cpp` build b11050 or newer is required for the `qwen3moe` graph. The
	exact model is a normal dense-equivalent FFN, so it is slower than the Q4
	source: the FFN is F16 and all 8 experts run.

	## Measurements

	Held-out text, 512-token context, CPU, `llama-perplexity`:

	\| model \| PPL \| ratio vs dense \|
	\|---\|---:\|---:\|
	\| dense Qwen3-4B Q4_K_M \| 17.6203 \| 1.000× \|
	\| MoE8 exact (all 8 experts) \| 17.6157 \| 0.9997× \|
	\| MoE8 top-6 ridge router \| 20.8920 \| 1.186× \|

	Logit comparison, exact vs dense:

	\| metric \| value \|
	\|---\|---:\|
	\| mean KL divergence \| 0.001350 \|
	\| same top token \| 96.47% \|
	\| PPL ratio \| 0.99972 \|

	Streaming, 4 tokens × 36 layers, real expert tensors, `madvise` after use:

	\| schedule \| peak RSS \| bytes touched \|
	\|---\|---:\|---:\|
	\| resident, no eviction \| 5.93 GB \| 21.5 GB \|
	\| streamed, evict after use \| 0.78 GB \| 21.5 GB \|

	The 0.78 GB figure is the executor alone, not a full serving stack. The exact
	model's FFN is 5.38 GB; the file is 6.25 GB.

	## What top-6 means

	The top-6 file is a lossy point on the quality/IO curve. It reads 75% of
	the FFN bytes and raises PPL from 17.62 to 20.89. The experts are slices of a
	dense FFN, not experts trained to produce complete updates; selecting a subset
	drops real terms. The exact file is the faithful one. See the project README
	for the oracle ceiling and the negative result on gradient-trained routers.

	## Calibration and provenance

	The ridge router was fitted on 384 token positions per layer from public project READMEs
	and specification documents, dumped with a llama.cpp eval callback. No model
	weights were trained. No private user data was used. The conversion scripts
	and full methodology live at:

	- GitHub: https://github.com/DrMkdaddy/qwen-moe-stream
	- Source model: https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507

	## Limitations

	- The exact model is a reformat, not a new model, and not a speedup. All 8
	experts run; only resident memory is improved by streaming.
	- The top-6 model is lossy and should not be presented as equivalent.
	- Only the FFN is MoE-ified. Attention, embeddings, and the LM head stay dense.
	- Measurements are from one CPU host with `-ngl 0`; GPU offload will change
	the memory and throughput numbers.
	- F16 experts from a Q4_K_M source are larger than the source file.

	## License

	The conversion code is MIT. The weights are derived from
	Qwen/Qwen3-4B-Instruct-2507 and are distributed under Apache-2.0.