README.md

---
language:
- en
- zh
- multilingual
license: other
license_name: stepfun-community-license
license_link: https://huggingface.co/SHSLab/Step-5-Preview-BF16/blob/main/LICENSE
library_name: llama.cpp
pipeline_tag: image-text-to-text
base_model: stepfun-ai/Step-5-Preview-BF16
tags:
- stepfun
- step-5
- gguf
- llama.cpp
- moe
- mixture-of-experts
- agentic
- coding
- software-engineering
- long-context
- 1m-context
- multimodal
- image-text-to-text
- text-generation
- image
- video
- sparse-attention
- gqa
- q3_k_m
- 3-bit
- quantization
- financial-analysis
- deep-research
- tool-calling
- parallel-tool-calling
- json-schema
---
```

```markdown
# Step-5-Preview-GGUF-Q3_K_M-3bit
```

---

# Step-5-Preview — GGUF (Q3_K_M, 3-bit)

<div align="center">

[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-SHSLab-yellow)](https://huggingface.co/SHSLab)
[![GitHub](https://img.shields.io/badge/GitHub-StepFun-181717?logo=github)](https://github.com/stepfun-ai)
[![Discord](https://img.shields.io/badge/Discord-Join%20Us-5865F2?logo=discord)](https://discord.gg/stepfun)
[![License](https://img.shields.io/badge/License-StepFun%20Community-blue)](https://huggingface.co/SHSLab/Step-5-Preview-BF16/blob/main/LICENSE)
[![Format](https://img.shields.io/badge/Format-GGUF-purple)]()
[![Quant](https://img.shields.io/badge/Quant-Q3__K__M%20%283--bit%29-red)]()
[![Base](https://img.shields.io/badge/Base-BF16%20safetensors-blue)]()
[![Context](https://img.shields.io/badge/Context-Up%20to%201M%20Tokens-green)]()

</div>

> **🔥 Step-5-Preview Q3_K_M GGUF is now available!**
>
> This is the **3-bit k-quant GGUF** build of Step-5-Preview, converted from the official
> **BF16 safetensors** checkpoint (`stepfun-ai/Step-5-Preview-BF16`) and packaged for
> **llama.cpp**-based inference. It runs on CPU + GPU hybrid setups and is intended for
> local deployment, low-VRAM environments, and quick experimentation.
>
> **A 4.1 GB vision `mmproj` file is included** for multimodal (image + video) input,
> using the Step3-VL perception encoder (`projector_type=step3vl`).

---

## 📖 Table of Contents

- [Introduction](#-introduction)
- [What's in this Repository](#-whats-in-this-repository)
- [Quantization Details](#-quantization-details)
- [Model Architecture](#-model-architecture)
- [Model Specifications](#-model-specifications)
- [Benchmark Reference](#-benchmark-reference)
- [Hardware Requirements](#-hardware-requirements)
- [Quickstart with llama.cpp](#-quickstart-with-llamacpp)
- [Multimodal (mmproj) Usage](#-multimodal-mmproj-usage)
- [Server Mode](#-server-mode)
- [Convert It Yourself](#-convert-it-yourself)
- [Known Limitations](#-known-limitations)
- [Performance Notes](#-performance-notes)
- [Citation](#-citation)
- [License](#-license)
- [Contact](#-contact)

---

## 🚀 Introduction

**Step-5-Preview** is StepFun's flagship foundation model, designed for **real-world agentic tasks** — AI coding, software engineering, professional knowledge work, and financial analysis. It is a **600B-parameter sparse Mixture-of-Experts (MoE)** model with **27B active parameters per token**, a **1M-token context window**, and native **text + image + video** input.

This repository contains the **Q3_K_M (3-bit) GGUF** conversion, produced with the official
`convert_hf_to_gguf.py` pipeline (see `step5-llamacpp.patch`) and `llama-quantize`.

> **💡 Why a 3-bit GGUF?**
>
> • **~5× smaller** than the BF16 checkpoint — fits on consumer and prosumer hardware.
> • **llama.cpp native** — pure CPU, CPU+GPU hybrid, or full GPU offload.
> • **Multimodal ready** — paired `mmproj` for image and video input.
> • **Agent-capable** — tool calling and long-context reasoning preserved.

---

## 📦 What's in this Repository

| File | Size (approx.) | Description |
|:---|:---|:---|
| `Step-5-Preview-Q3_K_M.gguf` | ~220–260 GB | Text model, 3-bit k-quant |
| `Step-5-Preview-mmproj-BF16.gguf` | ~4.1 GB | Vision projector (`projector_type=step3vl`) |
| `README.md` | — | This file |

> **Note:** Exact file sizes depend on the final shard layout. If the model is split, load
> with `-m Step-5-Preview-Q3_K_M-00001-of-000NN.gguf` — llama.cpp auto-loads the rest.

---

## ⚙️ Quantization Details

| Parameter | Value |
|:---|:---|
| **Base checkpoint** | `stepfun-ai/Step-5-Preview-BF16` (bfloat16 safetensors) |
| **Conversion tool** | `convert_hf_to_gguf.py` (with `step5-llamacpp.patch`) |
| **Quantization tool** | `llama-quantize` |
| **Quant type** | **Q3_K_M** (k-quant, mixed 3-bit) |
| **File type tag** | `Q3_K_M` |
| **Tokenizer pre-tokenizer** | `deepseek-v3` (identical BPE to DeepSeek-V3) |
| **Architecture tag** | `STEP35` |
| **Vision projector** | `step3vl` (Step3-VL perception encoder) |

### What Q3_K_M Means

Q3_K_M is llama.cpp's **mixed 3-bit k-quant** — most tensors are quantized to 3-bit, while
sensitive tensors (attention output, some FFN) are kept at higher precision (4–6 bit).
This gives a better quality/size trade-off than plain Q3_K_S but is larger.

| Quant | Bits (avg) | Relative Size | Relative Quality |
|:---|:---|:---|:---|
| **BF16** | 16 | 100% | Reference |
| **Q8_0** | 8 | ~50% | Near-lossless |
| **Q5_K_M** | ~5 | ~33% | Very high |
| **Q4_K_M** | ~4 | ~27% | High |
| **Q3_K_M** ← this repo | ~3 | **~20%** | **Moderate** |
| **Q2_K** | ~2 | ~15% | Noticeable loss |

> ⚠️ At 3-bit, expect **perplexity to increase** and some degradation on coding, math, and
> fine-grained reasoning tasks compared to BF16 or Q4_K_M. See [Known Limitations](#-known-limitations).

---

## 🏗️ Model Architecture

> The GGUF graph reuses the **STEP35** architecture path in llama.cpp, extended for Step-5.

### Core Trunk

| Component | Value |
|:---|:---|
| **Architecture** | Sparse Mixture-of-Experts (MoE) |
| **Total parameters** | 600B |
| **Active parameters** | 27B per token (~4.5% sparsity) |
| **Layers** | 95 (narrow-deep Transformer) |
| **Attention pattern** | Hybrid — **23 full_attention** + 72 sliding_attention |
| **SWA type** | `LLAMA_SWA_TYPE_STANDARD` |
| **MoE** | `moe_num_experts` + `moe_top_k` |
| **Tokenizer** | DeepSeek-V3 BPE (`tokenizer.ggml.pre = deepseek-v3`) |

### RoPE Configuration

| Layer Type | `rope_theta` | Rotary Fraction |
|:---|:---|:---|
| **full_attention** | `1e7` | **1/3** of `head_dim` |
| **sliding_attention** | `1e4` | **1.0** (full) |

> The GGUF explicitly writes `rope.dimension_count` (`head_dim/3`) and
> `rope.dimension_count_swa` (`head_dim`) so llama.cpp honours Step-5's partial RoPE.
> Without these keys, llama.cpp would fall back to the Step3p5 default of `head_dim/2`.

### Vision Encoder (mmproj)

| Parameter | Value |
|:---|:---|
| **Resolution** | 728 px |
| **Patch size** | 14 |
| **Width** | 1536 |
| **Layers** | 47 |
| **Downsampler** | stride-2 pair |
| **Projector** | `vit_large_projector` |
| **`projector_type`** | `step3vl` |

<details>
<summary><strong>🔬 Sparse GQA indexer — what happens in GGUF</strong></summary>

The BF16 checkpoint uses a **Sparse GQA** mechanism on the full_attention layers:

- **CSA (block compression) indexer**
- **top-k selection** over compressed KV blocks

This is **not modelled in llama.cpp yet**. During conversion, the affected tensors are
dropped and those layers fall back to **dense attention**:

- Tensors dropped: `*.sparse_indexer*`, `*.ssmax_s`
- Result: **correct output, but slower** and with **different long-context behaviour**
  than the reference BF16 model.

To re-enable sparse attention in the future, the `filter_tensors()` hook in
`Step5Model` must be removed once `attention_impl=sparse_gqa` exists in the graph builder.

</details>

---

## 📋 Model Specifications

| Category | Specification |
|:---|:---|
| **Model Name** | Step-5-Preview |
| **Repository** | `Step-5-Preview-GGUF-Q3_K_M-3bit` |
| **Base Checkpoint** | `stepfun-ai/Step-5-Preview-BF16` |
| **Format** | GGUF |
| **Quantization** | Q3_K_M (3-bit k-quant, mixed) |
| **Developer** | StepFun |
| **Architecture Tag** | `STEP35` |
| **Total Parameters** | 600B |
| **Active Parameters** | 27B per token (~4.5% sparsity) |
| **Layers** | 95 (23 full + 72 sliding attention) |
| **Context Window** | Up to 1,000,000 tokens (KV cache limited) |
| **Attention** | Hybrid full + sliding; dense fallback in GGUF |
| **Input Modalities** | Text, Image, Video (via mmproj) |
| **Output Modalities** | Text |
| **Video Formats** | MP4, QuickTime, Matroska (≤128 MB, ≤5 min recommended) |
| **Reasoning Effort** | `low` / `medium` / `high` (`xhigh`) |
| **Tool Calling** | Parallel, strict JSON schema |
| **Token Pre-tokenizer** | `deepseek-v3` |
| **License** | StepFun Community License |

---

## 📊 Benchmark Reference

This GGUF is a **quantized derivative** of the BF16 model. The scores below are for the
**BF16 reference model** and are provided for context — **Q3_K_M will score lower**,
especially on coding, math, and long-context tasks.

<div style="border: 1px solid #8884; border-radius: 12px; padding: 20px 24px; margin: 24px 0;">

<strong>Reference scores (BF16, <code>high</code> reasoning effort)</strong>

| Benchmark | Score |
|:---|:---|
| **GPQA Diamond** | 93.5% |
| **Humanity's Last Exam (HLE)** | 46.5% |
| **AA-LCR v1.1** | 88.3% |
| **DeepSWE v1.1** | 67.7% |
| **Terminal-Bench 4** | 33.3% |
| **CyberGym** | 84.7% |
| **SciCode** | 58.9% |
| **ProgramBench** | 80.5% |
| **FrontierFinance** | 66.4% |
| **DRACO** | 83.3% |
| **MMMU-Pro** | 76.0% |
| **Output Speed** | 99.8 tok/s (8× H100, BF16) |

</div>

<details>
<summary><strong>📉 Expected degradation at Q3_K_M</strong></summary>

Quantization at 3-bit affects tasks unevenly. Rough expectations (may vary):

| Task Type | Sensitivity to 3-bit | Notes |
|:---|:---|:---|
| Conversational / general text | Low | Usually near-identical |
| Summarization / extraction | Low–Moderate | Minor fluency loss |
| Coding / SWE | **High** | Syntax errors, subtler logic bugs |
| Math / formal reasoning | **High** | Multi-step chains degrade first |
| Long-context (100K+) | **High** | Compounded by dense fallback |
| Tool calling / JSON | Moderate | Schema adherence usually OK |
| Vision (mmproj is BF16) | Low | Encoder precision is preserved |

If you need maximum quality, use the **BF16 safetensors** or a **Q4_K_M / Q5_K_M** GGUF.

</details>

---

## 🖥️ Hardware Requirements

<div style="border: 1px solid #8884; border-radius: 12px; padding: 20px 24px; margin: 24px 0;">

<strong>📐 KV cache is the real constraint</strong>

The 3-bit weights are small, but the **1M-token context window is not**. KV cache memory
scales with context length and batch size. Realistic configurations below.

</div>

| Setup | RAM / VRAM | Notes |
|:---|:---|:---|
| **CPU-only (small context)** | 256 GB RAM | Slow but works; ~4–8K context |
| **CPU-only (recommended)** | 384–512 GB RAM | 32–64K context feasible |
| **CPU + GPU hybrid** | 256 GB RAM + 24 GB VRAM | Offload some layers to GPU |
| **CPU + GPU hybrid (pro)** | 256 GB RAM + 48–96 GB VRAM | Best single-node balance |
| **Full GPU offload** | ~240 GB VRAM (aggregate) | e.g. 4× A100 80GB / 4× H100 80GB |
| **1M context** | 512 GB RAM + paged KV | llama.cpp `--flash-attn` + `--cache-type-k/v q8_0` |

> **Tip:** Use `--n-gpu-layers N` to tune offload. Start with `-ngl 99` and reduce until
> it fits. For hybrid setups, offloading **attention layers first** usually helps most.

---

## ⚡ Quickstart with llama.cpp

### 1. Build llama.cpp

```bash
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON      # or -DGGML_METAL=ON, -DGGML_VULKAN=ON, etc.
cmake --build build --config Release -j
```

### 2. Download the model

```bash
# using huggingface-cli
huggingface-cli download SHSLab/Step-5-Preview-GGUF-Q3_K_M-3bit \
  --local-dir ./Step-5-Preview-Q3_K_M

# or with git-lfs
git lfs install
git clone https://huggingface.co/SHSLab/Step-5-Preview-GGUF-Q3_K_M-3bit
```

### 3. Run text generation

```bash
./build/bin/llama-cli \
  -m ./Step-5-Preview-Q3_K_M/Step-5-Preview-Q3_K_M.gguf \
  -p "Explain the significance of the Pareto Frontier in AI scaling." \
  -n 512 \
  -ngl 99 \
  --temp 0.7 \
  --top-p 0.95 \
  --ctx-size 32768
```

### 4. Run with a chat template

```bash
./build/bin/llama-cli \
  -m ./Step-5-Preview-Q3_K_M/Step-5-Preview-Q3_K_M.gguf \
  --conversation \
  -ngl 99 \
  --ctx-size 32768
```

### 5. Control reasoning effort

Step-5-Preview supports `low` / `medium` / `high` / `xhigh`. Pass it via the system prompt
or as a template argument, depending on the chat template exposed by the GGUF:

```bash
./build/bin/llama-cli \
  -m ./Step-5-Preview-Q3_K_M/Step-5-Preview-Q3_K_M.gguf \
  -sys "You are a helpful assistant. Reasoning effort: high." \
  -p "Write a Python function to merge two sorted lists." \
  -ngl 99
```

---

## 👁️ Multimodal (mmproj) Usage

The vision encoder is shipped as a separate **BF16 mmproj** file. This is intentional —
the projector stays high-precision while the language model is 3-bit.

### Image input

```bash
./build/bin/llama-mtmd-cli \
  -m ./Step-5-Preview-Q3_K_M/Step-5-Preview-Q3_K_M.gguf \
  --mmproj ./Step-5-Preview-Q3_K_M/Step-5-Preview-mmproj-BF16.gguf \
  --image ./screenshot.png \
  -p "Describe this UI and suggest improvements." \
  -ngl 99
```

### Video input

```bash
./build/bin/llama-mtmd-cli \
  -m ./Step-5-Preview-Q3_K_M/Step-5-Preview-Q3_K_M.gguf \
  --mmproj ./Step-5-Preview-Q3_K_M/Step-5-Preview-mmproj-BF16.gguf \
  --video ./demo.mp4 \
  -p "Summarize what happens in this video." \
  -ngl 99
```

> **Limits:** MP4, QuickTime, Matroska. Recommended ≤128 MB and ≤5 minutes per clip.

---

## 🌐 Server Mode

`llama-server` exposes an OpenAI-compatible HTTP API.

### Text-only

```bash
./build/bin/llama-server \
  -m ./Step-5-Preview-Q3_K_M/Step-5-Preview-Q3_K_M.gguf \
  -ngl 99 \
  --ctx-size 32768 \
  --host 0.0.0.0 \
  --port 8080
```

### With vision

```bash
./build/bin/llama-server \
  -m ./Step-5-Preview-Q3_K_M/Step-5-Preview-Q3_K_M.gguf \
  --mmproj ./Step-5-Preview-Q3_K_M/Step-5-Preview-mmproj-BF16.gguf \
  -ngl 99 \
  --ctx-size 32768 \
  --host 0.0.0.0 \
  --port 8080
```

### Query it

```python
from openai import OpenAI

client = OpenAI(api_key="none", base_url="http://localhost:8080/v1")

response = client.chat.completions.create(
    model="step-5-preview",
    messages=[
        {"role": "user", "content": "Write a Python function to merge two sorted lists."}
    ],
    max_tokens=512,
    temperature=0.7,
)

print(response.choices[0].message.content)
```

---

## 🔧 Convert It Yourself

The conversion requires the `step5-llamacpp.patch` applied on top of llama.cpp.

### 1. Apply the patch

```bash
cd llama.cpp
git checkout ce8caa6e60a03093351d6016a818720e0d46f0fb
git apply -p1 /path/to/step5-llamacpp.patch
# or: patch -p1 < /path/to/step5-llamacpp.patch
```

### 2. Convert BF16 safetensors → BF16 GGUF

```bash
python convert_hf_to_gguf.py /path/to/Step5_safetensors \
  --outtype bf16 \
  --outfile Step-5-Preview-BF16.gguf
```

### 3. Convert the vision projector

```bash
python convert_hf_to_gguf.py /path/to/Step5_safetensors \
  --mmproj \
  --outtype bf16 \
  --outfile Step-5-Preview-mmproj-BF16.gguf
```

### 4. Quantize to Q3_K_M

```bash
./build/bin/llama-quantize \
  Step-5-Preview-BF16.gguf \
  Step-5-Preview-Q3_K_M.gguf \
  Q3_K_M
```

<details>
<summary><strong>🧩 What the patch changes</strong></summary>

| File | Change |
|:---|:---|
| `conversion/__init__.py` | Routes `Step4ForCausalLM` and `MMGPTStepRoboticsForCausalLM` into the `step3` converter (text + mmproj maps). |
| `conversion/base.py` | Maps the Step-5 tokenizer hash to the `deepseek-v3` pre-tokenizer. |
| `conversion/step3.py` | Adds `Step5Model` (text) and `Step5VisionModel` (mmproj) on the `STEP35` arch; per-layer `rope_theta` by `layer_type`; writes `rope.dimension_count` / `rope.dimension_count_swa`; drops sparse-GQA indexer tensors. |
| `src/models/step35.cpp` | Only halves `n_rot_full` when `rope.dimension_count` is absent. |

</details>

---

## ⚠️ Known Limitations

1. **No sparse attention in GGUF.** The CSA indexer + top-k selection is dropped during
   conversion. Affected layers run **dense attention** — correct but slower, and
   **long-context quality differs** from the BF16 reference.

2. **23 of 95 layers remain full attention.** Combined with the dense fallback, this means
   long-context performance is not equivalent to the reference model until the indexer is
   implemented in llama.cpp.

3. **3-bit quantization loss.** Q3_K_M introduces noticeable degradation on coding, math,
   and fine-grained reasoning. Expect lower scores than the BF16 benchmark table.

4. **MTP / NextN tensors** convert through but are only used if a draft model is configured.

5. **KV cache dominates memory at long context.** The weights are small; the 1M-token
   context window is not. See [Hardware Requirements](#-hardware-requirements).

6. **Vision is separate.** The mmproj must be loaded explicitly; it is not bundled inside
   the text GGUF.

7. **Language coverage.** Primarily optimized for English and Chinese.

---

## ⚡ Performance Notes

| Factor | Impact |
|:---|:---|
| **Quant type** | Q3_K_M is ~20% of BF16 size; expect slower per-token speed on CPU than Q4_K_M due to less efficient kernels |
| **GPU offload** | `-ngl 99` offloads everything if VRAM allows; reduce if not |
| **Flash attention** | Add `--flash-attn` for lower KV memory and faster long-context |
| **KV quantization** | `--cache-type-k q8_0 --cache-type-v q8_0` roughly halves KV memory |
| **Batch size** | Larger batches improve throughput but increase KV memory |
| **Context length** | Larger `--ctx-size` allocates more KV cache upfront |

**Recommended flags for long context:**

```bash
./build/bin/llama-cli \
  -m ./Step-5-Preview-Q3_K_M/Step-5-Preview-Q3_K_M.gguf \
  -ngl 99 \
  --ctx-size 131072 \
  --flash-attn \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  -p "Summarize this codebase."
```

---

## 📚 Citation

If you use Step-5-Preview in your research, please cite:

```bibtex
@misc{stepfun2026step5preview,
  title        = {Step-5-Preview: A 600B Sparse MoE Foundation Model for Real-World Agentic Work},
  author       = {StepFun Team},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/SHSLab/Step-5-Preview-BF16}},
  note         = {Released September 20, 2026}
}
```

If you use this GGUF build, please also reference the base checkpoint and llama.cpp.

---

## 📜 License

Step-5-Preview is released under the **StepFun Community License**.
See the [LICENSE](https://huggingface.co/SHSLab/Step-5-Preview-BF16/blob/main/LICENSE) file for full terms.

> **⚠️ Usage Restrictions**
>
> • Commercial use is permitted under the StepFun Community License.
> • Redistribution must include the license and attribution.
> • See LICENSE for full details.

---

## 📬 Contact

- **Hugging Face:** [SHSLab](https://huggingface.co/SHSLab)
- **GitHub:** [github.com/stepfun-ai](https://github.com/stepfun-ai)
- **Discord:** [Join our Discord](https://discord.gg/stepfun)
- **Email:** [opensource@stepfun.com](mailto:opensource@stepfun.com)
- **Website:** [stepfun.com](https://stepfun.com)

---

<div align="center">
  <strong>⭐ If you find Step-5-Preview useful, please give us a star on GitHub and Hugging Face! ⭐</strong>
  <br><br>
  <em>Built with ❤️ by StepFun</em>
</div>