smalinin/Step-5-Preview-GGUF

Step-5 Preview GGUF

Calibrated mixed-precision GGUF weights derived from the official StepFun FP8 checkpoint, retaining the CSA indexer and all three MTP heads.

Three calibrated mixed-precision recipes have been prepared:

Quantization	Publication status	Text/MTP files
Q2 Antirez-like	Uploaded	8 parts, 182.472 GB / 169.940 GiB
IQ3_M AntirezLike	Uploaded	8 parts, 271.609 GB / 252.956 GiB
Q2_K-Protected2	Completed locally; upload planned	8 parts, 229.670 GB / 213.897 GiB

The uploaded Q2 Antirez-like files use the shorter name IQ2_XXS-GU-Q2_K-Down in the Q2_a directory.

Q2_K-Protected2 has already been generated and passed local integrity checks; it will be uploaded later. All three recipes preserve higher precision for common matrices, CSA and MTP. Their exact tensor types are compared below; the recipe names do not imply a uniform bit width for every matrix.

Use the experimental smalinin/llama.cpp, my_build branch. CSA is the default for Step-5 in this runtime; Dense remains an explicit comparison mode. Complete-model inference has been developed and tested only with CUDA GPUs, including CPU expert offload.

Reviewed on 2026-10-19. The build instructions below pin the tested merge commit 425badfe8. This card does not imply support in an arbitrary upstream llama.cpp build.

Model background

StepFun's official model card describes a sparse MoE model with approximately 600B total parameters and 27B active parameters per token, designed for coding, reasoning and agent workflows. The source model advertises up to 1M context and multimodal capabilities. Local runtime validation and quantization quality are described separately below.

The official configuration specifies:

Component	Configuration
Main text layers	92
Hidden size	4096
Routed experts / selected experts	352 / 8 per token
Attention heads / KV heads	64 / 4
Attention head dimension	192
Main attention pattern	23 CSA layers and 69 sliding-window layers
Sliding window	512 tokens
Rotary dimensions	64 for full/CSA attention; 192 for sliding attention
Embedded MTP layers	3, following the main text layers

The GGUF architecture identifier is step35; the fork handles the Step-5-specific configuration and tensors.

How these weights were made

All three quantizations were generated afresh from the official StepFun FP8 checkpoint. We also created a new importance matrix (imatrix) locally, covering the main model and all three MTP heads.

1. Downloaded the official FP8 weights and converted them to retained BF16 GGUF intermediates, applying the FP8 block scales during dequantization. The intermediates preserve the precision of the FP8 source.
2. Generated a new imatrix from our local calibration corpus, with CSA enabled. The main-model calibration processed 262,144 tokens, in 32 chunks with context 8192. All three MTP heads were additionally calibrated using corpus tokens in teacher-forced mode. The combined imatrix contains 1,210 entries, including all 30 quantizable MTP matrices.
3. Collected those calibration activations using the existing Q3_K_M model. This calibration model supplied the activation statistics; the weights used to produce the new GGUF quantizations came from the official FP8-to-BF16 conversion described above.
4. Quantized the retained BF16 weights using our new imatrix and the precision rules below. All 1,783 text/MTP tensors were retained, including 184 CSA tensors and 51 tensors belonging to the three MTP heads. FP8 scale tensors were absorbed into the converted weights. The vision tower was exported separately as BF16 and Q8_0 mmproj files.
5. Verified tensor names, shapes, types, data validity and split completeness, including unchanged CSA payloads relative to the converted source. The completed quantization manifests record SHA-256 and size for every output part. All eight published Q2 Antirez-like parts match the completed local job by SHA-256 and size.

Calibration covered 99.893% of routed layer/expert pairs. The remaining 33 pairs received no activations and use the quantizer's neutral importance fallback. This measures calibration coverage; general model quality requires separate evaluation.

Precision distribution for the available and planned recipes

Excerpt from the verified quantization plans:

Tensor group	Q2 Antirez-like — uploaded	IQ3_M AntirezLike — uploaded	Q2_K-Protected2 — upload planned
Routed MoE gate/up	IQ2_XXS	IQ3_S	Q2_K
Routed MoE down	Q2_K	Native IQ3_M mix: IQ3_S / Q4_K	Q3_K
Attention, including attention gates	Q8_0	Q8_0	Q8_0
Shared experts gate/up	Q8_0	Q8_0	Q8_0
Shared experts down	Q8_0	Q8_0	Q8_0
Ordinary dense FFN gate/up	Q8_0	Q8_0	Q8_0
Ordinary dense FFN down	Q8_0	Q8_0	Q8_0
Main output matrix	Q8_0	Q8_0	Q8_0
Token embeddings	BF16	BF16	BF16
Quantizable matrices in all three MTP heads, including output heads	Q6_K	Q6_K	Q6_K
CSA indexer Q/K/Z projections	BF16	BF16	BF16
Other CSA parameters, ssmax_s, routers, norms and biases	F32	F32	F32

For IQ3_M AntirezLike, routed down matrices in layers 3–10 use Q4_K (8 matrices); layers 11–90 use IQ3_S (80 matrices). All 176 routed gate/up matrices use IQ3_S. IQ3_M names a mixed-precision quantization preset; there is no separate IQ3_M tensor type. In Q2 Antirez-like, all 176 routed gate/up matrices use IQ2_XXS and all 88 routed down matrices use Q2_K. In Q2_K-Protected2, the corresponding matrices use Q2_K and Q3_K, respectively.

The recipes share the same protection for attention, shared experts, ordinary dense FFNs, output, embeddings, CSA and MTP. Their routed expert precision differs as shown above.

The main-and-MTP importance matrix is available as imatrix-step5-q3-csa-with-mtp.gguf:

SHA256: 17794d941b2bee3239fe8db35fb893644828ac81ca7ee3f32d85801ff9977c65

The repository also contains a main-model-only imatrix. Use the with-MTP version when reproducing this recipe. An imatrix is calibration data, not a model or a draft sidecar.

Files and memory requirements

Keep all eight parts of the chosen recipe together. The published Q2 Antirez-like files are in the Q2_a directory:

Q2_a/Step-5-Preview-IQ2_XXS-GU-Q2_K-Down-00001-of-00008.gguf
Q2_a/Step-5-Preview-IQ2_XXS-GU-Q2_K-Down-00002-of-00008.gguf
Q2_a/Step-5-Preview-IQ2_XXS-GU-Q2_K-Down-00003-of-00008.gguf
Q2_a/Step-5-Preview-IQ2_XXS-GU-Q2_K-Down-00004-of-00008.gguf
Q2_a/Step-5-Preview-IQ2_XXS-GU-Q2_K-Down-00005-of-00008.gguf
Q2_a/Step-5-Preview-IQ2_XXS-GU-Q2_K-Down-00006-of-00008.gguf
Q2_a/Step-5-Preview-IQ2_XXS-GU-Q2_K-Down-00007-of-00008.gguf
Q2_a/Step-5-Preview-IQ2_XXS-GU-Q2_K-Down-00008-of-00008.gguf

Pass the first part to -m; llama.cpp discovers the others automatically. Do not mix parts from different quantizations or releases.

Download	Disk size
All eight Q2 Antirez-like text/MTP parts	182.472 GB / 169.940 GiB
All eight IQ3_M AntirezLike text/MTP parts	271.609 GB / 252.956 GiB
All eight Q2_K-Protected2 text/MTP parts — upload planned	229.670 GB / 213.897 GiB
Optional BF16 projector	4.398 GB
Optional Q8_0 projector	2.747 GB

Only the selected text quantization and, for images, one projector are needed for inference. The imatrix is not loaded by the server.

RAM and VRAM must also accommodate KV caches, compute buffers and backend overhead. Auto-fit chooses placement; it does not remove those memory requirements. CSA retains original token-level K/V and adds an indexer cache: sparse attention does not mean the full KV cache is discarded. For one slot at context 131072, microbatch 512 and three MTP heads, matching main/draft F16 caches use approximately 11.89 GiB of cache tensors; matching Q8_0 caches use approximately 7.75 GiB. Those figures exclude weights, projector and compute workspace.

The performance tests used an Intel Core Ultra 7 265K and six NVIDIA GPUs with approximately 264 GiB aggregate VRAM, with some weights on CPU. These are the test-machine specifications, not a minimum GPU count for Q2 Antirez-like.

Experimental runtime support

Feature	Status in the pinned runtime
Text generation and chat	Implemented; full-model checks on restored Q3_K_M and locally generated IQ3_M
Regional CSA	Default for Step-5: eight-token regions, four provider groups, top-512 historical regions per provider plus the current region
Main attention	Reads selected original token-level K/V using indexed tensor-core Flash Attention
Indexer numerics	BF16 boundaries and E4M3 rounding follow the reference contract; independent numerical checks passed
Attention scale	Reference 1 / sqrt(192); ssmax_s is retained but unused by the corrected path
Dense mode	Optional; bypasses regional selection while retaining the corrected attention scale
KV cache	Matching F16/F16 or Q8_0/Q8_0 supported for CSA; canonical indexer K/Z stay BF16
Embedded MTP	Three heads; adaptive draft length enabled by default; separate draft file is optional
Cache with MTP	Prefix reuse, state restore, changed suffixes and speculative rejection tested
Tool calling	Real required-tool call/result round trips tested with and without MTP
Images	Separate BF16/Q8_0 projector; CPU and CUDA vision tested; MTP stays active with images and retained image history
Long context	Restored Q3_K_M tested through 142,980 input tokens at context 144000; latest paired IQ3_M speed test used 59,999 input tokens
Video and general multimodal quality	Not validated

CSA follows StepFun's published reference; it remains experimental and full-model execution is not established as bit-exact to the official runtime. The tensor-core attention path uses F16 arithmetic internally. See the runtime documentation for the numerical contract and validation limits.

Build with CUDA

Install Git, CMake, a C/C++ toolchain, a compatible NVIDIA driver and a CUDA Toolkit with nvcc. See the branch's build documentation.

git clone --branch my_build --single-branch \
  https://github.com/smalinin/llama.cpp.git llama-step5
cd llama-step5

git checkout 425badfe8c1c286a8984a454e991f045bb3008b2

cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA=ON \
  -DGGML_CUDA_FA=ON \
  -DGGML_NATIVE=ON

cmake --build build --config Release -j 8 --target llama-server

GGML_NATIVE=ON targets the build machine's CPU. The validated Ampere/Ada build additionally used -DCMAKE_CUDA_ARCHITECTURES="86-real;89-real"; add that only for an appropriate GPU set. Adjust build parallelism to available RAM.

Download the model

From the llama-step5 directory, install the Hugging Face CLI in a virtual environment and download the eight Q2 parts:

python3 -m venv .venv-hf
source .venv-hf/bin/activate
python -m pip install -U huggingface_hub

hf download smalinin/Step-5-Preview-GGUF \
  --include 'Q2_a/*.gguf' \
  --local-dir ./models/Step-5-Preview-GGUF

The CLI preserves the Q2_a/ subdirectory; the server command below uses that exact path. For IQ3_M AntirezLike, after publication completes, change the download pattern to '*IQ3_M-*.gguf' and select its first part in the server command.

Run the OpenAI-compatible server

This example uses CSA, a 144000-token context allocation, Q8_0 main K/V, auto-fit and up to three embedded MTP draft tokens. Draft K/V retain their separate F16 default.

STEP5_MODEL='./models/Step-5-Preview-GGUF/Q2_a/Step-5-Preview-IQ2_XXS-GU-Q2_K-Down-00001-of-00008.gguf'
STEP5_CTX=144000

./build/bin/llama-server \
  -m "$STEP5_MODEL" --alias step5 \
  -c "$STEP5_CTX" -b 2048 -ub 512 \
  --fit on --fit-target 1024 --fit-ctx "$STEP5_CTX" \
  --load-mode none \
  -t 20 -tb 20 -fa on \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --no-context-shift --no-warmup --no-mmproj \
  --parallel 1 --host 127.0.0.1 --port 8085 \
  --jinja --chat-template-kwargs '{"reasoning_effort":"high"}' \
  --temp 1.0 --top-p 0.95 --top-k 0 \
  --spec-type draft-mtp --spec-draft-n-max 3

Wait for model loaded and listening. The web UI is at http://127.0.0.1:8085; the API base is http://127.0.0.1:8085/v1.

• --fit on chooses GPU/CPU placement. --fit-target 1024 requests a 1024 MiB device margin; --fit-ctx prevents auto-fit from shortening the context below the specified value. Re-run fitting after changing context, microbatch, cache types, visible-device order or projector placement.
• Adjust -t 20 -tb 20 to the CPU. --load-mode none eagerly loads CPU-resident weights.
• --spec-draft-n-max 3 is a maximum; existing adaptive MTP may draft fewer tokens. Set LLAMA_MTP_ADAPTIVE=0 before launch for a fixed draft budget. For target-only decoding, replace both speculation options with --spec-type none --spec-draft-n-max 0. Pass --spec-type once.
• MTP accelerates generation; prompt prefill still runs the full target model and prepares draft state.
• For F16 main K/V, replace both q8_0 cache types with f16. To quantize draft K/V too, additionally pass --spec-draft-type-k q8_0 --spec-draft-type-v q8_0.
• For Dense, set LLAMA_STEP5_EXPERIMENTAL_DENSE=1 before launch and leave the explicit CSA flag unset.
• Prompt plus generated output must fit inside the selected context. The advertised 1M context has not been validated in this fork.

The sampling settings follow StepFun's recommendations. reasoning_effort is passed through the embedded chat template; this example uses high.

Make a chat request

curl http://127.0.0.1:8085/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "step5",
    "messages": [
      {"role": "user", "content": "Explain why binary search takes O(log n) time."}
    ],
    "temperature": 1.0,
    "top_p": 0.95,
    "top_k": 0,
    "max_tokens": 2048,
    "stream": false
  }'

For tools, send OpenAI-style tools definitions. The client executes returned tool calls and sends their results back as role: "tool" messages. Required-tool round trips were tested in target-only and MTP modes. Parallel tool-use reliability and strict JSON-schema compliance on arbitrary tasks have not been established.

Optional images: CPU or CUDA vision

Download one projector into the same model directory, for example:

hf download smalinin/Step-5-Preview-GGUF \
  mmproj-Step-5-Preview-Q8_0.gguf \
  --local-dir ./models/Step-5-Preview-GGUF

For CPU vision, replace --no-mmproj in the server command with:

--mmproj ./models/Step-5-Preview-GGUF/mmproj-Step-5-Preview-Q8_0.gguf \
--no-mmproj-offload

For CUDA vision, use this replacement instead:

--mmproj ./models/Step-5-Preview-GGUF/mmproj-Step-5-Preview-Q8_0.gguf

MTP remains enabled for image requests and follow-ups retaining images in their history. It speeds up text generation, not vision encoding. Local checks included synthetic colors/shapes, rendered-digit OCR and image-history cache reuse; they are not a general image/OCR/video evaluation.

Validation and measured performance

All three recipes — Q2 Antirez-like, IQ3_M AntirezLike and Q2_K-Protected2 — passed local conversion and quantization integrity checks. There is no broad Q2 quality comparison against the official FP8/BF16 checkpoint and no controlled Q2 throughput result in this card.

Runtime checks cover the independent CSA numerical oracle, CUDA indexed and Dense attention, F16/Q8_0 KV, all three MTP heads, speculative rollback, cache serialization and prefix reuse. The pinned my_build merge passed compilation and CPU/Ada/Ampere regression checks. Earlier restored Q3_K_M checks reached 142,980 input tokens with all eight archive facts retrieved. These finite checks do not establish general long-context accuracy.

IQ3_M-AntirezLike measurement

The following figures are for IQ3_M, whose publication is in progress. These are measurements of the completed local files and are not throughput results for Q2 Antirez-like.

Measured at 66176617d, before the my_build merge: 59,999 input tokens, context 144000, batch 2048, microbatch 512, Q8_0 main K/V, F16 draft K/V, fixed MTP 3, 20 CPU threads, Q8_0 projector on CUDA0. The requests contained no images. Both modes independently ran auto-fit with margins 1024,512,512,512,512,512 MiB and the GPU order 2,1,0,5,4,3.

Mode	Prefill, tokens/s, two runs	Generation, tokens/s, two runs
Dense	257.84 / 262.58	30.40 / 29.84
CSA	244.64 / 244.23	31.33 / 31.32

Prefill used uncached retrieval; generation used a separate cached follow-up producing 239 counting tokens. All eight facts were correct, counting was complete, and output token IDs matched between modes. Dense averaged 260.21 prefill tokens/s versus CSA's 244.44; CSA averaged 31.32 generation tokens/s versus Dense's 30.12. Draft acceptance and CPU weight placement differed. These measurements compare complete configurations and do not isolate attention kernels. They were not repeated on the merged commit or on Q2.

See the runtime performance notes for earlier fixtures, memory accounting and measurement limitations.

Limitations

• Quantization quality has not been comprehensively evaluated against the official FP8/BF16 checkpoint. Calibration used an existing Q3_K_M model, and a BF16 intermediate cannot recover precision lost in the FP8 source.
• CSA is implemented and enabled by default, but full-model numerical equivalence to the official runtime and the advertised 1M context are unverified. Dense uses all available keys instead of CSA selection and changes model behavior.
• Complete-model execution has only been validated with CUDA and CPU offload. ROCm, Vulkan, Metal, SYCL and complete CPU-only inference are unvalidated.
• Synthetic retrieval, image and tool tests do not establish general long-context, multimodal, agent or strict-schema reliability.

Samples

These are illustrative local outputs, not comparative benchmarks or tests of CSA correctness.

Q2 Antirez-like (IQ2_XXS-GU-Q2_K-Down)

Prompt:

create html with svg "pelican on a bicycle"

IQ3_M AntirezLike — publication in progress

The following examples were generated with the completed IQ3_M AntirezLike files that are being uploaded.

Prompt:

create html with svg "pelican on a bicycle"

Using OpenCode with tools and vision, prompt:

Indian Railways WDP-4D dual-cab diesel locomotive using Three.js

Attribution and license

• StepFun: official FP8 source weights, original BF16 vision weights, model configuration and reference implementation.
• TypeSafeAI and NeuroSenko: the earlier repaired Q3_K_M model used for activation calibration.
• ggml-org/llama.cpp: underlying inference engine and GGUF quantization tools.
• smalinin/llama.cpp, my_build: experimental Step-5 integration and runtime optimizations, merged from my_step5.

The official source weights are distributed under the StepFun Model License 1.0. Consult that license for the terms governing these derived weights. The runtime has its separate llama.cpp license.