txgsync's picture
|
download
raw
4.44 kB

Make a model folder

This guide makes a seed model folder from the original K2-Horizon-MoVA-36B-A4B. SNAPSHOT is the local Hugging Face snapshot of IFM/K2-Horizon-MoVA-36B-A4B. The prepared models are ready to use, so you do not need this guide to run a model.

GQA model with seeded experts

  1. Collect activation statistics, from about 160,000 tokens of plain text:

    OMLX_K2_MODEL=/path/to/omlx/patches/k2_horizon/k2_horizon_model.py python tools/mova_capture.py --text calib.txt --out out/actsq_moe.bin
    
  2. Search the seeds. This step takes about 3.4 hours on one M4 Max. The CUDA build runs the same search.

    out/bin/nslm-moe --model SNAPSHOT --act out/actsq_moe.bin --out out/blocks --scope gud --p4 --workers 2
    
  3. Pack the folder. The packer writes safetensors shards of at most 4.5 GB, the index, config.json, the tokenizer files and the MLX loader.

    out/bin/nslm-mova-pack --model SNAPSHOT --config p4mx --blk4 out/blocks --q4 v --out MODEL_DIR
    

Qwen3.8-Flash-Next experts

The search reads the fused expert tensors of the BF16 checkpoint. The Unsloth imatrix gives the activation weights, so no capture run is necessary.

out/bin/nslm-moe --model SNAPSHOT --imatrix imatrix_unsloth.gguf_file --out out/blocks --scope gud --p4

On HF Jobs, tools/hf_seedbank.sh takes IMATRIX=unsloth/Qwen3.8-Flash-Next-GGUF:imatrix_unsloth.gguf_file and ONLY=mlp.experts.. With ONLY, the job downloads only the shards that hold experts.

The release searches used these commands. ./src is a copy of the repository (git archive HEAD). The bucket keeps the block files. JOBS=moe3 gives P = 3, and JOBS=moe4 gives P = 4.

hf jobs run --detach --flavor rtx-pro-6000x4 --timeout 95m -v ./src:/src -v hf://buckets/USER/BUCKET:/out \
    -e SRC=/src -e MODEL=Qwen/Qwen3.8-Flash-Next@de4b8e4d43b917e7706784d8bb445c9af86a3540 -e ONLY=mlp.experts. \
    -e IMATRIX=unsloth/Qwen3.8-Flash-Next-GGUF:imatrix_unsloth.gguf_file -e OUT=/out/p4 -e JOBS=moe4 -e WORKERS=2 \
    nvidia/cuda:12.8.1-devel-ubuntu24.04 bash /src/tools/hf_seedbank.sh

To pack, start from an MLX folder of the model (mlx-community/Qwen3.8-Flash-Next-4bit-mtp). The packer copies its tensors and replaces the routed experts with seeds. --blk takes P = 3 blocks, and --blk4 takes P = 4 blocks.

out/bin/nslm-qwen4-pack --base MLX_DIR --blk4 out/blocks --out MODEL_DIR

On HF Jobs, tools/hf_qwen4_pack.sh packs on a CPU node, from and to mounted buckets:

hf jobs run --detach --flavor cpu-xl --timeout 2h -v ./src:/src -v hf://buckets/USER/BUCKET:/b \
    -e SRC=/src -e BASE=mlx-community/Qwen3.8-Flash-Next-4bit-mtp@825df623302aeebc434f44ca9122827b04f9a0f1 \
    -e BLK=/b/p4/experts-p4 -e TIER=4 -e OUT=/b/pack-p4 python:3.12 bash /src/tools/hf_qwen4_pack.sh

From Git Bash on Windows, set MSYS_NO_PATHCONV=1 first. Otherwise Git Bash changes the job's paths (/b/... to B:/...) and the job stops at once.

MLA projections

An MLA model folder also has model-mla-delta.safetensors. Read The model folder format. nslm-mova-pack --mla sets the encoding of the MLA projections and the per-head maps:

Option Encoding
(none) BF16
--mla q8, --mla q4 Affine Q8 or Q4
--mla p4 P = 4 seeds, 4.5 bits per weight. v_up stays in Q8 by default
--mla-q8 NAMES Keep some tensors in Q8. NAME@A-B limits it to layers A to B. --mla-q8 none makes all MLA tensors seeds

To make P = 4 seeds for the MLA projections:

  1. Capture the sums of squares of each projection input over the calibration text:

    out/bin/nslm-mova-mlacapture --model MLA_MODEL_DIR --text calib.txt --out out/actsq_mla.bin
    

    --xtx DIR also writes the X^T X matrix of each input, for GPTQ.

  2. Search each projection. The search does the per-head maps one head at a time.

    • Activation-weighted: nslm-moe --scope mla --p4. All 472 M weights take 536 s on an M4 Max.
    • GPTQ: nslm-moe --scope mla --p4 --xtx DIR. This takes 23 minutes on an M4 Max.
  3. Pack with nslm-mova-pack --mla p4 --mla-blk4 BLOCKS_DIR.

Read MLA projections in BF16, Q8, Q4 and seeds for the quality of each choice.

See also

Xet Storage Details

Size:
4.44 kB
·
Xet hash:
b72aa5dcb274b202a86da7a88a2e072b3c13c5206faa3226c21d3fca26ed9059

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.