txgsync/nslm-qwen38/src-e4735ad - Storage Bucket

NanoSeedLM

NanoSeedLM runs mixture-of-experts language models in less memory. It uses two methods:

• SeedLM replaces stored weights with seeds. The GPU makes the weights from the seeds at run time.
• TransMLA converts the model's attention to multi-head latent attention (MLA). The MLA model keeps less than half the KV cache of the original. Only the K2-Horizon-MoVA models use it.

The project has three parts:

• An inference engine in C99, with Metal kernels for Apple Silicon and CUDA kernels for NVIDIA GPUs.
• An MLX loader in each Hugging Face model repository, for use in oMLX or mlx-vlm.
• The tools that make the seeds and pack the models.

Supported models

Model family	Methods	oMLX or MLX	Metal engine	CUDA engine
IFM/K2-Horizon-MoVA-36B-A4B	Seeds, MLA	oMLX	Yes	Yes (p4mx-q4v; the MLA packs are not checked)
Qwen/Qwen3.8-Flash-Next	Seed experts	mlx-vlm (oMLX not yet)	Not yet	Yes (nslm-qwen4-gen only)
IFM/K2-Horizon-0.9B	Seeds	oMLX	No	No

The model list gives each published model and the runtimes that load it.

Should you use it?

For daily use, probably not. A seed model is not faster than an affine quantization (Q4, Q8) of the same model, and its quality is not better. A seed model is often a little smaller than an affine model of similar quality. That is the gain.

If you want the fastest or the most accurate version of a model on your machine, use an affine quantization of the original model. Use NanoSeedLM for one of these purposes:

• To get the longest K2-Horizon-MoVA context that your Mac can hold.
• To get the smallest model file at a given quality.
• To do research on seeded weights.

Highlights

• Seeds instead of stored weights. The model file keeps each block of 8 weights as a 16-bit seed and a few small numbers. The GPU makes the weights again from the seed inside each matrix multiply.
• A smaller KV cache. The K2-Horizon-MoVA MLA models keep less than half the KV cache of the original model. With the 8-bit cache, the full 512K-token context fits on a 128 GB Mac, and much memory stays free.
• Experts in host memory. The CUDA engine can keep the routed experts in RAM. A VRAM cache holds the recently used experts. Qwen3.8-Flash-Next runs on a GPU with 16 GB of VRAM.
• No patches in oMLX or mlx-vlm. Each model repository includes its own loader. Select Trust Remote Code for the model. The loader supplies the seed kernels and the MLA attention.
• The same engine on Metal and CUDA. The two engines compile the same C sources and run the same tests. Each GPU kernel must match a C reference. The two engines give the same quality on the same model.

The trade-offs

You get	You give
A model file a little smaller than an affine model of similar quality	Slower decode, because the GPU computes the weights
A KV cache less than half the size (MLA models)	Some quality, because the MLA conversion moves the model away from the original
Long contexts on smaller machines	Slower processing of long prompts, because MLA attention does more arithmetic

The MLA models also repeat themselves more often than the original in long generations. Set a limit on the output length. A small number of texts cause most of the MLA quality loss, and we do not know why yet. The measurements are in the documentation.

Get started

In oMLX (Mac, K2-Horizon)

1. Download a model, for example K2-Horizon-MoVA-36B-A4B-MLA-NanoSeedLM-P4.
2. In oMLX, select Trust Remote Code for the model.

For long contexts, read The MLX loader.

With the engine (Mac, K2-Horizon)

You need Apple Silicon and the Xcode command-line tools.

make
out/bin/nslm-chat --model MODEL_DIR "Why do tide pools matter?"
out/bin/nslm-serve --model MODEL_DIR --port 8080

nslm-serve is an OpenAI-compatible server. It supports streaming, tool calls and continuous batching. Read Run and serve.

In mlx-vlm (Mac, Qwen3.8-Flash-Next)

You need mlx-vlm 0.7.12 or later. The model card tells how to keep the PLE table on the SSD. Read The MLX loader.

python -m mlx_vlm.generate --model txgsync/Qwen3.8-Flash-Next-NanoSeedLM-P4 --trust-remote-code --prompt "Why do tide pools matter?"

With the engine (Windows or Linux)

You need an NVIDIA GPU of the Ampere generation or newer. K2-Horizon-MoVA needs 12 GB or more of VRAM. Qwen3.8-Flash-Next runs on a 16 GB GPU. Its routed experts stay in RAM, or the engine reads them from the model files (--stream-experts). Read Build and test and NVIDIA GPUs.

build/bin/nslm-qwen4-gen --model Qwen3.8-Flash-Next-NanoSeedLM-P4 --expert-cache-gb 8 --prompt "Why do tide pools matter?"

nslm-chat and nslm-serve do not run Qwen3.8-Flash-Next yet. Read Run and serve.

Models

Model	Attention	Weights	Size
K2-Horizon-MoVA-36B-A4B-NSLM-p4mx-q4v	GQA (original)	Seeds and affine	22.8 GB
K2-Horizon-MoVA-36B-A4B-MLA-NanoSeedLM-P4	MLA	Seeds	22.2 GB
K2-Horizon-MoVA-36B-A4B-MLA-NanoSeedLM-P3	MLA	Seeds	20.1 GB
K2-Horizon-MoVA-36B-A4B-MLA	MLA	BF16	75.6 GB
K2-Horizon-MoVA-36B-A4B-MLA-oQ4e, oQ6e, oQ8e	MLA	Affine, for comparison	22.9 to 40.6 GB
K2-Horizon-0.9B-NSLM-seeds-4bpw	GQA	Seeds	0.5 GB
Qwen3.8-Flash-Next-NanoSeedLM-P4	Original (DeltaNet and attention)	Seed experts, the rest 4-bit	105.7 GB
Qwen3.8-Flash-Next-NanoSeedLM-P3	Original (DeltaNet and attention)	Seed experts, the rest 4-bit	98.2 GB

How it works

SeedLM. A linear feedback shift register (LFSR) makes a sequence of numbers from a 16-bit seed. For each block of weights, a search finds the seed and the coefficients that come nearest to the original weights. The model file keeps only the seed and the coefficients. The SeedLM paper made the weights on an FPGA. NanoSeedLM makes them in GPU kernels. Read SeedLM.

MLA. The original model caches a key and a value for each group of attention heads and each token. The MLA model caches one small latent vector and one position key for each token. Attention reads the keys and values through the latent. Read Multi-head latent attention and The KV cache.

Parity. The Metal and CUDA engines share their C code, their tests and their arithmetic. Read Metal and CUDA parity.

Documentation

The documentation starts at docs/index.md. The experiment record is docs/log.md, in date order. AI agents can start at llms.txt.

Related work

• SeedLM, Shafipour et al., ICLR 2025. The method.
• TransMLA, Meng, Yao and Zhang, 2025. The conversion from GQA to MLA.
• More related work.

License

MIT. See LICENSE. The model weights have their own licenses.