Buckets:
| # NanoSeedLM | |
| NanoSeedLM runs mixture-of-experts language models in less memory. It uses two methods: | |
| - [SeedLM](https://arxiv.org/abs/2410.10714) replaces stored weights with seeds. The GPU makes the weights from the | |
| seeds at run time. | |
| - [TransMLA](https://arxiv.org/abs/2502.07864) converts the model's attention to multi-head latent attention (MLA). | |
| The MLA model keeps less than half the KV cache of the original. Only the K2-Horizon-MoVA models use it. | |
| The project has three parts: | |
| - An inference engine in C99, with Metal kernels for Apple Silicon and CUDA kernels for NVIDIA GPUs. | |
| - An MLX loader in each [Hugging Face model repository](docs/reference/models.md), for use in | |
| [oMLX](https://github.com/jundot/omlx) or mlx-vlm. | |
| - The tools that make the seeds and pack the models. | |
| ## Supported models | |
| | Model family | Methods | oMLX or MLX | Metal engine | CUDA engine | | |
| |---|---|---|---|---| | |
| | [IFM/K2-Horizon-MoVA-36B-A4B](https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B) | Seeds, MLA | oMLX | Yes | Yes (p4mx-q4v; the MLA packs are not checked) | | |
| | [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) | Seed experts | mlx-vlm (oMLX not yet) | Not yet | Yes (`nslm-qwen4-gen` only) | | |
| | [IFM/K2-Horizon-0.9B](https://huggingface.co/IFM/K2-Horizon-0.9B) | Seeds | oMLX | No | No | | |
| [The model list](docs/reference/models.md) gives each published model and the runtimes that load it. | |
| ## Should you use it? | |
| For daily use, probably not. A seed model is not faster than an affine quantization (Q4, Q8) of the same model, and | |
| its quality is not better. A seed model is often a little smaller than an affine model of similar quality. That is | |
| the gain. | |
| If you want the fastest or the most accurate version of a model on your machine, use an affine quantization of the | |
| original model. Use NanoSeedLM for one of these purposes: | |
| - To get the longest K2-Horizon-MoVA context that your Mac can hold. | |
| - To get the smallest model file at a given quality. | |
| - To do research on seeded weights. | |
| ## Highlights | |
| - **Seeds instead of stored weights.** The model file keeps each block of 8 weights as a 16-bit seed and a few small | |
| numbers. The GPU makes the weights again from the seed inside each matrix multiply. | |
| - **A smaller KV cache.** The K2-Horizon-MoVA MLA models keep less than half the KV cache of the original model. With | |
| the 8-bit cache, the full 512K-token context fits on a 128 GB Mac, and much memory stays free. | |
| - **Experts in host memory.** The CUDA engine can keep the routed experts in RAM. A VRAM cache holds the recently used | |
| experts. Qwen3.8-Flash-Next runs on a GPU with 16 GB of VRAM. | |
| - **No patches in oMLX or mlx-vlm.** Each model repository includes its own loader. Select **Trust Remote Code** for | |
| the model. The loader supplies the seed kernels and the MLA attention. | |
| - **The same engine on Metal and CUDA.** The two engines compile the same C sources and run the same tests. Each GPU | |
| kernel must match a C reference. The two engines give the same quality on the same model. | |
| ## The trade-offs | |
| | You get | You give | | |
| |---|---| | |
| | A model file a little smaller than an affine model of similar quality | Slower decode, because the GPU computes the weights | | |
| | A KV cache less than half the size (MLA models) | Some quality, because the MLA conversion moves the model away from the original | | |
| | Long contexts on smaller machines | Slower processing of long prompts, because MLA attention does more arithmetic | | |
| The MLA models also repeat themselves more often than the original in long generations. Set a limit on the output | |
| length. A small number of texts cause most of the MLA quality loss, and we do not know why yet. The measurements are | |
| in [the documentation](docs/index.md). | |
| ## Get started | |
| ### In oMLX (Mac, K2-Horizon) | |
| 1. Download a model, for example | |
| [K2-Horizon-MoVA-36B-A4B-MLA-NanoSeedLM-P4](https://huggingface.co/txgsync/K2-Horizon-MoVA-36B-A4B-MLA-NanoSeedLM-P4). | |
| 2. In oMLX, select **Trust Remote Code** for the model. | |
| For long contexts, read [The MLX loader](docs/guides/mlx.md). | |
| ### With the engine (Mac, K2-Horizon) | |
| You need Apple Silicon and the Xcode command-line tools. | |
| ```bash | |
| make | |
| out/bin/nslm-chat --model MODEL_DIR "Why do tide pools matter?" | |
| out/bin/nslm-serve --model MODEL_DIR --port 8080 | |
| ``` | |
| `nslm-serve` is an OpenAI-compatible server. It supports streaming, tool calls and continuous batching. Read | |
| [Run and serve](docs/guides/run-and-serve.md). | |
| ### In mlx-vlm (Mac, Qwen3.8-Flash-Next) | |
| You need mlx-vlm 0.7.12 or later. The model card tells how to keep the PLE table on the SSD. Read | |
| [The MLX loader](docs/guides/mlx.md). | |
| ```bash | |
| python -m mlx_vlm.generate --model txgsync/Qwen3.8-Flash-Next-NanoSeedLM-P4 --trust-remote-code --prompt "Why do tide pools matter?" | |
| ``` | |
| ### With the engine (Windows or Linux) | |
| You need an NVIDIA GPU of the Ampere generation or newer. K2-Horizon-MoVA needs 12 GB or more of VRAM. | |
| Qwen3.8-Flash-Next runs on a 16 GB GPU. Its routed experts stay in RAM, or the engine reads them from the model files | |
| (`--stream-experts`). Read [Build and test](docs/guides/build.md) and [NVIDIA GPUs](docs/guides/cuda.md). | |
| ```bash | |
| build/bin/nslm-qwen4-gen --model Qwen3.8-Flash-Next-NanoSeedLM-P4 --expert-cache-gb 8 --prompt "Why do tide pools matter?" | |
| ``` | |
| `nslm-chat` and `nslm-serve` do not run Qwen3.8-Flash-Next yet. Read | |
| [Run and serve](docs/guides/run-and-serve.md#qwen38-flash-next-on-cuda). | |
| ## Models | |
| | Model | Attention | Weights | Size | | |
| |---|---|---|---| | |
| | [K2-Horizon-MoVA-36B-A4B-NSLM-p4mx-q4v](https://huggingface.co/txgsync/K2-Horizon-MoVA-36B-A4B-NSLM-p4mx-q4v) | GQA (original) | Seeds and affine | 22.8 GB | | |
| | [K2-Horizon-MoVA-36B-A4B-MLA-NanoSeedLM-P4](https://huggingface.co/txgsync/K2-Horizon-MoVA-36B-A4B-MLA-NanoSeedLM-P4) | MLA | Seeds | 22.2 GB | | |
| | [K2-Horizon-MoVA-36B-A4B-MLA-NanoSeedLM-P3](https://huggingface.co/txgsync/K2-Horizon-MoVA-36B-A4B-MLA-NanoSeedLM-P3) | MLA | Seeds | 20.1 GB | | |
| | [K2-Horizon-MoVA-36B-A4B-MLA](https://huggingface.co/txgsync/K2-Horizon-MoVA-36B-A4B-MLA) | MLA | BF16 | 75.6 GB | | |
| | K2-Horizon-MoVA-36B-A4B-MLA-[oQ4e](https://huggingface.co/txgsync/K2-Horizon-MoVA-36B-A4B-MLA-oQ4e), [oQ6e](https://huggingface.co/txgsync/K2-Horizon-MoVA-36B-A4B-MLA-oQ6e), [oQ8e](https://huggingface.co/txgsync/K2-Horizon-MoVA-36B-A4B-MLA-oQ8e) | MLA | Affine, for comparison | 22.9 to 40.6 GB | | |
| | [K2-Horizon-0.9B-NSLM-seeds-4bpw](https://huggingface.co/txgsync/K2-Horizon-0.9B-NSLM-seeds-4bpw) | GQA | Seeds | 0.5 GB | | |
| | [Qwen3.8-Flash-Next-NanoSeedLM-P4](https://huggingface.co/txgsync/Qwen3.8-Flash-Next-NanoSeedLM-P4) | Original (DeltaNet and attention) | Seed experts, the rest 4-bit | 105.7 GB | | |
| | [Qwen3.8-Flash-Next-NanoSeedLM-P3](https://huggingface.co/txgsync/Qwen3.8-Flash-Next-NanoSeedLM-P3) | Original (DeltaNet and attention) | Seed experts, the rest 4-bit | 98.2 GB | | |
| ## How it works | |
| **SeedLM.** A linear feedback shift register (LFSR) makes a sequence of numbers from a 16-bit seed. For each block of | |
| weights, a search finds the seed and the coefficients that come nearest to the original weights. The model file keeps | |
| only the seed and the coefficients. The SeedLM paper made the weights on an FPGA. NanoSeedLM makes them in GPU | |
| kernels. Read [SeedLM](docs/concepts/seedlm.md). | |
| **MLA.** The original model caches a key and a value for each group of attention heads and each token. The MLA model | |
| caches one small latent vector and one position key for each token. Attention reads the keys and values through the | |
| latent. Read [Multi-head latent attention](docs/concepts/mla.md) and [The KV cache](docs/concepts/kv-cache.md). | |
| **Parity.** The Metal and CUDA engines share their C code, their tests and their arithmetic. Read | |
| [Metal and CUDA parity](docs/concepts/parity.md). | |
| ## Documentation | |
| The documentation starts at [docs/index.md](docs/index.md). The experiment record is [docs/log.md](docs/log.md), in | |
| date order. AI agents can start at [llms.txt](llms.txt). | |
| ## Related work | |
| - [SeedLM](https://arxiv.org/abs/2410.10714), Shafipour et al., ICLR 2025. The method. | |
| - [TransMLA](https://arxiv.org/abs/2502.07864), Meng, Yao and Zhang, 2025. The conversion from GQA to MLA. | |
| - [More related work](docs/reference/related-work.md). | |
| ## License | |
| MIT. See `LICENSE`. The model weights have their own licenses. | |
Xet Storage Details
- Size:
- 8.29 kB
- Xet hash:
- 55044361a666bf3d0b392ccd75db9a14e583b2842109b160947fbe535912616a
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.