|
Download README.md from Yougori/Ling-mini-2.0-Megakernel: direct link, hf CLI and curl.
- Browser
- Download file 8.5 kB
-
https://huggingface.co/Yougori/Ling-mini-2.0-Megakernel/resolve/main/README.md
- Command line
-
hf download hf://Yougori/Ling-mini-2.0-Megakernel/README.md
-
curl -L -o README.md https://huggingface.co/Yougori/Ling-mini-2.0-Megakernel/resolve/main/README.md
8.5 kB
| license: apache-2.0 | |
| base_model: inclusionAI/Ling-mini-2.0 | |
| base_model_relation: quantized | |
| pipeline_tag: text-generation | |
| language: | |
| - en | |
| tags: | |
| - megakernel | |
| - cuda | |
| - mixture-of-experts | |
| - 4-bit | |
| - fast-inference | |
| - ling | |
| # Ling-mini-2.0 Megakernel | |
| [Ling-mini-2.0](https://huggingface.co/inclusionAI/Ling-mini-2.0) (a 16B mixture of experts with | |
| 1.4B parameters active per token) at 4 bits, running in a CUDA megakernel: one GPU launch runs every | |
| layer of up to 256 tokens and chooses each next token itself. | |
| On one NVIDIA GeForce RTX 5090 Laptop GPU it writes 843 tokens per second in one chat, 1.68 times | |
| llama.cpp on the same model and GPU. With three chats at once it writes 1,337 tokens per second in | |
| total. The design and its measurements are described in the paper, `megakernel.pdf`. | |
|  | |
| ## Speed | |
| | Ling-mini-2.0, RTX 5090 Laptop GPU | llama.cpp | Megakernel | | | |
| |:--|--:|--:|--:| | |
| | One chat | 502 | **843** | 1.68× | | |
| | Two chats at once, in total | 784 | **1,096** | 1.40× | | |
| | Three chats at once, in total | 992 | **1,337** | 1.35× | | |
| | Four chats at once, in total | 1,164 | **1,371** | 1.18× | | |
| | Five chats at once, in total | 1,320 | **1,403** | 1.06× | | |
| Tokens per second while writing, greedy decoding, median of warm runs, measured on 6 November 2026 | |
| with the laptop GPU at its 175 W power limit. Every chat reads a 32-token prompt and then writes 256 | |
| tokens. llama.cpp is the CUDA build b12144 with bartowski's Q4_K_M file and flash attention on: | |
| `llama-bench` for one chat, `llama-batched-bench` for several. At eight chats llama.cpp reaches | |
| 1,616 tokens per second in total, because its batch shares each weight read between the chats; our | |
| chats do not share reads, and the provided apps run at most five at once. | |
|  | |
| ## What is different | |
| A normal runtime runs each token through the model as hundreds of small GPU programs, one after | |
| another, and the GPU sits partly idle between them. This kernel is one program that never stops | |
| between layers or tokens. | |
| 1. One persistent block of threads sits on every streaming multiprocessor (SM) for the whole launch. | |
| When a step needs the result of the step before, the blocks wait at a grid barrier instead of | |
| ending and starting again. | |
| 2. Steps are merged so that each layer waits only four times. The router's sums are added inside | |
| the attention output step, and the next layer's q, k and v inside the expert down step. This | |
| works because RMS normalization divides each token's vector by a single number: every block | |
| adds its own rows' share at once, and the division follows after the wait. | |
| 3. The router picks its 8 experts (and the shared one) inside the launch, and the kernel picks the | |
| next token itself, so nothing returns to the CPU between tokens. | |
| 4. The word table's 2,048 hidden dimensions are stored in order of importance. The kernel reads the | |
| important half of every row, then reads in full only the 4 best rows of each warp. It picks the | |
| same word as the full table at 99 to 99.5% of positions; where it differs, the two words were | |
| near ties. | |
| 5. Several chats run as copies of the kernel side by side, each on its own share of the SMs, with | |
| one copy of the model in memory. One chat leaves the GPU waiting at its barriers a good part of | |
| the time, and the other chats fill those waits. | |
| ## Quality | |
| All attention, MLP, expert and word-table matrices are rounded to 4 bits with one fp16 scale per 64 | |
| numbers (round to nearest with a per-group clip search). The router, its bias, the normalization | |
| gains and the input embeddings keep their original values. | |
| | 256 positions each | Web text | Python code | | |
| |:--|--:|--:| | |
| | Same top token as the same 4-bit weights run in fp32 | 98.8% | 98.8% | | |
| | KL to the same 4-bit weights in fp32 | 0.0017 | 0.0017 | | |
| | Same top token as the original bf16 model | 74.6% | 91.0% | | |
| | KL(original ∥ kernel) per token | 0.222 | 0.070 | | |
| | Perplexity, original bf16 model | 41.4 | 6.33 | | |
| | Perplexity, this kernel | 45.7 | 6.83 | | |
| The first two rows show that the kernel computes what its weights say. The rest is the cost of plain | |
| 4-bit rounding. llama.cpp's K-quants keep more of the original model; its quality was not measured | |
| here. In our chat tests the model answered simple questions and wrote correct short code. | |
| ## Requirements | |
| An NVIDIA GeForce RTX 50-series GPU (Blackwell, sm_120) with about 10 GB of free memory for one chat | |
| (12 GB for the several-chats app). Windows x64: `megakernel_ling.dll` is included. Other systems can | |
| build the library from `megakernel_ling.cu` (only Windows was tested). Python 3.10 or newer with | |
| PyTorch built for CUDA (install it from pytorch.org first), then the packages in | |
| `requirements.txt`. | |
| ## Usage | |
| ```bash | |
| hf download Yougori/Ling-mini-2.0-Megakernel --local-dir Ling-mini-2.0-Megakernel | |
| cd Ling-mini-2.0-Megakernel | |
| pip install -r requirements.txt | |
| ``` | |
| **Chat in a terminal:** | |
| ```bash | |
| python chat.py | |
| ``` | |
| **Several chats at once in a browser:** run the server, then open http://127.0.0.1:8790. Every pane | |
| is its own conversation; while several write, each gets its own copy of the kernel on a share of the | |
| GPU. | |
| ```bash | |
| python server.py | |
| ``` | |
| **Measure the speed:** | |
| ```bash | |
| python bench.py | |
| python bench.py --chats 3 | |
| ``` | |
| **In Python:** | |
| ```python | |
| from transformers import AutoTokenizer | |
| from ling_kernel import LingKernel | |
| tok = AutoTokenizer.from_pretrained(".") | |
| mk = LingKernel() | |
| ids = tok.apply_chat_template([{"role": "user", "content": "What is the capital of France?"}], | |
| tokenize=True, add_generation_prompt=True) | |
| first = mk.forced(ids)[-1] # reads the prompt; its last step chooses the first token | |
| new = [first] + mk.greedy(first, len(ids), 32) # 32 more tokens | |
| print(tok.decode(new, skip_special_tokens=True)) | |
| ``` | |
| To rebuild the library with the CUDA Toolkit 12.8 or newer, run `build.cmd` from an x64 Native Tools | |
| Command Prompt for Visual Studio 2022. | |
| ## Files | |
| | File | | | |
| |:--|:--| | |
| | `megakernel.pdf` | the paper: design, measurements and limitations | | |
| | `packed/layers.safetensors` | the 20 layers in the kernel's layout (8.7 GB) | | |
| | `packed/extras.safetensors` | 4-bit word table, its dimension order, layer 0's MLP, input embeddings, final norm (0.8 GB) | | |
| | `packed/meta.json` | layout numbers and SHA-256 of both files | | |
| | `megakernel_ling.cu`, `megakernel_ling.dll` | the kernel and its Windows build | | |
| | `ling_kernel.py` | loads the model and runs the kernel | | |
| | `chat.py`, `server.py`, `multichat.html`, `bench.py` | terminal chat, several-chats app, speed test | | |
| | `config.json`, `generation_config.json`, `tokenizer*.json`, `special_tokens_map.json` | inclusionAI's, unchanged | | |
| ## Limitations | |
| * Greedy decoding only: the kernel always takes the most likely token. There is no sampling. | |
| * The chat apps use Ling's chat template with thinking off and keep at most 4,096 tokens per | |
| conversation. | |
| * Speed depends on the GPU's power limit. On the same laptop with the GPU held to 90 W, one chat | |
| runs at about 660 to 720 tokens per second. | |
| * The kernel's floating-point sums run in a varying order, so two runs of the same prompt can choose | |
| differently where two tokens are almost equally likely. | |
| * Ling-mini-2.0's own limitations apply. It can be wrong, biased or repetitive. Do not use it for | |
| high-impact decisions. | |
| ## Licenses | |
| The 4-bit weights are derived from Ling-mini-2.0 by inclusionAI, which is licensed under the MIT | |
| License (`LICENSE-Ling`). The kernel and the Python code are licensed under the Apache License 2.0 | |
| (`LICENSE`). Changes from the original are listed in `NOTICE`. The configuration and tokenizer files | |
| are inclusionAI's, unchanged. The paper, `megakernel.pdf`, is licensed under CC BY 4.0 | |
| (`PAPER_LICENSE.txt`). | |
| ## Citation | |
| ```bibtex | |
| @misc{yougori2026megakernelpaper, | |
| title = {A Decoding Megakernel for a 16B Mixture of Experts on a Laptop GPU}, | |
| author = {Yougori}, | |
| year = {2026}, | |
| url = {https://huggingface.co/Yougori/Ling-mini-2.0-Megakernel/blob/main/megakernel.pdf} | |
| } | |
| @misc{yougori2026lingmegakernel, | |
| title = {Ling-mini-2.0 Megakernel}, | |
| author = {Yougori}, | |
| year = {2026}, | |
| url = {https://huggingface.co/Yougori/Ling-mini-2.0-Megakernel} | |
| } | |
| @misc{inclusionai2025lingmini2, | |
| title = {Ling-mini-2.0}, | |
| author = {{inclusionAI}}, | |
| year = {2025}, | |
| url = {https://huggingface.co/inclusionAI/Ling-mini-2.0} | |
| } | |
| ``` | |