README.md

---
license: apache-2.0
base_model: inclusionAI/LLaDA2.2-flash
base_model_relation: quantized
pipeline_tag: text-generation
tags:
- minnow
- diffusion
- quantized
- int4
- int8
- nvfp4
---

# LLaDA2.2-flash for minnow

Self-contained `.mnw` checkpoints of [InclusionAI's LLaDA2.2-flash](https://huggingface.co/inclusionAI/LLaDA2.2-flash) for [minnow](https://github.com/coder543/minnow), a Rust block-diffusion inference server with an OpenAI-compatible Chat Completions API. InclusionAI developed and trained the model; this repository provides format conversion and quantization.

## Checkpoints and precision

**Quantization applies to routed experts.** Their gate, up, and down projection weights use the formats below. Attention, embeddings, routers, dense/shared experts, and the output head retain their source floating-point weights, primarily BF16. Default CUDA execution uses BF16 for the surrounding model; the activation column describes inputs to the routed expert matrix multiplications.

| Checkpoint | Size | Routed expert weights | Expert activations on CUDA, by default |
| --- | ---: | --- | --- |
| `llada2.2-flash-bf16.mnw` | 191.66 GiB | Original floating-point weights, primarily BF16; no quantization. | BF16. |
| `llada2.2-flash-int8.mnw` | 100.11 GiB | Signed INT8; one FP16 scale per 128 weights. | BF16 (**W8A16**). |
| `llada2.2-flash-int4.mnw` | 53.61 GiB | Signed INT4; one FP16 scale per 128 weights. | BF16 (**W4A16**). |
| `llada2.2-flash-nvfp4.mnw` | 57.97 GiB | E2M1 FP4; E4M3 scales per 16 weights and an FP32 outer scale per expert matrix. | Dynamically quantized E2M1 FP4, with E4M3 scales per 16 values and an FP32 outer scale per input row (**W4A4**). |

**W** means weight bits and **A** means activation bits; A16 here means BF16. INT4 and INT8 are weight-only quantizations by default: CUDA kernels dequantize weights into BF16 registers for tensor-core multiplication without keeping a full expanded weight copy.

For either INT4 or INT8, `--int8-expert-activations` optionally selects **W4A8** or **W8A8** on Ampere-or-newer CUDA GPUs. This dynamically quantizes expert inputs to signed INT8 with FP32 scales per 128 values, uses INT32 group dot products, and combines scaled groups in FP32. Expert outputs return to BF16. It uses the same checkpoint and changes model numerics; it does not quantize the rest of the model's activations.

NVFP4 quantizes both expert weights and expert inputs. SM120/121 Blackwell uses native FP4 tensor cores. Ampere/Ada uses a BF16 tensor-core fallback that reconstructs the **already quantized FP4 operands**; the fallback retains FP4 activation quantization. `--int8-expert-activations` applies only to INT4/INT8 experts.

## Usage

Build current minnow using its [README](https://github.com/coder543/minnow#build-and-run). INT4 requires commit `0e6678f` or newer. Download and serve:

```sh
hf download coder543/LLaDA2.2-flash-minnow llada2.2-flash-int4.mnw --local-dir models
minnow --model models/llada2.2-flash-int4.mnw serve

# Optional integer expert activations, using the same checkpoint:
minnow --model models/llada2.2-flash-int4.mnw --int8-expert-activations serve
```

All files include configuration, tokenizer, and chat template. Allow additional memory for K/V, activations, and workspaces. These files require minnow and are not directly loadable by Transformers or llama.cpp. CPU fallback defaults to FP32 execution with one expert expanded at a time; the CUDA precision table does not describe CPU execution.

Quantization and activation mode can change outputs; upstream evaluation scores have not been re-measured for these files. See [RTX 3090 measurements](https://github.com/coder543/minnow/blob/main/docs/rtx3090.md) for comparisons under the documented settings.

Licensed under [Apache-2.0](LICENSE), matching the [upstream model](https://huggingface.co/inclusionAI/LLaDA2.2-flash).