Explain expert weight and activation precision

README.md

@@ -7,29 +7,46 @@ tags:
 - minnow
 - diffusion
 - quantized
+- int4
 - int8
 - nvfp4
 ---
 
 # LLaDA2.2-flash for minnow
 
-Self-contained `.mnw` checkpoints of [InclusionAI's LLaDA2.2-flash](https://huggingface.co/inclusionAI/LLaDA2.2-flash) for [minnow](https://github.com/coder543/minnow), a Rust block-diffusion inference server with an OpenAI-compatible Chat Completions API. Model development and training are credited to InclusionAI; this repository provides format conversion and quantization.
+Self-contained `.mnw` checkpoints of [InclusionAI's LLaDA2.2-flash](https://huggingface.co/inclusionAI/LLaDA2.2-flash) for [minnow](https://github.com/coder543/minnow), a Rust block-diffusion inference server with an OpenAI-compatible Chat Completions API. InclusionAI developed and trained the model; this repository provides format conversion and quantization.
 
-| Checkpoint | Format |
-| --- | --- |
-| `llada2.2-flash-nvfp4.mnw` | Native NVFP4 routed experts: FP4 weights and activations. CUDA execution requires SM120/121 Blackwell. |
-| `llada2.2-flash-int8.mnw` | INT8 routed expert weights, FP16 scales per 128 weights, BF16 CUDA activations. Ampere or newer. |
-| `llada2.2-flash-bf16.mnw` | Unquantized conversion preserving the original floating-point weight bytes, primarily BF16. |
+## Checkpoints and precision
 
-Only routed experts are quantized. All files include the configuration, tokenizer, and chat template. Quantization can change model outputs; upstream evaluation scores are not re-measured scores for these quantized files. The BF16 file is included as an unquantized alternative.
+**Quantization applies to routed experts.** Their gate, up, and down projection weights use the formats below. Attention, embeddings, routers, dense/shared experts, and the output head retain their source floating-point weights, primarily BF16. Default CUDA execution uses BF16 for the surrounding model; the activation column describes inputs to the routed expert matrix multiplications.
 
-Build minnow using its [README](https://github.com/coder543/minnow#build-and-run), then download and serve a checkpoint:
+| Checkpoint | Size | Routed expert weights | Expert activations on CUDA, by default |
+| --- | ---: | --- | --- |
+| `llada2.2-flash-bf16.mnw` | 191.66 GiB | Original floating-point weights, primarily BF16; no quantization. | BF16. |
+| `llada2.2-flash-int8.mnw` | 100.11 GiB | Signed INT8; one FP16 scale per 128 weights. | BF16 (**W8A16**). |
+| `llada2.2-flash-int4.mnw` | 53.61 GiB | Signed INT4; one FP16 scale per 128 weights. | BF16 (**W4A16**). |
+| `llada2.2-flash-nvfp4.mnw` | 57.97 GiB | E2M1 FP4; E4M3 scales per 16 weights and an FP32 outer scale per expert matrix. | Dynamically quantized E2M1 FP4, with E4M3 scales per 16 values and an FP32 outer scale per input row (**W4A4**). |
+
+**W** means weight bits and **A** means activation bits; A16 here means BF16. INT4 and INT8 are weight-only quantizations by default: CUDA kernels dequantize weights into BF16 registers for tensor-core multiplication without keeping a full expanded weight copy.
+
+For either INT4 or INT8, `--int8-expert-activations` optionally selects **W4A8** or **W8A8** on Ampere-or-newer CUDA GPUs. This dynamically quantizes expert inputs to signed INT8 with FP32 scales per 128 values, uses INT32 group dot products, and combines scaled groups in FP32. Expert outputs return to BF16. It uses the same checkpoint and changes model numerics; it does not quantize the rest of the model's activations.
+
+NVFP4 quantizes both expert weights and expert inputs. SM120/121 Blackwell uses native FP4 tensor cores. Ampere/Ada uses a BF16 tensor-core fallback that reconstructs the **already quantized FP4 operands**; the fallback retains FP4 activation quantization. `--int8-expert-activations` applies only to INT4/INT8 experts.
+
+## Usage
+
+Build current minnow using its [README](https://github.com/coder543/minnow#build-and-run). INT4 requires commit `0e6678f` or newer. Download and serve:
 
 ```sh
-hf download coder543/LLaDA2.2-flash-minnow llada2.2-flash-int8.mnw --local-dir models
-minnow --model models/llada2.2-flash-int8.mnw serve
+hf download coder543/LLaDA2.2-flash-minnow llada2.2-flash-int4.mnw --local-dir models
+minnow --model models/llada2.2-flash-int4.mnw serve
+
+# Optional integer expert activations, using the same checkpoint:
+minnow --model models/llada2.2-flash-int4.mnw --int8-expert-activations serve
 ```
 
-These files require minnow and are not directly loadable by Transformers or llama.cpp. Minnow also provides a slower CPU fallback. Allow memory beyond the checkpoint size for runtime state.
+All files include configuration, tokenizer, and chat template. Allow additional memory for K/V, activations, and workspaces. These files require minnow and are not directly loadable by Transformers or llama.cpp. CPU fallback defaults to FP32 execution with one expert expanded at a time; the CUDA precision table does not describe CPU execution.
+
+Quantization and activation mode can change outputs; upstream evaluation scores have not been re-measured for these files. See [RTX 3090 measurements](https://github.com/coder543/minnow/blob/main/docs/rtx3090.md) for comparisons under the documented settings.
 
-Licensed under [Apache-2.0](LICENSE), matching the [upstream model](https://huggingface.co/inclusionAI/LLaDA2.2-flash). See upstream for model details and usage guidance.
+Licensed under [Apache-2.0](LICENSE), matching the [upstream model](https://huggingface.co/inclusionAI/LLaDA2.2-flash).