README.md

---
license: apache-2.0
base_model: RecursiveSI/flint
base_model_relation: quantized
pipeline_tag: text-generation
library_name: gguf
tags: [gguf, rocmfpx, rocmfp4, amd, rocm, strix-halo, quantized, moe, coding, agentic, flint]
---

# Flint — ROCmFPX GGUF

> **Update 2026-10-24: much better 4-bit files. Please download again.** If you downloaded
> `flint-Q4_0_ROCMFP4_STRIX.gguf` or `flint-Q4_0_ROCMFP4_FAST.gguf`, replace it with one of the new 4-bit files below.
> They are about three times closer to the original model (KLD 0.18 instead of 0.54) and now beat a stock `Q4_K_M`
> while being smaller. The 8-bit and 6-bit files did not change.

[RecursiveSI/flint](https://huggingface.co/RecursiveSI/flint), Recursive's 39.8B mixture-of-experts model for coding
agents (about 2.5B active per token), quantized to the [ROCmFPX](https://github.com/charlie12345/ROCmFPX) formats, a
llama.cpp fork with low-bit weight types for AMD GPUs (Vulkan and HIP/ROCm).

**Start with `Q4_0_ROCMFP4_Q8ATTN`** (23.3 GB). It is closer to BF16 than a stock `Q4_K_M` of the same model while
being smaller. Use `Q6_0_ROCMFPX` or `Q8_0_ROCMFPX` if you have the memory.

> These files need [HarshaVardhan3002/ROCmFPX](https://github.com/HarshaVardhan3002/ROCmFPX) at commit `b130d4995`
> or newer. Stock llama.cpp, LM Studio, Ollama and other ROCmFPX builds do not load them. **These files have not been
> tested on AMD hardware yet.**
>
> Not an official release of Recursive. All credit for the model goes to its authors.

## Files and measured quality

Measured against the BF16 model (perplexity 19.1433) on WikiText-2 test, 40 chunks of 512 tokens.
**KLD** is the mean KL divergence of the next-token distribution from BF16 (lower is better). **Same top** is how
often the quant picks the same next token as BF16.

| File | Size | PPL | KLD | Same top | |
|---|---:|---:|---:|---:|---|
| `flint-Q8_0_ROCMFPX.gguf` | 41.1 GB | 19.1675 | 0.0747 | 91.02% | 8-bit, closest to BF16 |
| `flint-Q6_K_ROCMFPX.gguf` | 32.9 GB | 19.7449 | 0.0917 | 89.63% | 6-bit, almost as close as 8-bit |
| `flint-Q4_0_ROCMFP4_Q8ATTN.gguf` | 23.3 GB | 18.891 | 0.1777 | 84.96% | **Recommended 4-bit** (see below) |
| `flint-Q4_0_ROCMFP4_FAST_Q8ATTN.gguf` | 22.1 GB | 18.722 | 0.2071 | 84.23% | speed-first 4-bit: experts with one scale per block |

The 6-bit file is named `flint-Q6_K_ROCMFPX.gguf` so that Hugging Face lists it as a 6-bit quant. Its weights are the ROCmFPX `Q6_0_ROCMFPX` type, not a stock llama.cpp `Q6_K`, and it needs the same fork as the other files.

For comparison, a stock llama.cpp `Q4_K_M` of the same model (23 GB), scored the same way, measured PPL 19.92,
KLD 0.2449, same top 82.94%. Both 4-bit files here are closer to BF16 and smaller.

The 4-bit rows were measured in a second run with a fresh BF16 reference (perplexity 19.2141; the same method gives
KLD 0.0284 for the BF16 model itself, which is the floor of this measurement). The 8-bit and 6-bit rows come from the
first run.

### The 4-bit recipe

Flint has 256 small experts per layer, and they hold about 95% of the weights. The rest (attention, the shared
expert, the two dense layers at the start, and the output layer) is only about 2B weights, but measurements showed
that putting it at 4-bit caused most of the loss: with the experts kept at BF16 and everything else at 4-bit, KLD was
already 0.473. So both 4-bit files keep all of that at `Q8_0_ROCMFPX` (token embedding `Q6_K`) and only the routed
experts are 4-bit: dual-scale `Q4_0_ROCMFP4` in the recommended file, single-scale `Q4_0_ROCMFP4_FAST` in the speed
file. This costs about 1 GB over an all-4-bit file.

These replace the first 4-bit files (2026-10-24): `Q4_0_ROCMFP4_STRIX` (21.3 GB, KLD 0.5360) and
`Q4_0_ROCMFP4_FAST` (21.2 GB, KLD 0.5428), which also kept attention and the output layer at 4-bit.

Not published: `Q3_0_ROCMFPX` (22.1 GB, KLD 0.5523) is far behind the 4-bit files;
`Q2_0_ROCMFPX` does not work on this model.

## Why these files need a newer build

Flint's weights are about 20 times smaller than those of most models. ROCmFPX stores one small scale per block of 32
weights, and that scale has a fixed smallest step, so on Flint many blocks lost most of their precision: a first 8-bit
file had KLD 3.9 (same top 33%) ([ROCmFPX#118](https://github.com/charlie12345/ROCmFPX/issues/118)).

These files were made with an opt-in fix in `llama-quantize` (`LLAMA_ROCMFPX_TENSOR_SCALE=1`): each weight tensor (each
expert for mixture-of-experts tensors) is multiplied by a power of two before quantizing, and the inverse factor is
stored as a small extra `.scale` tensor that llama.cpp already applies to the matmul output. The factor is a power of
two, so the scaling itself loses nothing. The files therefore contain extra tensors that older builds reject.

## How these were made

1. `convert_hf_to_gguf.py --outtype bf16`.
2. Importance matrix from the BF16 model on the GPU over a general calibration text (`calibration_datav3`), 512-token
   chunks.
3. `LLAMA_ROCMFPX_TENSOR_SCALE=1 llama-quantize --imatrix`, one run per type, from
   [HarshaVardhan3002/ROCmFPX](https://github.com/HarshaVardhan3002/ROCmFPX) `b130d4995`. The 4-bit files use
   `Q4_0_ROCMFP4_STRIX` with `--tensor-type "attn_(q|k|v|output)=q8_0_rocmfpx" --tensor-type "_shexp=q8_0_rocmfpx"
   --tensor-type "blk\.[01]\.ffn_(gate|up|down)\.=q8_0_rocmfpx" --output-tensor-type q8_0_rocmfpx`, plus
   `--tensor-type "_exps=q4_0_rocmfp4"` for the recommended file.
4. Quality: the BF16 reference ran on an NVIDIA GPU with stock BF16 kernels. Each quant was expanded back to F16
   (`llama-quantize --allow-requantize ... F16`), which keeps its stored weights and scales, and scored on the same GPU
   with stock F16 kernels. The fork's own CUDA kernels for these types are not correct on NVIDIA, so they were not
   used.
   AMD GPU kernels compute the same math in a different order, so results on an AMD GPU can differ slightly.

## Run

```bash
git clone https://github.com/HarshaVardhan3002/ROCmFPX && cd ROCmFPX
cmake -B build -DGGML_VULKAN=ON          # or -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 (gfx1100 for RX 7900)
cmake --build build --target llama-server -j
build/bin/llama-server -m flint-Q4_0_ROCMFP4_Q8ATTN.gguf -ngl 999 -fa on --jinja --temp 1.0 --top-p 0.95 --top-k 20
```

Sampling settings are the ones Recursive used for its evaluations.

## License

Apache-2.0, same as the base model.