| --- |
| license: apache-2.0 |
| base_model: RecursiveSI/flint |
| base_model_relation: quantized |
| pipeline_tag: text-generation |
| library_name: gguf |
| tags: [gguf, rocmfpx, rocmfp4, amd, rocm, strix-halo, quantized, moe, coding, agentic, flint] |
| --- |
| |
| # Flint — ROCmFPX GGUF |
|
|
| > **Update 2026-10-24: much better 4-bit files. Please download again.** If you downloaded |
| > `flint-Q4_0_ROCMFP4_STRIX.gguf` or `flint-Q4_0_ROCMFP4_FAST.gguf`, replace it with one of the new 4-bit files below. |
| > They are about three times closer to the original model (KLD 0.18 instead of 0.54) and now beat a stock `Q4_K_M` |
| > while being smaller. The 8-bit and 6-bit files did not change. |
|
|
| [RecursiveSI/flint](https://huggingface.co/RecursiveSI/flint), Recursive's 39.8B mixture-of-experts model for coding |
| agents (about 2.5B active per token), quantized to the [ROCmFPX](https://github.com/charlie12345/ROCmFPX) formats, a |
| llama.cpp fork with low-bit weight types for AMD GPUs (Vulkan and HIP/ROCm). |
|
|
| **Start with `Q4_0_ROCMFP4_Q8ATTN`** (23.3 GB). It is closer to BF16 than a stock `Q4_K_M` of the same model while |
| being smaller. Use `Q6_0_ROCMFPX` or `Q8_0_ROCMFPX` if you have the memory. |
| |
| > These files need [HarshaVardhan3002/ROCmFPX](https://github.com/HarshaVardhan3002/ROCmFPX) at commit `b130d4995` |
| > or newer. Stock llama.cpp, LM Studio, Ollama and other ROCmFPX builds do not load them. **These files have not been |
| > tested on AMD hardware yet.** |
| > |
| > Not an official release of Recursive. All credit for the model goes to its authors. |
| |
| ## Files and measured quality |
| |
| Measured against the BF16 model (perplexity 19.1433) on WikiText-2 test, 40 chunks of 512 tokens. |
| **KLD** is the mean KL divergence of the next-token distribution from BF16 (lower is better). **Same top** is how |
| often the quant picks the same next token as BF16. |
| |
| | File | Size | PPL | KLD | Same top | | |
| |---|---:|---:|---:|---:|---| |
| | `flint-Q8_0_ROCMFPX.gguf` | 41.1 GB | 19.1675 | 0.0747 | 91.02% | 8-bit, closest to BF16 | |
| | `flint-Q6_K_ROCMFPX.gguf` | 32.9 GB | 19.7449 | 0.0917 | 89.63% | 6-bit, almost as close as 8-bit | |
| | `flint-Q4_0_ROCMFP4_Q8ATTN.gguf` | 23.3 GB | 18.891 | 0.1777 | 84.96% | **Recommended 4-bit** (see below) | |
| | `flint-Q4_0_ROCMFP4_FAST_Q8ATTN.gguf` | 22.1 GB | 18.722 | 0.2071 | 84.23% | speed-first 4-bit: experts with one scale per block | |
|
|
| The 6-bit file is named `flint-Q6_K_ROCMFPX.gguf` so that Hugging Face lists it as a 6-bit quant. Its weights are the ROCmFPX `Q6_0_ROCMFPX` type, not a stock llama.cpp `Q6_K`, and it needs the same fork as the other files. |
|
|
| For comparison, a stock llama.cpp `Q4_K_M` of the same model (23 GB), scored the same way, measured PPL 19.92, |
| KLD 0.2449, same top 82.94%. Both 4-bit files here are closer to BF16 and smaller. |
|
|
| The 4-bit rows were measured in a second run with a fresh BF16 reference (perplexity 19.2141; the same method gives |
| KLD 0.0284 for the BF16 model itself, which is the floor of this measurement). The 8-bit and 6-bit rows come from the |
| first run. |
|
|
| ### The 4-bit recipe |
|
|
| Flint has 256 small experts per layer, and they hold about 95% of the weights. The rest (attention, the shared |
| expert, the two dense layers at the start, and the output layer) is only about 2B weights, but measurements showed |
| that putting it at 4-bit caused most of the loss: with the experts kept at BF16 and everything else at 4-bit, KLD was |
| already 0.473. So both 4-bit files keep all of that at `Q8_0_ROCMFPX` (token embedding `Q6_K`) and only the routed |
| experts are 4-bit: dual-scale `Q4_0_ROCMFP4` in the recommended file, single-scale `Q4_0_ROCMFP4_FAST` in the speed |
| file. This costs about 1 GB over an all-4-bit file. |
|
|
| These replace the first 4-bit files (2026-10-24): `Q4_0_ROCMFP4_STRIX` (21.3 GB, KLD 0.5360) and |
| `Q4_0_ROCMFP4_FAST` (21.2 GB, KLD 0.5428), which also kept attention and the output layer at 4-bit. |
|
|
| Not published: `Q3_0_ROCMFPX` (22.1 GB, KLD 0.5523) is far behind the 4-bit files; |
| `Q2_0_ROCMFPX` does not work on this model. |
|
|
| ## Why these files need a newer build |
|
|
| Flint's weights are about 20 times smaller than those of most models. ROCmFPX stores one small scale per block of 32 |
| weights, and that scale has a fixed smallest step, so on Flint many blocks lost most of their precision: a first 8-bit |
| file had KLD 3.9 (same top 33%) ([ROCmFPX#118](https://github.com/charlie12345/ROCmFPX/issues/118)). |
|
|
| These files were made with an opt-in fix in `llama-quantize` (`LLAMA_ROCMFPX_TENSOR_SCALE=1`): each weight tensor (each |
| expert for mixture-of-experts tensors) is multiplied by a power of two before quantizing, and the inverse factor is |
| stored as a small extra `.scale` tensor that llama.cpp already applies to the matmul output. The factor is a power of |
| two, so the scaling itself loses nothing. The files therefore contain extra tensors that older builds reject. |
|
|
| ## How these were made |
|
|
| 1. `convert_hf_to_gguf.py --outtype bf16`. |
| 2. Importance matrix from the BF16 model on the GPU over a general calibration text (`calibration_datav3`), 512-token |
| chunks. |
| 3. `LLAMA_ROCMFPX_TENSOR_SCALE=1 llama-quantize --imatrix`, one run per type, from |
| [HarshaVardhan3002/ROCmFPX](https://github.com/HarshaVardhan3002/ROCmFPX) `b130d4995`. The 4-bit files use |
| `Q4_0_ROCMFP4_STRIX` with `--tensor-type "attn_(q|k|v|output)=q8_0_rocmfpx" --tensor-type "_shexp=q8_0_rocmfpx" |
| --tensor-type "blk\.[01]\.ffn_(gate|up|down)\.=q8_0_rocmfpx" --output-tensor-type q8_0_rocmfpx`, plus |
| `--tensor-type "_exps=q4_0_rocmfp4"` for the recommended file. |
| 4. Quality: the BF16 reference ran on an NVIDIA GPU with stock BF16 kernels. Each quant was expanded back to F16 |
| (`llama-quantize --allow-requantize ... F16`), which keeps its stored weights and scales, and scored on the same GPU |
| with stock F16 kernels. The fork's own CUDA kernels for these types are not correct on NVIDIA, so they were not |
| used. |
| AMD GPU kernels compute the same math in a different order, so results on an AMD GPU can differ slightly. |
|
|
| ## Run |
|
|
| ```bash |
| git clone https://github.com/HarshaVardhan3002/ROCmFPX && cd ROCmFPX |
| cmake -B build -DGGML_VULKAN=ON # or -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 (gfx1100 for RX 7900) |
| cmake --build build --target llama-server -j |
| build/bin/llama-server -m flint-Q4_0_ROCMFP4_Q8ATTN.gguf -ngl 999 -fa on --jinja --temp 1.0 --top-p 0.95 --top-k 20 |
| ``` |
|
|
| Sampling settings are the ones Recursive used for its evaluations. |
|
|
| ## License |
|
|
| Apache-2.0, same as the base model. |
|
|
