Flint ROCmFPX GGUF

.gitattributes

@@ -0,0 +1,41 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+flint-Q8_0_ROCMFPX.gguf filter=lfs diff=lfs merge=lfs -text
+flint-Q6_0_ROCMFPX.gguf filter=lfs diff=lfs merge=lfs -text
+flint-Q4_0_ROCMFP4_STRIX.gguf filter=lfs diff=lfs merge=lfs -text
+flint-Q4_0_ROCMFP4_FAST.gguf filter=lfs diff=lfs merge=lfs -text
+flint-Q4_0_ROCMFP4_Q8ATTN.gguf filter=lfs diff=lfs merge=lfs -text
+flint-Q4_0_ROCMFP4_FAST_Q8ATTN.gguf filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,105 @@
+---
+license: apache-2.0
+base_model: RecursiveSI/flint
+base_model_relation: quantized
+pipeline_tag: text-generation
+library_name: gguf
+tags: [gguf, rocmfpx, rocmfp4, amd, rocm, strix-halo, quantized, moe, coding, agentic, flint]
+---
+
+# Flint — ROCmFPX GGUF
+
+> **Update 2026-10-24: much better 4-bit files. Please download again.** If you downloaded
+> `flint-Q4_0_ROCMFP4_STRIX.gguf` or `flint-Q4_0_ROCMFP4_FAST.gguf`, replace it with one of the new 4-bit files below.
+> They are about three times closer to the original model (KLD 0.18 instead of 0.54) and now beat a stock `Q4_K_M`
+> while being smaller. The 8-bit and 6-bit files did not change.
+
+[RecursiveSI/flint](https://huggingface.co/RecursiveSI/flint), Recursive's 39.8B mixture-of-experts model for coding
+agents (about 2.5B active per token), quantized to the [ROCmFPX](https://github.com/charlie12345/ROCmFPX) formats, a
+llama.cpp fork with low-bit weight types for AMD GPUs (Vulkan and HIP/ROCm).
+
+**Start with `Q4_0_ROCMFP4_Q8ATTN`** (23.3 GB). It is closer to BF16 than a stock `Q4_K_M` of the same model while
+being smaller. Use `Q6_0_ROCMFPX` or `Q8_0_ROCMFPX` if you have the memory.
+
+> These files need [HarshaVardhan3002/ROCmFPX](https://github.com/HarshaVardhan3002/ROCmFPX) at commit `b130d4995`
+> or newer. Stock llama.cpp, LM Studio, Ollama and other ROCmFPX builds do not load them. **These files have not been
+> tested on AMD hardware yet.**
+>
+> Not an official release of Recursive. All credit for the model goes to its authors.
+
+## Files and measured quality
+
+Measured against the BF16 model (perplexity 19.1433) on WikiText-2 test, 40 chunks of 512 tokens.
+**KLD** is the mean KL divergence of the next-token distribution from BF16 (lower is better). **Same top** is how
+often the quant picks the same next token as BF16.
+
+| File | Size | PPL | KLD | Same top | |
+|---|---:|---:|---:|---:|---|
+| `flint-Q8_0_ROCMFPX.gguf` | 41.1 GB | 19.1675 | 0.0747 | 91.02% | 8-bit, closest to BF16 |
+| `flint-Q6_0_ROCMFPX.gguf` | 32.9 GB | 19.7449 | 0.0917 | 89.63% | 6-bit, almost as close as 8-bit |
+| `flint-Q4_0_ROCMFP4_Q8ATTN.gguf` | 23.3 GB | 18.891 | 0.1777 | 84.96% | **Recommended 4-bit** (see below) |
+| `flint-Q4_0_ROCMFP4_FAST_Q8ATTN.gguf` | 22.1 GB | 18.722 | 0.2071 | 84.23% | speed-first 4-bit: experts with one scale per block |
+
+For comparison, a stock llama.cpp `Q4_K_M` of the same model (23 GB), scored the same way, measured PPL 19.92,
+KLD 0.2449, same top 82.94%. Both 4-bit files here are closer to BF16 and smaller.
+
+The 4-bit rows were measured in a second run with a fresh BF16 reference (perplexity 19.2141; the same method gives
+KLD 0.0284 for the BF16 model itself, which is the floor of this measurement). The 8-bit and 6-bit rows come from the
+first run.
+
+### The 4-bit recipe
+
+Flint has 256 small experts per layer, and they hold about 95% of the weights. The rest (attention, the shared
+expert, the two dense layers at the start, and the output layer) is only about 2B weights, but measurements showed
+that putting it at 4-bit caused most of the loss: with the experts kept at BF16 and everything else at 4-bit, KLD was
+already 0.473. So both 4-bit files keep all of that at `Q8_0_ROCMFPX` (token embedding `Q6_K`) and only the routed
+experts are 4-bit: dual-scale `Q4_0_ROCMFP4` in the recommended file, single-scale `Q4_0_ROCMFP4_FAST` in the speed
+file. This costs about 1 GB over an all-4-bit file.
+
+These replace the first 4-bit files (2026-10-24): `Q4_0_ROCMFP4_STRIX` (21.3 GB, KLD 0.5360) and
+`Q4_0_ROCMFP4_FAST` (21.2 GB, KLD 0.5428), which also kept attention and the output layer at 4-bit.
+
+Not published: `Q3_0_ROCMFPX` (22.1 GB, KLD 0.5523) is far behind the 4-bit files;
+`Q2_0_ROCMFPX` does not work on this model.
+
+## Why these files need a newer build
+
+Flint's weights are about 20 times smaller than those of most models. ROCmFPX stores one small scale per block of 32
+weights, and that scale has a fixed smallest step, so on Flint many blocks lost most of their precision: a first 8-bit
+file had KLD 3.9 (same top 33%) ([ROCmFPX#118](https://github.com/charlie12345/ROCmFPX/issues/118)).
+
+These files were made with an opt-in fix in `llama-quantize` (`LLAMA_ROCMFPX_TENSOR_SCALE=1`): each weight tensor (each
+expert for mixture-of-experts tensors) is multiplied by a power of two before quantizing, and the inverse factor is
+stored as a small extra `.scale` tensor that llama.cpp already applies to the matmul output. The factor is a power of
+two, so the scaling itself loses nothing. The files therefore contain extra tensors that older builds reject.
+
+## How these were made
+
+1. `convert_hf_to_gguf.py --outtype bf16`.
+2. Importance matrix from the BF16 model on the GPU over a general calibration text (`calibration_datav3`), 512-token
+   chunks.
+3. `LLAMA_ROCMFPX_TENSOR_SCALE=1 llama-quantize --imatrix`, one run per type, from
+   [HarshaVardhan3002/ROCmFPX](https://github.com/HarshaVardhan3002/ROCmFPX) `b130d4995`. The 4-bit files use
+   `Q4_0_ROCMFP4_STRIX` with `--tensor-type "attn_(q|k|v|output)=q8_0_rocmfpx" --tensor-type "_shexp=q8_0_rocmfpx"
+   --tensor-type "blk\.[01]\.ffn_(gate|up|down)\.=q8_0_rocmfpx" --output-tensor-type q8_0_rocmfpx`, plus
+   `--tensor-type "_exps=q4_0_rocmfp4"` for the recommended file.
+4. Quality: the BF16 reference ran on an NVIDIA GPU with stock BF16 kernels. Each quant was expanded back to F16
+   (`llama-quantize --allow-requantize ... F16`), which keeps its stored weights and scales, and scored on the same GPU
+   with stock F16 kernels. The fork's own CUDA kernels for these types are not correct on NVIDIA, so they were not
+   used.
+   AMD GPU kernels compute the same math in a different order, so results on an AMD GPU can differ slightly.
+
+## Run
+
+```bash
+git clone https://github.com/HarshaVardhan3002/ROCmFPX && cd ROCmFPX
+cmake -B build -DGGML_VULKAN=ON          # or -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 (gfx1100 for RX 7900)
+cmake --build build --target llama-server -j
+build/bin/llama-server -m flint-Q4_0_ROCMFP4_Q8ATTN.gguf -ngl 999 -fa on --jinja --temp 1.0 --top-p 0.95 --top-k 20
+```
+
+Sampling settings are the ones Recursive used for its evaluations.
+
+## License
+
+Apache-2.0, same as the base model.

flint-Q4_0_ROCMFP4_FAST_Q8ATTN.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2a34f637851fd86e0ddbd4bc91f624553c7daa2b735ac0cbfad044fe63af145b
+size 22118466656

flint-Q4_0_ROCMFP4_Q8ATTN.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c5cd952ee8fcf53d1c6e77548571e4d26222402d5d3fdfe459f3e75860f0c7bd
+size 23301260384

flint-Q6_0_ROCMFPX.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7aae5f2e50510a7da77dd3627a037d01f0e858cadd6f6dfc0e84b4e54a20eb4a
+size 32901758048

flint-Q8_0_ROCMFPX.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7221b8960dd6af2118c7c3104ab47deefde0c59649bcfe3bb6d1ed44efa90584
+size 41108028512