Harsha001/Groot-19B-A3B-261023-ROCmFPX

Groot-19B-A3B-261023 — ROCmFPX GGUF

GrootTeam/Groot-19B-A3B-261023 is a reasoning model with 19B total parameters and about 3B active per token (mixture of experts, 128 experts, 8 per token). It starts from Qwen3.6-35B-A3B, reads images, uses tools, thinks before it answers by default, and has a 256K context. This repo has it in the ROCmFPX formats, a llama.cpp fork with low-bit weight types for AMD GPUs (Vulkan and HIP/ROCm).

Start with Groot-19B-A3B-261023-Q6_K_ROCMFPX.gguf (15.7 GB): it is almost as close to the full model as the 8-bit file and closer than a stock Q6_K. Take the 8-bit file if memory is no concern, and a 4-bit file when memory is tight.

These files need HarshaVardhan3002/ROCmFPX, built from 29 October 2026 or later (commit 642acd395, which loads per-tensor scales next to the skipped MTP layer). Stock llama.cpp, LM Studio, Ollama and other ROCmFPX builds may not load them. These files have not been tested on AMD hardware yet.

Not an official release of Groot. All credit for the model goes to GrootTeam. Their own GGUF (stock Q4_K_M) is at GrootTeam/Groot-19B-A3B-261023-GGUF.

Files

File	Size	KL to BF16	Same top token	PPL
Groot-19B-A3B-261023-Q8_0_ROCMFPX.gguf	19.6 GB	0.0100	95.0%	11.64	8-bit
Groot-19B-A3B-261023-Q6_K_ROCMFPX.gguf	15.7 GB	0.0127	94.4%	11.64	recommended, 6-bit (ROCmFPX Q6_0_ROCMFPX weights, see below)
Groot-19B-A3B-261023-Q4_0_ROCMFP4_STRIX_O8.gguf	10.4 GB	0.0623	86.6%	12.05	4-bit, Strix Halo recipe, embeddings at 8-bit
Groot-19B-A3B-261023-Q4_0_ROCMFP4_FAST.gguf	10.1 GB	0.0663	86.1%	12.10	smallest, speed-first 4-bit
mmproj-Groot-19B-A3B-261023-F16.gguf	0.9 GB				vision encoder, F16
mmproj-Groot-19B-A3B-261023-Q8_0.gguf	0.6 GB				vision encoder, Q8_0

The BF16 model scores PPL 11.60 on the same text. Stock llama.cpp files of this model, measured the same way:

Stock file	Size	KL to BF16	Same top token	PPL
Q8_0	20.2 GB	0.0075	96.1%	11.60
Q6_K	15.6 GB	0.0141	94.0%	11.63
Q4_K_M	11.7 GB	0.0319	90.8%	11.76

The 6-bit file here is closer to BF16 than a stock Q6_K at the same size. A stock Q8_0 is a little closer than the 8-bit file here, which is 0.6 GB smaller. At 4-bit, a stock Q4_K_M (like GrootTeam's own GGUF) stays clearly closer to the full model than the ROCmFP4 files here, for 1.3 to 1.6 GB more. The ROCmFP4 files are smaller and are meant for speed on AMD GPUs; pick the stock Q4_K_M if quality matters more than speed.

How this was measured: llama.cpp llama-perplexity --kl-divergence against the BF16 conversion of the original weights, 40 chunks of 512 tokens of English Wikipedia text (wikitext-2), on the CPU. KL is how far the file's next-token probabilities drift from BF16, averaged over every token; lower is better. "Same top token" is how often the file's most likely next token is the same as BF16's.

Math test (GSM8K)

The first 100 questions of the GSM8K test set, thinking switched off (enable_thinking: false), greedy decoding, at most 1024 tokens per answer, the answer read from a last line "Final answer: ".

File	Correct (of 100)
BF16	98
stock Q4_K_M	97
Q8_0_ROCMFPX	96
Q4_0_ROCMFP4_STRIX_O8	98
Q4_0_ROCMFP4_FAST	99

On 200 questions BF16 got 192, stock Q4_K_M 191 and the 8-bit file 190. With 100 questions a difference of one or two is noise, so none of the files lost measurable math ability. The 6-bit file answered 36 of its first 37 questions right before the run was stopped (it is very slow on the test machine's CPU, see below). BF16 and the stock file ran on a GPU, the ROCmFPX files on the CPU.

Other checks

• Every file answered "What is the capital of France?" correctly (greedy, CPU).
• Both vision encoders, with the 8-bit text file, described a test image (a newspaper front page) correctly.

Speed

These files were made and tested on an ARM machine (NVIDIA DGX Spark CPU, 20 cores), not on an AMD GPU, so there are no AMD speed numbers yet. On that CPU the fork's code for the 6-bit and 4-bit ROCmFPX types is slow, which says nothing about their speed on AMD GPUs, where they run on different code:

File	Prompt (tok/s)	Generation (tok/s)
stock Q4_K_M	108.9	24.7
Q8_0_ROCMFPX	99.1	19.3
Q6_K_ROCMFPX	2.9	0.8
Q4_0_ROCMFP4_STRIX_O8	22.9	2.3
Q4_0_ROCMFP4_FAST	22.3	2.2

(llama-bench, 512-token prompt, 128 generated tokens, 20 threads.) If you run on a CPU, use the 8-bit file or a stock file.

Notes on the files

All four files carry per-tensor scales: each weight matrix is multiplied by a power of two before quantizing and the inverse is stored as <name>.scale, which the fork multiplies back after the matrix product. This keeps small weights from getting stuck at the smallest ROCmFPX block scale. The token embedding uses a stock llama.cpp type (Q8_0 in the 8-bit and STRIX_O8 files, Q6_K in the 6-bit file; the FAST file keeps it in ROCmFP4 for speed).

The 6-bit file is named ...-Q6_K_ROCMFPX.gguf so that Hugging Face lists it as a 6-bit quant. Its weights are the ROCmFPX Q6_0_ROCMFPX type, not a stock llama.cpp Q6_K, and it needs the same fork as the other files.

The files keep the model's multi-token-prediction (MTP) layer. The fork does not use it yet and skips it when loading.

How these were made

1. Source: the original BF16 safetensors from GrootTeam/Groot-19B-A3B-261023, converted to a BF16 GGUF with convert_hf_to_gguf.py; vision encoders converted with --mmproj.
2. Importance matrix from the BF16 file over a general calibration text (calibration_datav3), 512-token chunks.
3. llama-quantize --imatrix from the BF16 file with LLAMA_ROCMFPX_TENSOR_SCALE=1, one run per type, from HarshaVardhan3002/ROCmFPX.

The only change from the original is the weight quantization. The tokenizer and chat template come unchanged from the original model.

Run

git clone https://github.com/HarshaVardhan3002/ROCmFPX && cd ROCmFPX
cmake -B build -DGGML_VULKAN=ON          # or -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 (gfx1100 for RX 7900)
cmake --build build --target llama-server -j
build/bin/llama-server -m Groot-19B-A3B-261023-Q6_K_ROCMFPX.gguf \
  --mmproj mmproj-Groot-19B-A3B-261023-F16.gguf -ngl 999 -c 131072 --jinja

GrootTeam's recommended sampling in thinking mode (the default) is temperature 1.0, top-p 0.95, top-k 20, min-p 0, presence penalty 1.5. To answer without thinking, send "chat_template_kwargs": {"enable_thinking": false} with the request and use temperature 0.7, top-p 0.8, top-k 20. They recommend keeping at least 128K of context for long thinking.

License

Apache-2.0, the same as the original model.