File size: 4,486 Bytes
bc27072
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2116290
bc27072
 
 
 
 
 
 
 
 
 
 
2116290
9890076
 
 
 
 
 
bc27072
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
---
license: apache-2.0
base_model: thinkingmachines/Inkling-2-Mini
base_model_relation: quantized
pipeline_tag: text-generation
library_name: gguf
tags: [gguf, rocmfpx, amd, vulkan, rocm, strix-halo, quantized, moe, inkling, inkling2]
---

# Inkling 2 Mini ROCmFPX (AMD GPUs and Strix Halo)

Hardware-aware GGUF quants of [Inkling 2 Mini by Thinking Machines](https://huggingface.co/thinkingmachines/Inkling-2-Mini),
built with the [ROCmFPX](https://github.com/HarshaVardhan3002/ROCmFPX) weight formats for AMD GPUs (Vulkan and ROCm).

Inkling 2 Mini: 26.1B parameters, 24 layers, 128 experts per layer with 8 active per token (about 3.1B active
parameters per layer pass). It is a looped model: layers 6 to 17 run three times per token, so a token passes through
48 layers. Text only here: the vision encoder and the multi-token prediction drafter are not included.

> Not an official release. All credit for the model goes to Thinking Machines.

## Files

| File | Size | Weights | Fits |
|---|---:|---|---|
| `Inkling-2-Mini-Q8_0_ROCMFPX.gguf` | 26.9 GB | everything 8-bit ROCmFPX (token embedding Q8_0) | best quality: Strix Halo, 32 GB AMD GPUs |
| `Inkling-2-Mini-Q4_0_ROCMFP4_STRIX_O8.gguf` | 15.5 GB | experts ROCmFP4, everything else 8-bit ROCmFPX | best smaller: 24 GB AMD GPUs, Strix Halo with room for long context |

## Quality

KL divergence against the original weights (Thinking Machines' NVFP4 experts kept as they are, the MXFP8 layers 20 to
23 stored as BF16), 40 chunks of 512 tokens of English Wikipedia text (wikitext-2), CPU. Lower is better.
"Same top token" is how often the file's most likely next token matches the original.

| Build | Size | Perplexity | KL to original | Same top token |
|---|---:|---:|---:|---:|
| Original weights (reference, not published) | 22.9 GB | 6.743 | | |
| **Q8_0_ROCMFPX** | 26.9 GB | 6.741 | 0.0039 | 97.2% |
| **Q4_0_ROCMFP4_STRIX_O8** | 15.5 GB | 6.784 | 0.0120 | 95.4% |
| stock Q8_0 (comparison, not uploaded) | 27.7 GB | 6.739 | 0.0035 | 97.5% |
| stock Q4_K_M (comparison, not uploaded) | 17.4 GB | 6.819 | 0.0174 | 94.1% |

At 4 bits the ROCmFP4 file is smaller than stock Q4_K_M and closer to the original on every measure. At 8 bits stock
Q8_0 is a little closer to the original (0.0035 against 0.0039) and 0.8 GB larger; the ROCmFPX file is the one built for
the AMD kernels.

The Q8 file is larger than the original checkpoint because the original experts are 4-bit NVFP4. 8 bits keep them
almost exactly; the 4-bit ROCmFP4 file is the size to pick when memory is tight.

Every file also passed a greedy chat check before upload ("What is the capital of France? Answer in one word." gives
"Paris" after a short thinking part).

## Running it

Inkling 2 Mini needs the looped `inkling2` graph, which is only in the ROCmFPX fork of llama.cpp (branch
`inkling2-mini`). Build it with Vulkan or ROCm/HIP, then for example:

```
llama-server -m Inkling-2-Mini-Q4_0_ROCMFP4_STRIX_O8.gguf -ngl 99 -c 16384 --jinja
```

Known limits of this first version:

- Only the CPU path has been checked against the original PyTorch model (it matches to about 5e-6 on a small test
  model, and the tokenizer matches exactly). Vulkan and ROCm have not been tried yet. Please report in the Community
  tab whether it runs, how fast, and how the answers look.
- llama.cpp has no parser for Inkling's message format yet. Answers come back with Inkling's own markers
  (`<|content_thinking|>`, `<|end_message|>` and so on): the thinking is not split from the answer and tool calls are
  not parsed. Plain chat works; agent use does not yet.
- The chat template accepts `reasoning_effort` as `none`, `minimal`, `low`, `medium`, `high`, `xhigh` or `max`,
  or as a number from 0 to 0.99 (default 0.9).

## How it was made

- Converted from the original checkpoint with the NVFP4 experts kept as they are, then requantized with
  `llama-quantize`. The per-expert NVFP4 scale is folded into the weights before requantizing.
- ROCmFPX tensor scales on (a power-of-two scale per tensor, stored as `.scale`), token embedding Q8_0.
- Importance matrix from a general text mix (prose, code, chat) on the original weights.

## Credits and license

- Model: [thinkingmachines/Inkling-2-Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini), Apache-2.0.
- Runtime: [llama.cpp](https://github.com/ggml-org/llama.cpp), with Inkling 2 and ROCmFPX support in
  [ROCmFPX](https://github.com/HarshaVardhan3002/ROCmFPX).

Apache-2.0. See `LICENSE`.