notforu777 commited on
Commit
11a7d4e
·
verified ·
1 Parent(s): f50b0d8

Card: early AD-IQ1_S and AD-IQ1_M

Browse files
Files changed (1) hide show
  1. README.md +4 -0
README.md CHANGED
@@ -36,8 +36,12 @@ GGUF files of [thinkingmachines/Inkling-2-Mini](https://huggingface.co/thinkingm
36
  | `Inkling-2-Mini-Q6_K.gguf` | 23.4 | llama.cpp preset Q6_K, no imatrix |
37
  | `Inkling-2-Mini-Q5_K_M.gguf` | 19.8 | llama.cpp preset Q5_K_M, no imatrix |
38
  | `Inkling-2-Mini-Q4_K_M.gguf` | 17.4 | llama.cpp preset Q4_K_M, no imatrix |
 
 
39
  | `mmproj-Inkling-2-Mini-F16.gguf` / `-BF16.gguf` | 0.64 | vision projector (image input) |
40
 
 
 
41
  **Coming next:** AD (Atomic Dynamic) quants made with an importance matrix for the 10–24 GB range, with every file's quality measured against BF16 on our neutral corpus (mean KLD and top-1 agreement). The presets above are the first files. A preset's name does not tell the whole story here: the expert down projection has rows of 1152, which k-quants cannot take, so llama.cpp swaps the type there: Q8_0 in all 24 layers of Q6_K, Q8_0 in half the layers and Q5_1 in the rest for Q5_K_M, Q8_0 and Q5_0 for Q4_K_M.
42
 
43
  ## Run
 
36
  | `Inkling-2-Mini-Q6_K.gguf` | 23.4 | llama.cpp preset Q6_K, no imatrix |
37
  | `Inkling-2-Mini-Q5_K_M.gguf` | 19.8 | llama.cpp preset Q5_K_M, no imatrix |
38
  | `Inkling-2-Mini-Q4_K_M.gguf` | 17.4 | llama.cpp preset Q4_K_M, no imatrix |
39
+ | `Inkling-2-Mini-AD-IQ1_M.gguf` | 7.26 | AD 1-bit: experts gate/up IQ1_M, down Q2_0; early imatrix, see below |
40
+ | `Inkling-2-Mini-AD-IQ1_S.gguf` | 6.88 | AD 1-bit: experts gate/up IQ1_S, down Q2_0; early imatrix, see below |
41
  | `mmproj-Inkling-2-Mini-F16.gguf` / `-BF16.gguf` | 0.64 | vision projector (image input) |
42
 
43
+ **The 1-bit files are early builds.** Their importance matrix covers about a quarter of our calibration corpus (the first chunks of two of six shards), because they were made while the full matrix was still being computed. Files built from the full matrix will replace them under the same names, and their measured quality will be added here. Expect 1-bit quality to be well below the 4-bit files: use them only when nothing larger fits. Both passed our checks (chat, reasoning, a tool call, an image question, Chinese) in llama-server on an H100.
44
+
45
  **Coming next:** AD (Atomic Dynamic) quants made with an importance matrix for the 10–24 GB range, with every file's quality measured against BF16 on our neutral corpus (mean KLD and top-1 agreement). The presets above are the first files. A preset's name does not tell the whole story here: the expert down projection has rows of 1152, which k-quants cannot take, so llama.cpp swaps the type there: Q8_0 in all 24 layers of Q6_K, Q8_0 in half the layers and Q5_1 in the rest for Q5_K_M, Q8_0 and Q5_0 for Q4_K_M.
46
 
47
  ## Run