NagusameCS commited on
Commit
d2c9b48
·
verified ·
1 Parent(s): 37d409c

model card: keep16u-iq2xs replaced; 2-bit cause documented

Browse files
Files changed (1) hide show
  1. README.md +6 -4
README.md CHANGED
@@ -24,9 +24,11 @@ defects. The previous files remain available in this repository's history.
24
  files keep, in each layer, the experts that layer actually uses most (full-model `llama-imatrix`, 32 × 512
25
  tokens of mixed calibration text).
26
  3. **keep12u had zeroed layers.** The last 6.5 GiB of the previous file (layers 48–60) was all zeros.
27
- 4. **The IQ2_XS files emitted garbage** under upstream llama.cpp and the Ollama build: repeated tokens, top-1
28
- agreement 0 %. The new IQ2_XS files are requantised from the new Q3_K files with that full-model imatrix,
29
- sliced to each tier's kept experts.
 
 
30
 
31
  ## Measured quality: read this before using them
32
 
@@ -40,6 +42,7 @@ and the top-1 agreement with it.
40
  |---|---|---|---|---|---|---|
41
  | `keep12u` | Q3_K_M | 32.2 | 7.8× / 53% | 22.3× / 31% | 17.8× / 30% | 118.1× / 22% |
42
  | `keep16u` | Q3_K_M | 39.3 | 7.2× / 55% | 22.9× / 32% | 18.3× / 31% | 129.9× / 21% |
 
43
 
44
  Previous files, for comparison (same test):
45
 
@@ -57,7 +60,6 @@ Larger tiers are more faithful (see the table).
57
  ## Files not (yet) replaced
58
 
59
  - `keep12u-iq2xs/DeepSeek-V4-Coder-keep12u-iq2xs.gguf`: **Known broken (previous file).** Emits garbage under upstream llama.cpp (top-1 agreement with the full model 0 %). A corrected requantisation replaces it once it passes the same checks as the tiers above.
60
- - `keep16u-iq2xs/DeepSeek-V4-Coder-keep16u-iq2xs.gguf`: **Known broken (previous file).** Emits garbage under upstream llama.cpp (top-1 agreement with the full model 0 %). A corrected requantisation replaces it once it passes the same checks as the tiers above.
61
  - `keep64-iq2xs/DeepSeek-V4-Coder-keep64-iq2xxs.gguf`: Unchanged. Mixed expert counts behind a full-width router: loads only with UltraTensor's llama.cpp fork, not in stock llama.cpp.
62
  - `keep8u-iq2xs/DeepSeek-V4-Coder-keep8u-iq2xs.gguf`: **Deprecated.** Emits garbage under upstream llama.cpp and the Ollama build (top-1 agreement 0 %), and keep8u itself is too small to be usable, so it is not requantised.
63
  - `keep8u/DeepSeek-V4-Coder-keep8u.gguf`: **Deprecated.** Eight experts per layer is not enough: both the previous file and a corrected rebuild degenerate into repetition ("a a a a ..."). Kept for reproducibility; do not use.
 
24
  files keep, in each layer, the experts that layer actually uses most (full-model `llama-imatrix`, 32 × 512
25
  tokens of mixed calibration text).
26
  3. **keep12u had zeroed layers.** The last 6.5 GiB of the previous file (layers 48–60) was all zeros.
27
+ 4. **The IQ2_XS files emitted garbage** under upstream llama.cpp: repeated tokens, top-1 agreement 0 %. The cause
28
+ is llama.cpp's IQ2_XS preset, which also puts attention, the indexer, the shared expert and the hyper-connection
29
+ mixers at 2 bits; a requant of our own with a correct imatrix failed the same way. In the new IQ2_XS files only
30
+ the routed-expert weights (most of the bytes) are IQ2_XS, with the full-model imatrix sliced to the kept
31
+ experts; every other tensor is Q4_K_M.
32
 
33
  ## Measured quality: read this before using them
34
 
 
42
  |---|---|---|---|---|---|---|
43
  | `keep12u` | Q3_K_M | 32.2 | 7.8× / 53% | 22.3× / 31% | 17.8× / 30% | 118.1× / 22% |
44
  | `keep16u` | Q3_K_M | 39.3 | 7.2× / 55% | 22.9× / 32% | 18.3× / 31% | 129.9× / 21% |
45
+ | `keep16u-iq2xs` | IQ2_XS | 31.1 | 9.0× / 52% | 26.6× / 30% | 20.5× / 30% | 143.1× / 20% |
46
 
47
  Previous files, for comparison (same test):
48
 
 
60
  ## Files not (yet) replaced
61
 
62
  - `keep12u-iq2xs/DeepSeek-V4-Coder-keep12u-iq2xs.gguf`: **Known broken (previous file).** Emits garbage under upstream llama.cpp (top-1 agreement with the full model 0 %). A corrected requantisation replaces it once it passes the same checks as the tiers above.
 
63
  - `keep64-iq2xs/DeepSeek-V4-Coder-keep64-iq2xxs.gguf`: Unchanged. Mixed expert counts behind a full-width router: loads only with UltraTensor's llama.cpp fork, not in stock llama.cpp.
64
  - `keep8u-iq2xs/DeepSeek-V4-Coder-keep8u-iq2xs.gguf`: **Deprecated.** Emits garbage under upstream llama.cpp and the Ollama build (top-1 agreement 0 %), and keep8u itself is too small to be usable, so it is not requantised.
65
  - `keep8u/DeepSeek-V4-Coder-keep8u.gguf`: **Deprecated.** Eight experts per layer is not enough: both the previous file and a corrected rebuild degenerate into repetition ("a a a a ..."). Kept for reproducibility; do not use.