Hugging Face's logo

gbuzhf
/
ICE-quantization

GGUF
ICE
ICE-Tiers
ice-quant
quantization
mixture-of-experts
llama.cpp
technical-report
Model card Files
xet
 Community
gbuzhf commited on
Commit
08ee00e
·
verified ·
1 Parent(s): f4fe197

ICE technical report v1.0 with full evidence appendix

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. README.md +138 -0
  2. appendix/SHA256SUMS.txt +58 -0
  3. appendix/measurements/ALLOCATION.txt +7 -0
  4. appendix/measurements/BAND_RESULT.txt +10 -0
  5. appendix/measurements/DENSE_RESULT.txt +8 -0
  6. appendix/measurements/FAM_RESULT.txt +8 -0
  7. appendix/measurements/FLOOR_RESULT.txt +1 -0
  8. appendix/measurements/ICE_RESULT.txt +10 -0
  9. appendix/measurements/PROBE_H_RESULT.txt +5 -0
  10. appendix/measurements/RESULT.txt +5 -0
  11. appendix/measurements/SENSITIVITY_SUMMARY.txt +42 -0
  12. appendix/measurements/V2_RESULT.txt +8 -0
  13. appendix/measurements/ice_final_ledger.txt +8 -0
  14. appendix/measurements/kld_A.txt +112 -0
  15. appendix/measurements/kld_B.txt +112 -0
  16. appendix/measurements/kld_BAND06.txt +112 -0
  17. appendix/measurements/kld_BAND12.txt +112 -0
  18. appendix/measurements/kld_BAND18.txt +112 -0
  19. appendix/measurements/kld_BAND25.txt +112 -0
  20. appendix/measurements/kld_C.txt +112 -0
  21. appendix/measurements/kld_CTRL-19G-ICE.txt +111 -0
  22. appendix/measurements/kld_CTRL-21G-ICE.txt +111 -0
  23. appendix/measurements/kld_CTRL-23G-ICE.txt +111 -0
  24. appendix/measurements/kld_CTRL-25G-ICE.txt +111 -0
  25. appendix/measurements/kld_D.txt +112 -0
  26. appendix/measurements/kld_DENSE_attention.txt +112 -0
  27. appendix/measurements/kld_DENSE_output.txt +112 -0
  28. appendix/measurements/kld_DENSE_token_embd.txt +112 -0
  29. appendix/measurements/kld_E.txt +112 -0
  30. appendix/measurements/kld_FAMEVEN-19G-ICE.txt +111 -0
  31. appendix/measurements/kld_FAMEVEN-21G-ICE.txt +111 -0
  32. appendix/measurements/kld_FAMSHALLOW-19G-ICE.txt +111 -0
  33. appendix/measurements/kld_FAMSHALLOW-21G-ICE.txt +111 -0
  34. appendix/measurements/kld_FLOORBF16.txt +111 -0
  35. appendix/measurements/kld_H.txt +112 -0
  36. appendix/measurements/kld_SHALLOW-19G-ICE.txt +111 -0
  37. appendix/measurements/kld_SHALLOW-21G-ICE.txt +111 -0
  38. appendix/measurements/kld_SHALLOW-23G-ICE.txt +111 -0
  39. appendix/measurements/kld_SHALLOW-25G-ICE.txt +111 -0
  40. appendix/measurements/kld_V2.txt +112 -0
  41. appendix/measurements/kldf_19G-ICE.txt +84 -0
  42. appendix/measurements/kldf_21G-ICE.txt +84 -0
  43. appendix/measurements/kldf_23G-ICE.txt +84 -0
  44. appendix/measurements/kldf_25G-ICE.txt +84 -0
  45. appendix/measurements/sensitivity.jsonl +0 -0
  46. appendix/original-ICE/PRINCIPLE.md +243 -0
  47. appendix/original-ICE/RECIPE.md +151 -0
  48. appendix/recipes/cfg_19G-ICE_ICEbase.txt +443 -0
  49. appendix/recipes/cfg_19G-ICE_previous.txt +443 -0
  50. appendix/recipes/cfg_21G-ICE_ICEbase.txt +443 -0
README.md ADDED
@@ -0,0 +1,138 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ tags:
4
+ - quantization
5
+ - gguf
6
+ - mixture-of-experts
7
+ - llama.cpp
8
+ - technical-report
9
+ ---
10
+
11
+ # ICE: Quantization by Error-Propagation Class in Sparse MoE Models
12
+
13
+ Technical report and full evidence for **ICE** (*Isolation of Compounding
14
+ Error*), a quantization bit-allocation method for sparse Mixture-of-Experts
15
+ checkpoints.
16
+
17
+ **Author:** Gökhan Buz ([`gbuzhf`](https://huggingface.co/gbuzhf))
18
+
19
+ ## Read the paper
20
+
21
+ **[`paper/ICE_Technical_Report.md`](paper/ICE_Technical_Report.md)**
22
+
23
+ ## What ICE is
24
+
25
+ Every quantizer in the GGUF ecosystem minimizes the same objective for every
26
+ tensor: the importance-matrix-weighted error of that tensor's output, **for the
27
+ current token**. That is the right objective for a tensor whose error dies with
28
+ the token. It is the wrong objective for three other kinds, and no published
29
+ method separates them.
30
+
31
+ ICE classifies every tensor by **how far its error travels**:
32
+
33
+ | class | mechanism | treatment |
34
+ |---|---|---|
35
+ | **DISCRETE** | error flips an argmax, so a different computation runs | exact (F32) |
36
+ | **RECURRENT** | error enters a state decay and compounds along the sequence | exact (F32) |
37
+ | **CACHED** | error is written to the KV cache once and re-read by every later token | near-exact (F16) |
38
+ | **INSTANT** | error affects this token only | this is where the budget lives |
39
+
40
+ The first three are 0.14% of the model, so freezing them is a line item rather
41
+ than a trade-off. In one line: **freeze what propagates, spend everything else on
42
+ the library.**
43
+
44
+ ## Headline results
45
+
46
+ Mean KL divergence against the bf16 checkpoint, WikiText-2, one harness for all
47
+ files.
48
+
49
+ | tier | size | mean KLD | nearest published tier | outcome |
50
+ |---|---:|---:|---|---|
51
+ | `23G-ICE` | 22.83 GB | 0.0361 | `UD-Q4_K_XL` 23.21 GB / 0.0380 | 0.38 GB smaller, 5.0% better |
52
+ | `23G-ICE` | 22.83 GB | 0.0361 | `APEX-I-Quality` 23.84 GB / 0.0415 | 1.01 GB smaller, 13.0% better |
53
+ | `25G-ICE` | 24.84 GB | 0.0303 | `APEX-I-Balanced` 26.28 GB / 0.0345 | 1.44 GB smaller, 12.2% better |
54
+ | `19G-ICE` | 18.82 GB | 0.0608 | `UD-IQ4_XS` 18.68 GB / 0.0723 | 15.8% better at +0.14 GB |
55
+
56
+ Across the twelve-tier comparison, nine tiers are Pareto-optimal and three are
57
+ strictly dominated. ICE does **not** win at the top of the ladder: `UD-Q5_K_S`
58
+ and `UD-Q6_K` are the two best files measured, which the paper explains rather
59
+ than omits (Law 4).
60
+
61
+ ## The four laws
62
+
63
+ 1. **Sparsity.** A bit on the always-on core is worth `E/k` bits on the expert
64
+ bank. Measured 31.8 against a predicted 32.
65
+ 2. **Convexity.** Error falls as `4^-b`, so allocation cleverness is capped at
66
+ **+0.139 bpw**. Two consequences: do no depth grading, and at a fixed average
67
+ always pick the **narrower type bracket** (measured +16.6% and +7.0% penalty
68
+ for widening).
69
+ 3. **Placement.** Inside a fixed bracket, shallow-first is worth about **−8.5%
70
+ KLD per bpw of gap** and **reverses below 0.47 bpw**. Depth gain measured
71
+ directly as `g(t) ≈ exp(−t/9.95)`.
72
+ 4. **The floor is epistemic.** Fitted on two independent harnesses at `k₀` =
73
+ 0.0205 and 0.0219, and confirmed by a direct probe at 0.018297. The best file
74
+ measured is 8% above it. **You cannot out-bit a wrong prior.**
75
+
76
+ ## What is in this repository
77
+
78
+ ```
79
+ paper/ICE_Technical_Report.md the report
80
+ appendix/original-ICE/ PRINCIPLE.md and RECIPE.md, the method as first written
81
+ appendix/recipes/ every tensor-type-file cited in the paper
82
+ appendix/measurements/ raw llama-perplexity output for every KLD quoted
83
+ appendix/SHA256SUMS.txt checksum for every file above
84
+ ```
85
+
86
+ ### `appendix/recipes/`
87
+
88
+ | suffix | meaning |
89
+ |---|---|
90
+ | `_ICEbase` | the current recipe, after the Law 3 revision |
91
+ | `_previous` | the recipe it replaced |
92
+ | `_comparison` | a published UD or APEX tier, as measured |
93
+
94
+ `cfg_21G-ICE_ICEbase.txt` and `cfg_21G-ICE_previous.txt` have the **same
95
+ SHA-256** (`93456d45b1fc0dab...`). That is not an oversight. 21G is the tier where
96
+ Law 3's condition does not hold, so the method's output is to change nothing, and
97
+ the identical checksum is the proof that nothing was changed.
98
+
99
+ Recipes for the three dominated tiers are not included: they were dropped after
100
+ the Pareto analysis and none was retained. Their measured numbers are in the
101
+ paper.
102
+
103
+ ## Reproducing a number
104
+
105
+ ```bash
106
+ llama-perplexity -m <file>.gguf -f wiki.test.raw \
107
+ -c 2048 --chunks 64 --kl-divergence-base base.kld --kl-divergence
108
+ ```
109
+
110
+ against a `base.kld` produced once from the bf16 checkpoint with the same corpus
111
+ and chunk count. Protocol details, including how the byte-identical control
112
+ variants are constructed, are in Appendix B.
113
+
114
+ The same recipe measured through three different paths gave 0.0345, 0.034162 and
115
+ 0.034290, a spread under 1%.
116
+
117
+ ## Negative results
118
+
119
+ Seven are documented at the same weight as the positive ones, including the
120
+ **retraction of a rule this work itself derived and shipped**: bumping `ffn_down`
121
+ above its sibling projections, which is standard practice, measured 11.3% worse
122
+ than uniform at identical size. One of three registered predictions also failed,
123
+ and it is scored as such.
124
+
125
+ ## On the comparison ladders
126
+
127
+ Unsloth Dynamic 2.0 and LocalAI APEX are the work of their respective authors.
128
+ They are measured here, not reproduced or modified. The comparison exists because
129
+ no individual publication can provide it: each ladder is published with its own
130
+ harness and its own reference, so the tiers are not comparable until someone puts
131
+ them on one. The same analysis that finds ICE tiers dominating three others also
132
+ finds two UD tiers to be the best files on the board and two APEX tiers to be the
133
+ only options below 18.5 GB.
134
+
135
+ ## Citing
136
+
137
+ Cite this repository. Model cards for ICE-quantized GGUFs link here for the
138
+ method description.
appendix/SHA256SUMS.txt ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 197b3ba93582e1cbdfa85777f95793f31b198ccd7a2c0037131e38de18843f61 *./measurements/ALLOCATION.txt
2
+ 460c97202411730e448b5a3d7047f556cc2c9a86f1415f78d25128a3d35e1443 *./measurements/BAND_RESULT.txt
3
+ 09ab6c5f7f5c648ccccc35c663c46a85c482bf1d9c13c9c4b0f37be8a57fb56d *./measurements/DENSE_RESULT.txt
4
+ 901f63d923864ebe24a53b8ef99502f29e8e3f1a778615452541fb7c3d244b82 *./measurements/FAM_RESULT.txt
5
+ d5db1b927e262c03e5c1ccd82db0ad813f4bcb4909b1d5eb7e04383b9a920220 *./measurements/FLOOR_RESULT.txt
6
+ 8c5fba182e91970383b2a90ec758946f28d6fbb732163afa40bac32106290457 *./measurements/ICE_RESULT.txt
7
+ c17a298c1ee4d2d808aa760cbd10c7b86dc5e5597c14692b49cbd206cf36a250 *./measurements/PROBE_H_RESULT.txt
8
+ 7461ec5e313b00a977c0e446f792c809969f1f1148c7d1571a3140c7a4baf29a *./measurements/RESULT.txt
9
+ e1d36c3b4bb25134691d4a596e5f98fb594d2ce3bc0f7997f82e3fafca6521d0 *./measurements/SENSITIVITY_SUMMARY.txt
10
+ c623c3a71eb6568fcbbe432abc707a7adf54c04a1f39df97ba912291cabda623 *./measurements/V2_RESULT.txt
11
+ 1a4a7fd427ea5572ec056535099c5d9efd578045ba505347a05c3e6d2576eeeb *./measurements/ice_final_ledger.txt
12
+ 1c027d2c9497ce413d37ce260b60a5df39a21a1cf44650c21fa68e4c4362b40b *./measurements/kld_A.txt
13
+ b50ae7929f494a32e6f97220c5084a83dbc772408bbc841366688eb07557b58c *./measurements/kld_B.txt
14
+ a94d51164840da70fde1be0a75b85d02387a90e238474578121ccad9df6fc4fa *./measurements/kld_BAND06.txt
15
+ 6acbf8a0d7537c74e89dcdf54457da9b8eb5471c1b8f6ceed217eb51f701ab44 *./measurements/kld_BAND12.txt
16
+ 804f3fd733597dc0c2befb2b3d3bb35ca01289e7343dcdaee7387c49415812bb *./measurements/kld_BAND18.txt
17
+ 0c71461bc53888542baf39fe9b7b455809e6cfe8f9ea8bcf4409e68d107941d7 *./measurements/kld_BAND25.txt
18
+ 1b2964ce70f4eb4967d7a5808882727cc603b01bd70c192d2659836756c88634 *./measurements/kld_C.txt
19
+ 8e0350a9a2701dbf79765d2dc80761ec31b60eade245c8ea1bc4b37527b3151f *./measurements/kld_CTRL-19G-ICE.txt
20
+ 24b2cd42e65233b49d00befd224aea275d37310a0527a29d390f8a836594ad08 *./measurements/kld_CTRL-21G-ICE.txt
21
+ c2d6c784625f82a8bcace8669b7a5972d5cbbd7f073846e17064ef9202b86cb7 *./measurements/kld_CTRL-23G-ICE.txt
22
+ 8e40a4f1bd4a1a3723fbcf8bd7911993a3d1b0e2b123bc13d7e4d8da28b0f96f *./measurements/kld_CTRL-25G-ICE.txt
23
+ 03926050213470e7299d468e0405f4223c9ba4feb5b30f61ce4ceba18c8d87d7 *./measurements/kld_D.txt
24
+ 0ddf6ace773708d984fb71a84bc5ea158702f23b74eac0c56359724b6174bc0a *./measurements/kld_DENSE_attention.txt
25
+ 31416b7bd2b70efd0110719b44cae4c0b916d852f45f745238440f7684bafdd6 *./measurements/kld_DENSE_output.txt
26
+ 41a4310ff9bd45a9d42290a1d7e43c16bd9482af12bac675e8fa1c7d572d2b59 *./measurements/kld_DENSE_token_embd.txt
27
+ c08f34da0fddcab51dbf293a05422c8c1f4033a082d395d4b85a0d568d05fab1 *./measurements/kld_E.txt
28
+ 5832beaaac14edaf9f8d3c3d219b779312294d4a13e8507a346e329b0dfe3675 *./measurements/kld_FAMEVEN-19G-ICE.txt
29
+ 93ee48836531052f0defaa8e69e250886861ca7ffe4af809bc2914b478ab60b4 *./measurements/kld_FAMEVEN-21G-ICE.txt
30
+ e4ce62ac6d90bfeced9e44c76bfd2ac5d82be1509f1875cc88c00acb466aecee *./measurements/kld_FAMSHALLOW-19G-ICE.txt
31
+ 5331e3f0f6347d3d742992d14f159f6a09e45b611559a9d478c620f1a8486a45 *./measurements/kld_FAMSHALLOW-21G-ICE.txt
32
+ 6f309879c98e8837d20fab8c9bc27f102d95c0f3db9acc34bcbe46be6cc8f0a1 *./measurements/kld_FLOORBF16.txt
33
+ 1a4643f4aa00527025c291f7f8e07bea7bae141bf7838282c4111606e3f70e97 *./measurements/kld_H.txt
34
+ 0d555044e7beea614bba8d558d2aa2fc8fc61b720615c21be8cab3655bbb5e64 *./measurements/kld_SHALLOW-19G-ICE.txt
35
+ 136cc786cb10b2334cc38976052bbe9a87fbdb62399db34a3eda8578739a1dc5 *./measurements/kld_SHALLOW-21G-ICE.txt
36
+ ede955af97f40cf9d6d7c9a7c07ea0c8dbb156ac54ffa9648ffae12032cdc8f6 *./measurements/kld_SHALLOW-23G-ICE.txt
37
+ 0bde773519317a049e5ff43bfeb3b3d8d65cf786e0e95ed35381c469d46c27c2 *./measurements/kld_SHALLOW-25G-ICE.txt
38
+ d5c2ade2ff18b37bef4760d18bff321214ed700c8c046258855ff7903eb6d981 *./measurements/kld_V2.txt
39
+ ba343c8e15674ae88ceb88e89f3a2c5a59d1fef3cf3dc79aea920107a6720027 *./measurements/kldf_19G-ICE.txt
40
+ f3d5aaf1a5f2863a31992443b40d2d9d718b75c9457f71856266daf808734afc *./measurements/kldf_21G-ICE.txt
41
+ 3b42f2045f78a390b0d57c50c218022e627a31e212e176aa6941c132fc848e07 *./measurements/kldf_23G-ICE.txt
42
+ b0f4e20583b69ba163d778a8d7e79c755c715a49a5ddec459c874395c487e5d9 *./measurements/kldf_25G-ICE.txt
43
+ 08d786bffc08ee3de0a0673f2a96ac321ef29bf71431cd4c643fc6cf335282e6 *./measurements/sensitivity.jsonl
44
+ b0cf96c35815e8dcb1edd8e8c7e03a126d5114309ed4e5e11dee77963f7177f7 *./original-ICE/PRINCIPLE.md
45
+ 2480e950488e268b90e10ee4c5f3556c456660af5361595bfc417fc5827c1ac8 *./original-ICE/RECIPE.md
46
+ efc76732cf05eb6fdf750336a0a9ac574c495b87b4bddc0bdb5c321a3b5295c1 *./recipes/cfg_19G-ICE_ICEbase.txt
47
+ 6a9d088d39692d2823daabc9467e011f447b91a481e39c38fa4b24fe18554f77 *./recipes/cfg_19G-ICE_previous.txt
48
+ 93456d45b1fc0dab59ac03c1a2773c5e74049b2d264241aa07fb6b67d27cbd79 *./recipes/cfg_21G-ICE_ICEbase.txt
49
+ 93456d45b1fc0dab59ac03c1a2773c5e74049b2d264241aa07fb6b67d27cbd79 *./recipes/cfg_21G-ICE_previous.txt
50
+ 2e3cbb15d39ffcf569e610efd8c94eb6988597c814f563609984809f9aac00b8 *./recipes/cfg_23G-ICE_ICEbase.txt
51
+ 1d461db13b2900514641f7e32c40ae7af257daf660ca184e1d2a56ff39eafa20 *./recipes/cfg_23G-ICE_previous.txt
52
+ 515f283171280f14f3913ed5caa7913cf42f73a3a6f5c942ef9530a294282558 *./recipes/cfg_25G-ICE_ICEbase.txt
53
+ 4fb6900bff49459a7948f0518d4008b7b13a1fc73b6c6f9cf343f0b5eca56024 *./recipes/cfg_25G-ICE_previous.txt
54
+ 0251845d1a524151c890ea491a7e1adc62ec3da67a888ea4cb401072e03ad9de *./recipes/cfg_APEX-I-Compact_comparison.txt
55
+ 0d3adc95d1848cb91231120af4d4e1c7e9a9d27e146c50124869ff2a993b57ab *./recipes/cfg_APEX-I-Mini_comparison.txt
56
+ e88b620a8c80cab33661be765ed9f01ec9a56d1ee9ae239d429a1f19e0aa5f2d *./recipes/cfg_UD-IQ4_XS_comparison.txt
57
+ d4f78a46b67f7105edf66f8c6a0a8a99dafa4337b6dbe707128908d837618cb5 *./recipes/cfg_UD-Q5_K_S_comparison.txt
58
+ cf561e0bbb74d0fdf3ea49cb85006c0221a180cfbeb180c91fb3ca160b7dc7d1 *./recipes/cfg_UD-Q6_K_comparison.txt
appendix/measurements/ALLOCATION.txt ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ control expert budget: 19.629 GB over 120 tensors (4.8750 bpw)
2
+ control predicted expert error S = 8.20534e+07
3
+
4
+ C per-tensor : 19.622 GB (-0.0380% vs control), S = 2.79971e+07 (-65.9%)
5
+ B block-level : 19.592 GB (-0.1897% vs control), S = 5.66933e+07 (-30.9%)
6
+ B -> /workspace/dosa/cfg_B.txt: 96 lines differ from control, all expert. mix {'Q5_K': 63, 'IQ4_XS': 51, 'IQ3_S': 6}
7
+ C -> /workspace/dosa/cfg_C.txt: 100 lines differ from control, all expert. mix {'Q6_K': 15, 'Q5_K': 55, 'Q4_K': 2, 'IQ4_XS': 12, 'IQ3_S': 36}
appendix/measurements/BAND_RESULT.txt ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ g(depth) band sweep — control's own Q5_K/Q4_K mix, 15-block band slid across depth
2
+ every band byte-identical to the control; depth is the only variable. 64 chunks.
3
+
4
+ band [ 0..14] centre 7.0 0.032407 +- 0.000458 (probe E)
5
+ band [ 6..20] centre 0.0 0.033014 ± 0.000492
6
+ band [12..26] centre 0.0 0.036760 ± 0.000572
7
+ band [18..32] centre 0.0 0.036107 ± 0.000517
8
+ band [25..39] centre 0.0 0.036136 ± 0.000542
9
+
10
+ reference: A even dither 0.034162 | D experts Q8_0 0.019018 (floor)
appendix/measurements/DENSE_RESULT.txt ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ Dense-floor decomposition, complete
2
+ reference D (experts Q8_0, nothing lifted) = 0.019018
3
+ true floor H (experts BF16) = 0.018297
4
+
5
+ role KLD with it lifted its share of the floor
6
+ output 0.018941 0.000077
7
+ token_embd 0.019217 -0.000199
8
+ attention 0.017510 0.001508
appendix/measurements/FAM_RESULT.txt ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ Same-family (Q4_K/Q3_K) vs the shipped mixed pair, byte-matched
2
+ 21G control (Q4_K/IQ4_XS) 0.039182 | shallow on that pair 0.039814 (+1.6%)
3
+ 19G control (IQ4_XS/IQ3_S) 0.060015 | shallow on that pair 0.058676 (-2.2%)
4
+
5
+ FAMEVEN-21G-ICE Mean KLD: 0.045673 ± 0.000564 Same top p: 91.346 ± 0.110 %
6
+ FAMSHALLOW-21G-ICE Mean KLD: 0.043766 ± 0.000525 Same top p: 91.647 ± 0.108 %
7
+ FAMEVEN-19G-ICE Mean KLD: 0.064213 ± 0.000727 Same top p: 89.953 ± 0.117 %
8
+ FAMSHALLOW-19G-ICE Mean KLD: 0.059623 ± 0.000672 Same top p: 90.315 ± 0.116 %
appendix/measurements/FLOOR_RESULT.txt ADDED
@@ -0,0 +1 @@
 
 
1
+ FLOOR-BF16 size=24589334592 Mean KLD: 0.031543 ± 0.000428
appendix/measurements/ICE_RESULT.txt ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ ICE base — shallow-first rule, each tier against its own control
2
+ quantiser: pinned d775b896 (CPU) | measurement: prebuilt 3653e6d (CUDA), 64 chunks
3
+ every CTRL/SHALLOW pair is byte-identical; absolute values not comparable
4
+ to the phase 2-4 CPU table, only within each pair.
5
+
6
+ tier control shallow delta
7
+ 19G-ICE 0.060015 0.058676 -2.2%
8
+ 21G-ICE 0.039182 0.039814 +1.6%
9
+ 23G-ICE 0.034290 0.032536 -5.1%
10
+ 25G-ICE 0.028955 0.028131 -2.8%
appendix/measurements/PROBE_H_RESULT.txt ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ Error-budget decomposition at the 23G-ICE dense configuration, 64 chunks
2
+
3
+ A control, experts 4.875 bpw 0.034162 +- 0.000455 92.579%
4
+ D experts Q8_0 0.019018 +- 0.000333 94.601%
5
+ H experts BF16 (lossless) Mean KLD: 0.018297 ± 0.000349 Same top p: 94.790 ± 0.087 %
appendix/measurements/RESULT.txt ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ DOSA Phase 2 — one byte budget (23G-ICE), three allocations, 64 chunks
2
+
3
+ A Mean KLD: 0.034162 ± 0.000455 Same top p: 92.579 ± 0.102 % size=22338172992
4
+ B Mean KLD: 0.042338 ± 0.000606 Same top p: 91.940 ± 0.106 % size=22300424256
5
+ C Mean KLD: 0.050559 ± 0.000627 Same top p: 91.239 ± 0.110 % size=22327687232
appendix/measurements/SENSITIVITY_SUMMARY.txt ADDED
@@ -0,0 +1,42 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ types: ['IQ3_S', 'IQ4_XS', 'Q4_K', 'Q5_K', 'Q6_K'] tensors per type: {'Q6_K': 120, 'Q5_K': 120, 'Q4_K': 120, 'IQ4_XS': 120, 'IQ3_S': 120}
2
+
3
+ IQ3_S s: min 3161 max 1.418e+07 ratio 4487.1x CV 1.022 rho(s, imatrix-energy) +0.9955 rho(s, raw err) +0.5514
4
+ IQ4_XS s: min 1260 max 4.511e+06 ratio 3580.4x CV 1.022 rho(s, imatrix-energy) +0.9951 rho(s, raw err) +0.6734
5
+ Q4_K s: min 904.6 max 3.825e+06 ratio 4228.4x CV 1.026 rho(s, imatrix-energy) +0.9952 rho(s, raw err) +0.6796
6
+ Q5_K s: min 241.2 max 9.681e+05 ratio 4013.5x CV 1.024 rho(s, imatrix-energy) +0.9951 rho(s, raw err) +0.6842
7
+ Q6_K s: min 61.66 max 2.394e+05 ratio 3882.3x CV 1.009 rho(s, imatrix-energy) +0.9956 rho(s, raw err) +0.6812
8
+
9
+ top 12 most sensitive tensors at IQ3_S (these must get the bits):
10
+ blk 38 gate s=1.418e+07
11
+ blk 37 gate s=1.416e+07
12
+ blk 36 gate s=1.356e+07
13
+ blk 38 up s=1.264e+07
14
+ blk 37 up s=1.255e+07
15
+ blk 39 gate s=1.218e+07
16
+ blk 35 gate s=1.21e+07
17
+ blk 36 up s=1.183e+07
18
+ blk 39 up s=1.039e+07
19
+ blk 35 up s=1.039e+07
20
+ blk 34 gate s=1.006e+07
21
+ blk 33 gate s=9.559e+06
22
+
23
+ bottom 12 (these can be starved):
24
+ blk 0 down s=3161
25
+ blk 2 down s=8085
26
+ blk 1 down s=8344
27
+ blk 3 down s=1.148e+04
28
+ blk 4 down s=1.596e+04
29
+ blk 5 down s=2.025e+04
30
+ blk 6 down s=2.267e+04
31
+ blk 7 down s=2.855e+04
32
+ blk 8 down s=3.134e+04
33
+ blk 9 down s=3.5e+04
34
+ blk 10 down s=3.822e+04
35
+ blk 12 down s=4.438e+04
36
+
37
+ role totals (share of total sensitivity):
38
+ IQ3_S gate= 51.4% up= 46.3% down= 2.3%
39
+ IQ4_XS gate= 51.4% up= 46.2% down= 2.4%
40
+ Q4_K gate= 51.3% up= 46.3% down= 2.4%
41
+ Q5_K gate= 51.3% up= 46.3% down= 2.4%
42
+ Q6_K gate= 51.0% up= 46.6% down= 2.4%
appendix/measurements/V2_RESULT.txt ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ Corrected allocation, validated — one byte budget, 64 chunks, same base.kld
2
+
3
+ A control (even dither) 0.034162 +- 0.000455 92.579%
4
+ E band[0..14] (crude shallow) 0.032407 +- 0.000458 92.824%
5
+ V2 corrected g(t)*s_t Mean KLD: 0.033769 ± 0.000483 Same top p: 92.589 ± 0.102 %
6
+
7
+ floor (H, experts lossless) 0.018297
8
+ V2 size: 22338172992 bytes
appendix/measurements/ice_final_ledger.txt ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ 04:45:16 KLD 25G-ICE: Mean KLD: 0.030304 ± 0.000779
2
+ 04:45:58 25G-ICE uploaded+verified 24849761792 B, freed
3
+ 04:48:32 KLD 23G-ICE: Mean KLD: 0.036069 ± 0.000868
4
+ 04:49:07 23G-ICE uploaded+verified 22836495872 B, freed
5
+ 04:52:08 KLD 21G-ICE: Mean KLD: 0.041192 ± 0.001140
6
+ 04:52:39 21G-ICE uploaded+verified 20848395776 B, freed
7
+ 04:57:04 KLD 19G-ICE: Mean KLD: 0.060844 ± 0.001185
8
+ 04:57:45 19G-ICE uploaded+verified 18822546944 B, freed
appendix/measurements/kld_A.txt ADDED
@@ -0,0 +1,112 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.18.483.679 I cmn init: llama threadpool init, n_threads = 30
2
+ 0.18.616.495 I
3
+ 0.18.616.655 I system_info: n_threads = 30 (n_threads_batch = 30) / 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
4
+ 0.18.972.875 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
5
+ 0.38.595.898 I kl_divergence: 19.62 seconds per pass - ETA 20.92 minutes
6
+
7
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
8
+ 1 5.8539 ± 0.4567 -0.00061 ± 0.01224 0.04542 ± 0.00344 7.033 ± 0.568 % 91.984 ± 0.849 %
9
+ 2 7.3443 ± 0.4257 0.00485 ± 0.00822 0.03957 ± 0.00196 5.940 ± 0.365 % 91.544 ± 0.615 %
10
+ 3 8.3070 ± 0.4097 0.01310 ± 0.00656 0.03660 ± 0.00154 5.525 ± 0.303 % 91.952 ± 0.491 %
11
+ 4 9.3665 ± 0.4262 0.00485 ± 0.00585 0.04045 ± 0.00213 5.620 ± 0.283 % 91.667 ± 0.432 %
12
+ 5 8.7243 ± 0.3493 0.00138 ± 0.00521 0.03966 ± 0.00177 5.542 ± 0.242 % 91.613 ± 0.388 %
13
+ 6 7.4068 ± 0.2619 0.00029 ± 0.00465 0.03864 ± 0.00159 5.582 ± 0.230 % 91.707 ± 0.352 %
14
+ 7 6.7449 ± 0.2167 -0.00776 ± 0.00442 0.03943 ± 0.00144 5.798 ± 0.213 % 91.761 ± 0.325 %
15
+ 8 6.5552 ± 0.1943 -0.00678 ± 0.00402 0.03812 ± 0.00128 5.718 ± 0.193 % 91.874 ± 0.302 %
16
+ 9 6.9515 ± 0.1960 -0.00642 ± 0.00379 0.03888 ± 0.00118 5.638 ± 0.176 % 91.745 ± 0.287 %
17
+ 10 7.0140 ± 0.1878 -0.00659 ± 0.00353 0.03783 ± 0.00109 5.567 ± 0.172 % 91.945 ± 0.269 %
18
+ 11 7.0196 ± 0.1780 -0.00632 ± 0.00334 0.03657 ± 0.00101 5.441 ± 0.161 % 92.127 ± 0.254 %
19
+ 12 7.2843 ± 0.1781 -0.00556 ± 0.00314 0.03544 ± 0.00093 5.327 ± 0.152 % 92.204 ± 0.242 %
20
+ 13 7.3923 ± 0.1743 -0.00611 ± 0.00303 0.03556 ± 0.00092 5.389 ± 0.146 % 92.300 ± 0.231 %
21
+ 14 7.4356 ± 0.1686 -0.00662 ± 0.00289 0.03486 ± 0.00086 5.328 ± 0.141 % 92.382 ± 0.222 %
22
+ 15 7.4746 ± 0.1636 -0.00919 ± 0.00279 0.03493 ± 0.00083 5.305 ± 0.134 % 92.297 ± 0.215 %
23
+ 16 7.6571 ± 0.1624 -0.00981 ± 0.00268 0.03449 ± 0.00078 5.240 ± 0.128 % 92.388 ± 0.207 %
24
+ 17 7.6643 ± 0.1568 -0.00982 ± 0.00256 0.03378 ± 0.00074 5.151 ± 0.122 % 92.473 ± 0.200 %
25
+ 18 7.7467 ± 0.1539 -0.00936 ± 0.00247 0.03347 ± 0.00071 5.116 ± 0.118 % 92.522 ± 0.194 %
26
+ 19 7.6600 ± 0.1484 -0.01121 ± 0.00240 0.03328 ± 0.00068 5.071 ± 0.113 % 92.525 ± 0.189 %
27
+ 20 7.4027 ± 0.1392 -0.01298 ± 0.00246 0.03649 ± 0.00088 5.420 ± 0.121 % 92.380 ± 0.185 %
28
+ 21 7.4475 ± 0.1366 -0.01238 ± 0.00242 0.03718 ± 0.00088 5.481 ± 0.120 % 92.282 ± 0.182 %
29
+ 22 7.5182 ± 0.1353 -0.01297 ± 0.00235 0.03680 ± 0.00084 5.460 ± 0.116 % 92.318 ± 0.178 %
30
+ 23 7.5483 ± 0.1326 -0.01247 ± 0.00227 0.03634 ± 0.00081 5.399 ± 0.112 % 92.341 ± 0.173 %
31
+ 24 7.6384 ± 0.1317 -0.01419 ± 0.00224 0.03690 ± 0.00082 5.461 ± 0.113 % 92.302 ± 0.170 %
32
+ 25 7.7048 ± 0.1305 -0.01397 ± 0.00219 0.03659 ± 0.00079 5.406 ± 0.110 % 92.289 ± 0.167 %
33
+ 26 7.6580 ± 0.1269 -0.01342 ± 0.00213 0.03624 ± 0.00076 5.382 ± 0.107 % 92.372 ± 0.163 %
34
+ 27 7.8097 ± 0.1274 -0.01350 ± 0.00207 0.03585 ± 0.00073 5.333 ± 0.104 % 92.365 ± 0.160 %
35
+ 28 7.8916 ± 0.1266 -0.01281 ± 0.00201 0.03543 ± 0.00071 5.287 ± 0.102 % 92.400 ± 0.157 %
36
+ 29 7.8961 ± 0.1247 -0.01180 ± 0.00198 0.03558 ± 0.00069 5.317 ± 0.099 % 92.436 ± 0.154 %
37
+ 30 7.9215 ± 0.1234 -0.01463 ± 0.00198 0.03588 ± 0.00068 5.321 ± 0.096 % 92.405 ± 0.151 %
38
+ 31 7.8122 ± 0.1193 -0.01463 ± 0.00194 0.03566 ± 0.00066 5.291 ± 0.094 % 92.461 ± 0.148 %
39
+ 32 7.6843 ± 0.1154 -0.01350 ± 0.00191 0.03561 ± 0.00065 5.335 ± 0.093 % 92.519 ± 0.145 %
40
+ 33 7.6706 ± 0.1136 -0.01319 ± 0.00188 0.03570 ± 0.00066 5.332 ± 0.091 % 92.521 ± 0.143 %
41
+ 34 7.6455 ± 0.1113 -0.01337 ± 0.00184 0.03539 ± 0.00064 5.313 ± 0.089 % 92.502 ± 0.141 %
42
+ 35 7.6425 ± 0.1097 -0.01357 ± 0.00181 0.03520 ± 0.00063 5.288 ± 0.087 % 92.501 ± 0.139 %
43
+ 36 7.6463 ± 0.1083 -0.01375 ± 0.00177 0.03497 ± 0.00061 5.275 ± 0.086 % 92.557 ± 0.137 %
44
+ 37 7.5036 ± 0.1043 -0.01342 ± 0.00174 0.03453 ± 0.00060 5.244 ± 0.084 % 92.608 ± 0.134 %
45
+ 38 7.3942 ± 0.1009 -0.01314 ± 0.00170 0.03422 ± 0.00058 5.220 ± 0.083 % 92.648 ± 0.132 %
46
+ 39 7.2821 ± 0.0977 -0.01254 ± 0.00168 0.03397 ± 0.00057 5.209 ± 0.082 % 92.656 ± 0.131 %
47
+ 40 7.1566 ± 0.0943 -0.01215 ± 0.00164 0.03359 ± 0.00056 5.195 ± 0.080 % 92.710 ± 0.129 %
48
+ 41 7.0789 ± 0.0918 -0.01169 ± 0.00162 0.03326 ± 0.00055 5.170 ± 0.079 % 92.723 ± 0.127 %
49
+ 42 7.1098 ± 0.0911 -0.01148 ± 0.00159 0.03316 ± 0.00053 5.157 ± 0.078 % 92.699 ± 0.126 %
50
+ 43 7.3081 ± 0.0935 -0.01146 ± 0.00158 0.03334 ± 0.00053 5.168 ± 0.079 % 92.650 ± 0.124 %
51
+ 44 7.4477 ± 0.0947 -0.01127 ± 0.00155 0.03326 ± 0.00052 5.135 ± 0.078 % 92.644 ± 0.123 %
52
+ 45 7.4301 ± 0.0933 -0.00966 ± 0.00155 0.03443 ± 0.00052 5.241 ± 0.077 % 92.553 ± 0.122 %
53
+ 46 7.4358 ± 0.0923 -0.00950 ± 0.00154 0.03468 ± 0.00052 5.239 ± 0.076 % 92.528 ± 0.121 %
54
+ 47 7.5004 ± 0.0923 -0.00967 ± 0.00153 0.03481 ± 0.00052 5.238 ± 0.075 % 92.527 ± 0.120 %
55
+ 48 7.4980 ± 0.0911 -0.00939 ± 0.00150 0.03464 ± 0.00051 5.223 ± 0.074 % 92.536 ± 0.119 %
56
+ 49 7.5493 ± 0.0909 -0.00915 ± 0.00149 0.03462 ± 0.00050 5.222 ± 0.074 % 92.493 ± 0.118 %
57
+ 50 7.5033 ± 0.0893 -0.00901 ± 0.00147 0.03437 ± 0.00050 5.203 ± 0.073 % 92.542 ± 0.116 %
58
+ 51 7.5390 ± 0.0888 -0.00849 ± 0.00145 0.03402 ± 0.00049 5.168 ± 0.072 % 92.561 ± 0.115 %
59
+ 52 7.4734 ± 0.0870 -0.00883 ± 0.00143 0.03388 ± 0.00048 5.161 ± 0.071 % 92.584 ± 0.114 %
60
+ 53 7.4441 ± 0.0857 -0.00835 ± 0.00141 0.03370 ± 0.00047 5.149 ± 0.070 % 92.599 ± 0.112 %
61
+ 54 7.4011 ± 0.0843 -0.00857 ± 0.00139 0.03348 ± 0.00047 5.139 ± 0.069 % 92.613 ± 0.111 %
62
+ 55 7.3598 ± 0.0829 -0.00913 ± 0.00139 0.03347 ± 0.00047 5.148 ± 0.069 % 92.610 ± 0.110 %
63
+ 56 7.3527 ± 0.0821 -0.00923 ± 0.00137 0.03343 ± 0.00046 5.146 ± 0.068 % 92.618 ± 0.109 %
64
+ 57 7.2747 ± 0.0803 -0.00914 ± 0.00136 0.03321 ± 0.00045 5.125 ± 0.067 % 92.653 ± 0.108 %
65
+ 58 7.3124 ± 0.0803 -0.00878 ± 0.00136 0.03371 ± 0.00046 5.165 ± 0.067 % 92.637 ± 0.107 %
66
+ 59 7.2845 ± 0.0792 -0.00861 ± 0.00135 0.03409 ± 0.00046 5.200 ± 0.067 % 92.631 ± 0.106 %
67
+ 60 7.2650 ± 0.0783 -0.00893 ± 0.00134 0.03392 ± 0.00045 5.185 ± 0.066 % 92.639 ± 0.105 %
68
+ 61 7.3309 ± 0.0787 -0.01074 ± 0.00135 0.03461 ± 0.00047 5.244 ± 0.067 % 92.580 ± 0.105 %
69
+ 62 7.3542 ± 0.0783 -0.01059 ± 0.00133 0.03446 ± 0.00047 5.224 ± 0.066 % 92.579 ± 0.104 %
70
+ 63 7.4011 ± 0.0783 -0.01029 ± 0.00132 0.03428 ± 0.00046 5.203 ± 0.066 % 92.574 ± 0.103 %
71
+ 64 7.4319 ± 0.0780 -0.01032 ± 0.00130 0.03416 ± 0.00046 5.189 ± 0.065 % 92.579 ± 0.102 %
72
+
73
+ ====== Perplexity statistics ======
74
+ Mean PPL(Q) : 7.431939 ± 0.078018
75
+ Mean PPL(base) : 7.509043 ± 0.079361
76
+ Cor(ln(PPL(Q)), ln(PPL(base))): 99.23%
77
+ Mean ln(PPL(Q)/PPL(base)) : -0.010321 ± 0.001305
78
+ Mean PPL(Q)/PPL(base) : 0.989732 ± 0.001291
79
+ Mean PPL(Q)-PPL(base) : -0.077104 ± 0.009826
80
+
81
+ ====== KL divergence statistics ======
82
+ Mean KLD: 0.034162 ± 0.000455
83
+ Maximum KLD: 7.973917
84
+ 99.9% KLD: 1.355159
85
+ 99.0% KLD: 0.361620
86
+ 95.0% KLD: 0.122376
87
+ 90.0% KLD: 0.071926
88
+ Median KLD: 0.011761
89
+ 10.0% KLD: 0.000184
90
+ 5.0% KLD: 0.000052
91
+ 1.0% KLD: 0.000000
92
+ 0.1% KLD: -0.000147
93
+ Minimum KLD: -0.001017
94
+
95
+ ====== Token probability statistics ======
96
+ Mean Δp: -0.126 ± 0.020 %
97
+ Maximum Δp: 88.179%
98
+ 99.9% Δp: 39.137%
99
+ 99.0% Δp: 15.728%
100
+ 95.0% Δp: 6.152%
101
+ 90.0% Δp: 3.233%
102
+ 75.0% Δp: 0.522%
103
+ Median Δp: -0.002%
104
+ 25.0% Δp: -0.758%
105
+ 10.0% Δp: -3.730%
106
+ 5.0% Δp: -6.619%
107
+ 1.0% Δp: -15.617%
108
+ 0.1% Δp: -41.049%
109
+ Minimum Δp: -93.760%
110
+ RMS Δp : 5.189 ± 0.065 %
111
+ Same top p: 92.579 ± 0.102 %
112
+
appendix/measurements/kld_B.txt ADDED
@@ -0,0 +1,112 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.02.564.663 I cmn init: llama threadpool init, n_threads = 30
2
+ 0.02.666.488 I
3
+ 0.02.666.677 I system_info: n_threads = 30 (n_threads_batch = 30) / 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
4
+ 0.03.039.270 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
5
+ 0.20.055.609 I kl_divergence: 17.02 seconds per pass - ETA 18.15 minutes
6
+
7
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
8
+ 1 5.7012 ± 0.4441 -0.02705 ± 0.01401 0.06733 ± 0.00483 8.250 ± 0.604 % 90.127 ± 0.933 %
9
+ 2 7.1282 ± 0.4100 -0.02501 ± 0.00929 0.05204 ± 0.00269 6.884 ± 0.400 % 90.616 ± 0.645 %
10
+ 3 8.0373 ± 0.3930 -0.01992 ± 0.00753 0.05041 ± 0.00320 6.614 ± 0.378 % 91.398 ± 0.506 %
11
+ 4 8.8810 ± 0.3972 -0.04837 ± 0.00730 0.06139 ± 0.00481 7.055 ± 0.370 % 91.251 ± 0.442 %
12
+ 5 8.3798 ± 0.3314 -0.03891 ± 0.00626 0.05611 ± 0.00388 6.615 ± 0.318 % 91.496 ± 0.390 %
13
+ 6 7.1462 ± 0.2492 -0.03553 ± 0.00556 0.05375 ± 0.00329 6.621 ± 0.277 % 91.544 ± 0.355 %
14
+ 7 6.5783 ± 0.2091 -0.03278 ± 0.00508 0.05161 ± 0.00285 6.541 ± 0.246 % 91.831 ± 0.324 %
15
+ 8 6.4147 ± 0.1882 -0.02846 ± 0.00460 0.04911 ± 0.00252 6.305 ± 0.224 % 91.997 ± 0.300 %
16
+ 9 6.7792 ± 0.1891 -0.03151 ± 0.00432 0.04906 ± 0.00226 6.283 ± 0.206 % 91.669 ± 0.288 %
17
+ 10 6.8538 ± 0.1816 -0.02970 ± 0.00401 0.04710 ± 0.00204 6.143 ± 0.195 % 91.623 ± 0.274 %
18
+ 11 6.8704 ± 0.1725 -0.02781 ± 0.00381 0.04566 ± 0.00187 6.055 ± 0.185 % 91.771 ± 0.259 %
19
+ 12 7.1196 ± 0.1722 -0.02844 ± 0.00357 0.04407 ± 0.00172 5.900 ± 0.175 % 91.805 ± 0.248 %
20
+ 13 7.2499 ± 0.1695 -0.02556 ± 0.00341 0.04322 ± 0.00159 5.848 ± 0.164 % 91.804 ± 0.238 %
21
+ 14 7.2938 ± 0.1640 -0.02588 ± 0.00325 0.04266 ± 0.00150 5.818 ± 0.161 % 91.824 ± 0.229 %
22
+ 15 7.3555 ± 0.1599 -0.02524 ± 0.00313 0.04204 ± 0.00141 5.792 ± 0.155 % 91.854 ± 0.221 %
23
+ 16 7.5456 ± 0.1590 -0.02447 ± 0.00301 0.04153 ± 0.00133 5.741 ± 0.149 % 91.874 ± 0.214 %
24
+ 17 7.5630 ± 0.1539 -0.02312 ± 0.00287 0.04045 ± 0.00126 5.648 ± 0.143 % 91.956 ± 0.206 %
25
+ 18 7.6457 ± 0.1511 -0.02248 ± 0.00276 0.03985 ± 0.00119 5.590 ± 0.137 % 92.044 ± 0.199 %
26
+ 19 7.5540 ± 0.1456 -0.02513 ± 0.00273 0.04080 ± 0.00116 5.648 ± 0.132 % 91.943 ± 0.195 %
27
+ 20 7.3209 ± 0.1369 -0.02410 ± 0.00276 0.04431 ± 0.00125 6.029 ± 0.138 % 91.799 ± 0.192 %
28
+ 21 7.3669 ± 0.1345 -0.02327 ± 0.00268 0.04427 ± 0.00120 6.030 ± 0.135 % 91.789 ± 0.187 %
29
+ 22 7.4308 ± 0.1330 -0.02465 ± 0.00261 0.04401 ± 0.00115 5.987 ± 0.130 % 91.789 ± 0.183 %
30
+ 23 7.4636 ± 0.1304 -0.02377 ± 0.00253 0.04347 ± 0.00110 5.922 ± 0.126 % 91.848 ± 0.178 %
31
+ 24 7.5419 ± 0.1293 -0.02691 ± 0.00250 0.04425 ± 0.00107 5.953 ± 0.122 % 91.768 ± 0.175 %
32
+ 25 7.5928 ± 0.1278 -0.02861 ± 0.00245 0.04426 ± 0.00104 5.914 ± 0.118 % 91.738 ± 0.172 %
33
+ 26 7.5481 ± 0.1243 -0.02788 ± 0.00239 0.04382 ± 0.00100 5.879 ± 0.115 % 91.804 ± 0.168 %
34
+ 27 7.6960 ± 0.1246 -0.02817 ± 0.00234 0.04353 ± 0.00097 5.842 ± 0.112 % 91.695 ± 0.166 %
35
+ 28 7.7640 ± 0.1236 -0.02911 ± 0.00229 0.04335 ± 0.00094 5.805 ± 0.110 % 91.670 ± 0.163 %
36
+ 29 7.7556 ± 0.1214 -0.02976 ± 0.00226 0.04368 ± 0.00092 5.876 ± 0.108 % 91.630 ± 0.161 %
37
+ 30 7.7690 ± 0.1199 -0.03408 ± 0.00226 0.04468 ± 0.00092 5.920 ± 0.107 % 91.561 ± 0.159 %
38
+ 31 7.6631 ± 0.1160 -0.03390 ± 0.00222 0.04437 ± 0.00089 5.890 ± 0.104 % 91.600 ± 0.156 %
39
+ 32 7.5283 ± 0.1119 -0.03402 ± 0.00218 0.04449 ± 0.00088 5.940 ± 0.103 % 91.664 ± 0.153 %
40
+ 33 7.5199 ± 0.1103 -0.03303 ± 0.00215 0.04456 ± 0.00088 5.926 ± 0.101 % 91.659 ± 0.150 %
41
+ 34 7.4961 ± 0.1081 -0.03312 ± 0.00210 0.04418 ± 0.00086 5.898 ± 0.099 % 91.680 ± 0.148 %
42
+ 35 7.4876 ± 0.1064 -0.03404 ± 0.00207 0.04433 ± 0.00084 5.909 ± 0.098 % 91.686 ± 0.146 %
43
+ 36 7.4965 ± 0.1051 -0.03353 ± 0.00202 0.04382 ± 0.00082 5.868 ± 0.096 % 91.759 ± 0.143 %
44
+ 37 7.3588 ± 0.1013 -0.03290 ± 0.00198 0.04310 ± 0.00080 5.831 ± 0.094 % 91.831 ± 0.141 %
45
+ 38 7.2564 ± 0.0981 -0.03197 ± 0.00194 0.04282 ± 0.00078 5.821 ± 0.093 % 91.881 ± 0.139 %
46
+ 39 7.1480 ± 0.0950 -0.03113 ± 0.00191 0.04246 ± 0.00078 5.809 ± 0.092 % 91.914 ± 0.136 %
47
+ 40 7.0277 ± 0.0917 -0.03032 ± 0.00187 0.04189 ± 0.00076 5.788 ± 0.090 % 91.982 ± 0.134 %
48
+ 41 6.9549 ± 0.0893 -0.02937 ± 0.00183 0.04132 ± 0.00074 5.749 ± 0.089 % 92.025 ± 0.132 %
49
+ 42 6.9911 ± 0.0888 -0.02832 ± 0.00181 0.04112 ± 0.00072 5.723 ± 0.087 % 92.029 ± 0.131 %
50
+ 43 7.1855 ± 0.0911 -0.02837 ± 0.00178 0.04099 ± 0.00071 5.693 ± 0.086 % 91.993 ± 0.129 %
51
+ 44 7.3203 ± 0.0922 -0.02853 ± 0.00176 0.04081 ± 0.00070 5.657 ± 0.085 % 91.958 ± 0.128 %
52
+ 45 7.2944 ± 0.0907 -0.02809 ± 0.00175 0.04157 ± 0.00069 5.708 ± 0.083 % 91.926 ± 0.127 %
53
+ 46 7.3024 ± 0.0897 -0.02760 ± 0.00173 0.04182 ± 0.00068 5.728 ± 0.082 % 91.940 ± 0.125 %
54
+ 47 7.3639 ± 0.0897 -0.02803 ± 0.00171 0.04178 ± 0.00067 5.716 ± 0.081 % 91.934 ± 0.124 %
55
+ 48 7.3643 ± 0.0886 -0.02739 ± 0.00168 0.04150 ± 0.00066 5.701 ± 0.081 % 91.948 ± 0.123 %
56
+ 49 7.4144 ± 0.0885 -0.02718 ± 0.00166 0.04154 ± 0.00066 5.689 ± 0.080 % 91.927 ± 0.122 %
57
+ 50 7.3716 ± 0.0869 -0.02672 ± 0.00164 0.04125 ± 0.00065 5.664 ± 0.079 % 91.957 ± 0.120 %
58
+ 51 7.4030 ± 0.0864 -0.02670 ± 0.00162 0.04101 ± 0.00064 5.650 ± 0.078 % 91.967 ± 0.119 %
59
+ 52 7.3367 ± 0.0846 -0.02728 ± 0.00160 0.04117 ± 0.00063 5.665 ± 0.077 % 91.949 ± 0.118 %
60
+ 53 7.3097 ± 0.0834 -0.02656 ± 0.00158 0.04091 ± 0.00062 5.644 ± 0.076 % 91.970 ± 0.117 %
61
+ 54 7.2701 ± 0.0820 -0.02641 ± 0.00156 0.04072 ± 0.00062 5.638 ± 0.076 % 91.972 ± 0.116 %
62
+ 55 7.2246 ± 0.0806 -0.02766 ± 0.00156 0.04078 ± 0.00061 5.640 ± 0.075 % 91.959 ± 0.115 %
63
+ 56 7.2170 ± 0.0799 -0.02785 ± 0.00155 0.04065 ± 0.00060 5.637 ± 0.074 % 91.960 ± 0.114 %
64
+ 57 7.1420 ± 0.0781 -0.02755 ± 0.00153 0.04048 ± 0.00060 5.625 ± 0.074 % 91.996 ± 0.112 %
65
+ 58 7.1770 ± 0.0780 -0.02747 ± 0.00153 0.04095 ± 0.00060 5.654 ± 0.073 % 91.974 ± 0.112 %
66
+ 59 7.1482 ± 0.0770 -0.02749 ± 0.00153 0.04131 ± 0.00059 5.687 ± 0.073 % 91.973 ± 0.111 %
67
+ 60 7.1321 ± 0.0762 -0.02739 ± 0.00151 0.04109 ± 0.00058 5.672 ± 0.072 % 92.007 ± 0.109 %
68
+ 61 7.1803 ± 0.0763 -0.03151 ± 0.00155 0.04317 ± 0.00063 5.787 ± 0.073 % 91.888 ± 0.109 %
69
+ 62 7.2037 ± 0.0759 -0.03127 ± 0.00153 0.04290 ± 0.00062 5.758 ± 0.073 % 91.909 ± 0.108 %
70
+ 63 7.2523 ± 0.0759 -0.03059 ± 0.00152 0.04255 ± 0.00061 5.730 ± 0.072 % 91.921 ± 0.107 %
71
+ 64 7.2833 ± 0.0757 -0.03052 ± 0.00150 0.04234 ± 0.00061 5.712 ± 0.071 % 91.940 ± 0.106 %
72
+
73
+ ====== Perplexity statistics ======
74
+ Mean PPL(Q) : 7.283348 ± 0.075668
75
+ Mean PPL(base) : 7.509043 ± 0.079361
76
+ Cor(ln(PPL(Q)), ln(PPL(base))): 98.99%
77
+ Mean ln(PPL(Q)/PPL(base)) : -0.030517 ± 0.001497
78
+ Mean PPL(Q)/PPL(base) : 0.969944 ± 0.001452
79
+ Mean PPL(Q)-PPL(base) : -0.225695 ± 0.011594
80
+
81
+ ====== KL divergence statistics ======
82
+ Mean KLD: 0.042338 ± 0.000606
83
+ Maximum KLD: 9.478714
84
+ 99.9% KLD: 1.668447
85
+ 99.0% KLD: 0.440673
86
+ 95.0% KLD: 0.152109
87
+ 90.0% KLD: 0.088946
88
+ Median KLD: 0.013976
89
+ 10.0% KLD: 0.000211
90
+ 5.0% KLD: 0.000057
91
+ 1.0% KLD: 0.000001
92
+ 0.1% KLD: -0.000129
93
+ Minimum KLD: -0.000870
94
+
95
+ ====== Token probability statistics ======
96
+ Mean Δp: -0.067 ± 0.022 %
97
+ Maximum Δp: 98.793%
98
+ 99.9% Δp: 48.776%
99
+ 99.0% Δp: 17.848%
100
+ 95.0% Δp: 6.781%
101
+ 90.0% Δp: 3.539%
102
+ 75.0% Δp: 0.592%
103
+ Median Δp: -0.001%
104
+ 25.0% Δp: -0.839%
105
+ 10.0% Δp: -4.098%
106
+ 5.0% Δp: -7.168%
107
+ 1.0% Δp: -16.669%
108
+ 0.1% Δp: -41.365%
109
+ Minimum Δp: -94.456%
110
+ RMS Δp : 5.712 ± 0.071 %
111
+ Same top p: 91.940 ± 0.106 %
112
+
appendix/measurements/kld_BAND06.txt ADDED
@@ -0,0 +1,112 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.17.135.914 I cmn init: llama threadpool init, n_threads = 30
2
+ 0.17.215.459 I
3
+ 0.17.215.634 I system_info: n_threads = 30 (n_threads_batch = 30) / 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
4
+ 0.17.538.935 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
5
+ 0.34.402.912 I kl_divergence: 16.86 seconds per pass - ETA 17.98 minutes
6
+
7
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
8
+ 1 5.7924 ± 0.4502 -0.01117 ± 0.01151 0.04248 ± 0.00330 6.239 ± 0.467 % 92.669 ± 0.815 %
9
+ 2 7.2587 ± 0.4205 -0.00687 ± 0.00817 0.03753 ± 0.00195 5.462 ± 0.295 % 92.229 ± 0.592 %
10
+ 3 8.1638 ± 0.4002 -0.00430 ± 0.00652 0.03487 ± 0.00158 5.105 ± 0.228 % 92.571 ± 0.473 %
11
+ 4 9.2047 ± 0.4160 -0.01257 ± 0.00602 0.04209 ± 0.00298 5.893 ± 0.316 % 92.107 ± 0.422 %
12
+ 5 8.6271 ± 0.3438 -0.00983 ± 0.00525 0.03977 ± 0.00241 5.649 ± 0.271 % 92.180 ± 0.375 %
13
+ 6 7.3396 ± 0.2584 -0.00882 ± 0.00466 0.03871 ± 0.00208 5.767 ± 0.247 % 92.147 ± 0.343 %
14
+ 7 6.7258 ± 0.2156 -0.01061 ± 0.00440 0.03873 ± 0.00186 5.919 ± 0.229 % 92.347 ± 0.314 %
15
+ 8 6.5318 ± 0.1932 -0.01036 ± 0.00403 0.03707 ± 0.00163 5.752 ± 0.209 % 92.424 ± 0.293 %
16
+ 9 6.9364 ± 0.1953 -0.00859 ± 0.00379 0.03770 ± 0.00149 5.702 ± 0.192 % 92.169 ± 0.280 %
17
+ 10 7.0078 ± 0.1874 -0.00749 ± 0.00352 0.03640 ± 0.00136 5.619 ± 0.184 % 92.336 ± 0.263 %
18
+ 11 7.0186 ± 0.1779 -0.00647 ± 0.00332 0.03518 ± 0.00125 5.481 ± 0.173 % 92.526 ± 0.248 %
19
+ 12 7.2650 ± 0.1772 -0.00821 ± 0.00314 0.03449 ± 0.00116 5.370 ± 0.163 % 92.587 ± 0.236 %
20
+ 13 7.3505 ± 0.1727 -0.01178 ± 0.00303 0.03440 ± 0.00109 5.363 ± 0.153 % 92.646 ± 0.226 %
21
+ 14 7.3981 ± 0.1672 -0.01168 ± 0.00288 0.03346 ± 0.00102 5.280 ± 0.148 % 92.738 ± 0.217 %
22
+ 15 7.4466 ± 0.1626 -0.01293 ± 0.00275 0.03318 ± 0.00097 5.248 ± 0.141 % 92.786 ± 0.209 %
23
+ 16 7.6356 ± 0.1617 -0.01261 ± 0.00264 0.03287 ± 0.00092 5.212 ± 0.137 % 92.827 ± 0.202 %
24
+ 17 7.6484 ± 0.1563 -0.01190 ± 0.00253 0.03238 ± 0.00088 5.122 ± 0.132 % 92.858 ± 0.195 %
25
+ 18 7.7318 ± 0.1534 -0.01128 ± 0.00244 0.03199 ± 0.00083 5.095 ± 0.127 % 92.875 ± 0.190 %
26
+ 19 7.6471 ± 0.1480 -0.01289 ± 0.00236 0.03168 ± 0.00079 5.045 ± 0.122 % 92.874 ± 0.185 %
27
+ 20 7.3906 ± 0.1389 -0.01462 ± 0.00241 0.03536 ± 0.00100 5.482 ± 0.132 % 92.659 ± 0.182 %
28
+ 21 7.4304 ± 0.1362 -0.01469 ± 0.00235 0.03585 ± 0.00098 5.514 ± 0.129 % 92.589 ± 0.179 %
29
+ 22 7.5080 ± 0.1350 -0.01432 ± 0.00228 0.03560 ± 0.00095 5.484 ± 0.126 % 92.673 ± 0.174 %
30
+ 23 7.5408 ± 0.1324 -0.01347 ± 0.00221 0.03524 ± 0.00091 5.419 ± 0.122 % 92.677 ± 0.170 %
31
+ 24 7.6315 ± 0.1315 -0.01510 ± 0.00217 0.03545 ± 0.00088 5.441 ± 0.119 % 92.669 ± 0.166 %
32
+ 25 7.6931 ± 0.1302 -0.01549 ± 0.00212 0.03519 ± 0.00085 5.389 ± 0.116 % 92.641 ± 0.163 %
33
+ 26 7.6440 ± 0.1266 -0.01526 ± 0.00206 0.03494 ± 0.00082 5.348 ± 0.112 % 92.691 ± 0.160 %
34
+ 27 7.7946 ± 0.1270 -0.01543 ± 0.00202 0.03470 ± 0.00079 5.306 ± 0.109 % 92.607 ± 0.157 %
35
+ 28 7.8638 ± 0.1259 -0.01634 ± 0.00198 0.03465 ± 0.00077 5.283 ± 0.107 % 92.602 ± 0.155 %
36
+ 29 7.8642 ± 0.1239 -0.01585 ± 0.00195 0.03455 ± 0.00075 5.294 ± 0.103 % 92.591 ± 0.152 %
37
+ 30 7.8943 ± 0.1228 -0.01807 ± 0.00194 0.03475 ± 0.00073 5.296 ± 0.100 % 92.590 ± 0.150 %
38
+ 31 7.7911 ± 0.1189 -0.01734 ± 0.00191 0.03451 ± 0.00071 5.280 ± 0.098 % 92.590 ± 0.147 %
39
+ 32 7.6621 ± 0.1149 -0.01640 ± 0.00187 0.03444 ± 0.00069 5.330 ± 0.098 % 92.647 ± 0.144 %
40
+ 33 7.6563 ± 0.1133 -0.01505 ± 0.00185 0.03471 ± 0.00072 5.331 ± 0.095 % 92.615 ± 0.142 %
41
+ 34 7.6386 ± 0.1112 -0.01429 ± 0.00181 0.03431 ± 0.00070 5.295 ± 0.093 % 92.625 ± 0.140 %
42
+ 35 7.6362 ± 0.1095 -0.01439 ± 0.00178 0.03420 ± 0.00069 5.279 ± 0.092 % 92.638 ± 0.138 %
43
+ 36 7.6413 ± 0.1081 -0.01440 ± 0.00174 0.03394 ± 0.00067 5.255 ± 0.090 % 92.696 ± 0.136 %
44
+ 37 7.4993 ± 0.1042 -0.01399 ± 0.00171 0.03352 ± 0.00066 5.227 ± 0.089 % 92.774 ± 0.133 %
45
+ 38 7.3882 ± 0.1008 -0.01397 ± 0.00168 0.03326 ± 0.00064 5.209 ± 0.087 % 92.815 ± 0.131 %
46
+ 39 7.2746 ± 0.0976 -0.01357 ± 0.00165 0.03292 ± 0.00062 5.185 ± 0.086 % 92.827 ± 0.129 %
47
+ 40 7.1513 ± 0.0942 -0.01290 ± 0.00161 0.03250 ± 0.00061 5.167 ± 0.084 % 92.864 ± 0.127 %
48
+ 41 7.0728 ± 0.0916 -0.01255 ± 0.00160 0.03225 ± 0.00060 5.159 ± 0.084 % 92.876 ± 0.126 %
49
+ 42 7.1047 ± 0.0910 -0.01221 ± 0.00157 0.03217 ± 0.00059 5.148 ± 0.083 % 92.880 ± 0.124 %
50
+ 43 7.3023 ± 0.0934 -0.01226 ± 0.00156 0.03220 ± 0.00058 5.134 ± 0.081 % 92.816 ± 0.123 %
51
+ 44 7.4446 ± 0.0946 -0.01169 ± 0.00154 0.03217 ± 0.00057 5.117 ± 0.080 % 92.802 ± 0.122 %
52
+ 45 7.4218 ± 0.0931 -0.01078 ± 0.00153 0.03307 ± 0.00057 5.189 ± 0.079 % 92.723 ± 0.121 %
53
+ 46 7.4257 ± 0.0921 -0.01086 ± 0.00152 0.03334 ± 0.00056 5.204 ± 0.078 % 92.703 ± 0.120 %
54
+ 47 7.4830 ± 0.0919 -0.01199 ± 0.00151 0.03359 ± 0.00056 5.208 ± 0.077 % 92.685 ± 0.119 %
55
+ 48 7.4818 ± 0.0908 -0.01155 ± 0.00149 0.03339 ± 0.00055 5.186 ± 0.076 % 92.691 ± 0.117 %
56
+ 49 7.5307 ± 0.0906 -0.01162 ± 0.00148 0.03357 ± 0.00056 5.185 ± 0.075 % 92.639 ± 0.117 %
57
+ 50 7.4880 ± 0.0890 -0.01104 ± 0.00146 0.03336 ± 0.00055 5.163 ± 0.074 % 92.661 ± 0.115 %
58
+ 51 7.5212 ± 0.0885 -0.01086 ± 0.00144 0.03314 ± 0.00054 5.137 ± 0.073 % 92.684 ± 0.114 %
59
+ 52 7.4573 ± 0.0868 -0.01098 ± 0.00142 0.03314 ± 0.00054 5.135 ± 0.072 % 92.682 ± 0.113 %
60
+ 53 7.4267 ± 0.0854 -0.01069 ± 0.00140 0.03293 ± 0.00053 5.116 ± 0.071 % 92.674 ± 0.112 %
61
+ 54 7.3858 ± 0.0840 -0.01063 ± 0.00138 0.03266 ± 0.00052 5.093 ± 0.070 % 92.707 ± 0.111 %
62
+ 55 7.3460 ± 0.0827 -0.01100 ± 0.00137 0.03259 ± 0.00051 5.100 ± 0.070 % 92.701 ± 0.110 %
63
+ 56 7.3408 ± 0.0819 -0.01085 ± 0.00136 0.03254 ± 0.00051 5.093 ± 0.069 % 92.698 ± 0.109 %
64
+ 57 7.2622 ± 0.0801 -0.01087 ± 0.00134 0.03233 ± 0.00050 5.074 ± 0.068 % 92.717 ± 0.108 %
65
+ 58 7.2991 ± 0.0801 -0.01060 ± 0.00135 0.03285 ± 0.00051 5.105 ± 0.068 % 92.685 ± 0.107 %
66
+ 59 7.2730 ± 0.0790 -0.01019 ± 0.00134 0.03303 ± 0.00050 5.122 ± 0.067 % 92.703 ± 0.106 %
67
+ 60 7.2581 ± 0.0782 -0.00988 ± 0.00132 0.03283 ± 0.00050 5.108 ± 0.066 % 92.731 ± 0.105 %
68
+ 61 7.3270 ± 0.0786 -0.01127 ± 0.00133 0.03354 ± 0.00051 5.171 ± 0.067 % 92.669 ± 0.104 %
69
+ 62 7.3500 ± 0.0782 -0.01116 ± 0.00132 0.03334 ± 0.00051 5.148 ± 0.066 % 92.683 ± 0.103 %
70
+ 63 7.3977 ± 0.0782 -0.01075 ± 0.00130 0.03318 ± 0.00050 5.131 ± 0.066 % 92.672 ± 0.103 %
71
+ 64 7.4287 ± 0.0780 -0.01075 ± 0.00129 0.03301 ± 0.00049 5.110 ± 0.065 % 92.682 ± 0.102 %
72
+
73
+ ====== Perplexity statistics ======
74
+ Mean PPL(Q) : 7.428722 ± 0.077970
75
+ Mean PPL(base) : 7.509043 ± 0.079361
76
+ Cor(ln(PPL(Q)), ln(PPL(base))): 99.26%
77
+ Mean ln(PPL(Q)/PPL(base)) : -0.010754 ± 0.001287
78
+ Mean PPL(Q)/PPL(base) : 0.989303 ± 0.001273
79
+ Mean PPL(Q)-PPL(base) : -0.080321 ± 0.009697
80
+
81
+ ====== KL divergence statistics ======
82
+ Mean KLD: 0.033014 ± 0.000492
83
+ Maximum KLD: 9.246984
84
+ 99.9% KLD: 1.432204
85
+ 99.0% KLD: 0.340676
86
+ 95.0% KLD: 0.116262
87
+ 90.0% KLD: 0.068759
88
+ Median KLD: 0.011378
89
+ 10.0% KLD: 0.000187
90
+ 5.0% KLD: 0.000054
91
+ 1.0% KLD: 0.000001
92
+ 0.1% KLD: -0.000133
93
+ Minimum KLD: -0.000665
94
+
95
+ ====== Token probability statistics ======
96
+ Mean Δp: -0.116 ± 0.020 %
97
+ Maximum Δp: 87.794%
98
+ 99.9% Δp: 38.577%
99
+ 99.0% Δp: 15.190%
100
+ 95.0% Δp: 5.955%
101
+ 90.0% Δp: 3.186%
102
+ 75.0% Δp: 0.519%
103
+ Median Δp: -0.002%
104
+ 25.0% Δp: -0.737%
105
+ 10.0% Δp: -3.626%
106
+ 5.0% Δp: -6.368%
107
+ 1.0% Δp: -15.612%
108
+ 0.1% Δp: -41.143%
109
+ Minimum Δp: -79.219%
110
+ RMS Δp : 5.110 ± 0.065 %
111
+ Same top p: 92.682 ± 0.102 %
112
+
appendix/measurements/kld_BAND12.txt ADDED
@@ -0,0 +1,112 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.16.986.383 I cmn init: llama threadpool init, n_threads = 30
2
+ 0.17.061.142 I
3
+ 0.17.061.321 I system_info: n_threads = 30 (n_threads_batch = 30) / 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
4
+ 0.17.408.470 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
5
+ 0.35.125.282 I kl_divergence: 17.72 seconds per pass - ETA 18.88 minutes
6
+
7
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
8
+ 1 5.8373 ± 0.4576 -0.00345 ± 0.01159 0.04655 ± 0.00395 6.691 ± 0.481 % 92.375 ± 0.830 %
9
+ 2 7.2588 ± 0.4204 -0.00685 ± 0.00822 0.04002 ± 0.00242 5.570 ± 0.301 % 91.691 ± 0.610 %
10
+ 3 8.1597 ± 0.4000 -0.00480 ± 0.00668 0.03813 ± 0.00242 5.400 ± 0.298 % 92.180 ± 0.485 %
11
+ 4 9.1457 ± 0.4128 -0.01900 ± 0.00620 0.04364 ± 0.00280 5.937 ± 0.319 % 92.058 ± 0.423 %
12
+ 5 8.5608 ± 0.3404 -0.01755 ± 0.00546 0.04213 ± 0.00231 5.750 ± 0.274 % 92.160 ± 0.376 %
13
+ 6 7.2651 ± 0.2551 -0.01903 ± 0.00496 0.04170 ± 0.00209 5.858 ± 0.260 % 92.229 ± 0.342 %
14
+ 7 6.6850 ± 0.2139 -0.01669 ± 0.00462 0.04137 ± 0.00184 5.946 ± 0.228 % 92.264 ± 0.316 %
15
+ 8 6.5020 ± 0.1921 -0.01494 ± 0.00419 0.03963 ± 0.00163 5.835 ± 0.211 % 92.339 ± 0.294 %
16
+ 9 6.8453 ± 0.1918 -0.02180 ± 0.00399 0.04057 ± 0.00147 5.847 ± 0.191 % 92.093 ± 0.281 %
17
+ 10 6.9130 ± 0.1840 -0.02110 ± 0.00372 0.03969 ± 0.00135 5.787 ± 0.178 % 92.053 ± 0.267 %
18
+ 11 6.9280 ± 0.1746 -0.01945 ± 0.00353 0.03833 ± 0.00124 5.654 ± 0.167 % 92.287 ± 0.252 %
19
+ 12 7.1772 ± 0.1743 -0.02038 ± 0.00332 0.03765 ± 0.00114 5.549 ± 0.157 % 92.384 ± 0.239 %
20
+ 13 7.2538 ± 0.1695 -0.02502 ± 0.00322 0.03784 ± 0.00108 5.579 ± 0.149 % 92.375 ± 0.230 %
21
+ 14 7.2995 ± 0.1640 -0.02510 ± 0.00305 0.03695 ± 0.00102 5.491 ± 0.144 % 92.431 ± 0.221 %
22
+ 15 7.3484 ± 0.1595 -0.02620 ± 0.00293 0.03656 ± 0.00096 5.439 ± 0.137 % 92.447 ± 0.213 %
23
+ 16 7.5288 ± 0.1584 -0.02670 ± 0.00281 0.03609 ± 0.00090 5.371 ± 0.131 % 92.534 ± 0.205 %
24
+ 17 7.5474 ± 0.1533 -0.02518 ± 0.00267 0.03559 ± 0.00088 5.283 ± 0.126 % 92.473 ± 0.200 %
25
+ 18 7.6360 ± 0.1506 -0.02374 ± 0.00259 0.03523 ± 0.00084 5.266 ± 0.122 % 92.544 ± 0.194 %
26
+ 19 7.5522 ± 0.1453 -0.02538 ± 0.00253 0.03524 ± 0.00080 5.240 ± 0.117 % 92.530 ± 0.189 %
27
+ 20 7.2853 ± 0.1360 -0.02897 ± 0.00263 0.04118 ± 0.00122 5.823 ± 0.127 % 92.287 ± 0.187 %
28
+ 21 7.3156 ± 0.1331 -0.03024 ± 0.00256 0.04156 ± 0.00118 5.828 ± 0.124 % 92.254 ± 0.182 %
29
+ 22 7.4000 ± 0.1322 -0.02881 ± 0.00249 0.04110 ± 0.00114 5.814 ± 0.121 % 92.304 ± 0.178 %
30
+ 23 7.4369 ± 0.1297 -0.02734 ± 0.00241 0.04050 ± 0.00109 5.748 ± 0.117 % 92.303 ± 0.174 %
31
+ 24 7.5173 ± 0.1287 -0.03017 ± 0.00237 0.04084 ± 0.00105 5.747 ± 0.114 % 92.265 ± 0.170 %
32
+ 25 7.5804 ± 0.1275 -0.03024 ± 0.00231 0.04042 ± 0.00101 5.684 ± 0.111 % 92.231 ± 0.167 %
33
+ 26 7.5344 ± 0.1240 -0.02970 ± 0.00225 0.04005 ± 0.00098 5.660 ± 0.108 % 92.248 ± 0.164 %
34
+ 27 7.6851 ± 0.1244 -0.02958 ± 0.00220 0.03985 ± 0.00095 5.628 ± 0.106 % 92.173 ± 0.162 %
35
+ 28 7.7605 ± 0.1235 -0.02956 ± 0.00216 0.03964 ± 0.00092 5.597 ± 0.104 % 92.169 ± 0.159 %
36
+ 29 7.7561 ± 0.1215 -0.02969 ± 0.00212 0.03952 ± 0.00089 5.590 ± 0.101 % 92.166 ± 0.156 %
37
+ 30 7.7880 ± 0.1204 -0.03163 ± 0.00211 0.03991 ± 0.00088 5.591 ± 0.098 % 92.121 ± 0.154 %
38
+ 31 7.6862 ± 0.1166 -0.03089 ± 0.00207 0.03957 ± 0.00086 5.576 ± 0.097 % 92.136 ± 0.151 %
39
+ 32 7.5589 ± 0.1127 -0.02997 ± 0.00203 0.03935 ± 0.00084 5.601 ± 0.095 % 92.198 ± 0.148 %
40
+ 33 7.5516 ± 0.1111 -0.02883 ± 0.00200 0.03952 ± 0.00086 5.600 ± 0.094 % 92.186 ± 0.146 %
41
+ 34 7.5339 ± 0.1090 -0.02808 ± 0.00195 0.03905 ± 0.00083 5.563 ± 0.092 % 92.180 ± 0.144 %
42
+ 35 7.5332 ± 0.1074 -0.02796 ± 0.00191 0.03882 ± 0.00081 5.540 ± 0.090 % 92.230 ± 0.141 %
43
+ 36 7.5368 ± 0.1060 -0.02817 ± 0.00188 0.03863 ± 0.00079 5.532 ± 0.088 % 92.280 ± 0.139 %
44
+ 37 7.3989 ± 0.1022 -0.02746 ± 0.00184 0.03812 ± 0.00077 5.503 ± 0.087 % 92.357 ± 0.137 %
45
+ 38 7.2947 ± 0.0989 -0.02670 ± 0.00181 0.03785 ± 0.00076 5.498 ± 0.086 % 92.386 ± 0.135 %
46
+ 39 7.1856 ± 0.0958 -0.02588 ± 0.00178 0.03744 ± 0.00074 5.471 ± 0.084 % 92.433 ± 0.132 %
47
+ 40 7.0648 ± 0.0925 -0.02506 ± 0.00174 0.03693 ± 0.00072 5.444 ± 0.083 % 92.483 ± 0.130 %
48
+ 41 6.9905 ± 0.0901 -0.02425 ± 0.00171 0.03670 ± 0.00072 5.436 ± 0.084 % 92.502 ± 0.129 %
49
+ 42 7.0226 ± 0.0895 -0.02383 ± 0.00169 0.03657 ± 0.00071 5.422 ± 0.082 % 92.529 ± 0.127 %
50
+ 43 7.2129 ± 0.0917 -0.02456 ± 0.00167 0.03658 ± 0.00069 5.399 ± 0.081 % 92.464 ± 0.126 %
51
+ 44 7.3512 ± 0.0929 -0.02432 ± 0.00164 0.03654 ± 0.00069 5.366 ± 0.080 % 92.442 ± 0.125 %
52
+ 45 7.3298 ± 0.0914 -0.02325 ± 0.00164 0.03729 ± 0.00068 5.413 ± 0.078 % 92.391 ± 0.124 %
53
+ 46 7.3342 ± 0.0904 -0.02326 ± 0.00163 0.03762 ± 0.00067 5.442 ± 0.078 % 92.358 ± 0.122 %
54
+ 47 7.3920 ± 0.0903 -0.02423 ± 0.00162 0.03782 ± 0.00067 5.453 ± 0.078 % 92.342 ± 0.121 %
55
+ 48 7.3923 ± 0.0892 -0.02359 ± 0.00159 0.03755 ± 0.00066 5.432 ± 0.077 % 92.347 ± 0.120 %
56
+ 49 7.4433 ± 0.0891 -0.02329 ± 0.00158 0.03771 ± 0.00066 5.437 ± 0.076 % 92.298 ± 0.119 %
57
+ 50 7.4021 ± 0.0875 -0.02258 ± 0.00156 0.03739 ± 0.00065 5.408 ± 0.075 % 92.346 ± 0.118 %
58
+ 51 7.4343 ± 0.0870 -0.02247 ± 0.00153 0.03716 ± 0.00064 5.383 ± 0.074 % 92.362 ± 0.116 %
59
+ 52 7.3731 ± 0.0853 -0.02234 ± 0.00152 0.03701 ± 0.00063 5.377 ± 0.074 % 92.390 ± 0.115 %
60
+ 53 7.3454 ± 0.0841 -0.02169 ± 0.00150 0.03678 ± 0.00062 5.358 ± 0.073 % 92.405 ± 0.114 %
61
+ 54 7.3064 ± 0.0827 -0.02144 ± 0.00148 0.03647 ± 0.00061 5.330 ± 0.072 % 92.435 ± 0.113 %
62
+ 55 7.2637 ± 0.0813 -0.02226 ± 0.00147 0.03644 ± 0.00060 5.341 ± 0.072 % 92.438 ± 0.111 %
63
+ 56 7.2521 ± 0.0805 -0.02300 ± 0.00145 0.03633 ± 0.00059 5.333 ± 0.071 % 92.452 ± 0.110 %
64
+ 57 7.1762 ± 0.0787 -0.02277 ± 0.00143 0.03610 ± 0.00058 5.309 ± 0.070 % 92.471 ± 0.109 %
65
+ 58 7.2089 ± 0.0786 -0.02303 ± 0.00143 0.03643 ± 0.00058 5.335 ± 0.070 % 92.443 ± 0.109 %
66
+ 59 7.1845 ± 0.0776 -0.02244 ± 0.00142 0.03685 ± 0.00058 5.370 ± 0.070 % 92.450 ± 0.108 %
67
+ 60 7.1689 ± 0.0768 -0.02224 ± 0.00141 0.03664 ± 0.00057 5.359 ± 0.069 % 92.470 ± 0.107 %
68
+ 61 7.2306 ± 0.0771 -0.02453 ± 0.00142 0.03742 ± 0.00060 5.414 ± 0.069 % 92.398 ± 0.106 %
69
+ 62 7.2544 ± 0.0767 -0.02425 ± 0.00140 0.03718 ± 0.00059 5.390 ± 0.069 % 92.418 ± 0.105 %
70
+ 63 7.3003 ± 0.0767 -0.02399 ± 0.00138 0.03693 ± 0.00058 5.367 ± 0.068 % 92.420 ± 0.104 %
71
+ 64 7.3310 ± 0.0764 -0.02399 ± 0.00137 0.03676 ± 0.00057 5.351 ± 0.067 % 92.432 ± 0.103 %
72
+
73
+ ====== Perplexity statistics ======
74
+ Mean PPL(Q) : 7.331023 ± 0.076448
75
+ Mean PPL(base) : 7.509043 ± 0.079361
76
+ Cor(ln(PPL(Q)), ln(PPL(base))): 99.16%
77
+ Mean ln(PPL(Q)/PPL(base)) : -0.023993 ± 0.001368
78
+ Mean PPL(Q)/PPL(base) : 0.976293 ± 0.001335
79
+ Mean PPL(Q)-PPL(base) : -0.178020 ± 0.010505
80
+
81
+ ====== KL divergence statistics ======
82
+ Mean KLD: 0.036760 ± 0.000572
83
+ Maximum KLD: 13.933623
84
+ 99.9% KLD: 1.553122
85
+ 99.0% KLD: 0.374749
86
+ 95.0% KLD: 0.129424
87
+ 90.0% KLD: 0.076851
88
+ Median KLD: 0.012635
89
+ 10.0% KLD: 0.000197
90
+ 5.0% KLD: 0.000057
91
+ 1.0% KLD: 0.000000
92
+ 0.1% KLD: -0.000150
93
+ Minimum KLD: -0.000764
94
+
95
+ ====== Token probability statistics ======
96
+ Mean Δp: -0.087 ± 0.021 %
97
+ Maximum Δp: 95.796%
98
+ 99.9% Δp: 42.865%
99
+ 99.0% Δp: 16.771%
100
+ 95.0% Δp: 6.344%
101
+ 90.0% Δp: 3.318%
102
+ 75.0% Δp: 0.545%
103
+ Median Δp: -0.002%
104
+ 25.0% Δp: -0.805%
105
+ 10.0% Δp: -3.825%
106
+ 5.0% Δp: -6.688%
107
+ 1.0% Δp: -15.859%
108
+ 0.1% Δp: -41.566%
109
+ Minimum Δp: -84.142%
110
+ RMS Δp : 5.351 ± 0.067 %
111
+ Same top p: 92.432 ± 0.103 %
112
+
appendix/measurements/kld_BAND18.txt ADDED
@@ -0,0 +1,112 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.16.985.356 I cmn init: llama threadpool init, n_threads = 30
2
+ 0.17.063.143 I
3
+ 0.17.063.322 I system_info: n_threads = 30 (n_threads_batch = 30) / 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
4
+ 0.17.385.102 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
5
+ 0.34.029.317 I kl_divergence: 16.64 seconds per pass - ETA 17.75 minutes
6
+
7
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
8
+ 1 5.8763 ± 0.4634 0.00320 ± 0.01175 0.04839 ± 0.00340 6.882 ± 0.440 % 92.669 ± 0.815 %
9
+ 2 7.2725 ± 0.4222 -0.00497 ± 0.00826 0.04185 ± 0.00218 5.833 ± 0.284 % 91.789 ± 0.607 %
10
+ 3 8.1713 ± 0.4014 -0.00338 ± 0.00686 0.03933 ± 0.00238 5.519 ± 0.291 % 92.147 ± 0.486 %
11
+ 4 9.2480 ± 0.4199 -0.00789 ± 0.00601 0.04268 ± 0.00235 5.719 ± 0.275 % 92.033 ± 0.423 %
12
+ 5 8.6243 ± 0.3445 -0.01016 ± 0.00541 0.04179 ± 0.00198 5.493 ± 0.233 % 92.102 ± 0.377 %
13
+ 6 7.3137 ± 0.2580 -0.01235 ± 0.00490 0.04131 ± 0.00181 5.592 ± 0.220 % 91.984 ± 0.347 %
14
+ 7 6.7390 ± 0.2168 -0.00865 ± 0.00459 0.04124 ± 0.00163 5.752 ± 0.201 % 92.096 ± 0.319 %
15
+ 8 6.5495 ± 0.1944 -0.00766 ± 0.00414 0.03952 ± 0.00145 5.643 ± 0.187 % 92.253 ± 0.296 %
16
+ 9 6.8928 ± 0.1940 -0.01489 ± 0.00399 0.04082 ± 0.00133 5.717 ± 0.173 % 91.778 ± 0.286 %
17
+ 10 6.9570 ± 0.1859 -0.01476 ± 0.00371 0.03960 ± 0.00121 5.630 ± 0.161 % 91.916 ± 0.270 %
18
+ 11 6.9632 ± 0.1762 -0.01439 ± 0.00350 0.03817 ± 0.00111 5.498 ± 0.150 % 92.038 ± 0.255 %
19
+ 12 7.2248 ± 0.1763 -0.01376 ± 0.00327 0.03712 ± 0.00103 5.391 ± 0.141 % 92.139 ± 0.243 %
20
+ 13 7.3174 ± 0.1721 -0.01630 ± 0.00316 0.03757 ± 0.00100 5.512 ± 0.137 % 92.142 ± 0.233 %
21
+ 14 7.3599 ± 0.1664 -0.01685 ± 0.00300 0.03662 ± 0.00094 5.432 ± 0.132 % 92.229 ± 0.224 %
22
+ 15 7.4089 ± 0.1619 -0.01801 ± 0.00289 0.03620 ± 0.00089 5.372 ± 0.126 % 92.232 ± 0.216 %
23
+ 16 7.5889 ± 0.1607 -0.01874 ± 0.00277 0.03571 ± 0.00084 5.297 ± 0.120 % 92.278 ± 0.209 %
24
+ 17 7.6016 ± 0.1553 -0.01804 ± 0.00264 0.03518 ± 0.00081 5.217 ± 0.115 % 92.272 ± 0.202 %
25
+ 18 7.6877 ± 0.1525 -0.01700 ± 0.00256 0.03494 ± 0.00077 5.197 ± 0.111 % 92.310 ± 0.196 %
26
+ 19 7.6032 ± 0.1471 -0.01864 ± 0.00249 0.03477 ± 0.00074 5.181 ± 0.108 % 92.350 ± 0.191 %
27
+ 20 7.3346 ± 0.1377 -0.02222 ± 0.00259 0.04057 ± 0.00117 5.726 ± 0.121 % 92.160 ± 0.188 %
28
+ 21 7.3669 ± 0.1348 -0.02326 ± 0.00252 0.04108 ± 0.00114 5.767 ± 0.120 % 92.124 ± 0.184 %
29
+ 22 7.4575 ± 0.1340 -0.02107 ± 0.00246 0.04071 ± 0.00110 5.749 ± 0.118 % 92.180 ± 0.179 %
30
+ 23 7.4924 ± 0.1315 -0.01991 ± 0.00238 0.04022 ± 0.00105 5.692 ± 0.114 % 92.180 ± 0.175 %
31
+ 24 7.5735 ± 0.1304 -0.02273 ± 0.00234 0.04062 ± 0.00102 5.699 ± 0.111 % 92.131 ± 0.172 %
32
+ 25 7.6377 ± 0.1292 -0.02272 ± 0.00229 0.04032 ± 0.00098 5.648 ± 0.108 % 92.074 ± 0.169 %
33
+ 26 7.5909 ± 0.1257 -0.02223 ± 0.00223 0.03993 ± 0.00095 5.622 ± 0.105 % 92.146 ± 0.165 %
34
+ 27 7.7399 ± 0.1260 -0.02247 ± 0.00218 0.03969 ± 0.00092 5.593 ± 0.103 % 92.104 ± 0.162 %
35
+ 28 7.8143 ± 0.1251 -0.02265 ± 0.00214 0.03943 ± 0.00089 5.557 ± 0.101 % 92.124 ± 0.159 %
36
+ 29 7.8128 ± 0.1231 -0.02241 ± 0.00210 0.03935 ± 0.00087 5.554 ± 0.098 % 92.136 ± 0.156 %
37
+ 30 7.8512 ± 0.1221 -0.02354 ± 0.00209 0.03963 ± 0.00086 5.575 ± 0.097 % 92.118 ± 0.154 %
38
+ 31 7.7512 ± 0.1183 -0.02248 ± 0.00205 0.03924 ± 0.00083 5.566 ± 0.095 % 92.145 ± 0.151 %
39
+ 32 7.6248 ± 0.1143 -0.02128 ± 0.00200 0.03896 ± 0.00081 5.589 ± 0.095 % 92.195 ± 0.148 %
40
+ 33 7.6174 ± 0.1127 -0.02015 ± 0.00197 0.03899 ± 0.00080 5.578 ± 0.093 % 92.168 ± 0.146 %
41
+ 34 7.5956 ± 0.1105 -0.01992 ± 0.00193 0.03857 ± 0.00078 5.540 ± 0.091 % 92.197 ± 0.144 %
42
+ 35 7.5945 ± 0.1090 -0.01987 ± 0.00189 0.03838 ± 0.00077 5.523 ± 0.089 % 92.216 ± 0.142 %
43
+ 36 7.5961 ± 0.1075 -0.02033 ± 0.00186 0.03823 ± 0.00075 5.522 ± 0.088 % 92.264 ± 0.139 %
44
+ 37 7.4552 ± 0.1035 -0.01988 ± 0.00182 0.03766 ± 0.00073 5.484 ± 0.086 % 92.336 ± 0.137 %
45
+ 38 7.3479 ± 0.1002 -0.01943 ± 0.00179 0.03736 ± 0.00071 5.465 ± 0.085 % 92.383 ± 0.135 %
46
+ 39 7.2368 ± 0.0970 -0.01878 ± 0.00176 0.03697 ± 0.00070 5.442 ± 0.083 % 92.420 ± 0.133 %
47
+ 40 7.1132 ± 0.0937 -0.01823 ± 0.00172 0.03648 ± 0.00068 5.415 ± 0.082 % 92.483 ± 0.130 %
48
+ 41 7.0368 ± 0.0911 -0.01766 ± 0.00169 0.03614 ± 0.00067 5.398 ± 0.081 % 92.478 ± 0.129 %
49
+ 42 7.0683 ± 0.0905 -0.01734 ± 0.00167 0.03607 ± 0.00065 5.384 ± 0.080 % 92.461 ± 0.127 %
50
+ 43 7.2577 ± 0.0927 -0.01838 ± 0.00165 0.03612 ± 0.00064 5.366 ± 0.078 % 92.391 ± 0.126 %
51
+ 44 7.3953 ± 0.0939 -0.01833 ± 0.00163 0.03607 ± 0.00064 5.333 ± 0.077 % 92.369 ± 0.125 %
52
+ 45 7.3718 ± 0.0924 -0.01754 ± 0.00162 0.03699 ± 0.00064 5.408 ± 0.077 % 92.291 ± 0.124 %
53
+ 46 7.3766 ± 0.0913 -0.01749 ± 0.00160 0.03728 ± 0.00063 5.423 ± 0.076 % 92.263 ± 0.123 %
54
+ 47 7.4364 ± 0.0913 -0.01823 ± 0.00159 0.03746 ± 0.00063 5.431 ± 0.076 % 92.248 ± 0.122 %
55
+ 48 7.4348 ± 0.0901 -0.01786 ± 0.00157 0.03724 ± 0.00062 5.409 ± 0.075 % 92.265 ± 0.121 %
56
+ 49 7.4874 ± 0.0900 -0.01738 ± 0.00155 0.03727 ± 0.00061 5.399 ± 0.074 % 92.258 ± 0.119 %
57
+ 50 7.4458 ± 0.0884 -0.01670 ± 0.00153 0.03699 ± 0.00060 5.374 ± 0.073 % 92.295 ± 0.118 %
58
+ 51 7.4767 ± 0.0879 -0.01679 ± 0.00151 0.03671 ± 0.00059 5.345 ± 0.072 % 92.308 ± 0.117 %
59
+ 52 7.4143 ± 0.0862 -0.01677 ± 0.00149 0.03649 ± 0.00058 5.346 ± 0.072 % 92.342 ± 0.115 %
60
+ 53 7.3854 ± 0.0849 -0.01627 ± 0.00147 0.03625 ± 0.00057 5.327 ± 0.071 % 92.340 ± 0.114 %
61
+ 54 7.3450 ± 0.0834 -0.01618 ± 0.00145 0.03596 ± 0.00056 5.305 ± 0.070 % 92.372 ± 0.113 %
62
+ 55 7.3025 ± 0.0821 -0.01693 ± 0.00144 0.03589 ± 0.00055 5.306 ± 0.069 % 92.356 ± 0.112 %
63
+ 56 7.2927 ± 0.0813 -0.01741 ± 0.00143 0.03575 ± 0.00054 5.301 ± 0.068 % 92.363 ± 0.111 %
64
+ 57 7.2164 ± 0.0795 -0.01719 ± 0.00141 0.03550 ± 0.00053 5.275 ± 0.067 % 92.396 ± 0.110 %
65
+ 58 7.2519 ± 0.0794 -0.01709 ± 0.00141 0.03594 ± 0.00054 5.300 ± 0.067 % 92.374 ± 0.109 %
66
+ 59 7.2252 ± 0.0784 -0.01678 ± 0.00140 0.03629 ± 0.00054 5.327 ± 0.066 % 92.387 ± 0.108 %
67
+ 60 7.2093 ± 0.0776 -0.01663 ± 0.00139 0.03607 ± 0.00053 5.312 ± 0.066 % 92.419 ± 0.107 %
68
+ 61 7.2727 ± 0.0779 -0.01871 ± 0.00140 0.03670 ± 0.00054 5.352 ± 0.066 % 92.367 ± 0.106 %
69
+ 62 7.2972 ± 0.0775 -0.01838 ± 0.00138 0.03652 ± 0.00053 5.332 ± 0.065 % 92.371 ± 0.105 %
70
+ 63 7.3434 ± 0.0775 -0.01811 ± 0.00137 0.03630 ± 0.00052 5.307 ± 0.064 % 92.375 ± 0.105 %
71
+ 64 7.3742 ± 0.0772 -0.01812 ± 0.00135 0.03611 ± 0.00052 5.287 ± 0.064 % 92.395 ± 0.104 %
72
+
73
+ ====== Perplexity statistics ======
74
+ Mean PPL(Q) : 7.374235 ± 0.077236
75
+ Mean PPL(base) : 7.509043 ± 0.079361
76
+ Cor(ln(PPL(Q)), ln(PPL(base))): 99.18%
77
+ Mean ln(PPL(Q)/PPL(base)) : -0.018116 ± 0.001349
78
+ Mean PPL(Q)/PPL(base) : 0.982047 ± 0.001325
79
+ Mean PPL(Q)-PPL(base) : -0.134808 ± 0.010239
80
+
81
+ ====== KL divergence statistics ======
82
+ Mean KLD: 0.036107 ± 0.000517
83
+ Maximum KLD: 13.522428
84
+ 99.9% KLD: 1.435223
85
+ 99.0% KLD: 0.376872
86
+ 95.0% KLD: 0.128838
87
+ 90.0% KLD: 0.076459
88
+ Median KLD: 0.012456
89
+ 10.0% KLD: 0.000194
90
+ 5.0% KLD: 0.000052
91
+ 1.0% KLD: -0.000001
92
+ 0.1% KLD: -0.000145
93
+ Minimum KLD: -0.000920
94
+
95
+ ====== Token probability statistics ======
96
+ Mean Δp: -0.041 ± 0.021 %
97
+ Maximum Δp: 86.424%
98
+ 99.9% Δp: 41.604%
99
+ 99.0% Δp: 16.736%
100
+ 95.0% Δp: 6.499%
101
+ 90.0% Δp: 3.429%
102
+ 75.0% Δp: 0.575%
103
+ Median Δp: -0.001%
104
+ 25.0% Δp: -0.755%
105
+ 10.0% Δp: -3.732%
106
+ 5.0% Δp: -6.596%
107
+ 1.0% Δp: -15.497%
108
+ 0.1% Δp: -41.141%
109
+ Minimum Δp: -81.360%
110
+ RMS Δp : 5.287 ± 0.064 %
111
+ Same top p: 92.395 ± 0.104 %
112
+
appendix/measurements/kld_BAND25.txt ADDED
@@ -0,0 +1,112 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.17.537.497 I cmn init: llama threadpool init, n_threads = 30
2
+ 0.17.621.111 I
3
+ 0.17.621.292 I system_info: n_threads = 30 (n_threads_batch = 30) / 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
4
+ 0.17.938.909 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
5
+ 0.34.761.984 I kl_divergence: 16.82 seconds per pass - ETA 17.93 minutes
6
+
7
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
8
+ 1 5.8081 ± 0.4553 -0.00847 ± 0.01156 0.05079 ± 0.00377 6.751 ± 0.522 % 92.473 ± 0.825 %
9
+ 2 7.2159 ± 0.4176 -0.01278 ± 0.00837 0.04366 ± 0.00234 5.803 ± 0.331 % 91.496 ± 0.617 %
10
+ 3 8.1543 ± 0.4011 -0.00546 ± 0.00684 0.04049 ± 0.00215 5.501 ± 0.277 % 91.952 ± 0.491 %
11
+ 4 9.2350 ± 0.4196 -0.00929 ± 0.00590 0.04387 ± 0.00244 5.734 ± 0.284 % 91.862 ± 0.427 %
12
+ 5 8.6439 ± 0.3462 -0.00789 ± 0.00542 0.04321 ± 0.00206 5.618 ± 0.248 % 91.808 ± 0.383 %
13
+ 6 7.3187 ± 0.2585 -0.01168 ± 0.00494 0.04293 ± 0.00183 5.775 ± 0.219 % 91.870 ± 0.349 %
14
+ 7 6.7112 ± 0.2159 -0.01278 ± 0.00462 0.04305 ± 0.00170 5.902 ± 0.203 % 91.956 ± 0.321 %
15
+ 8 6.5189 ± 0.1934 -0.01235 ± 0.00419 0.04130 ± 0.00151 5.797 ± 0.188 % 92.094 ± 0.298 %
16
+ 9 6.8673 ± 0.1932 -0.01861 ± 0.00400 0.04222 ± 0.00137 5.758 ± 0.172 % 91.756 ± 0.287 %
17
+ 10 6.9351 ± 0.1853 -0.01791 ± 0.00372 0.04082 ± 0.00125 5.649 ± 0.160 % 91.887 ± 0.270 %
18
+ 11 6.9501 ± 0.1759 -0.01627 ± 0.00351 0.03915 ± 0.00115 5.513 ± 0.150 % 92.127 ± 0.254 %
19
+ 12 7.2039 ± 0.1757 -0.01666 ± 0.00328 0.03797 ± 0.00105 5.396 ± 0.141 % 92.188 ± 0.242 %
20
+ 13 7.2932 ± 0.1713 -0.01961 ± 0.00317 0.03815 ± 0.00102 5.461 ± 0.135 % 92.187 ± 0.233 %
21
+ 14 7.3431 ± 0.1659 -0.01914 ± 0.00301 0.03718 ± 0.00096 5.363 ± 0.128 % 92.215 ± 0.224 %
22
+ 15 7.3929 ± 0.1615 -0.02017 ± 0.00290 0.03674 ± 0.00090 5.330 ± 0.123 % 92.167 ± 0.217 %
23
+ 16 7.5713 ± 0.1603 -0.02107 ± 0.00279 0.03617 ± 0.00085 5.253 ± 0.117 % 92.229 ± 0.209 %
24
+ 17 7.5870 ± 0.1550 -0.01995 ± 0.00266 0.03550 ± 0.00082 5.167 ± 0.112 % 92.283 ± 0.202 %
25
+ 18 7.6700 ± 0.1521 -0.01931 ± 0.00257 0.03522 ± 0.00079 5.131 ± 0.108 % 92.359 ± 0.196 %
26
+ 19 7.5885 ± 0.1468 -0.02058 ± 0.00250 0.03498 ± 0.00075 5.113 ± 0.104 % 92.355 ± 0.191 %
27
+ 20 7.3256 ± 0.1375 -0.02345 ± 0.00263 0.04138 ± 0.00126 5.739 ± 0.123 % 92.136 ± 0.188 %
28
+ 21 7.3610 ± 0.1347 -0.02406 ± 0.00257 0.04194 ± 0.00123 5.802 ± 0.123 % 92.059 ± 0.184 %
29
+ 22 7.4482 ± 0.1338 -0.02232 ± 0.00249 0.04143 ± 0.00118 5.768 ± 0.120 % 92.104 ± 0.180 %
30
+ 23 7.4840 ± 0.1313 -0.02103 ± 0.00242 0.04093 ± 0.00113 5.713 ± 0.117 % 92.129 ± 0.176 %
31
+ 24 7.5631 ± 0.1302 -0.02411 ± 0.00238 0.04128 ± 0.00109 5.737 ± 0.114 % 92.107 ± 0.172 %
32
+ 25 7.6293 ± 0.1291 -0.02381 ± 0.00232 0.04084 ± 0.00105 5.672 ± 0.111 % 92.051 ± 0.169 %
33
+ 26 7.5808 ± 0.1255 -0.02356 ± 0.00226 0.04038 ± 0.00101 5.625 ± 0.108 % 92.093 ± 0.165 %
34
+ 27 7.7289 ± 0.1258 -0.02390 ± 0.00221 0.04013 ± 0.00098 5.606 ± 0.106 % 92.039 ± 0.163 %
35
+ 28 7.8042 ± 0.1249 -0.02395 ± 0.00216 0.03982 ± 0.00095 5.561 ± 0.104 % 92.044 ± 0.160 %
36
+ 29 7.8032 ± 0.1229 -0.02364 ± 0.00212 0.03979 ± 0.00093 5.570 ± 0.101 % 92.069 ± 0.157 %
37
+ 30 7.8390 ± 0.1219 -0.02510 ± 0.00211 0.04002 ± 0.00091 5.588 ± 0.101 % 92.063 ± 0.154 %
38
+ 31 7.7383 ± 0.1180 -0.02414 ± 0.00206 0.03964 ± 0.00089 5.575 ± 0.100 % 92.092 ± 0.152 %
39
+ 32 7.6115 ± 0.1141 -0.02302 ± 0.00202 0.03942 ± 0.00087 5.616 ± 0.101 % 92.149 ± 0.149 %
40
+ 33 7.6031 ± 0.1125 -0.02203 ± 0.00199 0.03943 ± 0.00086 5.616 ± 0.099 % 92.156 ± 0.146 %
41
+ 34 7.5823 ± 0.1103 -0.02168 ± 0.00195 0.03900 ± 0.00084 5.574 ± 0.097 % 92.186 ± 0.144 %
42
+ 35 7.5808 ± 0.1086 -0.02167 ± 0.00191 0.03871 ± 0.00082 5.553 ± 0.095 % 92.219 ± 0.142 %
43
+ 36 7.5832 ± 0.1072 -0.02203 ± 0.00188 0.03847 ± 0.00080 5.540 ± 0.093 % 92.305 ± 0.139 %
44
+ 37 7.4434 ± 0.1033 -0.02147 ± 0.00184 0.03789 ± 0.00078 5.496 ± 0.092 % 92.373 ± 0.136 %
45
+ 38 7.3353 ± 0.0999 -0.02115 ± 0.00181 0.03754 ± 0.00076 5.474 ± 0.090 % 92.391 ± 0.134 %
46
+ 39 7.2231 ± 0.0968 -0.02067 ± 0.00177 0.03706 ± 0.00074 5.439 ± 0.088 % 92.433 ± 0.132 %
47
+ 40 7.1000 ± 0.0934 -0.02009 ± 0.00173 0.03658 ± 0.00072 5.424 ± 0.087 % 92.478 ± 0.130 %
48
+ 41 7.0232 ± 0.0909 -0.01959 ± 0.00171 0.03625 ± 0.00071 5.407 ± 0.086 % 92.483 ± 0.129 %
49
+ 42 7.0533 ± 0.0902 -0.01946 ± 0.00168 0.03613 ± 0.00069 5.388 ± 0.084 % 92.461 ± 0.127 %
50
+ 43 7.2419 ± 0.0924 -0.02055 ± 0.00166 0.03617 ± 0.00068 5.366 ± 0.083 % 92.412 ± 0.126 %
51
+ 44 7.3799 ± 0.0936 -0.02041 ± 0.00164 0.03612 ± 0.00068 5.338 ± 0.082 % 92.398 ± 0.125 %
52
+ 45 7.3566 ± 0.0921 -0.01961 ± 0.00163 0.03706 ± 0.00068 5.399 ± 0.081 % 92.349 ± 0.124 %
53
+ 46 7.3619 ± 0.0911 -0.01949 ± 0.00162 0.03725 ± 0.00066 5.405 ± 0.080 % 92.346 ± 0.123 %
54
+ 47 7.4222 ± 0.0910 -0.02015 ± 0.00160 0.03742 ± 0.00066 5.409 ± 0.080 % 92.328 ± 0.121 %
55
+ 48 7.4214 ± 0.0899 -0.01967 ± 0.00157 0.03716 ± 0.00065 5.387 ± 0.078 % 92.337 ± 0.120 %
56
+ 49 7.4741 ± 0.0897 -0.01916 ± 0.00156 0.03715 ± 0.00064 5.378 ± 0.078 % 92.322 ± 0.119 %
57
+ 50 7.4336 ± 0.0882 -0.01834 ± 0.00154 0.03688 ± 0.00063 5.352 ± 0.077 % 92.368 ± 0.117 %
58
+ 51 7.4652 ± 0.0877 -0.01833 ± 0.00152 0.03661 ± 0.00062 5.323 ± 0.076 % 92.398 ± 0.116 %
59
+ 52 7.4030 ± 0.0860 -0.01829 ± 0.00150 0.03635 ± 0.00061 5.318 ± 0.075 % 92.445 ± 0.115 %
60
+ 53 7.3744 ± 0.0847 -0.01776 ± 0.00148 0.03608 ± 0.00060 5.299 ± 0.074 % 92.447 ± 0.113 %
61
+ 54 7.3323 ± 0.0832 -0.01790 ± 0.00146 0.03583 ± 0.00059 5.289 ± 0.073 % 92.466 ± 0.112 %
62
+ 55 7.2899 ± 0.0819 -0.01867 ± 0.00145 0.03582 ± 0.00058 5.297 ± 0.073 % 92.468 ± 0.111 %
63
+ 56 7.2800 ± 0.0811 -0.01915 ± 0.00144 0.03569 ± 0.00057 5.291 ± 0.072 % 92.501 ± 0.110 %
64
+ 57 7.2027 ± 0.0793 -0.01909 ± 0.00142 0.03547 ± 0.00056 5.267 ± 0.071 % 92.535 ± 0.109 %
65
+ 58 7.2411 ± 0.0792 -0.01857 ± 0.00142 0.03593 ± 0.00056 5.297 ± 0.071 % 92.500 ± 0.108 %
66
+ 59 7.2130 ± 0.0782 -0.01848 ± 0.00141 0.03625 ± 0.00056 5.318 ± 0.070 % 92.498 ± 0.107 %
67
+ 60 7.1977 ± 0.0774 -0.01823 ± 0.00140 0.03604 ± 0.00056 5.307 ± 0.070 % 92.511 ± 0.106 %
68
+ 61 7.2596 ± 0.0777 -0.02052 ± 0.00141 0.03673 ± 0.00057 5.352 ± 0.069 % 92.441 ± 0.106 %
69
+ 62 7.2841 ± 0.0773 -0.02017 ± 0.00139 0.03655 ± 0.00056 5.333 ± 0.069 % 92.453 ± 0.105 %
70
+ 63 7.3306 ± 0.0773 -0.01985 ± 0.00137 0.03632 ± 0.00055 5.309 ± 0.068 % 92.445 ± 0.104 %
71
+ 64 7.3619 ± 0.0770 -0.01979 ± 0.00136 0.03614 ± 0.00054 5.292 ± 0.067 % 92.446 ± 0.103 %
72
+
73
+ ====== Perplexity statistics ======
74
+ Mean PPL(Q) : 7.361934 ± 0.077037
75
+ Mean PPL(base) : 7.509043 ± 0.079361
76
+ Cor(ln(PPL(Q)), ln(PPL(base))): 99.17%
77
+ Mean ln(PPL(Q)/PPL(base)) : -0.019785 ± 0.001359
78
+ Mean PPL(Q)/PPL(base) : 0.980409 ± 0.001333
79
+ Mean PPL(Q)-PPL(base) : -0.147109 ± 0.010342
80
+
81
+ ====== KL divergence statistics ======
82
+ Mean KLD: 0.036136 ± 0.000542
83
+ Maximum KLD: 13.558516
84
+ 99.9% KLD: 1.497926
85
+ 99.0% KLD: 0.382545
86
+ 95.0% KLD: 0.127583
87
+ 90.0% KLD: 0.075575
88
+ Median KLD: 0.012124
89
+ 10.0% KLD: 0.000183
90
+ 5.0% KLD: 0.000050
91
+ 1.0% KLD: 0.000000
92
+ 0.1% KLD: -0.000128
93
+ Minimum KLD: -0.000768
94
+
95
+ ====== Token probability statistics ======
96
+ Mean Δp: -0.027 ± 0.021 %
97
+ Maximum Δp: 86.379%
98
+ 99.9% Δp: 41.707%
99
+ 99.0% Δp: 16.631%
100
+ 95.0% Δp: 6.410%
101
+ 90.0% Δp: 3.362%
102
+ 75.0% Δp: 0.575%
103
+ Median Δp: -0.001%
104
+ 25.0% Δp: -0.736%
105
+ 10.0% Δp: -3.659%
106
+ 5.0% Δp: -6.538%
107
+ 1.0% Δp: -15.296%
108
+ 0.1% Δp: -39.546%
109
+ Minimum Δp: -94.852%
110
+ RMS Δp : 5.292 ± 0.067 %
111
+ Same top p: 92.446 ± 0.103 %
112
+
appendix/measurements/kld_C.txt ADDED
@@ -0,0 +1,112 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.03.125.524 I cmn init: llama threadpool init, n_threads = 30
2
+ 0.03.226.101 I
3
+ 0.03.226.311 I system_info: n_threads = 30 (n_threads_batch = 30) / 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
4
+ 0.03.580.111 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
5
+ 0.22.733.894 I kl_divergence: 19.15 seconds per pass - ETA 20.42 minutes
6
+
7
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
8
+ 1 5.7746 ± 0.4554 -0.01426 ± 0.01375 0.05820 ± 0.00457 6.691 ± 0.433 % 92.766 ± 0.810 %
9
+ 2 7.2331 ± 0.4201 -0.01040 ± 0.00946 0.05664 ± 0.00303 6.400 ± 0.407 % 90.665 ± 0.643 %
10
+ 3 8.2875 ± 0.4125 0.01074 ± 0.00791 0.05622 ± 0.00323 6.316 ± 0.358 % 91.007 ± 0.516 %
11
+ 4 9.4415 ± 0.4348 0.01283 ± 0.00714 0.06251 ± 0.00372 6.718 ± 0.336 % 90.909 ± 0.449 %
12
+ 5 8.8131 ± 0.3564 0.01150 ± 0.00618 0.05963 ± 0.00305 6.641 ± 0.292 % 90.850 ± 0.403 %
13
+ 6 7.4349 ± 0.2651 0.00408 ± 0.00553 0.05798 ± 0.00263 6.712 ± 0.259 % 90.860 ± 0.368 %
14
+ 7 6.8543 ± 0.2231 0.00833 ± 0.00538 0.05967 ± 0.00246 7.119 ± 0.250 % 90.951 ± 0.339 %
15
+ 8 6.6484 ± 0.1994 0.00732 ± 0.00490 0.05798 ± 0.00219 7.110 ± 0.233 % 91.080 ± 0.315 %
16
+ 9 7.0948 ± 0.2028 0.01398 ± 0.00471 0.05946 ± 0.00200 7.095 ± 0.214 % 90.605 ± 0.304 %
17
+ 10 7.1470 ± 0.1938 0.01218 ± 0.00438 0.05691 ± 0.00182 6.889 ± 0.200 % 90.811 ± 0.286 %
18
+ 11 7.1399 ± 0.1831 0.01068 ± 0.00410 0.05499 ± 0.00167 6.760 ± 0.187 % 90.918 ± 0.271 %
19
+ 12 7.4052 ± 0.1830 0.01090 ± 0.00385 0.05307 ± 0.00154 6.606 ± 0.177 % 91.072 ± 0.257 %
20
+ 13 7.5379 ± 0.1798 0.01339 ± 0.00367 0.05226 ± 0.00144 6.553 ± 0.167 % 91.097 ± 0.247 %
21
+ 14 7.5616 ± 0.1732 0.01018 ± 0.00349 0.05130 ± 0.00135 6.460 ± 0.161 % 91.251 ± 0.236 %
22
+ 15 7.6193 ± 0.1687 0.01000 ± 0.00338 0.05077 ± 0.00127 6.411 ± 0.153 % 91.307 ± 0.227 %
23
+ 16 7.8173 ± 0.1679 0.01090 ± 0.00326 0.05023 ± 0.00120 6.332 ± 0.146 % 91.245 ± 0.221 %
24
+ 17 7.8196 ± 0.1619 0.01025 ± 0.00312 0.04919 ± 0.00114 6.228 ± 0.140 % 91.254 ± 0.214 %
25
+ 18 7.8969 ± 0.1587 0.00985 ± 0.00300 0.04860 ± 0.00109 6.158 ± 0.136 % 91.262 ± 0.208 %
26
+ 19 7.8058 ± 0.1529 0.00765 ± 0.00290 0.04772 ± 0.00104 6.100 ± 0.132 % 91.352 ± 0.202 %
27
+ 20 7.5547 ± 0.1436 0.00734 ± 0.00293 0.05156 ± 0.00113 6.457 ± 0.134 % 91.266 ± 0.197 %
28
+ 21 7.5832 ± 0.1405 0.00567 ± 0.00285 0.05200 ± 0.00110 6.488 ± 0.131 % 91.179 ± 0.193 %
29
+ 22 7.6227 ± 0.1381 0.00084 ± 0.00282 0.05230 ± 0.00107 6.497 ± 0.127 % 91.180 ± 0.189 %
30
+ 23 7.6515 ± 0.1353 0.00110 ± 0.00273 0.05182 ± 0.00103 6.433 ± 0.123 % 91.181 ± 0.185 %
31
+ 24 7.7747 ± 0.1352 0.00349 ± 0.00268 0.05205 ± 0.00100 6.438 ± 0.121 % 91.178 ± 0.181 %
32
+ 25 7.8289 ± 0.1336 0.00202 ± 0.00261 0.05161 ± 0.00096 6.396 ± 0.118 % 91.155 ± 0.178 %
33
+ 26 7.7709 ± 0.1297 0.00121 ± 0.00255 0.05130 ± 0.00094 6.361 ± 0.114 % 91.236 ± 0.173 %
34
+ 27 7.9280 ± 0.1303 0.00153 ± 0.00247 0.05046 ± 0.00091 6.282 ± 0.112 % 91.260 ± 0.170 %
35
+ 28 7.9986 ± 0.1291 0.00065 ± 0.00241 0.05023 ± 0.00088 6.248 ± 0.109 % 91.213 ± 0.167 %
36
+ 29 8.0009 ± 0.1271 0.00138 ± 0.00238 0.05053 ± 0.00087 6.284 ± 0.108 % 91.202 ± 0.164 %
37
+ 30 8.0259 ± 0.1257 -0.00154 ± 0.00237 0.05119 ± 0.00089 6.288 ± 0.105 % 91.176 ± 0.162 %
38
+ 31 7.9212 ± 0.1218 -0.00078 ± 0.00232 0.05084 ± 0.00087 6.270 ± 0.103 % 91.193 ± 0.159 %
39
+ 32 7.7756 ± 0.1173 -0.00170 ± 0.00230 0.05161 ± 0.00087 6.333 ± 0.102 % 91.224 ± 0.156 %
40
+ 33 7.7737 ± 0.1158 0.00016 ± 0.00227 0.05201 ± 0.00087 6.333 ± 0.100 % 91.164 ± 0.154 %
41
+ 34 7.7450 ± 0.1134 -0.00045 ± 0.00223 0.05173 ± 0.00085 6.328 ± 0.099 % 91.145 ± 0.152 %
42
+ 35 7.7340 ± 0.1114 -0.00166 ± 0.00220 0.05170 ± 0.00083 6.316 ± 0.096 % 91.135 ± 0.150 %
43
+ 36 7.7347 ± 0.1099 -0.00225 ± 0.00216 0.05127 ± 0.00081 6.285 ± 0.095 % 91.194 ± 0.148 %
44
+ 37 7.5918 ± 0.1059 -0.00173 ± 0.00211 0.05055 ± 0.00079 6.247 ± 0.093 % 91.258 ± 0.145 %
45
+ 38 7.4811 ± 0.1025 -0.00147 ± 0.00207 0.05025 ± 0.00077 6.241 ± 0.092 % 91.316 ± 0.143 %
46
+ 39 7.3614 ± 0.0991 -0.00172 ± 0.00204 0.04986 ± 0.00077 6.213 ± 0.090 % 91.325 ± 0.141 %
47
+ 40 7.2331 ± 0.0956 -0.00152 ± 0.00200 0.04926 ± 0.00075 6.195 ± 0.089 % 91.388 ± 0.139 %
48
+ 41 7.1551 ± 0.0931 -0.00099 ± 0.00197 0.04880 ± 0.00073 6.172 ± 0.087 % 91.441 ± 0.137 %
49
+ 42 7.1853 ± 0.0924 -0.00092 ± 0.00194 0.04867 ± 0.00072 6.150 ± 0.086 % 91.440 ± 0.135 %
50
+ 43 7.3778 ± 0.0946 -0.00197 ± 0.00191 0.04858 ± 0.00070 6.120 ± 0.084 % 91.384 ± 0.134 %
51
+ 44 7.5153 ± 0.0957 -0.00223 ± 0.00189 0.04845 ± 0.00069 6.084 ± 0.083 % 91.360 ± 0.132 %
52
+ 45 7.4890 ± 0.0941 -0.00176 ± 0.00189 0.04962 ± 0.00069 6.155 ± 0.082 % 91.257 ± 0.132 %
53
+ 46 7.4918 ± 0.0930 -0.00200 ± 0.00187 0.05009 ± 0.00068 6.165 ± 0.081 % 91.253 ± 0.130 %
54
+ 47 7.5589 ± 0.0931 -0.00190 ± 0.00185 0.05003 ± 0.00067 6.157 ± 0.080 % 91.250 ± 0.129 %
55
+ 48 7.5548 ± 0.0919 -0.00185 ± 0.00182 0.04987 ± 0.00067 6.147 ± 0.080 % 91.239 ± 0.128 %
56
+ 49 7.6099 ± 0.0918 -0.00115 ± 0.00181 0.04986 ± 0.00066 6.132 ± 0.078 % 91.216 ± 0.126 %
57
+ 50 7.5628 ± 0.0901 -0.00111 ± 0.00178 0.04949 ± 0.00065 6.102 ± 0.077 % 91.245 ± 0.125 %
58
+ 51 7.5945 ± 0.0895 -0.00116 ± 0.00175 0.04900 ± 0.00063 6.065 ± 0.076 % 91.308 ± 0.123 %
59
+ 52 7.5239 ± 0.0877 -0.00209 ± 0.00173 0.04901 ± 0.00063 6.072 ± 0.076 % 91.313 ± 0.122 %
60
+ 53 7.4972 ± 0.0864 -0.00124 ± 0.00171 0.04879 ± 0.00062 6.059 ± 0.075 % 91.322 ± 0.121 %
61
+ 54 7.4540 ± 0.0849 -0.00144 ± 0.00169 0.04852 ± 0.00061 6.050 ± 0.074 % 91.333 ± 0.120 %
62
+ 55 7.4045 ± 0.0835 -0.00307 ± 0.00169 0.04858 ± 0.00061 6.058 ± 0.073 % 91.313 ± 0.119 %
63
+ 56 7.3970 ± 0.0827 -0.00322 ± 0.00167 0.04848 ± 0.00060 6.057 ± 0.073 % 91.332 ± 0.118 %
64
+ 57 7.3157 ± 0.0808 -0.00352 ± 0.00165 0.04821 ± 0.00059 6.043 ± 0.072 % 91.352 ± 0.116 %
65
+ 58 7.3534 ± 0.0808 -0.00318 ± 0.00165 0.04870 ± 0.00059 6.066 ± 0.071 % 91.344 ± 0.115 %
66
+ 59 7.3261 ± 0.0797 -0.00292 ± 0.00164 0.04942 ± 0.00061 6.119 ± 0.071 % 91.308 ± 0.115 %
67
+ 60 7.3074 ± 0.0788 -0.00311 ± 0.00162 0.04925 ± 0.00060 6.100 ± 0.070 % 91.323 ± 0.114 %
68
+ 61 7.3522 ± 0.0789 -0.00785 ± 0.00166 0.05137 ± 0.00065 6.191 ± 0.070 % 91.201 ± 0.113 %
69
+ 62 7.3762 ± 0.0785 -0.00760 ± 0.00164 0.05108 ± 0.00064 6.166 ± 0.069 % 91.207 ± 0.112 %
70
+ 63 7.4239 ± 0.0784 -0.00721 ± 0.00162 0.05074 ± 0.00063 6.141 ± 0.069 % 91.219 ± 0.111 %
71
+ 64 7.4532 ± 0.0782 -0.00746 ± 0.00161 0.05056 ± 0.00063 6.121 ± 0.068 % 91.239 ± 0.110 %
72
+
73
+ ====== Perplexity statistics ======
74
+ Mean PPL(Q) : 7.453244 ± 0.078172
75
+ Mean PPL(base) : 7.509043 ± 0.079361
76
+ Cor(ln(PPL(Q)), ln(PPL(base))): 98.84%
77
+ Mean ln(PPL(Q)/PPL(base)) : -0.007459 ± 0.001605
78
+ Mean PPL(Q)/PPL(base) : 0.992569 ± 0.001594
79
+ Mean PPL(Q)-PPL(base) : -0.055799 ± 0.012055
80
+
81
+ ====== KL divergence statistics ======
82
+ Mean KLD: 0.050559 ± 0.000627
83
+ Maximum KLD: 10.205813
84
+ 99.9% KLD: 1.798456
85
+ 99.0% KLD: 0.551951
86
+ 95.0% KLD: 0.186914
87
+ 90.0% KLD: 0.108423
88
+ Median KLD: 0.017439
89
+ 10.0% KLD: 0.000265
90
+ 5.0% KLD: 0.000074
91
+ 1.0% KLD: 0.000002
92
+ 0.1% KLD: -0.000127
93
+ Minimum KLD: -0.000620
94
+
95
+ ====== Token probability statistics ======
96
+ Mean Δp: -0.232 ± 0.024 %
97
+ Maximum Δp: 93.422%
98
+ 99.9% Δp: 46.367%
99
+ 99.0% Δp: 18.218%
100
+ 95.0% Δp: 7.112%
101
+ 90.0% Δp: 3.744%
102
+ 75.0% Δp: 0.576%
103
+ Median Δp: -0.006%
104
+ 25.0% Δp: -1.007%
105
+ 10.0% Δp: -4.662%
106
+ 5.0% Δp: -8.194%
107
+ 1.0% Δp: -19.743%
108
+ 0.1% Δp: -45.428%
109
+ Minimum Δp: -91.063%
110
+ RMS Δp : 6.121 ± 0.068 %
111
+ Same top p: 91.239 ± 0.110 %
112
+
appendix/measurements/kld_CTRL-19G-ICE.txt ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.04.811.851 I
2
+ 0.04.812.006 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
3
+ 0.05.135.998 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
4
+ 0.06.885.259 I kl_divergence: 1.75 seconds per pass - ETA 1.85 minutes
5
+
6
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
7
+ 1 5.7833 ± 0.4508 -0.01275 ± 0.01454 0.07915 ± 0.00579 8.002 ± 0.535 % 88.954 ± 0.981 %
8
+ 2 7.3898 ± 0.4323 0.01104 ± 0.01044 0.07059 ± 0.00350 7.018 ± 0.351 % 89.052 ± 0.690 %
9
+ 3 8.3892 ± 0.4164 0.02294 ± 0.00852 0.06800 ± 0.00354 7.187 ± 0.389 % 89.736 ± 0.548 %
10
+ 4 9.5061 ± 0.4348 0.01965 ± 0.00771 0.07443 ± 0.00389 7.507 ± 0.347 % 89.858 ± 0.472 %
11
+ 5 8.9037 ± 0.3584 0.02172 ± 0.00687 0.07195 ± 0.00322 7.344 ± 0.296 % 90.029 ± 0.419 %
12
+ 6 7.5269 ± 0.2667 0.01638 ± 0.00618 0.07035 ± 0.00282 7.487 ± 0.265 % 89.899 ± 0.385 %
13
+ 7 6.8182 ± 0.2190 0.00304 ± 0.00602 0.07240 ± 0.00270 7.797 ± 0.260 % 89.973 ± 0.355 %
14
+ 8 6.6126 ± 0.1958 0.00193 ± 0.00548 0.07114 ± 0.00243 7.705 ± 0.237 % 89.907 ± 0.333 %
15
+ 9 7.0222 ± 0.1980 0.00370 ± 0.00516 0.07151 ± 0.00221 7.605 ± 0.218 % 89.551 ± 0.319 %
16
+ 10 7.0805 ± 0.1896 0.00284 ± 0.00479 0.06901 ± 0.00202 7.495 ± 0.206 % 89.658 ± 0.301 %
17
+ 11 7.0838 ± 0.1795 0.00278 ± 0.00450 0.06675 ± 0.00186 7.341 ± 0.194 % 89.674 ± 0.287 %
18
+ 12 7.3486 ± 0.1795 0.00323 ± 0.00422 0.06445 ± 0.00171 7.170 ± 0.183 % 89.858 ± 0.272 %
19
+ 13 7.4886 ± 0.1768 0.00683 ± 0.00402 0.06393 ± 0.00161 7.177 ± 0.173 % 89.871 ± 0.262 %
20
+ 14 7.5258 ± 0.1708 0.00544 ± 0.00380 0.06179 ± 0.00150 7.006 ± 0.165 % 89.994 ± 0.251 %
21
+ 15 7.5721 ± 0.1661 0.00378 ± 0.00366 0.06139 ± 0.00142 6.982 ± 0.157 % 90.062 ± 0.242 %
22
+ 16 7.7601 ± 0.1650 0.00356 ± 0.00354 0.06090 ± 0.00135 6.907 ± 0.151 % 90.090 ± 0.234 %
23
+ 17 7.7677 ± 0.1594 0.00358 ± 0.00338 0.05950 ± 0.00128 6.795 ± 0.145 % 90.167 ± 0.226 %
24
+ 18 7.8469 ± 0.1564 0.00350 ± 0.00328 0.05918 ± 0.00123 6.759 ± 0.140 % 90.263 ± 0.218 %
25
+ 19 7.7659 ± 0.1510 0.00253 ± 0.00316 0.05823 ± 0.00119 6.666 ± 0.135 % 90.323 ± 0.212 %
26
+ 20 7.5403 ± 0.1425 0.00544 ± 0.00319 0.06375 ± 0.00132 7.154 ± 0.142 % 90.098 ± 0.209 %
27
+ 21 7.5886 ± 0.1399 0.00639 ± 0.00313 0.06403 ± 0.00131 7.142 ± 0.139 % 90.029 ± 0.204 %
28
+ 22 7.6349 ± 0.1378 0.00244 ± 0.00307 0.06454 ± 0.00127 7.161 ± 0.135 % 90.052 ± 0.200 %
29
+ 23 7.6638 ± 0.1350 0.00271 ± 0.00299 0.06391 ± 0.00122 7.082 ± 0.130 % 90.059 ± 0.195 %
30
+ 24 7.7682 ± 0.1344 0.00266 ± 0.00294 0.06388 ± 0.00119 7.054 ± 0.127 % 90.066 ± 0.191 %
31
+ 25 7.8383 ± 0.1333 0.00322 ± 0.00287 0.06307 ± 0.00115 6.984 ± 0.123 % 90.076 ± 0.187 %
32
+ 26 7.7847 ± 0.1295 0.00299 ± 0.00280 0.06286 ± 0.00111 6.970 ± 0.120 % 90.089 ± 0.183 %
33
+ 27 7.9389 ± 0.1300 0.00291 ± 0.00273 0.06214 ± 0.00108 6.902 ± 0.118 % 90.098 ± 0.180 %
34
+ 28 8.0109 ± 0.1289 0.00220 ± 0.00268 0.06180 ± 0.00105 6.870 ± 0.115 % 90.089 ± 0.177 %
35
+ 29 8.0076 ± 0.1267 0.00222 ± 0.00265 0.06273 ± 0.00106 7.014 ± 0.117 % 90.066 ± 0.174 %
36
+ 30 8.0317 ± 0.1254 -0.00082 ± 0.00262 0.06332 ± 0.00106 7.025 ± 0.115 % 90.033 ± 0.171 %
37
+ 31 7.9285 ± 0.1215 0.00014 ± 0.00257 0.06293 ± 0.00103 7.001 ± 0.113 % 90.039 ± 0.168 %
38
+ 32 7.7909 ± 0.1172 0.00027 ± 0.00254 0.06323 ± 0.00102 7.047 ± 0.112 % 90.112 ± 0.165 %
39
+ 33 7.7680 ± 0.1153 -0.00057 ± 0.00251 0.06365 ± 0.00104 7.050 ± 0.110 % 90.056 ± 0.163 %
40
+ 34 7.7427 ± 0.1129 -0.00074 ± 0.00246 0.06291 ± 0.00101 6.997 ± 0.108 % 90.124 ± 0.160 %
41
+ 35 7.7394 ± 0.1112 -0.00097 ± 0.00241 0.06247 ± 0.00098 6.953 ± 0.105 % 90.180 ± 0.157 %
42
+ 36 7.7470 ± 0.1098 -0.00066 ± 0.00237 0.06194 ± 0.00096 6.924 ± 0.103 % 90.255 ± 0.155 %
43
+ 37 7.6047 ± 0.1058 -0.00003 ± 0.00232 0.06115 ± 0.00094 6.887 ± 0.102 % 90.320 ± 0.152 %
44
+ 38 7.4948 ± 0.1024 0.00036 ± 0.00229 0.06103 ± 0.00092 6.910 ± 0.101 % 90.348 ± 0.150 %
45
+ 39 7.3786 ± 0.0991 0.00063 ± 0.00225 0.06084 ± 0.00091 6.922 ± 0.100 % 90.343 ± 0.148 %
46
+ 40 7.2511 ± 0.0957 0.00096 ± 0.00221 0.06020 ± 0.00090 6.903 ± 0.098 % 90.423 ± 0.145 %
47
+ 41 7.1732 ± 0.0931 0.00154 ± 0.00217 0.05985 ± 0.00088 6.911 ± 0.098 % 90.444 ± 0.144 %
48
+ 42 7.2045 ± 0.0924 0.00175 ± 0.00214 0.05966 ± 0.00086 6.883 ± 0.096 % 90.444 ± 0.142 %
49
+ 43 7.4009 ± 0.0947 0.00116 ± 0.00211 0.05951 ± 0.00085 6.841 ± 0.095 % 90.398 ± 0.140 %
50
+ 44 7.5456 ± 0.0960 0.00179 ± 0.00208 0.05923 ± 0.00084 6.804 ± 0.093 % 90.369 ± 0.139 %
51
+ 45 7.5222 ± 0.0944 0.00266 ± 0.00208 0.06082 ± 0.00083 6.890 ± 0.092 % 90.264 ± 0.138 %
52
+ 46 7.5250 ± 0.0934 0.00242 ± 0.00207 0.06140 ± 0.00083 6.901 ± 0.091 % 90.255 ± 0.137 %
53
+ 47 7.5876 ± 0.0933 0.00189 ± 0.00204 0.06135 ± 0.00082 6.894 ± 0.089 % 90.241 ± 0.135 %
54
+ 48 7.5800 ± 0.0921 0.00149 ± 0.00201 0.06127 ± 0.00081 6.890 ± 0.089 % 90.237 ± 0.134 %
55
+ 49 7.6268 ± 0.0918 0.00106 ± 0.00199 0.06110 ± 0.00079 6.886 ± 0.087 % 90.231 ± 0.133 %
56
+ 50 7.5809 ± 0.0902 0.00128 ± 0.00197 0.06070 ± 0.00078 6.859 ± 0.086 % 90.270 ± 0.131 %
57
+ 51 7.6211 ± 0.0897 0.00234 ± 0.00194 0.06019 ± 0.00077 6.834 ± 0.086 % 90.313 ± 0.129 %
58
+ 52 7.5574 ± 0.0880 0.00235 ± 0.00192 0.05998 ± 0.00076 6.822 ± 0.085 % 90.336 ± 0.128 %
59
+ 53 7.5264 ± 0.0867 0.00265 ± 0.00190 0.05957 ± 0.00075 6.804 ± 0.084 % 90.347 ± 0.127 %
60
+ 54 7.4824 ± 0.0852 0.00236 ± 0.00187 0.05922 ± 0.00074 6.790 ± 0.083 % 90.348 ± 0.126 %
61
+ 55 7.4401 ± 0.0838 0.00173 ± 0.00186 0.05912 ± 0.00073 6.794 ± 0.082 % 90.349 ± 0.124 %
62
+ 56 7.4364 ± 0.0831 0.00210 ± 0.00184 0.05908 ± 0.00072 6.797 ± 0.081 % 90.340 ± 0.123 %
63
+ 57 7.3560 ± 0.0812 0.00197 ± 0.00182 0.05879 ± 0.00071 6.777 ± 0.080 % 90.367 ± 0.122 %
64
+ 58 7.3832 ± 0.0810 0.00086 ± 0.00182 0.05948 ± 0.00072 6.798 ± 0.079 % 90.343 ± 0.121 %
65
+ 59 7.3519 ± 0.0799 0.00061 ± 0.00181 0.06006 ± 0.00072 6.833 ± 0.078 % 90.321 ± 0.120 %
66
+ 60 7.3320 ± 0.0790 0.00025 ± 0.00179 0.05984 ± 0.00071 6.812 ± 0.077 % 90.331 ± 0.119 %
67
+ 61 7.3975 ± 0.0794 -0.00170 ± 0.00179 0.06088 ± 0.00074 6.862 ± 0.078 % 90.278 ± 0.119 %
68
+ 62 7.4215 ± 0.0790 -0.00149 ± 0.00177 0.06057 ± 0.00073 6.830 ± 0.077 % 90.269 ± 0.118 %
69
+ 63 7.4680 ± 0.0789 -0.00128 ± 0.00175 0.06018 ± 0.00072 6.804 ± 0.076 % 90.287 ± 0.117 %
70
+ 64 7.4996 ± 0.0787 -0.00126 ± 0.00174 0.06001 ± 0.00071 6.782 ± 0.075 % 90.310 ± 0.116 %
71
+
72
+ ====== Perplexity statistics ======
73
+ Mean PPL(Q) : 7.499565 ± 0.078688
74
+ Mean PPL(base) : 7.509043 ± 0.079361
75
+ Cor(ln(PPL(Q)), ln(PPL(base))): 98.64%
76
+ Mean ln(PPL(Q)/PPL(base)) : -0.001263 ± 0.001736
77
+ Mean PPL(Q)/PPL(base) : 0.998738 ± 0.001734
78
+ Mean PPL(Q)-PPL(base) : -0.009478 ± 0.013033
79
+
80
+ ====== KL divergence statistics ======
81
+ Mean KLD: 0.060015 ± 0.000710
82
+ Maximum KLD: 8.653050
83
+ 99.9% KLD: 2.424127
84
+ 99.0% KLD: 0.638752
85
+ 95.0% KLD: 0.217372
86
+ 90.0% KLD: 0.128581
87
+ Median KLD: 0.020794
88
+ 10.0% KLD: 0.000334
89
+ 5.0% KLD: 0.000097
90
+ 1.0% KLD: 0.000003
91
+ 0.1% KLD: -0.000155
92
+ Minimum KLD: -0.000756
93
+
94
+ ====== Token probability statistics ======
95
+ Mean Δp: -0.403 ± 0.026 %
96
+ Maximum Δp: 97.079%
97
+ 99.9% Δp: 49.758%
98
+ 99.0% Δp: 19.289%
99
+ 95.0% Δp: 7.458%
100
+ 90.0% Δp: 3.961%
101
+ 75.0% Δp: 0.605%
102
+ Median Δp: -0.009%
103
+ 25.0% Δp: -1.154%
104
+ 10.0% Δp: -5.284%
105
+ 5.0% Δp: -9.417%
106
+ 1.0% Δp: -23.008%
107
+ 0.1% Δp: -53.005%
108
+ Minimum Δp: -94.057%
109
+ RMS Δp : 6.782 ± 0.075 %
110
+ Same top p: 90.310 ± 0.116 %
111
+
appendix/measurements/kld_CTRL-21G-ICE.txt ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.05.350.140 I
2
+ 0.05.350.263 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
3
+ 0.05.670.452 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
4
+ 0.07.506.851 I kl_divergence: 1.84 seconds per pass - ETA 1.95 minutes
5
+
6
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
7
+ 1 6.0309 ± 0.4838 0.02917 ± 0.01217 0.04986 ± 0.00417 6.052 ± 0.488 % 91.789 ± 0.859 %
8
+ 2 7.3944 ± 0.4341 0.01166 ± 0.00847 0.04562 ± 0.00267 5.548 ± 0.315 % 91.544 ± 0.615 %
9
+ 3 8.4114 ± 0.4198 0.02558 ± 0.00719 0.04565 ± 0.00286 5.658 ± 0.324 % 92.017 ± 0.489 %
10
+ 4 9.3801 ± 0.4280 0.00630 ± 0.00681 0.05353 ± 0.00409 6.182 ± 0.332 % 91.618 ± 0.433 %
11
+ 5 8.7711 ± 0.3529 0.00673 ± 0.00597 0.05056 ± 0.00331 6.000 ± 0.285 % 91.613 ± 0.388 %
12
+ 6 7.4346 ± 0.2640 0.00404 ± 0.00536 0.04938 ± 0.00282 6.135 ± 0.249 % 91.577 ± 0.355 %
13
+ 7 6.8170 ± 0.2205 0.00286 ± 0.00504 0.04905 ± 0.00249 6.299 ± 0.226 % 91.831 ± 0.324 %
14
+ 8 6.6178 ± 0.1975 0.00272 ± 0.00455 0.04742 ± 0.00220 6.192 ± 0.211 % 91.838 ± 0.303 %
15
+ 9 6.9964 ± 0.1983 0.00002 ± 0.00428 0.04768 ± 0.00199 6.124 ± 0.192 % 91.539 ± 0.290 %
16
+ 10 7.0684 ± 0.1902 0.00114 ± 0.00396 0.04543 ± 0.00180 5.937 ± 0.179 % 91.662 ± 0.273 %
17
+ 11 7.0694 ± 0.1801 0.00074 ± 0.00375 0.04390 ± 0.00165 5.803 ± 0.168 % 91.780 ± 0.259 %
18
+ 12 7.3282 ± 0.1798 0.00045 ± 0.00351 0.04247 ± 0.00152 5.688 ± 0.159 % 91.846 ± 0.247 %
19
+ 13 7.4336 ± 0.1758 -0.00055 ± 0.00336 0.04222 ± 0.00142 5.715 ± 0.150 % 91.819 ± 0.238 %
20
+ 14 7.4678 ± 0.1698 -0.00230 ± 0.00319 0.04106 ± 0.00132 5.616 ± 0.144 % 91.810 ± 0.229 %
21
+ 15 7.5214 ± 0.1653 -0.00295 ± 0.00306 0.04063 ± 0.00124 5.575 ± 0.136 % 91.795 ± 0.222 %
22
+ 16 7.6955 ± 0.1637 -0.00480 ± 0.00296 0.04048 ± 0.00117 5.528 ± 0.130 % 91.771 ± 0.215 %
23
+ 17 7.7066 ± 0.1583 -0.00431 ± 0.00283 0.03961 ± 0.00111 5.433 ± 0.125 % 91.864 ± 0.207 %
24
+ 18 7.7867 ± 0.1552 -0.00420 ± 0.00272 0.03927 ± 0.00105 5.393 ± 0.120 % 91.941 ± 0.201 %
25
+ 19 7.7076 ± 0.1499 -0.00501 ± 0.00263 0.03869 ± 0.00101 5.338 ± 0.116 % 92.041 ± 0.194 %
26
+ 20 7.4614 ± 0.1408 -0.00509 ± 0.00265 0.04178 ± 0.00107 5.731 ± 0.122 % 91.901 ± 0.191 %
27
+ 21 7.5006 ± 0.1381 -0.00528 ± 0.00260 0.04196 ± 0.00103 5.750 ± 0.119 % 91.840 ± 0.187 %
28
+ 22 7.5735 ± 0.1367 -0.00563 ± 0.00253 0.04162 ± 0.00099 5.710 ± 0.115 % 91.895 ± 0.182 %
29
+ 23 7.6001 ± 0.1339 -0.00564 ± 0.00245 0.04118 ± 0.00094 5.650 ± 0.112 % 91.908 ± 0.178 %
30
+ 24 7.6914 ± 0.1331 -0.00727 ± 0.00240 0.04140 ± 0.00092 5.675 ± 0.110 % 91.935 ± 0.174 %
31
+ 25 7.7503 ± 0.1317 -0.00807 ± 0.00236 0.04112 ± 0.00088 5.616 ± 0.107 % 91.922 ± 0.170 %
32
+ 26 7.6940 ± 0.1279 -0.00873 ± 0.00230 0.04098 ± 0.00086 5.615 ± 0.105 % 91.954 ± 0.167 %
33
+ 27 7.8477 ± 0.1283 -0.00865 ± 0.00224 0.04032 ± 0.00083 5.557 ± 0.102 % 91.952 ± 0.164 %
34
+ 28 7.9236 ± 0.1274 -0.00876 ± 0.00219 0.04002 ± 0.00080 5.539 ± 0.100 % 91.935 ± 0.161 %
35
+ 29 7.9324 ± 0.1256 -0.00721 ± 0.00216 0.04030 ± 0.00079 5.561 ± 0.098 % 91.930 ± 0.158 %
36
+ 30 7.9679 ± 0.1245 -0.00879 ± 0.00214 0.04029 ± 0.00077 5.557 ± 0.096 % 91.968 ± 0.155 %
37
+ 31 7.8584 ± 0.1204 -0.00873 ± 0.00210 0.04010 ± 0.00075 5.555 ± 0.094 % 91.978 ± 0.153 %
38
+ 32 7.7381 ± 0.1166 -0.00653 ± 0.00207 0.04039 ± 0.00075 5.651 ± 0.098 % 92.000 ± 0.150 %
39
+ 33 7.7236 ± 0.1148 -0.00630 ± 0.00204 0.04049 ± 0.00078 5.646 ± 0.096 % 92.017 ± 0.148 %
40
+ 34 7.6973 ± 0.1125 -0.00663 ± 0.00200 0.04014 ± 0.00076 5.628 ± 0.094 % 92.016 ± 0.145 %
41
+ 35 7.6902 ± 0.1107 -0.00734 ± 0.00196 0.04002 ± 0.00075 5.610 ± 0.092 % 92.018 ± 0.143 %
42
+ 36 7.6954 ± 0.1093 -0.00735 ± 0.00193 0.03969 ± 0.00073 5.572 ± 0.091 % 92.079 ± 0.141 %
43
+ 37 7.5515 ± 0.1053 -0.00705 ± 0.00189 0.03922 ± 0.00071 5.557 ± 0.090 % 92.143 ± 0.138 %
44
+ 38 7.4407 ± 0.1018 -0.00688 ± 0.00185 0.03892 ± 0.00070 5.549 ± 0.088 % 92.206 ± 0.136 %
45
+ 39 7.3249 ± 0.0985 -0.00668 ± 0.00182 0.03867 ± 0.00068 5.548 ± 0.087 % 92.217 ± 0.134 %
46
+ 40 7.1989 ± 0.0951 -0.00625 ± 0.00178 0.03835 ± 0.00067 5.556 ± 0.086 % 92.253 ± 0.132 %
47
+ 41 7.1195 ± 0.0925 -0.00597 ± 0.00175 0.03805 ± 0.00066 5.538 ± 0.084 % 92.247 ± 0.131 %
48
+ 42 7.1520 ± 0.0919 -0.00557 ± 0.00172 0.03790 ± 0.00064 5.522 ± 0.083 % 92.252 ± 0.129 %
49
+ 43 7.3406 ± 0.0940 -0.00702 ± 0.00171 0.03798 ± 0.00063 5.498 ± 0.082 % 92.194 ± 0.128 %
50
+ 44 7.4801 ± 0.0952 -0.00692 ± 0.00169 0.03787 ± 0.00062 5.471 ± 0.080 % 92.200 ± 0.126 %
51
+ 45 7.4620 ± 0.0938 -0.00538 ± 0.00169 0.03925 ± 0.00063 5.553 ± 0.080 % 92.091 ± 0.126 %
52
+ 46 7.4667 ± 0.0928 -0.00536 ± 0.00167 0.03939 ± 0.00062 5.562 ± 0.079 % 92.076 ± 0.125 %
53
+ 47 7.5273 ± 0.0927 -0.00608 ± 0.00166 0.03956 ± 0.00061 5.567 ± 0.078 % 92.068 ± 0.123 %
54
+ 48 7.5228 ± 0.0915 -0.00609 ± 0.00163 0.03938 ± 0.00060 5.545 ± 0.077 % 92.056 ± 0.122 %
55
+ 49 7.5703 ± 0.0912 -0.00637 ± 0.00162 0.03947 ± 0.00060 5.538 ± 0.076 % 92.016 ± 0.121 %
56
+ 50 7.5274 ± 0.0896 -0.00580 ± 0.00159 0.03923 ± 0.00059 5.518 ± 0.075 % 92.051 ± 0.120 %
57
+ 51 7.5583 ± 0.0891 -0.00593 ± 0.00157 0.03902 ± 0.00058 5.493 ± 0.074 % 92.042 ± 0.118 %
58
+ 52 7.4924 ± 0.0873 -0.00628 ± 0.00155 0.03891 ± 0.00058 5.491 ± 0.073 % 92.054 ± 0.117 %
59
+ 53 7.4618 ± 0.0860 -0.00598 ± 0.00153 0.03864 ± 0.00057 5.469 ± 0.072 % 92.056 ± 0.116 %
60
+ 54 7.4190 ± 0.0845 -0.00614 ± 0.00151 0.03836 ± 0.00056 5.446 ± 0.071 % 92.082 ± 0.115 %
61
+ 55 7.3764 ± 0.0832 -0.00687 ± 0.00150 0.03824 ± 0.00055 5.443 ± 0.070 % 92.086 ± 0.114 %
62
+ 56 7.3702 ± 0.0824 -0.00684 ± 0.00149 0.03823 ± 0.00054 5.451 ± 0.069 % 92.077 ± 0.113 %
63
+ 57 7.2910 ± 0.0805 -0.00691 ± 0.00147 0.03804 ± 0.00054 5.434 ± 0.069 % 92.104 ± 0.112 %
64
+ 58 7.3315 ± 0.0806 -0.00616 ± 0.00147 0.03854 ± 0.00054 5.445 ± 0.068 % 92.077 ± 0.111 %
65
+ 59 7.3036 ± 0.0795 -0.00599 ± 0.00147 0.03896 ± 0.00054 5.491 ± 0.068 % 92.080 ± 0.110 %
66
+ 60 7.2853 ± 0.0786 -0.00614 ± 0.00145 0.03879 ± 0.00054 5.480 ± 0.067 % 92.092 ± 0.109 %
67
+ 61 7.3472 ± 0.0789 -0.00852 ± 0.00146 0.03947 ± 0.00055 5.516 ± 0.067 % 92.021 ± 0.108 %
68
+ 62 7.3696 ± 0.0785 -0.00850 ± 0.00144 0.03930 ± 0.00054 5.494 ± 0.066 % 92.014 ± 0.108 %
69
+ 63 7.4212 ± 0.0785 -0.00757 ± 0.00143 0.03925 ± 0.00053 5.482 ± 0.065 % 92.006 ± 0.107 %
70
+ 64 7.4507 ± 0.0783 -0.00780 ± 0.00142 0.03918 ± 0.00053 5.470 ± 0.065 % 92.023 ± 0.106 %
71
+
72
+ ====== Perplexity statistics ======
73
+ Mean PPL(Q) : 7.450665 ± 0.078273
74
+ Mean PPL(base) : 7.509043 ± 0.079361
75
+ Cor(ln(PPL(Q)), ln(PPL(base))): 99.10%
76
+ Mean ln(PPL(Q)/PPL(base)) : -0.007805 ± 0.001417
77
+ Mean PPL(Q)/PPL(base) : 0.992226 ± 0.001406
78
+ Mean PPL(Q)-PPL(base) : -0.058378 ± 0.010646
79
+
80
+ ====== KL divergence statistics ======
81
+ Mean KLD: 0.039182 ± 0.000526
82
+ Maximum KLD: 9.558251
83
+ 99.9% KLD: 1.416135
84
+ 99.0% KLD: 0.419083
85
+ 95.0% KLD: 0.140715
86
+ 90.0% KLD: 0.083215
87
+ Median KLD: 0.013487
88
+ 10.0% KLD: 0.000211
89
+ 5.0% KLD: 0.000060
90
+ 1.0% KLD: 0.000001
91
+ 0.1% KLD: -0.000142
92
+ Minimum KLD: -0.000889
93
+
94
+ ====== Token probability statistics ======
95
+ Mean Δp: -0.164 ± 0.021 %
96
+ Maximum Δp: 87.223%
97
+ 99.9% Δp: 41.768%
98
+ 99.0% Δp: 16.309%
99
+ 95.0% Δp: 6.372%
100
+ 90.0% Δp: 3.369%
101
+ 75.0% Δp: 0.537%
102
+ Median Δp: -0.003%
103
+ 25.0% Δp: -0.836%
104
+ 10.0% Δp: -3.981%
105
+ 5.0% Δp: -7.032%
106
+ 1.0% Δp: -17.123%
107
+ 0.1% Δp: -43.687%
108
+ Minimum Δp: -98.439%
109
+ RMS Δp : 5.470 ± 0.065 %
110
+ Same top p: 92.023 ± 0.106 %
111
+
appendix/measurements/kld_CTRL-23G-ICE.txt ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.05.769.522 I
2
+ 0.05.769.657 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
3
+ 0.06.097.336 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
4
+ 0.08.038.108 I kl_divergence: 1.94 seconds per pass - ETA 2.07 minutes
5
+
6
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
7
+ 1 5.8119 ± 0.4530 -0.00781 ± 0.01065 0.04378 ± 0.00324 6.308 ± 0.402 % 92.278 ± 0.835 %
8
+ 2 7.2677 ± 0.4204 -0.00562 ± 0.00755 0.03899 ± 0.00193 5.422 ± 0.257 % 92.033 ± 0.599 %
9
+ 3 8.1933 ± 0.4015 -0.00069 ± 0.00637 0.03746 ± 0.00218 5.271 ± 0.277 % 92.603 ± 0.472 %
10
+ 4 9.2533 ± 0.4191 -0.00731 ± 0.00556 0.04194 ± 0.00252 5.552 ± 0.268 % 92.326 ± 0.416 %
11
+ 5 8.6475 ± 0.3446 -0.00747 ± 0.00487 0.04034 ± 0.00208 5.492 ± 0.239 % 92.493 ± 0.368 %
12
+ 6 7.3445 ± 0.2588 -0.00815 ± 0.00447 0.03997 ± 0.00190 5.654 ± 0.236 % 92.441 ± 0.337 %
13
+ 7 6.7462 ± 0.2167 -0.00758 ± 0.00420 0.04017 ± 0.00172 5.910 ± 0.225 % 92.417 ± 0.313 %
14
+ 8 6.5544 ± 0.1943 -0.00691 ± 0.00382 0.03855 ± 0.00152 5.755 ± 0.204 % 92.522 ± 0.291 %
15
+ 9 6.9689 ± 0.1969 -0.00391 ± 0.00365 0.03968 ± 0.00142 5.714 ± 0.186 % 92.147 ± 0.280 %
16
+ 10 7.0371 ± 0.1889 -0.00331 ± 0.00341 0.03818 ± 0.00129 5.608 ± 0.177 % 92.326 ± 0.263 %
17
+ 11 7.0359 ± 0.1787 -0.00401 ± 0.00324 0.03715 ± 0.00119 5.512 ± 0.166 % 92.491 ± 0.248 %
18
+ 12 7.3063 ± 0.1790 -0.00255 ± 0.00304 0.03589 ± 0.00110 5.378 ± 0.156 % 92.481 ± 0.238 %
19
+ 13 7.3794 ± 0.1739 -0.00786 ± 0.00296 0.03632 ± 0.00106 5.418 ± 0.147 % 92.518 ± 0.228 %
20
+ 14 7.4213 ± 0.1682 -0.00855 ± 0.00281 0.03526 ± 0.00099 5.315 ± 0.140 % 92.564 ± 0.219 %
21
+ 15 7.4698 ± 0.1635 -0.00982 ± 0.00272 0.03499 ± 0.00093 5.292 ± 0.134 % 92.616 ± 0.211 %
22
+ 16 7.6523 ± 0.1623 -0.01043 ± 0.00264 0.03465 ± 0.00088 5.217 ± 0.128 % 92.638 ± 0.204 %
23
+ 17 7.6601 ± 0.1568 -0.01036 ± 0.00251 0.03398 ± 0.00084 5.132 ± 0.122 % 92.651 ± 0.198 %
24
+ 18 7.7460 ± 0.1539 -0.00944 ± 0.00243 0.03378 ± 0.00081 5.119 ± 0.118 % 92.701 ± 0.192 %
25
+ 19 7.6641 ± 0.1486 -0.01067 ± 0.00236 0.03355 ± 0.00078 5.075 ± 0.114 % 92.699 ± 0.187 %
26
+ 20 7.4005 ± 0.1393 -0.01328 ± 0.00241 0.03721 ± 0.00091 5.497 ± 0.123 % 92.493 ± 0.184 %
27
+ 21 7.4419 ± 0.1366 -0.01313 ± 0.00237 0.03775 ± 0.00090 5.543 ± 0.122 % 92.417 ± 0.181 %
28
+ 22 7.5111 ± 0.1352 -0.01391 ± 0.00231 0.03753 ± 0.00087 5.530 ± 0.119 % 92.446 ± 0.176 %
29
+ 23 7.5419 ± 0.1325 -0.01332 ± 0.00223 0.03702 ± 0.00083 5.466 ± 0.115 % 92.443 ± 0.172 %
30
+ 24 7.6194 ± 0.1313 -0.01668 ± 0.00221 0.03766 ± 0.00083 5.504 ± 0.115 % 92.388 ± 0.169 %
31
+ 25 7.6863 ± 0.1302 -0.01637 ± 0.00215 0.03725 ± 0.00079 5.457 ± 0.111 % 92.336 ± 0.166 %
32
+ 26 7.6396 ± 0.1266 -0.01583 ± 0.00210 0.03699 ± 0.00077 5.451 ± 0.109 % 92.379 ± 0.163 %
33
+ 27 7.7923 ± 0.1271 -0.01574 ± 0.00204 0.03642 ± 0.00074 5.390 ± 0.107 % 92.346 ± 0.160 %
34
+ 28 7.8639 ± 0.1261 -0.01632 ± 0.00200 0.03620 ± 0.00072 5.352 ± 0.104 % 92.340 ± 0.157 %
35
+ 29 7.8698 ± 0.1242 -0.01514 ± 0.00197 0.03624 ± 0.00070 5.369 ± 0.101 % 92.345 ± 0.154 %
36
+ 30 7.9050 ± 0.1231 -0.01672 ± 0.00196 0.03658 ± 0.00069 5.380 ± 0.098 % 92.317 ± 0.152 %
37
+ 31 7.8014 ± 0.1192 -0.01602 ± 0.00192 0.03627 ± 0.00067 5.352 ± 0.096 % 92.347 ± 0.149 %
38
+ 32 7.6685 ± 0.1151 -0.01557 ± 0.00189 0.03625 ± 0.00066 5.386 ± 0.095 % 92.391 ± 0.147 %
39
+ 33 7.6595 ± 0.1135 -0.01464 ± 0.00186 0.03626 ± 0.00066 5.376 ± 0.093 % 92.393 ± 0.144 %
40
+ 34 7.6351 ± 0.1112 -0.01474 ± 0.00182 0.03591 ± 0.00064 5.350 ± 0.091 % 92.387 ± 0.142 %
41
+ 35 7.6315 ± 0.1095 -0.01501 ± 0.00178 0.03565 ± 0.00062 5.329 ± 0.089 % 92.401 ± 0.140 %
42
+ 36 7.6399 ± 0.1082 -0.01459 ± 0.00175 0.03532 ± 0.00061 5.307 ± 0.088 % 92.460 ± 0.138 %
43
+ 37 7.4974 ± 0.1042 -0.01424 ± 0.00172 0.03488 ± 0.00060 5.281 ± 0.087 % 92.507 ± 0.135 %
44
+ 38 7.3870 ± 0.1008 -0.01412 ± 0.00169 0.03455 ± 0.00058 5.262 ± 0.085 % 92.563 ± 0.133 %
45
+ 39 7.2748 ± 0.0976 -0.01354 ± 0.00166 0.03422 ± 0.00057 5.237 ± 0.084 % 92.596 ± 0.131 %
46
+ 40 7.1509 ± 0.0943 -0.01295 ± 0.00162 0.03379 ± 0.00056 5.217 ± 0.082 % 92.642 ± 0.129 %
47
+ 41 7.0727 ± 0.0917 -0.01257 ± 0.00160 0.03350 ± 0.00054 5.202 ± 0.081 % 92.642 ± 0.127 %
48
+ 42 7.1021 ± 0.0910 -0.01257 ± 0.00158 0.03337 ± 0.00053 5.186 ± 0.080 % 92.655 ± 0.126 %
49
+ 43 7.3006 ± 0.0934 -0.01248 ± 0.00156 0.03347 ± 0.00053 5.187 ± 0.081 % 92.621 ± 0.125 %
50
+ 44 7.4411 ± 0.0946 -0.01215 ± 0.00154 0.03351 ± 0.00054 5.159 ± 0.079 % 92.635 ± 0.123 %
51
+ 45 7.4173 ± 0.0931 -0.01138 ± 0.00154 0.03440 ± 0.00054 5.233 ± 0.078 % 92.553 ± 0.122 %
52
+ 46 7.4232 ± 0.0921 -0.01120 ± 0.00152 0.03460 ± 0.00053 5.238 ± 0.077 % 92.545 ± 0.121 %
53
+ 47 7.4825 ± 0.0920 -0.01206 ± 0.00151 0.03477 ± 0.00053 5.250 ± 0.077 % 92.540 ± 0.120 %
54
+ 48 7.4807 ± 0.0909 -0.01170 ± 0.00149 0.03472 ± 0.00053 5.242 ± 0.076 % 92.516 ± 0.119 %
55
+ 49 7.5288 ± 0.0906 -0.01187 ± 0.00148 0.03481 ± 0.00053 5.235 ± 0.075 % 92.481 ± 0.118 %
56
+ 50 7.4853 ± 0.0890 -0.01141 ± 0.00145 0.03454 ± 0.00052 5.212 ± 0.074 % 92.526 ± 0.116 %
57
+ 51 7.5203 ± 0.0885 -0.01097 ± 0.00143 0.03422 ± 0.00051 5.180 ± 0.073 % 92.557 ± 0.115 %
58
+ 52 7.4595 ± 0.0868 -0.01069 ± 0.00142 0.03407 ± 0.00050 5.174 ± 0.073 % 92.582 ± 0.114 %
59
+ 53 7.4304 ± 0.0856 -0.01020 ± 0.00140 0.03388 ± 0.00050 5.160 ± 0.072 % 92.569 ± 0.113 %
60
+ 54 7.3881 ± 0.0841 -0.01032 ± 0.00138 0.03369 ± 0.00049 5.159 ± 0.071 % 92.589 ± 0.111 %
61
+ 55 7.3447 ± 0.0827 -0.01118 ± 0.00138 0.03364 ± 0.00049 5.162 ± 0.070 % 92.598 ± 0.110 %
62
+ 56 7.3319 ± 0.0818 -0.01205 ± 0.00137 0.03365 ± 0.00048 5.158 ± 0.069 % 92.602 ± 0.109 %
63
+ 57 7.2539 ± 0.0800 -0.01201 ± 0.00135 0.03343 ± 0.00047 5.137 ± 0.068 % 92.638 ± 0.108 %
64
+ 58 7.2973 ± 0.0801 -0.01084 ± 0.00136 0.03401 ± 0.00048 5.174 ± 0.068 % 92.611 ± 0.107 %
65
+ 59 7.2734 ± 0.0791 -0.01013 ± 0.00135 0.03430 ± 0.00048 5.193 ± 0.067 % 92.601 ± 0.107 %
66
+ 60 7.2556 ± 0.0782 -0.01022 ± 0.00133 0.03412 ± 0.00047 5.180 ± 0.067 % 92.616 ± 0.106 %
67
+ 61 7.3304 ± 0.0787 -0.01082 ± 0.00133 0.03478 ± 0.00051 5.236 ± 0.069 % 92.576 ± 0.105 %
68
+ 62 7.3547 ± 0.0783 -0.01052 ± 0.00132 0.03461 ± 0.00050 5.211 ± 0.068 % 92.580 ± 0.104 %
69
+ 63 7.3994 ± 0.0783 -0.01051 ± 0.00131 0.03442 ± 0.00050 5.190 ± 0.067 % 92.562 ± 0.103 %
70
+ 64 7.4302 ± 0.0780 -0.01056 ± 0.00129 0.03429 ± 0.00049 5.172 ± 0.067 % 92.574 ± 0.102 %
71
+
72
+ ====== Perplexity statistics ======
73
+ Mean PPL(Q) : 7.430186 ± 0.078031
74
+ Mean PPL(base) : 7.509043 ± 0.079361
75
+ Cor(ln(PPL(Q)), ln(PPL(base))): 99.25%
76
+ Mean ln(PPL(Q)/PPL(base)) : -0.010557 ± 0.001292
77
+ Mean PPL(Q)/PPL(base) : 0.989498 ± 0.001278
78
+ Mean PPL(Q)-PPL(base) : -0.078857 ± 0.009729
79
+
80
+ ====== KL divergence statistics ======
81
+ Mean KLD: 0.034290 ± 0.000491
82
+ Maximum KLD: 7.434389
83
+ 99.9% KLD: 1.506679
84
+ 99.0% KLD: 0.346319
85
+ 95.0% KLD: 0.122503
86
+ 90.0% KLD: 0.072590
87
+ Median KLD: 0.011636
88
+ 10.0% KLD: 0.000186
89
+ 5.0% KLD: 0.000051
90
+ 1.0% KLD: -0.000000
91
+ 0.1% KLD: -0.000146
92
+ Minimum KLD: -0.000892
93
+
94
+ ====== Token probability statistics ======
95
+ Mean Δp: -0.102 ± 0.020 %
96
+ Maximum Δp: 88.208%
97
+ 99.9% Δp: 40.389%
98
+ 99.0% Δp: 15.604%
99
+ 95.0% Δp: 6.070%
100
+ 90.0% Δp: 3.272%
101
+ 75.0% Δp: 0.532%
102
+ Median Δp: -0.002%
103
+ 25.0% Δp: -0.746%
104
+ 10.0% Δp: -3.679%
105
+ 5.0% Δp: -6.481%
106
+ 1.0% Δp: -15.971%
107
+ 0.1% Δp: -39.102%
108
+ Minimum Δp: -96.886%
109
+ RMS Δp : 5.172 ± 0.067 %
110
+ Same top p: 92.574 ± 0.102 %
111
+
appendix/measurements/kld_CTRL-25G-ICE.txt ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.05.877.958 I
2
+ 0.05.878.099 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
3
+ 0.06.238.869 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
4
+ 0.08.156.863 I kl_divergence: 1.92 seconds per pass - ETA 2.03 minutes
5
+
6
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
7
+ 1 5.7968 ± 0.4527 -0.01041 ± 0.00981 0.03069 ± 0.00227 5.166 ± 0.382 % 93.451 ± 0.774 %
8
+ 2 7.3020 ± 0.4252 -0.00091 ± 0.00684 0.02912 ± 0.00142 4.744 ± 0.271 % 93.011 ± 0.564 %
9
+ 3 8.1596 ± 0.4009 -0.00481 ± 0.00566 0.02795 ± 0.00124 4.594 ± 0.269 % 93.288 ± 0.452 %
10
+ 4 9.2581 ± 0.4203 -0.00679 ± 0.00529 0.03488 ± 0.00253 5.058 ± 0.303 % 93.084 ± 0.397 %
11
+ 5 8.6612 ± 0.3465 -0.00588 ± 0.00463 0.03295 ± 0.00205 4.942 ± 0.255 % 93.255 ± 0.351 %
12
+ 6 7.3374 ± 0.2591 -0.00913 ± 0.00424 0.03346 ± 0.00189 5.239 ± 0.257 % 93.076 ± 0.324 %
13
+ 7 6.7062 ± 0.2151 -0.01352 ± 0.00399 0.03368 ± 0.00167 5.417 ± 0.229 % 93.185 ± 0.298 %
14
+ 8 6.5151 ± 0.1929 -0.01293 ± 0.00363 0.03236 ± 0.00148 5.303 ± 0.208 % 93.194 ± 0.278 %
15
+ 9 6.8917 ± 0.1939 -0.01506 ± 0.00342 0.03281 ± 0.00134 5.234 ± 0.190 % 92.962 ± 0.267 %
16
+ 10 6.9693 ± 0.1864 -0.01298 ± 0.00321 0.03179 ± 0.00125 5.168 ± 0.187 % 93.021 ± 0.252 %
17
+ 11 6.9712 ± 0.1765 -0.01324 ± 0.00305 0.03082 ± 0.00114 5.079 ± 0.176 % 93.148 ± 0.238 %
18
+ 12 7.2369 ± 0.1766 -0.01209 ± 0.00287 0.03005 ± 0.00106 4.965 ± 0.165 % 93.255 ± 0.226 %
19
+ 13 7.3133 ± 0.1718 -0.01686 ± 0.00282 0.03076 ± 0.00103 5.037 ± 0.155 % 93.180 ± 0.219 %
20
+ 14 7.3681 ± 0.1665 -0.01574 ± 0.00268 0.02986 ± 0.00096 4.939 ± 0.147 % 93.304 ± 0.209 %
21
+ 15 7.4124 ± 0.1617 -0.01753 ± 0.00262 0.02976 ± 0.00090 4.899 ± 0.139 % 93.262 ± 0.202 %
22
+ 16 7.5946 ± 0.1606 -0.01799 ± 0.00253 0.02968 ± 0.00085 4.850 ± 0.133 % 93.255 ± 0.196 %
23
+ 17 7.6106 ± 0.1553 -0.01684 ± 0.00241 0.02905 ± 0.00081 4.770 ± 0.128 % 93.215 ± 0.191 %
24
+ 18 7.6924 ± 0.1524 -0.01639 ± 0.00232 0.02870 ± 0.00077 4.753 ± 0.124 % 93.282 ± 0.184 %
25
+ 19 7.6046 ± 0.1469 -0.01846 ± 0.00227 0.02911 ± 0.00074 4.758 ± 0.118 % 93.199 ± 0.181 %
26
+ 20 7.3674 ± 0.1383 -0.01776 ± 0.00234 0.03277 ± 0.00096 5.213 ± 0.135 % 93.035 ± 0.178 %
27
+ 21 7.4083 ± 0.1357 -0.01765 ± 0.00227 0.03284 ± 0.00093 5.193 ± 0.132 % 93.032 ± 0.174 %
28
+ 22 7.4719 ± 0.1341 -0.01915 ± 0.00221 0.03261 ± 0.00089 5.168 ± 0.127 % 93.064 ± 0.169 %
29
+ 23 7.5057 ± 0.1316 -0.01813 ± 0.00213 0.03221 ± 0.00085 5.110 ± 0.123 % 93.077 ± 0.165 %
30
+ 24 7.5908 ± 0.1306 -0.02045 ± 0.00210 0.03247 ± 0.00083 5.121 ± 0.120 % 93.051 ± 0.162 %
31
+ 25 7.6495 ± 0.1292 -0.02117 ± 0.00206 0.03275 ± 0.00080 5.094 ± 0.116 % 92.966 ± 0.160 %
32
+ 26 7.6041 ± 0.1257 -0.02048 ± 0.00201 0.03241 ± 0.00077 5.069 ± 0.113 % 93.018 ± 0.156 %
33
+ 27 7.7582 ± 0.1262 -0.02012 ± 0.00196 0.03190 ± 0.00074 5.009 ± 0.110 % 93.031 ± 0.153 %
34
+ 28 7.8309 ± 0.1252 -0.02053 ± 0.00191 0.03171 ± 0.00072 4.974 ± 0.107 % 92.990 ± 0.151 %
35
+ 29 7.8302 ± 0.1232 -0.02018 ± 0.00187 0.03155 ± 0.00070 4.968 ± 0.104 % 92.989 ± 0.148 %
36
+ 30 7.8625 ± 0.1221 -0.02211 ± 0.00186 0.03178 ± 0.00068 4.971 ± 0.101 % 93.011 ± 0.146 %
37
+ 31 7.7611 ± 0.1183 -0.02119 ± 0.00183 0.03144 ± 0.00066 4.942 ± 0.099 % 93.047 ± 0.143 %
38
+ 32 7.6238 ± 0.1141 -0.02141 ± 0.00179 0.03123 ± 0.00065 4.954 ± 0.097 % 93.102 ± 0.140 %
39
+ 33 7.6209 ± 0.1126 -0.01969 ± 0.00177 0.03124 ± 0.00063 4.948 ± 0.094 % 93.086 ± 0.138 %
40
+ 34 7.5962 ± 0.1103 -0.01985 ± 0.00173 0.03099 ± 0.00061 4.922 ± 0.092 % 93.120 ± 0.136 %
41
+ 35 7.5948 ± 0.1087 -0.01983 ± 0.00170 0.03074 ± 0.00060 4.900 ± 0.091 % 93.135 ± 0.134 %
42
+ 36 7.5953 ± 0.1072 -0.02044 ± 0.00167 0.03058 ± 0.00059 4.892 ± 0.089 % 93.168 ± 0.131 %
43
+ 37 7.4546 ± 0.1033 -0.01997 ± 0.00163 0.03012 ± 0.00057 4.863 ± 0.088 % 93.252 ± 0.129 %
44
+ 38 7.3478 ± 0.1000 -0.01945 ± 0.00160 0.02989 ± 0.00056 4.851 ± 0.086 % 93.281 ± 0.127 %
45
+ 39 7.2352 ± 0.0968 -0.01900 ± 0.00157 0.02950 ± 0.00054 4.826 ± 0.085 % 93.300 ± 0.125 %
46
+ 40 7.1122 ± 0.0935 -0.01837 ± 0.00153 0.02915 ± 0.00053 4.809 ± 0.083 % 93.353 ± 0.123 %
47
+ 41 7.0336 ± 0.0909 -0.01811 ± 0.00151 0.02889 ± 0.00052 4.796 ± 0.082 % 93.396 ± 0.121 %
48
+ 42 7.0658 ± 0.0903 -0.01770 ± 0.00149 0.02876 ± 0.00051 4.777 ± 0.081 % 93.388 ± 0.120 %
49
+ 43 7.2564 ± 0.0925 -0.01855 ± 0.00147 0.02888 ± 0.00050 4.771 ± 0.080 % 93.321 ± 0.119 %
50
+ 44 7.3917 ± 0.0937 -0.01882 ± 0.00146 0.02892 ± 0.00050 4.747 ± 0.078 % 93.300 ± 0.118 %
51
+ 45 7.3665 ± 0.0921 -0.01826 ± 0.00145 0.02954 ± 0.00050 4.783 ± 0.077 % 93.286 ± 0.117 %
52
+ 46 7.3738 ± 0.0911 -0.01787 ± 0.00144 0.02965 ± 0.00049 4.784 ± 0.076 % 93.255 ± 0.116 %
53
+ 47 7.4335 ± 0.0911 -0.01863 ± 0.00142 0.02967 ± 0.00049 4.782 ± 0.074 % 93.216 ± 0.115 %
54
+ 48 7.4320 ± 0.0900 -0.01823 ± 0.00140 0.02950 ± 0.00048 4.768 ± 0.074 % 93.202 ± 0.114 %
55
+ 49 7.4812 ± 0.0897 -0.01821 ± 0.00139 0.02959 ± 0.00047 4.770 ± 0.073 % 93.159 ± 0.113 %
56
+ 50 7.4390 ± 0.0882 -0.01761 ± 0.00137 0.02938 ± 0.00046 4.750 ± 0.072 % 93.200 ± 0.111 %
57
+ 51 7.4705 ± 0.0876 -0.01761 ± 0.00135 0.02911 ± 0.00046 4.729 ± 0.071 % 93.228 ± 0.110 %
58
+ 52 7.4082 ± 0.0859 -0.01759 ± 0.00133 0.02897 ± 0.00045 4.726 ± 0.070 % 93.244 ± 0.109 %
59
+ 53 7.3794 ± 0.0847 -0.01708 ± 0.00132 0.02876 ± 0.00044 4.704 ± 0.069 % 93.244 ± 0.108 %
60
+ 54 7.3373 ± 0.0832 -0.01722 ± 0.00130 0.02857 ± 0.00043 4.695 ± 0.068 % 93.271 ± 0.107 %
61
+ 55 7.2946 ± 0.0819 -0.01802 ± 0.00129 0.02852 ± 0.00043 4.693 ± 0.067 % 93.257 ± 0.106 %
62
+ 56 7.2865 ± 0.0811 -0.01827 ± 0.00128 0.02843 ± 0.00042 4.688 ± 0.066 % 93.246 ± 0.105 %
63
+ 57 7.2102 ± 0.0793 -0.01805 ± 0.00126 0.02827 ± 0.00041 4.670 ± 0.066 % 93.276 ± 0.104 %
64
+ 58 7.2551 ± 0.0794 -0.01664 ± 0.00127 0.02876 ± 0.00042 4.701 ± 0.065 % 93.250 ± 0.103 %
65
+ 59 7.2291 ± 0.0783 -0.01624 ± 0.00126 0.02899 ± 0.00042 4.723 ± 0.064 % 93.253 ± 0.102 %
66
+ 60 7.2147 ± 0.0776 -0.01588 ± 0.00124 0.02885 ± 0.00041 4.709 ± 0.063 % 93.284 ± 0.101 %
67
+ 61 7.2897 ± 0.0781 -0.01639 ± 0.00125 0.02942 ± 0.00045 4.752 ± 0.065 % 93.258 ± 0.100 %
68
+ 62 7.3132 ± 0.0777 -0.01618 ± 0.00124 0.02929 ± 0.00044 4.734 ± 0.064 % 93.250 ± 0.100 %
69
+ 63 7.3601 ± 0.0777 -0.01584 ± 0.00122 0.02910 ± 0.00044 4.713 ± 0.063 % 93.252 ± 0.099 %
70
+ 64 7.3911 ± 0.0774 -0.01583 ± 0.00121 0.02896 ± 0.00043 4.697 ± 0.063 % 93.275 ± 0.098 %
71
+
72
+ ====== Perplexity statistics ======
73
+ Mean PPL(Q) : 7.391080 ± 0.077442
74
+ Mean PPL(base) : 7.509043 ± 0.079361
75
+ Cor(ln(PPL(Q)), ln(PPL(base))): 99.35%
76
+ Mean ln(PPL(Q)/PPL(base)) : -0.015834 ± 0.001208
77
+ Mean PPL(Q)/PPL(base) : 0.984291 ± 0.001189
78
+ Mean PPL(Q)-PPL(base) : -0.117963 ± 0.009175
79
+
80
+ ====== KL divergence statistics ======
81
+ Mean KLD: 0.028955 ± 0.000430
82
+ Maximum KLD: 9.086314
83
+ 99.9% KLD: 1.139082
84
+ 99.0% KLD: 0.292560
85
+ 95.0% KLD: 0.102705
86
+ 90.0% KLD: 0.061919
87
+ Median KLD: 0.009867
88
+ 10.0% KLD: 0.000162
89
+ 5.0% KLD: 0.000044
90
+ 1.0% KLD: -0.000002
91
+ 0.1% KLD: -0.000138
92
+ Minimum KLD: -0.000905
93
+
94
+ ====== Token probability statistics ======
95
+ Mean Δp: -0.068 ± 0.018 %
96
+ Maximum Δp: 88.211%
97
+ 99.9% Δp: 35.552%
98
+ 99.0% Δp: 14.238%
99
+ 95.0% Δp: 5.545%
100
+ 90.0% Δp: 2.995%
101
+ 75.0% Δp: 0.499%
102
+ Median Δp: -0.002%
103
+ 25.0% Δp: -0.658%
104
+ 10.0% Δp: -3.304%
105
+ 5.0% Δp: -5.901%
106
+ 1.0% Δp: -14.120%
107
+ 0.1% Δp: -35.247%
108
+ Minimum Δp: -92.155%
109
+ RMS Δp : 4.697 ± 0.063 %
110
+ Same top p: 93.275 ± 0.098 %
111
+
appendix/measurements/kld_D.txt ADDED
@@ -0,0 +1,112 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.03.422.955 I cmn init: llama threadpool init, n_threads = 30
2
+ 0.03.537.991 I
3
+ 0.03.538.201 I system_info: n_threads = 30 (n_threads_batch = 30) / 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
4
+ 0.03.878.781 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
5
+ 0.20.045.335 I kl_divergence: 16.17 seconds per pass - ETA 17.23 minutes
6
+
7
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
8
+ 1 5.8659 ± 0.4647 0.00144 ± 0.00731 0.01807 ± 0.00194 4.052 ± 0.474 % 95.308 ± 0.661 %
9
+ 2 7.3604 ± 0.4319 0.00705 ± 0.00518 0.01696 ± 0.00108 3.498 ± 0.284 % 94.819 ± 0.490 %
10
+ 3 8.2667 ± 0.4100 0.00823 ± 0.00427 0.01829 ± 0.00213 3.640 ± 0.395 % 94.591 ± 0.408 %
11
+ 4 9.2085 ± 0.4181 -0.01217 ± 0.00432 0.02478 ± 0.00247 4.429 ± 0.388 % 94.233 ± 0.364 %
12
+ 5 8.5944 ± 0.3434 -0.01363 ± 0.00374 0.02369 ± 0.00199 4.278 ± 0.326 % 94.409 ± 0.321 %
13
+ 6 7.3090 ± 0.2579 -0.01300 ± 0.00345 0.02350 ± 0.00172 4.515 ± 0.293 % 94.314 ± 0.296 %
14
+ 7 6.7287 ± 0.2165 -0.01017 ± 0.00331 0.02397 ± 0.00152 4.666 ± 0.254 % 94.344 ± 0.273 %
15
+ 8 6.5369 ± 0.1940 -0.00958 ± 0.00299 0.02268 ± 0.00134 4.498 ± 0.231 % 94.404 ± 0.254 %
16
+ 9 6.9176 ± 0.1951 -0.01131 ± 0.00283 0.02298 ± 0.00120 4.400 ± 0.211 % 94.081 ± 0.246 %
17
+ 10 6.9862 ± 0.1872 -0.01057 ± 0.00264 0.02201 ± 0.00109 4.271 ± 0.196 % 94.174 ± 0.232 %
18
+ 11 6.9930 ± 0.1775 -0.01011 ± 0.00255 0.02129 ± 0.00100 4.200 ± 0.183 % 94.313 ± 0.218 %
19
+ 12 7.2611 ± 0.1777 -0.00876 ± 0.00240 0.02069 ± 0.00092 4.119 ± 0.172 % 94.379 ± 0.208 %
20
+ 13 7.3936 ± 0.1749 -0.00594 ± 0.00233 0.02087 ± 0.00085 4.132 ± 0.160 % 94.383 ± 0.200 %
21
+ 14 7.4392 ± 0.1692 -0.00613 ± 0.00221 0.02025 ± 0.00080 4.056 ± 0.152 % 94.442 ± 0.191 %
22
+ 15 7.4934 ± 0.1646 -0.00668 ± 0.00214 0.02025 ± 0.00075 4.048 ± 0.143 % 94.435 ± 0.185 %
23
+ 16 7.6714 ± 0.1632 -0.00794 ± 0.00206 0.02011 ± 0.00070 3.995 ± 0.136 % 94.446 ± 0.179 %
24
+ 17 7.6769 ± 0.1575 -0.00817 ± 0.00199 0.01984 ± 0.00068 3.931 ± 0.131 % 94.486 ± 0.173 %
25
+ 18 7.7541 ± 0.1544 -0.00840 ± 0.00191 0.01958 ± 0.00064 3.900 ± 0.125 % 94.558 ± 0.167 %
26
+ 19 7.6663 ± 0.1488 -0.01038 ± 0.00187 0.01980 ± 0.00062 3.899 ± 0.119 % 94.474 ± 0.164 %
27
+ 20 7.4229 ± 0.1400 -0.01026 ± 0.00187 0.02122 ± 0.00067 4.188 ± 0.128 % 94.350 ± 0.161 %
28
+ 21 7.4659 ± 0.1374 -0.00991 ± 0.00182 0.02129 ± 0.00065 4.205 ± 0.126 % 94.354 ± 0.157 %
29
+ 22 7.5378 ± 0.1360 -0.01036 ± 0.00177 0.02115 ± 0.00063 4.200 ± 0.122 % 94.379 ± 0.154 %
30
+ 23 7.5680 ± 0.1333 -0.00987 ± 0.00172 0.02088 ± 0.00060 4.151 ± 0.118 % 94.411 ± 0.150 %
31
+ 24 7.6565 ± 0.1324 -0.01182 ± 0.00170 0.02125 ± 0.00059 4.170 ± 0.115 % 94.396 ± 0.147 %
32
+ 25 7.7201 ± 0.1311 -0.01198 ± 0.00166 0.02103 ± 0.00056 4.133 ± 0.112 % 94.362 ± 0.144 %
33
+ 26 7.6746 ± 0.1275 -0.01125 ± 0.00161 0.02078 ± 0.00054 4.100 ± 0.109 % 94.383 ± 0.141 %
34
+ 27 7.8278 ± 0.1280 -0.01118 ± 0.00157 0.02037 ± 0.00053 4.048 ± 0.106 % 94.388 ± 0.138 %
35
+ 28 7.9086 ± 0.1272 -0.01065 ± 0.00152 0.02008 ± 0.00051 4.013 ± 0.104 % 94.428 ± 0.136 %
36
+ 29 7.9133 ± 0.1253 -0.00963 ± 0.00150 0.02000 ± 0.00049 4.000 ± 0.101 % 94.475 ± 0.133 %
37
+ 30 7.9489 ± 0.1242 -0.01118 ± 0.00149 0.02016 ± 0.00048 4.002 ± 0.098 % 94.435 ± 0.131 %
38
+ 31 7.8374 ± 0.1201 -0.01141 ± 0.00146 0.01994 ± 0.00047 3.982 ± 0.095 % 94.463 ± 0.128 %
39
+ 32 7.7033 ± 0.1160 -0.01104 ± 0.00143 0.01978 ± 0.00046 3.963 ± 0.093 % 94.483 ± 0.126 %
40
+ 33 7.6887 ± 0.1142 -0.01084 ± 0.00142 0.01997 ± 0.00051 3.949 ± 0.091 % 94.484 ± 0.124 %
41
+ 34 7.6628 ± 0.1119 -0.01112 ± 0.00138 0.01967 ± 0.00050 3.924 ± 0.089 % 94.503 ± 0.122 %
42
+ 35 7.6616 ± 0.1103 -0.01107 ± 0.00136 0.01957 ± 0.00049 3.912 ± 0.087 % 94.526 ± 0.120 %
43
+ 36 7.6633 ± 0.1088 -0.01153 ± 0.00133 0.01952 ± 0.00048 3.913 ± 0.086 % 94.558 ± 0.118 %
44
+ 37 7.5207 ± 0.1048 -0.01114 ± 0.00130 0.01922 ± 0.00047 3.887 ± 0.084 % 94.616 ± 0.116 %
45
+ 38 7.4094 ± 0.1014 -0.01110 ± 0.00128 0.01903 ± 0.00045 3.879 ± 0.082 % 94.660 ± 0.114 %
46
+ 39 7.2959 ± 0.0981 -0.01065 ± 0.00126 0.01878 ± 0.00044 3.861 ± 0.081 % 94.674 ± 0.112 %
47
+ 40 7.1687 ± 0.0947 -0.01047 ± 0.00123 0.01851 ± 0.00043 3.842 ± 0.079 % 94.709 ± 0.111 %
48
+ 41 7.0902 ± 0.0921 -0.01009 ± 0.00121 0.01836 ± 0.00042 3.829 ± 0.078 % 94.714 ± 0.109 %
49
+ 42 7.1227 ± 0.0915 -0.00968 ± 0.00120 0.01835 ± 0.00042 3.824 ± 0.077 % 94.682 ± 0.108 %
50
+ 43 7.3184 ± 0.0938 -0.01005 ± 0.00119 0.01850 ± 0.00041 3.808 ± 0.076 % 94.633 ± 0.107 %
51
+ 44 7.4553 ± 0.0950 -0.01025 ± 0.00117 0.01851 ± 0.00040 3.796 ± 0.074 % 94.604 ± 0.106 %
52
+ 45 7.4320 ± 0.0935 -0.00941 ± 0.00117 0.01897 ± 0.00040 3.834 ± 0.073 % 94.545 ± 0.106 %
53
+ 46 7.4364 ± 0.0924 -0.00942 ± 0.00116 0.01897 ± 0.00039 3.833 ± 0.072 % 94.568 ± 0.104 %
54
+ 47 7.5008 ± 0.0924 -0.00961 ± 0.00115 0.01900 ± 0.00039 3.827 ± 0.071 % 94.576 ± 0.103 %
55
+ 48 7.4984 ± 0.0913 -0.00934 ± 0.00113 0.01895 ± 0.00038 3.837 ± 0.071 % 94.561 ± 0.102 %
56
+ 49 7.5469 ± 0.0910 -0.00946 ± 0.00112 0.01902 ± 0.00038 3.830 ± 0.070 % 94.546 ± 0.101 %
57
+ 50 7.5046 ± 0.0894 -0.00883 ± 0.00111 0.01890 ± 0.00037 3.812 ± 0.069 % 94.581 ± 0.100 %
58
+ 51 7.5383 ± 0.0889 -0.00858 ± 0.00109 0.01874 ± 0.00036 3.789 ± 0.068 % 94.599 ± 0.099 %
59
+ 52 7.4769 ± 0.0872 -0.00836 ± 0.00108 0.01861 ± 0.00036 3.781 ± 0.067 % 94.614 ± 0.098 %
60
+ 53 7.4483 ± 0.0859 -0.00779 ± 0.00107 0.01850 ± 0.00035 3.768 ± 0.066 % 94.631 ± 0.097 %
61
+ 54 7.4068 ± 0.0845 -0.00780 ± 0.00106 0.01844 ± 0.00035 3.771 ± 0.067 % 94.635 ± 0.096 %
62
+ 55 7.3647 ± 0.0831 -0.00845 ± 0.00105 0.01842 ± 0.00034 3.781 ± 0.065 % 94.643 ± 0.095 %
63
+ 56 7.3596 ± 0.0824 -0.00828 ± 0.00104 0.01843 ± 0.00034 3.779 ± 0.064 % 94.636 ± 0.094 %
64
+ 57 7.2814 ± 0.0805 -0.00822 ± 0.00103 0.01830 ± 0.00033 3.763 ± 0.064 % 94.667 ± 0.093 %
65
+ 58 7.3195 ± 0.0805 -0.00781 ± 0.00103 0.01873 ± 0.00034 3.799 ± 0.063 % 94.607 ± 0.093 %
66
+ 59 7.2925 ± 0.0795 -0.00751 ± 0.00102 0.01887 ± 0.00033 3.823 ± 0.063 % 94.589 ± 0.092 %
67
+ 60 7.2747 ± 0.0786 -0.00759 ± 0.00101 0.01876 ± 0.00033 3.810 ± 0.062 % 94.620 ± 0.091 %
68
+ 61 7.3427 ± 0.0790 -0.00914 ± 0.00102 0.01935 ± 0.00035 3.872 ± 0.063 % 94.556 ± 0.091 %
69
+ 62 7.3670 ± 0.0786 -0.00886 ± 0.00101 0.01926 ± 0.00034 3.854 ± 0.063 % 94.568 ± 0.090 %
70
+ 63 7.4126 ± 0.0786 -0.00873 ± 0.00100 0.01915 ± 0.00034 3.838 ± 0.062 % 94.569 ± 0.089 %
71
+ 64 7.4433 ± 0.0783 -0.00880 ± 0.00099 0.01902 ± 0.00033 3.818 ± 0.061 % 94.601 ± 0.088 %
72
+
73
+ ====== Perplexity statistics ======
74
+ Mean PPL(Q) : 7.443251 ± 0.078307
75
+ Mean PPL(base) : 7.509043 ± 0.079361
76
+ Cor(ln(PPL(Q)), ln(PPL(base))): 99.56%
77
+ Mean ln(PPL(Q)/PPL(base)) : -0.008800 ± 0.000988
78
+ Mean PPL(Q)/PPL(base) : 0.991238 ± 0.000980
79
+ Mean PPL(Q)-PPL(base) : -0.065792 ± 0.007455
80
+
81
+ ====== KL divergence statistics ======
82
+ Mean KLD: 0.019018 ± 0.000333
83
+ Maximum KLD: 8.796923
84
+ 99.9% KLD: 0.874047
85
+ 99.0% KLD: 0.187952
86
+ 95.0% KLD: 0.067044
87
+ 90.0% KLD: 0.039804
88
+ Median KLD: 0.006015
89
+ 10.0% KLD: 0.000109
90
+ 5.0% KLD: 0.000028
91
+ 1.0% KLD: -0.000005
92
+ 0.1% KLD: -0.000143
93
+ Minimum KLD: -0.000665
94
+
95
+ ====== Token probability statistics ======
96
+ Mean Δp: -0.035 ± 0.015 %
97
+ Maximum Δp: 88.388%
98
+ 99.9% Δp: 30.261%
99
+ 99.0% Δp: 11.186%
100
+ 95.0% Δp: 4.432%
101
+ 90.0% Δp: 2.353%
102
+ 75.0% Δp: 0.406%
103
+ Median Δp: -0.001%
104
+ 25.0% Δp: -0.495%
105
+ 10.0% Δp: -2.563%
106
+ 5.0% Δp: -4.613%
107
+ 1.0% Δp: -11.230%
108
+ 0.1% Δp: -28.803%
109
+ Minimum Δp: -95.338%
110
+ RMS Δp : 3.818 ± 0.061 %
111
+ Same top p: 94.601 ± 0.088 %
112
+
appendix/measurements/kld_DENSE_attention.txt ADDED
@@ -0,0 +1,112 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.09.661.082 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --no-mmap for better performance
2
+ 0.16.704.000 I
3
+ 0.16.704.120 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
4
+ 0.17.082.046 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
5
+ 0.20.971.352 I kl_divergence: 3.89 seconds per pass - ETA 4.13 minutes
6
+
7
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
8
+ 1 5.8468 ± 0.4599 -0.00183 ± 0.00730 0.01746 ± 0.00155 3.770 ± 0.305 % 95.601 ± 0.641 %
9
+ 2 7.2675 ± 0.4233 -0.00566 ± 0.00523 0.01668 ± 0.00093 3.546 ± 0.219 % 95.161 ± 0.475 %
10
+ 3 8.2695 ± 0.4095 0.00857 ± 0.00447 0.01827 ± 0.00187 3.716 ± 0.356 % 94.754 ± 0.403 %
11
+ 4 9.3899 ± 0.4303 0.00735 ± 0.00423 0.02218 ± 0.00197 4.221 ± 0.354 % 94.550 ± 0.355 %
12
+ 5 8.7220 ± 0.3510 0.00111 ± 0.00367 0.02109 ± 0.00159 4.053 ± 0.299 % 94.643 ± 0.315 %
13
+ 6 7.3765 ± 0.2617 -0.00381 ± 0.00329 0.02088 ± 0.00138 4.204 ± 0.266 % 94.575 ± 0.289 %
14
+ 7 6.7876 ± 0.2197 -0.00145 ± 0.00316 0.02140 ± 0.00122 4.319 ± 0.230 % 94.693 ± 0.265 %
15
+ 8 6.5882 ± 0.1966 -0.00176 ± 0.00287 0.02038 ± 0.00108 4.194 ± 0.208 % 94.807 ± 0.245 %
16
+ 9 6.9595 ± 0.1972 -0.00527 ± 0.00272 0.02070 ± 0.00097 4.162 ± 0.190 % 94.493 ± 0.238 %
17
+ 10 7.0222 ± 0.1889 -0.00542 ± 0.00254 0.01982 ± 0.00088 4.058 ± 0.176 % 94.575 ± 0.224 %
18
+ 11 7.0325 ± 0.1793 -0.00448 ± 0.00246 0.01911 ± 0.00081 3.975 ± 0.165 % 94.704 ± 0.211 %
19
+ 12 7.2911 ± 0.1791 -0.00462 ± 0.00231 0.01883 ± 0.00078 3.954 ± 0.163 % 94.762 ± 0.201 %
20
+ 13 7.4202 ± 0.1760 -0.00235 ± 0.00224 0.01891 ± 0.00073 3.958 ± 0.152 % 94.759 ± 0.193 %
21
+ 14 7.4691 ± 0.1704 -0.00213 ± 0.00212 0.01833 ± 0.00068 3.883 ± 0.144 % 94.777 ± 0.186 %
22
+ 15 7.5246 ± 0.1658 -0.00252 ± 0.00205 0.01816 ± 0.00064 3.852 ± 0.137 % 94.858 ± 0.178 %
23
+ 16 7.7089 ± 0.1646 -0.00306 ± 0.00197 0.01799 ± 0.00060 3.804 ± 0.130 % 94.831 ± 0.173 %
24
+ 17 7.7179 ± 0.1590 -0.00285 ± 0.00188 0.01757 ± 0.00057 3.738 ± 0.125 % 94.871 ± 0.167 %
25
+ 18 7.7964 ± 0.1558 -0.00296 ± 0.00181 0.01740 ± 0.00054 3.724 ± 0.120 % 94.922 ± 0.162 %
26
+ 19 7.7029 ± 0.1501 -0.00562 ± 0.00179 0.01797 ± 0.00053 3.745 ± 0.115 % 94.804 ± 0.159 %
27
+ 20 7.4532 ± 0.1411 -0.00618 ± 0.00179 0.01920 ± 0.00060 3.989 ± 0.124 % 94.751 ± 0.156 %
28
+ 21 7.4941 ± 0.1383 -0.00615 ± 0.00174 0.01933 ± 0.00058 4.017 ± 0.124 % 94.703 ± 0.153 %
29
+ 22 7.5594 ± 0.1367 -0.00750 ± 0.00170 0.01925 ± 0.00056 4.004 ± 0.120 % 94.695 ± 0.149 %
30
+ 23 7.5893 ± 0.1340 -0.00706 ± 0.00165 0.01911 ± 0.00053 3.970 ± 0.116 % 94.683 ± 0.146 %
31
+ 24 7.6861 ± 0.1332 -0.00797 ± 0.00162 0.01921 ± 0.00052 3.976 ± 0.112 % 94.681 ± 0.143 %
32
+ 25 7.7441 ± 0.1318 -0.00888 ± 0.00159 0.01925 ± 0.00050 3.968 ± 0.109 % 94.608 ± 0.141 %
33
+ 26 7.6982 ± 0.1282 -0.00819 ± 0.00155 0.01906 ± 0.00048 3.953 ± 0.107 % 94.631 ± 0.138 %
34
+ 27 7.8503 ± 0.1286 -0.00831 ± 0.00151 0.01874 ± 0.00046 3.907 ± 0.104 % 94.642 ± 0.136 %
35
+ 28 7.9227 ± 0.1276 -0.00887 ± 0.00147 0.01867 ± 0.00045 3.894 ± 0.102 % 94.634 ± 0.133 %
36
+ 29 7.9341 ± 0.1258 -0.00700 ± 0.00146 0.01876 ± 0.00044 3.915 ± 0.099 % 94.607 ± 0.131 %
37
+ 30 7.9724 ± 0.1248 -0.00823 ± 0.00145 0.01884 ± 0.00042 3.913 ± 0.096 % 94.591 ± 0.129 %
38
+ 31 7.8628 ± 0.1207 -0.00817 ± 0.00141 0.01861 ± 0.00041 3.899 ± 0.094 % 94.636 ± 0.127 %
39
+ 32 7.7284 ± 0.1166 -0.00779 ± 0.00139 0.01844 ± 0.00040 3.903 ± 0.092 % 94.688 ± 0.124 %
40
+ 33 7.7142 ± 0.1148 -0.00753 ± 0.00137 0.01854 ± 0.00042 3.899 ± 0.090 % 94.656 ± 0.122 %
41
+ 34 7.6892 ± 0.1125 -0.00768 ± 0.00134 0.01826 ± 0.00041 3.872 ± 0.088 % 94.690 ± 0.120 %
42
+ 35 7.6879 ± 0.1109 -0.00764 ± 0.00131 0.01810 ± 0.00040 3.853 ± 0.087 % 94.685 ± 0.119 %
43
+ 36 7.6913 ± 0.1094 -0.00788 ± 0.00129 0.01799 ± 0.00039 3.837 ± 0.085 % 94.719 ± 0.117 %
44
+ 37 7.5470 ± 0.1053 -0.00765 ± 0.00126 0.01771 ± 0.00038 3.814 ± 0.083 % 94.780 ± 0.114 %
45
+ 38 7.4355 ± 0.1019 -0.00759 ± 0.00123 0.01759 ± 0.00037 3.808 ± 0.081 % 94.817 ± 0.112 %
46
+ 39 7.3197 ± 0.0986 -0.00739 ± 0.00121 0.01736 ± 0.00036 3.786 ± 0.080 % 94.839 ± 0.111 %
47
+ 40 7.1936 ± 0.0952 -0.00700 ± 0.00119 0.01712 ± 0.00035 3.769 ± 0.078 % 94.868 ± 0.109 %
48
+ 41 7.1152 ± 0.0926 -0.00658 ± 0.00116 0.01692 ± 0.00034 3.742 ± 0.077 % 94.874 ± 0.108 %
49
+ 42 7.1467 ± 0.0920 -0.00630 ± 0.00115 0.01686 ± 0.00034 3.728 ± 0.076 % 94.861 ± 0.107 %
50
+ 43 7.3417 ± 0.0943 -0.00686 ± 0.00114 0.01706 ± 0.00033 3.727 ± 0.075 % 94.803 ± 0.106 %
51
+ 44 7.4763 ± 0.0954 -0.00743 ± 0.00113 0.01717 ± 0.00034 3.716 ± 0.074 % 94.784 ± 0.105 %
52
+ 45 7.4465 ± 0.0937 -0.00745 ± 0.00112 0.01741 ± 0.00033 3.742 ± 0.072 % 94.745 ± 0.104 %
53
+ 46 7.4502 ± 0.0927 -0.00757 ± 0.00111 0.01740 ± 0.00033 3.741 ± 0.071 % 94.760 ± 0.103 %
54
+ 47 7.5146 ± 0.0927 -0.00777 ± 0.00110 0.01750 ± 0.00033 3.745 ± 0.070 % 94.748 ± 0.102 %
55
+ 48 7.5117 ± 0.0915 -0.00757 ± 0.00108 0.01744 ± 0.00032 3.734 ± 0.069 % 94.734 ± 0.101 %
56
+ 49 7.5595 ± 0.0912 -0.00780 ± 0.00108 0.01764 ± 0.00034 3.753 ± 0.069 % 94.691 ± 0.100 %
57
+ 50 7.5162 ± 0.0896 -0.00730 ± 0.00107 0.01760 ± 0.00033 3.749 ± 0.068 % 94.714 ± 0.099 %
58
+ 51 7.5484 ± 0.0891 -0.00725 ± 0.00105 0.01743 ± 0.00033 3.726 ± 0.067 % 94.746 ± 0.098 %
59
+ 52 7.4851 ± 0.0874 -0.00726 ± 0.00104 0.01733 ± 0.00032 3.720 ± 0.066 % 94.765 ± 0.097 %
60
+ 53 7.4573 ± 0.0861 -0.00658 ± 0.00103 0.01727 ± 0.00032 3.711 ± 0.066 % 94.777 ± 0.096 %
61
+ 54 7.4161 ± 0.0846 -0.00654 ± 0.00102 0.01717 ± 0.00031 3.701 ± 0.065 % 94.779 ± 0.095 %
62
+ 55 7.3791 ± 0.0834 -0.00651 ± 0.00101 0.01709 ± 0.00031 3.699 ± 0.064 % 94.785 ± 0.094 %
63
+ 56 7.3702 ± 0.0826 -0.00684 ± 0.00100 0.01709 ± 0.00030 3.698 ± 0.063 % 94.779 ± 0.093 %
64
+ 57 7.2909 ± 0.0807 -0.00692 ± 0.00099 0.01699 ± 0.00030 3.688 ± 0.062 % 94.802 ± 0.092 %
65
+ 58 7.3278 ± 0.0807 -0.00667 ± 0.00099 0.01744 ± 0.00030 3.729 ± 0.062 % 94.740 ± 0.092 %
66
+ 59 7.2970 ± 0.0795 -0.00690 ± 0.00098 0.01756 ± 0.00030 3.747 ± 0.061 % 94.726 ± 0.091 %
67
+ 60 7.2801 ± 0.0787 -0.00685 ± 0.00097 0.01745 ± 0.00029 3.736 ± 0.060 % 94.761 ± 0.090 %
68
+ 61 7.3588 ± 0.0793 -0.00695 ± 0.00098 0.01783 ± 0.00031 3.785 ± 0.063 % 94.715 ± 0.090 %
69
+ 62 7.3824 ± 0.0789 -0.00676 ± 0.00097 0.01773 ± 0.00030 3.768 ± 0.062 % 94.712 ± 0.089 %
70
+ 63 7.4270 ± 0.0788 -0.00679 ± 0.00095 0.01762 ± 0.00030 3.751 ± 0.061 % 94.718 ± 0.088 %
71
+ 64 7.4586 ± 0.0786 -0.00674 ± 0.00094 0.01751 ± 0.00029 3.733 ± 0.061 % 94.743 ± 0.087 %
72
+
73
+ ====== Perplexity statistics ======
74
+ Mean PPL(Q) : 7.458614 ± 0.078559
75
+ Mean PPL(base) : 7.509043 ± 0.079361
76
+ Cor(ln(PPL(Q)), ln(PPL(base))): 99.60%
77
+ Mean ln(PPL(Q)/PPL(base)) : -0.006738 ± 0.000944
78
+ Mean PPL(Q)/PPL(base) : 0.993284 ± 0.000938
79
+ Mean PPL(Q)-PPL(base) : -0.050429 ± 0.007104
80
+
81
+ ====== KL divergence statistics ======
82
+ Mean KLD: 0.017510 ± 0.000293
83
+ Maximum KLD: 5.325312
84
+ 99.9% KLD: 0.723324
85
+ 99.0% KLD: 0.182327
86
+ 95.0% KLD: 0.061133
87
+ 90.0% KLD: 0.036994
88
+ Median KLD: 0.005638
89
+ 10.0% KLD: 0.000097
90
+ 5.0% KLD: 0.000024
91
+ 1.0% KLD: -0.000007
92
+ 0.1% KLD: -0.000136
93
+ Minimum KLD: -0.000574
94
+
95
+ ====== Token probability statistics ======
96
+ Mean Δp: -0.047 ± 0.015 %
97
+ Maximum Δp: 85.629%
98
+ 99.9% Δp: 30.170%
99
+ 99.0% Δp: 10.466%
100
+ 95.0% Δp: 4.281%
101
+ 90.0% Δp: 2.264%
102
+ 75.0% Δp: 0.386%
103
+ Median Δp: -0.000%
104
+ 25.0% Δp: -0.483%
105
+ 10.0% Δp: -2.518%
106
+ 5.0% Δp: -4.479%
107
+ 1.0% Δp: -10.870%
108
+ 0.1% Δp: -28.025%
109
+ Minimum Δp: -83.566%
110
+ RMS Δp : 3.733 ± 0.061 %
111
+ Same top p: 94.743 ± 0.087 %
112
+
appendix/measurements/kld_DENSE_output.txt ADDED
@@ -0,0 +1,112 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.03.235.962 I cmn init: llama threadpool init, n_threads = 30
2
+ 0.03.343.137 I
3
+ 0.03.343.344 I system_info: n_threads = 30 (n_threads_batch = 30) / 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
4
+ 0.03.695.165 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
5
+ 0.20.480.218 I kl_divergence: 16.78 seconds per pass - ETA 17.90 minutes
6
+
7
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
8
+ 1 5.8644 ± 0.4646 0.00118 ± 0.00729 0.01804 ± 0.00192 4.051 ± 0.471 % 95.210 ± 0.668 %
9
+ 2 7.3617 ± 0.4320 0.00723 ± 0.00517 0.01692 ± 0.00108 3.491 ± 0.282 % 95.015 ± 0.481 %
10
+ 3 8.2676 ± 0.4101 0.00834 ± 0.00426 0.01826 ± 0.00213 3.636 ± 0.395 % 94.656 ± 0.406 %
11
+ 4 9.2105 ± 0.4182 -0.01194 ± 0.00431 0.02473 ± 0.00247 4.421 ± 0.389 % 94.282 ± 0.363 %
12
+ 5 8.5944 ± 0.3434 -0.01362 ± 0.00373 0.02363 ± 0.00199 4.270 ± 0.326 % 94.428 ± 0.321 %
13
+ 6 7.3091 ± 0.2579 -0.01299 ± 0.00345 0.02345 ± 0.00171 4.508 ± 0.294 % 94.330 ± 0.295 %
14
+ 7 6.7285 ± 0.2165 -0.01020 ± 0.00331 0.02393 ± 0.00152 4.659 ± 0.254 % 94.372 ± 0.272 %
15
+ 8 6.5368 ± 0.1940 -0.00960 ± 0.00299 0.02264 ± 0.00134 4.490 ± 0.231 % 94.453 ± 0.253 %
16
+ 9 6.9172 ± 0.1951 -0.01136 ± 0.00283 0.02292 ± 0.00120 4.391 ± 0.211 % 94.102 ± 0.246 %
17
+ 10 6.9858 ± 0.1872 -0.01063 ± 0.00263 0.02194 ± 0.00109 4.262 ± 0.196 % 94.203 ± 0.231 %
18
+ 11 6.9922 ± 0.1775 -0.01023 ± 0.00255 0.02123 ± 0.00099 4.188 ± 0.183 % 94.330 ± 0.218 %
19
+ 12 7.2603 ± 0.1777 -0.00886 ± 0.00240 0.02062 ± 0.00092 4.109 ± 0.172 % 94.420 ± 0.207 %
20
+ 13 7.3928 ± 0.1748 -0.00605 ± 0.00232 0.02080 ± 0.00085 4.121 ± 0.160 % 94.428 ± 0.199 %
21
+ 14 7.4386 ± 0.1691 -0.00621 ± 0.00221 0.02019 ± 0.00079 4.046 ± 0.152 % 94.491 ± 0.191 %
22
+ 15 7.4926 ± 0.1646 -0.00677 ± 0.00213 0.02018 ± 0.00075 4.039 ± 0.143 % 94.467 ± 0.185 %
23
+ 16 7.6708 ± 0.1632 -0.00801 ± 0.00206 0.02004 ± 0.00070 3.986 ± 0.136 % 94.465 ± 0.179 %
24
+ 17 7.6764 ± 0.1575 -0.00823 ± 0.00199 0.01977 ± 0.00068 3.921 ± 0.131 % 94.509 ± 0.173 %
25
+ 18 7.7535 ± 0.1544 -0.00847 ± 0.00191 0.01950 ± 0.00064 3.890 ± 0.125 % 94.586 ± 0.167 %
26
+ 19 7.6659 ± 0.1488 -0.01043 ± 0.00186 0.01972 ± 0.00062 3.888 ± 0.119 % 94.505 ± 0.163 %
27
+ 20 7.4224 ± 0.1400 -0.01032 ± 0.00187 0.02115 ± 0.00067 4.180 ± 0.128 % 94.379 ± 0.161 %
28
+ 21 7.4654 ± 0.1374 -0.00998 ± 0.00182 0.02122 ± 0.00065 4.197 ± 0.126 % 94.396 ± 0.157 %
29
+ 22 7.5372 ± 0.1360 -0.01044 ± 0.00177 0.02108 ± 0.00062 4.192 ± 0.122 % 94.415 ± 0.153 %
30
+ 23 7.5674 ± 0.1333 -0.00995 ± 0.00172 0.02081 ± 0.00060 4.143 ± 0.119 % 94.445 ± 0.149 %
31
+ 24 7.6559 ± 0.1324 -0.01191 ± 0.00170 0.02118 ± 0.00059 4.161 ± 0.115 % 94.440 ± 0.146 %
32
+ 25 7.7195 ± 0.1311 -0.01207 ± 0.00166 0.02096 ± 0.00056 4.124 ± 0.112 % 94.405 ± 0.144 %
33
+ 26 7.6739 ± 0.1275 -0.01135 ± 0.00161 0.02071 ± 0.00054 4.090 ± 0.109 % 94.413 ± 0.141 %
34
+ 27 7.8272 ± 0.1280 -0.01126 ± 0.00156 0.02029 ± 0.00052 4.038 ± 0.107 % 94.403 ± 0.138 %
35
+ 28 7.9079 ± 0.1272 -0.01074 ± 0.00152 0.02000 ± 0.00051 4.003 ± 0.104 % 94.446 ± 0.135 %
36
+ 29 7.9125 ± 0.1253 -0.00972 ± 0.00150 0.01992 ± 0.00049 3.990 ± 0.101 % 94.492 ± 0.132 %
37
+ 30 7.9481 ± 0.1242 -0.01128 ± 0.00149 0.02009 ± 0.00048 3.993 ± 0.098 % 94.451 ± 0.131 %
38
+ 31 7.8366 ± 0.1201 -0.01151 ± 0.00146 0.01987 ± 0.00047 3.972 ± 0.095 % 94.479 ± 0.128 %
39
+ 32 7.7026 ± 0.1160 -0.01114 ± 0.00143 0.01971 ± 0.00046 3.953 ± 0.093 % 94.498 ± 0.126 %
40
+ 33 7.6880 ± 0.1142 -0.01092 ± 0.00141 0.01990 ± 0.00051 3.939 ± 0.091 % 94.496 ± 0.124 %
41
+ 34 7.6620 ± 0.1119 -0.01122 ± 0.00138 0.01960 ± 0.00050 3.914 ± 0.089 % 94.520 ± 0.122 %
42
+ 35 7.6608 ± 0.1102 -0.01117 ± 0.00135 0.01949 ± 0.00049 3.902 ± 0.087 % 94.537 ± 0.120 %
43
+ 36 7.6626 ± 0.1087 -0.01161 ± 0.00133 0.01945 ± 0.00048 3.903 ± 0.086 % 94.572 ± 0.118 %
44
+ 37 7.5200 ± 0.1048 -0.01123 ± 0.00130 0.01914 ± 0.00047 3.877 ± 0.084 % 94.634 ± 0.116 %
45
+ 38 7.4087 ± 0.1013 -0.01119 ± 0.00128 0.01896 ± 0.00045 3.869 ± 0.082 % 94.680 ± 0.114 %
46
+ 39 7.2951 ± 0.0981 -0.01076 ± 0.00125 0.01871 ± 0.00044 3.850 ± 0.081 % 94.699 ± 0.112 %
47
+ 40 7.1678 ± 0.0947 -0.01058 ± 0.00123 0.01843 ± 0.00043 3.831 ± 0.079 % 94.731 ± 0.110 %
48
+ 41 7.0895 ± 0.0921 -0.01019 ± 0.00121 0.01828 ± 0.00042 3.818 ± 0.078 % 94.736 ± 0.109 %
49
+ 42 7.1220 ± 0.0915 -0.00977 ± 0.00119 0.01828 ± 0.00042 3.813 ± 0.077 % 94.703 ± 0.108 %
50
+ 43 7.3177 ± 0.0938 -0.01015 ± 0.00118 0.01842 ± 0.00041 3.798 ± 0.076 % 94.658 ± 0.107 %
51
+ 44 7.4545 ± 0.0950 -0.01035 ± 0.00117 0.01843 ± 0.00040 3.786 ± 0.074 % 94.628 ± 0.106 %
52
+ 45 7.4313 ± 0.0934 -0.00950 ± 0.00117 0.01889 ± 0.00040 3.823 ± 0.073 % 94.565 ± 0.106 %
53
+ 46 7.4358 ± 0.0924 -0.00951 ± 0.00116 0.01889 ± 0.00039 3.822 ± 0.072 % 94.585 ± 0.104 %
54
+ 47 7.5003 ± 0.0924 -0.00968 ± 0.00115 0.01892 ± 0.00038 3.816 ± 0.071 % 94.586 ± 0.103 %
55
+ 48 7.4978 ± 0.0912 -0.00943 ± 0.00113 0.01887 ± 0.00038 3.827 ± 0.071 % 94.577 ± 0.102 %
56
+ 49 7.5463 ± 0.0910 -0.00955 ± 0.00112 0.01895 ± 0.00038 3.821 ± 0.070 % 94.564 ± 0.101 %
57
+ 50 7.5040 ± 0.0894 -0.00891 ± 0.00110 0.01883 ± 0.00037 3.802 ± 0.069 % 94.598 ± 0.100 %
58
+ 51 7.5377 ± 0.0889 -0.00866 ± 0.00109 0.01866 ± 0.00036 3.779 ± 0.068 % 94.616 ± 0.099 %
59
+ 52 7.4763 ± 0.0872 -0.00844 ± 0.00108 0.01853 ± 0.00036 3.771 ± 0.067 % 94.627 ± 0.098 %
60
+ 53 7.4476 ± 0.0859 -0.00788 ± 0.00107 0.01842 ± 0.00035 3.758 ± 0.066 % 94.644 ± 0.097 %
61
+ 54 7.4060 ± 0.0845 -0.00790 ± 0.00106 0.01836 ± 0.00035 3.762 ± 0.067 % 94.651 ± 0.096 %
62
+ 55 7.3638 ± 0.0831 -0.00857 ± 0.00105 0.01834 ± 0.00034 3.772 ± 0.066 % 94.654 ± 0.095 %
63
+ 56 7.3588 ± 0.0824 -0.00839 ± 0.00104 0.01836 ± 0.00034 3.770 ± 0.065 % 94.643 ± 0.094 %
64
+ 57 7.2806 ± 0.0805 -0.00833 ± 0.00103 0.01823 ± 0.00033 3.754 ± 0.064 % 94.665 ± 0.093 %
65
+ 58 7.3187 ± 0.0805 -0.00792 ± 0.00103 0.01866 ± 0.00034 3.790 ± 0.064 % 94.603 ± 0.093 %
66
+ 59 7.2917 ± 0.0794 -0.00761 ± 0.00102 0.01879 ± 0.00033 3.813 ± 0.063 % 94.594 ± 0.092 %
67
+ 60 7.2739 ± 0.0786 -0.00771 ± 0.00101 0.01869 ± 0.00033 3.800 ± 0.062 % 94.630 ± 0.091 %
68
+ 61 7.3419 ± 0.0790 -0.00924 ± 0.00102 0.01927 ± 0.00035 3.863 ± 0.064 % 94.568 ± 0.091 %
69
+ 62 7.3662 ± 0.0786 -0.00896 ± 0.00101 0.01919 ± 0.00034 3.844 ± 0.063 % 94.580 ± 0.090 %
70
+ 63 7.4118 ± 0.0785 -0.00883 ± 0.00100 0.01907 ± 0.00034 3.828 ± 0.062 % 94.582 ± 0.089 %
71
+ 64 7.4424 ± 0.0783 -0.00891 ± 0.00099 0.01894 ± 0.00033 3.808 ± 0.061 % 94.608 ± 0.088 %
72
+
73
+ ====== Perplexity statistics ======
74
+ Mean PPL(Q) : 7.442420 ± 0.078298
75
+ Mean PPL(base) : 7.509043 ± 0.079361
76
+ Cor(ln(PPL(Q)), ln(PPL(base))): 99.56%
77
+ Mean ln(PPL(Q)/PPL(base)) : -0.008912 ± 0.000987
78
+ Mean PPL(Q)/PPL(base) : 0.991128 ± 0.000978
79
+ Mean PPL(Q)-PPL(base) : -0.066623 ± 0.007447
80
+
81
+ ====== KL divergence statistics ======
82
+ Mean KLD: 0.018941 ± 0.000333
83
+ Maximum KLD: 8.800378
84
+ 99.9% KLD: 0.876907
85
+ 99.0% KLD: 0.185885
86
+ 95.0% KLD: 0.066776
87
+ 90.0% KLD: 0.039799
88
+ Median KLD: 0.005923
89
+ 10.0% KLD: 0.000108
90
+ 5.0% KLD: 0.000027
91
+ 1.0% KLD: -0.000006
92
+ 0.1% KLD: -0.000140
93
+ Minimum KLD: -0.000650
94
+
95
+ ====== Token probability statistics ======
96
+ Mean Δp: -0.031 ± 0.015 %
97
+ Maximum Δp: 88.434%
98
+ 99.9% Δp: 30.351%
99
+ 99.0% Δp: 11.128%
100
+ 95.0% Δp: 4.379%
101
+ 90.0% Δp: 2.336%
102
+ 75.0% Δp: 0.408%
103
+ Median Δp: -0.001%
104
+ 25.0% Δp: -0.485%
105
+ 10.0% Δp: -2.522%
106
+ 5.0% Δp: -4.595%
107
+ 1.0% Δp: -11.180%
108
+ 0.1% Δp: -28.533%
109
+ Minimum Δp: -95.342%
110
+ RMS Δp : 3.808 ± 0.061 %
111
+ Same top p: 94.608 ± 0.088 %
112
+
appendix/measurements/kld_DENSE_token_embd.txt ADDED
@@ -0,0 +1,112 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.09.672.714 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --no-mmap for better performance
2
+ 0.16.253.085 I
3
+ 0.16.253.234 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
4
+ 0.16.604.766 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
5
+ 0.20.363.884 I kl_divergence: 3.76 seconds per pass - ETA 4.00 minutes
6
+
7
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
8
+ 1 5.8764 ± 0.4641 0.00323 ± 0.00876 0.01848 ± 0.00136 3.540 ± 0.288 % 95.112 ± 0.674 %
9
+ 2 7.3544 ± 0.4303 0.00623 ± 0.00578 0.01767 ± 0.00083 3.247 ± 0.172 % 94.673 ± 0.497 %
10
+ 3 8.2482 ± 0.4081 0.00599 ± 0.00485 0.01922 ± 0.00184 3.652 ± 0.304 % 94.591 ± 0.408 %
11
+ 4 9.3084 ± 0.4253 -0.00137 ± 0.00471 0.02605 ± 0.00257 4.383 ± 0.385 % 94.086 ± 0.369 %
12
+ 5 8.6834 ± 0.3489 -0.00332 ± 0.00401 0.02405 ± 0.00207 4.188 ± 0.326 % 94.213 ± 0.327 %
13
+ 6 7.3818 ± 0.2620 -0.00309 ± 0.00363 0.02376 ± 0.00182 4.447 ± 0.308 % 94.151 ± 0.300 %
14
+ 7 6.7711 ± 0.2188 -0.00390 ± 0.00346 0.02411 ± 0.00160 4.597 ± 0.270 % 94.121 ± 0.278 %
15
+ 8 6.5775 ± 0.1962 -0.00340 ± 0.00312 0.02291 ± 0.00140 4.450 ± 0.245 % 94.159 ± 0.259 %
16
+ 9 6.9610 ± 0.1971 -0.00506 ± 0.00292 0.02298 ± 0.00126 4.351 ± 0.223 % 94.070 ± 0.246 %
17
+ 10 7.0167 ± 0.1886 -0.00621 ± 0.00274 0.02266 ± 0.00114 4.336 ± 0.206 % 94.066 ± 0.234 %
18
+ 11 7.0216 ± 0.1787 -0.00604 ± 0.00260 0.02173 ± 0.00104 4.235 ± 0.194 % 94.224 ± 0.220 %
19
+ 12 7.2814 ± 0.1786 -0.00596 ± 0.00242 0.02092 ± 0.00096 4.130 ± 0.182 % 94.347 ± 0.208 %
20
+ 13 7.4049 ± 0.1754 -0.00441 ± 0.00233 0.02083 ± 0.00089 4.128 ± 0.169 % 94.391 ± 0.200 %
21
+ 14 7.4451 ± 0.1695 -0.00534 ± 0.00221 0.02015 ± 0.00083 4.035 ± 0.161 % 94.463 ± 0.191 %
22
+ 15 7.4964 ± 0.1648 -0.00627 ± 0.00212 0.01995 ± 0.00078 4.027 ± 0.153 % 94.526 ± 0.184 %
23
+ 16 7.6950 ± 0.1642 -0.00486 ± 0.00206 0.01989 ± 0.00074 4.001 ± 0.145 % 94.508 ± 0.178 %
24
+ 17 7.7036 ± 0.1586 -0.00470 ± 0.00196 0.01951 ± 0.00070 3.941 ± 0.139 % 94.543 ± 0.172 %
25
+ 18 7.7764 ± 0.1553 -0.00553 ± 0.00192 0.01940 ± 0.00067 3.935 ± 0.134 % 94.548 ± 0.167 %
26
+ 19 7.6903 ± 0.1498 -0.00726 ± 0.00187 0.01945 ± 0.00064 3.922 ± 0.128 % 94.500 ± 0.164 %
27
+ 20 7.4468 ± 0.1409 -0.00704 ± 0.00189 0.02118 ± 0.00079 4.201 ± 0.139 % 94.443 ± 0.160 %
28
+ 21 7.4848 ± 0.1381 -0.00738 ± 0.00184 0.02128 ± 0.00077 4.227 ± 0.137 % 94.405 ± 0.157 %
29
+ 22 7.5589 ± 0.1367 -0.00756 ± 0.00179 0.02111 ± 0.00073 4.217 ± 0.134 % 94.424 ± 0.153 %
30
+ 23 7.5895 ± 0.1340 -0.00703 ± 0.00173 0.02084 ± 0.00070 4.162 ± 0.130 % 94.458 ± 0.149 %
31
+ 24 7.6824 ± 0.1331 -0.00845 ± 0.00172 0.02107 ± 0.00068 4.165 ± 0.125 % 94.457 ± 0.146 %
32
+ 25 7.7425 ± 0.1318 -0.00908 ± 0.00168 0.02110 ± 0.00065 4.142 ± 0.121 % 94.401 ± 0.144 %
33
+ 26 7.6911 ± 0.1280 -0.00912 ± 0.00164 0.02094 ± 0.00063 4.129 ± 0.118 % 94.417 ± 0.141 %
34
+ 27 7.8435 ± 0.1285 -0.00919 ± 0.00159 0.02057 ± 0.00061 4.076 ± 0.115 % 94.417 ± 0.138 %
35
+ 28 7.9221 ± 0.1276 -0.00895 ± 0.00155 0.02037 ± 0.00059 4.051 ± 0.112 % 94.421 ± 0.136 %
36
+ 29 7.9290 ± 0.1258 -0.00764 ± 0.00153 0.02037 ± 0.00057 4.051 ± 0.109 % 94.425 ± 0.133 %
37
+ 30 7.9699 ± 0.1248 -0.00855 ± 0.00152 0.02059 ± 0.00057 4.056 ± 0.105 % 94.422 ± 0.131 %
38
+ 31 7.8618 ± 0.1208 -0.00830 ± 0.00149 0.02031 ± 0.00055 4.025 ± 0.103 % 94.472 ± 0.128 %
39
+ 32 7.7236 ± 0.1165 -0.00841 ± 0.00145 0.02012 ± 0.00054 4.006 ± 0.100 % 94.523 ± 0.126 %
40
+ 33 7.7092 ± 0.1147 -0.00817 ± 0.00143 0.02031 ± 0.00059 4.014 ± 0.099 % 94.526 ± 0.124 %
41
+ 34 7.6843 ± 0.1124 -0.00832 ± 0.00140 0.02006 ± 0.00057 3.994 ± 0.098 % 94.560 ± 0.122 %
42
+ 35 7.6829 ± 0.1108 -0.00830 ± 0.00137 0.01997 ± 0.00056 3.975 ± 0.096 % 94.548 ± 0.120 %
43
+ 36 7.6945 ± 0.1095 -0.00747 ± 0.00135 0.01981 ± 0.00055 3.954 ± 0.094 % 94.594 ± 0.118 %
44
+ 37 7.5497 ± 0.1054 -0.00729 ± 0.00132 0.01951 ± 0.00054 3.931 ± 0.092 % 94.632 ± 0.116 %
45
+ 38 7.4379 ± 0.1020 -0.00725 ± 0.00129 0.01931 ± 0.00052 3.922 ± 0.090 % 94.670 ± 0.114 %
46
+ 39 7.3224 ± 0.0987 -0.00702 ± 0.00126 0.01908 ± 0.00051 3.902 ± 0.089 % 94.701 ± 0.112 %
47
+ 40 7.1945 ± 0.0953 -0.00687 ± 0.00124 0.01882 ± 0.00050 3.882 ± 0.087 % 94.761 ± 0.110 %
48
+ 41 7.1158 ± 0.0927 -0.00650 ± 0.00121 0.01860 ± 0.00049 3.858 ± 0.085 % 94.786 ± 0.109 %
49
+ 42 7.1464 ± 0.0920 -0.00635 ± 0.00120 0.01854 ± 0.00048 3.841 ± 0.084 % 94.770 ± 0.107 %
50
+ 43 7.3437 ± 0.0944 -0.00659 ± 0.00119 0.01866 ± 0.00047 3.839 ± 0.083 % 94.719 ± 0.107 %
51
+ 44 7.4800 ± 0.0955 -0.00694 ± 0.00118 0.01866 ± 0.00046 3.817 ± 0.081 % 94.715 ± 0.105 %
52
+ 45 7.4544 ± 0.0939 -0.00640 ± 0.00117 0.01913 ± 0.00045 3.865 ± 0.080 % 94.665 ± 0.105 %
53
+ 46 7.4623 ± 0.0929 -0.00595 ± 0.00116 0.01917 ± 0.00045 3.886 ± 0.079 % 94.653 ± 0.104 %
54
+ 47 7.5262 ± 0.0929 -0.00624 ± 0.00115 0.01930 ± 0.00044 3.887 ± 0.078 % 94.649 ± 0.103 %
55
+ 48 7.5232 ± 0.0917 -0.00603 ± 0.00113 0.01920 ± 0.00043 3.873 ± 0.077 % 94.646 ± 0.102 %
56
+ 49 7.5703 ± 0.0915 -0.00637 ± 0.00113 0.01933 ± 0.00044 3.873 ± 0.076 % 94.604 ± 0.101 %
57
+ 50 7.5248 ± 0.0898 -0.00614 ± 0.00111 0.01924 ± 0.00043 3.861 ± 0.075 % 94.630 ± 0.100 %
58
+ 51 7.5554 ± 0.0893 -0.00632 ± 0.00109 0.01912 ± 0.00042 3.845 ± 0.074 % 94.649 ± 0.099 %
59
+ 52 7.4910 ± 0.0875 -0.00648 ± 0.00108 0.01897 ± 0.00041 3.833 ± 0.073 % 94.673 ± 0.097 %
60
+ 53 7.4610 ± 0.0862 -0.00608 ± 0.00107 0.01883 ± 0.00041 3.816 ± 0.072 % 94.685 ± 0.096 %
61
+ 54 7.4193 ± 0.0847 -0.00610 ± 0.00105 0.01877 ± 0.00040 3.822 ± 0.072 % 94.694 ± 0.095 %
62
+ 55 7.3775 ± 0.0834 -0.00671 ± 0.00105 0.01874 ± 0.00040 3.824 ± 0.071 % 94.709 ± 0.094 %
63
+ 56 7.3709 ± 0.0826 -0.00675 ± 0.00104 0.01874 ± 0.00039 3.825 ± 0.070 % 94.706 ± 0.094 %
64
+ 57 7.2918 ± 0.0807 -0.00680 ± 0.00103 0.01865 ± 0.00038 3.817 ± 0.069 % 94.732 ± 0.093 %
65
+ 58 7.3267 ± 0.0806 -0.00682 ± 0.00103 0.01898 ± 0.00038 3.849 ± 0.068 % 94.688 ± 0.092 %
66
+ 59 7.2993 ± 0.0796 -0.00658 ± 0.00103 0.01919 ± 0.00038 3.903 ± 0.069 % 94.673 ± 0.091 %
67
+ 60 7.2804 ± 0.0787 -0.00680 ± 0.00101 0.01907 ± 0.00037 3.895 ± 0.068 % 94.684 ± 0.091 %
68
+ 61 7.3492 ± 0.0791 -0.00826 ± 0.00102 0.01954 ± 0.00039 3.928 ± 0.068 % 94.646 ± 0.090 %
69
+ 62 7.3725 ± 0.0787 -0.00810 ± 0.00101 0.01945 ± 0.00038 3.912 ± 0.067 % 94.649 ± 0.089 %
70
+ 63 7.4168 ± 0.0786 -0.00816 ± 0.00100 0.01933 ± 0.00038 3.897 ± 0.066 % 94.641 ± 0.089 %
71
+ 64 7.4477 ± 0.0784 -0.00820 ± 0.00099 0.01922 ± 0.00037 3.881 ± 0.065 % 94.662 ± 0.088 %
72
+
73
+ ====== Perplexity statistics ======
74
+ Mean PPL(Q) : 7.447739 ± 0.078390
75
+ Mean PPL(base) : 7.509043 ± 0.079361
76
+ Cor(ln(PPL(Q)), ln(PPL(base))): 99.56%
77
+ Mean ln(PPL(Q)/PPL(base)) : -0.008198 ± 0.000986
78
+ Mean PPL(Q)/PPL(base) : 0.991836 ± 0.000978
79
+ Mean PPL(Q)-PPL(base) : -0.061304 ± 0.007430
80
+
81
+ ====== KL divergence statistics ======
82
+ Mean KLD: 0.019217 ± 0.000370
83
+ Maximum KLD: 9.123656
84
+ 99.9% KLD: 0.882531
85
+ 99.0% KLD: 0.187328
86
+ 95.0% KLD: 0.065879
87
+ 90.0% KLD: 0.039818
88
+ Median KLD: 0.006113
89
+ 10.0% KLD: 0.000106
90
+ 5.0% KLD: 0.000028
91
+ 1.0% KLD: -0.000006
92
+ 0.1% KLD: -0.000140
93
+ Minimum KLD: -0.000965
94
+
95
+ ====== Token probability statistics ======
96
+ Mean Δp: -0.022 ± 0.015 %
97
+ Maximum Δp: 96.119%
98
+ 99.9% Δp: 30.920%
99
+ 99.0% Δp: 11.422%
100
+ 95.0% Δp: 4.564%
101
+ 90.0% Δp: 2.436%
102
+ 75.0% Δp: 0.407%
103
+ Median Δp: -0.001%
104
+ 25.0% Δp: -0.492%
105
+ 10.0% Δp: -2.569%
106
+ 5.0% Δp: -4.624%
107
+ 1.0% Δp: -11.075%
108
+ 0.1% Δp: -28.904%
109
+ Minimum Δp: -95.551%
110
+ RMS Δp : 3.881 ± 0.065 %
111
+ Same top p: 94.662 ± 0.088 %
112
+
appendix/measurements/kld_E.txt ADDED
@@ -0,0 +1,112 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.17.339.347 I cmn init: llama threadpool init, n_threads = 30
2
+ 0.17.459.928 I
3
+ 0.17.460.135 I system_info: n_threads = 30 (n_threads_batch = 30) / 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
4
+ 0.17.788.451 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
5
+ 0.36.252.973 I kl_divergence: 18.46 seconds per pass - ETA 19.68 minutes
6
+
7
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
8
+ 1 5.8952 ± 0.4664 0.00641 ± 0.01112 0.03817 ± 0.00334 5.677 ± 0.494 % 93.353 ± 0.779 %
9
+ 2 7.2979 ± 0.4250 -0.00149 ± 0.00814 0.03742 ± 0.00215 5.209 ± 0.333 % 92.229 ± 0.592 %
10
+ 3 8.2806 ± 0.4095 0.00991 ± 0.00631 0.03540 ± 0.00242 5.043 ± 0.345 % 92.506 ± 0.475 %
11
+ 4 9.3546 ± 0.4264 0.00358 ± 0.00583 0.04031 ± 0.00275 5.381 ± 0.298 % 92.424 ± 0.414 %
12
+ 5 8.7475 ± 0.3511 0.00403 ± 0.00513 0.03877 ± 0.00224 5.276 ± 0.252 % 92.512 ± 0.368 %
13
+ 6 7.4241 ± 0.2633 0.00263 ± 0.00459 0.03778 ± 0.00192 5.374 ± 0.222 % 92.473 ± 0.337 %
14
+ 7 6.7666 ± 0.2182 -0.00456 ± 0.00447 0.03869 ± 0.00175 5.666 ± 0.211 % 92.543 ± 0.310 %
15
+ 8 6.5838 ± 0.1960 -0.00243 ± 0.00404 0.03705 ± 0.00154 5.537 ± 0.193 % 92.608 ± 0.289 %
16
+ 9 6.9705 ± 0.1971 -0.00368 ± 0.00379 0.03755 ± 0.00140 5.526 ± 0.175 % 92.527 ± 0.274 %
17
+ 10 7.0337 ± 0.1888 -0.00379 ± 0.00351 0.03598 ± 0.00127 5.397 ± 0.164 % 92.581 ± 0.259 %
18
+ 11 7.0423 ± 0.1791 -0.00309 ± 0.00333 0.03487 ± 0.00117 5.292 ± 0.153 % 92.775 ± 0.244 %
19
+ 12 7.3222 ± 0.1797 -0.00037 ± 0.00314 0.03401 ± 0.00108 5.198 ± 0.144 % 92.799 ± 0.233 %
20
+ 13 7.4183 ± 0.1754 -0.00260 ± 0.00300 0.03370 ± 0.00101 5.180 ± 0.136 % 92.879 ± 0.223 %
21
+ 14 7.4640 ± 0.1696 -0.00280 ± 0.00284 0.03271 ± 0.00094 5.072 ± 0.129 % 92.934 ± 0.214 %
22
+ 15 7.5079 ± 0.1648 -0.00474 ± 0.00272 0.03264 ± 0.00089 5.069 ± 0.124 % 92.949 ± 0.207 %
23
+ 16 7.6843 ± 0.1634 -0.00625 ± 0.00264 0.03265 ± 0.00085 5.056 ± 0.124 % 92.980 ± 0.200 %
24
+ 17 7.6965 ± 0.1579 -0.00562 ± 0.00252 0.03189 ± 0.00080 4.976 ± 0.119 % 92.979 ± 0.194 %
25
+ 18 7.7709 ± 0.1546 -0.00624 ± 0.00243 0.03163 ± 0.00077 4.961 ± 0.117 % 92.973 ± 0.188 %
26
+ 19 7.6803 ± 0.1490 -0.00855 ± 0.00237 0.03172 ± 0.00075 4.935 ± 0.112 % 92.936 ± 0.184 %
27
+ 20 7.4365 ± 0.1401 -0.00843 ± 0.00240 0.03477 ± 0.00087 5.330 ± 0.118 % 92.717 ± 0.182 %
28
+ 21 7.4832 ± 0.1377 -0.00760 ± 0.00233 0.03518 ± 0.00085 5.364 ± 0.116 % 92.669 ± 0.178 %
29
+ 22 7.5511 ± 0.1361 -0.00860 ± 0.00227 0.03486 ± 0.00082 5.318 ± 0.112 % 92.717 ± 0.173 %
30
+ 23 7.5848 ± 0.1335 -0.00766 ± 0.00221 0.03452 ± 0.00078 5.265 ± 0.109 % 92.775 ± 0.169 %
31
+ 24 7.6835 ± 0.1329 -0.00830 ± 0.00215 0.03447 ± 0.00076 5.261 ± 0.106 % 92.787 ± 0.165 %
32
+ 25 7.7431 ± 0.1315 -0.00901 ± 0.00211 0.03452 ± 0.00074 5.244 ± 0.105 % 92.704 ± 0.163 %
33
+ 26 7.6940 ± 0.1278 -0.00874 ± 0.00206 0.03429 ± 0.00071 5.224 ± 0.102 % 92.729 ± 0.159 %
34
+ 27 7.8558 ± 0.1285 -0.00762 ± 0.00201 0.03379 ± 0.00069 5.162 ± 0.100 % 92.708 ± 0.156 %
35
+ 28 7.9391 ± 0.1278 -0.00680 ± 0.00196 0.03339 ± 0.00067 5.130 ± 0.098 % 92.711 ± 0.154 %
36
+ 29 7.9418 ± 0.1258 -0.00603 ± 0.00193 0.03344 ± 0.00065 5.149 ± 0.095 % 92.753 ± 0.151 %
37
+ 30 7.9775 ± 0.1247 -0.00759 ± 0.00192 0.03395 ± 0.00067 5.172 ± 0.093 % 92.711 ± 0.148 %
38
+ 31 7.8717 ± 0.1207 -0.00704 ± 0.00188 0.03363 ± 0.00066 5.143 ± 0.091 % 92.747 ± 0.146 %
39
+ 32 7.7376 ± 0.1166 -0.00660 ± 0.00185 0.03364 ± 0.00064 5.182 ± 0.091 % 92.794 ± 0.143 %
40
+ 33 7.7305 ± 0.1150 -0.00541 ± 0.00183 0.03386 ± 0.00068 5.188 ± 0.090 % 92.772 ± 0.141 %
41
+ 34 7.7047 ± 0.1126 -0.00566 ± 0.00179 0.03362 ± 0.00066 5.181 ± 0.089 % 92.775 ± 0.139 %
42
+ 35 7.6973 ± 0.1109 -0.00641 ± 0.00176 0.03343 ± 0.00065 5.158 ± 0.087 % 92.789 ± 0.137 %
43
+ 36 7.7052 ± 0.1095 -0.00607 ± 0.00173 0.03314 ± 0.00063 5.125 ± 0.086 % 92.842 ± 0.134 %
44
+ 37 7.5619 ± 0.1055 -0.00567 ± 0.00169 0.03272 ± 0.00062 5.101 ± 0.085 % 92.898 ± 0.132 %
45
+ 38 7.4490 ± 0.1020 -0.00577 ± 0.00166 0.03252 ± 0.00060 5.097 ± 0.084 % 92.946 ± 0.130 %
46
+ 39 7.3319 ± 0.0987 -0.00573 ± 0.00163 0.03224 ± 0.00059 5.086 ± 0.082 % 92.967 ± 0.128 %
47
+ 40 7.2043 ± 0.0953 -0.00552 ± 0.00160 0.03189 ± 0.00058 5.075 ± 0.081 % 93.011 ± 0.126 %
48
+ 41 7.1250 ± 0.0927 -0.00520 ± 0.00157 0.03163 ± 0.00056 5.066 ± 0.080 % 93.041 ± 0.124 %
49
+ 42 7.1564 ± 0.0921 -0.00495 ± 0.00155 0.03157 ± 0.00055 5.053 ± 0.078 % 93.046 ± 0.123 %
50
+ 43 7.3516 ± 0.0943 -0.00552 ± 0.00154 0.03159 ± 0.00054 5.042 ± 0.077 % 92.996 ± 0.122 %
51
+ 44 7.4859 ± 0.0954 -0.00615 ± 0.00152 0.03155 ± 0.00053 5.014 ± 0.076 % 92.977 ± 0.120 %
52
+ 45 7.4636 ± 0.0939 -0.00517 ± 0.00152 0.03248 ± 0.00053 5.101 ± 0.075 % 92.895 ± 0.120 %
53
+ 46 7.4688 ± 0.0929 -0.00507 ± 0.00150 0.03261 ± 0.00052 5.101 ± 0.074 % 92.879 ± 0.119 %
54
+ 47 7.5332 ± 0.0929 -0.00530 ± 0.00149 0.03276 ± 0.00052 5.111 ± 0.074 % 92.868 ± 0.117 %
55
+ 48 7.5313 ± 0.0917 -0.00496 ± 0.00147 0.03258 ± 0.00051 5.090 ± 0.073 % 92.852 ± 0.116 %
56
+ 49 7.5790 ± 0.0915 -0.00522 ± 0.00145 0.03266 ± 0.00051 5.100 ± 0.072 % 92.792 ± 0.116 %
57
+ 50 7.5357 ± 0.0899 -0.00470 ± 0.00144 0.03253 ± 0.00050 5.084 ± 0.071 % 92.807 ± 0.114 %
58
+ 51 7.5682 ± 0.0894 -0.00462 ± 0.00141 0.03229 ± 0.00049 5.058 ± 0.071 % 92.828 ± 0.113 %
59
+ 52 7.5035 ± 0.0876 -0.00481 ± 0.00140 0.03223 ± 0.00049 5.058 ± 0.070 % 92.845 ± 0.112 %
60
+ 53 7.4745 ± 0.0863 -0.00427 ± 0.00138 0.03201 ± 0.00048 5.047 ± 0.069 % 92.860 ± 0.111 %
61
+ 54 7.4300 ± 0.0848 -0.00466 ± 0.00136 0.03185 ± 0.00048 5.036 ± 0.068 % 92.851 ± 0.110 %
62
+ 55 7.3859 ± 0.0834 -0.00558 ± 0.00135 0.03180 ± 0.00047 5.036 ± 0.067 % 92.843 ± 0.109 %
63
+ 56 7.3764 ± 0.0826 -0.00600 ± 0.00134 0.03177 ± 0.00046 5.033 ± 0.067 % 92.840 ± 0.108 %
64
+ 57 7.2978 ± 0.0807 -0.00598 ± 0.00133 0.03163 ± 0.00046 5.019 ± 0.066 % 92.861 ± 0.107 %
65
+ 58 7.3384 ± 0.0807 -0.00522 ± 0.00133 0.03221 ± 0.00047 5.055 ± 0.065 % 92.824 ± 0.106 %
66
+ 59 7.3125 ± 0.0797 -0.00476 ± 0.00132 0.03244 ± 0.00046 5.077 ± 0.065 % 92.831 ± 0.105 %
67
+ 60 7.2948 ± 0.0788 -0.00484 ± 0.00131 0.03228 ± 0.00046 5.064 ± 0.064 % 92.846 ± 0.104 %
68
+ 61 7.3668 ± 0.0793 -0.00585 ± 0.00131 0.03291 ± 0.00048 5.118 ± 0.065 % 92.800 ± 0.103 %
69
+ 62 7.3895 ± 0.0789 -0.00581 ± 0.00130 0.03277 ± 0.00047 5.095 ± 0.065 % 92.793 ± 0.103 %
70
+ 63 7.4348 ± 0.0788 -0.00573 ± 0.00128 0.03254 ± 0.00046 5.072 ± 0.064 % 92.801 ± 0.102 %
71
+ 64 7.4663 ± 0.0786 -0.00570 ± 0.00127 0.03241 ± 0.00046 5.053 ± 0.063 % 92.824 ± 0.101 %
72
+
73
+ ====== Perplexity statistics ======
74
+ Mean PPL(Q) : 7.466348 ± 0.078584
75
+ Mean PPL(base) : 7.509043 ± 0.079361
76
+ Cor(ln(PPL(Q)), ln(PPL(base))): 99.28%
77
+ Mean ln(PPL(Q)/PPL(base)) : -0.005702 ± 0.001266
78
+ Mean PPL(Q)/PPL(base) : 0.994314 ± 0.001259
79
+ Mean PPL(Q)-PPL(base) : -0.042695 ± 0.009509
80
+
81
+ ====== KL divergence statistics ======
82
+ Mean KLD: 0.032407 ± 0.000458
83
+ Maximum KLD: 8.724359
84
+ 99.9% KLD: 1.273374
85
+ 99.0% KLD: 0.348172
86
+ 95.0% KLD: 0.114034
87
+ 90.0% KLD: 0.068035
88
+ Median KLD: 0.011113
89
+ 10.0% KLD: 0.000177
90
+ 5.0% KLD: 0.000051
91
+ 1.0% KLD: -0.000001
92
+ 0.1% KLD: -0.000145
93
+ Minimum KLD: -0.000786
94
+
95
+ ====== Token probability statistics ======
96
+ Mean Δp: -0.082 ± 0.020 %
97
+ Maximum Δp: 88.419%
98
+ 99.9% Δp: 41.174%
99
+ 99.0% Δp: 14.988%
100
+ 95.0% Δp: 5.948%
101
+ 90.0% Δp: 3.158%
102
+ 75.0% Δp: 0.522%
103
+ Median Δp: -0.002%
104
+ 25.0% Δp: -0.701%
105
+ 10.0% Δp: -3.491%
106
+ 5.0% Δp: -6.205%
107
+ 1.0% Δp: -15.510%
108
+ 0.1% Δp: -40.231%
109
+ Minimum Δp: -79.903%
110
+ RMS Δp : 5.053 ± 0.063 %
111
+ Same top p: 92.824 ± 0.101 %
112
+
appendix/measurements/kld_FAMEVEN-19G-ICE.txt ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.04.991.941 I
2
+ 0.04.992.079 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
3
+ 0.05.338.579 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
4
+ 0.07.271.162 I kl_divergence: 1.93 seconds per pass - ETA 2.05 minutes
5
+
6
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
7
+ 1 5.7950 ± 0.4493 -0.01073 ± 0.01896 0.09316 ± 0.00866 8.888 ± 0.638 % 89.247 ± 0.969 %
8
+ 2 7.3071 ± 0.4210 -0.00022 ± 0.01192 0.07817 ± 0.00477 7.722 ± 0.437 % 89.345 ± 0.682 %
9
+ 3 8.3139 ± 0.4107 0.01392 ± 0.00936 0.07181 ± 0.00345 7.195 ± 0.345 % 89.997 ± 0.542 %
10
+ 4 9.5243 ± 0.4368 0.02156 ± 0.00830 0.07863 ± 0.00371 7.839 ± 0.347 % 89.663 ± 0.476 %
11
+ 5 8.8988 ± 0.3592 0.02117 ± 0.00730 0.07634 ± 0.00310 7.645 ± 0.299 % 89.580 ± 0.427 %
12
+ 6 7.5311 ± 0.2682 0.01693 ± 0.00655 0.07475 ± 0.00280 7.687 ± 0.269 % 89.589 ± 0.390 %
13
+ 7 6.8595 ± 0.2220 0.00907 ± 0.00618 0.07432 ± 0.00253 7.785 ± 0.244 % 89.946 ± 0.355 %
14
+ 8 6.6461 ± 0.1981 0.00699 ± 0.00567 0.07279 ± 0.00227 7.750 ± 0.226 % 90.017 ± 0.331 %
15
+ 9 7.0477 ± 0.1998 0.00733 ± 0.00532 0.07421 ± 0.00208 7.755 ± 0.209 % 89.378 ± 0.321 %
16
+ 10 7.1171 ± 0.1917 0.00800 ± 0.00495 0.07126 ± 0.00192 7.562 ± 0.195 % 89.560 ± 0.302 %
17
+ 11 7.1094 ± 0.1812 0.00640 ± 0.00463 0.06922 ± 0.00176 7.413 ± 0.183 % 89.674 ± 0.287 %
18
+ 12 7.3597 ± 0.1805 0.00473 ± 0.00433 0.06679 ± 0.00162 7.261 ± 0.173 % 89.777 ± 0.273 %
19
+ 13 7.4888 ± 0.1773 0.00686 ± 0.00418 0.06626 ± 0.00152 7.242 ± 0.164 % 89.841 ± 0.262 %
20
+ 14 7.5315 ± 0.1715 0.00620 ± 0.00398 0.06466 ± 0.00143 7.125 ± 0.156 % 89.953 ± 0.251 %
21
+ 15 7.5688 ± 0.1664 0.00335 ± 0.00383 0.06427 ± 0.00135 7.078 ± 0.150 % 89.958 ± 0.243 %
22
+ 16 7.7331 ± 0.1645 0.00008 ± 0.00371 0.06454 ± 0.00129 7.013 ± 0.144 % 89.938 ± 0.235 %
23
+ 17 7.7479 ± 0.1591 0.00103 ± 0.00355 0.06339 ± 0.00123 6.935 ± 0.138 % 90.012 ± 0.227 %
24
+ 18 7.8233 ± 0.1558 0.00049 ± 0.00344 0.06311 ± 0.00119 6.907 ± 0.135 % 90.002 ± 0.221 %
25
+ 19 7.7307 ± 0.1502 -0.00202 ± 0.00333 0.06264 ± 0.00115 6.860 ± 0.130 % 90.040 ± 0.215 %
26
+ 20 7.5009 ± 0.1415 0.00019 ± 0.00338 0.06877 ± 0.00134 7.400 ± 0.136 % 89.785 ± 0.212 %
27
+ 21 7.5511 ± 0.1391 0.00143 ± 0.00333 0.06968 ± 0.00134 7.452 ± 0.135 % 89.685 ± 0.208 %
28
+ 22 7.6096 ± 0.1373 -0.00088 ± 0.00325 0.06916 ± 0.00128 7.402 ± 0.131 % 89.732 ± 0.202 %
29
+ 23 7.6365 ± 0.1346 -0.00086 ± 0.00316 0.06859 ± 0.00124 7.325 ± 0.127 % 89.740 ± 0.198 %
30
+ 24 7.7175 ± 0.1335 -0.00390 ± 0.00310 0.06897 ± 0.00121 7.344 ± 0.125 % 89.724 ± 0.194 %
31
+ 25 7.7694 ± 0.1319 -0.00562 ± 0.00303 0.06845 ± 0.00117 7.280 ± 0.122 % 89.728 ± 0.190 %
32
+ 26 7.7163 ± 0.1281 -0.00583 ± 0.00296 0.06821 ± 0.00114 7.270 ± 0.119 % 89.781 ± 0.186 %
33
+ 27 7.8726 ± 0.1287 -0.00548 ± 0.00289 0.06742 ± 0.00112 7.220 ± 0.118 % 89.783 ± 0.182 %
34
+ 28 7.9460 ± 0.1277 -0.00594 ± 0.00281 0.06670 ± 0.00108 7.170 ± 0.115 % 89.795 ± 0.179 %
35
+ 29 7.9431 ± 0.1255 -0.00587 ± 0.00278 0.06758 ± 0.00109 7.261 ± 0.114 % 89.736 ± 0.176 %
36
+ 30 7.9517 ± 0.1238 -0.01082 ± 0.00279 0.06873 ± 0.00108 7.310 ± 0.113 % 89.697 ± 0.174 %
37
+ 31 7.8467 ± 0.1199 -0.01022 ± 0.00273 0.06838 ± 0.00106 7.284 ± 0.111 % 89.736 ± 0.170 %
38
+ 32 7.7132 ± 0.1158 -0.00976 ± 0.00269 0.06835 ± 0.00105 7.304 ± 0.109 % 89.834 ± 0.167 %
39
+ 33 7.7086 ± 0.1143 -0.00825 ± 0.00265 0.06863 ± 0.00106 7.301 ± 0.106 % 89.804 ± 0.165 %
40
+ 34 7.6912 ± 0.1121 -0.00742 ± 0.00259 0.06802 ± 0.00104 7.269 ± 0.104 % 89.834 ± 0.162 %
41
+ 35 7.6850 ± 0.1104 -0.00801 ± 0.00255 0.06777 ± 0.00102 7.252 ± 0.102 % 89.865 ± 0.159 %
42
+ 36 7.6919 ± 0.1090 -0.00780 ± 0.00250 0.06723 ± 0.00100 7.238 ± 0.101 % 89.951 ± 0.157 %
43
+ 37 7.5482 ± 0.1050 -0.00749 ± 0.00245 0.06646 ± 0.00097 7.210 ± 0.100 % 90.027 ± 0.154 %
44
+ 38 7.4379 ± 0.1016 -0.00726 ± 0.00240 0.06622 ± 0.00095 7.217 ± 0.099 % 90.022 ± 0.152 %
45
+ 39 7.3246 ± 0.0984 -0.00672 ± 0.00236 0.06590 ± 0.00094 7.204 ± 0.097 % 90.029 ± 0.150 %
46
+ 40 7.1972 ± 0.0949 -0.00649 ± 0.00231 0.06515 ± 0.00092 7.184 ± 0.095 % 90.108 ± 0.148 %
47
+ 41 7.1200 ± 0.0924 -0.00590 ± 0.00228 0.06479 ± 0.00091 7.175 ± 0.095 % 90.115 ± 0.146 %
48
+ 42 7.1476 ± 0.0917 -0.00618 ± 0.00224 0.06458 ± 0.00090 7.147 ± 0.094 % 90.127 ± 0.144 %
49
+ 43 7.3445 ± 0.0940 -0.00648 ± 0.00221 0.06430 ± 0.00088 7.114 ± 0.092 % 90.116 ± 0.142 %
50
+ 44 7.4848 ± 0.0952 -0.00629 ± 0.00218 0.06407 ± 0.00086 7.079 ± 0.091 % 90.094 ± 0.141 %
51
+ 45 7.4618 ± 0.0937 -0.00540 ± 0.00218 0.06589 ± 0.00087 7.173 ± 0.090 % 89.995 ± 0.140 %
52
+ 46 7.4671 ± 0.0926 -0.00530 ± 0.00216 0.06619 ± 0.00086 7.176 ± 0.090 % 89.983 ± 0.138 %
53
+ 47 7.5312 ± 0.0927 -0.00557 ± 0.00213 0.06623 ± 0.00086 7.172 ± 0.089 % 89.959 ± 0.137 %
54
+ 48 7.5263 ± 0.0915 -0.00562 ± 0.00210 0.06603 ± 0.00085 7.172 ± 0.089 % 89.962 ± 0.136 %
55
+ 49 7.5728 ± 0.0912 -0.00604 ± 0.00208 0.06617 ± 0.00085 7.151 ± 0.087 % 89.916 ± 0.134 %
56
+ 50 7.5261 ± 0.0895 -0.00598 ± 0.00205 0.06577 ± 0.00084 7.126 ± 0.086 % 89.973 ± 0.133 %
57
+ 51 7.5554 ± 0.0890 -0.00632 ± 0.00202 0.06542 ± 0.00083 7.101 ± 0.086 % 89.970 ± 0.132 %
58
+ 52 7.4941 ± 0.0873 -0.00605 ± 0.00200 0.06508 ± 0.00082 7.092 ± 0.084 % 89.997 ± 0.130 %
59
+ 53 7.4668 ± 0.0860 -0.00531 ± 0.00197 0.06466 ± 0.00080 7.068 ± 0.083 % 90.002 ± 0.129 %
60
+ 54 7.4267 ± 0.0846 -0.00511 ± 0.00195 0.06416 ± 0.00079 7.036 ± 0.082 % 90.009 ± 0.128 %
61
+ 55 7.3815 ± 0.0832 -0.00618 ± 0.00193 0.06400 ± 0.00078 7.022 ± 0.081 % 90.006 ± 0.126 %
62
+ 56 7.3657 ± 0.0823 -0.00745 ± 0.00192 0.06381 ± 0.00077 7.013 ± 0.080 % 90.007 ± 0.125 %
63
+ 57 7.2855 ± 0.0804 -0.00766 ± 0.00189 0.06345 ± 0.00076 6.988 ± 0.079 % 90.046 ± 0.124 %
64
+ 58 7.3238 ± 0.0804 -0.00722 ± 0.00189 0.06400 ± 0.00075 7.007 ± 0.078 % 90.018 ± 0.123 %
65
+ 59 7.2974 ± 0.0793 -0.00684 ± 0.00187 0.06435 ± 0.00075 7.029 ± 0.077 % 89.986 ± 0.122 %
66
+ 60 7.2841 ± 0.0785 -0.00630 ± 0.00185 0.06411 ± 0.00074 7.023 ± 0.077 % 90.007 ± 0.121 %
67
+ 61 7.3678 ± 0.0792 -0.00572 ± 0.00185 0.06483 ± 0.00076 7.063 ± 0.077 % 89.964 ± 0.120 %
68
+ 62 7.3912 ± 0.0788 -0.00557 ± 0.00183 0.06465 ± 0.00075 7.046 ± 0.076 % 89.955 ± 0.119 %
69
+ 63 7.4318 ± 0.0786 -0.00614 ± 0.00181 0.06434 ± 0.00074 7.025 ± 0.075 % 89.942 ± 0.118 %
70
+ 64 7.4662 ± 0.0784 -0.00572 ± 0.00180 0.06421 ± 0.00073 7.014 ± 0.075 % 89.953 ± 0.117 %
71
+
72
+ ====== Perplexity statistics ======
73
+ Mean PPL(Q) : 7.466247 ± 0.078416
74
+ Mean PPL(base) : 7.509043 ± 0.079361
75
+ Cor(ln(PPL(Q)), ln(PPL(base))): 98.55%
76
+ Mean ln(PPL(Q)/PPL(base)) : -0.005716 ± 0.001798
77
+ Mean PPL(Q)/PPL(base) : 0.994301 ± 0.001787
78
+ Mean PPL(Q)-PPL(base) : -0.042796 ± 0.013484
79
+
80
+ ====== KL divergence statistics ======
81
+ Mean KLD: 0.064213 ± 0.000727
82
+ Maximum KLD: 9.456017
83
+ 99.9% KLD: 2.305098
84
+ 99.0% KLD: 0.692351
85
+ 95.0% KLD: 0.233270
86
+ 90.0% KLD: 0.138435
87
+ Median KLD: 0.022855
88
+ 10.0% KLD: 0.000336
89
+ 5.0% KLD: 0.000092
90
+ 1.0% KLD: 0.000004
91
+ 0.1% KLD: -0.000162
92
+ Minimum KLD: -0.000848
93
+
94
+ ====== Token probability statistics ======
95
+ Mean Δp: -0.228 ± 0.027 %
96
+ Maximum Δp: 95.687%
97
+ 99.9% Δp: 48.455%
98
+ 99.0% Δp: 21.099%
99
+ 95.0% Δp: 8.343%
100
+ 90.0% Δp: 4.527%
101
+ 75.0% Δp: 0.745%
102
+ Median Δp: -0.002%
103
+ 25.0% Δp: -1.012%
104
+ 10.0% Δp: -5.230%
105
+ 5.0% Δp: -9.215%
106
+ 1.0% Δp: -23.212%
107
+ 0.1% Δp: -57.430%
108
+ Minimum Δp: -95.678%
109
+ RMS Δp : 7.014 ± 0.075 %
110
+ Same top p: 89.953 ± 0.117 %
111
+
appendix/measurements/kld_FAMEVEN-21G-ICE.txt ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.05.623.084 I
2
+ 0.05.623.222 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
3
+ 0.05.938.775 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
4
+ 0.07.812.563 I kl_divergence: 1.87 seconds per pass - ETA 1.98 minutes
5
+
6
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
7
+ 1 5.8826 ± 0.4588 0.00428 ± 0.01447 0.06558 ± 0.00467 7.765 ± 0.503 % 91.300 ± 0.882 %
8
+ 2 7.3215 ± 0.4256 0.00175 ± 0.00961 0.05427 ± 0.00273 6.475 ± 0.329 % 90.860 ± 0.637 %
9
+ 3 8.2585 ± 0.4073 0.00724 ± 0.00774 0.05189 ± 0.00310 6.362 ± 0.364 % 90.779 ± 0.522 %
10
+ 4 9.2340 ± 0.4181 -0.00940 ± 0.00689 0.05722 ± 0.00323 6.552 ± 0.314 % 90.494 ± 0.459 %
11
+ 5 8.6545 ± 0.3457 -0.00665 ± 0.00602 0.05473 ± 0.00267 6.382 ± 0.274 % 90.694 ± 0.406 %
12
+ 6 7.3530 ± 0.2593 -0.00700 ± 0.00534 0.05358 ± 0.00233 6.497 ± 0.242 % 90.665 ± 0.371 %
13
+ 7 6.7665 ± 0.2178 -0.00458 ± 0.00498 0.05260 ± 0.00207 6.543 ± 0.218 % 90.923 ± 0.340 %
14
+ 8 6.5844 ± 0.1956 -0.00234 ± 0.00455 0.05111 ± 0.00184 6.438 ± 0.197 % 91.019 ± 0.316 %
15
+ 9 6.9223 ± 0.1948 -0.01063 ± 0.00441 0.05256 ± 0.00168 6.451 ± 0.182 % 90.616 ± 0.304 %
16
+ 10 6.9858 ± 0.1867 -0.01062 ± 0.00409 0.05104 ± 0.00153 6.300 ± 0.169 % 90.616 ± 0.288 %
17
+ 11 6.9964 ± 0.1771 -0.00963 ± 0.00385 0.04921 ± 0.00140 6.167 ± 0.159 % 90.829 ± 0.272 %
18
+ 12 7.2540 ± 0.1770 -0.00974 ± 0.00360 0.04766 ± 0.00131 6.078 ± 0.154 % 90.982 ± 0.259 %
19
+ 13 7.3378 ± 0.1724 -0.01352 ± 0.00347 0.04744 ± 0.00123 6.076 ± 0.147 % 91.067 ± 0.247 %
20
+ 14 7.3871 ± 0.1670 -0.01316 ± 0.00330 0.04603 ± 0.00115 5.961 ± 0.140 % 91.133 ± 0.238 %
21
+ 15 7.4326 ± 0.1622 -0.01482 ± 0.00317 0.04578 ± 0.00109 5.947 ± 0.135 % 91.183 ± 0.229 %
22
+ 16 7.6135 ± 0.1611 -0.01551 ± 0.00306 0.04538 ± 0.00103 5.898 ± 0.129 % 91.239 ± 0.221 %
23
+ 17 7.6265 ± 0.1558 -0.01476 ± 0.00292 0.04452 ± 0.00099 5.791 ± 0.124 % 91.289 ± 0.214 %
24
+ 18 7.7109 ± 0.1529 -0.01399 ± 0.00282 0.04422 ± 0.00095 5.756 ± 0.119 % 91.295 ± 0.208 %
25
+ 19 7.6209 ± 0.1474 -0.01632 ± 0.00275 0.04417 ± 0.00091 5.726 ± 0.115 % 91.331 ± 0.202 %
26
+ 20 7.3663 ± 0.1383 -0.01791 ± 0.00287 0.04977 ± 0.00109 6.289 ± 0.124 % 91.144 ± 0.199 %
27
+ 21 7.3977 ± 0.1355 -0.01909 ± 0.00279 0.04995 ± 0.00106 6.287 ± 0.121 % 91.100 ± 0.194 %
28
+ 22 7.4678 ± 0.1342 -0.01969 ± 0.00271 0.04961 ± 0.00102 6.256 ± 0.118 % 91.167 ± 0.189 %
29
+ 23 7.5020 ± 0.1316 -0.01864 ± 0.00264 0.04905 ± 0.00098 6.186 ± 0.115 % 91.160 ± 0.185 %
30
+ 24 7.5785 ± 0.1304 -0.02207 ± 0.00260 0.04959 ± 0.00096 6.220 ± 0.112 % 91.109 ± 0.182 %
31
+ 25 7.6290 ± 0.1288 -0.02386 ± 0.00255 0.04974 ± 0.00093 6.201 ± 0.109 % 91.042 ± 0.179 %
32
+ 26 7.5697 ± 0.1250 -0.02503 ± 0.00250 0.04975 ± 0.00090 6.210 ± 0.108 % 91.056 ± 0.175 %
33
+ 27 7.7225 ± 0.1255 -0.02473 ± 0.00244 0.04954 ± 0.00088 6.190 ± 0.106 % 90.960 ± 0.173 %
34
+ 28 7.7899 ± 0.1244 -0.02578 ± 0.00239 0.04914 ± 0.00085 6.153 ± 0.103 % 90.920 ± 0.170 %
35
+ 29 7.7814 ± 0.1222 -0.02643 ± 0.00236 0.04928 ± 0.00084 6.187 ± 0.101 % 90.926 ± 0.167 %
36
+ 30 7.8204 ± 0.1213 -0.02748 ± 0.00234 0.04957 ± 0.00083 6.221 ± 0.101 % 90.922 ± 0.164 %
37
+ 31 7.7205 ± 0.1175 -0.02644 ± 0.00229 0.04911 ± 0.00081 6.185 ± 0.099 % 91.004 ± 0.161 %
38
+ 32 7.5976 ± 0.1136 -0.02486 ± 0.00225 0.04884 ± 0.00079 6.206 ± 0.097 % 91.068 ± 0.158 %
39
+ 33 7.5905 ± 0.1121 -0.02369 ± 0.00222 0.04910 ± 0.00082 6.210 ± 0.096 % 91.028 ± 0.156 %
40
+ 34 7.5705 ± 0.1099 -0.02324 ± 0.00217 0.04851 ± 0.00080 6.170 ± 0.094 % 91.067 ± 0.153 %
41
+ 35 7.5703 ± 0.1083 -0.02306 ± 0.00213 0.04818 ± 0.00078 6.149 ± 0.092 % 91.116 ± 0.150 %
42
+ 36 7.5770 ± 0.1069 -0.02285 ± 0.00209 0.04774 ± 0.00076 6.122 ± 0.091 % 91.200 ± 0.148 %
43
+ 37 7.4347 ± 0.1030 -0.02264 ± 0.00204 0.04714 ± 0.00074 6.080 ± 0.089 % 91.250 ± 0.145 %
44
+ 38 7.3263 ± 0.0996 -0.02238 ± 0.00200 0.04688 ± 0.00072 6.069 ± 0.087 % 91.295 ± 0.143 %
45
+ 39 7.2174 ± 0.0965 -0.02147 ± 0.00197 0.04653 ± 0.00072 6.053 ± 0.086 % 91.363 ± 0.141 %
46
+ 40 7.0944 ± 0.0932 -0.02088 ± 0.00193 0.04598 ± 0.00070 6.030 ± 0.085 % 91.415 ± 0.138 %
47
+ 41 7.0202 ± 0.0907 -0.02001 ± 0.00190 0.04559 ± 0.00068 6.015 ± 0.083 % 91.438 ± 0.137 %
48
+ 42 7.0513 ± 0.0901 -0.01975 ± 0.00187 0.04559 ± 0.00067 6.002 ± 0.083 % 91.426 ± 0.135 %
49
+ 43 7.2378 ± 0.0922 -0.02112 ± 0.00185 0.04560 ± 0.00066 5.978 ± 0.081 % 91.350 ± 0.134 %
50
+ 44 7.3764 ± 0.0934 -0.02089 ± 0.00183 0.04556 ± 0.00067 5.951 ± 0.080 % 91.336 ± 0.133 %
51
+ 45 7.3525 ± 0.0918 -0.02016 ± 0.00183 0.04644 ± 0.00066 6.010 ± 0.079 % 91.257 ± 0.132 %
52
+ 46 7.3601 ± 0.0909 -0.01974 ± 0.00181 0.04679 ± 0.00065 6.020 ± 0.078 % 91.241 ± 0.130 %
53
+ 47 7.4224 ± 0.0909 -0.02012 ± 0.00179 0.04695 ± 0.00066 6.021 ± 0.078 % 91.256 ± 0.129 %
54
+ 48 7.4203 ± 0.0897 -0.01981 ± 0.00177 0.04672 ± 0.00064 6.014 ± 0.077 % 91.255 ± 0.127 %
55
+ 49 7.4723 ± 0.0896 -0.01940 ± 0.00175 0.04672 ± 0.00065 6.006 ± 0.076 % 91.240 ± 0.126 %
56
+ 50 7.4303 ± 0.0880 -0.01878 ± 0.00172 0.04649 ± 0.00064 5.991 ± 0.076 % 91.269 ± 0.125 %
57
+ 51 7.4613 ± 0.0875 -0.01886 ± 0.00170 0.04621 ± 0.00063 5.965 ± 0.075 % 91.281 ± 0.124 %
58
+ 52 7.3985 ± 0.0857 -0.01889 ± 0.00168 0.04595 ± 0.00062 5.958 ± 0.074 % 91.296 ± 0.122 %
59
+ 53 7.3695 ± 0.0845 -0.01842 ± 0.00166 0.04569 ± 0.00061 5.933 ± 0.073 % 91.293 ± 0.121 %
60
+ 54 7.3294 ± 0.0830 -0.01830 ± 0.00163 0.04530 ± 0.00060 5.912 ± 0.072 % 91.304 ± 0.120 %
61
+ 55 7.2835 ± 0.0817 -0.01954 ± 0.00163 0.04534 ± 0.00060 5.930 ± 0.073 % 91.309 ± 0.119 %
62
+ 56 7.2720 ± 0.0808 -0.02026 ± 0.00161 0.04529 ± 0.00059 5.935 ± 0.072 % 91.318 ± 0.118 %
63
+ 57 7.1959 ± 0.0790 -0.02003 ± 0.00159 0.04500 ± 0.00058 5.910 ± 0.071 % 91.348 ± 0.116 %
64
+ 58 7.2383 ± 0.0791 -0.01897 ± 0.00159 0.04532 ± 0.00058 5.918 ± 0.071 % 91.335 ± 0.115 %
65
+ 59 7.2091 ± 0.0780 -0.01901 ± 0.00158 0.04571 ± 0.00058 5.931 ± 0.070 % 91.337 ± 0.115 %
66
+ 60 7.1923 ± 0.0772 -0.01899 ± 0.00157 0.04543 ± 0.00058 5.912 ± 0.069 % 91.383 ± 0.113 %
67
+ 61 7.2561 ± 0.0775 -0.02101 ± 0.00158 0.04623 ± 0.00059 5.960 ± 0.069 % 91.315 ± 0.113 %
68
+ 62 7.2786 ± 0.0771 -0.02092 ± 0.00156 0.04609 ± 0.00058 5.940 ± 0.068 % 91.322 ± 0.112 %
69
+ 63 7.3208 ± 0.0770 -0.02119 ± 0.00154 0.04584 ± 0.00057 5.918 ± 0.067 % 91.322 ± 0.111 %
70
+ 64 7.3522 ± 0.0768 -0.02111 ± 0.00153 0.04567 ± 0.00056 5.899 ± 0.067 % 91.346 ± 0.110 %
71
+
72
+ ====== Perplexity statistics ======
73
+ Mean PPL(Q) : 7.352165 ± 0.076781
74
+ Mean PPL(base) : 7.509043 ± 0.079361
75
+ Cor(ln(PPL(Q)), ln(PPL(base))): 98.95%
76
+ Mean ln(PPL(Q)/PPL(base)) : -0.021113 ± 0.001527
77
+ Mean PPL(Q)/PPL(base) : 0.979108 ± 0.001495
78
+ Mean PPL(Q)-PPL(base) : -0.156878 ± 0.011597
79
+
80
+ ====== KL divergence statistics ======
81
+ Mean KLD: 0.045673 ± 0.000564
82
+ Maximum KLD: 8.806860
83
+ 99.9% KLD: 1.696481
84
+ 99.0% KLD: 0.482293
85
+ 95.0% KLD: 0.164943
86
+ 90.0% KLD: 0.097630
87
+ Median KLD: 0.016240
88
+ 10.0% KLD: 0.000249
89
+ 5.0% KLD: 0.000068
90
+ 1.0% KLD: 0.000001
91
+ 0.1% KLD: -0.000142
92
+ Minimum KLD: -0.000835
93
+
94
+ ====== Token probability statistics ======
95
+ Mean Δp: -0.117 ± 0.023 %
96
+ Maximum Δp: 99.093%
97
+ 99.9% Δp: 46.291%
98
+ 99.0% Δp: 18.363%
99
+ 95.0% Δp: 7.055%
100
+ 90.0% Δp: 3.785%
101
+ 75.0% Δp: 0.636%
102
+ Median Δp: -0.002%
103
+ 25.0% Δp: -0.900%
104
+ 10.0% Δp: -4.342%
105
+ 5.0% Δp: -7.493%
106
+ 1.0% Δp: -17.995%
107
+ 0.1% Δp: -46.845%
108
+ Minimum Δp: -84.509%
109
+ RMS Δp : 5.899 ± 0.067 %
110
+ Same top p: 91.346 ± 0.110 %
111
+
appendix/measurements/kld_FAMSHALLOW-19G-ICE.txt ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.04.790.165 I
2
+ 0.04.790.303 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
3
+ 0.05.137.141 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
4
+ 0.07.071.872 I kl_divergence: 1.93 seconds per pass - ETA 2.05 minutes
5
+
6
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
7
+ 1 5.8497 ± 0.4560 -0.00134 ± 0.01779 0.09095 ± 0.00738 8.904 ± 0.523 % 89.541 ± 0.957 %
8
+ 2 7.3600 ± 0.4273 0.00700 ± 0.01204 0.07810 ± 0.00425 7.677 ± 0.348 % 89.345 ± 0.682 %
9
+ 3 8.2810 ± 0.4072 0.00996 ± 0.00921 0.07238 ± 0.00356 7.357 ± 0.300 % 89.769 ± 0.547 %
10
+ 4 9.3159 ± 0.4222 -0.00057 ± 0.00822 0.07728 ± 0.00390 7.590 ± 0.293 % 89.467 ± 0.480 %
11
+ 5 8.7127 ± 0.3476 0.00004 ± 0.00731 0.07790 ± 0.00344 7.688 ± 0.279 % 89.247 ± 0.433 %
12
+ 6 7.3615 ± 0.2590 -0.00584 ± 0.00659 0.07602 ± 0.00299 7.727 ± 0.246 % 89.182 ± 0.396 %
13
+ 7 6.7284 ± 0.2157 -0.01021 ± 0.00611 0.07458 ± 0.00267 7.888 ± 0.234 % 89.582 ± 0.361 %
14
+ 8 6.5395 ± 0.1933 -0.00919 ± 0.00555 0.07186 ± 0.00237 7.697 ± 0.213 % 89.895 ± 0.333 %
15
+ 9 6.8902 ± 0.1932 -0.01527 ± 0.00521 0.07189 ± 0.00214 7.652 ± 0.196 % 89.475 ± 0.320 %
16
+ 10 6.9696 ± 0.1858 -0.01295 ± 0.00479 0.06846 ± 0.00194 7.447 ± 0.184 % 89.668 ± 0.301 %
17
+ 11 6.9870 ± 0.1765 -0.01097 ± 0.00449 0.06643 ± 0.00178 7.305 ± 0.173 % 89.834 ± 0.285 %
18
+ 12 7.2541 ± 0.1768 -0.00971 ± 0.00420 0.06420 ± 0.00164 7.128 ± 0.163 % 89.907 ± 0.272 %
19
+ 13 7.3406 ± 0.1722 -0.01313 ± 0.00403 0.06335 ± 0.00154 7.062 ± 0.156 % 89.969 ± 0.261 %
20
+ 14 7.4061 ± 0.1674 -0.01059 ± 0.00384 0.06193 ± 0.00145 6.980 ± 0.152 % 90.057 ± 0.250 %
21
+ 15 7.4616 ± 0.1630 -0.01092 ± 0.00368 0.06128 ± 0.00137 6.944 ± 0.146 % 90.068 ± 0.241 %
22
+ 16 7.6328 ± 0.1616 -0.01298 ± 0.00356 0.06086 ± 0.00130 6.892 ± 0.141 % 90.048 ± 0.234 %
23
+ 17 7.6512 ± 0.1564 -0.01152 ± 0.00341 0.05989 ± 0.00123 6.803 ± 0.135 % 90.213 ± 0.225 %
24
+ 18 7.7370 ± 0.1535 -0.01060 ± 0.00329 0.05952 ± 0.00118 6.745 ± 0.130 % 90.154 ± 0.220 %
25
+ 19 7.6502 ± 0.1481 -0.01249 ± 0.00319 0.05908 ± 0.00114 6.703 ± 0.125 % 90.194 ± 0.213 %
26
+ 20 7.3755 ± 0.1385 -0.01666 ± 0.00327 0.06529 ± 0.00130 7.302 ± 0.135 % 89.995 ± 0.210 %
27
+ 21 7.4133 ± 0.1357 -0.01698 ± 0.00320 0.06607 ± 0.00128 7.320 ± 0.132 % 89.871 ± 0.206 %
28
+ 22 7.5039 ± 0.1350 -0.01486 ± 0.00313 0.06575 ± 0.00124 7.319 ± 0.129 % 89.887 ± 0.201 %
29
+ 23 7.5433 ± 0.1325 -0.01314 ± 0.00304 0.06521 ± 0.00119 7.235 ± 0.125 % 89.940 ± 0.196 %
30
+ 24 7.6284 ± 0.1316 -0.01551 ± 0.00298 0.06574 ± 0.00116 7.261 ± 0.123 % 89.875 ± 0.193 %
31
+ 25 7.6836 ± 0.1301 -0.01672 ± 0.00292 0.06542 ± 0.00112 7.187 ± 0.120 % 89.826 ± 0.189 %
32
+ 26 7.6362 ± 0.1265 -0.01627 ± 0.00285 0.06505 ± 0.00109 7.155 ± 0.117 % 89.905 ± 0.185 %
33
+ 27 7.7886 ± 0.1270 -0.01621 ± 0.00278 0.06436 ± 0.00105 7.099 ± 0.115 % 89.906 ± 0.181 %
34
+ 28 7.8641 ± 0.1260 -0.01630 ± 0.00272 0.06374 ± 0.00102 7.050 ± 0.112 % 89.911 ± 0.178 %
35
+ 29 7.8719 ± 0.1242 -0.01488 ± 0.00267 0.06382 ± 0.00101 7.081 ± 0.110 % 89.905 ± 0.175 %
36
+ 30 7.9091 ± 0.1232 -0.01620 ± 0.00263 0.06389 ± 0.00098 7.085 ± 0.109 % 89.915 ± 0.172 %
37
+ 31 7.8051 ± 0.1193 -0.01554 ± 0.00258 0.06336 ± 0.00095 7.045 ± 0.107 % 89.976 ± 0.169 %
38
+ 32 7.6851 ± 0.1155 -0.01341 ± 0.00254 0.06357 ± 0.00094 7.139 ± 0.108 % 90.002 ± 0.166 %
39
+ 33 7.6773 ± 0.1139 -0.01232 ± 0.00251 0.06384 ± 0.00096 7.153 ± 0.105 % 89.982 ± 0.163 %
40
+ 34 7.6584 ± 0.1117 -0.01169 ± 0.00245 0.06327 ± 0.00094 7.103 ± 0.103 % 90.029 ± 0.161 %
41
+ 35 7.6583 ± 0.1101 -0.01150 ± 0.00240 0.06282 ± 0.00092 7.069 ± 0.101 % 90.091 ± 0.158 %
42
+ 36 7.6602 ± 0.1086 -0.01193 ± 0.00235 0.06221 ± 0.00089 7.034 ± 0.100 % 90.192 ± 0.155 %
43
+ 37 7.5164 ± 0.1046 -0.01172 ± 0.00230 0.06149 ± 0.00087 7.000 ± 0.098 % 90.238 ± 0.153 %
44
+ 38 7.4070 ± 0.1012 -0.01142 ± 0.00226 0.06143 ± 0.00086 7.012 ± 0.097 % 90.269 ± 0.150 %
45
+ 39 7.2957 ± 0.0980 -0.01068 ± 0.00223 0.06120 ± 0.00086 7.010 ± 0.096 % 90.280 ± 0.148 %
46
+ 40 7.1721 ± 0.0946 -0.00999 ± 0.00219 0.06071 ± 0.00085 7.008 ± 0.095 % 90.335 ± 0.146 %
47
+ 41 7.0954 ± 0.0921 -0.00936 ± 0.00216 0.06030 ± 0.00083 6.999 ± 0.093 % 90.327 ± 0.144 %
48
+ 42 7.1260 ± 0.0914 -0.00922 ± 0.00212 0.06011 ± 0.00081 6.972 ± 0.092 % 90.318 ± 0.143 %
49
+ 43 7.3176 ± 0.0937 -0.01016 ± 0.00209 0.05988 ± 0.00080 6.932 ± 0.091 % 90.270 ± 0.141 %
50
+ 44 7.4581 ± 0.0949 -0.00987 ± 0.00207 0.05981 ± 0.00080 6.908 ± 0.090 % 90.269 ± 0.140 %
51
+ 45 7.4340 ± 0.0933 -0.00914 ± 0.00206 0.06118 ± 0.00079 6.994 ± 0.089 % 90.197 ± 0.139 %
52
+ 46 7.4420 ± 0.0923 -0.00867 ± 0.00204 0.06171 ± 0.00079 7.043 ± 0.089 % 90.182 ± 0.137 %
53
+ 47 7.5077 ± 0.0924 -0.00869 ± 0.00202 0.06190 ± 0.00079 7.044 ± 0.089 % 90.183 ± 0.136 %
54
+ 48 7.5033 ± 0.0912 -0.00869 ± 0.00199 0.06147 ± 0.00078 7.013 ± 0.088 % 90.152 ± 0.134 %
55
+ 49 7.5551 ± 0.0910 -0.00837 ± 0.00197 0.06130 ± 0.00077 6.984 ± 0.087 % 90.151 ± 0.133 %
56
+ 50 7.5133 ± 0.0894 -0.00768 ± 0.00194 0.06091 ± 0.00076 6.960 ± 0.086 % 90.209 ± 0.131 %
57
+ 51 7.5463 ± 0.0889 -0.00752 ± 0.00191 0.06044 ± 0.00074 6.924 ± 0.085 % 90.227 ± 0.130 %
58
+ 52 7.4875 ± 0.0872 -0.00694 ± 0.00190 0.06012 ± 0.00073 6.901 ± 0.084 % 90.272 ± 0.128 %
59
+ 53 7.4579 ± 0.0860 -0.00650 ± 0.00187 0.05977 ± 0.00072 6.879 ± 0.082 % 90.269 ± 0.127 %
60
+ 54 7.4181 ± 0.0845 -0.00626 ± 0.00185 0.05936 ± 0.00071 6.855 ± 0.081 % 90.292 ± 0.126 %
61
+ 55 7.3750 ± 0.0832 -0.00706 ± 0.00183 0.05924 ± 0.00071 6.852 ± 0.081 % 90.287 ± 0.125 %
62
+ 56 7.3711 ± 0.0824 -0.00672 ± 0.00182 0.05911 ± 0.00070 6.845 ± 0.080 % 90.305 ± 0.124 %
63
+ 57 7.2922 ± 0.0806 -0.00674 ± 0.00179 0.05884 ± 0.00069 6.823 ± 0.080 % 90.353 ± 0.122 %
64
+ 58 7.3371 ± 0.0806 -0.00540 ± 0.00179 0.05941 ± 0.00069 6.860 ± 0.079 % 90.334 ± 0.121 %
65
+ 59 7.3107 ± 0.0796 -0.00501 ± 0.00178 0.05982 ± 0.00068 6.898 ± 0.079 % 90.336 ± 0.120 %
66
+ 60 7.2943 ± 0.0788 -0.00490 ± 0.00176 0.05962 ± 0.00068 6.886 ± 0.078 % 90.352 ± 0.119 %
67
+ 61 7.3682 ± 0.0793 -0.00567 ± 0.00176 0.06030 ± 0.00070 6.920 ± 0.078 % 90.307 ± 0.118 %
68
+ 62 7.3930 ± 0.0789 -0.00533 ± 0.00174 0.06003 ± 0.00069 6.894 ± 0.077 % 90.307 ± 0.117 %
69
+ 63 7.4412 ± 0.0789 -0.00487 ± 0.00173 0.05975 ± 0.00068 6.870 ± 0.076 % 90.299 ± 0.117 %
70
+ 64 7.4729 ± 0.0786 -0.00483 ± 0.00171 0.05962 ± 0.00067 6.853 ± 0.075 % 90.315 ± 0.116 %
71
+
72
+ ====== Perplexity statistics ======
73
+ Mean PPL(Q) : 7.472885 ± 0.078642
74
+ Mean PPL(base) : 7.509043 ± 0.079361
75
+ Cor(ln(PPL(Q)), ln(PPL(base))): 98.69%
76
+ Mean ln(PPL(Q)/PPL(base)) : -0.004827 ± 0.001709
77
+ Mean PPL(Q)/PPL(base) : 0.995185 ± 0.001701
78
+ Mean PPL(Q)-PPL(base) : -0.036158 ± 0.012821
79
+
80
+ ====== KL divergence statistics ======
81
+ Mean KLD: 0.059623 ± 0.000672
82
+ Maximum KLD: 9.654090
83
+ 99.9% KLD: 2.073871
84
+ 99.0% KLD: 0.625332
85
+ 95.0% KLD: 0.216637
86
+ 90.0% KLD: 0.129178
87
+ Median KLD: 0.022184
88
+ 10.0% KLD: 0.000334
89
+ 5.0% KLD: 0.000094
90
+ 1.0% KLD: 0.000003
91
+ 0.1% KLD: -0.000150
92
+ Minimum KLD: -0.000877
93
+
94
+ ====== Token probability statistics ======
95
+ Mean Δp: -0.170 ± 0.027 %
96
+ Maximum Δp: 98.598%
97
+ 99.9% Δp: 49.072%
98
+ 99.0% Δp: 20.323%
99
+ 95.0% Δp: 8.341%
100
+ 90.0% Δp: 4.538%
101
+ 75.0% Δp: 0.758%
102
+ Median Δp: -0.002%
103
+ 25.0% Δp: -0.975%
104
+ 10.0% Δp: -4.956%
105
+ 5.0% Δp: -8.921%
106
+ 1.0% Δp: -22.053%
107
+ 0.1% Δp: -53.988%
108
+ Minimum Δp: -95.964%
109
+ RMS Δp : 6.853 ± 0.075 %
110
+ Same top p: 90.315 ± 0.116 %
111
+
appendix/measurements/kld_FAMSHALLOW-21G-ICE.txt ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.05.245.318 I
2
+ 0.05.245.474 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
3
+ 0.05.600.527 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
4
+ 0.07.535.520 I kl_divergence: 1.93 seconds per pass - ETA 2.05 minutes
5
+
6
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
7
+ 1 5.8599 ± 0.4617 0.00040 ± 0.01310 0.05680 ± 0.00402 7.121 ± 0.486 % 92.766 ± 0.810 %
8
+ 2 7.3267 ± 0.4277 0.00246 ± 0.00922 0.05132 ± 0.00267 6.121 ± 0.312 % 92.082 ± 0.597 %
9
+ 3 8.2728 ± 0.4086 0.00896 ± 0.00751 0.05147 ± 0.00295 6.307 ± 0.348 % 92.245 ± 0.483 %
10
+ 4 9.2259 ± 0.4185 -0.01028 ± 0.00700 0.05822 ± 0.00366 6.682 ± 0.322 % 91.813 ± 0.429 %
11
+ 5 8.6431 ± 0.3452 -0.00797 ± 0.00613 0.05606 ± 0.00300 6.536 ± 0.280 % 91.652 ± 0.387 %
12
+ 6 7.3047 ± 0.2573 -0.01359 ± 0.00554 0.05524 ± 0.00262 6.543 ± 0.242 % 91.496 ± 0.356 %
13
+ 7 6.6738 ± 0.2137 -0.01837 ± 0.00511 0.05407 ± 0.00233 6.558 ± 0.224 % 91.579 ± 0.328 %
14
+ 8 6.4897 ± 0.1919 -0.01683 ± 0.00466 0.05194 ± 0.00205 6.372 ± 0.203 % 91.691 ± 0.305 %
15
+ 9 6.8583 ± 0.1925 -0.01991 ± 0.00438 0.05206 ± 0.00185 6.323 ± 0.186 % 91.365 ± 0.293 %
16
+ 10 6.9294 ± 0.1849 -0.01873 ± 0.00406 0.05000 ± 0.00167 6.168 ± 0.173 % 91.535 ± 0.275 %
17
+ 11 6.9450 ± 0.1756 -0.01700 ± 0.00385 0.04832 ± 0.00153 6.048 ± 0.162 % 91.629 ± 0.261 %
18
+ 12 7.2139 ± 0.1760 -0.01527 ± 0.00359 0.04654 ± 0.00141 5.899 ± 0.153 % 91.691 ± 0.249 %
19
+ 13 7.3085 ± 0.1717 -0.01751 ± 0.00346 0.04625 ± 0.00132 5.901 ± 0.147 % 91.729 ± 0.239 %
20
+ 14 7.3657 ± 0.1666 -0.01606 ± 0.00327 0.04497 ± 0.00124 5.794 ± 0.140 % 91.684 ± 0.231 %
21
+ 15 7.4120 ± 0.1620 -0.01760 ± 0.00314 0.04457 ± 0.00117 5.755 ± 0.134 % 91.685 ± 0.223 %
22
+ 16 7.5875 ± 0.1607 -0.01894 ± 0.00303 0.04419 ± 0.00111 5.692 ± 0.128 % 91.716 ± 0.215 %
23
+ 17 7.6020 ± 0.1554 -0.01798 ± 0.00289 0.04347 ± 0.00106 5.607 ± 0.123 % 91.731 ± 0.209 %
24
+ 18 7.6886 ± 0.1526 -0.01688 ± 0.00278 0.04314 ± 0.00101 5.573 ± 0.118 % 91.740 ± 0.203 %
25
+ 19 7.6035 ± 0.1472 -0.01861 ± 0.00271 0.04303 ± 0.00097 5.533 ± 0.113 % 91.773 ± 0.197 %
26
+ 20 7.3318 ± 0.1377 -0.02261 ± 0.00279 0.04736 ± 0.00106 6.029 ± 0.120 % 91.628 ± 0.194 %
27
+ 21 7.3643 ± 0.1348 -0.02361 ± 0.00272 0.04792 ± 0.00104 6.067 ± 0.118 % 91.505 ± 0.190 %
28
+ 22 7.4459 ± 0.1338 -0.02263 ± 0.00264 0.04748 ± 0.00100 6.057 ± 0.115 % 91.593 ± 0.185 %
29
+ 23 7.4842 ± 0.1313 -0.02100 ± 0.00257 0.04706 ± 0.00096 6.005 ± 0.111 % 91.555 ± 0.181 %
30
+ 24 7.5670 ± 0.1303 -0.02359 ± 0.00252 0.04732 ± 0.00093 6.038 ± 0.110 % 91.504 ± 0.178 %
31
+ 25 7.6188 ± 0.1288 -0.02519 ± 0.00248 0.04758 ± 0.00090 6.017 ± 0.107 % 91.406 ± 0.175 %
32
+ 26 7.5730 ± 0.1253 -0.02459 ± 0.00242 0.04725 ± 0.00087 5.986 ± 0.104 % 91.469 ± 0.171 %
33
+ 27 7.7241 ± 0.1258 -0.02452 ± 0.00236 0.04688 ± 0.00085 5.947 ± 0.102 % 91.438 ± 0.168 %
34
+ 28 7.8010 ± 0.1248 -0.02436 ± 0.00230 0.04646 ± 0.00082 5.914 ± 0.099 % 91.454 ± 0.165 %
35
+ 29 7.8071 ± 0.1230 -0.02314 ± 0.00227 0.04657 ± 0.00081 5.928 ± 0.097 % 91.492 ± 0.162 %
36
+ 30 7.8413 ± 0.1219 -0.02481 ± 0.00225 0.04672 ± 0.00078 5.936 ± 0.095 % 91.505 ± 0.159 %
37
+ 31 7.7410 ± 0.1181 -0.02379 ± 0.00220 0.04651 ± 0.00077 5.929 ± 0.093 % 91.521 ± 0.156 %
38
+ 32 7.6234 ± 0.1143 -0.02147 ± 0.00217 0.04651 ± 0.00075 5.977 ± 0.092 % 91.560 ± 0.154 %
39
+ 33 7.6195 ± 0.1128 -0.01988 ± 0.00215 0.04677 ± 0.00076 5.999 ± 0.091 % 91.543 ± 0.151 %
40
+ 34 7.6013 ± 0.1106 -0.01917 ± 0.00210 0.04636 ± 0.00074 5.969 ± 0.089 % 91.550 ± 0.149 %
41
+ 35 7.5997 ± 0.1090 -0.01918 ± 0.00206 0.04603 ± 0.00073 5.943 ± 0.088 % 91.563 ± 0.147 %
42
+ 36 7.6050 ± 0.1076 -0.01916 ± 0.00202 0.04571 ± 0.00071 5.924 ± 0.086 % 91.642 ± 0.144 %
43
+ 37 7.4638 ± 0.1037 -0.01873 ± 0.00198 0.04514 ± 0.00069 5.892 ± 0.085 % 91.707 ± 0.142 %
44
+ 38 7.3558 ± 0.1003 -0.01836 ± 0.00194 0.04490 ± 0.00068 5.881 ± 0.083 % 91.735 ± 0.140 %
45
+ 39 7.2456 ± 0.0972 -0.01756 ± 0.00191 0.04467 ± 0.00067 5.872 ± 0.083 % 91.729 ± 0.138 %
46
+ 40 7.1224 ± 0.0938 -0.01694 ± 0.00187 0.04429 ± 0.00066 5.870 ± 0.082 % 91.764 ± 0.136 %
47
+ 41 7.0461 ± 0.0913 -0.01633 ± 0.00185 0.04400 ± 0.00065 5.867 ± 0.081 % 91.748 ± 0.134 %
48
+ 42 7.0766 ± 0.0907 -0.01617 ± 0.00182 0.04388 ± 0.00064 5.850 ± 0.080 % 91.740 ± 0.133 %
49
+ 43 7.2669 ± 0.0929 -0.01710 ± 0.00180 0.04386 ± 0.00063 5.836 ± 0.080 % 91.696 ± 0.132 %
50
+ 44 7.4074 ± 0.0941 -0.01670 ± 0.00178 0.04384 ± 0.00062 5.810 ± 0.079 % 91.693 ± 0.130 %
51
+ 45 7.3837 ± 0.0926 -0.01592 ± 0.00177 0.04462 ± 0.00062 5.860 ± 0.078 % 91.643 ± 0.129 %
52
+ 46 7.3880 ± 0.0915 -0.01595 ± 0.00176 0.04489 ± 0.00061 5.871 ± 0.077 % 91.623 ± 0.128 %
53
+ 47 7.4507 ± 0.0915 -0.01631 ± 0.00174 0.04508 ± 0.00061 5.881 ± 0.076 % 91.639 ± 0.126 %
54
+ 48 7.4484 ± 0.0904 -0.01602 ± 0.00171 0.04482 ± 0.00060 5.855 ± 0.075 % 91.622 ± 0.125 %
55
+ 49 7.4995 ± 0.0902 -0.01576 ± 0.00169 0.04477 ± 0.00060 5.838 ± 0.074 % 91.559 ± 0.124 %
56
+ 50 7.4568 ± 0.0886 -0.01523 ± 0.00167 0.04445 ± 0.00059 5.808 ± 0.073 % 91.591 ± 0.123 %
57
+ 51 7.4908 ± 0.0881 -0.01491 ± 0.00164 0.04402 ± 0.00058 5.771 ± 0.072 % 91.618 ± 0.121 %
58
+ 52 7.4319 ± 0.0865 -0.01439 ± 0.00162 0.04379 ± 0.00057 5.765 ± 0.072 % 91.648 ± 0.120 %
59
+ 53 7.4030 ± 0.0852 -0.01389 ± 0.00160 0.04361 ± 0.00056 5.755 ± 0.071 % 91.634 ± 0.119 %
60
+ 54 7.3621 ± 0.0837 -0.01384 ± 0.00158 0.04332 ± 0.00055 5.740 ± 0.070 % 91.644 ± 0.118 %
61
+ 55 7.3207 ± 0.0824 -0.01444 ± 0.00157 0.04333 ± 0.00055 5.751 ± 0.069 % 91.638 ± 0.117 %
62
+ 56 7.3182 ± 0.0817 -0.01392 ± 0.00156 0.04323 ± 0.00054 5.749 ± 0.069 % 91.651 ± 0.116 %
63
+ 57 7.2408 ± 0.0799 -0.01381 ± 0.00154 0.04300 ± 0.00053 5.725 ± 0.068 % 91.696 ± 0.114 %
64
+ 58 7.2826 ± 0.0799 -0.01286 ± 0.00154 0.04343 ± 0.00053 5.740 ± 0.067 % 91.664 ± 0.113 %
65
+ 59 7.2560 ± 0.0789 -0.01253 ± 0.00153 0.04377 ± 0.00053 5.775 ± 0.067 % 91.661 ± 0.113 %
66
+ 60 7.2398 ± 0.0781 -0.01241 ± 0.00152 0.04361 ± 0.00053 5.767 ± 0.066 % 91.675 ± 0.112 %
67
+ 61 7.3097 ± 0.0785 -0.01364 ± 0.00152 0.04428 ± 0.00055 5.817 ± 0.067 % 91.645 ± 0.111 %
68
+ 62 7.3346 ± 0.0781 -0.01326 ± 0.00151 0.04405 ± 0.00054 5.796 ± 0.067 % 91.647 ± 0.110 %
69
+ 63 7.3842 ± 0.0781 -0.01257 ± 0.00149 0.04388 ± 0.00053 5.776 ± 0.066 % 91.645 ± 0.109 %
70
+ 64 7.4168 ± 0.0779 -0.01236 ± 0.00148 0.04377 ± 0.00052 5.766 ± 0.066 % 91.647 ± 0.108 %
71
+
72
+ ====== Perplexity statistics ======
73
+ Mean PPL(Q) : 7.416771 ± 0.077928
74
+ Mean PPL(base) : 7.509043 ± 0.079361
75
+ Cor(ln(PPL(Q)), ln(PPL(base))): 99.02%
76
+ Mean ln(PPL(Q)/PPL(base)) : -0.012364 ± 0.001478
77
+ Mean PPL(Q)/PPL(base) : 0.987712 ± 0.001460
78
+ Mean PPL(Q)-PPL(base) : -0.092272 ± 0.011112
79
+
80
+ ====== KL divergence statistics ======
81
+ Mean KLD: 0.043766 ± 0.000525
82
+ Maximum KLD: 7.441271
83
+ 99.9% KLD: 1.559258
84
+ 99.0% KLD: 0.433440
85
+ 95.0% KLD: 0.160038
86
+ 90.0% KLD: 0.094157
87
+ Median KLD: 0.016019
88
+ 10.0% KLD: 0.000250
89
+ 5.0% KLD: 0.000068
90
+ 1.0% KLD: -0.000002
91
+ 0.1% KLD: -0.000161
92
+ Minimum KLD: -0.000895
93
+
94
+ ====== Token probability statistics ======
95
+ Mean Δp: -0.043 ± 0.023 %
96
+ Maximum Δp: 86.543%
97
+ 99.9% Δp: 44.594%
98
+ 99.0% Δp: 17.577%
99
+ 95.0% Δp: 7.246%
100
+ 90.0% Δp: 3.927%
101
+ 75.0% Δp: 0.697%
102
+ Median Δp: -0.001%
103
+ 25.0% Δp: -0.794%
104
+ 10.0% Δp: -4.118%
105
+ 5.0% Δp: -7.346%
106
+ 1.0% Δp: -17.592%
107
+ 0.1% Δp: -42.874%
108
+ Minimum Δp: -92.951%
109
+ RMS Δp : 5.766 ± 0.066 %
110
+ Same top p: 91.647 ± 0.108 %
111
+
appendix/measurements/kld_FLOORBF16.txt ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.05.688.915 I
2
+ 0.05.689.070 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
3
+ 0.06.026.826 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
4
+ 0.07.886.668 I kl_divergence: 1.86 seconds per pass - ETA 1.98 minutes
5
+
6
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
7
+ 1 5.7728 ± 0.4515 -0.01457 ± 0.01164 0.04767 ± 0.00440 6.765 ± 0.711 % 92.571 ± 0.820 %
8
+ 2 7.2617 ± 0.4216 -0.00646 ± 0.00796 0.04046 ± 0.00245 5.682 ± 0.449 % 92.375 ± 0.587 %
9
+ 3 8.2520 ± 0.4070 0.00645 ± 0.00646 0.03668 ± 0.00179 5.332 ± 0.354 % 92.603 ± 0.472 %
10
+ 4 9.4089 ± 0.4299 0.00937 ± 0.00585 0.04196 ± 0.00255 5.485 ± 0.310 % 92.424 ± 0.414 %
11
+ 5 8.7836 ± 0.3531 0.00815 ± 0.00520 0.04031 ± 0.00214 5.441 ± 0.286 % 92.395 ± 0.371 %
12
+ 6 7.4347 ± 0.2640 0.00405 ± 0.00465 0.03911 ± 0.00184 5.539 ± 0.250 % 92.424 ± 0.338 %
13
+ 7 6.7862 ± 0.2190 -0.00166 ± 0.00440 0.03986 ± 0.00167 5.897 ± 0.238 % 92.403 ± 0.313 %
14
+ 8 6.5920 ± 0.1964 -0.00118 ± 0.00401 0.03829 ± 0.00148 5.794 ± 0.221 % 92.473 ± 0.292 %
15
+ 9 6.9771 ± 0.1975 -0.00273 ± 0.00379 0.03891 ± 0.00136 5.716 ± 0.201 % 92.223 ± 0.279 %
16
+ 10 7.0373 ± 0.1890 -0.00328 ± 0.00353 0.03763 ± 0.00125 5.625 ± 0.191 % 92.424 ± 0.262 %
17
+ 11 7.0430 ± 0.1792 -0.00299 ± 0.00334 0.03623 ± 0.00115 5.475 ± 0.180 % 92.642 ± 0.246 %
18
+ 12 7.3161 ± 0.1795 -0.00121 ± 0.00312 0.03492 ± 0.00106 5.328 ± 0.170 % 92.652 ± 0.236 %
19
+ 13 7.4034 ± 0.1749 -0.00460 ± 0.00300 0.03513 ± 0.00103 5.360 ± 0.161 % 92.676 ± 0.226 %
20
+ 14 7.4492 ± 0.1692 -0.00480 ± 0.00284 0.03409 ± 0.00096 5.276 ± 0.153 % 92.745 ± 0.217 %
21
+ 15 7.5013 ± 0.1646 -0.00562 ± 0.00272 0.03365 ± 0.00091 5.235 ± 0.145 % 92.812 ± 0.209 %
22
+ 16 7.6842 ± 0.1634 -0.00626 ± 0.00262 0.03325 ± 0.00085 5.152 ± 0.139 % 92.827 ± 0.202 %
23
+ 17 7.6928 ± 0.1579 -0.00611 ± 0.00250 0.03282 ± 0.00083 5.080 ± 0.133 % 92.853 ± 0.195 %
24
+ 18 7.7728 ± 0.1548 -0.00599 ± 0.00241 0.03251 ± 0.00079 5.052 ± 0.127 % 92.886 ± 0.189 %
25
+ 19 7.6953 ± 0.1496 -0.00661 ± 0.00231 0.03172 ± 0.00075 4.980 ± 0.123 % 92.982 ± 0.183 %
26
+ 20 7.4347 ± 0.1403 -0.00867 ± 0.00236 0.03469 ± 0.00081 5.395 ± 0.128 % 92.835 ± 0.180 %
27
+ 21 7.4739 ± 0.1376 -0.00884 ± 0.00231 0.03491 ± 0.00079 5.394 ± 0.125 % 92.762 ± 0.177 %
28
+ 22 7.5501 ± 0.1364 -0.00873 ± 0.00224 0.03462 ± 0.00076 5.369 ± 0.121 % 92.833 ± 0.172 %
29
+ 23 7.5795 ± 0.1336 -0.00835 ± 0.00217 0.03420 ± 0.00073 5.308 ± 0.117 % 92.817 ± 0.168 %
30
+ 24 7.6618 ± 0.1325 -0.01114 ± 0.00214 0.03461 ± 0.00071 5.312 ± 0.114 % 92.758 ± 0.165 %
31
+ 25 7.7324 ± 0.1314 -0.01039 ± 0.00209 0.03422 ± 0.00069 5.261 ± 0.111 % 92.708 ± 0.163 %
32
+ 26 7.6837 ± 0.1278 -0.01007 ± 0.00203 0.03387 ± 0.00067 5.221 ± 0.109 % 92.763 ± 0.159 %
33
+ 27 7.8384 ± 0.1283 -0.00983 ± 0.00198 0.03328 ± 0.00064 5.156 ± 0.106 % 92.777 ± 0.156 %
34
+ 28 7.9157 ± 0.1274 -0.00976 ± 0.00193 0.03299 ± 0.00063 5.118 ± 0.104 % 92.801 ± 0.153 %
35
+ 29 7.9252 ± 0.1256 -0.00812 ± 0.00190 0.03308 ± 0.00061 5.120 ± 0.101 % 92.827 ± 0.150 %
36
+ 30 7.9506 ± 0.1243 -0.01097 ± 0.00189 0.03350 ± 0.00060 5.130 ± 0.098 % 92.796 ± 0.148 %
37
+ 31 7.8418 ± 0.1202 -0.01085 ± 0.00185 0.03330 ± 0.00059 5.109 ± 0.096 % 92.811 ± 0.145 %
38
+ 32 7.7172 ± 0.1163 -0.00924 ± 0.00183 0.03324 ± 0.00057 5.132 ± 0.093 % 92.870 ± 0.142 %
39
+ 33 7.7067 ± 0.1146 -0.00850 ± 0.00180 0.03352 ± 0.00063 5.159 ± 0.093 % 92.852 ± 0.140 %
40
+ 34 7.6864 ± 0.1124 -0.00805 ± 0.00176 0.03318 ± 0.00061 5.133 ± 0.091 % 92.847 ± 0.138 %
41
+ 35 7.6777 ± 0.1106 -0.00896 ± 0.00173 0.03311 ± 0.00060 5.132 ± 0.090 % 92.847 ± 0.136 %
42
+ 36 7.6827 ± 0.1092 -0.00899 ± 0.00170 0.03284 ± 0.00059 5.102 ± 0.088 % 92.891 ± 0.134 %
43
+ 37 7.5389 ± 0.1052 -0.00872 ± 0.00166 0.03237 ± 0.00057 5.066 ± 0.086 % 92.949 ± 0.132 %
44
+ 38 7.4286 ± 0.1018 -0.00851 ± 0.00163 0.03212 ± 0.00056 5.053 ± 0.085 % 92.988 ± 0.130 %
45
+ 39 7.3144 ± 0.0985 -0.00812 ± 0.00160 0.03177 ± 0.00055 5.029 ± 0.083 % 93.002 ± 0.128 %
46
+ 40 7.1875 ± 0.0951 -0.00785 ± 0.00157 0.03141 ± 0.00054 5.013 ± 0.082 % 93.050 ± 0.126 %
47
+ 41 7.1082 ± 0.0925 -0.00755 ± 0.00155 0.03116 ± 0.00052 4.989 ± 0.080 % 93.076 ± 0.124 %
48
+ 42 7.1381 ± 0.0918 -0.00751 ± 0.00153 0.03110 ± 0.00051 4.984 ± 0.079 % 93.067 ± 0.123 %
49
+ 43 7.3385 ± 0.0942 -0.00731 ± 0.00150 0.03111 ± 0.00050 4.961 ± 0.078 % 93.005 ± 0.122 %
50
+ 44 7.4815 ± 0.0955 -0.00674 ± 0.00149 0.03111 ± 0.00051 4.941 ± 0.077 % 93.000 ± 0.120 %
51
+ 45 7.4583 ± 0.0940 -0.00587 ± 0.00148 0.03195 ± 0.00051 5.008 ± 0.076 % 92.916 ± 0.120 %
52
+ 46 7.4659 ± 0.0930 -0.00546 ± 0.00147 0.03212 ± 0.00050 5.019 ± 0.075 % 92.907 ± 0.118 %
53
+ 47 7.5287 ± 0.0929 -0.00589 ± 0.00145 0.03223 ± 0.00050 5.016 ± 0.074 % 92.902 ± 0.117 %
54
+ 48 7.5266 ± 0.0918 -0.00559 ± 0.00143 0.03204 ± 0.00049 4.993 ± 0.073 % 92.913 ± 0.116 %
55
+ 49 7.5789 ± 0.0916 -0.00524 ± 0.00142 0.03209 ± 0.00048 4.988 ± 0.072 % 92.872 ± 0.115 %
56
+ 50 7.5327 ± 0.0899 -0.00510 ± 0.00140 0.03187 ± 0.00047 4.971 ± 0.071 % 92.911 ± 0.113 %
57
+ 51 7.5679 ± 0.0895 -0.00467 ± 0.00138 0.03159 ± 0.00047 4.940 ± 0.070 % 92.935 ± 0.112 %
58
+ 52 7.5072 ± 0.0878 -0.00432 ± 0.00137 0.03142 ± 0.00046 4.939 ± 0.069 % 92.956 ± 0.111 %
59
+ 53 7.4789 ± 0.0865 -0.00368 ± 0.00135 0.03123 ± 0.00045 4.926 ± 0.068 % 92.971 ± 0.110 %
60
+ 54 7.4354 ± 0.0850 -0.00393 ± 0.00133 0.03101 ± 0.00045 4.916 ± 0.068 % 92.965 ± 0.109 %
61
+ 55 7.3950 ± 0.0837 -0.00434 ± 0.00132 0.03094 ± 0.00044 4.928 ± 0.068 % 92.955 ± 0.108 %
62
+ 56 7.3841 ± 0.0828 -0.00496 ± 0.00131 0.03093 ± 0.00043 4.924 ± 0.067 % 92.951 ± 0.107 %
63
+ 57 7.3054 ± 0.0810 -0.00493 ± 0.00130 0.03077 ± 0.00043 4.910 ± 0.066 % 92.960 ± 0.106 %
64
+ 58 7.3422 ± 0.0809 -0.00470 ± 0.00130 0.03127 ± 0.00043 4.952 ± 0.065 % 92.932 ± 0.105 %
65
+ 59 7.3161 ± 0.0799 -0.00428 ± 0.00130 0.03155 ± 0.00043 4.973 ± 0.065 % 92.932 ± 0.104 %
66
+ 60 7.2992 ± 0.0790 -0.00423 ± 0.00128 0.03137 ± 0.00043 4.958 ± 0.064 % 92.967 ± 0.103 %
67
+ 61 7.3703 ± 0.0795 -0.00539 ± 0.00129 0.03195 ± 0.00045 4.994 ± 0.064 % 92.915 ± 0.103 %
68
+ 62 7.3943 ± 0.0791 -0.00516 ± 0.00128 0.03182 ± 0.00044 4.973 ± 0.063 % 92.913 ± 0.102 %
69
+ 63 7.4403 ± 0.0790 -0.00499 ± 0.00126 0.03166 ± 0.00043 4.956 ± 0.063 % 92.900 ± 0.101 %
70
+ 64 7.4712 ± 0.0788 -0.00506 ± 0.00125 0.03154 ± 0.00043 4.942 ± 0.062 % 92.916 ± 0.100 %
71
+
72
+ ====== Perplexity statistics ======
73
+ Mean PPL(Q) : 7.471173 ± 0.078771
74
+ Mean PPL(base) : 7.509043 ± 0.079361
75
+ Cor(ln(PPL(Q)), ln(PPL(base))): 99.30%
76
+ Mean ln(PPL(Q)/PPL(base)) : -0.005056 ± 0.001250
77
+ Mean PPL(Q)/PPL(base) : 0.994957 ± 0.001244
78
+ Mean PPL(Q)-PPL(base) : -0.037870 ± 0.009381
79
+
80
+ ====== KL divergence statistics ======
81
+ Mean KLD: 0.031543 ± 0.000428
82
+ Maximum KLD: 9.499484
83
+ 99.9% KLD: 1.221953
84
+ 99.0% KLD: 0.321817
85
+ 95.0% KLD: 0.113814
86
+ 90.0% KLD: 0.068127
87
+ Median KLD: 0.010694
88
+ 10.0% KLD: 0.000164
89
+ 5.0% KLD: 0.000044
90
+ 1.0% KLD: -0.000003
91
+ 0.1% KLD: -0.000143
92
+ Minimum KLD: -0.000842
93
+
94
+ ====== Token probability statistics ======
95
+ Mean Δp: -0.058 ± 0.019 %
96
+ Maximum Δp: 89.304%
97
+ 99.9% Δp: 38.220%
98
+ 99.0% Δp: 15.078%
99
+ 95.0% Δp: 5.939%
100
+ 90.0% Δp: 3.167%
101
+ 75.0% Δp: 0.544%
102
+ Median Δp: -0.001%
103
+ 25.0% Δp: -0.655%
104
+ 10.0% Δp: -3.404%
105
+ 5.0% Δp: -6.134%
106
+ 1.0% Δp: -15.171%
107
+ 0.1% Δp: -38.071%
108
+ Minimum Δp: -75.561%
109
+ RMS Δp : 4.942 ± 0.062 %
110
+ Same top p: 92.916 ± 0.100 %
111
+
appendix/measurements/kld_H.txt ADDED
@@ -0,0 +1,112 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.05.196.457 I cmn init: llama threadpool init, n_threads = 30
2
+ 0.05.335.468 I
3
+ 0.05.335.647 I system_info: n_threads = 30 (n_threads_batch = 30) / 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
4
+ 0.05.679.180 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
5
+ 0.25.203.802 I kl_divergence: 19.52 seconds per pass - ETA 20.82 minutes
6
+
7
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
8
+ 1 5.9174 ± 0.4677 0.01017 ± 0.00896 0.02132 ± 0.00172 4.372 ± 0.419 % 95.112 ± 0.674 %
9
+ 2 7.3822 ± 0.4328 0.01001 ± 0.00579 0.01936 ± 0.00105 3.783 ± 0.282 % 94.282 ± 0.513 %
10
+ 3 8.2291 ± 0.4066 0.00367 ± 0.00461 0.01907 ± 0.00174 3.756 ± 0.311 % 94.884 ± 0.398 %
11
+ 4 9.3087 ± 0.4251 -0.00134 ± 0.00442 0.02442 ± 0.00201 4.340 ± 0.315 % 94.599 ± 0.353 %
12
+ 5 8.6947 ± 0.3495 -0.00202 ± 0.00375 0.02208 ± 0.00162 4.112 ± 0.269 % 94.663 ± 0.314 %
13
+ 6 7.3976 ± 0.2626 -0.00096 ± 0.00354 0.02327 ± 0.00158 4.439 ± 0.280 % 94.558 ± 0.290 %
14
+ 7 6.8377 ± 0.2217 0.00590 ± 0.00339 0.02421 ± 0.00143 4.653 ± 0.253 % 94.554 ± 0.268 %
15
+ 8 6.6282 ± 0.1982 0.00429 ± 0.00307 0.02290 ± 0.00126 4.498 ± 0.230 % 94.526 ± 0.251 %
16
+ 9 7.0033 ± 0.1987 0.00101 ± 0.00289 0.02285 ± 0.00113 4.411 ± 0.210 % 94.363 ± 0.240 %
17
+ 10 7.0613 ± 0.1902 0.00012 ± 0.00271 0.02207 ± 0.00102 4.316 ± 0.194 % 94.438 ± 0.227 %
18
+ 11 7.0625 ± 0.1802 -0.00023 ± 0.00261 0.02118 ± 0.00093 4.249 ± 0.182 % 94.570 ± 0.214 %
19
+ 12 7.3127 ± 0.1797 -0.00167 ± 0.00246 0.02062 ± 0.00086 4.172 ± 0.171 % 94.607 ± 0.204 %
20
+ 13 7.4186 ± 0.1758 -0.00257 ± 0.00236 0.02038 ± 0.00080 4.132 ± 0.160 % 94.631 ± 0.195 %
21
+ 14 7.4637 ± 0.1700 -0.00284 ± 0.00224 0.01969 ± 0.00075 4.048 ± 0.152 % 94.721 ± 0.187 %
22
+ 15 7.5143 ± 0.1653 -0.00389 ± 0.00216 0.01949 ± 0.00070 4.017 ± 0.143 % 94.774 ± 0.180 %
23
+ 16 7.6914 ± 0.1639 -0.00533 ± 0.00206 0.01928 ± 0.00066 3.961 ± 0.137 % 94.770 ± 0.174 %
24
+ 17 7.6975 ± 0.1583 -0.00549 ± 0.00197 0.01884 ± 0.00062 3.894 ± 0.131 % 94.819 ± 0.168 %
25
+ 18 7.7746 ± 0.1551 -0.00576 ± 0.00190 0.01865 ± 0.00059 3.881 ± 0.126 % 94.825 ± 0.163 %
26
+ 19 7.6863 ± 0.1495 -0.00778 ± 0.00185 0.01884 ± 0.00057 3.882 ± 0.120 % 94.752 ± 0.160 %
27
+ 20 7.4424 ± 0.1406 -0.00763 ± 0.00183 0.02033 ± 0.00072 4.101 ± 0.123 % 94.692 ± 0.157 %
28
+ 21 7.4876 ± 0.1380 -0.00701 ± 0.00179 0.02061 ± 0.00071 4.182 ± 0.128 % 94.628 ± 0.154 %
29
+ 22 7.5511 ± 0.1364 -0.00860 ± 0.00174 0.02055 ± 0.00068 4.163 ± 0.124 % 94.633 ± 0.150 %
30
+ 23 7.5812 ± 0.1337 -0.00813 ± 0.00169 0.02028 ± 0.00065 4.106 ± 0.120 % 94.666 ± 0.146 %
31
+ 24 7.6835 ± 0.1331 -0.00831 ± 0.00166 0.02047 ± 0.00063 4.124 ± 0.117 % 94.681 ± 0.143 %
32
+ 25 7.7448 ± 0.1318 -0.00878 ± 0.00162 0.02029 ± 0.00061 4.093 ± 0.114 % 94.663 ± 0.141 %
33
+ 26 7.6955 ± 0.1281 -0.00853 ± 0.00158 0.02014 ± 0.00059 4.085 ± 0.112 % 94.695 ± 0.137 %
34
+ 27 7.8462 ± 0.1285 -0.00884 ± 0.00154 0.01976 ± 0.00057 4.037 ± 0.109 % 94.692 ± 0.135 %
35
+ 28 7.9290 ± 0.1277 -0.00808 ± 0.00150 0.01946 ± 0.00055 3.999 ± 0.106 % 94.725 ± 0.132 %
36
+ 29 7.9318 ± 0.1258 -0.00729 ± 0.00148 0.01941 ± 0.00053 3.994 ± 0.103 % 94.715 ± 0.130 %
37
+ 30 7.9769 ± 0.1249 -0.00766 ± 0.00146 0.01933 ± 0.00052 3.977 ± 0.100 % 94.695 ± 0.128 %
38
+ 31 7.8678 ± 0.1208 -0.00754 ± 0.00143 0.01908 ± 0.00050 3.949 ± 0.098 % 94.740 ± 0.125 %
39
+ 32 7.7296 ± 0.1166 -0.00763 ± 0.00140 0.01902 ± 0.00049 3.962 ± 0.096 % 94.789 ± 0.123 %
40
+ 33 7.7175 ± 0.1149 -0.00710 ± 0.00139 0.01908 ± 0.00050 3.953 ± 0.094 % 94.772 ± 0.121 %
41
+ 34 7.6930 ± 0.1126 -0.00719 ± 0.00135 0.01881 ± 0.00048 3.925 ± 0.092 % 94.790 ± 0.119 %
42
+ 35 7.6901 ± 0.1109 -0.00735 ± 0.00133 0.01872 ± 0.00048 3.909 ± 0.090 % 94.788 ± 0.117 %
43
+ 36 7.6936 ± 0.1094 -0.00758 ± 0.00130 0.01862 ± 0.00046 3.894 ± 0.088 % 94.833 ± 0.115 %
44
+ 37 7.5492 ± 0.1054 -0.00735 ± 0.00127 0.01833 ± 0.00045 3.869 ± 0.087 % 94.898 ± 0.113 %
45
+ 38 7.4373 ± 0.1020 -0.00734 ± 0.00125 0.01818 ± 0.00044 3.855 ± 0.085 % 94.914 ± 0.111 %
46
+ 39 7.3208 ± 0.0987 -0.00724 ± 0.00123 0.01796 ± 0.00043 3.840 ± 0.083 % 94.934 ± 0.110 %
47
+ 40 7.1933 ± 0.0952 -0.00703 ± 0.00120 0.01772 ± 0.00042 3.825 ± 0.082 % 94.968 ± 0.108 %
48
+ 41 7.1144 ± 0.0926 -0.00669 ± 0.00118 0.01754 ± 0.00041 3.807 ± 0.080 % 94.986 ± 0.107 %
49
+ 42 7.1456 ± 0.0920 -0.00646 ± 0.00116 0.01748 ± 0.00040 3.789 ± 0.079 % 94.973 ± 0.105 %
50
+ 43 7.3403 ± 0.0942 -0.00706 ± 0.00116 0.01768 ± 0.00040 3.802 ± 0.080 % 94.894 ± 0.105 %
51
+ 44 7.4761 ± 0.0954 -0.00747 ± 0.00115 0.01774 ± 0.00040 3.787 ± 0.079 % 94.864 ± 0.104 %
52
+ 45 7.4483 ± 0.0938 -0.00721 ± 0.00114 0.01807 ± 0.00039 3.812 ± 0.077 % 94.826 ± 0.103 %
53
+ 46 7.4547 ± 0.0928 -0.00696 ± 0.00113 0.01809 ± 0.00039 3.809 ± 0.075 % 94.826 ± 0.102 %
54
+ 47 7.5195 ± 0.0928 -0.00713 ± 0.00112 0.01827 ± 0.00039 3.833 ± 0.078 % 94.815 ± 0.101 %
55
+ 48 7.5177 ± 0.0916 -0.00677 ± 0.00111 0.01819 ± 0.00039 3.819 ± 0.077 % 94.827 ± 0.100 %
56
+ 49 7.5661 ± 0.0914 -0.00692 ± 0.00110 0.01835 ± 0.00039 3.827 ± 0.076 % 94.787 ± 0.099 %
57
+ 50 7.5215 ± 0.0897 -0.00659 ± 0.00109 0.01825 ± 0.00039 3.810 ± 0.075 % 94.819 ± 0.098 %
58
+ 51 7.5548 ± 0.0892 -0.00640 ± 0.00107 0.01808 ± 0.00038 3.787 ± 0.074 % 94.834 ± 0.097 %
59
+ 52 7.4920 ± 0.0875 -0.00634 ± 0.00106 0.01798 ± 0.00037 3.780 ± 0.073 % 94.857 ± 0.096 %
60
+ 53 7.4630 ± 0.0862 -0.00582 ± 0.00105 0.01790 ± 0.00037 3.771 ± 0.072 % 94.839 ± 0.095 %
61
+ 54 7.4215 ± 0.0847 -0.00581 ± 0.00103 0.01784 ± 0.00036 3.780 ± 0.072 % 94.850 ± 0.094 %
62
+ 55 7.3804 ± 0.0834 -0.00633 ± 0.00103 0.01780 ± 0.00036 3.774 ± 0.071 % 94.855 ± 0.093 %
63
+ 56 7.3767 ± 0.0827 -0.00596 ± 0.00102 0.01779 ± 0.00035 3.778 ± 0.070 % 94.859 ± 0.092 %
64
+ 57 7.2984 ± 0.0808 -0.00589 ± 0.00100 0.01768 ± 0.00035 3.771 ± 0.069 % 94.891 ± 0.091 %
65
+ 58 7.3308 ± 0.0807 -0.00626 ± 0.00101 0.01817 ± 0.00036 3.825 ± 0.069 % 94.834 ± 0.091 %
66
+ 59 7.3039 ± 0.0797 -0.00595 ± 0.00101 0.01828 ± 0.00035 3.844 ± 0.068 % 94.809 ± 0.090 %
67
+ 60 7.2858 ± 0.0788 -0.00607 ± 0.00100 0.01819 ± 0.00035 3.836 ± 0.067 % 94.827 ± 0.089 %
68
+ 61 7.3585 ± 0.0793 -0.00699 ± 0.00100 0.01863 ± 0.00037 3.877 ± 0.068 % 94.784 ± 0.089 %
69
+ 62 7.3822 ± 0.0789 -0.00679 ± 0.00099 0.01851 ± 0.00036 3.858 ± 0.067 % 94.778 ± 0.088 %
70
+ 63 7.4281 ± 0.0788 -0.00664 ± 0.00098 0.01839 ± 0.00035 3.843 ± 0.066 % 94.783 ± 0.088 %
71
+ 64 7.4585 ± 0.0786 -0.00676 ± 0.00096 0.01830 ± 0.00035 3.828 ± 0.066 % 94.790 ± 0.087 %
72
+
73
+ ====== Perplexity statistics ======
74
+ Mean PPL(Q) : 7.458467 ± 0.078564
75
+ Mean PPL(base) : 7.509043 ± 0.079361
76
+ Cor(ln(PPL(Q)), ln(PPL(base))): 99.58%
77
+ Mean ln(PPL(Q)/PPL(base)) : -0.006758 ± 0.000964
78
+ Mean PPL(Q)/PPL(base) : 0.993265 ± 0.000958
79
+ Mean PPL(Q)-PPL(base) : -0.050576 ± 0.007257
80
+
81
+ ====== KL divergence statistics ======
82
+ Mean KLD: 0.018297 ± 0.000349
83
+ Maximum KLD: 7.798795
84
+ 99.9% KLD: 0.792242
85
+ 99.0% KLD: 0.180538
86
+ 95.0% KLD: 0.062892
87
+ 90.0% KLD: 0.037522
88
+ Median KLD: 0.005781
89
+ 10.0% KLD: 0.000103
90
+ 5.0% KLD: 0.000027
91
+ 1.0% KLD: -0.000005
92
+ 0.1% KLD: -0.000138
93
+ Minimum KLD: -0.000706
94
+
95
+ ====== Token probability statistics ======
96
+ Mean Δp: -0.014 ± 0.015 %
97
+ Maximum Δp: 88.045%
98
+ 99.9% Δp: 30.113%
99
+ 99.0% Δp: 11.269%
100
+ 95.0% Δp: 4.348%
101
+ 90.0% Δp: 2.350%
102
+ 75.0% Δp: 0.405%
103
+ Median Δp: -0.000%
104
+ 25.0% Δp: -0.478%
105
+ 10.0% Δp: -2.455%
106
+ 5.0% Δp: -4.492%
107
+ 1.0% Δp: -10.845%
108
+ 0.1% Δp: -26.883%
109
+ Minimum Δp: -97.726%
110
+ RMS Δp : 3.828 ± 0.066 %
111
+ Same top p: 94.790 ± 0.087 %
112
+
appendix/measurements/kld_SHALLOW-19G-ICE.txt ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.04.785.606 I
2
+ 0.04.785.737 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
3
+ 0.05.118.538 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
4
+ 0.06.917.749 I kl_divergence: 1.80 seconds per pass - ETA 1.92 minutes
5
+
6
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
7
+ 1 5.9463 ± 0.4643 0.01505 ± 0.01499 0.08661 ± 0.00811 8.274 ± 0.592 % 89.443 ± 0.961 %
8
+ 2 7.4650 ± 0.4361 0.02116 ± 0.01026 0.07104 ± 0.00436 7.223 ± 0.371 % 89.443 ± 0.680 %
9
+ 3 8.3322 ± 0.4101 0.01612 ± 0.00861 0.06969 ± 0.00404 7.412 ± 0.413 % 89.834 ± 0.546 %
10
+ 4 9.4202 ± 0.4280 0.01057 ± 0.00779 0.07592 ± 0.00408 7.556 ± 0.343 % 89.687 ± 0.475 %
11
+ 5 8.8371 ± 0.3542 0.01422 ± 0.00685 0.07243 ± 0.00338 7.322 ± 0.291 % 89.717 ± 0.425 %
12
+ 6 7.4746 ± 0.2640 0.00941 ± 0.00611 0.07048 ± 0.00297 7.398 ± 0.257 % 89.752 ± 0.387 %
13
+ 7 6.8185 ± 0.2192 0.00309 ± 0.00585 0.07181 ± 0.00288 7.786 ± 0.260 % 89.890 ± 0.356 %
14
+ 8 6.6259 ± 0.1963 0.00393 ± 0.00532 0.06926 ± 0.00256 7.623 ± 0.236 % 90.042 ± 0.331 %
15
+ 9 7.0245 ± 0.1979 0.00403 ± 0.00498 0.06973 ± 0.00233 7.531 ± 0.217 % 89.584 ± 0.318 %
16
+ 10 7.0920 ± 0.1897 0.00446 ± 0.00462 0.06742 ± 0.00211 7.395 ± 0.203 % 89.609 ± 0.302 %
17
+ 11 7.0837 ± 0.1791 0.00277 ± 0.00436 0.06581 ± 0.00194 7.306 ± 0.192 % 89.612 ± 0.288 %
18
+ 12 7.3487 ± 0.1791 0.00324 ± 0.00408 0.06367 ± 0.00179 7.144 ± 0.181 % 89.703 ± 0.274 %
19
+ 13 7.4709 ± 0.1758 0.00447 ± 0.00389 0.06251 ± 0.00167 7.108 ± 0.170 % 89.766 ± 0.263 %
20
+ 14 7.5105 ± 0.1699 0.00340 ± 0.00370 0.06086 ± 0.00157 6.985 ± 0.163 % 89.813 ± 0.253 %
21
+ 15 7.5804 ± 0.1659 0.00487 ± 0.00356 0.06024 ± 0.00148 6.926 ± 0.155 % 89.853 ± 0.244 %
22
+ 16 7.7541 ± 0.1643 0.00279 ± 0.00345 0.05999 ± 0.00140 6.887 ± 0.149 % 89.846 ± 0.236 %
23
+ 17 7.7624 ± 0.1586 0.00290 ± 0.00331 0.05876 ± 0.00132 6.781 ± 0.143 % 89.874 ± 0.229 %
24
+ 18 7.8483 ± 0.1556 0.00367 ± 0.00319 0.05818 ± 0.00126 6.740 ± 0.137 % 89.980 ± 0.221 %
25
+ 19 7.7638 ± 0.1502 0.00225 ± 0.00308 0.05719 ± 0.00120 6.664 ± 0.132 % 90.076 ± 0.214 %
26
+ 20 7.5078 ± 0.1409 0.00112 ± 0.00316 0.06251 ± 0.00134 7.143 ± 0.138 % 89.868 ± 0.211 %
27
+ 21 7.5513 ± 0.1383 0.00146 ± 0.00309 0.06277 ± 0.00130 7.160 ± 0.135 % 89.806 ± 0.206 %
28
+ 22 7.6127 ± 0.1367 -0.00048 ± 0.00301 0.06239 ± 0.00124 7.119 ± 0.130 % 89.834 ± 0.201 %
29
+ 23 7.6421 ± 0.1339 -0.00013 ± 0.00292 0.06174 ± 0.00120 7.064 ± 0.127 % 89.915 ± 0.196 %
30
+ 24 7.7141 ± 0.1326 -0.00434 ± 0.00289 0.06304 ± 0.00117 7.116 ± 0.124 % 89.866 ± 0.193 %
31
+ 25 7.7736 ± 0.1312 -0.00507 ± 0.00283 0.06279 ± 0.00114 7.086 ± 0.122 % 89.853 ± 0.189 %
32
+ 26 7.7164 ± 0.1273 -0.00583 ± 0.00277 0.06252 ± 0.00111 7.055 ± 0.119 % 89.920 ± 0.185 %
33
+ 27 7.8712 ± 0.1278 -0.00565 ± 0.00269 0.06180 ± 0.00107 6.995 ± 0.116 % 89.903 ± 0.181 %
34
+ 28 7.9431 ± 0.1267 -0.00630 ± 0.00263 0.06131 ± 0.00104 6.953 ± 0.114 % 89.921 ± 0.178 %
35
+ 29 7.9316 ± 0.1244 -0.00732 ± 0.00262 0.06211 ± 0.00103 7.041 ± 0.113 % 89.928 ± 0.175 %
36
+ 30 7.9718 ± 0.1235 -0.00830 ± 0.00258 0.06207 ± 0.00100 7.030 ± 0.110 % 89.935 ± 0.172 %
37
+ 31 7.8700 ± 0.1196 -0.00726 ± 0.00253 0.06164 ± 0.00098 7.011 ± 0.107 % 90.017 ± 0.168 %
38
+ 32 7.7375 ± 0.1155 -0.00660 ± 0.00248 0.06171 ± 0.00096 7.024 ± 0.106 % 90.124 ± 0.165 %
39
+ 33 7.7222 ± 0.1138 -0.00649 ± 0.00244 0.06170 ± 0.00096 7.017 ± 0.104 % 90.163 ± 0.162 %
40
+ 34 7.6971 ± 0.1114 -0.00666 ± 0.00239 0.06123 ± 0.00093 6.982 ± 0.102 % 90.141 ± 0.160 %
41
+ 35 7.6880 ± 0.1097 -0.00763 ± 0.00235 0.06113 ± 0.00091 6.970 ± 0.100 % 90.158 ± 0.157 %
42
+ 36 7.6952 ± 0.1083 -0.00737 ± 0.00230 0.06056 ± 0.00089 6.935 ± 0.099 % 90.247 ± 0.155 %
43
+ 37 7.5552 ± 0.1044 -0.00656 ± 0.00225 0.05988 ± 0.00087 6.903 ± 0.098 % 90.331 ± 0.152 %
44
+ 38 7.4468 ± 0.1010 -0.00607 ± 0.00221 0.05971 ± 0.00086 6.921 ± 0.096 % 90.312 ± 0.150 %
45
+ 39 7.3309 ± 0.0978 -0.00586 ± 0.00218 0.05931 ± 0.00084 6.899 ± 0.095 % 90.320 ± 0.148 %
46
+ 40 7.2053 ± 0.0944 -0.00537 ± 0.00214 0.05879 ± 0.00083 6.882 ± 0.093 % 90.367 ± 0.146 %
47
+ 41 7.1268 ± 0.0918 -0.00494 ± 0.00210 0.05848 ± 0.00081 6.896 ± 0.093 % 90.382 ± 0.144 %
48
+ 42 7.1609 ± 0.0912 -0.00433 ± 0.00207 0.05832 ± 0.00080 6.864 ± 0.091 % 90.367 ± 0.142 %
49
+ 43 7.3613 ± 0.0936 -0.00420 ± 0.00205 0.05823 ± 0.00078 6.836 ± 0.090 % 90.329 ± 0.141 %
50
+ 44 7.4945 ± 0.0947 -0.00500 ± 0.00202 0.05808 ± 0.00077 6.796 ± 0.088 % 90.320 ± 0.139 %
51
+ 45 7.4782 ± 0.0933 -0.00321 ± 0.00202 0.05953 ± 0.00078 6.875 ± 0.087 % 90.212 ± 0.138 %
52
+ 46 7.4825 ± 0.0922 -0.00324 ± 0.00200 0.06000 ± 0.00077 6.904 ± 0.086 % 90.221 ± 0.137 %
53
+ 47 7.5432 ± 0.0922 -0.00398 ± 0.00198 0.06029 ± 0.00078 6.918 ± 0.086 % 90.210 ± 0.136 %
54
+ 48 7.5434 ± 0.0911 -0.00335 ± 0.00195 0.05993 ± 0.00076 6.893 ± 0.085 % 90.217 ± 0.134 %
55
+ 49 7.5946 ± 0.0908 -0.00316 ± 0.00193 0.05980 ± 0.00075 6.873 ± 0.084 % 90.223 ± 0.133 %
56
+ 50 7.5515 ± 0.0893 -0.00260 ± 0.00191 0.05953 ± 0.00074 6.855 ± 0.083 % 90.260 ± 0.131 %
57
+ 51 7.5800 ± 0.0887 -0.00306 ± 0.00188 0.05920 ± 0.00073 6.834 ± 0.082 % 90.294 ± 0.130 %
58
+ 52 7.5216 ± 0.0870 -0.00239 ± 0.00187 0.05904 ± 0.00073 6.832 ± 0.082 % 90.324 ± 0.128 %
59
+ 53 7.4910 ± 0.0857 -0.00207 ± 0.00184 0.05867 ± 0.00072 6.809 ± 0.080 % 90.335 ± 0.127 %
60
+ 54 7.4497 ± 0.0843 -0.00201 ± 0.00182 0.05829 ± 0.00071 6.782 ± 0.079 % 90.364 ± 0.126 %
61
+ 55 7.4078 ± 0.0830 -0.00262 ± 0.00180 0.05809 ± 0.00070 6.777 ± 0.079 % 90.376 ± 0.124 %
62
+ 56 7.4081 ± 0.0823 -0.00171 ± 0.00179 0.05799 ± 0.00069 6.768 ± 0.078 % 90.370 ± 0.123 %
63
+ 57 7.3295 ± 0.0805 -0.00164 ± 0.00177 0.05764 ± 0.00068 6.750 ± 0.077 % 90.408 ± 0.122 %
64
+ 58 7.3808 ± 0.0806 0.00054 ± 0.00178 0.05854 ± 0.00068 6.782 ± 0.076 % 90.350 ± 0.121 %
65
+ 59 7.3512 ± 0.0795 0.00051 ± 0.00177 0.05897 ± 0.00068 6.809 ± 0.076 % 90.331 ± 0.120 %
66
+ 60 7.3329 ± 0.0787 0.00037 ± 0.00175 0.05879 ± 0.00068 6.804 ± 0.075 % 90.367 ± 0.119 %
67
+ 61 7.4050 ± 0.0792 -0.00069 ± 0.00175 0.05935 ± 0.00068 6.831 ± 0.074 % 90.295 ± 0.119 %
68
+ 62 7.4278 ± 0.0787 -0.00064 ± 0.00173 0.05907 ± 0.00067 6.801 ± 0.074 % 90.291 ± 0.118 %
69
+ 63 7.4789 ± 0.0788 0.00018 ± 0.00172 0.05876 ± 0.00066 6.783 ± 0.073 % 90.302 ± 0.117 %
70
+ 64 7.5098 ± 0.0785 0.00011 ± 0.00170 0.05868 ± 0.00065 6.772 ± 0.072 % 90.306 ± 0.116 %
71
+
72
+ ====== Perplexity statistics ======
73
+ Mean PPL(Q) : 7.509838 ± 0.078504
74
+ Mean PPL(base) : 7.509043 ± 0.079361
75
+ Cor(ln(PPL(Q)), ln(PPL(base))): 98.70%
76
+ Mean ln(PPL(Q)/PPL(base)) : 0.000106 ± 0.001698
77
+ Mean PPL(Q)/PPL(base) : 1.000106 ± 0.001698
78
+ Mean PPL(Q)-PPL(base) : 0.000795 ± 0.012750
79
+
80
+ ====== KL divergence statistics ======
81
+ Mean KLD: 0.058676 ± 0.000655
82
+ Maximum KLD: 7.914699
83
+ 99.9% KLD: 2.201383
84
+ 99.0% KLD: 0.616340
85
+ 95.0% KLD: 0.212113
86
+ 90.0% KLD: 0.124513
87
+ Median KLD: 0.022391
88
+ 10.0% KLD: 0.000394
89
+ 5.0% KLD: 0.000121
90
+ 1.0% KLD: 0.000005
91
+ 0.1% KLD: -0.000170
92
+ Minimum KLD: -0.000789
93
+
94
+ ====== Token probability statistics ======
95
+ Mean Δp: -0.653 ± 0.026 %
96
+ Maximum Δp: 94.651%
97
+ 99.9% Δp: 48.669%
98
+ 99.0% Δp: 18.822%
99
+ 95.0% Δp: 6.933%
100
+ 90.0% Δp: 3.611%
101
+ 75.0% Δp: 0.457%
102
+ Median Δp: -0.021%
103
+ 25.0% Δp: -1.462%
104
+ 10.0% Δp: -5.907%
105
+ 5.0% Δp: -10.087%
106
+ 1.0% Δp: -23.148%
107
+ 0.1% Δp: -51.657%
108
+ Minimum Δp: -95.869%
109
+ RMS Δp : 6.772 ± 0.072 %
110
+ Same top p: 90.306 ± 0.116 %
111
+
appendix/measurements/kld_SHALLOW-21G-ICE.txt ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.05.052.902 I
2
+ 0.05.053.047 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
3
+ 0.05.375.423 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
4
+ 0.07.258.596 I kl_divergence: 1.88 seconds per pass - ETA 2.00 minutes
5
+
6
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
7
+ 1 5.7187 ± 0.4467 -0.02398 ± 0.01286 0.05730 ± 0.00449 7.091 ± 0.508 % 91.496 ± 0.873 %
8
+ 2 7.1994 ± 0.4166 -0.01507 ± 0.00876 0.04875 ± 0.00267 6.045 ± 0.324 % 91.300 ± 0.623 %
9
+ 3 8.1447 ± 0.3993 -0.00664 ± 0.00720 0.04609 ± 0.00263 5.757 ± 0.283 % 91.724 ± 0.497 %
10
+ 4 9.1860 ± 0.4158 -0.01460 ± 0.00661 0.05211 ± 0.00337 6.261 ± 0.307 % 91.471 ± 0.437 %
11
+ 5 8.5931 ± 0.3425 -0.01378 ± 0.00587 0.05031 ± 0.00278 6.182 ± 0.276 % 91.437 ± 0.391 %
12
+ 6 7.2779 ± 0.2560 -0.01727 ± 0.00531 0.05000 ± 0.00245 6.267 ± 0.240 % 91.447 ± 0.357 %
13
+ 7 6.6583 ± 0.2131 -0.02069 ± 0.00498 0.05019 ± 0.00222 6.442 ± 0.228 % 91.524 ± 0.329 %
14
+ 8 6.4719 ± 0.1912 -0.01958 ± 0.00450 0.04837 ± 0.00196 6.274 ± 0.207 % 91.483 ± 0.309 %
15
+ 9 6.8339 ± 0.1917 -0.02347 ± 0.00423 0.04818 ± 0.00176 6.169 ± 0.189 % 91.257 ± 0.294 %
16
+ 10 6.9068 ± 0.1840 -0.02200 ± 0.00393 0.04623 ± 0.00160 6.004 ± 0.177 % 91.457 ± 0.276 %
17
+ 11 6.9259 ± 0.1750 -0.01976 ± 0.00372 0.04449 ± 0.00146 5.863 ± 0.166 % 91.567 ± 0.262 %
18
+ 12 7.1884 ± 0.1750 -0.01882 ± 0.00347 0.04272 ± 0.00134 5.714 ± 0.157 % 91.764 ± 0.248 %
19
+ 13 7.2835 ± 0.1709 -0.02094 ± 0.00333 0.04262 ± 0.00127 5.743 ± 0.151 % 91.781 ± 0.238 %
20
+ 14 7.3297 ± 0.1654 -0.02096 ± 0.00315 0.04144 ± 0.00118 5.624 ± 0.144 % 91.824 ± 0.229 %
21
+ 15 7.3780 ± 0.1609 -0.02218 ± 0.00304 0.04086 ± 0.00111 5.573 ± 0.136 % 91.834 ± 0.221 %
22
+ 16 7.5547 ± 0.1597 -0.02327 ± 0.00293 0.04049 ± 0.00105 5.534 ± 0.133 % 91.850 ± 0.214 %
23
+ 17 7.5715 ± 0.1545 -0.02200 ± 0.00280 0.03978 ± 0.00100 5.454 ± 0.128 % 91.881 ± 0.207 %
24
+ 18 7.6577 ± 0.1517 -0.02091 ± 0.00269 0.03940 ± 0.00095 5.417 ± 0.122 % 91.941 ± 0.201 %
25
+ 19 7.5722 ± 0.1463 -0.02274 ± 0.00262 0.03947 ± 0.00092 5.393 ± 0.117 % 91.938 ± 0.195 %
26
+ 20 7.3024 ± 0.1368 -0.02662 ± 0.00273 0.04421 ± 0.00103 5.971 ± 0.128 % 91.774 ± 0.192 %
27
+ 21 7.3335 ± 0.1340 -0.02781 ± 0.00266 0.04475 ± 0.00101 6.014 ± 0.126 % 91.719 ± 0.188 %
28
+ 22 7.4193 ± 0.1331 -0.02621 ± 0.00258 0.04431 ± 0.00096 5.981 ± 0.122 % 91.802 ± 0.183 %
29
+ 23 7.4548 ± 0.1306 -0.02494 ± 0.00251 0.04384 ± 0.00093 5.917 ± 0.118 % 91.793 ± 0.179 %
30
+ 24 7.5356 ± 0.1295 -0.02775 ± 0.00248 0.04429 ± 0.00090 5.932 ± 0.116 % 91.760 ± 0.175 %
31
+ 25 7.5857 ± 0.1279 -0.02954 ± 0.00244 0.04457 ± 0.00087 5.898 ± 0.112 % 91.644 ± 0.173 %
32
+ 26 7.5398 ± 0.1244 -0.02898 ± 0.00238 0.04421 ± 0.00084 5.864 ± 0.109 % 91.665 ± 0.169 %
33
+ 27 7.6883 ± 0.1248 -0.02917 ± 0.00232 0.04381 ± 0.00082 5.819 ± 0.106 % 91.630 ± 0.167 %
34
+ 28 7.7645 ± 0.1239 -0.02905 ± 0.00226 0.04333 ± 0.00079 5.770 ± 0.104 % 91.656 ± 0.163 %
35
+ 29 7.7683 ± 0.1220 -0.02812 ± 0.00223 0.04334 ± 0.00077 5.773 ± 0.101 % 91.728 ± 0.160 %
36
+ 30 7.7975 ± 0.1208 -0.03040 ± 0.00220 0.04341 ± 0.00075 5.768 ± 0.099 % 91.724 ± 0.157 %
37
+ 31 7.6986 ± 0.1170 -0.02927 ± 0.00215 0.04296 ± 0.00073 5.743 ± 0.097 % 91.789 ± 0.154 %
38
+ 32 7.5790 ± 0.1133 -0.02730 ± 0.00212 0.04288 ± 0.00072 5.793 ± 0.096 % 91.810 ± 0.152 %
39
+ 33 7.5742 ± 0.1117 -0.02584 ± 0.00209 0.04298 ± 0.00073 5.784 ± 0.094 % 91.810 ± 0.149 %
40
+ 34 7.5555 ± 0.1096 -0.02522 ± 0.00204 0.04256 ± 0.00071 5.760 ± 0.092 % 91.820 ± 0.147 %
41
+ 35 7.5567 ± 0.1080 -0.02485 ± 0.00200 0.04222 ± 0.00070 5.733 ± 0.091 % 91.864 ± 0.144 %
42
+ 36 7.5638 ± 0.1067 -0.02460 ± 0.00197 0.04191 ± 0.00068 5.709 ± 0.089 % 91.919 ± 0.142 %
43
+ 37 7.4249 ± 0.1028 -0.02395 ± 0.00192 0.04132 ± 0.00066 5.677 ± 0.088 % 91.984 ± 0.140 %
44
+ 38 7.3176 ± 0.0995 -0.02356 ± 0.00189 0.04106 ± 0.00065 5.660 ± 0.086 % 92.015 ± 0.137 %
45
+ 39 7.2091 ± 0.0964 -0.02261 ± 0.00185 0.04066 ± 0.00064 5.636 ± 0.085 % 92.062 ± 0.135 %
46
+ 40 7.0864 ± 0.0930 -0.02202 ± 0.00181 0.04022 ± 0.00062 5.623 ± 0.083 % 92.094 ± 0.133 %
47
+ 41 7.0101 ± 0.0905 -0.02145 ± 0.00178 0.03991 ± 0.00061 5.615 ± 0.082 % 92.096 ± 0.132 %
48
+ 42 7.0425 ± 0.0899 -0.02100 ± 0.00175 0.03975 ± 0.00060 5.592 ± 0.081 % 92.070 ± 0.130 %
49
+ 43 7.2332 ± 0.0921 -0.02176 ± 0.00173 0.03974 ± 0.00059 5.579 ± 0.080 % 91.998 ± 0.129 %
50
+ 44 7.3712 ± 0.0933 -0.02159 ± 0.00171 0.03970 ± 0.00059 5.548 ± 0.079 % 91.989 ± 0.128 %
51
+ 45 7.3470 ± 0.0918 -0.02091 ± 0.00171 0.04053 ± 0.00058 5.598 ± 0.078 % 91.928 ± 0.127 %
52
+ 46 7.3511 ± 0.0908 -0.02096 ± 0.00169 0.04095 ± 0.00058 5.639 ± 0.078 % 91.901 ± 0.126 %
53
+ 47 7.4146 ± 0.0908 -0.02117 ± 0.00167 0.04121 ± 0.00059 5.655 ± 0.077 % 91.889 ± 0.125 %
54
+ 48 7.4139 ± 0.0897 -0.02067 ± 0.00165 0.04098 ± 0.00058 5.632 ± 0.076 % 91.864 ± 0.123 %
55
+ 49 7.4637 ± 0.0895 -0.02055 ± 0.00163 0.04098 ± 0.00058 5.615 ± 0.075 % 91.835 ± 0.122 %
56
+ 50 7.4193 ± 0.0879 -0.02026 ± 0.00161 0.04066 ± 0.00057 5.595 ± 0.074 % 91.891 ± 0.121 %
57
+ 51 7.4527 ± 0.0874 -0.02000 ± 0.00158 0.04027 ± 0.00056 5.557 ± 0.073 % 91.925 ± 0.119 %
58
+ 52 7.3940 ± 0.0857 -0.01951 ± 0.00156 0.04001 ± 0.00055 5.553 ± 0.072 % 91.941 ± 0.118 %
59
+ 53 7.3647 ± 0.0845 -0.01907 ± 0.00154 0.03979 ± 0.00054 5.542 ± 0.072 % 91.946 ± 0.117 %
60
+ 54 7.3242 ± 0.0830 -0.01900 ± 0.00153 0.03953 ± 0.00053 5.531 ± 0.071 % 91.954 ± 0.116 %
61
+ 55 7.2811 ± 0.0817 -0.01987 ± 0.00152 0.03956 ± 0.00053 5.541 ± 0.070 % 91.956 ± 0.115 %
62
+ 56 7.2765 ± 0.0809 -0.01963 ± 0.00150 0.03942 ± 0.00052 5.530 ± 0.069 % 91.981 ± 0.113 %
63
+ 57 7.2012 ± 0.0792 -0.01930 ± 0.00149 0.03919 ± 0.00051 5.507 ± 0.068 % 92.024 ± 0.112 %
64
+ 58 7.2425 ± 0.0792 -0.01838 ± 0.00149 0.03954 ± 0.00051 5.530 ± 0.067 % 92.001 ± 0.111 %
65
+ 59 7.2161 ± 0.0782 -0.01805 ± 0.00148 0.03986 ± 0.00051 5.564 ± 0.067 % 91.991 ± 0.110 %
66
+ 60 7.2017 ± 0.0774 -0.01768 ± 0.00146 0.03970 ± 0.00050 5.551 ± 0.066 % 92.022 ± 0.109 %
67
+ 61 7.2691 ± 0.0778 -0.01921 ± 0.00147 0.04039 ± 0.00053 5.600 ± 0.067 % 91.973 ± 0.109 %
68
+ 62 7.2945 ± 0.0774 -0.01874 ± 0.00145 0.04018 ± 0.00052 5.576 ± 0.066 % 91.995 ± 0.108 %
69
+ 63 7.3445 ± 0.0774 -0.01795 ± 0.00144 0.04000 ± 0.00051 5.554 ± 0.066 % 92.001 ± 0.107 %
70
+ 64 7.3762 ± 0.0772 -0.01785 ± 0.00142 0.03981 ± 0.00051 5.535 ± 0.065 % 92.012 ± 0.106 %
71
+
72
+ ====== Perplexity statistics ======
73
+ Mean PPL(Q) : 7.376193 ± 0.077216
74
+ Mean PPL(base) : 7.509043 ± 0.079361
75
+ Cor(ln(PPL(Q)), ln(PPL(base))): 99.09%
76
+ Mean ln(PPL(Q)/PPL(base)) : -0.017850 ± 0.001420
77
+ Mean PPL(Q)/PPL(base) : 0.982308 ± 0.001395
78
+ Mean PPL(Q)-PPL(base) : -0.132850 ± 0.010757
79
+
80
+ ====== KL divergence statistics ======
81
+ Mean KLD: 0.039814 ± 0.000507
82
+ Maximum KLD: 7.651754
83
+ 99.9% KLD: 1.589447
84
+ 99.0% KLD: 0.412079
85
+ 95.0% KLD: 0.143817
86
+ 90.0% KLD: 0.085509
87
+ Median KLD: 0.013917
88
+ 10.0% KLD: 0.000220
89
+ 5.0% KLD: 0.000059
90
+ 1.0% KLD: -0.000000
91
+ 0.1% KLD: -0.000154
92
+ Minimum KLD: -0.000922
93
+
94
+ ====== Token probability statistics ======
95
+ Mean Δp: -0.062 ± 0.022 %
96
+ Maximum Δp: 88.205%
97
+ 99.9% Δp: 43.914%
98
+ 99.0% Δp: 17.167%
99
+ 95.0% Δp: 6.679%
100
+ 90.0% Δp: 3.536%
101
+ 75.0% Δp: 0.613%
102
+ Median Δp: -0.001%
103
+ 25.0% Δp: -0.769%
104
+ 10.0% Δp: -3.899%
105
+ 5.0% Δp: -6.876%
106
+ 1.0% Δp: -16.580%
107
+ 0.1% Δp: -43.277%
108
+ Minimum Δp: -82.493%
109
+ RMS Δp : 5.535 ± 0.065 %
110
+ Same top p: 92.012 ± 0.106 %
111
+
appendix/measurements/kld_SHALLOW-23G-ICE.txt ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.05.237.699 I
2
+ 0.05.237.836 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
3
+ 0.05.547.049 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
4
+ 0.07.354.412 I kl_divergence: 1.81 seconds per pass - ETA 1.92 minutes
5
+
6
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
7
+ 1 5.8750 ± 0.4631 0.00298 ± 0.01120 0.04176 ± 0.00334 6.156 ± 0.447 % 92.766 ± 0.810 %
8
+ 2 7.3620 ± 0.4290 0.00727 ± 0.00810 0.03874 ± 0.00206 5.421 ± 0.292 % 91.789 ± 0.607 %
9
+ 3 8.3347 ± 0.4119 0.01643 ± 0.00637 0.03639 ± 0.00261 5.195 ± 0.332 % 92.147 ± 0.486 %
10
+ 4 9.4556 ± 0.4321 0.01432 ± 0.00602 0.04304 ± 0.00328 5.649 ± 0.317 % 92.058 ± 0.423 %
11
+ 5 8.8320 ± 0.3554 0.01365 ± 0.00527 0.04060 ± 0.00266 5.441 ± 0.270 % 92.317 ± 0.372 %
12
+ 6 7.4854 ± 0.2659 0.01085 ± 0.00474 0.03951 ± 0.00228 5.542 ± 0.248 % 92.212 ± 0.342 %
13
+ 7 6.8518 ± 0.2216 0.00795 ± 0.00443 0.03993 ± 0.00204 5.738 ± 0.232 % 92.417 ± 0.313 %
14
+ 8 6.6513 ± 0.1985 0.00776 ± 0.00404 0.03851 ± 0.00180 5.620 ± 0.210 % 92.559 ± 0.290 %
15
+ 9 7.0608 ± 0.2005 0.00919 ± 0.00381 0.03937 ± 0.00168 5.553 ± 0.191 % 92.310 ± 0.278 %
16
+ 10 7.1294 ± 0.1924 0.00973 ± 0.00353 0.03763 ± 0.00152 5.450 ± 0.182 % 92.356 ± 0.263 %
17
+ 11 7.1238 ± 0.1819 0.00841 ± 0.00334 0.03628 ± 0.00139 5.347 ± 0.170 % 92.598 ± 0.247 %
18
+ 12 7.3873 ± 0.1818 0.00848 ± 0.00315 0.03516 ± 0.00129 5.248 ± 0.159 % 92.669 ± 0.235 %
19
+ 13 7.4601 ± 0.1766 0.00302 ± 0.00304 0.03513 ± 0.00120 5.265 ± 0.151 % 92.631 ± 0.227 %
20
+ 14 7.5034 ± 0.1707 0.00245 ± 0.00287 0.03400 ± 0.00112 5.141 ± 0.144 % 92.655 ± 0.218 %
21
+ 15 7.5494 ± 0.1660 0.00077 ± 0.00274 0.03380 ± 0.00106 5.115 ± 0.137 % 92.708 ± 0.210 %
22
+ 16 7.7246 ± 0.1645 -0.00102 ± 0.00267 0.03350 ± 0.00100 5.045 ± 0.131 % 92.797 ± 0.202 %
23
+ 17 7.7325 ± 0.1589 -0.00096 ± 0.00254 0.03270 ± 0.00094 4.964 ± 0.125 % 92.853 ± 0.195 %
24
+ 18 7.8068 ± 0.1556 -0.00162 ± 0.00246 0.03242 ± 0.00090 4.933 ± 0.120 % 92.837 ± 0.190 %
25
+ 19 7.7138 ± 0.1499 -0.00421 ± 0.00241 0.03265 ± 0.00086 4.929 ± 0.115 % 92.782 ± 0.186 %
26
+ 20 7.4622 ± 0.1409 -0.00498 ± 0.00247 0.03619 ± 0.00104 5.392 ± 0.124 % 92.542 ± 0.184 %
27
+ 21 7.5061 ± 0.1383 -0.00454 ± 0.00240 0.03664 ± 0.00102 5.412 ± 0.122 % 92.417 ± 0.181 %
28
+ 22 7.5743 ± 0.1368 -0.00553 ± 0.00233 0.03625 ± 0.00097 5.369 ± 0.117 % 92.442 ± 0.176 %
29
+ 23 7.6062 ± 0.1341 -0.00483 ± 0.00226 0.03582 ± 0.00093 5.309 ± 0.114 % 92.490 ± 0.172 %
30
+ 24 7.6949 ± 0.1331 -0.00683 ± 0.00222 0.03605 ± 0.00090 5.307 ± 0.111 % 92.485 ± 0.168 %
31
+ 25 7.7564 ± 0.1318 -0.00729 ± 0.00216 0.03579 ± 0.00087 5.262 ± 0.108 % 92.454 ± 0.165 %
32
+ 26 7.7098 ± 0.1282 -0.00668 ± 0.00210 0.03543 ± 0.00084 5.230 ± 0.105 % 92.492 ± 0.162 %
33
+ 27 7.8702 ± 0.1288 -0.00579 ± 0.00204 0.03479 ± 0.00081 5.168 ± 0.102 % 92.498 ± 0.159 %
34
+ 28 7.9397 ± 0.1277 -0.00674 ± 0.00199 0.03460 ± 0.00078 5.142 ± 0.100 % 92.519 ± 0.155 %
35
+ 29 7.9374 ± 0.1257 -0.00659 ± 0.00196 0.03463 ± 0.00076 5.159 ± 0.097 % 92.564 ± 0.152 %
36
+ 30 7.9846 ± 0.1248 -0.00670 ± 0.00194 0.03472 ± 0.00076 5.161 ± 0.095 % 92.577 ± 0.150 %
37
+ 31 7.8743 ± 0.1208 -0.00672 ± 0.00190 0.03438 ± 0.00074 5.133 ± 0.093 % 92.612 ± 0.147 %
38
+ 32 7.7400 ± 0.1166 -0.00629 ± 0.00187 0.03427 ± 0.00072 5.166 ± 0.091 % 92.675 ± 0.144 %
39
+ 33 7.7294 ± 0.1149 -0.00555 ± 0.00185 0.03445 ± 0.00075 5.167 ± 0.089 % 92.669 ± 0.142 %
40
+ 34 7.7030 ± 0.1126 -0.00589 ± 0.00181 0.03420 ± 0.00073 5.161 ± 0.088 % 92.660 ± 0.140 %
41
+ 35 7.6976 ± 0.1108 -0.00638 ± 0.00177 0.03394 ± 0.00071 5.135 ± 0.086 % 92.677 ± 0.138 %
42
+ 36 7.7021 ± 0.1094 -0.00647 ± 0.00174 0.03362 ± 0.00069 5.099 ± 0.084 % 92.737 ± 0.135 %
43
+ 37 7.5595 ± 0.1054 -0.00600 ± 0.00170 0.03321 ± 0.00067 5.070 ± 0.083 % 92.822 ± 0.133 %
44
+ 38 7.4468 ± 0.1019 -0.00606 ± 0.00167 0.03298 ± 0.00066 5.056 ± 0.082 % 92.874 ± 0.130 %
45
+ 39 7.3310 ± 0.0987 -0.00585 ± 0.00164 0.03260 ± 0.00064 5.028 ± 0.080 % 92.902 ± 0.129 %
46
+ 40 7.2031 ± 0.0952 -0.00567 ± 0.00161 0.03223 ± 0.00063 5.014 ± 0.079 % 92.945 ± 0.127 %
47
+ 41 7.1238 ± 0.0926 -0.00537 ± 0.00159 0.03195 ± 0.00061 5.002 ± 0.077 % 92.969 ± 0.125 %
48
+ 42 7.1531 ± 0.0920 -0.00541 ± 0.00156 0.03181 ± 0.00060 4.977 ± 0.076 % 92.974 ± 0.123 %
49
+ 43 7.3515 ± 0.0943 -0.00554 ± 0.00155 0.03181 ± 0.00059 4.955 ± 0.075 % 92.926 ± 0.122 %
50
+ 44 7.4877 ± 0.0954 -0.00591 ± 0.00153 0.03191 ± 0.00059 4.936 ± 0.074 % 92.909 ± 0.121 %
51
+ 45 7.4600 ± 0.0938 -0.00564 ± 0.00152 0.03275 ± 0.00059 5.005 ± 0.073 % 92.825 ± 0.120 %
52
+ 46 7.4668 ± 0.0928 -0.00535 ± 0.00151 0.03291 ± 0.00058 5.018 ± 0.072 % 92.815 ± 0.119 %
53
+ 47 7.5311 ± 0.0928 -0.00559 ± 0.00150 0.03309 ± 0.00058 5.043 ± 0.074 % 92.825 ± 0.118 %
54
+ 48 7.5285 ± 0.0917 -0.00534 ± 0.00147 0.03293 ± 0.00057 5.034 ± 0.073 % 92.817 ± 0.117 %
55
+ 49 7.5763 ± 0.0914 -0.00557 ± 0.00146 0.03304 ± 0.00057 5.039 ± 0.073 % 92.738 ± 0.116 %
56
+ 50 7.5310 ± 0.0898 -0.00533 ± 0.00144 0.03279 ± 0.00056 5.013 ± 0.072 % 92.770 ± 0.115 %
57
+ 51 7.5658 ± 0.0893 -0.00495 ± 0.00142 0.03248 ± 0.00055 4.979 ± 0.071 % 92.803 ± 0.113 %
58
+ 52 7.5013 ± 0.0875 -0.00510 ± 0.00141 0.03241 ± 0.00055 4.981 ± 0.070 % 92.825 ± 0.112 %
59
+ 53 7.4720 ± 0.0862 -0.00461 ± 0.00139 0.03215 ± 0.00054 4.964 ± 0.069 % 92.824 ± 0.111 %
60
+ 54 7.4284 ± 0.0847 -0.00488 ± 0.00137 0.03208 ± 0.00053 4.975 ± 0.069 % 92.815 ± 0.110 %
61
+ 55 7.3884 ± 0.0834 -0.00525 ± 0.00136 0.03198 ± 0.00052 4.970 ± 0.068 % 92.820 ± 0.109 %
62
+ 56 7.3790 ± 0.0826 -0.00565 ± 0.00135 0.03193 ± 0.00052 4.962 ± 0.067 % 92.827 ± 0.108 %
63
+ 57 7.3003 ± 0.0807 -0.00563 ± 0.00133 0.03178 ± 0.00051 4.945 ± 0.066 % 92.856 ± 0.107 %
64
+ 58 7.3455 ± 0.0808 -0.00426 ± 0.00134 0.03247 ± 0.00052 4.976 ± 0.065 % 92.793 ± 0.106 %
65
+ 59 7.3188 ± 0.0797 -0.00391 ± 0.00133 0.03268 ± 0.00051 5.008 ± 0.065 % 92.801 ± 0.105 %
66
+ 60 7.3013 ± 0.0789 -0.00394 ± 0.00131 0.03250 ± 0.00051 4.998 ± 0.064 % 92.833 ± 0.104 %
67
+ 61 7.3721 ± 0.0793 -0.00514 ± 0.00132 0.03303 ± 0.00051 5.059 ± 0.066 % 92.797 ± 0.103 %
68
+ 62 7.3946 ± 0.0789 -0.00511 ± 0.00130 0.03287 ± 0.00051 5.037 ± 0.065 % 92.809 ± 0.103 %
69
+ 63 7.4393 ± 0.0788 -0.00513 ± 0.00129 0.03266 ± 0.00050 5.017 ± 0.064 % 92.811 ± 0.102 %
70
+ 64 7.4707 ± 0.0786 -0.00512 ± 0.00128 0.03254 ± 0.00049 4.995 ± 0.064 % 92.835 ± 0.101 %
71
+
72
+ ====== Perplexity statistics ======
73
+ Mean PPL(Q) : 7.470682 ± 0.078596
74
+ Mean PPL(base) : 7.509043 ± 0.079361
75
+ Cor(ln(PPL(Q)), ln(PPL(base))): 99.27%
76
+ Mean ln(PPL(Q)/PPL(base)) : -0.005122 ± 0.001276
77
+ Mean PPL(Q)/PPL(base) : 0.994891 ± 0.001269
78
+ Mean PPL(Q)-PPL(base) : -0.038361 ± 0.009580
79
+
80
+ ====== KL divergence statistics ======
81
+ Mean KLD: 0.032536 ± 0.000493
82
+ Maximum KLD: 9.080469
83
+ 99.9% KLD: 1.318512
84
+ 99.0% KLD: 0.331990
85
+ 95.0% KLD: 0.113636
86
+ 90.0% KLD: 0.068610
87
+ Median KLD: 0.011154
88
+ 10.0% KLD: 0.000183
89
+ 5.0% KLD: 0.000051
90
+ 1.0% KLD: 0.000001
91
+ 0.1% KLD: -0.000143
92
+ Minimum KLD: -0.000774
93
+
94
+ ====== Token probability statistics ======
95
+ Mean Δp: -0.122 ± 0.020 %
96
+ Maximum Δp: 90.050%
97
+ 99.9% Δp: 38.701%
98
+ 99.0% Δp: 14.926%
99
+ 95.0% Δp: 5.790%
100
+ 90.0% Δp: 3.062%
101
+ 75.0% Δp: 0.512%
102
+ Median Δp: -0.002%
103
+ 25.0% Δp: -0.716%
104
+ 10.0% Δp: -3.562%
105
+ 5.0% Δp: -6.422%
106
+ 1.0% Δp: -15.571%
107
+ 0.1% Δp: -38.825%
108
+ Minimum Δp: -91.432%
109
+ RMS Δp : 4.995 ± 0.064 %
110
+ Same top p: 92.835 ± 0.101 %
111
+
appendix/measurements/kld_SHALLOW-25G-ICE.txt ADDED
@@ -0,0 +1,111 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.05.931.717 I
2
+ 0.05.931.865 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
3
+ 0.06.304.675 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
4
+ 0.08.330.832 I kl_divergence: 2.03 seconds per pass - ETA 2.15 minutes
5
+
6
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
7
+ 1 5.7739 ± 0.4525 -0.01437 ± 0.01007 0.02728 ± 0.00224 4.890 ± 0.409 % 94.330 ± 0.723 %
8
+ 2 7.2217 ± 0.4183 -0.01197 ± 0.00735 0.02713 ± 0.00138 4.572 ± 0.263 % 93.891 ± 0.530 %
9
+ 3 8.1465 ± 0.3994 -0.00641 ± 0.00600 0.02747 ± 0.00232 4.585 ± 0.340 % 93.939 ± 0.431 %
10
+ 4 9.2937 ± 0.4232 -0.00295 ± 0.00569 0.03424 ± 0.00280 5.137 ± 0.326 % 93.695 ± 0.380 %
11
+ 5 8.6837 ± 0.3478 -0.00329 ± 0.00489 0.03231 ± 0.00226 4.994 ± 0.279 % 93.744 ± 0.339 %
12
+ 6 7.3875 ± 0.2614 -0.00233 ± 0.00441 0.03176 ± 0.00195 5.139 ± 0.261 % 93.516 ± 0.314 %
13
+ 7 6.7593 ± 0.2177 -0.00563 ± 0.00416 0.03245 ± 0.00174 5.339 ± 0.232 % 93.451 ± 0.292 %
14
+ 8 6.5700 ± 0.1952 -0.00453 ± 0.00375 0.03102 ± 0.00153 5.216 ± 0.210 % 93.524 ± 0.272 %
15
+ 9 6.9545 ± 0.1965 -0.00598 ± 0.00357 0.03163 ± 0.00143 5.122 ± 0.192 % 93.331 ± 0.260 %
16
+ 10 7.0245 ± 0.1886 -0.00510 ± 0.00331 0.03035 ± 0.00130 5.029 ± 0.181 % 93.451 ± 0.245 %
17
+ 11 7.0327 ± 0.1789 -0.00446 ± 0.00316 0.02958 ± 0.00119 4.999 ± 0.173 % 93.486 ± 0.233 %
18
+ 12 7.2961 ± 0.1788 -0.00394 ± 0.00297 0.02883 ± 0.00110 4.891 ± 0.163 % 93.573 ± 0.221 %
19
+ 13 7.3748 ± 0.1740 -0.00849 ± 0.00287 0.02906 ± 0.00104 4.913 ± 0.153 % 93.518 ± 0.214 %
20
+ 14 7.4256 ± 0.1684 -0.00797 ± 0.00271 0.02828 ± 0.00097 4.831 ± 0.145 % 93.493 ± 0.206 %
21
+ 15 7.4745 ± 0.1638 -0.00919 ± 0.00260 0.02824 ± 0.00091 4.813 ± 0.137 % 93.470 ± 0.199 %
22
+ 16 7.6545 ± 0.1625 -0.01014 ± 0.00252 0.02806 ± 0.00086 4.751 ± 0.131 % 93.518 ± 0.192 %
23
+ 17 7.6653 ± 0.1571 -0.00968 ± 0.00240 0.02750 ± 0.00082 4.680 ± 0.125 % 93.537 ± 0.186 %
24
+ 18 7.7474 ± 0.1541 -0.00927 ± 0.00231 0.02714 ± 0.00077 4.632 ± 0.120 % 93.586 ± 0.181 %
25
+ 19 7.6547 ± 0.1484 -0.01190 ± 0.00227 0.02764 ± 0.00075 4.647 ± 0.115 % 93.471 ± 0.177 %
26
+ 20 7.4003 ± 0.1394 -0.01330 ± 0.00231 0.03094 ± 0.00099 4.991 ± 0.120 % 93.270 ± 0.175 %
27
+ 21 7.4460 ± 0.1368 -0.01258 ± 0.00224 0.03140 ± 0.00097 5.022 ± 0.119 % 93.209 ± 0.172 %
28
+ 22 7.5175 ± 0.1355 -0.01305 ± 0.00218 0.03110 ± 0.00093 4.981 ± 0.114 % 93.188 ± 0.168 %
29
+ 23 7.5511 ± 0.1329 -0.01210 ± 0.00211 0.03067 ± 0.00089 4.915 ± 0.111 % 93.200 ± 0.164 %
30
+ 24 7.6362 ± 0.1318 -0.01448 ± 0.00208 0.03113 ± 0.00086 4.960 ± 0.111 % 93.141 ± 0.161 %
31
+ 25 7.6993 ± 0.1306 -0.01468 ± 0.00202 0.03096 ± 0.00083 4.915 ± 0.107 % 93.095 ± 0.159 %
32
+ 26 7.6553 ± 0.1270 -0.01377 ± 0.00197 0.03062 ± 0.00080 4.877 ± 0.105 % 93.112 ± 0.155 %
33
+ 27 7.8166 ± 0.1277 -0.01262 ± 0.00191 0.03003 ± 0.00077 4.817 ± 0.102 % 93.121 ± 0.152 %
34
+ 28 7.8878 ± 0.1267 -0.01329 ± 0.00187 0.03002 ± 0.00075 4.806 ± 0.100 % 93.081 ± 0.150 %
35
+ 29 7.8846 ± 0.1246 -0.01326 ± 0.00184 0.02987 ± 0.00073 4.814 ± 0.097 % 93.097 ± 0.147 %
36
+ 30 7.9328 ± 0.1238 -0.01321 ± 0.00182 0.03006 ± 0.00073 4.828 ± 0.095 % 93.122 ± 0.144 %
37
+ 31 7.8258 ± 0.1198 -0.01289 ± 0.00178 0.02974 ± 0.00071 4.814 ± 0.094 % 93.148 ± 0.142 %
38
+ 32 7.6896 ± 0.1157 -0.01282 ± 0.00175 0.02974 ± 0.00069 4.835 ± 0.092 % 93.200 ± 0.139 %
39
+ 33 7.6796 ± 0.1140 -0.01201 ± 0.00172 0.02986 ± 0.00071 4.823 ± 0.090 % 93.184 ± 0.137 %
40
+ 34 7.6539 ± 0.1117 -0.01227 ± 0.00169 0.02967 ± 0.00069 4.821 ± 0.088 % 93.198 ± 0.135 %
41
+ 35 7.6515 ± 0.1100 -0.01239 ± 0.00165 0.02946 ± 0.00067 4.802 ± 0.087 % 93.216 ± 0.133 %
42
+ 36 7.6531 ± 0.1085 -0.01285 ± 0.00162 0.02928 ± 0.00066 4.788 ± 0.085 % 93.258 ± 0.131 %
43
+ 37 7.5110 ± 0.1045 -0.01242 ± 0.00159 0.02887 ± 0.00064 4.767 ± 0.084 % 93.316 ± 0.128 %
44
+ 38 7.4013 ± 0.1012 -0.01219 ± 0.00156 0.02862 ± 0.00062 4.749 ± 0.083 % 93.340 ± 0.126 %
45
+ 39 7.2877 ± 0.0979 -0.01177 ± 0.00153 0.02830 ± 0.00061 4.732 ± 0.081 % 93.378 ± 0.124 %
46
+ 40 7.1606 ± 0.0945 -0.01159 ± 0.00150 0.02794 ± 0.00059 4.713 ± 0.079 % 93.419 ± 0.123 %
47
+ 41 7.0829 ± 0.0919 -0.01113 ± 0.00148 0.02763 ± 0.00058 4.691 ± 0.078 % 93.448 ± 0.121 %
48
+ 42 7.1135 ± 0.0913 -0.01096 ± 0.00145 0.02752 ± 0.00057 4.677 ± 0.077 % 93.462 ± 0.119 %
49
+ 43 7.3094 ± 0.0936 -0.01128 ± 0.00144 0.02760 ± 0.00056 4.667 ± 0.076 % 93.421 ± 0.118 %
50
+ 44 7.4410 ± 0.0947 -0.01217 ± 0.00143 0.02778 ± 0.00057 4.651 ± 0.074 % 93.388 ± 0.117 %
51
+ 45 7.4161 ± 0.0931 -0.01155 ± 0.00142 0.02838 ± 0.00056 4.702 ± 0.073 % 93.329 ± 0.116 %
52
+ 46 7.4236 ± 0.0921 -0.01115 ± 0.00140 0.02848 ± 0.00055 4.714 ± 0.072 % 93.332 ± 0.115 %
53
+ 47 7.4851 ± 0.0921 -0.01171 ± 0.00139 0.02864 ± 0.00055 4.730 ± 0.073 % 93.320 ± 0.114 %
54
+ 48 7.4839 ± 0.0909 -0.01127 ± 0.00137 0.02849 ± 0.00054 4.717 ± 0.072 % 93.302 ± 0.113 %
55
+ 49 7.5329 ± 0.0907 -0.01132 ± 0.00136 0.02862 ± 0.00054 4.718 ± 0.071 % 93.267 ± 0.112 %
56
+ 50 7.4882 ± 0.0891 -0.01103 ± 0.00134 0.02839 ± 0.00053 4.693 ± 0.070 % 93.302 ± 0.111 %
57
+ 51 7.5201 ± 0.0886 -0.01100 ± 0.00132 0.02813 ± 0.00052 4.664 ± 0.069 % 93.351 ± 0.109 %
58
+ 52 7.4541 ± 0.0868 -0.01142 ± 0.00130 0.02805 ± 0.00052 4.677 ± 0.069 % 93.349 ± 0.108 %
59
+ 53 7.4245 ± 0.0855 -0.01098 ± 0.00128 0.02786 ± 0.00051 4.663 ± 0.068 % 93.349 ± 0.107 %
60
+ 54 7.3803 ± 0.0840 -0.01137 ± 0.00127 0.02777 ± 0.00050 4.669 ± 0.068 % 93.347 ± 0.106 %
61
+ 55 7.3414 ± 0.0827 -0.01162 ± 0.00126 0.02769 ± 0.00049 4.665 ± 0.067 % 93.351 ± 0.105 %
62
+ 56 7.3326 ± 0.0819 -0.01195 ± 0.00125 0.02764 ± 0.00049 4.657 ± 0.066 % 93.351 ± 0.104 %
63
+ 57 7.2551 ± 0.0801 -0.01184 ± 0.00123 0.02749 ± 0.00048 4.642 ± 0.065 % 93.377 ± 0.103 %
64
+ 58 7.2994 ± 0.0801 -0.01056 ± 0.00124 0.02799 ± 0.00048 4.663 ± 0.064 % 93.339 ± 0.102 %
65
+ 59 7.2727 ± 0.0791 -0.01023 ± 0.00123 0.02811 ± 0.00047 4.674 ± 0.063 % 93.355 ± 0.101 %
66
+ 60 7.2575 ± 0.0783 -0.00996 ± 0.00121 0.02799 ± 0.00047 4.670 ± 0.063 % 93.372 ± 0.100 %
67
+ 61 7.3299 ± 0.0788 -0.01088 ± 0.00123 0.02861 ± 0.00049 4.737 ± 0.066 % 93.321 ± 0.100 %
68
+ 62 7.3529 ± 0.0784 -0.01076 ± 0.00121 0.02848 ± 0.00048 4.718 ± 0.065 % 93.320 ± 0.099 %
69
+ 63 7.3985 ± 0.0783 -0.01063 ± 0.00120 0.02829 ± 0.00047 4.698 ± 0.064 % 93.319 ± 0.098 %
70
+ 64 7.4300 ± 0.0781 -0.01058 ± 0.00119 0.02813 ± 0.00046 4.678 ± 0.064 % 93.347 ± 0.097 %
71
+
72
+ ====== Perplexity statistics ======
73
+ Mean PPL(Q) : 7.429995 ± 0.078071
74
+ Mean PPL(base) : 7.509043 ± 0.079361
75
+ Cor(ln(PPL(Q)), ln(PPL(base))): 99.37%
76
+ Mean ln(PPL(Q)/PPL(base)) : -0.010583 ± 0.001187
77
+ Mean PPL(Q)/PPL(base) : 0.989473 ± 0.001174
78
+ Mean PPL(Q)-PPL(base) : -0.079048 ± 0.008947
79
+
80
+ ====== KL divergence statistics ======
81
+ Mean KLD: 0.028131 ± 0.000465
82
+ Maximum KLD: 7.775503
83
+ 99.9% KLD: 1.158895
84
+ 99.0% KLD: 0.279364
85
+ 95.0% KLD: 0.097497
86
+ 90.0% KLD: 0.058892
87
+ Median KLD: 0.009599
88
+ 10.0% KLD: 0.000162
89
+ 5.0% KLD: 0.000045
90
+ 1.0% KLD: -0.000002
91
+ 0.1% KLD: -0.000138
92
+ Minimum KLD: -0.000465
93
+
94
+ ====== Token probability statistics ======
95
+ Mean Δp: -0.081 ± 0.018 %
96
+ Maximum Δp: 88.574%
97
+ 99.9% Δp: 36.405%
98
+ 99.0% Δp: 14.296%
99
+ 95.0% Δp: 5.492%
100
+ 90.0% Δp: 2.904%
101
+ 75.0% Δp: 0.478%
102
+ Median Δp: -0.002%
103
+ 25.0% Δp: -0.675%
104
+ 10.0% Δp: -3.318%
105
+ 5.0% Δp: -5.794%
106
+ 1.0% Δp: -14.093%
107
+ 0.1% Δp: -35.491%
108
+ Minimum Δp: -95.025%
109
+ RMS Δp : 4.678 ± 0.064 %
110
+ Same top p: 93.347 ± 0.097 %
111
+
appendix/measurements/kld_V2.txt ADDED
@@ -0,0 +1,112 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.17.320.322 I cmn init: llama threadpool init, n_threads = 30
2
+ 0.17.399.393 I
3
+ 0.17.399.582 I system_info: n_threads = 30 (n_threads_batch = 30) / 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
4
+ 0.17.749.316 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1
5
+ 0.34.514.199 I kl_divergence: 16.76 seconds per pass - ETA 17.87 minutes
6
+
7
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
8
+ 1 5.8556 ± 0.4579 -0.00033 ± 0.01243 0.04597 ± 0.00466 6.745 ± 0.539 % 92.766 ± 0.810 %
9
+ 2 7.3205 ± 0.4252 0.00161 ± 0.00799 0.03994 ± 0.00263 5.564 ± 0.337 % 92.669 ± 0.576 %
10
+ 3 8.2603 ± 0.4067 0.00745 ± 0.00650 0.03789 ± 0.00277 5.449 ± 0.365 % 93.060 ± 0.459 %
11
+ 4 9.3600 ± 0.4263 0.00415 ± 0.00601 0.04408 ± 0.00323 5.675 ± 0.344 % 92.815 ± 0.404 %
12
+ 5 8.7404 ± 0.3505 0.00322 ± 0.00533 0.04209 ± 0.00262 5.472 ± 0.292 % 92.708 ± 0.364 %
13
+ 6 7.4182 ± 0.2629 0.00182 ± 0.00478 0.04148 ± 0.00227 5.655 ± 0.259 % 92.555 ± 0.335 %
14
+ 7 6.8036 ± 0.2197 0.00089 ± 0.00439 0.04090 ± 0.00199 5.702 ± 0.228 % 92.599 ± 0.309 %
15
+ 8 6.6056 ± 0.1968 0.00087 ± 0.00403 0.03920 ± 0.00176 5.582 ± 0.207 % 92.620 ± 0.289 %
16
+ 9 7.0203 ± 0.1990 0.00343 ± 0.00381 0.04021 ± 0.00162 5.562 ± 0.187 % 92.332 ± 0.277 %
17
+ 10 7.0867 ± 0.1909 0.00372 ± 0.00352 0.03846 ± 0.00147 5.410 ± 0.174 % 92.434 ± 0.261 %
18
+ 11 7.0841 ± 0.1806 0.00283 ± 0.00335 0.03724 ± 0.00135 5.327 ± 0.163 % 92.544 ± 0.248 %
19
+ 12 7.3491 ± 0.1806 0.00330 ± 0.00315 0.03617 ± 0.00124 5.232 ± 0.153 % 92.587 ± 0.236 %
20
+ 13 7.4453 ± 0.1762 0.00103 ± 0.00304 0.03572 ± 0.00116 5.257 ± 0.150 % 92.608 ± 0.227 %
21
+ 14 7.4843 ± 0.1703 -0.00009 ± 0.00289 0.03492 ± 0.00109 5.178 ± 0.144 % 92.634 ± 0.218 %
22
+ 15 7.5395 ± 0.1658 -0.00054 ± 0.00279 0.03445 ± 0.00102 5.136 ± 0.137 % 92.695 ± 0.210 %
23
+ 16 7.7470 ± 0.1655 0.00187 ± 0.00270 0.03438 ± 0.00097 5.092 ± 0.131 % 92.711 ± 0.203 %
24
+ 17 7.7518 ± 0.1597 0.00153 ± 0.00257 0.03369 ± 0.00092 5.011 ± 0.125 % 92.743 ± 0.197 %
25
+ 18 7.8313 ± 0.1566 0.00151 ± 0.00249 0.03335 ± 0.00087 4.987 ± 0.121 % 92.756 ± 0.191 %
26
+ 19 7.7506 ± 0.1512 0.00055 ± 0.00240 0.03276 ± 0.00083 4.929 ± 0.116 % 92.818 ± 0.185 %
27
+ 20 7.5014 ± 0.1422 0.00027 ± 0.00240 0.03577 ± 0.00096 5.322 ± 0.126 % 92.678 ± 0.182 %
28
+ 21 7.5371 ± 0.1393 -0.00042 ± 0.00235 0.03642 ± 0.00096 5.354 ± 0.125 % 92.557 ± 0.179 %
29
+ 22 7.6252 ± 0.1383 0.00117 ± 0.00229 0.03615 ± 0.00092 5.350 ± 0.122 % 92.633 ± 0.174 %
30
+ 23 7.6521 ± 0.1354 0.00117 ± 0.00222 0.03570 ± 0.00088 5.280 ± 0.118 % 92.673 ± 0.170 %
31
+ 24 7.7413 ± 0.1344 -0.00082 ± 0.00219 0.03606 ± 0.00087 5.335 ± 0.118 % 92.603 ± 0.167 %
32
+ 25 7.8118 ± 0.1333 -0.00017 ± 0.00213 0.03573 ± 0.00083 5.301 ± 0.116 % 92.559 ± 0.164 %
33
+ 26 7.7548 ± 0.1294 -0.00086 ± 0.00209 0.03544 ± 0.00081 5.266 ± 0.112 % 92.590 ± 0.161 %
34
+ 27 7.9028 ± 0.1297 -0.00166 ± 0.00204 0.03518 ± 0.00078 5.227 ± 0.110 % 92.535 ± 0.158 %
35
+ 28 7.9761 ± 0.1287 -0.00216 ± 0.00199 0.03485 ± 0.00075 5.185 ± 0.107 % 92.539 ± 0.155 %
36
+ 29 7.9867 ± 0.1269 -0.00040 ± 0.00198 0.03496 ± 0.00074 5.208 ± 0.104 % 92.520 ± 0.153 %
37
+ 30 8.0130 ± 0.1256 -0.00315 ± 0.00197 0.03534 ± 0.00072 5.214 ± 0.101 % 92.506 ± 0.150 %
38
+ 31 7.8997 ± 0.1214 -0.00349 ± 0.00194 0.03514 ± 0.00070 5.200 ± 0.099 % 92.549 ± 0.147 %
39
+ 32 7.7678 ± 0.1173 -0.00271 ± 0.00190 0.03505 ± 0.00068 5.241 ± 0.098 % 92.611 ± 0.145 %
40
+ 33 7.7570 ± 0.1156 -0.00199 ± 0.00188 0.03534 ± 0.00070 5.245 ± 0.096 % 92.580 ± 0.143 %
41
+ 34 7.7336 ± 0.1133 -0.00192 ± 0.00184 0.03489 ± 0.00068 5.213 ± 0.094 % 92.588 ± 0.140 %
42
+ 35 7.7318 ± 0.1117 -0.00194 ± 0.00180 0.03465 ± 0.00066 5.191 ± 0.092 % 92.582 ± 0.138 %
43
+ 36 7.7361 ± 0.1102 -0.00207 ± 0.00177 0.03441 ± 0.00064 5.180 ± 0.090 % 92.625 ± 0.136 %
44
+ 37 7.5889 ± 0.1061 -0.00211 ± 0.00173 0.03400 ± 0.00063 5.161 ± 0.090 % 92.655 ± 0.134 %
45
+ 38 7.4774 ± 0.1026 -0.00196 ± 0.00169 0.03374 ± 0.00062 5.153 ± 0.088 % 92.715 ± 0.132 %
46
+ 39 7.3609 ± 0.0994 -0.00178 ± 0.00166 0.03339 ± 0.00060 5.124 ± 0.086 % 92.726 ± 0.130 %
47
+ 40 7.2325 ± 0.0959 -0.00161 ± 0.00163 0.03300 ± 0.00059 5.104 ± 0.085 % 92.759 ± 0.128 %
48
+ 41 7.1515 ± 0.0932 -0.00148 ± 0.00161 0.03282 ± 0.00059 5.105 ± 0.085 % 92.743 ± 0.127 %
49
+ 42 7.1808 ± 0.0926 -0.00155 ± 0.00158 0.03276 ± 0.00057 5.093 ± 0.083 % 92.738 ± 0.125 %
50
+ 43 7.3764 ± 0.0949 -0.00215 ± 0.00157 0.03296 ± 0.00057 5.086 ± 0.082 % 92.680 ± 0.124 %
51
+ 44 7.5130 ± 0.0960 -0.00254 ± 0.00155 0.03301 ± 0.00058 5.062 ± 0.081 % 92.675 ± 0.123 %
52
+ 45 7.4880 ± 0.0944 -0.00190 ± 0.00155 0.03391 ± 0.00057 5.134 ± 0.079 % 92.619 ± 0.122 %
53
+ 46 7.4937 ± 0.0934 -0.00175 ± 0.00154 0.03406 ± 0.00056 5.147 ± 0.078 % 92.588 ± 0.121 %
54
+ 47 7.5551 ± 0.0933 -0.00240 ± 0.00152 0.03408 ± 0.00055 5.146 ± 0.077 % 92.577 ± 0.120 %
55
+ 48 7.5523 ± 0.0921 -0.00217 ± 0.00150 0.03387 ± 0.00054 5.129 ± 0.076 % 92.573 ± 0.118 %
56
+ 49 7.5993 ± 0.0918 -0.00255 ± 0.00148 0.03398 ± 0.00055 5.123 ± 0.075 % 92.545 ± 0.117 %
57
+ 50 7.5561 ± 0.0902 -0.00199 ± 0.00146 0.03380 ± 0.00054 5.102 ± 0.074 % 92.573 ± 0.116 %
58
+ 51 7.5900 ± 0.0897 -0.00175 ± 0.00144 0.03352 ± 0.00053 5.071 ± 0.073 % 92.602 ± 0.115 %
59
+ 52 7.5286 ± 0.0880 -0.00147 ± 0.00142 0.03336 ± 0.00053 5.064 ± 0.072 % 92.625 ± 0.113 %
60
+ 53 7.4982 ± 0.0867 -0.00110 ± 0.00140 0.03317 ± 0.00052 5.052 ± 0.071 % 92.630 ± 0.112 %
61
+ 54 7.4554 ± 0.0852 -0.00125 ± 0.00139 0.03299 ± 0.00051 5.036 ± 0.071 % 92.632 ± 0.111 %
62
+ 55 7.4096 ± 0.0838 -0.00238 ± 0.00138 0.03299 ± 0.00050 5.061 ± 0.072 % 92.638 ± 0.110 %
63
+ 56 7.3988 ± 0.0829 -0.00297 ± 0.00137 0.03300 ± 0.00050 5.058 ± 0.071 % 92.651 ± 0.109 %
64
+ 57 7.3204 ± 0.0811 -0.00287 ± 0.00135 0.03278 ± 0.00049 5.038 ± 0.070 % 92.670 ± 0.108 %
65
+ 58 7.3605 ± 0.0811 -0.00222 ± 0.00136 0.03339 ± 0.00050 5.068 ± 0.069 % 92.643 ± 0.107 %
66
+ 59 7.3360 ± 0.0800 -0.00156 ± 0.00135 0.03365 ± 0.00050 5.110 ± 0.069 % 92.621 ± 0.106 %
67
+ 60 7.3177 ± 0.0792 -0.00170 ± 0.00134 0.03350 ± 0.00049 5.100 ± 0.068 % 92.641 ± 0.105 %
68
+ 61 7.3813 ± 0.0795 -0.00389 ± 0.00135 0.03418 ± 0.00050 5.156 ± 0.069 % 92.582 ± 0.105 %
69
+ 62 7.4047 ± 0.0791 -0.00375 ± 0.00133 0.03400 ± 0.00050 5.132 ± 0.068 % 92.591 ± 0.104 %
70
+ 63 7.4554 ± 0.0791 -0.00297 ± 0.00132 0.03390 ± 0.00049 5.118 ± 0.067 % 92.582 ± 0.103 %
71
+ 64 7.4865 ± 0.0789 -0.00300 ± 0.00131 0.03377 ± 0.00048 5.103 ± 0.067 % 92.589 ± 0.102 %
72
+
73
+ ====== Perplexity statistics ======
74
+ Mean PPL(Q) : 7.486549 ± 0.078873
75
+ Mean PPL(base) : 7.509043 ± 0.079361
76
+ Cor(ln(PPL(Q)), ln(PPL(base))): 99.23%
77
+ Mean ln(PPL(Q)/PPL(base)) : -0.003000 ± 0.001307
78
+ Mean PPL(Q)/PPL(base) : 0.997004 ± 0.001303
79
+ Mean PPL(Q)-PPL(base) : -0.022494 ± 0.009811
80
+
81
+ ====== KL divergence statistics ======
82
+ Mean KLD: 0.033769 ± 0.000483
83
+ Maximum KLD: 6.496775
84
+ 99.9% KLD: 1.437661
85
+ 99.0% KLD: 0.341397
86
+ 95.0% KLD: 0.119217
87
+ 90.0% KLD: 0.071562
88
+ Median KLD: 0.011643
89
+ 10.0% KLD: 0.000193
90
+ 5.0% KLD: 0.000054
91
+ 1.0% KLD: -0.000000
92
+ 0.1% KLD: -0.000148
93
+ Minimum KLD: -0.000781
94
+
95
+ ====== Token probability statistics ======
96
+ Mean Δp: -0.118 ± 0.020 %
97
+ Maximum Δp: 92.131%
98
+ 99.9% Δp: 39.069%
99
+ 99.0% Δp: 15.335%
100
+ 95.0% Δp: 5.976%
101
+ 90.0% Δp: 3.202%
102
+ 75.0% Δp: 0.514%
103
+ Median Δp: -0.002%
104
+ 25.0% Δp: -0.739%
105
+ 10.0% Δp: -3.607%
106
+ 5.0% Δp: -6.416%
107
+ 1.0% Δp: -16.085%
108
+ 0.1% Δp: -39.140%
109
+ Minimum Δp: -98.490%
110
+ RMS Δp : 5.103 ± 0.067 %
111
+ Same top p: 92.589 ± 0.102 %
112
+
appendix/measurements/kldf_19G-ICE.txt ADDED
@@ -0,0 +1,84 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.00.800.239 W model has unused tensor blk.40.attn_norm.weight (size = 8192 bytes) -- ignoring
2
+ 0.00.800.253 W model has unused tensor blk.40.post_attention_norm.weight (size = 8192 bytes) -- ignoring
3
+ 0.00.800.259 W model has unused tensor blk.40.attn_q.weight (size = 17825792 bytes) -- ignoring
4
+ 0.00.800.262 W model has unused tensor blk.40.attn_k.weight (size = 2097152 bytes) -- ignoring
5
+ 0.00.800.264 W model has unused tensor blk.40.attn_v.weight (size = 2097152 bytes) -- ignoring
6
+ 0.00.800.272 W model has unused tensor blk.40.attn_output.weight (size = 8912896 bytes) -- ignoring
7
+ 0.00.800.274 W model has unused tensor blk.40.attn_q_norm.weight (size = 1024 bytes) -- ignoring
8
+ 0.00.800.276 W model has unused tensor blk.40.attn_k_norm.weight (size = 1024 bytes) -- ignoring
9
+ 0.00.800.279 W model has unused tensor blk.40.ffn_gate_inp.weight (size = 2097152 bytes) -- ignoring
10
+ 0.00.800.281 W model has unused tensor blk.40.ffn_down_exps.weight (size = 115343360 bytes) -- ignoring
11
+ 0.00.800.286 W model has unused tensor blk.40.ffn_gate_exps.weight (size = 115343360 bytes) -- ignoring
12
+ 0.00.800.288 W model has unused tensor blk.40.ffn_up_exps.weight (size = 115343360 bytes) -- ignoring
13
+ 0.00.800.291 W model has unused tensor blk.40.ffn_gate_inp_shexp.weight (size = 8192 bytes) -- ignoring
14
+ 0.00.800.293 W model has unused tensor blk.40.ffn_gate_shexp.weight (size = 1114112 bytes) -- ignoring
15
+ 0.00.800.296 W model has unused tensor blk.40.ffn_up_shexp.weight (size = 1114112 bytes) -- ignoring
16
+ 0.00.800.298 W model has unused tensor blk.40.ffn_down_shexp.weight (size = 1114112 bytes) -- ignoring
17
+ 0.00.800.303 W model has unused tensor blk.40.nextn.eh_proj.weight (size = 8912896 bytes) -- ignoring
18
+ 0.00.800.306 W model has unused tensor blk.40.nextn.enorm.weight (size = 8192 bytes) -- ignoring
19
+ 0.00.800.309 W model has unused tensor blk.40.nextn.hnorm.weight (size = 8192 bytes) -- ignoring
20
+ 0.00.800.316 W model has unused tensor blk.40.nextn.shared_head_norm.weight (size = 8192 bytes) -- ignoring
21
+ 0.02.303.475 I cmn init: llama threadpool init, n_threads = 40
22
+ 0.02.377.486 I
23
+ 0.02.377.635 I system_info: n_threads = 40 (n_threads_batch = 40) / 384 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
24
+ 0.02.583.472 I kl_divergence: computing over 16 chunks, n_ctx=2048, batch_size=2048, n_seq=1
25
+ 0.13.759.691 I kl_divergence: 11.18 seconds per pass - ETA 2.97 minutes
26
+
27
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
28
+ 1 5.7763 ± 0.4517 -0.00733 ± 0.01558 0.08572 ± 0.00664 7.576 ± 0.596 % 89.638 ± 0.953 %
29
+ 2 7.3577 ± 0.4291 0.00109 ± 0.01103 0.07566 ± 0.00395 6.863 ± 0.367 % 88.954 ± 0.693 %
30
+ 3 8.2653 ± 0.4068 0.00413 ± 0.00916 0.06999 ± 0.00319 6.901 ± 0.367 % 89.932 ± 0.543 %
31
+ 4 9.5749 ± 0.4396 0.01814 ± 0.00809 0.07343 ± 0.00304 7.469 ± 0.335 % 89.736 ± 0.474 %
32
+ 5 8.9691 ± 0.3626 0.02249 ± 0.00724 0.07149 ± 0.00258 7.401 ± 0.289 % 89.795 ± 0.423 %
33
+ 6 7.5834 ± 0.2701 0.01916 ± 0.00639 0.06990 ± 0.00232 7.387 ± 0.255 % 89.932 ± 0.384 %
34
+ 7 6.8837 ± 0.2225 0.01005 ± 0.00604 0.07056 ± 0.00224 7.745 ± 0.253 % 90.057 ± 0.354 %
35
+ 8 6.6786 ± 0.1991 0.00892 ± 0.00549 0.06905 ± 0.00206 7.640 ± 0.234 % 90.164 ± 0.329 %
36
+ 9 7.0691 ± 0.2003 0.00787 ± 0.00517 0.07019 ± 0.00191 7.558 ± 0.217 % 90.008 ± 0.313 %
37
+ 10 7.1264 ± 0.1916 0.00755 ± 0.00478 0.06774 ± 0.00174 7.409 ± 0.202 % 90.108 ± 0.295 %
38
+ 11 7.1329 ± 0.1817 0.00857 ± 0.00451 0.06571 ± 0.00160 7.281 ± 0.190 % 90.145 ± 0.281 %
39
+ 12 7.3853 ± 0.1810 0.00693 ± 0.00423 0.06374 ± 0.00148 7.132 ± 0.179 % 90.249 ± 0.268 %
40
+ 13 7.5271 ± 0.1784 0.01040 ± 0.00407 0.06351 ± 0.00141 7.132 ± 0.171 % 90.262 ± 0.257 %
41
+ 14 7.5569 ± 0.1721 0.00810 ± 0.00385 0.06164 ± 0.00133 7.003 ± 0.164 % 90.337 ± 0.247 %
42
+ 15 7.6073 ± 0.1674 0.00744 ± 0.00372 0.06097 ± 0.00125 6.941 ± 0.156 % 90.375 ± 0.238 %
43
+ 16 7.7786 ± 0.1658 0.00301 ± 0.00360 0.06084 ± 0.00118 6.878 ± 0.148 % 90.323 ± 0.231 %
44
+
45
+ ====== Perplexity statistics ======
46
+ Mean PPL(Q) : 7.778596 ± 0.165805
47
+ Mean PPL(base) : 7.755210 ± 0.166024
48
+ Cor(ln(PPL(Q)), ln(PPL(base))): 98.58%
49
+ Mean ln(PPL(Q)/PPL(base)) : 0.003011 ± 0.003598
50
+ Mean PPL(Q)/PPL(base) : 1.003016 ± 0.003609
51
+ Mean PPL(Q)-PPL(base) : 0.023386 ± 0.027939
52
+
53
+ ====== KL divergence statistics ======
54
+ Mean KLD: 0.060844 ± 0.001185
55
+ Maximum KLD: 5.031896
56
+ 99.9% KLD: 1.876680
57
+ 99.0% KLD: 0.615142
58
+ 95.0% KLD: 0.223469
59
+ 90.0% KLD: 0.137022
60
+ Median KLD: 0.023589
61
+ 10.0% KLD: 0.000344
62
+ 5.0% KLD: 0.000101
63
+ 1.0% KLD: 0.000006
64
+ 0.1% KLD: -0.000150
65
+ Minimum KLD: -0.000622
66
+
67
+ ====== Token probability statistics ======
68
+ Mean Δp: -0.459 ± 0.054 %
69
+ Maximum Δp: 95.224%
70
+ 99.9% Δp: 41.651%
71
+ 99.0% Δp: 19.851%
72
+ 95.0% Δp: 7.862%
73
+ 90.0% Δp: 4.076%
74
+ 75.0% Δp: 0.571%
75
+ Median Δp: -0.015%
76
+ 25.0% Δp: -1.288%
77
+ 10.0% Δp: -5.650%
78
+ 5.0% Δp: -9.632%
79
+ 1.0% Δp: -23.417%
80
+ 0.1% Δp: -55.582%
81
+ Minimum Δp: -96.108%
82
+ RMS Δp : 6.878 ± 0.148 %
83
+ Same top p: 90.323 ± 0.231 %
84
+
appendix/measurements/kldf_21G-ICE.txt ADDED
@@ -0,0 +1,84 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.00.801.364 W model has unused tensor blk.40.attn_norm.weight (size = 8192 bytes) -- ignoring
2
+ 0.00.801.378 W model has unused tensor blk.40.post_attention_norm.weight (size = 8192 bytes) -- ignoring
3
+ 0.00.801.384 W model has unused tensor blk.40.attn_q.weight (size = 17825792 bytes) -- ignoring
4
+ 0.00.801.386 W model has unused tensor blk.40.attn_k.weight (size = 2097152 bytes) -- ignoring
5
+ 0.00.801.389 W model has unused tensor blk.40.attn_v.weight (size = 2097152 bytes) -- ignoring
6
+ 0.00.801.397 W model has unused tensor blk.40.attn_output.weight (size = 8912896 bytes) -- ignoring
7
+ 0.00.801.399 W model has unused tensor blk.40.attn_q_norm.weight (size = 1024 bytes) -- ignoring
8
+ 0.00.801.401 W model has unused tensor blk.40.attn_k_norm.weight (size = 1024 bytes) -- ignoring
9
+ 0.00.801.404 W model has unused tensor blk.40.ffn_gate_inp.weight (size = 2097152 bytes) -- ignoring
10
+ 0.00.801.406 W model has unused tensor blk.40.ffn_down_exps.weight (size = 142606336 bytes) -- ignoring
11
+ 0.00.801.410 W model has unused tensor blk.40.ffn_gate_exps.weight (size = 142606336 bytes) -- ignoring
12
+ 0.00.801.413 W model has unused tensor blk.40.ffn_up_exps.weight (size = 142606336 bytes) -- ignoring
13
+ 0.00.801.415 W model has unused tensor blk.40.ffn_gate_inp_shexp.weight (size = 8192 bytes) -- ignoring
14
+ 0.00.801.418 W model has unused tensor blk.40.ffn_gate_shexp.weight (size = 1114112 bytes) -- ignoring
15
+ 0.00.801.420 W model has unused tensor blk.40.ffn_up_shexp.weight (size = 1114112 bytes) -- ignoring
16
+ 0.00.801.423 W model has unused tensor blk.40.ffn_down_shexp.weight (size = 1114112 bytes) -- ignoring
17
+ 0.00.801.426 W model has unused tensor blk.40.nextn.eh_proj.weight (size = 8912896 bytes) -- ignoring
18
+ 0.00.801.429 W model has unused tensor blk.40.nextn.enorm.weight (size = 8192 bytes) -- ignoring
19
+ 0.00.801.432 W model has unused tensor blk.40.nextn.hnorm.weight (size = 8192 bytes) -- ignoring
20
+ 0.00.801.439 W model has unused tensor blk.40.nextn.shared_head_norm.weight (size = 8192 bytes) -- ignoring
21
+ 0.12.710.938 I cmn init: llama threadpool init, n_threads = 40
22
+ 0.12.796.898 I
23
+ 0.12.797.045 I system_info: n_threads = 40 (n_threads_batch = 40) / 384 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
24
+ 0.12.979.626 I kl_divergence: computing over 16 chunks, n_ctx=2048, batch_size=2048, n_seq=1
25
+ 0.22.159.474 I kl_divergence: 9.18 seconds per pass - ETA 2.43 minutes
26
+
27
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
28
+ 1 5.8951 ± 0.4631 0.01303 ± 0.01142 0.04793 ± 0.00351 6.581 ± 0.671 % 92.180 ± 0.840 %
29
+ 2 7.4046 ± 0.4336 0.00745 ± 0.00812 0.04370 ± 0.00235 5.864 ± 0.407 % 92.033 ± 0.599 %
30
+ 3 8.4249 ± 0.4205 0.02325 ± 0.00714 0.04242 ± 0.00191 5.782 ± 0.351 % 92.408 ± 0.478 %
31
+ 4 9.3974 ± 0.4298 -0.00058 ± 0.00667 0.05186 ± 0.00354 6.426 ± 0.366 % 91.960 ± 0.425 %
32
+ 5 8.7808 ± 0.3539 0.00127 ± 0.00573 0.04943 ± 0.00288 6.224 ± 0.313 % 91.984 ± 0.380 %
33
+ 6 7.4381 ± 0.2644 -0.00019 ± 0.00511 0.04826 ± 0.00247 6.303 ± 0.273 % 91.838 ± 0.349 %
34
+ 7 6.8154 ± 0.2206 0.00008 ± 0.00484 0.04801 ± 0.00228 6.398 ± 0.263 % 92.068 ± 0.319 %
35
+ 8 6.6168 ± 0.1975 -0.00038 ± 0.00442 0.04652 ± 0.00203 6.264 ± 0.238 % 92.082 ± 0.298 %
36
+ 9 7.0203 ± 0.1994 0.00094 ± 0.00416 0.04747 ± 0.00189 6.256 ± 0.220 % 91.865 ± 0.285 %
37
+ 10 7.0889 ± 0.1910 0.00228 ± 0.00385 0.04543 ± 0.00171 6.096 ± 0.204 % 91.926 ± 0.269 %
38
+ 11 7.0839 ± 0.1807 0.00167 ± 0.00365 0.04410 ± 0.00158 5.983 ± 0.192 % 92.038 ± 0.255 %
39
+ 12 7.3335 ± 0.1801 -0.00011 ± 0.00343 0.04286 ± 0.00145 5.854 ± 0.180 % 92.074 ± 0.244 %
40
+ 13 7.4300 ± 0.1758 -0.00259 ± 0.00333 0.04271 ± 0.00136 5.838 ± 0.170 % 92.097 ± 0.234 %
41
+ 14 7.4669 ± 0.1698 -0.00387 ± 0.00314 0.04151 ± 0.00127 5.741 ± 0.162 % 92.138 ± 0.225 %
42
+ 15 7.5247 ± 0.1653 -0.00347 ± 0.00301 0.04093 ± 0.00119 5.679 ± 0.154 % 92.102 ± 0.218 %
43
+ 16 7.6961 ± 0.1638 -0.00765 ± 0.00296 0.04119 ± 0.00114 5.681 ± 0.148 % 92.027 ± 0.212 %
44
+
45
+ ====== Perplexity statistics ======
46
+ Mean PPL(Q) : 7.696145 ± 0.163779
47
+ Mean PPL(base) : 7.755210 ± 0.166024
48
+ Cor(ln(PPL(Q)), ln(PPL(base))): 99.04%
49
+ Mean ln(PPL(Q)/PPL(base)) : -0.007645 ± 0.002963
50
+ Mean PPL(Q)/PPL(base) : 0.992384 ± 0.002940
51
+ Mean PPL(Q)-PPL(base) : -0.059065 ± 0.022980
52
+
53
+ ====== KL divergence statistics ======
54
+ Mean KLD: 0.041192 ± 0.001140
55
+ Maximum KLD: 6.648515
56
+ 99.9% KLD: 1.664515
57
+ 99.0% KLD: 0.392024
58
+ 95.0% KLD: 0.142787
59
+ 90.0% KLD: 0.087165
60
+ Median KLD: 0.015146
61
+ 10.0% KLD: 0.000216
62
+ 5.0% KLD: 0.000057
63
+ 1.0% KLD: -0.000005
64
+ 0.1% KLD: -0.000155
65
+ Minimum KLD: -0.000405
66
+
67
+ ====== Token probability statistics ======
68
+ Mean Δp: -0.187 ± 0.044 %
69
+ Maximum Δp: 97.721%
70
+ 99.9% Δp: 40.671%
71
+ 99.0% Δp: 16.097%
72
+ 95.0% Δp: 6.943%
73
+ 90.0% Δp: 3.694%
74
+ 75.0% Δp: 0.586%
75
+ Median Δp: -0.002%
76
+ 25.0% Δp: -0.912%
77
+ 10.0% Δp: -4.338%
78
+ 5.0% Δp: -7.444%
79
+ 1.0% Δp: -17.527%
80
+ 0.1% Δp: -45.227%
81
+ Minimum Δp: -78.071%
82
+ RMS Δp : 5.681 ± 0.148 %
83
+ Same top p: 92.027 ± 0.212 %
84
+
appendix/measurements/kldf_23G-ICE.txt ADDED
@@ -0,0 +1,84 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.00.804.004 W model has unused tensor blk.40.attn_norm.weight (size = 8192 bytes) -- ignoring
2
+ 0.00.804.016 W model has unused tensor blk.40.post_attention_norm.weight (size = 8192 bytes) -- ignoring
3
+ 0.00.804.022 W model has unused tensor blk.40.attn_q.weight (size = 17825792 bytes) -- ignoring
4
+ 0.00.804.024 W model has unused tensor blk.40.attn_k.weight (size = 2097152 bytes) -- ignoring
5
+ 0.00.804.027 W model has unused tensor blk.40.attn_v.weight (size = 2097152 bytes) -- ignoring
6
+ 0.00.804.035 W model has unused tensor blk.40.attn_output.weight (size = 8912896 bytes) -- ignoring
7
+ 0.00.804.037 W model has unused tensor blk.40.attn_q_norm.weight (size = 1024 bytes) -- ignoring
8
+ 0.00.804.039 W model has unused tensor blk.40.attn_k_norm.weight (size = 1024 bytes) -- ignoring
9
+ 0.00.804.042 W model has unused tensor blk.40.ffn_gate_inp.weight (size = 2097152 bytes) -- ignoring
10
+ 0.00.804.044 W model has unused tensor blk.40.ffn_down_exps.weight (size = 150994944 bytes) -- ignoring
11
+ 0.00.804.048 W model has unused tensor blk.40.ffn_gate_exps.weight (size = 150994944 bytes) -- ignoring
12
+ 0.00.804.051 W model has unused tensor blk.40.ffn_up_exps.weight (size = 150994944 bytes) -- ignoring
13
+ 0.00.804.053 W model has unused tensor blk.40.ffn_gate_inp_shexp.weight (size = 8192 bytes) -- ignoring
14
+ 0.00.804.056 W model has unused tensor blk.40.ffn_gate_shexp.weight (size = 1114112 bytes) -- ignoring
15
+ 0.00.804.058 W model has unused tensor blk.40.ffn_up_shexp.weight (size = 1114112 bytes) -- ignoring
16
+ 0.00.804.061 W model has unused tensor blk.40.ffn_down_shexp.weight (size = 1114112 bytes) -- ignoring
17
+ 0.00.804.065 W model has unused tensor blk.40.nextn.eh_proj.weight (size = 8912896 bytes) -- ignoring
18
+ 0.00.804.068 W model has unused tensor blk.40.nextn.enorm.weight (size = 8192 bytes) -- ignoring
19
+ 0.00.804.071 W model has unused tensor blk.40.nextn.hnorm.weight (size = 8192 bytes) -- ignoring
20
+ 0.00.804.078 W model has unused tensor blk.40.nextn.shared_head_norm.weight (size = 8192 bytes) -- ignoring
21
+ 0.14.545.207 I cmn init: llama threadpool init, n_threads = 40
22
+ 0.14.635.257 I
23
+ 0.14.635.411 I system_info: n_threads = 40 (n_threads_batch = 40) / 384 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
24
+ 0.14.820.148 I kl_divergence: computing over 16 chunks, n_ctx=2048, batch_size=2048, n_seq=1
25
+ 0.24.019.678 I kl_divergence: 9.20 seconds per pass - ETA 2.45 minutes
26
+
27
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
28
+ 1 5.8444 ± 0.4565 0.00439 ± 0.01081 0.04593 ± 0.00390 6.689 ± 0.670 % 92.571 ± 0.820 %
29
+ 2 7.3246 ± 0.4255 -0.00342 ± 0.00759 0.04019 ± 0.00217 5.630 ± 0.410 % 91.935 ± 0.602 %
30
+ 3 8.3152 ± 0.4118 0.01015 ± 0.00619 0.03698 ± 0.00160 5.297 ± 0.336 % 92.571 ± 0.473 %
31
+ 4 9.4536 ± 0.4330 0.00539 ± 0.00561 0.04172 ± 0.00237 5.907 ± 0.337 % 92.351 ± 0.416 %
32
+ 5 8.8001 ± 0.3546 0.00347 ± 0.00493 0.04024 ± 0.00194 5.701 ± 0.285 % 92.532 ± 0.368 %
33
+ 6 7.4532 ± 0.2653 0.00184 ± 0.00441 0.03891 ± 0.00168 5.671 ± 0.245 % 92.669 ± 0.333 %
34
+ 7 6.7964 ± 0.2199 -0.00272 ± 0.00425 0.03871 ± 0.00155 5.823 ± 0.247 % 92.724 ± 0.307 %
35
+ 8 6.5984 ± 0.1968 -0.00316 ± 0.00390 0.03750 ± 0.00137 5.688 ± 0.224 % 92.705 ± 0.287 %
36
+ 9 7.0188 ± 0.1994 0.00074 ± 0.00372 0.03949 ± 0.00130 5.746 ± 0.204 % 92.288 ± 0.278 %
37
+ 10 7.0693 ± 0.1905 -0.00050 ± 0.00346 0.03819 ± 0.00118 5.606 ± 0.189 % 92.463 ± 0.261 %
38
+ 11 7.0643 ± 0.1801 -0.00110 ± 0.00327 0.03707 ± 0.00111 5.498 ± 0.178 % 92.606 ± 0.247 %
39
+ 12 7.3174 ± 0.1796 -0.00231 ± 0.00309 0.03610 ± 0.00103 5.388 ± 0.167 % 92.644 ± 0.236 %
40
+ 13 7.4127 ± 0.1753 -0.00491 ± 0.00300 0.03632 ± 0.00101 5.457 ± 0.160 % 92.646 ± 0.226 %
41
+ 14 7.4566 ± 0.1695 -0.00526 ± 0.00284 0.03531 ± 0.00094 5.378 ± 0.152 % 92.711 ± 0.217 %
42
+ 15 7.4957 ± 0.1645 -0.00732 ± 0.00274 0.03526 ± 0.00089 5.332 ± 0.145 % 92.727 ± 0.210 %
43
+ 16 7.6660 ± 0.1629 -0.01158 ± 0.00272 0.03607 ± 0.00087 5.354 ± 0.140 % 92.650 ± 0.204 %
44
+
45
+ ====== Perplexity statistics ======
46
+ Mean PPL(Q) : 7.665955 ± 0.162891
47
+ Mean PPL(base) : 7.755210 ± 0.166024
48
+ Cor(ln(PPL(Q)), ln(PPL(base))): 99.19%
49
+ Mean ln(PPL(Q)/PPL(base)) : -0.011576 ± 0.002723
50
+ Mean PPL(Q)/PPL(base) : 0.988491 ± 0.002692
51
+ Mean PPL(Q)-PPL(base) : -0.089255 ± 0.021195
52
+
53
+ ====== KL divergence statistics ======
54
+ Mean KLD: 0.036069 ± 0.000868
55
+ Maximum KLD: 5.588296
56
+ 99.9% KLD: 1.272136
57
+ 99.0% KLD: 0.331770
58
+ 95.0% KLD: 0.130636
59
+ 90.0% KLD: 0.080453
60
+ Median KLD: 0.013617
61
+ 10.0% KLD: 0.000199
62
+ 5.0% KLD: 0.000055
63
+ 1.0% KLD: -0.000006
64
+ 0.1% KLD: -0.000161
65
+ Minimum KLD: -0.000501
66
+
67
+ ====== Token probability statistics ======
68
+ Mean Δp: -0.162 ± 0.042 %
69
+ Maximum Δp: 75.679%
70
+ 99.9% Δp: 36.358%
71
+ 99.0% Δp: 15.326%
72
+ 95.0% Δp: 6.646%
73
+ 90.0% Δp: 3.477%
74
+ 75.0% Δp: 0.565%
75
+ Median Δp: -0.002%
76
+ 25.0% Δp: -0.827%
77
+ 10.0% Δp: -4.021%
78
+ 5.0% Δp: -7.009%
79
+ 1.0% Δp: -16.589%
80
+ 0.1% Δp: -49.141%
81
+ Minimum Δp: -92.408%
82
+ RMS Δp : 5.354 ± 0.140 %
83
+ Same top p: 92.650 ± 0.204 %
84
+
appendix/measurements/kldf_25G-ICE.txt ADDED
@@ -0,0 +1,84 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 0.00.792.992 W model has unused tensor blk.40.attn_norm.weight (size = 8192 bytes) -- ignoring
2
+ 0.00.793.005 W model has unused tensor blk.40.post_attention_norm.weight (size = 8192 bytes) -- ignoring
3
+ 0.00.793.011 W model has unused tensor blk.40.attn_q.weight (size = 17825792 bytes) -- ignoring
4
+ 0.00.793.014 W model has unused tensor blk.40.attn_k.weight (size = 2097152 bytes) -- ignoring
5
+ 0.00.793.016 W model has unused tensor blk.40.attn_v.weight (size = 2097152 bytes) -- ignoring
6
+ 0.00.793.024 W model has unused tensor blk.40.attn_output.weight (size = 8912896 bytes) -- ignoring
7
+ 0.00.793.026 W model has unused tensor blk.40.attn_q_norm.weight (size = 1024 bytes) -- ignoring
8
+ 0.00.793.028 W model has unused tensor blk.40.attn_k_norm.weight (size = 1024 bytes) -- ignoring
9
+ 0.00.793.031 W model has unused tensor blk.40.ffn_gate_inp.weight (size = 2097152 bytes) -- ignoring
10
+ 0.00.793.033 W model has unused tensor blk.40.ffn_down_exps.weight (size = 184549376 bytes) -- ignoring
11
+ 0.00.793.038 W model has unused tensor blk.40.ffn_gate_exps.weight (size = 184549376 bytes) -- ignoring
12
+ 0.00.793.041 W model has unused tensor blk.40.ffn_up_exps.weight (size = 184549376 bytes) -- ignoring
13
+ 0.00.793.043 W model has unused tensor blk.40.ffn_gate_inp_shexp.weight (size = 8192 bytes) -- ignoring
14
+ 0.00.793.046 W model has unused tensor blk.40.ffn_gate_shexp.weight (size = 1114112 bytes) -- ignoring
15
+ 0.00.793.048 W model has unused tensor blk.40.ffn_up_shexp.weight (size = 1114112 bytes) -- ignoring
16
+ 0.00.793.051 W model has unused tensor blk.40.ffn_down_shexp.weight (size = 1114112 bytes) -- ignoring
17
+ 0.00.793.055 W model has unused tensor blk.40.nextn.eh_proj.weight (size = 8912896 bytes) -- ignoring
18
+ 0.00.793.058 W model has unused tensor blk.40.nextn.enorm.weight (size = 8192 bytes) -- ignoring
19
+ 0.00.793.061 W model has unused tensor blk.40.nextn.hnorm.weight (size = 8192 bytes) -- ignoring
20
+ 0.00.793.068 W model has unused tensor blk.40.nextn.shared_head_norm.weight (size = 8192 bytes) -- ignoring
21
+ 0.05.671.948 I cmn init: llama threadpool init, n_threads = 40
22
+ 0.05.749.795 I
23
+ 0.05.749.952 I system_info: n_threads = 40 (n_threads_batch = 40) / 384 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
24
+ 0.05.937.173 I kl_divergence: computing over 16 chunks, n_ctx=2048, batch_size=2048, n_seq=1
25
+ 0.15.342.589 I kl_divergence: 9.41 seconds per pass - ETA 2.50 minutes
26
+
27
+ chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p
28
+ 1 5.7216 ± 0.4446 -0.01684 ± 0.00969 0.03757 ± 0.00273 5.983 ± 0.482 % 93.646 ± 0.763 %
29
+ 2 7.2893 ± 0.4253 -0.00825 ± 0.00691 0.03252 ± 0.00164 5.037 ± 0.297 % 93.157 ± 0.558 %
30
+ 3 8.1847 ± 0.4034 -0.00567 ± 0.00557 0.02982 ± 0.00132 4.860 ± 0.304 % 93.776 ± 0.436 %
31
+ 4 9.2966 ± 0.4234 -0.01135 ± 0.00510 0.03356 ± 0.00194 5.007 ± 0.244 % 93.426 ± 0.387 %
32
+ 5 8.6874 ± 0.3487 -0.00942 ± 0.00461 0.03272 ± 0.00160 4.857 ± 0.208 % 93.392 ± 0.347 %
33
+ 6 7.3567 ± 0.2607 -0.01119 ± 0.00416 0.03244 ± 0.00140 5.011 ± 0.187 % 93.271 ± 0.320 %
34
+ 7 6.6881 ± 0.2151 -0.01878 ± 0.00407 0.03406 ± 0.00147 5.272 ± 0.184 % 93.088 ± 0.300 %
35
+ 8 6.5149 ± 0.1935 -0.01590 ± 0.00368 0.03283 ± 0.00132 5.162 ± 0.168 % 93.218 ± 0.278 %
36
+ 9 6.8816 ± 0.1941 -0.01901 ± 0.00346 0.03371 ± 0.00126 5.151 ± 0.156 % 92.908 ± 0.268 %
37
+ 10 6.9492 ± 0.1862 -0.01763 ± 0.00322 0.03231 ± 0.00114 5.029 ± 0.146 % 93.109 ± 0.250 %
38
+ 11 6.9597 ± 0.1766 -0.01602 ± 0.00306 0.03132 ± 0.00106 4.957 ± 0.138 % 93.184 ± 0.238 %
39
+ 12 7.2214 ± 0.1766 -0.01551 ± 0.00287 0.03043 ± 0.00097 4.859 ± 0.130 % 93.231 ± 0.227 %
40
+ 13 7.3199 ± 0.1725 -0.01752 ± 0.00276 0.03056 ± 0.00092 4.898 ± 0.124 % 93.180 ± 0.219 %
41
+ 14 7.3819 ± 0.1673 -0.01533 ± 0.00262 0.02968 ± 0.00086 4.795 ± 0.118 % 93.248 ± 0.210 %
42
+ 15 7.4231 ± 0.1624 -0.01707 ± 0.00255 0.02990 ± 0.00082 4.801 ± 0.112 % 93.255 ± 0.202 %
43
+ 16 7.6111 ± 0.1614 -0.01876 ± 0.00251 0.03030 ± 0.00078 4.786 ± 0.109 % 93.163 ± 0.197 %
44
+
45
+ ====== Perplexity statistics ======
46
+ Mean PPL(Q) : 7.611108 ± 0.161433
47
+ Mean PPL(base) : 7.755210 ± 0.166024
48
+ Cor(ln(PPL(Q)), ln(PPL(base))): 99.31%
49
+ Mean ln(PPL(Q)/PPL(base)) : -0.018756 ± 0.002515
50
+ Mean PPL(Q)/PPL(base) : 0.981419 ± 0.002468
51
+ Mean PPL(Q)-PPL(base) : -0.144102 ± 0.019802
52
+
53
+ ====== KL divergence statistics ======
54
+ Mean KLD: 0.030304 ± 0.000779
55
+ Maximum KLD: 5.409998
56
+ 99.9% KLD: 1.048851
57
+ 99.0% KLD: 0.295999
58
+ 95.0% KLD: 0.108395
59
+ 90.0% KLD: 0.065665
60
+ Median KLD: 0.011116
61
+ 10.0% KLD: 0.000164
62
+ 5.0% KLD: 0.000046
63
+ 1.0% KLD: -0.000009
64
+ 0.1% KLD: -0.000142
65
+ Minimum KLD: -0.000403
66
+
67
+ ====== Token probability statistics ======
68
+ Mean Δp: -0.099 ± 0.037 %
69
+ Maximum Δp: 76.470%
70
+ 99.9% Δp: 33.668%
71
+ 99.0% Δp: 14.631%
72
+ 95.0% Δp: 5.945%
73
+ 90.0% Δp: 3.133%
74
+ 75.0% Δp: 0.527%
75
+ Median Δp: -0.002%
76
+ 25.0% Δp: -0.717%
77
+ 10.0% Δp: -3.579%
78
+ 5.0% Δp: -6.381%
79
+ 1.0% Δp: -14.571%
80
+ 0.1% Δp: -33.475%
81
+ Minimum Δp: -71.146%
82
+ RMS Δp : 4.786 ± 0.109 %
83
+ Same top p: 93.163 ± 0.197 %
84
+
appendix/measurements/sensitivity.jsonl ADDED
The diff for this file is too large to render. See raw diff
 
appendix/original-ICE/PRINCIPLE.md ADDED
@@ -0,0 +1,243 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ title: ICE — quantization by error-propagation class
3
+ created: 2026-08-21
4
+ status: measured — ladder shipped 2026-08-21, one derived rule refuted
5
+ scope: model-agnostic. Instantiated for Ornith-1.5-35B-A3B; written to be re-derived
6
+ for any sparse MoE.
7
+ ---
8
+
9
+ # ICE
10
+
11
+ **Freeze what propagates. Spend everything else on the library.**
12
+
13
+ Every quantizer in the GGUF ecosystem — llama.cpp's presets, Unsloth Dynamic,
14
+ mudler's APEX, AtomicChat's AD — minimises the same objective for every tensor:
15
+
16
+ > imatrix-weighted MSE of *this tensor's output, for the current token.*
17
+
18
+ That objective is correct for tensors whose error dies with the token. It is the
19
+ wrong objective for three other kinds of tensor, and nobody separates them.
20
+
21
+ ---
22
+
23
+ ## 1. The four error-propagation classes
24
+
25
+ Classify every tensor by **how far an error travels**, not by how big its
26
+ activations are.
27
+
28
+ | class | mechanism | failure shape | treatment |
29
+ |---|---|---|---|
30
+ | **DISCRETE** | error flips an argmax; a different computation runs | not graded — *categorical* | exact (F32) |
31
+ | **RECURRENT** | error enters a state decay and compounds multiplicatively along the sequence | grows with context length | exact (F32) |
32
+ | **CACHED** | error is written once and re-read by later tokens | frozen in, cannot be re-decided | near-exact (F16) |
33
+ | **INSTANT** | error affects this token's output only | graded, bounded | the budget lives here |
34
+
35
+ ### How to find them in an arbitrary architecture
36
+
37
+ - **DISCRETE** — anything feeding a `top_k`, `argmax`, or hard gate. In an MoE
38
+ that is the router (`ffn_gate_inp`). Also any hard routing/branching signal.
39
+ - **RECURRENT** — anything multiplying or parameterising a carried state: SSM/Mamba
40
+ decay and timestep terms (`ssm_alpha`, `ssm_beta`, `ssm_a`, `ssm_dt`), gated
41
+ linear-attention decay, any RNN gate. **Not** the state's output projection —
42
+ that one is INSTANT.
43
+ - **CACHED** — whatever is written into the KV cache: `attn_k`, `attn_v`. Note
44
+ `attn_q` is *not* cached (see §5 — this is the trap).
45
+ - **INSTANT** — everything else.
46
+
47
+ ### The finding that makes this cheap
48
+
49
+ On Ornith-1.5-35B-A3B the entire propagating set is **47 M parameters — 0.14% of
50
+ the model. Keeping all of it exact costs 0.15 GB.** It is not a trade-off; it is a
51
+ line item you can simply pay. Expect the same on any sparse MoE: routers and state
52
+ gates are structurally tiny.
53
+
54
+ ---
55
+
56
+ ## 2. The four measured laws
57
+
58
+ ### Law 1 — the sparsity law: always-on bits are worth `1/sparsity` routed bits
59
+
60
+ For anything that fires every token, compute-share and storage-cost are both
61
+ proportional to parameter count, so *work per byte* is a constant. Routed experts
62
+ divide that constant by the routing fraction.
63
+
64
+ Measured on Ornith-1.5 (8 of 256 active): every always-on role scores ~270 units of
65
+ work per GB-per-bpw; the experts score **8.5. The ratio is 32 — exactly 256/8.**
66
+
67
+ > **A bit spent on the always-on core is worth `E_total / E_active` bits spent on
68
+ > the expert bank.** Compute this ratio first for any new MoE; it sets the entire
69
+ > shape of the recipe.
70
+
71
+ ### Law 2 — the convexity law: allocation cleverness is capped
72
+
73
+ Quantization error falls as `4^-b`, which is convex. By Jensen, non-uniform
74
+ allocation only beats uniform when sensitivity varies enormously, and the gain
75
+ scales with the **logarithm** of the spread.
76
+
77
+ Measured, by water-filling over the real per-expert routing histogram:
78
+
79
+ | sensitivity spread | optimal gain over uniform |
80
+ |---|---|
81
+ | 46× (real expert routing) | **+0.139 bpw** |
82
+ | 3.7× (depth spread of `ffn_down`) | less |
83
+ | ~250× | ~1 bpw |
84
+
85
+ > **Corollary: stop tuning ladders.** This is why every published recipe lands
86
+ > within ~0.3 bpw of every other one. Per-expert allocation, depth grading, and
87
+ > per-layer bands are all worth less than 0.15 bpw. ICE therefore does **no depth
88
+ > grading at all** — an explicit refusal, not an oversight.
89
+
90
+ ### Law 3 — the leverage law: flat importance is the expensive case
91
+
92
+ The imatrix stores per-channel importance `h`. Measure its concentration with the
93
+ participation ratio `PR = (Σh)² / (n·Σh²)`.
94
+
95
+ **Spiky `h` (low PR) is a gift** — the quantizer knows which few channels to spend
96
+ the block scale on. **Flat `h` (high PR) is the hard case** — every channel matters
97
+ and there is no leverage. So *flat tensors need more bits.*
98
+
99
+ Measured on Ornith-1.5:
100
+
101
+ | tensor | PR/n |
102
+ |---|---:|
103
+ | `attn_k/q/v` | 0.0069 (~14 of 2048 channels) |
104
+ | `attn_qkv`, `attn_gate`, `ssm_alpha/beta` | 0.0128 |
105
+ | `ffn_gate_exps`, `ffn_up_exps` | 0.2048 |
106
+ | **`ffn_down_exps`** | **0.6104 — flattest in the model** |
107
+
108
+ > ~~`down` should sit `log4(PR_down / PR_gateup)` = 0.79 bpw above `gate`/`up`.~~
109
+ > **REFUTED BY MEASUREMENT, 2026-08-21.**
110
+
111
+ This looked like a derivation of what convention does by feel, and it agreed with
112
+ the best published rungs (UD-Q4_K_XL and AtomicChat AD-Q5_K-Q4_K both run ~1.0).
113
+ A controlled test at identical size says otherwise:
114
+
115
+ | expert allocation | size | mean KLD |
116
+ |---|---:|---:|
117
+ | `down` = `gate/up` + 0.79 | 20.82 GB | 0.046449 |
118
+ | **uniform (gap = 0)** | 20.85 GB | **0.041192** |
119
+
120
+ **Uniform is 11.3% better**, reproduced exactly on a rebuild (0.041192 twice).
121
+ Rebuilding the whole ladder uniform improved every rung: −7.3% / −11.3% / −3.3% /
122
+ −3.1% at 19/21/23/25 GB.
123
+
124
+ **What PR actually predicts is unknown.** It measures how concentrated the *input*
125
+ importance is, which is not the same as how much a tensor's error reaches the
126
+ output — `ffn_down` writes straight into the residual stream while `gate`/`up`
127
+ errors are attenuated through the gating nonlinearity first. Position in the
128
+ circuit is not captured by PR, and on this model it evidently dominates.
129
+
130
+ **Practical rule until someone measures better: allocate the expert stack
131
+ uniformly.** Do not bump `ffn_down`, despite convention.
132
+
133
+ ### Law 4 — the floor is epistemic, not numeric
134
+
135
+ Fit `KLD ≈ k₀ + c_d·4^(-bpw_dense) + c_e·4^(-bpw_expert)`. Two independent
136
+ harnesses (ours, AtomicChat's) agreed:
137
+
138
+ | | floor `k₀` | `c_e/c_d` |
139
+ |---|---:|---:|
140
+ | ours (wikitext/2048) | 0.0205 | 5.91 |
141
+ | AtomicChat (eval_neutral/4096) | 0.0219 | 7.22 |
142
+
143
+ `UD-Q6_K` measures 0.0221 against a 0.0205 floor — **it is 8% above the floor. The
144
+ top of the ladder is saturated.** Above ~5.5 expert bpw you are buying calibration
145
+ noise, not accuracy.
146
+
147
+ > **You cannot out-bit a wrong prior.** Quantization is lossy compression *guided by
148
+ > a belief about importance*. Improve the imatrix before adding bits.
149
+
150
+ **Caveat:** this fitted model ranks the extremes well and gets close calls wrong
151
+ (it misorders AtomicChat's `Q4_K_M` vs `AD-Q4_K-IQ4_XS`). Use it to find the floor
152
+ and the saturation point, never to choose between two candidate recipes.
153
+
154
+ ---
155
+
156
+ ## 3. The ICE assignment
157
+
158
+ ```
159
+ DISCRETE routers, hard gates F32
160
+ RECURRENT state decay / timestep terms F32
161
+ CACHED attn_k, attn_v F16
162
+ ALWAYS-ON output head, token_embd, attention, Q8_0
163
+ state output proj, shared expert
164
+ INSTANT routed experts the dial
165
+ down = gate = up (UNIFORM - measured, see Law 3)
166
+ ```
167
+
168
+ Plus one decision specific to speculative-decoding models:
169
+
170
+ > **A draft head does not need target-model precision.** Its errors are *verified*
171
+ > by the target, so they cost **acceptance**, not correctness. ICE pins only the
172
+ > draft block's projections and lets its experts follow the tier. On Ornith-1.5
173
+ > that frees ~0.40 GB. Everyone else pins the whole block Q8_0, drops it to Q4_0,
174
+ > or deletes it — nobody asks what a drafter actually needs.
175
+
176
+ ---
177
+
178
+ ## 4. Adapting ICE to a new MoE — the checklist
179
+
180
+ 1. **Inventory.** Dump the tensor list. Group by role. Compute params, % of file,
181
+ and % of *active* compute per role (weight routed experts by `k/E`).
182
+ 2. **Sparsity ratio `E/k`** → Law 1. This is your always-on-to-expert exchange rate.
183
+ 3. **Classify** every tensor into the four propagation classes (§1). Sum the
184
+ propagating set — if it is ≪1% of the model, freeze all of it.
185
+ 4. **PR from the imatrix** per role → Law 3. Informative about *quantizer leverage*,
186
+ but it did NOT predict the right expert split on Ornith-1.5 — allocate the expert
187
+ stack uniformly unless a controlled test on your model says otherwise.
188
+ `analysis/imatrix_anatomy.py` computes PR from the imatrix alone, no weights.
189
+ 5. **Routing histogram** from the imatrix `counts` → Law 2 → confirm depth/expert
190
+ grading is not worth the risk. `analysis/routing_skew.py`.
191
+ 6. **Fixed body** = propagating set + always-on core + draft projections.
192
+ Everything else is one dial: `size = body + E_params · bpw / 8`.
193
+ 7. **Bound the ladder.** Fit Law 4 on any measured tiers to find `k₀`; the useful
194
+ band runs from the cliff (where the expert term explodes) to saturation
195
+ (where the floor dominates). On Ornith-1.5 that was 17–27 GB — five rungs, not
196
+ eight.
197
+
198
+ ---
199
+
200
+ ## 5. What was tried and rejected — with numbers
201
+
202
+ Negative results, kept because they are the expensive part to rediscover.
203
+
204
+ - **Per-expert bit allocation by routing frequency.** Measured: load balancing
205
+ works (209.5 effective experts of 256, gini 0.331), and convexity caps the gain
206
+ at **+0.139 bpw**. A naive "hot 25% at 6.5, cold at 2.5" split is *worse than
207
+ uniform* (0.00586 vs 0.00552). **Dead.**
208
+ - **Demoting `attn_q`** on the argument that Q's error dies with the token while
209
+ K/V's persists. The arithmetic kills it: each query distributes attention weight
210
+ summing to 1, so total k-influence over a sequence is O(N) — **the same as q**.
211
+ Buys 63 MB, costs 0.17 active bpw. Kept only as an A/B flag.
212
+ - **Active bpw as a ranking metric.** Tested against measured KLD it misorders
213
+ **4 of our 8 tiers** (Spearman +0.95) and **10 of AtomicChat's 13** (+0.71).
214
+ It over-credits Q8_0-dense-path designs because error is not linear in bits.
215
+ **Use it to explain where bits went; rank on measured KLD.**
216
+ - **Depth grading.** Worth <0.15 bpw by Law 2, and the *direction* is unresolved:
217
+ PR says the middle layers need more bits (they are flattest), convention says
218
+ the edges do (they matter more functionally). PR measures quantizer leverage,
219
+ not functional sensitivity. Unresolved → not spent.
220
+
221
+ ---
222
+
223
+ ## 6. Open, and worth more than everything above
224
+
225
+ - **Orthogonal rotation.** `RMSNorm` is invariant to rotation but **not** to
226
+ scaling — so a rotation of the residual stream folds into the norm weights
227
+ exactly and for free, and yields a *standard GGUF*. With `attn_*` at PR 0.0069
228
+ the residual stream has ~14 dominant dimensions of 2048; spreading that energy
229
+ is worth an estimated ~0.9 bpw. Nobody has published rotation results for a
230
+ hybrid SSM/MoE, and the fixed SSM state may not tolerate it. **Test this next.**
231
+ - **Shared basis + per-expert delta.** `W_i = W̄ + Δ_i`, store `W̄` once. If
232
+ `‖Δ‖/‖W‖ ≈ ¼` that is ~2 bpw for 0.4% overhead, and `W̄ᵀx` is computed once per
233
+ token rather than once per expert. Needs a llama.cpp op. **The one measurement
234
+ that decides it is free: compute `‖Δ_i‖/‖W̄‖` from the bf16.** Counter-argument:
235
+ this architecture already has a shared expert, which may have absorbed the
236
+ common component.
237
+ - **`up`↔`down` scale migration.** `scale(up row j, s)` and `scale(down col j, 1/s)`
238
+ is *exactly* output-preserving. `up`'s output channels are rows and already get
239
+ their own k-quant scales (free), while `down`'s input channels are one element
240
+ in each of 2048 rows (polluting every row's range). Migrate the variance out of
241
+ `down`'s columns into `up`'s rows. Format-compatible, never done in GGUF.
242
+
243
+ Related: [[apex-standard-pipeline]] · [[map-transfers-imatrix-does-not]]
appendix/original-ICE/RECIPE.md ADDED
@@ -0,0 +1,151 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ title: ICE for Ornith-1.5-35B-A3B — the concrete instantiation
3
+ created: 2026-08-21
4
+ status: SHIPPED 2026-08-21 — 4 tiers live, uniform expert allocation
5
+ principle: see PRINCIPLE.md (model-agnostic)
6
+ ---
7
+
8
+ # ICE — Ornith-1.5-35B-A3B
9
+
10
+ The general method is in [PRINCIPLE.md](PRINCIPLE.md). This file is what it
11
+ produced for *this* model, with the arithmetic shown.
12
+
13
+ ## Architecture inputs
14
+
15
+ | | |
16
+ |---|---|
17
+ | blocks | 41 = 30 linear/SSM + 10 full-attention (3,7,…,39) + 1 MTP (`blk.40`) |
18
+ | residual width | 2048 |
19
+ | experts | 256, 8 active → **sparsity ratio 32** |
20
+ | expert FFN / shared FFN | 512 / 512 |
21
+ | KV heads | 2, head_dim 256, on 10 blocks only → ~20 KB/token |
22
+ | vocab | 248,320 → `output.weight` = 508.6 M in one tensor |
23
+
24
+ **Every block is 95.6% expert weights** (805.3 M of ~842 M). The surgical field is
25
+ the remaining ~35 M per block plus the two 508 M end-caps.
26
+
27
+ ## Active-compute geography (measured)
28
+
29
+ | role | params | % of file | % of *active* compute |
30
+ |---|---:|---:|---:|
31
+ | routed experts (8/256) | 32.21 B | 92.94% | 34.2% |
32
+ | attention | 1.03 B | 2.96% | **34.9%** |
33
+ | output head | 0.51 B | 1.47% | **17.3%** |
34
+ | ssm | 0.26 B | 0.74% | 8.7% |
35
+ | shared expert | 0.13 B | 0.36% | 4.3% |
36
+ | token_embd | 0.51 B | 1.47% | 0% (`get_rows`) |
37
+
38
+ **65.8% of active parameters are non-routed.** Active total 2.95 B — which is what
39
+ the "A3B" label means, and a useful check that the accounting is right.
40
+
41
+ ## The assignment
42
+
43
+ | tensor | × | params | class | type |
44
+ |---|---:|---:|---|---|
45
+ | `ffn_gate_inp`, `ffn_gate_inp_shexp` | 80 | 21.0 M | DISCRETE | F32 † |
46
+ | `ssm_conv1d` | 30 | 1.0 M | RECURRENT | F32 † |
47
+ | `ssm_alpha`, `ssm_beta` | 60 | 3.9 M | RECURRENT | **F32 (pinned)** |
48
+ | `ssm_a`, `ssm_dt`, all norms | — | 0.3 M | RECURRENT / — | F32 † |
49
+ | `attn_k`, `attn_v` | 20 | 21.0 M | **CACHED** | **F16** |
50
+ | `attn_q` | 10 | 167.8 M | INSTANT | Q8_0 |
51
+ | `attn_output` | 10 | 83.9 M | INSTANT | Q8_0 |
52
+ | `attn_qkv` | 30 | 503.3 M | INSTANT | Q8_0 |
53
+ | `attn_gate` | 30 | 251.7 M | INSTANT | Q8_0 |
54
+ | `ssm_out` | 30 | 251.7 M | INSTANT | Q8_0 |
55
+ | `ffn_*_shexp` | 120 | 125.8 M | always-on | Q8_0 |
56
+ | `output.weight` | 1 | 508.6 M | always-on | Q8_0 |
57
+ | `token_embd` | 1 | 508.6 M | — | Q8_0 |
58
+ | `blk.40.nextn.eh_proj` + blk.40 attn/shexp | — | 39.3 M | draft projections | Q8_0 |
59
+ | **`ffn_{down,gate,up}_exps`** | 123 | **33.02 B** | INSTANT | **the dial** |
60
+
61
+ † already F32 in llama.cpp: `ffn_gate_inp*` is excluded by name, `ssm_conv1d` has
62
+ first dim 4 (not a multiple of 256), 1-D tensors are never quantized. Pinning them
63
+ would be *inert* and fail GATE 0 — the intent is satisfied by the quantizer itself.
64
+
65
+ **Fixed body ≈ 2.6 GB.** Propagating set = 47 M params = **0.14% of the model**,
66
+ exact for 0.15 GB.
67
+
68
+ ### Expert split
69
+
70
+ **UNIFORM** — `down` = `gate` = `up`. The first ladder shipped a 0.79 bpw bump on
71
+ `down` derived from `log4(0.6104/0.2048)`; a controlled test at identical size
72
+ refuted it (0.041192 uniform vs 0.046449 bumped, **11.3% better uniform**), and the
73
+ ladder was rebuilt. See PRINCIPLE.md Law 3.
74
+
75
+ ### Draft head
76
+
77
+ `blk.40`'s **experts follow the tier**; only its projections are Q8_0. Draft errors
78
+ are verified by the target model, so they cost *acceptance*, not correctness.
79
+ Frees ~0.40 GB versus pinning the whole block.
80
+
81
+ ### No depth grading
82
+
83
+ Refused deliberately. Water-filling over the real routing histogram shows the
84
+ optimal gain is **+0.139 bpw**, and the direction of a depth profile is unresolved
85
+ (PR says middle, convention says edges). Not worth the risk budget.
86
+
87
+ ## The ladder
88
+
89
+ | tier | built | expert bpw | down = gate = up |
90
+ |---|---:|---:|---|
91
+ | `19G-ICE` | 18.82 GB | 3.899 | IQ3_S / IQ4_XS |
92
+ | `21G-ICE` | 20.84 GB | 4.383 | IQ4_XS / Q4_K |
93
+ | `23G-ICE` | 22.83 GB | 4.868 | Q4_K / Q5_K |
94
+ | `25G-ICE` | 24.84 GB | 5.353 | Q5_K / Q6_K |
95
+
96
+ GATE 0: **0 double-matched, 0 uncovered, 0 inert** on all four.
97
+
98
+ ## Predicted position (active bpw, pre-measurement)
99
+
100
+ | build | disk | ACTIVE bpw | dense | blk.40 |
101
+ |---|---:|---:|---:|---:|
102
+ | UD-IQ4_XS | 18.68 G | 6.762 | 2.47 G | 0.90 G |
103
+ | **19G-ICE** | 18.84 G | **7.191** | 2.61 G | 0.43 G |
104
+ | **23G-ICE** | 22.83 G | **7.521** | 2.61 G | 0.53 G |
105
+ | UD-Q4_K_XL | 23.21 G | 7.471 | 2.59 G | 0.90 G |
106
+ | **25G-ICE** | 24.84 G | **7.691** | 2.61 G | 0.53 G |
107
+ | UD-Q5_K_S | 25.83 G | 7.693 | 2.59 G | 0.90 G |
108
+
109
+ `25G-ICE` matches `UD-Q5_K_S`'s active precision at **0.99 GB less**.
110
+
111
+ ## Falsifiable predictions — scored
112
+
113
+ 1. ~~`25G-ICE` ≥ `UD-Q5_K_S` on measured KLD at ~1 GB less.~~ **FAILED.** 0.0303 vs
114
+ 0.0272; ICE is 0.99 GB smaller but not better. The 0.79 gap it rested on was then
115
+ refuted outright.
116
+ 2. **`19G-ICE` beats `UD-IQ4_XS`** — **HELD.** 0.060844 vs 0.0723, **15.8% better**
117
+ at 0.14 GB *smaller* (18.82 vs 18.68 GB is +0.14; after the uniform rebuild the
118
+ file is 18.82 GB and still wins).
119
+ 3. **Un-pinning `blk.40` costs <2 pp acceptance** — **HELD, direction unverified.**
120
+ 96.04% measured, above the 92.97% baseline, but on a different tier so not a
121
+ clean comparison. No collapse.
122
+
123
+ Same harness as the UD/APEX ladder — wikitext-2-raw, 16 chunks, 2048 ctx — so the
124
+ numbers drop straight into the existing table. Changing the harness would make the
125
+ comparison meaningless.
126
+
127
+ ## Results — shipped
128
+
129
+ | tier | size | mean KLD | top-1 | vs UD curve at same size |
130
+ |---|---:|---:|---:|---:|
131
+ | `25G-ICE` | 24.84 GB | 0.030304 | 93.16% | −1.9% |
132
+ | `23G-ICE` | 22.83 GB | 0.036069 | 92.65% | −6.3% |
133
+ | `21G-ICE` | 20.84 GB | 0.041192 | 92.03% | **−18.6%** |
134
+ | `19G-ICE` | 18.82 GB | 0.060844 | 89.72% | **−13.6%** |
135
+
136
+ **MTP draft acceptance: 96.04%** (388/404), measured on the shrunk `blk.40`.
137
+ Baseline with the block fully Q8_0 was 92.97% on a different tier, so this is not a
138
+ clean comparison — but it establishes the smaller draft head still drafts.
139
+
140
+ The advantage grows as size falls: at the top of the ladder the calibration floor
141
+ (~0.0185) dominates and no allocation choice can move it.
142
+
143
+ ### What did not survive measurement
144
+
145
+ | idea | verdict |
146
+ |---|---|
147
+ | `down` bumped above `gate`/`up` | **refuted, −11.3%** — ship uniform |
148
+ | `token_embd` → Q4_K | **not replicated** — helps at 21G, hurts at 19G |
149
+ | `attn_gate` / `ssm_out` → Q6_K | **harmful** — always-on, 32× law |
150
+ | `attn_q` demotion | dropped before shipping — arithmetic didn't hold |
151
+ | up↔down scale migration | **dead** — `down` columns uniform (p95/p05 = 1.09) |
appendix/recipes/cfg_19G-ICE_ICEbase.txt ADDED
@@ -0,0 +1,443 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ^output\.weight$=Q8_0
2
+ token_embd\.weight=Q8_0
3
+ blk\.0\.attn_gate\.weight=Q8_0
4
+ blk\.0\.attn_qkv\.weight=Q8_0
5
+ blk\.0\.ffn_down_exps\.weight=IQ4_XS
6
+ blk\.0\.ffn_down_shexp\.weight=Q8_0
7
+ blk\.0\.ffn_gate_exps\.weight=IQ4_XS
8
+ blk\.0\.ffn_gate_shexp\.weight=Q8_0
9
+ blk\.0\.ffn_up_exps\.weight=IQ4_XS
10
+ blk\.0\.ffn_up_shexp\.weight=Q8_0
11
+ blk\.0\.ssm_alpha\.weight=F32
12
+ blk\.0\.ssm_beta\.weight=F32
13
+ blk\.0\.ssm_out\.weight=Q8_0
14
+ blk\.1\.attn_gate\.weight=Q8_0
15
+ blk\.1\.attn_qkv\.weight=Q8_0
16
+ blk\.1\.ffn_down_exps\.weight=IQ4_XS
17
+ blk\.1\.ffn_down_shexp\.weight=Q8_0
18
+ blk\.1\.ffn_gate_exps\.weight=IQ4_XS
19
+ blk\.1\.ffn_gate_shexp\.weight=Q8_0
20
+ blk\.1\.ffn_up_exps\.weight=IQ4_XS
21
+ blk\.1\.ffn_up_shexp\.weight=Q8_0
22
+ blk\.1\.ssm_alpha\.weight=F32
23
+ blk\.1\.ssm_beta\.weight=F32
24
+ blk\.1\.ssm_out\.weight=Q8_0
25
+ blk\.10\.attn_gate\.weight=Q8_0
26
+ blk\.10\.attn_qkv\.weight=Q8_0
27
+ blk\.10\.ffn_down_exps\.weight=IQ4_XS
28
+ blk\.10\.ffn_down_shexp\.weight=Q8_0
29
+ blk\.10\.ffn_gate_exps\.weight=IQ4_XS
30
+ blk\.10\.ffn_gate_shexp\.weight=Q8_0
31
+ blk\.10\.ffn_up_exps\.weight=IQ4_XS
32
+ blk\.10\.ffn_up_shexp\.weight=Q8_0
33
+ blk\.10\.ssm_alpha\.weight=F32
34
+ blk\.10\.ssm_beta\.weight=F32
35
+ blk\.10\.ssm_out\.weight=Q8_0
36
+ blk\.11\.attn_k\.weight=F16
37
+ blk\.11\.attn_output\.weight=Q8_0
38
+ blk\.11\.attn_q\.weight=Q8_0
39
+ blk\.11\.attn_v\.weight=F16
40
+ blk\.11\.ffn_down_exps\.weight=IQ4_XS
41
+ blk\.11\.ffn_down_shexp\.weight=Q8_0
42
+ blk\.11\.ffn_gate_exps\.weight=IQ4_XS
43
+ blk\.11\.ffn_gate_shexp\.weight=Q8_0
44
+ blk\.11\.ffn_up_exps\.weight=IQ4_XS
45
+ blk\.11\.ffn_up_shexp\.weight=Q8_0
46
+ blk\.12\.attn_gate\.weight=Q8_0
47
+ blk\.12\.attn_qkv\.weight=Q8_0
48
+ blk\.12\.ffn_down_exps\.weight=IQ4_XS
49
+ blk\.12\.ffn_down_shexp\.weight=Q8_0
50
+ blk\.12\.ffn_gate_exps\.weight=IQ4_XS
51
+ blk\.12\.ffn_gate_shexp\.weight=Q8_0
52
+ blk\.12\.ffn_up_exps\.weight=IQ4_XS
53
+ blk\.12\.ffn_up_shexp\.weight=Q8_0
54
+ blk\.12\.ssm_alpha\.weight=F32
55
+ blk\.12\.ssm_beta\.weight=F32
56
+ blk\.12\.ssm_out\.weight=Q8_0
57
+ blk\.13\.attn_gate\.weight=Q8_0
58
+ blk\.13\.attn_qkv\.weight=Q8_0
59
+ blk\.13\.ffn_down_exps\.weight=IQ4_XS
60
+ blk\.13\.ffn_down_shexp\.weight=Q8_0
61
+ blk\.13\.ffn_gate_exps\.weight=IQ4_XS
62
+ blk\.13\.ffn_gate_shexp\.weight=Q8_0
63
+ blk\.13\.ffn_up_exps\.weight=IQ4_XS
64
+ blk\.13\.ffn_up_shexp\.weight=Q8_0
65
+ blk\.13\.ssm_alpha\.weight=F32
66
+ blk\.13\.ssm_beta\.weight=F32
67
+ blk\.13\.ssm_out\.weight=Q8_0
68
+ blk\.14\.attn_gate\.weight=Q8_0
69
+ blk\.14\.attn_qkv\.weight=Q8_0
70
+ blk\.14\.ffn_down_exps\.weight=IQ4_XS
71
+ blk\.14\.ffn_down_shexp\.weight=Q8_0
72
+ blk\.14\.ffn_gate_exps\.weight=IQ4_XS
73
+ blk\.14\.ffn_gate_shexp\.weight=Q8_0
74
+ blk\.14\.ffn_up_exps\.weight=IQ4_XS
75
+ blk\.14\.ffn_up_shexp\.weight=Q8_0
76
+ blk\.14\.ssm_alpha\.weight=F32
77
+ blk\.14\.ssm_beta\.weight=F32
78
+ blk\.14\.ssm_out\.weight=Q8_0
79
+ blk\.15\.attn_k\.weight=F16
80
+ blk\.15\.attn_output\.weight=Q8_0
81
+ blk\.15\.attn_q\.weight=Q8_0
82
+ blk\.15\.attn_v\.weight=F16
83
+ blk\.15\.ffn_down_exps\.weight=IQ4_XS
84
+ blk\.15\.ffn_down_shexp\.weight=Q8_0
85
+ blk\.15\.ffn_gate_exps\.weight=IQ4_XS
86
+ blk\.15\.ffn_gate_shexp\.weight=Q8_0
87
+ blk\.15\.ffn_up_exps\.weight=IQ4_XS
88
+ blk\.15\.ffn_up_shexp\.weight=Q8_0
89
+ blk\.16\.attn_gate\.weight=Q8_0
90
+ blk\.16\.attn_qkv\.weight=Q8_0
91
+ blk\.16\.ffn_down_exps\.weight=IQ4_XS
92
+ blk\.16\.ffn_down_shexp\.weight=Q8_0
93
+ blk\.16\.ffn_gate_exps\.weight=IQ4_XS
94
+ blk\.16\.ffn_gate_shexp\.weight=Q8_0
95
+ blk\.16\.ffn_up_exps\.weight=IQ4_XS
96
+ blk\.16\.ffn_up_shexp\.weight=Q8_0
97
+ blk\.16\.ssm_alpha\.weight=F32
98
+ blk\.16\.ssm_beta\.weight=F32
99
+ blk\.16\.ssm_out\.weight=Q8_0
100
+ blk\.17\.attn_gate\.weight=Q8_0
101
+ blk\.17\.attn_qkv\.weight=Q8_0
102
+ blk\.17\.ffn_down_exps\.weight=IQ4_XS
103
+ blk\.17\.ffn_down_shexp\.weight=Q8_0
104
+ blk\.17\.ffn_gate_exps\.weight=IQ4_XS
105
+ blk\.17\.ffn_gate_shexp\.weight=Q8_0
106
+ blk\.17\.ffn_up_exps\.weight=IQ4_XS
107
+ blk\.17\.ffn_up_shexp\.weight=Q8_0
108
+ blk\.17\.ssm_alpha\.weight=F32
109
+ blk\.17\.ssm_beta\.weight=F32
110
+ blk\.17\.ssm_out\.weight=Q8_0
111
+ blk\.18\.attn_gate\.weight=Q8_0
112
+ blk\.18\.attn_qkv\.weight=Q8_0
113
+ blk\.18\.ffn_down_exps\.weight=IQ4_XS
114
+ blk\.18\.ffn_down_shexp\.weight=Q8_0
115
+ blk\.18\.ffn_gate_exps\.weight=IQ4_XS
116
+ blk\.18\.ffn_gate_shexp\.weight=Q8_0
117
+ blk\.18\.ffn_up_exps\.weight=IQ4_XS
118
+ blk\.18\.ffn_up_shexp\.weight=Q8_0
119
+ blk\.18\.ssm_alpha\.weight=F32
120
+ blk\.18\.ssm_beta\.weight=F32
121
+ blk\.18\.ssm_out\.weight=Q8_0
122
+ blk\.19\.attn_k\.weight=F16
123
+ blk\.19\.attn_output\.weight=Q8_0
124
+ blk\.19\.attn_q\.weight=Q8_0
125
+ blk\.19\.attn_v\.weight=F16
126
+ blk\.19\.ffn_down_exps\.weight=IQ4_XS
127
+ blk\.19\.ffn_down_shexp\.weight=Q8_0
128
+ blk\.19\.ffn_gate_exps\.weight=IQ4_XS
129
+ blk\.19\.ffn_gate_shexp\.weight=Q8_0
130
+ blk\.19\.ffn_up_exps\.weight=IQ4_XS
131
+ blk\.19\.ffn_up_shexp\.weight=Q8_0
132
+ blk\.2\.attn_gate\.weight=Q8_0
133
+ blk\.2\.attn_qkv\.weight=Q8_0
134
+ blk\.2\.ffn_down_exps\.weight=IQ4_XS
135
+ blk\.2\.ffn_down_shexp\.weight=Q8_0
136
+ blk\.2\.ffn_gate_exps\.weight=IQ4_XS
137
+ blk\.2\.ffn_gate_shexp\.weight=Q8_0
138
+ blk\.2\.ffn_up_exps\.weight=IQ4_XS
139
+ blk\.2\.ffn_up_shexp\.weight=Q8_0
140
+ blk\.2\.ssm_alpha\.weight=F32
141
+ blk\.2\.ssm_beta\.weight=F32
142
+ blk\.2\.ssm_out\.weight=Q8_0
143
+ blk\.20\.attn_gate\.weight=Q8_0
144
+ blk\.20\.attn_qkv\.weight=Q8_0
145
+ blk\.20\.ffn_down_exps\.weight=IQ4_XS
146
+ blk\.20\.ffn_down_shexp\.weight=Q8_0
147
+ blk\.20\.ffn_gate_exps\.weight=IQ4_XS
148
+ blk\.20\.ffn_gate_shexp\.weight=Q8_0
149
+ blk\.20\.ffn_up_exps\.weight=IQ4_XS
150
+ blk\.20\.ffn_up_shexp\.weight=Q8_0
151
+ blk\.20\.ssm_alpha\.weight=F32
152
+ blk\.20\.ssm_beta\.weight=F32
153
+ blk\.20\.ssm_out\.weight=Q8_0
154
+ blk\.21\.attn_gate\.weight=Q8_0
155
+ blk\.21\.attn_qkv\.weight=Q8_0
156
+ blk\.21\.ffn_down_exps\.weight=IQ4_XS
157
+ blk\.21\.ffn_down_shexp\.weight=Q8_0
158
+ blk\.21\.ffn_gate_exps\.weight=IQ4_XS
159
+ blk\.21\.ffn_gate_shexp\.weight=Q8_0
160
+ blk\.21\.ffn_up_exps\.weight=IQ4_XS
161
+ blk\.21\.ffn_up_shexp\.weight=Q8_0
162
+ blk\.21\.ssm_alpha\.weight=F32
163
+ blk\.21\.ssm_beta\.weight=F32
164
+ blk\.21\.ssm_out\.weight=Q8_0
165
+ blk\.22\.attn_gate\.weight=Q8_0
166
+ blk\.22\.attn_qkv\.weight=Q8_0
167
+ blk\.22\.ffn_down_exps\.weight=IQ4_XS
168
+ blk\.22\.ffn_down_shexp\.weight=Q8_0
169
+ blk\.22\.ffn_gate_exps\.weight=IQ4_XS
170
+ blk\.22\.ffn_gate_shexp\.weight=Q8_0
171
+ blk\.22\.ffn_up_exps\.weight=IQ4_XS
172
+ blk\.22\.ffn_up_shexp\.weight=Q8_0
173
+ blk\.22\.ssm_alpha\.weight=F32
174
+ blk\.22\.ssm_beta\.weight=F32
175
+ blk\.22\.ssm_out\.weight=Q8_0
176
+ blk\.23\.attn_k\.weight=F16
177
+ blk\.23\.attn_output\.weight=Q8_0
178
+ blk\.23\.attn_q\.weight=Q8_0
179
+ blk\.23\.attn_v\.weight=F16
180
+ blk\.23\.ffn_down_exps\.weight=IQ3_S
181
+ blk\.23\.ffn_down_shexp\.weight=Q8_0
182
+ blk\.23\.ffn_gate_exps\.weight=IQ3_S
183
+ blk\.23\.ffn_gate_shexp\.weight=Q8_0
184
+ blk\.23\.ffn_up_exps\.weight=IQ3_S
185
+ blk\.23\.ffn_up_shexp\.weight=Q8_0
186
+ blk\.24\.attn_gate\.weight=Q8_0
187
+ blk\.24\.attn_qkv\.weight=Q8_0
188
+ blk\.24\.ffn_down_exps\.weight=IQ3_S
189
+ blk\.24\.ffn_down_shexp\.weight=Q8_0
190
+ blk\.24\.ffn_gate_exps\.weight=IQ3_S
191
+ blk\.24\.ffn_gate_shexp\.weight=Q8_0
192
+ blk\.24\.ffn_up_exps\.weight=IQ3_S
193
+ blk\.24\.ffn_up_shexp\.weight=Q8_0
194
+ blk\.24\.ssm_alpha\.weight=F32
195
+ blk\.24\.ssm_beta\.weight=F32
196
+ blk\.24\.ssm_out\.weight=Q8_0
197
+ blk\.25\.attn_gate\.weight=Q8_0
198
+ blk\.25\.attn_qkv\.weight=Q8_0
199
+ blk\.25\.ffn_down_exps\.weight=IQ3_S
200
+ blk\.25\.ffn_down_shexp\.weight=Q8_0
201
+ blk\.25\.ffn_gate_exps\.weight=IQ3_S
202
+ blk\.25\.ffn_gate_shexp\.weight=Q8_0
203
+ blk\.25\.ffn_up_exps\.weight=IQ3_S
204
+ blk\.25\.ffn_up_shexp\.weight=Q8_0
205
+ blk\.25\.ssm_alpha\.weight=F32
206
+ blk\.25\.ssm_beta\.weight=F32
207
+ blk\.25\.ssm_out\.weight=Q8_0
208
+ blk\.26\.attn_gate\.weight=Q8_0
209
+ blk\.26\.attn_qkv\.weight=Q8_0
210
+ blk\.26\.ffn_down_exps\.weight=IQ3_S
211
+ blk\.26\.ffn_down_shexp\.weight=Q8_0
212
+ blk\.26\.ffn_gate_exps\.weight=IQ3_S
213
+ blk\.26\.ffn_gate_shexp\.weight=Q8_0
214
+ blk\.26\.ffn_up_exps\.weight=IQ3_S
215
+ blk\.26\.ffn_up_shexp\.weight=Q8_0
216
+ blk\.26\.ssm_alpha\.weight=F32
217
+ blk\.26\.ssm_beta\.weight=F32
218
+ blk\.26\.ssm_out\.weight=Q8_0
219
+ blk\.27\.attn_k\.weight=F16
220
+ blk\.27\.attn_output\.weight=Q8_0
221
+ blk\.27\.attn_q\.weight=Q8_0
222
+ blk\.27\.attn_v\.weight=F16
223
+ blk\.27\.ffn_down_exps\.weight=IQ3_S
224
+ blk\.27\.ffn_down_shexp\.weight=Q8_0
225
+ blk\.27\.ffn_gate_exps\.weight=IQ3_S
226
+ blk\.27\.ffn_gate_shexp\.weight=Q8_0
227
+ blk\.27\.ffn_up_exps\.weight=IQ3_S
228
+ blk\.27\.ffn_up_shexp\.weight=Q8_0
229
+ blk\.28\.attn_gate\.weight=Q8_0
230
+ blk\.28\.attn_qkv\.weight=Q8_0
231
+ blk\.28\.ffn_down_exps\.weight=IQ3_S
232
+ blk\.28\.ffn_down_shexp\.weight=Q8_0
233
+ blk\.28\.ffn_gate_exps\.weight=IQ3_S
234
+ blk\.28\.ffn_gate_shexp\.weight=Q8_0
235
+ blk\.28\.ffn_up_exps\.weight=IQ3_S
236
+ blk\.28\.ffn_up_shexp\.weight=Q8_0
237
+ blk\.28\.ssm_alpha\.weight=F32
238
+ blk\.28\.ssm_beta\.weight=F32
239
+ blk\.28\.ssm_out\.weight=Q8_0
240
+ blk\.29\.attn_gate\.weight=Q8_0
241
+ blk\.29\.attn_qkv\.weight=Q8_0
242
+ blk\.29\.ffn_down_exps\.weight=IQ3_S
243
+ blk\.29\.ffn_down_shexp\.weight=Q8_0
244
+ blk\.29\.ffn_gate_exps\.weight=IQ3_S
245
+ blk\.29\.ffn_gate_shexp\.weight=Q8_0
246
+ blk\.29\.ffn_up_exps\.weight=IQ3_S
247
+ blk\.29\.ffn_up_shexp\.weight=Q8_0
248
+ blk\.29\.ssm_alpha\.weight=F32
249
+ blk\.29\.ssm_beta\.weight=F32
250
+ blk\.29\.ssm_out\.weight=Q8_0
251
+ blk\.3\.attn_k\.weight=F16
252
+ blk\.3\.attn_output\.weight=Q8_0
253
+ blk\.3\.attn_q\.weight=Q8_0
254
+ blk\.3\.attn_v\.weight=F16
255
+ blk\.3\.ffn_down_exps\.weight=IQ4_XS
256
+ blk\.3\.ffn_down_shexp\.weight=Q8_0
257
+ blk\.3\.ffn_gate_exps\.weight=IQ4_XS
258
+ blk\.3\.ffn_gate_shexp\.weight=Q8_0
259
+ blk\.3\.ffn_up_exps\.weight=IQ4_XS
260
+ blk\.3\.ffn_up_shexp\.weight=Q8_0
261
+ blk\.30\.attn_gate\.weight=Q8_0
262
+ blk\.30\.attn_qkv\.weight=Q8_0
263
+ blk\.30\.ffn_down_exps\.weight=IQ3_S
264
+ blk\.30\.ffn_down_shexp\.weight=Q8_0
265
+ blk\.30\.ffn_gate_exps\.weight=IQ3_S
266
+ blk\.30\.ffn_gate_shexp\.weight=Q8_0
267
+ blk\.30\.ffn_up_exps\.weight=IQ3_S
268
+ blk\.30\.ffn_up_shexp\.weight=Q8_0
269
+ blk\.30\.ssm_alpha\.weight=F32
270
+ blk\.30\.ssm_beta\.weight=F32
271
+ blk\.30\.ssm_out\.weight=Q8_0
272
+ blk\.31\.attn_k\.weight=F16
273
+ blk\.31\.attn_output\.weight=Q8_0
274
+ blk\.31\.attn_q\.weight=Q8_0
275
+ blk\.31\.attn_v\.weight=F16
276
+ blk\.31\.ffn_down_exps\.weight=IQ3_S
277
+ blk\.31\.ffn_down_shexp\.weight=Q8_0
278
+ blk\.31\.ffn_gate_exps\.weight=IQ3_S
279
+ blk\.31\.ffn_gate_shexp\.weight=Q8_0
280
+ blk\.31\.ffn_up_exps\.weight=IQ3_S
281
+ blk\.31\.ffn_up_shexp\.weight=Q8_0
282
+ blk\.32\.attn_gate\.weight=Q8_0
283
+ blk\.32\.attn_qkv\.weight=Q8_0
284
+ blk\.32\.ffn_down_exps\.weight=IQ3_S
285
+ blk\.32\.ffn_down_shexp\.weight=Q8_0
286
+ blk\.32\.ffn_gate_exps\.weight=IQ3_S
287
+ blk\.32\.ffn_gate_shexp\.weight=Q8_0
288
+ blk\.32\.ffn_up_exps\.weight=IQ3_S
289
+ blk\.32\.ffn_up_shexp\.weight=Q8_0
290
+ blk\.32\.ssm_alpha\.weight=F32
291
+ blk\.32\.ssm_beta\.weight=F32
292
+ blk\.32\.ssm_out\.weight=Q8_0
293
+ blk\.33\.attn_gate\.weight=Q8_0
294
+ blk\.33\.attn_qkv\.weight=Q8_0
295
+ blk\.33\.ffn_down_exps\.weight=IQ3_S
296
+ blk\.33\.ffn_down_shexp\.weight=Q8_0
297
+ blk\.33\.ffn_gate_exps\.weight=IQ3_S
298
+ blk\.33\.ffn_gate_shexp\.weight=Q8_0
299
+ blk\.33\.ffn_up_exps\.weight=IQ3_S
300
+ blk\.33\.ffn_up_shexp\.weight=Q8_0
301
+ blk\.33\.ssm_alpha\.weight=F32
302
+ blk\.33\.ssm_beta\.weight=F32
303
+ blk\.33\.ssm_out\.weight=Q8_0
304
+ blk\.34\.attn_gate\.weight=Q8_0
305
+ blk\.34\.attn_qkv\.weight=Q8_0
306
+ blk\.34\.ffn_down_exps\.weight=IQ3_S
307
+ blk\.34\.ffn_down_shexp\.weight=Q8_0
308
+ blk\.34\.ffn_gate_exps\.weight=IQ3_S
309
+ blk\.34\.ffn_gate_shexp\.weight=Q8_0
310
+ blk\.34\.ffn_up_exps\.weight=IQ3_S
311
+ blk\.34\.ffn_up_shexp\.weight=Q8_0
312
+ blk\.34\.ssm_alpha\.weight=F32
313
+ blk\.34\.ssm_beta\.weight=F32
314
+ blk\.34\.ssm_out\.weight=Q8_0
315
+ blk\.35\.attn_k\.weight=F16
316
+ blk\.35\.attn_output\.weight=Q8_0
317
+ blk\.35\.attn_q\.weight=Q8_0
318
+ blk\.35\.attn_v\.weight=F16
319
+ blk\.35\.ffn_down_exps\.weight=IQ3_S
320
+ blk\.35\.ffn_down_shexp\.weight=Q8_0
321
+ blk\.35\.ffn_gate_exps\.weight=IQ3_S
322
+ blk\.35\.ffn_gate_shexp\.weight=Q8_0
323
+ blk\.35\.ffn_up_exps\.weight=IQ3_S
324
+ blk\.35\.ffn_up_shexp\.weight=Q8_0
325
+ blk\.36\.attn_gate\.weight=Q8_0
326
+ blk\.36\.attn_qkv\.weight=Q8_0
327
+ blk\.36\.ffn_down_exps\.weight=IQ3_S
328
+ blk\.36\.ffn_down_shexp\.weight=Q8_0
329
+ blk\.36\.ffn_gate_exps\.weight=IQ3_S
330
+ blk\.36\.ffn_gate_shexp\.weight=Q8_0
331
+ blk\.36\.ffn_up_exps\.weight=IQ3_S
332
+ blk\.36\.ffn_up_shexp\.weight=Q8_0
333
+ blk\.36\.ssm_alpha\.weight=F32
334
+ blk\.36\.ssm_beta\.weight=F32
335
+ blk\.36\.ssm_out\.weight=Q8_0
336
+ blk\.37\.attn_gate\.weight=Q8_0
337
+ blk\.37\.attn_qkv\.weight=Q8_0
338
+ blk\.37\.ffn_down_exps\.weight=IQ3_S
339
+ blk\.37\.ffn_down_shexp\.weight=Q8_0
340
+ blk\.37\.ffn_gate_exps\.weight=IQ3_S
341
+ blk\.37\.ffn_gate_shexp\.weight=Q8_0
342
+ blk\.37\.ffn_up_exps\.weight=IQ3_S
343
+ blk\.37\.ffn_up_shexp\.weight=Q8_0
344
+ blk\.37\.ssm_alpha\.weight=F32
345
+ blk\.37\.ssm_beta\.weight=F32
346
+ blk\.37\.ssm_out\.weight=Q8_0
347
+ blk\.38\.attn_gate\.weight=Q8_0
348
+ blk\.38\.attn_qkv\.weight=Q8_0
349
+ blk\.38\.ffn_down_exps\.weight=IQ3_S
350
+ blk\.38\.ffn_down_shexp\.weight=Q8_0
351
+ blk\.38\.ffn_gate_exps\.weight=IQ3_S
352
+ blk\.38\.ffn_gate_shexp\.weight=Q8_0
353
+ blk\.38\.ffn_up_exps\.weight=IQ3_S
354
+ blk\.38\.ffn_up_shexp\.weight=Q8_0
355
+ blk\.38\.ssm_alpha\.weight=F32
356
+ blk\.38\.ssm_beta\.weight=F32
357
+ blk\.38\.ssm_out\.weight=Q8_0
358
+ blk\.39\.attn_k\.weight=F16
359
+ blk\.39\.attn_output\.weight=Q8_0
360
+ blk\.39\.attn_q\.weight=Q8_0
361
+ blk\.39\.attn_v\.weight=F16
362
+ blk\.39\.ffn_down_exps\.weight=IQ3_S
363
+ blk\.39\.ffn_down_shexp\.weight=Q8_0
364
+ blk\.39\.ffn_gate_exps\.weight=IQ3_S
365
+ blk\.39\.ffn_gate_shexp\.weight=Q8_0
366
+ blk\.39\.ffn_up_exps\.weight=IQ3_S
367
+ blk\.39\.ffn_up_shexp\.weight=Q8_0
368
+ blk\.4\.attn_gate\.weight=Q8_0
369
+ blk\.4\.attn_qkv\.weight=Q8_0
370
+ blk\.4\.ffn_down_exps\.weight=IQ4_XS
371
+ blk\.4\.ffn_down_shexp\.weight=Q8_0
372
+ blk\.4\.ffn_gate_exps\.weight=IQ4_XS
373
+ blk\.4\.ffn_gate_shexp\.weight=Q8_0
374
+ blk\.4\.ffn_up_exps\.weight=IQ4_XS
375
+ blk\.4\.ffn_up_shexp\.weight=Q8_0
376
+ blk\.4\.ssm_alpha\.weight=F32
377
+ blk\.4\.ssm_beta\.weight=F32
378
+ blk\.4\.ssm_out\.weight=Q8_0
379
+ blk\.40\.attn_k\.weight=F16
380
+ blk\.40\.attn_output\.weight=Q8_0
381
+ blk\.40\.attn_q\.weight=Q8_0
382
+ blk\.40\.attn_v\.weight=F16
383
+ blk\.40\.ffn_down_exps\.weight=IQ3_S
384
+ blk\.40\.ffn_down_shexp\.weight=Q8_0
385
+ blk\.40\.ffn_gate_exps\.weight=IQ3_S
386
+ blk\.40\.ffn_gate_shexp\.weight=Q8_0
387
+ blk\.40\.ffn_up_exps\.weight=IQ3_S
388
+ blk\.40\.ffn_up_shexp\.weight=Q8_0
389
+ blk\.40\.nextn\.eh_proj\.weight=Q8_0
390
+ blk\.5\.attn_gate\.weight=Q8_0
391
+ blk\.5\.attn_qkv\.weight=Q8_0
392
+ blk\.5\.ffn_down_exps\.weight=IQ4_XS
393
+ blk\.5\.ffn_down_shexp\.weight=Q8_0
394
+ blk\.5\.ffn_gate_exps\.weight=IQ4_XS
395
+ blk\.5\.ffn_gate_shexp\.weight=Q8_0
396
+ blk\.5\.ffn_up_exps\.weight=IQ4_XS
397
+ blk\.5\.ffn_up_shexp\.weight=Q8_0
398
+ blk\.5\.ssm_alpha\.weight=F32
399
+ blk\.5\.ssm_beta\.weight=F32
400
+ blk\.5\.ssm_out\.weight=Q8_0
401
+ blk\.6\.attn_gate\.weight=Q8_0
402
+ blk\.6\.attn_qkv\.weight=Q8_0
403
+ blk\.6\.ffn_down_exps\.weight=IQ4_XS
404
+ blk\.6\.ffn_down_shexp\.weight=Q8_0
405
+ blk\.6\.ffn_gate_exps\.weight=IQ4_XS
406
+ blk\.6\.ffn_gate_shexp\.weight=Q8_0
407
+ blk\.6\.ffn_up_exps\.weight=IQ4_XS
408
+ blk\.6\.ffn_up_shexp\.weight=Q8_0
409
+ blk\.6\.ssm_alpha\.weight=F32
410
+ blk\.6\.ssm_beta\.weight=F32
411
+ blk\.6\.ssm_out\.weight=Q8_0
412
+ blk\.7\.attn_k\.weight=F16
413
+ blk\.7\.attn_output\.weight=Q8_0
414
+ blk\.7\.attn_q\.weight=Q8_0
415
+ blk\.7\.attn_v\.weight=F16
416
+ blk\.7\.ffn_down_exps\.weight=IQ4_XS
417
+ blk\.7\.ffn_down_shexp\.weight=Q8_0
418
+ blk\.7\.ffn_gate_exps\.weight=IQ4_XS
419
+ blk\.7\.ffn_gate_shexp\.weight=Q8_0
420
+ blk\.7\.ffn_up_exps\.weight=IQ4_XS
421
+ blk\.7\.ffn_up_shexp\.weight=Q8_0
422
+ blk\.8\.attn_gate\.weight=Q8_0
423
+ blk\.8\.attn_qkv\.weight=Q8_0
424
+ blk\.8\.ffn_down_exps\.weight=IQ4_XS
425
+ blk\.8\.ffn_down_shexp\.weight=Q8_0
426
+ blk\.8\.ffn_gate_exps\.weight=IQ4_XS
427
+ blk\.8\.ffn_gate_shexp\.weight=Q8_0
428
+ blk\.8\.ffn_up_exps\.weight=IQ4_XS
429
+ blk\.8\.ffn_up_shexp\.weight=Q8_0
430
+ blk\.8\.ssm_alpha\.weight=F32
431
+ blk\.8\.ssm_beta\.weight=F32
432
+ blk\.8\.ssm_out\.weight=Q8_0
433
+ blk\.9\.attn_gate\.weight=Q8_0
434
+ blk\.9\.attn_qkv\.weight=Q8_0
435
+ blk\.9\.ffn_down_exps\.weight=IQ4_XS
436
+ blk\.9\.ffn_down_shexp\.weight=Q8_0
437
+ blk\.9\.ffn_gate_exps\.weight=IQ4_XS
438
+ blk\.9\.ffn_gate_shexp\.weight=Q8_0
439
+ blk\.9\.ffn_up_exps\.weight=IQ4_XS
440
+ blk\.9\.ffn_up_shexp\.weight=Q8_0
441
+ blk\.9\.ssm_alpha\.weight=F32
442
+ blk\.9\.ssm_beta\.weight=F32
443
+ blk\.9\.ssm_out\.weight=Q8_0
appendix/recipes/cfg_19G-ICE_previous.txt ADDED
@@ -0,0 +1,443 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ^output\.weight$=Q8_0
2
+ token_embd\.weight=Q8_0
3
+ blk\.0\.attn_gate\.weight=Q8_0
4
+ blk\.0\.attn_qkv\.weight=Q8_0
5
+ blk\.0\.ffn_down_exps\.weight=IQ4_XS
6
+ blk\.0\.ffn_down_shexp\.weight=Q8_0
7
+ blk\.0\.ffn_gate_exps\.weight=IQ4_XS
8
+ blk\.0\.ffn_gate_shexp\.weight=Q8_0
9
+ blk\.0\.ffn_up_exps\.weight=IQ4_XS
10
+ blk\.0\.ffn_up_shexp\.weight=Q8_0
11
+ blk\.0\.ssm_alpha\.weight=F32
12
+ blk\.0\.ssm_beta\.weight=F32
13
+ blk\.0\.ssm_out\.weight=Q8_0
14
+ blk\.1\.attn_gate\.weight=Q8_0
15
+ blk\.1\.attn_qkv\.weight=Q8_0
16
+ blk\.1\.ffn_down_exps\.weight=IQ3_S
17
+ blk\.1\.ffn_down_shexp\.weight=Q8_0
18
+ blk\.1\.ffn_gate_exps\.weight=IQ3_S
19
+ blk\.1\.ffn_gate_shexp\.weight=Q8_0
20
+ blk\.1\.ffn_up_exps\.weight=IQ3_S
21
+ blk\.1\.ffn_up_shexp\.weight=Q8_0
22
+ blk\.1\.ssm_alpha\.weight=F32
23
+ blk\.1\.ssm_beta\.weight=F32
24
+ blk\.1\.ssm_out\.weight=Q8_0
25
+ blk\.10\.attn_gate\.weight=Q8_0
26
+ blk\.10\.attn_qkv\.weight=Q8_0
27
+ blk\.10\.ffn_down_exps\.weight=IQ3_S
28
+ blk\.10\.ffn_down_shexp\.weight=Q8_0
29
+ blk\.10\.ffn_gate_exps\.weight=IQ3_S
30
+ blk\.10\.ffn_gate_shexp\.weight=Q8_0
31
+ blk\.10\.ffn_up_exps\.weight=IQ3_S
32
+ blk\.10\.ffn_up_shexp\.weight=Q8_0
33
+ blk\.10\.ssm_alpha\.weight=F32
34
+ blk\.10\.ssm_beta\.weight=F32
35
+ blk\.10\.ssm_out\.weight=Q8_0
36
+ blk\.11\.attn_k\.weight=F16
37
+ blk\.11\.attn_output\.weight=Q8_0
38
+ blk\.11\.attn_q\.weight=Q8_0
39
+ blk\.11\.attn_v\.weight=F16
40
+ blk\.11\.ffn_down_exps\.weight=IQ4_XS
41
+ blk\.11\.ffn_down_shexp\.weight=Q8_0
42
+ blk\.11\.ffn_gate_exps\.weight=IQ4_XS
43
+ blk\.11\.ffn_gate_shexp\.weight=Q8_0
44
+ blk\.11\.ffn_up_exps\.weight=IQ4_XS
45
+ blk\.11\.ffn_up_shexp\.weight=Q8_0
46
+ blk\.12\.attn_gate\.weight=Q8_0
47
+ blk\.12\.attn_qkv\.weight=Q8_0
48
+ blk\.12\.ffn_down_exps\.weight=IQ4_XS
49
+ blk\.12\.ffn_down_shexp\.weight=Q8_0
50
+ blk\.12\.ffn_gate_exps\.weight=IQ4_XS
51
+ blk\.12\.ffn_gate_shexp\.weight=Q8_0
52
+ blk\.12\.ffn_up_exps\.weight=IQ4_XS
53
+ blk\.12\.ffn_up_shexp\.weight=Q8_0
54
+ blk\.12\.ssm_alpha\.weight=F32
55
+ blk\.12\.ssm_beta\.weight=F32
56
+ blk\.12\.ssm_out\.weight=Q8_0
57
+ blk\.13\.attn_gate\.weight=Q8_0
58
+ blk\.13\.attn_qkv\.weight=Q8_0
59
+ blk\.13\.ffn_down_exps\.weight=IQ3_S
60
+ blk\.13\.ffn_down_shexp\.weight=Q8_0
61
+ blk\.13\.ffn_gate_exps\.weight=IQ3_S
62
+ blk\.13\.ffn_gate_shexp\.weight=Q8_0
63
+ blk\.13\.ffn_up_exps\.weight=IQ3_S
64
+ blk\.13\.ffn_up_shexp\.weight=Q8_0
65
+ blk\.13\.ssm_alpha\.weight=F32
66
+ blk\.13\.ssm_beta\.weight=F32
67
+ blk\.13\.ssm_out\.weight=Q8_0
68
+ blk\.14\.attn_gate\.weight=Q8_0
69
+ blk\.14\.attn_qkv\.weight=Q8_0
70
+ blk\.14\.ffn_down_exps\.weight=IQ4_XS
71
+ blk\.14\.ffn_down_shexp\.weight=Q8_0
72
+ blk\.14\.ffn_gate_exps\.weight=IQ4_XS
73
+ blk\.14\.ffn_gate_shexp\.weight=Q8_0
74
+ blk\.14\.ffn_up_exps\.weight=IQ4_XS
75
+ blk\.14\.ffn_up_shexp\.weight=Q8_0
76
+ blk\.14\.ssm_alpha\.weight=F32
77
+ blk\.14\.ssm_beta\.weight=F32
78
+ blk\.14\.ssm_out\.weight=Q8_0
79
+ blk\.15\.attn_k\.weight=F16
80
+ blk\.15\.attn_output\.weight=Q8_0
81
+ blk\.15\.attn_q\.weight=Q8_0
82
+ blk\.15\.attn_v\.weight=F16
83
+ blk\.15\.ffn_down_exps\.weight=IQ3_S
84
+ blk\.15\.ffn_down_shexp\.weight=Q8_0
85
+ blk\.15\.ffn_gate_exps\.weight=IQ3_S
86
+ blk\.15\.ffn_gate_shexp\.weight=Q8_0
87
+ blk\.15\.ffn_up_exps\.weight=IQ3_S
88
+ blk\.15\.ffn_up_shexp\.weight=Q8_0
89
+ blk\.16\.attn_gate\.weight=Q8_0
90
+ blk\.16\.attn_qkv\.weight=Q8_0
91
+ blk\.16\.ffn_down_exps\.weight=IQ4_XS
92
+ blk\.16\.ffn_down_shexp\.weight=Q8_0
93
+ blk\.16\.ffn_gate_exps\.weight=IQ4_XS
94
+ blk\.16\.ffn_gate_shexp\.weight=Q8_0
95
+ blk\.16\.ffn_up_exps\.weight=IQ4_XS
96
+ blk\.16\.ffn_up_shexp\.weight=Q8_0
97
+ blk\.16\.ssm_alpha\.weight=F32
98
+ blk\.16\.ssm_beta\.weight=F32
99
+ blk\.16\.ssm_out\.weight=Q8_0
100
+ blk\.17\.attn_gate\.weight=Q8_0
101
+ blk\.17\.attn_qkv\.weight=Q8_0
102
+ blk\.17\.ffn_down_exps\.weight=IQ3_S
103
+ blk\.17\.ffn_down_shexp\.weight=Q8_0
104
+ blk\.17\.ffn_gate_exps\.weight=IQ3_S
105
+ blk\.17\.ffn_gate_shexp\.weight=Q8_0
106
+ blk\.17\.ffn_up_exps\.weight=IQ3_S
107
+ blk\.17\.ffn_up_shexp\.weight=Q8_0
108
+ blk\.17\.ssm_alpha\.weight=F32
109
+ blk\.17\.ssm_beta\.weight=F32
110
+ blk\.17\.ssm_out\.weight=Q8_0
111
+ blk\.18\.attn_gate\.weight=Q8_0
112
+ blk\.18\.attn_qkv\.weight=Q8_0
113
+ blk\.18\.ffn_down_exps\.weight=IQ4_XS
114
+ blk\.18\.ffn_down_shexp\.weight=Q8_0
115
+ blk\.18\.ffn_gate_exps\.weight=IQ4_XS
116
+ blk\.18\.ffn_gate_shexp\.weight=Q8_0
117
+ blk\.18\.ffn_up_exps\.weight=IQ4_XS
118
+ blk\.18\.ffn_up_shexp\.weight=Q8_0
119
+ blk\.18\.ssm_alpha\.weight=F32
120
+ blk\.18\.ssm_beta\.weight=F32
121
+ blk\.18\.ssm_out\.weight=Q8_0
122
+ blk\.19\.attn_k\.weight=F16
123
+ blk\.19\.attn_output\.weight=Q8_0
124
+ blk\.19\.attn_q\.weight=Q8_0
125
+ blk\.19\.attn_v\.weight=F16
126
+ blk\.19\.ffn_down_exps\.weight=IQ3_S
127
+ blk\.19\.ffn_down_shexp\.weight=Q8_0
128
+ blk\.19\.ffn_gate_exps\.weight=IQ3_S
129
+ blk\.19\.ffn_gate_shexp\.weight=Q8_0
130
+ blk\.19\.ffn_up_exps\.weight=IQ3_S
131
+ blk\.19\.ffn_up_shexp\.weight=Q8_0
132
+ blk\.2\.attn_gate\.weight=Q8_0
133
+ blk\.2\.attn_qkv\.weight=Q8_0
134
+ blk\.2\.ffn_down_exps\.weight=IQ4_XS
135
+ blk\.2\.ffn_down_shexp\.weight=Q8_0
136
+ blk\.2\.ffn_gate_exps\.weight=IQ4_XS
137
+ blk\.2\.ffn_gate_shexp\.weight=Q8_0
138
+ blk\.2\.ffn_up_exps\.weight=IQ4_XS
139
+ blk\.2\.ffn_up_shexp\.weight=Q8_0
140
+ blk\.2\.ssm_alpha\.weight=F32
141
+ blk\.2\.ssm_beta\.weight=F32
142
+ blk\.2\.ssm_out\.weight=Q8_0
143
+ blk\.20\.attn_gate\.weight=Q8_0
144
+ blk\.20\.attn_qkv\.weight=Q8_0
145
+ blk\.20\.ffn_down_exps\.weight=IQ4_XS
146
+ blk\.20\.ffn_down_shexp\.weight=Q8_0
147
+ blk\.20\.ffn_gate_exps\.weight=IQ4_XS
148
+ blk\.20\.ffn_gate_shexp\.weight=Q8_0
149
+ blk\.20\.ffn_up_exps\.weight=IQ4_XS
150
+ blk\.20\.ffn_up_shexp\.weight=Q8_0
151
+ blk\.20\.ssm_alpha\.weight=F32
152
+ blk\.20\.ssm_beta\.weight=F32
153
+ blk\.20\.ssm_out\.weight=Q8_0
154
+ blk\.21\.attn_gate\.weight=Q8_0
155
+ blk\.21\.attn_qkv\.weight=Q8_0
156
+ blk\.21\.ffn_down_exps\.weight=IQ4_XS
157
+ blk\.21\.ffn_down_shexp\.weight=Q8_0
158
+ blk\.21\.ffn_gate_exps\.weight=IQ4_XS
159
+ blk\.21\.ffn_gate_shexp\.weight=Q8_0
160
+ blk\.21\.ffn_up_exps\.weight=IQ4_XS
161
+ blk\.21\.ffn_up_shexp\.weight=Q8_0
162
+ blk\.21\.ssm_alpha\.weight=F32
163
+ blk\.21\.ssm_beta\.weight=F32
164
+ blk\.21\.ssm_out\.weight=Q8_0
165
+ blk\.22\.attn_gate\.weight=Q8_0
166
+ blk\.22\.attn_qkv\.weight=Q8_0
167
+ blk\.22\.ffn_down_exps\.weight=IQ3_S
168
+ blk\.22\.ffn_down_shexp\.weight=Q8_0
169
+ blk\.22\.ffn_gate_exps\.weight=IQ3_S
170
+ blk\.22\.ffn_gate_shexp\.weight=Q8_0
171
+ blk\.22\.ffn_up_exps\.weight=IQ3_S
172
+ blk\.22\.ffn_up_shexp\.weight=Q8_0
173
+ blk\.22\.ssm_alpha\.weight=F32
174
+ blk\.22\.ssm_beta\.weight=F32
175
+ blk\.22\.ssm_out\.weight=Q8_0
176
+ blk\.23\.attn_k\.weight=F16
177
+ blk\.23\.attn_output\.weight=Q8_0
178
+ blk\.23\.attn_q\.weight=Q8_0
179
+ blk\.23\.attn_v\.weight=F16
180
+ blk\.23\.ffn_down_exps\.weight=IQ4_XS
181
+ blk\.23\.ffn_down_shexp\.weight=Q8_0
182
+ blk\.23\.ffn_gate_exps\.weight=IQ4_XS
183
+ blk\.23\.ffn_gate_shexp\.weight=Q8_0
184
+ blk\.23\.ffn_up_exps\.weight=IQ4_XS
185
+ blk\.23\.ffn_up_shexp\.weight=Q8_0
186
+ blk\.24\.attn_gate\.weight=Q8_0
187
+ blk\.24\.attn_qkv\.weight=Q8_0
188
+ blk\.24\.ffn_down_exps\.weight=IQ3_S
189
+ blk\.24\.ffn_down_shexp\.weight=Q8_0
190
+ blk\.24\.ffn_gate_exps\.weight=IQ3_S
191
+ blk\.24\.ffn_gate_shexp\.weight=Q8_0
192
+ blk\.24\.ffn_up_exps\.weight=IQ3_S
193
+ blk\.24\.ffn_up_shexp\.weight=Q8_0
194
+ blk\.24\.ssm_alpha\.weight=F32
195
+ blk\.24\.ssm_beta\.weight=F32
196
+ blk\.24\.ssm_out\.weight=Q8_0
197
+ blk\.25\.attn_gate\.weight=Q8_0
198
+ blk\.25\.attn_qkv\.weight=Q8_0
199
+ blk\.25\.ffn_down_exps\.weight=IQ4_XS
200
+ blk\.25\.ffn_down_shexp\.weight=Q8_0
201
+ blk\.25\.ffn_gate_exps\.weight=IQ4_XS
202
+ blk\.25\.ffn_gate_shexp\.weight=Q8_0
203
+ blk\.25\.ffn_up_exps\.weight=IQ4_XS
204
+ blk\.25\.ffn_up_shexp\.weight=Q8_0
205
+ blk\.25\.ssm_alpha\.weight=F32
206
+ blk\.25\.ssm_beta\.weight=F32
207
+ blk\.25\.ssm_out\.weight=Q8_0
208
+ blk\.26\.attn_gate\.weight=Q8_0
209
+ blk\.26\.attn_qkv\.weight=Q8_0
210
+ blk\.26\.ffn_down_exps\.weight=IQ3_S
211
+ blk\.26\.ffn_down_shexp\.weight=Q8_0
212
+ blk\.26\.ffn_gate_exps\.weight=IQ3_S
213
+ blk\.26\.ffn_gate_shexp\.weight=Q8_0
214
+ blk\.26\.ffn_up_exps\.weight=IQ3_S
215
+ blk\.26\.ffn_up_shexp\.weight=Q8_0
216
+ blk\.26\.ssm_alpha\.weight=F32
217
+ blk\.26\.ssm_beta\.weight=F32
218
+ blk\.26\.ssm_out\.weight=Q8_0
219
+ blk\.27\.attn_k\.weight=F16
220
+ blk\.27\.attn_output\.weight=Q8_0
221
+ blk\.27\.attn_q\.weight=Q8_0
222
+ blk\.27\.attn_v\.weight=F16
223
+ blk\.27\.ffn_down_exps\.weight=IQ4_XS
224
+ blk\.27\.ffn_down_shexp\.weight=Q8_0
225
+ blk\.27\.ffn_gate_exps\.weight=IQ4_XS
226
+ blk\.27\.ffn_gate_shexp\.weight=Q8_0
227
+ blk\.27\.ffn_up_exps\.weight=IQ4_XS
228
+ blk\.27\.ffn_up_shexp\.weight=Q8_0
229
+ blk\.28\.attn_gate\.weight=Q8_0
230
+ blk\.28\.attn_qkv\.weight=Q8_0
231
+ blk\.28\.ffn_down_exps\.weight=IQ3_S
232
+ blk\.28\.ffn_down_shexp\.weight=Q8_0
233
+ blk\.28\.ffn_gate_exps\.weight=IQ3_S
234
+ blk\.28\.ffn_gate_shexp\.weight=Q8_0
235
+ blk\.28\.ffn_up_exps\.weight=IQ3_S
236
+ blk\.28\.ffn_up_shexp\.weight=Q8_0
237
+ blk\.28\.ssm_alpha\.weight=F32
238
+ blk\.28\.ssm_beta\.weight=F32
239
+ blk\.28\.ssm_out\.weight=Q8_0
240
+ blk\.29\.attn_gate\.weight=Q8_0
241
+ blk\.29\.attn_qkv\.weight=Q8_0
242
+ blk\.29\.ffn_down_exps\.weight=IQ4_XS
243
+ blk\.29\.ffn_down_shexp\.weight=Q8_0
244
+ blk\.29\.ffn_gate_exps\.weight=IQ4_XS
245
+ blk\.29\.ffn_gate_shexp\.weight=Q8_0
246
+ blk\.29\.ffn_up_exps\.weight=IQ4_XS
247
+ blk\.29\.ffn_up_shexp\.weight=Q8_0
248
+ blk\.29\.ssm_alpha\.weight=F32
249
+ blk\.29\.ssm_beta\.weight=F32
250
+ blk\.29\.ssm_out\.weight=Q8_0
251
+ blk\.3\.attn_k\.weight=F16
252
+ blk\.3\.attn_output\.weight=Q8_0
253
+ blk\.3\.attn_q\.weight=Q8_0
254
+ blk\.3\.attn_v\.weight=F16
255
+ blk\.3\.ffn_down_exps\.weight=IQ3_S
256
+ blk\.3\.ffn_down_shexp\.weight=Q8_0
257
+ blk\.3\.ffn_gate_exps\.weight=IQ3_S
258
+ blk\.3\.ffn_gate_shexp\.weight=Q8_0
259
+ blk\.3\.ffn_up_exps\.weight=IQ3_S
260
+ blk\.3\.ffn_up_shexp\.weight=Q8_0
261
+ blk\.30\.attn_gate\.weight=Q8_0
262
+ blk\.30\.attn_qkv\.weight=Q8_0
263
+ blk\.30\.ffn_down_exps\.weight=IQ4_XS
264
+ blk\.30\.ffn_down_shexp\.weight=Q8_0
265
+ blk\.30\.ffn_gate_exps\.weight=IQ4_XS
266
+ blk\.30\.ffn_gate_shexp\.weight=Q8_0
267
+ blk\.30\.ffn_up_exps\.weight=IQ4_XS
268
+ blk\.30\.ffn_up_shexp\.weight=Q8_0
269
+ blk\.30\.ssm_alpha\.weight=F32
270
+ blk\.30\.ssm_beta\.weight=F32
271
+ blk\.30\.ssm_out\.weight=Q8_0
272
+ blk\.31\.attn_k\.weight=F16
273
+ blk\.31\.attn_output\.weight=Q8_0
274
+ blk\.31\.attn_q\.weight=Q8_0
275
+ blk\.31\.attn_v\.weight=F16
276
+ blk\.31\.ffn_down_exps\.weight=IQ3_S
277
+ blk\.31\.ffn_down_shexp\.weight=Q8_0
278
+ blk\.31\.ffn_gate_exps\.weight=IQ3_S
279
+ blk\.31\.ffn_gate_shexp\.weight=Q8_0
280
+ blk\.31\.ffn_up_exps\.weight=IQ3_S
281
+ blk\.31\.ffn_up_shexp\.weight=Q8_0
282
+ blk\.32\.attn_gate\.weight=Q8_0
283
+ blk\.32\.attn_qkv\.weight=Q8_0
284
+ blk\.32\.ffn_down_exps\.weight=IQ4_XS
285
+ blk\.32\.ffn_down_shexp\.weight=Q8_0
286
+ blk\.32\.ffn_gate_exps\.weight=IQ4_XS
287
+ blk\.32\.ffn_gate_shexp\.weight=Q8_0
288
+ blk\.32\.ffn_up_exps\.weight=IQ4_XS
289
+ blk\.32\.ffn_up_shexp\.weight=Q8_0
290
+ blk\.32\.ssm_alpha\.weight=F32
291
+ blk\.32\.ssm_beta\.weight=F32
292
+ blk\.32\.ssm_out\.weight=Q8_0
293
+ blk\.33\.attn_gate\.weight=Q8_0
294
+ blk\.33\.attn_qkv\.weight=Q8_0
295
+ blk\.33\.ffn_down_exps\.weight=IQ3_S
296
+ blk\.33\.ffn_down_shexp\.weight=Q8_0
297
+ blk\.33\.ffn_gate_exps\.weight=IQ3_S
298
+ blk\.33\.ffn_gate_shexp\.weight=Q8_0
299
+ blk\.33\.ffn_up_exps\.weight=IQ3_S
300
+ blk\.33\.ffn_up_shexp\.weight=Q8_0
301
+ blk\.33\.ssm_alpha\.weight=F32
302
+ blk\.33\.ssm_beta\.weight=F32
303
+ blk\.33\.ssm_out\.weight=Q8_0
304
+ blk\.34\.attn_gate\.weight=Q8_0
305
+ blk\.34\.attn_qkv\.weight=Q8_0
306
+ blk\.34\.ffn_down_exps\.weight=IQ4_XS
307
+ blk\.34\.ffn_down_shexp\.weight=Q8_0
308
+ blk\.34\.ffn_gate_exps\.weight=IQ4_XS
309
+ blk\.34\.ffn_gate_shexp\.weight=Q8_0
310
+ blk\.34\.ffn_up_exps\.weight=IQ4_XS
311
+ blk\.34\.ffn_up_shexp\.weight=Q8_0
312
+ blk\.34\.ssm_alpha\.weight=F32
313
+ blk\.34\.ssm_beta\.weight=F32
314
+ blk\.34\.ssm_out\.weight=Q8_0
315
+ blk\.35\.attn_k\.weight=F16
316
+ blk\.35\.attn_output\.weight=Q8_0
317
+ blk\.35\.attn_q\.weight=Q8_0
318
+ blk\.35\.attn_v\.weight=F16
319
+ blk\.35\.ffn_down_exps\.weight=IQ3_S
320
+ blk\.35\.ffn_down_shexp\.weight=Q8_0
321
+ blk\.35\.ffn_gate_exps\.weight=IQ3_S
322
+ blk\.35\.ffn_gate_shexp\.weight=Q8_0
323
+ blk\.35\.ffn_up_exps\.weight=IQ3_S
324
+ blk\.35\.ffn_up_shexp\.weight=Q8_0
325
+ blk\.36\.attn_gate\.weight=Q8_0
326
+ blk\.36\.attn_qkv\.weight=Q8_0
327
+ blk\.36\.ffn_down_exps\.weight=IQ4_XS
328
+ blk\.36\.ffn_down_shexp\.weight=Q8_0
329
+ blk\.36\.ffn_gate_exps\.weight=IQ4_XS
330
+ blk\.36\.ffn_gate_shexp\.weight=Q8_0
331
+ blk\.36\.ffn_up_exps\.weight=IQ4_XS
332
+ blk\.36\.ffn_up_shexp\.weight=Q8_0
333
+ blk\.36\.ssm_alpha\.weight=F32
334
+ blk\.36\.ssm_beta\.weight=F32
335
+ blk\.36\.ssm_out\.weight=Q8_0
336
+ blk\.37\.attn_gate\.weight=Q8_0
337
+ blk\.37\.attn_qkv\.weight=Q8_0
338
+ blk\.37\.ffn_down_exps\.weight=IQ4_XS
339
+ blk\.37\.ffn_down_shexp\.weight=Q8_0
340
+ blk\.37\.ffn_gate_exps\.weight=IQ4_XS
341
+ blk\.37\.ffn_gate_shexp\.weight=Q8_0
342
+ blk\.37\.ffn_up_exps\.weight=IQ4_XS
343
+ blk\.37\.ffn_up_shexp\.weight=Q8_0
344
+ blk\.37\.ssm_alpha\.weight=F32
345
+ blk\.37\.ssm_beta\.weight=F32
346
+ blk\.37\.ssm_out\.weight=Q8_0
347
+ blk\.38\.attn_gate\.weight=Q8_0
348
+ blk\.38\.attn_qkv\.weight=Q8_0
349
+ blk\.38\.ffn_down_exps\.weight=IQ3_S
350
+ blk\.38\.ffn_down_shexp\.weight=Q8_0
351
+ blk\.38\.ffn_gate_exps\.weight=IQ3_S
352
+ blk\.38\.ffn_gate_shexp\.weight=Q8_0
353
+ blk\.38\.ffn_up_exps\.weight=IQ3_S
354
+ blk\.38\.ffn_up_shexp\.weight=Q8_0
355
+ blk\.38\.ssm_alpha\.weight=F32
356
+ blk\.38\.ssm_beta\.weight=F32
357
+ blk\.38\.ssm_out\.weight=Q8_0
358
+ blk\.39\.attn_k\.weight=F16
359
+ blk\.39\.attn_output\.weight=Q8_0
360
+ blk\.39\.attn_q\.weight=Q8_0
361
+ blk\.39\.attn_v\.weight=F16
362
+ blk\.39\.ffn_down_exps\.weight=IQ4_XS
363
+ blk\.39\.ffn_down_shexp\.weight=Q8_0
364
+ blk\.39\.ffn_gate_exps\.weight=IQ4_XS
365
+ blk\.39\.ffn_gate_shexp\.weight=Q8_0
366
+ blk\.39\.ffn_up_exps\.weight=IQ4_XS
367
+ blk\.39\.ffn_up_shexp\.weight=Q8_0
368
+ blk\.4\.attn_gate\.weight=Q8_0
369
+ blk\.4\.attn_qkv\.weight=Q8_0
370
+ blk\.4\.ffn_down_exps\.weight=IQ4_XS
371
+ blk\.4\.ffn_down_shexp\.weight=Q8_0
372
+ blk\.4\.ffn_gate_exps\.weight=IQ4_XS
373
+ blk\.4\.ffn_gate_shexp\.weight=Q8_0
374
+ blk\.4\.ffn_up_exps\.weight=IQ4_XS
375
+ blk\.4\.ffn_up_shexp\.weight=Q8_0
376
+ blk\.4\.ssm_alpha\.weight=F32
377
+ blk\.4\.ssm_beta\.weight=F32
378
+ blk\.4\.ssm_out\.weight=Q8_0
379
+ blk\.40\.attn_k\.weight=F16
380
+ blk\.40\.attn_output\.weight=Q8_0
381
+ blk\.40\.attn_q\.weight=Q8_0
382
+ blk\.40\.attn_v\.weight=F16
383
+ blk\.40\.ffn_down_exps\.weight=IQ3_S
384
+ blk\.40\.ffn_down_shexp\.weight=Q8_0
385
+ blk\.40\.ffn_gate_exps\.weight=IQ3_S
386
+ blk\.40\.ffn_gate_shexp\.weight=Q8_0
387
+ blk\.40\.ffn_up_exps\.weight=IQ3_S
388
+ blk\.40\.ffn_up_shexp\.weight=Q8_0
389
+ blk\.40\.nextn\.eh_proj\.weight=Q8_0
390
+ blk\.5\.attn_gate\.weight=Q8_0
391
+ blk\.5\.attn_qkv\.weight=Q8_0
392
+ blk\.5\.ffn_down_exps\.weight=IQ4_XS
393
+ blk\.5\.ffn_down_shexp\.weight=Q8_0
394
+ blk\.5\.ffn_gate_exps\.weight=IQ4_XS
395
+ blk\.5\.ffn_gate_shexp\.weight=Q8_0
396
+ blk\.5\.ffn_up_exps\.weight=IQ4_XS
397
+ blk\.5\.ffn_up_shexp\.weight=Q8_0
398
+ blk\.5\.ssm_alpha\.weight=F32
399
+ blk\.5\.ssm_beta\.weight=F32
400
+ blk\.5\.ssm_out\.weight=Q8_0
401
+ blk\.6\.attn_gate\.weight=Q8_0
402
+ blk\.6\.attn_qkv\.weight=Q8_0
403
+ blk\.6\.ffn_down_exps\.weight=IQ3_S
404
+ blk\.6\.ffn_down_shexp\.weight=Q8_0
405
+ blk\.6\.ffn_gate_exps\.weight=IQ3_S
406
+ blk\.6\.ffn_gate_shexp\.weight=Q8_0
407
+ blk\.6\.ffn_up_exps\.weight=IQ3_S
408
+ blk\.6\.ffn_up_shexp\.weight=Q8_0
409
+ blk\.6\.ssm_alpha\.weight=F32
410
+ blk\.6\.ssm_beta\.weight=F32
411
+ blk\.6\.ssm_out\.weight=Q8_0
412
+ blk\.7\.attn_k\.weight=F16
413
+ blk\.7\.attn_output\.weight=Q8_0
414
+ blk\.7\.attn_q\.weight=Q8_0
415
+ blk\.7\.attn_v\.weight=F16
416
+ blk\.7\.ffn_down_exps\.weight=IQ4_XS
417
+ blk\.7\.ffn_down_shexp\.weight=Q8_0
418
+ blk\.7\.ffn_gate_exps\.weight=IQ4_XS
419
+ blk\.7\.ffn_gate_shexp\.weight=Q8_0
420
+ blk\.7\.ffn_up_exps\.weight=IQ4_XS
421
+ blk\.7\.ffn_up_shexp\.weight=Q8_0
422
+ blk\.8\.attn_gate\.weight=Q8_0
423
+ blk\.8\.attn_qkv\.weight=Q8_0
424
+ blk\.8\.ffn_down_exps\.weight=IQ3_S
425
+ blk\.8\.ffn_down_shexp\.weight=Q8_0
426
+ blk\.8\.ffn_gate_exps\.weight=IQ3_S
427
+ blk\.8\.ffn_gate_shexp\.weight=Q8_0
428
+ blk\.8\.ffn_up_exps\.weight=IQ3_S
429
+ blk\.8\.ffn_up_shexp\.weight=Q8_0
430
+ blk\.8\.ssm_alpha\.weight=F32
431
+ blk\.8\.ssm_beta\.weight=F32
432
+ blk\.8\.ssm_out\.weight=Q8_0
433
+ blk\.9\.attn_gate\.weight=Q8_0
434
+ blk\.9\.attn_qkv\.weight=Q8_0
435
+ blk\.9\.ffn_down_exps\.weight=IQ4_XS
436
+ blk\.9\.ffn_down_shexp\.weight=Q8_0
437
+ blk\.9\.ffn_gate_exps\.weight=IQ4_XS
438
+ blk\.9\.ffn_gate_shexp\.weight=Q8_0
439
+ blk\.9\.ffn_up_exps\.weight=IQ4_XS
440
+ blk\.9\.ffn_up_shexp\.weight=Q8_0
441
+ blk\.9\.ssm_alpha\.weight=F32
442
+ blk\.9\.ssm_beta\.weight=F32
443
+ blk\.9\.ssm_out\.weight=Q8_0
appendix/recipes/cfg_21G-ICE_ICEbase.txt ADDED
@@ -0,0 +1,443 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ^output\.weight$=Q8_0
2
+ token_embd\.weight=Q8_0
3
+ blk\.0\.attn_gate\.weight=Q8_0
4
+ blk\.0\.attn_qkv\.weight=Q8_0
5
+ blk\.0\.ffn_down_exps\.weight=Q4_K
6
+ blk\.0\.ffn_down_shexp\.weight=Q8_0
7
+ blk\.0\.ffn_gate_exps\.weight=Q4_K
8
+ blk\.0\.ffn_gate_shexp\.weight=Q8_0
9
+ blk\.0\.ffn_up_exps\.weight=Q4_K
10
+ blk\.0\.ffn_up_shexp\.weight=Q8_0
11
+ blk\.0\.ssm_alpha\.weight=F32
12
+ blk\.0\.ssm_beta\.weight=F32
13
+ blk\.0\.ssm_out\.weight=Q8_0
14
+ blk\.1\.attn_gate\.weight=Q8_0
15
+ blk\.1\.attn_qkv\.weight=Q8_0
16
+ blk\.1\.ffn_down_exps\.weight=IQ4_XS
17
+ blk\.1\.ffn_down_shexp\.weight=Q8_0
18
+ blk\.1\.ffn_gate_exps\.weight=IQ4_XS
19
+ blk\.1\.ffn_gate_shexp\.weight=Q8_0
20
+ blk\.1\.ffn_up_exps\.weight=IQ4_XS
21
+ blk\.1\.ffn_up_shexp\.weight=Q8_0
22
+ blk\.1\.ssm_alpha\.weight=F32
23
+ blk\.1\.ssm_beta\.weight=F32
24
+ blk\.1\.ssm_out\.weight=Q8_0
25
+ blk\.10\.attn_gate\.weight=Q8_0
26
+ blk\.10\.attn_qkv\.weight=Q8_0
27
+ blk\.10\.ffn_down_exps\.weight=IQ4_XS
28
+ blk\.10\.ffn_down_shexp\.weight=Q8_0
29
+ blk\.10\.ffn_gate_exps\.weight=IQ4_XS
30
+ blk\.10\.ffn_gate_shexp\.weight=Q8_0
31
+ blk\.10\.ffn_up_exps\.weight=IQ4_XS
32
+ blk\.10\.ffn_up_shexp\.weight=Q8_0
33
+ blk\.10\.ssm_alpha\.weight=F32
34
+ blk\.10\.ssm_beta\.weight=F32
35
+ blk\.10\.ssm_out\.weight=Q8_0
36
+ blk\.11\.attn_k\.weight=F16
37
+ blk\.11\.attn_output\.weight=Q8_0
38
+ blk\.11\.attn_q\.weight=Q8_0
39
+ blk\.11\.attn_v\.weight=F16
40
+ blk\.11\.ffn_down_exps\.weight=Q4_K
41
+ blk\.11\.ffn_down_shexp\.weight=Q8_0
42
+ blk\.11\.ffn_gate_exps\.weight=Q4_K
43
+ blk\.11\.ffn_gate_shexp\.weight=Q8_0
44
+ blk\.11\.ffn_up_exps\.weight=Q4_K
45
+ blk\.11\.ffn_up_shexp\.weight=Q8_0
46
+ blk\.12\.attn_gate\.weight=Q8_0
47
+ blk\.12\.attn_qkv\.weight=Q8_0
48
+ blk\.12\.ffn_down_exps\.weight=IQ4_XS
49
+ blk\.12\.ffn_down_shexp\.weight=Q8_0
50
+ blk\.12\.ffn_gate_exps\.weight=IQ4_XS
51
+ blk\.12\.ffn_gate_shexp\.weight=Q8_0
52
+ blk\.12\.ffn_up_exps\.weight=IQ4_XS
53
+ blk\.12\.ffn_up_shexp\.weight=Q8_0
54
+ blk\.12\.ssm_alpha\.weight=F32
55
+ blk\.12\.ssm_beta\.weight=F32
56
+ blk\.12\.ssm_out\.weight=Q8_0
57
+ blk\.13\.attn_gate\.weight=Q8_0
58
+ blk\.13\.attn_qkv\.weight=Q8_0
59
+ blk\.13\.ffn_down_exps\.weight=Q4_K
60
+ blk\.13\.ffn_down_shexp\.weight=Q8_0
61
+ blk\.13\.ffn_gate_exps\.weight=Q4_K
62
+ blk\.13\.ffn_gate_shexp\.weight=Q8_0
63
+ blk\.13\.ffn_up_exps\.weight=Q4_K
64
+ blk\.13\.ffn_up_shexp\.weight=Q8_0
65
+ blk\.13\.ssm_alpha\.weight=F32
66
+ blk\.13\.ssm_beta\.weight=F32
67
+ blk\.13\.ssm_out\.weight=Q8_0
68
+ blk\.14\.attn_gate\.weight=Q8_0
69
+ blk\.14\.attn_qkv\.weight=Q8_0
70
+ blk\.14\.ffn_down_exps\.weight=IQ4_XS
71
+ blk\.14\.ffn_down_shexp\.weight=Q8_0
72
+ blk\.14\.ffn_gate_exps\.weight=IQ4_XS
73
+ blk\.14\.ffn_gate_shexp\.weight=Q8_0
74
+ blk\.14\.ffn_up_exps\.weight=IQ4_XS
75
+ blk\.14\.ffn_up_shexp\.weight=Q8_0
76
+ blk\.14\.ssm_alpha\.weight=F32
77
+ blk\.14\.ssm_beta\.weight=F32
78
+ blk\.14\.ssm_out\.weight=Q8_0
79
+ blk\.15\.attn_k\.weight=F16
80
+ blk\.15\.attn_output\.weight=Q8_0
81
+ blk\.15\.attn_q\.weight=Q8_0
82
+ blk\.15\.attn_v\.weight=F16
83
+ blk\.15\.ffn_down_exps\.weight=Q4_K
84
+ blk\.15\.ffn_down_shexp\.weight=Q8_0
85
+ blk\.15\.ffn_gate_exps\.weight=Q4_K
86
+ blk\.15\.ffn_gate_shexp\.weight=Q8_0
87
+ blk\.15\.ffn_up_exps\.weight=Q4_K
88
+ blk\.15\.ffn_up_shexp\.weight=Q8_0
89
+ blk\.16\.attn_gate\.weight=Q8_0
90
+ blk\.16\.attn_qkv\.weight=Q8_0
91
+ blk\.16\.ffn_down_exps\.weight=IQ4_XS
92
+ blk\.16\.ffn_down_shexp\.weight=Q8_0
93
+ blk\.16\.ffn_gate_exps\.weight=IQ4_XS
94
+ blk\.16\.ffn_gate_shexp\.weight=Q8_0
95
+ blk\.16\.ffn_up_exps\.weight=IQ4_XS
96
+ blk\.16\.ffn_up_shexp\.weight=Q8_0
97
+ blk\.16\.ssm_alpha\.weight=F32
98
+ blk\.16\.ssm_beta\.weight=F32
99
+ blk\.16\.ssm_out\.weight=Q8_0
100
+ blk\.17\.attn_gate\.weight=Q8_0
101
+ blk\.17\.attn_qkv\.weight=Q8_0
102
+ blk\.17\.ffn_down_exps\.weight=Q4_K
103
+ blk\.17\.ffn_down_shexp\.weight=Q8_0
104
+ blk\.17\.ffn_gate_exps\.weight=Q4_K
105
+ blk\.17\.ffn_gate_shexp\.weight=Q8_0
106
+ blk\.17\.ffn_up_exps\.weight=Q4_K
107
+ blk\.17\.ffn_up_shexp\.weight=Q8_0
108
+ blk\.17\.ssm_alpha\.weight=F32
109
+ blk\.17\.ssm_beta\.weight=F32
110
+ blk\.17\.ssm_out\.weight=Q8_0
111
+ blk\.18\.attn_gate\.weight=Q8_0
112
+ blk\.18\.attn_qkv\.weight=Q8_0
113
+ blk\.18\.ffn_down_exps\.weight=IQ4_XS
114
+ blk\.18\.ffn_down_shexp\.weight=Q8_0
115
+ blk\.18\.ffn_gate_exps\.weight=IQ4_XS
116
+ blk\.18\.ffn_gate_shexp\.weight=Q8_0
117
+ blk\.18\.ffn_up_exps\.weight=IQ4_XS
118
+ blk\.18\.ffn_up_shexp\.weight=Q8_0
119
+ blk\.18\.ssm_alpha\.weight=F32
120
+ blk\.18\.ssm_beta\.weight=F32
121
+ blk\.18\.ssm_out\.weight=Q8_0
122
+ blk\.19\.attn_k\.weight=F16
123
+ blk\.19\.attn_output\.weight=Q8_0
124
+ blk\.19\.attn_q\.weight=Q8_0
125
+ blk\.19\.attn_v\.weight=F16
126
+ blk\.19\.ffn_down_exps\.weight=Q4_K
127
+ blk\.19\.ffn_down_shexp\.weight=Q8_0
128
+ blk\.19\.ffn_gate_exps\.weight=Q4_K
129
+ blk\.19\.ffn_gate_shexp\.weight=Q8_0
130
+ blk\.19\.ffn_up_exps\.weight=Q4_K
131
+ blk\.19\.ffn_up_shexp\.weight=Q8_0
132
+ blk\.2\.attn_gate\.weight=Q8_0
133
+ blk\.2\.attn_qkv\.weight=Q8_0
134
+ blk\.2\.ffn_down_exps\.weight=Q4_K
135
+ blk\.2\.ffn_down_shexp\.weight=Q8_0
136
+ blk\.2\.ffn_gate_exps\.weight=Q4_K
137
+ blk\.2\.ffn_gate_shexp\.weight=Q8_0
138
+ blk\.2\.ffn_up_exps\.weight=Q4_K
139
+ blk\.2\.ffn_up_shexp\.weight=Q8_0
140
+ blk\.2\.ssm_alpha\.weight=F32
141
+ blk\.2\.ssm_beta\.weight=F32
142
+ blk\.2\.ssm_out\.weight=Q8_0
143
+ blk\.20\.attn_gate\.weight=Q8_0
144
+ blk\.20\.attn_qkv\.weight=Q8_0
145
+ blk\.20\.ffn_down_exps\.weight=Q4_K
146
+ blk\.20\.ffn_down_shexp\.weight=Q8_0
147
+ blk\.20\.ffn_gate_exps\.weight=Q4_K
148
+ blk\.20\.ffn_gate_shexp\.weight=Q8_0
149
+ blk\.20\.ffn_up_exps\.weight=Q4_K
150
+ blk\.20\.ffn_up_shexp\.weight=Q8_0
151
+ blk\.20\.ssm_alpha\.weight=F32
152
+ blk\.20\.ssm_beta\.weight=F32
153
+ blk\.20\.ssm_out\.weight=Q8_0
154
+ blk\.21\.attn_gate\.weight=Q8_0
155
+ blk\.21\.attn_qkv\.weight=Q8_0
156
+ blk\.21\.ffn_down_exps\.weight=IQ4_XS
157
+ blk\.21\.ffn_down_shexp\.weight=Q8_0
158
+ blk\.21\.ffn_gate_exps\.weight=IQ4_XS
159
+ blk\.21\.ffn_gate_shexp\.weight=Q8_0
160
+ blk\.21\.ffn_up_exps\.weight=IQ4_XS
161
+ blk\.21\.ffn_up_shexp\.weight=Q8_0
162
+ blk\.21\.ssm_alpha\.weight=F32
163
+ blk\.21\.ssm_beta\.weight=F32
164
+ blk\.21\.ssm_out\.weight=Q8_0
165
+ blk\.22\.attn_gate\.weight=Q8_0
166
+ blk\.22\.attn_qkv\.weight=Q8_0
167
+ blk\.22\.ffn_down_exps\.weight=Q4_K
168
+ blk\.22\.ffn_down_shexp\.weight=Q8_0
169
+ blk\.22\.ffn_gate_exps\.weight=Q4_K
170
+ blk\.22\.ffn_gate_shexp\.weight=Q8_0
171
+ blk\.22\.ffn_up_exps\.weight=Q4_K
172
+ blk\.22\.ffn_up_shexp\.weight=Q8_0
173
+ blk\.22\.ssm_alpha\.weight=F32
174
+ blk\.22\.ssm_beta\.weight=F32
175
+ blk\.22\.ssm_out\.weight=Q8_0
176
+ blk\.23\.attn_k\.weight=F16
177
+ blk\.23\.attn_output\.weight=Q8_0
178
+ blk\.23\.attn_q\.weight=Q8_0
179
+ blk\.23\.attn_v\.weight=F16
180
+ blk\.23\.ffn_down_exps\.weight=IQ4_XS
181
+ blk\.23\.ffn_down_shexp\.weight=Q8_0
182
+ blk\.23\.ffn_gate_exps\.weight=IQ4_XS
183
+ blk\.23\.ffn_gate_shexp\.weight=Q8_0
184
+ blk\.23\.ffn_up_exps\.weight=IQ4_XS
185
+ blk\.23\.ffn_up_shexp\.weight=Q8_0
186
+ blk\.24\.attn_gate\.weight=Q8_0
187
+ blk\.24\.attn_qkv\.weight=Q8_0
188
+ blk\.24\.ffn_down_exps\.weight=Q4_K
189
+ blk\.24\.ffn_down_shexp\.weight=Q8_0
190
+ blk\.24\.ffn_gate_exps\.weight=Q4_K
191
+ blk\.24\.ffn_gate_shexp\.weight=Q8_0
192
+ blk\.24\.ffn_up_exps\.weight=Q4_K
193
+ blk\.24\.ffn_up_shexp\.weight=Q8_0
194
+ blk\.24\.ssm_alpha\.weight=F32
195
+ blk\.24\.ssm_beta\.weight=F32
196
+ blk\.24\.ssm_out\.weight=Q8_0
197
+ blk\.25\.attn_gate\.weight=Q8_0
198
+ blk\.25\.attn_qkv\.weight=Q8_0
199
+ blk\.25\.ffn_down_exps\.weight=IQ4_XS
200
+ blk\.25\.ffn_down_shexp\.weight=Q8_0
201
+ blk\.25\.ffn_gate_exps\.weight=IQ4_XS
202
+ blk\.25\.ffn_gate_shexp\.weight=Q8_0
203
+ blk\.25\.ffn_up_exps\.weight=IQ4_XS
204
+ blk\.25\.ffn_up_shexp\.weight=Q8_0
205
+ blk\.25\.ssm_alpha\.weight=F32
206
+ blk\.25\.ssm_beta\.weight=F32
207
+ blk\.25\.ssm_out\.weight=Q8_0
208
+ blk\.26\.attn_gate\.weight=Q8_0
209
+ blk\.26\.attn_qkv\.weight=Q8_0
210
+ blk\.26\.ffn_down_exps\.weight=Q4_K
211
+ blk\.26\.ffn_down_shexp\.weight=Q8_0
212
+ blk\.26\.ffn_gate_exps\.weight=Q4_K
213
+ blk\.26\.ffn_gate_shexp\.weight=Q8_0
214
+ blk\.26\.ffn_up_exps\.weight=Q4_K
215
+ blk\.26\.ffn_up_shexp\.weight=Q8_0
216
+ blk\.26\.ssm_alpha\.weight=F32
217
+ blk\.26\.ssm_beta\.weight=F32
218
+ blk\.26\.ssm_out\.weight=Q8_0
219
+ blk\.27\.attn_k\.weight=F16
220
+ blk\.27\.attn_output\.weight=Q8_0
221
+ blk\.27\.attn_q\.weight=Q8_0
222
+ blk\.27\.attn_v\.weight=F16
223
+ blk\.27\.ffn_down_exps\.weight=IQ4_XS
224
+ blk\.27\.ffn_down_shexp\.weight=Q8_0
225
+ blk\.27\.ffn_gate_exps\.weight=IQ4_XS
226
+ blk\.27\.ffn_gate_shexp\.weight=Q8_0
227
+ blk\.27\.ffn_up_exps\.weight=IQ4_XS
228
+ blk\.27\.ffn_up_shexp\.weight=Q8_0
229
+ blk\.28\.attn_gate\.weight=Q8_0
230
+ blk\.28\.attn_qkv\.weight=Q8_0
231
+ blk\.28\.ffn_down_exps\.weight=Q4_K
232
+ blk\.28\.ffn_down_shexp\.weight=Q8_0
233
+ blk\.28\.ffn_gate_exps\.weight=Q4_K
234
+ blk\.28\.ffn_gate_shexp\.weight=Q8_0
235
+ blk\.28\.ffn_up_exps\.weight=Q4_K
236
+ blk\.28\.ffn_up_shexp\.weight=Q8_0
237
+ blk\.28\.ssm_alpha\.weight=F32
238
+ blk\.28\.ssm_beta\.weight=F32
239
+ blk\.28\.ssm_out\.weight=Q8_0
240
+ blk\.29\.attn_gate\.weight=Q8_0
241
+ blk\.29\.attn_qkv\.weight=Q8_0
242
+ blk\.29\.ffn_down_exps\.weight=IQ4_XS
243
+ blk\.29\.ffn_down_shexp\.weight=Q8_0
244
+ blk\.29\.ffn_gate_exps\.weight=IQ4_XS
245
+ blk\.29\.ffn_gate_shexp\.weight=Q8_0
246
+ blk\.29\.ffn_up_exps\.weight=IQ4_XS
247
+ blk\.29\.ffn_up_shexp\.weight=Q8_0
248
+ blk\.29\.ssm_alpha\.weight=F32
249
+ blk\.29\.ssm_beta\.weight=F32
250
+ blk\.29\.ssm_out\.weight=Q8_0
251
+ blk\.3\.attn_k\.weight=F16
252
+ blk\.3\.attn_output\.weight=Q8_0
253
+ blk\.3\.attn_q\.weight=Q8_0
254
+ blk\.3\.attn_v\.weight=F16
255
+ blk\.3\.ffn_down_exps\.weight=IQ4_XS
256
+ blk\.3\.ffn_down_shexp\.weight=Q8_0
257
+ blk\.3\.ffn_gate_exps\.weight=IQ4_XS
258
+ blk\.3\.ffn_gate_shexp\.weight=Q8_0
259
+ blk\.3\.ffn_up_exps\.weight=IQ4_XS
260
+ blk\.3\.ffn_up_shexp\.weight=Q8_0
261
+ blk\.30\.attn_gate\.weight=Q8_0
262
+ blk\.30\.attn_qkv\.weight=Q8_0
263
+ blk\.30\.ffn_down_exps\.weight=Q4_K
264
+ blk\.30\.ffn_down_shexp\.weight=Q8_0
265
+ blk\.30\.ffn_gate_exps\.weight=Q4_K
266
+ blk\.30\.ffn_gate_shexp\.weight=Q8_0
267
+ blk\.30\.ffn_up_exps\.weight=Q4_K
268
+ blk\.30\.ffn_up_shexp\.weight=Q8_0
269
+ blk\.30\.ssm_alpha\.weight=F32
270
+ blk\.30\.ssm_beta\.weight=F32
271
+ blk\.30\.ssm_out\.weight=Q8_0
272
+ blk\.31\.attn_k\.weight=F16
273
+ blk\.31\.attn_output\.weight=Q8_0
274
+ blk\.31\.attn_q\.weight=Q8_0
275
+ blk\.31\.attn_v\.weight=F16
276
+ blk\.31\.ffn_down_exps\.weight=IQ4_XS
277
+ blk\.31\.ffn_down_shexp\.weight=Q8_0
278
+ blk\.31\.ffn_gate_exps\.weight=IQ4_XS
279
+ blk\.31\.ffn_gate_shexp\.weight=Q8_0
280
+ blk\.31\.ffn_up_exps\.weight=IQ4_XS
281
+ blk\.31\.ffn_up_shexp\.weight=Q8_0
282
+ blk\.32\.attn_gate\.weight=Q8_0
283
+ blk\.32\.attn_qkv\.weight=Q8_0
284
+ blk\.32\.ffn_down_exps\.weight=Q4_K
285
+ blk\.32\.ffn_down_shexp\.weight=Q8_0
286
+ blk\.32\.ffn_gate_exps\.weight=Q4_K
287
+ blk\.32\.ffn_gate_shexp\.weight=Q8_0
288
+ blk\.32\.ffn_up_exps\.weight=Q4_K
289
+ blk\.32\.ffn_up_shexp\.weight=Q8_0
290
+ blk\.32\.ssm_alpha\.weight=F32
291
+ blk\.32\.ssm_beta\.weight=F32
292
+ blk\.32\.ssm_out\.weight=Q8_0
293
+ blk\.33\.attn_gate\.weight=Q8_0
294
+ blk\.33\.attn_qkv\.weight=Q8_0
295
+ blk\.33\.ffn_down_exps\.weight=IQ4_XS
296
+ blk\.33\.ffn_down_shexp\.weight=Q8_0
297
+ blk\.33\.ffn_gate_exps\.weight=IQ4_XS
298
+ blk\.33\.ffn_gate_shexp\.weight=Q8_0
299
+ blk\.33\.ffn_up_exps\.weight=IQ4_XS
300
+ blk\.33\.ffn_up_shexp\.weight=Q8_0
301
+ blk\.33\.ssm_alpha\.weight=F32
302
+ blk\.33\.ssm_beta\.weight=F32
303
+ blk\.33\.ssm_out\.weight=Q8_0
304
+ blk\.34\.attn_gate\.weight=Q8_0
305
+ blk\.34\.attn_qkv\.weight=Q8_0
306
+ blk\.34\.ffn_down_exps\.weight=Q4_K
307
+ blk\.34\.ffn_down_shexp\.weight=Q8_0
308
+ blk\.34\.ffn_gate_exps\.weight=Q4_K
309
+ blk\.34\.ffn_gate_shexp\.weight=Q8_0
310
+ blk\.34\.ffn_up_exps\.weight=Q4_K
311
+ blk\.34\.ffn_up_shexp\.weight=Q8_0
312
+ blk\.34\.ssm_alpha\.weight=F32
313
+ blk\.34\.ssm_beta\.weight=F32
314
+ blk\.34\.ssm_out\.weight=Q8_0
315
+ blk\.35\.attn_k\.weight=F16
316
+ blk\.35\.attn_output\.weight=Q8_0
317
+ blk\.35\.attn_q\.weight=Q8_0
318
+ blk\.35\.attn_v\.weight=F16
319
+ blk\.35\.ffn_down_exps\.weight=Q4_K
320
+ blk\.35\.ffn_down_shexp\.weight=Q8_0
321
+ blk\.35\.ffn_gate_exps\.weight=Q4_K
322
+ blk\.35\.ffn_gate_shexp\.weight=Q8_0
323
+ blk\.35\.ffn_up_exps\.weight=Q4_K
324
+ blk\.35\.ffn_up_shexp\.weight=Q8_0
325
+ blk\.36\.attn_gate\.weight=Q8_0
326
+ blk\.36\.attn_qkv\.weight=Q8_0
327
+ blk\.36\.ffn_down_exps\.weight=IQ4_XS
328
+ blk\.36\.ffn_down_shexp\.weight=Q8_0
329
+ blk\.36\.ffn_gate_exps\.weight=IQ4_XS
330
+ blk\.36\.ffn_gate_shexp\.weight=Q8_0
331
+ blk\.36\.ffn_up_exps\.weight=IQ4_XS
332
+ blk\.36\.ffn_up_shexp\.weight=Q8_0
333
+ blk\.36\.ssm_alpha\.weight=F32
334
+ blk\.36\.ssm_beta\.weight=F32
335
+ blk\.36\.ssm_out\.weight=Q8_0
336
+ blk\.37\.attn_gate\.weight=Q8_0
337
+ blk\.37\.attn_qkv\.weight=Q8_0
338
+ blk\.37\.ffn_down_exps\.weight=Q4_K
339
+ blk\.37\.ffn_down_shexp\.weight=Q8_0
340
+ blk\.37\.ffn_gate_exps\.weight=Q4_K
341
+ blk\.37\.ffn_gate_shexp\.weight=Q8_0
342
+ blk\.37\.ffn_up_exps\.weight=Q4_K
343
+ blk\.37\.ffn_up_shexp\.weight=Q8_0
344
+ blk\.37\.ssm_alpha\.weight=F32
345
+ blk\.37\.ssm_beta\.weight=F32
346
+ blk\.37\.ssm_out\.weight=Q8_0
347
+ blk\.38\.attn_gate\.weight=Q8_0
348
+ blk\.38\.attn_qkv\.weight=Q8_0
349
+ blk\.38\.ffn_down_exps\.weight=IQ4_XS
350
+ blk\.38\.ffn_down_shexp\.weight=Q8_0
351
+ blk\.38\.ffn_gate_exps\.weight=IQ4_XS
352
+ blk\.38\.ffn_gate_shexp\.weight=Q8_0
353
+ blk\.38\.ffn_up_exps\.weight=IQ4_XS
354
+ blk\.38\.ffn_up_shexp\.weight=Q8_0
355
+ blk\.38\.ssm_alpha\.weight=F32
356
+ blk\.38\.ssm_beta\.weight=F32
357
+ blk\.38\.ssm_out\.weight=Q8_0
358
+ blk\.39\.attn_k\.weight=F16
359
+ blk\.39\.attn_output\.weight=Q8_0
360
+ blk\.39\.attn_q\.weight=Q8_0
361
+ blk\.39\.attn_v\.weight=F16
362
+ blk\.39\.ffn_down_exps\.weight=Q4_K
363
+ blk\.39\.ffn_down_shexp\.weight=Q8_0
364
+ blk\.39\.ffn_gate_exps\.weight=Q4_K
365
+ blk\.39\.ffn_gate_shexp\.weight=Q8_0
366
+ blk\.39\.ffn_up_exps\.weight=Q4_K
367
+ blk\.39\.ffn_up_shexp\.weight=Q8_0
368
+ blk\.4\.attn_gate\.weight=Q8_0
369
+ blk\.4\.attn_qkv\.weight=Q8_0
370
+ blk\.4\.ffn_down_exps\.weight=Q4_K
371
+ blk\.4\.ffn_down_shexp\.weight=Q8_0
372
+ blk\.4\.ffn_gate_exps\.weight=Q4_K
373
+ blk\.4\.ffn_gate_shexp\.weight=Q8_0
374
+ blk\.4\.ffn_up_exps\.weight=Q4_K
375
+ blk\.4\.ffn_up_shexp\.weight=Q8_0
376
+ blk\.4\.ssm_alpha\.weight=F32
377
+ blk\.4\.ssm_beta\.weight=F32
378
+ blk\.4\.ssm_out\.weight=Q8_0
379
+ blk\.40\.attn_k\.weight=F16
380
+ blk\.40\.attn_output\.weight=Q8_0
381
+ blk\.40\.attn_q\.weight=Q8_0
382
+ blk\.40\.attn_v\.weight=F16
383
+ blk\.40\.ffn_down_exps\.weight=IQ4_XS
384
+ blk\.40\.ffn_down_shexp\.weight=Q8_0
385
+ blk\.40\.ffn_gate_exps\.weight=IQ4_XS
386
+ blk\.40\.ffn_gate_shexp\.weight=Q8_0
387
+ blk\.40\.ffn_up_exps\.weight=IQ4_XS
388
+ blk\.40\.ffn_up_shexp\.weight=Q8_0
389
+ blk\.40\.nextn\.eh_proj\.weight=Q8_0
390
+ blk\.5\.attn_gate\.weight=Q8_0
391
+ blk\.5\.attn_qkv\.weight=Q8_0
392
+ blk\.5\.ffn_down_exps\.weight=IQ4_XS
393
+ blk\.5\.ffn_down_shexp\.weight=Q8_0
394
+ blk\.5\.ffn_gate_exps\.weight=IQ4_XS
395
+ blk\.5\.ffn_gate_shexp\.weight=Q8_0
396
+ blk\.5\.ffn_up_exps\.weight=IQ4_XS
397
+ blk\.5\.ffn_up_shexp\.weight=Q8_0
398
+ blk\.5\.ssm_alpha\.weight=F32
399
+ blk\.5\.ssm_beta\.weight=F32
400
+ blk\.5\.ssm_out\.weight=Q8_0
401
+ blk\.6\.attn_gate\.weight=Q8_0
402
+ blk\.6\.attn_qkv\.weight=Q8_0
403
+ blk\.6\.ffn_down_exps\.weight=Q4_K
404
+ blk\.6\.ffn_down_shexp\.weight=Q8_0
405
+ blk\.6\.ffn_gate_exps\.weight=Q4_K
406
+ blk\.6\.ffn_gate_shexp\.weight=Q8_0
407
+ blk\.6\.ffn_up_exps\.weight=Q4_K
408
+ blk\.6\.ffn_up_shexp\.weight=Q8_0
409
+ blk\.6\.ssm_alpha\.weight=F32
410
+ blk\.6\.ssm_beta\.weight=F32
411
+ blk\.6\.ssm_out\.weight=Q8_0
412
+ blk\.7\.attn_k\.weight=F16
413
+ blk\.7\.attn_output\.weight=Q8_0
414
+ blk\.7\.attn_q\.weight=Q8_0
415
+ blk\.7\.attn_v\.weight=F16
416
+ blk\.7\.ffn_down_exps\.weight=Q4_K
417
+ blk\.7\.ffn_down_shexp\.weight=Q8_0
418
+ blk\.7\.ffn_gate_exps\.weight=Q4_K
419
+ blk\.7\.ffn_gate_shexp\.weight=Q8_0
420
+ blk\.7\.ffn_up_exps\.weight=Q4_K
421
+ blk\.7\.ffn_up_shexp\.weight=Q8_0
422
+ blk\.8\.attn_gate\.weight=Q8_0
423
+ blk\.8\.attn_qkv\.weight=Q8_0
424
+ blk\.8\.ffn_down_exps\.weight=IQ4_XS
425
+ blk\.8\.ffn_down_shexp\.weight=Q8_0
426
+ blk\.8\.ffn_gate_exps\.weight=IQ4_XS
427
+ blk\.8\.ffn_gate_shexp\.weight=Q8_0
428
+ blk\.8\.ffn_up_exps\.weight=IQ4_XS
429
+ blk\.8\.ffn_up_shexp\.weight=Q8_0
430
+ blk\.8\.ssm_alpha\.weight=F32
431
+ blk\.8\.ssm_beta\.weight=F32
432
+ blk\.8\.ssm_out\.weight=Q8_0
433
+ blk\.9\.attn_gate\.weight=Q8_0
434
+ blk\.9\.attn_qkv\.weight=Q8_0
435
+ blk\.9\.ffn_down_exps\.weight=Q4_K
436
+ blk\.9\.ffn_down_shexp\.weight=Q8_0
437
+ blk\.9\.ffn_gate_exps\.weight=Q4_K
438
+ blk\.9\.ffn_gate_shexp\.weight=Q8_0
439
+ blk\.9\.ffn_up_exps\.weight=Q4_K
440
+ blk\.9\.ffn_up_shexp\.weight=Q8_0
441
+ blk\.9\.ssm_alpha\.weight=F32
442
+ blk\.9\.ssm_beta\.weight=F32
443
+ blk\.9\.ssm_out\.weight=Q8_0