diff --git a/README.md b/README.md new file mode 100644 index 0000000000000000000000000000000000000000..cf4a574a5ffe4f50ee4923373f1d51e1e27508e7 --- /dev/null +++ b/README.md @@ -0,0 +1,138 @@ +--- +license: apache-2.0 +tags: + - quantization + - gguf + - mixture-of-experts + - llama.cpp + - technical-report +--- + +# ICE: Quantization by Error-Propagation Class in Sparse MoE Models + +Technical report and full evidence for **ICE** (*Isolation of Compounding +Error*), a quantization bit-allocation method for sparse Mixture-of-Experts +checkpoints. + +**Author:** Gökhan Buz ([`gbuzhf`](https://huggingface.co/gbuzhf)) + +## Read the paper + +**[`paper/ICE_Technical_Report.md`](paper/ICE_Technical_Report.md)** + +## What ICE is + +Every quantizer in the GGUF ecosystem minimizes the same objective for every +tensor: the importance-matrix-weighted error of that tensor's output, **for the +current token**. That is the right objective for a tensor whose error dies with +the token. It is the wrong objective for three other kinds, and no published +method separates them. + +ICE classifies every tensor by **how far its error travels**: + +| class | mechanism | treatment | +|---|---|---| +| **DISCRETE** | error flips an argmax, so a different computation runs | exact (F32) | +| **RECURRENT** | error enters a state decay and compounds along the sequence | exact (F32) | +| **CACHED** | error is written to the KV cache once and re-read by every later token | near-exact (F16) | +| **INSTANT** | error affects this token only | this is where the budget lives | + +The first three are 0.14% of the model, so freezing them is a line item rather +than a trade-off. In one line: **freeze what propagates, spend everything else on +the library.** + +## Headline results + +Mean KL divergence against the bf16 checkpoint, WikiText-2, one harness for all +files. + +| tier | size | mean KLD | nearest published tier | outcome | +|---|---:|---:|---|---| +| `23G-ICE` | 22.83 GB | 0.0361 | `UD-Q4_K_XL` 23.21 GB / 0.0380 | 0.38 GB smaller, 5.0% better | +| `23G-ICE` | 22.83 GB | 0.0361 | `APEX-I-Quality` 23.84 GB / 0.0415 | 1.01 GB smaller, 13.0% better | +| `25G-ICE` | 24.84 GB | 0.0303 | `APEX-I-Balanced` 26.28 GB / 0.0345 | 1.44 GB smaller, 12.2% better | +| `19G-ICE` | 18.82 GB | 0.0608 | `UD-IQ4_XS` 18.68 GB / 0.0723 | 15.8% better at +0.14 GB | + +Across the twelve-tier comparison, nine tiers are Pareto-optimal and three are +strictly dominated. ICE does **not** win at the top of the ladder: `UD-Q5_K_S` +and `UD-Q6_K` are the two best files measured, which the paper explains rather +than omits (Law 4). + +## The four laws + +1. **Sparsity.** A bit on the always-on core is worth `E/k` bits on the expert + bank. Measured 31.8 against a predicted 32. +2. **Convexity.** Error falls as `4^-b`, so allocation cleverness is capped at + **+0.139 bpw**. Two consequences: do no depth grading, and at a fixed average + always pick the **narrower type bracket** (measured +16.6% and +7.0% penalty + for widening). +3. **Placement.** Inside a fixed bracket, shallow-first is worth about **−8.5% + KLD per bpw of gap** and **reverses below 0.47 bpw**. Depth gain measured + directly as `g(t) ≈ exp(−t/9.95)`. +4. **The floor is epistemic.** Fitted on two independent harnesses at `k₀` = + 0.0205 and 0.0219, and confirmed by a direct probe at 0.018297. The best file + measured is 8% above it. **You cannot out-bit a wrong prior.** + +## What is in this repository + +``` +paper/ICE_Technical_Report.md the report +appendix/original-ICE/ PRINCIPLE.md and RECIPE.md, the method as first written +appendix/recipes/ every tensor-type-file cited in the paper +appendix/measurements/ raw llama-perplexity output for every KLD quoted +appendix/SHA256SUMS.txt checksum for every file above +``` + +### `appendix/recipes/` + +| suffix | meaning | +|---|---| +| `_ICEbase` | the current recipe, after the Law 3 revision | +| `_previous` | the recipe it replaced | +| `_comparison` | a published UD or APEX tier, as measured | + +`cfg_21G-ICE_ICEbase.txt` and `cfg_21G-ICE_previous.txt` have the **same +SHA-256** (`93456d45b1fc0dab...`). That is not an oversight. 21G is the tier where +Law 3's condition does not hold, so the method's output is to change nothing, and +the identical checksum is the proof that nothing was changed. + +Recipes for the three dominated tiers are not included: they were dropped after +the Pareto analysis and none was retained. Their measured numbers are in the +paper. + +## Reproducing a number + +```bash +llama-perplexity -m .gguf -f wiki.test.raw \ + -c 2048 --chunks 64 --kl-divergence-base base.kld --kl-divergence +``` + +against a `base.kld` produced once from the bf16 checkpoint with the same corpus +and chunk count. Protocol details, including how the byte-identical control +variants are constructed, are in Appendix B. + +The same recipe measured through three different paths gave 0.0345, 0.034162 and +0.034290, a spread under 1%. + +## Negative results + +Seven are documented at the same weight as the positive ones, including the +**retraction of a rule this work itself derived and shipped**: bumping `ffn_down` +above its sibling projections, which is standard practice, measured 11.3% worse +than uniform at identical size. One of three registered predictions also failed, +and it is scored as such. + +## On the comparison ladders + +Unsloth Dynamic 2.0 and LocalAI APEX are the work of their respective authors. +They are measured here, not reproduced or modified. The comparison exists because +no individual publication can provide it: each ladder is published with its own +harness and its own reference, so the tiers are not comparable until someone puts +them on one. The same analysis that finds ICE tiers dominating three others also +finds two UD tiers to be the best files on the board and two APEX tiers to be the +only options below 18.5 GB. + +## Citing + +Cite this repository. Model cards for ICE-quantized GGUFs link here for the +method description. diff --git a/appendix/SHA256SUMS.txt b/appendix/SHA256SUMS.txt new file mode 100644 index 0000000000000000000000000000000000000000..63c9092f84d8a0f2da1dec0f8bad49b2f41b56b4 --- /dev/null +++ b/appendix/SHA256SUMS.txt @@ -0,0 +1,58 @@ +197b3ba93582e1cbdfa85777f95793f31b198ccd7a2c0037131e38de18843f61 *./measurements/ALLOCATION.txt +460c97202411730e448b5a3d7047f556cc2c9a86f1415f78d25128a3d35e1443 *./measurements/BAND_RESULT.txt +09ab6c5f7f5c648ccccc35c663c46a85c482bf1d9c13c9c4b0f37be8a57fb56d *./measurements/DENSE_RESULT.txt +901f63d923864ebe24a53b8ef99502f29e8e3f1a778615452541fb7c3d244b82 *./measurements/FAM_RESULT.txt +d5db1b927e262c03e5c1ccd82db0ad813f4bcb4909b1d5eb7e04383b9a920220 *./measurements/FLOOR_RESULT.txt +8c5fba182e91970383b2a90ec758946f28d6fbb732163afa40bac32106290457 *./measurements/ICE_RESULT.txt +c17a298c1ee4d2d808aa760cbd10c7b86dc5e5597c14692b49cbd206cf36a250 *./measurements/PROBE_H_RESULT.txt +7461ec5e313b00a977c0e446f792c809969f1f1148c7d1571a3140c7a4baf29a *./measurements/RESULT.txt +e1d36c3b4bb25134691d4a596e5f98fb594d2ce3bc0f7997f82e3fafca6521d0 *./measurements/SENSITIVITY_SUMMARY.txt +c623c3a71eb6568fcbbe432abc707a7adf54c04a1f39df97ba912291cabda623 *./measurements/V2_RESULT.txt +1a4a7fd427ea5572ec056535099c5d9efd578045ba505347a05c3e6d2576eeeb *./measurements/ice_final_ledger.txt +1c027d2c9497ce413d37ce260b60a5df39a21a1cf44650c21fa68e4c4362b40b *./measurements/kld_A.txt +b50ae7929f494a32e6f97220c5084a83dbc772408bbc841366688eb07557b58c *./measurements/kld_B.txt +a94d51164840da70fde1be0a75b85d02387a90e238474578121ccad9df6fc4fa *./measurements/kld_BAND06.txt +6acbf8a0d7537c74e89dcdf54457da9b8eb5471c1b8f6ceed217eb51f701ab44 *./measurements/kld_BAND12.txt +804f3fd733597dc0c2befb2b3d3bb35ca01289e7343dcdaee7387c49415812bb *./measurements/kld_BAND18.txt +0c71461bc53888542baf39fe9b7b455809e6cfe8f9ea8bcf4409e68d107941d7 *./measurements/kld_BAND25.txt +1b2964ce70f4eb4967d7a5808882727cc603b01bd70c192d2659836756c88634 *./measurements/kld_C.txt +8e0350a9a2701dbf79765d2dc80761ec31b60eade245c8ea1bc4b37527b3151f *./measurements/kld_CTRL-19G-ICE.txt +24b2cd42e65233b49d00befd224aea275d37310a0527a29d390f8a836594ad08 *./measurements/kld_CTRL-21G-ICE.txt +c2d6c784625f82a8bcace8669b7a5972d5cbbd7f073846e17064ef9202b86cb7 *./measurements/kld_CTRL-23G-ICE.txt +8e40a4f1bd4a1a3723fbcf8bd7911993a3d1b0e2b123bc13d7e4d8da28b0f96f *./measurements/kld_CTRL-25G-ICE.txt +03926050213470e7299d468e0405f4223c9ba4feb5b30f61ce4ceba18c8d87d7 *./measurements/kld_D.txt +0ddf6ace773708d984fb71a84bc5ea158702f23b74eac0c56359724b6174bc0a *./measurements/kld_DENSE_attention.txt +31416b7bd2b70efd0110719b44cae4c0b916d852f45f745238440f7684bafdd6 *./measurements/kld_DENSE_output.txt +41a4310ff9bd45a9d42290a1d7e43c16bd9482af12bac675e8fa1c7d572d2b59 *./measurements/kld_DENSE_token_embd.txt +c08f34da0fddcab51dbf293a05422c8c1f4033a082d395d4b85a0d568d05fab1 *./measurements/kld_E.txt +5832beaaac14edaf9f8d3c3d219b779312294d4a13e8507a346e329b0dfe3675 *./measurements/kld_FAMEVEN-19G-ICE.txt +93ee48836531052f0defaa8e69e250886861ca7ffe4af809bc2914b478ab60b4 *./measurements/kld_FAMEVEN-21G-ICE.txt +e4ce62ac6d90bfeced9e44c76bfd2ac5d82be1509f1875cc88c00acb466aecee *./measurements/kld_FAMSHALLOW-19G-ICE.txt +5331e3f0f6347d3d742992d14f159f6a09e45b611559a9d478c620f1a8486a45 *./measurements/kld_FAMSHALLOW-21G-ICE.txt +6f309879c98e8837d20fab8c9bc27f102d95c0f3db9acc34bcbe46be6cc8f0a1 *./measurements/kld_FLOORBF16.txt +1a4643f4aa00527025c291f7f8e07bea7bae141bf7838282c4111606e3f70e97 *./measurements/kld_H.txt +0d555044e7beea614bba8d558d2aa2fc8fc61b720615c21be8cab3655bbb5e64 *./measurements/kld_SHALLOW-19G-ICE.txt +136cc786cb10b2334cc38976052bbe9a87fbdb62399db34a3eda8578739a1dc5 *./measurements/kld_SHALLOW-21G-ICE.txt +ede955af97f40cf9d6d7c9a7c07ea0c8dbb156ac54ffa9648ffae12032cdc8f6 *./measurements/kld_SHALLOW-23G-ICE.txt +0bde773519317a049e5ff43bfeb3b3d8d65cf786e0e95ed35381c469d46c27c2 *./measurements/kld_SHALLOW-25G-ICE.txt +d5c2ade2ff18b37bef4760d18bff321214ed700c8c046258855ff7903eb6d981 *./measurements/kld_V2.txt +ba343c8e15674ae88ceb88e89f3a2c5a59d1fef3cf3dc79aea920107a6720027 *./measurements/kldf_19G-ICE.txt +f3d5aaf1a5f2863a31992443b40d2d9d718b75c9457f71856266daf808734afc *./measurements/kldf_21G-ICE.txt +3b42f2045f78a390b0d57c50c218022e627a31e212e176aa6941c132fc848e07 *./measurements/kldf_23G-ICE.txt +b0f4e20583b69ba163d778a8d7e79c755c715a49a5ddec459c874395c487e5d9 *./measurements/kldf_25G-ICE.txt +08d786bffc08ee3de0a0673f2a96ac321ef29bf71431cd4c643fc6cf335282e6 *./measurements/sensitivity.jsonl +b0cf96c35815e8dcb1edd8e8c7e03a126d5114309ed4e5e11dee77963f7177f7 *./original-ICE/PRINCIPLE.md +2480e950488e268b90e10ee4c5f3556c456660af5361595bfc417fc5827c1ac8 *./original-ICE/RECIPE.md +efc76732cf05eb6fdf750336a0a9ac574c495b87b4bddc0bdb5c321a3b5295c1 *./recipes/cfg_19G-ICE_ICEbase.txt +6a9d088d39692d2823daabc9467e011f447b91a481e39c38fa4b24fe18554f77 *./recipes/cfg_19G-ICE_previous.txt +93456d45b1fc0dab59ac03c1a2773c5e74049b2d264241aa07fb6b67d27cbd79 *./recipes/cfg_21G-ICE_ICEbase.txt +93456d45b1fc0dab59ac03c1a2773c5e74049b2d264241aa07fb6b67d27cbd79 *./recipes/cfg_21G-ICE_previous.txt +2e3cbb15d39ffcf569e610efd8c94eb6988597c814f563609984809f9aac00b8 *./recipes/cfg_23G-ICE_ICEbase.txt +1d461db13b2900514641f7e32c40ae7af257daf660ca184e1d2a56ff39eafa20 *./recipes/cfg_23G-ICE_previous.txt +515f283171280f14f3913ed5caa7913cf42f73a3a6f5c942ef9530a294282558 *./recipes/cfg_25G-ICE_ICEbase.txt +4fb6900bff49459a7948f0518d4008b7b13a1fc73b6c6f9cf343f0b5eca56024 *./recipes/cfg_25G-ICE_previous.txt +0251845d1a524151c890ea491a7e1adc62ec3da67a888ea4cb401072e03ad9de *./recipes/cfg_APEX-I-Compact_comparison.txt +0d3adc95d1848cb91231120af4d4e1c7e9a9d27e146c50124869ff2a993b57ab *./recipes/cfg_APEX-I-Mini_comparison.txt +e88b620a8c80cab33661be765ed9f01ec9a56d1ee9ae239d429a1f19e0aa5f2d *./recipes/cfg_UD-IQ4_XS_comparison.txt +d4f78a46b67f7105edf66f8c6a0a8a99dafa4337b6dbe707128908d837618cb5 *./recipes/cfg_UD-Q5_K_S_comparison.txt +cf561e0bbb74d0fdf3ea49cb85006c0221a180cfbeb180c91fb3ca160b7dc7d1 *./recipes/cfg_UD-Q6_K_comparison.txt diff --git a/appendix/measurements/ALLOCATION.txt b/appendix/measurements/ALLOCATION.txt new file mode 100644 index 0000000000000000000000000000000000000000..ebec7bd7f68d32d08450eedefd1ca81520722ee7 --- /dev/null +++ b/appendix/measurements/ALLOCATION.txt @@ -0,0 +1,7 @@ +control expert budget: 19.629 GB over 120 tensors (4.8750 bpw) +control predicted expert error S = 8.20534e+07 + +C per-tensor : 19.622 GB (-0.0380% vs control), S = 2.79971e+07 (-65.9%) +B block-level : 19.592 GB (-0.1897% vs control), S = 5.66933e+07 (-30.9%) + B -> /workspace/dosa/cfg_B.txt: 96 lines differ from control, all expert. mix {'Q5_K': 63, 'IQ4_XS': 51, 'IQ3_S': 6} + C -> /workspace/dosa/cfg_C.txt: 100 lines differ from control, all expert. mix {'Q6_K': 15, 'Q5_K': 55, 'Q4_K': 2, 'IQ4_XS': 12, 'IQ3_S': 36} diff --git a/appendix/measurements/BAND_RESULT.txt b/appendix/measurements/BAND_RESULT.txt new file mode 100644 index 0000000000000000000000000000000000000000..ad541e697c89aebb8070a9ac6ecca47676dc5330 --- /dev/null +++ b/appendix/measurements/BAND_RESULT.txt @@ -0,0 +1,10 @@ +g(depth) band sweep — control's own Q5_K/Q4_K mix, 15-block band slid across depth +every band byte-identical to the control; depth is the only variable. 64 chunks. + + band [ 0..14] centre 7.0 0.032407 +- 0.000458 (probe E) + band [ 6..20] centre 0.0 0.033014 ± 0.000492 + band [12..26] centre 0.0 0.036760 ± 0.000572 + band [18..32] centre 0.0 0.036107 ± 0.000517 + band [25..39] centre 0.0 0.036136 ± 0.000542 + + reference: A even dither 0.034162 | D experts Q8_0 0.019018 (floor) diff --git a/appendix/measurements/DENSE_RESULT.txt b/appendix/measurements/DENSE_RESULT.txt new file mode 100644 index 0000000000000000000000000000000000000000..8e3a2063ee2528105574460ceea344560e992156 --- /dev/null +++ b/appendix/measurements/DENSE_RESULT.txt @@ -0,0 +1,8 @@ +Dense-floor decomposition, complete +reference D (experts Q8_0, nothing lifted) = 0.019018 +true floor H (experts BF16) = 0.018297 + + role KLD with it lifted its share of the floor + output 0.018941 0.000077 + token_embd 0.019217 -0.000199 + attention 0.017510 0.001508 diff --git a/appendix/measurements/FAM_RESULT.txt b/appendix/measurements/FAM_RESULT.txt new file mode 100644 index 0000000000000000000000000000000000000000..49baafb5eca9c8d8e26b89c7028525a65191e954 --- /dev/null +++ b/appendix/measurements/FAM_RESULT.txt @@ -0,0 +1,8 @@ +Same-family (Q4_K/Q3_K) vs the shipped mixed pair, byte-matched + 21G control (Q4_K/IQ4_XS) 0.039182 | shallow on that pair 0.039814 (+1.6%) + 19G control (IQ4_XS/IQ3_S) 0.060015 | shallow on that pair 0.058676 (-2.2%) + + FAMEVEN-21G-ICE Mean KLD: 0.045673 ± 0.000564 Same top p: 91.346 ± 0.110 % + FAMSHALLOW-21G-ICE Mean KLD: 0.043766 ± 0.000525 Same top p: 91.647 ± 0.108 % + FAMEVEN-19G-ICE Mean KLD: 0.064213 ± 0.000727 Same top p: 89.953 ± 0.117 % + FAMSHALLOW-19G-ICE Mean KLD: 0.059623 ± 0.000672 Same top p: 90.315 ± 0.116 % diff --git a/appendix/measurements/FLOOR_RESULT.txt b/appendix/measurements/FLOOR_RESULT.txt new file mode 100644 index 0000000000000000000000000000000000000000..39f5d6d46a2075813e3a27f50886b0ed2172ffd9 --- /dev/null +++ b/appendix/measurements/FLOOR_RESULT.txt @@ -0,0 +1 @@ +FLOOR-BF16 size=24589334592 Mean KLD: 0.031543 ± 0.000428 diff --git a/appendix/measurements/ICE_RESULT.txt b/appendix/measurements/ICE_RESULT.txt new file mode 100644 index 0000000000000000000000000000000000000000..d2e07a9028db3fbf5053221e00336b67ebfee4b2 --- /dev/null +++ b/appendix/measurements/ICE_RESULT.txt @@ -0,0 +1,10 @@ +ICE base — shallow-first rule, each tier against its own control +quantiser: pinned d775b896 (CPU) | measurement: prebuilt 3653e6d (CUDA), 64 chunks +every CTRL/SHALLOW pair is byte-identical; absolute values not comparable +to the phase 2-4 CPU table, only within each pair. + +tier control shallow delta +19G-ICE 0.060015 0.058676 -2.2% +21G-ICE 0.039182 0.039814 +1.6% +23G-ICE 0.034290 0.032536 -5.1% +25G-ICE 0.028955 0.028131 -2.8% diff --git a/appendix/measurements/PROBE_H_RESULT.txt b/appendix/measurements/PROBE_H_RESULT.txt new file mode 100644 index 0000000000000000000000000000000000000000..cfa7e290e6d8b3494455a756961450c03084e38c --- /dev/null +++ b/appendix/measurements/PROBE_H_RESULT.txt @@ -0,0 +1,5 @@ +Error-budget decomposition at the 23G-ICE dense configuration, 64 chunks + + A control, experts 4.875 bpw 0.034162 +- 0.000455 92.579% + D experts Q8_0 0.019018 +- 0.000333 94.601% + H experts BF16 (lossless) Mean KLD: 0.018297 ± 0.000349 Same top p: 94.790 ± 0.087 % diff --git a/appendix/measurements/RESULT.txt b/appendix/measurements/RESULT.txt new file mode 100644 index 0000000000000000000000000000000000000000..ff5df20211da941fc89846b5f87a2c8cb836fa4d --- /dev/null +++ b/appendix/measurements/RESULT.txt @@ -0,0 +1,5 @@ +DOSA Phase 2 — one byte budget (23G-ICE), three allocations, 64 chunks + +A Mean KLD: 0.034162 ± 0.000455 Same top p: 92.579 ± 0.102 % size=22338172992 +B Mean KLD: 0.042338 ± 0.000606 Same top p: 91.940 ± 0.106 % size=22300424256 +C Mean KLD: 0.050559 ± 0.000627 Same top p: 91.239 ± 0.110 % size=22327687232 diff --git a/appendix/measurements/SENSITIVITY_SUMMARY.txt b/appendix/measurements/SENSITIVITY_SUMMARY.txt new file mode 100644 index 0000000000000000000000000000000000000000..866c930753d85219f936146a60ca1932dde68e99 --- /dev/null +++ b/appendix/measurements/SENSITIVITY_SUMMARY.txt @@ -0,0 +1,42 @@ +types: ['IQ3_S', 'IQ4_XS', 'Q4_K', 'Q5_K', 'Q6_K'] tensors per type: {'Q6_K': 120, 'Q5_K': 120, 'Q4_K': 120, 'IQ4_XS': 120, 'IQ3_S': 120} + +IQ3_S s: min 3161 max 1.418e+07 ratio 4487.1x CV 1.022 rho(s, imatrix-energy) +0.9955 rho(s, raw err) +0.5514 +IQ4_XS s: min 1260 max 4.511e+06 ratio 3580.4x CV 1.022 rho(s, imatrix-energy) +0.9951 rho(s, raw err) +0.6734 +Q4_K s: min 904.6 max 3.825e+06 ratio 4228.4x CV 1.026 rho(s, imatrix-energy) +0.9952 rho(s, raw err) +0.6796 +Q5_K s: min 241.2 max 9.681e+05 ratio 4013.5x CV 1.024 rho(s, imatrix-energy) +0.9951 rho(s, raw err) +0.6842 +Q6_K s: min 61.66 max 2.394e+05 ratio 3882.3x CV 1.009 rho(s, imatrix-energy) +0.9956 rho(s, raw err) +0.6812 + +top 12 most sensitive tensors at IQ3_S (these must get the bits): + blk 38 gate s=1.418e+07 + blk 37 gate s=1.416e+07 + blk 36 gate s=1.356e+07 + blk 38 up s=1.264e+07 + blk 37 up s=1.255e+07 + blk 39 gate s=1.218e+07 + blk 35 gate s=1.21e+07 + blk 36 up s=1.183e+07 + blk 39 up s=1.039e+07 + blk 35 up s=1.039e+07 + blk 34 gate s=1.006e+07 + blk 33 gate s=9.559e+06 + +bottom 12 (these can be starved): + blk 0 down s=3161 + blk 2 down s=8085 + blk 1 down s=8344 + blk 3 down s=1.148e+04 + blk 4 down s=1.596e+04 + blk 5 down s=2.025e+04 + blk 6 down s=2.267e+04 + blk 7 down s=2.855e+04 + blk 8 down s=3.134e+04 + blk 9 down s=3.5e+04 + blk 10 down s=3.822e+04 + blk 12 down s=4.438e+04 + +role totals (share of total sensitivity): + IQ3_S gate= 51.4% up= 46.3% down= 2.3% + IQ4_XS gate= 51.4% up= 46.2% down= 2.4% + Q4_K gate= 51.3% up= 46.3% down= 2.4% + Q5_K gate= 51.3% up= 46.3% down= 2.4% + Q6_K gate= 51.0% up= 46.6% down= 2.4% diff --git a/appendix/measurements/V2_RESULT.txt b/appendix/measurements/V2_RESULT.txt new file mode 100644 index 0000000000000000000000000000000000000000..c141979377db87d9de7caa87f92fa5303b2ae25d --- /dev/null +++ b/appendix/measurements/V2_RESULT.txt @@ -0,0 +1,8 @@ +Corrected allocation, validated — one byte budget, 64 chunks, same base.kld + + A control (even dither) 0.034162 +- 0.000455 92.579% + E band[0..14] (crude shallow) 0.032407 +- 0.000458 92.824% + V2 corrected g(t)*s_t Mean KLD: 0.033769 ± 0.000483 Same top p: 92.589 ± 0.102 % + + floor (H, experts lossless) 0.018297 + V2 size: 22338172992 bytes diff --git a/appendix/measurements/ice_final_ledger.txt b/appendix/measurements/ice_final_ledger.txt new file mode 100644 index 0000000000000000000000000000000000000000..e528990ffaa18f099b6da8d736c8e3582985b803 --- /dev/null +++ b/appendix/measurements/ice_final_ledger.txt @@ -0,0 +1,8 @@ +04:45:16 KLD 25G-ICE: Mean KLD: 0.030304 ± 0.000779 +04:45:58 25G-ICE uploaded+verified 24849761792 B, freed +04:48:32 KLD 23G-ICE: Mean KLD: 0.036069 ± 0.000868 +04:49:07 23G-ICE uploaded+verified 22836495872 B, freed +04:52:08 KLD 21G-ICE: Mean KLD: 0.041192 ± 0.001140 +04:52:39 21G-ICE uploaded+verified 20848395776 B, freed +04:57:04 KLD 19G-ICE: Mean KLD: 0.060844 ± 0.001185 +04:57:45 19G-ICE uploaded+verified 18822546944 B, freed diff --git a/appendix/measurements/kld_A.txt b/appendix/measurements/kld_A.txt new file mode 100644 index 0000000000000000000000000000000000000000..9fe2575242e0e3a19732fc0a0e10ab355907c0a6 --- /dev/null +++ b/appendix/measurements/kld_A.txt @@ -0,0 +1,112 @@ +0.18.483.679 I cmn init: llama threadpool init, n_threads = 30 +0.18.616.495 I +0.18.616.655 I system_info: n_threads = 30 (n_threads_batch = 30) / 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.18.972.875 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.38.595.898 I kl_divergence: 19.62 seconds per pass - ETA 20.92 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.8539 ± 0.4567 -0.00061 ± 0.01224 0.04542 ± 0.00344 7.033 ± 0.568 % 91.984 ± 0.849 % + 2 7.3443 ± 0.4257 0.00485 ± 0.00822 0.03957 ± 0.00196 5.940 ± 0.365 % 91.544 ± 0.615 % + 3 8.3070 ± 0.4097 0.01310 ± 0.00656 0.03660 ± 0.00154 5.525 ± 0.303 % 91.952 ± 0.491 % + 4 9.3665 ± 0.4262 0.00485 ± 0.00585 0.04045 ± 0.00213 5.620 ± 0.283 % 91.667 ± 0.432 % + 5 8.7243 ± 0.3493 0.00138 ± 0.00521 0.03966 ± 0.00177 5.542 ± 0.242 % 91.613 ± 0.388 % + 6 7.4068 ± 0.2619 0.00029 ± 0.00465 0.03864 ± 0.00159 5.582 ± 0.230 % 91.707 ± 0.352 % + 7 6.7449 ± 0.2167 -0.00776 ± 0.00442 0.03943 ± 0.00144 5.798 ± 0.213 % 91.761 ± 0.325 % + 8 6.5552 ± 0.1943 -0.00678 ± 0.00402 0.03812 ± 0.00128 5.718 ± 0.193 % 91.874 ± 0.302 % + 9 6.9515 ± 0.1960 -0.00642 ± 0.00379 0.03888 ± 0.00118 5.638 ± 0.176 % 91.745 ± 0.287 % + 10 7.0140 ± 0.1878 -0.00659 ± 0.00353 0.03783 ± 0.00109 5.567 ± 0.172 % 91.945 ± 0.269 % + 11 7.0196 ± 0.1780 -0.00632 ± 0.00334 0.03657 ± 0.00101 5.441 ± 0.161 % 92.127 ± 0.254 % + 12 7.2843 ± 0.1781 -0.00556 ± 0.00314 0.03544 ± 0.00093 5.327 ± 0.152 % 92.204 ± 0.242 % + 13 7.3923 ± 0.1743 -0.00611 ± 0.00303 0.03556 ± 0.00092 5.389 ± 0.146 % 92.300 ± 0.231 % + 14 7.4356 ± 0.1686 -0.00662 ± 0.00289 0.03486 ± 0.00086 5.328 ± 0.141 % 92.382 ± 0.222 % + 15 7.4746 ± 0.1636 -0.00919 ± 0.00279 0.03493 ± 0.00083 5.305 ± 0.134 % 92.297 ± 0.215 % + 16 7.6571 ± 0.1624 -0.00981 ± 0.00268 0.03449 ± 0.00078 5.240 ± 0.128 % 92.388 ± 0.207 % + 17 7.6643 ± 0.1568 -0.00982 ± 0.00256 0.03378 ± 0.00074 5.151 ± 0.122 % 92.473 ± 0.200 % + 18 7.7467 ± 0.1539 -0.00936 ± 0.00247 0.03347 ± 0.00071 5.116 ± 0.118 % 92.522 ± 0.194 % + 19 7.6600 ± 0.1484 -0.01121 ± 0.00240 0.03328 ± 0.00068 5.071 ± 0.113 % 92.525 ± 0.189 % + 20 7.4027 ± 0.1392 -0.01298 ± 0.00246 0.03649 ± 0.00088 5.420 ± 0.121 % 92.380 ± 0.185 % + 21 7.4475 ± 0.1366 -0.01238 ± 0.00242 0.03718 ± 0.00088 5.481 ± 0.120 % 92.282 ± 0.182 % + 22 7.5182 ± 0.1353 -0.01297 ± 0.00235 0.03680 ± 0.00084 5.460 ± 0.116 % 92.318 ± 0.178 % + 23 7.5483 ± 0.1326 -0.01247 ± 0.00227 0.03634 ± 0.00081 5.399 ± 0.112 % 92.341 ± 0.173 % + 24 7.6384 ± 0.1317 -0.01419 ± 0.00224 0.03690 ± 0.00082 5.461 ± 0.113 % 92.302 ± 0.170 % + 25 7.7048 ± 0.1305 -0.01397 ± 0.00219 0.03659 ± 0.00079 5.406 ± 0.110 % 92.289 ± 0.167 % + 26 7.6580 ± 0.1269 -0.01342 ± 0.00213 0.03624 ± 0.00076 5.382 ± 0.107 % 92.372 ± 0.163 % + 27 7.8097 ± 0.1274 -0.01350 ± 0.00207 0.03585 ± 0.00073 5.333 ± 0.104 % 92.365 ± 0.160 % + 28 7.8916 ± 0.1266 -0.01281 ± 0.00201 0.03543 ± 0.00071 5.287 ± 0.102 % 92.400 ± 0.157 % + 29 7.8961 ± 0.1247 -0.01180 ± 0.00198 0.03558 ± 0.00069 5.317 ± 0.099 % 92.436 ± 0.154 % + 30 7.9215 ± 0.1234 -0.01463 ± 0.00198 0.03588 ± 0.00068 5.321 ± 0.096 % 92.405 ± 0.151 % + 31 7.8122 ± 0.1193 -0.01463 ± 0.00194 0.03566 ± 0.00066 5.291 ± 0.094 % 92.461 ± 0.148 % + 32 7.6843 ± 0.1154 -0.01350 ± 0.00191 0.03561 ± 0.00065 5.335 ± 0.093 % 92.519 ± 0.145 % + 33 7.6706 ± 0.1136 -0.01319 ± 0.00188 0.03570 ± 0.00066 5.332 ± 0.091 % 92.521 ± 0.143 % + 34 7.6455 ± 0.1113 -0.01337 ± 0.00184 0.03539 ± 0.00064 5.313 ± 0.089 % 92.502 ± 0.141 % + 35 7.6425 ± 0.1097 -0.01357 ± 0.00181 0.03520 ± 0.00063 5.288 ± 0.087 % 92.501 ± 0.139 % + 36 7.6463 ± 0.1083 -0.01375 ± 0.00177 0.03497 ± 0.00061 5.275 ± 0.086 % 92.557 ± 0.137 % + 37 7.5036 ± 0.1043 -0.01342 ± 0.00174 0.03453 ± 0.00060 5.244 ± 0.084 % 92.608 ± 0.134 % + 38 7.3942 ± 0.1009 -0.01314 ± 0.00170 0.03422 ± 0.00058 5.220 ± 0.083 % 92.648 ± 0.132 % + 39 7.2821 ± 0.0977 -0.01254 ± 0.00168 0.03397 ± 0.00057 5.209 ± 0.082 % 92.656 ± 0.131 % + 40 7.1566 ± 0.0943 -0.01215 ± 0.00164 0.03359 ± 0.00056 5.195 ± 0.080 % 92.710 ± 0.129 % + 41 7.0789 ± 0.0918 -0.01169 ± 0.00162 0.03326 ± 0.00055 5.170 ± 0.079 % 92.723 ± 0.127 % + 42 7.1098 ± 0.0911 -0.01148 ± 0.00159 0.03316 ± 0.00053 5.157 ± 0.078 % 92.699 ± 0.126 % + 43 7.3081 ± 0.0935 -0.01146 ± 0.00158 0.03334 ± 0.00053 5.168 ± 0.079 % 92.650 ± 0.124 % + 44 7.4477 ± 0.0947 -0.01127 ± 0.00155 0.03326 ± 0.00052 5.135 ± 0.078 % 92.644 ± 0.123 % + 45 7.4301 ± 0.0933 -0.00966 ± 0.00155 0.03443 ± 0.00052 5.241 ± 0.077 % 92.553 ± 0.122 % + 46 7.4358 ± 0.0923 -0.00950 ± 0.00154 0.03468 ± 0.00052 5.239 ± 0.076 % 92.528 ± 0.121 % + 47 7.5004 ± 0.0923 -0.00967 ± 0.00153 0.03481 ± 0.00052 5.238 ± 0.075 % 92.527 ± 0.120 % + 48 7.4980 ± 0.0911 -0.00939 ± 0.00150 0.03464 ± 0.00051 5.223 ± 0.074 % 92.536 ± 0.119 % + 49 7.5493 ± 0.0909 -0.00915 ± 0.00149 0.03462 ± 0.00050 5.222 ± 0.074 % 92.493 ± 0.118 % + 50 7.5033 ± 0.0893 -0.00901 ± 0.00147 0.03437 ± 0.00050 5.203 ± 0.073 % 92.542 ± 0.116 % + 51 7.5390 ± 0.0888 -0.00849 ± 0.00145 0.03402 ± 0.00049 5.168 ± 0.072 % 92.561 ± 0.115 % + 52 7.4734 ± 0.0870 -0.00883 ± 0.00143 0.03388 ± 0.00048 5.161 ± 0.071 % 92.584 ± 0.114 % + 53 7.4441 ± 0.0857 -0.00835 ± 0.00141 0.03370 ± 0.00047 5.149 ± 0.070 % 92.599 ± 0.112 % + 54 7.4011 ± 0.0843 -0.00857 ± 0.00139 0.03348 ± 0.00047 5.139 ± 0.069 % 92.613 ± 0.111 % + 55 7.3598 ± 0.0829 -0.00913 ± 0.00139 0.03347 ± 0.00047 5.148 ± 0.069 % 92.610 ± 0.110 % + 56 7.3527 ± 0.0821 -0.00923 ± 0.00137 0.03343 ± 0.00046 5.146 ± 0.068 % 92.618 ± 0.109 % + 57 7.2747 ± 0.0803 -0.00914 ± 0.00136 0.03321 ± 0.00045 5.125 ± 0.067 % 92.653 ± 0.108 % + 58 7.3124 ± 0.0803 -0.00878 ± 0.00136 0.03371 ± 0.00046 5.165 ± 0.067 % 92.637 ± 0.107 % + 59 7.2845 ± 0.0792 -0.00861 ± 0.00135 0.03409 ± 0.00046 5.200 ± 0.067 % 92.631 ± 0.106 % + 60 7.2650 ± 0.0783 -0.00893 ± 0.00134 0.03392 ± 0.00045 5.185 ± 0.066 % 92.639 ± 0.105 % + 61 7.3309 ± 0.0787 -0.01074 ± 0.00135 0.03461 ± 0.00047 5.244 ± 0.067 % 92.580 ± 0.105 % + 62 7.3542 ± 0.0783 -0.01059 ± 0.00133 0.03446 ± 0.00047 5.224 ± 0.066 % 92.579 ± 0.104 % + 63 7.4011 ± 0.0783 -0.01029 ± 0.00132 0.03428 ± 0.00046 5.203 ± 0.066 % 92.574 ± 0.103 % + 64 7.4319 ± 0.0780 -0.01032 ± 0.00130 0.03416 ± 0.00046 5.189 ± 0.065 % 92.579 ± 0.102 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.431939 ± 0.078018 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 99.23% +Mean ln(PPL(Q)/PPL(base)) : -0.010321 ± 0.001305 +Mean PPL(Q)/PPL(base) : 0.989732 ± 0.001291 +Mean PPL(Q)-PPL(base) : -0.077104 ± 0.009826 + +====== KL divergence statistics ====== +Mean KLD: 0.034162 ± 0.000455 +Maximum KLD: 7.973917 +99.9% KLD: 1.355159 +99.0% KLD: 0.361620 +95.0% KLD: 0.122376 +90.0% KLD: 0.071926 +Median KLD: 0.011761 +10.0% KLD: 0.000184 + 5.0% KLD: 0.000052 + 1.0% KLD: 0.000000 + 0.1% KLD: -0.000147 +Minimum KLD: -0.001017 + +====== Token probability statistics ====== +Mean Δp: -0.126 ± 0.020 % +Maximum Δp: 88.179% +99.9% Δp: 39.137% +99.0% Δp: 15.728% +95.0% Δp: 6.152% +90.0% Δp: 3.233% +75.0% Δp: 0.522% +Median Δp: -0.002% +25.0% Δp: -0.758% +10.0% Δp: -3.730% + 5.0% Δp: -6.619% + 1.0% Δp: -15.617% + 0.1% Δp: -41.049% +Minimum Δp: -93.760% +RMS Δp : 5.189 ± 0.065 % +Same top p: 92.579 ± 0.102 % + diff --git a/appendix/measurements/kld_B.txt b/appendix/measurements/kld_B.txt new file mode 100644 index 0000000000000000000000000000000000000000..f1210fb027c304ad27b691c13e87a49cc02ee0f2 --- /dev/null +++ b/appendix/measurements/kld_B.txt @@ -0,0 +1,112 @@ +0.02.564.663 I cmn init: llama threadpool init, n_threads = 30 +0.02.666.488 I +0.02.666.677 I system_info: n_threads = 30 (n_threads_batch = 30) / 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.03.039.270 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.20.055.609 I kl_divergence: 17.02 seconds per pass - ETA 18.15 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.7012 ± 0.4441 -0.02705 ± 0.01401 0.06733 ± 0.00483 8.250 ± 0.604 % 90.127 ± 0.933 % + 2 7.1282 ± 0.4100 -0.02501 ± 0.00929 0.05204 ± 0.00269 6.884 ± 0.400 % 90.616 ± 0.645 % + 3 8.0373 ± 0.3930 -0.01992 ± 0.00753 0.05041 ± 0.00320 6.614 ± 0.378 % 91.398 ± 0.506 % + 4 8.8810 ± 0.3972 -0.04837 ± 0.00730 0.06139 ± 0.00481 7.055 ± 0.370 % 91.251 ± 0.442 % + 5 8.3798 ± 0.3314 -0.03891 ± 0.00626 0.05611 ± 0.00388 6.615 ± 0.318 % 91.496 ± 0.390 % + 6 7.1462 ± 0.2492 -0.03553 ± 0.00556 0.05375 ± 0.00329 6.621 ± 0.277 % 91.544 ± 0.355 % + 7 6.5783 ± 0.2091 -0.03278 ± 0.00508 0.05161 ± 0.00285 6.541 ± 0.246 % 91.831 ± 0.324 % + 8 6.4147 ± 0.1882 -0.02846 ± 0.00460 0.04911 ± 0.00252 6.305 ± 0.224 % 91.997 ± 0.300 % + 9 6.7792 ± 0.1891 -0.03151 ± 0.00432 0.04906 ± 0.00226 6.283 ± 0.206 % 91.669 ± 0.288 % + 10 6.8538 ± 0.1816 -0.02970 ± 0.00401 0.04710 ± 0.00204 6.143 ± 0.195 % 91.623 ± 0.274 % + 11 6.8704 ± 0.1725 -0.02781 ± 0.00381 0.04566 ± 0.00187 6.055 ± 0.185 % 91.771 ± 0.259 % + 12 7.1196 ± 0.1722 -0.02844 ± 0.00357 0.04407 ± 0.00172 5.900 ± 0.175 % 91.805 ± 0.248 % + 13 7.2499 ± 0.1695 -0.02556 ± 0.00341 0.04322 ± 0.00159 5.848 ± 0.164 % 91.804 ± 0.238 % + 14 7.2938 ± 0.1640 -0.02588 ± 0.00325 0.04266 ± 0.00150 5.818 ± 0.161 % 91.824 ± 0.229 % + 15 7.3555 ± 0.1599 -0.02524 ± 0.00313 0.04204 ± 0.00141 5.792 ± 0.155 % 91.854 ± 0.221 % + 16 7.5456 ± 0.1590 -0.02447 ± 0.00301 0.04153 ± 0.00133 5.741 ± 0.149 % 91.874 ± 0.214 % + 17 7.5630 ± 0.1539 -0.02312 ± 0.00287 0.04045 ± 0.00126 5.648 ± 0.143 % 91.956 ± 0.206 % + 18 7.6457 ± 0.1511 -0.02248 ± 0.00276 0.03985 ± 0.00119 5.590 ± 0.137 % 92.044 ± 0.199 % + 19 7.5540 ± 0.1456 -0.02513 ± 0.00273 0.04080 ± 0.00116 5.648 ± 0.132 % 91.943 ± 0.195 % + 20 7.3209 ± 0.1369 -0.02410 ± 0.00276 0.04431 ± 0.00125 6.029 ± 0.138 % 91.799 ± 0.192 % + 21 7.3669 ± 0.1345 -0.02327 ± 0.00268 0.04427 ± 0.00120 6.030 ± 0.135 % 91.789 ± 0.187 % + 22 7.4308 ± 0.1330 -0.02465 ± 0.00261 0.04401 ± 0.00115 5.987 ± 0.130 % 91.789 ± 0.183 % + 23 7.4636 ± 0.1304 -0.02377 ± 0.00253 0.04347 ± 0.00110 5.922 ± 0.126 % 91.848 ± 0.178 % + 24 7.5419 ± 0.1293 -0.02691 ± 0.00250 0.04425 ± 0.00107 5.953 ± 0.122 % 91.768 ± 0.175 % + 25 7.5928 ± 0.1278 -0.02861 ± 0.00245 0.04426 ± 0.00104 5.914 ± 0.118 % 91.738 ± 0.172 % + 26 7.5481 ± 0.1243 -0.02788 ± 0.00239 0.04382 ± 0.00100 5.879 ± 0.115 % 91.804 ± 0.168 % + 27 7.6960 ± 0.1246 -0.02817 ± 0.00234 0.04353 ± 0.00097 5.842 ± 0.112 % 91.695 ± 0.166 % + 28 7.7640 ± 0.1236 -0.02911 ± 0.00229 0.04335 ± 0.00094 5.805 ± 0.110 % 91.670 ± 0.163 % + 29 7.7556 ± 0.1214 -0.02976 ± 0.00226 0.04368 ± 0.00092 5.876 ± 0.108 % 91.630 ± 0.161 % + 30 7.7690 ± 0.1199 -0.03408 ± 0.00226 0.04468 ± 0.00092 5.920 ± 0.107 % 91.561 ± 0.159 % + 31 7.6631 ± 0.1160 -0.03390 ± 0.00222 0.04437 ± 0.00089 5.890 ± 0.104 % 91.600 ± 0.156 % + 32 7.5283 ± 0.1119 -0.03402 ± 0.00218 0.04449 ± 0.00088 5.940 ± 0.103 % 91.664 ± 0.153 % + 33 7.5199 ± 0.1103 -0.03303 ± 0.00215 0.04456 ± 0.00088 5.926 ± 0.101 % 91.659 ± 0.150 % + 34 7.4961 ± 0.1081 -0.03312 ± 0.00210 0.04418 ± 0.00086 5.898 ± 0.099 % 91.680 ± 0.148 % + 35 7.4876 ± 0.1064 -0.03404 ± 0.00207 0.04433 ± 0.00084 5.909 ± 0.098 % 91.686 ± 0.146 % + 36 7.4965 ± 0.1051 -0.03353 ± 0.00202 0.04382 ± 0.00082 5.868 ± 0.096 % 91.759 ± 0.143 % + 37 7.3588 ± 0.1013 -0.03290 ± 0.00198 0.04310 ± 0.00080 5.831 ± 0.094 % 91.831 ± 0.141 % + 38 7.2564 ± 0.0981 -0.03197 ± 0.00194 0.04282 ± 0.00078 5.821 ± 0.093 % 91.881 ± 0.139 % + 39 7.1480 ± 0.0950 -0.03113 ± 0.00191 0.04246 ± 0.00078 5.809 ± 0.092 % 91.914 ± 0.136 % + 40 7.0277 ± 0.0917 -0.03032 ± 0.00187 0.04189 ± 0.00076 5.788 ± 0.090 % 91.982 ± 0.134 % + 41 6.9549 ± 0.0893 -0.02937 ± 0.00183 0.04132 ± 0.00074 5.749 ± 0.089 % 92.025 ± 0.132 % + 42 6.9911 ± 0.0888 -0.02832 ± 0.00181 0.04112 ± 0.00072 5.723 ± 0.087 % 92.029 ± 0.131 % + 43 7.1855 ± 0.0911 -0.02837 ± 0.00178 0.04099 ± 0.00071 5.693 ± 0.086 % 91.993 ± 0.129 % + 44 7.3203 ± 0.0922 -0.02853 ± 0.00176 0.04081 ± 0.00070 5.657 ± 0.085 % 91.958 ± 0.128 % + 45 7.2944 ± 0.0907 -0.02809 ± 0.00175 0.04157 ± 0.00069 5.708 ± 0.083 % 91.926 ± 0.127 % + 46 7.3024 ± 0.0897 -0.02760 ± 0.00173 0.04182 ± 0.00068 5.728 ± 0.082 % 91.940 ± 0.125 % + 47 7.3639 ± 0.0897 -0.02803 ± 0.00171 0.04178 ± 0.00067 5.716 ± 0.081 % 91.934 ± 0.124 % + 48 7.3643 ± 0.0886 -0.02739 ± 0.00168 0.04150 ± 0.00066 5.701 ± 0.081 % 91.948 ± 0.123 % + 49 7.4144 ± 0.0885 -0.02718 ± 0.00166 0.04154 ± 0.00066 5.689 ± 0.080 % 91.927 ± 0.122 % + 50 7.3716 ± 0.0869 -0.02672 ± 0.00164 0.04125 ± 0.00065 5.664 ± 0.079 % 91.957 ± 0.120 % + 51 7.4030 ± 0.0864 -0.02670 ± 0.00162 0.04101 ± 0.00064 5.650 ± 0.078 % 91.967 ± 0.119 % + 52 7.3367 ± 0.0846 -0.02728 ± 0.00160 0.04117 ± 0.00063 5.665 ± 0.077 % 91.949 ± 0.118 % + 53 7.3097 ± 0.0834 -0.02656 ± 0.00158 0.04091 ± 0.00062 5.644 ± 0.076 % 91.970 ± 0.117 % + 54 7.2701 ± 0.0820 -0.02641 ± 0.00156 0.04072 ± 0.00062 5.638 ± 0.076 % 91.972 ± 0.116 % + 55 7.2246 ± 0.0806 -0.02766 ± 0.00156 0.04078 ± 0.00061 5.640 ± 0.075 % 91.959 ± 0.115 % + 56 7.2170 ± 0.0799 -0.02785 ± 0.00155 0.04065 ± 0.00060 5.637 ± 0.074 % 91.960 ± 0.114 % + 57 7.1420 ± 0.0781 -0.02755 ± 0.00153 0.04048 ± 0.00060 5.625 ± 0.074 % 91.996 ± 0.112 % + 58 7.1770 ± 0.0780 -0.02747 ± 0.00153 0.04095 ± 0.00060 5.654 ± 0.073 % 91.974 ± 0.112 % + 59 7.1482 ± 0.0770 -0.02749 ± 0.00153 0.04131 ± 0.00059 5.687 ± 0.073 % 91.973 ± 0.111 % + 60 7.1321 ± 0.0762 -0.02739 ± 0.00151 0.04109 ± 0.00058 5.672 ± 0.072 % 92.007 ± 0.109 % + 61 7.1803 ± 0.0763 -0.03151 ± 0.00155 0.04317 ± 0.00063 5.787 ± 0.073 % 91.888 ± 0.109 % + 62 7.2037 ± 0.0759 -0.03127 ± 0.00153 0.04290 ± 0.00062 5.758 ± 0.073 % 91.909 ± 0.108 % + 63 7.2523 ± 0.0759 -0.03059 ± 0.00152 0.04255 ± 0.00061 5.730 ± 0.072 % 91.921 ± 0.107 % + 64 7.2833 ± 0.0757 -0.03052 ± 0.00150 0.04234 ± 0.00061 5.712 ± 0.071 % 91.940 ± 0.106 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.283348 ± 0.075668 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 98.99% +Mean ln(PPL(Q)/PPL(base)) : -0.030517 ± 0.001497 +Mean PPL(Q)/PPL(base) : 0.969944 ± 0.001452 +Mean PPL(Q)-PPL(base) : -0.225695 ± 0.011594 + +====== KL divergence statistics ====== +Mean KLD: 0.042338 ± 0.000606 +Maximum KLD: 9.478714 +99.9% KLD: 1.668447 +99.0% KLD: 0.440673 +95.0% KLD: 0.152109 +90.0% KLD: 0.088946 +Median KLD: 0.013976 +10.0% KLD: 0.000211 + 5.0% KLD: 0.000057 + 1.0% KLD: 0.000001 + 0.1% KLD: -0.000129 +Minimum KLD: -0.000870 + +====== Token probability statistics ====== +Mean Δp: -0.067 ± 0.022 % +Maximum Δp: 98.793% +99.9% Δp: 48.776% +99.0% Δp: 17.848% +95.0% Δp: 6.781% +90.0% Δp: 3.539% +75.0% Δp: 0.592% +Median Δp: -0.001% +25.0% Δp: -0.839% +10.0% Δp: -4.098% + 5.0% Δp: -7.168% + 1.0% Δp: -16.669% + 0.1% Δp: -41.365% +Minimum Δp: -94.456% +RMS Δp : 5.712 ± 0.071 % +Same top p: 91.940 ± 0.106 % + diff --git a/appendix/measurements/kld_BAND06.txt b/appendix/measurements/kld_BAND06.txt new file mode 100644 index 0000000000000000000000000000000000000000..02865f98f3d29591937b203bd0b96894cd24c00c --- /dev/null +++ b/appendix/measurements/kld_BAND06.txt @@ -0,0 +1,112 @@ +0.17.135.914 I cmn init: llama threadpool init, n_threads = 30 +0.17.215.459 I +0.17.215.634 I system_info: n_threads = 30 (n_threads_batch = 30) / 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.17.538.935 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.34.402.912 I kl_divergence: 16.86 seconds per pass - ETA 17.98 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.7924 ± 0.4502 -0.01117 ± 0.01151 0.04248 ± 0.00330 6.239 ± 0.467 % 92.669 ± 0.815 % + 2 7.2587 ± 0.4205 -0.00687 ± 0.00817 0.03753 ± 0.00195 5.462 ± 0.295 % 92.229 ± 0.592 % + 3 8.1638 ± 0.4002 -0.00430 ± 0.00652 0.03487 ± 0.00158 5.105 ± 0.228 % 92.571 ± 0.473 % + 4 9.2047 ± 0.4160 -0.01257 ± 0.00602 0.04209 ± 0.00298 5.893 ± 0.316 % 92.107 ± 0.422 % + 5 8.6271 ± 0.3438 -0.00983 ± 0.00525 0.03977 ± 0.00241 5.649 ± 0.271 % 92.180 ± 0.375 % + 6 7.3396 ± 0.2584 -0.00882 ± 0.00466 0.03871 ± 0.00208 5.767 ± 0.247 % 92.147 ± 0.343 % + 7 6.7258 ± 0.2156 -0.01061 ± 0.00440 0.03873 ± 0.00186 5.919 ± 0.229 % 92.347 ± 0.314 % + 8 6.5318 ± 0.1932 -0.01036 ± 0.00403 0.03707 ± 0.00163 5.752 ± 0.209 % 92.424 ± 0.293 % + 9 6.9364 ± 0.1953 -0.00859 ± 0.00379 0.03770 ± 0.00149 5.702 ± 0.192 % 92.169 ± 0.280 % + 10 7.0078 ± 0.1874 -0.00749 ± 0.00352 0.03640 ± 0.00136 5.619 ± 0.184 % 92.336 ± 0.263 % + 11 7.0186 ± 0.1779 -0.00647 ± 0.00332 0.03518 ± 0.00125 5.481 ± 0.173 % 92.526 ± 0.248 % + 12 7.2650 ± 0.1772 -0.00821 ± 0.00314 0.03449 ± 0.00116 5.370 ± 0.163 % 92.587 ± 0.236 % + 13 7.3505 ± 0.1727 -0.01178 ± 0.00303 0.03440 ± 0.00109 5.363 ± 0.153 % 92.646 ± 0.226 % + 14 7.3981 ± 0.1672 -0.01168 ± 0.00288 0.03346 ± 0.00102 5.280 ± 0.148 % 92.738 ± 0.217 % + 15 7.4466 ± 0.1626 -0.01293 ± 0.00275 0.03318 ± 0.00097 5.248 ± 0.141 % 92.786 ± 0.209 % + 16 7.6356 ± 0.1617 -0.01261 ± 0.00264 0.03287 ± 0.00092 5.212 ± 0.137 % 92.827 ± 0.202 % + 17 7.6484 ± 0.1563 -0.01190 ± 0.00253 0.03238 ± 0.00088 5.122 ± 0.132 % 92.858 ± 0.195 % + 18 7.7318 ± 0.1534 -0.01128 ± 0.00244 0.03199 ± 0.00083 5.095 ± 0.127 % 92.875 ± 0.190 % + 19 7.6471 ± 0.1480 -0.01289 ± 0.00236 0.03168 ± 0.00079 5.045 ± 0.122 % 92.874 ± 0.185 % + 20 7.3906 ± 0.1389 -0.01462 ± 0.00241 0.03536 ± 0.00100 5.482 ± 0.132 % 92.659 ± 0.182 % + 21 7.4304 ± 0.1362 -0.01469 ± 0.00235 0.03585 ± 0.00098 5.514 ± 0.129 % 92.589 ± 0.179 % + 22 7.5080 ± 0.1350 -0.01432 ± 0.00228 0.03560 ± 0.00095 5.484 ± 0.126 % 92.673 ± 0.174 % + 23 7.5408 ± 0.1324 -0.01347 ± 0.00221 0.03524 ± 0.00091 5.419 ± 0.122 % 92.677 ± 0.170 % + 24 7.6315 ± 0.1315 -0.01510 ± 0.00217 0.03545 ± 0.00088 5.441 ± 0.119 % 92.669 ± 0.166 % + 25 7.6931 ± 0.1302 -0.01549 ± 0.00212 0.03519 ± 0.00085 5.389 ± 0.116 % 92.641 ± 0.163 % + 26 7.6440 ± 0.1266 -0.01526 ± 0.00206 0.03494 ± 0.00082 5.348 ± 0.112 % 92.691 ± 0.160 % + 27 7.7946 ± 0.1270 -0.01543 ± 0.00202 0.03470 ± 0.00079 5.306 ± 0.109 % 92.607 ± 0.157 % + 28 7.8638 ± 0.1259 -0.01634 ± 0.00198 0.03465 ± 0.00077 5.283 ± 0.107 % 92.602 ± 0.155 % + 29 7.8642 ± 0.1239 -0.01585 ± 0.00195 0.03455 ± 0.00075 5.294 ± 0.103 % 92.591 ± 0.152 % + 30 7.8943 ± 0.1228 -0.01807 ± 0.00194 0.03475 ± 0.00073 5.296 ± 0.100 % 92.590 ± 0.150 % + 31 7.7911 ± 0.1189 -0.01734 ± 0.00191 0.03451 ± 0.00071 5.280 ± 0.098 % 92.590 ± 0.147 % + 32 7.6621 ± 0.1149 -0.01640 ± 0.00187 0.03444 ± 0.00069 5.330 ± 0.098 % 92.647 ± 0.144 % + 33 7.6563 ± 0.1133 -0.01505 ± 0.00185 0.03471 ± 0.00072 5.331 ± 0.095 % 92.615 ± 0.142 % + 34 7.6386 ± 0.1112 -0.01429 ± 0.00181 0.03431 ± 0.00070 5.295 ± 0.093 % 92.625 ± 0.140 % + 35 7.6362 ± 0.1095 -0.01439 ± 0.00178 0.03420 ± 0.00069 5.279 ± 0.092 % 92.638 ± 0.138 % + 36 7.6413 ± 0.1081 -0.01440 ± 0.00174 0.03394 ± 0.00067 5.255 ± 0.090 % 92.696 ± 0.136 % + 37 7.4993 ± 0.1042 -0.01399 ± 0.00171 0.03352 ± 0.00066 5.227 ± 0.089 % 92.774 ± 0.133 % + 38 7.3882 ± 0.1008 -0.01397 ± 0.00168 0.03326 ± 0.00064 5.209 ± 0.087 % 92.815 ± 0.131 % + 39 7.2746 ± 0.0976 -0.01357 ± 0.00165 0.03292 ± 0.00062 5.185 ± 0.086 % 92.827 ± 0.129 % + 40 7.1513 ± 0.0942 -0.01290 ± 0.00161 0.03250 ± 0.00061 5.167 ± 0.084 % 92.864 ± 0.127 % + 41 7.0728 ± 0.0916 -0.01255 ± 0.00160 0.03225 ± 0.00060 5.159 ± 0.084 % 92.876 ± 0.126 % + 42 7.1047 ± 0.0910 -0.01221 ± 0.00157 0.03217 ± 0.00059 5.148 ± 0.083 % 92.880 ± 0.124 % + 43 7.3023 ± 0.0934 -0.01226 ± 0.00156 0.03220 ± 0.00058 5.134 ± 0.081 % 92.816 ± 0.123 % + 44 7.4446 ± 0.0946 -0.01169 ± 0.00154 0.03217 ± 0.00057 5.117 ± 0.080 % 92.802 ± 0.122 % + 45 7.4218 ± 0.0931 -0.01078 ± 0.00153 0.03307 ± 0.00057 5.189 ± 0.079 % 92.723 ± 0.121 % + 46 7.4257 ± 0.0921 -0.01086 ± 0.00152 0.03334 ± 0.00056 5.204 ± 0.078 % 92.703 ± 0.120 % + 47 7.4830 ± 0.0919 -0.01199 ± 0.00151 0.03359 ± 0.00056 5.208 ± 0.077 % 92.685 ± 0.119 % + 48 7.4818 ± 0.0908 -0.01155 ± 0.00149 0.03339 ± 0.00055 5.186 ± 0.076 % 92.691 ± 0.117 % + 49 7.5307 ± 0.0906 -0.01162 ± 0.00148 0.03357 ± 0.00056 5.185 ± 0.075 % 92.639 ± 0.117 % + 50 7.4880 ± 0.0890 -0.01104 ± 0.00146 0.03336 ± 0.00055 5.163 ± 0.074 % 92.661 ± 0.115 % + 51 7.5212 ± 0.0885 -0.01086 ± 0.00144 0.03314 ± 0.00054 5.137 ± 0.073 % 92.684 ± 0.114 % + 52 7.4573 ± 0.0868 -0.01098 ± 0.00142 0.03314 ± 0.00054 5.135 ± 0.072 % 92.682 ± 0.113 % + 53 7.4267 ± 0.0854 -0.01069 ± 0.00140 0.03293 ± 0.00053 5.116 ± 0.071 % 92.674 ± 0.112 % + 54 7.3858 ± 0.0840 -0.01063 ± 0.00138 0.03266 ± 0.00052 5.093 ± 0.070 % 92.707 ± 0.111 % + 55 7.3460 ± 0.0827 -0.01100 ± 0.00137 0.03259 ± 0.00051 5.100 ± 0.070 % 92.701 ± 0.110 % + 56 7.3408 ± 0.0819 -0.01085 ± 0.00136 0.03254 ± 0.00051 5.093 ± 0.069 % 92.698 ± 0.109 % + 57 7.2622 ± 0.0801 -0.01087 ± 0.00134 0.03233 ± 0.00050 5.074 ± 0.068 % 92.717 ± 0.108 % + 58 7.2991 ± 0.0801 -0.01060 ± 0.00135 0.03285 ± 0.00051 5.105 ± 0.068 % 92.685 ± 0.107 % + 59 7.2730 ± 0.0790 -0.01019 ± 0.00134 0.03303 ± 0.00050 5.122 ± 0.067 % 92.703 ± 0.106 % + 60 7.2581 ± 0.0782 -0.00988 ± 0.00132 0.03283 ± 0.00050 5.108 ± 0.066 % 92.731 ± 0.105 % + 61 7.3270 ± 0.0786 -0.01127 ± 0.00133 0.03354 ± 0.00051 5.171 ± 0.067 % 92.669 ± 0.104 % + 62 7.3500 ± 0.0782 -0.01116 ± 0.00132 0.03334 ± 0.00051 5.148 ± 0.066 % 92.683 ± 0.103 % + 63 7.3977 ± 0.0782 -0.01075 ± 0.00130 0.03318 ± 0.00050 5.131 ± 0.066 % 92.672 ± 0.103 % + 64 7.4287 ± 0.0780 -0.01075 ± 0.00129 0.03301 ± 0.00049 5.110 ± 0.065 % 92.682 ± 0.102 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.428722 ± 0.077970 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 99.26% +Mean ln(PPL(Q)/PPL(base)) : -0.010754 ± 0.001287 +Mean PPL(Q)/PPL(base) : 0.989303 ± 0.001273 +Mean PPL(Q)-PPL(base) : -0.080321 ± 0.009697 + +====== KL divergence statistics ====== +Mean KLD: 0.033014 ± 0.000492 +Maximum KLD: 9.246984 +99.9% KLD: 1.432204 +99.0% KLD: 0.340676 +95.0% KLD: 0.116262 +90.0% KLD: 0.068759 +Median KLD: 0.011378 +10.0% KLD: 0.000187 + 5.0% KLD: 0.000054 + 1.0% KLD: 0.000001 + 0.1% KLD: -0.000133 +Minimum KLD: -0.000665 + +====== Token probability statistics ====== +Mean Δp: -0.116 ± 0.020 % +Maximum Δp: 87.794% +99.9% Δp: 38.577% +99.0% Δp: 15.190% +95.0% Δp: 5.955% +90.0% Δp: 3.186% +75.0% Δp: 0.519% +Median Δp: -0.002% +25.0% Δp: -0.737% +10.0% Δp: -3.626% + 5.0% Δp: -6.368% + 1.0% Δp: -15.612% + 0.1% Δp: -41.143% +Minimum Δp: -79.219% +RMS Δp : 5.110 ± 0.065 % +Same top p: 92.682 ± 0.102 % + diff --git a/appendix/measurements/kld_BAND12.txt b/appendix/measurements/kld_BAND12.txt new file mode 100644 index 0000000000000000000000000000000000000000..5e681d4baeabd415c102b9c5e1f6799fd5eb2884 --- /dev/null +++ b/appendix/measurements/kld_BAND12.txt @@ -0,0 +1,112 @@ +0.16.986.383 I cmn init: llama threadpool init, n_threads = 30 +0.17.061.142 I +0.17.061.321 I system_info: n_threads = 30 (n_threads_batch = 30) / 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.17.408.470 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.35.125.282 I kl_divergence: 17.72 seconds per pass - ETA 18.88 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.8373 ± 0.4576 -0.00345 ± 0.01159 0.04655 ± 0.00395 6.691 ± 0.481 % 92.375 ± 0.830 % + 2 7.2588 ± 0.4204 -0.00685 ± 0.00822 0.04002 ± 0.00242 5.570 ± 0.301 % 91.691 ± 0.610 % + 3 8.1597 ± 0.4000 -0.00480 ± 0.00668 0.03813 ± 0.00242 5.400 ± 0.298 % 92.180 ± 0.485 % + 4 9.1457 ± 0.4128 -0.01900 ± 0.00620 0.04364 ± 0.00280 5.937 ± 0.319 % 92.058 ± 0.423 % + 5 8.5608 ± 0.3404 -0.01755 ± 0.00546 0.04213 ± 0.00231 5.750 ± 0.274 % 92.160 ± 0.376 % + 6 7.2651 ± 0.2551 -0.01903 ± 0.00496 0.04170 ± 0.00209 5.858 ± 0.260 % 92.229 ± 0.342 % + 7 6.6850 ± 0.2139 -0.01669 ± 0.00462 0.04137 ± 0.00184 5.946 ± 0.228 % 92.264 ± 0.316 % + 8 6.5020 ± 0.1921 -0.01494 ± 0.00419 0.03963 ± 0.00163 5.835 ± 0.211 % 92.339 ± 0.294 % + 9 6.8453 ± 0.1918 -0.02180 ± 0.00399 0.04057 ± 0.00147 5.847 ± 0.191 % 92.093 ± 0.281 % + 10 6.9130 ± 0.1840 -0.02110 ± 0.00372 0.03969 ± 0.00135 5.787 ± 0.178 % 92.053 ± 0.267 % + 11 6.9280 ± 0.1746 -0.01945 ± 0.00353 0.03833 ± 0.00124 5.654 ± 0.167 % 92.287 ± 0.252 % + 12 7.1772 ± 0.1743 -0.02038 ± 0.00332 0.03765 ± 0.00114 5.549 ± 0.157 % 92.384 ± 0.239 % + 13 7.2538 ± 0.1695 -0.02502 ± 0.00322 0.03784 ± 0.00108 5.579 ± 0.149 % 92.375 ± 0.230 % + 14 7.2995 ± 0.1640 -0.02510 ± 0.00305 0.03695 ± 0.00102 5.491 ± 0.144 % 92.431 ± 0.221 % + 15 7.3484 ± 0.1595 -0.02620 ± 0.00293 0.03656 ± 0.00096 5.439 ± 0.137 % 92.447 ± 0.213 % + 16 7.5288 ± 0.1584 -0.02670 ± 0.00281 0.03609 ± 0.00090 5.371 ± 0.131 % 92.534 ± 0.205 % + 17 7.5474 ± 0.1533 -0.02518 ± 0.00267 0.03559 ± 0.00088 5.283 ± 0.126 % 92.473 ± 0.200 % + 18 7.6360 ± 0.1506 -0.02374 ± 0.00259 0.03523 ± 0.00084 5.266 ± 0.122 % 92.544 ± 0.194 % + 19 7.5522 ± 0.1453 -0.02538 ± 0.00253 0.03524 ± 0.00080 5.240 ± 0.117 % 92.530 ± 0.189 % + 20 7.2853 ± 0.1360 -0.02897 ± 0.00263 0.04118 ± 0.00122 5.823 ± 0.127 % 92.287 ± 0.187 % + 21 7.3156 ± 0.1331 -0.03024 ± 0.00256 0.04156 ± 0.00118 5.828 ± 0.124 % 92.254 ± 0.182 % + 22 7.4000 ± 0.1322 -0.02881 ± 0.00249 0.04110 ± 0.00114 5.814 ± 0.121 % 92.304 ± 0.178 % + 23 7.4369 ± 0.1297 -0.02734 ± 0.00241 0.04050 ± 0.00109 5.748 ± 0.117 % 92.303 ± 0.174 % + 24 7.5173 ± 0.1287 -0.03017 ± 0.00237 0.04084 ± 0.00105 5.747 ± 0.114 % 92.265 ± 0.170 % + 25 7.5804 ± 0.1275 -0.03024 ± 0.00231 0.04042 ± 0.00101 5.684 ± 0.111 % 92.231 ± 0.167 % + 26 7.5344 ± 0.1240 -0.02970 ± 0.00225 0.04005 ± 0.00098 5.660 ± 0.108 % 92.248 ± 0.164 % + 27 7.6851 ± 0.1244 -0.02958 ± 0.00220 0.03985 ± 0.00095 5.628 ± 0.106 % 92.173 ± 0.162 % + 28 7.7605 ± 0.1235 -0.02956 ± 0.00216 0.03964 ± 0.00092 5.597 ± 0.104 % 92.169 ± 0.159 % + 29 7.7561 ± 0.1215 -0.02969 ± 0.00212 0.03952 ± 0.00089 5.590 ± 0.101 % 92.166 ± 0.156 % + 30 7.7880 ± 0.1204 -0.03163 ± 0.00211 0.03991 ± 0.00088 5.591 ± 0.098 % 92.121 ± 0.154 % + 31 7.6862 ± 0.1166 -0.03089 ± 0.00207 0.03957 ± 0.00086 5.576 ± 0.097 % 92.136 ± 0.151 % + 32 7.5589 ± 0.1127 -0.02997 ± 0.00203 0.03935 ± 0.00084 5.601 ± 0.095 % 92.198 ± 0.148 % + 33 7.5516 ± 0.1111 -0.02883 ± 0.00200 0.03952 ± 0.00086 5.600 ± 0.094 % 92.186 ± 0.146 % + 34 7.5339 ± 0.1090 -0.02808 ± 0.00195 0.03905 ± 0.00083 5.563 ± 0.092 % 92.180 ± 0.144 % + 35 7.5332 ± 0.1074 -0.02796 ± 0.00191 0.03882 ± 0.00081 5.540 ± 0.090 % 92.230 ± 0.141 % + 36 7.5368 ± 0.1060 -0.02817 ± 0.00188 0.03863 ± 0.00079 5.532 ± 0.088 % 92.280 ± 0.139 % + 37 7.3989 ± 0.1022 -0.02746 ± 0.00184 0.03812 ± 0.00077 5.503 ± 0.087 % 92.357 ± 0.137 % + 38 7.2947 ± 0.0989 -0.02670 ± 0.00181 0.03785 ± 0.00076 5.498 ± 0.086 % 92.386 ± 0.135 % + 39 7.1856 ± 0.0958 -0.02588 ± 0.00178 0.03744 ± 0.00074 5.471 ± 0.084 % 92.433 ± 0.132 % + 40 7.0648 ± 0.0925 -0.02506 ± 0.00174 0.03693 ± 0.00072 5.444 ± 0.083 % 92.483 ± 0.130 % + 41 6.9905 ± 0.0901 -0.02425 ± 0.00171 0.03670 ± 0.00072 5.436 ± 0.084 % 92.502 ± 0.129 % + 42 7.0226 ± 0.0895 -0.02383 ± 0.00169 0.03657 ± 0.00071 5.422 ± 0.082 % 92.529 ± 0.127 % + 43 7.2129 ± 0.0917 -0.02456 ± 0.00167 0.03658 ± 0.00069 5.399 ± 0.081 % 92.464 ± 0.126 % + 44 7.3512 ± 0.0929 -0.02432 ± 0.00164 0.03654 ± 0.00069 5.366 ± 0.080 % 92.442 ± 0.125 % + 45 7.3298 ± 0.0914 -0.02325 ± 0.00164 0.03729 ± 0.00068 5.413 ± 0.078 % 92.391 ± 0.124 % + 46 7.3342 ± 0.0904 -0.02326 ± 0.00163 0.03762 ± 0.00067 5.442 ± 0.078 % 92.358 ± 0.122 % + 47 7.3920 ± 0.0903 -0.02423 ± 0.00162 0.03782 ± 0.00067 5.453 ± 0.078 % 92.342 ± 0.121 % + 48 7.3923 ± 0.0892 -0.02359 ± 0.00159 0.03755 ± 0.00066 5.432 ± 0.077 % 92.347 ± 0.120 % + 49 7.4433 ± 0.0891 -0.02329 ± 0.00158 0.03771 ± 0.00066 5.437 ± 0.076 % 92.298 ± 0.119 % + 50 7.4021 ± 0.0875 -0.02258 ± 0.00156 0.03739 ± 0.00065 5.408 ± 0.075 % 92.346 ± 0.118 % + 51 7.4343 ± 0.0870 -0.02247 ± 0.00153 0.03716 ± 0.00064 5.383 ± 0.074 % 92.362 ± 0.116 % + 52 7.3731 ± 0.0853 -0.02234 ± 0.00152 0.03701 ± 0.00063 5.377 ± 0.074 % 92.390 ± 0.115 % + 53 7.3454 ± 0.0841 -0.02169 ± 0.00150 0.03678 ± 0.00062 5.358 ± 0.073 % 92.405 ± 0.114 % + 54 7.3064 ± 0.0827 -0.02144 ± 0.00148 0.03647 ± 0.00061 5.330 ± 0.072 % 92.435 ± 0.113 % + 55 7.2637 ± 0.0813 -0.02226 ± 0.00147 0.03644 ± 0.00060 5.341 ± 0.072 % 92.438 ± 0.111 % + 56 7.2521 ± 0.0805 -0.02300 ± 0.00145 0.03633 ± 0.00059 5.333 ± 0.071 % 92.452 ± 0.110 % + 57 7.1762 ± 0.0787 -0.02277 ± 0.00143 0.03610 ± 0.00058 5.309 ± 0.070 % 92.471 ± 0.109 % + 58 7.2089 ± 0.0786 -0.02303 ± 0.00143 0.03643 ± 0.00058 5.335 ± 0.070 % 92.443 ± 0.109 % + 59 7.1845 ± 0.0776 -0.02244 ± 0.00142 0.03685 ± 0.00058 5.370 ± 0.070 % 92.450 ± 0.108 % + 60 7.1689 ± 0.0768 -0.02224 ± 0.00141 0.03664 ± 0.00057 5.359 ± 0.069 % 92.470 ± 0.107 % + 61 7.2306 ± 0.0771 -0.02453 ± 0.00142 0.03742 ± 0.00060 5.414 ± 0.069 % 92.398 ± 0.106 % + 62 7.2544 ± 0.0767 -0.02425 ± 0.00140 0.03718 ± 0.00059 5.390 ± 0.069 % 92.418 ± 0.105 % + 63 7.3003 ± 0.0767 -0.02399 ± 0.00138 0.03693 ± 0.00058 5.367 ± 0.068 % 92.420 ± 0.104 % + 64 7.3310 ± 0.0764 -0.02399 ± 0.00137 0.03676 ± 0.00057 5.351 ± 0.067 % 92.432 ± 0.103 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.331023 ± 0.076448 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 99.16% +Mean ln(PPL(Q)/PPL(base)) : -0.023993 ± 0.001368 +Mean PPL(Q)/PPL(base) : 0.976293 ± 0.001335 +Mean PPL(Q)-PPL(base) : -0.178020 ± 0.010505 + +====== KL divergence statistics ====== +Mean KLD: 0.036760 ± 0.000572 +Maximum KLD: 13.933623 +99.9% KLD: 1.553122 +99.0% KLD: 0.374749 +95.0% KLD: 0.129424 +90.0% KLD: 0.076851 +Median KLD: 0.012635 +10.0% KLD: 0.000197 + 5.0% KLD: 0.000057 + 1.0% KLD: 0.000000 + 0.1% KLD: -0.000150 +Minimum KLD: -0.000764 + +====== Token probability statistics ====== +Mean Δp: -0.087 ± 0.021 % +Maximum Δp: 95.796% +99.9% Δp: 42.865% +99.0% Δp: 16.771% +95.0% Δp: 6.344% +90.0% Δp: 3.318% +75.0% Δp: 0.545% +Median Δp: -0.002% +25.0% Δp: -0.805% +10.0% Δp: -3.825% + 5.0% Δp: -6.688% + 1.0% Δp: -15.859% + 0.1% Δp: -41.566% +Minimum Δp: -84.142% +RMS Δp : 5.351 ± 0.067 % +Same top p: 92.432 ± 0.103 % + diff --git a/appendix/measurements/kld_BAND18.txt b/appendix/measurements/kld_BAND18.txt new file mode 100644 index 0000000000000000000000000000000000000000..94328ac223eff3620018681906ef19a40dc64bb3 --- /dev/null +++ b/appendix/measurements/kld_BAND18.txt @@ -0,0 +1,112 @@ +0.16.985.356 I cmn init: llama threadpool init, n_threads = 30 +0.17.063.143 I +0.17.063.322 I system_info: n_threads = 30 (n_threads_batch = 30) / 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.17.385.102 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.34.029.317 I kl_divergence: 16.64 seconds per pass - ETA 17.75 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.8763 ± 0.4634 0.00320 ± 0.01175 0.04839 ± 0.00340 6.882 ± 0.440 % 92.669 ± 0.815 % + 2 7.2725 ± 0.4222 -0.00497 ± 0.00826 0.04185 ± 0.00218 5.833 ± 0.284 % 91.789 ± 0.607 % + 3 8.1713 ± 0.4014 -0.00338 ± 0.00686 0.03933 ± 0.00238 5.519 ± 0.291 % 92.147 ± 0.486 % + 4 9.2480 ± 0.4199 -0.00789 ± 0.00601 0.04268 ± 0.00235 5.719 ± 0.275 % 92.033 ± 0.423 % + 5 8.6243 ± 0.3445 -0.01016 ± 0.00541 0.04179 ± 0.00198 5.493 ± 0.233 % 92.102 ± 0.377 % + 6 7.3137 ± 0.2580 -0.01235 ± 0.00490 0.04131 ± 0.00181 5.592 ± 0.220 % 91.984 ± 0.347 % + 7 6.7390 ± 0.2168 -0.00865 ± 0.00459 0.04124 ± 0.00163 5.752 ± 0.201 % 92.096 ± 0.319 % + 8 6.5495 ± 0.1944 -0.00766 ± 0.00414 0.03952 ± 0.00145 5.643 ± 0.187 % 92.253 ± 0.296 % + 9 6.8928 ± 0.1940 -0.01489 ± 0.00399 0.04082 ± 0.00133 5.717 ± 0.173 % 91.778 ± 0.286 % + 10 6.9570 ± 0.1859 -0.01476 ± 0.00371 0.03960 ± 0.00121 5.630 ± 0.161 % 91.916 ± 0.270 % + 11 6.9632 ± 0.1762 -0.01439 ± 0.00350 0.03817 ± 0.00111 5.498 ± 0.150 % 92.038 ± 0.255 % + 12 7.2248 ± 0.1763 -0.01376 ± 0.00327 0.03712 ± 0.00103 5.391 ± 0.141 % 92.139 ± 0.243 % + 13 7.3174 ± 0.1721 -0.01630 ± 0.00316 0.03757 ± 0.00100 5.512 ± 0.137 % 92.142 ± 0.233 % + 14 7.3599 ± 0.1664 -0.01685 ± 0.00300 0.03662 ± 0.00094 5.432 ± 0.132 % 92.229 ± 0.224 % + 15 7.4089 ± 0.1619 -0.01801 ± 0.00289 0.03620 ± 0.00089 5.372 ± 0.126 % 92.232 ± 0.216 % + 16 7.5889 ± 0.1607 -0.01874 ± 0.00277 0.03571 ± 0.00084 5.297 ± 0.120 % 92.278 ± 0.209 % + 17 7.6016 ± 0.1553 -0.01804 ± 0.00264 0.03518 ± 0.00081 5.217 ± 0.115 % 92.272 ± 0.202 % + 18 7.6877 ± 0.1525 -0.01700 ± 0.00256 0.03494 ± 0.00077 5.197 ± 0.111 % 92.310 ± 0.196 % + 19 7.6032 ± 0.1471 -0.01864 ± 0.00249 0.03477 ± 0.00074 5.181 ± 0.108 % 92.350 ± 0.191 % + 20 7.3346 ± 0.1377 -0.02222 ± 0.00259 0.04057 ± 0.00117 5.726 ± 0.121 % 92.160 ± 0.188 % + 21 7.3669 ± 0.1348 -0.02326 ± 0.00252 0.04108 ± 0.00114 5.767 ± 0.120 % 92.124 ± 0.184 % + 22 7.4575 ± 0.1340 -0.02107 ± 0.00246 0.04071 ± 0.00110 5.749 ± 0.118 % 92.180 ± 0.179 % + 23 7.4924 ± 0.1315 -0.01991 ± 0.00238 0.04022 ± 0.00105 5.692 ± 0.114 % 92.180 ± 0.175 % + 24 7.5735 ± 0.1304 -0.02273 ± 0.00234 0.04062 ± 0.00102 5.699 ± 0.111 % 92.131 ± 0.172 % + 25 7.6377 ± 0.1292 -0.02272 ± 0.00229 0.04032 ± 0.00098 5.648 ± 0.108 % 92.074 ± 0.169 % + 26 7.5909 ± 0.1257 -0.02223 ± 0.00223 0.03993 ± 0.00095 5.622 ± 0.105 % 92.146 ± 0.165 % + 27 7.7399 ± 0.1260 -0.02247 ± 0.00218 0.03969 ± 0.00092 5.593 ± 0.103 % 92.104 ± 0.162 % + 28 7.8143 ± 0.1251 -0.02265 ± 0.00214 0.03943 ± 0.00089 5.557 ± 0.101 % 92.124 ± 0.159 % + 29 7.8128 ± 0.1231 -0.02241 ± 0.00210 0.03935 ± 0.00087 5.554 ± 0.098 % 92.136 ± 0.156 % + 30 7.8512 ± 0.1221 -0.02354 ± 0.00209 0.03963 ± 0.00086 5.575 ± 0.097 % 92.118 ± 0.154 % + 31 7.7512 ± 0.1183 -0.02248 ± 0.00205 0.03924 ± 0.00083 5.566 ± 0.095 % 92.145 ± 0.151 % + 32 7.6248 ± 0.1143 -0.02128 ± 0.00200 0.03896 ± 0.00081 5.589 ± 0.095 % 92.195 ± 0.148 % + 33 7.6174 ± 0.1127 -0.02015 ± 0.00197 0.03899 ± 0.00080 5.578 ± 0.093 % 92.168 ± 0.146 % + 34 7.5956 ± 0.1105 -0.01992 ± 0.00193 0.03857 ± 0.00078 5.540 ± 0.091 % 92.197 ± 0.144 % + 35 7.5945 ± 0.1090 -0.01987 ± 0.00189 0.03838 ± 0.00077 5.523 ± 0.089 % 92.216 ± 0.142 % + 36 7.5961 ± 0.1075 -0.02033 ± 0.00186 0.03823 ± 0.00075 5.522 ± 0.088 % 92.264 ± 0.139 % + 37 7.4552 ± 0.1035 -0.01988 ± 0.00182 0.03766 ± 0.00073 5.484 ± 0.086 % 92.336 ± 0.137 % + 38 7.3479 ± 0.1002 -0.01943 ± 0.00179 0.03736 ± 0.00071 5.465 ± 0.085 % 92.383 ± 0.135 % + 39 7.2368 ± 0.0970 -0.01878 ± 0.00176 0.03697 ± 0.00070 5.442 ± 0.083 % 92.420 ± 0.133 % + 40 7.1132 ± 0.0937 -0.01823 ± 0.00172 0.03648 ± 0.00068 5.415 ± 0.082 % 92.483 ± 0.130 % + 41 7.0368 ± 0.0911 -0.01766 ± 0.00169 0.03614 ± 0.00067 5.398 ± 0.081 % 92.478 ± 0.129 % + 42 7.0683 ± 0.0905 -0.01734 ± 0.00167 0.03607 ± 0.00065 5.384 ± 0.080 % 92.461 ± 0.127 % + 43 7.2577 ± 0.0927 -0.01838 ± 0.00165 0.03612 ± 0.00064 5.366 ± 0.078 % 92.391 ± 0.126 % + 44 7.3953 ± 0.0939 -0.01833 ± 0.00163 0.03607 ± 0.00064 5.333 ± 0.077 % 92.369 ± 0.125 % + 45 7.3718 ± 0.0924 -0.01754 ± 0.00162 0.03699 ± 0.00064 5.408 ± 0.077 % 92.291 ± 0.124 % + 46 7.3766 ± 0.0913 -0.01749 ± 0.00160 0.03728 ± 0.00063 5.423 ± 0.076 % 92.263 ± 0.123 % + 47 7.4364 ± 0.0913 -0.01823 ± 0.00159 0.03746 ± 0.00063 5.431 ± 0.076 % 92.248 ± 0.122 % + 48 7.4348 ± 0.0901 -0.01786 ± 0.00157 0.03724 ± 0.00062 5.409 ± 0.075 % 92.265 ± 0.121 % + 49 7.4874 ± 0.0900 -0.01738 ± 0.00155 0.03727 ± 0.00061 5.399 ± 0.074 % 92.258 ± 0.119 % + 50 7.4458 ± 0.0884 -0.01670 ± 0.00153 0.03699 ± 0.00060 5.374 ± 0.073 % 92.295 ± 0.118 % + 51 7.4767 ± 0.0879 -0.01679 ± 0.00151 0.03671 ± 0.00059 5.345 ± 0.072 % 92.308 ± 0.117 % + 52 7.4143 ± 0.0862 -0.01677 ± 0.00149 0.03649 ± 0.00058 5.346 ± 0.072 % 92.342 ± 0.115 % + 53 7.3854 ± 0.0849 -0.01627 ± 0.00147 0.03625 ± 0.00057 5.327 ± 0.071 % 92.340 ± 0.114 % + 54 7.3450 ± 0.0834 -0.01618 ± 0.00145 0.03596 ± 0.00056 5.305 ± 0.070 % 92.372 ± 0.113 % + 55 7.3025 ± 0.0821 -0.01693 ± 0.00144 0.03589 ± 0.00055 5.306 ± 0.069 % 92.356 ± 0.112 % + 56 7.2927 ± 0.0813 -0.01741 ± 0.00143 0.03575 ± 0.00054 5.301 ± 0.068 % 92.363 ± 0.111 % + 57 7.2164 ± 0.0795 -0.01719 ± 0.00141 0.03550 ± 0.00053 5.275 ± 0.067 % 92.396 ± 0.110 % + 58 7.2519 ± 0.0794 -0.01709 ± 0.00141 0.03594 ± 0.00054 5.300 ± 0.067 % 92.374 ± 0.109 % + 59 7.2252 ± 0.0784 -0.01678 ± 0.00140 0.03629 ± 0.00054 5.327 ± 0.066 % 92.387 ± 0.108 % + 60 7.2093 ± 0.0776 -0.01663 ± 0.00139 0.03607 ± 0.00053 5.312 ± 0.066 % 92.419 ± 0.107 % + 61 7.2727 ± 0.0779 -0.01871 ± 0.00140 0.03670 ± 0.00054 5.352 ± 0.066 % 92.367 ± 0.106 % + 62 7.2972 ± 0.0775 -0.01838 ± 0.00138 0.03652 ± 0.00053 5.332 ± 0.065 % 92.371 ± 0.105 % + 63 7.3434 ± 0.0775 -0.01811 ± 0.00137 0.03630 ± 0.00052 5.307 ± 0.064 % 92.375 ± 0.105 % + 64 7.3742 ± 0.0772 -0.01812 ± 0.00135 0.03611 ± 0.00052 5.287 ± 0.064 % 92.395 ± 0.104 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.374235 ± 0.077236 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 99.18% +Mean ln(PPL(Q)/PPL(base)) : -0.018116 ± 0.001349 +Mean PPL(Q)/PPL(base) : 0.982047 ± 0.001325 +Mean PPL(Q)-PPL(base) : -0.134808 ± 0.010239 + +====== KL divergence statistics ====== +Mean KLD: 0.036107 ± 0.000517 +Maximum KLD: 13.522428 +99.9% KLD: 1.435223 +99.0% KLD: 0.376872 +95.0% KLD: 0.128838 +90.0% KLD: 0.076459 +Median KLD: 0.012456 +10.0% KLD: 0.000194 + 5.0% KLD: 0.000052 + 1.0% KLD: -0.000001 + 0.1% KLD: -0.000145 +Minimum KLD: -0.000920 + +====== Token probability statistics ====== +Mean Δp: -0.041 ± 0.021 % +Maximum Δp: 86.424% +99.9% Δp: 41.604% +99.0% Δp: 16.736% +95.0% Δp: 6.499% +90.0% Δp: 3.429% +75.0% Δp: 0.575% +Median Δp: -0.001% +25.0% Δp: -0.755% +10.0% Δp: -3.732% + 5.0% Δp: -6.596% + 1.0% Δp: -15.497% + 0.1% Δp: -41.141% +Minimum Δp: -81.360% +RMS Δp : 5.287 ± 0.064 % +Same top p: 92.395 ± 0.104 % + diff --git a/appendix/measurements/kld_BAND25.txt b/appendix/measurements/kld_BAND25.txt new file mode 100644 index 0000000000000000000000000000000000000000..0fc9045eb904f71fa36e23ad38b2717225cf01e0 --- /dev/null +++ b/appendix/measurements/kld_BAND25.txt @@ -0,0 +1,112 @@ +0.17.537.497 I cmn init: llama threadpool init, n_threads = 30 +0.17.621.111 I +0.17.621.292 I system_info: n_threads = 30 (n_threads_batch = 30) / 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.17.938.909 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.34.761.984 I kl_divergence: 16.82 seconds per pass - ETA 17.93 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.8081 ± 0.4553 -0.00847 ± 0.01156 0.05079 ± 0.00377 6.751 ± 0.522 % 92.473 ± 0.825 % + 2 7.2159 ± 0.4176 -0.01278 ± 0.00837 0.04366 ± 0.00234 5.803 ± 0.331 % 91.496 ± 0.617 % + 3 8.1543 ± 0.4011 -0.00546 ± 0.00684 0.04049 ± 0.00215 5.501 ± 0.277 % 91.952 ± 0.491 % + 4 9.2350 ± 0.4196 -0.00929 ± 0.00590 0.04387 ± 0.00244 5.734 ± 0.284 % 91.862 ± 0.427 % + 5 8.6439 ± 0.3462 -0.00789 ± 0.00542 0.04321 ± 0.00206 5.618 ± 0.248 % 91.808 ± 0.383 % + 6 7.3187 ± 0.2585 -0.01168 ± 0.00494 0.04293 ± 0.00183 5.775 ± 0.219 % 91.870 ± 0.349 % + 7 6.7112 ± 0.2159 -0.01278 ± 0.00462 0.04305 ± 0.00170 5.902 ± 0.203 % 91.956 ± 0.321 % + 8 6.5189 ± 0.1934 -0.01235 ± 0.00419 0.04130 ± 0.00151 5.797 ± 0.188 % 92.094 ± 0.298 % + 9 6.8673 ± 0.1932 -0.01861 ± 0.00400 0.04222 ± 0.00137 5.758 ± 0.172 % 91.756 ± 0.287 % + 10 6.9351 ± 0.1853 -0.01791 ± 0.00372 0.04082 ± 0.00125 5.649 ± 0.160 % 91.887 ± 0.270 % + 11 6.9501 ± 0.1759 -0.01627 ± 0.00351 0.03915 ± 0.00115 5.513 ± 0.150 % 92.127 ± 0.254 % + 12 7.2039 ± 0.1757 -0.01666 ± 0.00328 0.03797 ± 0.00105 5.396 ± 0.141 % 92.188 ± 0.242 % + 13 7.2932 ± 0.1713 -0.01961 ± 0.00317 0.03815 ± 0.00102 5.461 ± 0.135 % 92.187 ± 0.233 % + 14 7.3431 ± 0.1659 -0.01914 ± 0.00301 0.03718 ± 0.00096 5.363 ± 0.128 % 92.215 ± 0.224 % + 15 7.3929 ± 0.1615 -0.02017 ± 0.00290 0.03674 ± 0.00090 5.330 ± 0.123 % 92.167 ± 0.217 % + 16 7.5713 ± 0.1603 -0.02107 ± 0.00279 0.03617 ± 0.00085 5.253 ± 0.117 % 92.229 ± 0.209 % + 17 7.5870 ± 0.1550 -0.01995 ± 0.00266 0.03550 ± 0.00082 5.167 ± 0.112 % 92.283 ± 0.202 % + 18 7.6700 ± 0.1521 -0.01931 ± 0.00257 0.03522 ± 0.00079 5.131 ± 0.108 % 92.359 ± 0.196 % + 19 7.5885 ± 0.1468 -0.02058 ± 0.00250 0.03498 ± 0.00075 5.113 ± 0.104 % 92.355 ± 0.191 % + 20 7.3256 ± 0.1375 -0.02345 ± 0.00263 0.04138 ± 0.00126 5.739 ± 0.123 % 92.136 ± 0.188 % + 21 7.3610 ± 0.1347 -0.02406 ± 0.00257 0.04194 ± 0.00123 5.802 ± 0.123 % 92.059 ± 0.184 % + 22 7.4482 ± 0.1338 -0.02232 ± 0.00249 0.04143 ± 0.00118 5.768 ± 0.120 % 92.104 ± 0.180 % + 23 7.4840 ± 0.1313 -0.02103 ± 0.00242 0.04093 ± 0.00113 5.713 ± 0.117 % 92.129 ± 0.176 % + 24 7.5631 ± 0.1302 -0.02411 ± 0.00238 0.04128 ± 0.00109 5.737 ± 0.114 % 92.107 ± 0.172 % + 25 7.6293 ± 0.1291 -0.02381 ± 0.00232 0.04084 ± 0.00105 5.672 ± 0.111 % 92.051 ± 0.169 % + 26 7.5808 ± 0.1255 -0.02356 ± 0.00226 0.04038 ± 0.00101 5.625 ± 0.108 % 92.093 ± 0.165 % + 27 7.7289 ± 0.1258 -0.02390 ± 0.00221 0.04013 ± 0.00098 5.606 ± 0.106 % 92.039 ± 0.163 % + 28 7.8042 ± 0.1249 -0.02395 ± 0.00216 0.03982 ± 0.00095 5.561 ± 0.104 % 92.044 ± 0.160 % + 29 7.8032 ± 0.1229 -0.02364 ± 0.00212 0.03979 ± 0.00093 5.570 ± 0.101 % 92.069 ± 0.157 % + 30 7.8390 ± 0.1219 -0.02510 ± 0.00211 0.04002 ± 0.00091 5.588 ± 0.101 % 92.063 ± 0.154 % + 31 7.7383 ± 0.1180 -0.02414 ± 0.00206 0.03964 ± 0.00089 5.575 ± 0.100 % 92.092 ± 0.152 % + 32 7.6115 ± 0.1141 -0.02302 ± 0.00202 0.03942 ± 0.00087 5.616 ± 0.101 % 92.149 ± 0.149 % + 33 7.6031 ± 0.1125 -0.02203 ± 0.00199 0.03943 ± 0.00086 5.616 ± 0.099 % 92.156 ± 0.146 % + 34 7.5823 ± 0.1103 -0.02168 ± 0.00195 0.03900 ± 0.00084 5.574 ± 0.097 % 92.186 ± 0.144 % + 35 7.5808 ± 0.1086 -0.02167 ± 0.00191 0.03871 ± 0.00082 5.553 ± 0.095 % 92.219 ± 0.142 % + 36 7.5832 ± 0.1072 -0.02203 ± 0.00188 0.03847 ± 0.00080 5.540 ± 0.093 % 92.305 ± 0.139 % + 37 7.4434 ± 0.1033 -0.02147 ± 0.00184 0.03789 ± 0.00078 5.496 ± 0.092 % 92.373 ± 0.136 % + 38 7.3353 ± 0.0999 -0.02115 ± 0.00181 0.03754 ± 0.00076 5.474 ± 0.090 % 92.391 ± 0.134 % + 39 7.2231 ± 0.0968 -0.02067 ± 0.00177 0.03706 ± 0.00074 5.439 ± 0.088 % 92.433 ± 0.132 % + 40 7.1000 ± 0.0934 -0.02009 ± 0.00173 0.03658 ± 0.00072 5.424 ± 0.087 % 92.478 ± 0.130 % + 41 7.0232 ± 0.0909 -0.01959 ± 0.00171 0.03625 ± 0.00071 5.407 ± 0.086 % 92.483 ± 0.129 % + 42 7.0533 ± 0.0902 -0.01946 ± 0.00168 0.03613 ± 0.00069 5.388 ± 0.084 % 92.461 ± 0.127 % + 43 7.2419 ± 0.0924 -0.02055 ± 0.00166 0.03617 ± 0.00068 5.366 ± 0.083 % 92.412 ± 0.126 % + 44 7.3799 ± 0.0936 -0.02041 ± 0.00164 0.03612 ± 0.00068 5.338 ± 0.082 % 92.398 ± 0.125 % + 45 7.3566 ± 0.0921 -0.01961 ± 0.00163 0.03706 ± 0.00068 5.399 ± 0.081 % 92.349 ± 0.124 % + 46 7.3619 ± 0.0911 -0.01949 ± 0.00162 0.03725 ± 0.00066 5.405 ± 0.080 % 92.346 ± 0.123 % + 47 7.4222 ± 0.0910 -0.02015 ± 0.00160 0.03742 ± 0.00066 5.409 ± 0.080 % 92.328 ± 0.121 % + 48 7.4214 ± 0.0899 -0.01967 ± 0.00157 0.03716 ± 0.00065 5.387 ± 0.078 % 92.337 ± 0.120 % + 49 7.4741 ± 0.0897 -0.01916 ± 0.00156 0.03715 ± 0.00064 5.378 ± 0.078 % 92.322 ± 0.119 % + 50 7.4336 ± 0.0882 -0.01834 ± 0.00154 0.03688 ± 0.00063 5.352 ± 0.077 % 92.368 ± 0.117 % + 51 7.4652 ± 0.0877 -0.01833 ± 0.00152 0.03661 ± 0.00062 5.323 ± 0.076 % 92.398 ± 0.116 % + 52 7.4030 ± 0.0860 -0.01829 ± 0.00150 0.03635 ± 0.00061 5.318 ± 0.075 % 92.445 ± 0.115 % + 53 7.3744 ± 0.0847 -0.01776 ± 0.00148 0.03608 ± 0.00060 5.299 ± 0.074 % 92.447 ± 0.113 % + 54 7.3323 ± 0.0832 -0.01790 ± 0.00146 0.03583 ± 0.00059 5.289 ± 0.073 % 92.466 ± 0.112 % + 55 7.2899 ± 0.0819 -0.01867 ± 0.00145 0.03582 ± 0.00058 5.297 ± 0.073 % 92.468 ± 0.111 % + 56 7.2800 ± 0.0811 -0.01915 ± 0.00144 0.03569 ± 0.00057 5.291 ± 0.072 % 92.501 ± 0.110 % + 57 7.2027 ± 0.0793 -0.01909 ± 0.00142 0.03547 ± 0.00056 5.267 ± 0.071 % 92.535 ± 0.109 % + 58 7.2411 ± 0.0792 -0.01857 ± 0.00142 0.03593 ± 0.00056 5.297 ± 0.071 % 92.500 ± 0.108 % + 59 7.2130 ± 0.0782 -0.01848 ± 0.00141 0.03625 ± 0.00056 5.318 ± 0.070 % 92.498 ± 0.107 % + 60 7.1977 ± 0.0774 -0.01823 ± 0.00140 0.03604 ± 0.00056 5.307 ± 0.070 % 92.511 ± 0.106 % + 61 7.2596 ± 0.0777 -0.02052 ± 0.00141 0.03673 ± 0.00057 5.352 ± 0.069 % 92.441 ± 0.106 % + 62 7.2841 ± 0.0773 -0.02017 ± 0.00139 0.03655 ± 0.00056 5.333 ± 0.069 % 92.453 ± 0.105 % + 63 7.3306 ± 0.0773 -0.01985 ± 0.00137 0.03632 ± 0.00055 5.309 ± 0.068 % 92.445 ± 0.104 % + 64 7.3619 ± 0.0770 -0.01979 ± 0.00136 0.03614 ± 0.00054 5.292 ± 0.067 % 92.446 ± 0.103 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.361934 ± 0.077037 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 99.17% +Mean ln(PPL(Q)/PPL(base)) : -0.019785 ± 0.001359 +Mean PPL(Q)/PPL(base) : 0.980409 ± 0.001333 +Mean PPL(Q)-PPL(base) : -0.147109 ± 0.010342 + +====== KL divergence statistics ====== +Mean KLD: 0.036136 ± 0.000542 +Maximum KLD: 13.558516 +99.9% KLD: 1.497926 +99.0% KLD: 0.382545 +95.0% KLD: 0.127583 +90.0% KLD: 0.075575 +Median KLD: 0.012124 +10.0% KLD: 0.000183 + 5.0% KLD: 0.000050 + 1.0% KLD: 0.000000 + 0.1% KLD: -0.000128 +Minimum KLD: -0.000768 + +====== Token probability statistics ====== +Mean Δp: -0.027 ± 0.021 % +Maximum Δp: 86.379% +99.9% Δp: 41.707% +99.0% Δp: 16.631% +95.0% Δp: 6.410% +90.0% Δp: 3.362% +75.0% Δp: 0.575% +Median Δp: -0.001% +25.0% Δp: -0.736% +10.0% Δp: -3.659% + 5.0% Δp: -6.538% + 1.0% Δp: -15.296% + 0.1% Δp: -39.546% +Minimum Δp: -94.852% +RMS Δp : 5.292 ± 0.067 % +Same top p: 92.446 ± 0.103 % + diff --git a/appendix/measurements/kld_C.txt b/appendix/measurements/kld_C.txt new file mode 100644 index 0000000000000000000000000000000000000000..ff37ea2b34210d235a8041315c6c3f2917304026 --- /dev/null +++ b/appendix/measurements/kld_C.txt @@ -0,0 +1,112 @@ +0.03.125.524 I cmn init: llama threadpool init, n_threads = 30 +0.03.226.101 I +0.03.226.311 I system_info: n_threads = 30 (n_threads_batch = 30) / 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.03.580.111 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.22.733.894 I kl_divergence: 19.15 seconds per pass - ETA 20.42 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.7746 ± 0.4554 -0.01426 ± 0.01375 0.05820 ± 0.00457 6.691 ± 0.433 % 92.766 ± 0.810 % + 2 7.2331 ± 0.4201 -0.01040 ± 0.00946 0.05664 ± 0.00303 6.400 ± 0.407 % 90.665 ± 0.643 % + 3 8.2875 ± 0.4125 0.01074 ± 0.00791 0.05622 ± 0.00323 6.316 ± 0.358 % 91.007 ± 0.516 % + 4 9.4415 ± 0.4348 0.01283 ± 0.00714 0.06251 ± 0.00372 6.718 ± 0.336 % 90.909 ± 0.449 % + 5 8.8131 ± 0.3564 0.01150 ± 0.00618 0.05963 ± 0.00305 6.641 ± 0.292 % 90.850 ± 0.403 % + 6 7.4349 ± 0.2651 0.00408 ± 0.00553 0.05798 ± 0.00263 6.712 ± 0.259 % 90.860 ± 0.368 % + 7 6.8543 ± 0.2231 0.00833 ± 0.00538 0.05967 ± 0.00246 7.119 ± 0.250 % 90.951 ± 0.339 % + 8 6.6484 ± 0.1994 0.00732 ± 0.00490 0.05798 ± 0.00219 7.110 ± 0.233 % 91.080 ± 0.315 % + 9 7.0948 ± 0.2028 0.01398 ± 0.00471 0.05946 ± 0.00200 7.095 ± 0.214 % 90.605 ± 0.304 % + 10 7.1470 ± 0.1938 0.01218 ± 0.00438 0.05691 ± 0.00182 6.889 ± 0.200 % 90.811 ± 0.286 % + 11 7.1399 ± 0.1831 0.01068 ± 0.00410 0.05499 ± 0.00167 6.760 ± 0.187 % 90.918 ± 0.271 % + 12 7.4052 ± 0.1830 0.01090 ± 0.00385 0.05307 ± 0.00154 6.606 ± 0.177 % 91.072 ± 0.257 % + 13 7.5379 ± 0.1798 0.01339 ± 0.00367 0.05226 ± 0.00144 6.553 ± 0.167 % 91.097 ± 0.247 % + 14 7.5616 ± 0.1732 0.01018 ± 0.00349 0.05130 ± 0.00135 6.460 ± 0.161 % 91.251 ± 0.236 % + 15 7.6193 ± 0.1687 0.01000 ± 0.00338 0.05077 ± 0.00127 6.411 ± 0.153 % 91.307 ± 0.227 % + 16 7.8173 ± 0.1679 0.01090 ± 0.00326 0.05023 ± 0.00120 6.332 ± 0.146 % 91.245 ± 0.221 % + 17 7.8196 ± 0.1619 0.01025 ± 0.00312 0.04919 ± 0.00114 6.228 ± 0.140 % 91.254 ± 0.214 % + 18 7.8969 ± 0.1587 0.00985 ± 0.00300 0.04860 ± 0.00109 6.158 ± 0.136 % 91.262 ± 0.208 % + 19 7.8058 ± 0.1529 0.00765 ± 0.00290 0.04772 ± 0.00104 6.100 ± 0.132 % 91.352 ± 0.202 % + 20 7.5547 ± 0.1436 0.00734 ± 0.00293 0.05156 ± 0.00113 6.457 ± 0.134 % 91.266 ± 0.197 % + 21 7.5832 ± 0.1405 0.00567 ± 0.00285 0.05200 ± 0.00110 6.488 ± 0.131 % 91.179 ± 0.193 % + 22 7.6227 ± 0.1381 0.00084 ± 0.00282 0.05230 ± 0.00107 6.497 ± 0.127 % 91.180 ± 0.189 % + 23 7.6515 ± 0.1353 0.00110 ± 0.00273 0.05182 ± 0.00103 6.433 ± 0.123 % 91.181 ± 0.185 % + 24 7.7747 ± 0.1352 0.00349 ± 0.00268 0.05205 ± 0.00100 6.438 ± 0.121 % 91.178 ± 0.181 % + 25 7.8289 ± 0.1336 0.00202 ± 0.00261 0.05161 ± 0.00096 6.396 ± 0.118 % 91.155 ± 0.178 % + 26 7.7709 ± 0.1297 0.00121 ± 0.00255 0.05130 ± 0.00094 6.361 ± 0.114 % 91.236 ± 0.173 % + 27 7.9280 ± 0.1303 0.00153 ± 0.00247 0.05046 ± 0.00091 6.282 ± 0.112 % 91.260 ± 0.170 % + 28 7.9986 ± 0.1291 0.00065 ± 0.00241 0.05023 ± 0.00088 6.248 ± 0.109 % 91.213 ± 0.167 % + 29 8.0009 ± 0.1271 0.00138 ± 0.00238 0.05053 ± 0.00087 6.284 ± 0.108 % 91.202 ± 0.164 % + 30 8.0259 ± 0.1257 -0.00154 ± 0.00237 0.05119 ± 0.00089 6.288 ± 0.105 % 91.176 ± 0.162 % + 31 7.9212 ± 0.1218 -0.00078 ± 0.00232 0.05084 ± 0.00087 6.270 ± 0.103 % 91.193 ± 0.159 % + 32 7.7756 ± 0.1173 -0.00170 ± 0.00230 0.05161 ± 0.00087 6.333 ± 0.102 % 91.224 ± 0.156 % + 33 7.7737 ± 0.1158 0.00016 ± 0.00227 0.05201 ± 0.00087 6.333 ± 0.100 % 91.164 ± 0.154 % + 34 7.7450 ± 0.1134 -0.00045 ± 0.00223 0.05173 ± 0.00085 6.328 ± 0.099 % 91.145 ± 0.152 % + 35 7.7340 ± 0.1114 -0.00166 ± 0.00220 0.05170 ± 0.00083 6.316 ± 0.096 % 91.135 ± 0.150 % + 36 7.7347 ± 0.1099 -0.00225 ± 0.00216 0.05127 ± 0.00081 6.285 ± 0.095 % 91.194 ± 0.148 % + 37 7.5918 ± 0.1059 -0.00173 ± 0.00211 0.05055 ± 0.00079 6.247 ± 0.093 % 91.258 ± 0.145 % + 38 7.4811 ± 0.1025 -0.00147 ± 0.00207 0.05025 ± 0.00077 6.241 ± 0.092 % 91.316 ± 0.143 % + 39 7.3614 ± 0.0991 -0.00172 ± 0.00204 0.04986 ± 0.00077 6.213 ± 0.090 % 91.325 ± 0.141 % + 40 7.2331 ± 0.0956 -0.00152 ± 0.00200 0.04926 ± 0.00075 6.195 ± 0.089 % 91.388 ± 0.139 % + 41 7.1551 ± 0.0931 -0.00099 ± 0.00197 0.04880 ± 0.00073 6.172 ± 0.087 % 91.441 ± 0.137 % + 42 7.1853 ± 0.0924 -0.00092 ± 0.00194 0.04867 ± 0.00072 6.150 ± 0.086 % 91.440 ± 0.135 % + 43 7.3778 ± 0.0946 -0.00197 ± 0.00191 0.04858 ± 0.00070 6.120 ± 0.084 % 91.384 ± 0.134 % + 44 7.5153 ± 0.0957 -0.00223 ± 0.00189 0.04845 ± 0.00069 6.084 ± 0.083 % 91.360 ± 0.132 % + 45 7.4890 ± 0.0941 -0.00176 ± 0.00189 0.04962 ± 0.00069 6.155 ± 0.082 % 91.257 ± 0.132 % + 46 7.4918 ± 0.0930 -0.00200 ± 0.00187 0.05009 ± 0.00068 6.165 ± 0.081 % 91.253 ± 0.130 % + 47 7.5589 ± 0.0931 -0.00190 ± 0.00185 0.05003 ± 0.00067 6.157 ± 0.080 % 91.250 ± 0.129 % + 48 7.5548 ± 0.0919 -0.00185 ± 0.00182 0.04987 ± 0.00067 6.147 ± 0.080 % 91.239 ± 0.128 % + 49 7.6099 ± 0.0918 -0.00115 ± 0.00181 0.04986 ± 0.00066 6.132 ± 0.078 % 91.216 ± 0.126 % + 50 7.5628 ± 0.0901 -0.00111 ± 0.00178 0.04949 ± 0.00065 6.102 ± 0.077 % 91.245 ± 0.125 % + 51 7.5945 ± 0.0895 -0.00116 ± 0.00175 0.04900 ± 0.00063 6.065 ± 0.076 % 91.308 ± 0.123 % + 52 7.5239 ± 0.0877 -0.00209 ± 0.00173 0.04901 ± 0.00063 6.072 ± 0.076 % 91.313 ± 0.122 % + 53 7.4972 ± 0.0864 -0.00124 ± 0.00171 0.04879 ± 0.00062 6.059 ± 0.075 % 91.322 ± 0.121 % + 54 7.4540 ± 0.0849 -0.00144 ± 0.00169 0.04852 ± 0.00061 6.050 ± 0.074 % 91.333 ± 0.120 % + 55 7.4045 ± 0.0835 -0.00307 ± 0.00169 0.04858 ± 0.00061 6.058 ± 0.073 % 91.313 ± 0.119 % + 56 7.3970 ± 0.0827 -0.00322 ± 0.00167 0.04848 ± 0.00060 6.057 ± 0.073 % 91.332 ± 0.118 % + 57 7.3157 ± 0.0808 -0.00352 ± 0.00165 0.04821 ± 0.00059 6.043 ± 0.072 % 91.352 ± 0.116 % + 58 7.3534 ± 0.0808 -0.00318 ± 0.00165 0.04870 ± 0.00059 6.066 ± 0.071 % 91.344 ± 0.115 % + 59 7.3261 ± 0.0797 -0.00292 ± 0.00164 0.04942 ± 0.00061 6.119 ± 0.071 % 91.308 ± 0.115 % + 60 7.3074 ± 0.0788 -0.00311 ± 0.00162 0.04925 ± 0.00060 6.100 ± 0.070 % 91.323 ± 0.114 % + 61 7.3522 ± 0.0789 -0.00785 ± 0.00166 0.05137 ± 0.00065 6.191 ± 0.070 % 91.201 ± 0.113 % + 62 7.3762 ± 0.0785 -0.00760 ± 0.00164 0.05108 ± 0.00064 6.166 ± 0.069 % 91.207 ± 0.112 % + 63 7.4239 ± 0.0784 -0.00721 ± 0.00162 0.05074 ± 0.00063 6.141 ± 0.069 % 91.219 ± 0.111 % + 64 7.4532 ± 0.0782 -0.00746 ± 0.00161 0.05056 ± 0.00063 6.121 ± 0.068 % 91.239 ± 0.110 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.453244 ± 0.078172 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 98.84% +Mean ln(PPL(Q)/PPL(base)) : -0.007459 ± 0.001605 +Mean PPL(Q)/PPL(base) : 0.992569 ± 0.001594 +Mean PPL(Q)-PPL(base) : -0.055799 ± 0.012055 + +====== KL divergence statistics ====== +Mean KLD: 0.050559 ± 0.000627 +Maximum KLD: 10.205813 +99.9% KLD: 1.798456 +99.0% KLD: 0.551951 +95.0% KLD: 0.186914 +90.0% KLD: 0.108423 +Median KLD: 0.017439 +10.0% KLD: 0.000265 + 5.0% KLD: 0.000074 + 1.0% KLD: 0.000002 + 0.1% KLD: -0.000127 +Minimum KLD: -0.000620 + +====== Token probability statistics ====== +Mean Δp: -0.232 ± 0.024 % +Maximum Δp: 93.422% +99.9% Δp: 46.367% +99.0% Δp: 18.218% +95.0% Δp: 7.112% +90.0% Δp: 3.744% +75.0% Δp: 0.576% +Median Δp: -0.006% +25.0% Δp: -1.007% +10.0% Δp: -4.662% + 5.0% Δp: -8.194% + 1.0% Δp: -19.743% + 0.1% Δp: -45.428% +Minimum Δp: -91.063% +RMS Δp : 6.121 ± 0.068 % +Same top p: 91.239 ± 0.110 % + diff --git a/appendix/measurements/kld_CTRL-19G-ICE.txt b/appendix/measurements/kld_CTRL-19G-ICE.txt new file mode 100644 index 0000000000000000000000000000000000000000..3f80c47b687e00c556a848b736190d64c0bec975 --- /dev/null +++ b/appendix/measurements/kld_CTRL-19G-ICE.txt @@ -0,0 +1,111 @@ +0.04.811.851 I +0.04.812.006 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.05.135.998 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.06.885.259 I kl_divergence: 1.75 seconds per pass - ETA 1.85 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.7833 ± 0.4508 -0.01275 ± 0.01454 0.07915 ± 0.00579 8.002 ± 0.535 % 88.954 ± 0.981 % + 2 7.3898 ± 0.4323 0.01104 ± 0.01044 0.07059 ± 0.00350 7.018 ± 0.351 % 89.052 ± 0.690 % + 3 8.3892 ± 0.4164 0.02294 ± 0.00852 0.06800 ± 0.00354 7.187 ± 0.389 % 89.736 ± 0.548 % + 4 9.5061 ± 0.4348 0.01965 ± 0.00771 0.07443 ± 0.00389 7.507 ± 0.347 % 89.858 ± 0.472 % + 5 8.9037 ± 0.3584 0.02172 ± 0.00687 0.07195 ± 0.00322 7.344 ± 0.296 % 90.029 ± 0.419 % + 6 7.5269 ± 0.2667 0.01638 ± 0.00618 0.07035 ± 0.00282 7.487 ± 0.265 % 89.899 ± 0.385 % + 7 6.8182 ± 0.2190 0.00304 ± 0.00602 0.07240 ± 0.00270 7.797 ± 0.260 % 89.973 ± 0.355 % + 8 6.6126 ± 0.1958 0.00193 ± 0.00548 0.07114 ± 0.00243 7.705 ± 0.237 % 89.907 ± 0.333 % + 9 7.0222 ± 0.1980 0.00370 ± 0.00516 0.07151 ± 0.00221 7.605 ± 0.218 % 89.551 ± 0.319 % + 10 7.0805 ± 0.1896 0.00284 ± 0.00479 0.06901 ± 0.00202 7.495 ± 0.206 % 89.658 ± 0.301 % + 11 7.0838 ± 0.1795 0.00278 ± 0.00450 0.06675 ± 0.00186 7.341 ± 0.194 % 89.674 ± 0.287 % + 12 7.3486 ± 0.1795 0.00323 ± 0.00422 0.06445 ± 0.00171 7.170 ± 0.183 % 89.858 ± 0.272 % + 13 7.4886 ± 0.1768 0.00683 ± 0.00402 0.06393 ± 0.00161 7.177 ± 0.173 % 89.871 ± 0.262 % + 14 7.5258 ± 0.1708 0.00544 ± 0.00380 0.06179 ± 0.00150 7.006 ± 0.165 % 89.994 ± 0.251 % + 15 7.5721 ± 0.1661 0.00378 ± 0.00366 0.06139 ± 0.00142 6.982 ± 0.157 % 90.062 ± 0.242 % + 16 7.7601 ± 0.1650 0.00356 ± 0.00354 0.06090 ± 0.00135 6.907 ± 0.151 % 90.090 ± 0.234 % + 17 7.7677 ± 0.1594 0.00358 ± 0.00338 0.05950 ± 0.00128 6.795 ± 0.145 % 90.167 ± 0.226 % + 18 7.8469 ± 0.1564 0.00350 ± 0.00328 0.05918 ± 0.00123 6.759 ± 0.140 % 90.263 ± 0.218 % + 19 7.7659 ± 0.1510 0.00253 ± 0.00316 0.05823 ± 0.00119 6.666 ± 0.135 % 90.323 ± 0.212 % + 20 7.5403 ± 0.1425 0.00544 ± 0.00319 0.06375 ± 0.00132 7.154 ± 0.142 % 90.098 ± 0.209 % + 21 7.5886 ± 0.1399 0.00639 ± 0.00313 0.06403 ± 0.00131 7.142 ± 0.139 % 90.029 ± 0.204 % + 22 7.6349 ± 0.1378 0.00244 ± 0.00307 0.06454 ± 0.00127 7.161 ± 0.135 % 90.052 ± 0.200 % + 23 7.6638 ± 0.1350 0.00271 ± 0.00299 0.06391 ± 0.00122 7.082 ± 0.130 % 90.059 ± 0.195 % + 24 7.7682 ± 0.1344 0.00266 ± 0.00294 0.06388 ± 0.00119 7.054 ± 0.127 % 90.066 ± 0.191 % + 25 7.8383 ± 0.1333 0.00322 ± 0.00287 0.06307 ± 0.00115 6.984 ± 0.123 % 90.076 ± 0.187 % + 26 7.7847 ± 0.1295 0.00299 ± 0.00280 0.06286 ± 0.00111 6.970 ± 0.120 % 90.089 ± 0.183 % + 27 7.9389 ± 0.1300 0.00291 ± 0.00273 0.06214 ± 0.00108 6.902 ± 0.118 % 90.098 ± 0.180 % + 28 8.0109 ± 0.1289 0.00220 ± 0.00268 0.06180 ± 0.00105 6.870 ± 0.115 % 90.089 ± 0.177 % + 29 8.0076 ± 0.1267 0.00222 ± 0.00265 0.06273 ± 0.00106 7.014 ± 0.117 % 90.066 ± 0.174 % + 30 8.0317 ± 0.1254 -0.00082 ± 0.00262 0.06332 ± 0.00106 7.025 ± 0.115 % 90.033 ± 0.171 % + 31 7.9285 ± 0.1215 0.00014 ± 0.00257 0.06293 ± 0.00103 7.001 ± 0.113 % 90.039 ± 0.168 % + 32 7.7909 ± 0.1172 0.00027 ± 0.00254 0.06323 ± 0.00102 7.047 ± 0.112 % 90.112 ± 0.165 % + 33 7.7680 ± 0.1153 -0.00057 ± 0.00251 0.06365 ± 0.00104 7.050 ± 0.110 % 90.056 ± 0.163 % + 34 7.7427 ± 0.1129 -0.00074 ± 0.00246 0.06291 ± 0.00101 6.997 ± 0.108 % 90.124 ± 0.160 % + 35 7.7394 ± 0.1112 -0.00097 ± 0.00241 0.06247 ± 0.00098 6.953 ± 0.105 % 90.180 ± 0.157 % + 36 7.7470 ± 0.1098 -0.00066 ± 0.00237 0.06194 ± 0.00096 6.924 ± 0.103 % 90.255 ± 0.155 % + 37 7.6047 ± 0.1058 -0.00003 ± 0.00232 0.06115 ± 0.00094 6.887 ± 0.102 % 90.320 ± 0.152 % + 38 7.4948 ± 0.1024 0.00036 ± 0.00229 0.06103 ± 0.00092 6.910 ± 0.101 % 90.348 ± 0.150 % + 39 7.3786 ± 0.0991 0.00063 ± 0.00225 0.06084 ± 0.00091 6.922 ± 0.100 % 90.343 ± 0.148 % + 40 7.2511 ± 0.0957 0.00096 ± 0.00221 0.06020 ± 0.00090 6.903 ± 0.098 % 90.423 ± 0.145 % + 41 7.1732 ± 0.0931 0.00154 ± 0.00217 0.05985 ± 0.00088 6.911 ± 0.098 % 90.444 ± 0.144 % + 42 7.2045 ± 0.0924 0.00175 ± 0.00214 0.05966 ± 0.00086 6.883 ± 0.096 % 90.444 ± 0.142 % + 43 7.4009 ± 0.0947 0.00116 ± 0.00211 0.05951 ± 0.00085 6.841 ± 0.095 % 90.398 ± 0.140 % + 44 7.5456 ± 0.0960 0.00179 ± 0.00208 0.05923 ± 0.00084 6.804 ± 0.093 % 90.369 ± 0.139 % + 45 7.5222 ± 0.0944 0.00266 ± 0.00208 0.06082 ± 0.00083 6.890 ± 0.092 % 90.264 ± 0.138 % + 46 7.5250 ± 0.0934 0.00242 ± 0.00207 0.06140 ± 0.00083 6.901 ± 0.091 % 90.255 ± 0.137 % + 47 7.5876 ± 0.0933 0.00189 ± 0.00204 0.06135 ± 0.00082 6.894 ± 0.089 % 90.241 ± 0.135 % + 48 7.5800 ± 0.0921 0.00149 ± 0.00201 0.06127 ± 0.00081 6.890 ± 0.089 % 90.237 ± 0.134 % + 49 7.6268 ± 0.0918 0.00106 ± 0.00199 0.06110 ± 0.00079 6.886 ± 0.087 % 90.231 ± 0.133 % + 50 7.5809 ± 0.0902 0.00128 ± 0.00197 0.06070 ± 0.00078 6.859 ± 0.086 % 90.270 ± 0.131 % + 51 7.6211 ± 0.0897 0.00234 ± 0.00194 0.06019 ± 0.00077 6.834 ± 0.086 % 90.313 ± 0.129 % + 52 7.5574 ± 0.0880 0.00235 ± 0.00192 0.05998 ± 0.00076 6.822 ± 0.085 % 90.336 ± 0.128 % + 53 7.5264 ± 0.0867 0.00265 ± 0.00190 0.05957 ± 0.00075 6.804 ± 0.084 % 90.347 ± 0.127 % + 54 7.4824 ± 0.0852 0.00236 ± 0.00187 0.05922 ± 0.00074 6.790 ± 0.083 % 90.348 ± 0.126 % + 55 7.4401 ± 0.0838 0.00173 ± 0.00186 0.05912 ± 0.00073 6.794 ± 0.082 % 90.349 ± 0.124 % + 56 7.4364 ± 0.0831 0.00210 ± 0.00184 0.05908 ± 0.00072 6.797 ± 0.081 % 90.340 ± 0.123 % + 57 7.3560 ± 0.0812 0.00197 ± 0.00182 0.05879 ± 0.00071 6.777 ± 0.080 % 90.367 ± 0.122 % + 58 7.3832 ± 0.0810 0.00086 ± 0.00182 0.05948 ± 0.00072 6.798 ± 0.079 % 90.343 ± 0.121 % + 59 7.3519 ± 0.0799 0.00061 ± 0.00181 0.06006 ± 0.00072 6.833 ± 0.078 % 90.321 ± 0.120 % + 60 7.3320 ± 0.0790 0.00025 ± 0.00179 0.05984 ± 0.00071 6.812 ± 0.077 % 90.331 ± 0.119 % + 61 7.3975 ± 0.0794 -0.00170 ± 0.00179 0.06088 ± 0.00074 6.862 ± 0.078 % 90.278 ± 0.119 % + 62 7.4215 ± 0.0790 -0.00149 ± 0.00177 0.06057 ± 0.00073 6.830 ± 0.077 % 90.269 ± 0.118 % + 63 7.4680 ± 0.0789 -0.00128 ± 0.00175 0.06018 ± 0.00072 6.804 ± 0.076 % 90.287 ± 0.117 % + 64 7.4996 ± 0.0787 -0.00126 ± 0.00174 0.06001 ± 0.00071 6.782 ± 0.075 % 90.310 ± 0.116 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.499565 ± 0.078688 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 98.64% +Mean ln(PPL(Q)/PPL(base)) : -0.001263 ± 0.001736 +Mean PPL(Q)/PPL(base) : 0.998738 ± 0.001734 +Mean PPL(Q)-PPL(base) : -0.009478 ± 0.013033 + +====== KL divergence statistics ====== +Mean KLD: 0.060015 ± 0.000710 +Maximum KLD: 8.653050 +99.9% KLD: 2.424127 +99.0% KLD: 0.638752 +95.0% KLD: 0.217372 +90.0% KLD: 0.128581 +Median KLD: 0.020794 +10.0% KLD: 0.000334 + 5.0% KLD: 0.000097 + 1.0% KLD: 0.000003 + 0.1% KLD: -0.000155 +Minimum KLD: -0.000756 + +====== Token probability statistics ====== +Mean Δp: -0.403 ± 0.026 % +Maximum Δp: 97.079% +99.9% Δp: 49.758% +99.0% Δp: 19.289% +95.0% Δp: 7.458% +90.0% Δp: 3.961% +75.0% Δp: 0.605% +Median Δp: -0.009% +25.0% Δp: -1.154% +10.0% Δp: -5.284% + 5.0% Δp: -9.417% + 1.0% Δp: -23.008% + 0.1% Δp: -53.005% +Minimum Δp: -94.057% +RMS Δp : 6.782 ± 0.075 % +Same top p: 90.310 ± 0.116 % + diff --git a/appendix/measurements/kld_CTRL-21G-ICE.txt b/appendix/measurements/kld_CTRL-21G-ICE.txt new file mode 100644 index 0000000000000000000000000000000000000000..27f70c2f7080f847171d1c3d350b0250a0f07292 --- /dev/null +++ b/appendix/measurements/kld_CTRL-21G-ICE.txt @@ -0,0 +1,111 @@ +0.05.350.140 I +0.05.350.263 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.05.670.452 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.07.506.851 I kl_divergence: 1.84 seconds per pass - ETA 1.95 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 6.0309 ± 0.4838 0.02917 ± 0.01217 0.04986 ± 0.00417 6.052 ± 0.488 % 91.789 ± 0.859 % + 2 7.3944 ± 0.4341 0.01166 ± 0.00847 0.04562 ± 0.00267 5.548 ± 0.315 % 91.544 ± 0.615 % + 3 8.4114 ± 0.4198 0.02558 ± 0.00719 0.04565 ± 0.00286 5.658 ± 0.324 % 92.017 ± 0.489 % + 4 9.3801 ± 0.4280 0.00630 ± 0.00681 0.05353 ± 0.00409 6.182 ± 0.332 % 91.618 ± 0.433 % + 5 8.7711 ± 0.3529 0.00673 ± 0.00597 0.05056 ± 0.00331 6.000 ± 0.285 % 91.613 ± 0.388 % + 6 7.4346 ± 0.2640 0.00404 ± 0.00536 0.04938 ± 0.00282 6.135 ± 0.249 % 91.577 ± 0.355 % + 7 6.8170 ± 0.2205 0.00286 ± 0.00504 0.04905 ± 0.00249 6.299 ± 0.226 % 91.831 ± 0.324 % + 8 6.6178 ± 0.1975 0.00272 ± 0.00455 0.04742 ± 0.00220 6.192 ± 0.211 % 91.838 ± 0.303 % + 9 6.9964 ± 0.1983 0.00002 ± 0.00428 0.04768 ± 0.00199 6.124 ± 0.192 % 91.539 ± 0.290 % + 10 7.0684 ± 0.1902 0.00114 ± 0.00396 0.04543 ± 0.00180 5.937 ± 0.179 % 91.662 ± 0.273 % + 11 7.0694 ± 0.1801 0.00074 ± 0.00375 0.04390 ± 0.00165 5.803 ± 0.168 % 91.780 ± 0.259 % + 12 7.3282 ± 0.1798 0.00045 ± 0.00351 0.04247 ± 0.00152 5.688 ± 0.159 % 91.846 ± 0.247 % + 13 7.4336 ± 0.1758 -0.00055 ± 0.00336 0.04222 ± 0.00142 5.715 ± 0.150 % 91.819 ± 0.238 % + 14 7.4678 ± 0.1698 -0.00230 ± 0.00319 0.04106 ± 0.00132 5.616 ± 0.144 % 91.810 ± 0.229 % + 15 7.5214 ± 0.1653 -0.00295 ± 0.00306 0.04063 ± 0.00124 5.575 ± 0.136 % 91.795 ± 0.222 % + 16 7.6955 ± 0.1637 -0.00480 ± 0.00296 0.04048 ± 0.00117 5.528 ± 0.130 % 91.771 ± 0.215 % + 17 7.7066 ± 0.1583 -0.00431 ± 0.00283 0.03961 ± 0.00111 5.433 ± 0.125 % 91.864 ± 0.207 % + 18 7.7867 ± 0.1552 -0.00420 ± 0.00272 0.03927 ± 0.00105 5.393 ± 0.120 % 91.941 ± 0.201 % + 19 7.7076 ± 0.1499 -0.00501 ± 0.00263 0.03869 ± 0.00101 5.338 ± 0.116 % 92.041 ± 0.194 % + 20 7.4614 ± 0.1408 -0.00509 ± 0.00265 0.04178 ± 0.00107 5.731 ± 0.122 % 91.901 ± 0.191 % + 21 7.5006 ± 0.1381 -0.00528 ± 0.00260 0.04196 ± 0.00103 5.750 ± 0.119 % 91.840 ± 0.187 % + 22 7.5735 ± 0.1367 -0.00563 ± 0.00253 0.04162 ± 0.00099 5.710 ± 0.115 % 91.895 ± 0.182 % + 23 7.6001 ± 0.1339 -0.00564 ± 0.00245 0.04118 ± 0.00094 5.650 ± 0.112 % 91.908 ± 0.178 % + 24 7.6914 ± 0.1331 -0.00727 ± 0.00240 0.04140 ± 0.00092 5.675 ± 0.110 % 91.935 ± 0.174 % + 25 7.7503 ± 0.1317 -0.00807 ± 0.00236 0.04112 ± 0.00088 5.616 ± 0.107 % 91.922 ± 0.170 % + 26 7.6940 ± 0.1279 -0.00873 ± 0.00230 0.04098 ± 0.00086 5.615 ± 0.105 % 91.954 ± 0.167 % + 27 7.8477 ± 0.1283 -0.00865 ± 0.00224 0.04032 ± 0.00083 5.557 ± 0.102 % 91.952 ± 0.164 % + 28 7.9236 ± 0.1274 -0.00876 ± 0.00219 0.04002 ± 0.00080 5.539 ± 0.100 % 91.935 ± 0.161 % + 29 7.9324 ± 0.1256 -0.00721 ± 0.00216 0.04030 ± 0.00079 5.561 ± 0.098 % 91.930 ± 0.158 % + 30 7.9679 ± 0.1245 -0.00879 ± 0.00214 0.04029 ± 0.00077 5.557 ± 0.096 % 91.968 ± 0.155 % + 31 7.8584 ± 0.1204 -0.00873 ± 0.00210 0.04010 ± 0.00075 5.555 ± 0.094 % 91.978 ± 0.153 % + 32 7.7381 ± 0.1166 -0.00653 ± 0.00207 0.04039 ± 0.00075 5.651 ± 0.098 % 92.000 ± 0.150 % + 33 7.7236 ± 0.1148 -0.00630 ± 0.00204 0.04049 ± 0.00078 5.646 ± 0.096 % 92.017 ± 0.148 % + 34 7.6973 ± 0.1125 -0.00663 ± 0.00200 0.04014 ± 0.00076 5.628 ± 0.094 % 92.016 ± 0.145 % + 35 7.6902 ± 0.1107 -0.00734 ± 0.00196 0.04002 ± 0.00075 5.610 ± 0.092 % 92.018 ± 0.143 % + 36 7.6954 ± 0.1093 -0.00735 ± 0.00193 0.03969 ± 0.00073 5.572 ± 0.091 % 92.079 ± 0.141 % + 37 7.5515 ± 0.1053 -0.00705 ± 0.00189 0.03922 ± 0.00071 5.557 ± 0.090 % 92.143 ± 0.138 % + 38 7.4407 ± 0.1018 -0.00688 ± 0.00185 0.03892 ± 0.00070 5.549 ± 0.088 % 92.206 ± 0.136 % + 39 7.3249 ± 0.0985 -0.00668 ± 0.00182 0.03867 ± 0.00068 5.548 ± 0.087 % 92.217 ± 0.134 % + 40 7.1989 ± 0.0951 -0.00625 ± 0.00178 0.03835 ± 0.00067 5.556 ± 0.086 % 92.253 ± 0.132 % + 41 7.1195 ± 0.0925 -0.00597 ± 0.00175 0.03805 ± 0.00066 5.538 ± 0.084 % 92.247 ± 0.131 % + 42 7.1520 ± 0.0919 -0.00557 ± 0.00172 0.03790 ± 0.00064 5.522 ± 0.083 % 92.252 ± 0.129 % + 43 7.3406 ± 0.0940 -0.00702 ± 0.00171 0.03798 ± 0.00063 5.498 ± 0.082 % 92.194 ± 0.128 % + 44 7.4801 ± 0.0952 -0.00692 ± 0.00169 0.03787 ± 0.00062 5.471 ± 0.080 % 92.200 ± 0.126 % + 45 7.4620 ± 0.0938 -0.00538 ± 0.00169 0.03925 ± 0.00063 5.553 ± 0.080 % 92.091 ± 0.126 % + 46 7.4667 ± 0.0928 -0.00536 ± 0.00167 0.03939 ± 0.00062 5.562 ± 0.079 % 92.076 ± 0.125 % + 47 7.5273 ± 0.0927 -0.00608 ± 0.00166 0.03956 ± 0.00061 5.567 ± 0.078 % 92.068 ± 0.123 % + 48 7.5228 ± 0.0915 -0.00609 ± 0.00163 0.03938 ± 0.00060 5.545 ± 0.077 % 92.056 ± 0.122 % + 49 7.5703 ± 0.0912 -0.00637 ± 0.00162 0.03947 ± 0.00060 5.538 ± 0.076 % 92.016 ± 0.121 % + 50 7.5274 ± 0.0896 -0.00580 ± 0.00159 0.03923 ± 0.00059 5.518 ± 0.075 % 92.051 ± 0.120 % + 51 7.5583 ± 0.0891 -0.00593 ± 0.00157 0.03902 ± 0.00058 5.493 ± 0.074 % 92.042 ± 0.118 % + 52 7.4924 ± 0.0873 -0.00628 ± 0.00155 0.03891 ± 0.00058 5.491 ± 0.073 % 92.054 ± 0.117 % + 53 7.4618 ± 0.0860 -0.00598 ± 0.00153 0.03864 ± 0.00057 5.469 ± 0.072 % 92.056 ± 0.116 % + 54 7.4190 ± 0.0845 -0.00614 ± 0.00151 0.03836 ± 0.00056 5.446 ± 0.071 % 92.082 ± 0.115 % + 55 7.3764 ± 0.0832 -0.00687 ± 0.00150 0.03824 ± 0.00055 5.443 ± 0.070 % 92.086 ± 0.114 % + 56 7.3702 ± 0.0824 -0.00684 ± 0.00149 0.03823 ± 0.00054 5.451 ± 0.069 % 92.077 ± 0.113 % + 57 7.2910 ± 0.0805 -0.00691 ± 0.00147 0.03804 ± 0.00054 5.434 ± 0.069 % 92.104 ± 0.112 % + 58 7.3315 ± 0.0806 -0.00616 ± 0.00147 0.03854 ± 0.00054 5.445 ± 0.068 % 92.077 ± 0.111 % + 59 7.3036 ± 0.0795 -0.00599 ± 0.00147 0.03896 ± 0.00054 5.491 ± 0.068 % 92.080 ± 0.110 % + 60 7.2853 ± 0.0786 -0.00614 ± 0.00145 0.03879 ± 0.00054 5.480 ± 0.067 % 92.092 ± 0.109 % + 61 7.3472 ± 0.0789 -0.00852 ± 0.00146 0.03947 ± 0.00055 5.516 ± 0.067 % 92.021 ± 0.108 % + 62 7.3696 ± 0.0785 -0.00850 ± 0.00144 0.03930 ± 0.00054 5.494 ± 0.066 % 92.014 ± 0.108 % + 63 7.4212 ± 0.0785 -0.00757 ± 0.00143 0.03925 ± 0.00053 5.482 ± 0.065 % 92.006 ± 0.107 % + 64 7.4507 ± 0.0783 -0.00780 ± 0.00142 0.03918 ± 0.00053 5.470 ± 0.065 % 92.023 ± 0.106 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.450665 ± 0.078273 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 99.10% +Mean ln(PPL(Q)/PPL(base)) : -0.007805 ± 0.001417 +Mean PPL(Q)/PPL(base) : 0.992226 ± 0.001406 +Mean PPL(Q)-PPL(base) : -0.058378 ± 0.010646 + +====== KL divergence statistics ====== +Mean KLD: 0.039182 ± 0.000526 +Maximum KLD: 9.558251 +99.9% KLD: 1.416135 +99.0% KLD: 0.419083 +95.0% KLD: 0.140715 +90.0% KLD: 0.083215 +Median KLD: 0.013487 +10.0% KLD: 0.000211 + 5.0% KLD: 0.000060 + 1.0% KLD: 0.000001 + 0.1% KLD: -0.000142 +Minimum KLD: -0.000889 + +====== Token probability statistics ====== +Mean Δp: -0.164 ± 0.021 % +Maximum Δp: 87.223% +99.9% Δp: 41.768% +99.0% Δp: 16.309% +95.0% Δp: 6.372% +90.0% Δp: 3.369% +75.0% Δp: 0.537% +Median Δp: -0.003% +25.0% Δp: -0.836% +10.0% Δp: -3.981% + 5.0% Δp: -7.032% + 1.0% Δp: -17.123% + 0.1% Δp: -43.687% +Minimum Δp: -98.439% +RMS Δp : 5.470 ± 0.065 % +Same top p: 92.023 ± 0.106 % + diff --git a/appendix/measurements/kld_CTRL-23G-ICE.txt b/appendix/measurements/kld_CTRL-23G-ICE.txt new file mode 100644 index 0000000000000000000000000000000000000000..6a077f7928f567c951d3402957abd293cb1ef9d0 --- /dev/null +++ b/appendix/measurements/kld_CTRL-23G-ICE.txt @@ -0,0 +1,111 @@ +0.05.769.522 I +0.05.769.657 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.06.097.336 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.08.038.108 I kl_divergence: 1.94 seconds per pass - ETA 2.07 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.8119 ± 0.4530 -0.00781 ± 0.01065 0.04378 ± 0.00324 6.308 ± 0.402 % 92.278 ± 0.835 % + 2 7.2677 ± 0.4204 -0.00562 ± 0.00755 0.03899 ± 0.00193 5.422 ± 0.257 % 92.033 ± 0.599 % + 3 8.1933 ± 0.4015 -0.00069 ± 0.00637 0.03746 ± 0.00218 5.271 ± 0.277 % 92.603 ± 0.472 % + 4 9.2533 ± 0.4191 -0.00731 ± 0.00556 0.04194 ± 0.00252 5.552 ± 0.268 % 92.326 ± 0.416 % + 5 8.6475 ± 0.3446 -0.00747 ± 0.00487 0.04034 ± 0.00208 5.492 ± 0.239 % 92.493 ± 0.368 % + 6 7.3445 ± 0.2588 -0.00815 ± 0.00447 0.03997 ± 0.00190 5.654 ± 0.236 % 92.441 ± 0.337 % + 7 6.7462 ± 0.2167 -0.00758 ± 0.00420 0.04017 ± 0.00172 5.910 ± 0.225 % 92.417 ± 0.313 % + 8 6.5544 ± 0.1943 -0.00691 ± 0.00382 0.03855 ± 0.00152 5.755 ± 0.204 % 92.522 ± 0.291 % + 9 6.9689 ± 0.1969 -0.00391 ± 0.00365 0.03968 ± 0.00142 5.714 ± 0.186 % 92.147 ± 0.280 % + 10 7.0371 ± 0.1889 -0.00331 ± 0.00341 0.03818 ± 0.00129 5.608 ± 0.177 % 92.326 ± 0.263 % + 11 7.0359 ± 0.1787 -0.00401 ± 0.00324 0.03715 ± 0.00119 5.512 ± 0.166 % 92.491 ± 0.248 % + 12 7.3063 ± 0.1790 -0.00255 ± 0.00304 0.03589 ± 0.00110 5.378 ± 0.156 % 92.481 ± 0.238 % + 13 7.3794 ± 0.1739 -0.00786 ± 0.00296 0.03632 ± 0.00106 5.418 ± 0.147 % 92.518 ± 0.228 % + 14 7.4213 ± 0.1682 -0.00855 ± 0.00281 0.03526 ± 0.00099 5.315 ± 0.140 % 92.564 ± 0.219 % + 15 7.4698 ± 0.1635 -0.00982 ± 0.00272 0.03499 ± 0.00093 5.292 ± 0.134 % 92.616 ± 0.211 % + 16 7.6523 ± 0.1623 -0.01043 ± 0.00264 0.03465 ± 0.00088 5.217 ± 0.128 % 92.638 ± 0.204 % + 17 7.6601 ± 0.1568 -0.01036 ± 0.00251 0.03398 ± 0.00084 5.132 ± 0.122 % 92.651 ± 0.198 % + 18 7.7460 ± 0.1539 -0.00944 ± 0.00243 0.03378 ± 0.00081 5.119 ± 0.118 % 92.701 ± 0.192 % + 19 7.6641 ± 0.1486 -0.01067 ± 0.00236 0.03355 ± 0.00078 5.075 ± 0.114 % 92.699 ± 0.187 % + 20 7.4005 ± 0.1393 -0.01328 ± 0.00241 0.03721 ± 0.00091 5.497 ± 0.123 % 92.493 ± 0.184 % + 21 7.4419 ± 0.1366 -0.01313 ± 0.00237 0.03775 ± 0.00090 5.543 ± 0.122 % 92.417 ± 0.181 % + 22 7.5111 ± 0.1352 -0.01391 ± 0.00231 0.03753 ± 0.00087 5.530 ± 0.119 % 92.446 ± 0.176 % + 23 7.5419 ± 0.1325 -0.01332 ± 0.00223 0.03702 ± 0.00083 5.466 ± 0.115 % 92.443 ± 0.172 % + 24 7.6194 ± 0.1313 -0.01668 ± 0.00221 0.03766 ± 0.00083 5.504 ± 0.115 % 92.388 ± 0.169 % + 25 7.6863 ± 0.1302 -0.01637 ± 0.00215 0.03725 ± 0.00079 5.457 ± 0.111 % 92.336 ± 0.166 % + 26 7.6396 ± 0.1266 -0.01583 ± 0.00210 0.03699 ± 0.00077 5.451 ± 0.109 % 92.379 ± 0.163 % + 27 7.7923 ± 0.1271 -0.01574 ± 0.00204 0.03642 ± 0.00074 5.390 ± 0.107 % 92.346 ± 0.160 % + 28 7.8639 ± 0.1261 -0.01632 ± 0.00200 0.03620 ± 0.00072 5.352 ± 0.104 % 92.340 ± 0.157 % + 29 7.8698 ± 0.1242 -0.01514 ± 0.00197 0.03624 ± 0.00070 5.369 ± 0.101 % 92.345 ± 0.154 % + 30 7.9050 ± 0.1231 -0.01672 ± 0.00196 0.03658 ± 0.00069 5.380 ± 0.098 % 92.317 ± 0.152 % + 31 7.8014 ± 0.1192 -0.01602 ± 0.00192 0.03627 ± 0.00067 5.352 ± 0.096 % 92.347 ± 0.149 % + 32 7.6685 ± 0.1151 -0.01557 ± 0.00189 0.03625 ± 0.00066 5.386 ± 0.095 % 92.391 ± 0.147 % + 33 7.6595 ± 0.1135 -0.01464 ± 0.00186 0.03626 ± 0.00066 5.376 ± 0.093 % 92.393 ± 0.144 % + 34 7.6351 ± 0.1112 -0.01474 ± 0.00182 0.03591 ± 0.00064 5.350 ± 0.091 % 92.387 ± 0.142 % + 35 7.6315 ± 0.1095 -0.01501 ± 0.00178 0.03565 ± 0.00062 5.329 ± 0.089 % 92.401 ± 0.140 % + 36 7.6399 ± 0.1082 -0.01459 ± 0.00175 0.03532 ± 0.00061 5.307 ± 0.088 % 92.460 ± 0.138 % + 37 7.4974 ± 0.1042 -0.01424 ± 0.00172 0.03488 ± 0.00060 5.281 ± 0.087 % 92.507 ± 0.135 % + 38 7.3870 ± 0.1008 -0.01412 ± 0.00169 0.03455 ± 0.00058 5.262 ± 0.085 % 92.563 ± 0.133 % + 39 7.2748 ± 0.0976 -0.01354 ± 0.00166 0.03422 ± 0.00057 5.237 ± 0.084 % 92.596 ± 0.131 % + 40 7.1509 ± 0.0943 -0.01295 ± 0.00162 0.03379 ± 0.00056 5.217 ± 0.082 % 92.642 ± 0.129 % + 41 7.0727 ± 0.0917 -0.01257 ± 0.00160 0.03350 ± 0.00054 5.202 ± 0.081 % 92.642 ± 0.127 % + 42 7.1021 ± 0.0910 -0.01257 ± 0.00158 0.03337 ± 0.00053 5.186 ± 0.080 % 92.655 ± 0.126 % + 43 7.3006 ± 0.0934 -0.01248 ± 0.00156 0.03347 ± 0.00053 5.187 ± 0.081 % 92.621 ± 0.125 % + 44 7.4411 ± 0.0946 -0.01215 ± 0.00154 0.03351 ± 0.00054 5.159 ± 0.079 % 92.635 ± 0.123 % + 45 7.4173 ± 0.0931 -0.01138 ± 0.00154 0.03440 ± 0.00054 5.233 ± 0.078 % 92.553 ± 0.122 % + 46 7.4232 ± 0.0921 -0.01120 ± 0.00152 0.03460 ± 0.00053 5.238 ± 0.077 % 92.545 ± 0.121 % + 47 7.4825 ± 0.0920 -0.01206 ± 0.00151 0.03477 ± 0.00053 5.250 ± 0.077 % 92.540 ± 0.120 % + 48 7.4807 ± 0.0909 -0.01170 ± 0.00149 0.03472 ± 0.00053 5.242 ± 0.076 % 92.516 ± 0.119 % + 49 7.5288 ± 0.0906 -0.01187 ± 0.00148 0.03481 ± 0.00053 5.235 ± 0.075 % 92.481 ± 0.118 % + 50 7.4853 ± 0.0890 -0.01141 ± 0.00145 0.03454 ± 0.00052 5.212 ± 0.074 % 92.526 ± 0.116 % + 51 7.5203 ± 0.0885 -0.01097 ± 0.00143 0.03422 ± 0.00051 5.180 ± 0.073 % 92.557 ± 0.115 % + 52 7.4595 ± 0.0868 -0.01069 ± 0.00142 0.03407 ± 0.00050 5.174 ± 0.073 % 92.582 ± 0.114 % + 53 7.4304 ± 0.0856 -0.01020 ± 0.00140 0.03388 ± 0.00050 5.160 ± 0.072 % 92.569 ± 0.113 % + 54 7.3881 ± 0.0841 -0.01032 ± 0.00138 0.03369 ± 0.00049 5.159 ± 0.071 % 92.589 ± 0.111 % + 55 7.3447 ± 0.0827 -0.01118 ± 0.00138 0.03364 ± 0.00049 5.162 ± 0.070 % 92.598 ± 0.110 % + 56 7.3319 ± 0.0818 -0.01205 ± 0.00137 0.03365 ± 0.00048 5.158 ± 0.069 % 92.602 ± 0.109 % + 57 7.2539 ± 0.0800 -0.01201 ± 0.00135 0.03343 ± 0.00047 5.137 ± 0.068 % 92.638 ± 0.108 % + 58 7.2973 ± 0.0801 -0.01084 ± 0.00136 0.03401 ± 0.00048 5.174 ± 0.068 % 92.611 ± 0.107 % + 59 7.2734 ± 0.0791 -0.01013 ± 0.00135 0.03430 ± 0.00048 5.193 ± 0.067 % 92.601 ± 0.107 % + 60 7.2556 ± 0.0782 -0.01022 ± 0.00133 0.03412 ± 0.00047 5.180 ± 0.067 % 92.616 ± 0.106 % + 61 7.3304 ± 0.0787 -0.01082 ± 0.00133 0.03478 ± 0.00051 5.236 ± 0.069 % 92.576 ± 0.105 % + 62 7.3547 ± 0.0783 -0.01052 ± 0.00132 0.03461 ± 0.00050 5.211 ± 0.068 % 92.580 ± 0.104 % + 63 7.3994 ± 0.0783 -0.01051 ± 0.00131 0.03442 ± 0.00050 5.190 ± 0.067 % 92.562 ± 0.103 % + 64 7.4302 ± 0.0780 -0.01056 ± 0.00129 0.03429 ± 0.00049 5.172 ± 0.067 % 92.574 ± 0.102 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.430186 ± 0.078031 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 99.25% +Mean ln(PPL(Q)/PPL(base)) : -0.010557 ± 0.001292 +Mean PPL(Q)/PPL(base) : 0.989498 ± 0.001278 +Mean PPL(Q)-PPL(base) : -0.078857 ± 0.009729 + +====== KL divergence statistics ====== +Mean KLD: 0.034290 ± 0.000491 +Maximum KLD: 7.434389 +99.9% KLD: 1.506679 +99.0% KLD: 0.346319 +95.0% KLD: 0.122503 +90.0% KLD: 0.072590 +Median KLD: 0.011636 +10.0% KLD: 0.000186 + 5.0% KLD: 0.000051 + 1.0% KLD: -0.000000 + 0.1% KLD: -0.000146 +Minimum KLD: -0.000892 + +====== Token probability statistics ====== +Mean Δp: -0.102 ± 0.020 % +Maximum Δp: 88.208% +99.9% Δp: 40.389% +99.0% Δp: 15.604% +95.0% Δp: 6.070% +90.0% Δp: 3.272% +75.0% Δp: 0.532% +Median Δp: -0.002% +25.0% Δp: -0.746% +10.0% Δp: -3.679% + 5.0% Δp: -6.481% + 1.0% Δp: -15.971% + 0.1% Δp: -39.102% +Minimum Δp: -96.886% +RMS Δp : 5.172 ± 0.067 % +Same top p: 92.574 ± 0.102 % + diff --git a/appendix/measurements/kld_CTRL-25G-ICE.txt b/appendix/measurements/kld_CTRL-25G-ICE.txt new file mode 100644 index 0000000000000000000000000000000000000000..f7bd1a1be539e8fc2f0156a20f077b5f8eb8368f --- /dev/null +++ b/appendix/measurements/kld_CTRL-25G-ICE.txt @@ -0,0 +1,111 @@ +0.05.877.958 I +0.05.878.099 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.06.238.869 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.08.156.863 I kl_divergence: 1.92 seconds per pass - ETA 2.03 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.7968 ± 0.4527 -0.01041 ± 0.00981 0.03069 ± 0.00227 5.166 ± 0.382 % 93.451 ± 0.774 % + 2 7.3020 ± 0.4252 -0.00091 ± 0.00684 0.02912 ± 0.00142 4.744 ± 0.271 % 93.011 ± 0.564 % + 3 8.1596 ± 0.4009 -0.00481 ± 0.00566 0.02795 ± 0.00124 4.594 ± 0.269 % 93.288 ± 0.452 % + 4 9.2581 ± 0.4203 -0.00679 ± 0.00529 0.03488 ± 0.00253 5.058 ± 0.303 % 93.084 ± 0.397 % + 5 8.6612 ± 0.3465 -0.00588 ± 0.00463 0.03295 ± 0.00205 4.942 ± 0.255 % 93.255 ± 0.351 % + 6 7.3374 ± 0.2591 -0.00913 ± 0.00424 0.03346 ± 0.00189 5.239 ± 0.257 % 93.076 ± 0.324 % + 7 6.7062 ± 0.2151 -0.01352 ± 0.00399 0.03368 ± 0.00167 5.417 ± 0.229 % 93.185 ± 0.298 % + 8 6.5151 ± 0.1929 -0.01293 ± 0.00363 0.03236 ± 0.00148 5.303 ± 0.208 % 93.194 ± 0.278 % + 9 6.8917 ± 0.1939 -0.01506 ± 0.00342 0.03281 ± 0.00134 5.234 ± 0.190 % 92.962 ± 0.267 % + 10 6.9693 ± 0.1864 -0.01298 ± 0.00321 0.03179 ± 0.00125 5.168 ± 0.187 % 93.021 ± 0.252 % + 11 6.9712 ± 0.1765 -0.01324 ± 0.00305 0.03082 ± 0.00114 5.079 ± 0.176 % 93.148 ± 0.238 % + 12 7.2369 ± 0.1766 -0.01209 ± 0.00287 0.03005 ± 0.00106 4.965 ± 0.165 % 93.255 ± 0.226 % + 13 7.3133 ± 0.1718 -0.01686 ± 0.00282 0.03076 ± 0.00103 5.037 ± 0.155 % 93.180 ± 0.219 % + 14 7.3681 ± 0.1665 -0.01574 ± 0.00268 0.02986 ± 0.00096 4.939 ± 0.147 % 93.304 ± 0.209 % + 15 7.4124 ± 0.1617 -0.01753 ± 0.00262 0.02976 ± 0.00090 4.899 ± 0.139 % 93.262 ± 0.202 % + 16 7.5946 ± 0.1606 -0.01799 ± 0.00253 0.02968 ± 0.00085 4.850 ± 0.133 % 93.255 ± 0.196 % + 17 7.6106 ± 0.1553 -0.01684 ± 0.00241 0.02905 ± 0.00081 4.770 ± 0.128 % 93.215 ± 0.191 % + 18 7.6924 ± 0.1524 -0.01639 ± 0.00232 0.02870 ± 0.00077 4.753 ± 0.124 % 93.282 ± 0.184 % + 19 7.6046 ± 0.1469 -0.01846 ± 0.00227 0.02911 ± 0.00074 4.758 ± 0.118 % 93.199 ± 0.181 % + 20 7.3674 ± 0.1383 -0.01776 ± 0.00234 0.03277 ± 0.00096 5.213 ± 0.135 % 93.035 ± 0.178 % + 21 7.4083 ± 0.1357 -0.01765 ± 0.00227 0.03284 ± 0.00093 5.193 ± 0.132 % 93.032 ± 0.174 % + 22 7.4719 ± 0.1341 -0.01915 ± 0.00221 0.03261 ± 0.00089 5.168 ± 0.127 % 93.064 ± 0.169 % + 23 7.5057 ± 0.1316 -0.01813 ± 0.00213 0.03221 ± 0.00085 5.110 ± 0.123 % 93.077 ± 0.165 % + 24 7.5908 ± 0.1306 -0.02045 ± 0.00210 0.03247 ± 0.00083 5.121 ± 0.120 % 93.051 ± 0.162 % + 25 7.6495 ± 0.1292 -0.02117 ± 0.00206 0.03275 ± 0.00080 5.094 ± 0.116 % 92.966 ± 0.160 % + 26 7.6041 ± 0.1257 -0.02048 ± 0.00201 0.03241 ± 0.00077 5.069 ± 0.113 % 93.018 ± 0.156 % + 27 7.7582 ± 0.1262 -0.02012 ± 0.00196 0.03190 ± 0.00074 5.009 ± 0.110 % 93.031 ± 0.153 % + 28 7.8309 ± 0.1252 -0.02053 ± 0.00191 0.03171 ± 0.00072 4.974 ± 0.107 % 92.990 ± 0.151 % + 29 7.8302 ± 0.1232 -0.02018 ± 0.00187 0.03155 ± 0.00070 4.968 ± 0.104 % 92.989 ± 0.148 % + 30 7.8625 ± 0.1221 -0.02211 ± 0.00186 0.03178 ± 0.00068 4.971 ± 0.101 % 93.011 ± 0.146 % + 31 7.7611 ± 0.1183 -0.02119 ± 0.00183 0.03144 ± 0.00066 4.942 ± 0.099 % 93.047 ± 0.143 % + 32 7.6238 ± 0.1141 -0.02141 ± 0.00179 0.03123 ± 0.00065 4.954 ± 0.097 % 93.102 ± 0.140 % + 33 7.6209 ± 0.1126 -0.01969 ± 0.00177 0.03124 ± 0.00063 4.948 ± 0.094 % 93.086 ± 0.138 % + 34 7.5962 ± 0.1103 -0.01985 ± 0.00173 0.03099 ± 0.00061 4.922 ± 0.092 % 93.120 ± 0.136 % + 35 7.5948 ± 0.1087 -0.01983 ± 0.00170 0.03074 ± 0.00060 4.900 ± 0.091 % 93.135 ± 0.134 % + 36 7.5953 ± 0.1072 -0.02044 ± 0.00167 0.03058 ± 0.00059 4.892 ± 0.089 % 93.168 ± 0.131 % + 37 7.4546 ± 0.1033 -0.01997 ± 0.00163 0.03012 ± 0.00057 4.863 ± 0.088 % 93.252 ± 0.129 % + 38 7.3478 ± 0.1000 -0.01945 ± 0.00160 0.02989 ± 0.00056 4.851 ± 0.086 % 93.281 ± 0.127 % + 39 7.2352 ± 0.0968 -0.01900 ± 0.00157 0.02950 ± 0.00054 4.826 ± 0.085 % 93.300 ± 0.125 % + 40 7.1122 ± 0.0935 -0.01837 ± 0.00153 0.02915 ± 0.00053 4.809 ± 0.083 % 93.353 ± 0.123 % + 41 7.0336 ± 0.0909 -0.01811 ± 0.00151 0.02889 ± 0.00052 4.796 ± 0.082 % 93.396 ± 0.121 % + 42 7.0658 ± 0.0903 -0.01770 ± 0.00149 0.02876 ± 0.00051 4.777 ± 0.081 % 93.388 ± 0.120 % + 43 7.2564 ± 0.0925 -0.01855 ± 0.00147 0.02888 ± 0.00050 4.771 ± 0.080 % 93.321 ± 0.119 % + 44 7.3917 ± 0.0937 -0.01882 ± 0.00146 0.02892 ± 0.00050 4.747 ± 0.078 % 93.300 ± 0.118 % + 45 7.3665 ± 0.0921 -0.01826 ± 0.00145 0.02954 ± 0.00050 4.783 ± 0.077 % 93.286 ± 0.117 % + 46 7.3738 ± 0.0911 -0.01787 ± 0.00144 0.02965 ± 0.00049 4.784 ± 0.076 % 93.255 ± 0.116 % + 47 7.4335 ± 0.0911 -0.01863 ± 0.00142 0.02967 ± 0.00049 4.782 ± 0.074 % 93.216 ± 0.115 % + 48 7.4320 ± 0.0900 -0.01823 ± 0.00140 0.02950 ± 0.00048 4.768 ± 0.074 % 93.202 ± 0.114 % + 49 7.4812 ± 0.0897 -0.01821 ± 0.00139 0.02959 ± 0.00047 4.770 ± 0.073 % 93.159 ± 0.113 % + 50 7.4390 ± 0.0882 -0.01761 ± 0.00137 0.02938 ± 0.00046 4.750 ± 0.072 % 93.200 ± 0.111 % + 51 7.4705 ± 0.0876 -0.01761 ± 0.00135 0.02911 ± 0.00046 4.729 ± 0.071 % 93.228 ± 0.110 % + 52 7.4082 ± 0.0859 -0.01759 ± 0.00133 0.02897 ± 0.00045 4.726 ± 0.070 % 93.244 ± 0.109 % + 53 7.3794 ± 0.0847 -0.01708 ± 0.00132 0.02876 ± 0.00044 4.704 ± 0.069 % 93.244 ± 0.108 % + 54 7.3373 ± 0.0832 -0.01722 ± 0.00130 0.02857 ± 0.00043 4.695 ± 0.068 % 93.271 ± 0.107 % + 55 7.2946 ± 0.0819 -0.01802 ± 0.00129 0.02852 ± 0.00043 4.693 ± 0.067 % 93.257 ± 0.106 % + 56 7.2865 ± 0.0811 -0.01827 ± 0.00128 0.02843 ± 0.00042 4.688 ± 0.066 % 93.246 ± 0.105 % + 57 7.2102 ± 0.0793 -0.01805 ± 0.00126 0.02827 ± 0.00041 4.670 ± 0.066 % 93.276 ± 0.104 % + 58 7.2551 ± 0.0794 -0.01664 ± 0.00127 0.02876 ± 0.00042 4.701 ± 0.065 % 93.250 ± 0.103 % + 59 7.2291 ± 0.0783 -0.01624 ± 0.00126 0.02899 ± 0.00042 4.723 ± 0.064 % 93.253 ± 0.102 % + 60 7.2147 ± 0.0776 -0.01588 ± 0.00124 0.02885 ± 0.00041 4.709 ± 0.063 % 93.284 ± 0.101 % + 61 7.2897 ± 0.0781 -0.01639 ± 0.00125 0.02942 ± 0.00045 4.752 ± 0.065 % 93.258 ± 0.100 % + 62 7.3132 ± 0.0777 -0.01618 ± 0.00124 0.02929 ± 0.00044 4.734 ± 0.064 % 93.250 ± 0.100 % + 63 7.3601 ± 0.0777 -0.01584 ± 0.00122 0.02910 ± 0.00044 4.713 ± 0.063 % 93.252 ± 0.099 % + 64 7.3911 ± 0.0774 -0.01583 ± 0.00121 0.02896 ± 0.00043 4.697 ± 0.063 % 93.275 ± 0.098 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.391080 ± 0.077442 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 99.35% +Mean ln(PPL(Q)/PPL(base)) : -0.015834 ± 0.001208 +Mean PPL(Q)/PPL(base) : 0.984291 ± 0.001189 +Mean PPL(Q)-PPL(base) : -0.117963 ± 0.009175 + +====== KL divergence statistics ====== +Mean KLD: 0.028955 ± 0.000430 +Maximum KLD: 9.086314 +99.9% KLD: 1.139082 +99.0% KLD: 0.292560 +95.0% KLD: 0.102705 +90.0% KLD: 0.061919 +Median KLD: 0.009867 +10.0% KLD: 0.000162 + 5.0% KLD: 0.000044 + 1.0% KLD: -0.000002 + 0.1% KLD: -0.000138 +Minimum KLD: -0.000905 + +====== Token probability statistics ====== +Mean Δp: -0.068 ± 0.018 % +Maximum Δp: 88.211% +99.9% Δp: 35.552% +99.0% Δp: 14.238% +95.0% Δp: 5.545% +90.0% Δp: 2.995% +75.0% Δp: 0.499% +Median Δp: -0.002% +25.0% Δp: -0.658% +10.0% Δp: -3.304% + 5.0% Δp: -5.901% + 1.0% Δp: -14.120% + 0.1% Δp: -35.247% +Minimum Δp: -92.155% +RMS Δp : 4.697 ± 0.063 % +Same top p: 93.275 ± 0.098 % + diff --git a/appendix/measurements/kld_D.txt b/appendix/measurements/kld_D.txt new file mode 100644 index 0000000000000000000000000000000000000000..d844089e5f4ce56571fc588c1155695291f769d5 --- /dev/null +++ b/appendix/measurements/kld_D.txt @@ -0,0 +1,112 @@ +0.03.422.955 I cmn init: llama threadpool init, n_threads = 30 +0.03.537.991 I +0.03.538.201 I system_info: n_threads = 30 (n_threads_batch = 30) / 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.03.878.781 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.20.045.335 I kl_divergence: 16.17 seconds per pass - ETA 17.23 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.8659 ± 0.4647 0.00144 ± 0.00731 0.01807 ± 0.00194 4.052 ± 0.474 % 95.308 ± 0.661 % + 2 7.3604 ± 0.4319 0.00705 ± 0.00518 0.01696 ± 0.00108 3.498 ± 0.284 % 94.819 ± 0.490 % + 3 8.2667 ± 0.4100 0.00823 ± 0.00427 0.01829 ± 0.00213 3.640 ± 0.395 % 94.591 ± 0.408 % + 4 9.2085 ± 0.4181 -0.01217 ± 0.00432 0.02478 ± 0.00247 4.429 ± 0.388 % 94.233 ± 0.364 % + 5 8.5944 ± 0.3434 -0.01363 ± 0.00374 0.02369 ± 0.00199 4.278 ± 0.326 % 94.409 ± 0.321 % + 6 7.3090 ± 0.2579 -0.01300 ± 0.00345 0.02350 ± 0.00172 4.515 ± 0.293 % 94.314 ± 0.296 % + 7 6.7287 ± 0.2165 -0.01017 ± 0.00331 0.02397 ± 0.00152 4.666 ± 0.254 % 94.344 ± 0.273 % + 8 6.5369 ± 0.1940 -0.00958 ± 0.00299 0.02268 ± 0.00134 4.498 ± 0.231 % 94.404 ± 0.254 % + 9 6.9176 ± 0.1951 -0.01131 ± 0.00283 0.02298 ± 0.00120 4.400 ± 0.211 % 94.081 ± 0.246 % + 10 6.9862 ± 0.1872 -0.01057 ± 0.00264 0.02201 ± 0.00109 4.271 ± 0.196 % 94.174 ± 0.232 % + 11 6.9930 ± 0.1775 -0.01011 ± 0.00255 0.02129 ± 0.00100 4.200 ± 0.183 % 94.313 ± 0.218 % + 12 7.2611 ± 0.1777 -0.00876 ± 0.00240 0.02069 ± 0.00092 4.119 ± 0.172 % 94.379 ± 0.208 % + 13 7.3936 ± 0.1749 -0.00594 ± 0.00233 0.02087 ± 0.00085 4.132 ± 0.160 % 94.383 ± 0.200 % + 14 7.4392 ± 0.1692 -0.00613 ± 0.00221 0.02025 ± 0.00080 4.056 ± 0.152 % 94.442 ± 0.191 % + 15 7.4934 ± 0.1646 -0.00668 ± 0.00214 0.02025 ± 0.00075 4.048 ± 0.143 % 94.435 ± 0.185 % + 16 7.6714 ± 0.1632 -0.00794 ± 0.00206 0.02011 ± 0.00070 3.995 ± 0.136 % 94.446 ± 0.179 % + 17 7.6769 ± 0.1575 -0.00817 ± 0.00199 0.01984 ± 0.00068 3.931 ± 0.131 % 94.486 ± 0.173 % + 18 7.7541 ± 0.1544 -0.00840 ± 0.00191 0.01958 ± 0.00064 3.900 ± 0.125 % 94.558 ± 0.167 % + 19 7.6663 ± 0.1488 -0.01038 ± 0.00187 0.01980 ± 0.00062 3.899 ± 0.119 % 94.474 ± 0.164 % + 20 7.4229 ± 0.1400 -0.01026 ± 0.00187 0.02122 ± 0.00067 4.188 ± 0.128 % 94.350 ± 0.161 % + 21 7.4659 ± 0.1374 -0.00991 ± 0.00182 0.02129 ± 0.00065 4.205 ± 0.126 % 94.354 ± 0.157 % + 22 7.5378 ± 0.1360 -0.01036 ± 0.00177 0.02115 ± 0.00063 4.200 ± 0.122 % 94.379 ± 0.154 % + 23 7.5680 ± 0.1333 -0.00987 ± 0.00172 0.02088 ± 0.00060 4.151 ± 0.118 % 94.411 ± 0.150 % + 24 7.6565 ± 0.1324 -0.01182 ± 0.00170 0.02125 ± 0.00059 4.170 ± 0.115 % 94.396 ± 0.147 % + 25 7.7201 ± 0.1311 -0.01198 ± 0.00166 0.02103 ± 0.00056 4.133 ± 0.112 % 94.362 ± 0.144 % + 26 7.6746 ± 0.1275 -0.01125 ± 0.00161 0.02078 ± 0.00054 4.100 ± 0.109 % 94.383 ± 0.141 % + 27 7.8278 ± 0.1280 -0.01118 ± 0.00157 0.02037 ± 0.00053 4.048 ± 0.106 % 94.388 ± 0.138 % + 28 7.9086 ± 0.1272 -0.01065 ± 0.00152 0.02008 ± 0.00051 4.013 ± 0.104 % 94.428 ± 0.136 % + 29 7.9133 ± 0.1253 -0.00963 ± 0.00150 0.02000 ± 0.00049 4.000 ± 0.101 % 94.475 ± 0.133 % + 30 7.9489 ± 0.1242 -0.01118 ± 0.00149 0.02016 ± 0.00048 4.002 ± 0.098 % 94.435 ± 0.131 % + 31 7.8374 ± 0.1201 -0.01141 ± 0.00146 0.01994 ± 0.00047 3.982 ± 0.095 % 94.463 ± 0.128 % + 32 7.7033 ± 0.1160 -0.01104 ± 0.00143 0.01978 ± 0.00046 3.963 ± 0.093 % 94.483 ± 0.126 % + 33 7.6887 ± 0.1142 -0.01084 ± 0.00142 0.01997 ± 0.00051 3.949 ± 0.091 % 94.484 ± 0.124 % + 34 7.6628 ± 0.1119 -0.01112 ± 0.00138 0.01967 ± 0.00050 3.924 ± 0.089 % 94.503 ± 0.122 % + 35 7.6616 ± 0.1103 -0.01107 ± 0.00136 0.01957 ± 0.00049 3.912 ± 0.087 % 94.526 ± 0.120 % + 36 7.6633 ± 0.1088 -0.01153 ± 0.00133 0.01952 ± 0.00048 3.913 ± 0.086 % 94.558 ± 0.118 % + 37 7.5207 ± 0.1048 -0.01114 ± 0.00130 0.01922 ± 0.00047 3.887 ± 0.084 % 94.616 ± 0.116 % + 38 7.4094 ± 0.1014 -0.01110 ± 0.00128 0.01903 ± 0.00045 3.879 ± 0.082 % 94.660 ± 0.114 % + 39 7.2959 ± 0.0981 -0.01065 ± 0.00126 0.01878 ± 0.00044 3.861 ± 0.081 % 94.674 ± 0.112 % + 40 7.1687 ± 0.0947 -0.01047 ± 0.00123 0.01851 ± 0.00043 3.842 ± 0.079 % 94.709 ± 0.111 % + 41 7.0902 ± 0.0921 -0.01009 ± 0.00121 0.01836 ± 0.00042 3.829 ± 0.078 % 94.714 ± 0.109 % + 42 7.1227 ± 0.0915 -0.00968 ± 0.00120 0.01835 ± 0.00042 3.824 ± 0.077 % 94.682 ± 0.108 % + 43 7.3184 ± 0.0938 -0.01005 ± 0.00119 0.01850 ± 0.00041 3.808 ± 0.076 % 94.633 ± 0.107 % + 44 7.4553 ± 0.0950 -0.01025 ± 0.00117 0.01851 ± 0.00040 3.796 ± 0.074 % 94.604 ± 0.106 % + 45 7.4320 ± 0.0935 -0.00941 ± 0.00117 0.01897 ± 0.00040 3.834 ± 0.073 % 94.545 ± 0.106 % + 46 7.4364 ± 0.0924 -0.00942 ± 0.00116 0.01897 ± 0.00039 3.833 ± 0.072 % 94.568 ± 0.104 % + 47 7.5008 ± 0.0924 -0.00961 ± 0.00115 0.01900 ± 0.00039 3.827 ± 0.071 % 94.576 ± 0.103 % + 48 7.4984 ± 0.0913 -0.00934 ± 0.00113 0.01895 ± 0.00038 3.837 ± 0.071 % 94.561 ± 0.102 % + 49 7.5469 ± 0.0910 -0.00946 ± 0.00112 0.01902 ± 0.00038 3.830 ± 0.070 % 94.546 ± 0.101 % + 50 7.5046 ± 0.0894 -0.00883 ± 0.00111 0.01890 ± 0.00037 3.812 ± 0.069 % 94.581 ± 0.100 % + 51 7.5383 ± 0.0889 -0.00858 ± 0.00109 0.01874 ± 0.00036 3.789 ± 0.068 % 94.599 ± 0.099 % + 52 7.4769 ± 0.0872 -0.00836 ± 0.00108 0.01861 ± 0.00036 3.781 ± 0.067 % 94.614 ± 0.098 % + 53 7.4483 ± 0.0859 -0.00779 ± 0.00107 0.01850 ± 0.00035 3.768 ± 0.066 % 94.631 ± 0.097 % + 54 7.4068 ± 0.0845 -0.00780 ± 0.00106 0.01844 ± 0.00035 3.771 ± 0.067 % 94.635 ± 0.096 % + 55 7.3647 ± 0.0831 -0.00845 ± 0.00105 0.01842 ± 0.00034 3.781 ± 0.065 % 94.643 ± 0.095 % + 56 7.3596 ± 0.0824 -0.00828 ± 0.00104 0.01843 ± 0.00034 3.779 ± 0.064 % 94.636 ± 0.094 % + 57 7.2814 ± 0.0805 -0.00822 ± 0.00103 0.01830 ± 0.00033 3.763 ± 0.064 % 94.667 ± 0.093 % + 58 7.3195 ± 0.0805 -0.00781 ± 0.00103 0.01873 ± 0.00034 3.799 ± 0.063 % 94.607 ± 0.093 % + 59 7.2925 ± 0.0795 -0.00751 ± 0.00102 0.01887 ± 0.00033 3.823 ± 0.063 % 94.589 ± 0.092 % + 60 7.2747 ± 0.0786 -0.00759 ± 0.00101 0.01876 ± 0.00033 3.810 ± 0.062 % 94.620 ± 0.091 % + 61 7.3427 ± 0.0790 -0.00914 ± 0.00102 0.01935 ± 0.00035 3.872 ± 0.063 % 94.556 ± 0.091 % + 62 7.3670 ± 0.0786 -0.00886 ± 0.00101 0.01926 ± 0.00034 3.854 ± 0.063 % 94.568 ± 0.090 % + 63 7.4126 ± 0.0786 -0.00873 ± 0.00100 0.01915 ± 0.00034 3.838 ± 0.062 % 94.569 ± 0.089 % + 64 7.4433 ± 0.0783 -0.00880 ± 0.00099 0.01902 ± 0.00033 3.818 ± 0.061 % 94.601 ± 0.088 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.443251 ± 0.078307 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 99.56% +Mean ln(PPL(Q)/PPL(base)) : -0.008800 ± 0.000988 +Mean PPL(Q)/PPL(base) : 0.991238 ± 0.000980 +Mean PPL(Q)-PPL(base) : -0.065792 ± 0.007455 + +====== KL divergence statistics ====== +Mean KLD: 0.019018 ± 0.000333 +Maximum KLD: 8.796923 +99.9% KLD: 0.874047 +99.0% KLD: 0.187952 +95.0% KLD: 0.067044 +90.0% KLD: 0.039804 +Median KLD: 0.006015 +10.0% KLD: 0.000109 + 5.0% KLD: 0.000028 + 1.0% KLD: -0.000005 + 0.1% KLD: -0.000143 +Minimum KLD: -0.000665 + +====== Token probability statistics ====== +Mean Δp: -0.035 ± 0.015 % +Maximum Δp: 88.388% +99.9% Δp: 30.261% +99.0% Δp: 11.186% +95.0% Δp: 4.432% +90.0% Δp: 2.353% +75.0% Δp: 0.406% +Median Δp: -0.001% +25.0% Δp: -0.495% +10.0% Δp: -2.563% + 5.0% Δp: -4.613% + 1.0% Δp: -11.230% + 0.1% Δp: -28.803% +Minimum Δp: -95.338% +RMS Δp : 3.818 ± 0.061 % +Same top p: 94.601 ± 0.088 % + diff --git a/appendix/measurements/kld_DENSE_attention.txt b/appendix/measurements/kld_DENSE_attention.txt new file mode 100644 index 0000000000000000000000000000000000000000..88515fc34943905962ba61acd73f377c8e250ea4 --- /dev/null +++ b/appendix/measurements/kld_DENSE_attention.txt @@ -0,0 +1,112 @@ +0.09.661.082 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --no-mmap for better performance +0.16.704.000 I +0.16.704.120 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.17.082.046 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.20.971.352 I kl_divergence: 3.89 seconds per pass - ETA 4.13 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.8468 ± 0.4599 -0.00183 ± 0.00730 0.01746 ± 0.00155 3.770 ± 0.305 % 95.601 ± 0.641 % + 2 7.2675 ± 0.4233 -0.00566 ± 0.00523 0.01668 ± 0.00093 3.546 ± 0.219 % 95.161 ± 0.475 % + 3 8.2695 ± 0.4095 0.00857 ± 0.00447 0.01827 ± 0.00187 3.716 ± 0.356 % 94.754 ± 0.403 % + 4 9.3899 ± 0.4303 0.00735 ± 0.00423 0.02218 ± 0.00197 4.221 ± 0.354 % 94.550 ± 0.355 % + 5 8.7220 ± 0.3510 0.00111 ± 0.00367 0.02109 ± 0.00159 4.053 ± 0.299 % 94.643 ± 0.315 % + 6 7.3765 ± 0.2617 -0.00381 ± 0.00329 0.02088 ± 0.00138 4.204 ± 0.266 % 94.575 ± 0.289 % + 7 6.7876 ± 0.2197 -0.00145 ± 0.00316 0.02140 ± 0.00122 4.319 ± 0.230 % 94.693 ± 0.265 % + 8 6.5882 ± 0.1966 -0.00176 ± 0.00287 0.02038 ± 0.00108 4.194 ± 0.208 % 94.807 ± 0.245 % + 9 6.9595 ± 0.1972 -0.00527 ± 0.00272 0.02070 ± 0.00097 4.162 ± 0.190 % 94.493 ± 0.238 % + 10 7.0222 ± 0.1889 -0.00542 ± 0.00254 0.01982 ± 0.00088 4.058 ± 0.176 % 94.575 ± 0.224 % + 11 7.0325 ± 0.1793 -0.00448 ± 0.00246 0.01911 ± 0.00081 3.975 ± 0.165 % 94.704 ± 0.211 % + 12 7.2911 ± 0.1791 -0.00462 ± 0.00231 0.01883 ± 0.00078 3.954 ± 0.163 % 94.762 ± 0.201 % + 13 7.4202 ± 0.1760 -0.00235 ± 0.00224 0.01891 ± 0.00073 3.958 ± 0.152 % 94.759 ± 0.193 % + 14 7.4691 ± 0.1704 -0.00213 ± 0.00212 0.01833 ± 0.00068 3.883 ± 0.144 % 94.777 ± 0.186 % + 15 7.5246 ± 0.1658 -0.00252 ± 0.00205 0.01816 ± 0.00064 3.852 ± 0.137 % 94.858 ± 0.178 % + 16 7.7089 ± 0.1646 -0.00306 ± 0.00197 0.01799 ± 0.00060 3.804 ± 0.130 % 94.831 ± 0.173 % + 17 7.7179 ± 0.1590 -0.00285 ± 0.00188 0.01757 ± 0.00057 3.738 ± 0.125 % 94.871 ± 0.167 % + 18 7.7964 ± 0.1558 -0.00296 ± 0.00181 0.01740 ± 0.00054 3.724 ± 0.120 % 94.922 ± 0.162 % + 19 7.7029 ± 0.1501 -0.00562 ± 0.00179 0.01797 ± 0.00053 3.745 ± 0.115 % 94.804 ± 0.159 % + 20 7.4532 ± 0.1411 -0.00618 ± 0.00179 0.01920 ± 0.00060 3.989 ± 0.124 % 94.751 ± 0.156 % + 21 7.4941 ± 0.1383 -0.00615 ± 0.00174 0.01933 ± 0.00058 4.017 ± 0.124 % 94.703 ± 0.153 % + 22 7.5594 ± 0.1367 -0.00750 ± 0.00170 0.01925 ± 0.00056 4.004 ± 0.120 % 94.695 ± 0.149 % + 23 7.5893 ± 0.1340 -0.00706 ± 0.00165 0.01911 ± 0.00053 3.970 ± 0.116 % 94.683 ± 0.146 % + 24 7.6861 ± 0.1332 -0.00797 ± 0.00162 0.01921 ± 0.00052 3.976 ± 0.112 % 94.681 ± 0.143 % + 25 7.7441 ± 0.1318 -0.00888 ± 0.00159 0.01925 ± 0.00050 3.968 ± 0.109 % 94.608 ± 0.141 % + 26 7.6982 ± 0.1282 -0.00819 ± 0.00155 0.01906 ± 0.00048 3.953 ± 0.107 % 94.631 ± 0.138 % + 27 7.8503 ± 0.1286 -0.00831 ± 0.00151 0.01874 ± 0.00046 3.907 ± 0.104 % 94.642 ± 0.136 % + 28 7.9227 ± 0.1276 -0.00887 ± 0.00147 0.01867 ± 0.00045 3.894 ± 0.102 % 94.634 ± 0.133 % + 29 7.9341 ± 0.1258 -0.00700 ± 0.00146 0.01876 ± 0.00044 3.915 ± 0.099 % 94.607 ± 0.131 % + 30 7.9724 ± 0.1248 -0.00823 ± 0.00145 0.01884 ± 0.00042 3.913 ± 0.096 % 94.591 ± 0.129 % + 31 7.8628 ± 0.1207 -0.00817 ± 0.00141 0.01861 ± 0.00041 3.899 ± 0.094 % 94.636 ± 0.127 % + 32 7.7284 ± 0.1166 -0.00779 ± 0.00139 0.01844 ± 0.00040 3.903 ± 0.092 % 94.688 ± 0.124 % + 33 7.7142 ± 0.1148 -0.00753 ± 0.00137 0.01854 ± 0.00042 3.899 ± 0.090 % 94.656 ± 0.122 % + 34 7.6892 ± 0.1125 -0.00768 ± 0.00134 0.01826 ± 0.00041 3.872 ± 0.088 % 94.690 ± 0.120 % + 35 7.6879 ± 0.1109 -0.00764 ± 0.00131 0.01810 ± 0.00040 3.853 ± 0.087 % 94.685 ± 0.119 % + 36 7.6913 ± 0.1094 -0.00788 ± 0.00129 0.01799 ± 0.00039 3.837 ± 0.085 % 94.719 ± 0.117 % + 37 7.5470 ± 0.1053 -0.00765 ± 0.00126 0.01771 ± 0.00038 3.814 ± 0.083 % 94.780 ± 0.114 % + 38 7.4355 ± 0.1019 -0.00759 ± 0.00123 0.01759 ± 0.00037 3.808 ± 0.081 % 94.817 ± 0.112 % + 39 7.3197 ± 0.0986 -0.00739 ± 0.00121 0.01736 ± 0.00036 3.786 ± 0.080 % 94.839 ± 0.111 % + 40 7.1936 ± 0.0952 -0.00700 ± 0.00119 0.01712 ± 0.00035 3.769 ± 0.078 % 94.868 ± 0.109 % + 41 7.1152 ± 0.0926 -0.00658 ± 0.00116 0.01692 ± 0.00034 3.742 ± 0.077 % 94.874 ± 0.108 % + 42 7.1467 ± 0.0920 -0.00630 ± 0.00115 0.01686 ± 0.00034 3.728 ± 0.076 % 94.861 ± 0.107 % + 43 7.3417 ± 0.0943 -0.00686 ± 0.00114 0.01706 ± 0.00033 3.727 ± 0.075 % 94.803 ± 0.106 % + 44 7.4763 ± 0.0954 -0.00743 ± 0.00113 0.01717 ± 0.00034 3.716 ± 0.074 % 94.784 ± 0.105 % + 45 7.4465 ± 0.0937 -0.00745 ± 0.00112 0.01741 ± 0.00033 3.742 ± 0.072 % 94.745 ± 0.104 % + 46 7.4502 ± 0.0927 -0.00757 ± 0.00111 0.01740 ± 0.00033 3.741 ± 0.071 % 94.760 ± 0.103 % + 47 7.5146 ± 0.0927 -0.00777 ± 0.00110 0.01750 ± 0.00033 3.745 ± 0.070 % 94.748 ± 0.102 % + 48 7.5117 ± 0.0915 -0.00757 ± 0.00108 0.01744 ± 0.00032 3.734 ± 0.069 % 94.734 ± 0.101 % + 49 7.5595 ± 0.0912 -0.00780 ± 0.00108 0.01764 ± 0.00034 3.753 ± 0.069 % 94.691 ± 0.100 % + 50 7.5162 ± 0.0896 -0.00730 ± 0.00107 0.01760 ± 0.00033 3.749 ± 0.068 % 94.714 ± 0.099 % + 51 7.5484 ± 0.0891 -0.00725 ± 0.00105 0.01743 ± 0.00033 3.726 ± 0.067 % 94.746 ± 0.098 % + 52 7.4851 ± 0.0874 -0.00726 ± 0.00104 0.01733 ± 0.00032 3.720 ± 0.066 % 94.765 ± 0.097 % + 53 7.4573 ± 0.0861 -0.00658 ± 0.00103 0.01727 ± 0.00032 3.711 ± 0.066 % 94.777 ± 0.096 % + 54 7.4161 ± 0.0846 -0.00654 ± 0.00102 0.01717 ± 0.00031 3.701 ± 0.065 % 94.779 ± 0.095 % + 55 7.3791 ± 0.0834 -0.00651 ± 0.00101 0.01709 ± 0.00031 3.699 ± 0.064 % 94.785 ± 0.094 % + 56 7.3702 ± 0.0826 -0.00684 ± 0.00100 0.01709 ± 0.00030 3.698 ± 0.063 % 94.779 ± 0.093 % + 57 7.2909 ± 0.0807 -0.00692 ± 0.00099 0.01699 ± 0.00030 3.688 ± 0.062 % 94.802 ± 0.092 % + 58 7.3278 ± 0.0807 -0.00667 ± 0.00099 0.01744 ± 0.00030 3.729 ± 0.062 % 94.740 ± 0.092 % + 59 7.2970 ± 0.0795 -0.00690 ± 0.00098 0.01756 ± 0.00030 3.747 ± 0.061 % 94.726 ± 0.091 % + 60 7.2801 ± 0.0787 -0.00685 ± 0.00097 0.01745 ± 0.00029 3.736 ± 0.060 % 94.761 ± 0.090 % + 61 7.3588 ± 0.0793 -0.00695 ± 0.00098 0.01783 ± 0.00031 3.785 ± 0.063 % 94.715 ± 0.090 % + 62 7.3824 ± 0.0789 -0.00676 ± 0.00097 0.01773 ± 0.00030 3.768 ± 0.062 % 94.712 ± 0.089 % + 63 7.4270 ± 0.0788 -0.00679 ± 0.00095 0.01762 ± 0.00030 3.751 ± 0.061 % 94.718 ± 0.088 % + 64 7.4586 ± 0.0786 -0.00674 ± 0.00094 0.01751 ± 0.00029 3.733 ± 0.061 % 94.743 ± 0.087 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.458614 ± 0.078559 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 99.60% +Mean ln(PPL(Q)/PPL(base)) : -0.006738 ± 0.000944 +Mean PPL(Q)/PPL(base) : 0.993284 ± 0.000938 +Mean PPL(Q)-PPL(base) : -0.050429 ± 0.007104 + +====== KL divergence statistics ====== +Mean KLD: 0.017510 ± 0.000293 +Maximum KLD: 5.325312 +99.9% KLD: 0.723324 +99.0% KLD: 0.182327 +95.0% KLD: 0.061133 +90.0% KLD: 0.036994 +Median KLD: 0.005638 +10.0% KLD: 0.000097 + 5.0% KLD: 0.000024 + 1.0% KLD: -0.000007 + 0.1% KLD: -0.000136 +Minimum KLD: -0.000574 + +====== Token probability statistics ====== +Mean Δp: -0.047 ± 0.015 % +Maximum Δp: 85.629% +99.9% Δp: 30.170% +99.0% Δp: 10.466% +95.0% Δp: 4.281% +90.0% Δp: 2.264% +75.0% Δp: 0.386% +Median Δp: -0.000% +25.0% Δp: -0.483% +10.0% Δp: -2.518% + 5.0% Δp: -4.479% + 1.0% Δp: -10.870% + 0.1% Δp: -28.025% +Minimum Δp: -83.566% +RMS Δp : 3.733 ± 0.061 % +Same top p: 94.743 ± 0.087 % + diff --git a/appendix/measurements/kld_DENSE_output.txt b/appendix/measurements/kld_DENSE_output.txt new file mode 100644 index 0000000000000000000000000000000000000000..39402651e2a0cfe145e6be3ba0fdda8f97e235d3 --- /dev/null +++ b/appendix/measurements/kld_DENSE_output.txt @@ -0,0 +1,112 @@ +0.03.235.962 I cmn init: llama threadpool init, n_threads = 30 +0.03.343.137 I +0.03.343.344 I system_info: n_threads = 30 (n_threads_batch = 30) / 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.03.695.165 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.20.480.218 I kl_divergence: 16.78 seconds per pass - ETA 17.90 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.8644 ± 0.4646 0.00118 ± 0.00729 0.01804 ± 0.00192 4.051 ± 0.471 % 95.210 ± 0.668 % + 2 7.3617 ± 0.4320 0.00723 ± 0.00517 0.01692 ± 0.00108 3.491 ± 0.282 % 95.015 ± 0.481 % + 3 8.2676 ± 0.4101 0.00834 ± 0.00426 0.01826 ± 0.00213 3.636 ± 0.395 % 94.656 ± 0.406 % + 4 9.2105 ± 0.4182 -0.01194 ± 0.00431 0.02473 ± 0.00247 4.421 ± 0.389 % 94.282 ± 0.363 % + 5 8.5944 ± 0.3434 -0.01362 ± 0.00373 0.02363 ± 0.00199 4.270 ± 0.326 % 94.428 ± 0.321 % + 6 7.3091 ± 0.2579 -0.01299 ± 0.00345 0.02345 ± 0.00171 4.508 ± 0.294 % 94.330 ± 0.295 % + 7 6.7285 ± 0.2165 -0.01020 ± 0.00331 0.02393 ± 0.00152 4.659 ± 0.254 % 94.372 ± 0.272 % + 8 6.5368 ± 0.1940 -0.00960 ± 0.00299 0.02264 ± 0.00134 4.490 ± 0.231 % 94.453 ± 0.253 % + 9 6.9172 ± 0.1951 -0.01136 ± 0.00283 0.02292 ± 0.00120 4.391 ± 0.211 % 94.102 ± 0.246 % + 10 6.9858 ± 0.1872 -0.01063 ± 0.00263 0.02194 ± 0.00109 4.262 ± 0.196 % 94.203 ± 0.231 % + 11 6.9922 ± 0.1775 -0.01023 ± 0.00255 0.02123 ± 0.00099 4.188 ± 0.183 % 94.330 ± 0.218 % + 12 7.2603 ± 0.1777 -0.00886 ± 0.00240 0.02062 ± 0.00092 4.109 ± 0.172 % 94.420 ± 0.207 % + 13 7.3928 ± 0.1748 -0.00605 ± 0.00232 0.02080 ± 0.00085 4.121 ± 0.160 % 94.428 ± 0.199 % + 14 7.4386 ± 0.1691 -0.00621 ± 0.00221 0.02019 ± 0.00079 4.046 ± 0.152 % 94.491 ± 0.191 % + 15 7.4926 ± 0.1646 -0.00677 ± 0.00213 0.02018 ± 0.00075 4.039 ± 0.143 % 94.467 ± 0.185 % + 16 7.6708 ± 0.1632 -0.00801 ± 0.00206 0.02004 ± 0.00070 3.986 ± 0.136 % 94.465 ± 0.179 % + 17 7.6764 ± 0.1575 -0.00823 ± 0.00199 0.01977 ± 0.00068 3.921 ± 0.131 % 94.509 ± 0.173 % + 18 7.7535 ± 0.1544 -0.00847 ± 0.00191 0.01950 ± 0.00064 3.890 ± 0.125 % 94.586 ± 0.167 % + 19 7.6659 ± 0.1488 -0.01043 ± 0.00186 0.01972 ± 0.00062 3.888 ± 0.119 % 94.505 ± 0.163 % + 20 7.4224 ± 0.1400 -0.01032 ± 0.00187 0.02115 ± 0.00067 4.180 ± 0.128 % 94.379 ± 0.161 % + 21 7.4654 ± 0.1374 -0.00998 ± 0.00182 0.02122 ± 0.00065 4.197 ± 0.126 % 94.396 ± 0.157 % + 22 7.5372 ± 0.1360 -0.01044 ± 0.00177 0.02108 ± 0.00062 4.192 ± 0.122 % 94.415 ± 0.153 % + 23 7.5674 ± 0.1333 -0.00995 ± 0.00172 0.02081 ± 0.00060 4.143 ± 0.119 % 94.445 ± 0.149 % + 24 7.6559 ± 0.1324 -0.01191 ± 0.00170 0.02118 ± 0.00059 4.161 ± 0.115 % 94.440 ± 0.146 % + 25 7.7195 ± 0.1311 -0.01207 ± 0.00166 0.02096 ± 0.00056 4.124 ± 0.112 % 94.405 ± 0.144 % + 26 7.6739 ± 0.1275 -0.01135 ± 0.00161 0.02071 ± 0.00054 4.090 ± 0.109 % 94.413 ± 0.141 % + 27 7.8272 ± 0.1280 -0.01126 ± 0.00156 0.02029 ± 0.00052 4.038 ± 0.107 % 94.403 ± 0.138 % + 28 7.9079 ± 0.1272 -0.01074 ± 0.00152 0.02000 ± 0.00051 4.003 ± 0.104 % 94.446 ± 0.135 % + 29 7.9125 ± 0.1253 -0.00972 ± 0.00150 0.01992 ± 0.00049 3.990 ± 0.101 % 94.492 ± 0.132 % + 30 7.9481 ± 0.1242 -0.01128 ± 0.00149 0.02009 ± 0.00048 3.993 ± 0.098 % 94.451 ± 0.131 % + 31 7.8366 ± 0.1201 -0.01151 ± 0.00146 0.01987 ± 0.00047 3.972 ± 0.095 % 94.479 ± 0.128 % + 32 7.7026 ± 0.1160 -0.01114 ± 0.00143 0.01971 ± 0.00046 3.953 ± 0.093 % 94.498 ± 0.126 % + 33 7.6880 ± 0.1142 -0.01092 ± 0.00141 0.01990 ± 0.00051 3.939 ± 0.091 % 94.496 ± 0.124 % + 34 7.6620 ± 0.1119 -0.01122 ± 0.00138 0.01960 ± 0.00050 3.914 ± 0.089 % 94.520 ± 0.122 % + 35 7.6608 ± 0.1102 -0.01117 ± 0.00135 0.01949 ± 0.00049 3.902 ± 0.087 % 94.537 ± 0.120 % + 36 7.6626 ± 0.1087 -0.01161 ± 0.00133 0.01945 ± 0.00048 3.903 ± 0.086 % 94.572 ± 0.118 % + 37 7.5200 ± 0.1048 -0.01123 ± 0.00130 0.01914 ± 0.00047 3.877 ± 0.084 % 94.634 ± 0.116 % + 38 7.4087 ± 0.1013 -0.01119 ± 0.00128 0.01896 ± 0.00045 3.869 ± 0.082 % 94.680 ± 0.114 % + 39 7.2951 ± 0.0981 -0.01076 ± 0.00125 0.01871 ± 0.00044 3.850 ± 0.081 % 94.699 ± 0.112 % + 40 7.1678 ± 0.0947 -0.01058 ± 0.00123 0.01843 ± 0.00043 3.831 ± 0.079 % 94.731 ± 0.110 % + 41 7.0895 ± 0.0921 -0.01019 ± 0.00121 0.01828 ± 0.00042 3.818 ± 0.078 % 94.736 ± 0.109 % + 42 7.1220 ± 0.0915 -0.00977 ± 0.00119 0.01828 ± 0.00042 3.813 ± 0.077 % 94.703 ± 0.108 % + 43 7.3177 ± 0.0938 -0.01015 ± 0.00118 0.01842 ± 0.00041 3.798 ± 0.076 % 94.658 ± 0.107 % + 44 7.4545 ± 0.0950 -0.01035 ± 0.00117 0.01843 ± 0.00040 3.786 ± 0.074 % 94.628 ± 0.106 % + 45 7.4313 ± 0.0934 -0.00950 ± 0.00117 0.01889 ± 0.00040 3.823 ± 0.073 % 94.565 ± 0.106 % + 46 7.4358 ± 0.0924 -0.00951 ± 0.00116 0.01889 ± 0.00039 3.822 ± 0.072 % 94.585 ± 0.104 % + 47 7.5003 ± 0.0924 -0.00968 ± 0.00115 0.01892 ± 0.00038 3.816 ± 0.071 % 94.586 ± 0.103 % + 48 7.4978 ± 0.0912 -0.00943 ± 0.00113 0.01887 ± 0.00038 3.827 ± 0.071 % 94.577 ± 0.102 % + 49 7.5463 ± 0.0910 -0.00955 ± 0.00112 0.01895 ± 0.00038 3.821 ± 0.070 % 94.564 ± 0.101 % + 50 7.5040 ± 0.0894 -0.00891 ± 0.00110 0.01883 ± 0.00037 3.802 ± 0.069 % 94.598 ± 0.100 % + 51 7.5377 ± 0.0889 -0.00866 ± 0.00109 0.01866 ± 0.00036 3.779 ± 0.068 % 94.616 ± 0.099 % + 52 7.4763 ± 0.0872 -0.00844 ± 0.00108 0.01853 ± 0.00036 3.771 ± 0.067 % 94.627 ± 0.098 % + 53 7.4476 ± 0.0859 -0.00788 ± 0.00107 0.01842 ± 0.00035 3.758 ± 0.066 % 94.644 ± 0.097 % + 54 7.4060 ± 0.0845 -0.00790 ± 0.00106 0.01836 ± 0.00035 3.762 ± 0.067 % 94.651 ± 0.096 % + 55 7.3638 ± 0.0831 -0.00857 ± 0.00105 0.01834 ± 0.00034 3.772 ± 0.066 % 94.654 ± 0.095 % + 56 7.3588 ± 0.0824 -0.00839 ± 0.00104 0.01836 ± 0.00034 3.770 ± 0.065 % 94.643 ± 0.094 % + 57 7.2806 ± 0.0805 -0.00833 ± 0.00103 0.01823 ± 0.00033 3.754 ± 0.064 % 94.665 ± 0.093 % + 58 7.3187 ± 0.0805 -0.00792 ± 0.00103 0.01866 ± 0.00034 3.790 ± 0.064 % 94.603 ± 0.093 % + 59 7.2917 ± 0.0794 -0.00761 ± 0.00102 0.01879 ± 0.00033 3.813 ± 0.063 % 94.594 ± 0.092 % + 60 7.2739 ± 0.0786 -0.00771 ± 0.00101 0.01869 ± 0.00033 3.800 ± 0.062 % 94.630 ± 0.091 % + 61 7.3419 ± 0.0790 -0.00924 ± 0.00102 0.01927 ± 0.00035 3.863 ± 0.064 % 94.568 ± 0.091 % + 62 7.3662 ± 0.0786 -0.00896 ± 0.00101 0.01919 ± 0.00034 3.844 ± 0.063 % 94.580 ± 0.090 % + 63 7.4118 ± 0.0785 -0.00883 ± 0.00100 0.01907 ± 0.00034 3.828 ± 0.062 % 94.582 ± 0.089 % + 64 7.4424 ± 0.0783 -0.00891 ± 0.00099 0.01894 ± 0.00033 3.808 ± 0.061 % 94.608 ± 0.088 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.442420 ± 0.078298 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 99.56% +Mean ln(PPL(Q)/PPL(base)) : -0.008912 ± 0.000987 +Mean PPL(Q)/PPL(base) : 0.991128 ± 0.000978 +Mean PPL(Q)-PPL(base) : -0.066623 ± 0.007447 + +====== KL divergence statistics ====== +Mean KLD: 0.018941 ± 0.000333 +Maximum KLD: 8.800378 +99.9% KLD: 0.876907 +99.0% KLD: 0.185885 +95.0% KLD: 0.066776 +90.0% KLD: 0.039799 +Median KLD: 0.005923 +10.0% KLD: 0.000108 + 5.0% KLD: 0.000027 + 1.0% KLD: -0.000006 + 0.1% KLD: -0.000140 +Minimum KLD: -0.000650 + +====== Token probability statistics ====== +Mean Δp: -0.031 ± 0.015 % +Maximum Δp: 88.434% +99.9% Δp: 30.351% +99.0% Δp: 11.128% +95.0% Δp: 4.379% +90.0% Δp: 2.336% +75.0% Δp: 0.408% +Median Δp: -0.001% +25.0% Δp: -0.485% +10.0% Δp: -2.522% + 5.0% Δp: -4.595% + 1.0% Δp: -11.180% + 0.1% Δp: -28.533% +Minimum Δp: -95.342% +RMS Δp : 3.808 ± 0.061 % +Same top p: 94.608 ± 0.088 % + diff --git a/appendix/measurements/kld_DENSE_token_embd.txt b/appendix/measurements/kld_DENSE_token_embd.txt new file mode 100644 index 0000000000000000000000000000000000000000..86dadb4e69ff58f81c2b24a537ac3c4bd88f68fc --- /dev/null +++ b/appendix/measurements/kld_DENSE_token_embd.txt @@ -0,0 +1,112 @@ +0.09.672.714 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --no-mmap for better performance +0.16.253.085 I +0.16.253.234 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.16.604.766 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.20.363.884 I kl_divergence: 3.76 seconds per pass - ETA 4.00 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.8764 ± 0.4641 0.00323 ± 0.00876 0.01848 ± 0.00136 3.540 ± 0.288 % 95.112 ± 0.674 % + 2 7.3544 ± 0.4303 0.00623 ± 0.00578 0.01767 ± 0.00083 3.247 ± 0.172 % 94.673 ± 0.497 % + 3 8.2482 ± 0.4081 0.00599 ± 0.00485 0.01922 ± 0.00184 3.652 ± 0.304 % 94.591 ± 0.408 % + 4 9.3084 ± 0.4253 -0.00137 ± 0.00471 0.02605 ± 0.00257 4.383 ± 0.385 % 94.086 ± 0.369 % + 5 8.6834 ± 0.3489 -0.00332 ± 0.00401 0.02405 ± 0.00207 4.188 ± 0.326 % 94.213 ± 0.327 % + 6 7.3818 ± 0.2620 -0.00309 ± 0.00363 0.02376 ± 0.00182 4.447 ± 0.308 % 94.151 ± 0.300 % + 7 6.7711 ± 0.2188 -0.00390 ± 0.00346 0.02411 ± 0.00160 4.597 ± 0.270 % 94.121 ± 0.278 % + 8 6.5775 ± 0.1962 -0.00340 ± 0.00312 0.02291 ± 0.00140 4.450 ± 0.245 % 94.159 ± 0.259 % + 9 6.9610 ± 0.1971 -0.00506 ± 0.00292 0.02298 ± 0.00126 4.351 ± 0.223 % 94.070 ± 0.246 % + 10 7.0167 ± 0.1886 -0.00621 ± 0.00274 0.02266 ± 0.00114 4.336 ± 0.206 % 94.066 ± 0.234 % + 11 7.0216 ± 0.1787 -0.00604 ± 0.00260 0.02173 ± 0.00104 4.235 ± 0.194 % 94.224 ± 0.220 % + 12 7.2814 ± 0.1786 -0.00596 ± 0.00242 0.02092 ± 0.00096 4.130 ± 0.182 % 94.347 ± 0.208 % + 13 7.4049 ± 0.1754 -0.00441 ± 0.00233 0.02083 ± 0.00089 4.128 ± 0.169 % 94.391 ± 0.200 % + 14 7.4451 ± 0.1695 -0.00534 ± 0.00221 0.02015 ± 0.00083 4.035 ± 0.161 % 94.463 ± 0.191 % + 15 7.4964 ± 0.1648 -0.00627 ± 0.00212 0.01995 ± 0.00078 4.027 ± 0.153 % 94.526 ± 0.184 % + 16 7.6950 ± 0.1642 -0.00486 ± 0.00206 0.01989 ± 0.00074 4.001 ± 0.145 % 94.508 ± 0.178 % + 17 7.7036 ± 0.1586 -0.00470 ± 0.00196 0.01951 ± 0.00070 3.941 ± 0.139 % 94.543 ± 0.172 % + 18 7.7764 ± 0.1553 -0.00553 ± 0.00192 0.01940 ± 0.00067 3.935 ± 0.134 % 94.548 ± 0.167 % + 19 7.6903 ± 0.1498 -0.00726 ± 0.00187 0.01945 ± 0.00064 3.922 ± 0.128 % 94.500 ± 0.164 % + 20 7.4468 ± 0.1409 -0.00704 ± 0.00189 0.02118 ± 0.00079 4.201 ± 0.139 % 94.443 ± 0.160 % + 21 7.4848 ± 0.1381 -0.00738 ± 0.00184 0.02128 ± 0.00077 4.227 ± 0.137 % 94.405 ± 0.157 % + 22 7.5589 ± 0.1367 -0.00756 ± 0.00179 0.02111 ± 0.00073 4.217 ± 0.134 % 94.424 ± 0.153 % + 23 7.5895 ± 0.1340 -0.00703 ± 0.00173 0.02084 ± 0.00070 4.162 ± 0.130 % 94.458 ± 0.149 % + 24 7.6824 ± 0.1331 -0.00845 ± 0.00172 0.02107 ± 0.00068 4.165 ± 0.125 % 94.457 ± 0.146 % + 25 7.7425 ± 0.1318 -0.00908 ± 0.00168 0.02110 ± 0.00065 4.142 ± 0.121 % 94.401 ± 0.144 % + 26 7.6911 ± 0.1280 -0.00912 ± 0.00164 0.02094 ± 0.00063 4.129 ± 0.118 % 94.417 ± 0.141 % + 27 7.8435 ± 0.1285 -0.00919 ± 0.00159 0.02057 ± 0.00061 4.076 ± 0.115 % 94.417 ± 0.138 % + 28 7.9221 ± 0.1276 -0.00895 ± 0.00155 0.02037 ± 0.00059 4.051 ± 0.112 % 94.421 ± 0.136 % + 29 7.9290 ± 0.1258 -0.00764 ± 0.00153 0.02037 ± 0.00057 4.051 ± 0.109 % 94.425 ± 0.133 % + 30 7.9699 ± 0.1248 -0.00855 ± 0.00152 0.02059 ± 0.00057 4.056 ± 0.105 % 94.422 ± 0.131 % + 31 7.8618 ± 0.1208 -0.00830 ± 0.00149 0.02031 ± 0.00055 4.025 ± 0.103 % 94.472 ± 0.128 % + 32 7.7236 ± 0.1165 -0.00841 ± 0.00145 0.02012 ± 0.00054 4.006 ± 0.100 % 94.523 ± 0.126 % + 33 7.7092 ± 0.1147 -0.00817 ± 0.00143 0.02031 ± 0.00059 4.014 ± 0.099 % 94.526 ± 0.124 % + 34 7.6843 ± 0.1124 -0.00832 ± 0.00140 0.02006 ± 0.00057 3.994 ± 0.098 % 94.560 ± 0.122 % + 35 7.6829 ± 0.1108 -0.00830 ± 0.00137 0.01997 ± 0.00056 3.975 ± 0.096 % 94.548 ± 0.120 % + 36 7.6945 ± 0.1095 -0.00747 ± 0.00135 0.01981 ± 0.00055 3.954 ± 0.094 % 94.594 ± 0.118 % + 37 7.5497 ± 0.1054 -0.00729 ± 0.00132 0.01951 ± 0.00054 3.931 ± 0.092 % 94.632 ± 0.116 % + 38 7.4379 ± 0.1020 -0.00725 ± 0.00129 0.01931 ± 0.00052 3.922 ± 0.090 % 94.670 ± 0.114 % + 39 7.3224 ± 0.0987 -0.00702 ± 0.00126 0.01908 ± 0.00051 3.902 ± 0.089 % 94.701 ± 0.112 % + 40 7.1945 ± 0.0953 -0.00687 ± 0.00124 0.01882 ± 0.00050 3.882 ± 0.087 % 94.761 ± 0.110 % + 41 7.1158 ± 0.0927 -0.00650 ± 0.00121 0.01860 ± 0.00049 3.858 ± 0.085 % 94.786 ± 0.109 % + 42 7.1464 ± 0.0920 -0.00635 ± 0.00120 0.01854 ± 0.00048 3.841 ± 0.084 % 94.770 ± 0.107 % + 43 7.3437 ± 0.0944 -0.00659 ± 0.00119 0.01866 ± 0.00047 3.839 ± 0.083 % 94.719 ± 0.107 % + 44 7.4800 ± 0.0955 -0.00694 ± 0.00118 0.01866 ± 0.00046 3.817 ± 0.081 % 94.715 ± 0.105 % + 45 7.4544 ± 0.0939 -0.00640 ± 0.00117 0.01913 ± 0.00045 3.865 ± 0.080 % 94.665 ± 0.105 % + 46 7.4623 ± 0.0929 -0.00595 ± 0.00116 0.01917 ± 0.00045 3.886 ± 0.079 % 94.653 ± 0.104 % + 47 7.5262 ± 0.0929 -0.00624 ± 0.00115 0.01930 ± 0.00044 3.887 ± 0.078 % 94.649 ± 0.103 % + 48 7.5232 ± 0.0917 -0.00603 ± 0.00113 0.01920 ± 0.00043 3.873 ± 0.077 % 94.646 ± 0.102 % + 49 7.5703 ± 0.0915 -0.00637 ± 0.00113 0.01933 ± 0.00044 3.873 ± 0.076 % 94.604 ± 0.101 % + 50 7.5248 ± 0.0898 -0.00614 ± 0.00111 0.01924 ± 0.00043 3.861 ± 0.075 % 94.630 ± 0.100 % + 51 7.5554 ± 0.0893 -0.00632 ± 0.00109 0.01912 ± 0.00042 3.845 ± 0.074 % 94.649 ± 0.099 % + 52 7.4910 ± 0.0875 -0.00648 ± 0.00108 0.01897 ± 0.00041 3.833 ± 0.073 % 94.673 ± 0.097 % + 53 7.4610 ± 0.0862 -0.00608 ± 0.00107 0.01883 ± 0.00041 3.816 ± 0.072 % 94.685 ± 0.096 % + 54 7.4193 ± 0.0847 -0.00610 ± 0.00105 0.01877 ± 0.00040 3.822 ± 0.072 % 94.694 ± 0.095 % + 55 7.3775 ± 0.0834 -0.00671 ± 0.00105 0.01874 ± 0.00040 3.824 ± 0.071 % 94.709 ± 0.094 % + 56 7.3709 ± 0.0826 -0.00675 ± 0.00104 0.01874 ± 0.00039 3.825 ± 0.070 % 94.706 ± 0.094 % + 57 7.2918 ± 0.0807 -0.00680 ± 0.00103 0.01865 ± 0.00038 3.817 ± 0.069 % 94.732 ± 0.093 % + 58 7.3267 ± 0.0806 -0.00682 ± 0.00103 0.01898 ± 0.00038 3.849 ± 0.068 % 94.688 ± 0.092 % + 59 7.2993 ± 0.0796 -0.00658 ± 0.00103 0.01919 ± 0.00038 3.903 ± 0.069 % 94.673 ± 0.091 % + 60 7.2804 ± 0.0787 -0.00680 ± 0.00101 0.01907 ± 0.00037 3.895 ± 0.068 % 94.684 ± 0.091 % + 61 7.3492 ± 0.0791 -0.00826 ± 0.00102 0.01954 ± 0.00039 3.928 ± 0.068 % 94.646 ± 0.090 % + 62 7.3725 ± 0.0787 -0.00810 ± 0.00101 0.01945 ± 0.00038 3.912 ± 0.067 % 94.649 ± 0.089 % + 63 7.4168 ± 0.0786 -0.00816 ± 0.00100 0.01933 ± 0.00038 3.897 ± 0.066 % 94.641 ± 0.089 % + 64 7.4477 ± 0.0784 -0.00820 ± 0.00099 0.01922 ± 0.00037 3.881 ± 0.065 % 94.662 ± 0.088 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.447739 ± 0.078390 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 99.56% +Mean ln(PPL(Q)/PPL(base)) : -0.008198 ± 0.000986 +Mean PPL(Q)/PPL(base) : 0.991836 ± 0.000978 +Mean PPL(Q)-PPL(base) : -0.061304 ± 0.007430 + +====== KL divergence statistics ====== +Mean KLD: 0.019217 ± 0.000370 +Maximum KLD: 9.123656 +99.9% KLD: 0.882531 +99.0% KLD: 0.187328 +95.0% KLD: 0.065879 +90.0% KLD: 0.039818 +Median KLD: 0.006113 +10.0% KLD: 0.000106 + 5.0% KLD: 0.000028 + 1.0% KLD: -0.000006 + 0.1% KLD: -0.000140 +Minimum KLD: -0.000965 + +====== Token probability statistics ====== +Mean Δp: -0.022 ± 0.015 % +Maximum Δp: 96.119% +99.9% Δp: 30.920% +99.0% Δp: 11.422% +95.0% Δp: 4.564% +90.0% Δp: 2.436% +75.0% Δp: 0.407% +Median Δp: -0.001% +25.0% Δp: -0.492% +10.0% Δp: -2.569% + 5.0% Δp: -4.624% + 1.0% Δp: -11.075% + 0.1% Δp: -28.904% +Minimum Δp: -95.551% +RMS Δp : 3.881 ± 0.065 % +Same top p: 94.662 ± 0.088 % + diff --git a/appendix/measurements/kld_E.txt b/appendix/measurements/kld_E.txt new file mode 100644 index 0000000000000000000000000000000000000000..8dcc06a2eb36b654f541a178c337f4394913bc87 --- /dev/null +++ b/appendix/measurements/kld_E.txt @@ -0,0 +1,112 @@ +0.17.339.347 I cmn init: llama threadpool init, n_threads = 30 +0.17.459.928 I +0.17.460.135 I system_info: n_threads = 30 (n_threads_batch = 30) / 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.17.788.451 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.36.252.973 I kl_divergence: 18.46 seconds per pass - ETA 19.68 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.8952 ± 0.4664 0.00641 ± 0.01112 0.03817 ± 0.00334 5.677 ± 0.494 % 93.353 ± 0.779 % + 2 7.2979 ± 0.4250 -0.00149 ± 0.00814 0.03742 ± 0.00215 5.209 ± 0.333 % 92.229 ± 0.592 % + 3 8.2806 ± 0.4095 0.00991 ± 0.00631 0.03540 ± 0.00242 5.043 ± 0.345 % 92.506 ± 0.475 % + 4 9.3546 ± 0.4264 0.00358 ± 0.00583 0.04031 ± 0.00275 5.381 ± 0.298 % 92.424 ± 0.414 % + 5 8.7475 ± 0.3511 0.00403 ± 0.00513 0.03877 ± 0.00224 5.276 ± 0.252 % 92.512 ± 0.368 % + 6 7.4241 ± 0.2633 0.00263 ± 0.00459 0.03778 ± 0.00192 5.374 ± 0.222 % 92.473 ± 0.337 % + 7 6.7666 ± 0.2182 -0.00456 ± 0.00447 0.03869 ± 0.00175 5.666 ± 0.211 % 92.543 ± 0.310 % + 8 6.5838 ± 0.1960 -0.00243 ± 0.00404 0.03705 ± 0.00154 5.537 ± 0.193 % 92.608 ± 0.289 % + 9 6.9705 ± 0.1971 -0.00368 ± 0.00379 0.03755 ± 0.00140 5.526 ± 0.175 % 92.527 ± 0.274 % + 10 7.0337 ± 0.1888 -0.00379 ± 0.00351 0.03598 ± 0.00127 5.397 ± 0.164 % 92.581 ± 0.259 % + 11 7.0423 ± 0.1791 -0.00309 ± 0.00333 0.03487 ± 0.00117 5.292 ± 0.153 % 92.775 ± 0.244 % + 12 7.3222 ± 0.1797 -0.00037 ± 0.00314 0.03401 ± 0.00108 5.198 ± 0.144 % 92.799 ± 0.233 % + 13 7.4183 ± 0.1754 -0.00260 ± 0.00300 0.03370 ± 0.00101 5.180 ± 0.136 % 92.879 ± 0.223 % + 14 7.4640 ± 0.1696 -0.00280 ± 0.00284 0.03271 ± 0.00094 5.072 ± 0.129 % 92.934 ± 0.214 % + 15 7.5079 ± 0.1648 -0.00474 ± 0.00272 0.03264 ± 0.00089 5.069 ± 0.124 % 92.949 ± 0.207 % + 16 7.6843 ± 0.1634 -0.00625 ± 0.00264 0.03265 ± 0.00085 5.056 ± 0.124 % 92.980 ± 0.200 % + 17 7.6965 ± 0.1579 -0.00562 ± 0.00252 0.03189 ± 0.00080 4.976 ± 0.119 % 92.979 ± 0.194 % + 18 7.7709 ± 0.1546 -0.00624 ± 0.00243 0.03163 ± 0.00077 4.961 ± 0.117 % 92.973 ± 0.188 % + 19 7.6803 ± 0.1490 -0.00855 ± 0.00237 0.03172 ± 0.00075 4.935 ± 0.112 % 92.936 ± 0.184 % + 20 7.4365 ± 0.1401 -0.00843 ± 0.00240 0.03477 ± 0.00087 5.330 ± 0.118 % 92.717 ± 0.182 % + 21 7.4832 ± 0.1377 -0.00760 ± 0.00233 0.03518 ± 0.00085 5.364 ± 0.116 % 92.669 ± 0.178 % + 22 7.5511 ± 0.1361 -0.00860 ± 0.00227 0.03486 ± 0.00082 5.318 ± 0.112 % 92.717 ± 0.173 % + 23 7.5848 ± 0.1335 -0.00766 ± 0.00221 0.03452 ± 0.00078 5.265 ± 0.109 % 92.775 ± 0.169 % + 24 7.6835 ± 0.1329 -0.00830 ± 0.00215 0.03447 ± 0.00076 5.261 ± 0.106 % 92.787 ± 0.165 % + 25 7.7431 ± 0.1315 -0.00901 ± 0.00211 0.03452 ± 0.00074 5.244 ± 0.105 % 92.704 ± 0.163 % + 26 7.6940 ± 0.1278 -0.00874 ± 0.00206 0.03429 ± 0.00071 5.224 ± 0.102 % 92.729 ± 0.159 % + 27 7.8558 ± 0.1285 -0.00762 ± 0.00201 0.03379 ± 0.00069 5.162 ± 0.100 % 92.708 ± 0.156 % + 28 7.9391 ± 0.1278 -0.00680 ± 0.00196 0.03339 ± 0.00067 5.130 ± 0.098 % 92.711 ± 0.154 % + 29 7.9418 ± 0.1258 -0.00603 ± 0.00193 0.03344 ± 0.00065 5.149 ± 0.095 % 92.753 ± 0.151 % + 30 7.9775 ± 0.1247 -0.00759 ± 0.00192 0.03395 ± 0.00067 5.172 ± 0.093 % 92.711 ± 0.148 % + 31 7.8717 ± 0.1207 -0.00704 ± 0.00188 0.03363 ± 0.00066 5.143 ± 0.091 % 92.747 ± 0.146 % + 32 7.7376 ± 0.1166 -0.00660 ± 0.00185 0.03364 ± 0.00064 5.182 ± 0.091 % 92.794 ± 0.143 % + 33 7.7305 ± 0.1150 -0.00541 ± 0.00183 0.03386 ± 0.00068 5.188 ± 0.090 % 92.772 ± 0.141 % + 34 7.7047 ± 0.1126 -0.00566 ± 0.00179 0.03362 ± 0.00066 5.181 ± 0.089 % 92.775 ± 0.139 % + 35 7.6973 ± 0.1109 -0.00641 ± 0.00176 0.03343 ± 0.00065 5.158 ± 0.087 % 92.789 ± 0.137 % + 36 7.7052 ± 0.1095 -0.00607 ± 0.00173 0.03314 ± 0.00063 5.125 ± 0.086 % 92.842 ± 0.134 % + 37 7.5619 ± 0.1055 -0.00567 ± 0.00169 0.03272 ± 0.00062 5.101 ± 0.085 % 92.898 ± 0.132 % + 38 7.4490 ± 0.1020 -0.00577 ± 0.00166 0.03252 ± 0.00060 5.097 ± 0.084 % 92.946 ± 0.130 % + 39 7.3319 ± 0.0987 -0.00573 ± 0.00163 0.03224 ± 0.00059 5.086 ± 0.082 % 92.967 ± 0.128 % + 40 7.2043 ± 0.0953 -0.00552 ± 0.00160 0.03189 ± 0.00058 5.075 ± 0.081 % 93.011 ± 0.126 % + 41 7.1250 ± 0.0927 -0.00520 ± 0.00157 0.03163 ± 0.00056 5.066 ± 0.080 % 93.041 ± 0.124 % + 42 7.1564 ± 0.0921 -0.00495 ± 0.00155 0.03157 ± 0.00055 5.053 ± 0.078 % 93.046 ± 0.123 % + 43 7.3516 ± 0.0943 -0.00552 ± 0.00154 0.03159 ± 0.00054 5.042 ± 0.077 % 92.996 ± 0.122 % + 44 7.4859 ± 0.0954 -0.00615 ± 0.00152 0.03155 ± 0.00053 5.014 ± 0.076 % 92.977 ± 0.120 % + 45 7.4636 ± 0.0939 -0.00517 ± 0.00152 0.03248 ± 0.00053 5.101 ± 0.075 % 92.895 ± 0.120 % + 46 7.4688 ± 0.0929 -0.00507 ± 0.00150 0.03261 ± 0.00052 5.101 ± 0.074 % 92.879 ± 0.119 % + 47 7.5332 ± 0.0929 -0.00530 ± 0.00149 0.03276 ± 0.00052 5.111 ± 0.074 % 92.868 ± 0.117 % + 48 7.5313 ± 0.0917 -0.00496 ± 0.00147 0.03258 ± 0.00051 5.090 ± 0.073 % 92.852 ± 0.116 % + 49 7.5790 ± 0.0915 -0.00522 ± 0.00145 0.03266 ± 0.00051 5.100 ± 0.072 % 92.792 ± 0.116 % + 50 7.5357 ± 0.0899 -0.00470 ± 0.00144 0.03253 ± 0.00050 5.084 ± 0.071 % 92.807 ± 0.114 % + 51 7.5682 ± 0.0894 -0.00462 ± 0.00141 0.03229 ± 0.00049 5.058 ± 0.071 % 92.828 ± 0.113 % + 52 7.5035 ± 0.0876 -0.00481 ± 0.00140 0.03223 ± 0.00049 5.058 ± 0.070 % 92.845 ± 0.112 % + 53 7.4745 ± 0.0863 -0.00427 ± 0.00138 0.03201 ± 0.00048 5.047 ± 0.069 % 92.860 ± 0.111 % + 54 7.4300 ± 0.0848 -0.00466 ± 0.00136 0.03185 ± 0.00048 5.036 ± 0.068 % 92.851 ± 0.110 % + 55 7.3859 ± 0.0834 -0.00558 ± 0.00135 0.03180 ± 0.00047 5.036 ± 0.067 % 92.843 ± 0.109 % + 56 7.3764 ± 0.0826 -0.00600 ± 0.00134 0.03177 ± 0.00046 5.033 ± 0.067 % 92.840 ± 0.108 % + 57 7.2978 ± 0.0807 -0.00598 ± 0.00133 0.03163 ± 0.00046 5.019 ± 0.066 % 92.861 ± 0.107 % + 58 7.3384 ± 0.0807 -0.00522 ± 0.00133 0.03221 ± 0.00047 5.055 ± 0.065 % 92.824 ± 0.106 % + 59 7.3125 ± 0.0797 -0.00476 ± 0.00132 0.03244 ± 0.00046 5.077 ± 0.065 % 92.831 ± 0.105 % + 60 7.2948 ± 0.0788 -0.00484 ± 0.00131 0.03228 ± 0.00046 5.064 ± 0.064 % 92.846 ± 0.104 % + 61 7.3668 ± 0.0793 -0.00585 ± 0.00131 0.03291 ± 0.00048 5.118 ± 0.065 % 92.800 ± 0.103 % + 62 7.3895 ± 0.0789 -0.00581 ± 0.00130 0.03277 ± 0.00047 5.095 ± 0.065 % 92.793 ± 0.103 % + 63 7.4348 ± 0.0788 -0.00573 ± 0.00128 0.03254 ± 0.00046 5.072 ± 0.064 % 92.801 ± 0.102 % + 64 7.4663 ± 0.0786 -0.00570 ± 0.00127 0.03241 ± 0.00046 5.053 ± 0.063 % 92.824 ± 0.101 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.466348 ± 0.078584 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 99.28% +Mean ln(PPL(Q)/PPL(base)) : -0.005702 ± 0.001266 +Mean PPL(Q)/PPL(base) : 0.994314 ± 0.001259 +Mean PPL(Q)-PPL(base) : -0.042695 ± 0.009509 + +====== KL divergence statistics ====== +Mean KLD: 0.032407 ± 0.000458 +Maximum KLD: 8.724359 +99.9% KLD: 1.273374 +99.0% KLD: 0.348172 +95.0% KLD: 0.114034 +90.0% KLD: 0.068035 +Median KLD: 0.011113 +10.0% KLD: 0.000177 + 5.0% KLD: 0.000051 + 1.0% KLD: -0.000001 + 0.1% KLD: -0.000145 +Minimum KLD: -0.000786 + +====== Token probability statistics ====== +Mean Δp: -0.082 ± 0.020 % +Maximum Δp: 88.419% +99.9% Δp: 41.174% +99.0% Δp: 14.988% +95.0% Δp: 5.948% +90.0% Δp: 3.158% +75.0% Δp: 0.522% +Median Δp: -0.002% +25.0% Δp: -0.701% +10.0% Δp: -3.491% + 5.0% Δp: -6.205% + 1.0% Δp: -15.510% + 0.1% Δp: -40.231% +Minimum Δp: -79.903% +RMS Δp : 5.053 ± 0.063 % +Same top p: 92.824 ± 0.101 % + diff --git a/appendix/measurements/kld_FAMEVEN-19G-ICE.txt b/appendix/measurements/kld_FAMEVEN-19G-ICE.txt new file mode 100644 index 0000000000000000000000000000000000000000..9eb7fcd03f4740efeed6d1009f1bf7476a9c9c03 --- /dev/null +++ b/appendix/measurements/kld_FAMEVEN-19G-ICE.txt @@ -0,0 +1,111 @@ +0.04.991.941 I +0.04.992.079 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.05.338.579 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.07.271.162 I kl_divergence: 1.93 seconds per pass - ETA 2.05 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.7950 ± 0.4493 -0.01073 ± 0.01896 0.09316 ± 0.00866 8.888 ± 0.638 % 89.247 ± 0.969 % + 2 7.3071 ± 0.4210 -0.00022 ± 0.01192 0.07817 ± 0.00477 7.722 ± 0.437 % 89.345 ± 0.682 % + 3 8.3139 ± 0.4107 0.01392 ± 0.00936 0.07181 ± 0.00345 7.195 ± 0.345 % 89.997 ± 0.542 % + 4 9.5243 ± 0.4368 0.02156 ± 0.00830 0.07863 ± 0.00371 7.839 ± 0.347 % 89.663 ± 0.476 % + 5 8.8988 ± 0.3592 0.02117 ± 0.00730 0.07634 ± 0.00310 7.645 ± 0.299 % 89.580 ± 0.427 % + 6 7.5311 ± 0.2682 0.01693 ± 0.00655 0.07475 ± 0.00280 7.687 ± 0.269 % 89.589 ± 0.390 % + 7 6.8595 ± 0.2220 0.00907 ± 0.00618 0.07432 ± 0.00253 7.785 ± 0.244 % 89.946 ± 0.355 % + 8 6.6461 ± 0.1981 0.00699 ± 0.00567 0.07279 ± 0.00227 7.750 ± 0.226 % 90.017 ± 0.331 % + 9 7.0477 ± 0.1998 0.00733 ± 0.00532 0.07421 ± 0.00208 7.755 ± 0.209 % 89.378 ± 0.321 % + 10 7.1171 ± 0.1917 0.00800 ± 0.00495 0.07126 ± 0.00192 7.562 ± 0.195 % 89.560 ± 0.302 % + 11 7.1094 ± 0.1812 0.00640 ± 0.00463 0.06922 ± 0.00176 7.413 ± 0.183 % 89.674 ± 0.287 % + 12 7.3597 ± 0.1805 0.00473 ± 0.00433 0.06679 ± 0.00162 7.261 ± 0.173 % 89.777 ± 0.273 % + 13 7.4888 ± 0.1773 0.00686 ± 0.00418 0.06626 ± 0.00152 7.242 ± 0.164 % 89.841 ± 0.262 % + 14 7.5315 ± 0.1715 0.00620 ± 0.00398 0.06466 ± 0.00143 7.125 ± 0.156 % 89.953 ± 0.251 % + 15 7.5688 ± 0.1664 0.00335 ± 0.00383 0.06427 ± 0.00135 7.078 ± 0.150 % 89.958 ± 0.243 % + 16 7.7331 ± 0.1645 0.00008 ± 0.00371 0.06454 ± 0.00129 7.013 ± 0.144 % 89.938 ± 0.235 % + 17 7.7479 ± 0.1591 0.00103 ± 0.00355 0.06339 ± 0.00123 6.935 ± 0.138 % 90.012 ± 0.227 % + 18 7.8233 ± 0.1558 0.00049 ± 0.00344 0.06311 ± 0.00119 6.907 ± 0.135 % 90.002 ± 0.221 % + 19 7.7307 ± 0.1502 -0.00202 ± 0.00333 0.06264 ± 0.00115 6.860 ± 0.130 % 90.040 ± 0.215 % + 20 7.5009 ± 0.1415 0.00019 ± 0.00338 0.06877 ± 0.00134 7.400 ± 0.136 % 89.785 ± 0.212 % + 21 7.5511 ± 0.1391 0.00143 ± 0.00333 0.06968 ± 0.00134 7.452 ± 0.135 % 89.685 ± 0.208 % + 22 7.6096 ± 0.1373 -0.00088 ± 0.00325 0.06916 ± 0.00128 7.402 ± 0.131 % 89.732 ± 0.202 % + 23 7.6365 ± 0.1346 -0.00086 ± 0.00316 0.06859 ± 0.00124 7.325 ± 0.127 % 89.740 ± 0.198 % + 24 7.7175 ± 0.1335 -0.00390 ± 0.00310 0.06897 ± 0.00121 7.344 ± 0.125 % 89.724 ± 0.194 % + 25 7.7694 ± 0.1319 -0.00562 ± 0.00303 0.06845 ± 0.00117 7.280 ± 0.122 % 89.728 ± 0.190 % + 26 7.7163 ± 0.1281 -0.00583 ± 0.00296 0.06821 ± 0.00114 7.270 ± 0.119 % 89.781 ± 0.186 % + 27 7.8726 ± 0.1287 -0.00548 ± 0.00289 0.06742 ± 0.00112 7.220 ± 0.118 % 89.783 ± 0.182 % + 28 7.9460 ± 0.1277 -0.00594 ± 0.00281 0.06670 ± 0.00108 7.170 ± 0.115 % 89.795 ± 0.179 % + 29 7.9431 ± 0.1255 -0.00587 ± 0.00278 0.06758 ± 0.00109 7.261 ± 0.114 % 89.736 ± 0.176 % + 30 7.9517 ± 0.1238 -0.01082 ± 0.00279 0.06873 ± 0.00108 7.310 ± 0.113 % 89.697 ± 0.174 % + 31 7.8467 ± 0.1199 -0.01022 ± 0.00273 0.06838 ± 0.00106 7.284 ± 0.111 % 89.736 ± 0.170 % + 32 7.7132 ± 0.1158 -0.00976 ± 0.00269 0.06835 ± 0.00105 7.304 ± 0.109 % 89.834 ± 0.167 % + 33 7.7086 ± 0.1143 -0.00825 ± 0.00265 0.06863 ± 0.00106 7.301 ± 0.106 % 89.804 ± 0.165 % + 34 7.6912 ± 0.1121 -0.00742 ± 0.00259 0.06802 ± 0.00104 7.269 ± 0.104 % 89.834 ± 0.162 % + 35 7.6850 ± 0.1104 -0.00801 ± 0.00255 0.06777 ± 0.00102 7.252 ± 0.102 % 89.865 ± 0.159 % + 36 7.6919 ± 0.1090 -0.00780 ± 0.00250 0.06723 ± 0.00100 7.238 ± 0.101 % 89.951 ± 0.157 % + 37 7.5482 ± 0.1050 -0.00749 ± 0.00245 0.06646 ± 0.00097 7.210 ± 0.100 % 90.027 ± 0.154 % + 38 7.4379 ± 0.1016 -0.00726 ± 0.00240 0.06622 ± 0.00095 7.217 ± 0.099 % 90.022 ± 0.152 % + 39 7.3246 ± 0.0984 -0.00672 ± 0.00236 0.06590 ± 0.00094 7.204 ± 0.097 % 90.029 ± 0.150 % + 40 7.1972 ± 0.0949 -0.00649 ± 0.00231 0.06515 ± 0.00092 7.184 ± 0.095 % 90.108 ± 0.148 % + 41 7.1200 ± 0.0924 -0.00590 ± 0.00228 0.06479 ± 0.00091 7.175 ± 0.095 % 90.115 ± 0.146 % + 42 7.1476 ± 0.0917 -0.00618 ± 0.00224 0.06458 ± 0.00090 7.147 ± 0.094 % 90.127 ± 0.144 % + 43 7.3445 ± 0.0940 -0.00648 ± 0.00221 0.06430 ± 0.00088 7.114 ± 0.092 % 90.116 ± 0.142 % + 44 7.4848 ± 0.0952 -0.00629 ± 0.00218 0.06407 ± 0.00086 7.079 ± 0.091 % 90.094 ± 0.141 % + 45 7.4618 ± 0.0937 -0.00540 ± 0.00218 0.06589 ± 0.00087 7.173 ± 0.090 % 89.995 ± 0.140 % + 46 7.4671 ± 0.0926 -0.00530 ± 0.00216 0.06619 ± 0.00086 7.176 ± 0.090 % 89.983 ± 0.138 % + 47 7.5312 ± 0.0927 -0.00557 ± 0.00213 0.06623 ± 0.00086 7.172 ± 0.089 % 89.959 ± 0.137 % + 48 7.5263 ± 0.0915 -0.00562 ± 0.00210 0.06603 ± 0.00085 7.172 ± 0.089 % 89.962 ± 0.136 % + 49 7.5728 ± 0.0912 -0.00604 ± 0.00208 0.06617 ± 0.00085 7.151 ± 0.087 % 89.916 ± 0.134 % + 50 7.5261 ± 0.0895 -0.00598 ± 0.00205 0.06577 ± 0.00084 7.126 ± 0.086 % 89.973 ± 0.133 % + 51 7.5554 ± 0.0890 -0.00632 ± 0.00202 0.06542 ± 0.00083 7.101 ± 0.086 % 89.970 ± 0.132 % + 52 7.4941 ± 0.0873 -0.00605 ± 0.00200 0.06508 ± 0.00082 7.092 ± 0.084 % 89.997 ± 0.130 % + 53 7.4668 ± 0.0860 -0.00531 ± 0.00197 0.06466 ± 0.00080 7.068 ± 0.083 % 90.002 ± 0.129 % + 54 7.4267 ± 0.0846 -0.00511 ± 0.00195 0.06416 ± 0.00079 7.036 ± 0.082 % 90.009 ± 0.128 % + 55 7.3815 ± 0.0832 -0.00618 ± 0.00193 0.06400 ± 0.00078 7.022 ± 0.081 % 90.006 ± 0.126 % + 56 7.3657 ± 0.0823 -0.00745 ± 0.00192 0.06381 ± 0.00077 7.013 ± 0.080 % 90.007 ± 0.125 % + 57 7.2855 ± 0.0804 -0.00766 ± 0.00189 0.06345 ± 0.00076 6.988 ± 0.079 % 90.046 ± 0.124 % + 58 7.3238 ± 0.0804 -0.00722 ± 0.00189 0.06400 ± 0.00075 7.007 ± 0.078 % 90.018 ± 0.123 % + 59 7.2974 ± 0.0793 -0.00684 ± 0.00187 0.06435 ± 0.00075 7.029 ± 0.077 % 89.986 ± 0.122 % + 60 7.2841 ± 0.0785 -0.00630 ± 0.00185 0.06411 ± 0.00074 7.023 ± 0.077 % 90.007 ± 0.121 % + 61 7.3678 ± 0.0792 -0.00572 ± 0.00185 0.06483 ± 0.00076 7.063 ± 0.077 % 89.964 ± 0.120 % + 62 7.3912 ± 0.0788 -0.00557 ± 0.00183 0.06465 ± 0.00075 7.046 ± 0.076 % 89.955 ± 0.119 % + 63 7.4318 ± 0.0786 -0.00614 ± 0.00181 0.06434 ± 0.00074 7.025 ± 0.075 % 89.942 ± 0.118 % + 64 7.4662 ± 0.0784 -0.00572 ± 0.00180 0.06421 ± 0.00073 7.014 ± 0.075 % 89.953 ± 0.117 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.466247 ± 0.078416 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 98.55% +Mean ln(PPL(Q)/PPL(base)) : -0.005716 ± 0.001798 +Mean PPL(Q)/PPL(base) : 0.994301 ± 0.001787 +Mean PPL(Q)-PPL(base) : -0.042796 ± 0.013484 + +====== KL divergence statistics ====== +Mean KLD: 0.064213 ± 0.000727 +Maximum KLD: 9.456017 +99.9% KLD: 2.305098 +99.0% KLD: 0.692351 +95.0% KLD: 0.233270 +90.0% KLD: 0.138435 +Median KLD: 0.022855 +10.0% KLD: 0.000336 + 5.0% KLD: 0.000092 + 1.0% KLD: 0.000004 + 0.1% KLD: -0.000162 +Minimum KLD: -0.000848 + +====== Token probability statistics ====== +Mean Δp: -0.228 ± 0.027 % +Maximum Δp: 95.687% +99.9% Δp: 48.455% +99.0% Δp: 21.099% +95.0% Δp: 8.343% +90.0% Δp: 4.527% +75.0% Δp: 0.745% +Median Δp: -0.002% +25.0% Δp: -1.012% +10.0% Δp: -5.230% + 5.0% Δp: -9.215% + 1.0% Δp: -23.212% + 0.1% Δp: -57.430% +Minimum Δp: -95.678% +RMS Δp : 7.014 ± 0.075 % +Same top p: 89.953 ± 0.117 % + diff --git a/appendix/measurements/kld_FAMEVEN-21G-ICE.txt b/appendix/measurements/kld_FAMEVEN-21G-ICE.txt new file mode 100644 index 0000000000000000000000000000000000000000..43f153643ded625d16b9d5b75382b1d2ee2d613c --- /dev/null +++ b/appendix/measurements/kld_FAMEVEN-21G-ICE.txt @@ -0,0 +1,111 @@ +0.05.623.084 I +0.05.623.222 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.05.938.775 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.07.812.563 I kl_divergence: 1.87 seconds per pass - ETA 1.98 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.8826 ± 0.4588 0.00428 ± 0.01447 0.06558 ± 0.00467 7.765 ± 0.503 % 91.300 ± 0.882 % + 2 7.3215 ± 0.4256 0.00175 ± 0.00961 0.05427 ± 0.00273 6.475 ± 0.329 % 90.860 ± 0.637 % + 3 8.2585 ± 0.4073 0.00724 ± 0.00774 0.05189 ± 0.00310 6.362 ± 0.364 % 90.779 ± 0.522 % + 4 9.2340 ± 0.4181 -0.00940 ± 0.00689 0.05722 ± 0.00323 6.552 ± 0.314 % 90.494 ± 0.459 % + 5 8.6545 ± 0.3457 -0.00665 ± 0.00602 0.05473 ± 0.00267 6.382 ± 0.274 % 90.694 ± 0.406 % + 6 7.3530 ± 0.2593 -0.00700 ± 0.00534 0.05358 ± 0.00233 6.497 ± 0.242 % 90.665 ± 0.371 % + 7 6.7665 ± 0.2178 -0.00458 ± 0.00498 0.05260 ± 0.00207 6.543 ± 0.218 % 90.923 ± 0.340 % + 8 6.5844 ± 0.1956 -0.00234 ± 0.00455 0.05111 ± 0.00184 6.438 ± 0.197 % 91.019 ± 0.316 % + 9 6.9223 ± 0.1948 -0.01063 ± 0.00441 0.05256 ± 0.00168 6.451 ± 0.182 % 90.616 ± 0.304 % + 10 6.9858 ± 0.1867 -0.01062 ± 0.00409 0.05104 ± 0.00153 6.300 ± 0.169 % 90.616 ± 0.288 % + 11 6.9964 ± 0.1771 -0.00963 ± 0.00385 0.04921 ± 0.00140 6.167 ± 0.159 % 90.829 ± 0.272 % + 12 7.2540 ± 0.1770 -0.00974 ± 0.00360 0.04766 ± 0.00131 6.078 ± 0.154 % 90.982 ± 0.259 % + 13 7.3378 ± 0.1724 -0.01352 ± 0.00347 0.04744 ± 0.00123 6.076 ± 0.147 % 91.067 ± 0.247 % + 14 7.3871 ± 0.1670 -0.01316 ± 0.00330 0.04603 ± 0.00115 5.961 ± 0.140 % 91.133 ± 0.238 % + 15 7.4326 ± 0.1622 -0.01482 ± 0.00317 0.04578 ± 0.00109 5.947 ± 0.135 % 91.183 ± 0.229 % + 16 7.6135 ± 0.1611 -0.01551 ± 0.00306 0.04538 ± 0.00103 5.898 ± 0.129 % 91.239 ± 0.221 % + 17 7.6265 ± 0.1558 -0.01476 ± 0.00292 0.04452 ± 0.00099 5.791 ± 0.124 % 91.289 ± 0.214 % + 18 7.7109 ± 0.1529 -0.01399 ± 0.00282 0.04422 ± 0.00095 5.756 ± 0.119 % 91.295 ± 0.208 % + 19 7.6209 ± 0.1474 -0.01632 ± 0.00275 0.04417 ± 0.00091 5.726 ± 0.115 % 91.331 ± 0.202 % + 20 7.3663 ± 0.1383 -0.01791 ± 0.00287 0.04977 ± 0.00109 6.289 ± 0.124 % 91.144 ± 0.199 % + 21 7.3977 ± 0.1355 -0.01909 ± 0.00279 0.04995 ± 0.00106 6.287 ± 0.121 % 91.100 ± 0.194 % + 22 7.4678 ± 0.1342 -0.01969 ± 0.00271 0.04961 ± 0.00102 6.256 ± 0.118 % 91.167 ± 0.189 % + 23 7.5020 ± 0.1316 -0.01864 ± 0.00264 0.04905 ± 0.00098 6.186 ± 0.115 % 91.160 ± 0.185 % + 24 7.5785 ± 0.1304 -0.02207 ± 0.00260 0.04959 ± 0.00096 6.220 ± 0.112 % 91.109 ± 0.182 % + 25 7.6290 ± 0.1288 -0.02386 ± 0.00255 0.04974 ± 0.00093 6.201 ± 0.109 % 91.042 ± 0.179 % + 26 7.5697 ± 0.1250 -0.02503 ± 0.00250 0.04975 ± 0.00090 6.210 ± 0.108 % 91.056 ± 0.175 % + 27 7.7225 ± 0.1255 -0.02473 ± 0.00244 0.04954 ± 0.00088 6.190 ± 0.106 % 90.960 ± 0.173 % + 28 7.7899 ± 0.1244 -0.02578 ± 0.00239 0.04914 ± 0.00085 6.153 ± 0.103 % 90.920 ± 0.170 % + 29 7.7814 ± 0.1222 -0.02643 ± 0.00236 0.04928 ± 0.00084 6.187 ± 0.101 % 90.926 ± 0.167 % + 30 7.8204 ± 0.1213 -0.02748 ± 0.00234 0.04957 ± 0.00083 6.221 ± 0.101 % 90.922 ± 0.164 % + 31 7.7205 ± 0.1175 -0.02644 ± 0.00229 0.04911 ± 0.00081 6.185 ± 0.099 % 91.004 ± 0.161 % + 32 7.5976 ± 0.1136 -0.02486 ± 0.00225 0.04884 ± 0.00079 6.206 ± 0.097 % 91.068 ± 0.158 % + 33 7.5905 ± 0.1121 -0.02369 ± 0.00222 0.04910 ± 0.00082 6.210 ± 0.096 % 91.028 ± 0.156 % + 34 7.5705 ± 0.1099 -0.02324 ± 0.00217 0.04851 ± 0.00080 6.170 ± 0.094 % 91.067 ± 0.153 % + 35 7.5703 ± 0.1083 -0.02306 ± 0.00213 0.04818 ± 0.00078 6.149 ± 0.092 % 91.116 ± 0.150 % + 36 7.5770 ± 0.1069 -0.02285 ± 0.00209 0.04774 ± 0.00076 6.122 ± 0.091 % 91.200 ± 0.148 % + 37 7.4347 ± 0.1030 -0.02264 ± 0.00204 0.04714 ± 0.00074 6.080 ± 0.089 % 91.250 ± 0.145 % + 38 7.3263 ± 0.0996 -0.02238 ± 0.00200 0.04688 ± 0.00072 6.069 ± 0.087 % 91.295 ± 0.143 % + 39 7.2174 ± 0.0965 -0.02147 ± 0.00197 0.04653 ± 0.00072 6.053 ± 0.086 % 91.363 ± 0.141 % + 40 7.0944 ± 0.0932 -0.02088 ± 0.00193 0.04598 ± 0.00070 6.030 ± 0.085 % 91.415 ± 0.138 % + 41 7.0202 ± 0.0907 -0.02001 ± 0.00190 0.04559 ± 0.00068 6.015 ± 0.083 % 91.438 ± 0.137 % + 42 7.0513 ± 0.0901 -0.01975 ± 0.00187 0.04559 ± 0.00067 6.002 ± 0.083 % 91.426 ± 0.135 % + 43 7.2378 ± 0.0922 -0.02112 ± 0.00185 0.04560 ± 0.00066 5.978 ± 0.081 % 91.350 ± 0.134 % + 44 7.3764 ± 0.0934 -0.02089 ± 0.00183 0.04556 ± 0.00067 5.951 ± 0.080 % 91.336 ± 0.133 % + 45 7.3525 ± 0.0918 -0.02016 ± 0.00183 0.04644 ± 0.00066 6.010 ± 0.079 % 91.257 ± 0.132 % + 46 7.3601 ± 0.0909 -0.01974 ± 0.00181 0.04679 ± 0.00065 6.020 ± 0.078 % 91.241 ± 0.130 % + 47 7.4224 ± 0.0909 -0.02012 ± 0.00179 0.04695 ± 0.00066 6.021 ± 0.078 % 91.256 ± 0.129 % + 48 7.4203 ± 0.0897 -0.01981 ± 0.00177 0.04672 ± 0.00064 6.014 ± 0.077 % 91.255 ± 0.127 % + 49 7.4723 ± 0.0896 -0.01940 ± 0.00175 0.04672 ± 0.00065 6.006 ± 0.076 % 91.240 ± 0.126 % + 50 7.4303 ± 0.0880 -0.01878 ± 0.00172 0.04649 ± 0.00064 5.991 ± 0.076 % 91.269 ± 0.125 % + 51 7.4613 ± 0.0875 -0.01886 ± 0.00170 0.04621 ± 0.00063 5.965 ± 0.075 % 91.281 ± 0.124 % + 52 7.3985 ± 0.0857 -0.01889 ± 0.00168 0.04595 ± 0.00062 5.958 ± 0.074 % 91.296 ± 0.122 % + 53 7.3695 ± 0.0845 -0.01842 ± 0.00166 0.04569 ± 0.00061 5.933 ± 0.073 % 91.293 ± 0.121 % + 54 7.3294 ± 0.0830 -0.01830 ± 0.00163 0.04530 ± 0.00060 5.912 ± 0.072 % 91.304 ± 0.120 % + 55 7.2835 ± 0.0817 -0.01954 ± 0.00163 0.04534 ± 0.00060 5.930 ± 0.073 % 91.309 ± 0.119 % + 56 7.2720 ± 0.0808 -0.02026 ± 0.00161 0.04529 ± 0.00059 5.935 ± 0.072 % 91.318 ± 0.118 % + 57 7.1959 ± 0.0790 -0.02003 ± 0.00159 0.04500 ± 0.00058 5.910 ± 0.071 % 91.348 ± 0.116 % + 58 7.2383 ± 0.0791 -0.01897 ± 0.00159 0.04532 ± 0.00058 5.918 ± 0.071 % 91.335 ± 0.115 % + 59 7.2091 ± 0.0780 -0.01901 ± 0.00158 0.04571 ± 0.00058 5.931 ± 0.070 % 91.337 ± 0.115 % + 60 7.1923 ± 0.0772 -0.01899 ± 0.00157 0.04543 ± 0.00058 5.912 ± 0.069 % 91.383 ± 0.113 % + 61 7.2561 ± 0.0775 -0.02101 ± 0.00158 0.04623 ± 0.00059 5.960 ± 0.069 % 91.315 ± 0.113 % + 62 7.2786 ± 0.0771 -0.02092 ± 0.00156 0.04609 ± 0.00058 5.940 ± 0.068 % 91.322 ± 0.112 % + 63 7.3208 ± 0.0770 -0.02119 ± 0.00154 0.04584 ± 0.00057 5.918 ± 0.067 % 91.322 ± 0.111 % + 64 7.3522 ± 0.0768 -0.02111 ± 0.00153 0.04567 ± 0.00056 5.899 ± 0.067 % 91.346 ± 0.110 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.352165 ± 0.076781 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 98.95% +Mean ln(PPL(Q)/PPL(base)) : -0.021113 ± 0.001527 +Mean PPL(Q)/PPL(base) : 0.979108 ± 0.001495 +Mean PPL(Q)-PPL(base) : -0.156878 ± 0.011597 + +====== KL divergence statistics ====== +Mean KLD: 0.045673 ± 0.000564 +Maximum KLD: 8.806860 +99.9% KLD: 1.696481 +99.0% KLD: 0.482293 +95.0% KLD: 0.164943 +90.0% KLD: 0.097630 +Median KLD: 0.016240 +10.0% KLD: 0.000249 + 5.0% KLD: 0.000068 + 1.0% KLD: 0.000001 + 0.1% KLD: -0.000142 +Minimum KLD: -0.000835 + +====== Token probability statistics ====== +Mean Δp: -0.117 ± 0.023 % +Maximum Δp: 99.093% +99.9% Δp: 46.291% +99.0% Δp: 18.363% +95.0% Δp: 7.055% +90.0% Δp: 3.785% +75.0% Δp: 0.636% +Median Δp: -0.002% +25.0% Δp: -0.900% +10.0% Δp: -4.342% + 5.0% Δp: -7.493% + 1.0% Δp: -17.995% + 0.1% Δp: -46.845% +Minimum Δp: -84.509% +RMS Δp : 5.899 ± 0.067 % +Same top p: 91.346 ± 0.110 % + diff --git a/appendix/measurements/kld_FAMSHALLOW-19G-ICE.txt b/appendix/measurements/kld_FAMSHALLOW-19G-ICE.txt new file mode 100644 index 0000000000000000000000000000000000000000..2dba9a815f7c23ff33b68761ec69d9d99e0401ee --- /dev/null +++ b/appendix/measurements/kld_FAMSHALLOW-19G-ICE.txt @@ -0,0 +1,111 @@ +0.04.790.165 I +0.04.790.303 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.05.137.141 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.07.071.872 I kl_divergence: 1.93 seconds per pass - ETA 2.05 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.8497 ± 0.4560 -0.00134 ± 0.01779 0.09095 ± 0.00738 8.904 ± 0.523 % 89.541 ± 0.957 % + 2 7.3600 ± 0.4273 0.00700 ± 0.01204 0.07810 ± 0.00425 7.677 ± 0.348 % 89.345 ± 0.682 % + 3 8.2810 ± 0.4072 0.00996 ± 0.00921 0.07238 ± 0.00356 7.357 ± 0.300 % 89.769 ± 0.547 % + 4 9.3159 ± 0.4222 -0.00057 ± 0.00822 0.07728 ± 0.00390 7.590 ± 0.293 % 89.467 ± 0.480 % + 5 8.7127 ± 0.3476 0.00004 ± 0.00731 0.07790 ± 0.00344 7.688 ± 0.279 % 89.247 ± 0.433 % + 6 7.3615 ± 0.2590 -0.00584 ± 0.00659 0.07602 ± 0.00299 7.727 ± 0.246 % 89.182 ± 0.396 % + 7 6.7284 ± 0.2157 -0.01021 ± 0.00611 0.07458 ± 0.00267 7.888 ± 0.234 % 89.582 ± 0.361 % + 8 6.5395 ± 0.1933 -0.00919 ± 0.00555 0.07186 ± 0.00237 7.697 ± 0.213 % 89.895 ± 0.333 % + 9 6.8902 ± 0.1932 -0.01527 ± 0.00521 0.07189 ± 0.00214 7.652 ± 0.196 % 89.475 ± 0.320 % + 10 6.9696 ± 0.1858 -0.01295 ± 0.00479 0.06846 ± 0.00194 7.447 ± 0.184 % 89.668 ± 0.301 % + 11 6.9870 ± 0.1765 -0.01097 ± 0.00449 0.06643 ± 0.00178 7.305 ± 0.173 % 89.834 ± 0.285 % + 12 7.2541 ± 0.1768 -0.00971 ± 0.00420 0.06420 ± 0.00164 7.128 ± 0.163 % 89.907 ± 0.272 % + 13 7.3406 ± 0.1722 -0.01313 ± 0.00403 0.06335 ± 0.00154 7.062 ± 0.156 % 89.969 ± 0.261 % + 14 7.4061 ± 0.1674 -0.01059 ± 0.00384 0.06193 ± 0.00145 6.980 ± 0.152 % 90.057 ± 0.250 % + 15 7.4616 ± 0.1630 -0.01092 ± 0.00368 0.06128 ± 0.00137 6.944 ± 0.146 % 90.068 ± 0.241 % + 16 7.6328 ± 0.1616 -0.01298 ± 0.00356 0.06086 ± 0.00130 6.892 ± 0.141 % 90.048 ± 0.234 % + 17 7.6512 ± 0.1564 -0.01152 ± 0.00341 0.05989 ± 0.00123 6.803 ± 0.135 % 90.213 ± 0.225 % + 18 7.7370 ± 0.1535 -0.01060 ± 0.00329 0.05952 ± 0.00118 6.745 ± 0.130 % 90.154 ± 0.220 % + 19 7.6502 ± 0.1481 -0.01249 ± 0.00319 0.05908 ± 0.00114 6.703 ± 0.125 % 90.194 ± 0.213 % + 20 7.3755 ± 0.1385 -0.01666 ± 0.00327 0.06529 ± 0.00130 7.302 ± 0.135 % 89.995 ± 0.210 % + 21 7.4133 ± 0.1357 -0.01698 ± 0.00320 0.06607 ± 0.00128 7.320 ± 0.132 % 89.871 ± 0.206 % + 22 7.5039 ± 0.1350 -0.01486 ± 0.00313 0.06575 ± 0.00124 7.319 ± 0.129 % 89.887 ± 0.201 % + 23 7.5433 ± 0.1325 -0.01314 ± 0.00304 0.06521 ± 0.00119 7.235 ± 0.125 % 89.940 ± 0.196 % + 24 7.6284 ± 0.1316 -0.01551 ± 0.00298 0.06574 ± 0.00116 7.261 ± 0.123 % 89.875 ± 0.193 % + 25 7.6836 ± 0.1301 -0.01672 ± 0.00292 0.06542 ± 0.00112 7.187 ± 0.120 % 89.826 ± 0.189 % + 26 7.6362 ± 0.1265 -0.01627 ± 0.00285 0.06505 ± 0.00109 7.155 ± 0.117 % 89.905 ± 0.185 % + 27 7.7886 ± 0.1270 -0.01621 ± 0.00278 0.06436 ± 0.00105 7.099 ± 0.115 % 89.906 ± 0.181 % + 28 7.8641 ± 0.1260 -0.01630 ± 0.00272 0.06374 ± 0.00102 7.050 ± 0.112 % 89.911 ± 0.178 % + 29 7.8719 ± 0.1242 -0.01488 ± 0.00267 0.06382 ± 0.00101 7.081 ± 0.110 % 89.905 ± 0.175 % + 30 7.9091 ± 0.1232 -0.01620 ± 0.00263 0.06389 ± 0.00098 7.085 ± 0.109 % 89.915 ± 0.172 % + 31 7.8051 ± 0.1193 -0.01554 ± 0.00258 0.06336 ± 0.00095 7.045 ± 0.107 % 89.976 ± 0.169 % + 32 7.6851 ± 0.1155 -0.01341 ± 0.00254 0.06357 ± 0.00094 7.139 ± 0.108 % 90.002 ± 0.166 % + 33 7.6773 ± 0.1139 -0.01232 ± 0.00251 0.06384 ± 0.00096 7.153 ± 0.105 % 89.982 ± 0.163 % + 34 7.6584 ± 0.1117 -0.01169 ± 0.00245 0.06327 ± 0.00094 7.103 ± 0.103 % 90.029 ± 0.161 % + 35 7.6583 ± 0.1101 -0.01150 ± 0.00240 0.06282 ± 0.00092 7.069 ± 0.101 % 90.091 ± 0.158 % + 36 7.6602 ± 0.1086 -0.01193 ± 0.00235 0.06221 ± 0.00089 7.034 ± 0.100 % 90.192 ± 0.155 % + 37 7.5164 ± 0.1046 -0.01172 ± 0.00230 0.06149 ± 0.00087 7.000 ± 0.098 % 90.238 ± 0.153 % + 38 7.4070 ± 0.1012 -0.01142 ± 0.00226 0.06143 ± 0.00086 7.012 ± 0.097 % 90.269 ± 0.150 % + 39 7.2957 ± 0.0980 -0.01068 ± 0.00223 0.06120 ± 0.00086 7.010 ± 0.096 % 90.280 ± 0.148 % + 40 7.1721 ± 0.0946 -0.00999 ± 0.00219 0.06071 ± 0.00085 7.008 ± 0.095 % 90.335 ± 0.146 % + 41 7.0954 ± 0.0921 -0.00936 ± 0.00216 0.06030 ± 0.00083 6.999 ± 0.093 % 90.327 ± 0.144 % + 42 7.1260 ± 0.0914 -0.00922 ± 0.00212 0.06011 ± 0.00081 6.972 ± 0.092 % 90.318 ± 0.143 % + 43 7.3176 ± 0.0937 -0.01016 ± 0.00209 0.05988 ± 0.00080 6.932 ± 0.091 % 90.270 ± 0.141 % + 44 7.4581 ± 0.0949 -0.00987 ± 0.00207 0.05981 ± 0.00080 6.908 ± 0.090 % 90.269 ± 0.140 % + 45 7.4340 ± 0.0933 -0.00914 ± 0.00206 0.06118 ± 0.00079 6.994 ± 0.089 % 90.197 ± 0.139 % + 46 7.4420 ± 0.0923 -0.00867 ± 0.00204 0.06171 ± 0.00079 7.043 ± 0.089 % 90.182 ± 0.137 % + 47 7.5077 ± 0.0924 -0.00869 ± 0.00202 0.06190 ± 0.00079 7.044 ± 0.089 % 90.183 ± 0.136 % + 48 7.5033 ± 0.0912 -0.00869 ± 0.00199 0.06147 ± 0.00078 7.013 ± 0.088 % 90.152 ± 0.134 % + 49 7.5551 ± 0.0910 -0.00837 ± 0.00197 0.06130 ± 0.00077 6.984 ± 0.087 % 90.151 ± 0.133 % + 50 7.5133 ± 0.0894 -0.00768 ± 0.00194 0.06091 ± 0.00076 6.960 ± 0.086 % 90.209 ± 0.131 % + 51 7.5463 ± 0.0889 -0.00752 ± 0.00191 0.06044 ± 0.00074 6.924 ± 0.085 % 90.227 ± 0.130 % + 52 7.4875 ± 0.0872 -0.00694 ± 0.00190 0.06012 ± 0.00073 6.901 ± 0.084 % 90.272 ± 0.128 % + 53 7.4579 ± 0.0860 -0.00650 ± 0.00187 0.05977 ± 0.00072 6.879 ± 0.082 % 90.269 ± 0.127 % + 54 7.4181 ± 0.0845 -0.00626 ± 0.00185 0.05936 ± 0.00071 6.855 ± 0.081 % 90.292 ± 0.126 % + 55 7.3750 ± 0.0832 -0.00706 ± 0.00183 0.05924 ± 0.00071 6.852 ± 0.081 % 90.287 ± 0.125 % + 56 7.3711 ± 0.0824 -0.00672 ± 0.00182 0.05911 ± 0.00070 6.845 ± 0.080 % 90.305 ± 0.124 % + 57 7.2922 ± 0.0806 -0.00674 ± 0.00179 0.05884 ± 0.00069 6.823 ± 0.080 % 90.353 ± 0.122 % + 58 7.3371 ± 0.0806 -0.00540 ± 0.00179 0.05941 ± 0.00069 6.860 ± 0.079 % 90.334 ± 0.121 % + 59 7.3107 ± 0.0796 -0.00501 ± 0.00178 0.05982 ± 0.00068 6.898 ± 0.079 % 90.336 ± 0.120 % + 60 7.2943 ± 0.0788 -0.00490 ± 0.00176 0.05962 ± 0.00068 6.886 ± 0.078 % 90.352 ± 0.119 % + 61 7.3682 ± 0.0793 -0.00567 ± 0.00176 0.06030 ± 0.00070 6.920 ± 0.078 % 90.307 ± 0.118 % + 62 7.3930 ± 0.0789 -0.00533 ± 0.00174 0.06003 ± 0.00069 6.894 ± 0.077 % 90.307 ± 0.117 % + 63 7.4412 ± 0.0789 -0.00487 ± 0.00173 0.05975 ± 0.00068 6.870 ± 0.076 % 90.299 ± 0.117 % + 64 7.4729 ± 0.0786 -0.00483 ± 0.00171 0.05962 ± 0.00067 6.853 ± 0.075 % 90.315 ± 0.116 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.472885 ± 0.078642 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 98.69% +Mean ln(PPL(Q)/PPL(base)) : -0.004827 ± 0.001709 +Mean PPL(Q)/PPL(base) : 0.995185 ± 0.001701 +Mean PPL(Q)-PPL(base) : -0.036158 ± 0.012821 + +====== KL divergence statistics ====== +Mean KLD: 0.059623 ± 0.000672 +Maximum KLD: 9.654090 +99.9% KLD: 2.073871 +99.0% KLD: 0.625332 +95.0% KLD: 0.216637 +90.0% KLD: 0.129178 +Median KLD: 0.022184 +10.0% KLD: 0.000334 + 5.0% KLD: 0.000094 + 1.0% KLD: 0.000003 + 0.1% KLD: -0.000150 +Minimum KLD: -0.000877 + +====== Token probability statistics ====== +Mean Δp: -0.170 ± 0.027 % +Maximum Δp: 98.598% +99.9% Δp: 49.072% +99.0% Δp: 20.323% +95.0% Δp: 8.341% +90.0% Δp: 4.538% +75.0% Δp: 0.758% +Median Δp: -0.002% +25.0% Δp: -0.975% +10.0% Δp: -4.956% + 5.0% Δp: -8.921% + 1.0% Δp: -22.053% + 0.1% Δp: -53.988% +Minimum Δp: -95.964% +RMS Δp : 6.853 ± 0.075 % +Same top p: 90.315 ± 0.116 % + diff --git a/appendix/measurements/kld_FAMSHALLOW-21G-ICE.txt b/appendix/measurements/kld_FAMSHALLOW-21G-ICE.txt new file mode 100644 index 0000000000000000000000000000000000000000..b7abd7494b007fd5fcbf9569b5baaf57627d9a70 --- /dev/null +++ b/appendix/measurements/kld_FAMSHALLOW-21G-ICE.txt @@ -0,0 +1,111 @@ +0.05.245.318 I +0.05.245.474 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.05.600.527 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.07.535.520 I kl_divergence: 1.93 seconds per pass - ETA 2.05 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.8599 ± 0.4617 0.00040 ± 0.01310 0.05680 ± 0.00402 7.121 ± 0.486 % 92.766 ± 0.810 % + 2 7.3267 ± 0.4277 0.00246 ± 0.00922 0.05132 ± 0.00267 6.121 ± 0.312 % 92.082 ± 0.597 % + 3 8.2728 ± 0.4086 0.00896 ± 0.00751 0.05147 ± 0.00295 6.307 ± 0.348 % 92.245 ± 0.483 % + 4 9.2259 ± 0.4185 -0.01028 ± 0.00700 0.05822 ± 0.00366 6.682 ± 0.322 % 91.813 ± 0.429 % + 5 8.6431 ± 0.3452 -0.00797 ± 0.00613 0.05606 ± 0.00300 6.536 ± 0.280 % 91.652 ± 0.387 % + 6 7.3047 ± 0.2573 -0.01359 ± 0.00554 0.05524 ± 0.00262 6.543 ± 0.242 % 91.496 ± 0.356 % + 7 6.6738 ± 0.2137 -0.01837 ± 0.00511 0.05407 ± 0.00233 6.558 ± 0.224 % 91.579 ± 0.328 % + 8 6.4897 ± 0.1919 -0.01683 ± 0.00466 0.05194 ± 0.00205 6.372 ± 0.203 % 91.691 ± 0.305 % + 9 6.8583 ± 0.1925 -0.01991 ± 0.00438 0.05206 ± 0.00185 6.323 ± 0.186 % 91.365 ± 0.293 % + 10 6.9294 ± 0.1849 -0.01873 ± 0.00406 0.05000 ± 0.00167 6.168 ± 0.173 % 91.535 ± 0.275 % + 11 6.9450 ± 0.1756 -0.01700 ± 0.00385 0.04832 ± 0.00153 6.048 ± 0.162 % 91.629 ± 0.261 % + 12 7.2139 ± 0.1760 -0.01527 ± 0.00359 0.04654 ± 0.00141 5.899 ± 0.153 % 91.691 ± 0.249 % + 13 7.3085 ± 0.1717 -0.01751 ± 0.00346 0.04625 ± 0.00132 5.901 ± 0.147 % 91.729 ± 0.239 % + 14 7.3657 ± 0.1666 -0.01606 ± 0.00327 0.04497 ± 0.00124 5.794 ± 0.140 % 91.684 ± 0.231 % + 15 7.4120 ± 0.1620 -0.01760 ± 0.00314 0.04457 ± 0.00117 5.755 ± 0.134 % 91.685 ± 0.223 % + 16 7.5875 ± 0.1607 -0.01894 ± 0.00303 0.04419 ± 0.00111 5.692 ± 0.128 % 91.716 ± 0.215 % + 17 7.6020 ± 0.1554 -0.01798 ± 0.00289 0.04347 ± 0.00106 5.607 ± 0.123 % 91.731 ± 0.209 % + 18 7.6886 ± 0.1526 -0.01688 ± 0.00278 0.04314 ± 0.00101 5.573 ± 0.118 % 91.740 ± 0.203 % + 19 7.6035 ± 0.1472 -0.01861 ± 0.00271 0.04303 ± 0.00097 5.533 ± 0.113 % 91.773 ± 0.197 % + 20 7.3318 ± 0.1377 -0.02261 ± 0.00279 0.04736 ± 0.00106 6.029 ± 0.120 % 91.628 ± 0.194 % + 21 7.3643 ± 0.1348 -0.02361 ± 0.00272 0.04792 ± 0.00104 6.067 ± 0.118 % 91.505 ± 0.190 % + 22 7.4459 ± 0.1338 -0.02263 ± 0.00264 0.04748 ± 0.00100 6.057 ± 0.115 % 91.593 ± 0.185 % + 23 7.4842 ± 0.1313 -0.02100 ± 0.00257 0.04706 ± 0.00096 6.005 ± 0.111 % 91.555 ± 0.181 % + 24 7.5670 ± 0.1303 -0.02359 ± 0.00252 0.04732 ± 0.00093 6.038 ± 0.110 % 91.504 ± 0.178 % + 25 7.6188 ± 0.1288 -0.02519 ± 0.00248 0.04758 ± 0.00090 6.017 ± 0.107 % 91.406 ± 0.175 % + 26 7.5730 ± 0.1253 -0.02459 ± 0.00242 0.04725 ± 0.00087 5.986 ± 0.104 % 91.469 ± 0.171 % + 27 7.7241 ± 0.1258 -0.02452 ± 0.00236 0.04688 ± 0.00085 5.947 ± 0.102 % 91.438 ± 0.168 % + 28 7.8010 ± 0.1248 -0.02436 ± 0.00230 0.04646 ± 0.00082 5.914 ± 0.099 % 91.454 ± 0.165 % + 29 7.8071 ± 0.1230 -0.02314 ± 0.00227 0.04657 ± 0.00081 5.928 ± 0.097 % 91.492 ± 0.162 % + 30 7.8413 ± 0.1219 -0.02481 ± 0.00225 0.04672 ± 0.00078 5.936 ± 0.095 % 91.505 ± 0.159 % + 31 7.7410 ± 0.1181 -0.02379 ± 0.00220 0.04651 ± 0.00077 5.929 ± 0.093 % 91.521 ± 0.156 % + 32 7.6234 ± 0.1143 -0.02147 ± 0.00217 0.04651 ± 0.00075 5.977 ± 0.092 % 91.560 ± 0.154 % + 33 7.6195 ± 0.1128 -0.01988 ± 0.00215 0.04677 ± 0.00076 5.999 ± 0.091 % 91.543 ± 0.151 % + 34 7.6013 ± 0.1106 -0.01917 ± 0.00210 0.04636 ± 0.00074 5.969 ± 0.089 % 91.550 ± 0.149 % + 35 7.5997 ± 0.1090 -0.01918 ± 0.00206 0.04603 ± 0.00073 5.943 ± 0.088 % 91.563 ± 0.147 % + 36 7.6050 ± 0.1076 -0.01916 ± 0.00202 0.04571 ± 0.00071 5.924 ± 0.086 % 91.642 ± 0.144 % + 37 7.4638 ± 0.1037 -0.01873 ± 0.00198 0.04514 ± 0.00069 5.892 ± 0.085 % 91.707 ± 0.142 % + 38 7.3558 ± 0.1003 -0.01836 ± 0.00194 0.04490 ± 0.00068 5.881 ± 0.083 % 91.735 ± 0.140 % + 39 7.2456 ± 0.0972 -0.01756 ± 0.00191 0.04467 ± 0.00067 5.872 ± 0.083 % 91.729 ± 0.138 % + 40 7.1224 ± 0.0938 -0.01694 ± 0.00187 0.04429 ± 0.00066 5.870 ± 0.082 % 91.764 ± 0.136 % + 41 7.0461 ± 0.0913 -0.01633 ± 0.00185 0.04400 ± 0.00065 5.867 ± 0.081 % 91.748 ± 0.134 % + 42 7.0766 ± 0.0907 -0.01617 ± 0.00182 0.04388 ± 0.00064 5.850 ± 0.080 % 91.740 ± 0.133 % + 43 7.2669 ± 0.0929 -0.01710 ± 0.00180 0.04386 ± 0.00063 5.836 ± 0.080 % 91.696 ± 0.132 % + 44 7.4074 ± 0.0941 -0.01670 ± 0.00178 0.04384 ± 0.00062 5.810 ± 0.079 % 91.693 ± 0.130 % + 45 7.3837 ± 0.0926 -0.01592 ± 0.00177 0.04462 ± 0.00062 5.860 ± 0.078 % 91.643 ± 0.129 % + 46 7.3880 ± 0.0915 -0.01595 ± 0.00176 0.04489 ± 0.00061 5.871 ± 0.077 % 91.623 ± 0.128 % + 47 7.4507 ± 0.0915 -0.01631 ± 0.00174 0.04508 ± 0.00061 5.881 ± 0.076 % 91.639 ± 0.126 % + 48 7.4484 ± 0.0904 -0.01602 ± 0.00171 0.04482 ± 0.00060 5.855 ± 0.075 % 91.622 ± 0.125 % + 49 7.4995 ± 0.0902 -0.01576 ± 0.00169 0.04477 ± 0.00060 5.838 ± 0.074 % 91.559 ± 0.124 % + 50 7.4568 ± 0.0886 -0.01523 ± 0.00167 0.04445 ± 0.00059 5.808 ± 0.073 % 91.591 ± 0.123 % + 51 7.4908 ± 0.0881 -0.01491 ± 0.00164 0.04402 ± 0.00058 5.771 ± 0.072 % 91.618 ± 0.121 % + 52 7.4319 ± 0.0865 -0.01439 ± 0.00162 0.04379 ± 0.00057 5.765 ± 0.072 % 91.648 ± 0.120 % + 53 7.4030 ± 0.0852 -0.01389 ± 0.00160 0.04361 ± 0.00056 5.755 ± 0.071 % 91.634 ± 0.119 % + 54 7.3621 ± 0.0837 -0.01384 ± 0.00158 0.04332 ± 0.00055 5.740 ± 0.070 % 91.644 ± 0.118 % + 55 7.3207 ± 0.0824 -0.01444 ± 0.00157 0.04333 ± 0.00055 5.751 ± 0.069 % 91.638 ± 0.117 % + 56 7.3182 ± 0.0817 -0.01392 ± 0.00156 0.04323 ± 0.00054 5.749 ± 0.069 % 91.651 ± 0.116 % + 57 7.2408 ± 0.0799 -0.01381 ± 0.00154 0.04300 ± 0.00053 5.725 ± 0.068 % 91.696 ± 0.114 % + 58 7.2826 ± 0.0799 -0.01286 ± 0.00154 0.04343 ± 0.00053 5.740 ± 0.067 % 91.664 ± 0.113 % + 59 7.2560 ± 0.0789 -0.01253 ± 0.00153 0.04377 ± 0.00053 5.775 ± 0.067 % 91.661 ± 0.113 % + 60 7.2398 ± 0.0781 -0.01241 ± 0.00152 0.04361 ± 0.00053 5.767 ± 0.066 % 91.675 ± 0.112 % + 61 7.3097 ± 0.0785 -0.01364 ± 0.00152 0.04428 ± 0.00055 5.817 ± 0.067 % 91.645 ± 0.111 % + 62 7.3346 ± 0.0781 -0.01326 ± 0.00151 0.04405 ± 0.00054 5.796 ± 0.067 % 91.647 ± 0.110 % + 63 7.3842 ± 0.0781 -0.01257 ± 0.00149 0.04388 ± 0.00053 5.776 ± 0.066 % 91.645 ± 0.109 % + 64 7.4168 ± 0.0779 -0.01236 ± 0.00148 0.04377 ± 0.00052 5.766 ± 0.066 % 91.647 ± 0.108 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.416771 ± 0.077928 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 99.02% +Mean ln(PPL(Q)/PPL(base)) : -0.012364 ± 0.001478 +Mean PPL(Q)/PPL(base) : 0.987712 ± 0.001460 +Mean PPL(Q)-PPL(base) : -0.092272 ± 0.011112 + +====== KL divergence statistics ====== +Mean KLD: 0.043766 ± 0.000525 +Maximum KLD: 7.441271 +99.9% KLD: 1.559258 +99.0% KLD: 0.433440 +95.0% KLD: 0.160038 +90.0% KLD: 0.094157 +Median KLD: 0.016019 +10.0% KLD: 0.000250 + 5.0% KLD: 0.000068 + 1.0% KLD: -0.000002 + 0.1% KLD: -0.000161 +Minimum KLD: -0.000895 + +====== Token probability statistics ====== +Mean Δp: -0.043 ± 0.023 % +Maximum Δp: 86.543% +99.9% Δp: 44.594% +99.0% Δp: 17.577% +95.0% Δp: 7.246% +90.0% Δp: 3.927% +75.0% Δp: 0.697% +Median Δp: -0.001% +25.0% Δp: -0.794% +10.0% Δp: -4.118% + 5.0% Δp: -7.346% + 1.0% Δp: -17.592% + 0.1% Δp: -42.874% +Minimum Δp: -92.951% +RMS Δp : 5.766 ± 0.066 % +Same top p: 91.647 ± 0.108 % + diff --git a/appendix/measurements/kld_FLOORBF16.txt b/appendix/measurements/kld_FLOORBF16.txt new file mode 100644 index 0000000000000000000000000000000000000000..f910a44509d0d11f8b156d7b2d01103f0378bb18 --- /dev/null +++ b/appendix/measurements/kld_FLOORBF16.txt @@ -0,0 +1,111 @@ +0.05.688.915 I +0.05.689.070 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.06.026.826 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.07.886.668 I kl_divergence: 1.86 seconds per pass - ETA 1.98 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.7728 ± 0.4515 -0.01457 ± 0.01164 0.04767 ± 0.00440 6.765 ± 0.711 % 92.571 ± 0.820 % + 2 7.2617 ± 0.4216 -0.00646 ± 0.00796 0.04046 ± 0.00245 5.682 ± 0.449 % 92.375 ± 0.587 % + 3 8.2520 ± 0.4070 0.00645 ± 0.00646 0.03668 ± 0.00179 5.332 ± 0.354 % 92.603 ± 0.472 % + 4 9.4089 ± 0.4299 0.00937 ± 0.00585 0.04196 ± 0.00255 5.485 ± 0.310 % 92.424 ± 0.414 % + 5 8.7836 ± 0.3531 0.00815 ± 0.00520 0.04031 ± 0.00214 5.441 ± 0.286 % 92.395 ± 0.371 % + 6 7.4347 ± 0.2640 0.00405 ± 0.00465 0.03911 ± 0.00184 5.539 ± 0.250 % 92.424 ± 0.338 % + 7 6.7862 ± 0.2190 -0.00166 ± 0.00440 0.03986 ± 0.00167 5.897 ± 0.238 % 92.403 ± 0.313 % + 8 6.5920 ± 0.1964 -0.00118 ± 0.00401 0.03829 ± 0.00148 5.794 ± 0.221 % 92.473 ± 0.292 % + 9 6.9771 ± 0.1975 -0.00273 ± 0.00379 0.03891 ± 0.00136 5.716 ± 0.201 % 92.223 ± 0.279 % + 10 7.0373 ± 0.1890 -0.00328 ± 0.00353 0.03763 ± 0.00125 5.625 ± 0.191 % 92.424 ± 0.262 % + 11 7.0430 ± 0.1792 -0.00299 ± 0.00334 0.03623 ± 0.00115 5.475 ± 0.180 % 92.642 ± 0.246 % + 12 7.3161 ± 0.1795 -0.00121 ± 0.00312 0.03492 ± 0.00106 5.328 ± 0.170 % 92.652 ± 0.236 % + 13 7.4034 ± 0.1749 -0.00460 ± 0.00300 0.03513 ± 0.00103 5.360 ± 0.161 % 92.676 ± 0.226 % + 14 7.4492 ± 0.1692 -0.00480 ± 0.00284 0.03409 ± 0.00096 5.276 ± 0.153 % 92.745 ± 0.217 % + 15 7.5013 ± 0.1646 -0.00562 ± 0.00272 0.03365 ± 0.00091 5.235 ± 0.145 % 92.812 ± 0.209 % + 16 7.6842 ± 0.1634 -0.00626 ± 0.00262 0.03325 ± 0.00085 5.152 ± 0.139 % 92.827 ± 0.202 % + 17 7.6928 ± 0.1579 -0.00611 ± 0.00250 0.03282 ± 0.00083 5.080 ± 0.133 % 92.853 ± 0.195 % + 18 7.7728 ± 0.1548 -0.00599 ± 0.00241 0.03251 ± 0.00079 5.052 ± 0.127 % 92.886 ± 0.189 % + 19 7.6953 ± 0.1496 -0.00661 ± 0.00231 0.03172 ± 0.00075 4.980 ± 0.123 % 92.982 ± 0.183 % + 20 7.4347 ± 0.1403 -0.00867 ± 0.00236 0.03469 ± 0.00081 5.395 ± 0.128 % 92.835 ± 0.180 % + 21 7.4739 ± 0.1376 -0.00884 ± 0.00231 0.03491 ± 0.00079 5.394 ± 0.125 % 92.762 ± 0.177 % + 22 7.5501 ± 0.1364 -0.00873 ± 0.00224 0.03462 ± 0.00076 5.369 ± 0.121 % 92.833 ± 0.172 % + 23 7.5795 ± 0.1336 -0.00835 ± 0.00217 0.03420 ± 0.00073 5.308 ± 0.117 % 92.817 ± 0.168 % + 24 7.6618 ± 0.1325 -0.01114 ± 0.00214 0.03461 ± 0.00071 5.312 ± 0.114 % 92.758 ± 0.165 % + 25 7.7324 ± 0.1314 -0.01039 ± 0.00209 0.03422 ± 0.00069 5.261 ± 0.111 % 92.708 ± 0.163 % + 26 7.6837 ± 0.1278 -0.01007 ± 0.00203 0.03387 ± 0.00067 5.221 ± 0.109 % 92.763 ± 0.159 % + 27 7.8384 ± 0.1283 -0.00983 ± 0.00198 0.03328 ± 0.00064 5.156 ± 0.106 % 92.777 ± 0.156 % + 28 7.9157 ± 0.1274 -0.00976 ± 0.00193 0.03299 ± 0.00063 5.118 ± 0.104 % 92.801 ± 0.153 % + 29 7.9252 ± 0.1256 -0.00812 ± 0.00190 0.03308 ± 0.00061 5.120 ± 0.101 % 92.827 ± 0.150 % + 30 7.9506 ± 0.1243 -0.01097 ± 0.00189 0.03350 ± 0.00060 5.130 ± 0.098 % 92.796 ± 0.148 % + 31 7.8418 ± 0.1202 -0.01085 ± 0.00185 0.03330 ± 0.00059 5.109 ± 0.096 % 92.811 ± 0.145 % + 32 7.7172 ± 0.1163 -0.00924 ± 0.00183 0.03324 ± 0.00057 5.132 ± 0.093 % 92.870 ± 0.142 % + 33 7.7067 ± 0.1146 -0.00850 ± 0.00180 0.03352 ± 0.00063 5.159 ± 0.093 % 92.852 ± 0.140 % + 34 7.6864 ± 0.1124 -0.00805 ± 0.00176 0.03318 ± 0.00061 5.133 ± 0.091 % 92.847 ± 0.138 % + 35 7.6777 ± 0.1106 -0.00896 ± 0.00173 0.03311 ± 0.00060 5.132 ± 0.090 % 92.847 ± 0.136 % + 36 7.6827 ± 0.1092 -0.00899 ± 0.00170 0.03284 ± 0.00059 5.102 ± 0.088 % 92.891 ± 0.134 % + 37 7.5389 ± 0.1052 -0.00872 ± 0.00166 0.03237 ± 0.00057 5.066 ± 0.086 % 92.949 ± 0.132 % + 38 7.4286 ± 0.1018 -0.00851 ± 0.00163 0.03212 ± 0.00056 5.053 ± 0.085 % 92.988 ± 0.130 % + 39 7.3144 ± 0.0985 -0.00812 ± 0.00160 0.03177 ± 0.00055 5.029 ± 0.083 % 93.002 ± 0.128 % + 40 7.1875 ± 0.0951 -0.00785 ± 0.00157 0.03141 ± 0.00054 5.013 ± 0.082 % 93.050 ± 0.126 % + 41 7.1082 ± 0.0925 -0.00755 ± 0.00155 0.03116 ± 0.00052 4.989 ± 0.080 % 93.076 ± 0.124 % + 42 7.1381 ± 0.0918 -0.00751 ± 0.00153 0.03110 ± 0.00051 4.984 ± 0.079 % 93.067 ± 0.123 % + 43 7.3385 ± 0.0942 -0.00731 ± 0.00150 0.03111 ± 0.00050 4.961 ± 0.078 % 93.005 ± 0.122 % + 44 7.4815 ± 0.0955 -0.00674 ± 0.00149 0.03111 ± 0.00051 4.941 ± 0.077 % 93.000 ± 0.120 % + 45 7.4583 ± 0.0940 -0.00587 ± 0.00148 0.03195 ± 0.00051 5.008 ± 0.076 % 92.916 ± 0.120 % + 46 7.4659 ± 0.0930 -0.00546 ± 0.00147 0.03212 ± 0.00050 5.019 ± 0.075 % 92.907 ± 0.118 % + 47 7.5287 ± 0.0929 -0.00589 ± 0.00145 0.03223 ± 0.00050 5.016 ± 0.074 % 92.902 ± 0.117 % + 48 7.5266 ± 0.0918 -0.00559 ± 0.00143 0.03204 ± 0.00049 4.993 ± 0.073 % 92.913 ± 0.116 % + 49 7.5789 ± 0.0916 -0.00524 ± 0.00142 0.03209 ± 0.00048 4.988 ± 0.072 % 92.872 ± 0.115 % + 50 7.5327 ± 0.0899 -0.00510 ± 0.00140 0.03187 ± 0.00047 4.971 ± 0.071 % 92.911 ± 0.113 % + 51 7.5679 ± 0.0895 -0.00467 ± 0.00138 0.03159 ± 0.00047 4.940 ± 0.070 % 92.935 ± 0.112 % + 52 7.5072 ± 0.0878 -0.00432 ± 0.00137 0.03142 ± 0.00046 4.939 ± 0.069 % 92.956 ± 0.111 % + 53 7.4789 ± 0.0865 -0.00368 ± 0.00135 0.03123 ± 0.00045 4.926 ± 0.068 % 92.971 ± 0.110 % + 54 7.4354 ± 0.0850 -0.00393 ± 0.00133 0.03101 ± 0.00045 4.916 ± 0.068 % 92.965 ± 0.109 % + 55 7.3950 ± 0.0837 -0.00434 ± 0.00132 0.03094 ± 0.00044 4.928 ± 0.068 % 92.955 ± 0.108 % + 56 7.3841 ± 0.0828 -0.00496 ± 0.00131 0.03093 ± 0.00043 4.924 ± 0.067 % 92.951 ± 0.107 % + 57 7.3054 ± 0.0810 -0.00493 ± 0.00130 0.03077 ± 0.00043 4.910 ± 0.066 % 92.960 ± 0.106 % + 58 7.3422 ± 0.0809 -0.00470 ± 0.00130 0.03127 ± 0.00043 4.952 ± 0.065 % 92.932 ± 0.105 % + 59 7.3161 ± 0.0799 -0.00428 ± 0.00130 0.03155 ± 0.00043 4.973 ± 0.065 % 92.932 ± 0.104 % + 60 7.2992 ± 0.0790 -0.00423 ± 0.00128 0.03137 ± 0.00043 4.958 ± 0.064 % 92.967 ± 0.103 % + 61 7.3703 ± 0.0795 -0.00539 ± 0.00129 0.03195 ± 0.00045 4.994 ± 0.064 % 92.915 ± 0.103 % + 62 7.3943 ± 0.0791 -0.00516 ± 0.00128 0.03182 ± 0.00044 4.973 ± 0.063 % 92.913 ± 0.102 % + 63 7.4403 ± 0.0790 -0.00499 ± 0.00126 0.03166 ± 0.00043 4.956 ± 0.063 % 92.900 ± 0.101 % + 64 7.4712 ± 0.0788 -0.00506 ± 0.00125 0.03154 ± 0.00043 4.942 ± 0.062 % 92.916 ± 0.100 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.471173 ± 0.078771 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 99.30% +Mean ln(PPL(Q)/PPL(base)) : -0.005056 ± 0.001250 +Mean PPL(Q)/PPL(base) : 0.994957 ± 0.001244 +Mean PPL(Q)-PPL(base) : -0.037870 ± 0.009381 + +====== KL divergence statistics ====== +Mean KLD: 0.031543 ± 0.000428 +Maximum KLD: 9.499484 +99.9% KLD: 1.221953 +99.0% KLD: 0.321817 +95.0% KLD: 0.113814 +90.0% KLD: 0.068127 +Median KLD: 0.010694 +10.0% KLD: 0.000164 + 5.0% KLD: 0.000044 + 1.0% KLD: -0.000003 + 0.1% KLD: -0.000143 +Minimum KLD: -0.000842 + +====== Token probability statistics ====== +Mean Δp: -0.058 ± 0.019 % +Maximum Δp: 89.304% +99.9% Δp: 38.220% +99.0% Δp: 15.078% +95.0% Δp: 5.939% +90.0% Δp: 3.167% +75.0% Δp: 0.544% +Median Δp: -0.001% +25.0% Δp: -0.655% +10.0% Δp: -3.404% + 5.0% Δp: -6.134% + 1.0% Δp: -15.171% + 0.1% Δp: -38.071% +Minimum Δp: -75.561% +RMS Δp : 4.942 ± 0.062 % +Same top p: 92.916 ± 0.100 % + diff --git a/appendix/measurements/kld_H.txt b/appendix/measurements/kld_H.txt new file mode 100644 index 0000000000000000000000000000000000000000..1a5608bc3e64cad1900d9d5844837d46eb691c7d --- /dev/null +++ b/appendix/measurements/kld_H.txt @@ -0,0 +1,112 @@ +0.05.196.457 I cmn init: llama threadpool init, n_threads = 30 +0.05.335.468 I +0.05.335.647 I system_info: n_threads = 30 (n_threads_batch = 30) / 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.05.679.180 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.25.203.802 I kl_divergence: 19.52 seconds per pass - ETA 20.82 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.9174 ± 0.4677 0.01017 ± 0.00896 0.02132 ± 0.00172 4.372 ± 0.419 % 95.112 ± 0.674 % + 2 7.3822 ± 0.4328 0.01001 ± 0.00579 0.01936 ± 0.00105 3.783 ± 0.282 % 94.282 ± 0.513 % + 3 8.2291 ± 0.4066 0.00367 ± 0.00461 0.01907 ± 0.00174 3.756 ± 0.311 % 94.884 ± 0.398 % + 4 9.3087 ± 0.4251 -0.00134 ± 0.00442 0.02442 ± 0.00201 4.340 ± 0.315 % 94.599 ± 0.353 % + 5 8.6947 ± 0.3495 -0.00202 ± 0.00375 0.02208 ± 0.00162 4.112 ± 0.269 % 94.663 ± 0.314 % + 6 7.3976 ± 0.2626 -0.00096 ± 0.00354 0.02327 ± 0.00158 4.439 ± 0.280 % 94.558 ± 0.290 % + 7 6.8377 ± 0.2217 0.00590 ± 0.00339 0.02421 ± 0.00143 4.653 ± 0.253 % 94.554 ± 0.268 % + 8 6.6282 ± 0.1982 0.00429 ± 0.00307 0.02290 ± 0.00126 4.498 ± 0.230 % 94.526 ± 0.251 % + 9 7.0033 ± 0.1987 0.00101 ± 0.00289 0.02285 ± 0.00113 4.411 ± 0.210 % 94.363 ± 0.240 % + 10 7.0613 ± 0.1902 0.00012 ± 0.00271 0.02207 ± 0.00102 4.316 ± 0.194 % 94.438 ± 0.227 % + 11 7.0625 ± 0.1802 -0.00023 ± 0.00261 0.02118 ± 0.00093 4.249 ± 0.182 % 94.570 ± 0.214 % + 12 7.3127 ± 0.1797 -0.00167 ± 0.00246 0.02062 ± 0.00086 4.172 ± 0.171 % 94.607 ± 0.204 % + 13 7.4186 ± 0.1758 -0.00257 ± 0.00236 0.02038 ± 0.00080 4.132 ± 0.160 % 94.631 ± 0.195 % + 14 7.4637 ± 0.1700 -0.00284 ± 0.00224 0.01969 ± 0.00075 4.048 ± 0.152 % 94.721 ± 0.187 % + 15 7.5143 ± 0.1653 -0.00389 ± 0.00216 0.01949 ± 0.00070 4.017 ± 0.143 % 94.774 ± 0.180 % + 16 7.6914 ± 0.1639 -0.00533 ± 0.00206 0.01928 ± 0.00066 3.961 ± 0.137 % 94.770 ± 0.174 % + 17 7.6975 ± 0.1583 -0.00549 ± 0.00197 0.01884 ± 0.00062 3.894 ± 0.131 % 94.819 ± 0.168 % + 18 7.7746 ± 0.1551 -0.00576 ± 0.00190 0.01865 ± 0.00059 3.881 ± 0.126 % 94.825 ± 0.163 % + 19 7.6863 ± 0.1495 -0.00778 ± 0.00185 0.01884 ± 0.00057 3.882 ± 0.120 % 94.752 ± 0.160 % + 20 7.4424 ± 0.1406 -0.00763 ± 0.00183 0.02033 ± 0.00072 4.101 ± 0.123 % 94.692 ± 0.157 % + 21 7.4876 ± 0.1380 -0.00701 ± 0.00179 0.02061 ± 0.00071 4.182 ± 0.128 % 94.628 ± 0.154 % + 22 7.5511 ± 0.1364 -0.00860 ± 0.00174 0.02055 ± 0.00068 4.163 ± 0.124 % 94.633 ± 0.150 % + 23 7.5812 ± 0.1337 -0.00813 ± 0.00169 0.02028 ± 0.00065 4.106 ± 0.120 % 94.666 ± 0.146 % + 24 7.6835 ± 0.1331 -0.00831 ± 0.00166 0.02047 ± 0.00063 4.124 ± 0.117 % 94.681 ± 0.143 % + 25 7.7448 ± 0.1318 -0.00878 ± 0.00162 0.02029 ± 0.00061 4.093 ± 0.114 % 94.663 ± 0.141 % + 26 7.6955 ± 0.1281 -0.00853 ± 0.00158 0.02014 ± 0.00059 4.085 ± 0.112 % 94.695 ± 0.137 % + 27 7.8462 ± 0.1285 -0.00884 ± 0.00154 0.01976 ± 0.00057 4.037 ± 0.109 % 94.692 ± 0.135 % + 28 7.9290 ± 0.1277 -0.00808 ± 0.00150 0.01946 ± 0.00055 3.999 ± 0.106 % 94.725 ± 0.132 % + 29 7.9318 ± 0.1258 -0.00729 ± 0.00148 0.01941 ± 0.00053 3.994 ± 0.103 % 94.715 ± 0.130 % + 30 7.9769 ± 0.1249 -0.00766 ± 0.00146 0.01933 ± 0.00052 3.977 ± 0.100 % 94.695 ± 0.128 % + 31 7.8678 ± 0.1208 -0.00754 ± 0.00143 0.01908 ± 0.00050 3.949 ± 0.098 % 94.740 ± 0.125 % + 32 7.7296 ± 0.1166 -0.00763 ± 0.00140 0.01902 ± 0.00049 3.962 ± 0.096 % 94.789 ± 0.123 % + 33 7.7175 ± 0.1149 -0.00710 ± 0.00139 0.01908 ± 0.00050 3.953 ± 0.094 % 94.772 ± 0.121 % + 34 7.6930 ± 0.1126 -0.00719 ± 0.00135 0.01881 ± 0.00048 3.925 ± 0.092 % 94.790 ± 0.119 % + 35 7.6901 ± 0.1109 -0.00735 ± 0.00133 0.01872 ± 0.00048 3.909 ± 0.090 % 94.788 ± 0.117 % + 36 7.6936 ± 0.1094 -0.00758 ± 0.00130 0.01862 ± 0.00046 3.894 ± 0.088 % 94.833 ± 0.115 % + 37 7.5492 ± 0.1054 -0.00735 ± 0.00127 0.01833 ± 0.00045 3.869 ± 0.087 % 94.898 ± 0.113 % + 38 7.4373 ± 0.1020 -0.00734 ± 0.00125 0.01818 ± 0.00044 3.855 ± 0.085 % 94.914 ± 0.111 % + 39 7.3208 ± 0.0987 -0.00724 ± 0.00123 0.01796 ± 0.00043 3.840 ± 0.083 % 94.934 ± 0.110 % + 40 7.1933 ± 0.0952 -0.00703 ± 0.00120 0.01772 ± 0.00042 3.825 ± 0.082 % 94.968 ± 0.108 % + 41 7.1144 ± 0.0926 -0.00669 ± 0.00118 0.01754 ± 0.00041 3.807 ± 0.080 % 94.986 ± 0.107 % + 42 7.1456 ± 0.0920 -0.00646 ± 0.00116 0.01748 ± 0.00040 3.789 ± 0.079 % 94.973 ± 0.105 % + 43 7.3403 ± 0.0942 -0.00706 ± 0.00116 0.01768 ± 0.00040 3.802 ± 0.080 % 94.894 ± 0.105 % + 44 7.4761 ± 0.0954 -0.00747 ± 0.00115 0.01774 ± 0.00040 3.787 ± 0.079 % 94.864 ± 0.104 % + 45 7.4483 ± 0.0938 -0.00721 ± 0.00114 0.01807 ± 0.00039 3.812 ± 0.077 % 94.826 ± 0.103 % + 46 7.4547 ± 0.0928 -0.00696 ± 0.00113 0.01809 ± 0.00039 3.809 ± 0.075 % 94.826 ± 0.102 % + 47 7.5195 ± 0.0928 -0.00713 ± 0.00112 0.01827 ± 0.00039 3.833 ± 0.078 % 94.815 ± 0.101 % + 48 7.5177 ± 0.0916 -0.00677 ± 0.00111 0.01819 ± 0.00039 3.819 ± 0.077 % 94.827 ± 0.100 % + 49 7.5661 ± 0.0914 -0.00692 ± 0.00110 0.01835 ± 0.00039 3.827 ± 0.076 % 94.787 ± 0.099 % + 50 7.5215 ± 0.0897 -0.00659 ± 0.00109 0.01825 ± 0.00039 3.810 ± 0.075 % 94.819 ± 0.098 % + 51 7.5548 ± 0.0892 -0.00640 ± 0.00107 0.01808 ± 0.00038 3.787 ± 0.074 % 94.834 ± 0.097 % + 52 7.4920 ± 0.0875 -0.00634 ± 0.00106 0.01798 ± 0.00037 3.780 ± 0.073 % 94.857 ± 0.096 % + 53 7.4630 ± 0.0862 -0.00582 ± 0.00105 0.01790 ± 0.00037 3.771 ± 0.072 % 94.839 ± 0.095 % + 54 7.4215 ± 0.0847 -0.00581 ± 0.00103 0.01784 ± 0.00036 3.780 ± 0.072 % 94.850 ± 0.094 % + 55 7.3804 ± 0.0834 -0.00633 ± 0.00103 0.01780 ± 0.00036 3.774 ± 0.071 % 94.855 ± 0.093 % + 56 7.3767 ± 0.0827 -0.00596 ± 0.00102 0.01779 ± 0.00035 3.778 ± 0.070 % 94.859 ± 0.092 % + 57 7.2984 ± 0.0808 -0.00589 ± 0.00100 0.01768 ± 0.00035 3.771 ± 0.069 % 94.891 ± 0.091 % + 58 7.3308 ± 0.0807 -0.00626 ± 0.00101 0.01817 ± 0.00036 3.825 ± 0.069 % 94.834 ± 0.091 % + 59 7.3039 ± 0.0797 -0.00595 ± 0.00101 0.01828 ± 0.00035 3.844 ± 0.068 % 94.809 ± 0.090 % + 60 7.2858 ± 0.0788 -0.00607 ± 0.00100 0.01819 ± 0.00035 3.836 ± 0.067 % 94.827 ± 0.089 % + 61 7.3585 ± 0.0793 -0.00699 ± 0.00100 0.01863 ± 0.00037 3.877 ± 0.068 % 94.784 ± 0.089 % + 62 7.3822 ± 0.0789 -0.00679 ± 0.00099 0.01851 ± 0.00036 3.858 ± 0.067 % 94.778 ± 0.088 % + 63 7.4281 ± 0.0788 -0.00664 ± 0.00098 0.01839 ± 0.00035 3.843 ± 0.066 % 94.783 ± 0.088 % + 64 7.4585 ± 0.0786 -0.00676 ± 0.00096 0.01830 ± 0.00035 3.828 ± 0.066 % 94.790 ± 0.087 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.458467 ± 0.078564 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 99.58% +Mean ln(PPL(Q)/PPL(base)) : -0.006758 ± 0.000964 +Mean PPL(Q)/PPL(base) : 0.993265 ± 0.000958 +Mean PPL(Q)-PPL(base) : -0.050576 ± 0.007257 + +====== KL divergence statistics ====== +Mean KLD: 0.018297 ± 0.000349 +Maximum KLD: 7.798795 +99.9% KLD: 0.792242 +99.0% KLD: 0.180538 +95.0% KLD: 0.062892 +90.0% KLD: 0.037522 +Median KLD: 0.005781 +10.0% KLD: 0.000103 + 5.0% KLD: 0.000027 + 1.0% KLD: -0.000005 + 0.1% KLD: -0.000138 +Minimum KLD: -0.000706 + +====== Token probability statistics ====== +Mean Δp: -0.014 ± 0.015 % +Maximum Δp: 88.045% +99.9% Δp: 30.113% +99.0% Δp: 11.269% +95.0% Δp: 4.348% +90.0% Δp: 2.350% +75.0% Δp: 0.405% +Median Δp: -0.000% +25.0% Δp: -0.478% +10.0% Δp: -2.455% + 5.0% Δp: -4.492% + 1.0% Δp: -10.845% + 0.1% Δp: -26.883% +Minimum Δp: -97.726% +RMS Δp : 3.828 ± 0.066 % +Same top p: 94.790 ± 0.087 % + diff --git a/appendix/measurements/kld_SHALLOW-19G-ICE.txt b/appendix/measurements/kld_SHALLOW-19G-ICE.txt new file mode 100644 index 0000000000000000000000000000000000000000..e9d909f52a72f2d4be3b0ea2f56cd61092d5ab85 --- /dev/null +++ b/appendix/measurements/kld_SHALLOW-19G-ICE.txt @@ -0,0 +1,111 @@ +0.04.785.606 I +0.04.785.737 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.05.118.538 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.06.917.749 I kl_divergence: 1.80 seconds per pass - ETA 1.92 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.9463 ± 0.4643 0.01505 ± 0.01499 0.08661 ± 0.00811 8.274 ± 0.592 % 89.443 ± 0.961 % + 2 7.4650 ± 0.4361 0.02116 ± 0.01026 0.07104 ± 0.00436 7.223 ± 0.371 % 89.443 ± 0.680 % + 3 8.3322 ± 0.4101 0.01612 ± 0.00861 0.06969 ± 0.00404 7.412 ± 0.413 % 89.834 ± 0.546 % + 4 9.4202 ± 0.4280 0.01057 ± 0.00779 0.07592 ± 0.00408 7.556 ± 0.343 % 89.687 ± 0.475 % + 5 8.8371 ± 0.3542 0.01422 ± 0.00685 0.07243 ± 0.00338 7.322 ± 0.291 % 89.717 ± 0.425 % + 6 7.4746 ± 0.2640 0.00941 ± 0.00611 0.07048 ± 0.00297 7.398 ± 0.257 % 89.752 ± 0.387 % + 7 6.8185 ± 0.2192 0.00309 ± 0.00585 0.07181 ± 0.00288 7.786 ± 0.260 % 89.890 ± 0.356 % + 8 6.6259 ± 0.1963 0.00393 ± 0.00532 0.06926 ± 0.00256 7.623 ± 0.236 % 90.042 ± 0.331 % + 9 7.0245 ± 0.1979 0.00403 ± 0.00498 0.06973 ± 0.00233 7.531 ± 0.217 % 89.584 ± 0.318 % + 10 7.0920 ± 0.1897 0.00446 ± 0.00462 0.06742 ± 0.00211 7.395 ± 0.203 % 89.609 ± 0.302 % + 11 7.0837 ± 0.1791 0.00277 ± 0.00436 0.06581 ± 0.00194 7.306 ± 0.192 % 89.612 ± 0.288 % + 12 7.3487 ± 0.1791 0.00324 ± 0.00408 0.06367 ± 0.00179 7.144 ± 0.181 % 89.703 ± 0.274 % + 13 7.4709 ± 0.1758 0.00447 ± 0.00389 0.06251 ± 0.00167 7.108 ± 0.170 % 89.766 ± 0.263 % + 14 7.5105 ± 0.1699 0.00340 ± 0.00370 0.06086 ± 0.00157 6.985 ± 0.163 % 89.813 ± 0.253 % + 15 7.5804 ± 0.1659 0.00487 ± 0.00356 0.06024 ± 0.00148 6.926 ± 0.155 % 89.853 ± 0.244 % + 16 7.7541 ± 0.1643 0.00279 ± 0.00345 0.05999 ± 0.00140 6.887 ± 0.149 % 89.846 ± 0.236 % + 17 7.7624 ± 0.1586 0.00290 ± 0.00331 0.05876 ± 0.00132 6.781 ± 0.143 % 89.874 ± 0.229 % + 18 7.8483 ± 0.1556 0.00367 ± 0.00319 0.05818 ± 0.00126 6.740 ± 0.137 % 89.980 ± 0.221 % + 19 7.7638 ± 0.1502 0.00225 ± 0.00308 0.05719 ± 0.00120 6.664 ± 0.132 % 90.076 ± 0.214 % + 20 7.5078 ± 0.1409 0.00112 ± 0.00316 0.06251 ± 0.00134 7.143 ± 0.138 % 89.868 ± 0.211 % + 21 7.5513 ± 0.1383 0.00146 ± 0.00309 0.06277 ± 0.00130 7.160 ± 0.135 % 89.806 ± 0.206 % + 22 7.6127 ± 0.1367 -0.00048 ± 0.00301 0.06239 ± 0.00124 7.119 ± 0.130 % 89.834 ± 0.201 % + 23 7.6421 ± 0.1339 -0.00013 ± 0.00292 0.06174 ± 0.00120 7.064 ± 0.127 % 89.915 ± 0.196 % + 24 7.7141 ± 0.1326 -0.00434 ± 0.00289 0.06304 ± 0.00117 7.116 ± 0.124 % 89.866 ± 0.193 % + 25 7.7736 ± 0.1312 -0.00507 ± 0.00283 0.06279 ± 0.00114 7.086 ± 0.122 % 89.853 ± 0.189 % + 26 7.7164 ± 0.1273 -0.00583 ± 0.00277 0.06252 ± 0.00111 7.055 ± 0.119 % 89.920 ± 0.185 % + 27 7.8712 ± 0.1278 -0.00565 ± 0.00269 0.06180 ± 0.00107 6.995 ± 0.116 % 89.903 ± 0.181 % + 28 7.9431 ± 0.1267 -0.00630 ± 0.00263 0.06131 ± 0.00104 6.953 ± 0.114 % 89.921 ± 0.178 % + 29 7.9316 ± 0.1244 -0.00732 ± 0.00262 0.06211 ± 0.00103 7.041 ± 0.113 % 89.928 ± 0.175 % + 30 7.9718 ± 0.1235 -0.00830 ± 0.00258 0.06207 ± 0.00100 7.030 ± 0.110 % 89.935 ± 0.172 % + 31 7.8700 ± 0.1196 -0.00726 ± 0.00253 0.06164 ± 0.00098 7.011 ± 0.107 % 90.017 ± 0.168 % + 32 7.7375 ± 0.1155 -0.00660 ± 0.00248 0.06171 ± 0.00096 7.024 ± 0.106 % 90.124 ± 0.165 % + 33 7.7222 ± 0.1138 -0.00649 ± 0.00244 0.06170 ± 0.00096 7.017 ± 0.104 % 90.163 ± 0.162 % + 34 7.6971 ± 0.1114 -0.00666 ± 0.00239 0.06123 ± 0.00093 6.982 ± 0.102 % 90.141 ± 0.160 % + 35 7.6880 ± 0.1097 -0.00763 ± 0.00235 0.06113 ± 0.00091 6.970 ± 0.100 % 90.158 ± 0.157 % + 36 7.6952 ± 0.1083 -0.00737 ± 0.00230 0.06056 ± 0.00089 6.935 ± 0.099 % 90.247 ± 0.155 % + 37 7.5552 ± 0.1044 -0.00656 ± 0.00225 0.05988 ± 0.00087 6.903 ± 0.098 % 90.331 ± 0.152 % + 38 7.4468 ± 0.1010 -0.00607 ± 0.00221 0.05971 ± 0.00086 6.921 ± 0.096 % 90.312 ± 0.150 % + 39 7.3309 ± 0.0978 -0.00586 ± 0.00218 0.05931 ± 0.00084 6.899 ± 0.095 % 90.320 ± 0.148 % + 40 7.2053 ± 0.0944 -0.00537 ± 0.00214 0.05879 ± 0.00083 6.882 ± 0.093 % 90.367 ± 0.146 % + 41 7.1268 ± 0.0918 -0.00494 ± 0.00210 0.05848 ± 0.00081 6.896 ± 0.093 % 90.382 ± 0.144 % + 42 7.1609 ± 0.0912 -0.00433 ± 0.00207 0.05832 ± 0.00080 6.864 ± 0.091 % 90.367 ± 0.142 % + 43 7.3613 ± 0.0936 -0.00420 ± 0.00205 0.05823 ± 0.00078 6.836 ± 0.090 % 90.329 ± 0.141 % + 44 7.4945 ± 0.0947 -0.00500 ± 0.00202 0.05808 ± 0.00077 6.796 ± 0.088 % 90.320 ± 0.139 % + 45 7.4782 ± 0.0933 -0.00321 ± 0.00202 0.05953 ± 0.00078 6.875 ± 0.087 % 90.212 ± 0.138 % + 46 7.4825 ± 0.0922 -0.00324 ± 0.00200 0.06000 ± 0.00077 6.904 ± 0.086 % 90.221 ± 0.137 % + 47 7.5432 ± 0.0922 -0.00398 ± 0.00198 0.06029 ± 0.00078 6.918 ± 0.086 % 90.210 ± 0.136 % + 48 7.5434 ± 0.0911 -0.00335 ± 0.00195 0.05993 ± 0.00076 6.893 ± 0.085 % 90.217 ± 0.134 % + 49 7.5946 ± 0.0908 -0.00316 ± 0.00193 0.05980 ± 0.00075 6.873 ± 0.084 % 90.223 ± 0.133 % + 50 7.5515 ± 0.0893 -0.00260 ± 0.00191 0.05953 ± 0.00074 6.855 ± 0.083 % 90.260 ± 0.131 % + 51 7.5800 ± 0.0887 -0.00306 ± 0.00188 0.05920 ± 0.00073 6.834 ± 0.082 % 90.294 ± 0.130 % + 52 7.5216 ± 0.0870 -0.00239 ± 0.00187 0.05904 ± 0.00073 6.832 ± 0.082 % 90.324 ± 0.128 % + 53 7.4910 ± 0.0857 -0.00207 ± 0.00184 0.05867 ± 0.00072 6.809 ± 0.080 % 90.335 ± 0.127 % + 54 7.4497 ± 0.0843 -0.00201 ± 0.00182 0.05829 ± 0.00071 6.782 ± 0.079 % 90.364 ± 0.126 % + 55 7.4078 ± 0.0830 -0.00262 ± 0.00180 0.05809 ± 0.00070 6.777 ± 0.079 % 90.376 ± 0.124 % + 56 7.4081 ± 0.0823 -0.00171 ± 0.00179 0.05799 ± 0.00069 6.768 ± 0.078 % 90.370 ± 0.123 % + 57 7.3295 ± 0.0805 -0.00164 ± 0.00177 0.05764 ± 0.00068 6.750 ± 0.077 % 90.408 ± 0.122 % + 58 7.3808 ± 0.0806 0.00054 ± 0.00178 0.05854 ± 0.00068 6.782 ± 0.076 % 90.350 ± 0.121 % + 59 7.3512 ± 0.0795 0.00051 ± 0.00177 0.05897 ± 0.00068 6.809 ± 0.076 % 90.331 ± 0.120 % + 60 7.3329 ± 0.0787 0.00037 ± 0.00175 0.05879 ± 0.00068 6.804 ± 0.075 % 90.367 ± 0.119 % + 61 7.4050 ± 0.0792 -0.00069 ± 0.00175 0.05935 ± 0.00068 6.831 ± 0.074 % 90.295 ± 0.119 % + 62 7.4278 ± 0.0787 -0.00064 ± 0.00173 0.05907 ± 0.00067 6.801 ± 0.074 % 90.291 ± 0.118 % + 63 7.4789 ± 0.0788 0.00018 ± 0.00172 0.05876 ± 0.00066 6.783 ± 0.073 % 90.302 ± 0.117 % + 64 7.5098 ± 0.0785 0.00011 ± 0.00170 0.05868 ± 0.00065 6.772 ± 0.072 % 90.306 ± 0.116 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.509838 ± 0.078504 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 98.70% +Mean ln(PPL(Q)/PPL(base)) : 0.000106 ± 0.001698 +Mean PPL(Q)/PPL(base) : 1.000106 ± 0.001698 +Mean PPL(Q)-PPL(base) : 0.000795 ± 0.012750 + +====== KL divergence statistics ====== +Mean KLD: 0.058676 ± 0.000655 +Maximum KLD: 7.914699 +99.9% KLD: 2.201383 +99.0% KLD: 0.616340 +95.0% KLD: 0.212113 +90.0% KLD: 0.124513 +Median KLD: 0.022391 +10.0% KLD: 0.000394 + 5.0% KLD: 0.000121 + 1.0% KLD: 0.000005 + 0.1% KLD: -0.000170 +Minimum KLD: -0.000789 + +====== Token probability statistics ====== +Mean Δp: -0.653 ± 0.026 % +Maximum Δp: 94.651% +99.9% Δp: 48.669% +99.0% Δp: 18.822% +95.0% Δp: 6.933% +90.0% Δp: 3.611% +75.0% Δp: 0.457% +Median Δp: -0.021% +25.0% Δp: -1.462% +10.0% Δp: -5.907% + 5.0% Δp: -10.087% + 1.0% Δp: -23.148% + 0.1% Δp: -51.657% +Minimum Δp: -95.869% +RMS Δp : 6.772 ± 0.072 % +Same top p: 90.306 ± 0.116 % + diff --git a/appendix/measurements/kld_SHALLOW-21G-ICE.txt b/appendix/measurements/kld_SHALLOW-21G-ICE.txt new file mode 100644 index 0000000000000000000000000000000000000000..340d7fa73ddb38d1f00dacc18ca9a255ae7b68f2 --- /dev/null +++ b/appendix/measurements/kld_SHALLOW-21G-ICE.txt @@ -0,0 +1,111 @@ +0.05.052.902 I +0.05.053.047 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.05.375.423 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.07.258.596 I kl_divergence: 1.88 seconds per pass - ETA 2.00 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.7187 ± 0.4467 -0.02398 ± 0.01286 0.05730 ± 0.00449 7.091 ± 0.508 % 91.496 ± 0.873 % + 2 7.1994 ± 0.4166 -0.01507 ± 0.00876 0.04875 ± 0.00267 6.045 ± 0.324 % 91.300 ± 0.623 % + 3 8.1447 ± 0.3993 -0.00664 ± 0.00720 0.04609 ± 0.00263 5.757 ± 0.283 % 91.724 ± 0.497 % + 4 9.1860 ± 0.4158 -0.01460 ± 0.00661 0.05211 ± 0.00337 6.261 ± 0.307 % 91.471 ± 0.437 % + 5 8.5931 ± 0.3425 -0.01378 ± 0.00587 0.05031 ± 0.00278 6.182 ± 0.276 % 91.437 ± 0.391 % + 6 7.2779 ± 0.2560 -0.01727 ± 0.00531 0.05000 ± 0.00245 6.267 ± 0.240 % 91.447 ± 0.357 % + 7 6.6583 ± 0.2131 -0.02069 ± 0.00498 0.05019 ± 0.00222 6.442 ± 0.228 % 91.524 ± 0.329 % + 8 6.4719 ± 0.1912 -0.01958 ± 0.00450 0.04837 ± 0.00196 6.274 ± 0.207 % 91.483 ± 0.309 % + 9 6.8339 ± 0.1917 -0.02347 ± 0.00423 0.04818 ± 0.00176 6.169 ± 0.189 % 91.257 ± 0.294 % + 10 6.9068 ± 0.1840 -0.02200 ± 0.00393 0.04623 ± 0.00160 6.004 ± 0.177 % 91.457 ± 0.276 % + 11 6.9259 ± 0.1750 -0.01976 ± 0.00372 0.04449 ± 0.00146 5.863 ± 0.166 % 91.567 ± 0.262 % + 12 7.1884 ± 0.1750 -0.01882 ± 0.00347 0.04272 ± 0.00134 5.714 ± 0.157 % 91.764 ± 0.248 % + 13 7.2835 ± 0.1709 -0.02094 ± 0.00333 0.04262 ± 0.00127 5.743 ± 0.151 % 91.781 ± 0.238 % + 14 7.3297 ± 0.1654 -0.02096 ± 0.00315 0.04144 ± 0.00118 5.624 ± 0.144 % 91.824 ± 0.229 % + 15 7.3780 ± 0.1609 -0.02218 ± 0.00304 0.04086 ± 0.00111 5.573 ± 0.136 % 91.834 ± 0.221 % + 16 7.5547 ± 0.1597 -0.02327 ± 0.00293 0.04049 ± 0.00105 5.534 ± 0.133 % 91.850 ± 0.214 % + 17 7.5715 ± 0.1545 -0.02200 ± 0.00280 0.03978 ± 0.00100 5.454 ± 0.128 % 91.881 ± 0.207 % + 18 7.6577 ± 0.1517 -0.02091 ± 0.00269 0.03940 ± 0.00095 5.417 ± 0.122 % 91.941 ± 0.201 % + 19 7.5722 ± 0.1463 -0.02274 ± 0.00262 0.03947 ± 0.00092 5.393 ± 0.117 % 91.938 ± 0.195 % + 20 7.3024 ± 0.1368 -0.02662 ± 0.00273 0.04421 ± 0.00103 5.971 ± 0.128 % 91.774 ± 0.192 % + 21 7.3335 ± 0.1340 -0.02781 ± 0.00266 0.04475 ± 0.00101 6.014 ± 0.126 % 91.719 ± 0.188 % + 22 7.4193 ± 0.1331 -0.02621 ± 0.00258 0.04431 ± 0.00096 5.981 ± 0.122 % 91.802 ± 0.183 % + 23 7.4548 ± 0.1306 -0.02494 ± 0.00251 0.04384 ± 0.00093 5.917 ± 0.118 % 91.793 ± 0.179 % + 24 7.5356 ± 0.1295 -0.02775 ± 0.00248 0.04429 ± 0.00090 5.932 ± 0.116 % 91.760 ± 0.175 % + 25 7.5857 ± 0.1279 -0.02954 ± 0.00244 0.04457 ± 0.00087 5.898 ± 0.112 % 91.644 ± 0.173 % + 26 7.5398 ± 0.1244 -0.02898 ± 0.00238 0.04421 ± 0.00084 5.864 ± 0.109 % 91.665 ± 0.169 % + 27 7.6883 ± 0.1248 -0.02917 ± 0.00232 0.04381 ± 0.00082 5.819 ± 0.106 % 91.630 ± 0.167 % + 28 7.7645 ± 0.1239 -0.02905 ± 0.00226 0.04333 ± 0.00079 5.770 ± 0.104 % 91.656 ± 0.163 % + 29 7.7683 ± 0.1220 -0.02812 ± 0.00223 0.04334 ± 0.00077 5.773 ± 0.101 % 91.728 ± 0.160 % + 30 7.7975 ± 0.1208 -0.03040 ± 0.00220 0.04341 ± 0.00075 5.768 ± 0.099 % 91.724 ± 0.157 % + 31 7.6986 ± 0.1170 -0.02927 ± 0.00215 0.04296 ± 0.00073 5.743 ± 0.097 % 91.789 ± 0.154 % + 32 7.5790 ± 0.1133 -0.02730 ± 0.00212 0.04288 ± 0.00072 5.793 ± 0.096 % 91.810 ± 0.152 % + 33 7.5742 ± 0.1117 -0.02584 ± 0.00209 0.04298 ± 0.00073 5.784 ± 0.094 % 91.810 ± 0.149 % + 34 7.5555 ± 0.1096 -0.02522 ± 0.00204 0.04256 ± 0.00071 5.760 ± 0.092 % 91.820 ± 0.147 % + 35 7.5567 ± 0.1080 -0.02485 ± 0.00200 0.04222 ± 0.00070 5.733 ± 0.091 % 91.864 ± 0.144 % + 36 7.5638 ± 0.1067 -0.02460 ± 0.00197 0.04191 ± 0.00068 5.709 ± 0.089 % 91.919 ± 0.142 % + 37 7.4249 ± 0.1028 -0.02395 ± 0.00192 0.04132 ± 0.00066 5.677 ± 0.088 % 91.984 ± 0.140 % + 38 7.3176 ± 0.0995 -0.02356 ± 0.00189 0.04106 ± 0.00065 5.660 ± 0.086 % 92.015 ± 0.137 % + 39 7.2091 ± 0.0964 -0.02261 ± 0.00185 0.04066 ± 0.00064 5.636 ± 0.085 % 92.062 ± 0.135 % + 40 7.0864 ± 0.0930 -0.02202 ± 0.00181 0.04022 ± 0.00062 5.623 ± 0.083 % 92.094 ± 0.133 % + 41 7.0101 ± 0.0905 -0.02145 ± 0.00178 0.03991 ± 0.00061 5.615 ± 0.082 % 92.096 ± 0.132 % + 42 7.0425 ± 0.0899 -0.02100 ± 0.00175 0.03975 ± 0.00060 5.592 ± 0.081 % 92.070 ± 0.130 % + 43 7.2332 ± 0.0921 -0.02176 ± 0.00173 0.03974 ± 0.00059 5.579 ± 0.080 % 91.998 ± 0.129 % + 44 7.3712 ± 0.0933 -0.02159 ± 0.00171 0.03970 ± 0.00059 5.548 ± 0.079 % 91.989 ± 0.128 % + 45 7.3470 ± 0.0918 -0.02091 ± 0.00171 0.04053 ± 0.00058 5.598 ± 0.078 % 91.928 ± 0.127 % + 46 7.3511 ± 0.0908 -0.02096 ± 0.00169 0.04095 ± 0.00058 5.639 ± 0.078 % 91.901 ± 0.126 % + 47 7.4146 ± 0.0908 -0.02117 ± 0.00167 0.04121 ± 0.00059 5.655 ± 0.077 % 91.889 ± 0.125 % + 48 7.4139 ± 0.0897 -0.02067 ± 0.00165 0.04098 ± 0.00058 5.632 ± 0.076 % 91.864 ± 0.123 % + 49 7.4637 ± 0.0895 -0.02055 ± 0.00163 0.04098 ± 0.00058 5.615 ± 0.075 % 91.835 ± 0.122 % + 50 7.4193 ± 0.0879 -0.02026 ± 0.00161 0.04066 ± 0.00057 5.595 ± 0.074 % 91.891 ± 0.121 % + 51 7.4527 ± 0.0874 -0.02000 ± 0.00158 0.04027 ± 0.00056 5.557 ± 0.073 % 91.925 ± 0.119 % + 52 7.3940 ± 0.0857 -0.01951 ± 0.00156 0.04001 ± 0.00055 5.553 ± 0.072 % 91.941 ± 0.118 % + 53 7.3647 ± 0.0845 -0.01907 ± 0.00154 0.03979 ± 0.00054 5.542 ± 0.072 % 91.946 ± 0.117 % + 54 7.3242 ± 0.0830 -0.01900 ± 0.00153 0.03953 ± 0.00053 5.531 ± 0.071 % 91.954 ± 0.116 % + 55 7.2811 ± 0.0817 -0.01987 ± 0.00152 0.03956 ± 0.00053 5.541 ± 0.070 % 91.956 ± 0.115 % + 56 7.2765 ± 0.0809 -0.01963 ± 0.00150 0.03942 ± 0.00052 5.530 ± 0.069 % 91.981 ± 0.113 % + 57 7.2012 ± 0.0792 -0.01930 ± 0.00149 0.03919 ± 0.00051 5.507 ± 0.068 % 92.024 ± 0.112 % + 58 7.2425 ± 0.0792 -0.01838 ± 0.00149 0.03954 ± 0.00051 5.530 ± 0.067 % 92.001 ± 0.111 % + 59 7.2161 ± 0.0782 -0.01805 ± 0.00148 0.03986 ± 0.00051 5.564 ± 0.067 % 91.991 ± 0.110 % + 60 7.2017 ± 0.0774 -0.01768 ± 0.00146 0.03970 ± 0.00050 5.551 ± 0.066 % 92.022 ± 0.109 % + 61 7.2691 ± 0.0778 -0.01921 ± 0.00147 0.04039 ± 0.00053 5.600 ± 0.067 % 91.973 ± 0.109 % + 62 7.2945 ± 0.0774 -0.01874 ± 0.00145 0.04018 ± 0.00052 5.576 ± 0.066 % 91.995 ± 0.108 % + 63 7.3445 ± 0.0774 -0.01795 ± 0.00144 0.04000 ± 0.00051 5.554 ± 0.066 % 92.001 ± 0.107 % + 64 7.3762 ± 0.0772 -0.01785 ± 0.00142 0.03981 ± 0.00051 5.535 ± 0.065 % 92.012 ± 0.106 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.376193 ± 0.077216 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 99.09% +Mean ln(PPL(Q)/PPL(base)) : -0.017850 ± 0.001420 +Mean PPL(Q)/PPL(base) : 0.982308 ± 0.001395 +Mean PPL(Q)-PPL(base) : -0.132850 ± 0.010757 + +====== KL divergence statistics ====== +Mean KLD: 0.039814 ± 0.000507 +Maximum KLD: 7.651754 +99.9% KLD: 1.589447 +99.0% KLD: 0.412079 +95.0% KLD: 0.143817 +90.0% KLD: 0.085509 +Median KLD: 0.013917 +10.0% KLD: 0.000220 + 5.0% KLD: 0.000059 + 1.0% KLD: -0.000000 + 0.1% KLD: -0.000154 +Minimum KLD: -0.000922 + +====== Token probability statistics ====== +Mean Δp: -0.062 ± 0.022 % +Maximum Δp: 88.205% +99.9% Δp: 43.914% +99.0% Δp: 17.167% +95.0% Δp: 6.679% +90.0% Δp: 3.536% +75.0% Δp: 0.613% +Median Δp: -0.001% +25.0% Δp: -0.769% +10.0% Δp: -3.899% + 5.0% Δp: -6.876% + 1.0% Δp: -16.580% + 0.1% Δp: -43.277% +Minimum Δp: -82.493% +RMS Δp : 5.535 ± 0.065 % +Same top p: 92.012 ± 0.106 % + diff --git a/appendix/measurements/kld_SHALLOW-23G-ICE.txt b/appendix/measurements/kld_SHALLOW-23G-ICE.txt new file mode 100644 index 0000000000000000000000000000000000000000..580e6b9789b3cedbbd8c53447667d6f669d5dc08 --- /dev/null +++ b/appendix/measurements/kld_SHALLOW-23G-ICE.txt @@ -0,0 +1,111 @@ +0.05.237.699 I +0.05.237.836 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.05.547.049 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.07.354.412 I kl_divergence: 1.81 seconds per pass - ETA 1.92 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.8750 ± 0.4631 0.00298 ± 0.01120 0.04176 ± 0.00334 6.156 ± 0.447 % 92.766 ± 0.810 % + 2 7.3620 ± 0.4290 0.00727 ± 0.00810 0.03874 ± 0.00206 5.421 ± 0.292 % 91.789 ± 0.607 % + 3 8.3347 ± 0.4119 0.01643 ± 0.00637 0.03639 ± 0.00261 5.195 ± 0.332 % 92.147 ± 0.486 % + 4 9.4556 ± 0.4321 0.01432 ± 0.00602 0.04304 ± 0.00328 5.649 ± 0.317 % 92.058 ± 0.423 % + 5 8.8320 ± 0.3554 0.01365 ± 0.00527 0.04060 ± 0.00266 5.441 ± 0.270 % 92.317 ± 0.372 % + 6 7.4854 ± 0.2659 0.01085 ± 0.00474 0.03951 ± 0.00228 5.542 ± 0.248 % 92.212 ± 0.342 % + 7 6.8518 ± 0.2216 0.00795 ± 0.00443 0.03993 ± 0.00204 5.738 ± 0.232 % 92.417 ± 0.313 % + 8 6.6513 ± 0.1985 0.00776 ± 0.00404 0.03851 ± 0.00180 5.620 ± 0.210 % 92.559 ± 0.290 % + 9 7.0608 ± 0.2005 0.00919 ± 0.00381 0.03937 ± 0.00168 5.553 ± 0.191 % 92.310 ± 0.278 % + 10 7.1294 ± 0.1924 0.00973 ± 0.00353 0.03763 ± 0.00152 5.450 ± 0.182 % 92.356 ± 0.263 % + 11 7.1238 ± 0.1819 0.00841 ± 0.00334 0.03628 ± 0.00139 5.347 ± 0.170 % 92.598 ± 0.247 % + 12 7.3873 ± 0.1818 0.00848 ± 0.00315 0.03516 ± 0.00129 5.248 ± 0.159 % 92.669 ± 0.235 % + 13 7.4601 ± 0.1766 0.00302 ± 0.00304 0.03513 ± 0.00120 5.265 ± 0.151 % 92.631 ± 0.227 % + 14 7.5034 ± 0.1707 0.00245 ± 0.00287 0.03400 ± 0.00112 5.141 ± 0.144 % 92.655 ± 0.218 % + 15 7.5494 ± 0.1660 0.00077 ± 0.00274 0.03380 ± 0.00106 5.115 ± 0.137 % 92.708 ± 0.210 % + 16 7.7246 ± 0.1645 -0.00102 ± 0.00267 0.03350 ± 0.00100 5.045 ± 0.131 % 92.797 ± 0.202 % + 17 7.7325 ± 0.1589 -0.00096 ± 0.00254 0.03270 ± 0.00094 4.964 ± 0.125 % 92.853 ± 0.195 % + 18 7.8068 ± 0.1556 -0.00162 ± 0.00246 0.03242 ± 0.00090 4.933 ± 0.120 % 92.837 ± 0.190 % + 19 7.7138 ± 0.1499 -0.00421 ± 0.00241 0.03265 ± 0.00086 4.929 ± 0.115 % 92.782 ± 0.186 % + 20 7.4622 ± 0.1409 -0.00498 ± 0.00247 0.03619 ± 0.00104 5.392 ± 0.124 % 92.542 ± 0.184 % + 21 7.5061 ± 0.1383 -0.00454 ± 0.00240 0.03664 ± 0.00102 5.412 ± 0.122 % 92.417 ± 0.181 % + 22 7.5743 ± 0.1368 -0.00553 ± 0.00233 0.03625 ± 0.00097 5.369 ± 0.117 % 92.442 ± 0.176 % + 23 7.6062 ± 0.1341 -0.00483 ± 0.00226 0.03582 ± 0.00093 5.309 ± 0.114 % 92.490 ± 0.172 % + 24 7.6949 ± 0.1331 -0.00683 ± 0.00222 0.03605 ± 0.00090 5.307 ± 0.111 % 92.485 ± 0.168 % + 25 7.7564 ± 0.1318 -0.00729 ± 0.00216 0.03579 ± 0.00087 5.262 ± 0.108 % 92.454 ± 0.165 % + 26 7.7098 ± 0.1282 -0.00668 ± 0.00210 0.03543 ± 0.00084 5.230 ± 0.105 % 92.492 ± 0.162 % + 27 7.8702 ± 0.1288 -0.00579 ± 0.00204 0.03479 ± 0.00081 5.168 ± 0.102 % 92.498 ± 0.159 % + 28 7.9397 ± 0.1277 -0.00674 ± 0.00199 0.03460 ± 0.00078 5.142 ± 0.100 % 92.519 ± 0.155 % + 29 7.9374 ± 0.1257 -0.00659 ± 0.00196 0.03463 ± 0.00076 5.159 ± 0.097 % 92.564 ± 0.152 % + 30 7.9846 ± 0.1248 -0.00670 ± 0.00194 0.03472 ± 0.00076 5.161 ± 0.095 % 92.577 ± 0.150 % + 31 7.8743 ± 0.1208 -0.00672 ± 0.00190 0.03438 ± 0.00074 5.133 ± 0.093 % 92.612 ± 0.147 % + 32 7.7400 ± 0.1166 -0.00629 ± 0.00187 0.03427 ± 0.00072 5.166 ± 0.091 % 92.675 ± 0.144 % + 33 7.7294 ± 0.1149 -0.00555 ± 0.00185 0.03445 ± 0.00075 5.167 ± 0.089 % 92.669 ± 0.142 % + 34 7.7030 ± 0.1126 -0.00589 ± 0.00181 0.03420 ± 0.00073 5.161 ± 0.088 % 92.660 ± 0.140 % + 35 7.6976 ± 0.1108 -0.00638 ± 0.00177 0.03394 ± 0.00071 5.135 ± 0.086 % 92.677 ± 0.138 % + 36 7.7021 ± 0.1094 -0.00647 ± 0.00174 0.03362 ± 0.00069 5.099 ± 0.084 % 92.737 ± 0.135 % + 37 7.5595 ± 0.1054 -0.00600 ± 0.00170 0.03321 ± 0.00067 5.070 ± 0.083 % 92.822 ± 0.133 % + 38 7.4468 ± 0.1019 -0.00606 ± 0.00167 0.03298 ± 0.00066 5.056 ± 0.082 % 92.874 ± 0.130 % + 39 7.3310 ± 0.0987 -0.00585 ± 0.00164 0.03260 ± 0.00064 5.028 ± 0.080 % 92.902 ± 0.129 % + 40 7.2031 ± 0.0952 -0.00567 ± 0.00161 0.03223 ± 0.00063 5.014 ± 0.079 % 92.945 ± 0.127 % + 41 7.1238 ± 0.0926 -0.00537 ± 0.00159 0.03195 ± 0.00061 5.002 ± 0.077 % 92.969 ± 0.125 % + 42 7.1531 ± 0.0920 -0.00541 ± 0.00156 0.03181 ± 0.00060 4.977 ± 0.076 % 92.974 ± 0.123 % + 43 7.3515 ± 0.0943 -0.00554 ± 0.00155 0.03181 ± 0.00059 4.955 ± 0.075 % 92.926 ± 0.122 % + 44 7.4877 ± 0.0954 -0.00591 ± 0.00153 0.03191 ± 0.00059 4.936 ± 0.074 % 92.909 ± 0.121 % + 45 7.4600 ± 0.0938 -0.00564 ± 0.00152 0.03275 ± 0.00059 5.005 ± 0.073 % 92.825 ± 0.120 % + 46 7.4668 ± 0.0928 -0.00535 ± 0.00151 0.03291 ± 0.00058 5.018 ± 0.072 % 92.815 ± 0.119 % + 47 7.5311 ± 0.0928 -0.00559 ± 0.00150 0.03309 ± 0.00058 5.043 ± 0.074 % 92.825 ± 0.118 % + 48 7.5285 ± 0.0917 -0.00534 ± 0.00147 0.03293 ± 0.00057 5.034 ± 0.073 % 92.817 ± 0.117 % + 49 7.5763 ± 0.0914 -0.00557 ± 0.00146 0.03304 ± 0.00057 5.039 ± 0.073 % 92.738 ± 0.116 % + 50 7.5310 ± 0.0898 -0.00533 ± 0.00144 0.03279 ± 0.00056 5.013 ± 0.072 % 92.770 ± 0.115 % + 51 7.5658 ± 0.0893 -0.00495 ± 0.00142 0.03248 ± 0.00055 4.979 ± 0.071 % 92.803 ± 0.113 % + 52 7.5013 ± 0.0875 -0.00510 ± 0.00141 0.03241 ± 0.00055 4.981 ± 0.070 % 92.825 ± 0.112 % + 53 7.4720 ± 0.0862 -0.00461 ± 0.00139 0.03215 ± 0.00054 4.964 ± 0.069 % 92.824 ± 0.111 % + 54 7.4284 ± 0.0847 -0.00488 ± 0.00137 0.03208 ± 0.00053 4.975 ± 0.069 % 92.815 ± 0.110 % + 55 7.3884 ± 0.0834 -0.00525 ± 0.00136 0.03198 ± 0.00052 4.970 ± 0.068 % 92.820 ± 0.109 % + 56 7.3790 ± 0.0826 -0.00565 ± 0.00135 0.03193 ± 0.00052 4.962 ± 0.067 % 92.827 ± 0.108 % + 57 7.3003 ± 0.0807 -0.00563 ± 0.00133 0.03178 ± 0.00051 4.945 ± 0.066 % 92.856 ± 0.107 % + 58 7.3455 ± 0.0808 -0.00426 ± 0.00134 0.03247 ± 0.00052 4.976 ± 0.065 % 92.793 ± 0.106 % + 59 7.3188 ± 0.0797 -0.00391 ± 0.00133 0.03268 ± 0.00051 5.008 ± 0.065 % 92.801 ± 0.105 % + 60 7.3013 ± 0.0789 -0.00394 ± 0.00131 0.03250 ± 0.00051 4.998 ± 0.064 % 92.833 ± 0.104 % + 61 7.3721 ± 0.0793 -0.00514 ± 0.00132 0.03303 ± 0.00051 5.059 ± 0.066 % 92.797 ± 0.103 % + 62 7.3946 ± 0.0789 -0.00511 ± 0.00130 0.03287 ± 0.00051 5.037 ± 0.065 % 92.809 ± 0.103 % + 63 7.4393 ± 0.0788 -0.00513 ± 0.00129 0.03266 ± 0.00050 5.017 ± 0.064 % 92.811 ± 0.102 % + 64 7.4707 ± 0.0786 -0.00512 ± 0.00128 0.03254 ± 0.00049 4.995 ± 0.064 % 92.835 ± 0.101 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.470682 ± 0.078596 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 99.27% +Mean ln(PPL(Q)/PPL(base)) : -0.005122 ± 0.001276 +Mean PPL(Q)/PPL(base) : 0.994891 ± 0.001269 +Mean PPL(Q)-PPL(base) : -0.038361 ± 0.009580 + +====== KL divergence statistics ====== +Mean KLD: 0.032536 ± 0.000493 +Maximum KLD: 9.080469 +99.9% KLD: 1.318512 +99.0% KLD: 0.331990 +95.0% KLD: 0.113636 +90.0% KLD: 0.068610 +Median KLD: 0.011154 +10.0% KLD: 0.000183 + 5.0% KLD: 0.000051 + 1.0% KLD: 0.000001 + 0.1% KLD: -0.000143 +Minimum KLD: -0.000774 + +====== Token probability statistics ====== +Mean Δp: -0.122 ± 0.020 % +Maximum Δp: 90.050% +99.9% Δp: 38.701% +99.0% Δp: 14.926% +95.0% Δp: 5.790% +90.0% Δp: 3.062% +75.0% Δp: 0.512% +Median Δp: -0.002% +25.0% Δp: -0.716% +10.0% Δp: -3.562% + 5.0% Δp: -6.422% + 1.0% Δp: -15.571% + 0.1% Δp: -38.825% +Minimum Δp: -91.432% +RMS Δp : 4.995 ± 0.064 % +Same top p: 92.835 ± 0.101 % + diff --git a/appendix/measurements/kld_SHALLOW-25G-ICE.txt b/appendix/measurements/kld_SHALLOW-25G-ICE.txt new file mode 100644 index 0000000000000000000000000000000000000000..56cd078f1f99f9e2da6004e3e5fd1c33d0120734 --- /dev/null +++ b/appendix/measurements/kld_SHALLOW-25G-ICE.txt @@ -0,0 +1,111 @@ +0.05.931.717 I +0.05.931.865 I system_info: n_threads = 128 (n_threads_batch = 128) / 128 | CUDA : ARCHS = 610,700,750,800,860,890,900,1000,1200 | USE_GRAPHS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.06.304.675 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.08.330.832 I kl_divergence: 2.03 seconds per pass - ETA 2.15 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.7739 ± 0.4525 -0.01437 ± 0.01007 0.02728 ± 0.00224 4.890 ± 0.409 % 94.330 ± 0.723 % + 2 7.2217 ± 0.4183 -0.01197 ± 0.00735 0.02713 ± 0.00138 4.572 ± 0.263 % 93.891 ± 0.530 % + 3 8.1465 ± 0.3994 -0.00641 ± 0.00600 0.02747 ± 0.00232 4.585 ± 0.340 % 93.939 ± 0.431 % + 4 9.2937 ± 0.4232 -0.00295 ± 0.00569 0.03424 ± 0.00280 5.137 ± 0.326 % 93.695 ± 0.380 % + 5 8.6837 ± 0.3478 -0.00329 ± 0.00489 0.03231 ± 0.00226 4.994 ± 0.279 % 93.744 ± 0.339 % + 6 7.3875 ± 0.2614 -0.00233 ± 0.00441 0.03176 ± 0.00195 5.139 ± 0.261 % 93.516 ± 0.314 % + 7 6.7593 ± 0.2177 -0.00563 ± 0.00416 0.03245 ± 0.00174 5.339 ± 0.232 % 93.451 ± 0.292 % + 8 6.5700 ± 0.1952 -0.00453 ± 0.00375 0.03102 ± 0.00153 5.216 ± 0.210 % 93.524 ± 0.272 % + 9 6.9545 ± 0.1965 -0.00598 ± 0.00357 0.03163 ± 0.00143 5.122 ± 0.192 % 93.331 ± 0.260 % + 10 7.0245 ± 0.1886 -0.00510 ± 0.00331 0.03035 ± 0.00130 5.029 ± 0.181 % 93.451 ± 0.245 % + 11 7.0327 ± 0.1789 -0.00446 ± 0.00316 0.02958 ± 0.00119 4.999 ± 0.173 % 93.486 ± 0.233 % + 12 7.2961 ± 0.1788 -0.00394 ± 0.00297 0.02883 ± 0.00110 4.891 ± 0.163 % 93.573 ± 0.221 % + 13 7.3748 ± 0.1740 -0.00849 ± 0.00287 0.02906 ± 0.00104 4.913 ± 0.153 % 93.518 ± 0.214 % + 14 7.4256 ± 0.1684 -0.00797 ± 0.00271 0.02828 ± 0.00097 4.831 ± 0.145 % 93.493 ± 0.206 % + 15 7.4745 ± 0.1638 -0.00919 ± 0.00260 0.02824 ± 0.00091 4.813 ± 0.137 % 93.470 ± 0.199 % + 16 7.6545 ± 0.1625 -0.01014 ± 0.00252 0.02806 ± 0.00086 4.751 ± 0.131 % 93.518 ± 0.192 % + 17 7.6653 ± 0.1571 -0.00968 ± 0.00240 0.02750 ± 0.00082 4.680 ± 0.125 % 93.537 ± 0.186 % + 18 7.7474 ± 0.1541 -0.00927 ± 0.00231 0.02714 ± 0.00077 4.632 ± 0.120 % 93.586 ± 0.181 % + 19 7.6547 ± 0.1484 -0.01190 ± 0.00227 0.02764 ± 0.00075 4.647 ± 0.115 % 93.471 ± 0.177 % + 20 7.4003 ± 0.1394 -0.01330 ± 0.00231 0.03094 ± 0.00099 4.991 ± 0.120 % 93.270 ± 0.175 % + 21 7.4460 ± 0.1368 -0.01258 ± 0.00224 0.03140 ± 0.00097 5.022 ± 0.119 % 93.209 ± 0.172 % + 22 7.5175 ± 0.1355 -0.01305 ± 0.00218 0.03110 ± 0.00093 4.981 ± 0.114 % 93.188 ± 0.168 % + 23 7.5511 ± 0.1329 -0.01210 ± 0.00211 0.03067 ± 0.00089 4.915 ± 0.111 % 93.200 ± 0.164 % + 24 7.6362 ± 0.1318 -0.01448 ± 0.00208 0.03113 ± 0.00086 4.960 ± 0.111 % 93.141 ± 0.161 % + 25 7.6993 ± 0.1306 -0.01468 ± 0.00202 0.03096 ± 0.00083 4.915 ± 0.107 % 93.095 ± 0.159 % + 26 7.6553 ± 0.1270 -0.01377 ± 0.00197 0.03062 ± 0.00080 4.877 ± 0.105 % 93.112 ± 0.155 % + 27 7.8166 ± 0.1277 -0.01262 ± 0.00191 0.03003 ± 0.00077 4.817 ± 0.102 % 93.121 ± 0.152 % + 28 7.8878 ± 0.1267 -0.01329 ± 0.00187 0.03002 ± 0.00075 4.806 ± 0.100 % 93.081 ± 0.150 % + 29 7.8846 ± 0.1246 -0.01326 ± 0.00184 0.02987 ± 0.00073 4.814 ± 0.097 % 93.097 ± 0.147 % + 30 7.9328 ± 0.1238 -0.01321 ± 0.00182 0.03006 ± 0.00073 4.828 ± 0.095 % 93.122 ± 0.144 % + 31 7.8258 ± 0.1198 -0.01289 ± 0.00178 0.02974 ± 0.00071 4.814 ± 0.094 % 93.148 ± 0.142 % + 32 7.6896 ± 0.1157 -0.01282 ± 0.00175 0.02974 ± 0.00069 4.835 ± 0.092 % 93.200 ± 0.139 % + 33 7.6796 ± 0.1140 -0.01201 ± 0.00172 0.02986 ± 0.00071 4.823 ± 0.090 % 93.184 ± 0.137 % + 34 7.6539 ± 0.1117 -0.01227 ± 0.00169 0.02967 ± 0.00069 4.821 ± 0.088 % 93.198 ± 0.135 % + 35 7.6515 ± 0.1100 -0.01239 ± 0.00165 0.02946 ± 0.00067 4.802 ± 0.087 % 93.216 ± 0.133 % + 36 7.6531 ± 0.1085 -0.01285 ± 0.00162 0.02928 ± 0.00066 4.788 ± 0.085 % 93.258 ± 0.131 % + 37 7.5110 ± 0.1045 -0.01242 ± 0.00159 0.02887 ± 0.00064 4.767 ± 0.084 % 93.316 ± 0.128 % + 38 7.4013 ± 0.1012 -0.01219 ± 0.00156 0.02862 ± 0.00062 4.749 ± 0.083 % 93.340 ± 0.126 % + 39 7.2877 ± 0.0979 -0.01177 ± 0.00153 0.02830 ± 0.00061 4.732 ± 0.081 % 93.378 ± 0.124 % + 40 7.1606 ± 0.0945 -0.01159 ± 0.00150 0.02794 ± 0.00059 4.713 ± 0.079 % 93.419 ± 0.123 % + 41 7.0829 ± 0.0919 -0.01113 ± 0.00148 0.02763 ± 0.00058 4.691 ± 0.078 % 93.448 ± 0.121 % + 42 7.1135 ± 0.0913 -0.01096 ± 0.00145 0.02752 ± 0.00057 4.677 ± 0.077 % 93.462 ± 0.119 % + 43 7.3094 ± 0.0936 -0.01128 ± 0.00144 0.02760 ± 0.00056 4.667 ± 0.076 % 93.421 ± 0.118 % + 44 7.4410 ± 0.0947 -0.01217 ± 0.00143 0.02778 ± 0.00057 4.651 ± 0.074 % 93.388 ± 0.117 % + 45 7.4161 ± 0.0931 -0.01155 ± 0.00142 0.02838 ± 0.00056 4.702 ± 0.073 % 93.329 ± 0.116 % + 46 7.4236 ± 0.0921 -0.01115 ± 0.00140 0.02848 ± 0.00055 4.714 ± 0.072 % 93.332 ± 0.115 % + 47 7.4851 ± 0.0921 -0.01171 ± 0.00139 0.02864 ± 0.00055 4.730 ± 0.073 % 93.320 ± 0.114 % + 48 7.4839 ± 0.0909 -0.01127 ± 0.00137 0.02849 ± 0.00054 4.717 ± 0.072 % 93.302 ± 0.113 % + 49 7.5329 ± 0.0907 -0.01132 ± 0.00136 0.02862 ± 0.00054 4.718 ± 0.071 % 93.267 ± 0.112 % + 50 7.4882 ± 0.0891 -0.01103 ± 0.00134 0.02839 ± 0.00053 4.693 ± 0.070 % 93.302 ± 0.111 % + 51 7.5201 ± 0.0886 -0.01100 ± 0.00132 0.02813 ± 0.00052 4.664 ± 0.069 % 93.351 ± 0.109 % + 52 7.4541 ± 0.0868 -0.01142 ± 0.00130 0.02805 ± 0.00052 4.677 ± 0.069 % 93.349 ± 0.108 % + 53 7.4245 ± 0.0855 -0.01098 ± 0.00128 0.02786 ± 0.00051 4.663 ± 0.068 % 93.349 ± 0.107 % + 54 7.3803 ± 0.0840 -0.01137 ± 0.00127 0.02777 ± 0.00050 4.669 ± 0.068 % 93.347 ± 0.106 % + 55 7.3414 ± 0.0827 -0.01162 ± 0.00126 0.02769 ± 0.00049 4.665 ± 0.067 % 93.351 ± 0.105 % + 56 7.3326 ± 0.0819 -0.01195 ± 0.00125 0.02764 ± 0.00049 4.657 ± 0.066 % 93.351 ± 0.104 % + 57 7.2551 ± 0.0801 -0.01184 ± 0.00123 0.02749 ± 0.00048 4.642 ± 0.065 % 93.377 ± 0.103 % + 58 7.2994 ± 0.0801 -0.01056 ± 0.00124 0.02799 ± 0.00048 4.663 ± 0.064 % 93.339 ± 0.102 % + 59 7.2727 ± 0.0791 -0.01023 ± 0.00123 0.02811 ± 0.00047 4.674 ± 0.063 % 93.355 ± 0.101 % + 60 7.2575 ± 0.0783 -0.00996 ± 0.00121 0.02799 ± 0.00047 4.670 ± 0.063 % 93.372 ± 0.100 % + 61 7.3299 ± 0.0788 -0.01088 ± 0.00123 0.02861 ± 0.00049 4.737 ± 0.066 % 93.321 ± 0.100 % + 62 7.3529 ± 0.0784 -0.01076 ± 0.00121 0.02848 ± 0.00048 4.718 ± 0.065 % 93.320 ± 0.099 % + 63 7.3985 ± 0.0783 -0.01063 ± 0.00120 0.02829 ± 0.00047 4.698 ± 0.064 % 93.319 ± 0.098 % + 64 7.4300 ± 0.0781 -0.01058 ± 0.00119 0.02813 ± 0.00046 4.678 ± 0.064 % 93.347 ± 0.097 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.429995 ± 0.078071 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 99.37% +Mean ln(PPL(Q)/PPL(base)) : -0.010583 ± 0.001187 +Mean PPL(Q)/PPL(base) : 0.989473 ± 0.001174 +Mean PPL(Q)-PPL(base) : -0.079048 ± 0.008947 + +====== KL divergence statistics ====== +Mean KLD: 0.028131 ± 0.000465 +Maximum KLD: 7.775503 +99.9% KLD: 1.158895 +99.0% KLD: 0.279364 +95.0% KLD: 0.097497 +90.0% KLD: 0.058892 +Median KLD: 0.009599 +10.0% KLD: 0.000162 + 5.0% KLD: 0.000045 + 1.0% KLD: -0.000002 + 0.1% KLD: -0.000138 +Minimum KLD: -0.000465 + +====== Token probability statistics ====== +Mean Δp: -0.081 ± 0.018 % +Maximum Δp: 88.574% +99.9% Δp: 36.405% +99.0% Δp: 14.296% +95.0% Δp: 5.492% +90.0% Δp: 2.904% +75.0% Δp: 0.478% +Median Δp: -0.002% +25.0% Δp: -0.675% +10.0% Δp: -3.318% + 5.0% Δp: -5.794% + 1.0% Δp: -14.093% + 0.1% Δp: -35.491% +Minimum Δp: -95.025% +RMS Δp : 4.678 ± 0.064 % +Same top p: 93.347 ± 0.097 % + diff --git a/appendix/measurements/kld_V2.txt b/appendix/measurements/kld_V2.txt new file mode 100644 index 0000000000000000000000000000000000000000..d49a20e707424b8ddd9564a5a6079d734617ac2d --- /dev/null +++ b/appendix/measurements/kld_V2.txt @@ -0,0 +1,112 @@ +0.17.320.322 I cmn init: llama threadpool init, n_threads = 30 +0.17.399.393 I +0.17.399.582 I system_info: n_threads = 30 (n_threads_batch = 30) / 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.17.749.316 I kl_divergence: computing over 64 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.34.514.199 I kl_divergence: 16.76 seconds per pass - ETA 17.87 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.8556 ± 0.4579 -0.00033 ± 0.01243 0.04597 ± 0.00466 6.745 ± 0.539 % 92.766 ± 0.810 % + 2 7.3205 ± 0.4252 0.00161 ± 0.00799 0.03994 ± 0.00263 5.564 ± 0.337 % 92.669 ± 0.576 % + 3 8.2603 ± 0.4067 0.00745 ± 0.00650 0.03789 ± 0.00277 5.449 ± 0.365 % 93.060 ± 0.459 % + 4 9.3600 ± 0.4263 0.00415 ± 0.00601 0.04408 ± 0.00323 5.675 ± 0.344 % 92.815 ± 0.404 % + 5 8.7404 ± 0.3505 0.00322 ± 0.00533 0.04209 ± 0.00262 5.472 ± 0.292 % 92.708 ± 0.364 % + 6 7.4182 ± 0.2629 0.00182 ± 0.00478 0.04148 ± 0.00227 5.655 ± 0.259 % 92.555 ± 0.335 % + 7 6.8036 ± 0.2197 0.00089 ± 0.00439 0.04090 ± 0.00199 5.702 ± 0.228 % 92.599 ± 0.309 % + 8 6.6056 ± 0.1968 0.00087 ± 0.00403 0.03920 ± 0.00176 5.582 ± 0.207 % 92.620 ± 0.289 % + 9 7.0203 ± 0.1990 0.00343 ± 0.00381 0.04021 ± 0.00162 5.562 ± 0.187 % 92.332 ± 0.277 % + 10 7.0867 ± 0.1909 0.00372 ± 0.00352 0.03846 ± 0.00147 5.410 ± 0.174 % 92.434 ± 0.261 % + 11 7.0841 ± 0.1806 0.00283 ± 0.00335 0.03724 ± 0.00135 5.327 ± 0.163 % 92.544 ± 0.248 % + 12 7.3491 ± 0.1806 0.00330 ± 0.00315 0.03617 ± 0.00124 5.232 ± 0.153 % 92.587 ± 0.236 % + 13 7.4453 ± 0.1762 0.00103 ± 0.00304 0.03572 ± 0.00116 5.257 ± 0.150 % 92.608 ± 0.227 % + 14 7.4843 ± 0.1703 -0.00009 ± 0.00289 0.03492 ± 0.00109 5.178 ± 0.144 % 92.634 ± 0.218 % + 15 7.5395 ± 0.1658 -0.00054 ± 0.00279 0.03445 ± 0.00102 5.136 ± 0.137 % 92.695 ± 0.210 % + 16 7.7470 ± 0.1655 0.00187 ± 0.00270 0.03438 ± 0.00097 5.092 ± 0.131 % 92.711 ± 0.203 % + 17 7.7518 ± 0.1597 0.00153 ± 0.00257 0.03369 ± 0.00092 5.011 ± 0.125 % 92.743 ± 0.197 % + 18 7.8313 ± 0.1566 0.00151 ± 0.00249 0.03335 ± 0.00087 4.987 ± 0.121 % 92.756 ± 0.191 % + 19 7.7506 ± 0.1512 0.00055 ± 0.00240 0.03276 ± 0.00083 4.929 ± 0.116 % 92.818 ± 0.185 % + 20 7.5014 ± 0.1422 0.00027 ± 0.00240 0.03577 ± 0.00096 5.322 ± 0.126 % 92.678 ± 0.182 % + 21 7.5371 ± 0.1393 -0.00042 ± 0.00235 0.03642 ± 0.00096 5.354 ± 0.125 % 92.557 ± 0.179 % + 22 7.6252 ± 0.1383 0.00117 ± 0.00229 0.03615 ± 0.00092 5.350 ± 0.122 % 92.633 ± 0.174 % + 23 7.6521 ± 0.1354 0.00117 ± 0.00222 0.03570 ± 0.00088 5.280 ± 0.118 % 92.673 ± 0.170 % + 24 7.7413 ± 0.1344 -0.00082 ± 0.00219 0.03606 ± 0.00087 5.335 ± 0.118 % 92.603 ± 0.167 % + 25 7.8118 ± 0.1333 -0.00017 ± 0.00213 0.03573 ± 0.00083 5.301 ± 0.116 % 92.559 ± 0.164 % + 26 7.7548 ± 0.1294 -0.00086 ± 0.00209 0.03544 ± 0.00081 5.266 ± 0.112 % 92.590 ± 0.161 % + 27 7.9028 ± 0.1297 -0.00166 ± 0.00204 0.03518 ± 0.00078 5.227 ± 0.110 % 92.535 ± 0.158 % + 28 7.9761 ± 0.1287 -0.00216 ± 0.00199 0.03485 ± 0.00075 5.185 ± 0.107 % 92.539 ± 0.155 % + 29 7.9867 ± 0.1269 -0.00040 ± 0.00198 0.03496 ± 0.00074 5.208 ± 0.104 % 92.520 ± 0.153 % + 30 8.0130 ± 0.1256 -0.00315 ± 0.00197 0.03534 ± 0.00072 5.214 ± 0.101 % 92.506 ± 0.150 % + 31 7.8997 ± 0.1214 -0.00349 ± 0.00194 0.03514 ± 0.00070 5.200 ± 0.099 % 92.549 ± 0.147 % + 32 7.7678 ± 0.1173 -0.00271 ± 0.00190 0.03505 ± 0.00068 5.241 ± 0.098 % 92.611 ± 0.145 % + 33 7.7570 ± 0.1156 -0.00199 ± 0.00188 0.03534 ± 0.00070 5.245 ± 0.096 % 92.580 ± 0.143 % + 34 7.7336 ± 0.1133 -0.00192 ± 0.00184 0.03489 ± 0.00068 5.213 ± 0.094 % 92.588 ± 0.140 % + 35 7.7318 ± 0.1117 -0.00194 ± 0.00180 0.03465 ± 0.00066 5.191 ± 0.092 % 92.582 ± 0.138 % + 36 7.7361 ± 0.1102 -0.00207 ± 0.00177 0.03441 ± 0.00064 5.180 ± 0.090 % 92.625 ± 0.136 % + 37 7.5889 ± 0.1061 -0.00211 ± 0.00173 0.03400 ± 0.00063 5.161 ± 0.090 % 92.655 ± 0.134 % + 38 7.4774 ± 0.1026 -0.00196 ± 0.00169 0.03374 ± 0.00062 5.153 ± 0.088 % 92.715 ± 0.132 % + 39 7.3609 ± 0.0994 -0.00178 ± 0.00166 0.03339 ± 0.00060 5.124 ± 0.086 % 92.726 ± 0.130 % + 40 7.2325 ± 0.0959 -0.00161 ± 0.00163 0.03300 ± 0.00059 5.104 ± 0.085 % 92.759 ± 0.128 % + 41 7.1515 ± 0.0932 -0.00148 ± 0.00161 0.03282 ± 0.00059 5.105 ± 0.085 % 92.743 ± 0.127 % + 42 7.1808 ± 0.0926 -0.00155 ± 0.00158 0.03276 ± 0.00057 5.093 ± 0.083 % 92.738 ± 0.125 % + 43 7.3764 ± 0.0949 -0.00215 ± 0.00157 0.03296 ± 0.00057 5.086 ± 0.082 % 92.680 ± 0.124 % + 44 7.5130 ± 0.0960 -0.00254 ± 0.00155 0.03301 ± 0.00058 5.062 ± 0.081 % 92.675 ± 0.123 % + 45 7.4880 ± 0.0944 -0.00190 ± 0.00155 0.03391 ± 0.00057 5.134 ± 0.079 % 92.619 ± 0.122 % + 46 7.4937 ± 0.0934 -0.00175 ± 0.00154 0.03406 ± 0.00056 5.147 ± 0.078 % 92.588 ± 0.121 % + 47 7.5551 ± 0.0933 -0.00240 ± 0.00152 0.03408 ± 0.00055 5.146 ± 0.077 % 92.577 ± 0.120 % + 48 7.5523 ± 0.0921 -0.00217 ± 0.00150 0.03387 ± 0.00054 5.129 ± 0.076 % 92.573 ± 0.118 % + 49 7.5993 ± 0.0918 -0.00255 ± 0.00148 0.03398 ± 0.00055 5.123 ± 0.075 % 92.545 ± 0.117 % + 50 7.5561 ± 0.0902 -0.00199 ± 0.00146 0.03380 ± 0.00054 5.102 ± 0.074 % 92.573 ± 0.116 % + 51 7.5900 ± 0.0897 -0.00175 ± 0.00144 0.03352 ± 0.00053 5.071 ± 0.073 % 92.602 ± 0.115 % + 52 7.5286 ± 0.0880 -0.00147 ± 0.00142 0.03336 ± 0.00053 5.064 ± 0.072 % 92.625 ± 0.113 % + 53 7.4982 ± 0.0867 -0.00110 ± 0.00140 0.03317 ± 0.00052 5.052 ± 0.071 % 92.630 ± 0.112 % + 54 7.4554 ± 0.0852 -0.00125 ± 0.00139 0.03299 ± 0.00051 5.036 ± 0.071 % 92.632 ± 0.111 % + 55 7.4096 ± 0.0838 -0.00238 ± 0.00138 0.03299 ± 0.00050 5.061 ± 0.072 % 92.638 ± 0.110 % + 56 7.3988 ± 0.0829 -0.00297 ± 0.00137 0.03300 ± 0.00050 5.058 ± 0.071 % 92.651 ± 0.109 % + 57 7.3204 ± 0.0811 -0.00287 ± 0.00135 0.03278 ± 0.00049 5.038 ± 0.070 % 92.670 ± 0.108 % + 58 7.3605 ± 0.0811 -0.00222 ± 0.00136 0.03339 ± 0.00050 5.068 ± 0.069 % 92.643 ± 0.107 % + 59 7.3360 ± 0.0800 -0.00156 ± 0.00135 0.03365 ± 0.00050 5.110 ± 0.069 % 92.621 ± 0.106 % + 60 7.3177 ± 0.0792 -0.00170 ± 0.00134 0.03350 ± 0.00049 5.100 ± 0.068 % 92.641 ± 0.105 % + 61 7.3813 ± 0.0795 -0.00389 ± 0.00135 0.03418 ± 0.00050 5.156 ± 0.069 % 92.582 ± 0.105 % + 62 7.4047 ± 0.0791 -0.00375 ± 0.00133 0.03400 ± 0.00050 5.132 ± 0.068 % 92.591 ± 0.104 % + 63 7.4554 ± 0.0791 -0.00297 ± 0.00132 0.03390 ± 0.00049 5.118 ± 0.067 % 92.582 ± 0.103 % + 64 7.4865 ± 0.0789 -0.00300 ± 0.00131 0.03377 ± 0.00048 5.103 ± 0.067 % 92.589 ± 0.102 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.486549 ± 0.078873 +Mean PPL(base) : 7.509043 ± 0.079361 +Cor(ln(PPL(Q)), ln(PPL(base))): 99.23% +Mean ln(PPL(Q)/PPL(base)) : -0.003000 ± 0.001307 +Mean PPL(Q)/PPL(base) : 0.997004 ± 0.001303 +Mean PPL(Q)-PPL(base) : -0.022494 ± 0.009811 + +====== KL divergence statistics ====== +Mean KLD: 0.033769 ± 0.000483 +Maximum KLD: 6.496775 +99.9% KLD: 1.437661 +99.0% KLD: 0.341397 +95.0% KLD: 0.119217 +90.0% KLD: 0.071562 +Median KLD: 0.011643 +10.0% KLD: 0.000193 + 5.0% KLD: 0.000054 + 1.0% KLD: -0.000000 + 0.1% KLD: -0.000148 +Minimum KLD: -0.000781 + +====== Token probability statistics ====== +Mean Δp: -0.118 ± 0.020 % +Maximum Δp: 92.131% +99.9% Δp: 39.069% +99.0% Δp: 15.335% +95.0% Δp: 5.976% +90.0% Δp: 3.202% +75.0% Δp: 0.514% +Median Δp: -0.002% +25.0% Δp: -0.739% +10.0% Δp: -3.607% + 5.0% Δp: -6.416% + 1.0% Δp: -16.085% + 0.1% Δp: -39.140% +Minimum Δp: -98.490% +RMS Δp : 5.103 ± 0.067 % +Same top p: 92.589 ± 0.102 % + diff --git a/appendix/measurements/kldf_19G-ICE.txt b/appendix/measurements/kldf_19G-ICE.txt new file mode 100644 index 0000000000000000000000000000000000000000..b1884377af6e4bdd34054b58ebc0114ae6e77545 --- /dev/null +++ b/appendix/measurements/kldf_19G-ICE.txt @@ -0,0 +1,84 @@ +0.00.800.239 W model has unused tensor blk.40.attn_norm.weight (size = 8192 bytes) -- ignoring +0.00.800.253 W model has unused tensor blk.40.post_attention_norm.weight (size = 8192 bytes) -- ignoring +0.00.800.259 W model has unused tensor blk.40.attn_q.weight (size = 17825792 bytes) -- ignoring +0.00.800.262 W model has unused tensor blk.40.attn_k.weight (size = 2097152 bytes) -- ignoring +0.00.800.264 W model has unused tensor blk.40.attn_v.weight (size = 2097152 bytes) -- ignoring +0.00.800.272 W model has unused tensor blk.40.attn_output.weight (size = 8912896 bytes) -- ignoring +0.00.800.274 W model has unused tensor blk.40.attn_q_norm.weight (size = 1024 bytes) -- ignoring +0.00.800.276 W model has unused tensor blk.40.attn_k_norm.weight (size = 1024 bytes) -- ignoring +0.00.800.279 W model has unused tensor blk.40.ffn_gate_inp.weight (size = 2097152 bytes) -- ignoring +0.00.800.281 W model has unused tensor blk.40.ffn_down_exps.weight (size = 115343360 bytes) -- ignoring +0.00.800.286 W model has unused tensor blk.40.ffn_gate_exps.weight (size = 115343360 bytes) -- ignoring +0.00.800.288 W model has unused tensor blk.40.ffn_up_exps.weight (size = 115343360 bytes) -- ignoring +0.00.800.291 W model has unused tensor blk.40.ffn_gate_inp_shexp.weight (size = 8192 bytes) -- ignoring +0.00.800.293 W model has unused tensor blk.40.ffn_gate_shexp.weight (size = 1114112 bytes) -- ignoring +0.00.800.296 W model has unused tensor blk.40.ffn_up_shexp.weight (size = 1114112 bytes) -- ignoring +0.00.800.298 W model has unused tensor blk.40.ffn_down_shexp.weight (size = 1114112 bytes) -- ignoring +0.00.800.303 W model has unused tensor blk.40.nextn.eh_proj.weight (size = 8912896 bytes) -- ignoring +0.00.800.306 W model has unused tensor blk.40.nextn.enorm.weight (size = 8192 bytes) -- ignoring +0.00.800.309 W model has unused tensor blk.40.nextn.hnorm.weight (size = 8192 bytes) -- ignoring +0.00.800.316 W model has unused tensor blk.40.nextn.shared_head_norm.weight (size = 8192 bytes) -- ignoring +0.02.303.475 I cmn init: llama threadpool init, n_threads = 40 +0.02.377.486 I +0.02.377.635 I system_info: n_threads = 40 (n_threads_batch = 40) / 384 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.02.583.472 I kl_divergence: computing over 16 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.13.759.691 I kl_divergence: 11.18 seconds per pass - ETA 2.97 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.7763 ± 0.4517 -0.00733 ± 0.01558 0.08572 ± 0.00664 7.576 ± 0.596 % 89.638 ± 0.953 % + 2 7.3577 ± 0.4291 0.00109 ± 0.01103 0.07566 ± 0.00395 6.863 ± 0.367 % 88.954 ± 0.693 % + 3 8.2653 ± 0.4068 0.00413 ± 0.00916 0.06999 ± 0.00319 6.901 ± 0.367 % 89.932 ± 0.543 % + 4 9.5749 ± 0.4396 0.01814 ± 0.00809 0.07343 ± 0.00304 7.469 ± 0.335 % 89.736 ± 0.474 % + 5 8.9691 ± 0.3626 0.02249 ± 0.00724 0.07149 ± 0.00258 7.401 ± 0.289 % 89.795 ± 0.423 % + 6 7.5834 ± 0.2701 0.01916 ± 0.00639 0.06990 ± 0.00232 7.387 ± 0.255 % 89.932 ± 0.384 % + 7 6.8837 ± 0.2225 0.01005 ± 0.00604 0.07056 ± 0.00224 7.745 ± 0.253 % 90.057 ± 0.354 % + 8 6.6786 ± 0.1991 0.00892 ± 0.00549 0.06905 ± 0.00206 7.640 ± 0.234 % 90.164 ± 0.329 % + 9 7.0691 ± 0.2003 0.00787 ± 0.00517 0.07019 ± 0.00191 7.558 ± 0.217 % 90.008 ± 0.313 % + 10 7.1264 ± 0.1916 0.00755 ± 0.00478 0.06774 ± 0.00174 7.409 ± 0.202 % 90.108 ± 0.295 % + 11 7.1329 ± 0.1817 0.00857 ± 0.00451 0.06571 ± 0.00160 7.281 ± 0.190 % 90.145 ± 0.281 % + 12 7.3853 ± 0.1810 0.00693 ± 0.00423 0.06374 ± 0.00148 7.132 ± 0.179 % 90.249 ± 0.268 % + 13 7.5271 ± 0.1784 0.01040 ± 0.00407 0.06351 ± 0.00141 7.132 ± 0.171 % 90.262 ± 0.257 % + 14 7.5569 ± 0.1721 0.00810 ± 0.00385 0.06164 ± 0.00133 7.003 ± 0.164 % 90.337 ± 0.247 % + 15 7.6073 ± 0.1674 0.00744 ± 0.00372 0.06097 ± 0.00125 6.941 ± 0.156 % 90.375 ± 0.238 % + 16 7.7786 ± 0.1658 0.00301 ± 0.00360 0.06084 ± 0.00118 6.878 ± 0.148 % 90.323 ± 0.231 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.778596 ± 0.165805 +Mean PPL(base) : 7.755210 ± 0.166024 +Cor(ln(PPL(Q)), ln(PPL(base))): 98.58% +Mean ln(PPL(Q)/PPL(base)) : 0.003011 ± 0.003598 +Mean PPL(Q)/PPL(base) : 1.003016 ± 0.003609 +Mean PPL(Q)-PPL(base) : 0.023386 ± 0.027939 + +====== KL divergence statistics ====== +Mean KLD: 0.060844 ± 0.001185 +Maximum KLD: 5.031896 +99.9% KLD: 1.876680 +99.0% KLD: 0.615142 +95.0% KLD: 0.223469 +90.0% KLD: 0.137022 +Median KLD: 0.023589 +10.0% KLD: 0.000344 + 5.0% KLD: 0.000101 + 1.0% KLD: 0.000006 + 0.1% KLD: -0.000150 +Minimum KLD: -0.000622 + +====== Token probability statistics ====== +Mean Δp: -0.459 ± 0.054 % +Maximum Δp: 95.224% +99.9% Δp: 41.651% +99.0% Δp: 19.851% +95.0% Δp: 7.862% +90.0% Δp: 4.076% +75.0% Δp: 0.571% +Median Δp: -0.015% +25.0% Δp: -1.288% +10.0% Δp: -5.650% + 5.0% Δp: -9.632% + 1.0% Δp: -23.417% + 0.1% Δp: -55.582% +Minimum Δp: -96.108% +RMS Δp : 6.878 ± 0.148 % +Same top p: 90.323 ± 0.231 % + diff --git a/appendix/measurements/kldf_21G-ICE.txt b/appendix/measurements/kldf_21G-ICE.txt new file mode 100644 index 0000000000000000000000000000000000000000..fd6d999168f06bb6ac5f47c2c6801853bbcaeeec --- /dev/null +++ b/appendix/measurements/kldf_21G-ICE.txt @@ -0,0 +1,84 @@ +0.00.801.364 W model has unused tensor blk.40.attn_norm.weight (size = 8192 bytes) -- ignoring +0.00.801.378 W model has unused tensor blk.40.post_attention_norm.weight (size = 8192 bytes) -- ignoring +0.00.801.384 W model has unused tensor blk.40.attn_q.weight (size = 17825792 bytes) -- ignoring +0.00.801.386 W model has unused tensor blk.40.attn_k.weight (size = 2097152 bytes) -- ignoring +0.00.801.389 W model has unused tensor blk.40.attn_v.weight (size = 2097152 bytes) -- ignoring +0.00.801.397 W model has unused tensor blk.40.attn_output.weight (size = 8912896 bytes) -- ignoring +0.00.801.399 W model has unused tensor blk.40.attn_q_norm.weight (size = 1024 bytes) -- ignoring +0.00.801.401 W model has unused tensor blk.40.attn_k_norm.weight (size = 1024 bytes) -- ignoring +0.00.801.404 W model has unused tensor blk.40.ffn_gate_inp.weight (size = 2097152 bytes) -- ignoring +0.00.801.406 W model has unused tensor blk.40.ffn_down_exps.weight (size = 142606336 bytes) -- ignoring +0.00.801.410 W model has unused tensor blk.40.ffn_gate_exps.weight (size = 142606336 bytes) -- ignoring +0.00.801.413 W model has unused tensor blk.40.ffn_up_exps.weight (size = 142606336 bytes) -- ignoring +0.00.801.415 W model has unused tensor blk.40.ffn_gate_inp_shexp.weight (size = 8192 bytes) -- ignoring +0.00.801.418 W model has unused tensor blk.40.ffn_gate_shexp.weight (size = 1114112 bytes) -- ignoring +0.00.801.420 W model has unused tensor blk.40.ffn_up_shexp.weight (size = 1114112 bytes) -- ignoring +0.00.801.423 W model has unused tensor blk.40.ffn_down_shexp.weight (size = 1114112 bytes) -- ignoring +0.00.801.426 W model has unused tensor blk.40.nextn.eh_proj.weight (size = 8912896 bytes) -- ignoring +0.00.801.429 W model has unused tensor blk.40.nextn.enorm.weight (size = 8192 bytes) -- ignoring +0.00.801.432 W model has unused tensor blk.40.nextn.hnorm.weight (size = 8192 bytes) -- ignoring +0.00.801.439 W model has unused tensor blk.40.nextn.shared_head_norm.weight (size = 8192 bytes) -- ignoring +0.12.710.938 I cmn init: llama threadpool init, n_threads = 40 +0.12.796.898 I +0.12.797.045 I system_info: n_threads = 40 (n_threads_batch = 40) / 384 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.12.979.626 I kl_divergence: computing over 16 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.22.159.474 I kl_divergence: 9.18 seconds per pass - ETA 2.43 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.8951 ± 0.4631 0.01303 ± 0.01142 0.04793 ± 0.00351 6.581 ± 0.671 % 92.180 ± 0.840 % + 2 7.4046 ± 0.4336 0.00745 ± 0.00812 0.04370 ± 0.00235 5.864 ± 0.407 % 92.033 ± 0.599 % + 3 8.4249 ± 0.4205 0.02325 ± 0.00714 0.04242 ± 0.00191 5.782 ± 0.351 % 92.408 ± 0.478 % + 4 9.3974 ± 0.4298 -0.00058 ± 0.00667 0.05186 ± 0.00354 6.426 ± 0.366 % 91.960 ± 0.425 % + 5 8.7808 ± 0.3539 0.00127 ± 0.00573 0.04943 ± 0.00288 6.224 ± 0.313 % 91.984 ± 0.380 % + 6 7.4381 ± 0.2644 -0.00019 ± 0.00511 0.04826 ± 0.00247 6.303 ± 0.273 % 91.838 ± 0.349 % + 7 6.8154 ± 0.2206 0.00008 ± 0.00484 0.04801 ± 0.00228 6.398 ± 0.263 % 92.068 ± 0.319 % + 8 6.6168 ± 0.1975 -0.00038 ± 0.00442 0.04652 ± 0.00203 6.264 ± 0.238 % 92.082 ± 0.298 % + 9 7.0203 ± 0.1994 0.00094 ± 0.00416 0.04747 ± 0.00189 6.256 ± 0.220 % 91.865 ± 0.285 % + 10 7.0889 ± 0.1910 0.00228 ± 0.00385 0.04543 ± 0.00171 6.096 ± 0.204 % 91.926 ± 0.269 % + 11 7.0839 ± 0.1807 0.00167 ± 0.00365 0.04410 ± 0.00158 5.983 ± 0.192 % 92.038 ± 0.255 % + 12 7.3335 ± 0.1801 -0.00011 ± 0.00343 0.04286 ± 0.00145 5.854 ± 0.180 % 92.074 ± 0.244 % + 13 7.4300 ± 0.1758 -0.00259 ± 0.00333 0.04271 ± 0.00136 5.838 ± 0.170 % 92.097 ± 0.234 % + 14 7.4669 ± 0.1698 -0.00387 ± 0.00314 0.04151 ± 0.00127 5.741 ± 0.162 % 92.138 ± 0.225 % + 15 7.5247 ± 0.1653 -0.00347 ± 0.00301 0.04093 ± 0.00119 5.679 ± 0.154 % 92.102 ± 0.218 % + 16 7.6961 ± 0.1638 -0.00765 ± 0.00296 0.04119 ± 0.00114 5.681 ± 0.148 % 92.027 ± 0.212 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.696145 ± 0.163779 +Mean PPL(base) : 7.755210 ± 0.166024 +Cor(ln(PPL(Q)), ln(PPL(base))): 99.04% +Mean ln(PPL(Q)/PPL(base)) : -0.007645 ± 0.002963 +Mean PPL(Q)/PPL(base) : 0.992384 ± 0.002940 +Mean PPL(Q)-PPL(base) : -0.059065 ± 0.022980 + +====== KL divergence statistics ====== +Mean KLD: 0.041192 ± 0.001140 +Maximum KLD: 6.648515 +99.9% KLD: 1.664515 +99.0% KLD: 0.392024 +95.0% KLD: 0.142787 +90.0% KLD: 0.087165 +Median KLD: 0.015146 +10.0% KLD: 0.000216 + 5.0% KLD: 0.000057 + 1.0% KLD: -0.000005 + 0.1% KLD: -0.000155 +Minimum KLD: -0.000405 + +====== Token probability statistics ====== +Mean Δp: -0.187 ± 0.044 % +Maximum Δp: 97.721% +99.9% Δp: 40.671% +99.0% Δp: 16.097% +95.0% Δp: 6.943% +90.0% Δp: 3.694% +75.0% Δp: 0.586% +Median Δp: -0.002% +25.0% Δp: -0.912% +10.0% Δp: -4.338% + 5.0% Δp: -7.444% + 1.0% Δp: -17.527% + 0.1% Δp: -45.227% +Minimum Δp: -78.071% +RMS Δp : 5.681 ± 0.148 % +Same top p: 92.027 ± 0.212 % + diff --git a/appendix/measurements/kldf_23G-ICE.txt b/appendix/measurements/kldf_23G-ICE.txt new file mode 100644 index 0000000000000000000000000000000000000000..22595d58b25308981e3edb53a83dcb8b2b48f57f --- /dev/null +++ b/appendix/measurements/kldf_23G-ICE.txt @@ -0,0 +1,84 @@ +0.00.804.004 W model has unused tensor blk.40.attn_norm.weight (size = 8192 bytes) -- ignoring +0.00.804.016 W model has unused tensor blk.40.post_attention_norm.weight (size = 8192 bytes) -- ignoring +0.00.804.022 W model has unused tensor blk.40.attn_q.weight (size = 17825792 bytes) -- ignoring +0.00.804.024 W model has unused tensor blk.40.attn_k.weight (size = 2097152 bytes) -- ignoring +0.00.804.027 W model has unused tensor blk.40.attn_v.weight (size = 2097152 bytes) -- ignoring +0.00.804.035 W model has unused tensor blk.40.attn_output.weight (size = 8912896 bytes) -- ignoring +0.00.804.037 W model has unused tensor blk.40.attn_q_norm.weight (size = 1024 bytes) -- ignoring +0.00.804.039 W model has unused tensor blk.40.attn_k_norm.weight (size = 1024 bytes) -- ignoring +0.00.804.042 W model has unused tensor blk.40.ffn_gate_inp.weight (size = 2097152 bytes) -- ignoring +0.00.804.044 W model has unused tensor blk.40.ffn_down_exps.weight (size = 150994944 bytes) -- ignoring +0.00.804.048 W model has unused tensor blk.40.ffn_gate_exps.weight (size = 150994944 bytes) -- ignoring +0.00.804.051 W model has unused tensor blk.40.ffn_up_exps.weight (size = 150994944 bytes) -- ignoring +0.00.804.053 W model has unused tensor blk.40.ffn_gate_inp_shexp.weight (size = 8192 bytes) -- ignoring +0.00.804.056 W model has unused tensor blk.40.ffn_gate_shexp.weight (size = 1114112 bytes) -- ignoring +0.00.804.058 W model has unused tensor blk.40.ffn_up_shexp.weight (size = 1114112 bytes) -- ignoring +0.00.804.061 W model has unused tensor blk.40.ffn_down_shexp.weight (size = 1114112 bytes) -- ignoring +0.00.804.065 W model has unused tensor blk.40.nextn.eh_proj.weight (size = 8912896 bytes) -- ignoring +0.00.804.068 W model has unused tensor blk.40.nextn.enorm.weight (size = 8192 bytes) -- ignoring +0.00.804.071 W model has unused tensor blk.40.nextn.hnorm.weight (size = 8192 bytes) -- ignoring +0.00.804.078 W model has unused tensor blk.40.nextn.shared_head_norm.weight (size = 8192 bytes) -- ignoring +0.14.545.207 I cmn init: llama threadpool init, n_threads = 40 +0.14.635.257 I +0.14.635.411 I system_info: n_threads = 40 (n_threads_batch = 40) / 384 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.14.820.148 I kl_divergence: computing over 16 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.24.019.678 I kl_divergence: 9.20 seconds per pass - ETA 2.45 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.8444 ± 0.4565 0.00439 ± 0.01081 0.04593 ± 0.00390 6.689 ± 0.670 % 92.571 ± 0.820 % + 2 7.3246 ± 0.4255 -0.00342 ± 0.00759 0.04019 ± 0.00217 5.630 ± 0.410 % 91.935 ± 0.602 % + 3 8.3152 ± 0.4118 0.01015 ± 0.00619 0.03698 ± 0.00160 5.297 ± 0.336 % 92.571 ± 0.473 % + 4 9.4536 ± 0.4330 0.00539 ± 0.00561 0.04172 ± 0.00237 5.907 ± 0.337 % 92.351 ± 0.416 % + 5 8.8001 ± 0.3546 0.00347 ± 0.00493 0.04024 ± 0.00194 5.701 ± 0.285 % 92.532 ± 0.368 % + 6 7.4532 ± 0.2653 0.00184 ± 0.00441 0.03891 ± 0.00168 5.671 ± 0.245 % 92.669 ± 0.333 % + 7 6.7964 ± 0.2199 -0.00272 ± 0.00425 0.03871 ± 0.00155 5.823 ± 0.247 % 92.724 ± 0.307 % + 8 6.5984 ± 0.1968 -0.00316 ± 0.00390 0.03750 ± 0.00137 5.688 ± 0.224 % 92.705 ± 0.287 % + 9 7.0188 ± 0.1994 0.00074 ± 0.00372 0.03949 ± 0.00130 5.746 ± 0.204 % 92.288 ± 0.278 % + 10 7.0693 ± 0.1905 -0.00050 ± 0.00346 0.03819 ± 0.00118 5.606 ± 0.189 % 92.463 ± 0.261 % + 11 7.0643 ± 0.1801 -0.00110 ± 0.00327 0.03707 ± 0.00111 5.498 ± 0.178 % 92.606 ± 0.247 % + 12 7.3174 ± 0.1796 -0.00231 ± 0.00309 0.03610 ± 0.00103 5.388 ± 0.167 % 92.644 ± 0.236 % + 13 7.4127 ± 0.1753 -0.00491 ± 0.00300 0.03632 ± 0.00101 5.457 ± 0.160 % 92.646 ± 0.226 % + 14 7.4566 ± 0.1695 -0.00526 ± 0.00284 0.03531 ± 0.00094 5.378 ± 0.152 % 92.711 ± 0.217 % + 15 7.4957 ± 0.1645 -0.00732 ± 0.00274 0.03526 ± 0.00089 5.332 ± 0.145 % 92.727 ± 0.210 % + 16 7.6660 ± 0.1629 -0.01158 ± 0.00272 0.03607 ± 0.00087 5.354 ± 0.140 % 92.650 ± 0.204 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.665955 ± 0.162891 +Mean PPL(base) : 7.755210 ± 0.166024 +Cor(ln(PPL(Q)), ln(PPL(base))): 99.19% +Mean ln(PPL(Q)/PPL(base)) : -0.011576 ± 0.002723 +Mean PPL(Q)/PPL(base) : 0.988491 ± 0.002692 +Mean PPL(Q)-PPL(base) : -0.089255 ± 0.021195 + +====== KL divergence statistics ====== +Mean KLD: 0.036069 ± 0.000868 +Maximum KLD: 5.588296 +99.9% KLD: 1.272136 +99.0% KLD: 0.331770 +95.0% KLD: 0.130636 +90.0% KLD: 0.080453 +Median KLD: 0.013617 +10.0% KLD: 0.000199 + 5.0% KLD: 0.000055 + 1.0% KLD: -0.000006 + 0.1% KLD: -0.000161 +Minimum KLD: -0.000501 + +====== Token probability statistics ====== +Mean Δp: -0.162 ± 0.042 % +Maximum Δp: 75.679% +99.9% Δp: 36.358% +99.0% Δp: 15.326% +95.0% Δp: 6.646% +90.0% Δp: 3.477% +75.0% Δp: 0.565% +Median Δp: -0.002% +25.0% Δp: -0.827% +10.0% Δp: -4.021% + 5.0% Δp: -7.009% + 1.0% Δp: -16.589% + 0.1% Δp: -49.141% +Minimum Δp: -92.408% +RMS Δp : 5.354 ± 0.140 % +Same top p: 92.650 ± 0.204 % + diff --git a/appendix/measurements/kldf_25G-ICE.txt b/appendix/measurements/kldf_25G-ICE.txt new file mode 100644 index 0000000000000000000000000000000000000000..ee015c91d551af22dbc8407c5722e1b2150812ea --- /dev/null +++ b/appendix/measurements/kldf_25G-ICE.txt @@ -0,0 +1,84 @@ +0.00.792.992 W model has unused tensor blk.40.attn_norm.weight (size = 8192 bytes) -- ignoring +0.00.793.005 W model has unused tensor blk.40.post_attention_norm.weight (size = 8192 bytes) -- ignoring +0.00.793.011 W model has unused tensor blk.40.attn_q.weight (size = 17825792 bytes) -- ignoring +0.00.793.014 W model has unused tensor blk.40.attn_k.weight (size = 2097152 bytes) -- ignoring +0.00.793.016 W model has unused tensor blk.40.attn_v.weight (size = 2097152 bytes) -- ignoring +0.00.793.024 W model has unused tensor blk.40.attn_output.weight (size = 8912896 bytes) -- ignoring +0.00.793.026 W model has unused tensor blk.40.attn_q_norm.weight (size = 1024 bytes) -- ignoring +0.00.793.028 W model has unused tensor blk.40.attn_k_norm.weight (size = 1024 bytes) -- ignoring +0.00.793.031 W model has unused tensor blk.40.ffn_gate_inp.weight (size = 2097152 bytes) -- ignoring +0.00.793.033 W model has unused tensor blk.40.ffn_down_exps.weight (size = 184549376 bytes) -- ignoring +0.00.793.038 W model has unused tensor blk.40.ffn_gate_exps.weight (size = 184549376 bytes) -- ignoring +0.00.793.041 W model has unused tensor blk.40.ffn_up_exps.weight (size = 184549376 bytes) -- ignoring +0.00.793.043 W model has unused tensor blk.40.ffn_gate_inp_shexp.weight (size = 8192 bytes) -- ignoring +0.00.793.046 W model has unused tensor blk.40.ffn_gate_shexp.weight (size = 1114112 bytes) -- ignoring +0.00.793.048 W model has unused tensor blk.40.ffn_up_shexp.weight (size = 1114112 bytes) -- ignoring +0.00.793.051 W model has unused tensor blk.40.ffn_down_shexp.weight (size = 1114112 bytes) -- ignoring +0.00.793.055 W model has unused tensor blk.40.nextn.eh_proj.weight (size = 8912896 bytes) -- ignoring +0.00.793.058 W model has unused tensor blk.40.nextn.enorm.weight (size = 8192 bytes) -- ignoring +0.00.793.061 W model has unused tensor blk.40.nextn.hnorm.weight (size = 8192 bytes) -- ignoring +0.00.793.068 W model has unused tensor blk.40.nextn.shared_head_norm.weight (size = 8192 bytes) -- ignoring +0.05.671.948 I cmn init: llama threadpool init, n_threads = 40 +0.05.749.795 I +0.05.749.952 I system_info: n_threads = 40 (n_threads_batch = 40) / 384 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | +0.05.937.173 I kl_divergence: computing over 16 chunks, n_ctx=2048, batch_size=2048, n_seq=1 +0.15.342.589 I kl_divergence: 9.41 seconds per pass - ETA 2.50 minutes + +chunk PPL ln(PPL(Q)/PPL(base)) KL Divergence Δp RMS Same top p + 1 5.7216 ± 0.4446 -0.01684 ± 0.00969 0.03757 ± 0.00273 5.983 ± 0.482 % 93.646 ± 0.763 % + 2 7.2893 ± 0.4253 -0.00825 ± 0.00691 0.03252 ± 0.00164 5.037 ± 0.297 % 93.157 ± 0.558 % + 3 8.1847 ± 0.4034 -0.00567 ± 0.00557 0.02982 ± 0.00132 4.860 ± 0.304 % 93.776 ± 0.436 % + 4 9.2966 ± 0.4234 -0.01135 ± 0.00510 0.03356 ± 0.00194 5.007 ± 0.244 % 93.426 ± 0.387 % + 5 8.6874 ± 0.3487 -0.00942 ± 0.00461 0.03272 ± 0.00160 4.857 ± 0.208 % 93.392 ± 0.347 % + 6 7.3567 ± 0.2607 -0.01119 ± 0.00416 0.03244 ± 0.00140 5.011 ± 0.187 % 93.271 ± 0.320 % + 7 6.6881 ± 0.2151 -0.01878 ± 0.00407 0.03406 ± 0.00147 5.272 ± 0.184 % 93.088 ± 0.300 % + 8 6.5149 ± 0.1935 -0.01590 ± 0.00368 0.03283 ± 0.00132 5.162 ± 0.168 % 93.218 ± 0.278 % + 9 6.8816 ± 0.1941 -0.01901 ± 0.00346 0.03371 ± 0.00126 5.151 ± 0.156 % 92.908 ± 0.268 % + 10 6.9492 ± 0.1862 -0.01763 ± 0.00322 0.03231 ± 0.00114 5.029 ± 0.146 % 93.109 ± 0.250 % + 11 6.9597 ± 0.1766 -0.01602 ± 0.00306 0.03132 ± 0.00106 4.957 ± 0.138 % 93.184 ± 0.238 % + 12 7.2214 ± 0.1766 -0.01551 ± 0.00287 0.03043 ± 0.00097 4.859 ± 0.130 % 93.231 ± 0.227 % + 13 7.3199 ± 0.1725 -0.01752 ± 0.00276 0.03056 ± 0.00092 4.898 ± 0.124 % 93.180 ± 0.219 % + 14 7.3819 ± 0.1673 -0.01533 ± 0.00262 0.02968 ± 0.00086 4.795 ± 0.118 % 93.248 ± 0.210 % + 15 7.4231 ± 0.1624 -0.01707 ± 0.00255 0.02990 ± 0.00082 4.801 ± 0.112 % 93.255 ± 0.202 % + 16 7.6111 ± 0.1614 -0.01876 ± 0.00251 0.03030 ± 0.00078 4.786 ± 0.109 % 93.163 ± 0.197 % + +====== Perplexity statistics ====== +Mean PPL(Q) : 7.611108 ± 0.161433 +Mean PPL(base) : 7.755210 ± 0.166024 +Cor(ln(PPL(Q)), ln(PPL(base))): 99.31% +Mean ln(PPL(Q)/PPL(base)) : -0.018756 ± 0.002515 +Mean PPL(Q)/PPL(base) : 0.981419 ± 0.002468 +Mean PPL(Q)-PPL(base) : -0.144102 ± 0.019802 + +====== KL divergence statistics ====== +Mean KLD: 0.030304 ± 0.000779 +Maximum KLD: 5.409998 +99.9% KLD: 1.048851 +99.0% KLD: 0.295999 +95.0% KLD: 0.108395 +90.0% KLD: 0.065665 +Median KLD: 0.011116 +10.0% KLD: 0.000164 + 5.0% KLD: 0.000046 + 1.0% KLD: -0.000009 + 0.1% KLD: -0.000142 +Minimum KLD: -0.000403 + +====== Token probability statistics ====== +Mean Δp: -0.099 ± 0.037 % +Maximum Δp: 76.470% +99.9% Δp: 33.668% +99.0% Δp: 14.631% +95.0% Δp: 5.945% +90.0% Δp: 3.133% +75.0% Δp: 0.527% +Median Δp: -0.002% +25.0% Δp: -0.717% +10.0% Δp: -3.579% + 5.0% Δp: -6.381% + 1.0% Δp: -14.571% + 0.1% Δp: -33.475% +Minimum Δp: -71.146% +RMS Δp : 4.786 ± 0.109 % +Same top p: 93.163 ± 0.197 % + diff --git a/appendix/measurements/sensitivity.jsonl b/appendix/measurements/sensitivity.jsonl new file mode 100644 index 0000000000000000000000000000000000000000..e94e5a445053a98f87f82a3bc079568af365d439 --- /dev/null +++ b/appendix/measurements/sensitivity.jsonl @@ -0,0 +1,600 @@ +{"tensor": "blk.0.ffn_down_exps.weight", "block": 0, "role": "down", "type": "Q6_K", "s": 61.65974366628854, "err_l2": 3.885833740234375, "energy": 1594305.5174544854, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.0.ffn_gate_exps.weight", "block": 0, "role": "gate", "type": "Q6_K", "s": 8722.722517432312, "err_l2": 4.310602188110352, "energy": 1558006241.509983, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.0.ffn_up_exps.weight", "block": 0, "role": "up", "type": "Q6_K", "s": 8293.844507298692, "err_l2": 4.043476104736328, "energy": 1558006241.509983, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.1.ffn_down_exps.weight", "block": 1, "role": "down", "type": "Q6_K", "s": 148.40433461287418, "err_l2": 8.792551040649414, "energy": 2793671.7302908897, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.1.ffn_gate_exps.weight", "block": 1, "role": "gate", "type": "Q6_K", "s": 19179.921745263306, "err_l2": 7.918911457061768, "energy": 1528231540.1870728, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.1.ffn_up_exps.weight", "block": 1, "role": "up", "type": "Q6_K", "s": 18372.12213424377, "err_l2": 7.502530574798584, "energy": 1528231540.1870728, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.2.ffn_down_exps.weight", "block": 2, "role": "down", "type": "Q6_K", "s": 138.5214119023555, "err_l2": 8.70921802520752, "energy": 2394420.025505662, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.2.ffn_gate_exps.weight", "block": 2, "role": "gate", "type": "Q6_K", "s": 22531.969596581566, "err_l2": 8.501482963562012, "energy": 1614167290.0648804, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.2.ffn_up_exps.weight", "block": 2, "role": "up", "type": "Q6_K", "s": 21711.524281495793, "err_l2": 8.173916816711426, "energy": 1614167290.0648804, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.3.ffn_down_exps.weight", "block": 3, "role": "down", "type": "Q6_K", "s": 194.663888707422, "err_l2": 8.62031078338623, "energy": 3253016.4095531586, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.3.ffn_gate_exps.weight", "block": 3, "role": "gate", "type": "Q6_K", "s": 29165.36333808167, "err_l2": 8.65600872039795, "energy": 1987198435.1838608, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.3.ffn_up_exps.weight", "block": 3, "role": "up", "type": "Q6_K", "s": 27719.899813540436, "err_l2": 8.281763076782227, "energy": 1987198435.1838608, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.4.ffn_down_exps.weight", "block": 4, "role": "down", "type": "Q6_K", "s": 271.9846326088101, "err_l2": 8.561132431030273, "energy": 4543147.314295173, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.4.ffn_gate_exps.weight", "block": 4, "role": "gate", "type": "Q6_K", "s": 33605.28046366811, "err_l2": 9.120293617248535, "energy": 2219785664.976015, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.4.ffn_up_exps.weight", "block": 4, "role": "up", "type": "Q6_K", "s": 30961.082725922886, "err_l2": 8.452974319458008, "energy": 2219785664.976015, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.5.ffn_down_exps.weight", "block": 5, "role": "down", "type": "Q6_K", "s": 343.9703593460114, "err_l2": 8.565292358398438, "energy": 5663699.233736992, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.5.ffn_gate_exps.weight", "block": 5, "role": "gate", "type": "Q6_K", "s": 37175.75811867472, "err_l2": 9.255672454833984, "energy": 2404538256.884094, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.5.ffn_up_exps.weight", "block": 5, "role": "up", "type": "Q6_K", "s": 33890.176958525706, "err_l2": 8.472661972045898, "energy": 2404538256.884094, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.6.ffn_down_exps.weight", "block": 6, "role": "down", "type": "Q6_K", "s": 383.46539756557496, "err_l2": 8.541509628295898, "energy": 6342106.4899379015, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.6.ffn_gate_exps.weight", "block": 6, "role": "gate", "type": "Q6_K", "s": 38326.83843144321, "err_l2": 9.313314437866211, "energy": 2495879819.4156494, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.6.ffn_up_exps.weight", "block": 6, "role": "up", "type": "Q6_K", "s": 35055.87635212023, "err_l2": 8.4568510055542, "energy": 2495879819.4156494, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.7.ffn_down_exps.weight", "block": 7, "role": "down", "type": "Q6_K", "s": 480.5134190982183, "err_l2": 8.466245651245117, "energy": 8081860.904105902, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.7.ffn_gate_exps.weight", "block": 7, "role": "gate", "type": "Q6_K", "s": 48319.97398497647, "err_l2": 9.312021255493164, "energy": 3024648344.3563538, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.7.ffn_up_exps.weight", "block": 7, "role": "up", "type": "Q6_K", "s": 43869.28078833384, "err_l2": 8.358713150024414, "energy": 3024648344.3563538, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.8.ffn_down_exps.weight", "block": 8, "role": "down", "type": "Q6_K", "s": 527.0905369263719, "err_l2": 8.366340637207031, "energy": 8960753.594052434, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.8.ffn_gate_exps.weight", "block": 8, "role": "gate", "type": "Q6_K", "s": 53442.297922986574, "err_l2": 9.41165828704834, "energy": 3319484814.762665, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.8.ffn_up_exps.weight", "block": 8, "role": "up", "type": "Q6_K", "s": 47327.85567293315, "err_l2": 8.197868347167969, "energy": 3319484814.762665, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.9.ffn_down_exps.weight", "block": 9, "role": "down", "type": "Q6_K", "s": 590.7201014389407, "err_l2": 8.609736442565918, "energy": 9568770.147459984, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.9.ffn_gate_exps.weight", "block": 9, "role": "gate", "type": "Q6_K", "s": 54013.08434677724, "err_l2": 9.61959171295166, "energy": 3361266052.4784546, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.9.ffn_up_exps.weight", "block": 9, "role": "up", "type": "Q6_K", "s": 49521.857111821824, "err_l2": 8.598270416259766, "energy": 3361266052.4784546, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.10.ffn_down_exps.weight", "block": 10, "role": "down", "type": "Q6_K", "s": 644.7502809699487, "err_l2": 8.759125709533691, "energy": 10111794.935325623, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.10.ffn_gate_exps.weight", "block": 10, "role": "gate", "type": "Q6_K", "s": 50724.71734158883, "err_l2": 9.70076847076416, "energy": 3193724210.449585, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.10.ffn_up_exps.weight", "block": 10, "role": "up", "type": "Q6_K", "s": 47883.680888665185, "err_l2": 8.969681739807129, "energy": 3193724210.449585, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.11.ffn_down_exps.weight", "block": 11, "role": "down", "type": "Q6_K", "s": 878.0528585487092, "err_l2": 8.80418586730957, "energy": 13562770.490616083, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.11.ffn_gate_exps.weight", "block": 11, "role": "gate", "type": "Q6_K", "s": 58194.77397592106, "err_l2": 9.588287353515625, "energy": 3610363311.6185303, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.11.ffn_up_exps.weight", "block": 11, "role": "up", "type": "Q6_K", "s": 55283.89997364741, "err_l2": 8.98556900024414, "energy": 3610363311.6185303, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.12.ffn_down_exps.weight", "block": 12, "role": "down", "type": "Q6_K", "s": 745.8334262779887, "err_l2": 8.573540687561035, "energy": 11890972.488930225, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.12.ffn_gate_exps.weight", "block": 12, "role": "gate", "type": "Q6_K", "s": 59671.55988838575, "err_l2": 9.60045337677002, "energy": 3644765104.288269, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.12.ffn_up_exps.weight", "block": 12, "role": "up", "type": "Q6_K", "s": 55711.748573403645, "err_l2": 8.746289253234863, "energy": 3644765104.288269, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.13.ffn_down_exps.weight", "block": 13, "role": "down", "type": "Q6_K", "s": 929.5455463533548, "err_l2": 8.67822551727295, "energy": 14734612.844225299, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.13.ffn_gate_exps.weight", "block": 13, "role": "gate", "type": "Q6_K", "s": 66365.35864998066, "err_l2": 9.633715629577637, "energy": 3992092145.1657104, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.13.ffn_up_exps.weight", "block": 13, "role": "up", "type": "Q6_K", "s": 61474.40204864574, "err_l2": 8.79011058807373, "energy": 3992092145.1657104, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.14.ffn_down_exps.weight", "block": 14, "role": "down", "type": "Q6_K", "s": 854.4091443223264, "err_l2": 8.732540130615234, "energy": 13554481.034674168, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.14.ffn_gate_exps.weight", "block": 14, "role": "gate", "type": "Q6_K", "s": 62511.49527712904, "err_l2": 9.638384819030762, "energy": 3737719776.7888184, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.14.ffn_up_exps.weight", "block": 14, "role": "up", "type": "Q6_K", "s": 57921.828785579564, "err_l2": 8.90190601348877, "energy": 3737719776.7888184, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.15.ffn_down_exps.weight", "block": 15, "role": "down", "type": "Q6_K", "s": 1020.8314061481782, "err_l2": 8.787332534790039, "energy": 15960954.2438488, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.15.ffn_gate_exps.weight", "block": 15, "role": "gate", "type": "Q6_K", "s": 70285.51821137413, "err_l2": 9.602424621582031, "energy": 4167948470.2155457, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.15.ffn_up_exps.weight", "block": 15, "role": "up", "type": "Q6_K", "s": 66022.4728037704, "err_l2": 8.990856170654297, "energy": 4167948470.2155457, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.16.ffn_down_exps.weight", "block": 16, "role": "down", "type": "Q6_K", "s": 1116.4102610779942, "err_l2": 8.951882362365723, "energy": 17199945.656352043, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.16.ffn_gate_exps.weight", "block": 16, "role": "gate", "type": "Q6_K", "s": 67159.61719045154, "err_l2": 9.460735321044922, "energy": 4078872020.2729187, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.16.ffn_up_exps.weight", "block": 16, "role": "up", "type": "Q6_K", "s": 65717.80323865861, "err_l2": 9.189590454101562, "energy": 4078872020.2729187, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.17.ffn_down_exps.weight", "block": 17, "role": "down", "type": "Q6_K", "s": 1043.9322677924329, "err_l2": 8.932032585144043, "energy": 16259451.176771164, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.17.ffn_gate_exps.weight", "block": 17, "role": "gate", "type": "Q6_K", "s": 71656.23375834893, "err_l2": 9.645599365234375, "energy": 4243044980.7101135, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.17.ffn_up_exps.weight", "block": 17, "role": "up", "type": "Q6_K", "s": 67865.26001842087, "err_l2": 9.152010917663574, "energy": 4243044980.7101135, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.18.ffn_down_exps.weight", "block": 18, "role": "down", "type": "Q6_K", "s": 1174.2995250488148, "err_l2": 9.020037651062012, "energy": 18323826.341524124, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.18.ffn_gate_exps.weight", "block": 18, "role": "gate", "type": "Q6_K", "s": 75146.73675179266, "err_l2": 9.76032543182373, "energy": 4469796752.81366, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.18.ffn_up_exps.weight", "block": 18, "role": "up", "type": "Q6_K", "s": 71141.8821130805, "err_l2": 9.164133071899414, "energy": 4469796752.81366, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.19.ffn_down_exps.weight", "block": 19, "role": "down", "type": "Q6_K", "s": 1397.2072411390614, "err_l2": 9.054471015930176, "energy": 21314045.25215578, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.19.ffn_gate_exps.weight", "block": 19, "role": "gate", "type": "Q6_K", "s": 85024.3250469029, "err_l2": 9.873744010925293, "energy": 4942801179.769653, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.19.ffn_up_exps.weight", "block": 19, "role": "up", "type": "Q6_K", "s": 79050.59845768427, "err_l2": 9.13430404663086, "energy": 4942801179.769653, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.20.ffn_down_exps.weight", "block": 20, "role": "down", "type": "Q6_K", "s": 1193.8208372636389, "err_l2": 8.714624404907227, "energy": 19533477.948932648, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.20.ffn_gate_exps.weight", "block": 20, "role": "gate", "type": "Q6_K", "s": 93936.09562868436, "err_l2": 10.087002754211426, "energy": 5309558766.605164, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.20.ffn_up_exps.weight", "block": 20, "role": "up", "type": "Q6_K", "s": 81003.9965837499, "err_l2": 8.589920997619629, "energy": 5309558766.605164, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.21.ffn_down_exps.weight", "block": 21, "role": "down", "type": "Q6_K", "s": 1268.477338359586, "err_l2": 8.829290390014648, "energy": 19783092.441063404, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.21.ffn_gate_exps.weight", "block": 21, "role": "gate", "type": "Q6_K", "s": 89153.03123862503, "err_l2": 10.184690475463867, "energy": 5133279711.727417, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.21.ffn_up_exps.weight", "block": 21, "role": "up", "type": "Q6_K", "s": 80956.5434693456, "err_l2": 8.897253036499023, "energy": 5133279711.727417, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.22.ffn_down_exps.weight", "block": 22, "role": "down", "type": "Q6_K", "s": 1331.085023340426, "err_l2": 9.098003387451172, "energy": 20124856.359775662, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.22.ffn_gate_exps.weight", "block": 22, "role": "gate", "type": "Q6_K", "s": 80280.28005235989, "err_l2": 10.041524887084961, "energy": 4779006383.581612, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.22.ffn_up_exps.weight", "block": 22, "role": "up", "type": "Q6_K", "s": 76851.75900875675, "err_l2": 9.456496238708496, "energy": 4779006383.581612, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.23.ffn_down_exps.weight", "block": 23, "role": "down", "type": "Q6_K", "s": 1430.0273684200804, "err_l2": 9.193745613098145, "energy": 21217019.830352783, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.23.ffn_gate_exps.weight", "block": 23, "role": "gate", "type": "Q6_K", "s": 82235.5453172375, "err_l2": 9.947522163391113, "energy": 4948538410.368652, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.23.ffn_up_exps.weight", "block": 23, "role": "up", "type": "Q6_K", "s": 80103.02427891325, "err_l2": 9.578818321228027, "energy": 4948538410.368652, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.24.ffn_down_exps.weight", "block": 24, "role": "down", "type": "Q6_K", "s": 1275.1341928522415, "err_l2": 9.05922794342041, "energy": 18962488.084349155, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.24.ffn_gate_exps.weight", "block": 24, "role": "gate", "type": "Q6_K", "s": 84735.6399841808, "err_l2": 10.00178050994873, "energy": 4985729777.105591, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.24.ffn_up_exps.weight", "block": 24, "role": "up", "type": "Q6_K", "s": 81105.26489435471, "err_l2": 9.371387481689453, "energy": 4985729777.105591, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.25.ffn_down_exps.weight", "block": 25, "role": "down", "type": "Q6_K", "s": 1471.8125726908947, "err_l2": 9.173450469970703, "energy": 22012922.12262726, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.25.ffn_gate_exps.weight", "block": 25, "role": "gate", "type": "Q6_K", "s": 88877.57772513098, "err_l2": 10.159884452819824, "energy": 5205305004.696228, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.25.ffn_up_exps.weight", "block": 25, "role": "up", "type": "Q6_K", "s": 84201.61485677405, "err_l2": 9.435264587402344, "energy": 5205305004.696228, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.26.ffn_down_exps.weight", "block": 26, "role": "down", "type": "Q6_K", "s": 1439.5012481931294, "err_l2": 9.330008506774902, "energy": 21354014.354220867, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.26.ffn_gate_exps.weight", "block": 26, "role": "gate", "type": "Q6_K", "s": 85007.04580712454, "err_l2": 10.10197639465332, "energy": 4994181607.238647, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.26.ffn_up_exps.weight", "block": 26, "role": "up", "type": "Q6_K", "s": 81488.45601827816, "err_l2": 9.597506523132324, "energy": 4994181607.238647, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.27.ffn_down_exps.weight", "block": 27, "role": "down", "type": "Q6_K", "s": 1874.6524468043333, "err_l2": 9.482582092285156, "energy": 27613217.047315598, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.27.ffn_gate_exps.weight", "block": 27, "role": "gate", "type": "Q6_K", "s": 94456.3493506003, "err_l2": 10.339587211608887, "energy": 5427490272.732422, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.27.ffn_up_exps.weight", "block": 27, "role": "up", "type": "Q6_K", "s": 89914.27719804135, "err_l2": 9.72640323638916, "energy": 5427490272.732422, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.28.ffn_down_exps.weight", "block": 28, "role": "down", "type": "Q6_K", "s": 2076.218094406911, "err_l2": 9.665953636169434, "energy": 30576922.299942017, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.28.ffn_gate_exps.weight", "block": 28, "role": "gate", "type": "Q6_K", "s": 91510.50576180585, "err_l2": 10.356928825378418, "energy": 5276713432.383301, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.28.ffn_up_exps.weight", "block": 28, "role": "up", "type": "Q6_K", "s": 88310.35027305048, "err_l2": 9.885608673095703, "energy": 5276713432.383301, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.29.ffn_down_exps.weight", "block": 29, "role": "down", "type": "Q6_K", "s": 2212.726617004802, "err_l2": 9.694920539855957, "energy": 32332774.765409946, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.29.ffn_gate_exps.weight", "block": 29, "role": "gate", "type": "Q6_K", "s": 97499.78375664815, "err_l2": 10.660562515258789, "energy": 5504049370.36377, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.29.ffn_up_exps.weight", "block": 29, "role": "up", "type": "Q6_K", "s": 91897.93780554387, "err_l2": 9.917572021484375, "energy": 5504049370.36377, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.30.ffn_down_exps.weight", "block": 30, "role": "down", "type": "Q6_K", "s": 2604.830508125893, "err_l2": 9.77155876159668, "energy": 38829847.31696892, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.30.ffn_gate_exps.weight", "block": 30, "role": "gate", "type": "Q6_K", "s": 106013.58716137402, "err_l2": 10.893744468688965, "energy": 5947769125.285889, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.30.ffn_up_exps.weight", "block": 30, "role": "up", "type": "Q6_K", "s": 98007.02158948682, "err_l2": 9.889787673950195, "energy": 5947769125.285889, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.31.ffn_down_exps.weight", "block": 31, "role": "down", "type": "Q6_K", "s": 3755.6380678989362, "err_l2": 10.009068489074707, "energy": 54921343.842552185, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.31.ffn_gate_exps.weight", "block": 31, "role": "gate", "type": "Q6_K", "s": 127706.65311999206, "err_l2": 11.104010581970215, "energy": 6885215013.473389, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.31.ffn_up_exps.weight", "block": 31, "role": "up", "type": "Q6_K", "s": 116107.41715943682, "err_l2": 9.935404777526855, "energy": 6885215013.473389, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.32.ffn_down_exps.weight", "block": 32, "role": "down", "type": "Q6_K", "s": 5419.216371409959, "err_l2": 10.30760669708252, "energy": 78752141.1644249, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.32.ffn_gate_exps.weight", "block": 32, "role": "gate", "type": "Q6_K", "s": 146615.02528301408, "err_l2": 11.722517013549805, "energy": 7527876637.479248, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.32.ffn_up_exps.weight", "block": 32, "role": "up", "type": "Q6_K", "s": 130415.48649023584, "err_l2": 10.297945976257324, "energy": 7527876637.479248, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.33.ffn_down_exps.weight", "block": 33, "role": "down", "type": "Q6_K", "s": 7402.562256213854, "err_l2": 10.608491897583008, "energy": 106669200.57768059, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.33.ffn_gate_exps.weight", "block": 33, "role": "gate", "type": "Q6_K", "s": 166124.52656686984, "err_l2": 12.083194732666016, "energy": 8332150965.168579, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.33.ffn_up_exps.weight", "block": 33, "role": "up", "type": "Q6_K", "s": 146514.3376861333, "err_l2": 10.465734481811523, "energy": 8332150965.168579, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.34.ffn_down_exps.weight", "block": 34, "role": "down", "type": "Q6_K", "s": 8786.196860568489, "err_l2": 10.727540969848633, "energy": 124581003.01006317, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.34.ffn_gate_exps.weight", "block": 34, "role": "gate", "type": "Q6_K", "s": 173709.50603974838, "err_l2": 12.139286994934082, "energy": 8550464978.343994, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.34.ffn_up_exps.weight", "block": 34, "role": "up", "type": "Q6_K", "s": 151468.5884759203, "err_l2": 10.507402420043945, "energy": 8550464978.343994, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.35.ffn_down_exps.weight", "block": 35, "role": "down", "type": "Q6_K", "s": 9837.389107325424, "err_l2": 10.972063064575195, "energy": 137147195.31022453, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.35.ffn_gate_exps.weight", "block": 35, "role": "gate", "type": "Q6_K", "s": 203612.5961699222, "err_l2": 11.975159645080566, "energy": 9771642959.588196, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.35.ffn_up_exps.weight", "block": 35, "role": "up", "type": "Q6_K", "s": 176970.1247174816, "err_l2": 10.345810890197754, "energy": 9771642959.588196, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.36.ffn_down_exps.weight", "block": 36, "role": "down", "type": "Q6_K", "s": 12436.333185258036, "err_l2": 11.332387924194336, "energy": 167412832.3093624, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.36.ffn_gate_exps.weight", "block": 36, "role": "gate", "type": "Q6_K", "s": 228308.68432479096, "err_l2": 12.16110897064209, "energy": 10746321069.68451, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.36.ffn_up_exps.weight", "block": 36, "role": "up", "type": "Q6_K", "s": 201685.5476832193, "err_l2": 10.716617584228516, "energy": 10746321069.68451, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.37.ffn_down_exps.weight", "block": 37, "role": "down", "type": "Q6_K", "s": 19558.64982240983, "err_l2": 11.664541244506836, "energy": 260268775.01239014, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.37.ffn_gate_exps.weight", "block": 37, "role": "gate", "type": "Q6_K", "s": 238226.23803732393, "err_l2": 12.103915214538574, "energy": 11005630810.807129, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.37.ffn_up_exps.weight", "block": 37, "role": "up", "type": "Q6_K", "s": 212890.86107131225, "err_l2": 10.931743621826172, "energy": 11005630810.807129, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.38.ffn_down_exps.weight", "block": 38, "role": "down", "type": "Q6_K", "s": 30125.380921080847, "err_l2": 11.86742115020752, "energy": 406006067.78154135, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.38.ffn_gate_exps.weight", "block": 38, "role": "gate", "type": "Q6_K", "s": 239383.28675979914, "err_l2": 12.628026008605957, "energy": 10846040824.944519, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.38.ffn_up_exps.weight", "block": 38, "role": "up", "type": "Q6_K", "s": 215139.75773200358, "err_l2": 11.336996078491211, "energy": 10846040824.944519, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.39.ffn_down_exps.weight", "block": 39, "role": "down", "type": "Q6_K", "s": 46629.0579221778, "err_l2": 12.394109725952148, "energy": 639124982.3202152, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.39.ffn_gate_exps.weight", "block": 39, "role": "gate", "type": "Q6_K", "s": 205630.18799030103, "err_l2": 14.826573371887207, "energy": 8206865322.349945, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.39.ffn_up_exps.weight", "block": 39, "role": "up", "type": "Q6_K", "s": 177035.46223435458, "err_l2": 12.565495491027832, "energy": 8206865322.349945, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.0.ffn_down_exps.weight", "block": 0, "role": "down", "type": "Q5_K", "s": 241.20914259914753, "err_l2": 27.6664981842041, "energy": 1594305.5174544854, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.0.ffn_gate_exps.weight", "block": 0, "role": "gate", "type": "Q5_K", "s": 28831.12725776301, "err_l2": 17.198810577392578, "energy": 1558006241.509983, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.0.ffn_up_exps.weight", "block": 0, "role": "up", "type": "Q5_K", "s": 27122.513480744616, "err_l2": 16.094871520996094, "energy": 1558006241.509983, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.1.ffn_down_exps.weight", "block": 1, "role": "down", "type": "Q5_K", "s": 593.7382079016369, "err_l2": 36.34273910522461, "energy": 2793671.7302908897, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.1.ffn_gate_exps.weight", "block": 1, "role": "gate", "type": "Q5_K", "s": 71539.52374965082, "err_l2": 32.338043212890625, "energy": 1528231540.1870728, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.1.ffn_up_exps.weight", "block": 1, "role": "up", "type": "Q5_K", "s": 67977.3764422464, "err_l2": 30.60460662841797, "energy": 1528231540.1870728, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.2.ffn_down_exps.weight", "block": 2, "role": "down", "type": "Q5_K", "s": 561.3234184941497, "err_l2": 36.03813552856445, "energy": 2394420.025505662, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.2.ffn_gate_exps.weight", "block": 2, "role": "gate", "type": "Q5_K", "s": 86113.98703534614, "err_l2": 34.82948303222656, "energy": 1614167290.0648804, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.2.ffn_up_exps.weight", "block": 2, "role": "up", "type": "Q5_K", "s": 82585.73597951153, "err_l2": 33.46082305908203, "energy": 1614167290.0648804, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.3.ffn_down_exps.weight", "block": 3, "role": "down", "type": "Q5_K", "s": 793.1075085322504, "err_l2": 35.51857376098633, "energy": 3253016.4095531586, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.3.ffn_gate_exps.weight", "block": 3, "role": "gate", "type": "Q5_K", "s": 113301.67780107616, "err_l2": 35.52821350097656, "energy": 1987198435.1838608, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.3.ffn_up_exps.weight", "block": 3, "role": "up", "type": "Q5_K", "s": 106766.83005379223, "err_l2": 33.95730972290039, "energy": 1987198435.1838608, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.4.ffn_down_exps.weight", "block": 4, "role": "down", "type": "Q5_K", "s": 1108.861776379431, "err_l2": 35.201072692871094, "energy": 4543147.314295173, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.4.ffn_gate_exps.weight", "block": 4, "role": "gate", "type": "Q5_K", "s": 128483.98560895282, "err_l2": 37.39972686767578, "energy": 2219785664.976015, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.4.ffn_up_exps.weight", "block": 4, "role": "up", "type": "Q5_K", "s": 116916.68844289132, "err_l2": 34.624420166015625, "energy": 2219785664.976015, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.5.ffn_down_exps.weight", "block": 5, "role": "down", "type": "Q5_K", "s": 1406.488848662763, "err_l2": 35.18010330200195, "energy": 5663699.233736992, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.5.ffn_gate_exps.weight", "block": 5, "role": "gate", "type": "Q5_K", "s": 142759.8349721835, "err_l2": 37.97281265258789, "energy": 2404538256.884094, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.5.ffn_up_exps.weight", "block": 5, "role": "up", "type": "Q5_K", "s": 128416.07556612711, "err_l2": 34.72708511352539, "energy": 2404538256.884094, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.6.ffn_down_exps.weight", "block": 6, "role": "down", "type": "Q5_K", "s": 1566.5687139173328, "err_l2": 35.09141540527344, "energy": 6342106.4899379015, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.6.ffn_gate_exps.weight", "block": 6, "role": "gate", "type": "Q5_K", "s": 146953.86989336918, "err_l2": 38.20359802246094, "energy": 2495879819.4156494, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.6.ffn_up_exps.weight", "block": 6, "role": "up", "type": "Q5_K", "s": 132118.24242444383, "err_l2": 34.63741683959961, "energy": 2495879819.4156494, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.7.ffn_down_exps.weight", "block": 7, "role": "down", "type": "Q5_K", "s": 1961.366460626264, "err_l2": 34.78459930419922, "energy": 8081860.904105902, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.7.ffn_gate_exps.weight", "block": 7, "role": "gate", "type": "Q5_K", "s": 189232.95673546416, "err_l2": 38.29631042480469, "energy": 3024648344.3563538, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.7.ffn_up_exps.weight", "block": 7, "role": "up", "type": "Q5_K", "s": 169170.67856858988, "err_l2": 34.33823776245117, "energy": 3024648344.3563538, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.8.ffn_down_exps.weight", "block": 8, "role": "down", "type": "Q5_K", "s": 2150.599333537757, "err_l2": 34.39470672607422, "energy": 8960753.594052434, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.8.ffn_gate_exps.weight", "block": 8, "role": "gate", "type": "Q5_K", "s": 211108.5407232453, "err_l2": 38.737388610839844, "energy": 3319484814.762665, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.8.ffn_up_exps.weight", "block": 8, "role": "up", "type": "Q5_K", "s": 183457.09004031922, "err_l2": 33.66750717163086, "energy": 3319484814.762665, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.9.ffn_down_exps.weight", "block": 9, "role": "down", "type": "Q5_K", "s": 2417.1632929348602, "err_l2": 35.395694732666016, "energy": 9568770.147459984, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.9.ffn_gate_exps.weight", "block": 9, "role": "gate", "type": "Q5_K", "s": 210331.67998679404, "err_l2": 39.524513244628906, "energy": 3361266052.4784546, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.9.ffn_up_exps.weight", "block": 9, "role": "up", "type": "Q5_K", "s": 188935.389681114, "err_l2": 35.26238250732422, "energy": 3361266052.4784546, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.10.ffn_down_exps.weight", "block": 10, "role": "down", "type": "Q5_K", "s": 2643.5119483037765, "err_l2": 35.99646759033203, "energy": 10111794.935325623, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.10.ffn_gate_exps.weight", "block": 10, "role": "gate", "type": "Q5_K", "s": 195096.06104415772, "err_l2": 39.7840576171875, "energy": 3193724210.449585, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.10.ffn_up_exps.weight", "block": 10, "role": "up", "type": "Q5_K", "s": 181132.67917377438, "err_l2": 36.73598861694336, "energy": 3193724210.449585, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.11.ffn_down_exps.weight", "block": 11, "role": "down", "type": "Q5_K", "s": 3600.7514658217938, "err_l2": 36.2430419921875, "energy": 13562770.490616083, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.11.ffn_gate_exps.weight", "block": 11, "role": "gate", "type": "Q5_K", "s": 226704.08438546569, "err_l2": 39.421730041503906, "energy": 3610363311.6185303, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.11.ffn_up_exps.weight", "block": 11, "role": "up", "type": "Q5_K", "s": 212357.76811110892, "err_l2": 36.913108825683594, "energy": 3610363311.6185303, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.12.ffn_down_exps.weight", "block": 12, "role": "down", "type": "Q5_K", "s": 3055.4603375535225, "err_l2": 35.26779556274414, "energy": 11890972.488930225, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.12.ffn_gate_exps.weight", "block": 12, "role": "gate", "type": "Q5_K", "s": 236089.57367006355, "err_l2": 39.52536392211914, "energy": 3644765104.288269, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.12.ffn_up_exps.weight", "block": 12, "role": "up", "type": "Q5_K", "s": 216868.78989727295, "err_l2": 35.956634521484375, "energy": 3644765104.288269, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.13.ffn_down_exps.weight", "block": 13, "role": "down", "type": "Q5_K", "s": 3804.966137320595, "err_l2": 35.70102310180664, "energy": 14734612.844225299, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.13.ffn_gate_exps.weight", "block": 13, "role": "gate", "type": "Q5_K", "s": 263877.0844030417, "err_l2": 39.6608772277832, "energy": 3992092145.1657104, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.13.ffn_up_exps.weight", "block": 13, "role": "up", "type": "Q5_K", "s": 240736.41343200492, "err_l2": 36.15342712402344, "energy": 3992092145.1657104, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.14.ffn_down_exps.weight", "block": 14, "role": "down", "type": "Q5_K", "s": 3500.13005663359, "err_l2": 35.94085693359375, "energy": 13554481.034674168, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.14.ffn_gate_exps.weight", "block": 14, "role": "gate", "type": "Q5_K", "s": 248749.04674866365, "err_l2": 39.697303771972656, "energy": 3737719776.7888184, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.14.ffn_up_exps.weight", "block": 14, "role": "up", "type": "Q5_K", "s": 227361.04143534432, "err_l2": 36.62542724609375, "energy": 3737719776.7888184, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.15.ffn_down_exps.weight", "block": 15, "role": "down", "type": "Q5_K", "s": 4187.448782302002, "err_l2": 36.1639518737793, "energy": 15960954.2438488, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.15.ffn_gate_exps.weight", "block": 15, "role": "gate", "type": "Q5_K", "s": 280215.8938726947, "err_l2": 39.547096252441406, "energy": 4167948470.2155457, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.15.ffn_up_exps.weight", "block": 15, "role": "up", "type": "Q5_K", "s": 259843.97046211144, "err_l2": 37.007938385009766, "energy": 4167948470.2155457, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.16.ffn_down_exps.weight", "block": 16, "role": "down", "type": "Q5_K", "s": 4576.235849039853, "err_l2": 36.827247619628906, "energy": 17199945.656352043, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.16.ffn_gate_exps.weight", "block": 16, "role": "gate", "type": "Q5_K", "s": 267757.36422190384, "err_l2": 38.949066162109375, "energy": 4078872020.2729187, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.16.ffn_up_exps.weight", "block": 16, "role": "up", "type": "Q5_K", "s": 259744.95273058757, "err_l2": 37.80992126464844, "energy": 4078872020.2729187, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.17.ffn_down_exps.weight", "block": 17, "role": "down", "type": "Q5_K", "s": 4278.018765138681, "err_l2": 36.73914337158203, "energy": 16259451.176771164, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.17.ffn_gate_exps.weight", "block": 17, "role": "gate", "type": "Q5_K", "s": 284979.63614452886, "err_l2": 39.68003845214844, "energy": 4243044980.7101135, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.17.ffn_up_exps.weight", "block": 17, "role": "up", "type": "Q5_K", "s": 267414.91302014596, "err_l2": 37.64183807373047, "energy": 4243044980.7101135, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.18.ffn_down_exps.weight", "block": 18, "role": "down", "type": "Q5_K", "s": 4809.607574994471, "err_l2": 37.08259582519531, "energy": 18323826.341524124, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.18.ffn_gate_exps.weight", "block": 18, "role": "gate", "type": "Q5_K", "s": 297605.6909760426, "err_l2": 40.1346321105957, "energy": 4469796752.81366, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.18.ffn_up_exps.weight", "block": 18, "role": "up", "type": "Q5_K", "s": 278465.8565783062, "err_l2": 37.6745719909668, "energy": 4469796752.81366, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.19.ffn_down_exps.weight", "block": 19, "role": "down", "type": "Q5_K", "s": 5737.706008539612, "err_l2": 37.24573516845703, "energy": 21314045.25215578, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.19.ffn_gate_exps.weight", "block": 19, "role": "gate", "type": "Q5_K", "s": 339425.8130118332, "err_l2": 40.634342193603516, "energy": 4942801179.769653, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.19.ffn_up_exps.weight", "block": 19, "role": "up", "type": "Q5_K", "s": 312482.86468401697, "err_l2": 37.58784866333008, "energy": 4942801179.769653, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.20.ffn_down_exps.weight", "block": 20, "role": "down", "type": "Q5_K", "s": 4857.96909253863, "err_l2": 35.864986419677734, "energy": 19533477.948932648, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.20.ffn_gate_exps.weight", "block": 20, "role": "gate", "type": "Q5_K", "s": 377975.24384990823, "err_l2": 41.558143615722656, "energy": 5309558766.605164, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.20.ffn_up_exps.weight", "block": 20, "role": "up", "type": "Q5_K", "s": 320704.9481656009, "err_l2": 35.32932662963867, "energy": 5309558766.605164, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.21.ffn_down_exps.weight", "block": 21, "role": "down", "type": "Q5_K", "s": 5190.917080725259, "err_l2": 36.32001876831055, "energy": 19783092.441063404, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.21.ffn_gate_exps.weight", "block": 21, "role": "gate", "type": "Q5_K", "s": 356207.6878807908, "err_l2": 41.88343811035156, "energy": 5133279711.727417, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.21.ffn_up_exps.weight", "block": 21, "role": "up", "type": "Q5_K", "s": 318889.20447148697, "err_l2": 36.56703567504883, "energy": 5133279711.727417, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.22.ffn_down_exps.weight", "block": 22, "role": "down", "type": "Q5_K", "s": 5439.420321516767, "err_l2": 37.38140869140625, "energy": 20124856.359775662, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.22.ffn_gate_exps.weight", "block": 22, "role": "gate", "type": "Q5_K", "s": 315603.692009368, "err_l2": 41.21792984008789, "energy": 4779006383.581612, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.22.ffn_up_exps.weight", "block": 22, "role": "up", "type": "Q5_K", "s": 297925.895953572, "err_l2": 38.81044006347656, "energy": 4779006383.581612, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.23.ffn_down_exps.weight", "block": 23, "role": "down", "type": "Q5_K", "s": 5861.5112354581615, "err_l2": 37.87303161621094, "energy": 21217019.830352783, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.23.ffn_gate_exps.weight", "block": 23, "role": "gate", "type": "Q5_K", "s": 323308.7827608487, "err_l2": 40.8481330871582, "energy": 4948538410.368652, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.23.ffn_up_exps.weight", "block": 23, "role": "up", "type": "Q5_K", "s": 310746.32575217733, "err_l2": 39.33385467529297, "energy": 4948538410.368652, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.24.ffn_down_exps.weight", "block": 24, "role": "down", "type": "Q5_K", "s": 5237.80331068404, "err_l2": 37.29518127441406, "energy": 18962488.084349155, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.24.ffn_gate_exps.weight", "block": 24, "role": "gate", "type": "Q5_K", "s": 336370.95884986804, "err_l2": 41.16030502319336, "energy": 4985729777.105591, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.24.ffn_up_exps.weight", "block": 24, "role": "up", "type": "Q5_K", "s": 317419.84282737726, "err_l2": 38.52948760986328, "energy": 4985729777.105591, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.25.ffn_down_exps.weight", "block": 25, "role": "down", "type": "Q5_K", "s": 6035.49297169152, "err_l2": 37.752620697021484, "energy": 22012922.12262726, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.25.ffn_gate_exps.weight", "block": 25, "role": "gate", "type": "Q5_K", "s": 352520.4758013772, "err_l2": 41.77400207519531, "energy": 5205305004.696228, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.25.ffn_up_exps.weight", "block": 25, "role": "up", "type": "Q5_K", "s": 329329.2676082775, "err_l2": 38.77091979980469, "energy": 5205305004.696228, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.26.ffn_down_exps.weight", "block": 26, "role": "down", "type": "Q5_K", "s": 5905.434285376993, "err_l2": 38.394927978515625, "energy": 21354014.354220867, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.26.ffn_gate_exps.weight", "block": 26, "role": "gate", "type": "Q5_K", "s": 336417.99369685404, "err_l2": 41.54265213012695, "energy": 4994181607.238647, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.26.ffn_up_exps.weight", "block": 26, "role": "up", "type": "Q5_K", "s": 317694.87916039187, "err_l2": 39.44468688964844, "energy": 4994181607.238647, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.27.ffn_down_exps.weight", "block": 27, "role": "down", "type": "Q5_K", "s": 7679.18709662083, "err_l2": 39.00667190551758, "energy": 27613217.047315598, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.27.ffn_gate_exps.weight", "block": 27, "role": "gate", "type": "Q5_K", "s": 373609.7741563236, "err_l2": 42.5060920715332, "energy": 5427490272.732422, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.27.ffn_up_exps.weight", "block": 27, "role": "up", "type": "Q5_K", "s": 351261.4287928231, "err_l2": 39.98457336425781, "energy": 5427490272.732422, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.28.ffn_down_exps.weight", "block": 28, "role": "down", "type": "Q5_K", "s": 8496.632866007296, "err_l2": 39.70487976074219, "energy": 30576922.299942017, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.28.ffn_gate_exps.weight", "block": 28, "role": "gate", "type": "Q5_K", "s": 362365.79538141, "err_l2": 42.5920524597168, "energy": 5276713432.383301, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.28.ffn_up_exps.weight", "block": 28, "role": "up", "type": "Q5_K", "s": 346363.1633809811, "err_l2": 40.63900375366211, "energy": 5276713432.383301, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.29.ffn_down_exps.weight", "block": 29, "role": "down", "type": "Q5_K", "s": 9032.494000243041, "err_l2": 39.814308166503906, "energy": 32332774.765409946, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.29.ffn_gate_exps.weight", "block": 29, "role": "gate", "type": "Q5_K", "s": 384869.5987431255, "err_l2": 43.75163650512695, "energy": 5504049370.36377, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.29.ffn_up_exps.weight", "block": 29, "role": "up", "type": "Q5_K", "s": 358771.948391652, "err_l2": 40.70647430419922, "energy": 5504049370.36377, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.30.ffn_down_exps.weight", "block": 30, "role": "down", "type": "Q5_K", "s": 10652.259806958573, "err_l2": 40.085601806640625, "energy": 38829847.31696892, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.30.ffn_gate_exps.weight", "block": 30, "role": "gate", "type": "Q5_K", "s": 418469.57331328845, "err_l2": 44.71826934814453, "energy": 5947769125.285889, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.30.ffn_up_exps.weight", "block": 30, "role": "up", "type": "Q5_K", "s": 381448.8734584397, "err_l2": 40.577484130859375, "energy": 5947769125.285889, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.31.ffn_down_exps.weight", "block": 31, "role": "down", "type": "Q5_K", "s": 15345.58225335298, "err_l2": 41.0534782409668, "energy": 54921343.842552185, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.31.ffn_gate_exps.weight", "block": 31, "role": "gate", "type": "Q5_K", "s": 508574.59275703196, "err_l2": 45.62510299682617, "energy": 6885215013.473389, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.31.ffn_up_exps.weight", "block": 31, "role": "up", "type": "Q5_K", "s": 457769.864224897, "err_l2": 40.830467224121094, "energy": 6885215013.473389, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.32.ffn_down_exps.weight", "block": 32, "role": "down", "type": "Q5_K", "s": 22062.240868700144, "err_l2": 42.10927200317383, "energy": 78752141.1644249, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.32.ffn_gate_exps.weight", "block": 32, "role": "gate", "type": "Q5_K", "s": 580104.8710670602, "err_l2": 47.868560791015625, "energy": 7527876637.479248, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.32.ffn_up_exps.weight", "block": 32, "role": "up", "type": "Q5_K", "s": 512667.9882985299, "err_l2": 42.16421127319336, "energy": 7527876637.479248, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.33.ffn_down_exps.weight", "block": 33, "role": "down", "type": "Q5_K", "s": 29984.114049467964, "err_l2": 43.222564697265625, "energy": 106669200.57768059, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.33.ffn_gate_exps.weight", "block": 33, "role": "gate", "type": "Q5_K", "s": 658498.1887282872, "err_l2": 49.32149124145508, "energy": 8332150965.168579, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.33.ffn_up_exps.weight", "block": 33, "role": "up", "type": "Q5_K", "s": 577161.6412442178, "err_l2": 42.83669662475586, "energy": 8332150965.168579, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.34.ffn_down_exps.weight", "block": 34, "role": "down", "type": "Q5_K", "s": 35658.693236137624, "err_l2": 43.708187103271484, "energy": 124581003.01006317, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.34.ffn_gate_exps.weight", "block": 34, "role": "gate", "type": "Q5_K", "s": 691190.0691991765, "err_l2": 49.664024353027344, "energy": 8550464978.343994, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.34.ffn_up_exps.weight", "block": 34, "role": "up", "type": "Q5_K", "s": 598368.3130182894, "err_l2": 43.06099319458008, "energy": 8550464978.343994, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.35.ffn_down_exps.weight", "block": 35, "role": "down", "type": "Q5_K", "s": 40000.20582083017, "err_l2": 44.798744201660156, "energy": 137147195.31022453, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.35.ffn_gate_exps.weight", "block": 35, "role": "gate", "type": "Q5_K", "s": 826091.3806369101, "err_l2": 49.26906204223633, "energy": 9771642959.588196, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.35.ffn_up_exps.weight", "block": 35, "role": "up", "type": "Q5_K", "s": 713593.2033253899, "err_l2": 42.56196212768555, "energy": 9771642959.588196, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.36.ffn_down_exps.weight", "block": 36, "role": "down", "type": "Q5_K", "s": 50560.54375551507, "err_l2": 46.16388702392578, "energy": 167412832.3093624, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.36.ffn_gate_exps.weight", "block": 36, "role": "gate", "type": "Q5_K", "s": 926680.6895918397, "err_l2": 49.911338806152344, "energy": 10746321069.68451, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.36.ffn_up_exps.weight", "block": 36, "role": "up", "type": "Q5_K", "s": 815203.1572886803, "err_l2": 44.05369567871094, "energy": 10746321069.68451, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.37.ffn_down_exps.weight", "block": 37, "role": "down", "type": "Q5_K", "s": 79581.28731709403, "err_l2": 47.33296203613281, "energy": 260268775.01239014, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.37.ffn_gate_exps.weight", "block": 37, "role": "gate", "type": "Q5_K", "s": 968097.3733590397, "err_l2": 49.674285888671875, "energy": 11005630810.807129, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.37.ffn_up_exps.weight", "block": 37, "role": "up", "type": "Q5_K", "s": 863018.4359576661, "err_l2": 44.95008087158203, "energy": 11005630810.807129, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.38.ffn_down_exps.weight", "block": 38, "role": "down", "type": "Q5_K", "s": 122166.80806186717, "err_l2": 47.919700622558594, "energy": 406006067.78154135, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.38.ffn_gate_exps.weight", "block": 38, "role": "gate", "type": "Q5_K", "s": 967960.8151923514, "err_l2": 51.708770751953125, "energy": 10846040824.944519, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.38.ffn_up_exps.weight", "block": 38, "role": "up", "type": "Q5_K", "s": 866555.8165808959, "err_l2": 46.57724380493164, "energy": 10846040824.944519, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.39.ffn_down_exps.weight", "block": 39, "role": "down", "type": "Q5_K", "s": 187147.79168050908, "err_l2": 49.72294998168945, "energy": 639124982.3202152, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.39.ffn_gate_exps.weight", "block": 39, "role": "gate", "type": "Q5_K", "s": 829950.0277944624, "err_l2": 60.77745819091797, "energy": 8206865322.349945, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.39.ffn_up_exps.weight", "block": 39, "role": "up", "type": "Q5_K", "s": 708736.8571432877, "err_l2": 51.61329650878906, "energy": 8206865322.349945, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.0.ffn_down_exps.weight", "block": 0, "role": "down", "type": "Q4_K", "s": 904.5606738143085, "err_l2": 114.46932983398438, "energy": 1594305.5174544854, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.0.ffn_gate_exps.weight", "block": 0, "role": "gate", "type": "Q4_K", "s": 109003.30879723736, "err_l2": 69.7431869506836, "energy": 1558006241.509983, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.0.ffn_up_exps.weight", "block": 0, "role": "up", "type": "Q4_K", "s": 102647.4724255079, "err_l2": 65.25181579589844, "energy": 1558006241.509983, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.1.ffn_down_exps.weight", "block": 1, "role": "down", "type": "Q4_K", "s": 2286.235266923706, "err_l2": 147.88865661621094, "energy": 2793671.7302908897, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.1.ffn_gate_exps.weight", "block": 1, "role": "gate", "type": "Q4_K", "s": 278604.65535756457, "err_l2": 130.5421600341797, "energy": 1528231540.1870728, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.1.ffn_up_exps.weight", "block": 1, "role": "up", "type": "Q4_K", "s": 264666.54483893863, "err_l2": 123.53633880615234, "energy": 1528231540.1870728, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.2.ffn_down_exps.weight", "block": 2, "role": "down", "type": "Q4_K", "s": 2202.278893513486, "err_l2": 145.50555419921875, "energy": 2394420.025505662, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.2.ffn_gate_exps.weight", "block": 2, "role": "gate", "type": "Q4_K", "s": 337127.0199521544, "err_l2": 140.430419921875, "energy": 1614167290.0648804, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.2.ffn_up_exps.weight", "block": 2, "role": "up", "type": "Q4_K", "s": 323364.3246664765, "err_l2": 134.91009521484375, "energy": 1614167290.0648804, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.3.ffn_down_exps.weight", "block": 3, "role": "down", "type": "Q4_K", "s": 3127.2112012129, "err_l2": 143.1685333251953, "energy": 3253016.4095531586, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.3.ffn_gate_exps.weight", "block": 3, "role": "gate", "type": "Q4_K", "s": 444616.69118109276, "err_l2": 143.18673706054688, "energy": 1987198435.1838608, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.3.ffn_up_exps.weight", "block": 3, "role": "up", "type": "Q4_K", "s": 419080.3379855831, "err_l2": 136.8494873046875, "energy": 1987198435.1838608, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.4.ffn_down_exps.weight", "block": 4, "role": "down", "type": "Q4_K", "s": 4367.785301662576, "err_l2": 141.95372009277344, "energy": 4543147.314295173, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.4.ffn_gate_exps.weight", "block": 4, "role": "gate", "type": "Q4_K", "s": 503354.9157890483, "err_l2": 150.6708984375, "energy": 2219785664.976015, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.4.ffn_up_exps.weight", "block": 4, "role": "up", "type": "Q4_K", "s": 458115.8183072881, "err_l2": 139.49392700195312, "energy": 2219785664.976015, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.5.ffn_down_exps.weight", "block": 5, "role": "down", "type": "Q4_K", "s": 5547.923257796393, "err_l2": 141.80618286132812, "energy": 5663699.233736992, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.5.ffn_gate_exps.weight", "block": 5, "role": "gate", "type": "Q4_K", "s": 559771.5778741615, "err_l2": 152.912841796875, "energy": 2404538256.884094, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.5.ffn_up_exps.weight", "block": 5, "role": "up", "type": "Q4_K", "s": 503709.2305842041, "err_l2": 139.8680877685547, "energy": 2404538256.884094, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.6.ffn_down_exps.weight", "block": 6, "role": "down", "type": "Q4_K", "s": 6182.172462447363, "err_l2": 141.38768005371094, "energy": 6342106.4899379015, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.6.ffn_gate_exps.weight", "block": 6, "role": "gate", "type": "Q4_K", "s": 576054.7506977739, "err_l2": 153.81326293945312, "energy": 2495879819.4156494, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.6.ffn_up_exps.weight", "block": 6, "role": "up", "type": "Q4_K", "s": 518101.3856846899, "err_l2": 139.4791259765625, "energy": 2495879819.4156494, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.7.ffn_down_exps.weight", "block": 7, "role": "down", "type": "Q4_K", "s": 7751.770356168581, "err_l2": 140.0110321044922, "energy": 8081860.904105902, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.7.ffn_gate_exps.weight", "block": 7, "role": "gate", "type": "Q4_K", "s": 744397.3443245585, "err_l2": 154.1573486328125, "energy": 3024648344.3563538, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.7.ffn_up_exps.weight", "block": 7, "role": "up", "type": "Q4_K", "s": 665760.8256388255, "err_l2": 138.22482299804688, "energy": 3024648344.3563538, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.8.ffn_down_exps.weight", "block": 8, "role": "down", "type": "Q4_K", "s": 8492.063094696681, "err_l2": 138.483642578125, "energy": 8960753.594052434, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.8.ffn_gate_exps.weight", "block": 8, "role": "gate", "type": "Q4_K", "s": 831231.6198254562, "err_l2": 155.89459228515625, "energy": 3319484814.762665, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.8.ffn_up_exps.weight", "block": 8, "role": "up", "type": "Q4_K", "s": 722673.5590085953, "err_l2": 135.47213745117188, "energy": 3319484814.762665, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.9.ffn_down_exps.weight", "block": 9, "role": "down", "type": "Q4_K", "s": 9553.75114448638, "err_l2": 142.46319580078125, "energy": 9568770.147459984, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.9.ffn_gate_exps.weight", "block": 9, "role": "gate", "type": "Q4_K", "s": 825478.1249887358, "err_l2": 159.20936584472656, "energy": 3361266052.4784546, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.9.ffn_up_exps.weight", "block": 9, "role": "up", "type": "Q4_K", "s": 741618.4252258086, "err_l2": 142.0214080810547, "energy": 3361266052.4784546, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.10.ffn_down_exps.weight", "block": 10, "role": "down", "type": "Q4_K", "s": 10461.471371797026, "err_l2": 144.85719299316406, "energy": 10111794.935325623, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.10.ffn_gate_exps.weight", "block": 10, "role": "gate", "type": "Q4_K", "s": 763888.1998486838, "err_l2": 160.34303283691406, "energy": 3193724210.449585, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.10.ffn_up_exps.weight", "block": 10, "role": "up", "type": "Q4_K", "s": 709346.8216617616, "err_l2": 148.0549774169922, "energy": 3193724210.449585, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.11.ffn_down_exps.weight", "block": 11, "role": "down", "type": "Q4_K", "s": 14253.278066675439, "err_l2": 145.75210571289062, "energy": 13562770.490616083, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.11.ffn_gate_exps.weight", "block": 11, "role": "gate", "type": "Q4_K", "s": 890438.5040536918, "err_l2": 158.7361602783203, "energy": 3610363311.6185303, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.11.ffn_up_exps.weight", "block": 11, "role": "up", "type": "Q4_K", "s": 834511.0825568002, "err_l2": 148.63836669921875, "energy": 3610363311.6185303, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.12.ffn_down_exps.weight", "block": 12, "role": "down", "type": "Q4_K", "s": 12095.453870020778, "err_l2": 141.7747344970703, "energy": 11890972.488930225, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.12.ffn_gate_exps.weight", "block": 12, "role": "gate", "type": "Q4_K", "s": 929753.3876399586, "err_l2": 159.04293823242188, "energy": 3644765104.288269, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.12.ffn_up_exps.weight", "block": 12, "role": "up", "type": "Q4_K", "s": 854486.9534617885, "err_l2": 144.67784118652344, "energy": 3644765104.288269, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.13.ffn_down_exps.weight", "block": 13, "role": "down", "type": "Q4_K", "s": 15058.819132094404, "err_l2": 143.56585693359375, "energy": 14734612.844225299, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.13.ffn_gate_exps.weight", "block": 13, "role": "gate", "type": "Q4_K", "s": 1039826.6368090335, "err_l2": 159.5388641357422, "energy": 3992092145.1657104, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.13.ffn_up_exps.weight", "block": 13, "role": "up", "type": "Q4_K", "s": 949435.9156870559, "err_l2": 145.44171142578125, "energy": 3992092145.1657104, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.14.ffn_down_exps.weight", "block": 14, "role": "down", "type": "Q4_K", "s": 13843.984930755834, "err_l2": 144.48953247070312, "energy": 13554481.034674168, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.14.ffn_gate_exps.weight", "block": 14, "role": "gate", "type": "Q4_K", "s": 980884.8576365163, "err_l2": 159.6781463623047, "energy": 3737719776.7888184, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.14.ffn_up_exps.weight", "block": 14, "role": "up", "type": "Q4_K", "s": 897147.6857806524, "err_l2": 147.32298278808594, "energy": 3737719776.7888184, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.15.ffn_down_exps.weight", "block": 15, "role": "down", "type": "Q4_K", "s": 16578.33850762985, "err_l2": 145.33197021484375, "energy": 15960954.2438488, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.15.ffn_gate_exps.weight", "block": 15, "role": "gate", "type": "Q4_K", "s": 1106053.3792676793, "err_l2": 159.0278778076172, "energy": 4167948470.2155457, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.15.ffn_up_exps.weight", "block": 15, "role": "up", "type": "Q4_K", "s": 1026187.1340636057, "err_l2": 148.79632568359375, "energy": 4167948470.2155457, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.16.ffn_down_exps.weight", "block": 16, "role": "down", "type": "Q4_K", "s": 18101.282901151168, "err_l2": 148.02835083007812, "energy": 17199945.656352043, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.16.ffn_gate_exps.weight", "block": 16, "role": "gate", "type": "Q4_K", "s": 1056751.3175107618, "err_l2": 156.603271484375, "energy": 4078872020.2729187, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.16.ffn_up_exps.weight", "block": 16, "role": "up", "type": "Q4_K", "s": 1025596.2617089952, "err_l2": 152.02288818359375, "energy": 4078872020.2729187, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.17.ffn_down_exps.weight", "block": 17, "role": "down", "type": "Q4_K", "s": 16937.067006507878, "err_l2": 147.66860961914062, "energy": 16259451.176771164, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.17.ffn_gate_exps.weight", "block": 17, "role": "gate", "type": "Q4_K", "s": 1123830.8874538988, "err_l2": 159.5614776611328, "energy": 4243044980.7101135, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.17.ffn_up_exps.weight", "block": 17, "role": "up", "type": "Q4_K", "s": 1055264.778983379, "err_l2": 151.3518829345703, "energy": 4243044980.7101135, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.18.ffn_down_exps.weight", "block": 18, "role": "down", "type": "Q4_K", "s": 19020.889615901044, "err_l2": 149.10885620117188, "energy": 18323826.341524124, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.18.ffn_gate_exps.weight", "block": 18, "role": "gate", "type": "Q4_K", "s": 1172739.0909496634, "err_l2": 161.4282989501953, "energy": 4469796752.81366, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.18.ffn_up_exps.weight", "block": 18, "role": "up", "type": "Q4_K", "s": 1098021.5706024128, "err_l2": 151.52981567382812, "energy": 4469796752.81366, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.19.ffn_down_exps.weight", "block": 19, "role": "down", "type": "Q4_K", "s": 22713.38000082114, "err_l2": 149.7120819091797, "energy": 21314045.25215578, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.19.ffn_gate_exps.weight", "block": 19, "role": "gate", "type": "Q4_K", "s": 1339365.2300284891, "err_l2": 163.34063720703125, "energy": 4942801179.769653, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.19.ffn_up_exps.weight", "block": 19, "role": "up", "type": "Q4_K", "s": 1234238.0313683825, "err_l2": 151.0924530029297, "energy": 4942801179.769653, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.20.ffn_down_exps.weight", "block": 20, "role": "down", "type": "Q4_K", "s": 19187.085592627798, "err_l2": 144.27554321289062, "energy": 19533477.948932648, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.20.ffn_gate_exps.weight", "block": 20, "role": "gate", "type": "Q4_K", "s": 1493096.0128068011, "err_l2": 167.0875244140625, "energy": 5309558766.605164, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.20.ffn_up_exps.weight", "block": 20, "role": "up", "type": "Q4_K", "s": 1267937.750920109, "err_l2": 142.01596069335938, "energy": 5309558766.605164, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.21.ffn_down_exps.weight", "block": 21, "role": "down", "type": "Q4_K", "s": 20530.863076498474, "err_l2": 146.06817626953125, "energy": 19783092.441063404, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.21.ffn_gate_exps.weight", "block": 21, "role": "gate", "type": "Q4_K", "s": 1404305.248386441, "err_l2": 168.4664764404297, "energy": 5133279711.727417, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.21.ffn_up_exps.weight", "block": 21, "role": "up", "type": "Q4_K", "s": 1257875.5942074456, "err_l2": 147.0392303466797, "energy": 5133279711.727417, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.22.ffn_down_exps.weight", "block": 22, "role": "down", "type": "Q4_K", "s": 21480.323126896288, "err_l2": 150.27906799316406, "energy": 20124856.359775662, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.22.ffn_gate_exps.weight", "block": 22, "role": "gate", "type": "Q4_K", "s": 1239997.2652860908, "err_l2": 165.95860290527344, "energy": 4779006383.581612, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.22.ffn_up_exps.weight", "block": 22, "role": "up", "type": "Q4_K", "s": 1171184.482194255, "err_l2": 156.2431182861328, "energy": 4779006383.581612, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.23.ffn_down_exps.weight", "block": 23, "role": "down", "type": "Q4_K", "s": 23215.21563532881, "err_l2": 152.19277954101562, "energy": 21217019.830352783, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.23.ffn_gate_exps.weight", "block": 23, "role": "gate", "type": "Q4_K", "s": 1271343.116664958, "err_l2": 164.36404418945312, "energy": 4948538410.368652, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.23.ffn_up_exps.weight", "block": 23, "role": "up", "type": "Q4_K", "s": 1222449.351907585, "err_l2": 158.25314331054688, "energy": 4948538410.368652, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.24.ffn_down_exps.weight", "block": 24, "role": "down", "type": "Q4_K", "s": 20764.811404115688, "err_l2": 149.78382873535156, "energy": 18962488.084349155, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.24.ffn_gate_exps.weight", "block": 24, "role": "gate", "type": "Q4_K", "s": 1325560.4974175675, "err_l2": 165.5309600830078, "energy": 4985729777.105591, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.24.ffn_up_exps.weight", "block": 24, "role": "up", "type": "Q4_K", "s": 1251637.4805001717, "err_l2": 154.93333435058594, "energy": 4985729777.105591, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.25.ffn_down_exps.weight", "block": 25, "role": "down", "type": "Q4_K", "s": 23887.292627975847, "err_l2": 151.75502014160156, "energy": 22012922.12262726, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.25.ffn_gate_exps.weight", "block": 25, "role": "gate", "type": "Q4_K", "s": 1388184.32441362, "err_l2": 168.03384399414062, "energy": 5205305004.696228, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.25.ffn_up_exps.weight", "block": 25, "role": "up", "type": "Q4_K", "s": 1297989.5490480454, "err_l2": 155.94760131835938, "energy": 5205305004.696228, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.26.ffn_down_exps.weight", "block": 26, "role": "down", "type": "Q4_K", "s": 23359.13935824708, "err_l2": 154.30184936523438, "energy": 21354014.354220867, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.26.ffn_gate_exps.weight", "block": 26, "role": "gate", "type": "Q4_K", "s": 1324641.6400366693, "err_l2": 167.13731384277344, "energy": 4994181607.238647, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.26.ffn_up_exps.weight", "block": 26, "role": "up", "type": "Q4_K", "s": 1251517.172096246, "err_l2": 158.68096923828125, "energy": 4994181607.238647, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.27.ffn_down_exps.weight", "block": 27, "role": "down", "type": "Q4_K", "s": 30328.125938532896, "err_l2": 156.8855743408203, "energy": 27613217.047315598, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.27.ffn_gate_exps.weight", "block": 27, "role": "gate", "type": "Q4_K", "s": 1470424.3580718576, "err_l2": 171.0621337890625, "energy": 5427490272.732422, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.27.ffn_up_exps.weight", "block": 27, "role": "up", "type": "Q4_K", "s": 1382969.9334289453, "err_l2": 160.85391235351562, "energy": 5427490272.732422, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.28.ffn_down_exps.weight", "block": 28, "role": "down", "type": "Q4_K", "s": 33501.644135256814, "err_l2": 159.860595703125, "energy": 30576922.299942017, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.28.ffn_gate_exps.weight", "block": 28, "role": "gate", "type": "Q4_K", "s": 1425016.9439697345, "err_l2": 171.46156311035156, "energy": 5276713432.383301, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.28.ffn_up_exps.weight", "block": 28, "role": "up", "type": "Q4_K", "s": 1363078.6097184534, "err_l2": 163.59422302246094, "energy": 5276713432.383301, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.29.ffn_down_exps.weight", "block": 29, "role": "down", "type": "Q4_K", "s": 35560.15628591796, "err_l2": 160.37210083007812, "energy": 32332774.765409946, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.29.ffn_gate_exps.weight", "block": 29, "role": "gate", "type": "Q4_K", "s": 1510377.1595138572, "err_l2": 176.29962158203125, "energy": 5504049370.36377, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.29.ffn_up_exps.weight", "block": 29, "role": "up", "type": "Q4_K", "s": 1409029.9411262888, "err_l2": 164.0252685546875, "energy": 5504049370.36377, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.30.ffn_down_exps.weight", "block": 30, "role": "down", "type": "Q4_K", "s": 41917.79067354375, "err_l2": 161.5420379638672, "energy": 38829847.31696892, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.30.ffn_gate_exps.weight", "block": 30, "role": "gate", "type": "Q4_K", "s": 1642079.5280428468, "err_l2": 180.16162109375, "energy": 5947769125.285889, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.30.ffn_up_exps.weight", "block": 30, "role": "up", "type": "Q4_K", "s": 1497880.5778222496, "err_l2": 163.4658966064453, "energy": 5947769125.285889, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.31.ffn_down_exps.weight", "block": 31, "role": "down", "type": "Q4_K", "s": 60279.762060155306, "err_l2": 165.54791259765625, "energy": 54921343.842552185, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.31.ffn_gate_exps.weight", "block": 31, "role": "gate", "type": "Q4_K", "s": 1998600.861498341, "err_l2": 183.78244018554688, "energy": 6885215013.473389, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.31.ffn_up_exps.weight", "block": 31, "role": "up", "type": "Q4_K", "s": 1800608.3001958097, "err_l2": 164.41419982910156, "energy": 6885215013.473389, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.32.ffn_down_exps.weight", "block": 32, "role": "down", "type": "Q4_K", "s": 86113.53062897554, "err_l2": 170.2176513671875, "energy": 78752141.1644249, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.32.ffn_gate_exps.weight", "block": 32, "role": "gate", "type": "Q4_K", "s": 2268189.5347059513, "err_l2": 193.1435546875, "energy": 7527876637.479248, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.32.ffn_up_exps.weight", "block": 32, "role": "up", "type": "Q4_K", "s": 2007690.590525881, "err_l2": 170.0263214111328, "energy": 7527876637.479248, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.33.ffn_down_exps.weight", "block": 33, "role": "down", "type": "Q4_K", "s": 116517.94076608548, "err_l2": 174.8612518310547, "energy": 106669200.57768059, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.33.ffn_gate_exps.weight", "block": 33, "role": "gate", "type": "Q4_K", "s": 2575970.8580002342, "err_l2": 198.9849853515625, "energy": 8332150965.168579, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.33.ffn_up_exps.weight", "block": 33, "role": "up", "type": "Q4_K", "s": 2261560.4564967453, "err_l2": 172.78236389160156, "energy": 8332150965.168579, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.34.ffn_down_exps.weight", "block": 34, "role": "down", "type": "Q4_K", "s": 138844.5602793648, "err_l2": 176.6785125732422, "energy": 124581003.01006317, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.34.ffn_gate_exps.weight", "block": 34, "role": "gate", "type": "Q4_K", "s": 2708457.494306678, "err_l2": 200.23069763183594, "energy": 8550464978.343994, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.34.ffn_up_exps.weight", "block": 34, "role": "up", "type": "Q4_K", "s": 2347195.44641228, "err_l2": 173.54344177246094, "energy": 8550464978.343994, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.35.ffn_down_exps.weight", "block": 35, "role": "down", "type": "Q4_K", "s": 155774.89483261143, "err_l2": 181.27171325683594, "energy": 137147195.31022453, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.35.ffn_gate_exps.weight", "block": 35, "role": "gate", "type": "Q4_K", "s": 3260481.6191513753, "err_l2": 198.25904846191406, "energy": 9771642959.588196, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.35.ffn_up_exps.weight", "block": 35, "role": "up", "type": "Q4_K", "s": 2818234.0720600123, "err_l2": 171.2122344970703, "energy": 9771642959.588196, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.36.ffn_down_exps.weight", "block": 36, "role": "down", "type": "Q4_K", "s": 196529.67111484383, "err_l2": 186.93150329589844, "energy": 167412832.3093624, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.36.ffn_gate_exps.weight", "block": 36, "role": "gate", "type": "Q4_K", "s": 3657960.2179009407, "err_l2": 200.87742614746094, "energy": 10746321069.68451, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.36.ffn_up_exps.weight", "block": 36, "role": "up", "type": "Q4_K", "s": 3218541.0318971556, "err_l2": 177.18356323242188, "energy": 10746321069.68451, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.37.ffn_down_exps.weight", "block": 37, "role": "down", "type": "Q4_K", "s": 308105.257826097, "err_l2": 191.95196533203125, "energy": 260268775.01239014, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.37.ffn_gate_exps.weight", "block": 37, "role": "gate", "type": "Q4_K", "s": 3824885.490338578, "err_l2": 199.86402893066406, "energy": 11005630810.807129, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.37.ffn_up_exps.weight", "block": 37, "role": "up", "type": "Q4_K", "s": 3410990.7677083793, "err_l2": 180.7641143798828, "energy": 11005630810.807129, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.38.ffn_down_exps.weight", "block": 38, "role": "down", "type": "Q4_K", "s": 470263.8502647374, "err_l2": 194.58363342285156, "energy": 406006067.78154135, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.38.ffn_gate_exps.weight", "block": 38, "role": "gate", "type": "Q4_K", "s": 3821390.12925755, "err_l2": 207.9365234375, "energy": 10846040824.944519, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.38.ffn_up_exps.weight", "block": 38, "role": "up", "type": "Q4_K", "s": 3422778.59115517, "err_l2": 187.2698516845703, "energy": 10846040824.944519, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.39.ffn_down_exps.weight", "block": 39, "role": "down", "type": "Q4_K", "s": 712354.9798758981, "err_l2": 202.43844604492188, "energy": 639124982.3202152, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.39.ffn_gate_exps.weight", "block": 39, "role": "gate", "type": "Q4_K", "s": 3279904.206502961, "err_l2": 244.09165954589844, "energy": 8206865322.349945, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.39.ffn_up_exps.weight", "block": 39, "role": "up", "type": "Q4_K", "s": 2801581.4633014114, "err_l2": 207.33071899414062, "energy": 8206865322.349945, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.0.ffn_down_exps.weight", "block": 0, "role": "down", "type": "IQ4_XS", "s": 1259.8207992760326, "err_l2": 81.13033294677734, "energy": 1594305.5174544854, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.0.ffn_gate_exps.weight", "block": 0, "role": "gate", "type": "IQ4_XS", "s": 144455.1384179555, "err_l2": 78.18563079833984, "energy": 1558006241.509983, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.0.ffn_up_exps.weight", "block": 0, "role": "up", "type": "IQ4_XS", "s": 134987.4882631177, "err_l2": 73.1617202758789, "energy": 1558006241.509983, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.1.ffn_down_exps.weight", "block": 1, "role": "down", "type": "IQ4_XS", "s": 2810.1344985543587, "err_l2": 162.6932830810547, "energy": 2793671.7302908897, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.1.ffn_gate_exps.weight", "block": 1, "role": "gate", "type": "IQ4_XS", "s": 339425.25549406506, "err_l2": 146.4847412109375, "energy": 1528231540.1870728, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.1.ffn_up_exps.weight", "block": 1, "role": "up", "type": "IQ4_XS", "s": 322413.8550113233, "err_l2": 138.52975463867188, "energy": 1528231540.1870728, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.2.ffn_down_exps.weight", "block": 2, "role": "down", "type": "IQ4_XS", "s": 2619.534305754281, "err_l2": 162.68707275390625, "energy": 2394420.025505662, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.2.ffn_gate_exps.weight", "block": 2, "role": "gate", "type": "IQ4_XS", "s": 404157.5522056011, "err_l2": 157.6260986328125, "energy": 1614167290.0648804, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.2.ffn_up_exps.weight", "block": 2, "role": "up", "type": "IQ4_XS", "s": 387542.5783650017, "err_l2": 151.35287475585938, "energy": 1614167290.0648804, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.3.ffn_down_exps.weight", "block": 3, "role": "down", "type": "IQ4_XS", "s": 3678.9798111831265, "err_l2": 160.96746826171875, "energy": 3253016.4095531586, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.3.ffn_gate_exps.weight", "block": 3, "role": "gate", "type": "IQ4_XS", "s": 529923.470753506, "err_l2": 160.75413513183594, "energy": 1987198435.1838608, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.3.ffn_up_exps.weight", "block": 3, "role": "up", "type": "IQ4_XS", "s": 498855.4799163559, "err_l2": 153.464599609375, "energy": 1987198435.1838608, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.4.ffn_down_exps.weight", "block": 4, "role": "down", "type": "IQ4_XS", "s": 5139.768291076599, "err_l2": 159.60592651367188, "energy": 4543147.314295173, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.4.ffn_gate_exps.weight", "block": 4, "role": "gate", "type": "IQ4_XS", "s": 604720.4459662072, "err_l2": 169.3643035888672, "energy": 2219785664.976015, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.4.ffn_up_exps.weight", "block": 4, "role": "up", "type": "IQ4_XS", "s": 550247.1667894692, "err_l2": 156.6046905517578, "energy": 2219785664.976015, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.5.ffn_down_exps.weight", "block": 5, "role": "down", "type": "IQ4_XS", "s": 6503.682446369712, "err_l2": 159.74485778808594, "energy": 5663699.233736992, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.5.ffn_gate_exps.weight", "block": 5, "role": "gate", "type": "IQ4_XS", "s": 670632.6317971624, "err_l2": 171.97320556640625, "energy": 2404538256.884094, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.5.ffn_up_exps.weight", "block": 5, "role": "up", "type": "IQ4_XS", "s": 603300.8419966964, "err_l2": 157.01963806152344, "energy": 2404538256.884094, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.6.ffn_down_exps.weight", "block": 6, "role": "down", "type": "IQ4_XS", "s": 7247.613407809629, "err_l2": 159.3434295654297, "energy": 6342106.4899379015, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.6.ffn_gate_exps.weight", "block": 6, "role": "gate", "type": "IQ4_XS", "s": 689721.4002039246, "err_l2": 172.92996215820312, "energy": 2495879819.4156494, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.6.ffn_up_exps.weight", "block": 6, "role": "up", "type": "IQ4_XS", "s": 620855.0477943119, "err_l2": 156.55101013183594, "energy": 2495879819.4156494, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.7.ffn_down_exps.weight", "block": 7, "role": "down", "type": "IQ4_XS", "s": 9062.466310708063, "err_l2": 158.04388427734375, "energy": 8081860.904105902, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.7.ffn_gate_exps.weight", "block": 7, "role": "gate", "type": "IQ4_XS", "s": 882304.0442765035, "err_l2": 173.41123962402344, "energy": 3024648344.3563538, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.7.ffn_up_exps.weight", "block": 7, "role": "up", "type": "IQ4_XS", "s": 788486.5120166934, "err_l2": 155.129638671875, "energy": 3024648344.3563538, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.8.ffn_down_exps.weight", "block": 8, "role": "down", "type": "IQ4_XS", "s": 9946.181972783643, "err_l2": 156.21725463867188, "energy": 8960753.594052434, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.8.ffn_gate_exps.weight", "block": 8, "role": "gate", "type": "IQ4_XS", "s": 982649.3310827628, "err_l2": 175.57708740234375, "energy": 3319484814.762665, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.8.ffn_up_exps.weight", "block": 8, "role": "up", "type": "IQ4_XS", "s": 853721.874117513, "err_l2": 152.10794067382812, "energy": 3319484814.762665, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.9.ffn_down_exps.weight", "block": 9, "role": "down", "type": "IQ4_XS", "s": 11150.721154573253, "err_l2": 160.69696044921875, "energy": 9568770.147459984, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.9.ffn_gate_exps.weight", "block": 9, "role": "gate", "type": "IQ4_XS", "s": 982058.1796713608, "err_l2": 178.76356506347656, "energy": 3361266052.4784546, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.9.ffn_up_exps.weight", "block": 9, "role": "up", "type": "IQ4_XS", "s": 882104.9900405867, "err_l2": 159.04322814941406, "energy": 3361266052.4784546, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.10.ffn_down_exps.weight", "block": 10, "role": "down", "type": "IQ4_XS", "s": 12164.027696207073, "err_l2": 163.46441650390625, "energy": 10111794.935325623, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.10.ffn_gate_exps.weight", "block": 10, "role": "gate", "type": "IQ4_XS", "s": 913272.0245548842, "err_l2": 179.7646942138672, "energy": 3193724210.449585, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.10.ffn_up_exps.weight", "block": 10, "role": "up", "type": "IQ4_XS", "s": 848192.5413026639, "err_l2": 165.65567016601562, "energy": 3193724210.449585, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.11.ffn_down_exps.weight", "block": 11, "role": "down", "type": "IQ4_XS", "s": 16537.641570376967, "err_l2": 164.2898712158203, "energy": 13562770.490616083, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.11.ffn_gate_exps.weight", "block": 11, "role": "gate", "type": "IQ4_XS", "s": 1057560.5918540258, "err_l2": 178.02964782714844, "energy": 3610363311.6185303, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.11.ffn_up_exps.weight", "block": 11, "role": "up", "type": "IQ4_XS", "s": 989991.1621813548, "err_l2": 166.3628387451172, "energy": 3610363311.6185303, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.12.ffn_down_exps.weight", "block": 12, "role": "down", "type": "IQ4_XS", "s": 14051.786283671889, "err_l2": 160.22509765625, "energy": 11890972.488930225, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.12.ffn_gate_exps.weight", "block": 12, "role": "gate", "type": "IQ4_XS", "s": 1094661.7731498063, "err_l2": 178.55311584472656, "energy": 3644765104.288269, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.12.ffn_up_exps.weight", "block": 12, "role": "up", "type": "IQ4_XS", "s": 1004580.6612261212, "err_l2": 161.95059204101562, "energy": 3644765104.288269, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.13.ffn_down_exps.weight", "block": 13, "role": "down", "type": "IQ4_XS", "s": 17513.78525853192, "err_l2": 162.09388732910156, "energy": 14734612.844225299, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.13.ffn_gate_exps.weight", "block": 13, "role": "gate", "type": "IQ4_XS", "s": 1223332.9658601657, "err_l2": 179.21742248535156, "energy": 3992092145.1657104, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.13.ffn_up_exps.weight", "block": 13, "role": "up", "type": "IQ4_XS", "s": 1114307.8859709743, "err_l2": 162.79237365722656, "energy": 3992092145.1657104, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.14.ffn_down_exps.weight", "block": 14, "role": "down", "type": "IQ4_XS", "s": 16115.019052806085, "err_l2": 163.3257598876953, "energy": 13554481.034674168, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.14.ffn_gate_exps.weight", "block": 14, "role": "gate", "type": "IQ4_XS", "s": 1151806.60830066, "err_l2": 179.25596618652344, "energy": 3737719776.7888184, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.14.ffn_up_exps.weight", "block": 14, "role": "up", "type": "IQ4_XS", "s": 1051931.7977624536, "err_l2": 164.96002197265625, "energy": 3737719776.7888184, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.15.ffn_down_exps.weight", "block": 15, "role": "down", "type": "IQ4_XS", "s": 19259.89066919859, "err_l2": 164.33636474609375, "energy": 15960954.2438488, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.15.ffn_gate_exps.weight", "block": 15, "role": "gate", "type": "IQ4_XS", "s": 1298410.3387051183, "err_l2": 178.6608123779297, "energy": 4167948470.2155457, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.15.ffn_up_exps.weight", "block": 15, "role": "up", "type": "IQ4_XS", "s": 1202047.4464933036, "err_l2": 166.7231903076172, "energy": 4167948470.2155457, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.16.ffn_down_exps.weight", "block": 16, "role": "down", "type": "IQ4_XS", "s": 21042.551388577693, "err_l2": 167.31991577148438, "energy": 17199945.656352043, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.16.ffn_gate_exps.weight", "block": 16, "role": "gate", "type": "IQ4_XS", "s": 1239286.4835024155, "err_l2": 175.79080200195312, "energy": 4078872020.2729187, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.16.ffn_up_exps.weight", "block": 16, "role": "up", "type": "IQ4_XS", "s": 1200338.9907984857, "err_l2": 170.30967712402344, "energy": 4078872020.2729187, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.17.ffn_down_exps.weight", "block": 17, "role": "down", "type": "IQ4_XS", "s": 19660.991951557968, "err_l2": 166.93028259277344, "energy": 16259451.176771164, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.17.ffn_gate_exps.weight", "block": 17, "role": "gate", "type": "IQ4_XS", "s": 1319644.102496407, "err_l2": 179.0358123779297, "energy": 4243044980.7101135, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.17.ffn_up_exps.weight", "block": 17, "role": "up", "type": "IQ4_XS", "s": 1235756.9761406153, "err_l2": 169.5189971923828, "energy": 4243044980.7101135, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.18.ffn_down_exps.weight", "block": 18, "role": "down", "type": "IQ4_XS", "s": 22116.003337872717, "err_l2": 168.5118408203125, "energy": 18323826.341524124, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.18.ffn_gate_exps.weight", "block": 18, "role": "gate", "type": "IQ4_XS", "s": 1381527.8427209149, "err_l2": 181.2363739013672, "energy": 4469796752.81366, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.18.ffn_up_exps.weight", "block": 18, "role": "up", "type": "IQ4_XS", "s": 1290383.947973005, "err_l2": 169.7872314453125, "energy": 4469796752.81366, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.19.ffn_down_exps.weight", "block": 19, "role": "down", "type": "IQ4_XS", "s": 26350.38198105933, "err_l2": 169.20541381835938, "energy": 21314045.25215578, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.19.ffn_gate_exps.weight", "block": 19, "role": "gate", "type": "IQ4_XS", "s": 1572916.264355122, "err_l2": 183.60552978515625, "energy": 4942801179.769653, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.19.ffn_up_exps.weight", "block": 19, "role": "up", "type": "IQ4_XS", "s": 1444066.2327120642, "err_l2": 169.4422607421875, "energy": 4942801179.769653, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.20.ffn_down_exps.weight", "block": 20, "role": "down", "type": "IQ4_XS", "s": 22443.59875495332, "err_l2": 162.91847229003906, "energy": 19533477.948932648, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.20.ffn_gate_exps.weight", "block": 20, "role": "gate", "type": "IQ4_XS", "s": 1751987.2164654492, "err_l2": 188.39930725097656, "energy": 5309558766.605164, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.20.ffn_up_exps.weight", "block": 20, "role": "up", "type": "IQ4_XS", "s": 1481386.301068045, "err_l2": 159.48374938964844, "energy": 5309558766.605164, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.21.ffn_down_exps.weight", "block": 21, "role": "down", "type": "IQ4_XS", "s": 23880.028409866536, "err_l2": 164.85488891601562, "energy": 19783092.441063404, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.21.ffn_gate_exps.weight", "block": 21, "role": "gate", "type": "IQ4_XS", "s": 1650307.1033646653, "err_l2": 189.37741088867188, "energy": 5133279711.727417, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.21.ffn_up_exps.weight", "block": 21, "role": "up", "type": "IQ4_XS", "s": 1472907.870193245, "err_l2": 164.7708740234375, "energy": 5133279711.727417, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.22.ffn_down_exps.weight", "block": 22, "role": "down", "type": "IQ4_XS", "s": 24981.200120458052, "err_l2": 169.77471923828125, "energy": 20124856.359775662, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.22.ffn_gate_exps.weight", "block": 22, "role": "gate", "type": "IQ4_XS", "s": 1466288.5233050324, "err_l2": 186.03541564941406, "energy": 4779006383.581612, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.22.ffn_up_exps.weight", "block": 22, "role": "up", "type": "IQ4_XS", "s": 1382385.9051855155, "err_l2": 174.73709106445312, "energy": 4779006383.581612, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.23.ffn_down_exps.weight", "block": 23, "role": "down", "type": "IQ4_XS", "s": 26886.61532217691, "err_l2": 171.75901794433594, "energy": 21217019.830352783, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.23.ffn_gate_exps.weight", "block": 23, "role": "gate", "type": "IQ4_XS", "s": 1503493.0289595628, "err_l2": 184.47232055664062, "energy": 4948538410.368652, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.23.ffn_up_exps.weight", "block": 23, "role": "up", "type": "IQ4_XS", "s": 1443031.5044482534, "err_l2": 177.228271484375, "energy": 4948538410.368652, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.24.ffn_down_exps.weight", "block": 24, "role": "down", "type": "IQ4_XS", "s": 23998.113584667815, "err_l2": 169.4828643798828, "energy": 18962488.084349155, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.24.ffn_gate_exps.weight", "block": 24, "role": "gate", "type": "IQ4_XS", "s": 1559672.502231276, "err_l2": 185.9626007080078, "energy": 4985729777.105591, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.24.ffn_up_exps.weight", "block": 24, "role": "up", "type": "IQ4_XS", "s": 1469476.5815079226, "err_l2": 173.56227111816406, "energy": 4985729777.105591, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.25.ffn_down_exps.weight", "block": 25, "role": "down", "type": "IQ4_XS", "s": 27702.808401248636, "err_l2": 171.40052795410156, "energy": 22012922.12262726, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.25.ffn_gate_exps.weight", "block": 25, "role": "gate", "type": "IQ4_XS", "s": 1636387.9592345927, "err_l2": 188.69606018066406, "energy": 5205305004.696228, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.25.ffn_up_exps.weight", "block": 25, "role": "up", "type": "IQ4_XS", "s": 1527230.016242748, "err_l2": 174.68353271484375, "energy": 5205305004.696228, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.26.ffn_down_exps.weight", "block": 26, "role": "down", "type": "IQ4_XS", "s": 27114.96862881472, "err_l2": 174.39450073242188, "energy": 21354014.354220867, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.26.ffn_gate_exps.weight", "block": 26, "role": "gate", "type": "IQ4_XS", "s": 1562910.096352114, "err_l2": 187.68505859375, "energy": 4994181607.238647, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.26.ffn_up_exps.weight", "block": 26, "role": "up", "type": "IQ4_XS", "s": 1474973.1949190728, "err_l2": 177.83119201660156, "energy": 4994181607.238647, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.27.ffn_down_exps.weight", "block": 27, "role": "down", "type": "IQ4_XS", "s": 35305.585671233785, "err_l2": 177.130615234375, "energy": 27613217.047315598, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.27.ffn_gate_exps.weight", "block": 27, "role": "gate", "type": "IQ4_XS", "s": 1737112.0673812428, "err_l2": 192.0638885498047, "energy": 5427490272.732422, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.27.ffn_up_exps.weight", "block": 27, "role": "up", "type": "IQ4_XS", "s": 1631963.4376121087, "err_l2": 180.29029846191406, "energy": 5427490272.732422, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.28.ffn_down_exps.weight", "block": 28, "role": "down", "type": "IQ4_XS", "s": 39095.15756100568, "err_l2": 180.17601013183594, "energy": 30576922.299942017, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.28.ffn_gate_exps.weight", "block": 28, "role": "gate", "type": "IQ4_XS", "s": 1684585.1351257523, "err_l2": 192.27969360351562, "energy": 5276713432.383301, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.28.ffn_up_exps.weight", "block": 28, "role": "up", "type": "IQ4_XS", "s": 1608196.221090164, "err_l2": 183.16058349609375, "energy": 5276713432.383301, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.29.ffn_down_exps.weight", "block": 29, "role": "down", "type": "IQ4_XS", "s": 41576.29858749395, "err_l2": 180.5443878173828, "energy": 32332774.765409946, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.29.ffn_gate_exps.weight", "block": 29, "role": "gate", "type": "IQ4_XS", "s": 1792501.1311253053, "err_l2": 197.41737365722656, "energy": 5504049370.36377, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.29.ffn_up_exps.weight", "block": 29, "role": "up", "type": "IQ4_XS", "s": 1667006.4344755835, "err_l2": 183.33831787109375, "energy": 5504049370.36377, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.30.ffn_down_exps.weight", "block": 30, "role": "down", "type": "IQ4_XS", "s": 49030.58505621785, "err_l2": 181.8170928955078, "energy": 38829847.31696892, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.30.ffn_gate_exps.weight", "block": 30, "role": "gate", "type": "IQ4_XS", "s": 1949623.984885451, "err_l2": 202.01275634765625, "energy": 5947769125.285889, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.30.ffn_up_exps.weight", "block": 30, "role": "up", "type": "IQ4_XS", "s": 1775199.3162880957, "err_l2": 182.9893798828125, "energy": 5947769125.285889, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.31.ffn_down_exps.weight", "block": 31, "role": "down", "type": "IQ4_XS", "s": 70722.34582280106, "err_l2": 186.094970703125, "energy": 54921343.842552185, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.31.ffn_gate_exps.weight", "block": 31, "role": "gate", "type": "IQ4_XS", "s": 2367664.859962002, "err_l2": 206.2225799560547, "energy": 6885215013.473389, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.31.ffn_up_exps.weight", "block": 31, "role": "up", "type": "IQ4_XS", "s": 2124809.3811483807, "err_l2": 184.1732177734375, "energy": 6885215013.473389, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.32.ffn_down_exps.weight", "block": 32, "role": "down", "type": "IQ4_XS", "s": 101720.83557744388, "err_l2": 190.5087890625, "energy": 78752141.1644249, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.32.ffn_gate_exps.weight", "block": 32, "role": "gate", "type": "IQ4_XS", "s": 2716697.1928113825, "err_l2": 215.96151733398438, "energy": 7527876637.479248, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.32.ffn_up_exps.weight", "block": 32, "role": "up", "type": "IQ4_XS", "s": 2380231.6349136136, "err_l2": 189.73739624023438, "energy": 7527876637.479248, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.33.ffn_down_exps.weight", "block": 33, "role": "down", "type": "IQ4_XS", "s": 138452.12302130135, "err_l2": 195.35267639160156, "energy": 106669200.57768059, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.33.ffn_gate_exps.weight", "block": 33, "role": "gate", "type": "IQ4_XS", "s": 3080675.8768702606, "err_l2": 222.5367889404297, "energy": 8332150965.168579, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.33.ffn_up_exps.weight", "block": 33, "role": "up", "type": "IQ4_XS", "s": 2674968.3096753447, "err_l2": 192.8326873779297, "energy": 8332150965.168579, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.34.ffn_down_exps.weight", "block": 34, "role": "down", "type": "IQ4_XS", "s": 163992.1713414014, "err_l2": 197.61561584472656, "energy": 124581003.01006317, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.34.ffn_gate_exps.weight", "block": 34, "role": "gate", "type": "IQ4_XS", "s": 3222098.350915965, "err_l2": 224.4082794189453, "energy": 8550464978.343994, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.34.ffn_up_exps.weight", "block": 34, "role": "up", "type": "IQ4_XS", "s": 2770207.1053777714, "err_l2": 194.0430908203125, "energy": 8550464978.343994, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.35.ffn_down_exps.weight", "block": 35, "role": "down", "type": "IQ4_XS", "s": 184397.16518947826, "err_l2": 202.8323211669922, "energy": 137147195.31022453, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.35.ffn_gate_exps.weight", "block": 35, "role": "gate", "type": "IQ4_XS", "s": 3830352.6226438815, "err_l2": 223.34866333007812, "energy": 9771642959.588196, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.35.ffn_up_exps.weight", "block": 35, "role": "up", "type": "IQ4_XS", "s": 3288237.3784090206, "err_l2": 192.45388793945312, "energy": 9771642959.588196, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.36.ffn_down_exps.weight", "block": 36, "role": "down", "type": "IQ4_XS", "s": 232099.68282068818, "err_l2": 208.89959716796875, "energy": 167412832.3093624, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.36.ffn_gate_exps.weight", "block": 36, "role": "gate", "type": "IQ4_XS", "s": 4297845.560476509, "err_l2": 226.4549560546875, "energy": 10746321069.68451, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.36.ffn_up_exps.weight", "block": 36, "role": "up", "type": "IQ4_XS", "s": 3751336.359109651, "err_l2": 199.18943786621094, "energy": 10746321069.68451, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.37.ffn_down_exps.weight", "block": 37, "role": "down", "type": "IQ4_XS", "s": 364362.14639169566, "err_l2": 214.06065368652344, "energy": 260268775.01239014, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.37.ffn_gate_exps.weight", "block": 37, "role": "gate", "type": "IQ4_XS", "s": 4493249.183317599, "err_l2": 225.3658447265625, "energy": 11005630810.807129, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.37.ffn_up_exps.weight", "block": 37, "role": "up", "type": "IQ4_XS", "s": 3976901.3934555533, "err_l2": 203.40878295898438, "energy": 11005630810.807129, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.38.ffn_down_exps.weight", "block": 38, "role": "down", "type": "IQ4_XS", "s": 561756.3104592973, "err_l2": 216.874267578125, "energy": 406006067.78154135, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.38.ffn_gate_exps.weight", "block": 38, "role": "gate", "type": "IQ4_XS", "s": 4510618.338562746, "err_l2": 234.75888061523438, "energy": 10846040824.944519, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.38.ffn_up_exps.weight", "block": 38, "role": "up", "type": "IQ4_XS", "s": 4005111.9444749975, "err_l2": 210.9281768798828, "energy": 10846040824.944519, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.39.ffn_down_exps.weight", "block": 39, "role": "down", "type": "IQ4_XS", "s": 870368.030860716, "err_l2": 224.1531219482422, "energy": 639124982.3202152, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.39.ffn_gate_exps.weight", "block": 39, "role": "gate", "type": "IQ4_XS", "s": 3858338.486249575, "err_l2": 276.1679382324219, "energy": 8206865322.349945, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.39.ffn_up_exps.weight", "block": 39, "role": "up", "type": "IQ4_XS", "s": 3284342.176323951, "err_l2": 234.19215393066406, "energy": 8206865322.349945, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.0.ffn_down_exps.weight", "block": 0, "role": "down", "type": "IQ3_S", "s": 3160.7944293372657, "err_l2": 900.9968872070312, "energy": 1594305.5174544854, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.0.ffn_gate_exps.weight", "block": 0, "role": "gate", "type": "IQ3_S", "s": 452754.15963359905, "err_l2": 256.8077697753906, "energy": 1558006241.509983, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.0.ffn_up_exps.weight", "block": 0, "role": "up", "type": "IQ3_S", "s": 416146.1085763802, "err_l2": 240.3582763671875, "energy": 1558006241.509983, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.1.ffn_down_exps.weight", "block": 1, "role": "down", "type": "IQ3_S", "s": 8343.977181851402, "err_l2": 584.8770141601562, "energy": 2793671.7302908897, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.1.ffn_gate_exps.weight", "block": 1, "role": "gate", "type": "IQ3_S", "s": 1065540.1581653825, "err_l2": 479.3675537109375, "energy": 1528231540.1870728, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.1.ffn_up_exps.weight", "block": 1, "role": "up", "type": "IQ3_S", "s": 1012191.5792834234, "err_l2": 453.1082458496094, "energy": 1528231540.1870728, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.2.ffn_down_exps.weight", "block": 2, "role": "down", "type": "IQ3_S", "s": 8084.653966804856, "err_l2": 558.1129150390625, "energy": 2394420.025505662, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.2.ffn_gate_exps.weight", "block": 2, "role": "gate", "type": "IQ3_S", "s": 1273410.61171046, "err_l2": 516.1437377929688, "energy": 1614167290.0648804, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.2.ffn_up_exps.weight", "block": 2, "role": "up", "type": "IQ3_S", "s": 1220276.903842408, "err_l2": 495.2116394042969, "energy": 1614167290.0648804, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.3.ffn_down_exps.weight", "block": 3, "role": "down", "type": "IQ3_S", "s": 11480.68925839384, "err_l2": 541.4755249023438, "energy": 3253016.4095531586, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.3.ffn_gate_exps.weight", "block": 3, "role": "gate", "type": "IQ3_S", "s": 1672856.0036041602, "err_l2": 527.1359252929688, "energy": 1987198435.1838608, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.3.ffn_up_exps.weight", "block": 3, "role": "up", "type": "IQ3_S", "s": 1571752.4605223876, "err_l2": 502.90753173828125, "energy": 1987198435.1838608, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.4.ffn_down_exps.weight", "block": 4, "role": "down", "type": "IQ3_S", "s": 15962.299250743374, "err_l2": 534.6857299804688, "energy": 4543147.314295173, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.4.ffn_gate_exps.weight", "block": 4, "role": "gate", "type": "IQ3_S", "s": 1913655.917284216, "err_l2": 554.4131469726562, "energy": 2219785664.976015, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.4.ffn_up_exps.weight", "block": 4, "role": "up", "type": "IQ3_S", "s": 1736992.7733108406, "err_l2": 512.450927734375, "energy": 2219785664.976015, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.5.ffn_down_exps.weight", "block": 5, "role": "down", "type": "IQ3_S", "s": 20251.82982183481, "err_l2": 531.4661865234375, "energy": 5663699.233736992, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.5.ffn_gate_exps.weight", "block": 5, "role": "gate", "type": "IQ3_S", "s": 2125959.787134381, "err_l2": 562.4515380859375, "energy": 2404538256.884094, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.5.ffn_up_exps.weight", "block": 5, "role": "up", "type": "IQ3_S", "s": 1907856.5756103725, "err_l2": 513.3391723632812, "energy": 2404538256.884094, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.6.ffn_down_exps.weight", "block": 6, "role": "down", "type": "IQ3_S", "s": 22670.314739696358, "err_l2": 529.2758178710938, "energy": 6342106.4899379015, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.6.ffn_gate_exps.weight", "block": 6, "role": "gate", "type": "IQ3_S", "s": 2187844.4262000327, "err_l2": 565.199951171875, "energy": 2495879819.4156494, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.6.ffn_up_exps.weight", "block": 6, "role": "up", "type": "IQ3_S", "s": 1964646.9109667481, "err_l2": 511.3644714355469, "energy": 2495879819.4156494, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.7.ffn_down_exps.weight", "block": 7, "role": "down", "type": "IQ3_S", "s": 28553.410499008663, "err_l2": 523.9112548828125, "energy": 8081860.904105902, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.7.ffn_gate_exps.weight", "block": 7, "role": "gate", "type": "IQ3_S", "s": 2803698.718461978, "err_l2": 566.6865844726562, "energy": 3024648344.3563538, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.7.ffn_up_exps.weight", "block": 7, "role": "up", "type": "IQ3_S", "s": 2501471.2529095113, "err_l2": 506.50299072265625, "energy": 3024648344.3563538, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.8.ffn_down_exps.weight", "block": 8, "role": "down", "type": "IQ3_S", "s": 31335.630735910585, "err_l2": 517.8521728515625, "energy": 8960753.594052434, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.8.ffn_gate_exps.weight", "block": 8, "role": "gate", "type": "IQ3_S", "s": 3128232.3492311677, "err_l2": 573.94775390625, "energy": 3319484814.762665, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.8.ffn_up_exps.weight", "block": 8, "role": "up", "type": "IQ3_S", "s": 2712931.4672748186, "err_l2": 496.7440490722656, "energy": 3319484814.762665, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.9.ffn_down_exps.weight", "block": 9, "role": "down", "type": "IQ3_S", "s": 34995.00676708239, "err_l2": 533.0400390625, "energy": 9568770.147459984, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.9.ffn_gate_exps.weight", "block": 9, "role": "gate", "type": "IQ3_S", "s": 3110818.898361415, "err_l2": 583.6638793945312, "energy": 3361266052.4784546, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.9.ffn_up_exps.weight", "block": 9, "role": "up", "type": "IQ3_S", "s": 2791247.654129577, "err_l2": 518.638916015625, "energy": 3361266052.4784546, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.10.ffn_down_exps.weight", "block": 10, "role": "down", "type": "IQ3_S", "s": 38222.915436906114, "err_l2": 542.0826416015625, "energy": 10111794.935325623, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.10.ffn_gate_exps.weight", "block": 10, "role": "gate", "type": "IQ3_S", "s": 2884344.5722378963, "err_l2": 586.2544555664062, "energy": 3193724210.449585, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.10.ffn_up_exps.weight", "block": 10, "role": "up", "type": "IQ3_S", "s": 2676315.448412295, "err_l2": 539.7728881835938, "energy": 3193724210.449585, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.11.ffn_down_exps.weight", "block": 11, "role": "down", "type": "IQ3_S", "s": 51985.0681169213, "err_l2": 547.7119140625, "energy": 13562770.490616083, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.11.ffn_gate_exps.weight", "block": 11, "role": "gate", "type": "IQ3_S", "s": 3347142.01602877, "err_l2": 580.8025512695312, "energy": 3610363311.6185303, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.11.ffn_up_exps.weight", "block": 11, "role": "up", "type": "IQ3_S", "s": 3132459.9042833927, "err_l2": 542.5391235351562, "energy": 3610363311.6185303, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.12.ffn_down_exps.weight", "block": 12, "role": "down", "type": "IQ3_S", "s": 44382.34442701971, "err_l2": 530.018310546875, "energy": 11890972.488930225, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.12.ffn_gate_exps.weight", "block": 12, "role": "gate", "type": "IQ3_S", "s": 3476009.0308052665, "err_l2": 582.969482421875, "energy": 3644765104.288269, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.12.ffn_up_exps.weight", "block": 12, "role": "up", "type": "IQ3_S", "s": 3186878.573856535, "err_l2": 528.1942749023438, "energy": 3644765104.288269, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.13.ffn_down_exps.weight", "block": 13, "role": "down", "type": "IQ3_S", "s": 55185.17201409374, "err_l2": 537.3656005859375, "energy": 14734612.844225299, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.13.ffn_gate_exps.weight", "block": 13, "role": "gate", "type": "IQ3_S", "s": 3885324.193518669, "err_l2": 585.0238037109375, "energy": 3992092145.1657104, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.13.ffn_up_exps.weight", "block": 13, "role": "up", "type": "IQ3_S", "s": 3534521.612839847, "err_l2": 530.9894409179688, "energy": 3992092145.1657104, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.14.ffn_down_exps.weight", "block": 14, "role": "down", "type": "IQ3_S", "s": 50753.171225795966, "err_l2": 541.4236450195312, "energy": 13554481.034674168, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.14.ffn_gate_exps.weight", "block": 14, "role": "gate", "type": "IQ3_S", "s": 3660414.3198930626, "err_l2": 585.3523559570312, "energy": 3737719776.7888184, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.14.ffn_up_exps.weight", "block": 14, "role": "up", "type": "IQ3_S", "s": 3339606.7686456465, "err_l2": 538.2454833984375, "energy": 3737719776.7888184, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.15.ffn_down_exps.weight", "block": 15, "role": "down", "type": "IQ3_S", "s": 60703.34377456959, "err_l2": 544.146484375, "energy": 15960954.2438488, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.15.ffn_gate_exps.weight", "block": 15, "role": "gate", "type": "IQ3_S", "s": 4130555.2522207904, "err_l2": 583.3648681640625, "energy": 4167948470.2155457, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.15.ffn_up_exps.weight", "block": 15, "role": "up", "type": "IQ3_S", "s": 3820981.0486747427, "err_l2": 544.2249755859375, "energy": 4167948470.2155457, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.16.ffn_down_exps.weight", "block": 16, "role": "down", "type": "IQ3_S", "s": 66112.4353527574, "err_l2": 554.1646728515625, "energy": 17199945.656352043, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.16.ffn_gate_exps.weight", "block": 16, "role": "gate", "type": "IQ3_S", "s": 3935996.5798871657, "err_l2": 574.068115234375, "energy": 4078872020.2729187, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.16.ffn_up_exps.weight", "block": 16, "role": "up", "type": "IQ3_S", "s": 3809271.459034695, "err_l2": 556.0358276367188, "energy": 4078872020.2729187, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.17.ffn_down_exps.weight", "block": 17, "role": "down", "type": "IQ3_S", "s": 61944.8942565209, "err_l2": 553.5111694335938, "energy": 16259451.176771164, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.17.ffn_gate_exps.weight", "block": 17, "role": "gate", "type": "IQ3_S", "s": 4189540.0714451014, "err_l2": 584.1781616210938, "energy": 4243044980.7101135, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.17.ffn_up_exps.weight", "block": 17, "role": "up", "type": "IQ3_S", "s": 3921894.8869724707, "err_l2": 553.0812377929688, "energy": 4243044980.7101135, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.18.ffn_down_exps.weight", "block": 18, "role": "down", "type": "IQ3_S", "s": 69559.6849082556, "err_l2": 558.6499633789062, "energy": 18323826.341524124, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.18.ffn_gate_exps.weight", "block": 18, "role": "gate", "type": "IQ3_S", "s": 4387660.424556547, "err_l2": 591.2513427734375, "energy": 4469796752.81366, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.18.ffn_up_exps.weight", "block": 18, "role": "up", "type": "IQ3_S", "s": 4097229.072300347, "err_l2": 553.648193359375, "energy": 4469796752.81366, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.19.ffn_down_exps.weight", "block": 19, "role": "down", "type": "IQ3_S", "s": 82883.42718849362, "err_l2": 560.3446655273438, "energy": 21314045.25215578, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.19.ffn_gate_exps.weight", "block": 19, "role": "gate", "type": "IQ3_S", "s": 5003290.408384014, "err_l2": 599.171142578125, "energy": 4942801179.769653, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.19.ffn_up_exps.weight", "block": 19, "role": "up", "type": "IQ3_S", "s": 4592570.0684832195, "err_l2": 552.9580078125, "energy": 4942801179.769653, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.20.ffn_down_exps.weight", "block": 20, "role": "down", "type": "IQ3_S", "s": 70892.01597780899, "err_l2": 538.7665405273438, "energy": 19533477.948932648, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.20.ffn_gate_exps.weight", "block": 20, "role": "gate", "type": "IQ3_S", "s": 5588287.778330978, "err_l2": 616.052734375, "energy": 5309558766.605164, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.20.ffn_up_exps.weight", "block": 20, "role": "up", "type": "IQ3_S", "s": 4717590.169043346, "err_l2": 521.070068359375, "energy": 5309558766.605164, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.21.ffn_down_exps.weight", "block": 21, "role": "down", "type": "IQ3_S", "s": 75113.77535767024, "err_l2": 546.2060546875, "energy": 19783092.441063404, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.21.ffn_gate_exps.weight", "block": 21, "role": "gate", "type": "IQ3_S", "s": 5239496.9021850685, "err_l2": 618.1743774414062, "energy": 5133279711.727417, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.21.ffn_up_exps.weight", "block": 21, "role": "up", "type": "IQ3_S", "s": 4671810.241320427, "err_l2": 537.6951293945312, "energy": 5133279711.727417, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.22.ffn_down_exps.weight", "block": 22, "role": "down", "type": "IQ3_S", "s": 78430.21688908852, "err_l2": 561.7735595703125, "energy": 20124856.359775662, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.22.ffn_gate_exps.weight", "block": 22, "role": "gate", "type": "IQ3_S", "s": 4635056.921830076, "err_l2": 606.8456420898438, "energy": 4779006383.581612, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.22.ffn_up_exps.weight", "block": 22, "role": "up", "type": "IQ3_S", "s": 4367302.804227735, "err_l2": 569.844970703125, "energy": 4779006383.581612, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.23.ffn_down_exps.weight", "block": 23, "role": "down", "type": "IQ3_S", "s": 84691.32044463974, "err_l2": 571.7097778320312, "energy": 21217019.830352783, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.23.ffn_gate_exps.weight", "block": 23, "role": "gate", "type": "IQ3_S", "s": 4761784.981807351, "err_l2": 601.6629638671875, "energy": 4948538410.368652, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.23.ffn_up_exps.weight", "block": 23, "role": "up", "type": "IQ3_S", "s": 4566655.829891256, "err_l2": 577.8573608398438, "energy": 4948538410.368652, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.24.ffn_down_exps.weight", "block": 24, "role": "down", "type": "IQ3_S", "s": 75919.04941504156, "err_l2": 559.82470703125, "energy": 18962488.084349155, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.24.ffn_gate_exps.weight", "block": 24, "role": "gate", "type": "IQ3_S", "s": 4952154.961190764, "err_l2": 606.8809204101562, "energy": 4985729777.105591, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.24.ffn_up_exps.weight", "block": 24, "role": "up", "type": "IQ3_S", "s": 4659728.720314053, "err_l2": 565.8052368164062, "energy": 4985729777.105591, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.25.ffn_down_exps.weight", "block": 25, "role": "down", "type": "IQ3_S", "s": 87138.85709335162, "err_l2": 567.681640625, "energy": 22012922.12262726, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.25.ffn_gate_exps.weight", "block": 25, "role": "gate", "type": "IQ3_S", "s": 5192218.898353936, "err_l2": 615.70458984375, "energy": 5205305004.696228, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.25.ffn_up_exps.weight", "block": 25, "role": "up", "type": "IQ3_S", "s": 4842031.056555591, "err_l2": 569.7493896484375, "energy": 5205305004.696228, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.26.ffn_down_exps.weight", "block": 26, "role": "down", "type": "IQ3_S", "s": 85218.41889404264, "err_l2": 578.304443359375, "energy": 21354014.354220867, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.26.ffn_gate_exps.weight", "block": 26, "role": "gate", "type": "IQ3_S", "s": 4959794.111734957, "err_l2": 612.7774658203125, "energy": 4994181607.238647, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.26.ffn_up_exps.weight", "block": 26, "role": "up", "type": "IQ3_S", "s": 4677559.00006406, "err_l2": 580.3008422851562, "energy": 4994181607.238647, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.27.ffn_down_exps.weight", "block": 27, "role": "down", "type": "IQ3_S", "s": 110375.81788056428, "err_l2": 587.4334716796875, "energy": 27613217.047315598, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.27.ffn_gate_exps.weight", "block": 27, "role": "gate", "type": "IQ3_S", "s": 5505800.031905194, "err_l2": 626.7807006835938, "energy": 5427490272.732422, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.27.ffn_up_exps.weight", "block": 27, "role": "up", "type": "IQ3_S", "s": 5169619.890955507, "err_l2": 588.3776245117188, "energy": 5427490272.732422, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.28.ffn_down_exps.weight", "block": 28, "role": "down", "type": "IQ3_S", "s": 121570.828985493, "err_l2": 598.8751220703125, "energy": 30576922.299942017, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.28.ffn_gate_exps.weight", "block": 28, "role": "gate", "type": "IQ3_S", "s": 5321585.063166202, "err_l2": 628.0255126953125, "energy": 5276713432.383301, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.28.ffn_up_exps.weight", "block": 28, "role": "up", "type": "IQ3_S", "s": 5078092.675738056, "err_l2": 598.3042602539062, "energy": 5276713432.383301, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.29.ffn_down_exps.weight", "block": 29, "role": "down", "type": "IQ3_S", "s": 128718.79921440108, "err_l2": 601.0684204101562, "energy": 32332774.765409946, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.29.ffn_gate_exps.weight", "block": 29, "role": "gate", "type": "IQ3_S", "s": 5631039.976415399, "err_l2": 645.246826171875, "energy": 5504049370.36377, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.29.ffn_up_exps.weight", "block": 29, "role": "up", "type": "IQ3_S", "s": 5234378.314924892, "err_l2": 599.2750244140625, "energy": 5504049370.36377, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.30.ffn_down_exps.weight", "block": 30, "role": "down", "type": "IQ3_S", "s": 151681.3860599562, "err_l2": 605.2285766601562, "energy": 38829847.31696892, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.30.ffn_gate_exps.weight", "block": 30, "role": "gate", "type": "IQ3_S", "s": 6140667.023403802, "err_l2": 660.4652099609375, "energy": 5947769125.285889, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.30.ffn_up_exps.weight", "block": 30, "role": "up", "type": "IQ3_S", "s": 5589263.423614544, "err_l2": 597.9803466796875, "energy": 5947769125.285889, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.31.ffn_down_exps.weight", "block": 31, "role": "down", "type": "IQ3_S", "s": 217128.74858495774, "err_l2": 619.8214111328125, "energy": 54921343.842552185, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.31.ffn_gate_exps.weight", "block": 31, "role": "gate", "type": "IQ3_S", "s": 7465612.496332629, "err_l2": 673.8700561523438, "energy": 6885215013.473389, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.31.ffn_up_exps.weight", "block": 31, "role": "up", "type": "IQ3_S", "s": 6697993.1553127775, "err_l2": 601.9542846679688, "energy": 6885215013.473389, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.32.ffn_down_exps.weight", "block": 32, "role": "down", "type": "IQ3_S", "s": 306529.0965501912, "err_l2": 636.7525024414062, "energy": 78752141.1644249, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.32.ffn_gate_exps.weight", "block": 32, "role": "gate", "type": "IQ3_S", "s": 8443505.821339225, "err_l2": 706.1597900390625, "energy": 7527876637.479248, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.32.ffn_up_exps.weight", "block": 32, "role": "up", "type": "IQ3_S", "s": 7412783.01796599, "err_l2": 621.2427978515625, "energy": 7527876637.479248, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.33.ffn_down_exps.weight", "block": 33, "role": "down", "type": "IQ3_S", "s": 413249.17071542935, "err_l2": 652.964599609375, "energy": 106669200.57768059, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.33.ffn_gate_exps.weight", "block": 33, "role": "gate", "type": "IQ3_S", "s": 9559343.506460926, "err_l2": 728.4646606445312, "energy": 8332150965.168579, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.33.ffn_up_exps.weight", "block": 33, "role": "up", "type": "IQ3_S", "s": 8320369.6111924425, "err_l2": 632.2214965820312, "energy": 8332150965.168579, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.34.ffn_down_exps.weight", "block": 34, "role": "down", "type": "IQ3_S", "s": 490158.1868941054, "err_l2": 661.0261840820312, "energy": 124581003.01006317, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.34.ffn_gate_exps.weight", "block": 34, "role": "gate", "type": "IQ3_S", "s": 10056521.74306483, "err_l2": 734.4813232421875, "energy": 8550464978.343994, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.34.ffn_up_exps.weight", "block": 34, "role": "up", "type": "IQ3_S", "s": 8655460.903246213, "err_l2": 635.8094482421875, "energy": 8550464978.343994, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.35.ffn_down_exps.weight", "block": 35, "role": "down", "type": "IQ3_S", "s": 551968.5209622283, "err_l2": 678.5155639648438, "energy": 137147195.31022453, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.35.ffn_gate_exps.weight", "block": 35, "role": "gate", "type": "IQ3_S", "s": 12102038.101962328, "err_l2": 732.8733520507812, "energy": 9771642959.588196, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.35.ffn_up_exps.weight", "block": 35, "role": "up", "type": "IQ3_S", "s": 10392423.710442733, "err_l2": 631.5198974609375, "energy": 9771642959.588196, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.36.ffn_down_exps.weight", "block": 36, "role": "down", "type": "IQ3_S", "s": 691096.0368499237, "err_l2": 700.4812622070312, "energy": 167412832.3093624, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.36.ffn_gate_exps.weight", "block": 36, "role": "gate", "type": "IQ3_S", "s": 13560377.77815588, "err_l2": 743.4298706054688, "energy": 10746321069.68451, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.36.ffn_up_exps.weight", "block": 36, "role": "up", "type": "IQ3_S", "s": 11833460.838188414, "err_l2": 654.0140991210938, "energy": 10746321069.68451, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.37.ffn_down_exps.weight", "block": 37, "role": "down", "type": "IQ3_S", "s": 1065288.5766277083, "err_l2": 724.123779296875, "energy": 260268775.01239014, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.37.ffn_gate_exps.weight", "block": 37, "role": "gate", "type": "IQ3_S", "s": 14161707.869068626, "err_l2": 740.1130981445312, "energy": 11005630810.807129, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.37.ffn_up_exps.weight", "block": 37, "role": "up", "type": "IQ3_S", "s": 12550725.946321322, "err_l2": 668.5313720703125, "energy": 11005630810.807129, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.38.ffn_down_exps.weight", "block": 38, "role": "down", "type": "IQ3_S", "s": 1617751.5282598408, "err_l2": 736.3215942382812, "energy": 406006067.78154135, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.38.ffn_gate_exps.weight", "block": 38, "role": "gate", "type": "IQ3_S", "s": 14182660.3448626, "err_l2": 770.3104248046875, "energy": 10846040824.944519, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.38.ffn_up_exps.weight", "block": 38, "role": "up", "type": "IQ3_S", "s": 12637990.40340672, "err_l2": 693.5491333007812, "energy": 10846040824.944519, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.39.ffn_down_exps.weight", "block": 39, "role": "down", "type": "IQ3_S", "s": 2426980.065219186, "err_l2": 769.3417358398438, "energy": 639124982.3202152, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.39.ffn_gate_exps.weight", "block": 39, "role": "gate", "type": "IQ3_S", "s": 12182885.787531016, "err_l2": 906.9334716796875, "energy": 8206865322.349945, "counts": 2347008.0, "params": 268435456} +{"tensor": "blk.39.ffn_up_exps.weight", "block": 39, "role": "up", "type": "IQ3_S", "s": 10393573.322986204, "err_l2": 770.9765625, "energy": 8206865322.349945, "counts": 2347008.0, "params": 268435456} diff --git a/appendix/original-ICE/PRINCIPLE.md b/appendix/original-ICE/PRINCIPLE.md new file mode 100644 index 0000000000000000000000000000000000000000..87e6f035aed03802fa6dacb6a7fb3a8cc2739c50 --- /dev/null +++ b/appendix/original-ICE/PRINCIPLE.md @@ -0,0 +1,243 @@ +--- +title: ICE — quantization by error-propagation class +created: 2026-08-21 +status: measured — ladder shipped 2026-08-21, one derived rule refuted +scope: model-agnostic. Instantiated for Ornith-1.5-35B-A3B; written to be re-derived + for any sparse MoE. +--- + +# ICE + +**Freeze what propagates. Spend everything else on the library.** + +Every quantizer in the GGUF ecosystem — llama.cpp's presets, Unsloth Dynamic, +mudler's APEX, AtomicChat's AD — minimises the same objective for every tensor: + +> imatrix-weighted MSE of *this tensor's output, for the current token.* + +That objective is correct for tensors whose error dies with the token. It is the +wrong objective for three other kinds of tensor, and nobody separates them. + +--- + +## 1. The four error-propagation classes + +Classify every tensor by **how far an error travels**, not by how big its +activations are. + +| class | mechanism | failure shape | treatment | +|---|---|---|---| +| **DISCRETE** | error flips an argmax; a different computation runs | not graded — *categorical* | exact (F32) | +| **RECURRENT** | error enters a state decay and compounds multiplicatively along the sequence | grows with context length | exact (F32) | +| **CACHED** | error is written once and re-read by later tokens | frozen in, cannot be re-decided | near-exact (F16) | +| **INSTANT** | error affects this token's output only | graded, bounded | the budget lives here | + +### How to find them in an arbitrary architecture + +- **DISCRETE** — anything feeding a `top_k`, `argmax`, or hard gate. In an MoE + that is the router (`ffn_gate_inp`). Also any hard routing/branching signal. +- **RECURRENT** — anything multiplying or parameterising a carried state: SSM/Mamba + decay and timestep terms (`ssm_alpha`, `ssm_beta`, `ssm_a`, `ssm_dt`), gated + linear-attention decay, any RNN gate. **Not** the state's output projection — + that one is INSTANT. +- **CACHED** — whatever is written into the KV cache: `attn_k`, `attn_v`. Note + `attn_q` is *not* cached (see §5 — this is the trap). +- **INSTANT** — everything else. + +### The finding that makes this cheap + +On Ornith-1.5-35B-A3B the entire propagating set is **47 M parameters — 0.14% of +the model. Keeping all of it exact costs 0.15 GB.** It is not a trade-off; it is a +line item you can simply pay. Expect the same on any sparse MoE: routers and state +gates are structurally tiny. + +--- + +## 2. The four measured laws + +### Law 1 — the sparsity law: always-on bits are worth `1/sparsity` routed bits + +For anything that fires every token, compute-share and storage-cost are both +proportional to parameter count, so *work per byte* is a constant. Routed experts +divide that constant by the routing fraction. + +Measured on Ornith-1.5 (8 of 256 active): every always-on role scores ~270 units of +work per GB-per-bpw; the experts score **8.5. The ratio is 32 — exactly 256/8.** + +> **A bit spent on the always-on core is worth `E_total / E_active` bits spent on +> the expert bank.** Compute this ratio first for any new MoE; it sets the entire +> shape of the recipe. + +### Law 2 — the convexity law: allocation cleverness is capped + +Quantization error falls as `4^-b`, which is convex. By Jensen, non-uniform +allocation only beats uniform when sensitivity varies enormously, and the gain +scales with the **logarithm** of the spread. + +Measured, by water-filling over the real per-expert routing histogram: + +| sensitivity spread | optimal gain over uniform | +|---|---| +| 46× (real expert routing) | **+0.139 bpw** | +| 3.7× (depth spread of `ffn_down`) | less | +| ~250× | ~1 bpw | + +> **Corollary: stop tuning ladders.** This is why every published recipe lands +> within ~0.3 bpw of every other one. Per-expert allocation, depth grading, and +> per-layer bands are all worth less than 0.15 bpw. ICE therefore does **no depth +> grading at all** — an explicit refusal, not an oversight. + +### Law 3 — the leverage law: flat importance is the expensive case + +The imatrix stores per-channel importance `h`. Measure its concentration with the +participation ratio `PR = (Σh)² / (n·Σh²)`. + +**Spiky `h` (low PR) is a gift** — the quantizer knows which few channels to spend +the block scale on. **Flat `h` (high PR) is the hard case** — every channel matters +and there is no leverage. So *flat tensors need more bits.* + +Measured on Ornith-1.5: + +| tensor | PR/n | +|---|---:| +| `attn_k/q/v` | 0.0069 (~14 of 2048 channels) | +| `attn_qkv`, `attn_gate`, `ssm_alpha/beta` | 0.0128 | +| `ffn_gate_exps`, `ffn_up_exps` | 0.2048 | +| **`ffn_down_exps`** | **0.6104 — flattest in the model** | + +> ~~`down` should sit `log4(PR_down / PR_gateup)` = 0.79 bpw above `gate`/`up`.~~ +> **REFUTED BY MEASUREMENT, 2026-08-21.** + +This looked like a derivation of what convention does by feel, and it agreed with +the best published rungs (UD-Q4_K_XL and AtomicChat AD-Q5_K-Q4_K both run ~1.0). +A controlled test at identical size says otherwise: + +| expert allocation | size | mean KLD | +|---|---:|---:| +| `down` = `gate/up` + 0.79 | 20.82 GB | 0.046449 | +| **uniform (gap = 0)** | 20.85 GB | **0.041192** | + +**Uniform is 11.3% better**, reproduced exactly on a rebuild (0.041192 twice). +Rebuilding the whole ladder uniform improved every rung: −7.3% / −11.3% / −3.3% / +−3.1% at 19/21/23/25 GB. + +**What PR actually predicts is unknown.** It measures how concentrated the *input* +importance is, which is not the same as how much a tensor's error reaches the +output — `ffn_down` writes straight into the residual stream while `gate`/`up` +errors are attenuated through the gating nonlinearity first. Position in the +circuit is not captured by PR, and on this model it evidently dominates. + +**Practical rule until someone measures better: allocate the expert stack +uniformly.** Do not bump `ffn_down`, despite convention. + +### Law 4 — the floor is epistemic, not numeric + +Fit `KLD ≈ k₀ + c_d·4^(-bpw_dense) + c_e·4^(-bpw_expert)`. Two independent +harnesses (ours, AtomicChat's) agreed: + +| | floor `k₀` | `c_e/c_d` | +|---|---:|---:| +| ours (wikitext/2048) | 0.0205 | 5.91 | +| AtomicChat (eval_neutral/4096) | 0.0219 | 7.22 | + +`UD-Q6_K` measures 0.0221 against a 0.0205 floor — **it is 8% above the floor. The +top of the ladder is saturated.** Above ~5.5 expert bpw you are buying calibration +noise, not accuracy. + +> **You cannot out-bit a wrong prior.** Quantization is lossy compression *guided by +> a belief about importance*. Improve the imatrix before adding bits. + +**Caveat:** this fitted model ranks the extremes well and gets close calls wrong +(it misorders AtomicChat's `Q4_K_M` vs `AD-Q4_K-IQ4_XS`). Use it to find the floor +and the saturation point, never to choose between two candidate recipes. + +--- + +## 3. The ICE assignment + +``` +DISCRETE routers, hard gates F32 +RECURRENT state decay / timestep terms F32 +CACHED attn_k, attn_v F16 +ALWAYS-ON output head, token_embd, attention, Q8_0 + state output proj, shared expert +INSTANT routed experts the dial + down = gate = up (UNIFORM - measured, see Law 3) +``` + +Plus one decision specific to speculative-decoding models: + +> **A draft head does not need target-model precision.** Its errors are *verified* +> by the target, so they cost **acceptance**, not correctness. ICE pins only the +> draft block's projections and lets its experts follow the tier. On Ornith-1.5 +> that frees ~0.40 GB. Everyone else pins the whole block Q8_0, drops it to Q4_0, +> or deletes it — nobody asks what a drafter actually needs. + +--- + +## 4. Adapting ICE to a new MoE — the checklist + +1. **Inventory.** Dump the tensor list. Group by role. Compute params, % of file, + and % of *active* compute per role (weight routed experts by `k/E`). +2. **Sparsity ratio `E/k`** → Law 1. This is your always-on-to-expert exchange rate. +3. **Classify** every tensor into the four propagation classes (§1). Sum the + propagating set — if it is ≪1% of the model, freeze all of it. +4. **PR from the imatrix** per role → Law 3. Informative about *quantizer leverage*, + but it did NOT predict the right expert split on Ornith-1.5 — allocate the expert + stack uniformly unless a controlled test on your model says otherwise. + `analysis/imatrix_anatomy.py` computes PR from the imatrix alone, no weights. +5. **Routing histogram** from the imatrix `counts` → Law 2 → confirm depth/expert + grading is not worth the risk. `analysis/routing_skew.py`. +6. **Fixed body** = propagating set + always-on core + draft projections. + Everything else is one dial: `size = body + E_params · bpw / 8`. +7. **Bound the ladder.** Fit Law 4 on any measured tiers to find `k₀`; the useful + band runs from the cliff (where the expert term explodes) to saturation + (where the floor dominates). On Ornith-1.5 that was 17–27 GB — five rungs, not + eight. + +--- + +## 5. What was tried and rejected — with numbers + +Negative results, kept because they are the expensive part to rediscover. + +- **Per-expert bit allocation by routing frequency.** Measured: load balancing + works (209.5 effective experts of 256, gini 0.331), and convexity caps the gain + at **+0.139 bpw**. A naive "hot 25% at 6.5, cold at 2.5" split is *worse than + uniform* (0.00586 vs 0.00552). **Dead.** +- **Demoting `attn_q`** on the argument that Q's error dies with the token while + K/V's persists. The arithmetic kills it: each query distributes attention weight + summing to 1, so total k-influence over a sequence is O(N) — **the same as q**. + Buys 63 MB, costs 0.17 active bpw. Kept only as an A/B flag. +- **Active bpw as a ranking metric.** Tested against measured KLD it misorders + **4 of our 8 tiers** (Spearman +0.95) and **10 of AtomicChat's 13** (+0.71). + It over-credits Q8_0-dense-path designs because error is not linear in bits. + **Use it to explain where bits went; rank on measured KLD.** +- **Depth grading.** Worth <0.15 bpw by Law 2, and the *direction* is unresolved: + PR says the middle layers need more bits (they are flattest), convention says + the edges do (they matter more functionally). PR measures quantizer leverage, + not functional sensitivity. Unresolved → not spent. + +--- + +## 6. Open, and worth more than everything above + +- **Orthogonal rotation.** `RMSNorm` is invariant to rotation but **not** to + scaling — so a rotation of the residual stream folds into the norm weights + exactly and for free, and yields a *standard GGUF*. With `attn_*` at PR 0.0069 + the residual stream has ~14 dominant dimensions of 2048; spreading that energy + is worth an estimated ~0.9 bpw. Nobody has published rotation results for a + hybrid SSM/MoE, and the fixed SSM state may not tolerate it. **Test this next.** +- **Shared basis + per-expert delta.** `W_i = W̄ + Δ_i`, store `W̄` once. If + `‖Δ‖/‖W‖ ≈ ¼` that is ~2 bpw for 0.4% overhead, and `W̄ᵀx` is computed once per + token rather than once per expert. Needs a llama.cpp op. **The one measurement + that decides it is free: compute `‖Δ_i‖/‖W̄‖` from the bf16.** Counter-argument: + this architecture already has a shared expert, which may have absorbed the + common component. +- **`up`↔`down` scale migration.** `scale(up row j, s)` and `scale(down col j, 1/s)` + is *exactly* output-preserving. `up`'s output channels are rows and already get + their own k-quant scales (free), while `down`'s input channels are one element + in each of 2048 rows (polluting every row's range). Migrate the variance out of + `down`'s columns into `up`'s rows. Format-compatible, never done in GGUF. + +Related: [[apex-standard-pipeline]] · [[map-transfers-imatrix-does-not]] diff --git a/appendix/original-ICE/RECIPE.md b/appendix/original-ICE/RECIPE.md new file mode 100644 index 0000000000000000000000000000000000000000..0b41207479430d87480635d6156d70484051d297 --- /dev/null +++ b/appendix/original-ICE/RECIPE.md @@ -0,0 +1,151 @@ +--- +title: ICE for Ornith-1.5-35B-A3B — the concrete instantiation +created: 2026-08-21 +status: SHIPPED 2026-08-21 — 4 tiers live, uniform expert allocation +principle: see PRINCIPLE.md (model-agnostic) +--- + +# ICE — Ornith-1.5-35B-A3B + +The general method is in [PRINCIPLE.md](PRINCIPLE.md). This file is what it +produced for *this* model, with the arithmetic shown. + +## Architecture inputs + +| | | +|---|---| +| blocks | 41 = 30 linear/SSM + 10 full-attention (3,7,…,39) + 1 MTP (`blk.40`) | +| residual width | 2048 | +| experts | 256, 8 active → **sparsity ratio 32** | +| expert FFN / shared FFN | 512 / 512 | +| KV heads | 2, head_dim 256, on 10 blocks only → ~20 KB/token | +| vocab | 248,320 → `output.weight` = 508.6 M in one tensor | + +**Every block is 95.6% expert weights** (805.3 M of ~842 M). The surgical field is +the remaining ~35 M per block plus the two 508 M end-caps. + +## Active-compute geography (measured) + +| role | params | % of file | % of *active* compute | +|---|---:|---:|---:| +| routed experts (8/256) | 32.21 B | 92.94% | 34.2% | +| attention | 1.03 B | 2.96% | **34.9%** | +| output head | 0.51 B | 1.47% | **17.3%** | +| ssm | 0.26 B | 0.74% | 8.7% | +| shared expert | 0.13 B | 0.36% | 4.3% | +| token_embd | 0.51 B | 1.47% | 0% (`get_rows`) | + +**65.8% of active parameters are non-routed.** Active total 2.95 B — which is what +the "A3B" label means, and a useful check that the accounting is right. + +## The assignment + +| tensor | × | params | class | type | +|---|---:|---:|---|---| +| `ffn_gate_inp`, `ffn_gate_inp_shexp` | 80 | 21.0 M | DISCRETE | F32 † | +| `ssm_conv1d` | 30 | 1.0 M | RECURRENT | F32 † | +| `ssm_alpha`, `ssm_beta` | 60 | 3.9 M | RECURRENT | **F32 (pinned)** | +| `ssm_a`, `ssm_dt`, all norms | — | 0.3 M | RECURRENT / — | F32 † | +| `attn_k`, `attn_v` | 20 | 21.0 M | **CACHED** | **F16** | +| `attn_q` | 10 | 167.8 M | INSTANT | Q8_0 | +| `attn_output` | 10 | 83.9 M | INSTANT | Q8_0 | +| `attn_qkv` | 30 | 503.3 M | INSTANT | Q8_0 | +| `attn_gate` | 30 | 251.7 M | INSTANT | Q8_0 | +| `ssm_out` | 30 | 251.7 M | INSTANT | Q8_0 | +| `ffn_*_shexp` | 120 | 125.8 M | always-on | Q8_0 | +| `output.weight` | 1 | 508.6 M | always-on | Q8_0 | +| `token_embd` | 1 | 508.6 M | — | Q8_0 | +| `blk.40.nextn.eh_proj` + blk.40 attn/shexp | — | 39.3 M | draft projections | Q8_0 | +| **`ffn_{down,gate,up}_exps`** | 123 | **33.02 B** | INSTANT | **the dial** | + +† already F32 in llama.cpp: `ffn_gate_inp*` is excluded by name, `ssm_conv1d` has +first dim 4 (not a multiple of 256), 1-D tensors are never quantized. Pinning them +would be *inert* and fail GATE 0 — the intent is satisfied by the quantizer itself. + +**Fixed body ≈ 2.6 GB.** Propagating set = 47 M params = **0.14% of the model**, +exact for 0.15 GB. + +### Expert split + +**UNIFORM** — `down` = `gate` = `up`. The first ladder shipped a 0.79 bpw bump on +`down` derived from `log4(0.6104/0.2048)`; a controlled test at identical size +refuted it (0.041192 uniform vs 0.046449 bumped, **11.3% better uniform**), and the +ladder was rebuilt. See PRINCIPLE.md Law 3. + +### Draft head + +`blk.40`'s **experts follow the tier**; only its projections are Q8_0. Draft errors +are verified by the target model, so they cost *acceptance*, not correctness. +Frees ~0.40 GB versus pinning the whole block. + +### No depth grading + +Refused deliberately. Water-filling over the real routing histogram shows the +optimal gain is **+0.139 bpw**, and the direction of a depth profile is unresolved +(PR says middle, convention says edges). Not worth the risk budget. + +## The ladder + +| tier | built | expert bpw | down = gate = up | +|---|---:|---:|---| +| `19G-ICE` | 18.82 GB | 3.899 | IQ3_S / IQ4_XS | +| `21G-ICE` | 20.84 GB | 4.383 | IQ4_XS / Q4_K | +| `23G-ICE` | 22.83 GB | 4.868 | Q4_K / Q5_K | +| `25G-ICE` | 24.84 GB | 5.353 | Q5_K / Q6_K | + +GATE 0: **0 double-matched, 0 uncovered, 0 inert** on all four. + +## Predicted position (active bpw, pre-measurement) + +| build | disk | ACTIVE bpw | dense | blk.40 | +|---|---:|---:|---:|---:| +| UD-IQ4_XS | 18.68 G | 6.762 | 2.47 G | 0.90 G | +| **19G-ICE** | 18.84 G | **7.191** | 2.61 G | 0.43 G | +| **23G-ICE** | 22.83 G | **7.521** | 2.61 G | 0.53 G | +| UD-Q4_K_XL | 23.21 G | 7.471 | 2.59 G | 0.90 G | +| **25G-ICE** | 24.84 G | **7.691** | 2.61 G | 0.53 G | +| UD-Q5_K_S | 25.83 G | 7.693 | 2.59 G | 0.90 G | + +`25G-ICE` matches `UD-Q5_K_S`'s active precision at **0.99 GB less**. + +## Falsifiable predictions — scored + +1. ~~`25G-ICE` ≥ `UD-Q5_K_S` on measured KLD at ~1 GB less.~~ **FAILED.** 0.0303 vs + 0.0272; ICE is 0.99 GB smaller but not better. The 0.79 gap it rested on was then + refuted outright. +2. **`19G-ICE` beats `UD-IQ4_XS`** — **HELD.** 0.060844 vs 0.0723, **15.8% better** + at 0.14 GB *smaller* (18.82 vs 18.68 GB is +0.14; after the uniform rebuild the + file is 18.82 GB and still wins). +3. **Un-pinning `blk.40` costs <2 pp acceptance** — **HELD, direction unverified.** + 96.04% measured, above the 92.97% baseline, but on a different tier so not a + clean comparison. No collapse. + +Same harness as the UD/APEX ladder — wikitext-2-raw, 16 chunks, 2048 ctx — so the +numbers drop straight into the existing table. Changing the harness would make the +comparison meaningless. + +## Results — shipped + +| tier | size | mean KLD | top-1 | vs UD curve at same size | +|---|---:|---:|---:|---:| +| `25G-ICE` | 24.84 GB | 0.030304 | 93.16% | −1.9% | +| `23G-ICE` | 22.83 GB | 0.036069 | 92.65% | −6.3% | +| `21G-ICE` | 20.84 GB | 0.041192 | 92.03% | **−18.6%** | +| `19G-ICE` | 18.82 GB | 0.060844 | 89.72% | **−13.6%** | + +**MTP draft acceptance: 96.04%** (388/404), measured on the shrunk `blk.40`. +Baseline with the block fully Q8_0 was 92.97% on a different tier, so this is not a +clean comparison — but it establishes the smaller draft head still drafts. + +The advantage grows as size falls: at the top of the ladder the calibration floor +(~0.0185) dominates and no allocation choice can move it. + +### What did not survive measurement + +| idea | verdict | +|---|---| +| `down` bumped above `gate`/`up` | **refuted, −11.3%** — ship uniform | +| `token_embd` → Q4_K | **not replicated** — helps at 21G, hurts at 19G | +| `attn_gate` / `ssm_out` → Q6_K | **harmful** — always-on, 32× law | +| `attn_q` demotion | dropped before shipping — arithmetic didn't hold | +| up↔down scale migration | **dead** — `down` columns uniform (p95/p05 = 1.09) | diff --git a/appendix/recipes/cfg_19G-ICE_ICEbase.txt b/appendix/recipes/cfg_19G-ICE_ICEbase.txt new file mode 100644 index 0000000000000000000000000000000000000000..46bf07ff93c65f8a5d1bd25d29c8cc2dbb540e93 --- /dev/null +++ b/appendix/recipes/cfg_19G-ICE_ICEbase.txt @@ -0,0 +1,443 @@ +^output\.weight$=Q8_0 +token_embd\.weight=Q8_0 +blk\.0\.attn_gate\.weight=Q8_0 +blk\.0\.attn_qkv\.weight=Q8_0 +blk\.0\.ffn_down_exps\.weight=IQ4_XS +blk\.0\.ffn_down_shexp\.weight=Q8_0 +blk\.0\.ffn_gate_exps\.weight=IQ4_XS +blk\.0\.ffn_gate_shexp\.weight=Q8_0 +blk\.0\.ffn_up_exps\.weight=IQ4_XS +blk\.0\.ffn_up_shexp\.weight=Q8_0 +blk\.0\.ssm_alpha\.weight=F32 +blk\.0\.ssm_beta\.weight=F32 +blk\.0\.ssm_out\.weight=Q8_0 +blk\.1\.attn_gate\.weight=Q8_0 +blk\.1\.attn_qkv\.weight=Q8_0 +blk\.1\.ffn_down_exps\.weight=IQ4_XS +blk\.1\.ffn_down_shexp\.weight=Q8_0 +blk\.1\.ffn_gate_exps\.weight=IQ4_XS +blk\.1\.ffn_gate_shexp\.weight=Q8_0 +blk\.1\.ffn_up_exps\.weight=IQ4_XS +blk\.1\.ffn_up_shexp\.weight=Q8_0 +blk\.1\.ssm_alpha\.weight=F32 +blk\.1\.ssm_beta\.weight=F32 +blk\.1\.ssm_out\.weight=Q8_0 +blk\.10\.attn_gate\.weight=Q8_0 +blk\.10\.attn_qkv\.weight=Q8_0 +blk\.10\.ffn_down_exps\.weight=IQ4_XS +blk\.10\.ffn_down_shexp\.weight=Q8_0 +blk\.10\.ffn_gate_exps\.weight=IQ4_XS +blk\.10\.ffn_gate_shexp\.weight=Q8_0 +blk\.10\.ffn_up_exps\.weight=IQ4_XS +blk\.10\.ffn_up_shexp\.weight=Q8_0 +blk\.10\.ssm_alpha\.weight=F32 +blk\.10\.ssm_beta\.weight=F32 +blk\.10\.ssm_out\.weight=Q8_0 +blk\.11\.attn_k\.weight=F16 +blk\.11\.attn_output\.weight=Q8_0 +blk\.11\.attn_q\.weight=Q8_0 +blk\.11\.attn_v\.weight=F16 +blk\.11\.ffn_down_exps\.weight=IQ4_XS +blk\.11\.ffn_down_shexp\.weight=Q8_0 +blk\.11\.ffn_gate_exps\.weight=IQ4_XS +blk\.11\.ffn_gate_shexp\.weight=Q8_0 +blk\.11\.ffn_up_exps\.weight=IQ4_XS +blk\.11\.ffn_up_shexp\.weight=Q8_0 +blk\.12\.attn_gate\.weight=Q8_0 +blk\.12\.attn_qkv\.weight=Q8_0 +blk\.12\.ffn_down_exps\.weight=IQ4_XS +blk\.12\.ffn_down_shexp\.weight=Q8_0 +blk\.12\.ffn_gate_exps\.weight=IQ4_XS +blk\.12\.ffn_gate_shexp\.weight=Q8_0 +blk\.12\.ffn_up_exps\.weight=IQ4_XS +blk\.12\.ffn_up_shexp\.weight=Q8_0 +blk\.12\.ssm_alpha\.weight=F32 +blk\.12\.ssm_beta\.weight=F32 +blk\.12\.ssm_out\.weight=Q8_0 +blk\.13\.attn_gate\.weight=Q8_0 +blk\.13\.attn_qkv\.weight=Q8_0 +blk\.13\.ffn_down_exps\.weight=IQ4_XS +blk\.13\.ffn_down_shexp\.weight=Q8_0 +blk\.13\.ffn_gate_exps\.weight=IQ4_XS +blk\.13\.ffn_gate_shexp\.weight=Q8_0 +blk\.13\.ffn_up_exps\.weight=IQ4_XS +blk\.13\.ffn_up_shexp\.weight=Q8_0 +blk\.13\.ssm_alpha\.weight=F32 +blk\.13\.ssm_beta\.weight=F32 +blk\.13\.ssm_out\.weight=Q8_0 +blk\.14\.attn_gate\.weight=Q8_0 +blk\.14\.attn_qkv\.weight=Q8_0 +blk\.14\.ffn_down_exps\.weight=IQ4_XS +blk\.14\.ffn_down_shexp\.weight=Q8_0 +blk\.14\.ffn_gate_exps\.weight=IQ4_XS +blk\.14\.ffn_gate_shexp\.weight=Q8_0 +blk\.14\.ffn_up_exps\.weight=IQ4_XS +blk\.14\.ffn_up_shexp\.weight=Q8_0 +blk\.14\.ssm_alpha\.weight=F32 +blk\.14\.ssm_beta\.weight=F32 +blk\.14\.ssm_out\.weight=Q8_0 +blk\.15\.attn_k\.weight=F16 +blk\.15\.attn_output\.weight=Q8_0 +blk\.15\.attn_q\.weight=Q8_0 +blk\.15\.attn_v\.weight=F16 +blk\.15\.ffn_down_exps\.weight=IQ4_XS +blk\.15\.ffn_down_shexp\.weight=Q8_0 +blk\.15\.ffn_gate_exps\.weight=IQ4_XS +blk\.15\.ffn_gate_shexp\.weight=Q8_0 +blk\.15\.ffn_up_exps\.weight=IQ4_XS +blk\.15\.ffn_up_shexp\.weight=Q8_0 +blk\.16\.attn_gate\.weight=Q8_0 +blk\.16\.attn_qkv\.weight=Q8_0 +blk\.16\.ffn_down_exps\.weight=IQ4_XS +blk\.16\.ffn_down_shexp\.weight=Q8_0 +blk\.16\.ffn_gate_exps\.weight=IQ4_XS +blk\.16\.ffn_gate_shexp\.weight=Q8_0 +blk\.16\.ffn_up_exps\.weight=IQ4_XS +blk\.16\.ffn_up_shexp\.weight=Q8_0 +blk\.16\.ssm_alpha\.weight=F32 +blk\.16\.ssm_beta\.weight=F32 +blk\.16\.ssm_out\.weight=Q8_0 +blk\.17\.attn_gate\.weight=Q8_0 +blk\.17\.attn_qkv\.weight=Q8_0 +blk\.17\.ffn_down_exps\.weight=IQ4_XS +blk\.17\.ffn_down_shexp\.weight=Q8_0 +blk\.17\.ffn_gate_exps\.weight=IQ4_XS +blk\.17\.ffn_gate_shexp\.weight=Q8_0 +blk\.17\.ffn_up_exps\.weight=IQ4_XS +blk\.17\.ffn_up_shexp\.weight=Q8_0 +blk\.17\.ssm_alpha\.weight=F32 +blk\.17\.ssm_beta\.weight=F32 +blk\.17\.ssm_out\.weight=Q8_0 +blk\.18\.attn_gate\.weight=Q8_0 +blk\.18\.attn_qkv\.weight=Q8_0 +blk\.18\.ffn_down_exps\.weight=IQ4_XS +blk\.18\.ffn_down_shexp\.weight=Q8_0 +blk\.18\.ffn_gate_exps\.weight=IQ4_XS +blk\.18\.ffn_gate_shexp\.weight=Q8_0 +blk\.18\.ffn_up_exps\.weight=IQ4_XS +blk\.18\.ffn_up_shexp\.weight=Q8_0 +blk\.18\.ssm_alpha\.weight=F32 +blk\.18\.ssm_beta\.weight=F32 +blk\.18\.ssm_out\.weight=Q8_0 +blk\.19\.attn_k\.weight=F16 +blk\.19\.attn_output\.weight=Q8_0 +blk\.19\.attn_q\.weight=Q8_0 +blk\.19\.attn_v\.weight=F16 +blk\.19\.ffn_down_exps\.weight=IQ4_XS +blk\.19\.ffn_down_shexp\.weight=Q8_0 +blk\.19\.ffn_gate_exps\.weight=IQ4_XS +blk\.19\.ffn_gate_shexp\.weight=Q8_0 +blk\.19\.ffn_up_exps\.weight=IQ4_XS +blk\.19\.ffn_up_shexp\.weight=Q8_0 +blk\.2\.attn_gate\.weight=Q8_0 +blk\.2\.attn_qkv\.weight=Q8_0 +blk\.2\.ffn_down_exps\.weight=IQ4_XS +blk\.2\.ffn_down_shexp\.weight=Q8_0 +blk\.2\.ffn_gate_exps\.weight=IQ4_XS +blk\.2\.ffn_gate_shexp\.weight=Q8_0 +blk\.2\.ffn_up_exps\.weight=IQ4_XS +blk\.2\.ffn_up_shexp\.weight=Q8_0 +blk\.2\.ssm_alpha\.weight=F32 +blk\.2\.ssm_beta\.weight=F32 +blk\.2\.ssm_out\.weight=Q8_0 +blk\.20\.attn_gate\.weight=Q8_0 +blk\.20\.attn_qkv\.weight=Q8_0 +blk\.20\.ffn_down_exps\.weight=IQ4_XS +blk\.20\.ffn_down_shexp\.weight=Q8_0 +blk\.20\.ffn_gate_exps\.weight=IQ4_XS +blk\.20\.ffn_gate_shexp\.weight=Q8_0 +blk\.20\.ffn_up_exps\.weight=IQ4_XS +blk\.20\.ffn_up_shexp\.weight=Q8_0 +blk\.20\.ssm_alpha\.weight=F32 +blk\.20\.ssm_beta\.weight=F32 +blk\.20\.ssm_out\.weight=Q8_0 +blk\.21\.attn_gate\.weight=Q8_0 +blk\.21\.attn_qkv\.weight=Q8_0 +blk\.21\.ffn_down_exps\.weight=IQ4_XS +blk\.21\.ffn_down_shexp\.weight=Q8_0 +blk\.21\.ffn_gate_exps\.weight=IQ4_XS +blk\.21\.ffn_gate_shexp\.weight=Q8_0 +blk\.21\.ffn_up_exps\.weight=IQ4_XS +blk\.21\.ffn_up_shexp\.weight=Q8_0 +blk\.21\.ssm_alpha\.weight=F32 +blk\.21\.ssm_beta\.weight=F32 +blk\.21\.ssm_out\.weight=Q8_0 +blk\.22\.attn_gate\.weight=Q8_0 +blk\.22\.attn_qkv\.weight=Q8_0 +blk\.22\.ffn_down_exps\.weight=IQ4_XS +blk\.22\.ffn_down_shexp\.weight=Q8_0 +blk\.22\.ffn_gate_exps\.weight=IQ4_XS +blk\.22\.ffn_gate_shexp\.weight=Q8_0 +blk\.22\.ffn_up_exps\.weight=IQ4_XS +blk\.22\.ffn_up_shexp\.weight=Q8_0 +blk\.22\.ssm_alpha\.weight=F32 +blk\.22\.ssm_beta\.weight=F32 +blk\.22\.ssm_out\.weight=Q8_0 +blk\.23\.attn_k\.weight=F16 +blk\.23\.attn_output\.weight=Q8_0 +blk\.23\.attn_q\.weight=Q8_0 +blk\.23\.attn_v\.weight=F16 +blk\.23\.ffn_down_exps\.weight=IQ3_S +blk\.23\.ffn_down_shexp\.weight=Q8_0 +blk\.23\.ffn_gate_exps\.weight=IQ3_S +blk\.23\.ffn_gate_shexp\.weight=Q8_0 +blk\.23\.ffn_up_exps\.weight=IQ3_S +blk\.23\.ffn_up_shexp\.weight=Q8_0 +blk\.24\.attn_gate\.weight=Q8_0 +blk\.24\.attn_qkv\.weight=Q8_0 +blk\.24\.ffn_down_exps\.weight=IQ3_S +blk\.24\.ffn_down_shexp\.weight=Q8_0 +blk\.24\.ffn_gate_exps\.weight=IQ3_S +blk\.24\.ffn_gate_shexp\.weight=Q8_0 +blk\.24\.ffn_up_exps\.weight=IQ3_S +blk\.24\.ffn_up_shexp\.weight=Q8_0 +blk\.24\.ssm_alpha\.weight=F32 +blk\.24\.ssm_beta\.weight=F32 +blk\.24\.ssm_out\.weight=Q8_0 +blk\.25\.attn_gate\.weight=Q8_0 +blk\.25\.attn_qkv\.weight=Q8_0 +blk\.25\.ffn_down_exps\.weight=IQ3_S +blk\.25\.ffn_down_shexp\.weight=Q8_0 +blk\.25\.ffn_gate_exps\.weight=IQ3_S +blk\.25\.ffn_gate_shexp\.weight=Q8_0 +blk\.25\.ffn_up_exps\.weight=IQ3_S +blk\.25\.ffn_up_shexp\.weight=Q8_0 +blk\.25\.ssm_alpha\.weight=F32 +blk\.25\.ssm_beta\.weight=F32 +blk\.25\.ssm_out\.weight=Q8_0 +blk\.26\.attn_gate\.weight=Q8_0 +blk\.26\.attn_qkv\.weight=Q8_0 +blk\.26\.ffn_down_exps\.weight=IQ3_S +blk\.26\.ffn_down_shexp\.weight=Q8_0 +blk\.26\.ffn_gate_exps\.weight=IQ3_S +blk\.26\.ffn_gate_shexp\.weight=Q8_0 +blk\.26\.ffn_up_exps\.weight=IQ3_S +blk\.26\.ffn_up_shexp\.weight=Q8_0 +blk\.26\.ssm_alpha\.weight=F32 +blk\.26\.ssm_beta\.weight=F32 +blk\.26\.ssm_out\.weight=Q8_0 +blk\.27\.attn_k\.weight=F16 +blk\.27\.attn_output\.weight=Q8_0 +blk\.27\.attn_q\.weight=Q8_0 +blk\.27\.attn_v\.weight=F16 +blk\.27\.ffn_down_exps\.weight=IQ3_S +blk\.27\.ffn_down_shexp\.weight=Q8_0 +blk\.27\.ffn_gate_exps\.weight=IQ3_S +blk\.27\.ffn_gate_shexp\.weight=Q8_0 +blk\.27\.ffn_up_exps\.weight=IQ3_S +blk\.27\.ffn_up_shexp\.weight=Q8_0 +blk\.28\.attn_gate\.weight=Q8_0 +blk\.28\.attn_qkv\.weight=Q8_0 +blk\.28\.ffn_down_exps\.weight=IQ3_S +blk\.28\.ffn_down_shexp\.weight=Q8_0 +blk\.28\.ffn_gate_exps\.weight=IQ3_S +blk\.28\.ffn_gate_shexp\.weight=Q8_0 +blk\.28\.ffn_up_exps\.weight=IQ3_S +blk\.28\.ffn_up_shexp\.weight=Q8_0 +blk\.28\.ssm_alpha\.weight=F32 +blk\.28\.ssm_beta\.weight=F32 +blk\.28\.ssm_out\.weight=Q8_0 +blk\.29\.attn_gate\.weight=Q8_0 +blk\.29\.attn_qkv\.weight=Q8_0 +blk\.29\.ffn_down_exps\.weight=IQ3_S +blk\.29\.ffn_down_shexp\.weight=Q8_0 +blk\.29\.ffn_gate_exps\.weight=IQ3_S +blk\.29\.ffn_gate_shexp\.weight=Q8_0 +blk\.29\.ffn_up_exps\.weight=IQ3_S +blk\.29\.ffn_up_shexp\.weight=Q8_0 +blk\.29\.ssm_alpha\.weight=F32 +blk\.29\.ssm_beta\.weight=F32 +blk\.29\.ssm_out\.weight=Q8_0 +blk\.3\.attn_k\.weight=F16 +blk\.3\.attn_output\.weight=Q8_0 +blk\.3\.attn_q\.weight=Q8_0 +blk\.3\.attn_v\.weight=F16 +blk\.3\.ffn_down_exps\.weight=IQ4_XS +blk\.3\.ffn_down_shexp\.weight=Q8_0 +blk\.3\.ffn_gate_exps\.weight=IQ4_XS +blk\.3\.ffn_gate_shexp\.weight=Q8_0 +blk\.3\.ffn_up_exps\.weight=IQ4_XS +blk\.3\.ffn_up_shexp\.weight=Q8_0 +blk\.30\.attn_gate\.weight=Q8_0 +blk\.30\.attn_qkv\.weight=Q8_0 +blk\.30\.ffn_down_exps\.weight=IQ3_S +blk\.30\.ffn_down_shexp\.weight=Q8_0 +blk\.30\.ffn_gate_exps\.weight=IQ3_S +blk\.30\.ffn_gate_shexp\.weight=Q8_0 +blk\.30\.ffn_up_exps\.weight=IQ3_S +blk\.30\.ffn_up_shexp\.weight=Q8_0 +blk\.30\.ssm_alpha\.weight=F32 +blk\.30\.ssm_beta\.weight=F32 +blk\.30\.ssm_out\.weight=Q8_0 +blk\.31\.attn_k\.weight=F16 +blk\.31\.attn_output\.weight=Q8_0 +blk\.31\.attn_q\.weight=Q8_0 +blk\.31\.attn_v\.weight=F16 +blk\.31\.ffn_down_exps\.weight=IQ3_S +blk\.31\.ffn_down_shexp\.weight=Q8_0 +blk\.31\.ffn_gate_exps\.weight=IQ3_S +blk\.31\.ffn_gate_shexp\.weight=Q8_0 +blk\.31\.ffn_up_exps\.weight=IQ3_S +blk\.31\.ffn_up_shexp\.weight=Q8_0 +blk\.32\.attn_gate\.weight=Q8_0 +blk\.32\.attn_qkv\.weight=Q8_0 +blk\.32\.ffn_down_exps\.weight=IQ3_S +blk\.32\.ffn_down_shexp\.weight=Q8_0 +blk\.32\.ffn_gate_exps\.weight=IQ3_S +blk\.32\.ffn_gate_shexp\.weight=Q8_0 +blk\.32\.ffn_up_exps\.weight=IQ3_S +blk\.32\.ffn_up_shexp\.weight=Q8_0 +blk\.32\.ssm_alpha\.weight=F32 +blk\.32\.ssm_beta\.weight=F32 +blk\.32\.ssm_out\.weight=Q8_0 +blk\.33\.attn_gate\.weight=Q8_0 +blk\.33\.attn_qkv\.weight=Q8_0 +blk\.33\.ffn_down_exps\.weight=IQ3_S +blk\.33\.ffn_down_shexp\.weight=Q8_0 +blk\.33\.ffn_gate_exps\.weight=IQ3_S +blk\.33\.ffn_gate_shexp\.weight=Q8_0 +blk\.33\.ffn_up_exps\.weight=IQ3_S +blk\.33\.ffn_up_shexp\.weight=Q8_0 +blk\.33\.ssm_alpha\.weight=F32 +blk\.33\.ssm_beta\.weight=F32 +blk\.33\.ssm_out\.weight=Q8_0 +blk\.34\.attn_gate\.weight=Q8_0 +blk\.34\.attn_qkv\.weight=Q8_0 +blk\.34\.ffn_down_exps\.weight=IQ3_S +blk\.34\.ffn_down_shexp\.weight=Q8_0 +blk\.34\.ffn_gate_exps\.weight=IQ3_S +blk\.34\.ffn_gate_shexp\.weight=Q8_0 +blk\.34\.ffn_up_exps\.weight=IQ3_S +blk\.34\.ffn_up_shexp\.weight=Q8_0 +blk\.34\.ssm_alpha\.weight=F32 +blk\.34\.ssm_beta\.weight=F32 +blk\.34\.ssm_out\.weight=Q8_0 +blk\.35\.attn_k\.weight=F16 +blk\.35\.attn_output\.weight=Q8_0 +blk\.35\.attn_q\.weight=Q8_0 +blk\.35\.attn_v\.weight=F16 +blk\.35\.ffn_down_exps\.weight=IQ3_S +blk\.35\.ffn_down_shexp\.weight=Q8_0 +blk\.35\.ffn_gate_exps\.weight=IQ3_S +blk\.35\.ffn_gate_shexp\.weight=Q8_0 +blk\.35\.ffn_up_exps\.weight=IQ3_S +blk\.35\.ffn_up_shexp\.weight=Q8_0 +blk\.36\.attn_gate\.weight=Q8_0 +blk\.36\.attn_qkv\.weight=Q8_0 +blk\.36\.ffn_down_exps\.weight=IQ3_S +blk\.36\.ffn_down_shexp\.weight=Q8_0 +blk\.36\.ffn_gate_exps\.weight=IQ3_S +blk\.36\.ffn_gate_shexp\.weight=Q8_0 +blk\.36\.ffn_up_exps\.weight=IQ3_S +blk\.36\.ffn_up_shexp\.weight=Q8_0 +blk\.36\.ssm_alpha\.weight=F32 +blk\.36\.ssm_beta\.weight=F32 +blk\.36\.ssm_out\.weight=Q8_0 +blk\.37\.attn_gate\.weight=Q8_0 +blk\.37\.attn_qkv\.weight=Q8_0 +blk\.37\.ffn_down_exps\.weight=IQ3_S +blk\.37\.ffn_down_shexp\.weight=Q8_0 +blk\.37\.ffn_gate_exps\.weight=IQ3_S +blk\.37\.ffn_gate_shexp\.weight=Q8_0 +blk\.37\.ffn_up_exps\.weight=IQ3_S +blk\.37\.ffn_up_shexp\.weight=Q8_0 +blk\.37\.ssm_alpha\.weight=F32 +blk\.37\.ssm_beta\.weight=F32 +blk\.37\.ssm_out\.weight=Q8_0 +blk\.38\.attn_gate\.weight=Q8_0 +blk\.38\.attn_qkv\.weight=Q8_0 +blk\.38\.ffn_down_exps\.weight=IQ3_S +blk\.38\.ffn_down_shexp\.weight=Q8_0 +blk\.38\.ffn_gate_exps\.weight=IQ3_S +blk\.38\.ffn_gate_shexp\.weight=Q8_0 +blk\.38\.ffn_up_exps\.weight=IQ3_S +blk\.38\.ffn_up_shexp\.weight=Q8_0 +blk\.38\.ssm_alpha\.weight=F32 +blk\.38\.ssm_beta\.weight=F32 +blk\.38\.ssm_out\.weight=Q8_0 +blk\.39\.attn_k\.weight=F16 +blk\.39\.attn_output\.weight=Q8_0 +blk\.39\.attn_q\.weight=Q8_0 +blk\.39\.attn_v\.weight=F16 +blk\.39\.ffn_down_exps\.weight=IQ3_S +blk\.39\.ffn_down_shexp\.weight=Q8_0 +blk\.39\.ffn_gate_exps\.weight=IQ3_S +blk\.39\.ffn_gate_shexp\.weight=Q8_0 +blk\.39\.ffn_up_exps\.weight=IQ3_S +blk\.39\.ffn_up_shexp\.weight=Q8_0 +blk\.4\.attn_gate\.weight=Q8_0 +blk\.4\.attn_qkv\.weight=Q8_0 +blk\.4\.ffn_down_exps\.weight=IQ4_XS +blk\.4\.ffn_down_shexp\.weight=Q8_0 +blk\.4\.ffn_gate_exps\.weight=IQ4_XS +blk\.4\.ffn_gate_shexp\.weight=Q8_0 +blk\.4\.ffn_up_exps\.weight=IQ4_XS +blk\.4\.ffn_up_shexp\.weight=Q8_0 +blk\.4\.ssm_alpha\.weight=F32 +blk\.4\.ssm_beta\.weight=F32 +blk\.4\.ssm_out\.weight=Q8_0 +blk\.40\.attn_k\.weight=F16 +blk\.40\.attn_output\.weight=Q8_0 +blk\.40\.attn_q\.weight=Q8_0 +blk\.40\.attn_v\.weight=F16 +blk\.40\.ffn_down_exps\.weight=IQ3_S +blk\.40\.ffn_down_shexp\.weight=Q8_0 +blk\.40\.ffn_gate_exps\.weight=IQ3_S +blk\.40\.ffn_gate_shexp\.weight=Q8_0 +blk\.40\.ffn_up_exps\.weight=IQ3_S +blk\.40\.ffn_up_shexp\.weight=Q8_0 +blk\.40\.nextn\.eh_proj\.weight=Q8_0 +blk\.5\.attn_gate\.weight=Q8_0 +blk\.5\.attn_qkv\.weight=Q8_0 +blk\.5\.ffn_down_exps\.weight=IQ4_XS +blk\.5\.ffn_down_shexp\.weight=Q8_0 +blk\.5\.ffn_gate_exps\.weight=IQ4_XS +blk\.5\.ffn_gate_shexp\.weight=Q8_0 +blk\.5\.ffn_up_exps\.weight=IQ4_XS +blk\.5\.ffn_up_shexp\.weight=Q8_0 +blk\.5\.ssm_alpha\.weight=F32 +blk\.5\.ssm_beta\.weight=F32 +blk\.5\.ssm_out\.weight=Q8_0 +blk\.6\.attn_gate\.weight=Q8_0 +blk\.6\.attn_qkv\.weight=Q8_0 +blk\.6\.ffn_down_exps\.weight=IQ4_XS +blk\.6\.ffn_down_shexp\.weight=Q8_0 +blk\.6\.ffn_gate_exps\.weight=IQ4_XS +blk\.6\.ffn_gate_shexp\.weight=Q8_0 +blk\.6\.ffn_up_exps\.weight=IQ4_XS +blk\.6\.ffn_up_shexp\.weight=Q8_0 +blk\.6\.ssm_alpha\.weight=F32 +blk\.6\.ssm_beta\.weight=F32 +blk\.6\.ssm_out\.weight=Q8_0 +blk\.7\.attn_k\.weight=F16 +blk\.7\.attn_output\.weight=Q8_0 +blk\.7\.attn_q\.weight=Q8_0 +blk\.7\.attn_v\.weight=F16 +blk\.7\.ffn_down_exps\.weight=IQ4_XS +blk\.7\.ffn_down_shexp\.weight=Q8_0 +blk\.7\.ffn_gate_exps\.weight=IQ4_XS +blk\.7\.ffn_gate_shexp\.weight=Q8_0 +blk\.7\.ffn_up_exps\.weight=IQ4_XS +blk\.7\.ffn_up_shexp\.weight=Q8_0 +blk\.8\.attn_gate\.weight=Q8_0 +blk\.8\.attn_qkv\.weight=Q8_0 +blk\.8\.ffn_down_exps\.weight=IQ4_XS +blk\.8\.ffn_down_shexp\.weight=Q8_0 +blk\.8\.ffn_gate_exps\.weight=IQ4_XS +blk\.8\.ffn_gate_shexp\.weight=Q8_0 +blk\.8\.ffn_up_exps\.weight=IQ4_XS +blk\.8\.ffn_up_shexp\.weight=Q8_0 +blk\.8\.ssm_alpha\.weight=F32 +blk\.8\.ssm_beta\.weight=F32 +blk\.8\.ssm_out\.weight=Q8_0 +blk\.9\.attn_gate\.weight=Q8_0 +blk\.9\.attn_qkv\.weight=Q8_0 +blk\.9\.ffn_down_exps\.weight=IQ4_XS +blk\.9\.ffn_down_shexp\.weight=Q8_0 +blk\.9\.ffn_gate_exps\.weight=IQ4_XS +blk\.9\.ffn_gate_shexp\.weight=Q8_0 +blk\.9\.ffn_up_exps\.weight=IQ4_XS +blk\.9\.ffn_up_shexp\.weight=Q8_0 +blk\.9\.ssm_alpha\.weight=F32 +blk\.9\.ssm_beta\.weight=F32 +blk\.9\.ssm_out\.weight=Q8_0 diff --git a/appendix/recipes/cfg_19G-ICE_previous.txt b/appendix/recipes/cfg_19G-ICE_previous.txt new file mode 100644 index 0000000000000000000000000000000000000000..470d5b594008645892690568f20a5073908502c8 --- /dev/null +++ b/appendix/recipes/cfg_19G-ICE_previous.txt @@ -0,0 +1,443 @@ +^output\.weight$=Q8_0 +token_embd\.weight=Q8_0 +blk\.0\.attn_gate\.weight=Q8_0 +blk\.0\.attn_qkv\.weight=Q8_0 +blk\.0\.ffn_down_exps\.weight=IQ4_XS +blk\.0\.ffn_down_shexp\.weight=Q8_0 +blk\.0\.ffn_gate_exps\.weight=IQ4_XS +blk\.0\.ffn_gate_shexp\.weight=Q8_0 +blk\.0\.ffn_up_exps\.weight=IQ4_XS +blk\.0\.ffn_up_shexp\.weight=Q8_0 +blk\.0\.ssm_alpha\.weight=F32 +blk\.0\.ssm_beta\.weight=F32 +blk\.0\.ssm_out\.weight=Q8_0 +blk\.1\.attn_gate\.weight=Q8_0 +blk\.1\.attn_qkv\.weight=Q8_0 +blk\.1\.ffn_down_exps\.weight=IQ3_S +blk\.1\.ffn_down_shexp\.weight=Q8_0 +blk\.1\.ffn_gate_exps\.weight=IQ3_S +blk\.1\.ffn_gate_shexp\.weight=Q8_0 +blk\.1\.ffn_up_exps\.weight=IQ3_S +blk\.1\.ffn_up_shexp\.weight=Q8_0 +blk\.1\.ssm_alpha\.weight=F32 +blk\.1\.ssm_beta\.weight=F32 +blk\.1\.ssm_out\.weight=Q8_0 +blk\.10\.attn_gate\.weight=Q8_0 +blk\.10\.attn_qkv\.weight=Q8_0 +blk\.10\.ffn_down_exps\.weight=IQ3_S +blk\.10\.ffn_down_shexp\.weight=Q8_0 +blk\.10\.ffn_gate_exps\.weight=IQ3_S +blk\.10\.ffn_gate_shexp\.weight=Q8_0 +blk\.10\.ffn_up_exps\.weight=IQ3_S +blk\.10\.ffn_up_shexp\.weight=Q8_0 +blk\.10\.ssm_alpha\.weight=F32 +blk\.10\.ssm_beta\.weight=F32 +blk\.10\.ssm_out\.weight=Q8_0 +blk\.11\.attn_k\.weight=F16 +blk\.11\.attn_output\.weight=Q8_0 +blk\.11\.attn_q\.weight=Q8_0 +blk\.11\.attn_v\.weight=F16 +blk\.11\.ffn_down_exps\.weight=IQ4_XS +blk\.11\.ffn_down_shexp\.weight=Q8_0 +blk\.11\.ffn_gate_exps\.weight=IQ4_XS +blk\.11\.ffn_gate_shexp\.weight=Q8_0 +blk\.11\.ffn_up_exps\.weight=IQ4_XS +blk\.11\.ffn_up_shexp\.weight=Q8_0 +blk\.12\.attn_gate\.weight=Q8_0 +blk\.12\.attn_qkv\.weight=Q8_0 +blk\.12\.ffn_down_exps\.weight=IQ4_XS +blk\.12\.ffn_down_shexp\.weight=Q8_0 +blk\.12\.ffn_gate_exps\.weight=IQ4_XS +blk\.12\.ffn_gate_shexp\.weight=Q8_0 +blk\.12\.ffn_up_exps\.weight=IQ4_XS +blk\.12\.ffn_up_shexp\.weight=Q8_0 +blk\.12\.ssm_alpha\.weight=F32 +blk\.12\.ssm_beta\.weight=F32 +blk\.12\.ssm_out\.weight=Q8_0 +blk\.13\.attn_gate\.weight=Q8_0 +blk\.13\.attn_qkv\.weight=Q8_0 +blk\.13\.ffn_down_exps\.weight=IQ3_S +blk\.13\.ffn_down_shexp\.weight=Q8_0 +blk\.13\.ffn_gate_exps\.weight=IQ3_S +blk\.13\.ffn_gate_shexp\.weight=Q8_0 +blk\.13\.ffn_up_exps\.weight=IQ3_S +blk\.13\.ffn_up_shexp\.weight=Q8_0 +blk\.13\.ssm_alpha\.weight=F32 +blk\.13\.ssm_beta\.weight=F32 +blk\.13\.ssm_out\.weight=Q8_0 +blk\.14\.attn_gate\.weight=Q8_0 +blk\.14\.attn_qkv\.weight=Q8_0 +blk\.14\.ffn_down_exps\.weight=IQ4_XS +blk\.14\.ffn_down_shexp\.weight=Q8_0 +blk\.14\.ffn_gate_exps\.weight=IQ4_XS +blk\.14\.ffn_gate_shexp\.weight=Q8_0 +blk\.14\.ffn_up_exps\.weight=IQ4_XS +blk\.14\.ffn_up_shexp\.weight=Q8_0 +blk\.14\.ssm_alpha\.weight=F32 +blk\.14\.ssm_beta\.weight=F32 +blk\.14\.ssm_out\.weight=Q8_0 +blk\.15\.attn_k\.weight=F16 +blk\.15\.attn_output\.weight=Q8_0 +blk\.15\.attn_q\.weight=Q8_0 +blk\.15\.attn_v\.weight=F16 +blk\.15\.ffn_down_exps\.weight=IQ3_S +blk\.15\.ffn_down_shexp\.weight=Q8_0 +blk\.15\.ffn_gate_exps\.weight=IQ3_S +blk\.15\.ffn_gate_shexp\.weight=Q8_0 +blk\.15\.ffn_up_exps\.weight=IQ3_S +blk\.15\.ffn_up_shexp\.weight=Q8_0 +blk\.16\.attn_gate\.weight=Q8_0 +blk\.16\.attn_qkv\.weight=Q8_0 +blk\.16\.ffn_down_exps\.weight=IQ4_XS +blk\.16\.ffn_down_shexp\.weight=Q8_0 +blk\.16\.ffn_gate_exps\.weight=IQ4_XS +blk\.16\.ffn_gate_shexp\.weight=Q8_0 +blk\.16\.ffn_up_exps\.weight=IQ4_XS +blk\.16\.ffn_up_shexp\.weight=Q8_0 +blk\.16\.ssm_alpha\.weight=F32 +blk\.16\.ssm_beta\.weight=F32 +blk\.16\.ssm_out\.weight=Q8_0 +blk\.17\.attn_gate\.weight=Q8_0 +blk\.17\.attn_qkv\.weight=Q8_0 +blk\.17\.ffn_down_exps\.weight=IQ3_S +blk\.17\.ffn_down_shexp\.weight=Q8_0 +blk\.17\.ffn_gate_exps\.weight=IQ3_S +blk\.17\.ffn_gate_shexp\.weight=Q8_0 +blk\.17\.ffn_up_exps\.weight=IQ3_S +blk\.17\.ffn_up_shexp\.weight=Q8_0 +blk\.17\.ssm_alpha\.weight=F32 +blk\.17\.ssm_beta\.weight=F32 +blk\.17\.ssm_out\.weight=Q8_0 +blk\.18\.attn_gate\.weight=Q8_0 +blk\.18\.attn_qkv\.weight=Q8_0 +blk\.18\.ffn_down_exps\.weight=IQ4_XS +blk\.18\.ffn_down_shexp\.weight=Q8_0 +blk\.18\.ffn_gate_exps\.weight=IQ4_XS +blk\.18\.ffn_gate_shexp\.weight=Q8_0 +blk\.18\.ffn_up_exps\.weight=IQ4_XS +blk\.18\.ffn_up_shexp\.weight=Q8_0 +blk\.18\.ssm_alpha\.weight=F32 +blk\.18\.ssm_beta\.weight=F32 +blk\.18\.ssm_out\.weight=Q8_0 +blk\.19\.attn_k\.weight=F16 +blk\.19\.attn_output\.weight=Q8_0 +blk\.19\.attn_q\.weight=Q8_0 +blk\.19\.attn_v\.weight=F16 +blk\.19\.ffn_down_exps\.weight=IQ3_S +blk\.19\.ffn_down_shexp\.weight=Q8_0 +blk\.19\.ffn_gate_exps\.weight=IQ3_S +blk\.19\.ffn_gate_shexp\.weight=Q8_0 +blk\.19\.ffn_up_exps\.weight=IQ3_S +blk\.19\.ffn_up_shexp\.weight=Q8_0 +blk\.2\.attn_gate\.weight=Q8_0 +blk\.2\.attn_qkv\.weight=Q8_0 +blk\.2\.ffn_down_exps\.weight=IQ4_XS +blk\.2\.ffn_down_shexp\.weight=Q8_0 +blk\.2\.ffn_gate_exps\.weight=IQ4_XS +blk\.2\.ffn_gate_shexp\.weight=Q8_0 +blk\.2\.ffn_up_exps\.weight=IQ4_XS +blk\.2\.ffn_up_shexp\.weight=Q8_0 +blk\.2\.ssm_alpha\.weight=F32 +blk\.2\.ssm_beta\.weight=F32 +blk\.2\.ssm_out\.weight=Q8_0 +blk\.20\.attn_gate\.weight=Q8_0 +blk\.20\.attn_qkv\.weight=Q8_0 +blk\.20\.ffn_down_exps\.weight=IQ4_XS +blk\.20\.ffn_down_shexp\.weight=Q8_0 +blk\.20\.ffn_gate_exps\.weight=IQ4_XS +blk\.20\.ffn_gate_shexp\.weight=Q8_0 +blk\.20\.ffn_up_exps\.weight=IQ4_XS +blk\.20\.ffn_up_shexp\.weight=Q8_0 +blk\.20\.ssm_alpha\.weight=F32 +blk\.20\.ssm_beta\.weight=F32 +blk\.20\.ssm_out\.weight=Q8_0 +blk\.21\.attn_gate\.weight=Q8_0 +blk\.21\.attn_qkv\.weight=Q8_0 +blk\.21\.ffn_down_exps\.weight=IQ4_XS +blk\.21\.ffn_down_shexp\.weight=Q8_0 +blk\.21\.ffn_gate_exps\.weight=IQ4_XS +blk\.21\.ffn_gate_shexp\.weight=Q8_0 +blk\.21\.ffn_up_exps\.weight=IQ4_XS +blk\.21\.ffn_up_shexp\.weight=Q8_0 +blk\.21\.ssm_alpha\.weight=F32 +blk\.21\.ssm_beta\.weight=F32 +blk\.21\.ssm_out\.weight=Q8_0 +blk\.22\.attn_gate\.weight=Q8_0 +blk\.22\.attn_qkv\.weight=Q8_0 +blk\.22\.ffn_down_exps\.weight=IQ3_S +blk\.22\.ffn_down_shexp\.weight=Q8_0 +blk\.22\.ffn_gate_exps\.weight=IQ3_S +blk\.22\.ffn_gate_shexp\.weight=Q8_0 +blk\.22\.ffn_up_exps\.weight=IQ3_S +blk\.22\.ffn_up_shexp\.weight=Q8_0 +blk\.22\.ssm_alpha\.weight=F32 +blk\.22\.ssm_beta\.weight=F32 +blk\.22\.ssm_out\.weight=Q8_0 +blk\.23\.attn_k\.weight=F16 +blk\.23\.attn_output\.weight=Q8_0 +blk\.23\.attn_q\.weight=Q8_0 +blk\.23\.attn_v\.weight=F16 +blk\.23\.ffn_down_exps\.weight=IQ4_XS +blk\.23\.ffn_down_shexp\.weight=Q8_0 +blk\.23\.ffn_gate_exps\.weight=IQ4_XS +blk\.23\.ffn_gate_shexp\.weight=Q8_0 +blk\.23\.ffn_up_exps\.weight=IQ4_XS +blk\.23\.ffn_up_shexp\.weight=Q8_0 +blk\.24\.attn_gate\.weight=Q8_0 +blk\.24\.attn_qkv\.weight=Q8_0 +blk\.24\.ffn_down_exps\.weight=IQ3_S +blk\.24\.ffn_down_shexp\.weight=Q8_0 +blk\.24\.ffn_gate_exps\.weight=IQ3_S +blk\.24\.ffn_gate_shexp\.weight=Q8_0 +blk\.24\.ffn_up_exps\.weight=IQ3_S +blk\.24\.ffn_up_shexp\.weight=Q8_0 +blk\.24\.ssm_alpha\.weight=F32 +blk\.24\.ssm_beta\.weight=F32 +blk\.24\.ssm_out\.weight=Q8_0 +blk\.25\.attn_gate\.weight=Q8_0 +blk\.25\.attn_qkv\.weight=Q8_0 +blk\.25\.ffn_down_exps\.weight=IQ4_XS +blk\.25\.ffn_down_shexp\.weight=Q8_0 +blk\.25\.ffn_gate_exps\.weight=IQ4_XS +blk\.25\.ffn_gate_shexp\.weight=Q8_0 +blk\.25\.ffn_up_exps\.weight=IQ4_XS +blk\.25\.ffn_up_shexp\.weight=Q8_0 +blk\.25\.ssm_alpha\.weight=F32 +blk\.25\.ssm_beta\.weight=F32 +blk\.25\.ssm_out\.weight=Q8_0 +blk\.26\.attn_gate\.weight=Q8_0 +blk\.26\.attn_qkv\.weight=Q8_0 +blk\.26\.ffn_down_exps\.weight=IQ3_S +blk\.26\.ffn_down_shexp\.weight=Q8_0 +blk\.26\.ffn_gate_exps\.weight=IQ3_S +blk\.26\.ffn_gate_shexp\.weight=Q8_0 +blk\.26\.ffn_up_exps\.weight=IQ3_S +blk\.26\.ffn_up_shexp\.weight=Q8_0 +blk\.26\.ssm_alpha\.weight=F32 +blk\.26\.ssm_beta\.weight=F32 +blk\.26\.ssm_out\.weight=Q8_0 +blk\.27\.attn_k\.weight=F16 +blk\.27\.attn_output\.weight=Q8_0 +blk\.27\.attn_q\.weight=Q8_0 +blk\.27\.attn_v\.weight=F16 +blk\.27\.ffn_down_exps\.weight=IQ4_XS +blk\.27\.ffn_down_shexp\.weight=Q8_0 +blk\.27\.ffn_gate_exps\.weight=IQ4_XS +blk\.27\.ffn_gate_shexp\.weight=Q8_0 +blk\.27\.ffn_up_exps\.weight=IQ4_XS +blk\.27\.ffn_up_shexp\.weight=Q8_0 +blk\.28\.attn_gate\.weight=Q8_0 +blk\.28\.attn_qkv\.weight=Q8_0 +blk\.28\.ffn_down_exps\.weight=IQ3_S +blk\.28\.ffn_down_shexp\.weight=Q8_0 +blk\.28\.ffn_gate_exps\.weight=IQ3_S +blk\.28\.ffn_gate_shexp\.weight=Q8_0 +blk\.28\.ffn_up_exps\.weight=IQ3_S +blk\.28\.ffn_up_shexp\.weight=Q8_0 +blk\.28\.ssm_alpha\.weight=F32 +blk\.28\.ssm_beta\.weight=F32 +blk\.28\.ssm_out\.weight=Q8_0 +blk\.29\.attn_gate\.weight=Q8_0 +blk\.29\.attn_qkv\.weight=Q8_0 +blk\.29\.ffn_down_exps\.weight=IQ4_XS +blk\.29\.ffn_down_shexp\.weight=Q8_0 +blk\.29\.ffn_gate_exps\.weight=IQ4_XS +blk\.29\.ffn_gate_shexp\.weight=Q8_0 +blk\.29\.ffn_up_exps\.weight=IQ4_XS +blk\.29\.ffn_up_shexp\.weight=Q8_0 +blk\.29\.ssm_alpha\.weight=F32 +blk\.29\.ssm_beta\.weight=F32 +blk\.29\.ssm_out\.weight=Q8_0 +blk\.3\.attn_k\.weight=F16 +blk\.3\.attn_output\.weight=Q8_0 +blk\.3\.attn_q\.weight=Q8_0 +blk\.3\.attn_v\.weight=F16 +blk\.3\.ffn_down_exps\.weight=IQ3_S +blk\.3\.ffn_down_shexp\.weight=Q8_0 +blk\.3\.ffn_gate_exps\.weight=IQ3_S +blk\.3\.ffn_gate_shexp\.weight=Q8_0 +blk\.3\.ffn_up_exps\.weight=IQ3_S +blk\.3\.ffn_up_shexp\.weight=Q8_0 +blk\.30\.attn_gate\.weight=Q8_0 +blk\.30\.attn_qkv\.weight=Q8_0 +blk\.30\.ffn_down_exps\.weight=IQ4_XS +blk\.30\.ffn_down_shexp\.weight=Q8_0 +blk\.30\.ffn_gate_exps\.weight=IQ4_XS +blk\.30\.ffn_gate_shexp\.weight=Q8_0 +blk\.30\.ffn_up_exps\.weight=IQ4_XS +blk\.30\.ffn_up_shexp\.weight=Q8_0 +blk\.30\.ssm_alpha\.weight=F32 +blk\.30\.ssm_beta\.weight=F32 +blk\.30\.ssm_out\.weight=Q8_0 +blk\.31\.attn_k\.weight=F16 +blk\.31\.attn_output\.weight=Q8_0 +blk\.31\.attn_q\.weight=Q8_0 +blk\.31\.attn_v\.weight=F16 +blk\.31\.ffn_down_exps\.weight=IQ3_S +blk\.31\.ffn_down_shexp\.weight=Q8_0 +blk\.31\.ffn_gate_exps\.weight=IQ3_S +blk\.31\.ffn_gate_shexp\.weight=Q8_0 +blk\.31\.ffn_up_exps\.weight=IQ3_S +blk\.31\.ffn_up_shexp\.weight=Q8_0 +blk\.32\.attn_gate\.weight=Q8_0 +blk\.32\.attn_qkv\.weight=Q8_0 +blk\.32\.ffn_down_exps\.weight=IQ4_XS +blk\.32\.ffn_down_shexp\.weight=Q8_0 +blk\.32\.ffn_gate_exps\.weight=IQ4_XS +blk\.32\.ffn_gate_shexp\.weight=Q8_0 +blk\.32\.ffn_up_exps\.weight=IQ4_XS +blk\.32\.ffn_up_shexp\.weight=Q8_0 +blk\.32\.ssm_alpha\.weight=F32 +blk\.32\.ssm_beta\.weight=F32 +blk\.32\.ssm_out\.weight=Q8_0 +blk\.33\.attn_gate\.weight=Q8_0 +blk\.33\.attn_qkv\.weight=Q8_0 +blk\.33\.ffn_down_exps\.weight=IQ3_S +blk\.33\.ffn_down_shexp\.weight=Q8_0 +blk\.33\.ffn_gate_exps\.weight=IQ3_S +blk\.33\.ffn_gate_shexp\.weight=Q8_0 +blk\.33\.ffn_up_exps\.weight=IQ3_S +blk\.33\.ffn_up_shexp\.weight=Q8_0 +blk\.33\.ssm_alpha\.weight=F32 +blk\.33\.ssm_beta\.weight=F32 +blk\.33\.ssm_out\.weight=Q8_0 +blk\.34\.attn_gate\.weight=Q8_0 +blk\.34\.attn_qkv\.weight=Q8_0 +blk\.34\.ffn_down_exps\.weight=IQ4_XS +blk\.34\.ffn_down_shexp\.weight=Q8_0 +blk\.34\.ffn_gate_exps\.weight=IQ4_XS +blk\.34\.ffn_gate_shexp\.weight=Q8_0 +blk\.34\.ffn_up_exps\.weight=IQ4_XS +blk\.34\.ffn_up_shexp\.weight=Q8_0 +blk\.34\.ssm_alpha\.weight=F32 +blk\.34\.ssm_beta\.weight=F32 +blk\.34\.ssm_out\.weight=Q8_0 +blk\.35\.attn_k\.weight=F16 +blk\.35\.attn_output\.weight=Q8_0 +blk\.35\.attn_q\.weight=Q8_0 +blk\.35\.attn_v\.weight=F16 +blk\.35\.ffn_down_exps\.weight=IQ3_S +blk\.35\.ffn_down_shexp\.weight=Q8_0 +blk\.35\.ffn_gate_exps\.weight=IQ3_S +blk\.35\.ffn_gate_shexp\.weight=Q8_0 +blk\.35\.ffn_up_exps\.weight=IQ3_S +blk\.35\.ffn_up_shexp\.weight=Q8_0 +blk\.36\.attn_gate\.weight=Q8_0 +blk\.36\.attn_qkv\.weight=Q8_0 +blk\.36\.ffn_down_exps\.weight=IQ4_XS +blk\.36\.ffn_down_shexp\.weight=Q8_0 +blk\.36\.ffn_gate_exps\.weight=IQ4_XS +blk\.36\.ffn_gate_shexp\.weight=Q8_0 +blk\.36\.ffn_up_exps\.weight=IQ4_XS +blk\.36\.ffn_up_shexp\.weight=Q8_0 +blk\.36\.ssm_alpha\.weight=F32 +blk\.36\.ssm_beta\.weight=F32 +blk\.36\.ssm_out\.weight=Q8_0 +blk\.37\.attn_gate\.weight=Q8_0 +blk\.37\.attn_qkv\.weight=Q8_0 +blk\.37\.ffn_down_exps\.weight=IQ4_XS +blk\.37\.ffn_down_shexp\.weight=Q8_0 +blk\.37\.ffn_gate_exps\.weight=IQ4_XS +blk\.37\.ffn_gate_shexp\.weight=Q8_0 +blk\.37\.ffn_up_exps\.weight=IQ4_XS +blk\.37\.ffn_up_shexp\.weight=Q8_0 +blk\.37\.ssm_alpha\.weight=F32 +blk\.37\.ssm_beta\.weight=F32 +blk\.37\.ssm_out\.weight=Q8_0 +blk\.38\.attn_gate\.weight=Q8_0 +blk\.38\.attn_qkv\.weight=Q8_0 +blk\.38\.ffn_down_exps\.weight=IQ3_S +blk\.38\.ffn_down_shexp\.weight=Q8_0 +blk\.38\.ffn_gate_exps\.weight=IQ3_S +blk\.38\.ffn_gate_shexp\.weight=Q8_0 +blk\.38\.ffn_up_exps\.weight=IQ3_S +blk\.38\.ffn_up_shexp\.weight=Q8_0 +blk\.38\.ssm_alpha\.weight=F32 +blk\.38\.ssm_beta\.weight=F32 +blk\.38\.ssm_out\.weight=Q8_0 +blk\.39\.attn_k\.weight=F16 +blk\.39\.attn_output\.weight=Q8_0 +blk\.39\.attn_q\.weight=Q8_0 +blk\.39\.attn_v\.weight=F16 +blk\.39\.ffn_down_exps\.weight=IQ4_XS +blk\.39\.ffn_down_shexp\.weight=Q8_0 +blk\.39\.ffn_gate_exps\.weight=IQ4_XS +blk\.39\.ffn_gate_shexp\.weight=Q8_0 +blk\.39\.ffn_up_exps\.weight=IQ4_XS +blk\.39\.ffn_up_shexp\.weight=Q8_0 +blk\.4\.attn_gate\.weight=Q8_0 +blk\.4\.attn_qkv\.weight=Q8_0 +blk\.4\.ffn_down_exps\.weight=IQ4_XS +blk\.4\.ffn_down_shexp\.weight=Q8_0 +blk\.4\.ffn_gate_exps\.weight=IQ4_XS +blk\.4\.ffn_gate_shexp\.weight=Q8_0 +blk\.4\.ffn_up_exps\.weight=IQ4_XS +blk\.4\.ffn_up_shexp\.weight=Q8_0 +blk\.4\.ssm_alpha\.weight=F32 +blk\.4\.ssm_beta\.weight=F32 +blk\.4\.ssm_out\.weight=Q8_0 +blk\.40\.attn_k\.weight=F16 +blk\.40\.attn_output\.weight=Q8_0 +blk\.40\.attn_q\.weight=Q8_0 +blk\.40\.attn_v\.weight=F16 +blk\.40\.ffn_down_exps\.weight=IQ3_S +blk\.40\.ffn_down_shexp\.weight=Q8_0 +blk\.40\.ffn_gate_exps\.weight=IQ3_S +blk\.40\.ffn_gate_shexp\.weight=Q8_0 +blk\.40\.ffn_up_exps\.weight=IQ3_S +blk\.40\.ffn_up_shexp\.weight=Q8_0 +blk\.40\.nextn\.eh_proj\.weight=Q8_0 +blk\.5\.attn_gate\.weight=Q8_0 +blk\.5\.attn_qkv\.weight=Q8_0 +blk\.5\.ffn_down_exps\.weight=IQ4_XS +blk\.5\.ffn_down_shexp\.weight=Q8_0 +blk\.5\.ffn_gate_exps\.weight=IQ4_XS +blk\.5\.ffn_gate_shexp\.weight=Q8_0 +blk\.5\.ffn_up_exps\.weight=IQ4_XS +blk\.5\.ffn_up_shexp\.weight=Q8_0 +blk\.5\.ssm_alpha\.weight=F32 +blk\.5\.ssm_beta\.weight=F32 +blk\.5\.ssm_out\.weight=Q8_0 +blk\.6\.attn_gate\.weight=Q8_0 +blk\.6\.attn_qkv\.weight=Q8_0 +blk\.6\.ffn_down_exps\.weight=IQ3_S +blk\.6\.ffn_down_shexp\.weight=Q8_0 +blk\.6\.ffn_gate_exps\.weight=IQ3_S +blk\.6\.ffn_gate_shexp\.weight=Q8_0 +blk\.6\.ffn_up_exps\.weight=IQ3_S +blk\.6\.ffn_up_shexp\.weight=Q8_0 +blk\.6\.ssm_alpha\.weight=F32 +blk\.6\.ssm_beta\.weight=F32 +blk\.6\.ssm_out\.weight=Q8_0 +blk\.7\.attn_k\.weight=F16 +blk\.7\.attn_output\.weight=Q8_0 +blk\.7\.attn_q\.weight=Q8_0 +blk\.7\.attn_v\.weight=F16 +blk\.7\.ffn_down_exps\.weight=IQ4_XS +blk\.7\.ffn_down_shexp\.weight=Q8_0 +blk\.7\.ffn_gate_exps\.weight=IQ4_XS +blk\.7\.ffn_gate_shexp\.weight=Q8_0 +blk\.7\.ffn_up_exps\.weight=IQ4_XS +blk\.7\.ffn_up_shexp\.weight=Q8_0 +blk\.8\.attn_gate\.weight=Q8_0 +blk\.8\.attn_qkv\.weight=Q8_0 +blk\.8\.ffn_down_exps\.weight=IQ3_S +blk\.8\.ffn_down_shexp\.weight=Q8_0 +blk\.8\.ffn_gate_exps\.weight=IQ3_S +blk\.8\.ffn_gate_shexp\.weight=Q8_0 +blk\.8\.ffn_up_exps\.weight=IQ3_S +blk\.8\.ffn_up_shexp\.weight=Q8_0 +blk\.8\.ssm_alpha\.weight=F32 +blk\.8\.ssm_beta\.weight=F32 +blk\.8\.ssm_out\.weight=Q8_0 +blk\.9\.attn_gate\.weight=Q8_0 +blk\.9\.attn_qkv\.weight=Q8_0 +blk\.9\.ffn_down_exps\.weight=IQ4_XS +blk\.9\.ffn_down_shexp\.weight=Q8_0 +blk\.9\.ffn_gate_exps\.weight=IQ4_XS +blk\.9\.ffn_gate_shexp\.weight=Q8_0 +blk\.9\.ffn_up_exps\.weight=IQ4_XS +blk\.9\.ffn_up_shexp\.weight=Q8_0 +blk\.9\.ssm_alpha\.weight=F32 +blk\.9\.ssm_beta\.weight=F32 +blk\.9\.ssm_out\.weight=Q8_0 diff --git a/appendix/recipes/cfg_21G-ICE_ICEbase.txt b/appendix/recipes/cfg_21G-ICE_ICEbase.txt new file mode 100644 index 0000000000000000000000000000000000000000..64dae9ca078748d20fd9f48f426a9b17bd19aa99 --- /dev/null +++ b/appendix/recipes/cfg_21G-ICE_ICEbase.txt @@ -0,0 +1,443 @@ +^output\.weight$=Q8_0 +token_embd\.weight=Q8_0 +blk\.0\.attn_gate\.weight=Q8_0 +blk\.0\.attn_qkv\.weight=Q8_0 +blk\.0\.ffn_down_exps\.weight=Q4_K +blk\.0\.ffn_down_shexp\.weight=Q8_0 +blk\.0\.ffn_gate_exps\.weight=Q4_K +blk\.0\.ffn_gate_shexp\.weight=Q8_0 +blk\.0\.ffn_up_exps\.weight=Q4_K +blk\.0\.ffn_up_shexp\.weight=Q8_0 +blk\.0\.ssm_alpha\.weight=F32 +blk\.0\.ssm_beta\.weight=F32 +blk\.0\.ssm_out\.weight=Q8_0 +blk\.1\.attn_gate\.weight=Q8_0 +blk\.1\.attn_qkv\.weight=Q8_0 +blk\.1\.ffn_down_exps\.weight=IQ4_XS +blk\.1\.ffn_down_shexp\.weight=Q8_0 +blk\.1\.ffn_gate_exps\.weight=IQ4_XS +blk\.1\.ffn_gate_shexp\.weight=Q8_0 +blk\.1\.ffn_up_exps\.weight=IQ4_XS +blk\.1\.ffn_up_shexp\.weight=Q8_0 +blk\.1\.ssm_alpha\.weight=F32 +blk\.1\.ssm_beta\.weight=F32 +blk\.1\.ssm_out\.weight=Q8_0 +blk\.10\.attn_gate\.weight=Q8_0 +blk\.10\.attn_qkv\.weight=Q8_0 +blk\.10\.ffn_down_exps\.weight=IQ4_XS +blk\.10\.ffn_down_shexp\.weight=Q8_0 +blk\.10\.ffn_gate_exps\.weight=IQ4_XS +blk\.10\.ffn_gate_shexp\.weight=Q8_0 +blk\.10\.ffn_up_exps\.weight=IQ4_XS +blk\.10\.ffn_up_shexp\.weight=Q8_0 +blk\.10\.ssm_alpha\.weight=F32 +blk\.10\.ssm_beta\.weight=F32 +blk\.10\.ssm_out\.weight=Q8_0 +blk\.11\.attn_k\.weight=F16 +blk\.11\.attn_output\.weight=Q8_0 +blk\.11\.attn_q\.weight=Q8_0 +blk\.11\.attn_v\.weight=F16 +blk\.11\.ffn_down_exps\.weight=Q4_K +blk\.11\.ffn_down_shexp\.weight=Q8_0 +blk\.11\.ffn_gate_exps\.weight=Q4_K +blk\.11\.ffn_gate_shexp\.weight=Q8_0 +blk\.11\.ffn_up_exps\.weight=Q4_K +blk\.11\.ffn_up_shexp\.weight=Q8_0 +blk\.12\.attn_gate\.weight=Q8_0 +blk\.12\.attn_qkv\.weight=Q8_0 +blk\.12\.ffn_down_exps\.weight=IQ4_XS +blk\.12\.ffn_down_shexp\.weight=Q8_0 +blk\.12\.ffn_gate_exps\.weight=IQ4_XS +blk\.12\.ffn_gate_shexp\.weight=Q8_0 +blk\.12\.ffn_up_exps\.weight=IQ4_XS +blk\.12\.ffn_up_shexp\.weight=Q8_0 +blk\.12\.ssm_alpha\.weight=F32 +blk\.12\.ssm_beta\.weight=F32 +blk\.12\.ssm_out\.weight=Q8_0 +blk\.13\.attn_gate\.weight=Q8_0 +blk\.13\.attn_qkv\.weight=Q8_0 +blk\.13\.ffn_down_exps\.weight=Q4_K +blk\.13\.ffn_down_shexp\.weight=Q8_0 +blk\.13\.ffn_gate_exps\.weight=Q4_K +blk\.13\.ffn_gate_shexp\.weight=Q8_0 +blk\.13\.ffn_up_exps\.weight=Q4_K +blk\.13\.ffn_up_shexp\.weight=Q8_0 +blk\.13\.ssm_alpha\.weight=F32 +blk\.13\.ssm_beta\.weight=F32 +blk\.13\.ssm_out\.weight=Q8_0 +blk\.14\.attn_gate\.weight=Q8_0 +blk\.14\.attn_qkv\.weight=Q8_0 +blk\.14\.ffn_down_exps\.weight=IQ4_XS +blk\.14\.ffn_down_shexp\.weight=Q8_0 +blk\.14\.ffn_gate_exps\.weight=IQ4_XS +blk\.14\.ffn_gate_shexp\.weight=Q8_0 +blk\.14\.ffn_up_exps\.weight=IQ4_XS +blk\.14\.ffn_up_shexp\.weight=Q8_0 +blk\.14\.ssm_alpha\.weight=F32 +blk\.14\.ssm_beta\.weight=F32 +blk\.14\.ssm_out\.weight=Q8_0 +blk\.15\.attn_k\.weight=F16 +blk\.15\.attn_output\.weight=Q8_0 +blk\.15\.attn_q\.weight=Q8_0 +blk\.15\.attn_v\.weight=F16 +blk\.15\.ffn_down_exps\.weight=Q4_K +blk\.15\.ffn_down_shexp\.weight=Q8_0 +blk\.15\.ffn_gate_exps\.weight=Q4_K +blk\.15\.ffn_gate_shexp\.weight=Q8_0 +blk\.15\.ffn_up_exps\.weight=Q4_K +blk\.15\.ffn_up_shexp\.weight=Q8_0 +blk\.16\.attn_gate\.weight=Q8_0 +blk\.16\.attn_qkv\.weight=Q8_0 +blk\.16\.ffn_down_exps\.weight=IQ4_XS +blk\.16\.ffn_down_shexp\.weight=Q8_0 +blk\.16\.ffn_gate_exps\.weight=IQ4_XS +blk\.16\.ffn_gate_shexp\.weight=Q8_0 +blk\.16\.ffn_up_exps\.weight=IQ4_XS +blk\.16\.ffn_up_shexp\.weight=Q8_0 +blk\.16\.ssm_alpha\.weight=F32 +blk\.16\.ssm_beta\.weight=F32 +blk\.16\.ssm_out\.weight=Q8_0 +blk\.17\.attn_gate\.weight=Q8_0 +blk\.17\.attn_qkv\.weight=Q8_0 +blk\.17\.ffn_down_exps\.weight=Q4_K +blk\.17\.ffn_down_shexp\.weight=Q8_0 +blk\.17\.ffn_gate_exps\.weight=Q4_K +blk\.17\.ffn_gate_shexp\.weight=Q8_0 +blk\.17\.ffn_up_exps\.weight=Q4_K +blk\.17\.ffn_up_shexp\.weight=Q8_0 +blk\.17\.ssm_alpha\.weight=F32 +blk\.17\.ssm_beta\.weight=F32 +blk\.17\.ssm_out\.weight=Q8_0 +blk\.18\.attn_gate\.weight=Q8_0 +blk\.18\.attn_qkv\.weight=Q8_0 +blk\.18\.ffn_down_exps\.weight=IQ4_XS +blk\.18\.ffn_down_shexp\.weight=Q8_0 +blk\.18\.ffn_gate_exps\.weight=IQ4_XS +blk\.18\.ffn_gate_shexp\.weight=Q8_0 +blk\.18\.ffn_up_exps\.weight=IQ4_XS +blk\.18\.ffn_up_shexp\.weight=Q8_0 +blk\.18\.ssm_alpha\.weight=F32 +blk\.18\.ssm_beta\.weight=F32 +blk\.18\.ssm_out\.weight=Q8_0 +blk\.19\.attn_k\.weight=F16 +blk\.19\.attn_output\.weight=Q8_0 +blk\.19\.attn_q\.weight=Q8_0 +blk\.19\.attn_v\.weight=F16 +blk\.19\.ffn_down_exps\.weight=Q4_K +blk\.19\.ffn_down_shexp\.weight=Q8_0 +blk\.19\.ffn_gate_exps\.weight=Q4_K +blk\.19\.ffn_gate_shexp\.weight=Q8_0 +blk\.19\.ffn_up_exps\.weight=Q4_K +blk\.19\.ffn_up_shexp\.weight=Q8_0 +blk\.2\.attn_gate\.weight=Q8_0 +blk\.2\.attn_qkv\.weight=Q8_0 +blk\.2\.ffn_down_exps\.weight=Q4_K +blk\.2\.ffn_down_shexp\.weight=Q8_0 +blk\.2\.ffn_gate_exps\.weight=Q4_K +blk\.2\.ffn_gate_shexp\.weight=Q8_0 +blk\.2\.ffn_up_exps\.weight=Q4_K +blk\.2\.ffn_up_shexp\.weight=Q8_0 +blk\.2\.ssm_alpha\.weight=F32 +blk\.2\.ssm_beta\.weight=F32 +blk\.2\.ssm_out\.weight=Q8_0 +blk\.20\.attn_gate\.weight=Q8_0 +blk\.20\.attn_qkv\.weight=Q8_0 +blk\.20\.ffn_down_exps\.weight=Q4_K +blk\.20\.ffn_down_shexp\.weight=Q8_0 +blk\.20\.ffn_gate_exps\.weight=Q4_K +blk\.20\.ffn_gate_shexp\.weight=Q8_0 +blk\.20\.ffn_up_exps\.weight=Q4_K +blk\.20\.ffn_up_shexp\.weight=Q8_0 +blk\.20\.ssm_alpha\.weight=F32 +blk\.20\.ssm_beta\.weight=F32 +blk\.20\.ssm_out\.weight=Q8_0 +blk\.21\.attn_gate\.weight=Q8_0 +blk\.21\.attn_qkv\.weight=Q8_0 +blk\.21\.ffn_down_exps\.weight=IQ4_XS +blk\.21\.ffn_down_shexp\.weight=Q8_0 +blk\.21\.ffn_gate_exps\.weight=IQ4_XS +blk\.21\.ffn_gate_shexp\.weight=Q8_0 +blk\.21\.ffn_up_exps\.weight=IQ4_XS +blk\.21\.ffn_up_shexp\.weight=Q8_0 +blk\.21\.ssm_alpha\.weight=F32 +blk\.21\.ssm_beta\.weight=F32 +blk\.21\.ssm_out\.weight=Q8_0 +blk\.22\.attn_gate\.weight=Q8_0 +blk\.22\.attn_qkv\.weight=Q8_0 +blk\.22\.ffn_down_exps\.weight=Q4_K +blk\.22\.ffn_down_shexp\.weight=Q8_0 +blk\.22\.ffn_gate_exps\.weight=Q4_K +blk\.22\.ffn_gate_shexp\.weight=Q8_0 +blk\.22\.ffn_up_exps\.weight=Q4_K +blk\.22\.ffn_up_shexp\.weight=Q8_0 +blk\.22\.ssm_alpha\.weight=F32 +blk\.22\.ssm_beta\.weight=F32 +blk\.22\.ssm_out\.weight=Q8_0 +blk\.23\.attn_k\.weight=F16 +blk\.23\.attn_output\.weight=Q8_0 +blk\.23\.attn_q\.weight=Q8_0 +blk\.23\.attn_v\.weight=F16 +blk\.23\.ffn_down_exps\.weight=IQ4_XS +blk\.23\.ffn_down_shexp\.weight=Q8_0 +blk\.23\.ffn_gate_exps\.weight=IQ4_XS +blk\.23\.ffn_gate_shexp\.weight=Q8_0 +blk\.23\.ffn_up_exps\.weight=IQ4_XS +blk\.23\.ffn_up_shexp\.weight=Q8_0 +blk\.24\.attn_gate\.weight=Q8_0 +blk\.24\.attn_qkv\.weight=Q8_0 +blk\.24\.ffn_down_exps\.weight=Q4_K +blk\.24\.ffn_down_shexp\.weight=Q8_0 +blk\.24\.ffn_gate_exps\.weight=Q4_K +blk\.24\.ffn_gate_shexp\.weight=Q8_0 +blk\.24\.ffn_up_exps\.weight=Q4_K +blk\.24\.ffn_up_shexp\.weight=Q8_0 +blk\.24\.ssm_alpha\.weight=F32 +blk\.24\.ssm_beta\.weight=F32 +blk\.24\.ssm_out\.weight=Q8_0 +blk\.25\.attn_gate\.weight=Q8_0 +blk\.25\.attn_qkv\.weight=Q8_0 +blk\.25\.ffn_down_exps\.weight=IQ4_XS +blk\.25\.ffn_down_shexp\.weight=Q8_0 +blk\.25\.ffn_gate_exps\.weight=IQ4_XS +blk\.25\.ffn_gate_shexp\.weight=Q8_0 +blk\.25\.ffn_up_exps\.weight=IQ4_XS +blk\.25\.ffn_up_shexp\.weight=Q8_0 +blk\.25\.ssm_alpha\.weight=F32 +blk\.25\.ssm_beta\.weight=F32 +blk\.25\.ssm_out\.weight=Q8_0 +blk\.26\.attn_gate\.weight=Q8_0 +blk\.26\.attn_qkv\.weight=Q8_0 +blk\.26\.ffn_down_exps\.weight=Q4_K +blk\.26\.ffn_down_shexp\.weight=Q8_0 +blk\.26\.ffn_gate_exps\.weight=Q4_K +blk\.26\.ffn_gate_shexp\.weight=Q8_0 +blk\.26\.ffn_up_exps\.weight=Q4_K +blk\.26\.ffn_up_shexp\.weight=Q8_0 +blk\.26\.ssm_alpha\.weight=F32 +blk\.26\.ssm_beta\.weight=F32 +blk\.26\.ssm_out\.weight=Q8_0 +blk\.27\.attn_k\.weight=F16 +blk\.27\.attn_output\.weight=Q8_0 +blk\.27\.attn_q\.weight=Q8_0 +blk\.27\.attn_v\.weight=F16 +blk\.27\.ffn_down_exps\.weight=IQ4_XS +blk\.27\.ffn_down_shexp\.weight=Q8_0 +blk\.27\.ffn_gate_exps\.weight=IQ4_XS +blk\.27\.ffn_gate_shexp\.weight=Q8_0 +blk\.27\.ffn_up_exps\.weight=IQ4_XS +blk\.27\.ffn_up_shexp\.weight=Q8_0 +blk\.28\.attn_gate\.weight=Q8_0 +blk\.28\.attn_qkv\.weight=Q8_0 +blk\.28\.ffn_down_exps\.weight=Q4_K +blk\.28\.ffn_down_shexp\.weight=Q8_0 +blk\.28\.ffn_gate_exps\.weight=Q4_K +blk\.28\.ffn_gate_shexp\.weight=Q8_0 +blk\.28\.ffn_up_exps\.weight=Q4_K +blk\.28\.ffn_up_shexp\.weight=Q8_0 +blk\.28\.ssm_alpha\.weight=F32 +blk\.28\.ssm_beta\.weight=F32 +blk\.28\.ssm_out\.weight=Q8_0 +blk\.29\.attn_gate\.weight=Q8_0 +blk\.29\.attn_qkv\.weight=Q8_0 +blk\.29\.ffn_down_exps\.weight=IQ4_XS +blk\.29\.ffn_down_shexp\.weight=Q8_0 +blk\.29\.ffn_gate_exps\.weight=IQ4_XS +blk\.29\.ffn_gate_shexp\.weight=Q8_0 +blk\.29\.ffn_up_exps\.weight=IQ4_XS +blk\.29\.ffn_up_shexp\.weight=Q8_0 +blk\.29\.ssm_alpha\.weight=F32 +blk\.29\.ssm_beta\.weight=F32 +blk\.29\.ssm_out\.weight=Q8_0 +blk\.3\.attn_k\.weight=F16 +blk\.3\.attn_output\.weight=Q8_0 +blk\.3\.attn_q\.weight=Q8_0 +blk\.3\.attn_v\.weight=F16 +blk\.3\.ffn_down_exps\.weight=IQ4_XS +blk\.3\.ffn_down_shexp\.weight=Q8_0 +blk\.3\.ffn_gate_exps\.weight=IQ4_XS +blk\.3\.ffn_gate_shexp\.weight=Q8_0 +blk\.3\.ffn_up_exps\.weight=IQ4_XS +blk\.3\.ffn_up_shexp\.weight=Q8_0 +blk\.30\.attn_gate\.weight=Q8_0 +blk\.30\.attn_qkv\.weight=Q8_0 +blk\.30\.ffn_down_exps\.weight=Q4_K +blk\.30\.ffn_down_shexp\.weight=Q8_0 +blk\.30\.ffn_gate_exps\.weight=Q4_K +blk\.30\.ffn_gate_shexp\.weight=Q8_0 +blk\.30\.ffn_up_exps\.weight=Q4_K +blk\.30\.ffn_up_shexp\.weight=Q8_0 +blk\.30\.ssm_alpha\.weight=F32 +blk\.30\.ssm_beta\.weight=F32 +blk\.30\.ssm_out\.weight=Q8_0 +blk\.31\.attn_k\.weight=F16 +blk\.31\.attn_output\.weight=Q8_0 +blk\.31\.attn_q\.weight=Q8_0 +blk\.31\.attn_v\.weight=F16 +blk\.31\.ffn_down_exps\.weight=IQ4_XS +blk\.31\.ffn_down_shexp\.weight=Q8_0 +blk\.31\.ffn_gate_exps\.weight=IQ4_XS +blk\.31\.ffn_gate_shexp\.weight=Q8_0 +blk\.31\.ffn_up_exps\.weight=IQ4_XS +blk\.31\.ffn_up_shexp\.weight=Q8_0 +blk\.32\.attn_gate\.weight=Q8_0 +blk\.32\.attn_qkv\.weight=Q8_0 +blk\.32\.ffn_down_exps\.weight=Q4_K +blk\.32\.ffn_down_shexp\.weight=Q8_0 +blk\.32\.ffn_gate_exps\.weight=Q4_K +blk\.32\.ffn_gate_shexp\.weight=Q8_0 +blk\.32\.ffn_up_exps\.weight=Q4_K +blk\.32\.ffn_up_shexp\.weight=Q8_0 +blk\.32\.ssm_alpha\.weight=F32 +blk\.32\.ssm_beta\.weight=F32 +blk\.32\.ssm_out\.weight=Q8_0 +blk\.33\.attn_gate\.weight=Q8_0 +blk\.33\.attn_qkv\.weight=Q8_0 +blk\.33\.ffn_down_exps\.weight=IQ4_XS +blk\.33\.ffn_down_shexp\.weight=Q8_0 +blk\.33\.ffn_gate_exps\.weight=IQ4_XS +blk\.33\.ffn_gate_shexp\.weight=Q8_0 +blk\.33\.ffn_up_exps\.weight=IQ4_XS +blk\.33\.ffn_up_shexp\.weight=Q8_0 +blk\.33\.ssm_alpha\.weight=F32 +blk\.33\.ssm_beta\.weight=F32 +blk\.33\.ssm_out\.weight=Q8_0 +blk\.34\.attn_gate\.weight=Q8_0 +blk\.34\.attn_qkv\.weight=Q8_0 +blk\.34\.ffn_down_exps\.weight=Q4_K +blk\.34\.ffn_down_shexp\.weight=Q8_0 +blk\.34\.ffn_gate_exps\.weight=Q4_K +blk\.34\.ffn_gate_shexp\.weight=Q8_0 +blk\.34\.ffn_up_exps\.weight=Q4_K +blk\.34\.ffn_up_shexp\.weight=Q8_0 +blk\.34\.ssm_alpha\.weight=F32 +blk\.34\.ssm_beta\.weight=F32 +blk\.34\.ssm_out\.weight=Q8_0 +blk\.35\.attn_k\.weight=F16 +blk\.35\.attn_output\.weight=Q8_0 +blk\.35\.attn_q\.weight=Q8_0 +blk\.35\.attn_v\.weight=F16 +blk\.35\.ffn_down_exps\.weight=Q4_K +blk\.35\.ffn_down_shexp\.weight=Q8_0 +blk\.35\.ffn_gate_exps\.weight=Q4_K +blk\.35\.ffn_gate_shexp\.weight=Q8_0 +blk\.35\.ffn_up_exps\.weight=Q4_K +blk\.35\.ffn_up_shexp\.weight=Q8_0 +blk\.36\.attn_gate\.weight=Q8_0 +blk\.36\.attn_qkv\.weight=Q8_0 +blk\.36\.ffn_down_exps\.weight=IQ4_XS +blk\.36\.ffn_down_shexp\.weight=Q8_0 +blk\.36\.ffn_gate_exps\.weight=IQ4_XS +blk\.36\.ffn_gate_shexp\.weight=Q8_0 +blk\.36\.ffn_up_exps\.weight=IQ4_XS +blk\.36\.ffn_up_shexp\.weight=Q8_0 +blk\.36\.ssm_alpha\.weight=F32 +blk\.36\.ssm_beta\.weight=F32 +blk\.36\.ssm_out\.weight=Q8_0 +blk\.37\.attn_gate\.weight=Q8_0 +blk\.37\.attn_qkv\.weight=Q8_0 +blk\.37\.ffn_down_exps\.weight=Q4_K +blk\.37\.ffn_down_shexp\.weight=Q8_0 +blk\.37\.ffn_gate_exps\.weight=Q4_K +blk\.37\.ffn_gate_shexp\.weight=Q8_0 +blk\.37\.ffn_up_exps\.weight=Q4_K +blk\.37\.ffn_up_shexp\.weight=Q8_0 +blk\.37\.ssm_alpha\.weight=F32 +blk\.37\.ssm_beta\.weight=F32 +blk\.37\.ssm_out\.weight=Q8_0 +blk\.38\.attn_gate\.weight=Q8_0 +blk\.38\.attn_qkv\.weight=Q8_0 +blk\.38\.ffn_down_exps\.weight=IQ4_XS +blk\.38\.ffn_down_shexp\.weight=Q8_0 +blk\.38\.ffn_gate_exps\.weight=IQ4_XS +blk\.38\.ffn_gate_shexp\.weight=Q8_0 +blk\.38\.ffn_up_exps\.weight=IQ4_XS +blk\.38\.ffn_up_shexp\.weight=Q8_0 +blk\.38\.ssm_alpha\.weight=F32 +blk\.38\.ssm_beta\.weight=F32 +blk\.38\.ssm_out\.weight=Q8_0 +blk\.39\.attn_k\.weight=F16 +blk\.39\.attn_output\.weight=Q8_0 +blk\.39\.attn_q\.weight=Q8_0 +blk\.39\.attn_v\.weight=F16 +blk\.39\.ffn_down_exps\.weight=Q4_K +blk\.39\.ffn_down_shexp\.weight=Q8_0 +blk\.39\.ffn_gate_exps\.weight=Q4_K +blk\.39\.ffn_gate_shexp\.weight=Q8_0 +blk\.39\.ffn_up_exps\.weight=Q4_K +blk\.39\.ffn_up_shexp\.weight=Q8_0 +blk\.4\.attn_gate\.weight=Q8_0 +blk\.4\.attn_qkv\.weight=Q8_0 +blk\.4\.ffn_down_exps\.weight=Q4_K +blk\.4\.ffn_down_shexp\.weight=Q8_0 +blk\.4\.ffn_gate_exps\.weight=Q4_K +blk\.4\.ffn_gate_shexp\.weight=Q8_0 +blk\.4\.ffn_up_exps\.weight=Q4_K +blk\.4\.ffn_up_shexp\.weight=Q8_0 +blk\.4\.ssm_alpha\.weight=F32 +blk\.4\.ssm_beta\.weight=F32 +blk\.4\.ssm_out\.weight=Q8_0 +blk\.40\.attn_k\.weight=F16 +blk\.40\.attn_output\.weight=Q8_0 +blk\.40\.attn_q\.weight=Q8_0 +blk\.40\.attn_v\.weight=F16 +blk\.40\.ffn_down_exps\.weight=IQ4_XS +blk\.40\.ffn_down_shexp\.weight=Q8_0 +blk\.40\.ffn_gate_exps\.weight=IQ4_XS +blk\.40\.ffn_gate_shexp\.weight=Q8_0 +blk\.40\.ffn_up_exps\.weight=IQ4_XS +blk\.40\.ffn_up_shexp\.weight=Q8_0 +blk\.40\.nextn\.eh_proj\.weight=Q8_0 +blk\.5\.attn_gate\.weight=Q8_0 +blk\.5\.attn_qkv\.weight=Q8_0 +blk\.5\.ffn_down_exps\.weight=IQ4_XS +blk\.5\.ffn_down_shexp\.weight=Q8_0 +blk\.5\.ffn_gate_exps\.weight=IQ4_XS +blk\.5\.ffn_gate_shexp\.weight=Q8_0 +blk\.5\.ffn_up_exps\.weight=IQ4_XS +blk\.5\.ffn_up_shexp\.weight=Q8_0 +blk\.5\.ssm_alpha\.weight=F32 +blk\.5\.ssm_beta\.weight=F32 +blk\.5\.ssm_out\.weight=Q8_0 +blk\.6\.attn_gate\.weight=Q8_0 +blk\.6\.attn_qkv\.weight=Q8_0 +blk\.6\.ffn_down_exps\.weight=Q4_K +blk\.6\.ffn_down_shexp\.weight=Q8_0 +blk\.6\.ffn_gate_exps\.weight=Q4_K +blk\.6\.ffn_gate_shexp\.weight=Q8_0 +blk\.6\.ffn_up_exps\.weight=Q4_K +blk\.6\.ffn_up_shexp\.weight=Q8_0 +blk\.6\.ssm_alpha\.weight=F32 +blk\.6\.ssm_beta\.weight=F32 +blk\.6\.ssm_out\.weight=Q8_0 +blk\.7\.attn_k\.weight=F16 +blk\.7\.attn_output\.weight=Q8_0 +blk\.7\.attn_q\.weight=Q8_0 +blk\.7\.attn_v\.weight=F16 +blk\.7\.ffn_down_exps\.weight=Q4_K +blk\.7\.ffn_down_shexp\.weight=Q8_0 +blk\.7\.ffn_gate_exps\.weight=Q4_K +blk\.7\.ffn_gate_shexp\.weight=Q8_0 +blk\.7\.ffn_up_exps\.weight=Q4_K +blk\.7\.ffn_up_shexp\.weight=Q8_0 +blk\.8\.attn_gate\.weight=Q8_0 +blk\.8\.attn_qkv\.weight=Q8_0 +blk\.8\.ffn_down_exps\.weight=IQ4_XS +blk\.8\.ffn_down_shexp\.weight=Q8_0 +blk\.8\.ffn_gate_exps\.weight=IQ4_XS +blk\.8\.ffn_gate_shexp\.weight=Q8_0 +blk\.8\.ffn_up_exps\.weight=IQ4_XS +blk\.8\.ffn_up_shexp\.weight=Q8_0 +blk\.8\.ssm_alpha\.weight=F32 +blk\.8\.ssm_beta\.weight=F32 +blk\.8\.ssm_out\.weight=Q8_0 +blk\.9\.attn_gate\.weight=Q8_0 +blk\.9\.attn_qkv\.weight=Q8_0 +blk\.9\.ffn_down_exps\.weight=Q4_K +blk\.9\.ffn_down_shexp\.weight=Q8_0 +blk\.9\.ffn_gate_exps\.weight=Q4_K +blk\.9\.ffn_gate_shexp\.weight=Q8_0 +blk\.9\.ffn_up_exps\.weight=Q4_K +blk\.9\.ffn_up_shexp\.weight=Q8_0 +blk\.9\.ssm_alpha\.weight=F32 +blk\.9\.ssm_beta\.weight=F32 +blk\.9\.ssm_out\.weight=Q8_0 diff --git a/appendix/recipes/cfg_21G-ICE_previous.txt b/appendix/recipes/cfg_21G-ICE_previous.txt new file mode 100644 index 0000000000000000000000000000000000000000..64dae9ca078748d20fd9f48f426a9b17bd19aa99 --- /dev/null +++ b/appendix/recipes/cfg_21G-ICE_previous.txt @@ -0,0 +1,443 @@ +^output\.weight$=Q8_0 +token_embd\.weight=Q8_0 +blk\.0\.attn_gate\.weight=Q8_0 +blk\.0\.attn_qkv\.weight=Q8_0 +blk\.0\.ffn_down_exps\.weight=Q4_K +blk\.0\.ffn_down_shexp\.weight=Q8_0 +blk\.0\.ffn_gate_exps\.weight=Q4_K +blk\.0\.ffn_gate_shexp\.weight=Q8_0 +blk\.0\.ffn_up_exps\.weight=Q4_K +blk\.0\.ffn_up_shexp\.weight=Q8_0 +blk\.0\.ssm_alpha\.weight=F32 +blk\.0\.ssm_beta\.weight=F32 +blk\.0\.ssm_out\.weight=Q8_0 +blk\.1\.attn_gate\.weight=Q8_0 +blk\.1\.attn_qkv\.weight=Q8_0 +blk\.1\.ffn_down_exps\.weight=IQ4_XS +blk\.1\.ffn_down_shexp\.weight=Q8_0 +blk\.1\.ffn_gate_exps\.weight=IQ4_XS +blk\.1\.ffn_gate_shexp\.weight=Q8_0 +blk\.1\.ffn_up_exps\.weight=IQ4_XS +blk\.1\.ffn_up_shexp\.weight=Q8_0 +blk\.1\.ssm_alpha\.weight=F32 +blk\.1\.ssm_beta\.weight=F32 +blk\.1\.ssm_out\.weight=Q8_0 +blk\.10\.attn_gate\.weight=Q8_0 +blk\.10\.attn_qkv\.weight=Q8_0 +blk\.10\.ffn_down_exps\.weight=IQ4_XS +blk\.10\.ffn_down_shexp\.weight=Q8_0 +blk\.10\.ffn_gate_exps\.weight=IQ4_XS +blk\.10\.ffn_gate_shexp\.weight=Q8_0 +blk\.10\.ffn_up_exps\.weight=IQ4_XS +blk\.10\.ffn_up_shexp\.weight=Q8_0 +blk\.10\.ssm_alpha\.weight=F32 +blk\.10\.ssm_beta\.weight=F32 +blk\.10\.ssm_out\.weight=Q8_0 +blk\.11\.attn_k\.weight=F16 +blk\.11\.attn_output\.weight=Q8_0 +blk\.11\.attn_q\.weight=Q8_0 +blk\.11\.attn_v\.weight=F16 +blk\.11\.ffn_down_exps\.weight=Q4_K +blk\.11\.ffn_down_shexp\.weight=Q8_0 +blk\.11\.ffn_gate_exps\.weight=Q4_K +blk\.11\.ffn_gate_shexp\.weight=Q8_0 +blk\.11\.ffn_up_exps\.weight=Q4_K +blk\.11\.ffn_up_shexp\.weight=Q8_0 +blk\.12\.attn_gate\.weight=Q8_0 +blk\.12\.attn_qkv\.weight=Q8_0 +blk\.12\.ffn_down_exps\.weight=IQ4_XS +blk\.12\.ffn_down_shexp\.weight=Q8_0 +blk\.12\.ffn_gate_exps\.weight=IQ4_XS +blk\.12\.ffn_gate_shexp\.weight=Q8_0 +blk\.12\.ffn_up_exps\.weight=IQ4_XS +blk\.12\.ffn_up_shexp\.weight=Q8_0 +blk\.12\.ssm_alpha\.weight=F32 +blk\.12\.ssm_beta\.weight=F32 +blk\.12\.ssm_out\.weight=Q8_0 +blk\.13\.attn_gate\.weight=Q8_0 +blk\.13\.attn_qkv\.weight=Q8_0 +blk\.13\.ffn_down_exps\.weight=Q4_K +blk\.13\.ffn_down_shexp\.weight=Q8_0 +blk\.13\.ffn_gate_exps\.weight=Q4_K +blk\.13\.ffn_gate_shexp\.weight=Q8_0 +blk\.13\.ffn_up_exps\.weight=Q4_K +blk\.13\.ffn_up_shexp\.weight=Q8_0 +blk\.13\.ssm_alpha\.weight=F32 +blk\.13\.ssm_beta\.weight=F32 +blk\.13\.ssm_out\.weight=Q8_0 +blk\.14\.attn_gate\.weight=Q8_0 +blk\.14\.attn_qkv\.weight=Q8_0 +blk\.14\.ffn_down_exps\.weight=IQ4_XS +blk\.14\.ffn_down_shexp\.weight=Q8_0 +blk\.14\.ffn_gate_exps\.weight=IQ4_XS +blk\.14\.ffn_gate_shexp\.weight=Q8_0 +blk\.14\.ffn_up_exps\.weight=IQ4_XS +blk\.14\.ffn_up_shexp\.weight=Q8_0 +blk\.14\.ssm_alpha\.weight=F32 +blk\.14\.ssm_beta\.weight=F32 +blk\.14\.ssm_out\.weight=Q8_0 +blk\.15\.attn_k\.weight=F16 +blk\.15\.attn_output\.weight=Q8_0 +blk\.15\.attn_q\.weight=Q8_0 +blk\.15\.attn_v\.weight=F16 +blk\.15\.ffn_down_exps\.weight=Q4_K +blk\.15\.ffn_down_shexp\.weight=Q8_0 +blk\.15\.ffn_gate_exps\.weight=Q4_K +blk\.15\.ffn_gate_shexp\.weight=Q8_0 +blk\.15\.ffn_up_exps\.weight=Q4_K +blk\.15\.ffn_up_shexp\.weight=Q8_0 +blk\.16\.attn_gate\.weight=Q8_0 +blk\.16\.attn_qkv\.weight=Q8_0 +blk\.16\.ffn_down_exps\.weight=IQ4_XS +blk\.16\.ffn_down_shexp\.weight=Q8_0 +blk\.16\.ffn_gate_exps\.weight=IQ4_XS +blk\.16\.ffn_gate_shexp\.weight=Q8_0 +blk\.16\.ffn_up_exps\.weight=IQ4_XS +blk\.16\.ffn_up_shexp\.weight=Q8_0 +blk\.16\.ssm_alpha\.weight=F32 +blk\.16\.ssm_beta\.weight=F32 +blk\.16\.ssm_out\.weight=Q8_0 +blk\.17\.attn_gate\.weight=Q8_0 +blk\.17\.attn_qkv\.weight=Q8_0 +blk\.17\.ffn_down_exps\.weight=Q4_K +blk\.17\.ffn_down_shexp\.weight=Q8_0 +blk\.17\.ffn_gate_exps\.weight=Q4_K +blk\.17\.ffn_gate_shexp\.weight=Q8_0 +blk\.17\.ffn_up_exps\.weight=Q4_K +blk\.17\.ffn_up_shexp\.weight=Q8_0 +blk\.17\.ssm_alpha\.weight=F32 +blk\.17\.ssm_beta\.weight=F32 +blk\.17\.ssm_out\.weight=Q8_0 +blk\.18\.attn_gate\.weight=Q8_0 +blk\.18\.attn_qkv\.weight=Q8_0 +blk\.18\.ffn_down_exps\.weight=IQ4_XS +blk\.18\.ffn_down_shexp\.weight=Q8_0 +blk\.18\.ffn_gate_exps\.weight=IQ4_XS +blk\.18\.ffn_gate_shexp\.weight=Q8_0 +blk\.18\.ffn_up_exps\.weight=IQ4_XS +blk\.18\.ffn_up_shexp\.weight=Q8_0 +blk\.18\.ssm_alpha\.weight=F32 +blk\.18\.ssm_beta\.weight=F32 +blk\.18\.ssm_out\.weight=Q8_0 +blk\.19\.attn_k\.weight=F16 +blk\.19\.attn_output\.weight=Q8_0 +blk\.19\.attn_q\.weight=Q8_0 +blk\.19\.attn_v\.weight=F16 +blk\.19\.ffn_down_exps\.weight=Q4_K +blk\.19\.ffn_down_shexp\.weight=Q8_0 +blk\.19\.ffn_gate_exps\.weight=Q4_K +blk\.19\.ffn_gate_shexp\.weight=Q8_0 +blk\.19\.ffn_up_exps\.weight=Q4_K +blk\.19\.ffn_up_shexp\.weight=Q8_0 +blk\.2\.attn_gate\.weight=Q8_0 +blk\.2\.attn_qkv\.weight=Q8_0 +blk\.2\.ffn_down_exps\.weight=Q4_K +blk\.2\.ffn_down_shexp\.weight=Q8_0 +blk\.2\.ffn_gate_exps\.weight=Q4_K +blk\.2\.ffn_gate_shexp\.weight=Q8_0 +blk\.2\.ffn_up_exps\.weight=Q4_K +blk\.2\.ffn_up_shexp\.weight=Q8_0 +blk\.2\.ssm_alpha\.weight=F32 +blk\.2\.ssm_beta\.weight=F32 +blk\.2\.ssm_out\.weight=Q8_0 +blk\.20\.attn_gate\.weight=Q8_0 +blk\.20\.attn_qkv\.weight=Q8_0 +blk\.20\.ffn_down_exps\.weight=Q4_K +blk\.20\.ffn_down_shexp\.weight=Q8_0 +blk\.20\.ffn_gate_exps\.weight=Q4_K +blk\.20\.ffn_gate_shexp\.weight=Q8_0 +blk\.20\.ffn_up_exps\.weight=Q4_K +blk\.20\.ffn_up_shexp\.weight=Q8_0 +blk\.20\.ssm_alpha\.weight=F32 +blk\.20\.ssm_beta\.weight=F32 +blk\.20\.ssm_out\.weight=Q8_0 +blk\.21\.attn_gate\.weight=Q8_0 +blk\.21\.attn_qkv\.weight=Q8_0 +blk\.21\.ffn_down_exps\.weight=IQ4_XS +blk\.21\.ffn_down_shexp\.weight=Q8_0 +blk\.21\.ffn_gate_exps\.weight=IQ4_XS +blk\.21\.ffn_gate_shexp\.weight=Q8_0 +blk\.21\.ffn_up_exps\.weight=IQ4_XS +blk\.21\.ffn_up_shexp\.weight=Q8_0 +blk\.21\.ssm_alpha\.weight=F32 +blk\.21\.ssm_beta\.weight=F32 +blk\.21\.ssm_out\.weight=Q8_0 +blk\.22\.attn_gate\.weight=Q8_0 +blk\.22\.attn_qkv\.weight=Q8_0 +blk\.22\.ffn_down_exps\.weight=Q4_K +blk\.22\.ffn_down_shexp\.weight=Q8_0 +blk\.22\.ffn_gate_exps\.weight=Q4_K +blk\.22\.ffn_gate_shexp\.weight=Q8_0 +blk\.22\.ffn_up_exps\.weight=Q4_K +blk\.22\.ffn_up_shexp\.weight=Q8_0 +blk\.22\.ssm_alpha\.weight=F32 +blk\.22\.ssm_beta\.weight=F32 +blk\.22\.ssm_out\.weight=Q8_0 +blk\.23\.attn_k\.weight=F16 +blk\.23\.attn_output\.weight=Q8_0 +blk\.23\.attn_q\.weight=Q8_0 +blk\.23\.attn_v\.weight=F16 +blk\.23\.ffn_down_exps\.weight=IQ4_XS +blk\.23\.ffn_down_shexp\.weight=Q8_0 +blk\.23\.ffn_gate_exps\.weight=IQ4_XS +blk\.23\.ffn_gate_shexp\.weight=Q8_0 +blk\.23\.ffn_up_exps\.weight=IQ4_XS +blk\.23\.ffn_up_shexp\.weight=Q8_0 +blk\.24\.attn_gate\.weight=Q8_0 +blk\.24\.attn_qkv\.weight=Q8_0 +blk\.24\.ffn_down_exps\.weight=Q4_K +blk\.24\.ffn_down_shexp\.weight=Q8_0 +blk\.24\.ffn_gate_exps\.weight=Q4_K +blk\.24\.ffn_gate_shexp\.weight=Q8_0 +blk\.24\.ffn_up_exps\.weight=Q4_K +blk\.24\.ffn_up_shexp\.weight=Q8_0 +blk\.24\.ssm_alpha\.weight=F32 +blk\.24\.ssm_beta\.weight=F32 +blk\.24\.ssm_out\.weight=Q8_0 +blk\.25\.attn_gate\.weight=Q8_0 +blk\.25\.attn_qkv\.weight=Q8_0 +blk\.25\.ffn_down_exps\.weight=IQ4_XS +blk\.25\.ffn_down_shexp\.weight=Q8_0 +blk\.25\.ffn_gate_exps\.weight=IQ4_XS +blk\.25\.ffn_gate_shexp\.weight=Q8_0 +blk\.25\.ffn_up_exps\.weight=IQ4_XS +blk\.25\.ffn_up_shexp\.weight=Q8_0 +blk\.25\.ssm_alpha\.weight=F32 +blk\.25\.ssm_beta\.weight=F32 +blk\.25\.ssm_out\.weight=Q8_0 +blk\.26\.attn_gate\.weight=Q8_0 +blk\.26\.attn_qkv\.weight=Q8_0 +blk\.26\.ffn_down_exps\.weight=Q4_K +blk\.26\.ffn_down_shexp\.weight=Q8_0 +blk\.26\.ffn_gate_exps\.weight=Q4_K +blk\.26\.ffn_gate_shexp\.weight=Q8_0 +blk\.26\.ffn_up_exps\.weight=Q4_K +blk\.26\.ffn_up_shexp\.weight=Q8_0 +blk\.26\.ssm_alpha\.weight=F32 +blk\.26\.ssm_beta\.weight=F32 +blk\.26\.ssm_out\.weight=Q8_0 +blk\.27\.attn_k\.weight=F16 +blk\.27\.attn_output\.weight=Q8_0 +blk\.27\.attn_q\.weight=Q8_0 +blk\.27\.attn_v\.weight=F16 +blk\.27\.ffn_down_exps\.weight=IQ4_XS +blk\.27\.ffn_down_shexp\.weight=Q8_0 +blk\.27\.ffn_gate_exps\.weight=IQ4_XS +blk\.27\.ffn_gate_shexp\.weight=Q8_0 +blk\.27\.ffn_up_exps\.weight=IQ4_XS +blk\.27\.ffn_up_shexp\.weight=Q8_0 +blk\.28\.attn_gate\.weight=Q8_0 +blk\.28\.attn_qkv\.weight=Q8_0 +blk\.28\.ffn_down_exps\.weight=Q4_K +blk\.28\.ffn_down_shexp\.weight=Q8_0 +blk\.28\.ffn_gate_exps\.weight=Q4_K +blk\.28\.ffn_gate_shexp\.weight=Q8_0 +blk\.28\.ffn_up_exps\.weight=Q4_K +blk\.28\.ffn_up_shexp\.weight=Q8_0 +blk\.28\.ssm_alpha\.weight=F32 +blk\.28\.ssm_beta\.weight=F32 +blk\.28\.ssm_out\.weight=Q8_0 +blk\.29\.attn_gate\.weight=Q8_0 +blk\.29\.attn_qkv\.weight=Q8_0 +blk\.29\.ffn_down_exps\.weight=IQ4_XS +blk\.29\.ffn_down_shexp\.weight=Q8_0 +blk\.29\.ffn_gate_exps\.weight=IQ4_XS +blk\.29\.ffn_gate_shexp\.weight=Q8_0 +blk\.29\.ffn_up_exps\.weight=IQ4_XS +blk\.29\.ffn_up_shexp\.weight=Q8_0 +blk\.29\.ssm_alpha\.weight=F32 +blk\.29\.ssm_beta\.weight=F32 +blk\.29\.ssm_out\.weight=Q8_0 +blk\.3\.attn_k\.weight=F16 +blk\.3\.attn_output\.weight=Q8_0 +blk\.3\.attn_q\.weight=Q8_0 +blk\.3\.attn_v\.weight=F16 +blk\.3\.ffn_down_exps\.weight=IQ4_XS +blk\.3\.ffn_down_shexp\.weight=Q8_0 +blk\.3\.ffn_gate_exps\.weight=IQ4_XS +blk\.3\.ffn_gate_shexp\.weight=Q8_0 +blk\.3\.ffn_up_exps\.weight=IQ4_XS +blk\.3\.ffn_up_shexp\.weight=Q8_0 +blk\.30\.attn_gate\.weight=Q8_0 +blk\.30\.attn_qkv\.weight=Q8_0 +blk\.30\.ffn_down_exps\.weight=Q4_K +blk\.30\.ffn_down_shexp\.weight=Q8_0 +blk\.30\.ffn_gate_exps\.weight=Q4_K +blk\.30\.ffn_gate_shexp\.weight=Q8_0 +blk\.30\.ffn_up_exps\.weight=Q4_K +blk\.30\.ffn_up_shexp\.weight=Q8_0 +blk\.30\.ssm_alpha\.weight=F32 +blk\.30\.ssm_beta\.weight=F32 +blk\.30\.ssm_out\.weight=Q8_0 +blk\.31\.attn_k\.weight=F16 +blk\.31\.attn_output\.weight=Q8_0 +blk\.31\.attn_q\.weight=Q8_0 +blk\.31\.attn_v\.weight=F16 +blk\.31\.ffn_down_exps\.weight=IQ4_XS +blk\.31\.ffn_down_shexp\.weight=Q8_0 +blk\.31\.ffn_gate_exps\.weight=IQ4_XS +blk\.31\.ffn_gate_shexp\.weight=Q8_0 +blk\.31\.ffn_up_exps\.weight=IQ4_XS +blk\.31\.ffn_up_shexp\.weight=Q8_0 +blk\.32\.attn_gate\.weight=Q8_0 +blk\.32\.attn_qkv\.weight=Q8_0 +blk\.32\.ffn_down_exps\.weight=Q4_K +blk\.32\.ffn_down_shexp\.weight=Q8_0 +blk\.32\.ffn_gate_exps\.weight=Q4_K +blk\.32\.ffn_gate_shexp\.weight=Q8_0 +blk\.32\.ffn_up_exps\.weight=Q4_K +blk\.32\.ffn_up_shexp\.weight=Q8_0 +blk\.32\.ssm_alpha\.weight=F32 +blk\.32\.ssm_beta\.weight=F32 +blk\.32\.ssm_out\.weight=Q8_0 +blk\.33\.attn_gate\.weight=Q8_0 +blk\.33\.attn_qkv\.weight=Q8_0 +blk\.33\.ffn_down_exps\.weight=IQ4_XS +blk\.33\.ffn_down_shexp\.weight=Q8_0 +blk\.33\.ffn_gate_exps\.weight=IQ4_XS +blk\.33\.ffn_gate_shexp\.weight=Q8_0 +blk\.33\.ffn_up_exps\.weight=IQ4_XS +blk\.33\.ffn_up_shexp\.weight=Q8_0 +blk\.33\.ssm_alpha\.weight=F32 +blk\.33\.ssm_beta\.weight=F32 +blk\.33\.ssm_out\.weight=Q8_0 +blk\.34\.attn_gate\.weight=Q8_0 +blk\.34\.attn_qkv\.weight=Q8_0 +blk\.34\.ffn_down_exps\.weight=Q4_K +blk\.34\.ffn_down_shexp\.weight=Q8_0 +blk\.34\.ffn_gate_exps\.weight=Q4_K +blk\.34\.ffn_gate_shexp\.weight=Q8_0 +blk\.34\.ffn_up_exps\.weight=Q4_K +blk\.34\.ffn_up_shexp\.weight=Q8_0 +blk\.34\.ssm_alpha\.weight=F32 +blk\.34\.ssm_beta\.weight=F32 +blk\.34\.ssm_out\.weight=Q8_0 +blk\.35\.attn_k\.weight=F16 +blk\.35\.attn_output\.weight=Q8_0 +blk\.35\.attn_q\.weight=Q8_0 +blk\.35\.attn_v\.weight=F16 +blk\.35\.ffn_down_exps\.weight=Q4_K +blk\.35\.ffn_down_shexp\.weight=Q8_0 +blk\.35\.ffn_gate_exps\.weight=Q4_K +blk\.35\.ffn_gate_shexp\.weight=Q8_0 +blk\.35\.ffn_up_exps\.weight=Q4_K +blk\.35\.ffn_up_shexp\.weight=Q8_0 +blk\.36\.attn_gate\.weight=Q8_0 +blk\.36\.attn_qkv\.weight=Q8_0 +blk\.36\.ffn_down_exps\.weight=IQ4_XS +blk\.36\.ffn_down_shexp\.weight=Q8_0 +blk\.36\.ffn_gate_exps\.weight=IQ4_XS +blk\.36\.ffn_gate_shexp\.weight=Q8_0 +blk\.36\.ffn_up_exps\.weight=IQ4_XS +blk\.36\.ffn_up_shexp\.weight=Q8_0 +blk\.36\.ssm_alpha\.weight=F32 +blk\.36\.ssm_beta\.weight=F32 +blk\.36\.ssm_out\.weight=Q8_0 +blk\.37\.attn_gate\.weight=Q8_0 +blk\.37\.attn_qkv\.weight=Q8_0 +blk\.37\.ffn_down_exps\.weight=Q4_K +blk\.37\.ffn_down_shexp\.weight=Q8_0 +blk\.37\.ffn_gate_exps\.weight=Q4_K +blk\.37\.ffn_gate_shexp\.weight=Q8_0 +blk\.37\.ffn_up_exps\.weight=Q4_K +blk\.37\.ffn_up_shexp\.weight=Q8_0 +blk\.37\.ssm_alpha\.weight=F32 +blk\.37\.ssm_beta\.weight=F32 +blk\.37\.ssm_out\.weight=Q8_0 +blk\.38\.attn_gate\.weight=Q8_0 +blk\.38\.attn_qkv\.weight=Q8_0 +blk\.38\.ffn_down_exps\.weight=IQ4_XS +blk\.38\.ffn_down_shexp\.weight=Q8_0 +blk\.38\.ffn_gate_exps\.weight=IQ4_XS +blk\.38\.ffn_gate_shexp\.weight=Q8_0 +blk\.38\.ffn_up_exps\.weight=IQ4_XS +blk\.38\.ffn_up_shexp\.weight=Q8_0 +blk\.38\.ssm_alpha\.weight=F32 +blk\.38\.ssm_beta\.weight=F32 +blk\.38\.ssm_out\.weight=Q8_0 +blk\.39\.attn_k\.weight=F16 +blk\.39\.attn_output\.weight=Q8_0 +blk\.39\.attn_q\.weight=Q8_0 +blk\.39\.attn_v\.weight=F16 +blk\.39\.ffn_down_exps\.weight=Q4_K +blk\.39\.ffn_down_shexp\.weight=Q8_0 +blk\.39\.ffn_gate_exps\.weight=Q4_K +blk\.39\.ffn_gate_shexp\.weight=Q8_0 +blk\.39\.ffn_up_exps\.weight=Q4_K +blk\.39\.ffn_up_shexp\.weight=Q8_0 +blk\.4\.attn_gate\.weight=Q8_0 +blk\.4\.attn_qkv\.weight=Q8_0 +blk\.4\.ffn_down_exps\.weight=Q4_K +blk\.4\.ffn_down_shexp\.weight=Q8_0 +blk\.4\.ffn_gate_exps\.weight=Q4_K +blk\.4\.ffn_gate_shexp\.weight=Q8_0 +blk\.4\.ffn_up_exps\.weight=Q4_K +blk\.4\.ffn_up_shexp\.weight=Q8_0 +blk\.4\.ssm_alpha\.weight=F32 +blk\.4\.ssm_beta\.weight=F32 +blk\.4\.ssm_out\.weight=Q8_0 +blk\.40\.attn_k\.weight=F16 +blk\.40\.attn_output\.weight=Q8_0 +blk\.40\.attn_q\.weight=Q8_0 +blk\.40\.attn_v\.weight=F16 +blk\.40\.ffn_down_exps\.weight=IQ4_XS +blk\.40\.ffn_down_shexp\.weight=Q8_0 +blk\.40\.ffn_gate_exps\.weight=IQ4_XS +blk\.40\.ffn_gate_shexp\.weight=Q8_0 +blk\.40\.ffn_up_exps\.weight=IQ4_XS +blk\.40\.ffn_up_shexp\.weight=Q8_0 +blk\.40\.nextn\.eh_proj\.weight=Q8_0 +blk\.5\.attn_gate\.weight=Q8_0 +blk\.5\.attn_qkv\.weight=Q8_0 +blk\.5\.ffn_down_exps\.weight=IQ4_XS +blk\.5\.ffn_down_shexp\.weight=Q8_0 +blk\.5\.ffn_gate_exps\.weight=IQ4_XS +blk\.5\.ffn_gate_shexp\.weight=Q8_0 +blk\.5\.ffn_up_exps\.weight=IQ4_XS +blk\.5\.ffn_up_shexp\.weight=Q8_0 +blk\.5\.ssm_alpha\.weight=F32 +blk\.5\.ssm_beta\.weight=F32 +blk\.5\.ssm_out\.weight=Q8_0 +blk\.6\.attn_gate\.weight=Q8_0 +blk\.6\.attn_qkv\.weight=Q8_0 +blk\.6\.ffn_down_exps\.weight=Q4_K +blk\.6\.ffn_down_shexp\.weight=Q8_0 +blk\.6\.ffn_gate_exps\.weight=Q4_K +blk\.6\.ffn_gate_shexp\.weight=Q8_0 +blk\.6\.ffn_up_exps\.weight=Q4_K +blk\.6\.ffn_up_shexp\.weight=Q8_0 +blk\.6\.ssm_alpha\.weight=F32 +blk\.6\.ssm_beta\.weight=F32 +blk\.6\.ssm_out\.weight=Q8_0 +blk\.7\.attn_k\.weight=F16 +blk\.7\.attn_output\.weight=Q8_0 +blk\.7\.attn_q\.weight=Q8_0 +blk\.7\.attn_v\.weight=F16 +blk\.7\.ffn_down_exps\.weight=Q4_K +blk\.7\.ffn_down_shexp\.weight=Q8_0 +blk\.7\.ffn_gate_exps\.weight=Q4_K +blk\.7\.ffn_gate_shexp\.weight=Q8_0 +blk\.7\.ffn_up_exps\.weight=Q4_K +blk\.7\.ffn_up_shexp\.weight=Q8_0 +blk\.8\.attn_gate\.weight=Q8_0 +blk\.8\.attn_qkv\.weight=Q8_0 +blk\.8\.ffn_down_exps\.weight=IQ4_XS +blk\.8\.ffn_down_shexp\.weight=Q8_0 +blk\.8\.ffn_gate_exps\.weight=IQ4_XS +blk\.8\.ffn_gate_shexp\.weight=Q8_0 +blk\.8\.ffn_up_exps\.weight=IQ4_XS +blk\.8\.ffn_up_shexp\.weight=Q8_0 +blk\.8\.ssm_alpha\.weight=F32 +blk\.8\.ssm_beta\.weight=F32 +blk\.8\.ssm_out\.weight=Q8_0 +blk\.9\.attn_gate\.weight=Q8_0 +blk\.9\.attn_qkv\.weight=Q8_0 +blk\.9\.ffn_down_exps\.weight=Q4_K +blk\.9\.ffn_down_shexp\.weight=Q8_0 +blk\.9\.ffn_gate_exps\.weight=Q4_K +blk\.9\.ffn_gate_shexp\.weight=Q8_0 +blk\.9\.ffn_up_exps\.weight=Q4_K +blk\.9\.ffn_up_shexp\.weight=Q8_0 +blk\.9\.ssm_alpha\.weight=F32 +blk\.9\.ssm_beta\.weight=F32 +blk\.9\.ssm_out\.weight=Q8_0 diff --git a/appendix/recipes/cfg_23G-ICE_ICEbase.txt b/appendix/recipes/cfg_23G-ICE_ICEbase.txt new file mode 100644 index 0000000000000000000000000000000000000000..10643d3cc1dd392ab627263c81e0823d0e06c2b2 --- /dev/null +++ b/appendix/recipes/cfg_23G-ICE_ICEbase.txt @@ -0,0 +1,443 @@ +^output\.weight$=Q8_0 +token_embd\.weight=Q8_0 +blk\.0\.attn_gate\.weight=Q8_0 +blk\.0\.attn_qkv\.weight=Q8_0 +blk\.0\.ffn_down_exps\.weight=Q5_K +blk\.0\.ffn_down_shexp\.weight=Q8_0 +blk\.0\.ffn_gate_exps\.weight=Q5_K +blk\.0\.ffn_gate_shexp\.weight=Q8_0 +blk\.0\.ffn_up_exps\.weight=Q5_K +blk\.0\.ffn_up_shexp\.weight=Q8_0 +blk\.0\.ssm_alpha\.weight=F32 +blk\.0\.ssm_beta\.weight=F32 +blk\.0\.ssm_out\.weight=Q8_0 +blk\.1\.attn_gate\.weight=Q8_0 +blk\.1\.attn_qkv\.weight=Q8_0 +blk\.1\.ffn_down_exps\.weight=Q5_K +blk\.1\.ffn_down_shexp\.weight=Q8_0 +blk\.1\.ffn_gate_exps\.weight=Q5_K +blk\.1\.ffn_gate_shexp\.weight=Q8_0 +blk\.1\.ffn_up_exps\.weight=Q5_K +blk\.1\.ffn_up_shexp\.weight=Q8_0 +blk\.1\.ssm_alpha\.weight=F32 +blk\.1\.ssm_beta\.weight=F32 +blk\.1\.ssm_out\.weight=Q8_0 +blk\.10\.attn_gate\.weight=Q8_0 +blk\.10\.attn_qkv\.weight=Q8_0 +blk\.10\.ffn_down_exps\.weight=Q5_K +blk\.10\.ffn_down_shexp\.weight=Q8_0 +blk\.10\.ffn_gate_exps\.weight=Q5_K +blk\.10\.ffn_gate_shexp\.weight=Q8_0 +blk\.10\.ffn_up_exps\.weight=Q5_K +blk\.10\.ffn_up_shexp\.weight=Q8_0 +blk\.10\.ssm_alpha\.weight=F32 +blk\.10\.ssm_beta\.weight=F32 +blk\.10\.ssm_out\.weight=Q8_0 +blk\.11\.attn_k\.weight=F16 +blk\.11\.attn_output\.weight=Q8_0 +blk\.11\.attn_q\.weight=Q8_0 +blk\.11\.attn_v\.weight=F16 +blk\.11\.ffn_down_exps\.weight=Q5_K +blk\.11\.ffn_down_shexp\.weight=Q8_0 +blk\.11\.ffn_gate_exps\.weight=Q5_K +blk\.11\.ffn_gate_shexp\.weight=Q8_0 +blk\.11\.ffn_up_exps\.weight=Q5_K +blk\.11\.ffn_up_shexp\.weight=Q8_0 +blk\.12\.attn_gate\.weight=Q8_0 +blk\.12\.attn_qkv\.weight=Q8_0 +blk\.12\.ffn_down_exps\.weight=Q5_K +blk\.12\.ffn_down_shexp\.weight=Q8_0 +blk\.12\.ffn_gate_exps\.weight=Q5_K +blk\.12\.ffn_gate_shexp\.weight=Q8_0 +blk\.12\.ffn_up_exps\.weight=Q5_K +blk\.12\.ffn_up_shexp\.weight=Q8_0 +blk\.12\.ssm_alpha\.weight=F32 +blk\.12\.ssm_beta\.weight=F32 +blk\.12\.ssm_out\.weight=Q8_0 +blk\.13\.attn_gate\.weight=Q8_0 +blk\.13\.attn_qkv\.weight=Q8_0 +blk\.13\.ffn_down_exps\.weight=Q5_K +blk\.13\.ffn_down_shexp\.weight=Q8_0 +blk\.13\.ffn_gate_exps\.weight=Q5_K +blk\.13\.ffn_gate_shexp\.weight=Q8_0 +blk\.13\.ffn_up_exps\.weight=Q5_K +blk\.13\.ffn_up_shexp\.weight=Q8_0 +blk\.13\.ssm_alpha\.weight=F32 +blk\.13\.ssm_beta\.weight=F32 +blk\.13\.ssm_out\.weight=Q8_0 +blk\.14\.attn_gate\.weight=Q8_0 +blk\.14\.attn_qkv\.weight=Q8_0 +blk\.14\.ffn_down_exps\.weight=Q5_K +blk\.14\.ffn_down_shexp\.weight=Q8_0 +blk\.14\.ffn_gate_exps\.weight=Q5_K +blk\.14\.ffn_gate_shexp\.weight=Q8_0 +blk\.14\.ffn_up_exps\.weight=Q5_K +blk\.14\.ffn_up_shexp\.weight=Q8_0 +blk\.14\.ssm_alpha\.weight=F32 +blk\.14\.ssm_beta\.weight=F32 +blk\.14\.ssm_out\.weight=Q8_0 +blk\.15\.attn_k\.weight=F16 +blk\.15\.attn_output\.weight=Q8_0 +blk\.15\.attn_q\.weight=Q8_0 +blk\.15\.attn_v\.weight=F16 +blk\.15\.ffn_down_exps\.weight=Q4_K +blk\.15\.ffn_down_shexp\.weight=Q8_0 +blk\.15\.ffn_gate_exps\.weight=Q4_K +blk\.15\.ffn_gate_shexp\.weight=Q8_0 +blk\.15\.ffn_up_exps\.weight=Q4_K +blk\.15\.ffn_up_shexp\.weight=Q8_0 +blk\.16\.attn_gate\.weight=Q8_0 +blk\.16\.attn_qkv\.weight=Q8_0 +blk\.16\.ffn_down_exps\.weight=Q4_K +blk\.16\.ffn_down_shexp\.weight=Q8_0 +blk\.16\.ffn_gate_exps\.weight=Q4_K +blk\.16\.ffn_gate_shexp\.weight=Q8_0 +blk\.16\.ffn_up_exps\.weight=Q4_K +blk\.16\.ffn_up_shexp\.weight=Q8_0 +blk\.16\.ssm_alpha\.weight=F32 +blk\.16\.ssm_beta\.weight=F32 +blk\.16\.ssm_out\.weight=Q8_0 +blk\.17\.attn_gate\.weight=Q8_0 +blk\.17\.attn_qkv\.weight=Q8_0 +blk\.17\.ffn_down_exps\.weight=Q4_K +blk\.17\.ffn_down_shexp\.weight=Q8_0 +blk\.17\.ffn_gate_exps\.weight=Q4_K +blk\.17\.ffn_gate_shexp\.weight=Q8_0 +blk\.17\.ffn_up_exps\.weight=Q4_K +blk\.17\.ffn_up_shexp\.weight=Q8_0 +blk\.17\.ssm_alpha\.weight=F32 +blk\.17\.ssm_beta\.weight=F32 +blk\.17\.ssm_out\.weight=Q8_0 +blk\.18\.attn_gate\.weight=Q8_0 +blk\.18\.attn_qkv\.weight=Q8_0 +blk\.18\.ffn_down_exps\.weight=Q4_K +blk\.18\.ffn_down_shexp\.weight=Q8_0 +blk\.18\.ffn_gate_exps\.weight=Q4_K +blk\.18\.ffn_gate_shexp\.weight=Q8_0 +blk\.18\.ffn_up_exps\.weight=Q4_K +blk\.18\.ffn_up_shexp\.weight=Q8_0 +blk\.18\.ssm_alpha\.weight=F32 +blk\.18\.ssm_beta\.weight=F32 +blk\.18\.ssm_out\.weight=Q8_0 +blk\.19\.attn_k\.weight=F16 +blk\.19\.attn_output\.weight=Q8_0 +blk\.19\.attn_q\.weight=Q8_0 +blk\.19\.attn_v\.weight=F16 +blk\.19\.ffn_down_exps\.weight=Q4_K +blk\.19\.ffn_down_shexp\.weight=Q8_0 +blk\.19\.ffn_gate_exps\.weight=Q4_K +blk\.19\.ffn_gate_shexp\.weight=Q8_0 +blk\.19\.ffn_up_exps\.weight=Q4_K +blk\.19\.ffn_up_shexp\.weight=Q8_0 +blk\.2\.attn_gate\.weight=Q8_0 +blk\.2\.attn_qkv\.weight=Q8_0 +blk\.2\.ffn_down_exps\.weight=Q5_K +blk\.2\.ffn_down_shexp\.weight=Q8_0 +blk\.2\.ffn_gate_exps\.weight=Q5_K +blk\.2\.ffn_gate_shexp\.weight=Q8_0 +blk\.2\.ffn_up_exps\.weight=Q5_K +blk\.2\.ffn_up_shexp\.weight=Q8_0 +blk\.2\.ssm_alpha\.weight=F32 +blk\.2\.ssm_beta\.weight=F32 +blk\.2\.ssm_out\.weight=Q8_0 +blk\.20\.attn_gate\.weight=Q8_0 +blk\.20\.attn_qkv\.weight=Q8_0 +blk\.20\.ffn_down_exps\.weight=Q4_K +blk\.20\.ffn_down_shexp\.weight=Q8_0 +blk\.20\.ffn_gate_exps\.weight=Q4_K +blk\.20\.ffn_gate_shexp\.weight=Q8_0 +blk\.20\.ffn_up_exps\.weight=Q4_K +blk\.20\.ffn_up_shexp\.weight=Q8_0 +blk\.20\.ssm_alpha\.weight=F32 +blk\.20\.ssm_beta\.weight=F32 +blk\.20\.ssm_out\.weight=Q8_0 +blk\.21\.attn_gate\.weight=Q8_0 +blk\.21\.attn_qkv\.weight=Q8_0 +blk\.21\.ffn_down_exps\.weight=Q4_K +blk\.21\.ffn_down_shexp\.weight=Q8_0 +blk\.21\.ffn_gate_exps\.weight=Q4_K +blk\.21\.ffn_gate_shexp\.weight=Q8_0 +blk\.21\.ffn_up_exps\.weight=Q4_K +blk\.21\.ffn_up_shexp\.weight=Q8_0 +blk\.21\.ssm_alpha\.weight=F32 +blk\.21\.ssm_beta\.weight=F32 +blk\.21\.ssm_out\.weight=Q8_0 +blk\.22\.attn_gate\.weight=Q8_0 +blk\.22\.attn_qkv\.weight=Q8_0 +blk\.22\.ffn_down_exps\.weight=Q4_K +blk\.22\.ffn_down_shexp\.weight=Q8_0 +blk\.22\.ffn_gate_exps\.weight=Q4_K +blk\.22\.ffn_gate_shexp\.weight=Q8_0 +blk\.22\.ffn_up_exps\.weight=Q4_K +blk\.22\.ffn_up_shexp\.weight=Q8_0 +blk\.22\.ssm_alpha\.weight=F32 +blk\.22\.ssm_beta\.weight=F32 +blk\.22\.ssm_out\.weight=Q8_0 +blk\.23\.attn_k\.weight=F16 +blk\.23\.attn_output\.weight=Q8_0 +blk\.23\.attn_q\.weight=Q8_0 +blk\.23\.attn_v\.weight=F16 +blk\.23\.ffn_down_exps\.weight=Q4_K +blk\.23\.ffn_down_shexp\.weight=Q8_0 +blk\.23\.ffn_gate_exps\.weight=Q4_K +blk\.23\.ffn_gate_shexp\.weight=Q8_0 +blk\.23\.ffn_up_exps\.weight=Q4_K +blk\.23\.ffn_up_shexp\.weight=Q8_0 +blk\.24\.attn_gate\.weight=Q8_0 +blk\.24\.attn_qkv\.weight=Q8_0 +blk\.24\.ffn_down_exps\.weight=Q4_K +blk\.24\.ffn_down_shexp\.weight=Q8_0 +blk\.24\.ffn_gate_exps\.weight=Q4_K +blk\.24\.ffn_gate_shexp\.weight=Q8_0 +blk\.24\.ffn_up_exps\.weight=Q4_K +blk\.24\.ffn_up_shexp\.weight=Q8_0 +blk\.24\.ssm_alpha\.weight=F32 +blk\.24\.ssm_beta\.weight=F32 +blk\.24\.ssm_out\.weight=Q8_0 +blk\.25\.attn_gate\.weight=Q8_0 +blk\.25\.attn_qkv\.weight=Q8_0 +blk\.25\.ffn_down_exps\.weight=Q4_K +blk\.25\.ffn_down_shexp\.weight=Q8_0 +blk\.25\.ffn_gate_exps\.weight=Q4_K +blk\.25\.ffn_gate_shexp\.weight=Q8_0 +blk\.25\.ffn_up_exps\.weight=Q4_K +blk\.25\.ffn_up_shexp\.weight=Q8_0 +blk\.25\.ssm_alpha\.weight=F32 +blk\.25\.ssm_beta\.weight=F32 +blk\.25\.ssm_out\.weight=Q8_0 +blk\.26\.attn_gate\.weight=Q8_0 +blk\.26\.attn_qkv\.weight=Q8_0 +blk\.26\.ffn_down_exps\.weight=Q4_K +blk\.26\.ffn_down_shexp\.weight=Q8_0 +blk\.26\.ffn_gate_exps\.weight=Q4_K +blk\.26\.ffn_gate_shexp\.weight=Q8_0 +blk\.26\.ffn_up_exps\.weight=Q4_K +blk\.26\.ffn_up_shexp\.weight=Q8_0 +blk\.26\.ssm_alpha\.weight=F32 +blk\.26\.ssm_beta\.weight=F32 +blk\.26\.ssm_out\.weight=Q8_0 +blk\.27\.attn_k\.weight=F16 +blk\.27\.attn_output\.weight=Q8_0 +blk\.27\.attn_q\.weight=Q8_0 +blk\.27\.attn_v\.weight=F16 +blk\.27\.ffn_down_exps\.weight=Q4_K +blk\.27\.ffn_down_shexp\.weight=Q8_0 +blk\.27\.ffn_gate_exps\.weight=Q4_K +blk\.27\.ffn_gate_shexp\.weight=Q8_0 +blk\.27\.ffn_up_exps\.weight=Q4_K +blk\.27\.ffn_up_shexp\.weight=Q8_0 +blk\.28\.attn_gate\.weight=Q8_0 +blk\.28\.attn_qkv\.weight=Q8_0 +blk\.28\.ffn_down_exps\.weight=Q4_K +blk\.28\.ffn_down_shexp\.weight=Q8_0 +blk\.28\.ffn_gate_exps\.weight=Q4_K +blk\.28\.ffn_gate_shexp\.weight=Q8_0 +blk\.28\.ffn_up_exps\.weight=Q4_K +blk\.28\.ffn_up_shexp\.weight=Q8_0 +blk\.28\.ssm_alpha\.weight=F32 +blk\.28\.ssm_beta\.weight=F32 +blk\.28\.ssm_out\.weight=Q8_0 +blk\.29\.attn_gate\.weight=Q8_0 +blk\.29\.attn_qkv\.weight=Q8_0 +blk\.29\.ffn_down_exps\.weight=Q4_K +blk\.29\.ffn_down_shexp\.weight=Q8_0 +blk\.29\.ffn_gate_exps\.weight=Q4_K +blk\.29\.ffn_gate_shexp\.weight=Q8_0 +blk\.29\.ffn_up_exps\.weight=Q4_K +blk\.29\.ffn_up_shexp\.weight=Q8_0 +blk\.29\.ssm_alpha\.weight=F32 +blk\.29\.ssm_beta\.weight=F32 +blk\.29\.ssm_out\.weight=Q8_0 +blk\.3\.attn_k\.weight=F16 +blk\.3\.attn_output\.weight=Q8_0 +blk\.3\.attn_q\.weight=Q8_0 +blk\.3\.attn_v\.weight=F16 +blk\.3\.ffn_down_exps\.weight=Q5_K +blk\.3\.ffn_down_shexp\.weight=Q8_0 +blk\.3\.ffn_gate_exps\.weight=Q5_K +blk\.3\.ffn_gate_shexp\.weight=Q8_0 +blk\.3\.ffn_up_exps\.weight=Q5_K +blk\.3\.ffn_up_shexp\.weight=Q8_0 +blk\.30\.attn_gate\.weight=Q8_0 +blk\.30\.attn_qkv\.weight=Q8_0 +blk\.30\.ffn_down_exps\.weight=Q4_K +blk\.30\.ffn_down_shexp\.weight=Q8_0 +blk\.30\.ffn_gate_exps\.weight=Q4_K +blk\.30\.ffn_gate_shexp\.weight=Q8_0 +blk\.30\.ffn_up_exps\.weight=Q4_K +blk\.30\.ffn_up_shexp\.weight=Q8_0 +blk\.30\.ssm_alpha\.weight=F32 +blk\.30\.ssm_beta\.weight=F32 +blk\.30\.ssm_out\.weight=Q8_0 +blk\.31\.attn_k\.weight=F16 +blk\.31\.attn_output\.weight=Q8_0 +blk\.31\.attn_q\.weight=Q8_0 +blk\.31\.attn_v\.weight=F16 +blk\.31\.ffn_down_exps\.weight=Q4_K +blk\.31\.ffn_down_shexp\.weight=Q8_0 +blk\.31\.ffn_gate_exps\.weight=Q4_K +blk\.31\.ffn_gate_shexp\.weight=Q8_0 +blk\.31\.ffn_up_exps\.weight=Q4_K +blk\.31\.ffn_up_shexp\.weight=Q8_0 +blk\.32\.attn_gate\.weight=Q8_0 +blk\.32\.attn_qkv\.weight=Q8_0 +blk\.32\.ffn_down_exps\.weight=Q4_K +blk\.32\.ffn_down_shexp\.weight=Q8_0 +blk\.32\.ffn_gate_exps\.weight=Q4_K +blk\.32\.ffn_gate_shexp\.weight=Q8_0 +blk\.32\.ffn_up_exps\.weight=Q4_K +blk\.32\.ffn_up_shexp\.weight=Q8_0 +blk\.32\.ssm_alpha\.weight=F32 +blk\.32\.ssm_beta\.weight=F32 +blk\.32\.ssm_out\.weight=Q8_0 +blk\.33\.attn_gate\.weight=Q8_0 +blk\.33\.attn_qkv\.weight=Q8_0 +blk\.33\.ffn_down_exps\.weight=Q4_K +blk\.33\.ffn_down_shexp\.weight=Q8_0 +blk\.33\.ffn_gate_exps\.weight=Q4_K +blk\.33\.ffn_gate_shexp\.weight=Q8_0 +blk\.33\.ffn_up_exps\.weight=Q4_K +blk\.33\.ffn_up_shexp\.weight=Q8_0 +blk\.33\.ssm_alpha\.weight=F32 +blk\.33\.ssm_beta\.weight=F32 +blk\.33\.ssm_out\.weight=Q8_0 +blk\.34\.attn_gate\.weight=Q8_0 +blk\.34\.attn_qkv\.weight=Q8_0 +blk\.34\.ffn_down_exps\.weight=Q4_K +blk\.34\.ffn_down_shexp\.weight=Q8_0 +blk\.34\.ffn_gate_exps\.weight=Q4_K +blk\.34\.ffn_gate_shexp\.weight=Q8_0 +blk\.34\.ffn_up_exps\.weight=Q4_K +blk\.34\.ffn_up_shexp\.weight=Q8_0 +blk\.34\.ssm_alpha\.weight=F32 +blk\.34\.ssm_beta\.weight=F32 +blk\.34\.ssm_out\.weight=Q8_0 +blk\.35\.attn_k\.weight=F16 +blk\.35\.attn_output\.weight=Q8_0 +blk\.35\.attn_q\.weight=Q8_0 +blk\.35\.attn_v\.weight=F16 +blk\.35\.ffn_down_exps\.weight=Q4_K +blk\.35\.ffn_down_shexp\.weight=Q8_0 +blk\.35\.ffn_gate_exps\.weight=Q4_K +blk\.35\.ffn_gate_shexp\.weight=Q8_0 +blk\.35\.ffn_up_exps\.weight=Q4_K +blk\.35\.ffn_up_shexp\.weight=Q8_0 +blk\.36\.attn_gate\.weight=Q8_0 +blk\.36\.attn_qkv\.weight=Q8_0 +blk\.36\.ffn_down_exps\.weight=Q4_K +blk\.36\.ffn_down_shexp\.weight=Q8_0 +blk\.36\.ffn_gate_exps\.weight=Q4_K +blk\.36\.ffn_gate_shexp\.weight=Q8_0 +blk\.36\.ffn_up_exps\.weight=Q4_K +blk\.36\.ffn_up_shexp\.weight=Q8_0 +blk\.36\.ssm_alpha\.weight=F32 +blk\.36\.ssm_beta\.weight=F32 +blk\.36\.ssm_out\.weight=Q8_0 +blk\.37\.attn_gate\.weight=Q8_0 +blk\.37\.attn_qkv\.weight=Q8_0 +blk\.37\.ffn_down_exps\.weight=Q4_K +blk\.37\.ffn_down_shexp\.weight=Q8_0 +blk\.37\.ffn_gate_exps\.weight=Q4_K +blk\.37\.ffn_gate_shexp\.weight=Q8_0 +blk\.37\.ffn_up_exps\.weight=Q4_K +blk\.37\.ffn_up_shexp\.weight=Q8_0 +blk\.37\.ssm_alpha\.weight=F32 +blk\.37\.ssm_beta\.weight=F32 +blk\.37\.ssm_out\.weight=Q8_0 +blk\.38\.attn_gate\.weight=Q8_0 +blk\.38\.attn_qkv\.weight=Q8_0 +blk\.38\.ffn_down_exps\.weight=Q4_K +blk\.38\.ffn_down_shexp\.weight=Q8_0 +blk\.38\.ffn_gate_exps\.weight=Q4_K +blk\.38\.ffn_gate_shexp\.weight=Q8_0 +blk\.38\.ffn_up_exps\.weight=Q4_K +blk\.38\.ffn_up_shexp\.weight=Q8_0 +blk\.38\.ssm_alpha\.weight=F32 +blk\.38\.ssm_beta\.weight=F32 +blk\.38\.ssm_out\.weight=Q8_0 +blk\.39\.attn_k\.weight=F16 +blk\.39\.attn_output\.weight=Q8_0 +blk\.39\.attn_q\.weight=Q8_0 +blk\.39\.attn_v\.weight=F16 +blk\.39\.ffn_down_exps\.weight=Q4_K +blk\.39\.ffn_down_shexp\.weight=Q8_0 +blk\.39\.ffn_gate_exps\.weight=Q4_K +blk\.39\.ffn_gate_shexp\.weight=Q8_0 +blk\.39\.ffn_up_exps\.weight=Q4_K +blk\.39\.ffn_up_shexp\.weight=Q8_0 +blk\.4\.attn_gate\.weight=Q8_0 +blk\.4\.attn_qkv\.weight=Q8_0 +blk\.4\.ffn_down_exps\.weight=Q5_K +blk\.4\.ffn_down_shexp\.weight=Q8_0 +blk\.4\.ffn_gate_exps\.weight=Q5_K +blk\.4\.ffn_gate_shexp\.weight=Q8_0 +blk\.4\.ffn_up_exps\.weight=Q5_K +blk\.4\.ffn_up_shexp\.weight=Q8_0 +blk\.4\.ssm_alpha\.weight=F32 +blk\.4\.ssm_beta\.weight=F32 +blk\.4\.ssm_out\.weight=Q8_0 +blk\.40\.attn_k\.weight=F16 +blk\.40\.attn_output\.weight=Q8_0 +blk\.40\.attn_q\.weight=Q8_0 +blk\.40\.attn_v\.weight=F16 +blk\.40\.ffn_down_exps\.weight=Q4_K +blk\.40\.ffn_down_shexp\.weight=Q8_0 +blk\.40\.ffn_gate_exps\.weight=Q4_K +blk\.40\.ffn_gate_shexp\.weight=Q8_0 +blk\.40\.ffn_up_exps\.weight=Q4_K +blk\.40\.ffn_up_shexp\.weight=Q8_0 +blk\.40\.nextn\.eh_proj\.weight=Q8_0 +blk\.5\.attn_gate\.weight=Q8_0 +blk\.5\.attn_qkv\.weight=Q8_0 +blk\.5\.ffn_down_exps\.weight=Q5_K +blk\.5\.ffn_down_shexp\.weight=Q8_0 +blk\.5\.ffn_gate_exps\.weight=Q5_K +blk\.5\.ffn_gate_shexp\.weight=Q8_0 +blk\.5\.ffn_up_exps\.weight=Q5_K +blk\.5\.ffn_up_shexp\.weight=Q8_0 +blk\.5\.ssm_alpha\.weight=F32 +blk\.5\.ssm_beta\.weight=F32 +blk\.5\.ssm_out\.weight=Q8_0 +blk\.6\.attn_gate\.weight=Q8_0 +blk\.6\.attn_qkv\.weight=Q8_0 +blk\.6\.ffn_down_exps\.weight=Q5_K +blk\.6\.ffn_down_shexp\.weight=Q8_0 +blk\.6\.ffn_gate_exps\.weight=Q5_K +blk\.6\.ffn_gate_shexp\.weight=Q8_0 +blk\.6\.ffn_up_exps\.weight=Q5_K +blk\.6\.ffn_up_shexp\.weight=Q8_0 +blk\.6\.ssm_alpha\.weight=F32 +blk\.6\.ssm_beta\.weight=F32 +blk\.6\.ssm_out\.weight=Q8_0 +blk\.7\.attn_k\.weight=F16 +blk\.7\.attn_output\.weight=Q8_0 +blk\.7\.attn_q\.weight=Q8_0 +blk\.7\.attn_v\.weight=F16 +blk\.7\.ffn_down_exps\.weight=Q5_K +blk\.7\.ffn_down_shexp\.weight=Q8_0 +blk\.7\.ffn_gate_exps\.weight=Q5_K +blk\.7\.ffn_gate_shexp\.weight=Q8_0 +blk\.7\.ffn_up_exps\.weight=Q5_K +blk\.7\.ffn_up_shexp\.weight=Q8_0 +blk\.8\.attn_gate\.weight=Q8_0 +blk\.8\.attn_qkv\.weight=Q8_0 +blk\.8\.ffn_down_exps\.weight=Q5_K +blk\.8\.ffn_down_shexp\.weight=Q8_0 +blk\.8\.ffn_gate_exps\.weight=Q5_K +blk\.8\.ffn_gate_shexp\.weight=Q8_0 +blk\.8\.ffn_up_exps\.weight=Q5_K +blk\.8\.ffn_up_shexp\.weight=Q8_0 +blk\.8\.ssm_alpha\.weight=F32 +blk\.8\.ssm_beta\.weight=F32 +blk\.8\.ssm_out\.weight=Q8_0 +blk\.9\.attn_gate\.weight=Q8_0 +blk\.9\.attn_qkv\.weight=Q8_0 +blk\.9\.ffn_down_exps\.weight=Q5_K +blk\.9\.ffn_down_shexp\.weight=Q8_0 +blk\.9\.ffn_gate_exps\.weight=Q5_K +blk\.9\.ffn_gate_shexp\.weight=Q8_0 +blk\.9\.ffn_up_exps\.weight=Q5_K +blk\.9\.ffn_up_shexp\.weight=Q8_0 +blk\.9\.ssm_alpha\.weight=F32 +blk\.9\.ssm_beta\.weight=F32 +blk\.9\.ssm_out\.weight=Q8_0 diff --git a/appendix/recipes/cfg_23G-ICE_previous.txt b/appendix/recipes/cfg_23G-ICE_previous.txt new file mode 100644 index 0000000000000000000000000000000000000000..65292e1b4f9efc663e29f5133ff48a2b3d58179b --- /dev/null +++ b/appendix/recipes/cfg_23G-ICE_previous.txt @@ -0,0 +1,443 @@ +^output\.weight$=Q8_0 +token_embd\.weight=Q8_0 +blk\.0\.attn_gate\.weight=Q8_0 +blk\.0\.attn_qkv\.weight=Q8_0 +blk\.0\.ffn_down_exps\.weight=Q5_K +blk\.0\.ffn_down_shexp\.weight=Q8_0 +blk\.0\.ffn_gate_exps\.weight=Q5_K +blk\.0\.ffn_gate_shexp\.weight=Q8_0 +blk\.0\.ffn_up_exps\.weight=Q5_K +blk\.0\.ffn_up_shexp\.weight=Q8_0 +blk\.0\.ssm_alpha\.weight=F32 +blk\.0\.ssm_beta\.weight=F32 +blk\.0\.ssm_out\.weight=Q8_0 +blk\.1\.attn_gate\.weight=Q8_0 +blk\.1\.attn_qkv\.weight=Q8_0 +blk\.1\.ffn_down_exps\.weight=Q4_K +blk\.1\.ffn_down_shexp\.weight=Q8_0 +blk\.1\.ffn_gate_exps\.weight=Q4_K +blk\.1\.ffn_gate_shexp\.weight=Q8_0 +blk\.1\.ffn_up_exps\.weight=Q4_K +blk\.1\.ffn_up_shexp\.weight=Q8_0 +blk\.1\.ssm_alpha\.weight=F32 +blk\.1\.ssm_beta\.weight=F32 +blk\.1\.ssm_out\.weight=Q8_0 +blk\.10\.attn_gate\.weight=Q8_0 +blk\.10\.attn_qkv\.weight=Q8_0 +blk\.10\.ffn_down_exps\.weight=Q4_K +blk\.10\.ffn_down_shexp\.weight=Q8_0 +blk\.10\.ffn_gate_exps\.weight=Q4_K +blk\.10\.ffn_gate_shexp\.weight=Q8_0 +blk\.10\.ffn_up_exps\.weight=Q4_K +blk\.10\.ffn_up_shexp\.weight=Q8_0 +blk\.10\.ssm_alpha\.weight=F32 +blk\.10\.ssm_beta\.weight=F32 +blk\.10\.ssm_out\.weight=Q8_0 +blk\.11\.attn_k\.weight=F16 +blk\.11\.attn_output\.weight=Q8_0 +blk\.11\.attn_q\.weight=Q8_0 +blk\.11\.attn_v\.weight=F16 +blk\.11\.ffn_down_exps\.weight=Q5_K +blk\.11\.ffn_down_shexp\.weight=Q8_0 +blk\.11\.ffn_gate_exps\.weight=Q5_K +blk\.11\.ffn_gate_shexp\.weight=Q8_0 +blk\.11\.ffn_up_exps\.weight=Q5_K +blk\.11\.ffn_up_shexp\.weight=Q8_0 +blk\.12\.attn_gate\.weight=Q8_0 +blk\.12\.attn_qkv\.weight=Q8_0 +blk\.12\.ffn_down_exps\.weight=Q4_K +blk\.12\.ffn_down_shexp\.weight=Q8_0 +blk\.12\.ffn_gate_exps\.weight=Q4_K +blk\.12\.ffn_gate_shexp\.weight=Q8_0 +blk\.12\.ffn_up_exps\.weight=Q4_K +blk\.12\.ffn_up_shexp\.weight=Q8_0 +blk\.12\.ssm_alpha\.weight=F32 +blk\.12\.ssm_beta\.weight=F32 +blk\.12\.ssm_out\.weight=Q8_0 +blk\.13\.attn_gate\.weight=Q8_0 +blk\.13\.attn_qkv\.weight=Q8_0 +blk\.13\.ffn_down_exps\.weight=Q4_K +blk\.13\.ffn_down_shexp\.weight=Q8_0 +blk\.13\.ffn_gate_exps\.weight=Q4_K +blk\.13\.ffn_gate_shexp\.weight=Q8_0 +blk\.13\.ffn_up_exps\.weight=Q4_K +blk\.13\.ffn_up_shexp\.weight=Q8_0 +blk\.13\.ssm_alpha\.weight=F32 +blk\.13\.ssm_beta\.weight=F32 +blk\.13\.ssm_out\.weight=Q8_0 +blk\.14\.attn_gate\.weight=Q8_0 +blk\.14\.attn_qkv\.weight=Q8_0 +blk\.14\.ffn_down_exps\.weight=Q5_K +blk\.14\.ffn_down_shexp\.weight=Q8_0 +blk\.14\.ffn_gate_exps\.weight=Q5_K +blk\.14\.ffn_gate_shexp\.weight=Q8_0 +blk\.14\.ffn_up_exps\.weight=Q5_K +blk\.14\.ffn_up_shexp\.weight=Q8_0 +blk\.14\.ssm_alpha\.weight=F32 +blk\.14\.ssm_beta\.weight=F32 +blk\.14\.ssm_out\.weight=Q8_0 +blk\.15\.attn_k\.weight=F16 +blk\.15\.attn_output\.weight=Q8_0 +blk\.15\.attn_q\.weight=Q8_0 +blk\.15\.attn_v\.weight=F16 +blk\.15\.ffn_down_exps\.weight=Q4_K +blk\.15\.ffn_down_shexp\.weight=Q8_0 +blk\.15\.ffn_gate_exps\.weight=Q4_K +blk\.15\.ffn_gate_shexp\.weight=Q8_0 +blk\.15\.ffn_up_exps\.weight=Q4_K +blk\.15\.ffn_up_shexp\.weight=Q8_0 +blk\.16\.attn_gate\.weight=Q8_0 +blk\.16\.attn_qkv\.weight=Q8_0 +blk\.16\.ffn_down_exps\.weight=Q5_K +blk\.16\.ffn_down_shexp\.weight=Q8_0 +blk\.16\.ffn_gate_exps\.weight=Q5_K +blk\.16\.ffn_gate_shexp\.weight=Q8_0 +blk\.16\.ffn_up_exps\.weight=Q5_K +blk\.16\.ffn_up_shexp\.weight=Q8_0 +blk\.16\.ssm_alpha\.weight=F32 +blk\.16\.ssm_beta\.weight=F32 +blk\.16\.ssm_out\.weight=Q8_0 +blk\.17\.attn_gate\.weight=Q8_0 +blk\.17\.attn_qkv\.weight=Q8_0 +blk\.17\.ffn_down_exps\.weight=Q4_K +blk\.17\.ffn_down_shexp\.weight=Q8_0 +blk\.17\.ffn_gate_exps\.weight=Q4_K +blk\.17\.ffn_gate_shexp\.weight=Q8_0 +blk\.17\.ffn_up_exps\.weight=Q4_K +blk\.17\.ffn_up_shexp\.weight=Q8_0 +blk\.17\.ssm_alpha\.weight=F32 +blk\.17\.ssm_beta\.weight=F32 +blk\.17\.ssm_out\.weight=Q8_0 +blk\.18\.attn_gate\.weight=Q8_0 +blk\.18\.attn_qkv\.weight=Q8_0 +blk\.18\.ffn_down_exps\.weight=Q4_K +blk\.18\.ffn_down_shexp\.weight=Q8_0 +blk\.18\.ffn_gate_exps\.weight=Q4_K +blk\.18\.ffn_gate_shexp\.weight=Q8_0 +blk\.18\.ffn_up_exps\.weight=Q4_K +blk\.18\.ffn_up_shexp\.weight=Q8_0 +blk\.18\.ssm_alpha\.weight=F32 +blk\.18\.ssm_beta\.weight=F32 +blk\.18\.ssm_out\.weight=Q8_0 +blk\.19\.attn_k\.weight=F16 +blk\.19\.attn_output\.weight=Q8_0 +blk\.19\.attn_q\.weight=Q8_0 +blk\.19\.attn_v\.weight=F16 +blk\.19\.ffn_down_exps\.weight=Q5_K +blk\.19\.ffn_down_shexp\.weight=Q8_0 +blk\.19\.ffn_gate_exps\.weight=Q5_K +blk\.19\.ffn_gate_shexp\.weight=Q8_0 +blk\.19\.ffn_up_exps\.weight=Q5_K +blk\.19\.ffn_up_shexp\.weight=Q8_0 +blk\.2\.attn_gate\.weight=Q8_0 +blk\.2\.attn_qkv\.weight=Q8_0 +blk\.2\.ffn_down_exps\.weight=Q4_K +blk\.2\.ffn_down_shexp\.weight=Q8_0 +blk\.2\.ffn_gate_exps\.weight=Q4_K +blk\.2\.ffn_gate_shexp\.weight=Q8_0 +blk\.2\.ffn_up_exps\.weight=Q4_K +blk\.2\.ffn_up_shexp\.weight=Q8_0 +blk\.2\.ssm_alpha\.weight=F32 +blk\.2\.ssm_beta\.weight=F32 +blk\.2\.ssm_out\.weight=Q8_0 +blk\.20\.attn_gate\.weight=Q8_0 +blk\.20\.attn_qkv\.weight=Q8_0 +blk\.20\.ffn_down_exps\.weight=Q4_K +blk\.20\.ffn_down_shexp\.weight=Q8_0 +blk\.20\.ffn_gate_exps\.weight=Q4_K +blk\.20\.ffn_gate_shexp\.weight=Q8_0 +blk\.20\.ffn_up_exps\.weight=Q4_K +blk\.20\.ffn_up_shexp\.weight=Q8_0 +blk\.20\.ssm_alpha\.weight=F32 +blk\.20\.ssm_beta\.weight=F32 +blk\.20\.ssm_out\.weight=Q8_0 +blk\.21\.attn_gate\.weight=Q8_0 +blk\.21\.attn_qkv\.weight=Q8_0 +blk\.21\.ffn_down_exps\.weight=Q4_K +blk\.21\.ffn_down_shexp\.weight=Q8_0 +blk\.21\.ffn_gate_exps\.weight=Q4_K +blk\.21\.ffn_gate_shexp\.weight=Q8_0 +blk\.21\.ffn_up_exps\.weight=Q4_K +blk\.21\.ffn_up_shexp\.weight=Q8_0 +blk\.21\.ssm_alpha\.weight=F32 +blk\.21\.ssm_beta\.weight=F32 +blk\.21\.ssm_out\.weight=Q8_0 +blk\.22\.attn_gate\.weight=Q8_0 +blk\.22\.attn_qkv\.weight=Q8_0 +blk\.22\.ffn_down_exps\.weight=Q5_K +blk\.22\.ffn_down_shexp\.weight=Q8_0 +blk\.22\.ffn_gate_exps\.weight=Q5_K +blk\.22\.ffn_gate_shexp\.weight=Q8_0 +blk\.22\.ffn_up_exps\.weight=Q5_K +blk\.22\.ffn_up_shexp\.weight=Q8_0 +blk\.22\.ssm_alpha\.weight=F32 +blk\.22\.ssm_beta\.weight=F32 +blk\.22\.ssm_out\.weight=Q8_0 +blk\.23\.attn_k\.weight=F16 +blk\.23\.attn_output\.weight=Q8_0 +blk\.23\.attn_q\.weight=Q8_0 +blk\.23\.attn_v\.weight=F16 +blk\.23\.ffn_down_exps\.weight=Q4_K +blk\.23\.ffn_down_shexp\.weight=Q8_0 +blk\.23\.ffn_gate_exps\.weight=Q4_K +blk\.23\.ffn_gate_shexp\.weight=Q8_0 +blk\.23\.ffn_up_exps\.weight=Q4_K +blk\.23\.ffn_up_shexp\.weight=Q8_0 +blk\.24\.attn_gate\.weight=Q8_0 +blk\.24\.attn_qkv\.weight=Q8_0 +blk\.24\.ffn_down_exps\.weight=Q4_K +blk\.24\.ffn_down_shexp\.weight=Q8_0 +blk\.24\.ffn_gate_exps\.weight=Q4_K +blk\.24\.ffn_gate_shexp\.weight=Q8_0 +blk\.24\.ffn_up_exps\.weight=Q4_K +blk\.24\.ffn_up_shexp\.weight=Q8_0 +blk\.24\.ssm_alpha\.weight=F32 +blk\.24\.ssm_beta\.weight=F32 +blk\.24\.ssm_out\.weight=Q8_0 +blk\.25\.attn_gate\.weight=Q8_0 +blk\.25\.attn_qkv\.weight=Q8_0 +blk\.25\.ffn_down_exps\.weight=Q5_K +blk\.25\.ffn_down_shexp\.weight=Q8_0 +blk\.25\.ffn_gate_exps\.weight=Q5_K +blk\.25\.ffn_gate_shexp\.weight=Q8_0 +blk\.25\.ffn_up_exps\.weight=Q5_K +blk\.25\.ffn_up_shexp\.weight=Q8_0 +blk\.25\.ssm_alpha\.weight=F32 +blk\.25\.ssm_beta\.weight=F32 +blk\.25\.ssm_out\.weight=Q8_0 +blk\.26\.attn_gate\.weight=Q8_0 +blk\.26\.attn_qkv\.weight=Q8_0 +blk\.26\.ffn_down_exps\.weight=Q4_K +blk\.26\.ffn_down_shexp\.weight=Q8_0 +blk\.26\.ffn_gate_exps\.weight=Q4_K +blk\.26\.ffn_gate_shexp\.weight=Q8_0 +blk\.26\.ffn_up_exps\.weight=Q4_K +blk\.26\.ffn_up_shexp\.weight=Q8_0 +blk\.26\.ssm_alpha\.weight=F32 +blk\.26\.ssm_beta\.weight=F32 +blk\.26\.ssm_out\.weight=Q8_0 +blk\.27\.attn_k\.weight=F16 +blk\.27\.attn_output\.weight=Q8_0 +blk\.27\.attn_q\.weight=Q8_0 +blk\.27\.attn_v\.weight=F16 +blk\.27\.ffn_down_exps\.weight=Q5_K +blk\.27\.ffn_down_shexp\.weight=Q8_0 +blk\.27\.ffn_gate_exps\.weight=Q5_K +blk\.27\.ffn_gate_shexp\.weight=Q8_0 +blk\.27\.ffn_up_exps\.weight=Q5_K +blk\.27\.ffn_up_shexp\.weight=Q8_0 +blk\.28\.attn_gate\.weight=Q8_0 +blk\.28\.attn_qkv\.weight=Q8_0 +blk\.28\.ffn_down_exps\.weight=Q4_K +blk\.28\.ffn_down_shexp\.weight=Q8_0 +blk\.28\.ffn_gate_exps\.weight=Q4_K +blk\.28\.ffn_gate_shexp\.weight=Q8_0 +blk\.28\.ffn_up_exps\.weight=Q4_K +blk\.28\.ffn_up_shexp\.weight=Q8_0 +blk\.28\.ssm_alpha\.weight=F32 +blk\.28\.ssm_beta\.weight=F32 +blk\.28\.ssm_out\.weight=Q8_0 +blk\.29\.attn_gate\.weight=Q8_0 +blk\.29\.attn_qkv\.weight=Q8_0 +blk\.29\.ffn_down_exps\.weight=Q4_K +blk\.29\.ffn_down_shexp\.weight=Q8_0 +blk\.29\.ffn_gate_exps\.weight=Q4_K +blk\.29\.ffn_gate_shexp\.weight=Q8_0 +blk\.29\.ffn_up_exps\.weight=Q4_K +blk\.29\.ffn_up_shexp\.weight=Q8_0 +blk\.29\.ssm_alpha\.weight=F32 +blk\.29\.ssm_beta\.weight=F32 +blk\.29\.ssm_out\.weight=Q8_0 +blk\.3\.attn_k\.weight=F16 +blk\.3\.attn_output\.weight=Q8_0 +blk\.3\.attn_q\.weight=Q8_0 +blk\.3\.attn_v\.weight=F16 +blk\.3\.ffn_down_exps\.weight=Q5_K +blk\.3\.ffn_down_shexp\.weight=Q8_0 +blk\.3\.ffn_gate_exps\.weight=Q5_K +blk\.3\.ffn_gate_shexp\.weight=Q8_0 +blk\.3\.ffn_up_exps\.weight=Q5_K +blk\.3\.ffn_up_shexp\.weight=Q8_0 +blk\.30\.attn_gate\.weight=Q8_0 +blk\.30\.attn_qkv\.weight=Q8_0 +blk\.30\.ffn_down_exps\.weight=Q5_K +blk\.30\.ffn_down_shexp\.weight=Q8_0 +blk\.30\.ffn_gate_exps\.weight=Q5_K +blk\.30\.ffn_gate_shexp\.weight=Q8_0 +blk\.30\.ffn_up_exps\.weight=Q5_K +blk\.30\.ffn_up_shexp\.weight=Q8_0 +blk\.30\.ssm_alpha\.weight=F32 +blk\.30\.ssm_beta\.weight=F32 +blk\.30\.ssm_out\.weight=Q8_0 +blk\.31\.attn_k\.weight=F16 +blk\.31\.attn_output\.weight=Q8_0 +blk\.31\.attn_q\.weight=Q8_0 +blk\.31\.attn_v\.weight=F16 +blk\.31\.ffn_down_exps\.weight=Q4_K +blk\.31\.ffn_down_shexp\.weight=Q8_0 +blk\.31\.ffn_gate_exps\.weight=Q4_K +blk\.31\.ffn_gate_shexp\.weight=Q8_0 +blk\.31\.ffn_up_exps\.weight=Q4_K +blk\.31\.ffn_up_shexp\.weight=Q8_0 +blk\.32\.attn_gate\.weight=Q8_0 +blk\.32\.attn_qkv\.weight=Q8_0 +blk\.32\.ffn_down_exps\.weight=Q4_K +blk\.32\.ffn_down_shexp\.weight=Q8_0 +blk\.32\.ffn_gate_exps\.weight=Q4_K +blk\.32\.ffn_gate_shexp\.weight=Q8_0 +blk\.32\.ffn_up_exps\.weight=Q4_K +blk\.32\.ffn_up_shexp\.weight=Q8_0 +blk\.32\.ssm_alpha\.weight=F32 +blk\.32\.ssm_beta\.weight=F32 +blk\.32\.ssm_out\.weight=Q8_0 +blk\.33\.attn_gate\.weight=Q8_0 +blk\.33\.attn_qkv\.weight=Q8_0 +blk\.33\.ffn_down_exps\.weight=Q5_K +blk\.33\.ffn_down_shexp\.weight=Q8_0 +blk\.33\.ffn_gate_exps\.weight=Q5_K +blk\.33\.ffn_gate_shexp\.weight=Q8_0 +blk\.33\.ffn_up_exps\.weight=Q5_K +blk\.33\.ffn_up_shexp\.weight=Q8_0 +blk\.33\.ssm_alpha\.weight=F32 +blk\.33\.ssm_beta\.weight=F32 +blk\.33\.ssm_out\.weight=Q8_0 +blk\.34\.attn_gate\.weight=Q8_0 +blk\.34\.attn_qkv\.weight=Q8_0 +blk\.34\.ffn_down_exps\.weight=Q4_K +blk\.34\.ffn_down_shexp\.weight=Q8_0 +blk\.34\.ffn_gate_exps\.weight=Q4_K +blk\.34\.ffn_gate_shexp\.weight=Q8_0 +blk\.34\.ffn_up_exps\.weight=Q4_K +blk\.34\.ffn_up_shexp\.weight=Q8_0 +blk\.34\.ssm_alpha\.weight=F32 +blk\.34\.ssm_beta\.weight=F32 +blk\.34\.ssm_out\.weight=Q8_0 +blk\.35\.attn_k\.weight=F16 +blk\.35\.attn_output\.weight=Q8_0 +blk\.35\.attn_q\.weight=Q8_0 +blk\.35\.attn_v\.weight=F16 +blk\.35\.ffn_down_exps\.weight=Q4_K +blk\.35\.ffn_down_shexp\.weight=Q8_0 +blk\.35\.ffn_gate_exps\.weight=Q4_K +blk\.35\.ffn_gate_shexp\.weight=Q8_0 +blk\.35\.ffn_up_exps\.weight=Q4_K +blk\.35\.ffn_up_shexp\.weight=Q8_0 +blk\.36\.attn_gate\.weight=Q8_0 +blk\.36\.attn_qkv\.weight=Q8_0 +blk\.36\.ffn_down_exps\.weight=Q5_K +blk\.36\.ffn_down_shexp\.weight=Q8_0 +blk\.36\.ffn_gate_exps\.weight=Q5_K +blk\.36\.ffn_gate_shexp\.weight=Q8_0 +blk\.36\.ffn_up_exps\.weight=Q5_K +blk\.36\.ffn_up_shexp\.weight=Q8_0 +blk\.36\.ssm_alpha\.weight=F32 +blk\.36\.ssm_beta\.weight=F32 +blk\.36\.ssm_out\.weight=Q8_0 +blk\.37\.attn_gate\.weight=Q8_0 +blk\.37\.attn_qkv\.weight=Q8_0 +blk\.37\.ffn_down_exps\.weight=Q4_K +blk\.37\.ffn_down_shexp\.weight=Q8_0 +blk\.37\.ffn_gate_exps\.weight=Q4_K +blk\.37\.ffn_gate_shexp\.weight=Q8_0 +blk\.37\.ffn_up_exps\.weight=Q4_K +blk\.37\.ffn_up_shexp\.weight=Q8_0 +blk\.37\.ssm_alpha\.weight=F32 +blk\.37\.ssm_beta\.weight=F32 +blk\.37\.ssm_out\.weight=Q8_0 +blk\.38\.attn_gate\.weight=Q8_0 +blk\.38\.attn_qkv\.weight=Q8_0 +blk\.38\.ffn_down_exps\.weight=Q5_K +blk\.38\.ffn_down_shexp\.weight=Q8_0 +blk\.38\.ffn_gate_exps\.weight=Q5_K +blk\.38\.ffn_gate_shexp\.weight=Q8_0 +blk\.38\.ffn_up_exps\.weight=Q5_K +blk\.38\.ffn_up_shexp\.weight=Q8_0 +blk\.38\.ssm_alpha\.weight=F32 +blk\.38\.ssm_beta\.weight=F32 +blk\.38\.ssm_out\.weight=Q8_0 +blk\.39\.attn_k\.weight=F16 +blk\.39\.attn_output\.weight=Q8_0 +blk\.39\.attn_q\.weight=Q8_0 +blk\.39\.attn_v\.weight=F16 +blk\.39\.ffn_down_exps\.weight=Q4_K +blk\.39\.ffn_down_shexp\.weight=Q8_0 +blk\.39\.ffn_gate_exps\.weight=Q4_K +blk\.39\.ffn_gate_shexp\.weight=Q8_0 +blk\.39\.ffn_up_exps\.weight=Q4_K +blk\.39\.ffn_up_shexp\.weight=Q8_0 +blk\.4\.attn_gate\.weight=Q8_0 +blk\.4\.attn_qkv\.weight=Q8_0 +blk\.4\.ffn_down_exps\.weight=Q4_K +blk\.4\.ffn_down_shexp\.weight=Q8_0 +blk\.4\.ffn_gate_exps\.weight=Q4_K +blk\.4\.ffn_gate_shexp\.weight=Q8_0 +blk\.4\.ffn_up_exps\.weight=Q4_K +blk\.4\.ffn_up_shexp\.weight=Q8_0 +blk\.4\.ssm_alpha\.weight=F32 +blk\.4\.ssm_beta\.weight=F32 +blk\.4\.ssm_out\.weight=Q8_0 +blk\.40\.attn_k\.weight=F16 +blk\.40\.attn_output\.weight=Q8_0 +blk\.40\.attn_q\.weight=Q8_0 +blk\.40\.attn_v\.weight=F16 +blk\.40\.ffn_down_exps\.weight=Q4_K +blk\.40\.ffn_down_shexp\.weight=Q8_0 +blk\.40\.ffn_gate_exps\.weight=Q4_K +blk\.40\.ffn_gate_shexp\.weight=Q8_0 +blk\.40\.ffn_up_exps\.weight=Q4_K +blk\.40\.ffn_up_shexp\.weight=Q8_0 +blk\.40\.nextn\.eh_proj\.weight=Q8_0 +blk\.5\.attn_gate\.weight=Q8_0 +blk\.5\.attn_qkv\.weight=Q8_0 +blk\.5\.ffn_down_exps\.weight=Q5_K +blk\.5\.ffn_down_shexp\.weight=Q8_0 +blk\.5\.ffn_gate_exps\.weight=Q5_K +blk\.5\.ffn_gate_shexp\.weight=Q8_0 +blk\.5\.ffn_up_exps\.weight=Q5_K +blk\.5\.ffn_up_shexp\.weight=Q8_0 +blk\.5\.ssm_alpha\.weight=F32 +blk\.5\.ssm_beta\.weight=F32 +blk\.5\.ssm_out\.weight=Q8_0 +blk\.6\.attn_gate\.weight=Q8_0 +blk\.6\.attn_qkv\.weight=Q8_0 +blk\.6\.ffn_down_exps\.weight=Q4_K +blk\.6\.ffn_down_shexp\.weight=Q8_0 +blk\.6\.ffn_gate_exps\.weight=Q4_K +blk\.6\.ffn_gate_shexp\.weight=Q8_0 +blk\.6\.ffn_up_exps\.weight=Q4_K +blk\.6\.ffn_up_shexp\.weight=Q8_0 +blk\.6\.ssm_alpha\.weight=F32 +blk\.6\.ssm_beta\.weight=F32 +blk\.6\.ssm_out\.weight=Q8_0 +blk\.7\.attn_k\.weight=F16 +blk\.7\.attn_output\.weight=Q8_0 +blk\.7\.attn_q\.weight=Q8_0 +blk\.7\.attn_v\.weight=F16 +blk\.7\.ffn_down_exps\.weight=Q4_K +blk\.7\.ffn_down_shexp\.weight=Q8_0 +blk\.7\.ffn_gate_exps\.weight=Q4_K +blk\.7\.ffn_gate_shexp\.weight=Q8_0 +blk\.7\.ffn_up_exps\.weight=Q4_K +blk\.7\.ffn_up_shexp\.weight=Q8_0 +blk\.8\.attn_gate\.weight=Q8_0 +blk\.8\.attn_qkv\.weight=Q8_0 +blk\.8\.ffn_down_exps\.weight=Q5_K +blk\.8\.ffn_down_shexp\.weight=Q8_0 +blk\.8\.ffn_gate_exps\.weight=Q5_K +blk\.8\.ffn_gate_shexp\.weight=Q8_0 +blk\.8\.ffn_up_exps\.weight=Q5_K +blk\.8\.ffn_up_shexp\.weight=Q8_0 +blk\.8\.ssm_alpha\.weight=F32 +blk\.8\.ssm_beta\.weight=F32 +blk\.8\.ssm_out\.weight=Q8_0 +blk\.9\.attn_gate\.weight=Q8_0 +blk\.9\.attn_qkv\.weight=Q8_0 +blk\.9\.ffn_down_exps\.weight=Q4_K +blk\.9\.ffn_down_shexp\.weight=Q8_0 +blk\.9\.ffn_gate_exps\.weight=Q4_K +blk\.9\.ffn_gate_shexp\.weight=Q8_0 +blk\.9\.ffn_up_exps\.weight=Q4_K +blk\.9\.ffn_up_shexp\.weight=Q8_0 +blk\.9\.ssm_alpha\.weight=F32 +blk\.9\.ssm_beta\.weight=F32 +blk\.9\.ssm_out\.weight=Q8_0 diff --git a/appendix/recipes/cfg_25G-ICE_ICEbase.txt b/appendix/recipes/cfg_25G-ICE_ICEbase.txt new file mode 100644 index 0000000000000000000000000000000000000000..0dbbdb115e4c18bbcffcaf545238ae6551fbc42a --- /dev/null +++ b/appendix/recipes/cfg_25G-ICE_ICEbase.txt @@ -0,0 +1,443 @@ +^output\.weight$=Q8_0 +token_embd\.weight=Q8_0 +blk\.0\.attn_gate\.weight=Q8_0 +blk\.0\.attn_qkv\.weight=Q8_0 +blk\.0\.ffn_down_exps\.weight=Q5_K +blk\.0\.ffn_down_shexp\.weight=Q8_0 +blk\.0\.ffn_gate_exps\.weight=Q5_K +blk\.0\.ffn_gate_shexp\.weight=Q8_0 +blk\.0\.ffn_up_exps\.weight=Q5_K +blk\.0\.ffn_up_shexp\.weight=Q8_0 +blk\.0\.ssm_alpha\.weight=F32 +blk\.0\.ssm_beta\.weight=F32 +blk\.0\.ssm_out\.weight=Q8_0 +blk\.1\.attn_gate\.weight=Q8_0 +blk\.1\.attn_qkv\.weight=Q8_0 +blk\.1\.ffn_down_exps\.weight=Q5_K +blk\.1\.ffn_down_shexp\.weight=Q8_0 +blk\.1\.ffn_gate_exps\.weight=Q5_K +blk\.1\.ffn_gate_shexp\.weight=Q8_0 +blk\.1\.ffn_up_exps\.weight=Q5_K +blk\.1\.ffn_up_shexp\.weight=Q8_0 +blk\.1\.ssm_alpha\.weight=F32 +blk\.1\.ssm_beta\.weight=F32 +blk\.1\.ssm_out\.weight=Q8_0 +blk\.10\.attn_gate\.weight=Q8_0 +blk\.10\.attn_qkv\.weight=Q8_0 +blk\.10\.ffn_down_exps\.weight=Q5_K +blk\.10\.ffn_down_shexp\.weight=Q8_0 +blk\.10\.ffn_gate_exps\.weight=Q5_K +blk\.10\.ffn_gate_shexp\.weight=Q8_0 +blk\.10\.ffn_up_exps\.weight=Q5_K +blk\.10\.ffn_up_shexp\.weight=Q8_0 +blk\.10\.ssm_alpha\.weight=F32 +blk\.10\.ssm_beta\.weight=F32 +blk\.10\.ssm_out\.weight=Q8_0 +blk\.11\.attn_k\.weight=F16 +blk\.11\.attn_output\.weight=Q8_0 +blk\.11\.attn_q\.weight=Q8_0 +blk\.11\.attn_v\.weight=F16 +blk\.11\.ffn_down_exps\.weight=Q5_K +blk\.11\.ffn_down_shexp\.weight=Q8_0 +blk\.11\.ffn_gate_exps\.weight=Q5_K +blk\.11\.ffn_gate_shexp\.weight=Q8_0 +blk\.11\.ffn_up_exps\.weight=Q5_K +blk\.11\.ffn_up_shexp\.weight=Q8_0 +blk\.12\.attn_gate\.weight=Q8_0 +blk\.12\.attn_qkv\.weight=Q8_0 +blk\.12\.ffn_down_exps\.weight=Q5_K +blk\.12\.ffn_down_shexp\.weight=Q8_0 +blk\.12\.ffn_gate_exps\.weight=Q5_K +blk\.12\.ffn_gate_shexp\.weight=Q8_0 +blk\.12\.ffn_up_exps\.weight=Q5_K +blk\.12\.ffn_up_shexp\.weight=Q8_0 +blk\.12\.ssm_alpha\.weight=F32 +blk\.12\.ssm_beta\.weight=F32 +blk\.12\.ssm_out\.weight=Q8_0 +blk\.13\.attn_gate\.weight=Q8_0 +blk\.13\.attn_qkv\.weight=Q8_0 +blk\.13\.ffn_down_exps\.weight=Q5_K +blk\.13\.ffn_down_shexp\.weight=Q8_0 +blk\.13\.ffn_gate_exps\.weight=Q5_K +blk\.13\.ffn_gate_shexp\.weight=Q8_0 +blk\.13\.ffn_up_exps\.weight=Q5_K +blk\.13\.ffn_up_shexp\.weight=Q8_0 +blk\.13\.ssm_alpha\.weight=F32 +blk\.13\.ssm_beta\.weight=F32 +blk\.13\.ssm_out\.weight=Q8_0 +blk\.14\.attn_gate\.weight=Q8_0 +blk\.14\.attn_qkv\.weight=Q8_0 +blk\.14\.ffn_down_exps\.weight=Q5_K +blk\.14\.ffn_down_shexp\.weight=Q8_0 +blk\.14\.ffn_gate_exps\.weight=Q5_K +blk\.14\.ffn_gate_shexp\.weight=Q8_0 +blk\.14\.ffn_up_exps\.weight=Q5_K +blk\.14\.ffn_up_shexp\.weight=Q8_0 +blk\.14\.ssm_alpha\.weight=F32 +blk\.14\.ssm_beta\.weight=F32 +blk\.14\.ssm_out\.weight=Q8_0 +blk\.15\.attn_k\.weight=F16 +blk\.15\.attn_output\.weight=Q8_0 +blk\.15\.attn_q\.weight=Q8_0 +blk\.15\.attn_v\.weight=F16 +blk\.15\.ffn_down_exps\.weight=Q5_K +blk\.15\.ffn_down_shexp\.weight=Q8_0 +blk\.15\.ffn_gate_exps\.weight=Q5_K +blk\.15\.ffn_gate_shexp\.weight=Q8_0 +blk\.15\.ffn_up_exps\.weight=Q5_K +blk\.15\.ffn_up_shexp\.weight=Q8_0 +blk\.16\.attn_gate\.weight=Q8_0 +blk\.16\.attn_qkv\.weight=Q8_0 +blk\.16\.ffn_down_exps\.weight=Q5_K +blk\.16\.ffn_down_shexp\.weight=Q8_0 +blk\.16\.ffn_gate_exps\.weight=Q5_K +blk\.16\.ffn_gate_shexp\.weight=Q8_0 +blk\.16\.ffn_up_exps\.weight=Q5_K +blk\.16\.ffn_up_shexp\.weight=Q8_0 +blk\.16\.ssm_alpha\.weight=F32 +blk\.16\.ssm_beta\.weight=F32 +blk\.16\.ssm_out\.weight=Q8_0 +blk\.17\.attn_gate\.weight=Q8_0 +blk\.17\.attn_qkv\.weight=Q8_0 +blk\.17\.ffn_down_exps\.weight=Q5_K +blk\.17\.ffn_down_shexp\.weight=Q8_0 +blk\.17\.ffn_gate_exps\.weight=Q5_K +blk\.17\.ffn_gate_shexp\.weight=Q8_0 +blk\.17\.ffn_up_exps\.weight=Q5_K +blk\.17\.ffn_up_shexp\.weight=Q8_0 +blk\.17\.ssm_alpha\.weight=F32 +blk\.17\.ssm_beta\.weight=F32 +blk\.17\.ssm_out\.weight=Q8_0 +blk\.18\.attn_gate\.weight=Q8_0 +blk\.18\.attn_qkv\.weight=Q8_0 +blk\.18\.ffn_down_exps\.weight=Q5_K +blk\.18\.ffn_down_shexp\.weight=Q8_0 +blk\.18\.ffn_gate_exps\.weight=Q5_K +blk\.18\.ffn_gate_shexp\.weight=Q8_0 +blk\.18\.ffn_up_exps\.weight=Q5_K +blk\.18\.ffn_up_shexp\.weight=Q8_0 +blk\.18\.ssm_alpha\.weight=F32 +blk\.18\.ssm_beta\.weight=F32 +blk\.18\.ssm_out\.weight=Q8_0 +blk\.19\.attn_k\.weight=F16 +blk\.19\.attn_output\.weight=Q8_0 +blk\.19\.attn_q\.weight=Q8_0 +blk\.19\.attn_v\.weight=F16 +blk\.19\.ffn_down_exps\.weight=Q5_K +blk\.19\.ffn_down_shexp\.weight=Q8_0 +blk\.19\.ffn_gate_exps\.weight=Q5_K +blk\.19\.ffn_gate_shexp\.weight=Q8_0 +blk\.19\.ffn_up_exps\.weight=Q5_K +blk\.19\.ffn_up_shexp\.weight=Q8_0 +blk\.2\.attn_gate\.weight=Q8_0 +blk\.2\.attn_qkv\.weight=Q8_0 +blk\.2\.ffn_down_exps\.weight=Q5_K +blk\.2\.ffn_down_shexp\.weight=Q8_0 +blk\.2\.ffn_gate_exps\.weight=Q5_K +blk\.2\.ffn_gate_shexp\.weight=Q8_0 +blk\.2\.ffn_up_exps\.weight=Q5_K +blk\.2\.ffn_up_shexp\.weight=Q8_0 +blk\.2\.ssm_alpha\.weight=F32 +blk\.2\.ssm_beta\.weight=F32 +blk\.2\.ssm_out\.weight=Q8_0 +blk\.20\.attn_gate\.weight=Q8_0 +blk\.20\.attn_qkv\.weight=Q8_0 +blk\.20\.ffn_down_exps\.weight=Q5_K +blk\.20\.ffn_down_shexp\.weight=Q8_0 +blk\.20\.ffn_gate_exps\.weight=Q5_K +blk\.20\.ffn_gate_shexp\.weight=Q8_0 +blk\.20\.ffn_up_exps\.weight=Q5_K +blk\.20\.ffn_up_shexp\.weight=Q8_0 +blk\.20\.ssm_alpha\.weight=F32 +blk\.20\.ssm_beta\.weight=F32 +blk\.20\.ssm_out\.weight=Q8_0 +blk\.21\.attn_gate\.weight=Q8_0 +blk\.21\.attn_qkv\.weight=Q8_0 +blk\.21\.ffn_down_exps\.weight=Q5_K +blk\.21\.ffn_down_shexp\.weight=Q8_0 +blk\.21\.ffn_gate_exps\.weight=Q5_K +blk\.21\.ffn_gate_shexp\.weight=Q8_0 +blk\.21\.ffn_up_exps\.weight=Q5_K +blk\.21\.ffn_up_shexp\.weight=Q8_0 +blk\.21\.ssm_alpha\.weight=F32 +blk\.21\.ssm_beta\.weight=F32 +blk\.21\.ssm_out\.weight=Q8_0 +blk\.22\.attn_gate\.weight=Q8_0 +blk\.22\.attn_qkv\.weight=Q8_0 +blk\.22\.ffn_down_exps\.weight=Q5_K +blk\.22\.ffn_down_shexp\.weight=Q8_0 +blk\.22\.ffn_gate_exps\.weight=Q5_K +blk\.22\.ffn_gate_shexp\.weight=Q8_0 +blk\.22\.ffn_up_exps\.weight=Q5_K +blk\.22\.ffn_up_shexp\.weight=Q8_0 +blk\.22\.ssm_alpha\.weight=F32 +blk\.22\.ssm_beta\.weight=F32 +blk\.22\.ssm_out\.weight=Q8_0 +blk\.23\.attn_k\.weight=F16 +blk\.23\.attn_output\.weight=Q8_0 +blk\.23\.attn_q\.weight=Q8_0 +blk\.23\.attn_v\.weight=F16 +blk\.23\.ffn_down_exps\.weight=Q5_K +blk\.23\.ffn_down_shexp\.weight=Q8_0 +blk\.23\.ffn_gate_exps\.weight=Q5_K +blk\.23\.ffn_gate_shexp\.weight=Q8_0 +blk\.23\.ffn_up_exps\.weight=Q5_K +blk\.23\.ffn_up_shexp\.weight=Q8_0 +blk\.24\.attn_gate\.weight=Q8_0 +blk\.24\.attn_qkv\.weight=Q8_0 +blk\.24\.ffn_down_exps\.weight=Q5_K +blk\.24\.ffn_down_shexp\.weight=Q8_0 +blk\.24\.ffn_gate_exps\.weight=Q5_K +blk\.24\.ffn_gate_shexp\.weight=Q8_0 +blk\.24\.ffn_up_exps\.weight=Q5_K +blk\.24\.ffn_up_shexp\.weight=Q8_0 +blk\.24\.ssm_alpha\.weight=F32 +blk\.24\.ssm_beta\.weight=F32 +blk\.24\.ssm_out\.weight=Q8_0 +blk\.25\.attn_gate\.weight=Q8_0 +blk\.25\.attn_qkv\.weight=Q8_0 +blk\.25\.ffn_down_exps\.weight=Q5_K +blk\.25\.ffn_down_shexp\.weight=Q8_0 +blk\.25\.ffn_gate_exps\.weight=Q5_K +blk\.25\.ffn_gate_shexp\.weight=Q8_0 +blk\.25\.ffn_up_exps\.weight=Q5_K +blk\.25\.ffn_up_shexp\.weight=Q8_0 +blk\.25\.ssm_alpha\.weight=F32 +blk\.25\.ssm_beta\.weight=F32 +blk\.25\.ssm_out\.weight=Q8_0 +blk\.26\.attn_gate\.weight=Q8_0 +blk\.26\.attn_qkv\.weight=Q8_0 +blk\.26\.ffn_down_exps\.weight=Q5_K +blk\.26\.ffn_down_shexp\.weight=Q8_0 +blk\.26\.ffn_gate_exps\.weight=Q5_K +blk\.26\.ffn_gate_shexp\.weight=Q8_0 +blk\.26\.ffn_up_exps\.weight=Q5_K +blk\.26\.ffn_up_shexp\.weight=Q8_0 +blk\.26\.ssm_alpha\.weight=F32 +blk\.26\.ssm_beta\.weight=F32 +blk\.26\.ssm_out\.weight=Q8_0 +blk\.27\.attn_k\.weight=F16 +blk\.27\.attn_output\.weight=Q8_0 +blk\.27\.attn_q\.weight=Q8_0 +blk\.27\.attn_v\.weight=F16 +blk\.27\.ffn_down_exps\.weight=Q5_K +blk\.27\.ffn_down_shexp\.weight=Q8_0 +blk\.27\.ffn_gate_exps\.weight=Q5_K +blk\.27\.ffn_gate_shexp\.weight=Q8_0 +blk\.27\.ffn_up_exps\.weight=Q5_K +blk\.27\.ffn_up_shexp\.weight=Q8_0 +blk\.28\.attn_gate\.weight=Q8_0 +blk\.28\.attn_qkv\.weight=Q8_0 +blk\.28\.ffn_down_exps\.weight=Q5_K +blk\.28\.ffn_down_shexp\.weight=Q8_0 +blk\.28\.ffn_gate_exps\.weight=Q5_K +blk\.28\.ffn_gate_shexp\.weight=Q8_0 +blk\.28\.ffn_up_exps\.weight=Q5_K +blk\.28\.ffn_up_shexp\.weight=Q8_0 +blk\.28\.ssm_alpha\.weight=F32 +blk\.28\.ssm_beta\.weight=F32 +blk\.28\.ssm_out\.weight=Q8_0 +blk\.29\.attn_gate\.weight=Q8_0 +blk\.29\.attn_qkv\.weight=Q8_0 +blk\.29\.ffn_down_exps\.weight=Q5_K +blk\.29\.ffn_down_shexp\.weight=Q8_0 +blk\.29\.ffn_gate_exps\.weight=Q5_K +blk\.29\.ffn_gate_shexp\.weight=Q8_0 +blk\.29\.ffn_up_exps\.weight=Q5_K +blk\.29\.ffn_up_shexp\.weight=Q8_0 +blk\.29\.ssm_alpha\.weight=F32 +blk\.29\.ssm_beta\.weight=F32 +blk\.29\.ssm_out\.weight=Q8_0 +blk\.3\.attn_k\.weight=F16 +blk\.3\.attn_output\.weight=Q8_0 +blk\.3\.attn_q\.weight=Q8_0 +blk\.3\.attn_v\.weight=F16 +blk\.3\.ffn_down_exps\.weight=Q5_K +blk\.3\.ffn_down_shexp\.weight=Q8_0 +blk\.3\.ffn_gate_exps\.weight=Q5_K +blk\.3\.ffn_gate_shexp\.weight=Q8_0 +blk\.3\.ffn_up_exps\.weight=Q5_K +blk\.3\.ffn_up_shexp\.weight=Q8_0 +blk\.30\.attn_gate\.weight=Q8_0 +blk\.30\.attn_qkv\.weight=Q8_0 +blk\.30\.ffn_down_exps\.weight=Q5_K +blk\.30\.ffn_down_shexp\.weight=Q8_0 +blk\.30\.ffn_gate_exps\.weight=Q5_K +blk\.30\.ffn_gate_shexp\.weight=Q8_0 +blk\.30\.ffn_up_exps\.weight=Q5_K +blk\.30\.ffn_up_shexp\.weight=Q8_0 +blk\.30\.ssm_alpha\.weight=F32 +blk\.30\.ssm_beta\.weight=F32 +blk\.30\.ssm_out\.weight=Q8_0 +blk\.31\.attn_k\.weight=F16 +blk\.31\.attn_output\.weight=Q8_0 +blk\.31\.attn_q\.weight=Q8_0 +blk\.31\.attn_v\.weight=F16 +blk\.31\.ffn_down_exps\.weight=Q5_K +blk\.31\.ffn_down_shexp\.weight=Q8_0 +blk\.31\.ffn_gate_exps\.weight=Q5_K +blk\.31\.ffn_gate_shexp\.weight=Q8_0 +blk\.31\.ffn_up_exps\.weight=Q5_K +blk\.31\.ffn_up_shexp\.weight=Q8_0 +blk\.32\.attn_gate\.weight=Q8_0 +blk\.32\.attn_qkv\.weight=Q8_0 +blk\.32\.ffn_down_exps\.weight=Q5_K +blk\.32\.ffn_down_shexp\.weight=Q8_0 +blk\.32\.ffn_gate_exps\.weight=Q5_K +blk\.32\.ffn_gate_shexp\.weight=Q8_0 +blk\.32\.ffn_up_exps\.weight=Q5_K +blk\.32\.ffn_up_shexp\.weight=Q8_0 +blk\.32\.ssm_alpha\.weight=F32 +blk\.32\.ssm_beta\.weight=F32 +blk\.32\.ssm_out\.weight=Q8_0 +blk\.33\.attn_gate\.weight=Q8_0 +blk\.33\.attn_qkv\.weight=Q8_0 +blk\.33\.ffn_down_exps\.weight=Q5_K +blk\.33\.ffn_down_shexp\.weight=Q8_0 +blk\.33\.ffn_gate_exps\.weight=Q5_K +blk\.33\.ffn_gate_shexp\.weight=Q8_0 +blk\.33\.ffn_up_exps\.weight=Q5_K +blk\.33\.ffn_up_shexp\.weight=Q8_0 +blk\.33\.ssm_alpha\.weight=F32 +blk\.33\.ssm_beta\.weight=F32 +blk\.33\.ssm_out\.weight=Q8_0 +blk\.34\.attn_gate\.weight=Q8_0 +blk\.34\.attn_qkv\.weight=Q8_0 +blk\.34\.ffn_down_exps\.weight=Q4_K +blk\.34\.ffn_down_shexp\.weight=Q8_0 +blk\.34\.ffn_gate_exps\.weight=Q4_K +blk\.34\.ffn_gate_shexp\.weight=Q8_0 +blk\.34\.ffn_up_exps\.weight=Q4_K +blk\.34\.ffn_up_shexp\.weight=Q8_0 +blk\.34\.ssm_alpha\.weight=F32 +blk\.34\.ssm_beta\.weight=F32 +blk\.34\.ssm_out\.weight=Q8_0 +blk\.35\.attn_k\.weight=F16 +blk\.35\.attn_output\.weight=Q8_0 +blk\.35\.attn_q\.weight=Q8_0 +blk\.35\.attn_v\.weight=F16 +blk\.35\.ffn_down_exps\.weight=Q4_K +blk\.35\.ffn_down_shexp\.weight=Q8_0 +blk\.35\.ffn_gate_exps\.weight=Q4_K +blk\.35\.ffn_gate_shexp\.weight=Q8_0 +blk\.35\.ffn_up_exps\.weight=Q4_K +blk\.35\.ffn_up_shexp\.weight=Q8_0 +blk\.36\.attn_gate\.weight=Q8_0 +blk\.36\.attn_qkv\.weight=Q8_0 +blk\.36\.ffn_down_exps\.weight=Q4_K +blk\.36\.ffn_down_shexp\.weight=Q8_0 +blk\.36\.ffn_gate_exps\.weight=Q4_K +blk\.36\.ffn_gate_shexp\.weight=Q8_0 +blk\.36\.ffn_up_exps\.weight=Q4_K +blk\.36\.ffn_up_shexp\.weight=Q8_0 +blk\.36\.ssm_alpha\.weight=F32 +blk\.36\.ssm_beta\.weight=F32 +blk\.36\.ssm_out\.weight=Q8_0 +blk\.37\.attn_gate\.weight=Q8_0 +blk\.37\.attn_qkv\.weight=Q8_0 +blk\.37\.ffn_down_exps\.weight=Q4_K +blk\.37\.ffn_down_shexp\.weight=Q8_0 +blk\.37\.ffn_gate_exps\.weight=Q4_K +blk\.37\.ffn_gate_shexp\.weight=Q8_0 +blk\.37\.ffn_up_exps\.weight=Q4_K +blk\.37\.ffn_up_shexp\.weight=Q8_0 +blk\.37\.ssm_alpha\.weight=F32 +blk\.37\.ssm_beta\.weight=F32 +blk\.37\.ssm_out\.weight=Q8_0 +blk\.38\.attn_gate\.weight=Q8_0 +blk\.38\.attn_qkv\.weight=Q8_0 +blk\.38\.ffn_down_exps\.weight=Q4_K +blk\.38\.ffn_down_shexp\.weight=Q8_0 +blk\.38\.ffn_gate_exps\.weight=Q4_K +blk\.38\.ffn_gate_shexp\.weight=Q8_0 +blk\.38\.ffn_up_exps\.weight=Q4_K +blk\.38\.ffn_up_shexp\.weight=Q8_0 +blk\.38\.ssm_alpha\.weight=F32 +blk\.38\.ssm_beta\.weight=F32 +blk\.38\.ssm_out\.weight=Q8_0 +blk\.39\.attn_k\.weight=F16 +blk\.39\.attn_output\.weight=Q8_0 +blk\.39\.attn_q\.weight=Q8_0 +blk\.39\.attn_v\.weight=F16 +blk\.39\.ffn_down_exps\.weight=Q4_K +blk\.39\.ffn_down_shexp\.weight=Q8_0 +blk\.39\.ffn_gate_exps\.weight=Q4_K +blk\.39\.ffn_gate_shexp\.weight=Q8_0 +blk\.39\.ffn_up_exps\.weight=Q4_K +blk\.39\.ffn_up_shexp\.weight=Q8_0 +blk\.4\.attn_gate\.weight=Q8_0 +blk\.4\.attn_qkv\.weight=Q8_0 +blk\.4\.ffn_down_exps\.weight=Q5_K +blk\.4\.ffn_down_shexp\.weight=Q8_0 +blk\.4\.ffn_gate_exps\.weight=Q5_K +blk\.4\.ffn_gate_shexp\.weight=Q8_0 +blk\.4\.ffn_up_exps\.weight=Q5_K +blk\.4\.ffn_up_shexp\.weight=Q8_0 +blk\.4\.ssm_alpha\.weight=F32 +blk\.4\.ssm_beta\.weight=F32 +blk\.4\.ssm_out\.weight=Q8_0 +blk\.40\.attn_k\.weight=F16 +blk\.40\.attn_output\.weight=Q8_0 +blk\.40\.attn_q\.weight=Q8_0 +blk\.40\.attn_v\.weight=F16 +blk\.40\.ffn_down_exps\.weight=Q5_K +blk\.40\.ffn_down_shexp\.weight=Q8_0 +blk\.40\.ffn_gate_exps\.weight=Q5_K +blk\.40\.ffn_gate_shexp\.weight=Q8_0 +blk\.40\.ffn_up_exps\.weight=Q5_K +blk\.40\.ffn_up_shexp\.weight=Q8_0 +blk\.40\.nextn\.eh_proj\.weight=Q8_0 +blk\.5\.attn_gate\.weight=Q8_0 +blk\.5\.attn_qkv\.weight=Q8_0 +blk\.5\.ffn_down_exps\.weight=Q5_K +blk\.5\.ffn_down_shexp\.weight=Q8_0 +blk\.5\.ffn_gate_exps\.weight=Q5_K +blk\.5\.ffn_gate_shexp\.weight=Q8_0 +blk\.5\.ffn_up_exps\.weight=Q5_K +blk\.5\.ffn_up_shexp\.weight=Q8_0 +blk\.5\.ssm_alpha\.weight=F32 +blk\.5\.ssm_beta\.weight=F32 +blk\.5\.ssm_out\.weight=Q8_0 +blk\.6\.attn_gate\.weight=Q8_0 +blk\.6\.attn_qkv\.weight=Q8_0 +blk\.6\.ffn_down_exps\.weight=Q5_K +blk\.6\.ffn_down_shexp\.weight=Q8_0 +blk\.6\.ffn_gate_exps\.weight=Q5_K +blk\.6\.ffn_gate_shexp\.weight=Q8_0 +blk\.6\.ffn_up_exps\.weight=Q5_K +blk\.6\.ffn_up_shexp\.weight=Q8_0 +blk\.6\.ssm_alpha\.weight=F32 +blk\.6\.ssm_beta\.weight=F32 +blk\.6\.ssm_out\.weight=Q8_0 +blk\.7\.attn_k\.weight=F16 +blk\.7\.attn_output\.weight=Q8_0 +blk\.7\.attn_q\.weight=Q8_0 +blk\.7\.attn_v\.weight=F16 +blk\.7\.ffn_down_exps\.weight=Q5_K +blk\.7\.ffn_down_shexp\.weight=Q8_0 +blk\.7\.ffn_gate_exps\.weight=Q5_K +blk\.7\.ffn_gate_shexp\.weight=Q8_0 +blk\.7\.ffn_up_exps\.weight=Q5_K +blk\.7\.ffn_up_shexp\.weight=Q8_0 +blk\.8\.attn_gate\.weight=Q8_0 +blk\.8\.attn_qkv\.weight=Q8_0 +blk\.8\.ffn_down_exps\.weight=Q5_K +blk\.8\.ffn_down_shexp\.weight=Q8_0 +blk\.8\.ffn_gate_exps\.weight=Q5_K +blk\.8\.ffn_gate_shexp\.weight=Q8_0 +blk\.8\.ffn_up_exps\.weight=Q5_K +blk\.8\.ffn_up_shexp\.weight=Q8_0 +blk\.8\.ssm_alpha\.weight=F32 +blk\.8\.ssm_beta\.weight=F32 +blk\.8\.ssm_out\.weight=Q8_0 +blk\.9\.attn_gate\.weight=Q8_0 +blk\.9\.attn_qkv\.weight=Q8_0 +blk\.9\.ffn_down_exps\.weight=Q5_K +blk\.9\.ffn_down_shexp\.weight=Q8_0 +blk\.9\.ffn_gate_exps\.weight=Q5_K +blk\.9\.ffn_gate_shexp\.weight=Q8_0 +blk\.9\.ffn_up_exps\.weight=Q5_K +blk\.9\.ffn_up_shexp\.weight=Q8_0 +blk\.9\.ssm_alpha\.weight=F32 +blk\.9\.ssm_beta\.weight=F32 +blk\.9\.ssm_out\.weight=Q8_0 diff --git a/appendix/recipes/cfg_25G-ICE_previous.txt b/appendix/recipes/cfg_25G-ICE_previous.txt new file mode 100644 index 0000000000000000000000000000000000000000..8fa67b888e87c36c1f9f0e8b9fcc475548b0209a --- /dev/null +++ b/appendix/recipes/cfg_25G-ICE_previous.txt @@ -0,0 +1,443 @@ +^output\.weight$=Q8_0 +token_embd\.weight=Q8_0 +blk\.0\.attn_gate\.weight=Q8_0 +blk\.0\.attn_qkv\.weight=Q8_0 +blk\.0\.ffn_down_exps\.weight=Q5_K +blk\.0\.ffn_down_shexp\.weight=Q8_0 +blk\.0\.ffn_gate_exps\.weight=Q5_K +blk\.0\.ffn_gate_shexp\.weight=Q8_0 +blk\.0\.ffn_up_exps\.weight=Q5_K +blk\.0\.ffn_up_shexp\.weight=Q8_0 +blk\.0\.ssm_alpha\.weight=F32 +blk\.0\.ssm_beta\.weight=F32 +blk\.0\.ssm_out\.weight=Q8_0 +blk\.1\.attn_gate\.weight=Q8_0 +blk\.1\.attn_qkv\.weight=Q8_0 +blk\.1\.ffn_down_exps\.weight=Q5_K +blk\.1\.ffn_down_shexp\.weight=Q8_0 +blk\.1\.ffn_gate_exps\.weight=Q5_K +blk\.1\.ffn_gate_shexp\.weight=Q8_0 +blk\.1\.ffn_up_exps\.weight=Q5_K +blk\.1\.ffn_up_shexp\.weight=Q8_0 +blk\.1\.ssm_alpha\.weight=F32 +blk\.1\.ssm_beta\.weight=F32 +blk\.1\.ssm_out\.weight=Q8_0 +blk\.10\.attn_gate\.weight=Q8_0 +blk\.10\.attn_qkv\.weight=Q8_0 +blk\.10\.ffn_down_exps\.weight=Q4_K +blk\.10\.ffn_down_shexp\.weight=Q8_0 +blk\.10\.ffn_gate_exps\.weight=Q4_K +blk\.10\.ffn_gate_shexp\.weight=Q8_0 +blk\.10\.ffn_up_exps\.weight=Q4_K +blk\.10\.ffn_up_shexp\.weight=Q8_0 +blk\.10\.ssm_alpha\.weight=F32 +blk\.10\.ssm_beta\.weight=F32 +blk\.10\.ssm_out\.weight=Q8_0 +blk\.11\.attn_k\.weight=F16 +blk\.11\.attn_output\.weight=Q8_0 +blk\.11\.attn_q\.weight=Q8_0 +blk\.11\.attn_v\.weight=F16 +blk\.11\.ffn_down_exps\.weight=Q5_K +blk\.11\.ffn_down_shexp\.weight=Q8_0 +blk\.11\.ffn_gate_exps\.weight=Q5_K +blk\.11\.ffn_gate_shexp\.weight=Q8_0 +blk\.11\.ffn_up_exps\.weight=Q5_K +blk\.11\.ffn_up_shexp\.weight=Q8_0 +blk\.12\.attn_gate\.weight=Q8_0 +blk\.12\.attn_qkv\.weight=Q8_0 +blk\.12\.ffn_down_exps\.weight=Q5_K +blk\.12\.ffn_down_shexp\.weight=Q8_0 +blk\.12\.ffn_gate_exps\.weight=Q5_K +blk\.12\.ffn_gate_shexp\.weight=Q8_0 +blk\.12\.ffn_up_exps\.weight=Q5_K +blk\.12\.ffn_up_shexp\.weight=Q8_0 +blk\.12\.ssm_alpha\.weight=F32 +blk\.12\.ssm_beta\.weight=F32 +blk\.12\.ssm_out\.weight=Q8_0 +blk\.13\.attn_gate\.weight=Q8_0 +blk\.13\.attn_qkv\.weight=Q8_0 +blk\.13\.ffn_down_exps\.weight=Q5_K +blk\.13\.ffn_down_shexp\.weight=Q8_0 +blk\.13\.ffn_gate_exps\.weight=Q5_K +blk\.13\.ffn_gate_shexp\.weight=Q8_0 +blk\.13\.ffn_up_exps\.weight=Q5_K +blk\.13\.ffn_up_shexp\.weight=Q8_0 +blk\.13\.ssm_alpha\.weight=F32 +blk\.13\.ssm_beta\.weight=F32 +blk\.13\.ssm_out\.weight=Q8_0 +blk\.14\.attn_gate\.weight=Q8_0 +blk\.14\.attn_qkv\.weight=Q8_0 +blk\.14\.ffn_down_exps\.weight=Q5_K +blk\.14\.ffn_down_shexp\.weight=Q8_0 +blk\.14\.ffn_gate_exps\.weight=Q5_K +blk\.14\.ffn_gate_shexp\.weight=Q8_0 +blk\.14\.ffn_up_exps\.weight=Q5_K +blk\.14\.ffn_up_shexp\.weight=Q8_0 +blk\.14\.ssm_alpha\.weight=F32 +blk\.14\.ssm_beta\.weight=F32 +blk\.14\.ssm_out\.weight=Q8_0 +blk\.15\.attn_k\.weight=F16 +blk\.15\.attn_output\.weight=Q8_0 +blk\.15\.attn_q\.weight=Q8_0 +blk\.15\.attn_v\.weight=F16 +blk\.15\.ffn_down_exps\.weight=Q5_K +blk\.15\.ffn_down_shexp\.weight=Q8_0 +blk\.15\.ffn_gate_exps\.weight=Q5_K +blk\.15\.ffn_gate_shexp\.weight=Q8_0 +blk\.15\.ffn_up_exps\.weight=Q5_K +blk\.15\.ffn_up_shexp\.weight=Q8_0 +blk\.16\.attn_gate\.weight=Q8_0 +blk\.16\.attn_qkv\.weight=Q8_0 +blk\.16\.ffn_down_exps\.weight=Q5_K +blk\.16\.ffn_down_shexp\.weight=Q8_0 +blk\.16\.ffn_gate_exps\.weight=Q5_K +blk\.16\.ffn_gate_shexp\.weight=Q8_0 +blk\.16\.ffn_up_exps\.weight=Q5_K +blk\.16\.ffn_up_shexp\.weight=Q8_0 +blk\.16\.ssm_alpha\.weight=F32 +blk\.16\.ssm_beta\.weight=F32 +blk\.16\.ssm_out\.weight=Q8_0 +blk\.17\.attn_gate\.weight=Q8_0 +blk\.17\.attn_qkv\.weight=Q8_0 +blk\.17\.ffn_down_exps\.weight=Q4_K +blk\.17\.ffn_down_shexp\.weight=Q8_0 +blk\.17\.ffn_gate_exps\.weight=Q4_K +blk\.17\.ffn_gate_shexp\.weight=Q8_0 +blk\.17\.ffn_up_exps\.weight=Q4_K +blk\.17\.ffn_up_shexp\.weight=Q8_0 +blk\.17\.ssm_alpha\.weight=F32 +blk\.17\.ssm_beta\.weight=F32 +blk\.17\.ssm_out\.weight=Q8_0 +blk\.18\.attn_gate\.weight=Q8_0 +blk\.18\.attn_qkv\.weight=Q8_0 +blk\.18\.ffn_down_exps\.weight=Q5_K +blk\.18\.ffn_down_shexp\.weight=Q8_0 +blk\.18\.ffn_gate_exps\.weight=Q5_K +blk\.18\.ffn_gate_shexp\.weight=Q8_0 +blk\.18\.ffn_up_exps\.weight=Q5_K +blk\.18\.ffn_up_shexp\.weight=Q8_0 +blk\.18\.ssm_alpha\.weight=F32 +blk\.18\.ssm_beta\.weight=F32 +blk\.18\.ssm_out\.weight=Q8_0 +blk\.19\.attn_k\.weight=F16 +blk\.19\.attn_output\.weight=Q8_0 +blk\.19\.attn_q\.weight=Q8_0 +blk\.19\.attn_v\.weight=F16 +blk\.19\.ffn_down_exps\.weight=Q5_K +blk\.19\.ffn_down_shexp\.weight=Q8_0 +blk\.19\.ffn_gate_exps\.weight=Q5_K +blk\.19\.ffn_gate_shexp\.weight=Q8_0 +blk\.19\.ffn_up_exps\.weight=Q5_K +blk\.19\.ffn_up_shexp\.weight=Q8_0 +blk\.2\.attn_gate\.weight=Q8_0 +blk\.2\.attn_qkv\.weight=Q8_0 +blk\.2\.ffn_down_exps\.weight=Q5_K +blk\.2\.ffn_down_shexp\.weight=Q8_0 +blk\.2\.ffn_gate_exps\.weight=Q5_K +blk\.2\.ffn_gate_shexp\.weight=Q8_0 +blk\.2\.ffn_up_exps\.weight=Q5_K +blk\.2\.ffn_up_shexp\.weight=Q8_0 +blk\.2\.ssm_alpha\.weight=F32 +blk\.2\.ssm_beta\.weight=F32 +blk\.2\.ssm_out\.weight=Q8_0 +blk\.20\.attn_gate\.weight=Q8_0 +blk\.20\.attn_qkv\.weight=Q8_0 +blk\.20\.ffn_down_exps\.weight=Q5_K +blk\.20\.ffn_down_shexp\.weight=Q8_0 +blk\.20\.ffn_gate_exps\.weight=Q5_K +blk\.20\.ffn_gate_shexp\.weight=Q8_0 +blk\.20\.ffn_up_exps\.weight=Q5_K +blk\.20\.ffn_up_shexp\.weight=Q8_0 +blk\.20\.ssm_alpha\.weight=F32 +blk\.20\.ssm_beta\.weight=F32 +blk\.20\.ssm_out\.weight=Q8_0 +blk\.21\.attn_gate\.weight=Q8_0 +blk\.21\.attn_qkv\.weight=Q8_0 +blk\.21\.ffn_down_exps\.weight=Q5_K +blk\.21\.ffn_down_shexp\.weight=Q8_0 +blk\.21\.ffn_gate_exps\.weight=Q5_K +blk\.21\.ffn_gate_shexp\.weight=Q8_0 +blk\.21\.ffn_up_exps\.weight=Q5_K +blk\.21\.ffn_up_shexp\.weight=Q8_0 +blk\.21\.ssm_alpha\.weight=F32 +blk\.21\.ssm_beta\.weight=F32 +blk\.21\.ssm_out\.weight=Q8_0 +blk\.22\.attn_gate\.weight=Q8_0 +blk\.22\.attn_qkv\.weight=Q8_0 +blk\.22\.ffn_down_exps\.weight=Q5_K +blk\.22\.ffn_down_shexp\.weight=Q8_0 +blk\.22\.ffn_gate_exps\.weight=Q5_K +blk\.22\.ffn_gate_shexp\.weight=Q8_0 +blk\.22\.ffn_up_exps\.weight=Q5_K +blk\.22\.ffn_up_shexp\.weight=Q8_0 +blk\.22\.ssm_alpha\.weight=F32 +blk\.22\.ssm_beta\.weight=F32 +blk\.22\.ssm_out\.weight=Q8_0 +blk\.23\.attn_k\.weight=F16 +blk\.23\.attn_output\.weight=Q8_0 +blk\.23\.attn_q\.weight=Q8_0 +blk\.23\.attn_v\.weight=F16 +blk\.23\.ffn_down_exps\.weight=Q5_K +blk\.23\.ffn_down_shexp\.weight=Q8_0 +blk\.23\.ffn_gate_exps\.weight=Q5_K +blk\.23\.ffn_gate_shexp\.weight=Q8_0 +blk\.23\.ffn_up_exps\.weight=Q5_K +blk\.23\.ffn_up_shexp\.weight=Q8_0 +blk\.24\.attn_gate\.weight=Q8_0 +blk\.24\.attn_qkv\.weight=Q8_0 +blk\.24\.ffn_down_exps\.weight=Q4_K +blk\.24\.ffn_down_shexp\.weight=Q8_0 +blk\.24\.ffn_gate_exps\.weight=Q4_K +blk\.24\.ffn_gate_shexp\.weight=Q8_0 +blk\.24\.ffn_up_exps\.weight=Q4_K +blk\.24\.ffn_up_shexp\.weight=Q8_0 +blk\.24\.ssm_alpha\.weight=F32 +blk\.24\.ssm_beta\.weight=F32 +blk\.24\.ssm_out\.weight=Q8_0 +blk\.25\.attn_gate\.weight=Q8_0 +blk\.25\.attn_qkv\.weight=Q8_0 +blk\.25\.ffn_down_exps\.weight=Q5_K +blk\.25\.ffn_down_shexp\.weight=Q8_0 +blk\.25\.ffn_gate_exps\.weight=Q5_K +blk\.25\.ffn_gate_shexp\.weight=Q8_0 +blk\.25\.ffn_up_exps\.weight=Q5_K +blk\.25\.ffn_up_shexp\.weight=Q8_0 +blk\.25\.ssm_alpha\.weight=F32 +blk\.25\.ssm_beta\.weight=F32 +blk\.25\.ssm_out\.weight=Q8_0 +blk\.26\.attn_gate\.weight=Q8_0 +blk\.26\.attn_qkv\.weight=Q8_0 +blk\.26\.ffn_down_exps\.weight=Q5_K +blk\.26\.ffn_down_shexp\.weight=Q8_0 +blk\.26\.ffn_gate_exps\.weight=Q5_K +blk\.26\.ffn_gate_shexp\.weight=Q8_0 +blk\.26\.ffn_up_exps\.weight=Q5_K +blk\.26\.ffn_up_shexp\.weight=Q8_0 +blk\.26\.ssm_alpha\.weight=F32 +blk\.26\.ssm_beta\.weight=F32 +blk\.26\.ssm_out\.weight=Q8_0 +blk\.27\.attn_k\.weight=F16 +blk\.27\.attn_output\.weight=Q8_0 +blk\.27\.attn_q\.weight=Q8_0 +blk\.27\.attn_v\.weight=F16 +blk\.27\.ffn_down_exps\.weight=Q5_K +blk\.27\.ffn_down_shexp\.weight=Q8_0 +blk\.27\.ffn_gate_exps\.weight=Q5_K +blk\.27\.ffn_gate_shexp\.weight=Q8_0 +blk\.27\.ffn_up_exps\.weight=Q5_K +blk\.27\.ffn_up_shexp\.weight=Q8_0 +blk\.28\.attn_gate\.weight=Q8_0 +blk\.28\.attn_qkv\.weight=Q8_0 +blk\.28\.ffn_down_exps\.weight=Q5_K +blk\.28\.ffn_down_shexp\.weight=Q8_0 +blk\.28\.ffn_gate_exps\.weight=Q5_K +blk\.28\.ffn_gate_shexp\.weight=Q8_0 +blk\.28\.ffn_up_exps\.weight=Q5_K +blk\.28\.ffn_up_shexp\.weight=Q8_0 +blk\.28\.ssm_alpha\.weight=F32 +blk\.28\.ssm_beta\.weight=F32 +blk\.28\.ssm_out\.weight=Q8_0 +blk\.29\.attn_gate\.weight=Q8_0 +blk\.29\.attn_qkv\.weight=Q8_0 +blk\.29\.ffn_down_exps\.weight=Q5_K +blk\.29\.ffn_down_shexp\.weight=Q8_0 +blk\.29\.ffn_gate_exps\.weight=Q5_K +blk\.29\.ffn_gate_shexp\.weight=Q8_0 +blk\.29\.ffn_up_exps\.weight=Q5_K +blk\.29\.ffn_up_shexp\.weight=Q8_0 +blk\.29\.ssm_alpha\.weight=F32 +blk\.29\.ssm_beta\.weight=F32 +blk\.29\.ssm_out\.weight=Q8_0 +blk\.3\.attn_k\.weight=F16 +blk\.3\.attn_output\.weight=Q8_0 +blk\.3\.attn_q\.weight=Q8_0 +blk\.3\.attn_v\.weight=F16 +blk\.3\.ffn_down_exps\.weight=Q4_K +blk\.3\.ffn_down_shexp\.weight=Q8_0 +blk\.3\.ffn_gate_exps\.weight=Q4_K +blk\.3\.ffn_gate_shexp\.weight=Q8_0 +blk\.3\.ffn_up_exps\.weight=Q4_K +blk\.3\.ffn_up_shexp\.weight=Q8_0 +blk\.30\.attn_gate\.weight=Q8_0 +blk\.30\.attn_qkv\.weight=Q8_0 +blk\.30\.ffn_down_exps\.weight=Q5_K +blk\.30\.ffn_down_shexp\.weight=Q8_0 +blk\.30\.ffn_gate_exps\.weight=Q5_K +blk\.30\.ffn_gate_shexp\.weight=Q8_0 +blk\.30\.ffn_up_exps\.weight=Q5_K +blk\.30\.ffn_up_shexp\.weight=Q8_0 +blk\.30\.ssm_alpha\.weight=F32 +blk\.30\.ssm_beta\.weight=F32 +blk\.30\.ssm_out\.weight=Q8_0 +blk\.31\.attn_k\.weight=F16 +blk\.31\.attn_output\.weight=Q8_0 +blk\.31\.attn_q\.weight=Q8_0 +blk\.31\.attn_v\.weight=F16 +blk\.31\.ffn_down_exps\.weight=Q4_K +blk\.31\.ffn_down_shexp\.weight=Q8_0 +blk\.31\.ffn_gate_exps\.weight=Q4_K +blk\.31\.ffn_gate_shexp\.weight=Q8_0 +blk\.31\.ffn_up_exps\.weight=Q4_K +blk\.31\.ffn_up_shexp\.weight=Q8_0 +blk\.32\.attn_gate\.weight=Q8_0 +blk\.32\.attn_qkv\.weight=Q8_0 +blk\.32\.ffn_down_exps\.weight=Q5_K +blk\.32\.ffn_down_shexp\.weight=Q8_0 +blk\.32\.ffn_gate_exps\.weight=Q5_K +blk\.32\.ffn_gate_shexp\.weight=Q8_0 +blk\.32\.ffn_up_exps\.weight=Q5_K +blk\.32\.ffn_up_shexp\.weight=Q8_0 +blk\.32\.ssm_alpha\.weight=F32 +blk\.32\.ssm_beta\.weight=F32 +blk\.32\.ssm_out\.weight=Q8_0 +blk\.33\.attn_gate\.weight=Q8_0 +blk\.33\.attn_qkv\.weight=Q8_0 +blk\.33\.ffn_down_exps\.weight=Q5_K +blk\.33\.ffn_down_shexp\.weight=Q8_0 +blk\.33\.ffn_gate_exps\.weight=Q5_K +blk\.33\.ffn_gate_shexp\.weight=Q8_0 +blk\.33\.ffn_up_exps\.weight=Q5_K +blk\.33\.ffn_up_shexp\.weight=Q8_0 +blk\.33\.ssm_alpha\.weight=F32 +blk\.33\.ssm_beta\.weight=F32 +blk\.33\.ssm_out\.weight=Q8_0 +blk\.34\.attn_gate\.weight=Q8_0 +blk\.34\.attn_qkv\.weight=Q8_0 +blk\.34\.ffn_down_exps\.weight=Q5_K +blk\.34\.ffn_down_shexp\.weight=Q8_0 +blk\.34\.ffn_gate_exps\.weight=Q5_K +blk\.34\.ffn_gate_shexp\.weight=Q8_0 +blk\.34\.ffn_up_exps\.weight=Q5_K +blk\.34\.ffn_up_shexp\.weight=Q8_0 +blk\.34\.ssm_alpha\.weight=F32 +blk\.34\.ssm_beta\.weight=F32 +blk\.34\.ssm_out\.weight=Q8_0 +blk\.35\.attn_k\.weight=F16 +blk\.35\.attn_output\.weight=Q8_0 +blk\.35\.attn_q\.weight=Q8_0 +blk\.35\.attn_v\.weight=F16 +blk\.35\.ffn_down_exps\.weight=Q5_K +blk\.35\.ffn_down_shexp\.weight=Q8_0 +blk\.35\.ffn_gate_exps\.weight=Q5_K +blk\.35\.ffn_gate_shexp\.weight=Q8_0 +blk\.35\.ffn_up_exps\.weight=Q5_K +blk\.35\.ffn_up_shexp\.weight=Q8_0 +blk\.36\.attn_gate\.weight=Q8_0 +blk\.36\.attn_qkv\.weight=Q8_0 +blk\.36\.ffn_down_exps\.weight=Q5_K +blk\.36\.ffn_down_shexp\.weight=Q8_0 +blk\.36\.ffn_gate_exps\.weight=Q5_K +blk\.36\.ffn_gate_shexp\.weight=Q8_0 +blk\.36\.ffn_up_exps\.weight=Q5_K +blk\.36\.ffn_up_shexp\.weight=Q8_0 +blk\.36\.ssm_alpha\.weight=F32 +blk\.36\.ssm_beta\.weight=F32 +blk\.36\.ssm_out\.weight=Q8_0 +blk\.37\.attn_gate\.weight=Q8_0 +blk\.37\.attn_qkv\.weight=Q8_0 +blk\.37\.ffn_down_exps\.weight=Q5_K +blk\.37\.ffn_down_shexp\.weight=Q8_0 +blk\.37\.ffn_gate_exps\.weight=Q5_K +blk\.37\.ffn_gate_shexp\.weight=Q8_0 +blk\.37\.ffn_up_exps\.weight=Q5_K +blk\.37\.ffn_up_shexp\.weight=Q8_0 +blk\.37\.ssm_alpha\.weight=F32 +blk\.37\.ssm_beta\.weight=F32 +blk\.37\.ssm_out\.weight=Q8_0 +blk\.38\.attn_gate\.weight=Q8_0 +blk\.38\.attn_qkv\.weight=Q8_0 +blk\.38\.ffn_down_exps\.weight=Q4_K +blk\.38\.ffn_down_shexp\.weight=Q8_0 +blk\.38\.ffn_gate_exps\.weight=Q4_K +blk\.38\.ffn_gate_shexp\.weight=Q8_0 +blk\.38\.ffn_up_exps\.weight=Q4_K +blk\.38\.ffn_up_shexp\.weight=Q8_0 +blk\.38\.ssm_alpha\.weight=F32 +blk\.38\.ssm_beta\.weight=F32 +blk\.38\.ssm_out\.weight=Q8_0 +blk\.39\.attn_k\.weight=F16 +blk\.39\.attn_output\.weight=Q8_0 +blk\.39\.attn_q\.weight=Q8_0 +blk\.39\.attn_v\.weight=F16 +blk\.39\.ffn_down_exps\.weight=Q5_K +blk\.39\.ffn_down_shexp\.weight=Q8_0 +blk\.39\.ffn_gate_exps\.weight=Q5_K +blk\.39\.ffn_gate_shexp\.weight=Q8_0 +blk\.39\.ffn_up_exps\.weight=Q5_K +blk\.39\.ffn_up_shexp\.weight=Q8_0 +blk\.4\.attn_gate\.weight=Q8_0 +blk\.4\.attn_qkv\.weight=Q8_0 +blk\.4\.ffn_down_exps\.weight=Q5_K +blk\.4\.ffn_down_shexp\.weight=Q8_0 +blk\.4\.ffn_gate_exps\.weight=Q5_K +blk\.4\.ffn_gate_shexp\.weight=Q8_0 +blk\.4\.ffn_up_exps\.weight=Q5_K +blk\.4\.ffn_up_shexp\.weight=Q8_0 +blk\.4\.ssm_alpha\.weight=F32 +blk\.4\.ssm_beta\.weight=F32 +blk\.4\.ssm_out\.weight=Q8_0 +blk\.40\.attn_k\.weight=F16 +blk\.40\.attn_output\.weight=Q8_0 +blk\.40\.attn_q\.weight=Q8_0 +blk\.40\.attn_v\.weight=F16 +blk\.40\.ffn_down_exps\.weight=Q5_K +blk\.40\.ffn_down_shexp\.weight=Q8_0 +blk\.40\.ffn_gate_exps\.weight=Q5_K +blk\.40\.ffn_gate_shexp\.weight=Q8_0 +blk\.40\.ffn_up_exps\.weight=Q5_K +blk\.40\.ffn_up_shexp\.weight=Q8_0 +blk\.40\.nextn\.eh_proj\.weight=Q8_0 +blk\.5\.attn_gate\.weight=Q8_0 +blk\.5\.attn_qkv\.weight=Q8_0 +blk\.5\.ffn_down_exps\.weight=Q5_K +blk\.5\.ffn_down_shexp\.weight=Q8_0 +blk\.5\.ffn_gate_exps\.weight=Q5_K +blk\.5\.ffn_gate_shexp\.weight=Q8_0 +blk\.5\.ffn_up_exps\.weight=Q5_K +blk\.5\.ffn_up_shexp\.weight=Q8_0 +blk\.5\.ssm_alpha\.weight=F32 +blk\.5\.ssm_beta\.weight=F32 +blk\.5\.ssm_out\.weight=Q8_0 +blk\.6\.attn_gate\.weight=Q8_0 +blk\.6\.attn_qkv\.weight=Q8_0 +blk\.6\.ffn_down_exps\.weight=Q5_K +blk\.6\.ffn_down_shexp\.weight=Q8_0 +blk\.6\.ffn_gate_exps\.weight=Q5_K +blk\.6\.ffn_gate_shexp\.weight=Q8_0 +blk\.6\.ffn_up_exps\.weight=Q5_K +blk\.6\.ffn_up_shexp\.weight=Q8_0 +blk\.6\.ssm_alpha\.weight=F32 +blk\.6\.ssm_beta\.weight=F32 +blk\.6\.ssm_out\.weight=Q8_0 +blk\.7\.attn_k\.weight=F16 +blk\.7\.attn_output\.weight=Q8_0 +blk\.7\.attn_q\.weight=Q8_0 +blk\.7\.attn_v\.weight=F16 +blk\.7\.ffn_down_exps\.weight=Q5_K +blk\.7\.ffn_down_shexp\.weight=Q8_0 +blk\.7\.ffn_gate_exps\.weight=Q5_K +blk\.7\.ffn_gate_shexp\.weight=Q8_0 +blk\.7\.ffn_up_exps\.weight=Q5_K +blk\.7\.ffn_up_shexp\.weight=Q8_0 +blk\.8\.attn_gate\.weight=Q8_0 +blk\.8\.attn_qkv\.weight=Q8_0 +blk\.8\.ffn_down_exps\.weight=Q5_K +blk\.8\.ffn_down_shexp\.weight=Q8_0 +blk\.8\.ffn_gate_exps\.weight=Q5_K +blk\.8\.ffn_gate_shexp\.weight=Q8_0 +blk\.8\.ffn_up_exps\.weight=Q5_K +blk\.8\.ffn_up_shexp\.weight=Q8_0 +blk\.8\.ssm_alpha\.weight=F32 +blk\.8\.ssm_beta\.weight=F32 +blk\.8\.ssm_out\.weight=Q8_0 +blk\.9\.attn_gate\.weight=Q8_0 +blk\.9\.attn_qkv\.weight=Q8_0 +blk\.9\.ffn_down_exps\.weight=Q5_K +blk\.9\.ffn_down_shexp\.weight=Q8_0 +blk\.9\.ffn_gate_exps\.weight=Q5_K +blk\.9\.ffn_gate_shexp\.weight=Q8_0 +blk\.9\.ffn_up_exps\.weight=Q5_K +blk\.9\.ffn_up_shexp\.weight=Q8_0 +blk\.9\.ssm_alpha\.weight=F32 +blk\.9\.ssm_beta\.weight=F32 +blk\.9\.ssm_out\.weight=Q8_0 diff --git a/appendix/recipes/cfg_APEX-I-Compact_comparison.txt b/appendix/recipes/cfg_APEX-I-Compact_comparison.txt new file mode 100644 index 0000000000000000000000000000000000000000..65960e6bf4ca2440c53304509907f63aebeb70a2 --- /dev/null +++ b/appendix/recipes/cfg_APEX-I-Compact_comparison.txt @@ -0,0 +1,443 @@ +^output\.weight$=Q8_0 +token_embd\.weight=Q4_K +blk\.0\.attn_gate\.weight=Q4_K +blk\.0\.attn_qkv\.weight=Q4_K +blk\.0\.ffn_down_exps\.weight=Q4_K +blk\.0\.ffn_down_shexp\.weight=Q6_K +blk\.0\.ffn_gate_exps\.weight=Q4_K +blk\.0\.ffn_gate_shexp\.weight=Q6_K +blk\.0\.ffn_up_exps\.weight=Q4_K +blk\.0\.ffn_up_shexp\.weight=Q6_K +blk\.0\.ssm_alpha\.weight=Q4_K +blk\.0\.ssm_beta\.weight=Q4_K +blk\.0\.ssm_out\.weight=Q4_K +blk\.1\.attn_gate\.weight=Q4_K +blk\.1\.attn_qkv\.weight=Q4_K +blk\.1\.ffn_down_exps\.weight=Q4_K +blk\.1\.ffn_down_shexp\.weight=Q6_K +blk\.1\.ffn_gate_exps\.weight=Q4_K +blk\.1\.ffn_gate_shexp\.weight=Q6_K +blk\.1\.ffn_up_exps\.weight=Q4_K +blk\.1\.ffn_up_shexp\.weight=Q6_K +blk\.1\.ssm_alpha\.weight=Q4_K +blk\.1\.ssm_beta\.weight=Q4_K +blk\.1\.ssm_out\.weight=Q4_K +blk\.2\.attn_gate\.weight=Q4_K +blk\.2\.attn_qkv\.weight=Q4_K +blk\.2\.ffn_down_exps\.weight=Q4_K +blk\.2\.ffn_down_shexp\.weight=Q6_K +blk\.2\.ffn_gate_exps\.weight=Q4_K +blk\.2\.ffn_gate_shexp\.weight=Q6_K +blk\.2\.ffn_up_exps\.weight=Q4_K +blk\.2\.ffn_up_shexp\.weight=Q6_K +blk\.2\.ssm_alpha\.weight=Q4_K +blk\.2\.ssm_beta\.weight=Q4_K +blk\.2\.ssm_out\.weight=Q4_K +blk\.3\.attn_k\.weight=Q6_K +blk\.3\.attn_output\.weight=Q4_K +blk\.3\.attn_q\.weight=Q4_K +blk\.3\.attn_v\.weight=Q6_K +blk\.3\.ffn_down_exps\.weight=Q4_K +blk\.3\.ffn_down_shexp\.weight=Q6_K +blk\.3\.ffn_gate_exps\.weight=Q4_K +blk\.3\.ffn_gate_shexp\.weight=Q6_K +blk\.3\.ffn_up_exps\.weight=Q4_K +blk\.3\.ffn_up_shexp\.weight=Q6_K +blk\.4\.attn_gate\.weight=Q4_K +blk\.4\.attn_qkv\.weight=Q4_K +blk\.4\.ffn_down_exps\.weight=Q4_K +blk\.4\.ffn_down_shexp\.weight=Q6_K +blk\.4\.ffn_gate_exps\.weight=Q4_K +blk\.4\.ffn_gate_shexp\.weight=Q6_K +blk\.4\.ffn_up_exps\.weight=Q4_K +blk\.4\.ffn_up_shexp\.weight=Q6_K +blk\.4\.ssm_alpha\.weight=Q4_K +blk\.4\.ssm_beta\.weight=Q4_K +blk\.4\.ssm_out\.weight=Q4_K +blk\.5\.attn_gate\.weight=Q4_K +blk\.5\.attn_qkv\.weight=Q4_K +blk\.5\.ffn_down_exps\.weight=Q3_K +blk\.5\.ffn_down_shexp\.weight=Q6_K +blk\.5\.ffn_gate_exps\.weight=Q3_K +blk\.5\.ffn_gate_shexp\.weight=Q6_K +blk\.5\.ffn_up_exps\.weight=Q3_K +blk\.5\.ffn_up_shexp\.weight=Q6_K +blk\.5\.ssm_alpha\.weight=Q4_K +blk\.5\.ssm_beta\.weight=Q4_K +blk\.5\.ssm_out\.weight=Q4_K +blk\.6\.attn_gate\.weight=Q4_K +blk\.6\.attn_qkv\.weight=Q4_K +blk\.6\.ffn_down_exps\.weight=Q3_K +blk\.6\.ffn_down_shexp\.weight=Q6_K +blk\.6\.ffn_gate_exps\.weight=Q3_K +blk\.6\.ffn_gate_shexp\.weight=Q6_K +blk\.6\.ffn_up_exps\.weight=Q3_K +blk\.6\.ffn_up_shexp\.weight=Q6_K +blk\.6\.ssm_alpha\.weight=Q4_K +blk\.6\.ssm_beta\.weight=Q4_K +blk\.6\.ssm_out\.weight=Q4_K +blk\.7\.attn_k\.weight=Q6_K +blk\.7\.attn_output\.weight=Q4_K +blk\.7\.attn_q\.weight=Q4_K +blk\.7\.attn_v\.weight=Q6_K +blk\.7\.ffn_down_exps\.weight=Q3_K +blk\.7\.ffn_down_shexp\.weight=Q6_K +blk\.7\.ffn_gate_exps\.weight=Q3_K +blk\.7\.ffn_gate_shexp\.weight=Q6_K +blk\.7\.ffn_up_exps\.weight=Q3_K +blk\.7\.ffn_up_shexp\.weight=Q6_K +blk\.8\.attn_gate\.weight=Q4_K +blk\.8\.attn_qkv\.weight=Q4_K +blk\.8\.ffn_down_exps\.weight=Q3_K +blk\.8\.ffn_down_shexp\.weight=Q6_K +blk\.8\.ffn_gate_exps\.weight=Q3_K +blk\.8\.ffn_gate_shexp\.weight=Q6_K +blk\.8\.ffn_up_exps\.weight=Q3_K +blk\.8\.ffn_up_shexp\.weight=Q6_K +blk\.8\.ssm_alpha\.weight=Q4_K +blk\.8\.ssm_beta\.weight=Q4_K +blk\.8\.ssm_out\.weight=Q4_K +blk\.9\.attn_gate\.weight=Q4_K +blk\.9\.attn_qkv\.weight=Q4_K +blk\.9\.ffn_down_exps\.weight=Q3_K +blk\.9\.ffn_down_shexp\.weight=Q6_K +blk\.9\.ffn_gate_exps\.weight=Q3_K +blk\.9\.ffn_gate_shexp\.weight=Q6_K +blk\.9\.ffn_up_exps\.weight=Q3_K +blk\.9\.ffn_up_shexp\.weight=Q6_K +blk\.9\.ssm_alpha\.weight=Q4_K +blk\.9\.ssm_beta\.weight=Q4_K +blk\.9\.ssm_out\.weight=Q4_K +blk\.10\.attn_gate\.weight=Q4_K +blk\.10\.attn_qkv\.weight=Q4_K +blk\.10\.ffn_down_exps\.weight=Q3_K +blk\.10\.ffn_down_shexp\.weight=Q6_K +blk\.10\.ffn_gate_exps\.weight=Q3_K +blk\.10\.ffn_gate_shexp\.weight=Q6_K +blk\.10\.ffn_up_exps\.weight=Q3_K +blk\.10\.ffn_up_shexp\.weight=Q6_K +blk\.10\.ssm_alpha\.weight=Q4_K +blk\.10\.ssm_beta\.weight=Q4_K +blk\.10\.ssm_out\.weight=Q4_K +blk\.11\.attn_k\.weight=Q6_K +blk\.11\.attn_output\.weight=Q4_K +blk\.11\.attn_q\.weight=Q4_K +blk\.11\.attn_v\.weight=Q6_K +blk\.11\.ffn_down_exps\.weight=Q3_K +blk\.11\.ffn_down_shexp\.weight=Q6_K +blk\.11\.ffn_gate_exps\.weight=Q3_K +blk\.11\.ffn_gate_shexp\.weight=Q6_K +blk\.11\.ffn_up_exps\.weight=Q3_K +blk\.11\.ffn_up_shexp\.weight=Q6_K +blk\.12\.attn_gate\.weight=Q4_K +blk\.12\.attn_qkv\.weight=Q4_K +blk\.12\.ffn_down_exps\.weight=Q3_K +blk\.12\.ffn_down_shexp\.weight=Q6_K +blk\.12\.ffn_gate_exps\.weight=Q3_K +blk\.12\.ffn_gate_shexp\.weight=Q6_K +blk\.12\.ffn_up_exps\.weight=Q3_K +blk\.12\.ffn_up_shexp\.weight=Q6_K +blk\.12\.ssm_alpha\.weight=Q4_K +blk\.12\.ssm_beta\.weight=Q4_K +blk\.12\.ssm_out\.weight=Q4_K +blk\.13\.attn_gate\.weight=Q4_K +blk\.13\.attn_qkv\.weight=Q4_K +blk\.13\.ffn_down_exps\.weight=Q3_K +blk\.13\.ffn_down_shexp\.weight=Q6_K +blk\.13\.ffn_gate_exps\.weight=Q3_K +blk\.13\.ffn_gate_shexp\.weight=Q6_K +blk\.13\.ffn_up_exps\.weight=Q3_K +blk\.13\.ffn_up_shexp\.weight=Q6_K +blk\.13\.ssm_alpha\.weight=Q4_K +blk\.13\.ssm_beta\.weight=Q4_K +blk\.13\.ssm_out\.weight=Q4_K +blk\.14\.attn_gate\.weight=Q4_K +blk\.14\.attn_qkv\.weight=Q4_K +blk\.14\.ffn_down_exps\.weight=Q3_K +blk\.14\.ffn_down_shexp\.weight=Q6_K +blk\.14\.ffn_gate_exps\.weight=Q3_K +blk\.14\.ffn_gate_shexp\.weight=Q6_K +blk\.14\.ffn_up_exps\.weight=Q3_K +blk\.14\.ffn_up_shexp\.weight=Q6_K +blk\.14\.ssm_alpha\.weight=Q4_K +blk\.14\.ssm_beta\.weight=Q4_K +blk\.14\.ssm_out\.weight=Q4_K +blk\.15\.attn_k\.weight=Q6_K +blk\.15\.attn_output\.weight=Q4_K +blk\.15\.attn_q\.weight=Q4_K +blk\.15\.attn_v\.weight=Q6_K +blk\.15\.ffn_down_exps\.weight=Q3_K +blk\.15\.ffn_down_shexp\.weight=Q6_K +blk\.15\.ffn_gate_exps\.weight=Q3_K +blk\.15\.ffn_gate_shexp\.weight=Q6_K +blk\.15\.ffn_up_exps\.weight=Q3_K +blk\.15\.ffn_up_shexp\.weight=Q6_K +blk\.16\.attn_gate\.weight=Q4_K +blk\.16\.attn_qkv\.weight=Q4_K +blk\.16\.ffn_down_exps\.weight=Q3_K +blk\.16\.ffn_down_shexp\.weight=Q6_K +blk\.16\.ffn_gate_exps\.weight=Q3_K +blk\.16\.ffn_gate_shexp\.weight=Q6_K +blk\.16\.ffn_up_exps\.weight=Q3_K +blk\.16\.ffn_up_shexp\.weight=Q6_K +blk\.16\.ssm_alpha\.weight=Q4_K +blk\.16\.ssm_beta\.weight=Q4_K +blk\.16\.ssm_out\.weight=Q4_K +blk\.17\.attn_gate\.weight=Q4_K +blk\.17\.attn_qkv\.weight=Q4_K +blk\.17\.ffn_down_exps\.weight=Q3_K +blk\.17\.ffn_down_shexp\.weight=Q6_K +blk\.17\.ffn_gate_exps\.weight=Q3_K +blk\.17\.ffn_gate_shexp\.weight=Q6_K +blk\.17\.ffn_up_exps\.weight=Q3_K +blk\.17\.ffn_up_shexp\.weight=Q6_K +blk\.17\.ssm_alpha\.weight=Q4_K +blk\.17\.ssm_beta\.weight=Q4_K +blk\.17\.ssm_out\.weight=Q4_K +blk\.18\.attn_gate\.weight=Q4_K +blk\.18\.attn_qkv\.weight=Q4_K +blk\.18\.ffn_down_exps\.weight=Q3_K +blk\.18\.ffn_down_shexp\.weight=Q6_K +blk\.18\.ffn_gate_exps\.weight=Q3_K +blk\.18\.ffn_gate_shexp\.weight=Q6_K +blk\.18\.ffn_up_exps\.weight=Q3_K +blk\.18\.ffn_up_shexp\.weight=Q6_K +blk\.18\.ssm_alpha\.weight=Q4_K +blk\.18\.ssm_beta\.weight=Q4_K +blk\.18\.ssm_out\.weight=Q4_K +blk\.19\.attn_k\.weight=Q6_K +blk\.19\.attn_output\.weight=Q4_K +blk\.19\.attn_q\.weight=Q4_K +blk\.19\.attn_v\.weight=Q6_K +blk\.19\.ffn_down_exps\.weight=Q3_K +blk\.19\.ffn_down_shexp\.weight=Q6_K +blk\.19\.ffn_gate_exps\.weight=Q3_K +blk\.19\.ffn_gate_shexp\.weight=Q6_K +blk\.19\.ffn_up_exps\.weight=Q3_K +blk\.19\.ffn_up_shexp\.weight=Q6_K +blk\.20\.attn_gate\.weight=Q4_K +blk\.20\.attn_qkv\.weight=Q4_K +blk\.20\.ffn_down_exps\.weight=Q3_K +blk\.20\.ffn_down_shexp\.weight=Q6_K +blk\.20\.ffn_gate_exps\.weight=Q3_K +blk\.20\.ffn_gate_shexp\.weight=Q6_K +blk\.20\.ffn_up_exps\.weight=Q3_K +blk\.20\.ffn_up_shexp\.weight=Q6_K +blk\.20\.ssm_alpha\.weight=Q4_K +blk\.20\.ssm_beta\.weight=Q4_K +blk\.20\.ssm_out\.weight=Q4_K +blk\.21\.attn_gate\.weight=Q4_K +blk\.21\.attn_qkv\.weight=Q4_K +blk\.21\.ffn_down_exps\.weight=Q3_K +blk\.21\.ffn_down_shexp\.weight=Q6_K +blk\.21\.ffn_gate_exps\.weight=Q3_K +blk\.21\.ffn_gate_shexp\.weight=Q6_K +blk\.21\.ffn_up_exps\.weight=Q3_K +blk\.21\.ffn_up_shexp\.weight=Q6_K +blk\.21\.ssm_alpha\.weight=Q4_K +blk\.21\.ssm_beta\.weight=Q4_K +blk\.21\.ssm_out\.weight=Q4_K +blk\.22\.attn_gate\.weight=Q4_K +blk\.22\.attn_qkv\.weight=Q4_K +blk\.22\.ffn_down_exps\.weight=Q3_K +blk\.22\.ffn_down_shexp\.weight=Q6_K +blk\.22\.ffn_gate_exps\.weight=Q3_K +blk\.22\.ffn_gate_shexp\.weight=Q6_K +blk\.22\.ffn_up_exps\.weight=Q3_K +blk\.22\.ffn_up_shexp\.weight=Q6_K +blk\.22\.ssm_alpha\.weight=Q4_K +blk\.22\.ssm_beta\.weight=Q4_K +blk\.22\.ssm_out\.weight=Q4_K +blk\.23\.attn_k\.weight=Q6_K +blk\.23\.attn_output\.weight=Q4_K +blk\.23\.attn_q\.weight=Q4_K +blk\.23\.attn_v\.weight=Q6_K +blk\.23\.ffn_down_exps\.weight=Q3_K +blk\.23\.ffn_down_shexp\.weight=Q6_K +blk\.23\.ffn_gate_exps\.weight=Q3_K +blk\.23\.ffn_gate_shexp\.weight=Q6_K +blk\.23\.ffn_up_exps\.weight=Q3_K +blk\.23\.ffn_up_shexp\.weight=Q6_K +blk\.24\.attn_gate\.weight=Q4_K +blk\.24\.attn_qkv\.weight=Q4_K +blk\.24\.ffn_down_exps\.weight=Q3_K +blk\.24\.ffn_down_shexp\.weight=Q6_K +blk\.24\.ffn_gate_exps\.weight=Q3_K +blk\.24\.ffn_gate_shexp\.weight=Q6_K +blk\.24\.ffn_up_exps\.weight=Q3_K +blk\.24\.ffn_up_shexp\.weight=Q6_K +blk\.24\.ssm_alpha\.weight=Q4_K +blk\.24\.ssm_beta\.weight=Q4_K +blk\.24\.ssm_out\.weight=Q4_K +blk\.25\.attn_gate\.weight=Q4_K +blk\.25\.attn_qkv\.weight=Q4_K +blk\.25\.ffn_down_exps\.weight=Q3_K +blk\.25\.ffn_down_shexp\.weight=Q6_K +blk\.25\.ffn_gate_exps\.weight=Q3_K +blk\.25\.ffn_gate_shexp\.weight=Q6_K +blk\.25\.ffn_up_exps\.weight=Q3_K +blk\.25\.ffn_up_shexp\.weight=Q6_K +blk\.25\.ssm_alpha\.weight=Q4_K +blk\.25\.ssm_beta\.weight=Q4_K +blk\.25\.ssm_out\.weight=Q4_K +blk\.26\.attn_gate\.weight=Q4_K +blk\.26\.attn_qkv\.weight=Q4_K +blk\.26\.ffn_down_exps\.weight=Q3_K +blk\.26\.ffn_down_shexp\.weight=Q6_K +blk\.26\.ffn_gate_exps\.weight=Q3_K +blk\.26\.ffn_gate_shexp\.weight=Q6_K +blk\.26\.ffn_up_exps\.weight=Q3_K +blk\.26\.ffn_up_shexp\.weight=Q6_K +blk\.26\.ssm_alpha\.weight=Q4_K +blk\.26\.ssm_beta\.weight=Q4_K +blk\.26\.ssm_out\.weight=Q4_K +blk\.27\.attn_k\.weight=Q6_K +blk\.27\.attn_output\.weight=Q4_K +blk\.27\.attn_q\.weight=Q4_K +blk\.27\.attn_v\.weight=Q6_K +blk\.27\.ffn_down_exps\.weight=Q3_K +blk\.27\.ffn_down_shexp\.weight=Q6_K +blk\.27\.ffn_gate_exps\.weight=Q3_K +blk\.27\.ffn_gate_shexp\.weight=Q6_K +blk\.27\.ffn_up_exps\.weight=Q3_K +blk\.27\.ffn_up_shexp\.weight=Q6_K +blk\.28\.attn_gate\.weight=Q4_K +blk\.28\.attn_qkv\.weight=Q4_K +blk\.28\.ffn_down_exps\.weight=Q3_K +blk\.28\.ffn_down_shexp\.weight=Q6_K +blk\.28\.ffn_gate_exps\.weight=Q3_K +blk\.28\.ffn_gate_shexp\.weight=Q6_K +blk\.28\.ffn_up_exps\.weight=Q3_K +blk\.28\.ffn_up_shexp\.weight=Q6_K +blk\.28\.ssm_alpha\.weight=Q4_K +blk\.28\.ssm_beta\.weight=Q4_K +blk\.28\.ssm_out\.weight=Q4_K +blk\.29\.attn_gate\.weight=Q4_K +blk\.29\.attn_qkv\.weight=Q4_K +blk\.29\.ffn_down_exps\.weight=Q3_K +blk\.29\.ffn_down_shexp\.weight=Q6_K +blk\.29\.ffn_gate_exps\.weight=Q3_K +blk\.29\.ffn_gate_shexp\.weight=Q6_K +blk\.29\.ffn_up_exps\.weight=Q3_K +blk\.29\.ffn_up_shexp\.weight=Q6_K +blk\.29\.ssm_alpha\.weight=Q4_K +blk\.29\.ssm_beta\.weight=Q4_K +blk\.29\.ssm_out\.weight=Q4_K +blk\.30\.attn_gate\.weight=Q4_K +blk\.30\.attn_qkv\.weight=Q4_K +blk\.30\.ffn_down_exps\.weight=Q3_K +blk\.30\.ffn_down_shexp\.weight=Q6_K +blk\.30\.ffn_gate_exps\.weight=Q3_K +blk\.30\.ffn_gate_shexp\.weight=Q6_K +blk\.30\.ffn_up_exps\.weight=Q3_K +blk\.30\.ffn_up_shexp\.weight=Q6_K +blk\.30\.ssm_alpha\.weight=Q4_K +blk\.30\.ssm_beta\.weight=Q4_K +blk\.30\.ssm_out\.weight=Q4_K +blk\.31\.attn_k\.weight=Q6_K +blk\.31\.attn_output\.weight=Q4_K +blk\.31\.attn_q\.weight=Q4_K +blk\.31\.attn_v\.weight=Q6_K +blk\.31\.ffn_down_exps\.weight=Q3_K +blk\.31\.ffn_down_shexp\.weight=Q6_K +blk\.31\.ffn_gate_exps\.weight=Q3_K +blk\.31\.ffn_gate_shexp\.weight=Q6_K +blk\.31\.ffn_up_exps\.weight=Q3_K +blk\.31\.ffn_up_shexp\.weight=Q6_K +blk\.32\.attn_gate\.weight=Q4_K +blk\.32\.attn_qkv\.weight=Q4_K +blk\.32\.ffn_down_exps\.weight=Q3_K +blk\.32\.ffn_down_shexp\.weight=Q6_K +blk\.32\.ffn_gate_exps\.weight=Q3_K +blk\.32\.ffn_gate_shexp\.weight=Q6_K +blk\.32\.ffn_up_exps\.weight=Q3_K +blk\.32\.ffn_up_shexp\.weight=Q6_K +blk\.32\.ssm_alpha\.weight=Q4_K +blk\.32\.ssm_beta\.weight=Q4_K +blk\.32\.ssm_out\.weight=Q4_K +blk\.33\.attn_gate\.weight=Q4_K +blk\.33\.attn_qkv\.weight=Q4_K +blk\.33\.ffn_down_exps\.weight=Q3_K +blk\.33\.ffn_down_shexp\.weight=Q6_K +blk\.33\.ffn_gate_exps\.weight=Q3_K +blk\.33\.ffn_gate_shexp\.weight=Q6_K +blk\.33\.ffn_up_exps\.weight=Q3_K +blk\.33\.ffn_up_shexp\.weight=Q6_K +blk\.33\.ssm_alpha\.weight=Q4_K +blk\.33\.ssm_beta\.weight=Q4_K +blk\.33\.ssm_out\.weight=Q4_K +blk\.34\.attn_gate\.weight=Q4_K +blk\.34\.attn_qkv\.weight=Q4_K +blk\.34\.ffn_down_exps\.weight=Q3_K +blk\.34\.ffn_down_shexp\.weight=Q6_K +blk\.34\.ffn_gate_exps\.weight=Q3_K +blk\.34\.ffn_gate_shexp\.weight=Q6_K +blk\.34\.ffn_up_exps\.weight=Q3_K +blk\.34\.ffn_up_shexp\.weight=Q6_K +blk\.34\.ssm_alpha\.weight=Q4_K +blk\.34\.ssm_beta\.weight=Q4_K +blk\.34\.ssm_out\.weight=Q4_K +blk\.35\.attn_k\.weight=Q6_K +blk\.35\.attn_output\.weight=Q4_K +blk\.35\.attn_q\.weight=Q4_K +blk\.35\.attn_v\.weight=Q6_K +blk\.35\.ffn_down_exps\.weight=Q4_K +blk\.35\.ffn_down_shexp\.weight=Q6_K +blk\.35\.ffn_gate_exps\.weight=Q4_K +blk\.35\.ffn_gate_shexp\.weight=Q6_K +blk\.35\.ffn_up_exps\.weight=Q4_K +blk\.35\.ffn_up_shexp\.weight=Q6_K +blk\.36\.attn_gate\.weight=Q4_K +blk\.36\.attn_qkv\.weight=Q4_K +blk\.36\.ffn_down_exps\.weight=Q4_K +blk\.36\.ffn_down_shexp\.weight=Q6_K +blk\.36\.ffn_gate_exps\.weight=Q4_K +blk\.36\.ffn_gate_shexp\.weight=Q6_K +blk\.36\.ffn_up_exps\.weight=Q4_K +blk\.36\.ffn_up_shexp\.weight=Q6_K +blk\.36\.ssm_alpha\.weight=Q4_K +blk\.36\.ssm_beta\.weight=Q4_K +blk\.36\.ssm_out\.weight=Q4_K +blk\.37\.attn_gate\.weight=Q4_K +blk\.37\.attn_qkv\.weight=Q4_K +blk\.37\.ffn_down_exps\.weight=Q4_K +blk\.37\.ffn_down_shexp\.weight=Q6_K +blk\.37\.ffn_gate_exps\.weight=Q4_K +blk\.37\.ffn_gate_shexp\.weight=Q6_K +blk\.37\.ffn_up_exps\.weight=Q4_K +blk\.37\.ffn_up_shexp\.weight=Q6_K +blk\.37\.ssm_alpha\.weight=Q4_K +blk\.37\.ssm_beta\.weight=Q4_K +blk\.37\.ssm_out\.weight=Q4_K +blk\.38\.attn_gate\.weight=Q4_K +blk\.38\.attn_qkv\.weight=Q4_K +blk\.38\.ffn_down_exps\.weight=Q4_K +blk\.38\.ffn_down_shexp\.weight=Q6_K +blk\.38\.ffn_gate_exps\.weight=Q4_K +blk\.38\.ffn_gate_shexp\.weight=Q6_K +blk\.38\.ffn_up_exps\.weight=Q4_K +blk\.38\.ffn_up_shexp\.weight=Q6_K +blk\.38\.ssm_alpha\.weight=Q4_K +blk\.38\.ssm_beta\.weight=Q4_K +blk\.38\.ssm_out\.weight=Q4_K +blk\.39\.attn_k\.weight=Q6_K +blk\.39\.attn_output\.weight=Q4_K +blk\.39\.attn_q\.weight=Q4_K +blk\.39\.attn_v\.weight=Q6_K +blk\.39\.ffn_down_exps\.weight=Q4_K +blk\.39\.ffn_down_shexp\.weight=Q6_K +blk\.39\.ffn_gate_exps\.weight=Q4_K +blk\.39\.ffn_gate_shexp\.weight=Q6_K +blk\.39\.ffn_up_exps\.weight=Q4_K +blk\.39\.ffn_up_shexp\.weight=Q6_K +blk\.40\.attn_k\.weight=Q8_0 +blk\.40\.attn_output\.weight=Q8_0 +blk\.40\.attn_q\.weight=Q8_0 +blk\.40\.attn_v\.weight=Q8_0 +blk\.40\.ffn_down_exps\.weight=Q8_0 +blk\.40\.ffn_down_shexp\.weight=Q8_0 +blk\.40\.ffn_gate_exps\.weight=Q8_0 +blk\.40\.ffn_gate_shexp\.weight=Q8_0 +blk\.40\.ffn_up_exps\.weight=Q8_0 +blk\.40\.ffn_up_shexp\.weight=Q8_0 +blk\.40\.nextn\.eh_proj\.weight=Q8_0 diff --git a/appendix/recipes/cfg_APEX-I-Mini_comparison.txt b/appendix/recipes/cfg_APEX-I-Mini_comparison.txt new file mode 100644 index 0000000000000000000000000000000000000000..8f7a120cf74938d24f1a3a0fae8479bf884440db --- /dev/null +++ b/appendix/recipes/cfg_APEX-I-Mini_comparison.txt @@ -0,0 +1,443 @@ +^output\.weight$=Q6_K +token_embd\.weight=Q3_K +blk\.0\.attn_gate\.weight=Q4_K +blk\.0\.attn_qkv\.weight=Q4_K +blk\.0\.ffn_down_exps\.weight=Q3_K +blk\.0\.ffn_down_shexp\.weight=Q5_K +blk\.0\.ffn_gate_exps\.weight=Q3_K +blk\.0\.ffn_gate_shexp\.weight=Q5_K +blk\.0\.ffn_up_exps\.weight=Q3_K +blk\.0\.ffn_up_shexp\.weight=Q5_K +blk\.0\.ssm_alpha\.weight=Q4_K +blk\.0\.ssm_beta\.weight=Q4_K +blk\.0\.ssm_out\.weight=Q4_K +blk\.1\.attn_gate\.weight=Q4_K +blk\.1\.attn_qkv\.weight=Q4_K +blk\.1\.ffn_down_exps\.weight=Q3_K +blk\.1\.ffn_down_shexp\.weight=Q5_K +blk\.1\.ffn_gate_exps\.weight=Q3_K +blk\.1\.ffn_gate_shexp\.weight=Q5_K +blk\.1\.ffn_up_exps\.weight=Q3_K +blk\.1\.ffn_up_shexp\.weight=Q5_K +blk\.1\.ssm_alpha\.weight=Q4_K +blk\.1\.ssm_beta\.weight=Q4_K +blk\.1\.ssm_out\.weight=Q4_K +blk\.2\.attn_gate\.weight=Q4_K +blk\.2\.attn_qkv\.weight=Q4_K +blk\.2\.ffn_down_exps\.weight=Q3_K +blk\.2\.ffn_down_shexp\.weight=Q5_K +blk\.2\.ffn_gate_exps\.weight=Q3_K +blk\.2\.ffn_gate_shexp\.weight=Q5_K +blk\.2\.ffn_up_exps\.weight=Q3_K +blk\.2\.ffn_up_shexp\.weight=Q5_K +blk\.2\.ssm_alpha\.weight=Q4_K +blk\.2\.ssm_beta\.weight=Q4_K +blk\.2\.ssm_out\.weight=Q4_K +blk\.3\.attn_k\.weight=Q6_K +blk\.3\.attn_output\.weight=Q3_K +blk\.3\.attn_q\.weight=Q3_K +blk\.3\.attn_v\.weight=Q6_K +blk\.3\.ffn_down_exps\.weight=Q3_K +blk\.3\.ffn_down_shexp\.weight=Q5_K +blk\.3\.ffn_gate_exps\.weight=Q3_K +blk\.3\.ffn_gate_shexp\.weight=Q5_K +blk\.3\.ffn_up_exps\.weight=Q3_K +blk\.3\.ffn_up_shexp\.weight=Q5_K +blk\.4\.attn_gate\.weight=Q3_K +blk\.4\.attn_qkv\.weight=Q3_K +blk\.4\.ffn_down_exps\.weight=Q3_K +blk\.4\.ffn_down_shexp\.weight=Q5_K +blk\.4\.ffn_gate_exps\.weight=Q3_K +blk\.4\.ffn_gate_shexp\.weight=Q5_K +blk\.4\.ffn_up_exps\.weight=Q3_K +blk\.4\.ffn_up_shexp\.weight=Q5_K +blk\.4\.ssm_alpha\.weight=Q3_K +blk\.4\.ssm_beta\.weight=Q3_K +blk\.4\.ssm_out\.weight=Q3_K +blk\.5\.attn_gate\.weight=Q3_K +blk\.5\.attn_qkv\.weight=Q3_K +blk\.5\.ffn_down_exps\.weight=Q3_K +blk\.5\.ffn_down_shexp\.weight=Q4_K +blk\.5\.ffn_gate_exps\.weight=Q3_K +blk\.5\.ffn_gate_shexp\.weight=Q4_K +blk\.5\.ffn_up_exps\.weight=Q3_K +blk\.5\.ffn_up_shexp\.weight=Q4_K +blk\.5\.ssm_alpha\.weight=Q3_K +blk\.5\.ssm_beta\.weight=Q3_K +blk\.5\.ssm_out\.weight=Q3_K +blk\.6\.attn_gate\.weight=Q3_K +blk\.6\.attn_qkv\.weight=Q3_K +blk\.6\.ffn_down_exps\.weight=Q3_K +blk\.6\.ffn_down_shexp\.weight=Q4_K +blk\.6\.ffn_gate_exps\.weight=Q3_K +blk\.6\.ffn_gate_shexp\.weight=Q4_K +blk\.6\.ffn_up_exps\.weight=Q3_K +blk\.6\.ffn_up_shexp\.weight=Q4_K +blk\.6\.ssm_alpha\.weight=Q3_K +blk\.6\.ssm_beta\.weight=Q3_K +blk\.6\.ssm_out\.weight=Q3_K +blk\.7\.attn_k\.weight=Q6_K +blk\.7\.attn_output\.weight=Q3_K +blk\.7\.attn_q\.weight=Q3_K +blk\.7\.attn_v\.weight=Q6_K +blk\.7\.ffn_down_exps\.weight=Q3_K +blk\.7\.ffn_down_shexp\.weight=Q4_K +blk\.7\.ffn_gate_exps\.weight=Q3_K +blk\.7\.ffn_gate_shexp\.weight=Q4_K +blk\.7\.ffn_up_exps\.weight=Q3_K +blk\.7\.ffn_up_shexp\.weight=Q4_K +blk\.8\.attn_gate\.weight=Q3_K +blk\.8\.attn_qkv\.weight=Q3_K +blk\.8\.ffn_down_exps\.weight=Q3_K +blk\.8\.ffn_down_shexp\.weight=Q4_K +blk\.8\.ffn_gate_exps\.weight=Q3_K +blk\.8\.ffn_gate_shexp\.weight=Q4_K +blk\.8\.ffn_up_exps\.weight=Q3_K +blk\.8\.ffn_up_shexp\.weight=Q4_K +blk\.8\.ssm_alpha\.weight=Q3_K +blk\.8\.ssm_beta\.weight=Q3_K +blk\.8\.ssm_out\.weight=Q3_K +blk\.9\.attn_gate\.weight=Q3_K +blk\.9\.attn_qkv\.weight=Q3_K +blk\.9\.ffn_down_exps\.weight=Q3_K +blk\.9\.ffn_down_shexp\.weight=Q4_K +blk\.9\.ffn_gate_exps\.weight=Q3_K +blk\.9\.ffn_gate_shexp\.weight=Q4_K +blk\.9\.ffn_up_exps\.weight=Q3_K +blk\.9\.ffn_up_shexp\.weight=Q4_K +blk\.9\.ssm_alpha\.weight=Q3_K +blk\.9\.ssm_beta\.weight=Q3_K +blk\.9\.ssm_out\.weight=Q3_K +blk\.10\.attn_gate\.weight=Q3_K +blk\.10\.attn_qkv\.weight=Q3_K +blk\.10\.ffn_down_exps\.weight=IQ2_S +blk\.10\.ffn_down_shexp\.weight=Q4_K +blk\.10\.ffn_gate_exps\.weight=IQ2_S +blk\.10\.ffn_gate_shexp\.weight=Q4_K +blk\.10\.ffn_up_exps\.weight=IQ2_S +blk\.10\.ffn_up_shexp\.weight=Q4_K +blk\.10\.ssm_alpha\.weight=Q3_K +blk\.10\.ssm_beta\.weight=Q3_K +blk\.10\.ssm_out\.weight=Q3_K +blk\.11\.attn_k\.weight=Q6_K +blk\.11\.attn_output\.weight=Q3_K +blk\.11\.attn_q\.weight=Q3_K +blk\.11\.attn_v\.weight=Q6_K +blk\.11\.ffn_down_exps\.weight=IQ2_S +blk\.11\.ffn_down_shexp\.weight=Q4_K +blk\.11\.ffn_gate_exps\.weight=IQ2_S +blk\.11\.ffn_gate_shexp\.weight=Q4_K +blk\.11\.ffn_up_exps\.weight=IQ2_S +blk\.11\.ffn_up_shexp\.weight=Q4_K +blk\.12\.attn_gate\.weight=Q3_K +blk\.12\.attn_qkv\.weight=Q3_K +blk\.12\.ffn_down_exps\.weight=IQ2_S +blk\.12\.ffn_down_shexp\.weight=Q4_K +blk\.12\.ffn_gate_exps\.weight=IQ2_S +blk\.12\.ffn_gate_shexp\.weight=Q4_K +blk\.12\.ffn_up_exps\.weight=IQ2_S +blk\.12\.ffn_up_shexp\.weight=Q4_K +blk\.12\.ssm_alpha\.weight=Q3_K +blk\.12\.ssm_beta\.weight=Q3_K +blk\.12\.ssm_out\.weight=Q3_K +blk\.13\.attn_gate\.weight=Q3_K +blk\.13\.attn_qkv\.weight=Q3_K +blk\.13\.ffn_down_exps\.weight=IQ2_S +blk\.13\.ffn_down_shexp\.weight=Q4_K +blk\.13\.ffn_gate_exps\.weight=IQ2_S +blk\.13\.ffn_gate_shexp\.weight=Q4_K +blk\.13\.ffn_up_exps\.weight=IQ2_S +blk\.13\.ffn_up_shexp\.weight=Q4_K +blk\.13\.ssm_alpha\.weight=Q3_K +blk\.13\.ssm_beta\.weight=Q3_K +blk\.13\.ssm_out\.weight=Q3_K +blk\.14\.attn_gate\.weight=Q3_K +blk\.14\.attn_qkv\.weight=Q3_K +blk\.14\.ffn_down_exps\.weight=IQ2_S +blk\.14\.ffn_down_shexp\.weight=Q4_K +blk\.14\.ffn_gate_exps\.weight=IQ2_S +blk\.14\.ffn_gate_shexp\.weight=Q4_K +blk\.14\.ffn_up_exps\.weight=IQ2_S +blk\.14\.ffn_up_shexp\.weight=Q4_K +blk\.14\.ssm_alpha\.weight=Q3_K +blk\.14\.ssm_beta\.weight=Q3_K +blk\.14\.ssm_out\.weight=Q3_K +blk\.15\.attn_k\.weight=Q6_K +blk\.15\.attn_output\.weight=Q3_K +blk\.15\.attn_q\.weight=Q3_K +blk\.15\.attn_v\.weight=Q6_K +blk\.15\.ffn_down_exps\.weight=IQ2_S +blk\.15\.ffn_down_shexp\.weight=Q4_K +blk\.15\.ffn_gate_exps\.weight=IQ2_S +blk\.15\.ffn_gate_shexp\.weight=Q4_K +blk\.15\.ffn_up_exps\.weight=IQ2_S +blk\.15\.ffn_up_shexp\.weight=Q4_K +blk\.16\.attn_gate\.weight=Q3_K +blk\.16\.attn_qkv\.weight=Q3_K +blk\.16\.ffn_down_exps\.weight=IQ2_S +blk\.16\.ffn_down_shexp\.weight=Q4_K +blk\.16\.ffn_gate_exps\.weight=IQ2_S +blk\.16\.ffn_gate_shexp\.weight=Q4_K +blk\.16\.ffn_up_exps\.weight=IQ2_S +blk\.16\.ffn_up_shexp\.weight=Q4_K +blk\.16\.ssm_alpha\.weight=Q3_K +blk\.16\.ssm_beta\.weight=Q3_K +blk\.16\.ssm_out\.weight=Q3_K +blk\.17\.attn_gate\.weight=Q3_K +blk\.17\.attn_qkv\.weight=Q3_K +blk\.17\.ffn_down_exps\.weight=IQ2_S +blk\.17\.ffn_down_shexp\.weight=Q4_K +blk\.17\.ffn_gate_exps\.weight=IQ2_S +blk\.17\.ffn_gate_shexp\.weight=Q4_K +blk\.17\.ffn_up_exps\.weight=IQ2_S +blk\.17\.ffn_up_shexp\.weight=Q4_K +blk\.17\.ssm_alpha\.weight=Q3_K +blk\.17\.ssm_beta\.weight=Q3_K +blk\.17\.ssm_out\.weight=Q3_K +blk\.18\.attn_gate\.weight=Q3_K +blk\.18\.attn_qkv\.weight=Q3_K +blk\.18\.ffn_down_exps\.weight=IQ2_S +blk\.18\.ffn_down_shexp\.weight=Q4_K +blk\.18\.ffn_gate_exps\.weight=IQ2_S +blk\.18\.ffn_gate_shexp\.weight=Q4_K +blk\.18\.ffn_up_exps\.weight=IQ2_S +blk\.18\.ffn_up_shexp\.weight=Q4_K +blk\.18\.ssm_alpha\.weight=Q3_K +blk\.18\.ssm_beta\.weight=Q3_K +blk\.18\.ssm_out\.weight=Q3_K +blk\.19\.attn_k\.weight=Q6_K +blk\.19\.attn_output\.weight=Q3_K +blk\.19\.attn_q\.weight=Q3_K +blk\.19\.attn_v\.weight=Q6_K +blk\.19\.ffn_down_exps\.weight=IQ2_S +blk\.19\.ffn_down_shexp\.weight=Q4_K +blk\.19\.ffn_gate_exps\.weight=IQ2_S +blk\.19\.ffn_gate_shexp\.weight=Q4_K +blk\.19\.ffn_up_exps\.weight=IQ2_S +blk\.19\.ffn_up_shexp\.weight=Q4_K +blk\.20\.attn_gate\.weight=Q3_K +blk\.20\.attn_qkv\.weight=Q3_K +blk\.20\.ffn_down_exps\.weight=IQ2_S +blk\.20\.ffn_down_shexp\.weight=Q4_K +blk\.20\.ffn_gate_exps\.weight=IQ2_S +blk\.20\.ffn_gate_shexp\.weight=Q4_K +blk\.20\.ffn_up_exps\.weight=IQ2_S +blk\.20\.ffn_up_shexp\.weight=Q4_K +blk\.20\.ssm_alpha\.weight=Q3_K +blk\.20\.ssm_beta\.weight=Q3_K +blk\.20\.ssm_out\.weight=Q3_K +blk\.21\.attn_gate\.weight=Q3_K +blk\.21\.attn_qkv\.weight=Q3_K +blk\.21\.ffn_down_exps\.weight=IQ2_S +blk\.21\.ffn_down_shexp\.weight=Q4_K +blk\.21\.ffn_gate_exps\.weight=IQ2_S +blk\.21\.ffn_gate_shexp\.weight=Q4_K +blk\.21\.ffn_up_exps\.weight=IQ2_S +blk\.21\.ffn_up_shexp\.weight=Q4_K +blk\.21\.ssm_alpha\.weight=Q3_K +blk\.21\.ssm_beta\.weight=Q3_K +blk\.21\.ssm_out\.weight=Q3_K +blk\.22\.attn_gate\.weight=Q3_K +blk\.22\.attn_qkv\.weight=Q3_K +blk\.22\.ffn_down_exps\.weight=IQ2_S +blk\.22\.ffn_down_shexp\.weight=Q4_K +blk\.22\.ffn_gate_exps\.weight=IQ2_S +blk\.22\.ffn_gate_shexp\.weight=Q4_K +blk\.22\.ffn_up_exps\.weight=IQ2_S +blk\.22\.ffn_up_shexp\.weight=Q4_K +blk\.22\.ssm_alpha\.weight=Q3_K +blk\.22\.ssm_beta\.weight=Q3_K +blk\.22\.ssm_out\.weight=Q3_K +blk\.23\.attn_k\.weight=Q6_K +blk\.23\.attn_output\.weight=Q3_K +blk\.23\.attn_q\.weight=Q3_K +blk\.23\.attn_v\.weight=Q6_K +blk\.23\.ffn_down_exps\.weight=IQ2_S +blk\.23\.ffn_down_shexp\.weight=Q4_K +blk\.23\.ffn_gate_exps\.weight=IQ2_S +blk\.23\.ffn_gate_shexp\.weight=Q4_K +blk\.23\.ffn_up_exps\.weight=IQ2_S +blk\.23\.ffn_up_shexp\.weight=Q4_K +blk\.24\.attn_gate\.weight=Q3_K +blk\.24\.attn_qkv\.weight=Q3_K +blk\.24\.ffn_down_exps\.weight=IQ2_S +blk\.24\.ffn_down_shexp\.weight=Q4_K +blk\.24\.ffn_gate_exps\.weight=IQ2_S +blk\.24\.ffn_gate_shexp\.weight=Q4_K +blk\.24\.ffn_up_exps\.weight=IQ2_S +blk\.24\.ffn_up_shexp\.weight=Q4_K +blk\.24\.ssm_alpha\.weight=Q3_K +blk\.24\.ssm_beta\.weight=Q3_K +blk\.24\.ssm_out\.weight=Q3_K +blk\.25\.attn_gate\.weight=Q3_K +blk\.25\.attn_qkv\.weight=Q3_K +blk\.25\.ffn_down_exps\.weight=IQ2_S +blk\.25\.ffn_down_shexp\.weight=Q4_K +blk\.25\.ffn_gate_exps\.weight=IQ2_S +blk\.25\.ffn_gate_shexp\.weight=Q4_K +blk\.25\.ffn_up_exps\.weight=IQ2_S +blk\.25\.ffn_up_shexp\.weight=Q4_K +blk\.25\.ssm_alpha\.weight=Q3_K +blk\.25\.ssm_beta\.weight=Q3_K +blk\.25\.ssm_out\.weight=Q3_K +blk\.26\.attn_gate\.weight=Q3_K +blk\.26\.attn_qkv\.weight=Q3_K +blk\.26\.ffn_down_exps\.weight=IQ2_S +blk\.26\.ffn_down_shexp\.weight=Q4_K +blk\.26\.ffn_gate_exps\.weight=IQ2_S +blk\.26\.ffn_gate_shexp\.weight=Q4_K +blk\.26\.ffn_up_exps\.weight=IQ2_S +blk\.26\.ffn_up_shexp\.weight=Q4_K +blk\.26\.ssm_alpha\.weight=Q3_K +blk\.26\.ssm_beta\.weight=Q3_K +blk\.26\.ssm_out\.weight=Q3_K +blk\.27\.attn_k\.weight=Q6_K +blk\.27\.attn_output\.weight=Q3_K +blk\.27\.attn_q\.weight=Q3_K +blk\.27\.attn_v\.weight=Q6_K +blk\.27\.ffn_down_exps\.weight=IQ2_S +blk\.27\.ffn_down_shexp\.weight=Q4_K +blk\.27\.ffn_gate_exps\.weight=IQ2_S +blk\.27\.ffn_gate_shexp\.weight=Q4_K +blk\.27\.ffn_up_exps\.weight=IQ2_S +blk\.27\.ffn_up_shexp\.weight=Q4_K +blk\.28\.attn_gate\.weight=Q3_K +blk\.28\.attn_qkv\.weight=Q3_K +blk\.28\.ffn_down_exps\.weight=IQ2_S +blk\.28\.ffn_down_shexp\.weight=Q4_K +blk\.28\.ffn_gate_exps\.weight=IQ2_S +blk\.28\.ffn_gate_shexp\.weight=Q4_K +blk\.28\.ffn_up_exps\.weight=IQ2_S +blk\.28\.ffn_up_shexp\.weight=Q4_K +blk\.28\.ssm_alpha\.weight=Q3_K +blk\.28\.ssm_beta\.weight=Q3_K +blk\.28\.ssm_out\.weight=Q3_K +blk\.29\.attn_gate\.weight=Q3_K +blk\.29\.attn_qkv\.weight=Q3_K +blk\.29\.ffn_down_exps\.weight=IQ2_S +blk\.29\.ffn_down_shexp\.weight=Q4_K +blk\.29\.ffn_gate_exps\.weight=IQ2_S +blk\.29\.ffn_gate_shexp\.weight=Q4_K +blk\.29\.ffn_up_exps\.weight=IQ2_S +blk\.29\.ffn_up_shexp\.weight=Q4_K +blk\.29\.ssm_alpha\.weight=Q3_K +blk\.29\.ssm_beta\.weight=Q3_K +blk\.29\.ssm_out\.weight=Q3_K +blk\.30\.attn_gate\.weight=Q3_K +blk\.30\.attn_qkv\.weight=Q3_K +blk\.30\.ffn_down_exps\.weight=Q3_K +blk\.30\.ffn_down_shexp\.weight=Q4_K +blk\.30\.ffn_gate_exps\.weight=Q3_K +blk\.30\.ffn_gate_shexp\.weight=Q4_K +blk\.30\.ffn_up_exps\.weight=Q3_K +blk\.30\.ffn_up_shexp\.weight=Q4_K +blk\.30\.ssm_alpha\.weight=Q3_K +blk\.30\.ssm_beta\.weight=Q3_K +blk\.30\.ssm_out\.weight=Q3_K +blk\.31\.attn_k\.weight=Q6_K +blk\.31\.attn_output\.weight=Q3_K +blk\.31\.attn_q\.weight=Q3_K +blk\.31\.attn_v\.weight=Q6_K +blk\.31\.ffn_down_exps\.weight=Q3_K +blk\.31\.ffn_down_shexp\.weight=Q4_K +blk\.31\.ffn_gate_exps\.weight=Q3_K +blk\.31\.ffn_gate_shexp\.weight=Q4_K +blk\.31\.ffn_up_exps\.weight=Q3_K +blk\.31\.ffn_up_shexp\.weight=Q4_K +blk\.32\.attn_gate\.weight=Q3_K +blk\.32\.attn_qkv\.weight=Q3_K +blk\.32\.ffn_down_exps\.weight=Q3_K +blk\.32\.ffn_down_shexp\.weight=Q4_K +blk\.32\.ffn_gate_exps\.weight=Q3_K +blk\.32\.ffn_gate_shexp\.weight=Q4_K +blk\.32\.ffn_up_exps\.weight=Q3_K +blk\.32\.ffn_up_shexp\.weight=Q4_K +blk\.32\.ssm_alpha\.weight=Q3_K +blk\.32\.ssm_beta\.weight=Q3_K +blk\.32\.ssm_out\.weight=Q3_K +blk\.33\.attn_gate\.weight=Q3_K +blk\.33\.attn_qkv\.weight=Q3_K +blk\.33\.ffn_down_exps\.weight=Q3_K +blk\.33\.ffn_down_shexp\.weight=Q4_K +blk\.33\.ffn_gate_exps\.weight=Q3_K +blk\.33\.ffn_gate_shexp\.weight=Q4_K +blk\.33\.ffn_up_exps\.weight=Q3_K +blk\.33\.ffn_up_shexp\.weight=Q4_K +blk\.33\.ssm_alpha\.weight=Q3_K +blk\.33\.ssm_beta\.weight=Q3_K +blk\.33\.ssm_out\.weight=Q3_K +blk\.34\.attn_gate\.weight=Q3_K +blk\.34\.attn_qkv\.weight=Q3_K +blk\.34\.ffn_down_exps\.weight=Q3_K +blk\.34\.ffn_down_shexp\.weight=Q4_K +blk\.34\.ffn_gate_exps\.weight=Q3_K +blk\.34\.ffn_gate_shexp\.weight=Q4_K +blk\.34\.ffn_up_exps\.weight=Q3_K +blk\.34\.ffn_up_shexp\.weight=Q4_K +blk\.34\.ssm_alpha\.weight=Q3_K +blk\.34\.ssm_beta\.weight=Q3_K +blk\.34\.ssm_out\.weight=Q3_K +blk\.35\.attn_k\.weight=Q6_K +blk\.35\.attn_output\.weight=Q3_K +blk\.35\.attn_q\.weight=Q3_K +blk\.35\.attn_v\.weight=Q6_K +blk\.35\.ffn_down_exps\.weight=Q3_K +blk\.35\.ffn_down_shexp\.weight=Q5_K +blk\.35\.ffn_gate_exps\.weight=Q3_K +blk\.35\.ffn_gate_shexp\.weight=Q5_K +blk\.35\.ffn_up_exps\.weight=Q3_K +blk\.35\.ffn_up_shexp\.weight=Q5_K +blk\.36\.attn_gate\.weight=Q3_K +blk\.36\.attn_qkv\.weight=Q3_K +blk\.36\.ffn_down_exps\.weight=Q3_K +blk\.36\.ffn_down_shexp\.weight=Q5_K +blk\.36\.ffn_gate_exps\.weight=Q3_K +blk\.36\.ffn_gate_shexp\.weight=Q5_K +blk\.36\.ffn_up_exps\.weight=Q3_K +blk\.36\.ffn_up_shexp\.weight=Q5_K +blk\.36\.ssm_alpha\.weight=Q3_K +blk\.36\.ssm_beta\.weight=Q3_K +blk\.36\.ssm_out\.weight=Q3_K +blk\.37\.attn_gate\.weight=Q4_K +blk\.37\.attn_qkv\.weight=Q4_K +blk\.37\.ffn_down_exps\.weight=Q3_K +blk\.37\.ffn_down_shexp\.weight=Q5_K +blk\.37\.ffn_gate_exps\.weight=Q3_K +blk\.37\.ffn_gate_shexp\.weight=Q5_K +blk\.37\.ffn_up_exps\.weight=Q3_K +blk\.37\.ffn_up_shexp\.weight=Q5_K +blk\.37\.ssm_alpha\.weight=Q4_K +blk\.37\.ssm_beta\.weight=Q4_K +blk\.37\.ssm_out\.weight=Q4_K +blk\.38\.attn_gate\.weight=Q4_K +blk\.38\.attn_qkv\.weight=Q4_K +blk\.38\.ffn_down_exps\.weight=Q3_K +blk\.38\.ffn_down_shexp\.weight=Q5_K +blk\.38\.ffn_gate_exps\.weight=Q3_K +blk\.38\.ffn_gate_shexp\.weight=Q5_K +blk\.38\.ffn_up_exps\.weight=Q3_K +blk\.38\.ffn_up_shexp\.weight=Q5_K +blk\.38\.ssm_alpha\.weight=Q4_K +blk\.38\.ssm_beta\.weight=Q4_K +blk\.38\.ssm_out\.weight=Q4_K +blk\.39\.attn_k\.weight=Q6_K +blk\.39\.attn_output\.weight=Q4_K +blk\.39\.attn_q\.weight=Q4_K +blk\.39\.attn_v\.weight=Q6_K +blk\.39\.ffn_down_exps\.weight=Q3_K +blk\.39\.ffn_down_shexp\.weight=Q5_K +blk\.39\.ffn_gate_exps\.weight=Q3_K +blk\.39\.ffn_gate_shexp\.weight=Q5_K +blk\.39\.ffn_up_exps\.weight=Q3_K +blk\.39\.ffn_up_shexp\.weight=Q5_K +blk\.40\.attn_k\.weight=Q8_0 +blk\.40\.attn_output\.weight=Q8_0 +blk\.40\.attn_q\.weight=Q8_0 +blk\.40\.attn_v\.weight=Q8_0 +blk\.40\.ffn_down_exps\.weight=Q8_0 +blk\.40\.ffn_down_shexp\.weight=Q8_0 +blk\.40\.ffn_gate_exps\.weight=Q8_0 +blk\.40\.ffn_gate_shexp\.weight=Q8_0 +blk\.40\.ffn_up_exps\.weight=Q8_0 +blk\.40\.ffn_up_shexp\.weight=Q8_0 +blk\.40\.nextn\.eh_proj\.weight=Q8_0 diff --git a/appendix/recipes/cfg_UD-IQ4_XS_comparison.txt b/appendix/recipes/cfg_UD-IQ4_XS_comparison.txt new file mode 100644 index 0000000000000000000000000000000000000000..f517e0be37a1e2f81963e49a598b51e4cea9b5ad --- /dev/null +++ b/appendix/recipes/cfg_UD-IQ4_XS_comparison.txt @@ -0,0 +1,443 @@ +^output\.weight$=Q6_K +token_embd\.weight=Q8_0 +blk\.0\.attn_gate\.weight=Q8_0 +blk\.0\.attn_qkv\.weight=Q8_0 +blk\.0\.ffn_down_exps\.weight=IQ4_XS +blk\.0\.ffn_down_shexp\.weight=Q8_0 +blk\.0\.ffn_gate_exps\.weight=IQ3_S +blk\.0\.ffn_gate_shexp\.weight=Q8_0 +blk\.0\.ffn_up_exps\.weight=IQ3_S +blk\.0\.ffn_up_shexp\.weight=Q8_0 +blk\.0\.ssm_alpha\.weight=F32 +blk\.0\.ssm_beta\.weight=F32 +blk\.0\.ssm_out\.weight=Q8_0 +blk\.1\.attn_gate\.weight=Q8_0 +blk\.1\.attn_qkv\.weight=Q8_0 +blk\.1\.ffn_down_exps\.weight=IQ4_XS +blk\.1\.ffn_down_shexp\.weight=Q8_0 +blk\.1\.ffn_gate_exps\.weight=IQ3_S +blk\.1\.ffn_gate_shexp\.weight=Q8_0 +blk\.1\.ffn_up_exps\.weight=IQ3_S +blk\.1\.ffn_up_shexp\.weight=Q8_0 +blk\.1\.ssm_alpha\.weight=F32 +blk\.1\.ssm_beta\.weight=F32 +blk\.1\.ssm_out\.weight=Q8_0 +blk\.2\.attn_gate\.weight=Q8_0 +blk\.2\.attn_qkv\.weight=Q8_0 +blk\.2\.ffn_down_exps\.weight=IQ4_XS +blk\.2\.ffn_down_shexp\.weight=Q8_0 +blk\.2\.ffn_gate_exps\.weight=IQ3_S +blk\.2\.ffn_gate_shexp\.weight=Q8_0 +blk\.2\.ffn_up_exps\.weight=IQ3_S +blk\.2\.ffn_up_shexp\.weight=Q8_0 +blk\.2\.ssm_alpha\.weight=F32 +blk\.2\.ssm_beta\.weight=F32 +blk\.2\.ssm_out\.weight=Q8_0 +blk\.3\.attn_k\.weight=Q8_0 +blk\.3\.attn_output\.weight=Q8_0 +blk\.3\.attn_q\.weight=Q8_0 +blk\.3\.attn_v\.weight=Q8_0 +blk\.3\.ffn_down_exps\.weight=IQ4_XS +blk\.3\.ffn_down_shexp\.weight=Q8_0 +blk\.3\.ffn_gate_exps\.weight=IQ3_S +blk\.3\.ffn_gate_shexp\.weight=Q8_0 +blk\.3\.ffn_up_exps\.weight=IQ3_S +blk\.3\.ffn_up_shexp\.weight=Q8_0 +blk\.4\.attn_gate\.weight=Q8_0 +blk\.4\.attn_qkv\.weight=Q8_0 +blk\.4\.ffn_down_exps\.weight=IQ4_XS +blk\.4\.ffn_down_shexp\.weight=Q8_0 +blk\.4\.ffn_gate_exps\.weight=IQ3_S +blk\.4\.ffn_gate_shexp\.weight=Q8_0 +blk\.4\.ffn_up_exps\.weight=IQ3_S +blk\.4\.ffn_up_shexp\.weight=Q8_0 +blk\.4\.ssm_alpha\.weight=F32 +blk\.4\.ssm_beta\.weight=F32 +blk\.4\.ssm_out\.weight=Q8_0 +blk\.5\.attn_gate\.weight=Q8_0 +blk\.5\.attn_qkv\.weight=Q8_0 +blk\.5\.ffn_down_exps\.weight=IQ4_XS +blk\.5\.ffn_down_shexp\.weight=Q8_0 +blk\.5\.ffn_gate_exps\.weight=IQ3_S +blk\.5\.ffn_gate_shexp\.weight=Q8_0 +blk\.5\.ffn_up_exps\.weight=IQ3_S +blk\.5\.ffn_up_shexp\.weight=Q8_0 +blk\.5\.ssm_alpha\.weight=F32 +blk\.5\.ssm_beta\.weight=F32 +blk\.5\.ssm_out\.weight=Q8_0 +blk\.6\.attn_gate\.weight=Q8_0 +blk\.6\.attn_qkv\.weight=Q8_0 +blk\.6\.ffn_down_exps\.weight=IQ4_XS +blk\.6\.ffn_down_shexp\.weight=Q8_0 +blk\.6\.ffn_gate_exps\.weight=IQ3_S +blk\.6\.ffn_gate_shexp\.weight=Q8_0 +blk\.6\.ffn_up_exps\.weight=IQ3_S +blk\.6\.ffn_up_shexp\.weight=Q8_0 +blk\.6\.ssm_alpha\.weight=F32 +blk\.6\.ssm_beta\.weight=F32 +blk\.6\.ssm_out\.weight=Q8_0 +blk\.7\.attn_k\.weight=Q8_0 +blk\.7\.attn_output\.weight=Q8_0 +blk\.7\.attn_q\.weight=Q8_0 +blk\.7\.attn_v\.weight=Q8_0 +blk\.7\.ffn_down_exps\.weight=IQ4_XS +blk\.7\.ffn_down_shexp\.weight=Q8_0 +blk\.7\.ffn_gate_exps\.weight=IQ3_S +blk\.7\.ffn_gate_shexp\.weight=Q8_0 +blk\.7\.ffn_up_exps\.weight=IQ3_S +blk\.7\.ffn_up_shexp\.weight=Q8_0 +blk\.8\.attn_gate\.weight=Q8_0 +blk\.8\.attn_qkv\.weight=Q8_0 +blk\.8\.ffn_down_exps\.weight=IQ4_XS +blk\.8\.ffn_down_shexp\.weight=Q8_0 +blk\.8\.ffn_gate_exps\.weight=IQ3_S +blk\.8\.ffn_gate_shexp\.weight=Q8_0 +blk\.8\.ffn_up_exps\.weight=IQ3_S +blk\.8\.ffn_up_shexp\.weight=Q8_0 +blk\.8\.ssm_alpha\.weight=F32 +blk\.8\.ssm_beta\.weight=F32 +blk\.8\.ssm_out\.weight=Q8_0 +blk\.9\.attn_gate\.weight=Q8_0 +blk\.9\.attn_qkv\.weight=Q8_0 +blk\.9\.ffn_down_exps\.weight=IQ4_XS +blk\.9\.ffn_down_shexp\.weight=Q8_0 +blk\.9\.ffn_gate_exps\.weight=IQ3_S +blk\.9\.ffn_gate_shexp\.weight=Q8_0 +blk\.9\.ffn_up_exps\.weight=IQ3_S +blk\.9\.ffn_up_shexp\.weight=Q8_0 +blk\.9\.ssm_alpha\.weight=F32 +blk\.9\.ssm_beta\.weight=F32 +blk\.9\.ssm_out\.weight=Q8_0 +blk\.10\.attn_gate\.weight=Q8_0 +blk\.10\.attn_qkv\.weight=Q8_0 +blk\.10\.ffn_down_exps\.weight=IQ4_XS +blk\.10\.ffn_down_shexp\.weight=Q8_0 +blk\.10\.ffn_gate_exps\.weight=IQ3_S +blk\.10\.ffn_gate_shexp\.weight=Q8_0 +blk\.10\.ffn_up_exps\.weight=IQ3_S +blk\.10\.ffn_up_shexp\.weight=Q8_0 +blk\.10\.ssm_alpha\.weight=F32 +blk\.10\.ssm_beta\.weight=F32 +blk\.10\.ssm_out\.weight=Q8_0 +blk\.11\.attn_k\.weight=Q8_0 +blk\.11\.attn_output\.weight=Q8_0 +blk\.11\.attn_q\.weight=Q8_0 +blk\.11\.attn_v\.weight=Q8_0 +blk\.11\.ffn_down_exps\.weight=IQ4_XS +blk\.11\.ffn_down_shexp\.weight=Q8_0 +blk\.11\.ffn_gate_exps\.weight=IQ3_S +blk\.11\.ffn_gate_shexp\.weight=Q8_0 +blk\.11\.ffn_up_exps\.weight=IQ3_S +blk\.11\.ffn_up_shexp\.weight=Q8_0 +blk\.12\.attn_gate\.weight=Q8_0 +blk\.12\.attn_qkv\.weight=Q8_0 +blk\.12\.ffn_down_exps\.weight=IQ4_XS +blk\.12\.ffn_down_shexp\.weight=Q8_0 +blk\.12\.ffn_gate_exps\.weight=IQ3_S +blk\.12\.ffn_gate_shexp\.weight=Q8_0 +blk\.12\.ffn_up_exps\.weight=IQ3_S +blk\.12\.ffn_up_shexp\.weight=Q8_0 +blk\.12\.ssm_alpha\.weight=F32 +blk\.12\.ssm_beta\.weight=F32 +blk\.12\.ssm_out\.weight=Q8_0 +blk\.13\.attn_gate\.weight=Q8_0 +blk\.13\.attn_qkv\.weight=Q8_0 +blk\.13\.ffn_down_exps\.weight=IQ4_XS +blk\.13\.ffn_down_shexp\.weight=Q8_0 +blk\.13\.ffn_gate_exps\.weight=IQ3_S +blk\.13\.ffn_gate_shexp\.weight=Q8_0 +blk\.13\.ffn_up_exps\.weight=IQ3_S +blk\.13\.ffn_up_shexp\.weight=Q8_0 +blk\.13\.ssm_alpha\.weight=F32 +blk\.13\.ssm_beta\.weight=F32 +blk\.13\.ssm_out\.weight=Q8_0 +blk\.14\.attn_gate\.weight=Q8_0 +blk\.14\.attn_qkv\.weight=Q8_0 +blk\.14\.ffn_down_exps\.weight=IQ4_XS +blk\.14\.ffn_down_shexp\.weight=Q8_0 +blk\.14\.ffn_gate_exps\.weight=IQ3_S +blk\.14\.ffn_gate_shexp\.weight=Q8_0 +blk\.14\.ffn_up_exps\.weight=IQ3_S +blk\.14\.ffn_up_shexp\.weight=Q8_0 +blk\.14\.ssm_alpha\.weight=F32 +blk\.14\.ssm_beta\.weight=F32 +blk\.14\.ssm_out\.weight=Q8_0 +blk\.15\.attn_k\.weight=Q8_0 +blk\.15\.attn_output\.weight=Q8_0 +blk\.15\.attn_q\.weight=Q8_0 +blk\.15\.attn_v\.weight=Q8_0 +blk\.15\.ffn_down_exps\.weight=IQ4_XS +blk\.15\.ffn_down_shexp\.weight=Q8_0 +blk\.15\.ffn_gate_exps\.weight=IQ3_S +blk\.15\.ffn_gate_shexp\.weight=Q8_0 +blk\.15\.ffn_up_exps\.weight=IQ3_S +blk\.15\.ffn_up_shexp\.weight=Q8_0 +blk\.16\.attn_gate\.weight=Q8_0 +blk\.16\.attn_qkv\.weight=Q8_0 +blk\.16\.ffn_down_exps\.weight=IQ4_XS +blk\.16\.ffn_down_shexp\.weight=Q8_0 +blk\.16\.ffn_gate_exps\.weight=IQ3_S +blk\.16\.ffn_gate_shexp\.weight=Q8_0 +blk\.16\.ffn_up_exps\.weight=IQ3_S +blk\.16\.ffn_up_shexp\.weight=Q8_0 +blk\.16\.ssm_alpha\.weight=F32 +blk\.16\.ssm_beta\.weight=F32 +blk\.16\.ssm_out\.weight=Q8_0 +blk\.17\.attn_gate\.weight=Q8_0 +blk\.17\.attn_qkv\.weight=Q8_0 +blk\.17\.ffn_down_exps\.weight=IQ4_XS +blk\.17\.ffn_down_shexp\.weight=Q8_0 +blk\.17\.ffn_gate_exps\.weight=IQ3_S +blk\.17\.ffn_gate_shexp\.weight=Q8_0 +blk\.17\.ffn_up_exps\.weight=IQ3_S +blk\.17\.ffn_up_shexp\.weight=Q8_0 +blk\.17\.ssm_alpha\.weight=F32 +blk\.17\.ssm_beta\.weight=F32 +blk\.17\.ssm_out\.weight=Q8_0 +blk\.18\.attn_gate\.weight=Q8_0 +blk\.18\.attn_qkv\.weight=Q8_0 +blk\.18\.ffn_down_exps\.weight=IQ4_XS +blk\.18\.ffn_down_shexp\.weight=Q8_0 +blk\.18\.ffn_gate_exps\.weight=IQ3_S +blk\.18\.ffn_gate_shexp\.weight=Q8_0 +blk\.18\.ffn_up_exps\.weight=IQ3_S +blk\.18\.ffn_up_shexp\.weight=Q8_0 +blk\.18\.ssm_alpha\.weight=F32 +blk\.18\.ssm_beta\.weight=F32 +blk\.18\.ssm_out\.weight=Q8_0 +blk\.19\.attn_k\.weight=Q8_0 +blk\.19\.attn_output\.weight=Q8_0 +blk\.19\.attn_q\.weight=Q8_0 +blk\.19\.attn_v\.weight=Q8_0 +blk\.19\.ffn_down_exps\.weight=IQ4_XS +blk\.19\.ffn_down_shexp\.weight=Q8_0 +blk\.19\.ffn_gate_exps\.weight=IQ3_S +blk\.19\.ffn_gate_shexp\.weight=Q8_0 +blk\.19\.ffn_up_exps\.weight=IQ3_S +blk\.19\.ffn_up_shexp\.weight=Q8_0 +blk\.20\.attn_gate\.weight=Q8_0 +blk\.20\.attn_qkv\.weight=Q8_0 +blk\.20\.ffn_down_exps\.weight=IQ4_XS +blk\.20\.ffn_down_shexp\.weight=Q8_0 +blk\.20\.ffn_gate_exps\.weight=IQ3_S +blk\.20\.ffn_gate_shexp\.weight=Q8_0 +blk\.20\.ffn_up_exps\.weight=IQ3_S +blk\.20\.ffn_up_shexp\.weight=Q8_0 +blk\.20\.ssm_alpha\.weight=F32 +blk\.20\.ssm_beta\.weight=F32 +blk\.20\.ssm_out\.weight=Q8_0 +blk\.21\.attn_gate\.weight=Q8_0 +blk\.21\.attn_qkv\.weight=Q8_0 +blk\.21\.ffn_down_exps\.weight=IQ4_XS +blk\.21\.ffn_down_shexp\.weight=Q8_0 +blk\.21\.ffn_gate_exps\.weight=IQ3_S +blk\.21\.ffn_gate_shexp\.weight=Q8_0 +blk\.21\.ffn_up_exps\.weight=IQ3_S +blk\.21\.ffn_up_shexp\.weight=Q8_0 +blk\.21\.ssm_alpha\.weight=F32 +blk\.21\.ssm_beta\.weight=F32 +blk\.21\.ssm_out\.weight=Q8_0 +blk\.22\.attn_gate\.weight=Q8_0 +blk\.22\.attn_qkv\.weight=Q8_0 +blk\.22\.ffn_down_exps\.weight=IQ4_XS +blk\.22\.ffn_down_shexp\.weight=Q8_0 +blk\.22\.ffn_gate_exps\.weight=IQ3_S +blk\.22\.ffn_gate_shexp\.weight=Q8_0 +blk\.22\.ffn_up_exps\.weight=IQ3_S +blk\.22\.ffn_up_shexp\.weight=Q8_0 +blk\.22\.ssm_alpha\.weight=F32 +blk\.22\.ssm_beta\.weight=F32 +blk\.22\.ssm_out\.weight=Q8_0 +blk\.23\.attn_k\.weight=Q8_0 +blk\.23\.attn_output\.weight=Q8_0 +blk\.23\.attn_q\.weight=Q8_0 +blk\.23\.attn_v\.weight=Q8_0 +blk\.23\.ffn_down_exps\.weight=IQ4_XS +blk\.23\.ffn_down_shexp\.weight=Q8_0 +blk\.23\.ffn_gate_exps\.weight=IQ3_S +blk\.23\.ffn_gate_shexp\.weight=Q8_0 +blk\.23\.ffn_up_exps\.weight=IQ3_S +blk\.23\.ffn_up_shexp\.weight=Q8_0 +blk\.24\.attn_gate\.weight=Q8_0 +blk\.24\.attn_qkv\.weight=Q8_0 +blk\.24\.ffn_down_exps\.weight=IQ4_XS +blk\.24\.ffn_down_shexp\.weight=Q8_0 +blk\.24\.ffn_gate_exps\.weight=IQ3_S +blk\.24\.ffn_gate_shexp\.weight=Q8_0 +blk\.24\.ffn_up_exps\.weight=IQ3_S +blk\.24\.ffn_up_shexp\.weight=Q8_0 +blk\.24\.ssm_alpha\.weight=F32 +blk\.24\.ssm_beta\.weight=F32 +blk\.24\.ssm_out\.weight=Q8_0 +blk\.25\.attn_gate\.weight=Q8_0 +blk\.25\.attn_qkv\.weight=Q8_0 +blk\.25\.ffn_down_exps\.weight=IQ4_XS +blk\.25\.ffn_down_shexp\.weight=Q8_0 +blk\.25\.ffn_gate_exps\.weight=IQ3_S +blk\.25\.ffn_gate_shexp\.weight=Q8_0 +blk\.25\.ffn_up_exps\.weight=IQ3_S +blk\.25\.ffn_up_shexp\.weight=Q8_0 +blk\.25\.ssm_alpha\.weight=F32 +blk\.25\.ssm_beta\.weight=F32 +blk\.25\.ssm_out\.weight=Q8_0 +blk\.26\.attn_gate\.weight=Q8_0 +blk\.26\.attn_qkv\.weight=Q8_0 +blk\.26\.ffn_down_exps\.weight=IQ4_XS +blk\.26\.ffn_down_shexp\.weight=Q8_0 +blk\.26\.ffn_gate_exps\.weight=IQ3_S +blk\.26\.ffn_gate_shexp\.weight=Q8_0 +blk\.26\.ffn_up_exps\.weight=IQ3_S +blk\.26\.ffn_up_shexp\.weight=Q8_0 +blk\.26\.ssm_alpha\.weight=F32 +blk\.26\.ssm_beta\.weight=F32 +blk\.26\.ssm_out\.weight=Q8_0 +blk\.27\.attn_k\.weight=Q8_0 +blk\.27\.attn_output\.weight=Q8_0 +blk\.27\.attn_q\.weight=Q8_0 +blk\.27\.attn_v\.weight=Q8_0 +blk\.27\.ffn_down_exps\.weight=IQ4_XS +blk\.27\.ffn_down_shexp\.weight=Q8_0 +blk\.27\.ffn_gate_exps\.weight=IQ3_S +blk\.27\.ffn_gate_shexp\.weight=Q8_0 +blk\.27\.ffn_up_exps\.weight=IQ3_S +blk\.27\.ffn_up_shexp\.weight=Q8_0 +blk\.28\.attn_gate\.weight=Q8_0 +blk\.28\.attn_qkv\.weight=Q8_0 +blk\.28\.ffn_down_exps\.weight=IQ4_XS +blk\.28\.ffn_down_shexp\.weight=Q8_0 +blk\.28\.ffn_gate_exps\.weight=IQ3_S +blk\.28\.ffn_gate_shexp\.weight=Q8_0 +blk\.28\.ffn_up_exps\.weight=IQ3_S +blk\.28\.ffn_up_shexp\.weight=Q8_0 +blk\.28\.ssm_alpha\.weight=F32 +blk\.28\.ssm_beta\.weight=F32 +blk\.28\.ssm_out\.weight=Q8_0 +blk\.29\.attn_gate\.weight=Q8_0 +blk\.29\.attn_qkv\.weight=Q8_0 +blk\.29\.ffn_down_exps\.weight=IQ4_XS +blk\.29\.ffn_down_shexp\.weight=Q8_0 +blk\.29\.ffn_gate_exps\.weight=IQ3_S +blk\.29\.ffn_gate_shexp\.weight=Q8_0 +blk\.29\.ffn_up_exps\.weight=IQ3_S +blk\.29\.ffn_up_shexp\.weight=Q8_0 +blk\.29\.ssm_alpha\.weight=F32 +blk\.29\.ssm_beta\.weight=F32 +blk\.29\.ssm_out\.weight=Q8_0 +blk\.30\.attn_gate\.weight=Q8_0 +blk\.30\.attn_qkv\.weight=Q8_0 +blk\.30\.ffn_down_exps\.weight=IQ4_XS +blk\.30\.ffn_down_shexp\.weight=Q8_0 +blk\.30\.ffn_gate_exps\.weight=IQ3_S +blk\.30\.ffn_gate_shexp\.weight=Q8_0 +blk\.30\.ffn_up_exps\.weight=IQ3_S +blk\.30\.ffn_up_shexp\.weight=Q8_0 +blk\.30\.ssm_alpha\.weight=F32 +blk\.30\.ssm_beta\.weight=F32 +blk\.30\.ssm_out\.weight=Q8_0 +blk\.31\.attn_k\.weight=Q8_0 +blk\.31\.attn_output\.weight=Q8_0 +blk\.31\.attn_q\.weight=Q8_0 +blk\.31\.attn_v\.weight=Q8_0 +blk\.31\.ffn_down_exps\.weight=IQ4_XS +blk\.31\.ffn_down_shexp\.weight=Q8_0 +blk\.31\.ffn_gate_exps\.weight=IQ3_S +blk\.31\.ffn_gate_shexp\.weight=Q8_0 +blk\.31\.ffn_up_exps\.weight=IQ3_S +blk\.31\.ffn_up_shexp\.weight=Q8_0 +blk\.32\.attn_gate\.weight=Q8_0 +blk\.32\.attn_qkv\.weight=Q8_0 +blk\.32\.ffn_down_exps\.weight=IQ4_XS +blk\.32\.ffn_down_shexp\.weight=Q8_0 +blk\.32\.ffn_gate_exps\.weight=IQ3_S +blk\.32\.ffn_gate_shexp\.weight=Q8_0 +blk\.32\.ffn_up_exps\.weight=IQ3_S +blk\.32\.ffn_up_shexp\.weight=Q8_0 +blk\.32\.ssm_alpha\.weight=F32 +blk\.32\.ssm_beta\.weight=F32 +blk\.32\.ssm_out\.weight=Q8_0 +blk\.33\.attn_gate\.weight=Q8_0 +blk\.33\.attn_qkv\.weight=Q8_0 +blk\.33\.ffn_down_exps\.weight=IQ4_XS +blk\.33\.ffn_down_shexp\.weight=Q8_0 +blk\.33\.ffn_gate_exps\.weight=IQ3_S +blk\.33\.ffn_gate_shexp\.weight=Q8_0 +blk\.33\.ffn_up_exps\.weight=IQ3_S +blk\.33\.ffn_up_shexp\.weight=Q8_0 +blk\.33\.ssm_alpha\.weight=F32 +blk\.33\.ssm_beta\.weight=F32 +blk\.33\.ssm_out\.weight=Q8_0 +blk\.34\.attn_gate\.weight=Q8_0 +blk\.34\.attn_qkv\.weight=Q8_0 +blk\.34\.ffn_down_exps\.weight=Q6_K +blk\.34\.ffn_down_shexp\.weight=Q8_0 +blk\.34\.ffn_gate_exps\.weight=IQ3_S +blk\.34\.ffn_gate_shexp\.weight=Q8_0 +blk\.34\.ffn_up_exps\.weight=IQ3_S +blk\.34\.ffn_up_shexp\.weight=Q8_0 +blk\.34\.ssm_alpha\.weight=F32 +blk\.34\.ssm_beta\.weight=F32 +blk\.34\.ssm_out\.weight=Q8_0 +blk\.35\.attn_k\.weight=Q8_0 +blk\.35\.attn_output\.weight=Q8_0 +blk\.35\.attn_q\.weight=Q8_0 +blk\.35\.attn_v\.weight=Q8_0 +blk\.35\.ffn_down_exps\.weight=IQ4_XS +blk\.35\.ffn_down_shexp\.weight=Q8_0 +blk\.35\.ffn_gate_exps\.weight=IQ3_S +blk\.35\.ffn_gate_shexp\.weight=Q8_0 +blk\.35\.ffn_up_exps\.weight=IQ3_S +blk\.35\.ffn_up_shexp\.weight=Q8_0 +blk\.36\.attn_gate\.weight=Q8_0 +blk\.36\.attn_qkv\.weight=Q8_0 +blk\.36\.ffn_down_exps\.weight=IQ4_XS +blk\.36\.ffn_down_shexp\.weight=Q8_0 +blk\.36\.ffn_gate_exps\.weight=IQ3_S +blk\.36\.ffn_gate_shexp\.weight=Q8_0 +blk\.36\.ffn_up_exps\.weight=IQ3_S +blk\.36\.ffn_up_shexp\.weight=Q8_0 +blk\.36\.ssm_alpha\.weight=F32 +blk\.36\.ssm_beta\.weight=F32 +blk\.36\.ssm_out\.weight=Q8_0 +blk\.37\.attn_gate\.weight=Q8_0 +blk\.37\.attn_qkv\.weight=Q8_0 +blk\.37\.ffn_down_exps\.weight=IQ4_XS +blk\.37\.ffn_down_shexp\.weight=Q8_0 +blk\.37\.ffn_gate_exps\.weight=IQ3_S +blk\.37\.ffn_gate_shexp\.weight=Q8_0 +blk\.37\.ffn_up_exps\.weight=IQ3_S +blk\.37\.ffn_up_shexp\.weight=Q8_0 +blk\.37\.ssm_alpha\.weight=F32 +blk\.37\.ssm_beta\.weight=F32 +blk\.37\.ssm_out\.weight=Q8_0 +blk\.38\.attn_gate\.weight=Q8_0 +blk\.38\.attn_qkv\.weight=Q8_0 +blk\.38\.ffn_down_exps\.weight=Q6_K +blk\.38\.ffn_down_shexp\.weight=Q8_0 +blk\.38\.ffn_gate_exps\.weight=IQ3_S +blk\.38\.ffn_gate_shexp\.weight=Q8_0 +blk\.38\.ffn_up_exps\.weight=IQ3_S +blk\.38\.ffn_up_shexp\.weight=Q8_0 +blk\.38\.ssm_alpha\.weight=F32 +blk\.38\.ssm_beta\.weight=F32 +blk\.38\.ssm_out\.weight=Q8_0 +blk\.39\.attn_k\.weight=Q8_0 +blk\.39\.attn_output\.weight=Q8_0 +blk\.39\.attn_q\.weight=Q8_0 +blk\.39\.attn_v\.weight=Q8_0 +blk\.39\.ffn_down_exps\.weight=Q6_K +blk\.39\.ffn_down_shexp\.weight=Q8_0 +blk\.39\.ffn_gate_exps\.weight=IQ4_XS +blk\.39\.ffn_gate_shexp\.weight=Q8_0 +blk\.39\.ffn_up_exps\.weight=IQ4_XS +blk\.39\.ffn_up_shexp\.weight=Q8_0 +blk\.40\.attn_k\.weight=Q8_0 +blk\.40\.attn_output\.weight=Q8_0 +blk\.40\.attn_q\.weight=Q8_0 +blk\.40\.attn_v\.weight=Q8_0 +blk\.40\.ffn_down_exps\.weight=Q8_0 +blk\.40\.ffn_down_shexp\.weight=Q8_0 +blk\.40\.ffn_gate_exps\.weight=Q8_0 +blk\.40\.ffn_gate_shexp\.weight=Q8_0 +blk\.40\.ffn_up_exps\.weight=Q8_0 +blk\.40\.ffn_up_shexp\.weight=Q8_0 +blk\.40\.nextn\.eh_proj\.weight=Q8_0 diff --git a/appendix/recipes/cfg_UD-Q5_K_S_comparison.txt b/appendix/recipes/cfg_UD-Q5_K_S_comparison.txt new file mode 100644 index 0000000000000000000000000000000000000000..5508ebbd669b5bfc0ab99766fb30cbace96f30af --- /dev/null +++ b/appendix/recipes/cfg_UD-Q5_K_S_comparison.txt @@ -0,0 +1,443 @@ +^output\.weight$=Q8_0 +token_embd\.weight=Q8_0 +blk\.0\.attn_gate\.weight=Q8_0 +blk\.0\.attn_qkv\.weight=Q8_0 +blk\.0\.ffn_down_exps\.weight=Q5_K +blk\.0\.ffn_down_shexp\.weight=Q8_0 +blk\.0\.ffn_gate_exps\.weight=Q5_K +blk\.0\.ffn_gate_shexp\.weight=Q8_0 +blk\.0\.ffn_up_exps\.weight=Q5_K +blk\.0\.ffn_up_shexp\.weight=Q8_0 +blk\.0\.ssm_alpha\.weight=F32 +blk\.0\.ssm_beta\.weight=F32 +blk\.0\.ssm_out\.weight=Q8_0 +blk\.1\.attn_gate\.weight=Q8_0 +blk\.1\.attn_qkv\.weight=Q8_0 +blk\.1\.ffn_down_exps\.weight=Q5_K +blk\.1\.ffn_down_shexp\.weight=Q8_0 +blk\.1\.ffn_gate_exps\.weight=Q5_K +blk\.1\.ffn_gate_shexp\.weight=Q8_0 +blk\.1\.ffn_up_exps\.weight=Q5_K +blk\.1\.ffn_up_shexp\.weight=Q8_0 +blk\.1\.ssm_alpha\.weight=F32 +blk\.1\.ssm_beta\.weight=F32 +blk\.1\.ssm_out\.weight=Q8_0 +blk\.2\.attn_gate\.weight=Q8_0 +blk\.2\.attn_qkv\.weight=Q8_0 +blk\.2\.ffn_down_exps\.weight=Q5_K +blk\.2\.ffn_down_shexp\.weight=Q8_0 +blk\.2\.ffn_gate_exps\.weight=Q5_K +blk\.2\.ffn_gate_shexp\.weight=Q8_0 +blk\.2\.ffn_up_exps\.weight=Q5_K +blk\.2\.ffn_up_shexp\.weight=Q8_0 +blk\.2\.ssm_alpha\.weight=F32 +blk\.2\.ssm_beta\.weight=F32 +blk\.2\.ssm_out\.weight=Q8_0 +blk\.3\.attn_k\.weight=Q8_0 +blk\.3\.attn_output\.weight=Q8_0 +blk\.3\.attn_q\.weight=Q8_0 +blk\.3\.attn_v\.weight=Q8_0 +blk\.3\.ffn_down_exps\.weight=Q5_K +blk\.3\.ffn_down_shexp\.weight=Q8_0 +blk\.3\.ffn_gate_exps\.weight=Q5_K +blk\.3\.ffn_gate_shexp\.weight=Q8_0 +blk\.3\.ffn_up_exps\.weight=Q5_K +blk\.3\.ffn_up_shexp\.weight=Q8_0 +blk\.4\.attn_gate\.weight=Q8_0 +blk\.4\.attn_qkv\.weight=Q8_0 +blk\.4\.ffn_down_exps\.weight=Q5_K +blk\.4\.ffn_down_shexp\.weight=Q8_0 +blk\.4\.ffn_gate_exps\.weight=Q5_K +blk\.4\.ffn_gate_shexp\.weight=Q8_0 +blk\.4\.ffn_up_exps\.weight=Q5_K +blk\.4\.ffn_up_shexp\.weight=Q8_0 +blk\.4\.ssm_alpha\.weight=F32 +blk\.4\.ssm_beta\.weight=F32 +blk\.4\.ssm_out\.weight=Q8_0 +blk\.5\.attn_gate\.weight=Q8_0 +blk\.5\.attn_qkv\.weight=Q8_0 +blk\.5\.ffn_down_exps\.weight=Q5_K +blk\.5\.ffn_down_shexp\.weight=Q8_0 +blk\.5\.ffn_gate_exps\.weight=Q5_K +blk\.5\.ffn_gate_shexp\.weight=Q8_0 +blk\.5\.ffn_up_exps\.weight=Q5_K +blk\.5\.ffn_up_shexp\.weight=Q8_0 +blk\.5\.ssm_alpha\.weight=F32 +blk\.5\.ssm_beta\.weight=F32 +blk\.5\.ssm_out\.weight=Q8_0 +blk\.6\.attn_gate\.weight=Q8_0 +blk\.6\.attn_qkv\.weight=Q8_0 +blk\.6\.ffn_down_exps\.weight=Q5_K +blk\.6\.ffn_down_shexp\.weight=Q8_0 +blk\.6\.ffn_gate_exps\.weight=Q5_K +blk\.6\.ffn_gate_shexp\.weight=Q8_0 +blk\.6\.ffn_up_exps\.weight=Q5_K +blk\.6\.ffn_up_shexp\.weight=Q8_0 +blk\.6\.ssm_alpha\.weight=F32 +blk\.6\.ssm_beta\.weight=F32 +blk\.6\.ssm_out\.weight=Q8_0 +blk\.7\.attn_k\.weight=Q8_0 +blk\.7\.attn_output\.weight=Q8_0 +blk\.7\.attn_q\.weight=Q8_0 +blk\.7\.attn_v\.weight=Q8_0 +blk\.7\.ffn_down_exps\.weight=Q5_K +blk\.7\.ffn_down_shexp\.weight=Q8_0 +blk\.7\.ffn_gate_exps\.weight=Q5_K +blk\.7\.ffn_gate_shexp\.weight=Q8_0 +blk\.7\.ffn_up_exps\.weight=Q5_K +blk\.7\.ffn_up_shexp\.weight=Q8_0 +blk\.8\.attn_gate\.weight=Q8_0 +blk\.8\.attn_qkv\.weight=Q8_0 +blk\.8\.ffn_down_exps\.weight=Q5_K +blk\.8\.ffn_down_shexp\.weight=Q8_0 +blk\.8\.ffn_gate_exps\.weight=Q5_K +blk\.8\.ffn_gate_shexp\.weight=Q8_0 +blk\.8\.ffn_up_exps\.weight=Q5_K +blk\.8\.ffn_up_shexp\.weight=Q8_0 +blk\.8\.ssm_alpha\.weight=F32 +blk\.8\.ssm_beta\.weight=F32 +blk\.8\.ssm_out\.weight=Q8_0 +blk\.9\.attn_gate\.weight=Q8_0 +blk\.9\.attn_qkv\.weight=Q8_0 +blk\.9\.ffn_down_exps\.weight=Q5_K +blk\.9\.ffn_down_shexp\.weight=Q8_0 +blk\.9\.ffn_gate_exps\.weight=Q5_K +blk\.9\.ffn_gate_shexp\.weight=Q8_0 +blk\.9\.ffn_up_exps\.weight=Q5_K +blk\.9\.ffn_up_shexp\.weight=Q8_0 +blk\.9\.ssm_alpha\.weight=F32 +blk\.9\.ssm_beta\.weight=F32 +blk\.9\.ssm_out\.weight=Q8_0 +blk\.10\.attn_gate\.weight=Q8_0 +blk\.10\.attn_qkv\.weight=Q8_0 +blk\.10\.ffn_down_exps\.weight=Q5_K +blk\.10\.ffn_down_shexp\.weight=Q8_0 +blk\.10\.ffn_gate_exps\.weight=Q5_K +blk\.10\.ffn_gate_shexp\.weight=Q8_0 +blk\.10\.ffn_up_exps\.weight=Q5_K +blk\.10\.ffn_up_shexp\.weight=Q8_0 +blk\.10\.ssm_alpha\.weight=F32 +blk\.10\.ssm_beta\.weight=F32 +blk\.10\.ssm_out\.weight=Q8_0 +blk\.11\.attn_k\.weight=Q8_0 +blk\.11\.attn_output\.weight=Q8_0 +blk\.11\.attn_q\.weight=Q8_0 +blk\.11\.attn_v\.weight=Q8_0 +blk\.11\.ffn_down_exps\.weight=Q5_K +blk\.11\.ffn_down_shexp\.weight=Q8_0 +blk\.11\.ffn_gate_exps\.weight=Q5_K +blk\.11\.ffn_gate_shexp\.weight=Q8_0 +blk\.11\.ffn_up_exps\.weight=Q5_K +blk\.11\.ffn_up_shexp\.weight=Q8_0 +blk\.12\.attn_gate\.weight=Q8_0 +blk\.12\.attn_qkv\.weight=Q8_0 +blk\.12\.ffn_down_exps\.weight=Q5_K +blk\.12\.ffn_down_shexp\.weight=Q8_0 +blk\.12\.ffn_gate_exps\.weight=Q5_K +blk\.12\.ffn_gate_shexp\.weight=Q8_0 +blk\.12\.ffn_up_exps\.weight=Q5_K +blk\.12\.ffn_up_shexp\.weight=Q8_0 +blk\.12\.ssm_alpha\.weight=F32 +blk\.12\.ssm_beta\.weight=F32 +blk\.12\.ssm_out\.weight=Q8_0 +blk\.13\.attn_gate\.weight=Q8_0 +blk\.13\.attn_qkv\.weight=Q8_0 +blk\.13\.ffn_down_exps\.weight=Q5_K +blk\.13\.ffn_down_shexp\.weight=Q8_0 +blk\.13\.ffn_gate_exps\.weight=Q5_K +blk\.13\.ffn_gate_shexp\.weight=Q8_0 +blk\.13\.ffn_up_exps\.weight=Q5_K +blk\.13\.ffn_up_shexp\.weight=Q8_0 +blk\.13\.ssm_alpha\.weight=F32 +blk\.13\.ssm_beta\.weight=F32 +blk\.13\.ssm_out\.weight=Q8_0 +blk\.14\.attn_gate\.weight=Q8_0 +blk\.14\.attn_qkv\.weight=Q8_0 +blk\.14\.ffn_down_exps\.weight=Q5_K +blk\.14\.ffn_down_shexp\.weight=Q8_0 +blk\.14\.ffn_gate_exps\.weight=Q5_K +blk\.14\.ffn_gate_shexp\.weight=Q8_0 +blk\.14\.ffn_up_exps\.weight=Q5_K +blk\.14\.ffn_up_shexp\.weight=Q8_0 +blk\.14\.ssm_alpha\.weight=F32 +blk\.14\.ssm_beta\.weight=F32 +blk\.14\.ssm_out\.weight=Q8_0 +blk\.15\.attn_k\.weight=Q8_0 +blk\.15\.attn_output\.weight=Q8_0 +blk\.15\.attn_q\.weight=Q8_0 +blk\.15\.attn_v\.weight=Q8_0 +blk\.15\.ffn_down_exps\.weight=Q5_K +blk\.15\.ffn_down_shexp\.weight=Q8_0 +blk\.15\.ffn_gate_exps\.weight=Q5_K +blk\.15\.ffn_gate_shexp\.weight=Q8_0 +blk\.15\.ffn_up_exps\.weight=Q5_K +blk\.15\.ffn_up_shexp\.weight=Q8_0 +blk\.16\.attn_gate\.weight=Q8_0 +blk\.16\.attn_qkv\.weight=Q8_0 +blk\.16\.ffn_down_exps\.weight=Q5_K +blk\.16\.ffn_down_shexp\.weight=Q8_0 +blk\.16\.ffn_gate_exps\.weight=Q5_K +blk\.16\.ffn_gate_shexp\.weight=Q8_0 +blk\.16\.ffn_up_exps\.weight=Q5_K +blk\.16\.ffn_up_shexp\.weight=Q8_0 +blk\.16\.ssm_alpha\.weight=F32 +blk\.16\.ssm_beta\.weight=F32 +blk\.16\.ssm_out\.weight=Q8_0 +blk\.17\.attn_gate\.weight=Q8_0 +blk\.17\.attn_qkv\.weight=Q8_0 +blk\.17\.ffn_down_exps\.weight=Q5_K +blk\.17\.ffn_down_shexp\.weight=Q8_0 +blk\.17\.ffn_gate_exps\.weight=Q5_K +blk\.17\.ffn_gate_shexp\.weight=Q8_0 +blk\.17\.ffn_up_exps\.weight=Q5_K +blk\.17\.ffn_up_shexp\.weight=Q8_0 +blk\.17\.ssm_alpha\.weight=F32 +blk\.17\.ssm_beta\.weight=F32 +blk\.17\.ssm_out\.weight=Q8_0 +blk\.18\.attn_gate\.weight=Q8_0 +blk\.18\.attn_qkv\.weight=Q8_0 +blk\.18\.ffn_down_exps\.weight=Q5_K +blk\.18\.ffn_down_shexp\.weight=Q8_0 +blk\.18\.ffn_gate_exps\.weight=Q5_K +blk\.18\.ffn_gate_shexp\.weight=Q8_0 +blk\.18\.ffn_up_exps\.weight=Q5_K +blk\.18\.ffn_up_shexp\.weight=Q8_0 +blk\.18\.ssm_alpha\.weight=F32 +blk\.18\.ssm_beta\.weight=F32 +blk\.18\.ssm_out\.weight=Q8_0 +blk\.19\.attn_k\.weight=Q8_0 +blk\.19\.attn_output\.weight=Q8_0 +blk\.19\.attn_q\.weight=Q8_0 +blk\.19\.attn_v\.weight=Q8_0 +blk\.19\.ffn_down_exps\.weight=Q5_K +blk\.19\.ffn_down_shexp\.weight=Q8_0 +blk\.19\.ffn_gate_exps\.weight=Q5_K +blk\.19\.ffn_gate_shexp\.weight=Q8_0 +blk\.19\.ffn_up_exps\.weight=Q5_K +blk\.19\.ffn_up_shexp\.weight=Q8_0 +blk\.20\.attn_gate\.weight=Q8_0 +blk\.20\.attn_qkv\.weight=Q8_0 +blk\.20\.ffn_down_exps\.weight=Q5_K +blk\.20\.ffn_down_shexp\.weight=Q8_0 +blk\.20\.ffn_gate_exps\.weight=Q5_K +blk\.20\.ffn_gate_shexp\.weight=Q8_0 +blk\.20\.ffn_up_exps\.weight=Q5_K +blk\.20\.ffn_up_shexp\.weight=Q8_0 +blk\.20\.ssm_alpha\.weight=F32 +blk\.20\.ssm_beta\.weight=F32 +blk\.20\.ssm_out\.weight=Q8_0 +blk\.21\.attn_gate\.weight=Q8_0 +blk\.21\.attn_qkv\.weight=Q8_0 +blk\.21\.ffn_down_exps\.weight=Q5_K +blk\.21\.ffn_down_shexp\.weight=Q8_0 +blk\.21\.ffn_gate_exps\.weight=Q5_K +blk\.21\.ffn_gate_shexp\.weight=Q8_0 +blk\.21\.ffn_up_exps\.weight=Q5_K +blk\.21\.ffn_up_shexp\.weight=Q8_0 +blk\.21\.ssm_alpha\.weight=F32 +blk\.21\.ssm_beta\.weight=F32 +blk\.21\.ssm_out\.weight=Q8_0 +blk\.22\.attn_gate\.weight=Q8_0 +blk\.22\.attn_qkv\.weight=Q8_0 +blk\.22\.ffn_down_exps\.weight=Q5_K +blk\.22\.ffn_down_shexp\.weight=Q8_0 +blk\.22\.ffn_gate_exps\.weight=Q5_K +blk\.22\.ffn_gate_shexp\.weight=Q8_0 +blk\.22\.ffn_up_exps\.weight=Q5_K +blk\.22\.ffn_up_shexp\.weight=Q8_0 +blk\.22\.ssm_alpha\.weight=F32 +blk\.22\.ssm_beta\.weight=F32 +blk\.22\.ssm_out\.weight=Q8_0 +blk\.23\.attn_k\.weight=Q8_0 +blk\.23\.attn_output\.weight=Q8_0 +blk\.23\.attn_q\.weight=Q8_0 +blk\.23\.attn_v\.weight=Q8_0 +blk\.23\.ffn_down_exps\.weight=Q5_K +blk\.23\.ffn_down_shexp\.weight=Q8_0 +blk\.23\.ffn_gate_exps\.weight=Q5_K +blk\.23\.ffn_gate_shexp\.weight=Q8_0 +blk\.23\.ffn_up_exps\.weight=Q5_K +blk\.23\.ffn_up_shexp\.weight=Q8_0 +blk\.24\.attn_gate\.weight=Q8_0 +blk\.24\.attn_qkv\.weight=Q8_0 +blk\.24\.ffn_down_exps\.weight=Q5_K +blk\.24\.ffn_down_shexp\.weight=Q8_0 +blk\.24\.ffn_gate_exps\.weight=Q5_K +blk\.24\.ffn_gate_shexp\.weight=Q8_0 +blk\.24\.ffn_up_exps\.weight=Q5_K +blk\.24\.ffn_up_shexp\.weight=Q8_0 +blk\.24\.ssm_alpha\.weight=F32 +blk\.24\.ssm_beta\.weight=F32 +blk\.24\.ssm_out\.weight=Q8_0 +blk\.25\.attn_gate\.weight=Q8_0 +blk\.25\.attn_qkv\.weight=Q8_0 +blk\.25\.ffn_down_exps\.weight=Q5_K +blk\.25\.ffn_down_shexp\.weight=Q8_0 +blk\.25\.ffn_gate_exps\.weight=Q5_K +blk\.25\.ffn_gate_shexp\.weight=Q8_0 +blk\.25\.ffn_up_exps\.weight=Q5_K +blk\.25\.ffn_up_shexp\.weight=Q8_0 +blk\.25\.ssm_alpha\.weight=F32 +blk\.25\.ssm_beta\.weight=F32 +blk\.25\.ssm_out\.weight=Q8_0 +blk\.26\.attn_gate\.weight=Q8_0 +blk\.26\.attn_qkv\.weight=Q8_0 +blk\.26\.ffn_down_exps\.weight=Q5_K +blk\.26\.ffn_down_shexp\.weight=Q8_0 +blk\.26\.ffn_gate_exps\.weight=Q5_K +blk\.26\.ffn_gate_shexp\.weight=Q8_0 +blk\.26\.ffn_up_exps\.weight=Q5_K +blk\.26\.ffn_up_shexp\.weight=Q8_0 +blk\.26\.ssm_alpha\.weight=F32 +blk\.26\.ssm_beta\.weight=F32 +blk\.26\.ssm_out\.weight=Q8_0 +blk\.27\.attn_k\.weight=Q8_0 +blk\.27\.attn_output\.weight=Q8_0 +blk\.27\.attn_q\.weight=Q8_0 +blk\.27\.attn_v\.weight=Q8_0 +blk\.27\.ffn_down_exps\.weight=Q5_K +blk\.27\.ffn_down_shexp\.weight=Q8_0 +blk\.27\.ffn_gate_exps\.weight=Q5_K +blk\.27\.ffn_gate_shexp\.weight=Q8_0 +blk\.27\.ffn_up_exps\.weight=Q5_K +blk\.27\.ffn_up_shexp\.weight=Q8_0 +blk\.28\.attn_gate\.weight=Q8_0 +blk\.28\.attn_qkv\.weight=Q8_0 +blk\.28\.ffn_down_exps\.weight=Q5_K +blk\.28\.ffn_down_shexp\.weight=Q8_0 +blk\.28\.ffn_gate_exps\.weight=Q5_K +blk\.28\.ffn_gate_shexp\.weight=Q8_0 +blk\.28\.ffn_up_exps\.weight=Q5_K +blk\.28\.ffn_up_shexp\.weight=Q8_0 +blk\.28\.ssm_alpha\.weight=F32 +blk\.28\.ssm_beta\.weight=F32 +blk\.28\.ssm_out\.weight=Q8_0 +blk\.29\.attn_gate\.weight=Q8_0 +blk\.29\.attn_qkv\.weight=Q8_0 +blk\.29\.ffn_down_exps\.weight=Q5_K +blk\.29\.ffn_down_shexp\.weight=Q8_0 +blk\.29\.ffn_gate_exps\.weight=Q5_K +blk\.29\.ffn_gate_shexp\.weight=Q8_0 +blk\.29\.ffn_up_exps\.weight=Q5_K +blk\.29\.ffn_up_shexp\.weight=Q8_0 +blk\.29\.ssm_alpha\.weight=F32 +blk\.29\.ssm_beta\.weight=F32 +blk\.29\.ssm_out\.weight=Q8_0 +blk\.30\.attn_gate\.weight=Q8_0 +blk\.30\.attn_qkv\.weight=Q8_0 +blk\.30\.ffn_down_exps\.weight=Q5_K +blk\.30\.ffn_down_shexp\.weight=Q8_0 +blk\.30\.ffn_gate_exps\.weight=Q5_K +blk\.30\.ffn_gate_shexp\.weight=Q8_0 +blk\.30\.ffn_up_exps\.weight=Q5_K +blk\.30\.ffn_up_shexp\.weight=Q8_0 +blk\.30\.ssm_alpha\.weight=F32 +blk\.30\.ssm_beta\.weight=F32 +blk\.30\.ssm_out\.weight=Q8_0 +blk\.31\.attn_k\.weight=Q8_0 +blk\.31\.attn_output\.weight=Q8_0 +blk\.31\.attn_q\.weight=Q8_0 +blk\.31\.attn_v\.weight=Q8_0 +blk\.31\.ffn_down_exps\.weight=Q5_K +blk\.31\.ffn_down_shexp\.weight=Q8_0 +blk\.31\.ffn_gate_exps\.weight=Q5_K +blk\.31\.ffn_gate_shexp\.weight=Q8_0 +blk\.31\.ffn_up_exps\.weight=Q5_K +blk\.31\.ffn_up_shexp\.weight=Q8_0 +blk\.32\.attn_gate\.weight=Q8_0 +blk\.32\.attn_qkv\.weight=Q8_0 +blk\.32\.ffn_down_exps\.weight=Q5_K +blk\.32\.ffn_down_shexp\.weight=Q8_0 +blk\.32\.ffn_gate_exps\.weight=Q5_K +blk\.32\.ffn_gate_shexp\.weight=Q8_0 +blk\.32\.ffn_up_exps\.weight=Q5_K +blk\.32\.ffn_up_shexp\.weight=Q8_0 +blk\.32\.ssm_alpha\.weight=F32 +blk\.32\.ssm_beta\.weight=F32 +blk\.32\.ssm_out\.weight=Q8_0 +blk\.33\.attn_gate\.weight=Q8_0 +blk\.33\.attn_qkv\.weight=Q8_0 +blk\.33\.ffn_down_exps\.weight=Q5_K +blk\.33\.ffn_down_shexp\.weight=Q8_0 +blk\.33\.ffn_gate_exps\.weight=Q5_K +blk\.33\.ffn_gate_shexp\.weight=Q8_0 +blk\.33\.ffn_up_exps\.weight=Q5_K +blk\.33\.ffn_up_shexp\.weight=Q8_0 +blk\.33\.ssm_alpha\.weight=F32 +blk\.33\.ssm_beta\.weight=F32 +blk\.33\.ssm_out\.weight=Q8_0 +blk\.34\.attn_gate\.weight=Q8_0 +blk\.34\.attn_qkv\.weight=Q8_0 +blk\.34\.ffn_down_exps\.weight=Q6_K +blk\.34\.ffn_down_shexp\.weight=Q8_0 +blk\.34\.ffn_gate_exps\.weight=Q5_K +blk\.34\.ffn_gate_shexp\.weight=Q8_0 +blk\.34\.ffn_up_exps\.weight=Q5_K +blk\.34\.ffn_up_shexp\.weight=Q8_0 +blk\.34\.ssm_alpha\.weight=F32 +blk\.34\.ssm_beta\.weight=F32 +blk\.34\.ssm_out\.weight=Q8_0 +blk\.35\.attn_k\.weight=Q8_0 +blk\.35\.attn_output\.weight=Q8_0 +blk\.35\.attn_q\.weight=Q8_0 +blk\.35\.attn_v\.weight=Q8_0 +blk\.35\.ffn_down_exps\.weight=Q5_K +blk\.35\.ffn_down_shexp\.weight=Q8_0 +blk\.35\.ffn_gate_exps\.weight=Q5_K +blk\.35\.ffn_gate_shexp\.weight=Q8_0 +blk\.35\.ffn_up_exps\.weight=Q5_K +blk\.35\.ffn_up_shexp\.weight=Q8_0 +blk\.36\.attn_gate\.weight=Q8_0 +blk\.36\.attn_qkv\.weight=Q8_0 +blk\.36\.ffn_down_exps\.weight=Q5_K +blk\.36\.ffn_down_shexp\.weight=Q8_0 +blk\.36\.ffn_gate_exps\.weight=Q5_K +blk\.36\.ffn_gate_shexp\.weight=Q8_0 +blk\.36\.ffn_up_exps\.weight=Q5_K +blk\.36\.ffn_up_shexp\.weight=Q8_0 +blk\.36\.ssm_alpha\.weight=F32 +blk\.36\.ssm_beta\.weight=F32 +blk\.36\.ssm_out\.weight=Q8_0 +blk\.37\.attn_gate\.weight=Q8_0 +blk\.37\.attn_qkv\.weight=Q8_0 +blk\.37\.ffn_down_exps\.weight=Q5_K +blk\.37\.ffn_down_shexp\.weight=Q8_0 +blk\.37\.ffn_gate_exps\.weight=Q5_K +blk\.37\.ffn_gate_shexp\.weight=Q8_0 +blk\.37\.ffn_up_exps\.weight=Q5_K +blk\.37\.ffn_up_shexp\.weight=Q8_0 +blk\.37\.ssm_alpha\.weight=F32 +blk\.37\.ssm_beta\.weight=F32 +blk\.37\.ssm_out\.weight=Q8_0 +blk\.38\.attn_gate\.weight=Q8_0 +blk\.38\.attn_qkv\.weight=Q8_0 +blk\.38\.ffn_down_exps\.weight=Q6_K +blk\.38\.ffn_down_shexp\.weight=Q8_0 +blk\.38\.ffn_gate_exps\.weight=Q5_K +blk\.38\.ffn_gate_shexp\.weight=Q8_0 +blk\.38\.ffn_up_exps\.weight=Q5_K +blk\.38\.ffn_up_shexp\.weight=Q8_0 +blk\.38\.ssm_alpha\.weight=F32 +blk\.38\.ssm_beta\.weight=F32 +blk\.38\.ssm_out\.weight=Q8_0 +blk\.39\.attn_k\.weight=Q8_0 +blk\.39\.attn_output\.weight=Q8_0 +blk\.39\.attn_q\.weight=Q8_0 +blk\.39\.attn_v\.weight=Q8_0 +blk\.39\.ffn_down_exps\.weight=Q6_K +blk\.39\.ffn_down_shexp\.weight=Q8_0 +blk\.39\.ffn_gate_exps\.weight=Q5_K +blk\.39\.ffn_gate_shexp\.weight=Q8_0 +blk\.39\.ffn_up_exps\.weight=Q5_K +blk\.39\.ffn_up_shexp\.weight=Q8_0 +blk\.40\.attn_k\.weight=Q8_0 +blk\.40\.attn_output\.weight=Q8_0 +blk\.40\.attn_q\.weight=Q8_0 +blk\.40\.attn_v\.weight=Q8_0 +blk\.40\.ffn_down_exps\.weight=Q8_0 +blk\.40\.ffn_down_shexp\.weight=Q8_0 +blk\.40\.ffn_gate_exps\.weight=Q8_0 +blk\.40\.ffn_gate_shexp\.weight=Q8_0 +blk\.40\.ffn_up_exps\.weight=Q8_0 +blk\.40\.ffn_up_shexp\.weight=Q8_0 +blk\.40\.nextn\.eh_proj\.weight=Q8_0 diff --git a/appendix/recipes/cfg_UD-Q6_K_comparison.txt b/appendix/recipes/cfg_UD-Q6_K_comparison.txt new file mode 100644 index 0000000000000000000000000000000000000000..865e9d31f7f9cf666cca9218ee8798462971abc1 --- /dev/null +++ b/appendix/recipes/cfg_UD-Q6_K_comparison.txt @@ -0,0 +1,443 @@ +^output\.weight$=Q8_0 +token_embd\.weight=Q8_0 +blk\.0\.attn_gate\.weight=Q8_0 +blk\.0\.attn_qkv\.weight=Q8_0 +blk\.0\.ffn_down_exps\.weight=Q6_K +blk\.0\.ffn_down_shexp\.weight=Q8_0 +blk\.0\.ffn_gate_exps\.weight=Q6_K +blk\.0\.ffn_gate_shexp\.weight=Q8_0 +blk\.0\.ffn_up_exps\.weight=Q6_K +blk\.0\.ffn_up_shexp\.weight=Q8_0 +blk\.0\.ssm_alpha\.weight=F32 +blk\.0\.ssm_beta\.weight=F32 +blk\.0\.ssm_out\.weight=Q8_0 +blk\.1\.attn_gate\.weight=Q8_0 +blk\.1\.attn_qkv\.weight=Q8_0 +blk\.1\.ffn_down_exps\.weight=Q6_K +blk\.1\.ffn_down_shexp\.weight=Q8_0 +blk\.1\.ffn_gate_exps\.weight=Q6_K +blk\.1\.ffn_gate_shexp\.weight=Q8_0 +blk\.1\.ffn_up_exps\.weight=Q6_K +blk\.1\.ffn_up_shexp\.weight=Q8_0 +blk\.1\.ssm_alpha\.weight=F32 +blk\.1\.ssm_beta\.weight=F32 +blk\.1\.ssm_out\.weight=Q8_0 +blk\.2\.attn_gate\.weight=Q8_0 +blk\.2\.attn_qkv\.weight=Q8_0 +blk\.2\.ffn_down_exps\.weight=Q6_K +blk\.2\.ffn_down_shexp\.weight=Q8_0 +blk\.2\.ffn_gate_exps\.weight=Q6_K +blk\.2\.ffn_gate_shexp\.weight=Q8_0 +blk\.2\.ffn_up_exps\.weight=Q6_K +blk\.2\.ffn_up_shexp\.weight=Q8_0 +blk\.2\.ssm_alpha\.weight=F32 +blk\.2\.ssm_beta\.weight=F32 +blk\.2\.ssm_out\.weight=Q8_0 +blk\.3\.attn_k\.weight=Q8_0 +blk\.3\.attn_output\.weight=Q8_0 +blk\.3\.attn_q\.weight=Q8_0 +blk\.3\.attn_v\.weight=Q8_0 +blk\.3\.ffn_down_exps\.weight=Q6_K +blk\.3\.ffn_down_shexp\.weight=Q8_0 +blk\.3\.ffn_gate_exps\.weight=Q6_K +blk\.3\.ffn_gate_shexp\.weight=Q8_0 +blk\.3\.ffn_up_exps\.weight=Q6_K +blk\.3\.ffn_up_shexp\.weight=Q8_0 +blk\.4\.attn_gate\.weight=Q8_0 +blk\.4\.attn_qkv\.weight=Q8_0 +blk\.4\.ffn_down_exps\.weight=Q6_K +blk\.4\.ffn_down_shexp\.weight=Q8_0 +blk\.4\.ffn_gate_exps\.weight=Q6_K +blk\.4\.ffn_gate_shexp\.weight=Q8_0 +blk\.4\.ffn_up_exps\.weight=Q6_K +blk\.4\.ffn_up_shexp\.weight=Q8_0 +blk\.4\.ssm_alpha\.weight=F32 +blk\.4\.ssm_beta\.weight=F32 +blk\.4\.ssm_out\.weight=Q8_0 +blk\.5\.attn_gate\.weight=Q8_0 +blk\.5\.attn_qkv\.weight=Q8_0 +blk\.5\.ffn_down_exps\.weight=Q6_K +blk\.5\.ffn_down_shexp\.weight=Q8_0 +blk\.5\.ffn_gate_exps\.weight=Q6_K +blk\.5\.ffn_gate_shexp\.weight=Q8_0 +blk\.5\.ffn_up_exps\.weight=Q6_K +blk\.5\.ffn_up_shexp\.weight=Q8_0 +blk\.5\.ssm_alpha\.weight=F32 +blk\.5\.ssm_beta\.weight=F32 +blk\.5\.ssm_out\.weight=Q8_0 +blk\.6\.attn_gate\.weight=Q8_0 +blk\.6\.attn_qkv\.weight=Q8_0 +blk\.6\.ffn_down_exps\.weight=Q6_K +blk\.6\.ffn_down_shexp\.weight=Q8_0 +blk\.6\.ffn_gate_exps\.weight=Q6_K +blk\.6\.ffn_gate_shexp\.weight=Q8_0 +blk\.6\.ffn_up_exps\.weight=Q6_K +blk\.6\.ffn_up_shexp\.weight=Q8_0 +blk\.6\.ssm_alpha\.weight=F32 +blk\.6\.ssm_beta\.weight=F32 +blk\.6\.ssm_out\.weight=Q8_0 +blk\.7\.attn_k\.weight=Q8_0 +blk\.7\.attn_output\.weight=Q8_0 +blk\.7\.attn_q\.weight=Q8_0 +blk\.7\.attn_v\.weight=Q8_0 +blk\.7\.ffn_down_exps\.weight=Q6_K +blk\.7\.ffn_down_shexp\.weight=Q8_0 +blk\.7\.ffn_gate_exps\.weight=Q6_K +blk\.7\.ffn_gate_shexp\.weight=Q8_0 +blk\.7\.ffn_up_exps\.weight=Q6_K +blk\.7\.ffn_up_shexp\.weight=Q8_0 +blk\.8\.attn_gate\.weight=Q8_0 +blk\.8\.attn_qkv\.weight=Q8_0 +blk\.8\.ffn_down_exps\.weight=Q6_K +blk\.8\.ffn_down_shexp\.weight=Q8_0 +blk\.8\.ffn_gate_exps\.weight=Q6_K +blk\.8\.ffn_gate_shexp\.weight=Q8_0 +blk\.8\.ffn_up_exps\.weight=Q6_K +blk\.8\.ffn_up_shexp\.weight=Q8_0 +blk\.8\.ssm_alpha\.weight=F32 +blk\.8\.ssm_beta\.weight=F32 +blk\.8\.ssm_out\.weight=Q8_0 +blk\.9\.attn_gate\.weight=Q8_0 +blk\.9\.attn_qkv\.weight=Q8_0 +blk\.9\.ffn_down_exps\.weight=Q6_K +blk\.9\.ffn_down_shexp\.weight=Q8_0 +blk\.9\.ffn_gate_exps\.weight=Q6_K +blk\.9\.ffn_gate_shexp\.weight=Q8_0 +blk\.9\.ffn_up_exps\.weight=Q6_K +blk\.9\.ffn_up_shexp\.weight=Q8_0 +blk\.9\.ssm_alpha\.weight=F32 +blk\.9\.ssm_beta\.weight=F32 +blk\.9\.ssm_out\.weight=Q8_0 +blk\.10\.attn_gate\.weight=Q8_0 +blk\.10\.attn_qkv\.weight=Q8_0 +blk\.10\.ffn_down_exps\.weight=Q6_K +blk\.10\.ffn_down_shexp\.weight=Q8_0 +blk\.10\.ffn_gate_exps\.weight=Q6_K +blk\.10\.ffn_gate_shexp\.weight=Q8_0 +blk\.10\.ffn_up_exps\.weight=Q6_K +blk\.10\.ffn_up_shexp\.weight=Q8_0 +blk\.10\.ssm_alpha\.weight=F32 +blk\.10\.ssm_beta\.weight=F32 +blk\.10\.ssm_out\.weight=Q8_0 +blk\.11\.attn_k\.weight=Q8_0 +blk\.11\.attn_output\.weight=Q8_0 +blk\.11\.attn_q\.weight=Q8_0 +blk\.11\.attn_v\.weight=Q8_0 +blk\.11\.ffn_down_exps\.weight=Q6_K +blk\.11\.ffn_down_shexp\.weight=Q8_0 +blk\.11\.ffn_gate_exps\.weight=Q6_K +blk\.11\.ffn_gate_shexp\.weight=Q8_0 +blk\.11\.ffn_up_exps\.weight=Q6_K +blk\.11\.ffn_up_shexp\.weight=Q8_0 +blk\.12\.attn_gate\.weight=Q8_0 +blk\.12\.attn_qkv\.weight=Q8_0 +blk\.12\.ffn_down_exps\.weight=Q6_K +blk\.12\.ffn_down_shexp\.weight=Q8_0 +blk\.12\.ffn_gate_exps\.weight=Q6_K +blk\.12\.ffn_gate_shexp\.weight=Q8_0 +blk\.12\.ffn_up_exps\.weight=Q6_K +blk\.12\.ffn_up_shexp\.weight=Q8_0 +blk\.12\.ssm_alpha\.weight=F32 +blk\.12\.ssm_beta\.weight=F32 +blk\.12\.ssm_out\.weight=Q8_0 +blk\.13\.attn_gate\.weight=Q8_0 +blk\.13\.attn_qkv\.weight=Q8_0 +blk\.13\.ffn_down_exps\.weight=Q6_K +blk\.13\.ffn_down_shexp\.weight=Q8_0 +blk\.13\.ffn_gate_exps\.weight=Q6_K +blk\.13\.ffn_gate_shexp\.weight=Q8_0 +blk\.13\.ffn_up_exps\.weight=Q6_K +blk\.13\.ffn_up_shexp\.weight=Q8_0 +blk\.13\.ssm_alpha\.weight=F32 +blk\.13\.ssm_beta\.weight=F32 +blk\.13\.ssm_out\.weight=Q8_0 +blk\.14\.attn_gate\.weight=Q8_0 +blk\.14\.attn_qkv\.weight=Q8_0 +blk\.14\.ffn_down_exps\.weight=Q6_K +blk\.14\.ffn_down_shexp\.weight=Q8_0 +blk\.14\.ffn_gate_exps\.weight=Q6_K +blk\.14\.ffn_gate_shexp\.weight=Q8_0 +blk\.14\.ffn_up_exps\.weight=Q6_K +blk\.14\.ffn_up_shexp\.weight=Q8_0 +blk\.14\.ssm_alpha\.weight=F32 +blk\.14\.ssm_beta\.weight=F32 +blk\.14\.ssm_out\.weight=Q8_0 +blk\.15\.attn_k\.weight=Q8_0 +blk\.15\.attn_output\.weight=Q8_0 +blk\.15\.attn_q\.weight=Q8_0 +blk\.15\.attn_v\.weight=Q8_0 +blk\.15\.ffn_down_exps\.weight=Q6_K +blk\.15\.ffn_down_shexp\.weight=Q8_0 +blk\.15\.ffn_gate_exps\.weight=Q6_K +blk\.15\.ffn_gate_shexp\.weight=Q8_0 +blk\.15\.ffn_up_exps\.weight=Q6_K +blk\.15\.ffn_up_shexp\.weight=Q8_0 +blk\.16\.attn_gate\.weight=Q8_0 +blk\.16\.attn_qkv\.weight=Q8_0 +blk\.16\.ffn_down_exps\.weight=Q6_K +blk\.16\.ffn_down_shexp\.weight=Q8_0 +blk\.16\.ffn_gate_exps\.weight=Q6_K +blk\.16\.ffn_gate_shexp\.weight=Q8_0 +blk\.16\.ffn_up_exps\.weight=Q6_K +blk\.16\.ffn_up_shexp\.weight=Q8_0 +blk\.16\.ssm_alpha\.weight=F32 +blk\.16\.ssm_beta\.weight=F32 +blk\.16\.ssm_out\.weight=Q8_0 +blk\.17\.attn_gate\.weight=Q8_0 +blk\.17\.attn_qkv\.weight=Q8_0 +blk\.17\.ffn_down_exps\.weight=Q6_K +blk\.17\.ffn_down_shexp\.weight=Q8_0 +blk\.17\.ffn_gate_exps\.weight=Q6_K +blk\.17\.ffn_gate_shexp\.weight=Q8_0 +blk\.17\.ffn_up_exps\.weight=Q6_K +blk\.17\.ffn_up_shexp\.weight=Q8_0 +blk\.17\.ssm_alpha\.weight=F32 +blk\.17\.ssm_beta\.weight=F32 +blk\.17\.ssm_out\.weight=Q8_0 +blk\.18\.attn_gate\.weight=Q8_0 +blk\.18\.attn_qkv\.weight=Q8_0 +blk\.18\.ffn_down_exps\.weight=Q6_K +blk\.18\.ffn_down_shexp\.weight=Q8_0 +blk\.18\.ffn_gate_exps\.weight=Q6_K +blk\.18\.ffn_gate_shexp\.weight=Q8_0 +blk\.18\.ffn_up_exps\.weight=Q6_K +blk\.18\.ffn_up_shexp\.weight=Q8_0 +blk\.18\.ssm_alpha\.weight=F32 +blk\.18\.ssm_beta\.weight=F32 +blk\.18\.ssm_out\.weight=Q8_0 +blk\.19\.attn_k\.weight=Q8_0 +blk\.19\.attn_output\.weight=Q8_0 +blk\.19\.attn_q\.weight=Q8_0 +blk\.19\.attn_v\.weight=Q8_0 +blk\.19\.ffn_down_exps\.weight=Q6_K +blk\.19\.ffn_down_shexp\.weight=Q8_0 +blk\.19\.ffn_gate_exps\.weight=Q6_K +blk\.19\.ffn_gate_shexp\.weight=Q8_0 +blk\.19\.ffn_up_exps\.weight=Q6_K +blk\.19\.ffn_up_shexp\.weight=Q8_0 +blk\.20\.attn_gate\.weight=Q8_0 +blk\.20\.attn_qkv\.weight=Q8_0 +blk\.20\.ffn_down_exps\.weight=Q6_K +blk\.20\.ffn_down_shexp\.weight=Q8_0 +blk\.20\.ffn_gate_exps\.weight=Q6_K +blk\.20\.ffn_gate_shexp\.weight=Q8_0 +blk\.20\.ffn_up_exps\.weight=Q6_K +blk\.20\.ffn_up_shexp\.weight=Q8_0 +blk\.20\.ssm_alpha\.weight=F32 +blk\.20\.ssm_beta\.weight=F32 +blk\.20\.ssm_out\.weight=Q8_0 +blk\.21\.attn_gate\.weight=Q8_0 +blk\.21\.attn_qkv\.weight=Q8_0 +blk\.21\.ffn_down_exps\.weight=Q6_K +blk\.21\.ffn_down_shexp\.weight=Q8_0 +blk\.21\.ffn_gate_exps\.weight=Q6_K +blk\.21\.ffn_gate_shexp\.weight=Q8_0 +blk\.21\.ffn_up_exps\.weight=Q6_K +blk\.21\.ffn_up_shexp\.weight=Q8_0 +blk\.21\.ssm_alpha\.weight=F32 +blk\.21\.ssm_beta\.weight=F32 +blk\.21\.ssm_out\.weight=Q8_0 +blk\.22\.attn_gate\.weight=Q8_0 +blk\.22\.attn_qkv\.weight=Q8_0 +blk\.22\.ffn_down_exps\.weight=Q6_K +blk\.22\.ffn_down_shexp\.weight=Q8_0 +blk\.22\.ffn_gate_exps\.weight=Q6_K +blk\.22\.ffn_gate_shexp\.weight=Q8_0 +blk\.22\.ffn_up_exps\.weight=Q6_K +blk\.22\.ffn_up_shexp\.weight=Q8_0 +blk\.22\.ssm_alpha\.weight=F32 +blk\.22\.ssm_beta\.weight=F32 +blk\.22\.ssm_out\.weight=Q8_0 +blk\.23\.attn_k\.weight=Q8_0 +blk\.23\.attn_output\.weight=Q8_0 +blk\.23\.attn_q\.weight=Q8_0 +blk\.23\.attn_v\.weight=Q8_0 +blk\.23\.ffn_down_exps\.weight=Q6_K +blk\.23\.ffn_down_shexp\.weight=Q8_0 +blk\.23\.ffn_gate_exps\.weight=Q6_K +blk\.23\.ffn_gate_shexp\.weight=Q8_0 +blk\.23\.ffn_up_exps\.weight=Q6_K +blk\.23\.ffn_up_shexp\.weight=Q8_0 +blk\.24\.attn_gate\.weight=Q8_0 +blk\.24\.attn_qkv\.weight=Q8_0 +blk\.24\.ffn_down_exps\.weight=Q6_K +blk\.24\.ffn_down_shexp\.weight=Q8_0 +blk\.24\.ffn_gate_exps\.weight=Q6_K +blk\.24\.ffn_gate_shexp\.weight=Q8_0 +blk\.24\.ffn_up_exps\.weight=Q6_K +blk\.24\.ffn_up_shexp\.weight=Q8_0 +blk\.24\.ssm_alpha\.weight=F32 +blk\.24\.ssm_beta\.weight=F32 +blk\.24\.ssm_out\.weight=Q8_0 +blk\.25\.attn_gate\.weight=Q8_0 +blk\.25\.attn_qkv\.weight=Q8_0 +blk\.25\.ffn_down_exps\.weight=Q6_K +blk\.25\.ffn_down_shexp\.weight=Q8_0 +blk\.25\.ffn_gate_exps\.weight=Q6_K +blk\.25\.ffn_gate_shexp\.weight=Q8_0 +blk\.25\.ffn_up_exps\.weight=Q6_K +blk\.25\.ffn_up_shexp\.weight=Q8_0 +blk\.25\.ssm_alpha\.weight=F32 +blk\.25\.ssm_beta\.weight=F32 +blk\.25\.ssm_out\.weight=Q8_0 +blk\.26\.attn_gate\.weight=Q8_0 +blk\.26\.attn_qkv\.weight=Q8_0 +blk\.26\.ffn_down_exps\.weight=Q6_K +blk\.26\.ffn_down_shexp\.weight=Q8_0 +blk\.26\.ffn_gate_exps\.weight=Q6_K +blk\.26\.ffn_gate_shexp\.weight=Q8_0 +blk\.26\.ffn_up_exps\.weight=Q6_K +blk\.26\.ffn_up_shexp\.weight=Q8_0 +blk\.26\.ssm_alpha\.weight=F32 +blk\.26\.ssm_beta\.weight=F32 +blk\.26\.ssm_out\.weight=Q8_0 +blk\.27\.attn_k\.weight=Q8_0 +blk\.27\.attn_output\.weight=Q8_0 +blk\.27\.attn_q\.weight=Q8_0 +blk\.27\.attn_v\.weight=Q8_0 +blk\.27\.ffn_down_exps\.weight=Q6_K +blk\.27\.ffn_down_shexp\.weight=Q8_0 +blk\.27\.ffn_gate_exps\.weight=Q6_K +blk\.27\.ffn_gate_shexp\.weight=Q8_0 +blk\.27\.ffn_up_exps\.weight=Q6_K +blk\.27\.ffn_up_shexp\.weight=Q8_0 +blk\.28\.attn_gate\.weight=Q8_0 +blk\.28\.attn_qkv\.weight=Q8_0 +blk\.28\.ffn_down_exps\.weight=Q6_K +blk\.28\.ffn_down_shexp\.weight=Q8_0 +blk\.28\.ffn_gate_exps\.weight=Q6_K +blk\.28\.ffn_gate_shexp\.weight=Q8_0 +blk\.28\.ffn_up_exps\.weight=Q6_K +blk\.28\.ffn_up_shexp\.weight=Q8_0 +blk\.28\.ssm_alpha\.weight=F32 +blk\.28\.ssm_beta\.weight=F32 +blk\.28\.ssm_out\.weight=Q8_0 +blk\.29\.attn_gate\.weight=Q8_0 +blk\.29\.attn_qkv\.weight=Q8_0 +blk\.29\.ffn_down_exps\.weight=Q6_K +blk\.29\.ffn_down_shexp\.weight=Q8_0 +blk\.29\.ffn_gate_exps\.weight=Q6_K +blk\.29\.ffn_gate_shexp\.weight=Q8_0 +blk\.29\.ffn_up_exps\.weight=Q6_K +blk\.29\.ffn_up_shexp\.weight=Q8_0 +blk\.29\.ssm_alpha\.weight=F32 +blk\.29\.ssm_beta\.weight=F32 +blk\.29\.ssm_out\.weight=Q8_0 +blk\.30\.attn_gate\.weight=Q8_0 +blk\.30\.attn_qkv\.weight=Q8_0 +blk\.30\.ffn_down_exps\.weight=Q6_K +blk\.30\.ffn_down_shexp\.weight=Q8_0 +blk\.30\.ffn_gate_exps\.weight=Q6_K +blk\.30\.ffn_gate_shexp\.weight=Q8_0 +blk\.30\.ffn_up_exps\.weight=Q6_K +blk\.30\.ffn_up_shexp\.weight=Q8_0 +blk\.30\.ssm_alpha\.weight=F32 +blk\.30\.ssm_beta\.weight=F32 +blk\.30\.ssm_out\.weight=Q8_0 +blk\.31\.attn_k\.weight=Q8_0 +blk\.31\.attn_output\.weight=Q8_0 +blk\.31\.attn_q\.weight=Q8_0 +blk\.31\.attn_v\.weight=Q8_0 +blk\.31\.ffn_down_exps\.weight=Q6_K +blk\.31\.ffn_down_shexp\.weight=Q8_0 +blk\.31\.ffn_gate_exps\.weight=Q6_K +blk\.31\.ffn_gate_shexp\.weight=Q8_0 +blk\.31\.ffn_up_exps\.weight=Q6_K +blk\.31\.ffn_up_shexp\.weight=Q8_0 +blk\.32\.attn_gate\.weight=Q8_0 +blk\.32\.attn_qkv\.weight=Q8_0 +blk\.32\.ffn_down_exps\.weight=Q6_K +blk\.32\.ffn_down_shexp\.weight=Q8_0 +blk\.32\.ffn_gate_exps\.weight=Q6_K +blk\.32\.ffn_gate_shexp\.weight=Q8_0 +blk\.32\.ffn_up_exps\.weight=Q6_K +blk\.32\.ffn_up_shexp\.weight=Q8_0 +blk\.32\.ssm_alpha\.weight=F32 +blk\.32\.ssm_beta\.weight=F32 +blk\.32\.ssm_out\.weight=Q8_0 +blk\.33\.attn_gate\.weight=Q8_0 +blk\.33\.attn_qkv\.weight=Q8_0 +blk\.33\.ffn_down_exps\.weight=Q6_K +blk\.33\.ffn_down_shexp\.weight=Q8_0 +blk\.33\.ffn_gate_exps\.weight=Q6_K +blk\.33\.ffn_gate_shexp\.weight=Q8_0 +blk\.33\.ffn_up_exps\.weight=Q6_K +blk\.33\.ffn_up_shexp\.weight=Q8_0 +blk\.33\.ssm_alpha\.weight=F32 +blk\.33\.ssm_beta\.weight=F32 +blk\.33\.ssm_out\.weight=Q8_0 +blk\.34\.attn_gate\.weight=Q8_0 +blk\.34\.attn_qkv\.weight=Q8_0 +blk\.34\.ffn_down_exps\.weight=Q8_0 +blk\.34\.ffn_down_shexp\.weight=Q8_0 +blk\.34\.ffn_gate_exps\.weight=Q6_K +blk\.34\.ffn_gate_shexp\.weight=Q8_0 +blk\.34\.ffn_up_exps\.weight=Q6_K +blk\.34\.ffn_up_shexp\.weight=Q8_0 +blk\.34\.ssm_alpha\.weight=F32 +blk\.34\.ssm_beta\.weight=F32 +blk\.34\.ssm_out\.weight=Q8_0 +blk\.35\.attn_k\.weight=Q8_0 +blk\.35\.attn_output\.weight=Q8_0 +blk\.35\.attn_q\.weight=Q8_0 +blk\.35\.attn_v\.weight=Q8_0 +blk\.35\.ffn_down_exps\.weight=Q6_K +blk\.35\.ffn_down_shexp\.weight=Q8_0 +blk\.35\.ffn_gate_exps\.weight=Q6_K +blk\.35\.ffn_gate_shexp\.weight=Q8_0 +blk\.35\.ffn_up_exps\.weight=Q6_K +blk\.35\.ffn_up_shexp\.weight=Q8_0 +blk\.36\.attn_gate\.weight=Q8_0 +blk\.36\.attn_qkv\.weight=Q8_0 +blk\.36\.ffn_down_exps\.weight=Q6_K +blk\.36\.ffn_down_shexp\.weight=Q8_0 +blk\.36\.ffn_gate_exps\.weight=Q6_K +blk\.36\.ffn_gate_shexp\.weight=Q8_0 +blk\.36\.ffn_up_exps\.weight=Q6_K +blk\.36\.ffn_up_shexp\.weight=Q8_0 +blk\.36\.ssm_alpha\.weight=F32 +blk\.36\.ssm_beta\.weight=F32 +blk\.36\.ssm_out\.weight=Q8_0 +blk\.37\.attn_gate\.weight=Q8_0 +blk\.37\.attn_qkv\.weight=Q8_0 +blk\.37\.ffn_down_exps\.weight=Q6_K +blk\.37\.ffn_down_shexp\.weight=Q8_0 +blk\.37\.ffn_gate_exps\.weight=Q6_K +blk\.37\.ffn_gate_shexp\.weight=Q8_0 +blk\.37\.ffn_up_exps\.weight=Q6_K +blk\.37\.ffn_up_shexp\.weight=Q8_0 +blk\.37\.ssm_alpha\.weight=F32 +blk\.37\.ssm_beta\.weight=F32 +blk\.37\.ssm_out\.weight=Q8_0 +blk\.38\.attn_gate\.weight=Q8_0 +blk\.38\.attn_qkv\.weight=Q8_0 +blk\.38\.ffn_down_exps\.weight=Q8_0 +blk\.38\.ffn_down_shexp\.weight=Q8_0 +blk\.38\.ffn_gate_exps\.weight=Q6_K +blk\.38\.ffn_gate_shexp\.weight=Q8_0 +blk\.38\.ffn_up_exps\.weight=Q6_K +blk\.38\.ffn_up_shexp\.weight=Q8_0 +blk\.38\.ssm_alpha\.weight=F32 +blk\.38\.ssm_beta\.weight=F32 +blk\.38\.ssm_out\.weight=Q8_0 +blk\.39\.attn_k\.weight=Q8_0 +blk\.39\.attn_output\.weight=Q8_0 +blk\.39\.attn_q\.weight=Q8_0 +blk\.39\.attn_v\.weight=Q8_0 +blk\.39\.ffn_down_exps\.weight=Q8_0 +blk\.39\.ffn_down_shexp\.weight=Q8_0 +blk\.39\.ffn_gate_exps\.weight=Q6_K +blk\.39\.ffn_gate_shexp\.weight=Q8_0 +blk\.39\.ffn_up_exps\.weight=Q6_K +blk\.39\.ffn_up_shexp\.weight=Q8_0 +blk\.40\.attn_k\.weight=Q8_0 +blk\.40\.attn_output\.weight=Q8_0 +blk\.40\.attn_q\.weight=Q8_0 +blk\.40\.attn_v\.weight=Q8_0 +blk\.40\.ffn_down_exps\.weight=Q8_0 +blk\.40\.ffn_down_shexp\.weight=Q8_0 +blk\.40\.ffn_gate_exps\.weight=Q8_0 +blk\.40\.ffn_gate_shexp\.weight=Q8_0 +blk\.40\.ffn_up_exps\.weight=Q8_0 +blk\.40\.ffn_up_shexp\.weight=Q8_0 +blk\.40\.nextn\.eh_proj\.weight=Q8_0 diff --git a/paper/ICE_Technical_Report.md b/paper/ICE_Technical_Report.md new file mode 100644 index 0000000000000000000000000000000000000000..6176f20ebc1b933b7cdd2ac859e6596870852b89 --- /dev/null +++ b/paper/ICE_Technical_Report.md @@ -0,0 +1,1229 @@ +# ICE: Quantization by Error-Propagation Class in Sparse Mixture-of-Experts Models + +**Gökhan Buz** + +Independent researcher, `gbuzhf` on Hugging Face + +Version 1.0, August 2026 + +> **ICE** stands for *Isolation of Compounding Error*: the method finds the +> tensors whose quantization error compounds beyond the token that produced it, +> and isolates them out of the bit budget entirely. + +--- + +## Abstract + +Every quantizer in the GGUF ecosystem minimizes the same objective for every +tensor: the importance-matrix-weighted mean squared error of that tensor's +output, for the current token. That objective is correct for tensors whose error +dies with the token that produced it. It is the wrong objective for three other +kinds of tensor, and no published method separates them. + +This report describes ICE, which classifies every tensor by **how far its +quantization error travels** rather than by how large its activations are, and +allocates bits accordingly. Errors that flip a discrete routing argmax, that +enter a recurrent state decay, or that are written into the KV cache do not +decay with the token: they are re-read or compounded by every later token, so +their cost grows with context length. ICE holds that set exact and spends +everything else on the expert bank. + +The set is small enough that this is a line item rather than a trade-off. On the +model studied here the entire propagating set is 47 M parameters, 0.14% of the +checkpoint. Roughly half of it is already held exact by upstream `llama.cpp`; the +part ICE actually pins is **27.0 M parameters, 0.078% of the model, for about +0.03 GB.** + +The method is evaluated against two published mixed-precision ladders, Unsloth +Dynamic 2.0 and LocalAI APEX, on a common harness: mean KL divergence against the +bf16 checkpoint on WikiText-2. Across a twelve-tier comparison, nine tiers are +Pareto-optimal and three are strictly dominated. At matched size the 23 GB ICE +tier is 5.0% closer to bf16 than the nearest UD tier and 13.0% closer than the +nearest APEX tier, while being smaller than both. + +Four quantitative laws came out of the measurements, and they matter more than +the ladder: + +* **Law 1, sparsity.** In an MoE with `E` experts and `k` active, a bit spent on + the always-on core is worth `E/k` bits spent on the expert bank. Measured: + 270 against 8.5 units of work per GB-per-bpw, a ratio of **31.8 against a + predicted 32**. +* **Law 2, convexity.** Quantization error falls as `4^-b`. By Jensen, cleverness + in allocation is capped: water-filling over the real routing histogram beats + uniform by only **+0.139 bpw**, so ICE does no depth grading at all. The same + convexity says that at a fixed average, the **narrower type bracket always + wins**, measured at +16.6% and +7.0% for widening. +* **Law 3, placement.** Within a fixed bracket, assigning the high type + shallow-first is worth about **−8.5% KLD per bpw of gap** (r = −0.894) and + **reverses below a gap of 0.47 bpw**. The value of a bit falls with depth as + `g(t) ≈ exp(−t/9.95)`, so a block-0 bit is worth about 50 times a block-39 bit. +* **Law 4, the floor is epistemic.** Fitting `KLD ≈ k₀ + c_d·4^(−b_dense) + + c_e·4^(−b_expert)` on two independent harnesses gives `k₀` = 0.0205 and 0.0219. + The best tier measured sits 8% above that floor. Above roughly 5.5 expert bpw + you are buying calibration noise. **You cannot out-bit a wrong prior.** + +The negative results are reported at the same weight as the positive ones, +because they are the expensive part to rediscover. Ranking expert tensors by +imatrix-weighted sensitivity, the standard approach, made things monotonically +worse (+48.0% per tensor, +23.9% block-coupled). Bumping `ffn_down` above its +sibling projections, which is standard practice and which this work originally +derived from first principles, was **11.3% worse** than uniform at identical +size and was retracted. And lifting the entire dense path to bf16 bought +0.00122 KLD/GB against an expert-side rate of 0.0027 KLD/GB, which closes what +had looked like the largest remaining direction. + +--- + +## 1. Introduction + +### 1.1 The problem + +A 35-billion-parameter MoE checkpoint with 3 billion active parameters behaves +like neither a 35B dense model nor a 3B dense model. Its byte distribution and +its compute distribution disagree by a factor of nearly three: + +| role group | params | % of file | % of active compute | +|---|---:|---:|---:| +| routed experts (8 of 256 fire) | 32.21 B | 92.94% | 34.2% | +| attention | 1.03 B | 2.96% | 34.9% | +| output head | 0.51 B | 1.47% | 17.3% | +| SSM / state | 0.26 B | 0.74% | 8.7% | +| shared expert | 0.13 B | 0.36% | 4.3% | +| `token_embd` | 0.51 B | 1.47% | 0% (`get_rows`) | + +**65.8% of the active parameters are not routed.** Active total is 2.95 B, which +is what the "A3B" label means and a useful check that the accounting is right. + +Every quantization decision on such a model is a decision about how to split a +byte budget between a very large, very sparsely used store of weights and a +small, densely used one. Published ladders take reasonable but different +positions on that split, and until they are measured on one harness against one +reference there is no way to tell which position is correct. + +### 1.2 Why activation magnitude is the wrong ranking axis + +The signal available to a llama.cpp practitioner is the importance matrix +(imatrix), which accumulates `E[x·xᵀ]` per tensor over a calibration corpus. It +is a good and cheap statistic and it is what llama.cpp's own mixes and both +comparison ladders build on. + +But `E[x·xᵀ]` is the **input** side of the Gauss-Newton (Fisher) approximation to +the loss Hessian only. The full second-order term needs the output-side +sensitivity as well, and the two are not interchangeable. Two failure modes +follow, both of which were measured here (Section 7.1): + +1. **A depth bias that is mechanical, not informational.** Activation norms grow + with depth in a residual stream, so any ranking proportional to input energy + nominates deep tensors. On this model the imatrix's top-8 blocks were + [32..39]; direct measurement puts the highest-value blocks at [0..7]. +2. **A role bias against `ffn_down`.** The down-projection's input is + post-activation and small in norm, but its output is written straight into the + residual stream. Input energy under-weights it in exact proportion to how much + the activation function suppressed its input. + +There is a deeper problem than either bias. The imatrix objective is defined +**per token**. A tensor whose error is still doing damage a thousand tokens later +is scored as if it were not. + +### 1.3 The claim + +ICE asks a different question about each tensor. Not how strongly it is driven, +but **how long the error it introduces survives**. + +In one line: **freeze what propagates, spend everything else on the library.** + +### 1.4 Contributions + +1. **A four-class taxonomy of quantization error by propagation distance**, with + a procedure for finding each class in an arbitrary architecture (Section 3.1). +2. **Four measured laws** governing MoE bit allocation: the sparsity exchange + rate, the convexity cap, the placement rule, and the epistemic floor + (Section 4). Law 4 is cross-validated against an independent third-party + harness. +3. **A twelve-tier controlled comparison** against two published ladders on one + harness and one reference, with a Pareto analysis that eliminates three tiers + as dominated (Sections 6.1, 6.2). +4. **A direct measurement of the depth gain function** by sliding a fixed-width + band across depth at constant file size (Section 6.3). +5. **An error-budget decomposition** proving the dense path is correctly fed and + closing it as an optimization direction (Section 6.4). +6. **Seven documented negative results**, including the retraction of a rule this + work originally derived and shipped (Section 7). + +--- + +## 2. Background + +### 2.1 The model + +All measurements are on **Ornith-1.5-35B-A3B** (`qwen35moe` in llama.cpp) and its +abliterated derivative. The structure below is read from the checkpoint's own +tensor table. + +| | | +|---|---| +| blocks | 41 = 30 linear/SSM + 10 full-attention (indices 3, 7, ..., 39) + 1 MTP (`blk.40`) | +| residual width | 2048 | +| experts | 256, 8 active, so **sparsity ratio `E/k` = 32** | +| KV | 2 heads, head_dim 256, on 10 blocks only, about 20 KB/token | +| vocab | 248,320, so `output.weight` is 508.6 M in a single tensor | +| trunk parameters | 34,660,610,688 | + +**Every block is 95.6% expert weight** (805.3 M of about 842 M). The surgical +field is the remaining 35 M per block plus the two 508 M end caps. + +Two structural facts drive the method. First, the hybrid 3:1 layout means only 10 +of 40 trunk blocks maintain a KV cache, so the cached class is a quarter the size +it would be on a uniformly attentive model of the same depth. Second, `blk.40` is +a multi-token-prediction draft head whose proposals are verified by the target +model, which changes what its errors cost. + +Per-role parameter counts, which the whole allocation problem turns on: + +| role | tensors | params | % of model | +|---|---:|---:|---:| +| `ffn_{down,gate,up}_exps` | 120 | 32,212,254,720 | 92.94% | +| `token_embd` | 1 | 508,559,360 | 1.47% | +| `output` | 1 | 508,559,360 | 1.47% | +| `attn_qkv` (fused, SSM blocks) | 30 | 503,316,480 | 1.45% | +| `attn_gate` | 30 | 251,658,240 | 0.73% | +| `ssm_out` | 30 | 251,658,240 | 0.73% | +| `attn_q` | 10 | 167,772,160 | 0.48% | +| `attn_output` | 10 | 83,886,080 | 0.24% | +| shared experts | 120 | 125,829,120 | 0.36% | +| `ffn_gate_inp` (routers) | 40 | 20,971,520 | 0.06% | +| `attn_k` | 10 | 10,485,760 | 0.03% | +| `attn_v` | 10 | 10,485,760 | 0.03% | +| `ssm_alpha`, `ssm_beta` | 60 | 3,932,160 | 0.01% | +| `ssm_conv1d` | 30 | 983,040 | 0.00% | +| norms and misc | 231 | ~176,128 | 0.00% | + +### 2.2 Quantization in llama.cpp + +Two format families are relevant: + +* **k-quants** (`Q3_K`, `Q4_K`, `Q5_K`, `Q6_K`): block-scaled with a super-block + hierarchy, nominal 3.4375 / 4.5 / 5.5 / 6.5625 bpw. +* **IQ quants** (`IQ3_S`, `IQ4_XS`, `IQ4_NL`): codebook based, nominal + 3.44 / 4.25 / 4.5 bpw. + +A recipe is a `--tensor-type-file`: a list of regex-to-type rules applied per +tensor. This is what makes controlled experiments possible. Two recipes that +assign the same *number* of tensors to each type produce files of identical size, +so any KLD difference between them is attributable purely to *which* tensors got +which type. Every comparison in Sections 6 and 7 is constructed that way. + +Some tensors are never quantized by upstream regardless of recipe: `ffn_gate_inp` +is excluded by name, `ssm_conv1d` has a first dimension of 4 and is not a +multiple of the block size, and 1-D tensors including all norms are skipped. This +matters for honest accounting and is treated in Section 3.2. + +### 2.3 The metric + +All quality numbers are **mean KL divergence against the bf16 checkpoint's own +token distribution**, via `llama-perplexity --kl-divergence` on WikiText-2 raw +test. KLD is preferred to perplexity because it measures distance from the model +being approximated rather than from the ground truth text, because it stays +sensitive in the regime of interest (tiers below differ by 40% in KLD while their +perplexity ratios differ in the third decimal), and because it is deterministic. + +### 2.4 Prior ladders + +**Unsloth Dynamic 2.0 (UD)** assigns bits per tensor with a calibration-driven +policy and, in the variants measured, pins the dense path at Q8_0 across every +size tier. This gives high active bpw at a given file size, and on the measured +ladder the UD tiers are the strongest files at the large end. + +**LocalAI APEX** uses a sensitivity-driven allocation with a documented +methodology and a published technical report, which is the structural model for +this document. Its tiers span a wider size range, down to 14.24 GB. + +**AtomicChat AD** appears once, in Section 4.4, as an independent harness whose +published measurements were used to cross-validate Law 4. Its tiers are not +otherwise compared here. + +The comparison in Section 6.1 is not an argument that any ladder is badly +designed. It is a measurement of where each sits on a size-versus-fidelity plane +when all of them are evaluated identically, which is information no individual +publication can provide because each uses its own harness and its own reference. + +### 2.5 Related literature + +The problem addressed here is the format-constrained version of the second-order +weight-sensitivity problem studied by OBD and OBS and, in the modern LLM setting, +by GPTQ, AWQ and SqueezeLLM. Those methods solve for *values* within a fixed +format. ICE solves for *format assignment* across tensors and takes the values +from the stock quantizer, so the two are complementary. Full references are in +Section 10. + +--- + +## 3. Method + +### 3.1 The four error-propagation classes + +Classify every tensor by how far an error travels, not by how big its activations +are. + +| class | mechanism | failure shape | treatment | +|---|---|---|---| +| **DISCRETE** | error flips an argmax, so a different computation runs | categorical, not graded | exact (F32) | +| **RECURRENT** | error enters a state decay and compounds multiplicatively along the sequence | grows with context length | exact (F32) | +| **CACHED** | error is written once and re-read by later tokens | frozen in, cannot be re-decided | near-exact (F16) | +| **INSTANT** | error affects this token's output only | graded, bounded | this is where the budget lives | + +**How to find them in an arbitrary architecture.** + +* **DISCRETE**: anything feeding a `top_k`, `argmax` or hard gate. In an MoE that + is the router (`ffn_gate_inp`), plus any hard routing or branching signal. +* **RECURRENT**: anything multiplying or parameterizing a carried state. SSM and + Mamba decay and timestep terms (`ssm_alpha`, `ssm_beta`, `ssm_a`, `ssm_dt`), + gated linear-attention decay, any RNN gate. **Not** the state's output + projection, which is INSTANT. +* **CACHED**: whatever is written into the KV cache, so `attn_k` and `attn_v`. + Note that `attn_q` is not cached, which is the trap discussed in Section 7.4. +* **INSTANT**: everything else. + +The three non-INSTANT classes share one property that the per-token imatrix +objective cannot see: **their cost is a function of context length.** That is the +property the method is named for. + +### 3.2 Rule 1: freeze the propagating set + +On this model the entire propagating set is **47 M parameters, 0.14% of the +checkpoint**, and holding all of it exact costs 0.15 GB. It is not a trade-off, +it is a line item you can simply pay. Expect the same on any sparse MoE, because +routers and state gates are structurally tiny. + +An honest accounting splits that 47 M in two. + +**Already exact without any recipe rule (22.3 M params).** `ffn_gate_inp` is +excluded by name in `src/llama-quant.cpp`; `ssm_conv1d` has a first dimension of +4, not a multiple of the block size; `ssm_a`, `ssm_dt` and all norms are 1-D and +never quantized. Pinning any of these would be **inert**, and ICE's recipe +validator rejects inert rules. The propagation argument still explains why no +ladder should try to reclaim those bytes, but ICE does not get credit for them. + +**Actually pinned by ICE (27.0 M params, 0.078%):** + +| tensor | blocks | params | class | assigned | bytes | +|---|---:|---:|---|---|---:| +| `ssm_alpha` | 30 | 1,966,080 | RECURRENT | F32 | 7.86 MB | +| `ssm_beta` | 30 | 1,966,080 | RECURRENT | F32 | 7.86 MB | +| `attn_k` | 11 | 11,534,336 | CACHED | F16 | 23.07 MB | +| `attn_v` | 11 | 11,534,336 | CACHED | F16 | 23.07 MB | +| **total** | **82 tensors** | **27,000,832** | | | **61.86 MB** | + +The same tensors at the dense path's Q8_0 would occupy 28.7 MB, so **Rule 1's +incremental cost is 33 MB**, about 0.03 GB. + +Everything always-on but INSTANT (`attn_q`, `attn_output`, `attn_qkv`, +`attn_gate`, `ssm_out`, shared experts, `output`, `token_embd`) is held at Q8_0. +Section 4.1 explains why that is not generosity but arithmetic. + +### 3.3 Rule 2: the draft head follows the tier + +A draft head does not need target-model precision. `blk.40` is an MTP block whose +proposals are verified by the target model, so its errors cost **acceptance +rate**, not correctness. ICE pins only the draft block's projections at Q8_0 and +lets its experts follow the tier. + +The saving is exact: `blk.40` holds 3 × 268,435,456 = 805,306,368 expert +parameters, which occupy 855 MB at Q8_0 and 453 MB at the 23G tier's Q4_K. +**Rule 2 frees 402 MB per tier**, an order of magnitude more than Rule 1 spends. + +Other ladders pin the whole block at Q8_0, drop it to Q4_0, or delete it. The +question of what a drafter actually needs does not appear to have been asked. +Measured draft acceptance with the shrunk block is **96.04%** (388 of 404), so +the saving is not paid for out of throughput. See Section 6.6 for why this is not +a clean A/B. + +### 3.4 Rule 3: uniform width across FFN roles + +`ffn_down`, `ffn_gate` and `ffn_up` receive the **same** bit-width. + +This contradicts standard practice and it contradicts what this work originally +shipped. The first ladder put `ffn_down` 0.79 bpw above its siblings, derived +from the imatrix participation ratio (Section 4.2). A controlled test at +identical size refuted it: **0.041192 uniform against 0.046449 bumped, 11.3% +better uniform**, reproduced exactly on a rebuild. Rebuilding the whole ladder +uniform improved every rung, by 7.3%, 11.3%, 3.3% and 3.1% at 19, 21, 23 and +25 GB. + +The retraction is discussed in Section 7.2. Practical rule until someone measures +better: **allocate the expert stack uniformly, despite convention.** + +### 3.5 Rule 4: choose the narrowest bracket the budget allows + +A tier's expert budget is met by mixing two types, a high and a low, in whatever +ratio hits the target size. There is usually more than one pair that can hit a +given average. **Always choose the pair with the smallest bit-width separation**, +by the convexity argument in Section 4.2. The enumeration is over a small +discrete set, so it is exhaustive rather than heuristic. + +### 3.6 Rule 5: place the high type shallow-first, only if the gap exceeds 0.47 bpw + +Given the pair from Rule 4 and the ratio that hits the budget, the remaining +freedom is *which* blocks get the high type. Measurement (Section 4.3) says +assign them shallow-first, but only when the two types are far enough apart to +make position meaningful. Below about half a bit of separation the attempt costs +more than it returns. + +This is a conditional rule and the condition is checkable before any file is +built. On the one tier in the ladder whose gap is 0.25 bpw, applying it anyway +made the result 1.6% worse. + +### 3.7 The algorithm + +``` +Input: target file size S, model M, imatrix I (for the quantizer, not for ranking) +Output: a tensor-type-file recipe + +1. classify every tensor into DISCRETE / RECURRENT / CACHED / INSTANT + pin the RECURRENT and CACHED tensors that upstream would otherwise quantize + reject any rule that would be inert # Rule 1 + +2. always-on INSTANT tensors -> Q8_0 # Law 1 + S_remaining <- S - size(everything assigned so far) + +3. blk.40 projections -> Q8_0; blk.40 experts join the pool # Rule 2 + +4. b_target <- S_remaining / n_expert_params + enumerate all (high, low) type pairs bracketing b_target + choose the pair minimizing (bpw_high - bpw_low) # Rule 4 / Law 2 + n_high <- the count that hits b_target + +5. if (bpw_high - bpw_low) > 0.47: # Rule 5 / Law 3 + high type -> blocks [0 .. n_high-1] # shallow-first + else: + high type -> even Bresenham dither across depth + gate, up and down always move together # Rule 3 + +6. emit; verify rule count, zero double-matches, zero uncovered, zero inert +``` + +Step 6 is not optional. On all four shipped tiers it reports **0 double-matched, +0 uncovered, 0 inert**, and it is the check that catches a regex which did not +match what it was intended to match, something `llama-quantize` will not warn +about. + +### 3.8 Adapting ICE to a new MoE + +1. **Inventory.** Dump the tensor list, group by role, compute params, share of + file, and share of *active* compute (weighting routed experts by `k/E`). +2. **Sparsity ratio `E/k`.** This is the always-on-to-expert exchange rate and it + sets the entire shape of the recipe (Law 1). +3. **Classify** every tensor into the four classes. Sum the propagating set. If + it is well under 1% of the model, freeze all of it. +4. **Confirm grading is not worth it** by water-filling over the routing + histogram (Law 2). +5. **Fixed body** = propagating set + always-on core + draft projections. + Everything else is one dial: `size = body + E_params · bpw / 8`. +6. **Bound the ladder.** Fit Law 4 on any measured tiers to find `k₀`. The useful + band runs from the cliff, where the expert term explodes, to saturation, where + the floor dominates. On this model that was 17 to 27 GB. + +### 3.9 What ICE deliberately does not do + +**No per-layer graded bit ladder.** Refused explicitly, not overlooked. Law 2 +caps the gain at +0.139 bpw and the *direction* of a depth profile was unresolved +at the time: the participation ratio says the middle blocks need more bits +because they are flattest, convention says the edges do because they matter more +functionally. + +Note this does not conflict with Rule 5. Rule 5 is a binary partition between two +already-chosen types at a fixed count, which changes no bit-widths. A graded +ladder changes the bit-widths themselves and pays the convexity penalty for it. + +**No imatrix-derived tensor ranking.** Three variants were tested and all three +lost (Section 7.1). + +--- + +## 4. The four measured laws + +### 4.1 Law 1, sparsity: always-on bits are worth `E/k` routed bits + +For anything that fires on every token, compute share and storage cost are both +proportional to parameter count, so *work per byte* is a constant. Routed experts +divide that constant by the routing fraction. + +Measured on this model, with 8 of 256 experts active: every always-on role scores +about **270 units of work per GB-per-bpw**; the experts score **8.5**. The ratio +is **31.8, against a predicted `E/k` = 32.** + +> A bit spent on the always-on core is worth `E_total / E_active` bits spent on +> the expert bank. Compute this ratio first for any new MoE. + +This is why the always-on core sits at Q8_0 in every tier and why the ladder's +entire dynamic range lives in the experts. It also explains a negative result: +lifting `attn_gate` or `ssm_out` to Q6_K is *harmful*, because they are always-on +and the 32× law says those bytes belong in the experts. + +### 4.2 Law 2, convexity: allocation cleverness is capped + +Quantization error falls as `4^-b`, which is convex. By Jensen's inequality, +non-uniform allocation beats uniform only when sensitivity varies enormously, and +the gain scales with the **logarithm** of the spread. + +Measured by water-filling over the real per-expert routing histogram: + +| sensitivity spread | optimal gain over uniform | +|---|---| +| 46× (real expert routing) | **+0.139 bpw** | +| 3.7× (depth spread of `ffn_down`) | less | +| ~250× (hypothetical) | ~1 bpw | + +Routing on this model is well balanced (209.5 effective experts of 256, Gini +0.331), which is precisely why the spread is only 46×. + +> **Corollary: stop tuning ladders.** This is why every published recipe lands +> within about 0.3 bpw of every other one. Per-expert allocation, depth grading +> and per-layer bands are all worth less than 0.15 bpw. + +**The second consequence of the same convexity** is the bracket rule. At a fixed +average bit-width, a tight bracket incurs strictly less expected cost than a wide +one. Measured at constant file size: + +| tier | narrow pair | wide pair | penalty for widening | +|---|---|---|---:| +| 21G | Q4_K / IQ4_XS (gap 0.25) 0.039182 | Q4_K / Q3_K (gap 1.06) 0.045673 | **+16.6%** | +| 19G | IQ4_XS / IQ3_S (gap 0.81) 0.060015 | Q4_K / Q3_K (gap 1.06) 0.064213 | **+7.0%** | + +Both wide alternatives hit the same file size, so the penalty is entirely bracket +width. This is Rule 4. + +An empirical check on the `4^-b` form: fitting the four shipped tiers after +subtracting the Law 4 floor gives a decay of roughly 0.65 to 0.75 bpw per halving +of excess KLD, against 0.5 bpw for a pure `4^-b`. The residual difference is the +dense path, which does not move across the ladder. + +### 4.3 Law 3, placement: gain scales with the type gap + +Within a fixed bracket, where the high type goes still matters. Two measurements +establish the rule, both byte-identical by construction. + +**The depth gain function.** A 15-block window of the high type was slid across +depth at constant count, on the 23G tier: + +| band position | mean KLD | +|---|---:| +| blocks [0..14] | **0.032407** | +| blocks [6..20] | 0.033014 | +| even dither (reference) | 0.034162 | +| blocks [18..32] | 0.036107 | +| blocks [25..39] | 0.036136 | +| blocks [12..26] | 0.036760 | + +Fitting an exponential gives **`g(t) ≈ exp(−t/9.95)`**, RMS residual 0.00077 +against roughly 0.0005 error bars. A bit in block 0 is worth about **50 times** a +bit in block 39. + +Two things a simple decay model does not capture. First, `[12..26]` and the even +dither share a centre of mass but differ by 0.0026, so **concentration matters +independently of position**. Second, `[12..26]` is the worst row despite being +centred, because it leaves blocks [0..11] entirely on the low type: **starving +the shallow region costs more than enriching the deep region helps.** + +**The conditional.** Applying shallow-first across six byte-identical pairs: + +| tier | pair | gap (bpw) | dither | shallow | gain | +|---|---|---:|---:|---:|---:| +| 21G | Q4_K / IQ4_XS | 0.25 | 0.039182 | 0.039814 | **+1.6%** | +| 19G | IQ4_XS / IQ3_S | 0.81 | 0.060015 | 0.058676 | −2.2% | +| 25G | Q5_K / Q4_K | 1.00 | 0.028955 | 0.028131 | −2.8% | +| 23G | Q5_K / Q4_K | 1.00 | 0.034290 | 0.032536 | **−5.1%** | +| 21G | Q4_K / Q3_K | 1.06 | 0.045673 | 0.043766 | −4.2% | +| 19G | Q4_K / Q3_K | 1.06 | 0.064213 | 0.059623 | −7.1% | + +Linear fit: **gain ≈ −8.5% per bpw of gap, r = −0.894, zero-crossing at +0.47 bpw.** + +The 21G row is the important one. It is the only tier with a sub-0.5 bpw gap and +the only one where shallow-first made things worse. The law predicted the sign +before that measurement was taken, which is the reason to treat it as a law +rather than a fit through six points. + +**Laws 2 and 3 pull against each other.** Widening the gap makes placement more +valuable (Law 3) but costs more than placement can recover (Law 2). The +resolution is ICE's allocation policy: **use the narrowest bracket the budget +allows, then apply shallow-first only where that bracket is still wider than +about 0.5 bpw.** Law 2 dominates; placement is a second-order correction inside +an already-chosen bracket, never a reason to widen one. + +### 4.4 Law 4, the floor is epistemic, not numeric + +Fitting `KLD ≈ k₀ + c_d·4^(−b_dense) + c_e·4^(−b_expert)` on two independent +harnesses: + +| harness | floor `k₀` | `c_e/c_d` | +|---|---:|---:| +| this work (WikiText-2, ctx 2048) | 0.0205 | 5.91 | +| AtomicChat AD (eval_neutral, ctx 4096) | 0.0219 | 7.22 | + +The best file measured, `UD-Q6_K`, scores 0.0221 against a 0.0205 floor, so it is +**8% above the floor: the top of the ladder is saturated.** Above roughly +5.5 expert bpw you are buying calibration noise, not accuracy. + +> **You cannot out-bit a wrong prior.** Quantization is lossy compression guided +> by a belief about importance. Improve the imatrix before adding bits. + +A direct probe later confirmed the fit from a completely different direction. +Holding the experts at Q8_0 and quantizing only the dense path measures +**0.018297** on the abliterated checkpoint (Section 6.4), against a fitted `k₀` +of 0.0205 and a card estimate of about 0.0185 on the clean one. An extrapolated +intercept and a direct measurement agreeing to within about 10% is the strongest +evidence available that the floor is real. + +**Caveat, stated when the law was formulated and still true.** This fitted model +ranks the extremes well and gets close calls wrong: it misorders AtomicChat's +`Q4_K_M` against `AD-Q4_K-IQ4_XS`. Use it to find the floor and the saturation +point, never to choose between two candidate recipes. + +--- + +## 5. Experimental setup + +| item | value | +|---|---| +| base checkpoints | `Ornith-1.5-35B-A3B` (bf16) and its abliterated derivative (bf16) | +| quantizer | upstream `llama.cpp`, pinned commit, unmodified | +| importance matrix | one generic imatrix, identical across every file compared | +| corpus | WikiText-2 raw test | +| context | 2048 | +| chunks | 16 for the twelve-tier ladder comparison, **64** for every controlled experiment | +| metric | mean KLD against the corresponding bf16 checkpoint | +| hardware | rented single-GPU instances, plus CPU-only for the bf16 reference pass | + +### 5.1 Controls + +Three disciplines make the comparisons interpretable, and their absence is the +usual reason quantization comparisons are not: + +1. **One reference per model.** Every KLD in a given table is against the same + bf16 logits file. Numbers are never compared across base models; where both + checkpoints appear they are in separate tables and labelled. +2. **Byte-identical variants.** Every experiment in Sections 6 and 7 permutes + type assignments rather than changing type counts, so the variant matches its + control in size, usually to the byte. A KLD difference is then a pure recipe + effect. +3. **One harness.** Same binary, corpus, context and chunk count throughout. + +### 5.2 Reproducibility + +The same recipe measured through three different paths: + +| path | chunks | device | mean KLD | +|---|---:|---|---:| +| published model card | 16 | CPU | 0.0345 | +| controlled study, phase 2 | 64 | CPU | 0.034162 | +| controlled study, phase 5 | 64 | GPU | 0.034290 | + +Spread under 1%. KLD against a fixed reference is deterministic; chunk count +tightens the error bar without moving the mean, and the CPU/GPU difference is +floating-point accumulation order. + +**Error bars.** At 16 chunks the reported bars are 0.0008 to 0.0012; at 64 chunks +they are 0.0005 to 0.0007. All significance statements below use the 64-chunk +figure. + +--- + +## 6. Results + +### 6.1 The twelve-tier comparison + +All twelve files, one harness, one reference, 16 chunks, clean base model. Sorted +by mean KLD. + +| tier | size | mean KLD | 99% KLD | 99.9% KLD | PPL ratio | same top-1 | active bpw | file bpw | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| `UD-Q6_K` | 30.20 GB | **0.0221** | 0.220 | 0.809 | 0.9957 | 93.85% | 8.063 | 6.804 | +| `UD-Q5_K_S` | 25.83 GB | **0.0272** | 0.269 | 1.030 | 0.9862 | 93.51% | 7.693 | 5.820 | +| **`25G-ICE`** | 24.84 GB | **0.0303** | 0.296 | 1.049 | 0.9814 | 93.16% | 7.686 | 5.597 | +| `APEX-I-Balanced` | 26.28 GB | **0.0345** | 0.328 | 1.401 | 0.9907 | 92.55% | 6.913 | 5.922 | +| **`23G-ICE`** | 22.83 GB | **0.0361** | 0.332 | 1.272 | 0.9885 | 92.65% | 7.523 | 5.143 | +| `UD-Q4_K_XL` | 23.21 GB | **0.0380** | 0.389 | 1.405 | 0.9740 | 92.47% | 7.471 | 5.230 | +| **`21G-ICE`** | 20.84 GB | **0.0412** | 0.392 | 1.665 | 0.9924 | 92.03% | 7.357 | 4.695 | +| `APEX-I-Quality` | 23.84 GB | **0.0415** | 0.424 | 1.386 | 0.9843 | 91.98% | 6.699 | 5.371 | +| **`19G-ICE`** | 18.82 GB | **0.0608** | 0.615 | 1.877 | 1.0030 | 90.32% | 7.192 | 4.240 | +| `UD-IQ4_XS` | 18.68 GB | **0.0723** | 0.758 | 2.464 | 1.0526 | 89.46% | 6.762 | 4.209 | +| `APEX-I-Compact` | 17.56 GB | **0.0954** | 0.899 | 2.962 | 1.0101 | 87.83% | 5.228 | 3.956 | +| `APEX-I-Mini` | 14.24 GB | **0.2608** | 2.489 | 5.915 | 1.2281 | 80.49% | 4.180 | 3.208 | + +Read by size-matched pairs, which is the only fair way to read it: + +* `23G-ICE` (22.83 GB) against `UD-Q4_K_XL` (23.21 GB): **0.38 GB smaller, 5.0% + closer to bf16.** +* `23G-ICE` against `APEX-I-Quality` (23.84 GB): **1.01 GB smaller, 13.0% + closer.** +* `25G-ICE` (24.84 GB) against `APEX-I-Balanced` (26.28 GB): **1.44 GB smaller, + 12.2% closer.** +* `19G-ICE` (18.82 GB) against `UD-IQ4_XS` (18.68 GB): **15.8% closer** at + +0.14 GB. +* At the top, `UD-Q5_K_S` and `UD-Q6_K` are **the two best files measured** and + nothing in the ICE ladder reaches them. This agrees with Law 4: above about + 25 GB the expert term is saturated and the dense path is what remains, which is + the regime the UD policy of pinning everything dense at Q8_0 is built for. +* At the bottom, `APEX-I-Compact` and `APEX-I-Mini` are the only tiers below + 18.5 GB. ICE does not operate there and offers no evidence about that regime. + +**The advantage grows as size falls**, from −1.9% at 25 GB to −18.6% at 21 GB +against the UD curve, which is Law 4 again: at the top the floor dominates and no +allocation choice can move it. + +**A note on active bpw.** The column is included because it explains where the +bits went, but it must not be used to rank. Tested against measured KLD it +misorders 4 of the 8 UD and APEX tiers (Spearman +0.95), and 10 of AtomicChat's +13 on a second ladder (+0.71). It is a linear average of bit-widths while error +is not linear in bits, so Q8_0-dense-path designs come out flattered. + +### 6.2 Pareto analysis: twelve tiers reduce to nine + +A tier is worth publishing only if nothing else is both smaller and closer to +bf16. + +**Pareto-optimal (nine):** `APEX-I-Mini` (14.24 / 0.2608), `APEX-I-Compact` +(17.56 / 0.0954), `UD-IQ4_XS` (18.68 / 0.0723), **`19G-ICE`** (18.82 / 0.0608), +**`21G-ICE`** (20.84 / 0.0412), **`23G-ICE`** (22.83 / 0.0361), **`25G-ICE`** +(24.84 / 0.0303), `UD-Q5_K_S` (25.83 / 0.0272), `UD-Q6_K` (30.20 / 0.0221). + +**Dominated (three):** + +| covered tier | covered by | +|---|---| +| `UD-Q4_K_XL` (23.21 GB / 0.0380) | `23G-ICE`: 0.38 GB smaller, 5.0% better | +| `APEX-I-Quality` (23.84 GB / 0.0415) | `23G-ICE`: 1.01 GB smaller, 13.0% better | +| `APEX-I-Balanced` (26.28 GB / 0.0345) | `UD-Q5_K_S`: 0.45 GB smaller, 21.1% better; also `25G-ICE`: 1.44 GB smaller, 12.2% better | + +This is the most useful output of the comparison, and it is worth being precise +about its scope. It says that at these three operating points, on this model, +with this harness, another published file is strictly better on both axes. It +does not say those ladders are worse in general: the same analysis keeps two UD +tiers as the best files on the board and two APEX tiers as the only options below +18.5 GB. + +### 6.3 Applying Law 3 to the shipped ladder + +Each recipe byte-identical to the tier it replaces, abliterated base, 64 chunks: + +| tier | gap | shipped | revised | change | decision | +|---|---:|---:|---:|---:|---| +| 19G | 0.81 | 0.060015 | **0.058676** | −2.2% (1.4σ) | adopt | +| 21G | 0.25 | **0.039182** | 0.039814 | +1.6% | **do not change** | +| 23G | 1.00 | 0.034290 | **0.032536** | **−5.1% (3.6σ)** | adopt | +| 25G | 1.00 | 0.028955 | **0.028131** | −2.8% (1.3σ) | adopt | + +Only 23G is individually significant at conventional thresholds. 19G and 25G are +directionally consistent at about 1.3σ each and are adopted because the law they +confirm was fitted on data including an independent significant point and a +correctly predicted failure. 21G is left unchanged, which is the honest output of +a conditional rule when the condition does not hold. Its revised recipe file has +the **same SHA-256** as the one it replaces. + +At the local slope of each tier's size-versus-quality curve these gains are worth +approximately 0.14, 0, 0.49 and 0.34 GB. + +### 6.4 The error budget, and why the dense path is closed + +Decomposing total damage at the 23G operating point by holding one side exact: + +| | mean KLD | share of damage | bytes | share of bytes | +|---|---:|---:|---|---:| +| dense path | 0.018297 | 53.6% | 2.7 GB | 12% | +| routed experts at 4.875 bpw | 0.015993 | 46.4% | 19.6 GB | 88% | + +The dense path causes more than half the damage in an eighth of the bytes, which +looks like an obvious misallocation. It is not, and the deciding measurement is +direct: + +| variant | mean KLD | size | +|---|---:|---:| +| control 23G | 0.034290 | 22.34 GB | +| all 238 dense tensors lifted to BF16 | **0.031543** | 24.59 GB | + +Gain 0.002747 for +2.25 GB is **0.00122 KLD/GB**, against a measured expert-side +rate of **0.0027 KLD/GB**. Those bytes are worth **2.2× more in the experts.** + +**The dense path is not underfed, it is correctly fed.** The 0.018297 figure is +the intrinsic cost of quantizing the dense path at all, which is Law 4's floor +seen from another angle, not a misallocation waiting to be recovered. The ICE +split of 88% of bytes to experts is already on the right side of the trade and +this direction is closed. + +**Sub-additivity.** Dense-only damage (0.002747) plus expert-only damage +(0.015993) is 0.018740, while the two together give 0.034290. The perturbations +are strongly sub-additive, so per-role "share of the floor" was never a +meaningful quantity, and per-role attribution of a joint quantization error +should be treated with suspicion generally. + +### 6.5 Draft acceptance + +Measured acceptance with the shrunk `blk.40`: **96.04%** (388 of 404). The +baseline with the block fully pinned at Q8_0 was 92.97%, but on a different tier, +so **this is not a clean A/B and is not claimed as one.** What it establishes is +that the smaller draft head still drafts and there is no collapse, which is the +condition Rule 2 needs. + +### 6.6 Falsifiable predictions, scored + +Three predictions were registered before measurement. + +1. ~~`25G-ICE` beats `UD-Q5_K_S` on measured KLD at about 1 GB less.~~ + **FAILED.** 0.0303 against 0.0272. ICE is 0.99 GB smaller but not better. The + prediction rested on the 0.79 bpw `ffn_down` gap, which was then refuted + outright (Section 7.2). +2. **`19G-ICE` beats `UD-IQ4_XS`.** **HELD**, 0.0608 against 0.0723, 15.8% + better. +3. **Un-pinning `blk.40` costs under 2 pp of acceptance.** **HELD, direction + unverified**, for the reason in Section 6.5. + +One of three failed, and the failure took a derived rule down with it. That is +recorded here rather than quietly dropped because a method that only reports its +successful predictions is not making predictions. + +--- + +## 7. Negative results + +Seven results reported at the same weight as the positive ones. Four of them +refute standard practice and one retracts a rule this work itself shipped. + +### 7.1 Imatrix-weighted sensitivity ranking (three variants, all lost) + +Score each of the 120 routed-expert tensors by `s_t = Σ_j a_j ‖ΔW‖²` using the +imatrix activations, then spend the high type on the top scorers. On the 23G +tier, byte-identical: + +| variant | result against control | +|---|---:| +| per-tensor ranking, roles free | **+48.0%** | +| block-coupled (gate, up, down move together) | **+23.9%** | +| propagation-corrected `g(t)·s_t`, roles free | −1.2% (0.6σ, noise) | + +The first two are catastrophic; the third is indistinguishable from doing +nothing. The diagnosis is Section 1.2: `s_t` grows with depth mechanically, its +top-8 blocks being [32..39] while the measured truth is [0..7]. Coupling the +roles halves the damage without fixing the direction. + +The third variant is the most informative failure. Multiplying by the measured +`g(t)` fixes the depth axis and *still* does not beat the control, because `s_t` +is also wrong on the role axis: it starves all 40 `ffn_down` tensors and gives +back exactly what the depth correction won. + +**Conclusion: imatrix activation energy is not a usable ranking signal for +inter-tensor bit allocation on this architecture.** It remains the right input to +the quantizer itself, which is how ICE uses it. + +### 7.2 The `ffn_down` bump, and the retraction of the leverage rule + +The imatrix stores per-channel importance `h`. Its concentration can be measured +with the participation ratio `PR = (Σh)² / (n·Σh²)`. Spiky `h` (low PR) is a gift +because the quantizer knows where to spend its block scale; flat `h` (high PR) is +the hard case. So flat tensors should need more bits. Measured: + +| tensor | PR/n | +|---|---:| +| `attn_k`, `attn_q`, `attn_v` | 0.0069 (about 14 of 2048 channels) | +| `attn_qkv`, `attn_gate`, `ssm_alpha`, `ssm_beta` | 0.0128 | +| `ffn_gate_exps`, `ffn_up_exps` | 0.2048 | +| **`ffn_down_exps`** | **0.6104, flattest in the model** | + +This gave a clean derivation: `down` should sit `log4(0.6104/0.2048)` = 0.79 bpw +above `gate` and `up`. It agreed with convention, it agreed with the best +published rungs (`UD-Q4_K_XL` and AtomicChat's `AD-Q5_K-Q4_K` both run about +1.0), and it shipped. + +**It is wrong.** A controlled test at identical size: + +| expert allocation | size | mean KLD | +|---|---:|---:| +| `down` = `gate`/`up` + 0.79 | 20.82 GB | 0.046449 | +| **uniform** | 20.85 GB | **0.041192** | + +**Uniform is 11.3% better**, reproduced exactly on a rebuild. The whole ladder +was rebuilt uniform, improving every rung by 7.3%, 11.3%, 3.3% and 3.1%. + +**What PR actually predicts is unknown.** It measures how concentrated the +*input* importance is, which is not how much a tensor's error reaches the output. +`ffn_down` writes straight into the residual stream while `gate` and `up` errors +are attenuated through the gating nonlinearity first. Position in the circuit is +not captured by PR and on this model it evidently dominates. + +This is the same defect as Section 7.1, seen from a different direction: an +input-side statistic used as an output-side one. + +### 7.3 Per-expert allocation by routing frequency + +Load balancing works on this model (209.5 effective experts of 256, Gini 0.331) +and Law 2 caps the achievable gain at +0.139 bpw. A naive "hot 25% at 6.5 bpw, +cold at 2.5" split is **worse than uniform**, 0.00586 against 0.00552. **Dead.** + +### 7.4 Demoting `attn_q` + +The tempting corollary of the CACHED class: if K and V persist and Q does not, +demote Q below the always-on rate. **The arithmetic kills it.** Each query +distributes attention weight summing to 1, so the total k-influence accumulated +over a sequence is O(N), the same as q. Buys 63 MB and costs 0.17 active bpw. +Dropped before shipping. + +This is why Rule 1 promotes K and V rather than demoting Q, and the distinction +is not cosmetic. + +### 7.5 Dense role lifting + +Experts held at Q8_0, one dense role at a time lifted to BF16, reference +D = 0.019018: + +| role lifted | mean KLD | bytes added | rate (KLD/GB) | +|---|---:|---:|---:| +| output head | 0.018941 | +0.48 GB | 0.00016 | +| `token_embd` | 0.019217 | +0.48 GB | negative | +| attention | 0.017510 | +0.94 GB | 0.0016 | + +Against an expert-side rate of 0.0027 KLD/GB none is worth doing, and +`token_embd` is worth undoing. Section 6.4 generalizes this to the whole dense +path. + +Separately, demoting `token_embd` to Q4_K **did not replicate**: it helps at 21G +and hurts at 19G. Lifting `attn_gate` or `ssm_out` to Q6_K is **harmful**, +which is Law 1 doing its job. + +### 7.6 Same-family type swaps + +An early hypothesis held that mixing quantization *families* was itself harmful, +since a k-quant's and an IQ quant's error distributions are not commensurate. +Both 19G and 21G have same-family alternatives at their exact budgets: + +| tier | pair | dither | shallow | +|---|---|---:|---:| +| 21G | Q4_K / IQ4_XS (shipped, mixed) | **0.039182** | 0.039814 | +| 21G | Q4_K / Q3_K (same family) | 0.045673 | 0.043766 | +| 19G | IQ4_XS / IQ3_S (shipped, same family) | 0.060015 | **0.058676** | +| 19G | Q4_K / Q3_K (same family) | 0.064213 | 0.059623 | + +Both swaps lose, but this is **not** clean evidence for the family hypothesis, +because forcing the same family also forced a much wider bpw gap, and Law 3 shows +the gap is the real variable. The family hypothesis is **confounded and +unresolved**, and is listed as open work rather than confirmed. + +### 7.7 `up` to `down` scale migration + +`scale(up row j, s)` and `scale(down col j, 1/s)` is exactly output-preserving, +so variance could in principle be migrated out of `down`'s columns (which pollute +every row's quantization range) into `up`'s rows (which get their own k-quant +scales for free). **Dead on this model:** `down`'s columns are already uniform, +p95/p05 = 1.09. There is nothing to migrate. + +--- + +## 8. Limitations + +1. **One model family.** Every measurement is on `qwen35moe` at 35B-A3B. The + decay constant 9.95, the 0.47 bpw zero-crossing, the +0.139 bpw cap and the + floor `k₀` are fitted on this architecture. The *form* of the laws should + transfer; the constants should be re-measured. Law 1 is the exception, being + arithmetic rather than a fit. +2. **One evaluation corpus.** WikiText-2 raw test only. KLD against bf16 is a + faithfulness metric, not a capability metric, and says nothing directly about + code, multilingual or long-context behaviour. This last gap is awkward, since + the method's central claim is about context length. +3. **One imatrix, and a generic one.** Every experiment reuses the same generic + importance matrix. By Law 4 this is the largest untested variable in the work. +4. **Error bars.** 1σ is 0.0005 to 0.0007 at 64 chunks. Only the 23G placement + result clears 3σ; 19G and 25G are adopted on the strength of the law rather + than their individual significance, which is stated rather than hidden. +5. **Rule 1 is not expressible on a fused-QKV checkpoint.** It needs `attn_k` and + `attn_v` as separate tensors. On this model 30 of 41 blocks fuse them into + `attn_qkv`; those blocks happen to carry no KV cache, but on an architecture + that fuses QKV *and* attends, Rule 1 would require splitting the tensor at + conversion time, which a `--tensor-type-file` cannot do. +6. **Rule 2 applies only to checkpoints carrying a draft head**, and its + acceptance evidence is not a clean A/B (Section 6.5). +7. **The family-versus-gap confound** in Section 7.6 is unresolved. +8. **Comparison scope.** Two published ladders at twelve tiers on one model. Not + a survey. + +--- + +## 9. Conclusion and open work + +ICE classifies tensors by how far their quantization error travels rather than by +how strongly they are activated. The three classes whose error outlives the +current token turn out to be 0.14% of the checkpoint, so freezing them is a line +item rather than a trade-off, and the recovered budget goes to the expert bank. + +The resulting ladder is Pareto-optimal at its operating points against two +published ladders measured on one harness, dominating three of their tiers, and +the advantage grows as the budget falls. It does not win at the top of the +ladder, where Law 4 says the floor dominates and the measurements agree. + +The four laws are more portable than the ladder. They say: compute `E/k` first +because it sets everything; stop tuning allocations because convexity caps the +gain at about 0.14 bpw; place shallow-first only when the bracket is wide enough +to make position mean anything; and stop adding bits near the floor because you +cannot out-bit a wrong prior. + +They also close two directions that looked promising: imatrix-weighted +sensitivity ranking, and moving bytes from the experts to the dense path. + +**Open work, ranked by expected value.** + +1. **A domain-matched imatrix.** Untouched throughout, and Law 4 says it is worth + more than any remaining allocation change. +2. **Orthogonal rotation of the residual stream.** RMSNorm is invariant to + rotation but not to scaling, so a rotation folds into the norm weights exactly + and for free, and yields a standard GGUF. With attention at PR 0.0069 the + residual stream has about 14 dominant dimensions of 2048, and spreading that + energy is worth an estimated 0.9 bpw, which is larger than everything in this + report combined. Nobody has published rotation results for a hybrid SSM/MoE + and the fixed SSM state may not tolerate it. +3. **Shared basis plus per-expert delta.** Store `W̄` once and `Δ_i` per expert. + If `‖Δ‖/‖W‖ ≈ ¼` that is about 2 bpw for 0.4% overhead. Needs a llama.cpp op, + but the deciding measurement is free: compute `‖Δ_i‖/‖W̄‖` from the bf16. The + counter-argument is that this architecture already has a shared expert which + may have absorbed the common component. +4. **Graded placement profiles** on the wide-gap tiers, since every placement + variant tested is a step function while the measured `g(t)` is smooth. +5. **Disentangling family from gap** (Section 7.6). +6. **Re-measuring the constants on a second architecture**, the only way to learn + whether `g(t)`'s decay length is a property of this model or of residual + transformers generally. + +--- + +## 10. References + +**Quantization formats and tooling** + +1. G. Gerganov and the `llama.cpp` contributors. *llama.cpp: LLM inference in + C/C++*. https://github.com/ggml-org/llama.cpp + The k-quant and IQ formats, `llama-quantize`, the `--tensor-type-file` + mechanism and the `--kl-divergence` evaluation path are all from this project. + Every file measured here was produced by an unmodified build of it. +2. I. Kawrakow. K-quant format design (`Q3_K` through `Q6_K`), `llama.cpp` + PR #1684 and subsequent work. +3. I. Kawrakow. IQ-quant and codebook format design (`IQ3_S`, `IQ4_XS`, + `IQ4_NL`), `llama.cpp`. +4. The importance-matrix mechanism in `llama.cpp`, which accumulates per-tensor + activation second moments over a calibration corpus and feeds the quantizer's + scale search. + +**Second-order weight sensitivity, the basis for Sections 1.2 and 7.2** + +5. Y. LeCun, J. Denker and S. Solla. *Optimal Brain Damage*. NeurIPS 1989. +6. B. Hassibi and D. Stork. *Second Order Derivatives for Network Pruning: + Optimal Brain Surgeon*. NeurIPS 1992. +7. E. Frantar, S. Ashkboos, T. Hoefler and D. Alistarh. *GPTQ: Accurate + Post-Training Quantization for Generative Pre-trained Transformers*. + ICLR 2023. The Hessian-based formulation whose Gauss-Newton approximation the + imatrix represents the input half of. +8. J. Lin et al. *AWQ: Activation-aware Weight Quantization for LLM Compression + and Acceleration*. MLSys 2024. The activation-magnitude approach that + Section 7.1 finds insufficient for inter-tensor allocation on MoE. +9. S. Kim et al. *SqueezeLLM: Dense-and-Sparse Quantization*. ICML 2024. + Sensitivity-weighted non-uniform quantization, the closest prior work in + spirit, though solving for values rather than format assignment. + +**Allocation theory** + +10. C. Shannon. *Coding Theorems for a Discrete Source with a Fidelity + Criterion*. IRE 1959. The `4^-b` rate-distortion behaviour underlying Law 2. +11. Reverse water-filling for Gaussian sources, as in T. Cover and J. Thomas, + *Elements of Information Theory*, ch. 10. The allocation procedure used to + compute Law 2's +0.139 bpw bound. + +**Mixture-of-Experts, state-space models and speculative decoding** + +12. N. Shazeer et al. *Outrageously Large Neural Networks: The Sparsely-Gated + Mixture-of-Experts Layer*. ICLR 2017. The top-k routing whose discreteness + defines the DISCRETE class. +13. W. Fedus, B. Zoph and N. Shazeer. *Switch Transformers*. JMLR 2022. +14. A. Gu and T. Dao. *Mamba: Linear-Time Sequence Modeling with Selective State + Spaces*. 2023. The state decay terms that define the RECURRENT class. +15. Y. Leviathan, M. Kalman and Y. Matias. *Fast Inference from Transformers via + Speculative Decoding*. ICML 2023. The verification property that makes + Rule 2 sound. +16. F. Gloeckle et al. *Better & Faster Large Language Models via Multi-token + Prediction*. ICML 2024. The MTP head structure used by `blk.40`. + +**Comparison ladders** + +17. Unsloth. *Unsloth Dynamic 2.0 GGUFs*. https://docs.unsloth.ai + A calibration-driven per-tensor mixed-precision policy. Used here as a + comparison ladder; its `Q5_K_S` and `Q6_K` tiers are the two best files in + the Section 6.1 measurement. +18. E. Di Giacinto, R. Palethorpe and the LocalAI team. *APEX: Adaptive Precision + EXpert quantization*. LocalAI, 2025. + https://github.com/localai-org/apex-quant + A documented sensitivity-driven allocation method with a published technical + report, which is the structural model for this document. +19. AtomicChat `AD` GGUF ladder, used in Section 4.4 as an independent harness + for cross-validating Law 4 and in Section 6.1 for the active-bpw ranking + check. + +**Data and checkpoints** + +20. S. Merity, C. Xiong, J. Bradbury and R. Socher. *Pointer Sentinel Mixture + Models*. ICLR 2017. WikiText-2, raw test split, as distributed with the + `llama.cpp` CI dataset. +21. `ornith-ai/Ornith-1.5-35B-A3B`, the bf16 checkpoint and its MTP head, and the + abliterated derivative used for the controlled experiments. + +--- + +## Appendix A: recipe format and the shipped rules + +Each ICE tier is a `--tensor-type-file` of 443 rules, written out per tensor +rather than as wildcards, so the file is a complete and auditable statement with +no regex precedence to reason about. Excerpts are verbatim from the shipped 23G +recipe. + +**A.1 Global rules (2):** + +``` +^output\.weight$=Q8_0 +token_embd\.weight=Q8_0 +``` + +**A.2 Rule 1, the pinned propagating set.** 60 state-gate rules across the 30 SSM +blocks, 22 K/V rules across the 11 attention blocks: + +``` +blk\.0\.ssm_alpha\.weight=F32 +blk\.0\.ssm_beta\.weight=F32 +... +blk\.11\.attn_k\.weight=F16 +blk\.11\.attn_v\.weight=F16 +``` + +Routers, `ssm_conv1d`, `ssm_a`, `ssm_dt` and the norms do not appear. Upstream +holds them exact already, so a rule would be inert and the validator rejects it. + +**A.3 Rule 2, the draft block.** All 11 of `blk.40`'s rules, from the 23G tier: + +``` +blk\.40\.attn_k\.weight=F16 <- Rule 1 applies to the draft block too +blk\.40\.attn_v\.weight=F16 +blk\.40\.attn_q\.weight=Q8_0 +blk\.40\.attn_output\.weight=Q8_0 +blk\.40\.nextn\.eh_proj\.weight=Q8_0 +blk\.40\.ffn_down_shexp\.weight=Q8_0 +blk\.40\.ffn_gate_shexp\.weight=Q8_0 +blk\.40\.ffn_up_shexp\.weight=Q8_0 +blk\.40\.ffn_down_exps\.weight=Q4_K <- follows the tier, not pinned Q8_0 +blk\.40\.ffn_gate_exps\.weight=Q4_K +blk\.40\.ffn_up_exps\.weight=Q4_K +``` + +The three `_exps` lines are the whole of Rule 2. They read `IQ3_S` at 19G, +`IQ4_XS` at 21G, `Q4_K` at 23G and `Q5_K` at 25G, so the draft block is a full +member of the expert pool and is eligible for the high type when the tier is +generous enough to reach it. + +**A.4 Rules 3 to 5, the expert body.** One rule per routed-expert tensor, with +`gate`, `up` and `down` always receiving the same type within a block (Rule 3). +The pair comes from Rule 4 and the block assignment from Rule 5. Verified against +the shipped files: + +| tier | pair (high / low) | gap | expert bpw | high-type blocks | count | +|---|---|---:|---:|---|---:| +| 19G | IQ4_XS / IQ3_S | 0.81 | 3.899 | `[0..22]` shallow-first | 23 / 41 | +| 21G | Q4_K / IQ4_XS | 0.25 | 4.383 | `0 2 4 6 7 9 11 13 15 17 19 20 22 24 26 28 30 32 34 35 37 39` (even dither) | 22 / 41 | +| 23G | Q5_K / Q4_K | 1.00 | 4.868 | `[0..14]` shallow-first | 15 / 41 | +| 25G | Q5_K / Q4_K | 1.00 | 5.353 | `[0..33]` plus `blk.40`, shallow-first | 35 / 41 | + +21G is the tier where Rule 5's condition fails, so its block set is the even +dither, byte-for-byte identical to the previous revision. That row is the method +declining to change something, which is the point of having a condition. + +**A.5 Validating a recipe before building.** Four checks, all cheap, all of which +have caught a real error at least once: + +1. **Rule count.** All four tiers have exactly 443 rules. +2. **Zero double-matched.** No tensor matched by two rules. +3. **Zero uncovered.** No intended tensor left to the base ftype by accident. +4. **Zero inert.** No rule that assigns a type the quantizer would have used + anyway, which would silently misrepresent what the recipe does. + +Plus one build-time check: sum the per-tensor sizes implied by the rules and +compare to the built file. A mismatch means a regex did not match what it was +meant to, and `llama-quantize` will not warn about that. + +--- + +## Appendix B: measurement protocol + +**B.1 The bf16 reference**, once per base model, reused for every tier: + +``` +llama-perplexity -m base-bf16.gguf -f wiki.test.raw \ + -c 2048 --chunks 64 --kl-divergence-base base.kld +``` + +CPU-bound on a 69 GB file, roughly 25 minutes on a 30-core machine. + +**B.2 A tier:** + +``` +llama-perplexity -m tier.gguf -f wiki.test.raw \ + -c 2048 --chunks 64 --kl-divergence-base base.kld --kl-divergence +``` + +**B.3 Constructing a byte-identical variant.** Count how many blocks receive the +high type in the control. Produce the variant by *permuting* which blocks those +are, never by changing the count. The files then match in size, usually to the +byte. If they do not, the variant is invalid and must not be compared. + +**B.4 Error bars.** 1σ is 0.0005 to 0.0007 at 64 chunks and 0.0008 to 0.0012 at +16. Differences below about 1.5% on a 0.03 mean KLD are not interpretable as +single measurements, only as confirmations of a rule established elsewhere, and +are labelled as such in Section 6.3. + +--- + +## Appendix C: index of evidence + +``` +appendix/original-ICE/PRINCIPLE.md the model-agnostic method, as first written +appendix/original-ICE/RECIPE.md its instantiation for this model, arithmetic shown +appendix/recipes/ every tensor-type-file cited +appendix/measurements/ raw llama-perplexity output for every KLD quoted +appendix/SHA256SUMS.txt checksum for every file above +``` + +`appendix/measurements/` also holds `sensitivity.jsonl`, the 600-row per-tensor +sensitivity map used to construct the attacks that Section 7.1 reports as +failures, and `ice_final_ledger.txt`, the original build log with its error bars. + +Recipes for the three dominated tiers are not included: they were dropped after +the Pareto analysis and no recipe was retained. Their measured numbers are in +Section 6.1. + +--- + +## Appendix D: the propagating set in full + +| tensor | blocks | params | class | status | +|---|---:|---:|---|---| +| `ffn_gate_inp`, `ffn_gate_inp_shexp` | 80 | 21,053,440 | DISCRETE | already F32, excluded by name upstream | +| `ssm_conv1d` | 30 | 983,040 | RECURRENT | already F32, first dim 4 | +| `ssm_a`, `ssm_dt`, all norms | 100+ | ~264,000 | RECURRENT / n/a | already F32, 1-D | +| `ssm_alpha` | 30 | 1,966,080 | RECURRENT | **pinned F32 by ICE** | +| `ssm_beta` | 30 | 1,966,080 | RECURRENT | **pinned F32 by ICE** | +| `attn_k` | 11 | 11,534,336 | CACHED | **pinned F16 by ICE** | +| `attn_v` | 11 | 11,534,336 | CACHED | **pinned F16 by ICE** | +| **whole propagating set** | | **~47.3 M (0.14%)** | | 0.15 GB | +| **of which ICE pins** | **82 tensors** | **27,000,832 (0.078%)** | | 61.86 MB, +33 MB over Q8_0 | + +**Excluded and why:** `attn_q` (167.8 M) and `attn_output` (83.9 M) are INSTANT +and held at the always-on Q8_0; see Section 7.4 for the arithmetic that killed +demoting them further. `attn_qkv` (503.3 M) fuses Q, K and V on the 30 SSM +blocks, so the class cannot be separated out there; those blocks carry no KV +cache, but it is a structural limit rather than a design choice (Section 8.5). + +--- + +*This report describes work done independently. The comparison ladders in +Section 6 are the published work of their respective authors and are measured +here, not reproduced or modified.*