File size: 16,402 Bytes
6f11713 5fce406 39f8d71 6f11713 2761ff1 6f11713 3172aad 6f11713 39f8d71 6f11713 e8c37dc c190030 541d5f8 e8c37dc 5809518 f152c96 e8c37dc 5fce406 6f11713 e8c37dc 3172aad 5fce406 6f11713 3172aad 6f11713 e8c37dc 6f11713 e8c37dc f152c96 5fce406 3172aad 2761ff1 5fce406 6f11713 3172aad | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 | # Pruning Evaluation Progress
Last updated: 2026-05-30 UTC.
## Scope
Evaluation cache: `artifacts/data/calibration/c0543d66aa2199f7/chunks.npy`
Baseline: BF16 `poolside/Laguna-XS.2`, full 4k cache, 1024 chunks.
Importance artifact: `artifacts/runs/heapr_paper_128x2048_4xl40s_alllayers_fused_device_b4/atomic_scores.npy`
## Decision Summary
The primary quality experiment preserves Laguna's static/shared expert and original top-8 routed-expert
router. It zeroes importance-ranked 64-atom blocks only inside routed expert tensors.
| Native grouped pruning | Full-cache loss | PPL delta | GSM8K-CoT strict EM | Interpretation |
| --- | ---: | ---: | ---: | --- |
| BF16 baseline | 2.347142 | 0.00% | 0.931818 | unpruned anchor |
| 10% | 2.363612 | +1.66% | 0.909091 | strongest deployment candidate |
| 20% | 2.416442 | +7.18% | 0.780303 | upper-bound tradeoff point |
| 25% | 2.466706 | +12.70% | 0.681818 | stress point; quality loss is substantial |
| 40% | 2.683520 | +39.99% | not run | loss indicates excessive degradation |
GSM8K-CoT values use the same first 10% subset (`132` examples), fixed eight-shot prompt, and greedy
decoding for every row.
| Native grouped pruning | Full-cache loss | Full MMLU 0-shot accuracy | MMLU-STEM 5-shot accuracy | GSM8K-CoT 8-shot strict EM |
| --- | ---: | ---: | ---: | ---: |
| BF16 baseline | 2.347142 | 0.733514 | 0.690771 | 0.931818 |
| 10% | 2.363612 | 0.735437 | 0.693942 | 0.909091 |
| 20% | 2.416442 | 0.734012 | 0.679036 | 0.780303 |
| 25% | 2.466706 | 0.725965 | 0.666350 | 0.681818 |
## Loss / Perplexity
| Run | Pruning interpretation | Mean loss | Delta loss | Perplexity | PPL delta | Artifact |
| --- | --- | ---: | ---: | ---: | ---: | --- |
| BF16 baseline | unpruned | 2.347443 | 0.000000 | 10.458797 | 0.00% | `artifacts/runs/baselines/bf16_4k_full_4xl40s_loss.json` |
| BF16 baseline, explicit static cache | unpruned | 2.347142 | 0.000000 | 10.455647 | 0.00% | `artifacts/runs/baselines/bf16_4k_full_4xl40s_static_cache_b4_loss.json` |
| Atomic 10% | lowest atomic scores zeroed globally | 2.362229 | +0.014786 | 10.614584 | +1.49% | `artifacts/runs/pruned_loss_full_4k_4xl40s/atomic_pruned_10pct_loss.json` |
| Atomic 20% | lowest atomic scores zeroed globally | 2.412423 | +0.064979 | 11.160969 | +6.71% | `artifacts/runs/pruned_loss_full_4k_4xl40s/atomic_pruned_20pct_loss.json` |
| Atomic 40% | lowest atomic scores zeroed globally | 2.705508 | +0.358065 | 14.961917 | +43.06% | `artifacts/runs/pruned_loss_full_4k_4xl40s/atomic_pruned_40pct_loss.json` |
| Group 10% diagnostic | importance-sorted 64-atom groups zeroed inside original parent routing | 2.362365 | +0.014922 | 10.616028 | +1.50% | `artifacts/runs/pruned_loss_full_4k_4xl40s/group_pruned_10pct_loss.json` |
| Repacked group 10% | independently routed 64-atom child groups, fixed top-64 backfill | 2.404768 | +0.057626 | 11.075862 | +5.93% | `artifacts/runs/pruned_loss_full_4k_4xl40s/repacked_group_pruned_10pct_loss.json` |
| Repacked group 20% | independently routed 64-atom child groups, fixed top-64 backfill | 2.517318 | +0.170176 | 12.395307 | +18.55% | `artifacts/runs/pruned_loss_full_4k_4xl40s/repacked_group_pruned_20pct_loss.json` |
| Repacked group 40% | independently routed 64-atom child groups, fixed top-64 backfill | 2.928546 | +0.581404 | 18.700425 | +78.85% | `artifacts/runs/pruned_loss_full_4k_4xl40s/repacked_group_pruned_40pct_loss.json` |
| Native grouped 10% | original top-8 parent routing; pruned 64-atom groups zeroed in place | 2.363612 | +0.016469 | 10.629271 | +1.66% | `artifacts/runs/native_group_loss_full_4k_4xl40s/group_pruned_10pct_loss.json` |
| Random native grouped 10% | random seed 20260530; original top-8 parent routing; same minimum-retention rule | 2.480595 | +0.133453 | 11.948376 | +14.28% | `artifacts/runs/random_group_10pct/group_pruned_10pct_loss.json` |
| Native grouped 20% | original top-8 parent routing; pruned 64-atom groups zeroed in place | 2.416442 | +0.069300 | 11.205921 | +7.18% | `artifacts/runs/native_group_loss_full_4k_4xl40s/group_pruned_20pct_loss.json` |
| Native grouped 25% | original top-8 parent routing; pruned 64-atom groups zeroed in place | 2.466706 | +0.119564 | 11.783569 | +12.70% | `artifacts/runs/native_group_loss_full_4k_4xl40s/group_pruned_25pct_loss.json` |
| Random native grouped 25% | random seed 20260530; original top-8 parent routing; same minimum-retention rule | 2.561328 | +0.214186 | 12.953011 | +23.89% | `artifacts/runs/random_group_25pct/group_pruned_25pct_loss.json` |
| Native grouped 40% | original top-8 parent routing; pruned 64-atom groups zeroed in place | 2.683520 | +0.336378 | 14.636528 | +39.99% | `artifacts/runs/native_group_loss_full_4k_4xl40s/group_pruned_40pct_loss.json` |
## Expanded Grouped Router Validation
| Run | Chunks | Mean loss | Perplexity | Artifact |
| --- | ---: | ---: | ---: | --- |
| BF16 baseline, first full-cache 16 chunks | 16 | 2.475593 | 11.888757 | `artifacts/runs/baselines/bf16_4k_full_first16_4xl40s_loss.json` |
| Repacked grouped, 0% pruned, first full-cache 16 chunks | 16 | 2.477158 | 11.907380 | `artifacts/runs/pruned_loss_full_4k_4xl40s/repacked_group_pruned_00pct_loss_pilot16.json` |
| Repacked grouped, 10% pruned, first full-cache 16 chunks | 16 | 2.515994 | 12.378907 | `artifacts/runs/pruned_loss_full_4k_4xl40s/repacked_group_pruned_10pct_loss_pilot16.json` |
## Native Grouped Pruning Pilot
The primary grouped experiment now preserves Laguna's original top-8 parent-expert router and native
forward. Pruned 64-atom groups are zeroed inside each original expert tensor. This measures grouped
structural-pruning quality soundly, although the current native kernel does not realize the potential
deployment speedup.
Per-layer 25% mask statistics: `artifacts/reports/native_group_pruning_stats.md`
| Run | Chunks | Mean loss | Perplexity | Artifact |
| --- | ---: | ---: | ---: | --- |
| Native grouped 10% | 16 | 2.482592 | 11.972254 | `artifacts/runs/native_group_loss_pilot16/group_pruned_10pct_loss.json` |
| Native grouped 20% | 16 | 2.527117 | 12.517363 | `artifacts/runs/native_group_loss_pilot16/group_pruned_20pct_loss.json` |
| Native grouped 40% | 16 | 2.765421 | 15.885724 | `artifacts/runs/native_group_loss_pilot16/group_pruned_40pct_loss.json` |
## Downstream Evaluation
| Run | Scope | Result | Artifact |
| --- | --- | --- | --- |
| Full MMLU 0-shot baseline smoke | 0.1% limit; 58 subject samples | harness completed; accuracy 0.879310 | `artifacts/runs/downstream/mmlu_baseline_smoke.json` |
| Full MMLU 0-shot baseline | full 14,042 samples | accuracy 0.733514; STEM slice 0.702188 | `artifacts/runs/downstream/mmlu_baseline_full.json` |
| Full MMLU 0-shot atomic 10% | full 14,042 samples | accuracy 0.737715; delta +0.004202 | `artifacts/runs/downstream/mmlu_atomic_10pct_full.json` |
| Full MMLU 0-shot atomic 20% | full 14,042 samples | accuracy 0.734511; delta +0.000997 | `artifacts/runs/downstream/mmlu_atomic_20pct_full.json` |
| Full MMLU 0-shot atomic 40% | full 14,042 samples | accuracy 0.681028; delta -0.052485 | `artifacts/runs/downstream/mmlu_atomic_40pct_full.json` |
| Full MMLU 0-shot native grouped 10% | full 14,042 samples | accuracy 0.735437; delta +0.001923 | `artifacts/runs/downstream/mmlu_native_group_10pct_full.json` |
| Full MMLU 0-shot native grouped 20% | full 14,042 samples | accuracy 0.734012; delta +0.000499 | `artifacts/runs/downstream/mmlu_native_group_20pct_full.json` |
| Full MMLU 0-shot native grouped 25% | full 14,042 samples | accuracy 0.725965; delta -0.007549; STEM slice 0.688233 | `artifacts/runs/downstream/mmlu_native_group_25pct_full.json` |
| MMLU-STEM 5-shot baseline | report-metric-aligned local lm-eval; 3,153 samples | accuracy 0.690771 | `artifacts/runs/downstream/mmlu_stem_5shot_baseline_full.json` |
| MMLU-STEM 5-shot native grouped 10% | report-metric-aligned local lm-eval; 3,153 samples | accuracy 0.693942; delta +0.003172 | `artifacts/runs/downstream/mmlu_stem_5shot_native_group_10pct_full.json` |
| MMLU-STEM 5-shot native grouped 20% | report-metric-aligned local lm-eval; 3,153 samples | accuracy 0.679036; delta -0.011735 | `artifacts/runs/downstream/mmlu_stem_5shot_native_group_20pct_full.json` |
| MMLU-STEM 5-shot native grouped 25% | report-metric-aligned local lm-eval; 3,153 samples | accuracy 0.666350; delta -0.024421 | `artifacts/runs/downstream/mmlu_stem_5shot_native_group_25pct_full.json` |
| GSM8K-CoT 8-shot baseline | first 10%; 132 paired samples | strict EM 0.931818; flexible EM 0.886364 | `artifacts/runs/downstream/gsm8k_cot_baseline_10pct.json` |
| GSM8K-CoT 8-shot native grouped 10% | first 10%; 132 paired samples | strict EM 0.909091; delta -0.022727; flexible EM 0.863636; delta -0.022727 | `artifacts/runs/downstream/gsm8k_cot_native_group_10pct_10pct.json` |
| GSM8K-CoT 8-shot native grouped 20% | first 10%; 132 paired samples | strict EM 0.780303; delta -0.151515; flexible EM 0.742424; delta -0.143939 | `artifacts/runs/downstream/gsm8k_cot_native_group_20pct_10pct.json` |
| GSM8K-CoT 8-shot native grouped 25% | first 10%; 132 paired samples | strict EM 0.681818; delta -0.250000; flexible EM 0.689394; delta -0.196970 | `artifacts/runs/downstream/gsm8k_cot_native_group_25pct_10pct.json` |
| CRUXEval-O CoT baseline smoke | one function; 10 sampled generations; executable pass@1 | harness completed; pass@1 0.0 | `artifacts/runs/downstream/cruxeval_output_cot_baseline_smoke.json` |
| SWE-bench Lite baseline smoke | one oracle-prepared instance: `astropy__astropy-12907` | official grader completed; generated patch failed to apply | `artifacts/runs/downstream/swebench_baseline_smoke_predictions.jsonl` |
The SWE-bench smoke validates generation, prediction serialization, Docker availability, and official harness invocation. It is not a quality estimate: the single generated continuation produced a malformed patch and the grader recorded one patch-apply error. Further SWE-bench work is paused while the basic native-grouped benchmarks run.
Additional small benchmark selected from the Laguna technical-report table: `gsm8k_cot`. The installed
`lm-eval` task is configured for the report-aligned 8-shot chain-of-thought exact-match setup. Run it
for the baseline and native-grouped 25% variant after MMLU. Because a full run takes more than one hour
per model, use the same first 10% subset for both variants. Runs explicitly cap generation at 256 new
tokens because Laguna's shipped generation config otherwise requests up to 4096.
The report-metric-aligned local MMLU-STEM 5-shot baseline is `0.690771`, below the Laguna technical
report's `0.781`. The local `lm-eval` comparison remains useful because the unpruned and pruned models
run through the same harness, but it does not exactly reproduce the report's evaluation stack.
The GSM8K-CoT comparison is likewise a matched local-harness comparison rather than an exact
reproduction of Poolside's internal framework. The local task uses eight fixed chain-of-thought
demonstrations and greedy decoding (`do_sample=False`). An audit of the paired 10% run confirmed
identical prompts and test examples. The 256-token cap did not bind: generated responses were short,
and the grouped-model errors were predominantly incorrect arithmetic answers rather than truncations.
CRUXEval-O CoT executable grading was smoke-tested successfully. The paired coding subset was stopped
after `84 / 400` baseline generations when the remaining session was time-boxed. The smoke validates
the local generation and executable-grader path but is not a coding-quality estimate.
## Grouped Compute Note
For grouped pruning, the simple deployment interpretation is the pruned group fraction. The prioritized
25% experiment removes exactly 19,968 of 79,872 64-wide groups. That is the meaningful reduction if a
grouped MoE kernel only materializes retained child experts.
The primary native grouped evaluator preserves the original top-8 parent router and zeroes pruned
groups in place. Its current kernel still computes those zeroed blocks, so observed runtime is a
quality-measurement harness rather than a realized speedup. The separately reported repacked-child
exploration uses fixed top-64 backfill and is not the primary result.
## Method Notes
- Atomic pruning was evaluated in memory by loading the BF16 model, zeroing pruned atom rows/columns in the packed expert tensors, and running the native Laguna forward on the full 4k cache.
- Current fixed-shape loss runs pass an explicit Hugging Face `StaticCache` into each direct forward call with `use_cache=True`, avoiding Laguna's internal dynamic-cache construction path.
- Atomic and diagnostic-group deltas above use the earlier BF16 baseline. Repacked-group deltas use the explicit-static-cache BF16 baseline, matching those runs' cache path.
- The grouped 10% diagnostic is not the full grouped mini-expert routing experiment. It keeps the original parent-expert router/top-8 behavior and only zeroes 64-wide groups inside selected parent experts.
- That original-router grouped path is now the primary experimental quality measurement: it directly applies 64-atom structural pruning while preserving the source model's routing semantics.
- For grouped mini-experts as independently routable child experts, the evaluator needs to route over child groups. A plausible inherited-router test is: expand each parent expert score to its 8 child groups, mask pruned child groups, select top 64 child groups per token, and evaluate those group contributions. Under no pruning this collapses to the original top-8 parents; under group pruning it can select child groups from more than 8 parent experts.
- The repacked grouped runtime now implements that inherited-router test by repacking each sparse layer from 256 parent experts x 512 atoms into 2048 child experts x 64 atoms, expanding router rows to child groups, selecting top 64 child groups, and assigning each selected group its parent expert's normalized router weight. The 0% pruned first-16 validation above checks that this preserves the unpruned model.
## Current Status
- Completed: BF16 full-cache baseline.
- Completed: atomic 10%, 20%, and 40% full-cache loss.
- Completed but diagnostic-only: constrained grouped 10% full-cache loss.
- Stopped: constrained grouped 20%/40%, because it does not answer the independently-routable grouped mini-expert question.
- Completed: repacked grouped 10% full-cache loss with explicit static cache.
- Completed: repacked grouped 20% full-cache loss with explicit static cache.
- Completed: repacked grouped 40% full-cache loss with explicit static cache.
- Completed: unpruned full-cache loss with explicit static cache and directly comparable grouped deltas.
- Completed: baseline MMLU and SWE-bench Lite smoke paths.
- Completed: full baseline and atomic 10%/20%/40% MMLU comparison runs.
- Stopped: full repacked-group MMLU. Repacked child routing over-selects groups for the primary experiment and remains exploratory only.
- Completed: native original-router grouped 10%/20%/40% loss pilot on 16 chunks.
- Completed: native original-router grouped 10%/20%/40% full-cache loss.
- Priority changed by request: evaluate baseline versus native original-router grouped 25% only.
- Completed: native original-router grouped 25% full-cache loss.
- Completed: full-MMLU 0-shot native grouped 25%. This is useful auxiliary evidence but does not match the tech-report MMLU-STEM 5-shot row.
- Completed: report-metric-aligned local MMLU-STEM 5-shot baseline and native grouped 25%.
- Stopped: full GSM8K-CoT baseline after runtime projection exceeded one hour.
- Completed: paired first-10% GSM8K-CoT baseline and native grouped 25%.
- Completed: paired first-10% GSM8K-CoT native grouped 10% and 20% curve points.
- Stopped by request: CRUXEval-O CoT paired subset after executable-grader smoke validation, due to the remaining time box.
- Completed: random native grouped 25% full-cache loss control with seed 20260530. It is +0.094622 loss worse than importance-ranked 25% pruning.
- Completed: random native grouped 10% full-cache loss control with seed 20260530. It is +0.116984 loss worse than importance-ranked 10% pruning.
- Skipped from the grouped MMLU downstream matrix: native grouped 10%, 20%, and 40%.
- Paused by request: further SWE-bench evaluation.
- Validated: full test suite `28 passed`; Python compilation and `git diff --check` passed.
|