Do pruning and quantization compose? Compressed model states (ESANN 2027 study, v2)
This repository holds 316 compressed model states of 6 base models, every state evaluated in the v2 study of the order of unstructured pruning and weight quantization under Hessian-based (OBS) compensation: pruning alone, quantization alone, and the two combined in three orders (prune then quantize, quantize then prune, and both in one joint sweep), plus the ablation arms that test why quantizing first hurts.
- Code, result lines and per-item benchmark logs: https://github.com/martin-cousseau/esann2027-compose (branch
claude/busy-hopper-ghtynk) - One Weights & Biases run per state (metrics, layer diagnostics, links back here): https://wandb.ai/martin-css/esann2027-compose-v2
- Paper: in preparation for ESANN 2027; citation below once available.
Each file stores the compressed linear layers of the decoder blocks in an exact compact form (integer codes, per-group scale and zero point, sparsity bitmap). Decoding gives the evaluated fp32 weights bit for bit; embeddings, norms and the LM head are the unchanged base model.
Load a state
pip install torch transformers safetensors huggingface_hub numpy
python load_esann2027.py models/SmolLM2-135M/SmolLM2-135M__p_then_q__s0.5000__b4__seed0__float32.safetensors
from huggingface_hub import hf_hub_download
import load_esann2027 as L # this repository's loader
path = hf_hub_download("martincousseau/esann2027-compose-v2", "models/SmolLM2-135M/SmolLM2-135M__p_then_q__s0.5000__b4__seed0__float32.safetensors")
model, meta = L.load(path) # transformers model in fp32, compressed weights in place
File names
models/<model>/<model>__<order>__s<sparsity>__b<bits>__seed<seed>__float32.safetensors, with orders dense, prune (pruning only, b16), quant (quantization only, s0.0000), joint, p_then_q, q_then_p and the ablation arms q_then_p_refit (grid re-fitted after pruning), q_then_p_pmask (the mask pruning alone would choose), q_then_p_pmask_refit (both) and p_then_q_stale (P->Q with the grid of the unpruned weights). seed is the calibration-sample seed.
Format esann2027-v2-compact-1
JSON metadata under the safetensors metadata key esann2027 (state, commit, code hash, group size, and one entry per layer: key <block>.<module>, shape, kind). Per kind:
| kind | tensors | decoding |
|---|---|---|
quant |
<key>.codes uint8 (two 4-bit codes per byte, low nibble first, when bits <= 4), <key>.scale fp32 and <key>.zero uint8 per (row, group of 128 columns), optional <key>.mask bitmap |
W = scale * (code - zero), then W[mask] = 0 |
sparse_fp32 |
<key>.mask bitmap, <key>.values fp32 kept weights (row-major) |
zeros, kept values in place |
fp32 |
<key>.weight |
as stored |
Bitmaps use numpy.packbits order. The bitmap is stored to tell pruned weights from weights that round to zero; the bits-per-weight columns below count it as 1 bit per weight whenever the state is pruned.
How the states were made
All states: fp32 model and Hessians on one GPU type (NVIDIA L4), TF32 off, deterministic kernels, the OBS sweep of the paper's code (src/core.py), 128 calibration windows of 2048 tokens from C4, asymmetric min-max quantization grid per group of 128 weights. Evaluation: WikiText-2 test perplexity, C4 validation perplexity (256 windows), KL divergence and top-1 agreement with the dense model, and (E1, E4) nine zero-shot tasks with lm-evaluation-harness 0.4.13 on full splits with per-item logs. Every result line records the GPU, driver, Colab image, library versions, commit and code hash; see the GitHub repository.
States by experiment family: E1 (the paper's grid in fp32): 72, E3 (Q->P mechanism arms): 40, E4 (extra frontier points): 15, E5 (calibration seeds 1 and 2): 120, E6 (the grid on the larger models): 59, E6B (larger-model states re-run with benchmarks): 10.
States
bits/w = bits per weight of the compressed linear layers with a 1-bit mask counted for pruned states (the per-group scale and zero point add (16 + b)/128 bits for quantized states and are not included); PPL = WikiText-2 test perplexity; KL = mean KL divergence to the dense model on WikiText-2. Pruning-only states are counted at 16 bits per kept weight (the base models' precision; they are stored here in fp32). The dense states are the base models themselves and have no file here.
SmolLM2-135M (85 states; base model HuggingFaceTB/SmolLM2-135M)
SmolLM2-360M (85 states; base model HuggingFaceTB/SmolLM2-360M)
Qwen2.5-0.5B (77 states; base model Qwen/Qwen2.5-0.5B)
Qwen2.5-1.5B (23 states; base model Qwen/Qwen2.5-1.5B)
SmolLM2-1.7B (23 states; base model HuggingFaceTB/SmolLM2-1.7B)
OLMo-2-0425-1B (23 states; base model allenai/OLMo-2-0425-1B)
License
Derived from the base models, all released under Apache-2.0; the compressed states keep that license.
Citation
A citation will be added when the ESANN 2027 paper is available.
Model tree for martincousseau/esann2027-compose-v2
Base model
HuggingFaceTB/SmolLM2-1.7B