OverRep

Recovery checkpoints for Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured LLM Pruning (EMNLP 2026). Paper · Code

Built with Llama.

Each folder holds the two recovery blocks of one pruned model, already merged into plain Hugging Face decoder-layer weights. For a mask T:I, layers T+1 … T+I of the base model are removed and the blocks at T and T+I+1 are replaced by the weights in layer_T.safetensors. The result is a standard pruned model with no extra parameters.

Backbone	Base model	Pruning	Folder	Reasoning RP (%)	Generation RP (%)
LLaMA2-7B	meta-llama/Llama-2-7b-hf	25%	llama-2-7b/25pct	92.6	61.9
LLaMA2-7B	meta-llama/Llama-2-7b-hf	50%	llama-2-7b/50pct	76.9	29.4
LLaMA2-13B	meta-llama/Llama-2-13b-hf	25%	llama-2-13b/25pct	95.3	67.2
LLaMA2-13B	meta-llama/Llama-2-13b-hf	50%	llama-2-13b/50pct	85.5	42.1
LLaMA3.2-3B	meta-llama/Llama-3.2-3B	25%	llama-3.2-3b/25pct	90.2	47.1
LLaMA3.2-3B	meta-llama/Llama-3.2-3B	50%	llama-3.2-3b/50pct	72.3	10.7
LLaMA3.2-3B (OverRep-KD)	meta-llama/Llama-3.2-3B	25%	llama-3.2-3b/25pct-kd	91.8	61.0
LLaMA3.2-3B (OverRep-KD)	meta-llama/Llama-3.2-3B	50%	llama-3.2-3b/50pct-kd	73.4	26.7
LLaMA3.1-8B	meta-llama/Llama-3.1-8B	25%	llama-3.1-8b/25pct	91.3	50.0
LLaMA3.1-8B	meta-llama/Llama-3.1-8B	50%	llama-3.1-8b/50pct	69.5	14.7

The last two columns are retained performance (RP), not accuracy: the pruned model's average score divided by the dense base model's average score, × 100 (100 = no loss from pruning). Reasoning RP averages ten 0-shot tasks (ARC-Easy, ARC-Challenge, BoolQ, HellaSwag, MathQA, MMLU, OpenBookQA, PIQA, RACE, WinoGrande); Generation RP averages CoQA (F1, 0-shot), GSM8K (EM, 8-shot) and TriviaQA (EM, 5-shot). Values are those reported in the paper.

Every folder contains layer_T.safetensors and overrep_config.json (base_model, include_layer).

Usage

Install the code from GitHub, download one folder, and evaluate it with the mask T:I from its overrep_config.json (include_layer):

hf download SoongE/OverRep --include "llama-3.2-3b/25pct/*" --local-dir checkpoints

overrep-eval deploy_model hellaswag --custom \
  --tokenizer-name meta-llama/Llama-3.2-3B --include-layer 17:9 \
  --checkpoint checkpoints/llama-3.2-3b/25pct

The base models are gated on Hugging Face; accept their licenses before downloading them.

License

The checkpoints are derivatives of the Llama base models and are distributed under their licenses: llama-2-* under the Llama 2 Community License, llama-3.1-8b under the Llama 3.1 Community License, and llama-3.2-3b under the Llama 3.2 Community License. See LICENSE.md and NOTICE.

Citation

@inproceedings{oh2026overrep,
  title     = {Train Overcomplete, Deploy Compact: Scaling Recovery Capacity for Structured {LLM} Pruning},
  author    = {Oh, Seungmin and Lee, Donggeon and Ryu, Jongbin},
  booktitle = {Proceedings of the Conference on Empirical Methods in Natural Language Processing},
  year      = {2026},
  publisher = {Association for Computational Linguistics},
  url       = {https://arxiv.org/abs/2609.06974}
}