Advancing SVD-based LLM Compression via Layer-Wise Error Model Search
Paper • 2610.32878 • Published • 5
LEMS & KFAC-SVD Compression
This collection hosts the compressed models evaluated in our paper: "Layer-wise Error Modeling Search (LEMS) and KFAC-SVD".
-
MoritzMo123/kfac-svd_uniform_llama-3-8b_0.6
Text Generation • 5B • Updated • 179Note Llama-3-8B compressed with KFAC-SVD and UNIFORM rank allocation (60% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_llama-3-8b_0.7
Text Generation • 6B • Updated • 209Note Llama-3-8B compressed with KFAC-SVD and UNIFORM rank allocation (70% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_llama-3-8b_0.8
Text Generation • 7B • Updated • 168Note Llama-3-8B compressed with KFAC-SVD and UNIFORM rank allocation (80% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_llama-3-8b_0.9
Text Generation • 7B • Updated • 192Note Llama-3-8B compressed with KFAC-SVD and UNIFORM rank allocation (90% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_llama-3-8b_0.6
Text Generation • 5B • Updated • 203Note Llama-3-8B compressed with KFAC-SVD and LEMS rank allocation (60% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_llama-3-8b_0.7
Text Generation • 6B • Updated • 182Note Llama-3-8B compressed with KFAC-SVD and LEMS rank allocation (70% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_llama-3-8b_0.8
Text Generation • 7B • Updated • 218Note Llama-3-8B compressed with KFAC-SVD and LEMS rank allocation (80% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_llama-3-8b_0.9
Text Generation • 7B • Updated • 203Note Llama-3-8B compressed with KFAC-SVD and LEMS rank allocation (90% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_Qwen3-8B_0.6
Text Generation • 5B • Updated • 183Note Qwen3-8B compressed with KFAC-SVD and UNIFORM rank allocation (60% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_Qwen3-8B_0.7
Text Generation • 6B • Updated • 172Note Qwen3-8B compressed with KFAC-SVD and UNIFORM rank allocation (70% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_Qwen3-8B_0.8
Text Generation • 7B • Updated • 185Note Qwen3-8B compressed with KFAC-SVD and UNIFORM rank allocation (80% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_Qwen3-8B_0.9
Text Generation • 7B • Updated • 199Note Qwen3-8B compressed with KFAC-SVD and UNIFORM rank allocation (90% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_Qwen3-8B_0.6
Text Generation • 5B • Updated • 231 • 1Note Qwen3-8B compressed with KFAC-SVD and LEMS rank allocation (60% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_Qwen3-8B_0.7
Text Generation • 6B • Updated • 211Note Qwen3-8B compressed with KFAC-SVD and LEMS rank allocation (70% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_Qwen3-8B_0.8
Text Generation • 7B • Updated • 250Note Qwen3-8B compressed with KFAC-SVD and LEMS rank allocation (80% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_Qwen3-8B_0.9
Text Generation • 7B • Updated • 214Note Qwen3-8B compressed with KFAC-SVD and LEMS rank allocation (90% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_mistral-7b_0.6
Text Generation • 4B • Updated • 199Note Mistral-7B compressed with KFAC-SVD and UNIFORM rank allocation (60% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_mistral-7b_0.7
Text Generation • 5B • Updated • 173Note Mistral-7B compressed with KFAC-SVD and UNIFORM rank allocation (70% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_mistral-7b_0.8
Text Generation • 6B • Updated • 191Note Mistral-7B compressed with KFAC-SVD and UNIFORM rank allocation (80% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_mistral-7b_0.9
Text Generation • 7B • Updated • 204Note Mistral-7B compressed with KFAC-SVD and UNIFORM rank allocation (90% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_mistral-7b_0.6
Text Generation • 4B • Updated • 203Note Mistral-7B compressed with KFAC-SVD and LEMS rank allocation (60% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_mistral-7b_0.7
Text Generation • 5B • Updated • 204Note Mistral-7B compressed with KFAC-SVD and LEMS rank allocation (70% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_mistral-7b_0.8
Text Generation • 6B • Updated • 228Note Mistral-7B compressed with KFAC-SVD and LEMS rank allocation (80% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_mistral-7b_0.9
Text Generation • 7B • Updated • 206Note Mistral-7B compressed with KFAC-SVD and LEMS rank allocation (90% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_llama-2-7b_0.6
Text Generation • 4B • Updated • 26 • 1Note Llama-2-7B compressed with KFAC-SVD and UNIFORM rank allocation (60% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_llama-2-7b_0.8
Text Generation • 5B • Updated • 23Note Llama-2-7B compressed with KFAC-SVD and UNIFORM rank allocation (80% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_llama-2-7b_0.6
Text Generation • 4B • Updated • 26 • 1Note Llama-2-7B compressed with KFAC-SVD and LEMS rank allocation (60% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_llama-2-7b_0.8
Text Generation • 5B • Updated • 29Note Llama-2-7B compressed with KFAC-SVD and LEMS rank allocation (80% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_llama-7b-hf_0.6
Text Generation • 4B • Updated • 10Note Llama-7B compressed with KFAC-SVD and uniform rank allocation (60% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_llama-7b-hf_0.8
Text Generation • 5B • Updated • 11Note Llama-7B compressed with KFAC-SVD and uniform rank allocation (80% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_llama-7b-hf_0.6
Text Generation • 4B • Updated • 12Note Llama-7B compressed with KFAC-SVD and LEMS rank allocation (60% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_llama-7b-hf_0.8
Text Generation • 5B • Updated • 10Note Llama-7B compressed with KFAC-SVD and LEMS rank allocation (80% of total linear parameters remaining), without fine-tuning.