MoritzMo123/kfac-svd_uniform_llama-3-8b_0.6
Text Generation • 5B • Updated • 159
LEMS & KFAC-SVD Compression
This collection hosts the compressed models evaluated in our paper: "Layer-wise Error Modeling Search (LEMS) and KFAC-SVD".
Note Llama-3-8B compressed with KFAC-SVD and UNIFORM rank allocation (60% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_llama-3-8b_0.7
Text Generation • 6B • Updated • 185Note Llama-3-8B compressed with KFAC-SVD and UNIFORM rank allocation (70% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_llama-3-8b_0.8
Text Generation • 7B • Updated • 147Note Llama-3-8B compressed with KFAC-SVD and UNIFORM rank allocation (80% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_llama-3-8b_0.9
Text Generation • 7B • Updated • 168Note Llama-3-8B compressed with KFAC-SVD and UNIFORM rank allocation (90% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_llama-3-8b_0.6
Text Generation • 5B • Updated • 180Note Llama-3-8B compressed with KFAC-SVD and LEMS rank allocation (60% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_llama-3-8b_0.7
Text Generation • 6B • Updated • 150Note Llama-3-8B compressed with KFAC-SVD and LEMS rank allocation (70% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_llama-3-8b_0.8
Text Generation • 7B • Updated • 186Note Llama-3-8B compressed with KFAC-SVD and LEMS rank allocation (80% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_llama-3-8b_0.9
Text Generation • 7B • Updated • 177Note Llama-3-8B compressed with KFAC-SVD and LEMS rank allocation (90% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_Qwen3-8B_0.6
Text Generation • 5B • Updated • 158Note Qwen3-8B compressed with KFAC-SVD and UNIFORM rank allocation (60% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_Qwen3-8B_0.7
Text Generation • 6B • Updated • 153Note Qwen3-8B compressed with KFAC-SVD and UNIFORM rank allocation (70% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_Qwen3-8B_0.8
Text Generation • 7B • Updated • 161Note Qwen3-8B compressed with KFAC-SVD and UNIFORM rank allocation (80% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_Qwen3-8B_0.9
Text Generation • 7B • Updated • 174Note Qwen3-8B compressed with KFAC-SVD and UNIFORM rank allocation (90% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_Qwen3-8B_0.6
Text Generation • 5B • Updated • 230 • 1Note Qwen3-8B compressed with KFAC-SVD and LEMS rank allocation (60% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_Qwen3-8B_0.7
Text Generation • 6B • Updated • 184Note Qwen3-8B compressed with KFAC-SVD and LEMS rank allocation (70% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_Qwen3-8B_0.8
Text Generation • 7B • Updated • 203Note Qwen3-8B compressed with KFAC-SVD and LEMS rank allocation (80% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_Qwen3-8B_0.9
Text Generation • 7B • Updated • 180Note Qwen3-8B compressed with KFAC-SVD and LEMS rank allocation (90% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_mistral-7b_0.6
Text Generation • 4B • Updated • 175Note Mistral-7B compressed with KFAC-SVD and UNIFORM rank allocation (60% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_mistral-7b_0.7
Text Generation • 5B • Updated • 153Note Mistral-7B compressed with KFAC-SVD and UNIFORM rank allocation (70% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_mistral-7b_0.8
Text Generation • 6B • Updated • 170Note Mistral-7B compressed with KFAC-SVD and UNIFORM rank allocation (80% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_mistral-7b_0.9
Text Generation • 7B • Updated • 177Note Mistral-7B compressed with KFAC-SVD and UNIFORM rank allocation (90% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_mistral-7b_0.6
Text Generation • 4B • Updated • 177Note Mistral-7B compressed with KFAC-SVD and LEMS rank allocation (60% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_mistral-7b_0.7
Text Generation • 5B • Updated • 184Note Mistral-7B compressed with KFAC-SVD and LEMS rank allocation (70% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_mistral-7b_0.8
Text Generation • 6B • Updated • 198Note Mistral-7B compressed with KFAC-SVD and LEMS rank allocation (80% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_mistral-7b_0.9
Text Generation • 7B • Updated • 177Note Mistral-7B compressed with KFAC-SVD and LEMS rank allocation (90% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_llama-2-7b_0.6
Text Generation • 4B • Updated • 30 • 1Note Llama-2-7B compressed with KFAC-SVD and UNIFORM rank allocation (60% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_uniform_llama-2-7b_0.8
Text Generation • 5B • Updated • 27Note Llama-2-7B compressed with KFAC-SVD and UNIFORM rank allocation (80% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_llama-2-7b_0.6
Text Generation • 4B • Updated • 30 • 1Note Llama-2-7B compressed with KFAC-SVD and LEMS rank allocation (60% of total linear parameters remaining), without fine-tuning.
MoritzMo123/kfac-svd_lems_llama-2-7b_0.8
Text Generation • 5B • Updated • 34Note Llama-2-7B compressed with KFAC-SVD and LEMS rank allocation (80% of total linear parameters remaining), without fine-tuning.
-
Advancing SVD-based LLM Compression via Layer-Wise Error Model Search
Paper • 2610.32878 • Published • 1