convaiinnovations/laya-multilingual
Text Classification • 0.3B • Updated • 275
Ian Andolina
iandol
AI & ML interests
None yet
Recent Activity
liked a model 4 days ago
convaiinnovations/laya-multilingual liked a model about 2 months ago
deepseek-ai/DeepSeek-V4-Flash-0731 liked a dataset 10 months ago
openai/gdpvalOrganizations
liked a
model 4 days ago
liked a
model about 2 months ago
liked a
dataset 10 months ago
upvoted a paper about 1 year ago
reacted to bartowski's post with 👍 about 2 years ago
Post
10154
So turns out I've been spreading a bit of misinformation when it comes to imatrix in llama.cpp
It starts true; imatrix runs the model against a corpus of text and tracks the activation of weights to determine which are most important
However what the quantization then does with that information is where I was wrong.
I think I made the accidental connection between imatrix and exllamav2's measuring, where ExLlamaV2 decides how many bits to assign to which weight depending on the goal BPW
Instead, what llama.cpp with imatrix does is it attempts to select a scale for a quantization block that most accurately returns the important weights to their original values, ie minimizing the dequantization error based on the importance of activations
The mildly surprising part is that it actually just does a relatively brute force search, it picks a bunch of scales and tries each and sees which one results in the minimum error for weights deemed important in the group
But yeah, turns out, the quantization scheme is always the same, it's just that the scaling has a bit more logic to it when you use imatrix
Huge shoutout to @compilade for helping me wrap my head around it - feel free to add/correct as well if I've messed something up
It starts true; imatrix runs the model against a corpus of text and tracks the activation of weights to determine which are most important
However what the quantization then does with that information is where I was wrong.
I think I made the accidental connection between imatrix and exllamav2's measuring, where ExLlamaV2 decides how many bits to assign to which weight depending on the goal BPW
Instead, what llama.cpp with imatrix does is it attempts to select a scale for a quantization block that most accurately returns the important weights to their original values, ie minimizing the dequantization error based on the importance of activations
The mildly surprising part is that it actually just does a relatively brute force search, it picks a bunch of scales and tries each and sees which one results in the minimum error for weights deemed important in the group
But yeah, turns out, the quantization scheme is always the same, it's just that the scaling has a bit more logic to it when you use imatrix
Huge shoutout to @compilade for helping me wrap my head around it - feel free to add/correct as well if I've messed something up
- 5 replies
liked 2
models about 2 years ago
liked a
model over 2 years ago
liked a
Space over 2 years ago
liked 2
models over 2 years ago
upvoted a collection over 2 years ago
liked 8
models over 2 years ago
parler-tts/parler_tts_mini_v0.1
Text-to-Speech • 0.6B • Updated • 2.31k • 358
jetmoe/jetmoe-8b
Text Generation • 9B • Updated • 12.6k • 253
ai21labs/Jamba-v0.1
Text Generation • 52B • Updated • 8.54k • 1.19k
bartowski/stable-code-instruct-3b-GGUF
Text Generation • 3B • Updated • 730 • 22
ezelikman/quietstar-8-ahead
Text Generation • 7B • Updated • 92 • 90
Nexusflow/Starling-LM-7B-beta
Text Generation • 7B • Updated • 1.68k • • 343
bartowski/Mistral-Evolved-11b-v0.1-GGUF
Text Generation • 11B • Updated • 1.9k • 7
NousResearch/Hermes-2-Pro-Mistral-7B-GGUF
7B • Updated • 42.5k • 261