| --- |
| title: README |
| emoji: 🏛️ |
| colorFrom: gray |
| colorTo: yellow |
| sdk: static |
| pinned: false |
| --- |
| |
| <p align="center"> |
| <img src="image-1789184498432.png" alt="ARCHSLOTH — the summit of quantization" width="620"> |
| </p> |
|
|
| <p align="center"> |
| <b>Most quantized weights ship with an adjective.<br>Ours ship with a table.</b> |
| </p> |
|
|
| --- |
|
|
| ## What we publish |
|
|
| Every ARCHSLOTH build arrives with its own evaluation attached: the sample count, the harness, the exact file it was measured against, and the failures it produced. Not a claim that it is accurate — the numbers, so you can check. |
|
|
| **If we cannot show the numbers, we do not publish the file.** |
|
|
| --- |
|
|
| ## Three rules we hold ourselves to |
|
|
| ### 1 · Q4 is the floor |
|
|
| We do not ship below 4-bit. Sub-4-bit builds make a model small, and they also make it unreliable in ways perplexity will not show you. If a model does not fit your memory at Q4, that is a hardware answer, not a quantization answer — and we would rather tell you that than hand you a file that loads and then quietly fails. |
|
|
| ### 2 · Calibration is not language-neutral |
|
|
| An importance matrix built on English text protects English. |
|
|
| We calibrate across Korean, English, and reasoning data, and we report all three **separately**. A build that holds its English scores while losing Korean looks healthy in aggregate and is not. We have measured exactly that failure in publicly available expert-pruned builds of a 124B model: |
|
|
| | Axis | Original weights | Expert-pruned build | |
| |---|---|---| |
| | Korean knowledge | 12 / 14 | **0 / 14** | |
| | English knowledge | 8 / 8 | **8 / 8** | |
|
|
| The English column is why the failure goes unnoticed. Asked a Korean question, the pruned build answered in Chinese, then repeated itself until it hit the token limit. Aggregate accuracy barely moved. |
|
|
| ### 3 · Failures get names |
|
|
| "78% accurate" hides what actually broke. Our evaluations separate **empty output**, **language collapse**, **runaway repetition**, **malformed tool calls**, **wrong arguments**, and **plainly wrong answers** — because a model that answers Korean questions in Chinese and a model that gets a date wrong need completely different fixes, and one number cannot tell them apart. |
|
|
| --- |
|
|
| ## How we measure |
|
|
| - **Denominators are always printed.** A score with no denominator is not a score. |
| - **Controls are always included.** We report a domain we did *not* optimize for, so you can tell a real gain from a shifted baseline. |
| - **The measuring tool is tested before the model is.** Our graders are mutation-tested: we feed them known failures and confirm each is caught under a *different* name. Two real defects in our own grader were caught this way before a single model was scored. |
| - **Every comparison names the exact file.** Repository, filename, bit width, harness, flags. If you cannot reproduce it, we should not have printed it. |
|
|
| --- |
|
|
| ## Roadmap |
|
|
| Korean-calibrated Q4 builds of widely used open models. Each one published with a side-by-side evaluation against the most-downloaded existing quant of the same model at the same bit width. |
|
|
| Same model, same bits, same harness. The only variable is the calibration. |
|
|
| **We will publish the ones that win. We will not publish the ones that do not.** |
|
|
| --- |
|
|
| ## Who we are |
|
|
| A team in Korea that builds inference acceleration and on-device models. We ship weights that run on CPU-only mini PCs and on Android tablets, with no GPU — which is how we learned that a quant can pass every aggregate benchmark and still be broken in the language our users actually speak. |
|
|
| We are new here. This page has no download counts on it yet, and we would rather it stayed that way until the tables underneath are ours. |
|
|
| --- |
|
|
| <sub>Open weights, permissively licensed. Reproductions welcome — if a number here does not replicate on your hardware, open a discussion and we will correct it.</sub> |
|
|
