Krafla 0.8.0 (0b527a0)

README.md

@@ -25,22 +25,27 @@ INT4 and INT8 conversions of [HuggingFaceTB/SmolLM2-360M-Instruct](https://huggi
 
 | File | Weights | Size | Perplexity | Prefill | Decode | SHA-256 |
 |------|---------|-----:|-----------:|--------:|-------:|---------|
-| `SmolLM2-360M-Instruct-Q4_0.mlb` | Q4_0, every matrix 4-bit | 194 MiB | 21.73 | 755 tok/s | 154.9 tok/s | `4269a4b315b7913f`… |
-| `SmolLM2-360M-Instruct-Q8_0.mlb` | Q8_0, every matrix 8-bit | 367 MiB | 17.45 | 510 tok/s | 100.5 tok/s | `1bf18d4f21de7a7c`… |
+| `SmolLM2-360M-Instruct-Q4_0.mlb` | Q4_0, every matrix 4-bit | 194 MiB | 21.73 | 758 tok/s | 173.6 tok/s | `1b6c4f2992bfa579`… |
+| `SmolLM2-360M-Instruct-Q8_0.mlb` | Q8_0, every matrix 8-bit | 367 MiB | 17.45 | 510 tok/s | 90.2 tok/s | `410bd0a3daade795`… |
 | `tokenizer.mlt` | tokenizer | 1138 KiB | | | | `02f356d659655d21`… |
 
-Measured on a GitHub runner (AMD EPYC 9V45 96-Core Processor, 4 threads) with Krafla's `avx2` build. Perplexity is measured on the wikitext-2 test set, 4 chunks of 512 tokens (`tools/perplexity.py --ctx 512 --chunks 4`). Lower is better, and it is only comparable between files of the same model. Speed comes from `ml_bench -n 128` (128 prompt tokens, then 128 decoded tokens). The files are converted with a 4096-position context. The full hashes are in `SHA256SUMS`.
+Measured on a GitHub runner (AMD EPYC 9V45 96-Core Processor, 4 threads) with Krafla's `avx2` build. Perplexity is measured on the wikitext-2 test set, 4 chunks of 512 tokens (`tools/perplexity.py --ctx 512 --chunks 4`). Lower is better, and it is only comparable between files of the same model. Speed comes from `ml_bench -n 128` (128 prompt tokens, then 128 decoded tokens). The full hashes are in `SHA256SUMS`.
 
 ## Run
 
-Download a [release](https://github.com/JonPark0/krafla/releases) of Krafla (version 0.6.1 or later; Linux x86-64, AArch64 and ARMv7, Windows x86-64) and two files from here:
+Download a [release](https://github.com/JonPark0/krafla/releases) of Krafla (version 0.8.0 or later; Linux x86-64, AArch64 and ARMv7, Windows x86-64) and two files from here:
 
 ```sh
 curl -LO https://huggingface.co/jonpark0/SmolLM2-360M-Instruct-krafla/resolve/main/SmolLM2-360M-Instruct-Q4_0.mlb
 curl -LO https://huggingface.co/jonpark0/SmolLM2-360M-Instruct-krafla/resolve/main/tokenizer.mlt
 ./ml_run SmolLM2-360M-Instruct-Q4_0.mlb -z tokenizer.mlt --chat smollm -t 4
+
+# or serve it with the OpenAI API at http://127.0.0.1:8080/v1
+./ml_server SmolLM2-360M-Instruct-Q4_0.mlb -z tokenizer.mlt --chat smollm -t 4
 ```
 
+The files record the model's 8192-position context. `ml_run` and `ml_server` use 4096 positions of it unless you pass `--ctx N`, which sets the size of the KV cache.
+
 The release's `ml_run` runs the most capable build your CPU supports. A greedy one-turn reply from `SmolLM2-360M-Instruct-Q4_0.mlb` to "What is the capital of France? Answer in one sentence.":
 
 ```text
@@ -49,12 +54,12 @@ The capital of France is Paris.
 
 ## How these files were made
 
-With Krafla 0.6.1 (commit [`93ea003`](https://github.com/JonPark0/krafla/commit/93ea00305954ebaf501af2f3e956f9436484f06d)) on 2026-11-02, from the safetensors of [HuggingFaceTB/SmolLM2-360M-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct):
+With Krafla 0.8.0 (commit [`0b527a0`](https://github.com/JonPark0/krafla/commit/0b527a04aebd4234ed3c2197bfbb0248a1628ac7)) on 2026-11-03, from the safetensors of [HuggingFaceTB/SmolLM2-360M-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct):
 
 ```sh
 python3 tools/convert_tokenizer.py SmolLM2-360M-Instruct/tokenizer.json tokenizer.mlt
-python3 tools/convert.py hf SmolLM2-360M-Instruct SmolLM2-360M-Instruct-Q4_0.mlb --wtype q4 --max-seq 4096
-python3 tools/convert.py hf SmolLM2-360M-Instruct SmolLM2-360M-Instruct-Q8_0.mlb --wtype q8 --max-seq 4096
+python3 tools/convert.py hf SmolLM2-360M-Instruct SmolLM2-360M-Instruct-Q4_0.mlb --wtype q4
+python3 tools/convert.py hf SmolLM2-360M-Instruct SmolLM2-360M-Instruct-Q8_0.mlb --wtype q8
 ```
 
 Q4_0 and Q8_0 keep one scale per block of 32 weights (f16 scales: 4.5 and 8.5 bits per weight). See the [user guide](https://github.com/JonPark0/krafla/blob/main/docs/user-guide.md#q8-q4-or-mixed) for the formats and [the comparison with llama.cpp](https://github.com/JonPark0/krafla/blob/main/docs/benchmarks.md).

SHA256SUMS

@@ -1,3 +1,3 @@
-4269a4b315b7913fc9e272c5e67c1e0beaaa8694a9a5ee85e8f8451c2c9c9f38  SmolLM2-360M-Instruct-Q4_0.mlb
-1bf18d4f21de7a7c9cb0129925caf0eac8adca3db7d6be8de9905796018bb894  SmolLM2-360M-Instruct-Q8_0.mlb
+1b6c4f2992bfa579a2419b7d3a9b8710d7a3f81a73667889aba1393d31d97d4c  SmolLM2-360M-Instruct-Q4_0.mlb
+410bd0a3daade7957ec2fe11460bdb6c1fb57caf3e3b1bf4540ad0214ea1b2ef  SmolLM2-360M-Instruct-Q8_0.mlb
 02f356d659655d214e96ceb64af2f6adc1df3a1c51c04b8543e2e15957736c1b  tokenizer.mlt

SmolLM2-360M-Instruct-Q4_0.mlb

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4269a4b315b7913fc9e272c5e67c1e0beaaa8694a9a5ee85e8f8451c2c9c9f38
+oid sha256:1b6c4f2992bfa579a2419b7d3a9b8710d7a3f81a73667889aba1393d31d97d4c
 size 203739136

SmolLM2-360M-Instruct-Q8_0.mlb

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1bf18d4f21de7a7c9cb0129925caf0eac8adca3db7d6be8de9905796018bb894
+oid sha256:410bd0a3daade7957ec2fe11460bdb6c1fb57caf3e3b1bf4540ad0214ea1b2ef
 size 384618496