# bench

Three checks behind the numbers on the model card. All of them run on CPU.

| Script | What it proves |
|---|---|
| `eval_bpc.py` | bits/char, perplexity and top-k against the base model; exits 1 on regression |
| `bench_cpu.sh` | prompt and generation tok/s of a GGUF build, plus a sample answer |
| `chat_smoke.py` | the chat checkpoint answers instead of continuing the prompt |

```bash
pip install torch transformers datasets

# continuation model: 20k tokens of Russian web text against the base
python bench/eval_bpc.py --model Rahspide/MNN-Pocket-0.3B-RU \
    --base taobao-mnn/MNN-Pocket-0.3B-Base --dataset HuggingFaceFW/fineweb-2 --config rus_Cyrl

# quantised build on your own llama.cpp
LLAMA_CLI=/path/to/llama-cli bash bench/bench_cpu.sh gguf/MNN-Pocket-0.3B-RU-Q4_K_M.gguf

# assistant: template in place, answers not echoed
python bench/chat_smoke.py --model Rahspide/MNN-Pocket-0.3B-RU-Chat --cpu
```

Notes:

- `eval_bpc.py` uses bits/char, not perplexity, as its comparison unit: perplexity is not comparable between
  models with different tokenizers, bits per character is.
- fp32 is the default so the numbers match the card. `--bf16` is roughly 15% faster on AVX-512 machines and
  moves the third decimal.
- `bench_cpu.sh` parses llama.cpp's own timing line, so it measures the binary you actually ship. It also
  accepts a decimal comma, which some locales print.