File size: 3,443 Bytes
e5235fb | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | # Low-bit Marco-Nano-Instruct: quantisation and evaluation code
The codebook and quantiser code and the benchmark and evaluation scripts behind the 1-, 2- and 3-bit GGUF builds of
[Marco-Nano-Instruct](https://huggingface.co/ATH-MaaS/Marco-Nano-Instruct) (8B parameters, 0.6B active, 232 experts).
The training code (the trainer, losses, data pipeline and run configs) is not published.
This is research code, published as it was used. Machine-specific paths are replaced by placeholders (`/path/to/data`,
`/path/to/repo`, `/path/to/llama.cpp`), so set them for your machine before running anything.
## What is here
| Path | Contents |
|---|---|
| `src/lowbit/quant/` | The quantisers and codebook formats: ternary and N-bit grids, the IQ codebooks (IQ2_XXS, IQ2_S, IQ3_S, IQ4_NL), importance weighting, GPTQ-style rounding |
| `src/lowbit/export/` | GGUF reading and writing for those formats, bit-exact with llama.cpp |
| `src/lowbit/model/` | The quantised student model the export and KL tools load |
| `src/lowbit/eval/`, `src/lowbit/gen/` | Evaluation helpers (GSM8K scoring, retention, and the RS suites, which are fixed question subsets of the benchmarks for comparing a quantised model with the original) and the loop detector |
| `scripts/` | Quantisation and GGUF tools and the benchmarks. Each file's docstring says what it does and how to run it |
| `configs/eval/`, `configs/precision/` | The evaluation suites and the per-part formats |
| `tests/` | Unit tests for the formats, GGUF round trips and the evaluation code. Some need a GPU or tens of GB of memory |
## Quantisation and GGUF tools
Some file names carry the internal task or decision they came from: `t27` is the GGUF export task, `d58` the
codebook-training decision and `cb` stands for codebook.
- `scripts/ptq_baseline.py`: llama.cpp's own quantisations of a model (imatrix) and their KL divergence to the bf16
model (`llama-perplexity --kl-divergence`, 40 × 4,096 held-out tokens).
- `scripts/cb_import.py` / `scripts/cb_gate.py`: read a llama.cpp GGUF's codebook codes and scales into the quantised
model, and check that it exports back bit-exactly.
- `scripts/t27_gguf.py build` (GGUF export): write a quantised model as a GGUF in its format.
- `scripts/tq1_repack.py`: repack 2-bit gate and up projections as llama.cpp's ternary TQ1_0.
- `scripts/rtn_gguf.py`, `scripts/imatrix_to_importance.py`: round-to-nearest quantisation with importance weights.
## Benchmarks and evaluation
- `scripts/compare_quality.py`: multiple choice (MMLU-Redux, MMLU-Pro, Global-MMLU, Belebele), GSM8K, MATH-500 and
IFEval through each model's own chat template, one llama-server per model.
- `scripts/code_eval.py`: HumanEval+ and MBPP+ with EvalPlus's tests in a no-network Docker sandbox.
- `scripts/ship_rule.py`: paired comparison of two files on the same items (2-standard-error rule) plus KL.
- `scripts/sampling_check.py`: looping replies under different sampling settings.
- `scripts/pod_kl.py`: quick KL of a quantised model against the original with PyTorch.
- `scripts/card_tables.py` (card tables), `scripts/d58_scaling_chart.py` (scaling chart): the model card's tables and
chart from the result files.
## Requirements
- Python 3.12, PyTorch, and the packages in `pyproject.toml`.
- llama.cpp for quantisation, KL and serving. Q2_0 and TQ1_0 are standard llama.cpp types.
- Docker for the code benchmark.
## Licence
Apache-2.0 (see `LICENSE`).
|