File size: 1,376 Bytes
8465953 c0f8664 8465953 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 | # Scripts
Tooling used to produce this checkpoint and the numbers in the model card. None
of it is needed to use the model.
Install the dependencies first:
```bash
uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python torch transformers bitsandbytes accelerate sentencepiece
```
## `quantize.py`
Rebuilds the 4-bit NF4 checkpoint from the original fp32 weights and writes a
self-contained repository (weights, tokenizer files, `.gitattributes`) that can be
uploaded directly. Optional argument: output directory, defaulting to
`../models/gradient-ai-text-detector-4bit`.
```bash
python scripts/quantize.py
```
It keeps the classifier head in fp32, because bitsandbytes' packed CPU kernel
requires each quantized layer's output dimension to divide evenly by its block
size and the head is `[1, 1024]`. It then asserts that no quantized layer would
break that kernel, and prints a reload sanity value.
## `bench_quant.py`
Compares fp32, bf16, and NF4 4-bit on CPU and Apple Silicon MPS, measuring
resident memory, batch latency, and the maximum probability change against the
fp32 reference.
```bash
python scripts/bench_quant.py
```
## `eval_quant.py`
Measures how much quantization moves individual scores: max and mean absolute
change in P(AI) against fp32, and how many verdicts flip at the 0.5 threshold.
```bash
python scripts/eval_quant.py
``` |