- epic_quant/eval/smoke.py: CPU-runnable smoke test that chains the
engine's forward_one_layer through all 42 layers on a fixed prompt
and reports top-10 next-token predictions. Validates the engine
produces a coherent forward end-to-end on real E4B weights.
- epic_quant/eval/kaggle_notebook.ipynb: full eval sweep for a Kaggle
GPU box (T4 or P100). Three evals:
- WikiText-103 PPL (50K tokens, 512-token chunks)
- MMLU Pro 4-shot accuracy (200-question subsample, loglik over A/B/C/D)
- MRCR v2 8-needle retrieval (8K context; 128K OOMs on T4)
Compares 1.58 / 3 / 4 / 16-bit policies against the published
Gemma 4 E4B 69.4% MMLU Pro and 25.4% MRCR v2 8-needle 128K.
Smoke test ran on this 13.8 GB RAM / CPU box:
- 16-bit: 1.37 s/layer, top-1 token id 26069
- 3-bit: 2.50 s/layer, top-1 token id 26069 (matches)
Top-1 is robust; top-10 overlap is 3/10, consistent with the
L2 recon numbers.
