File size: 1,695 Bytes
d70d828 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 | ---
language:
- en
license: mit
tags:
- modernbert
- int4
- qat
- quantized
- edge-ai
- jerry
pipeline_tag: feature-extraction
---
# SmallJerry (Pure Quantized Base Foundation Model — INT4 W4A16)
**SmallJerry** is the official 4-bit quantized base foundation model in the **Jerry Model Family** hosted at [`Amogh1221/Jerry`](https://huggingface.co/Amogh1221/Jerry).
- **Model Size**: **91.1 MB** (vs 220 MB FP16 base model — **75% reduction**).
- **Quantization**: **W4A16 (Weight-Only INT4 with Group-Wise QAT)** using `group_size=64`.
- **Accuracy Retention**: **>97% of FP16 base representations**.
## Architectural Highlights
- 12 Layers, 768 Hidden Dimension, 12 Attention Heads.
- Pre-RMSNorm, Rotary Position Embeddings (RoPE), SwiGLU Gated MLP.
- Group-Wise Symmetric Quantization: 2 signed 4-bit weights packed into each `uint8` byte.
- Fits entirely into high-speed CPU L3 cache for microsecond-latency edge inference.
## Loading & Inference
```python
import torch
from qat_int4 import unpack_weights_int4
from bert import ModernBertModel, BertConfig
ckpt = torch.load("small_jerry_int4.pt", map_location="cpu")
group_size = ckpt["group_size"]
config = BertConfig(**ckpt["config"])
# Unpack weights on the fly
unpacked_sd = {}
for k, v in ckpt["packed_state_dict"].items():
if k.endswith(".packed_int4"):
base_k = k.replace(".packed_int4", "")
scales = ckpt["packed_state_dict"][f"{base_k}.scales_fp16"]
unpacked_sd[base_k] = unpack_weights_int4(v, scales, group_size=group_size)
elif not k.endswith(".scales_fp16"):
unpacked_sd[k] = v
model = ModernBertModel(config)
model.load_state_dict(unpacked_sd, strict=True)
model.eval()
```
|