FOCUS NVFP4 scales for phi-4 (our reproduction), 1,024 calibration windows
This is our reproduction of FOCUS (“FOCUS: FP4 Optimization via Coupled-Relaxation and Dual-Granularity Scaling”, arXiv:2608.01847), made as a baseline for the FlipQuant paper. It is not an official release by the FOCUS authors. It is calibrated on 1,024 windows, four times the 256 of edgeai-lab/phi-4-FOCUS-NVFP4.
What this is
focus.pt holds the learned FOCUS scale parameters for microsoft/phi-4 at revision 2db69c1c3e91a05d2c64a3185acfbaf36f744e25, for every quantized linear layer (160 layers: all text-model linear layers except the LM head). Per layer, in FP32:
| field | count | role |
|---|---|---|
m |
one per 16-element block (851,968,000 in total) | the block-scale factor (Coupled-Relaxation Scaling); starts at 1 |
q |
one per 8-element sub-block, 2 per block (1,703,936,000 in total) | the sub-block relaxation logit (Dual-Granularity Scaling); coefficient sigmoid(q), starts at q = 6 |
s2 |
one per tensor | the NVFP4 tensor scale amax(abs(W)) / (6 · 448), frozen (it follows from the base weights) |
m and q are in row-major order of the weight [out_features, in_features], 16-element blocks along in_features. Each layer also has act_gscale = None: activations use one scale per token, so there is no static activation scale. The file also stores the calibration settings (config) and the base model's id and revision (source).
It is not a quantized checkpoint: it holds no weights. Using it needs the BF16 base model at the pinned revision and the FOCUS implementation, which turns the base weights and these parameters into standard NVFP4 codes. For a block b of 16 weights with sub-blocks k = 1, 2 of 8:
S_b = e4m3( clamp( max|w_b| · m_b / (6 · s2), 2^-6, 448 ) ) # the block's E4M3 scale
c_i = E2M1( clamp( w_i / (S_b · sigmoid(q_k) · s2), -6, 6 ) ) # element i in sub-block k
w'_i = c_i · S_b · s2 # the deployed (dequantized) weight
E2M1 rounds to the nearest FP4 value (ties to even); a block whose scaled maximum is 0 gets scale 1 before the clamp. sigmoid(q) only changes which code each element gets: the stored scales are S_b and s2, so the deployed model is plain NVFP4 (E2M1 elements, one E4M3 scale per 16 elements, one FP32 scale per tensor) and runs on standard NVFP4 kernels with no extra metadata or cost.
The code (our FOCUS implementation and the evaluation) will be released with the paper.
Calibration
The hyperparameters follow the FOCUS paper's NVFP4 setting:
- learning rate 5e-3 for the block-scale factors
m, 1e-3 for the relaxation logitsq; - 2 sub-blocks of 8 elements per 16-element block;
- KL-Top loss with k = 1000 (the BF16 model's top-1000 tokens per position, both distributions renormalized over them);
- AdamW (betas 0.9 / 0.999, weight decay 0) with a constant learning rate;
- 1 epoch, global batch 32;
qinitialized to 6 (andmto 1); seed 42 (the data order).
Only the scale parameters train; the model weights are frozen. The teacher is the unquantized BF16 model.
Differences from the paper:
- Calibration data: 1,024 windows of 512 tokens (524,288 tokens). The first 128 are the calibration record's: the FlipQuant paper's math/code windows (OpenWebMath @ fde8ef8, CodeParrot-clean @ 35a59fb), built by the paper's rule with this model's tokenizer. The other 896 come from the release extension rule (seed 20260930; one window per document, half from each source). It skips the record's fit documents, the development records and the published C4 evaluation documents (128 / 192 / 0 skipped). The set contains the release's 256 windows, window for window, so the 256-window version's fit set is a subset of this one. Every window is recorded by document sha256, offset and token sha256 (sha256 of the token-hash list:
a78f561bb85616860e72d42c4c86a07b41a7ead127f52a03caf5f406e06a4d61). A word 13-gram audit of the study's 2,048-window set, which contains these windows, against every evaluation text found no shared 13-gram. The evaluation texts are WikiText-2 test, the C4 evaluation documents, the six zero-shot tasks, GSM8K test and its few-shot exemplars. The audit's positive control detects evaluation items spliced into windows, 30 of 30 for each item. That is 524,288 tokens and 32 optimizer steps (the 256-window state: 8). The paper uses 1,248 WikiText-2 training samples of 2,048 tokens (39 steps). - Activations, during calibration and evaluation: FourOverSix with one scale per token, as in the FlipQuant paper's protocol. The paper quantizes activations to NVFP4 with a static per-tensor scale.
Cost on one NVIDIA RTX PRO 6000 Blackwell Workstation Edition (sm_120), micro-batch 8 (4 forward/backward passes per step), activation checkpointing per decoder block:
| training (32 steps) | end to end (model load, data, training, save) | peak GPU memory |
|---|---|---|
| 27.5 min | 28.3 min | 91.0 GiB |
The peak is PyTorch's peak allocated memory on the GPU (torch.cuda.max_memory_allocated). The training time includes the BF16 teacher's forward passes.
Results
From the FlipQuant paper's calibration-size study. The numbers are also in ppl_summary.json.
Native sm_120 kernels on one NVIDIA RTX PRO 6000 Blackwell. Every quantized row uses per-token FourOverSix activations, and every row is evaluated on the same data:
- PPL: WikiText-2 test (141 windows of 2,048 tokens) and C4 validation (256 windows of 2,048 tokens), FlipQuant's
--paper-convention; - zero-shot: PIQA, HellaSwag, WinoGrande, ARC-Easy, ARC-Challenge and LAMBADA (lm-eval 0.4.11, full evaluation splits; acc_norm for PIQA, HellaSwag and ARC, acc for WinoGrande and LAMBADA). Avg. is their mean;
- GSM8K: lm-eval's
gsm8k_llama: 8-shot chain of thought with the chat template, greedy decoding, at most 1,024 new tokens, strict-match accuracy ± 2 SE.
| method | zero-shot Avg. (%) | GSM8K strict (%) | WikiText-2 PPL | C4 PPL |
|---|---|---|---|---|
| BF16 | 73.53 | 93.33 ± 1.37 | 6.462 | 10.310 |
| FourOverSix (reference) | 72.39 | 91.74 ± 1.52 | 6.662 | 10.544 |
FOCUS, 256 windows (phi-4-FOCUS-NVFP4) |
73.14 | 92.72 ± 1.43 | 6.628 | 10.523 |
| FOCUS, 1,024 windows (this repository) | 73.05 | 91.51 ± 1.54 | 6.641 | 10.536 |
Zero-shot accuracy (%) per task:
| method | PIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | LAMBADA |
|---|---|---|---|---|---|---|
| BF16 | 81.12 | 81.95 | 76.95 | 72.77 | 56.14 | 72.27 |
| FourOverSix (reference) | 80.69 | 81.11 | 74.27 | 71.89 | 54.86 | 71.55 |
FOCUS, 256 windows (phi-4-FOCUS-NVFP4) |
80.52 | 81.48 | 76.32 | 72.81 | 55.38 | 72.33 |
| FOCUS, 1,024 windows (this repository) | 81.28 | 81.32 | 75.77 | 73.19 | 54.52 | 72.19 |
This repository against its references, paired.
- Zero-shot: pooled over all 21,848 questions of the six tasks.
- GSM8K: strict-match over its 1,319 problems.
- Format: difference in points (questions only this row gets right / only the reference gets right, exact McNemar p).
- ΔNLL: the mean paired per-window difference in nats per token, ± 2 SE; negative is lower NLL.
*: p < 0.05, or |Δ| > 2 SE.
| comparison | zero-shot | GSM8K | ΔNLL WikiText-2 | ΔNLL C4 |
|---|---|---|---|---|
| vs FourOverSix | +0.51* (630/519, p=0.0012) | -0.23 (29/32, p=0.8) | -0.0032 ± 0.0017* | -0.0007 ± 0.0012 |
| vs its 256-window version (phi-4-FOCUS-NVFP4) | -0.08 (461/478, p=0.6) | -1.21* (19/35, p=0.04) | +0.0019 ± 0.0013* | +0.0012 ± 0.0008* |
How to read the file
With PyTorch alone:
import torch
state = torch.load("focus.pt", map_location="cpu", weights_only=True)
state["config"] # the calibration settings
state["source"] # the base model id and revision
layer = state["layers"]["model.layers.0.self_attn.o_proj"]
layer["m"], layer["q"], layer["s2"] # FP32 tensors (see above)
License
This state is released under the same license as the base model, the MIT License.
LICENSE is a copy of the base repository's license file (microsoft/phi-4).
Files
| file | bytes | sha256 |
|---|---|---|
LICENSE |
1,084 | fa8235e5b48faca34e3ca98cf4f694ef08bd216d28b58071a1f85b1d50cb814d |
focus.pt |
10,223,745,627 | aa44f7c49b0af62d77d6fa4211ab28c01da6777325a63c1876d3bb0adbacfcb6 |
ppl_summary.json |
6,656 | 528913846188f7dd43066d7d18f47649b704d89429abf56c70bf658397c3c0bb |
focus.pt: the FOCUS state ({config, layers, source}; above). It is the calibration output, unchanged.ppl_summary.json: the numbers above, the evaluation settings, the calibration settings and cost.LICENSE: see License.
Model tree for edgeai-lab/phi-4-FOCUS-NVFP4-Calib1k
Base model
microsoft/phi-4