HawkBearPig/flint-FP8-NVFP4

Flint — mixed FP8/NVFP4, 35.74 GB

35.738 GB of weight tensors: 55.09% smaller than BF16 and 14.36% smaller than all-expert FP8. This checkpoint implements the budget36-swap1 projection policy from the footprint-constrained Flint study. It contains 23,040 FP8 expert matrices, 13,056 NVFP4 expert matrices, and 638 unchanged BF16 tensors. No routed expert matrix remains BF16.

Of all model parameters, 60.72% are FP8, 34.41% are NVFP4, and 4.87% are BF16. The BF16 tensors include routers, shared experts, attention, sinks, norms, embeddings, the output head, and the initial two dense layers. They retain the source bytes. Activations remain BF16; there is no retraining or runtime recovery.

The policy was selected by text fidelity within a fixed storage budget, then checked on separate text and task fixtures. It has 91.89% next-token agreement with BF16, versus 93.59% for all-expert FP8 on the same text. The smaller model's higher task point estimates below are uncertain; they do not establish superior or equivalent general capability. This is a measured storage/quality tradeoff, not a claim of a globally optimal quantizer.

Compatibility: inference requires Flint architecture support and the mixed weight layout documented below. This is a custom safetensors format, not a claimed drop-in export for a particular inference framework. The included correctness reference expands the weights to BF16 and needs roughly 80 GB of GPU weight memory. Native inference directly from compressed weights has not been qualified. The 35.738 GB figure measures stored tensors, not total serving VRAM; KV cache, activations and workspace are additional.

Source: RecursiveSI/flint, revision 2a247be0c0aec895ccb0ac7ca92229ec6df37288. The source contains 39,785,287,456 parameters and 79,570,574,912 BF16 tensor bytes. This release was exported and checked on an RTX PRO 6000 Blackwell Server GPU.

Exact precision allocation

Layer numbers are zero-based. Each entry applies to all 256 routed experts in that layer. Every weight outside these routed projections remains BF16.

Layers	Gate	Up	Down
2–11, 13–30, 32	FP8	FP8	FP8
12	FP8	NVFP4	FP8
31, 33–36, 38–48	NVFP4	NVFP4	NVFP4
37	NVFP4	NVFP4	FP8

The machine-readable allocation is in precision-policy.json and config.json → quantization_config → routed_experts → precision_by_layer. The two projection exceptions are part of the frozen, evaluated policy.

Stored format	Parameter share	Tensor bytes, including scales
FP8	60.72%	24,165,089,280
NVFP4	34.41%	7,700,794,368
BF16	4.87%	3,871,776,320
Total	100%	35,737,659,968

Safetensors headers, tokenizer and diagnostic files are additional disk bytes. This is a separate checkpoint from the earlier 30.802 GB early-FP8/late-NVFP4 release.

Held-out quality comparison

The study evaluated 72 complete policies under 28–48 GB budgets after profiling all 47 MoE layers in three baseline contexts and 564 local projection changes. Finalists were frozen before the independent checks below. The original study and all policy comparisons are also included in diagnostics/qualification/ as a historical snapshot.

The text holdout contains 49,140 scored targets in 12 disjoint 4,096-token windows, balanced across prose, code and math. Previously used source rows were excluded. Code/math text likelihood is not a coding or reasoning task score.

Checkpoint	Tensor GB	Agreement with BF16	KL from BF16	Text NLL
BF16 source	79.571	100%	0	2.214730
All routed experts FP8	41.730	93.59%	0.072432	2.217737
This mixed checkpoint	35.738	91.89%	0.090109	2.219417

Agreement measures matching next-token choices on identical input text, not answer correctness. Lower KL means closer probability distributions. These results establish greater numerical drift than all-expert FP8.

The task check uses all 500 OpenBookQA test questions, without supporting facts, and 1,024 HellaSwag validation questions excluded from the earlier subsets. This is a custom zero-shot, untemplated continuation-likelihood runner, not an official harness run or an assessment of free-form generation. acc_norm divides each continuation log-likelihood by its character count.

Checkpoint	OpenBookQA acc_norm	HellaSwag acc_norm	OpenBookQA raw accuracy	HellaSwag raw accuracy
BF16 source	42.20%	75.68%	32.60%	59.77%
All routed experts FP8	41.60%	75.78%	31.80%	60.25%
This mixed checkpoint	43.00%	76.37%	32.80%	60.06%

For this checkpoint minus BF16, normalized-accuracy differences are +0.80 percentage points on OpenBookQA, with a paired 95% interval of −0.80 to +2.40, and +0.68 points on HellaSwag, with an interval of −0.29 to +1.66. Compared with all-expert FP8, the corresponding intervals are −0.40 to +3.40 and −0.39 to +1.56 points. They include zero. These marginal intervals use 10,000 question-level bootstrap draws and are not adjusted for multiple comparisons. No equivalence margin was specified.

Normalized answer agreement with BF16 is 94.40% on OpenBookQA and 96.68% on HellaSwag. The mixed model changes 6 correct OpenBookQA answers to wrong and 10 wrong answers to correct; corresponding HellaSwag counts are 11 and 18. Similar aggregate accuracy can therefore hide different errors. Longer reasoning, coding-agent tasks and broad generation quality remain unqualified.

All policies use matching operation shapes and actual FlashInfer CUDA fused-add RMSNorm, with the same PyTorch attention/expert reference. Process swap is zero. These results make no serving throughput claim.

Evaluation summary · Per-question likelihoods and paired comparisons · Per-window text results · Footprint/quality analysis

Export integrity and generation checks

The export audit checks all 48 shard hashes and every tensor's name, shape, dtype and value. Every quantized tensor matches the evaluated candidate bytes; all 638 preserved tensors and tokenizer/template/license files match the source. The serialized checkpoint also replays all 49,140 held-out text targets through the mixed-format decoder: every per-window metric exactly matches the frozen policy evaluation. This replay is an export check, not additional independent quality evidence.

Four greedy chat smoke checks cover a fixed phrase, arithmetic, a small Python function and JSON output. Their full outputs and token IDs are included. They check loading, decoding, the chat template and basic completion; they do not establish general generation quality.

Tensor audit · Export replay · Generation transcripts · Shard manifest

Weight encoding

FP8 expert tensors retain the *.weight name and use E4M3 values with one FP32 scale per 128×128 block under *.weight_scale_inv. Despite that suffix, the stored value is a multiplicative scale:

W[row, col] = float32(weight[row, col]) * weight_scale_inv[row // 128, col // 128]

FP8 scales are block absmax / 448; zero blocks use scale 1.

NVFP4 expert matrices use *.weight_packed (uint8, two E2M1 codes per byte), *.weight_scale (E4M3, one per 16 consecutive input coordinates), and *.weight_global_scale (one positive FP32 reciprocal matrix scale). The low nibble encodes the even input coordinate:

W[row, col] = e2m1(code[row, col]) * float32(weight_scale[row, col // 16]) / weight_global_scale

The E2M1 magnitudes are 0, 0.5, 1, 1.5, 2, 3, 4 and 6, with a sign bit. The global reciprocal scale is 2688 / matrix absmax, or 1 for a zero matrix. For each 16-weight block, the recipe tries 31 scale factors from 0.70 to 1.30 around block absmax, rounds scales to E4M3 and minimizes squared weight error. Both quantization recipes are calibration-free. The precision assignment was chosen using selection-text fidelity; evaluation labels did not fit weights. The index, configuration and manifest specify the complete tensor contract.

Reproduction

The reproduction/ directory includes the quantizer, decoder, fused evaluator, policy search, frozen fixtures, generation prompts, package versions and source hashes. diagnostics/qualification/ preserves the selection and holdout results, rank uncertainty, complete policy maps and local-search comparisons. The historical report describes the policy before export; this repository now contains that policy's actual weights.

With the pinned source snapshot in source/, a compatible CUDA toolchain and the recorded Python packages, the following builds the selected formats and exports them. Use empty work/ and checkpoint/ directories; the tools refuse to overwrite candidate weights.

python reproduction/flint_quantize.py prepare \
  --source source --work work --corpus reproduction/corpus-layer-qualification.json \
  --precision-policy precision-policy.json
python reproduction/flint_quantize.py export \
  --source source --work work --corpus reproduction/corpus-layer-qualification.json \
  --precision-policy precision-policy.json --output checkpoint
python reproduction/flint_quantize.py audit \
  --source source --work work --corpus reproduction/corpus-layer-qualification.json \
  --output checkpoint
python reproduction/flint_export_validation.py \
  --source source --checkpoint checkpoint \
  --corpus reproduction/corpus-layer-qualification.json \
  --expected diagnostics/qualification/qualification-validation.json \
  --policy budget36-swap1 --output work/export-validation.json
python reproduction/flint_fused_generate.py \
  --checkpoint-dir checkpoint --prompts-json reproduction/smoke-prompts.json \
  --max-new-tokens 512 --output work/generation-smoke.json

The measured reference uses Torch 2.8.0+cu128, Triton 3.4.0 and FlashInfer 0.6.16.post1 CUDA normalization, compiled with CUDA 12.9 on SM120. Reproducing exact floating-point scores can depend on kernels, hardware and operation shapes. The public fixtures are no longer unseen data for further tuning. The source license, tokenizer, chat template and generation configuration are preserved without modification.