Inkling-2-Small-NVFP4-CSF

Multimodal checkpoint of Inkling 2 Small with NVFP4 routed experts in all 48 decoder layers, calibrated static activation scales, losslessly compressed expert block scales, and MXFP8 DSpark feed-forward weights.

Component Stored precision
Decoder routed experts NVFP4; E2M1 values, E4M3 block scales per 16 weights, FP32 global scale and FP32 calibrated input scale per projection; 48 layers, 384 experts, 55,296 matrices
Decoder attention projections BF16
Router weights and biases, short-convolution weights, attention-residual queries FP32; 368 tensors
Embedding, output head, normalization and other decoder tensors BF16
Vision tower BF16
DSpark feed-forward weights (mtp.safetensors) MXFP8; E4M3 values, UE8M0 scales per 32 weights; 15 matrices
Other DSpark tensors BF16; 69 tensors, including attention, normalization, fusion, and the Markov head

The decoder has 1,418 tensors outside the routed experts: 1,050 BF16 and 368 FP32. The FP32 router and short-convolution weights are exact upcasts of the source BF16 values.

Purpose

The upstream NVFP4 release, thinkingmachines/Inkling-2-Small, suits expert-parallel serving better than tensor-parallel serving, and it mixes two expert formats:

  • Its NVFP4 experts require per-token activation quantization: each token's scale comes from the maximum of that token's own activations. With expert parallelism every expert runs whole on one device, so that maximum is local. With tensor parallelism each device holds a slice of every expert's intermediate channels, so the devices must exchange per-token maxima inside every expert layer before the down projection can run.
  • Its experts are NVFP4 in decoder layers 0 through 39 and MXFP8 in layers 40 through 47. A runtime needs two expert formats and two sets of kernels, and many local inference stacks support only one of them.

This checkpoint uses the layout most local runtimes already serve: NVFP4 experts in every decoder layer, in the standard Model Optimizer format, with calibrated static activation scales. No scale depends on a token's activations at run time, so tensor-parallel devices run their expert slices independently.

Source and conversion

The pinned weight source is thinkingmachines/Inkling-2-Small-MXFP8 at revision 086a047e8b7ff2ac902dd52355328ddb66b16914. The activation scales are model-inputscales.safetensors from local-inference-lab/Inkling-2-Small at revision 1b77722adf8327b12b57af05cd2f370669b62657, stored unchanged.

Each source expert is decoded from MXFP8 to BF16, with its w13 rows de-interleaved into gate (even rows) and up (odd rows) projections, and packed as NVFP4 by NVIDIA Model Optimizer at commit cc6963dea0a6f01d90d66c7699baf3884df5f64a. Each expert has its own global scale: gate and up share max(|gate|, |up|) / 2688, and down uses max(|down|) / 2688. Block scales are block_max / (6 * global_scale) rounded to E4M3. No scale search, AWQ, or smoothing is applied. Each packed byte holds the even input column in its low nibble.

Each input_scale equals the calibrated input maximum divided by 2688. Gate and up share one value per layer; each down projection has its own value. All other decoder and vision tensors retain their source values. The 15 DSpark feed-forward matrices are converted from BF16 to MXFP8 by Model Optimizer; the other 69 DSpark tensors match the source byte-for-byte.

Tensors use the parameter names of Inkling2ForConditionalGeneration in Transformers 5.23.0, with routed experts stored per expert as model.language_model.layers.L.mlp.experts.E.{gate_proj,up_proj,down_proj}. Three groups keep their source spelling: model.language_model.embed_norm.weight, model.language_model.layers.L.self_attn.{k,v}_sconv.weight_proj on the eight global-attention layers, and model.vision_tower.embeddings.position_embedding.

Storage contract

The checkpoint contains 48 safetensors shards of at most 9.1 GB, the input-scale shard model-inputscales.safetensors, and the DSpark file mtp.safetensors. model.safetensors.index.json maps 443,786 stored tensors, holding 327,558,040,424 bytes, to the shards and the input-scale shard. mtp.safetensors holds 99 tensors and is not indexed.

Scale encoding: weight_scale_encoding: "csf_32x2_v1". The 55,296 compressed matrices are exclusively the routed-expert E4M3 block scales. Each logical weight_scale tensor is represented by five planes: uint8 .csf32_bases, uint8 .csf32_codes, uint32 .csf32_page_bases, uint16 .csf32_offsets, and uint16 .csf32_exceptions. The planes occupy 19,883,694,952 bytes; the scales they encode occupy 36,691,771,392 bytes. Packed weights, global scales, input scales, and DSpark MXFP8 scales remain uncompressed.

Runtime requirements

config.json and hf_quant_config.json use quant_method: "modelopt", quant_algo: "NVFP4", group_size: 16, and weight_scale_encoding: "csf_32x2_v1". The ignore list excludes the output head, the vision tower, and every decoder layer's router, attention, and short-convolution modules. hf_quant_config.json declares the DSpark precision groups under mtp.parameter_groups: a BF16 group of 69 tensors and an mxfp8_nearest group of 15 tensors with block shape [1, 32].

A runtime must reconstruct the E4M3 block scales from the csf32 planes, or execute them directly, before running the routed experts. The static input scales support NVFP4 activation quantization without per-token scale computation.

Validation

Every compressed scale plane passed lossless reconstruction against the uncompressed E4M3 block scales with a decoder independent of the encoder. All other tensors in the numbered shards match the uncompressed conversion byte-for-byte. For that conversion, sampled experts in decoder layers 2, 40, and 47 reproduce Model Optimizer's packed values, block scales, and global scales exactly from an independently decoded source; all non-expert tensors match the cast source tensors; the stored input scales are byte-identical to the published file; and the DSpark MXFP8 matrices equal Model Optimizer's conversion.

License

LICENSE, NOTICE, and LICENSES/ hold the license and attribution materials for this work, including the upstream model's Apache License, Version 2.0.

Downloads last month
100
Safetensors
Model size
634B params
Tensor type
F32
ยท
U32
ยท
BF16
ยท
U16
ยท
U8
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for local-inference-lab/Inkling-2-Small-NVFP4-CSF

Quantized
(5)
this model
Quantizations
1 model