kekzle/Flint-NVFP4

Flint NVFP4 (SafeTensors, ModelOpt layout, for vLLM and imp)

Recursive Flint (39.8 B parameters, about 2.5 B active per token) as NVFP4 SafeTensors. The quantized tensors are Recursive's own: this repo is a lossless repack of the official RecursiveSI/flint-nvfp4-gguf into the ModelOpt NVFP4 layout, so the checkpoint loads in vLLM and in imp without a GGUF reader. 74.1 GiB of BF16 is 21.7 GiB here and fits one 32 GB card.

Not affiliated with Recursive. Model details, benchmarks, chat conventions and limitations: the Flint model card.

What is in the files

Source of the quantized tensors	flint-nvfp4.gguf from RecursiveSI/flint-nvfp4-gguf, 22 863 084 160 bytes
Repack	vllm.models.flint.gguf_nvfp4 (vLLM 0.33.0): E2M1 codes, E4M3 block scales and the F32 tensor scale are copied, not recomputed
Layout	per Linear <name>.weight (U8, two codes per byte), <name>.weight_scale (F8_E4M3, one per 16 columns), <name>.weight_scale_2 (F32)
Quantized	36 439 Linears: attention q/k/v/o, the two dense MLPs, shared experts, all 256 x 47 routed experts
Source precision (BF16)	model.embed_tokens, the 47 routers mlp.gate, norms, attention sinks
lm_head	NVFP4 in the GGUF; written here as BF16 = code x block scale x tensor scale (one rounding to BF16), the form vLLM and imp both load
Declaration	config.json quantization_config (quant_method: modelopt, quant_algo: W4A16_NVFP4) and hf_quant_config.json
Tensors	109 612 in 15 shards

Use it

# vLLM >= 0.33.0 (the first release with FlintForCausalLM); verified on 0.33.0 with a local copy, RTX 5090
vllm serve kekzle/Flint-NVFP4 --served-model-name flint --max-model-len 8192 --gpu-memory-utilization 0.90

# imp (builds with Flint support, after v0.67.1)
docker run --rm --gpus all -p 8080:8080 -v /path/to/models:/models \
  ghcr.io/kekzl/imp:latest \
  imp-server --host 0.0.0.0 --model /models/Flint-NVFP4

Recommended sampling (Recursive): temperature 1.0, top-p 0.95, top-k 20. Use the shipped chat template and keep reasoning_content in the conversation history.

Measured

Perplexity on the same 4 095 rows (the first 4 096 tokens of imp's ppl_corpus_45k.txt, no BOS). The reference is a CPU FP32 forward written from Recursive's SGLang model code; it keeps activations at full precision (W4A16).

Weights	Runtime	PPL
BF16 (RecursiveSI/flint)	CPU reference	12.717
this repo	CPU reference, W4A16	12.861 (NLL +0.011 +- 0.016 against BF16)
this repo	imp, RTX 5090 (prefill quantizes activations, W4A4)	13.841

• Against BF16 the next-token argmax agrees on 82.2 % of the rows (KL 0.221 +- 0.014): the distribution moves, the mean loss does not.
• vLLM 0.33.0 on an RTX 5090 (32 GB): loads, answers, 31 125 MiB at --gpu-memory-utilization 0.90. vLLM's Flint kernels target SM100 cards; on the 5090 it runs its stock FP4 Marlin path.
• imp on the same card: chat, reasoning split, OpenAI and Anthropic tool calls; about 350 tokens/s for 600-1200 token answers, one stream.

License

Apache-2.0, as the source repos. The LICENSE file is Recursive's.