Instructions to use FutureMLS-Lab/flint-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use FutureMLS-Lab/flint-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="FutureMLS-Lab/flint-NVFP4")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("FutureMLS-Lab/flint-NVFP4")
model = AutoModelForCausalLM.from_pretrained("FutureMLS-Lab/flint-NVFP4", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use FutureMLS-Lab/flint-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "FutureMLS-Lab/flint-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FutureMLS-Lab/flint-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/FutureMLS-Lab/flint-NVFP4

SGLang

How to use FutureMLS-Lab/flint-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "FutureMLS-Lab/flint-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FutureMLS-Lab/flint-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "FutureMLS-Lab/flint-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FutureMLS-Lab/flint-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use FutureMLS-Lab/flint-NVFP4 with Docker Model Runner:
```
docker model run hf.co/FutureMLS-Lab/flint-NVFP4
```

flint-NVFP4

NVFP4 quantization of RecursiveSI/flint (revision 2a247be), Recursive's 39.8B-total / ~2.5B-active mixture-of-experts model for coding agents. The checkpoint is 23.5 GB instead of 79.6 GB, so one Blackwell GPU keeps most of its memory for KV cache.

What is quantized

The 12,032 routed experts of MoE layers 2–48 (gate_proj, up_proj, down_proj) are static W4A4 NVFP4: FP4 E2M1 weights in 16-element blocks with FP8 E4M3 block scales and FP32 global scales, in NVIDIA ModelOpt's per-expert layout (weight, weight_scale, weight_scale_2, input_scale).
Every other tensor is the unchanged BF16 source tensor: attention and its learned sinks, routers, shared experts, the dense layers 0–1, norms, embeddings and lm_head.

Evaluation

B300, SGLang 0.5.16 with flint_sglang 0.1.4 and the base repository's serve_b300.sh flags, greedy decoding.

	BF16	NVFP4
GSM8K, first 250 test questions	96.4	96.0
Held-out NLL, 32 × 8,192-token coding-agent trajectories	4.816	4.757
Weights in GPU memory	~75 GB	23.6 GB

Flint's top-k routing makes per-token outputs sensitive to any numerical change: two BF16 implementations of the model differ in held-out NLL by about ±0.6%, so NLL differences of this size are noise rather than a quality change.

Usage

NVFP4 needs a Blackwell GPU; this checkpoint was validated on B300. Install the Flint SGLang kit from the base repository (sglang/install.sh, or its Dockerfile), then point the kit's serve script at this checkpoint. SGLang reads the ModelOpt NVFP4 configuration from config.json, and the kit's default flags (FA4 attention, flashinfer_trtllm_routed MoE) work unchanged.

hf download FutureMLS-Lab/flint-NVFP4 --local-dir flint-NVFP4
hf download RecursiveSI/flint --include "sglang/*" --local-dir flint-kit
MODEL=$PWD/flint-NVFP4 bash flint-kit/sglang/serve_b300.sh

Use the chat template and sampling defaults shipped with the checkpoint, as described in the base model card.

License

Apache-2.0, the same license as the base model. See RecursiveSI/flint for the model's intended uses and limitations.

Downloads last month: -

Safetensors

Model size

21B params

Tensor type

BF16

Model tree for FutureMLS-Lab/flint-NVFP4

Base model

RecursiveSI/flint

Quantized

(16)

this model