Pilcothink/Raon-1-nano-NVFP4

I quantized KRAFTON's Raon-1-nano, a Korean–English language model, to NVFP4 using post-training quantization (PTQ). All 24 MLP blocks use 4-bit weights and activations. The per-layer embedding (PLE) table uses 4-bit weights. The three safetensors weight files total 3.79 GB. Base model · My Hugging Face profile

Quantization coverage

Component Storage and inference
MLP gate_proj, up_proj, down_proj NVFP4 W4A4. Each block of 16 E2M1 weights has an E4M3 scale, with an additional FP32 global scale. The gate and up projections share a global scale so vLLM can fuse them.
Per-layer embedding table Weight-only NVFP4. At inference time the loader gathers the rows for the input token IDs and dequantizes those rows to BF16.
Input embeddings and lm_head Share BF16 weights.
RDA and MLA attention, mHC, normalization Retain the weight precision of the base checkpoint.

PTQ calibration data

I used NVIDIA Model Optimizer for PTQ, without quantization-aware distillation (QAD) or further fine-tuning. I chose the three public datasets below and sampled conversations in a 3:2:1 ratio to cover Korean question answering, English dialogue, and function calling. This calibration mix is my choice for this release. Model Optimizer source revision

Calibration dataset (pinned revision) Source examples Sampling ratio by conversation Conversations used for PTQ Input tokens after chat formatting
beomi/KoAlpaca-v1.1a Korean questions and answers from train 3/6 256 (50.00%) 45,893 (16.76%)
HuggingFaceH4/ultrachat_200k English conversations from train_sft 2/6 181 (35.35%) 166,061 (60.65%)
NousResearch/hermes-function-calling-v1 Tool definitions, user requests, and function calls from func-calling-singleturn.json 1/6 75 (14.65%) 61,870 (22.59%)
Total 6/6 512 (100%) 273,824 (100%)

For every six conversations sampled, the pipeline drew three from KoAlpaca, two from UltraChat, and one from Hermes. It removed duplicate user prompts, shuffled with seed 42, then split the data into 512 calibration, 1,024 training, and 128 evaluation conversations. Only the calibration split was used for this checkpoint. With the original Raon tokenizer and chat template, the longest conversation was 3,502 tokens, so all 512 fit within the 4,096-token calibration limit. UltraChat accounts for 35.35% of the conversations and 60.65% of the tokens because its dialogues are longer.

Serving

I’ll publish the serving command together with the Docker image once it’s optimized for this checkpoint. The loader must support both nano_nvfp4_recipe=mlp-w4a4-v1 and nano_ple_format=nvfp4-e2m1-v1 in config.json.

BF16 and NVFP4 evaluation protocol

The comparison uses the same examples, sampling parameters, output budget, and concurrency for both checkpoints on all five benchmarks. Temperature and presence penalty follow KRAFTON's general-use recommendations. Raon sampling recommendations

Setting Value for this comparison
Hardware and software One NVIDIA GeForce RTX 5090 (SM120a), driver 595.99.02, vLLM 0.34.0rc0, FlashInfer 0.7.3
Serving Each checkpoint runs on a single GPU with BF16 compute, TRITON_MLA, and eager execution. The quantized MLP uses --linear-backend b12x.
Context and scheduling max_model_len=65536, max_num_seqs=500, max_num_batched_tokens=4096, gpu_memory_utilization=0.75. Prefix caching and FlashInfer autotuning are disabled.
Sampling temperature=0.1, presence_penalty=1.5, max_tokens=8192, seed=42. The output budget includes reasoning and the final answer. Requests use the engine defaults for top-p and top-k.
Chat template and parsing enable_thinking=true, truncate_history_thinking=false, raon_1 reasoning and tool parsers, automatic tool choice enabled
Client Up to 250 concurrent requests. Each completed task frees a slot for the next item. Request timeout is 3,600 seconds.
Benchmark Data and scoring
KMMLU-Redux Zero-shot generation on 2,587 Korean four-choice questions. The evaluator extracts A–D from the final answer. An unparseable answer counts as incorrect.
MMLU-Redux 2.0 Zero-shot generation across 57 subjects in English. EvalScope 1.15.0 scores accuracy using the Redux answer corrections.
IFBench EvalScope 1.15.0 scores the final answer. Prompt-level loose and strict accuracy are reported separately.
PersonaMem v2 The 5,000 text multiple-choice questions use their full 32K conversation histories. Choice order follows the authors' per-item shuffle with PYTHONHASHSEED=42.
BFCL v4 EvalScope 1.15.0 with Berkeley bfcl-eval==2025.10.27.1 and native function calling. Scores are grouped by non-live calls, live calls, irrelevance, and multi-turn tasks. live_relevance is diagnostic.

The 17 selected BFCL categories are simple_python, simple_java, simple_javascript, multiple, parallel, parallel_multiple, irrelevance, live_simple, live_multiple, live_parallel, live_parallel_multiple, live_irrelevance, live_relevance, multi_turn_base, multi_turn_miss_func, multi_turn_miss_param, and multi_turn_long_context. A request adapter forwards the shared sampling settings and output limit to the BFCL handler.

Evaluation results

Scores use a 0–100 scale. The last column gives the NVFP4 score minus the BF16 score in percentage points. Results are pending.

Benchmark Evaluator Metric BF16 base NVFP4 PTQ Difference (pp)
KMMLU-Redux Custom card_mcq.py (kmmlu-redux) Accuracy
MMLU-Redux 2.0 EvalScope 1.15.0 (mmlu_redux) Accuracy
IFBench EvalScope 1.15.0 (ifbench) Prompt-level loose accuracy
IFBench EvalScope 1.15.0 (ifbench) Prompt-level strict accuracy
PersonaMem v2 · 32K Custom card_mcq.py (personamem-v2-32k) Text MCQ accuracy
BFCL v4 · Non-live EvalScope 1.15.0 + Berkeley bfcl-eval 2025.10.27.1 Function-calling accuracy
BFCL v4 · Live EvalScope 1.15.0 + Berkeley bfcl-eval 2025.10.27.1 Function-calling accuracy
BFCL v4 · Irrelevance EvalScope 1.15.0 + Berkeley bfcl-eval 2025.10.27.1 Correctly avoiding unnecessary calls (%)
BFCL v4 · Multi-turn EvalScope 1.15.0 + Berkeley bfcl-eval 2025.10.27.1 Multi-turn accuracy

License

KRAFTON created the base model. This release includes the quantized weights and modified quantization configuration, the original model code and tokenizer, and the Apache License 2.0. License text

Downloads last month
60
Safetensors
Model size
4B params
Tensor type
F32
·
BF16
·
U8
·
F8_E4M3
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Pilcothink/Raon-1-nano-NVFP4

Quantized
(7)
this model