Pilcothink/Raon-1-nano-NVFP4
I quantized KRAFTON's Raon-1-nano, a Korean–English language model, to NVFP4 using post-training quantization (PTQ). All 24 MLP blocks use 4-bit weights and activations. The per-layer embedding (PLE) table uses 4-bit weights. The three safetensors weight files total 3.79 GB. Base model · My Hugging Face profile
Quantization coverage
| Component | Storage and inference |
|---|---|
MLP gate_proj, up_proj, down_proj |
NVFP4 W4A4. Each block of 16 E2M1 weights has an E4M3 scale, with an additional FP32 global scale. The gate and up projections share a global scale so vLLM can fuse them. |
| Per-layer embedding table | Weight-only NVFP4. At inference time the loader gathers the rows for the input token IDs and dequantizes those rows to BF16. |
Input embeddings and lm_head |
Share BF16 weights. |
| RDA and MLA attention, mHC, normalization | Retain the weight precision of the base checkpoint. |
PTQ calibration data
I used NVIDIA Model Optimizer for PTQ, without quantization-aware distillation (QAD) or further fine-tuning. I chose the three public datasets below and sampled conversations in a 3:2:1 ratio to cover Korean question answering, English dialogue, and function calling. This calibration mix is my choice for this release. Model Optimizer source revision
| Calibration dataset (pinned revision) | Source examples | Sampling ratio by conversation | Conversations used for PTQ | Input tokens after chat formatting |
|---|---|---|---|---|
| beomi/KoAlpaca-v1.1a | Korean questions and answers from train |
3/6 | 256 (50.00%) | 45,893 (16.76%) |
| HuggingFaceH4/ultrachat_200k | English conversations from train_sft |
2/6 | 181 (35.35%) | 166,061 (60.65%) |
| NousResearch/hermes-function-calling-v1 | Tool definitions, user requests, and function calls from func-calling-singleturn.json |
1/6 | 75 (14.65%) | 61,870 (22.59%) |
| Total | 6/6 | 512 (100%) | 273,824 (100%) |
For every six conversations sampled, the pipeline drew three from KoAlpaca, two from UltraChat, and one from Hermes. It removed duplicate user prompts, shuffled with seed 42, then split the data into 512 calibration, 1,024 training, and 128 evaluation conversations. Only the calibration split was used for this checkpoint. With the original Raon tokenizer and chat template, the longest conversation was 3,502 tokens, so all 512 fit within the 4,096-token calibration limit. UltraChat accounts for 35.35% of the conversations and 60.65% of the tokens because its dialogues are longer.
Serving
I’ll publish the serving command together with the Docker image once it’s optimized for this checkpoint. The loader must support both nano_nvfp4_recipe=mlp-w4a4-v1 and nano_ple_format=nvfp4-e2m1-v1 in config.json.
BF16 and NVFP4 evaluation protocol
The comparison uses the same examples, sampling parameters, output budget, and concurrency for both checkpoints on all five benchmarks. Temperature and presence penalty follow KRAFTON's general-use recommendations. Raon sampling recommendations
| Setting | Value for this comparison |
|---|---|
| Hardware and software | One NVIDIA GeForce RTX 5090 (SM120a), driver 595.99.02, vLLM 0.34.0rc0, FlashInfer 0.7.3 |
| Serving | Each checkpoint runs on a single GPU with BF16 compute, TRITON_MLA, and eager execution. The quantized MLP uses --linear-backend b12x. |
| Context and scheduling | max_model_len=65536, max_num_seqs=500, max_num_batched_tokens=4096, gpu_memory_utilization=0.75. Prefix caching and FlashInfer autotuning are disabled. |
| Sampling | temperature=0.1, presence_penalty=1.5, max_tokens=8192, seed=42. The output budget includes reasoning and the final answer. Requests use the engine defaults for top-p and top-k. |
| Chat template and parsing | enable_thinking=true, truncate_history_thinking=false, raon_1 reasoning and tool parsers, automatic tool choice enabled |
| Client | Up to 250 concurrent requests. Each completed task frees a slot for the next item. Request timeout is 3,600 seconds. |
| Benchmark | Data and scoring |
|---|---|
| KMMLU-Redux | Zero-shot generation on 2,587 Korean four-choice questions. The evaluator extracts A–D from the final answer. An unparseable answer counts as incorrect. |
| MMLU-Redux 2.0 | Zero-shot generation across 57 subjects in English. EvalScope 1.15.0 scores accuracy using the Redux answer corrections. |
| IFBench | EvalScope 1.15.0 scores the final answer. Prompt-level loose and strict accuracy are reported separately. |
| PersonaMem v2 | The 5,000 text multiple-choice questions use their full 32K conversation histories. Choice order follows the authors' per-item shuffle with PYTHONHASHSEED=42. |
| BFCL v4 | EvalScope 1.15.0 with Berkeley bfcl-eval==2025.10.27.1 and native function calling. Scores are grouped by non-live calls, live calls, irrelevance, and multi-turn tasks. live_relevance is diagnostic. |
The 17 selected BFCL categories are simple_python, simple_java, simple_javascript, multiple, parallel, parallel_multiple, irrelevance, live_simple, live_multiple, live_parallel, live_parallel_multiple, live_irrelevance, live_relevance, multi_turn_base, multi_turn_miss_func, multi_turn_miss_param, and multi_turn_long_context. A request adapter forwards the shared sampling settings and output limit to the BFCL handler.
Evaluation results
Scores use a 0–100 scale. The last column gives the NVFP4 score minus the BF16 score in percentage points. Results are pending.
| Benchmark | Evaluator | Metric | BF16 base | NVFP4 PTQ | Difference (pp) |
|---|---|---|---|---|---|
| KMMLU-Redux | Custom card_mcq.py (kmmlu-redux) |
Accuracy | |||
| MMLU-Redux 2.0 | EvalScope 1.15.0 (mmlu_redux) |
Accuracy | |||
| IFBench | EvalScope 1.15.0 (ifbench) |
Prompt-level loose accuracy | |||
| IFBench | EvalScope 1.15.0 (ifbench) |
Prompt-level strict accuracy | |||
| PersonaMem v2 · 32K | Custom card_mcq.py (personamem-v2-32k) |
Text MCQ accuracy | |||
| BFCL v4 · Non-live | EvalScope 1.15.0 + Berkeley bfcl-eval 2025.10.27.1 |
Function-calling accuracy | |||
| BFCL v4 · Live | EvalScope 1.15.0 + Berkeley bfcl-eval 2025.10.27.1 |
Function-calling accuracy | |||
| BFCL v4 · Irrelevance | EvalScope 1.15.0 + Berkeley bfcl-eval 2025.10.27.1 |
Correctly avoiding unnecessary calls (%) | |||
| BFCL v4 · Multi-turn | EvalScope 1.15.0 + Berkeley bfcl-eval 2025.10.27.1 |
Multi-turn accuracy |
License
KRAFTON created the base model. This release includes the quantized weights and modified quantization configuration, the original model code and tokenizer, and the Apache License 2.0. License text
- Downloads last month
- 60
Model tree for Pilcothink/Raon-1-nano-NVFP4
Base model
KRAFTON/Raon-1-nano