--- license: other license_name: nvidia-open-model-agreement license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-agreement/ base_model: autotrust/Qwen3.8-Flash-Next-E384-FP4-Local base_model_relation: merge pipeline_tag: image-text-to-text library_name: vllm language: - en - zh tags: - qwen3.8 - moe - nvfp4 - modelopt - mtp - rl - grpo --- # Qwen3.8-Flash-Next-E384-FP4-V2 A compact build of [nvidia/Qwen3.8-Flash-Next-FP4-Local](https://huggingface.co/nvidia/Qwen3.8-Flash-Next-FP4-Local) found by **Neural Architecture Search (NAS)** over the routed-expert configuration: it keeps **384 of the 512 routed experts** in every decoder MoE layer and in the MTP layer. Unchanged from the original: - top-10 routing and the shared experts; - attention, the n-gram PLE table, the vision tower and the tokenizer. This is an unofficial derivative, not produced or endorsed by Alibaba or NVIDIA. | | original (512 experts) | **E384** | |---|---|---| | weights | 102.14 GB = 95.12 GiB | **85.66 GB = 79.78 GiB** (−15.35 GiB, −16.1 %) | | routed experts / layer (decoder + MTP) | 512 | 384 | | active experts / token | 10 + shared | 10 + shared | Full merged checkpoint of the E384 FP4 model after GRPO-style reinforcement learning with a LoRA adapter. **This is a complete model, not an adapter**: all 21 weight shards, the tokenizer, configs and MTP files are included. It is an unofficial derivative and is not endorsed by Alibaba or NVIDIA. ## Serving Validated with vLLM 0.31 on one B200. The included `vllm_patch/sitecustomize.py` is needed for this checkpoint's FP4/CSF/PLE weight layout. From this directory: ```bash export PYTHONPATH="$PWD/vllm_patch${PYTHONPATH:+:$PYTHONPATH}" vllm serve . --served-model-name e384-rl --port 8000 \ --tensor-parallel-size 1 --dtype bfloat16 --kv-cache-dtype fp8 \ --quantization modelopt_mixed --mamba-cache-mode align \ --enable-prefix-caching --enable-chunked-prefill \ --max-model-len 72000 --max-num-seqs 128 --gpu-memory-utilization 0.90 \ --reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice ``` `serve_vllm.sh` is the equivalent portable launcher. The model was benchmarked through this serving path, including the FP8 KV cache and the Qwen reasoning/tool parsers. ## Training and checkpoint contents - Three GRPO-style LoRA updates with verifiable rewards; the final round targets short-budget CVSS completion and structured findings emission. - LoRA rank 32, alpha 64. The routed experts, router, shared expert and PLE are frozen. - The GDN projection deltas are folded at full precision and those projections are stored as BF16; the corresponding entries are removed from `quantization_config.quantized_layers`. Other model tensors retain the base checkpoint format. - The final adapter was merged into the weights before this repository was created. No adapter loading step is needed. ## Full benchmark results Same benchmark harness and serving settings for all rows. These are single runs; GPQA has only 198 questions and is correspondingly noisy. AIME 2026 uses 30 problems x 4 samples with xhigh reasoning and a 64k generation budget. Base numbers are the previously published reference numbers, not a rerun. | benchmark | base reference | this model | |---|---:|---:| | GPQA-Diamond (low, 32k) | 77.3 | **75.76** | | MMLU (14,042) | 86.9 | **87.37** | | C-Eval val (1,346) | 89.0 | **89.97** | | AIME 2026 (xhigh, 64k) | 93.3 | **95.0** | | HumanEval (164, greedy) | 97.6 | **98.78** | | MMMU val (900) | 79.4 | **78.22** | | tool calling (800) | — | **77.88** | Detailed per-task counts and truncation statistics are in `benchmark_results.json`. ### AIME reasoning length Compared with the preceding RL checkpoint on the same 120 AIME samples, total completion tokens fell 2.05% (19,897 to 19,490 per answer on average), while accuracy rose from 92.5 to 95.0. The median was essentially unchanged (16,528 to 16,675 tokens); the reduction came mainly from the long tail (P90 58,272 to 45,339) and fewer 65,536-token truncations (9 to 6). `completion_tokens` includes both the reasoning trace and the final answer. ## Behavior notes and limitations On the same training-prompt evaluation used during the final RL round, CVSS3 completion within a 2,200 token budget rose from 39% to 70%, CVSS4 from 18% to 60%, and structured-findings emission reward from 0.609 to 0.917. Math and guard rewards did not degrade in that evaluation. These are training-prompt measurements, not an independent behavior benchmark. **CVSS arithmetic is still not reliable**: among completed answers, exact agreement with the reference implementation was about 4.5% for CVSS3 and 4.2% for CVSS4 in that evaluation. This checkpoint should not be described as having fully fixed CVSS scoring. See `behavior_results.json` for the detailed comparison.