ctech-fle-gt / README.md
ctech123's picture
Duplicate from ctech123/Qwen3.8-Flash-Next-E384-FP4-V2
d07994b
|
Raw History Blame Contribute Delete
4.89 kB
---
license: other
license_name: nvidia-open-model-agreement
license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-agreement/
base_model: autotrust/Qwen3.8-Flash-Next-E384-FP4-Local
base_model_relation: merge
pipeline_tag: image-text-to-text
library_name: vllm
language:
- en
- zh
tags:
- qwen3.8
- moe
- nvfp4
- modelopt
- mtp
- rl
- grpo
---
# Qwen3.8-Flash-Next-E384-FP4-V2
A compact build of [nvidia/Qwen3.8-Flash-Next-FP4-Local](https://huggingface.co/nvidia/Qwen3.8-Flash-Next-FP4-Local) found by
**Neural Architecture Search (NAS)** over the routed-expert configuration: it keeps **384 of the 512 routed experts** in every
decoder MoE layer and in the MTP layer.
Unchanged from the original:
- top-10 routing and the shared experts;
- attention, the n-gram PLE table, the vision tower and the tokenizer.
This is an unofficial derivative, not produced or endorsed by Alibaba or NVIDIA.
| | original (512 experts) | **E384** |
|---|---|---|
| weights | 102.14 GB = 95.12 GiB | **85.66 GB = 79.78 GiB** (−15.35 GiB, −16.1 %) |
| routed experts / layer (decoder + MTP) | 512 | 384 |
| active experts / token | 10 + shared | 10 + shared |
Full merged checkpoint of the E384 FP4 model after GRPO-style reinforcement learning with a LoRA
adapter. **This is a complete model, not an adapter**: all 21 weight shards, the tokenizer, configs and
MTP files are included. It is an unofficial derivative and is not endorsed by Alibaba or NVIDIA.
## Serving
Validated with vLLM 0.31 on one B200. The included `vllm_patch/sitecustomize.py` is needed for this
checkpoint's FP4/CSF/PLE weight layout. From this directory:
```bash
export PYTHONPATH="$PWD/vllm_patch${PYTHONPATH:+:$PYTHONPATH}"
vllm serve . --served-model-name e384-rl --port 8000 \
--tensor-parallel-size 1 --dtype bfloat16 --kv-cache-dtype fp8 \
--quantization modelopt_mixed --mamba-cache-mode align \
--enable-prefix-caching --enable-chunked-prefill \
--max-model-len 72000 --max-num-seqs 128 --gpu-memory-utilization 0.90 \
--reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice
```
`serve_vllm.sh` is the equivalent portable launcher. The model was benchmarked through this serving
path, including the FP8 KV cache and the Qwen reasoning/tool parsers.
## Training and checkpoint contents
- Three GRPO-style LoRA updates with verifiable rewards; the final round targets short-budget CVSS
completion and structured findings emission.
- LoRA rank 32, alpha 64. The routed experts, router, shared expert and PLE are frozen.
- The GDN projection deltas are folded at full precision and those projections are stored as BF16;
the corresponding entries are removed from `quantization_config.quantized_layers`. Other model
tensors retain the base checkpoint format.
- The final adapter was merged into the weights before this repository was created. No adapter loading
step is needed.
## Full benchmark results
Same benchmark harness and serving settings for all rows. These are single runs; GPQA has only 198
questions and is correspondingly noisy. AIME 2026 uses 30 problems x 4 samples with xhigh reasoning and
a 64k generation budget. Base numbers are the previously published reference numbers, not a rerun.
| benchmark | base reference | this model |
|---|---:|---:|
| GPQA-Diamond (low, 32k) | 77.3 | **75.76** |
| MMLU (14,042) | 86.9 | **87.37** |
| C-Eval val (1,346) | 89.0 | **89.97** |
| AIME 2026 (xhigh, 64k) | 93.3 | **95.0** |
| HumanEval (164, greedy) | 97.6 | **98.78** |
| MMMU val (900) | 79.4 | **78.22** |
| tool calling (800) | — | **77.88** |
Detailed per-task counts and truncation statistics are in `benchmark_results.json`.
### AIME reasoning length
Compared with the preceding RL checkpoint on the same 120 AIME samples, total completion tokens fell
2.05% (19,897 to 19,490 per answer on average), while accuracy rose from 92.5 to 95.0. The median was
essentially unchanged (16,528 to 16,675 tokens); the reduction came mainly from the long tail (P90
58,272 to 45,339) and fewer 65,536-token truncations (9 to 6). `completion_tokens` includes both the
reasoning trace and the final answer.
## Behavior notes and limitations
On the same training-prompt evaluation used during the final RL round, CVSS3 completion within a 2,200
token budget rose from 39% to 70%, CVSS4 from 18% to 60%, and structured-findings emission reward from
0.609 to 0.917. Math and guard rewards did not degrade in that evaluation. These are training-prompt
measurements, not an independent behavior benchmark.
**CVSS arithmetic is still not reliable**: among completed answers, exact agreement with the reference
implementation was about 4.5% for CVSS3 and 4.2% for CVSS4 in that evaluation. This checkpoint should not
be described as having fully fixed CVSS scoring. See `behavior_results.json` for the detailed comparison.