|
Download README.md from ctech123/ctech-fle-gt: direct link, hf CLI and curl.
- Browser
- Download file 4.89 kB
-
https://huggingface.co/ctech123/ctech-fle-gt/resolve/main/README.md
- Command line
-
hf download hf://ctech123/ctech-fle-gt/README.md
-
curl -L -o README.md https://huggingface.co/ctech123/ctech-fle-gt/resolve/main/README.md
4.89 kB
| license: other | |
| license_name: nvidia-open-model-agreement | |
| license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-agreement/ | |
| base_model: autotrust/Qwen3.8-Flash-Next-E384-FP4-Local | |
| base_model_relation: merge | |
| pipeline_tag: image-text-to-text | |
| library_name: vllm | |
| language: | |
| - en | |
| - zh | |
| tags: | |
| - qwen3.8 | |
| - moe | |
| - nvfp4 | |
| - modelopt | |
| - mtp | |
| - rl | |
| - grpo | |
| # Qwen3.8-Flash-Next-E384-FP4-V2 | |
| A compact build of [nvidia/Qwen3.8-Flash-Next-FP4-Local](https://huggingface.co/nvidia/Qwen3.8-Flash-Next-FP4-Local) found by | |
| **Neural Architecture Search (NAS)** over the routed-expert configuration: it keeps **384 of the 512 routed experts** in every | |
| decoder MoE layer and in the MTP layer. | |
| Unchanged from the original: | |
| - top-10 routing and the shared experts; | |
| - attention, the n-gram PLE table, the vision tower and the tokenizer. | |
| This is an unofficial derivative, not produced or endorsed by Alibaba or NVIDIA. | |
| | | original (512 experts) | **E384** | | |
| |---|---|---| | |
| | weights | 102.14 GB = 95.12 GiB | **85.66 GB = 79.78 GiB** (−15.35 GiB, −16.1 %) | | |
| | routed experts / layer (decoder + MTP) | 512 | 384 | | |
| | active experts / token | 10 + shared | 10 + shared | | |
| Full merged checkpoint of the E384 FP4 model after GRPO-style reinforcement learning with a LoRA | |
| adapter. **This is a complete model, not an adapter**: all 21 weight shards, the tokenizer, configs and | |
| MTP files are included. It is an unofficial derivative and is not endorsed by Alibaba or NVIDIA. | |
| ## Serving | |
| Validated with vLLM 0.31 on one B200. The included `vllm_patch/sitecustomize.py` is needed for this | |
| checkpoint's FP4/CSF/PLE weight layout. From this directory: | |
| ```bash | |
| export PYTHONPATH="$PWD/vllm_patch${PYTHONPATH:+:$PYTHONPATH}" | |
| vllm serve . --served-model-name e384-rl --port 8000 \ | |
| --tensor-parallel-size 1 --dtype bfloat16 --kv-cache-dtype fp8 \ | |
| --quantization modelopt_mixed --mamba-cache-mode align \ | |
| --enable-prefix-caching --enable-chunked-prefill \ | |
| --max-model-len 72000 --max-num-seqs 128 --gpu-memory-utilization 0.90 \ | |
| --reasoning-parser qwen3 --tool-call-parser qwen3_xml --enable-auto-tool-choice | |
| ``` | |
| `serve_vllm.sh` is the equivalent portable launcher. The model was benchmarked through this serving | |
| path, including the FP8 KV cache and the Qwen reasoning/tool parsers. | |
| ## Training and checkpoint contents | |
| - Three GRPO-style LoRA updates with verifiable rewards; the final round targets short-budget CVSS | |
| completion and structured findings emission. | |
| - LoRA rank 32, alpha 64. The routed experts, router, shared expert and PLE are frozen. | |
| - The GDN projection deltas are folded at full precision and those projections are stored as BF16; | |
| the corresponding entries are removed from `quantization_config.quantized_layers`. Other model | |
| tensors retain the base checkpoint format. | |
| - The final adapter was merged into the weights before this repository was created. No adapter loading | |
| step is needed. | |
| ## Full benchmark results | |
| Same benchmark harness and serving settings for all rows. These are single runs; GPQA has only 198 | |
| questions and is correspondingly noisy. AIME 2026 uses 30 problems x 4 samples with xhigh reasoning and | |
| a 64k generation budget. Base numbers are the previously published reference numbers, not a rerun. | |
| | benchmark | base reference | this model | | |
| |---|---:|---:| | |
| | GPQA-Diamond (low, 32k) | 77.3 | **75.76** | | |
| | MMLU (14,042) | 86.9 | **87.37** | | |
| | C-Eval val (1,346) | 89.0 | **89.97** | | |
| | AIME 2026 (xhigh, 64k) | 93.3 | **95.0** | | |
| | HumanEval (164, greedy) | 97.6 | **98.78** | | |
| | MMMU val (900) | 79.4 | **78.22** | | |
| | tool calling (800) | — | **77.88** | | |
| Detailed per-task counts and truncation statistics are in `benchmark_results.json`. | |
| ### AIME reasoning length | |
| Compared with the preceding RL checkpoint on the same 120 AIME samples, total completion tokens fell | |
| 2.05% (19,897 to 19,490 per answer on average), while accuracy rose from 92.5 to 95.0. The median was | |
| essentially unchanged (16,528 to 16,675 tokens); the reduction came mainly from the long tail (P90 | |
| 58,272 to 45,339) and fewer 65,536-token truncations (9 to 6). `completion_tokens` includes both the | |
| reasoning trace and the final answer. | |
| ## Behavior notes and limitations | |
| On the same training-prompt evaluation used during the final RL round, CVSS3 completion within a 2,200 | |
| token budget rose from 39% to 70%, CVSS4 from 18% to 60%, and structured-findings emission reward from | |
| 0.609 to 0.917. Math and guard rewards did not degrade in that evaluation. These are training-prompt | |
| measurements, not an independent behavior benchmark. | |
| **CVSS arithmetic is still not reliable**: among completed answers, exact agreement with the reference | |
| implementation was about 4.5% for CVSS3 and 4.2% for CVSS4 in that evaluation. This checkpoint should not | |
| be described as having fully fixed CVSS scoring. See `behavior_results.json` for the detailed comparison. | |