# SenseNova-Loop-9B NVFP4 + MTP on GB10 (sm_121): compatibility scout, 2026-10-30

Labels: **[V]** verified here (code read at a pinned SHA, or a command run on tux). **[I]** inferred, not yet run.
Top criterion (added by the user): **MTP speculative decoding plus an NVFP4 target in one runtime.**

## Ranking
| # | Path | GEMM | MTP | Status |
|---|---|---|---|---|
| 1 | **SGLang PR #47281 overlay + 2 small patches** on `lmsysorg/sglang:nightly-dev-cu13-20261029-c10e0e17` | **W4A4** flashinfer `mm_fp4` (cutlass/cudnn) | NEXTN via the stock `Qwen3_5ForCausalLMMTP`, BF16 draft (needs patch 2) | best; patches untested end-to-end |
| 2 | Same as #1 with `--fp4-gemm-backend marlin` | **W4A16** Marlin NVFP4 | same | fallback if W4A4 accuracy or kernels misbehave |
| 3 | vLLM 0.27.1 (local image) + out-of-tree sensenova_loop | W4A4 CUTLASS works on sm121 [V] | must port loop **and** wire MTP | big effort |
| 4 | LightLLM `main` | **no NVFP4/ModelOpt** (awq/w8a8/fp8 only) [V] | has sensenova_loop + MTP (`eagle_with_att`) [V] | BF16/FP8 reference only; fa3 defaults are Hopper-only |
| 5 | HF transformers + custom NVFP4 Linear | W4A4 `torch._scaled_mm` fp4 or flashinfer [V] | **none**: we would write draft+verify ourselves | correctness oracle only |

## 1. SGLang PR #47281 [V]
- Open, unmerged, CI red. Fork `WANDY666/sglang` branch `feat/sensenova-loop`. **Head `336c54ac1e721197806d94b6929f567446a59693`.**
  Merge base `c10e0e17735f7981c7cc46891eed6908b7701e58`, which is exactly the commit of nightly `nightly-dev-cu13-20261029-c10e0e17`, so the overlay is exact.
  Validated by the author on H800 BF16 only (logit cosine 0.984 to 0.9999 vs HF). **No MTP and no quantized test.**
- Overlay = 7 runtime files (in `overlay/python/sglang/`): `srt/models/sensenova_loop.py` (new), `srt/configs/sensenova_loop.py` (new),
  `srt/configs/{__init__,model_config}.py`, `srt/layers/rotary_embedding/mrope_rope_index.py`,
  `srt/multimodal/processors/qwen_vl.py`, `srt/utils/hf_transformers/common.py`. They need the very new `layer_boundary.MHCState`,
  `sglang.kernels.ops.layernorm.mhc`, and `runtime_context`, so **don't overlay onto release images ≤ v0.5.22**.
- Loop: the config expands to 56 execution layers (0..3, 4..27×2, 28..31). The KV and GDN pools are per execution layer. Re-visits are
  `copy.copy` of the physical layer with new RadixAttention/RadixLinearAttention objects (layer_id = execution index) and shared params.
  `load_weights` remaps `model.language_model.layers.{phys}.<any suffix>` to the first execution index, so weight_scale, weight_scale_2 and input_scale
  need no extra mapping. mHC routers are raw FP32 Parameters with F.linear and are never quantized.
- **Patch 1, `patches/sensenova_loop_quant.patch` (required for NVFP4):** `get_layer()` builds decoder layers **without
  `quant_config`**, so every decoder Linear would be BF16 and packed uint8 weights fail to load. Pass `quant_config=quant_config`. Also build
  the physical layer with prefix `...layers.{physical_id}`, because ModelOpt `exclude_modules` uses physical names and layers 28..31 run at
  execution ids 52..55 (otherwise their `in_proj_a/b` would not be excluded). Alternative: rewrite exclude_modules to globs
  (`*linear_attn.in_proj_a*`, `*linear_attn.in_proj_b*`, `lm_head`, `*visual*`, `mtp*`).
- Base code that already handles NVFP4 [V]: the GDN packed loaders cover weight/weight_scale/input_scale. Scalar `weight_scale_2` with a tuple shard
  is broadcast by `MergedColumnParallelLinear.weight_loader_v2`. The BF16 fused in_proj fast-path turns itself off when quantized.
- FP4 backend: `initialize_fp4_gemm_config()` picks `auto`, which resolves to cutedsl on sm100, marlin on 8.0 ≤ cc < 10, and **`flashinfer_cutlass`** otherwise (sm121).

## 2. MTP (top criterion)
- (a) **The PR does not implement MTP.** The draft-arch rewrite in `configs/model_config.py` (Qwen3.5 list, ~L1169) does not include
  `SenseNovaLoopForConditionalGeneration`, so a NEXTN draft would instantiate the whole loop model [V]. Even with the arch added,
  `SenseNovaLoopTextConfig.layers_block_type` follows `layer_types` (56 entries, [0] = linear_attention), so the single MTP layer would be
  built as GDN instead of full attention [V, code]. **Patch 2, `patches/model_config_mtp_draft.patch`:** for draft + SenseNovaLoop, deep-copy the
  config, set `num_hidden_layers=1`, `layer_types=["full_attention"]`, `full_attention_interval=1`, `execution_plan=(0,)`, `mhc=None`,
  `num_nextn_predict_layers=1`, and arch `Qwen3_5ForCausalLMMTP`. This mirrors the existing Qwen4Exp block. [Patch written, not run.]
- Draft class `Qwen3_5ForCausalLMMTP` (`models/qwen3_5_mtp.py`) loads names `mtp.*` (fc, pre_fc_norm_embedding/hidden,
  layers.0.{self_attn,mlp,norms}, norm). Upstream `mtp.safetensors` has exactly these 15 BF16 tensors [V], and `embed_tokens` and
  `lm_head` are shared from the target via `get_embed_and_head`. The draft takes the target's final-norm hidden state. The PR's loop forward returns
  `final_norm(...)`, the same as Qwen3.5 [V code]. Whether SenseNova trained MTP on that same tensor is [I] (the README says MTP is "Available").
- Flags [I, standard SGLang]: `--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1
  --speculative-num-draft-tokens 4` (the draft model path defaults to the target). GDN spec-verify state is per layer_id, so visits are distinct [I].
- (b) NVFP4 target + BF16 draft: `_mtp_quant_config()` returns **None** for `modelopt_fp4` with a serialized NVFP4 checkpoint, so the draft is built BF16 with
  no patch [V]. A draft-vs-main quantization mismatch is auto-resolved for draft models [V]. Keep `mtp*` in exclude_modules regardless.
- Checkpoint layout SGLang expects: `mtp.*` keys must be **listed in `model.safetensors.index.json`**. The loader iterates the index. The
  "add unindexed mtp.safetensors" hack is GLM-only. Upstream already indexes them into `mtp.safetensors` (871 keys) [V]. ModelOpt
  `export_hf_checkpoint` carries unplaced weights such as MTP heads into the export and adds them to exclusions [V code]. **After export, verify that `mtp.*` (15 keys)
  is in the new index. If not, copy `mtp.safetensors` and add its keys to the index.**
- (c) LightLLM: has `lightllm/models/sensenova_loop` and `test/start_scripts/sensenova_loop/sensenova_loop_mtp.sh`
  (`--mtp_mode eagle_with_att --mtp_step 3 --mtp_draft_model_dir <separate dir>`, fa3 attention) at main `ccf4e33d0ce9…`, but there is
  no FP4 quantization module in `lightllm/common/quantization`, so it does not meet the NVFP4 requirement. Useful as an MTP acceptance-rate reference on another GPU.
- (d) The HF runtime cannot use `mtp.*` (ignored keys), so draft and verify would have to be written by hand. It ranks last.

## 3. Images [V]
- Local: `vllm/vllm-openai:v0.27.1-aarch64` and `...-dflash2-nvfp4-hotfix-aarch64` (vllm 0.27.1, torch 2.13.0+cu130, flashinfer
  0.6.16.post3, transformers 5.15.0). No local SGLang image or venv (`~/sglang-cache` holds only JIT caches; `~/venvs` holds vllm025 and vllm-026).
- Pulled (the 1 allowed pull, since it is the exact PR base): `lmsysorg/sglang:nightly-dev-cu13-20261029-c10e0e17`,
  index `sha256:78142f9f4b7fb3ee27f59065615dc7bb263f643ed256c9299c3861086ade4d47`, arm64
  `sha256:5efa08069f2822f313fc3e5764ce72238f72cb9ca8744e9fa0f46358fd2dd643` (15.7 GB). Its pyproject pins flashinfer 0.7.2.

## 4. vLLM 0.27.1 [V facts / I effort]
Has `Qwen3_5ForConditionalGeneration` and `Qwen3_5MTP`, plus the NVFP4 linear kernels cutlass (works on sm121), marlin and emulation (flashinfer cutedsl is sm10x only).
A loop port must give each visit distinct Attention/GDN instances (vLLM keys KV and mamba state by layer prefix) and make `Qwen3_5MTP` see
the loop's output. That is about 1 to 2 days with CUDA-graph and prefix-cache risk. Not recommended while SGLang is viable.

## 5. Kernel smoke tests on GB10 [V]
`docker run --rm --gpus all -v $PWD:/w:ro --entrypoint python3 vllm/vllm-openai:v0.27.1-aarch64 /w/smoke_fp4.py` (and `smoke_fp4_b.py`).
Peak GPU memory 141 MB. N = K = 4096, ModelOpt scale convention (gs = 448·6/amax, alpha = 1/(gx·gw)).
```
flashinfer mm_fp4: cutlass OK, cudnn OK, auto OK; trtllm & cute-dsl -> BackendSupportedError (capability 121)
kernel vs emulated-FP4 reference rel_err 0.00166 (cutlass & cudnn)   [FP4 vs BF16 noise on random data: 0.134]
M=128 : cutlass 44us (98 TFLOPS)  cudnn 73us
M=2048: cutlass 225us (306 TFLOPS) cudnn 221us | bf16 torch 779us
M=1   : cutlass 1066us (!)  cudnn 63us        -> decode: prefer --fp4-gemm-backend flashinfer_cudnn (or autotune)
vllm ops.cutlass_scaled_fp4_mm OK; torch._scaled_mm(float4_e2m1fn_x2, e4m3 swizzled scales) OK (identical results)
flashinfer mm_bf16_fp4 (W4A16): unsupported on cc121 (cutlass/trtllm); cudnn path needs cuDNN >= 9.23.1 (image 9.20)
```
SGLang-image checks (PR unit tests with overlay, flashinfer 0.7.2 mm_fp4, Marlin W4A16, config expansion): `./run_sglang_smoke.sh`.
**NOT YET RUN:** the image pull was still in progress at hand-off (`pull.log`; pid of `docker pull` still alive). Run the script once it finishes.

## 6. ModelOpt [V unless marked]
- PyPI stable **0.47.0** pins `transformers<5.15` and is **incompatible** with 5.18.0. **0.48.0rc1** (2026-10-19) pins `>=5.5,<5.19`, so use it in a fresh venv.
- `default_disabled_quantizers.yaml` disables `*linear_attn.in_proj_a*`, `*linear_attn.in_proj_b*`, `*linear_attn.conv1d*`, `*lm_head*`,
  `*visual*`, `*vision_tower*`, `mtp.*`, `*router*`, `*mlp.gate.*`, and Embedding. GDN-state quantizers are off by default.
  `NVFP4_DEFAULT_CFG` = max calibration with w4a4_nvfp4_nvfp4. mhc_* are not nn.Linear, so they are not wrapped.
- Modules visited twice: quantizers are per module, and max calibration takes the running max over all calls, so both visits are covered [I]. Export walks each physical module once.
- Remote code: `hf_model_type` is only a key for MoE specs and special cases, and the dense path is generic. It copies `*.py`. The resmooth step does `model(torch.ones(1,2))` [I: fine, text-only].
- Output: `hf_quant_config.json` `{"producer":{"name":"modelopt",...},"quantization":{"quant_algo":"NVFP4","group_size":16,
  "kv_cache_quant_algo":null,"exclude_modules":[prefix-wildcards, physical names]}}`, plus config.json `quantization_config.quant_method="modelopt"`.
  Tensors: `weight` uint8 [N,K/2], `weight_scale` e4m3 [N,K/16], `weight_scale_2` fp32, `input_scale` fp32.
- NVIDIA playbook (build.nvidia.com/station/nvfp4-quantization): Spark uses `nvcr.io/nvidia/tensorrt-llm/release:spark-single-gpu-dev` with ModelOpt
  0.41.0 (`examples/llm_ptq/scripts/huggingface_example.sh --quant nvfp4 --tasks quant`) and `trtllm-serve`. Station uses vllm 25.12.post1 with ModelOpt 0.47.0.
  Both are too old for transformers 5.18, so use them only as a pattern.

## 7. Serve command (recipe, [I])
```
docker run --rm --gpus all --ipc=host -p 30000:30000 -v $OUT:/model:ro <7 overlay -v mounts, see run_sglang_smoke.sh> \
  lmsysorg/sglang:nightly-dev-cu13-20261029-c10e0e17 python3 -m sglang.launch_server --model-path /model \
  --quantization modelopt_fp4 --fp4-gemm-backend flashinfer_cudnn --mem-fraction-static 0.5 --port 30000 \
  --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4
```
Port 30000 avoids 8000/3000/18080-18105.

## 8. Risks
- The PR is unmerged with CI red and BF16-only validation. Patches 1 and 2 are new and unrun against real weights. MTP on the loop model in SGLang is untested by anyone [I].
- The physical-prefix change only renames layers 28..31. Anything that parses the layer id from the prefix would see the physical id [I].
- GDN triton/fla kernels and spec-decode GDN state tracking on sm121 have not been exercised. M=1 cutlass mm_fp4 is slow, so benchmark cudnn.
- Looping doubles the use of every quantized weight. Keep mHC, in_proj_a/b, lm_head, vision and MTP in BF16, and evaluate against BF16 (and measure MTP acceptance against BF16).