Fix expert_swiglu_limit_list in modeling

#10

Fixes the missing SwiGLU clamps in the HF modeling code to align with SGLang, addressing discussion #9.
Thanks for the detailed investigation and helpful feedback


Smoke Test Results: Using the full Ling-3.0-flash checkpoint, we compared the original implementation, the new implementation, and the new implementation with clipping disabled.

  • Backward compatibility: With clipping disabled, generated token IDs, prefill logits, and fixed-input cached-decode logits matched the original implementation exactly (maximum difference: 0).
  • Clipping takes effect: Clipping was triggered in 189 of 1,981 sampled MLPs during inference. The maximum same-input prefill-logit difference was 7.84375.
  • Numerical correctness: Captured MLP outputs matched the expected SwiGLU clipping formula with maximum error: 0. Per-layer thresholds, shared-expert configuration, and out-of-range MTP behavior passed validation.
  • Inference coverage: All 9 full-model inference runs and 13 component checks passed. All three variants returned 579 for the arithmetic case.
michael-qiu changed pull request status to open
michael-qiu changed pull request status to merged

Sign up or log in to comment