hy4-flash-concise / README.md
cloudyu's picture
Rename to hy4-flash-concise; add alpha-0.05 results
1f51adb verified
|
Raw History Blame Contribute Delete
7.04 kB
metadata
license: other
license_name: tencent-hy-license
license_link: https://huggingface.co/tencent/Hy4-flash/blob/main/LICENSE
base_model:
  - tencent/Hy4-flash
  - atakans/Hy4-flash-NVFP4
tags:
  - hy4-flash
  - weight-editing
  - weight-space-programming
  - reasoning
  - nvfp4
  - moe
library_name: transformers
pipeline_tag: text-generation

hy4-flash-concise

Hy4-flash that thinks less. Nine shared-expert down-projections of tencent/Hy4-flash are rewritten with a rank-one, gradient-free edit (weight-space programming). There are no gradient steps and no fine-tuning data. The edit comes from the model's own activations and is compiled into nine weight matrices.

Thinking tokens drop by 37–55% on every benchmark we ran. Accuracy also drops, by a little on GPQA and by a meaningful amount on code and instruction following (see below). Use this model when shorter, cheaper reasoning matters more than the last few points of accuracy.

Status: experimental. The root of this repo is the aggressive edit (α = 0.1). A milder edit (α = 0.05) that keeps reasoning and code accuracy within noise is in alpha-0.05/.

What was changed

Base weights atakans/Hy4-flash-NVFP4 (NVFP4 MLPs, FP8 Engram tables, BF16 elsewhere), derived from tencent/Hy4-flash
Edited tensors model.layers.{22,24,…,38}.mlp.shared_experts.down_proj.weight (9 matrices)
Edit W ← W_bf16_original + α · n · d″ uᵀ / ‖u‖², α = 0.1, stored in BF16 (excluded from NVFP4)
Relative size ‖ΔW‖/‖W‖ = 0.56–1.09% per layer
Everything else byte-identical to atakans/Hy4-flash-NVFP4
  • d is the residual-stream direction that separates the last 48 thinking tokens before </think> from the rest of the thinking, measured on 288 MATH-train (level 5) traces. d″ corrects d for the FFN post-norm (combine_output_norm), so after that RMSNorm the written vector points along d.
  • u is the mean input to the shared-expert down-projection on thinking tokens, and n is the mean norm of the FFN output before the post-norm.
  • The shared expert runs on every token, so a rank-one term in its down-projection acts like a constant steering vector, compiled into the weights.
  • Directions, layers and α were chosen on MATH train only. No evaluation benchmark was used for selection.

The direction tensors and scripts are in edit/.

Results

Setup: vLLM, this checkpoint against the matched base (the same nine matrices in BF16, unedited), thinking on (reasoning_effort: high), temperature 0.9, max 32k tokens, one H200. Δ is paired by problem, with a 95% bootstrap CI.

Benchmark Base concise (α=0.1) Δ accuracy [95% CI] Δ thinking tokens [95% CI]
GPQA Diamond (198 × 2) 91.2 88.6 −2.5 [−5.3, +0.3] −39.6% [−43.3, −35.6]
LiveCodeBench v6 (175, 2025-01…04) 84.6 78.3 −6.3 [−12.0, −0.6] −36.8% [−43.3, −30.3]
IFBench (300, loose) 76.7 66.0 −10.7 [−16.0, −5.3] −54.8% [−58.3, −51.0]
13 coding/math/science tasks × 3 (held-out) 39/39 38/39 −2.6 −54.2%
MATH train screen (64 × 2) 92.2 89.8 −2.3 [−6.2, +0.8] −40.2% [−47.8, −29.9]

Mean thinking tokens, base → concise: GPQA 5448 → 3291, LCB 8564 → 5409, IFBench 5275 → 2382, held-out tasks 5415 → 2482.

Milder version: alpha-0.05/

Benchmark Base α=0.05 Δ accuracy [95% CI] Δ thinking tokens [95% CI]
GPQA Diamond (198 × 2) 91.2 89.6 −1.5 [−3.8, +0.5] −20.6% [−25.6, −15.7]
LiveCodeBench v6 (175) 84.6 82.3 −2.3 [−8.0, +3.4] −25.0% [−31.2, −18.8]
IFBench (300, loose) 76.7 70.3 −6.3 [−11.0, −2.0] −36.6% [−40.8, −31.7]
13 held-out tasks × 3 39/39 37/39 −5.1 −29.1%

Which one to pick: choose alpha-0.05 if accuracy on reasoning and code must hold. Choose the root (α = 0.1) for the largest saving. Both lose points on strict instruction following (IFBench).

Control. A norm-matched random edit (same layers, same ‖ΔW‖, random direction orthogonal to d) on the MATH screen gives +0.0 points of accuracy and +8% tokens. The token saving comes from the direction, not from the perturbation.

Dose response (MATH screen): α = 0.05 → −23% tokens; α = 0.075 → −32%; α = 0.1 → −40%; α = 0.3 → answers become wrong; α = 1.0 → the model loops on </think>. Do not raise α.

Usage

vLLM (Hopper / Blackwell)

Hy4-flash needs the vllm/vllm-openai:hy4-flash image. NVFP4 expert loading needs the vllm/models/hy_v4_flash/nvidia/model.py from atakann/dgx-spark-recipes (copy included as edit/vllm_hy_v4_flash_model.py). On DGX Spark (SM120), use that recipe's full image.

vllm serve autotrust/hy4-flash-concise --served-model-name hy4-flash \
  --dtype bfloat16 --max-model-len 40960 --max-num-seqs 64 \
  --reasoning-parser hy_v4 --limit-mm-per-prompt '{"image":0,"video":0}'
# milder version (local path):
hf download autotrust/hy4-flash-concise --include "alpha-0.05/*" --local-dir hy4-flash-concise
vllm serve hy4-flash-concise/alpha-0.05 --served-model-name hy4-flash --reasoning-parser hy_v4 ...

On one H200: 97 GiB of weights on the GPU, Engram tables in host RAM, about 1.5k tok/s aggregate decode at 100 concurrent requests.

llama.cpp (GGUF, e.g. DGX Spark)

gguf-lora/hy4-flash-concise-lora.gguf (167 KB) applies the same rank-one edit to the stock AngelSlim/Hy4-flash-GGUF files as a LoRA adapter on blk.{22..38 even}.ffn_down_shexp. The scale is α (0.1 = root version, 0.05 = milder version):

llama-server -m Hy4-flash-SPARK-4Bit-Fast.gguf ... --lora-scaled hy4-flash-concise-lora.gguf:0.1

The adapter was validated on Q4_K_M (MATH screen: −38% tokens, −2.3 points). It is not separately validated on the SPARK-4Bit builds. Note that llama-server will not batch requests whose per-request LoRA settings differ, so set the scale globally.

Limitations

  • Accuracy loss on code (−6 points) and instruction following (−11 points) at α = 0.1 is statistically significant.
  • The direction was read from a Q4_K_M GGUF of the same model. Q8_0 shared experts differ from BF16 by 0.6%.
  • The edit was tested only with thinking on. With no_think it is untested.
  • One edit form, one layer set, single runs per benchmark.

License and attribution

Tencent Hy License 1.0 (see LICENSE), inherited from tencent/Hy4-flash. Modified from atakans/Hy4-flash-NVFP4: nine shared_experts.down_proj tensors replaced as described above, and the hf_quant_config.json exclude list extended to cover them.

Method after weight-space programming (Takehi et al., 2026) and the norm-matched replication by Concept Country (2026).