--- license: other license_name: tencent-hy-license license_link: https://huggingface.co/tencent/Hy4-flash/blob/main/LICENSE base_model: - tencent/Hy4-flash - atakans/Hy4-flash-NVFP4 tags: - hy4-flash - weight-editing - weight-space-programming - reasoning - nvfp4 - moe library_name: transformers pipeline_tag: text-generation --- # hy4-flash-concise **Hy4-flash that thinks less.** Nine shared-expert down-projections of [tencent/Hy4-flash](https://huggingface.co/tencent/Hy4-flash) are rewritten with a rank-one, gradient-free edit (weight-space programming). There are no gradient steps and no fine-tuning data. The edit comes from the model's own activations and is compiled into nine weight matrices. Thinking tokens drop by **37–55%** on every benchmark we ran. Accuracy also drops, by a little on GPQA and by a meaningful amount on code and instruction following (see below). Use this model when shorter, cheaper reasoning matters more than the last few points of accuracy. > Status: experimental. The root of this repo is the **aggressive** edit (α = 0.1). A **milder** edit (α = 0.05) that keeps > reasoning and code accuracy within noise is in [`alpha-0.05/`](alpha-0.05/). ## What was changed | | | |---|---| | Base weights | `atakans/Hy4-flash-NVFP4` (NVFP4 MLPs, FP8 Engram tables, BF16 elsewhere), derived from `tencent/Hy4-flash` | | Edited tensors | `model.layers.{22,24,…,38}.mlp.shared_experts.down_proj.weight` (9 matrices) | | Edit | `W ← W_bf16_original + α · n · d″ uᵀ / ‖u‖²`, α = 0.1, stored in **BF16** (excluded from NVFP4) | | Relative size | ‖ΔW‖/‖W‖ = 0.56–1.09% per layer | | Everything else | byte-identical to `atakans/Hy4-flash-NVFP4` | * `d` is the residual-stream direction that separates the last 48 thinking tokens before `` from the rest of the thinking, measured on 288 MATH-train (level 5) traces. `d″` corrects `d` for the FFN post-norm (`combine_output_norm`), so after that RMSNorm the written vector points along `d`. * `u` is the mean input to the shared-expert down-projection on thinking tokens, and `n` is the mean norm of the FFN output before the post-norm. * The shared expert runs on every token, so a rank-one term in its down-projection acts like a constant steering vector, compiled into the weights. * Directions, layers and α were chosen on MATH train only. No evaluation benchmark was used for selection. The direction tensors and scripts are in [`edit/`](edit/). ## Results Setup: vLLM, this checkpoint against the matched base (the same nine matrices in BF16, unedited), thinking on (`reasoning_effort: high`), temperature 0.9, max 32k tokens, one H200. Δ is paired by problem, with a 95% bootstrap CI. | Benchmark | Base | **concise (α=0.1)** | Δ accuracy [95% CI] | Δ thinking tokens [95% CI] | |---|---|---|---|---| | GPQA Diamond (198 × 2) | 91.2 | 88.6 | −2.5 [−5.3, +0.3] | **−39.6%** [−43.3, −35.6] | | LiveCodeBench v6 (175, 2025-01…04) | 84.6 | 78.3 | −6.3 [−12.0, −0.6] | **−36.8%** [−43.3, −30.3] | | IFBench (300, loose) | 76.7 | 66.0 | −10.7 [−16.0, −5.3] | **−54.8%** [−58.3, −51.0] | | 13 coding/math/science tasks × 3 (held-out) | 39/39 | 38/39 | −2.6 | **−54.2%** | | MATH train screen (64 × 2) | 92.2 | 89.8 | −2.3 [−6.2, +0.8] | −40.2% [−47.8, −29.9] | Mean thinking tokens, base → concise: GPQA 5448 → 3291, LCB 8564 → 5409, IFBench 5275 → 2382, held-out tasks 5415 → 2482. ### Milder version: `alpha-0.05/` | Benchmark | Base | **α=0.05** | Δ accuracy [95% CI] | Δ thinking tokens [95% CI] | |---|---|---|---|---| | GPQA Diamond (198 × 2) | 91.2 | 89.6 | −1.5 [−3.8, +0.5] | −20.6% [−25.6, −15.7] | | LiveCodeBench v6 (175) | 84.6 | 82.3 | −2.3 [−8.0, +3.4] | −25.0% [−31.2, −18.8] | | IFBench (300, loose) | 76.7 | 70.3 | −6.3 [−11.0, −2.0] | −36.6% [−40.8, −31.7] | | 13 held-out tasks × 3 | 39/39 | 37/39 | −5.1 | −29.1% | Which one to pick: choose `alpha-0.05` if accuracy on reasoning and code must hold. Choose the root (α = 0.1) for the largest saving. Both lose points on strict instruction following (IFBench). **Control.** A norm-matched random edit (same layers, same ‖ΔW‖, random direction orthogonal to `d`) on the MATH screen gives +0.0 points of accuracy and +8% tokens. The token saving comes from the direction, not from the perturbation. **Dose response** (MATH screen): α = 0.05 → −23% tokens; α = 0.075 → −32%; α = 0.1 → −40%; α = 0.3 → answers become wrong; α = 1.0 → the model loops on ``. Do not raise α. ## Usage ### vLLM (Hopper / Blackwell) Hy4-flash needs the `vllm/vllm-openai:hy4-flash` image. NVFP4 expert loading needs the `vllm/models/hy_v4_flash/nvidia/model.py` from [atakann/dgx-spark-recipes](https://github.com/atakann/dgx-spark-recipes) (copy included as `edit/vllm_hy_v4_flash_model.py`). On DGX Spark (SM120), use that recipe's full image. ```bash vllm serve autotrust/hy4-flash-concise --served-model-name hy4-flash \ --dtype bfloat16 --max-model-len 40960 --max-num-seqs 64 \ --reasoning-parser hy_v4 --limit-mm-per-prompt '{"image":0,"video":0}' # milder version (local path): hf download autotrust/hy4-flash-concise --include "alpha-0.05/*" --local-dir hy4-flash-concise vllm serve hy4-flash-concise/alpha-0.05 --served-model-name hy4-flash --reasoning-parser hy_v4 ... ``` On one H200: 97 GiB of weights on the GPU, Engram tables in host RAM, about 1.5k tok/s aggregate decode at 100 concurrent requests. ### llama.cpp (GGUF, e.g. DGX Spark) `gguf-lora/hy4-flash-concise-lora.gguf` (167 KB) applies the same rank-one edit to the stock [AngelSlim/Hy4-flash-GGUF](https://huggingface.co/AngelSlim/Hy4-flash-GGUF) files as a LoRA adapter on `blk.{22..38 even}.ffn_down_shexp`. The scale is α (0.1 = root version, 0.05 = milder version): ```bash llama-server -m Hy4-flash-SPARK-4Bit-Fast.gguf ... --lora-scaled hy4-flash-concise-lora.gguf:0.1 ``` The adapter was validated on Q4_K_M (MATH screen: −38% tokens, −2.3 points). It is **not** separately validated on the SPARK-4Bit builds. Note that llama-server will not batch requests whose per-request LoRA settings differ, so set the scale globally. ## Limitations * Accuracy loss on code (−6 points) and instruction following (−11 points) at α = 0.1 is statistically significant. * The direction was read from a Q4_K_M GGUF of the same model. Q8_0 shared experts differ from BF16 by 0.6%. * The edit was tested only with thinking on. With `no_think` it is untested. * One edit form, one layer set, single runs per benchmark. ## License and attribution Tencent Hy License 1.0 (see `LICENSE`), inherited from tencent/Hy4-flash. Modified from atakans/Hy4-flash-NVFP4: nine `shared_experts.down_proj` tensors replaced as described above, and the `hf_quant_config.json` exclude list extended to cover them. Method after *weight-space programming* (Takehi et al., 2026) and the norm-matched replication by Concept Country (2026).