hy4-flash-concise / README.md
cloudyu's picture
Rename to hy4-flash-concise; add alpha-0.05 results
1f51adb verified
|
Raw History Blame Contribute Delete
7.04 kB
---
license: other
license_name: tencent-hy-license
license_link: https://huggingface.co/tencent/Hy4-flash/blob/main/LICENSE
base_model:
- tencent/Hy4-flash
- atakans/Hy4-flash-NVFP4
tags:
- hy4-flash
- weight-editing
- weight-space-programming
- reasoning
- nvfp4
- moe
library_name: transformers
pipeline_tag: text-generation
---
# hy4-flash-concise
**Hy4-flash that thinks less.** Nine shared-expert down-projections of [tencent/Hy4-flash](https://huggingface.co/tencent/Hy4-flash)
are rewritten with a rank-one, gradient-free edit (weight-space programming). There are no gradient steps and no
fine-tuning data. The edit comes from the model's own activations and is compiled into nine weight matrices.
Thinking tokens drop by **37–55%** on every benchmark we ran. Accuracy also drops, by a little on GPQA and by a
meaningful amount on code and instruction following (see below). Use this model when shorter, cheaper
reasoning matters more than the last few points of accuracy.
> Status: experimental. The root of this repo is the **aggressive** edit (α = 0.1). A **milder** edit (α = 0.05) that keeps
> reasoning and code accuracy within noise is in [`alpha-0.05/`](alpha-0.05/).
## What was changed
| | |
|---|---|
| Base weights | `atakans/Hy4-flash-NVFP4` (NVFP4 MLPs, FP8 Engram tables, BF16 elsewhere), derived from `tencent/Hy4-flash` |
| Edited tensors | `model.layers.{22,24,…,38}.mlp.shared_experts.down_proj.weight` (9 matrices) |
| Edit | `W ← W_bf16_original + α · n · d″ uᵀ / ‖u‖²`, α = 0.1, stored in **BF16** (excluded from NVFP4) |
| Relative size | ‖ΔW‖/‖W‖ = 0.56–1.09% per layer |
| Everything else | byte-identical to `atakans/Hy4-flash-NVFP4` |
* `d` is the residual-stream direction that separates the last 48 thinking tokens before `</think>` from the rest of
the thinking, measured on 288 MATH-train (level 5) traces. `d″` corrects `d` for the FFN post-norm
(`combine_output_norm`), so after that RMSNorm the written vector points along `d`.
* `u` is the mean input to the shared-expert down-projection on thinking tokens, and `n` is the mean norm of the
FFN output before the post-norm.
* The shared expert runs on every token, so a rank-one term in its down-projection acts like a constant steering
vector, compiled into the weights.
* Directions, layers and α were chosen on MATH train only. No evaluation benchmark was used for selection.
The direction tensors and scripts are in [`edit/`](edit/).
## Results
Setup: vLLM, this checkpoint against the matched base (the same nine matrices in BF16, unedited), thinking on
(`reasoning_effort: high`), temperature 0.9, max 32k tokens, one H200. Δ is paired by problem, with a 95%
bootstrap CI.
| Benchmark | Base | **concise (α=0.1)** | Δ accuracy [95% CI] | Δ thinking tokens [95% CI] |
|---|---|---|---|---|
| GPQA Diamond (198 × 2) | 91.2 | 88.6 | −2.5 [−5.3, +0.3] | **−39.6%** [−43.3, −35.6] |
| LiveCodeBench v6 (175, 2025-01…04) | 84.6 | 78.3 | −6.3 [−12.0, −0.6] | **−36.8%** [−43.3, −30.3] |
| IFBench (300, loose) | 76.7 | 66.0 | −10.7 [−16.0, −5.3] | **−54.8%** [−58.3, −51.0] |
| 13 coding/math/science tasks × 3 (held-out) | 39/39 | 38/39 | −2.6 | **−54.2%** |
| MATH train screen (64 × 2) | 92.2 | 89.8 | −2.3 [−6.2, +0.8] | −40.2% [−47.8, −29.9] |
Mean thinking tokens, base → concise: GPQA 5448 → 3291, LCB 8564 → 5409, IFBench 5275 → 2382, held-out tasks 5415 → 2482.
### Milder version: `alpha-0.05/`
| Benchmark | Base | **α=0.05** | Δ accuracy [95% CI] | Δ thinking tokens [95% CI] |
|---|---|---|---|---|
| GPQA Diamond (198 × 2) | 91.2 | 89.6 | −1.5 [−3.8, +0.5] | −20.6% [−25.6, −15.7] |
| LiveCodeBench v6 (175) | 84.6 | 82.3 | −2.3 [−8.0, +3.4] | −25.0% [−31.2, −18.8] |
| IFBench (300, loose) | 76.7 | 70.3 | −6.3 [−11.0, −2.0] | −36.6% [−40.8, −31.7] |
| 13 held-out tasks × 3 | 39/39 | 37/39 | −5.1 | −29.1% |
Which one to pick: choose `alpha-0.05` if accuracy on reasoning and code must hold. Choose the root (α = 0.1) for
the largest saving. Both lose points on strict instruction following (IFBench).
**Control.** A norm-matched random edit (same layers, same ‖ΔW‖, random direction orthogonal to `d`) on the
MATH screen gives +0.0 points of accuracy and +8% tokens. The token saving comes from the direction, not from the
perturbation.
**Dose response** (MATH screen): α = 0.05 → −23% tokens; α = 0.075 → −32%; α = 0.1 → −40%; α = 0.3 → answers
become wrong; α = 1.0 → the model loops on `</think>`. Do not raise α.
## Usage
### vLLM (Hopper / Blackwell)
Hy4-flash needs the `vllm/vllm-openai:hy4-flash` image. NVFP4 expert loading needs the
`vllm/models/hy_v4_flash/nvidia/model.py` from
[atakann/dgx-spark-recipes](https://github.com/atakann/dgx-spark-recipes) (copy included as
`edit/vllm_hy_v4_flash_model.py`). On DGX Spark (SM120), use that recipe's full image.
```bash
vllm serve autotrust/hy4-flash-concise --served-model-name hy4-flash \
--dtype bfloat16 --max-model-len 40960 --max-num-seqs 64 \
--reasoning-parser hy_v4 --limit-mm-per-prompt '{"image":0,"video":0}'
# milder version (local path):
hf download autotrust/hy4-flash-concise --include "alpha-0.05/*" --local-dir hy4-flash-concise
vllm serve hy4-flash-concise/alpha-0.05 --served-model-name hy4-flash --reasoning-parser hy_v4 ...
```
On one H200: 97 GiB of weights on the GPU, Engram tables in host RAM, about 1.5k tok/s aggregate decode at 100
concurrent requests.
### llama.cpp (GGUF, e.g. DGX Spark)
`gguf-lora/hy4-flash-concise-lora.gguf` (167 KB) applies the same rank-one edit to the stock
[AngelSlim/Hy4-flash-GGUF](https://huggingface.co/AngelSlim/Hy4-flash-GGUF) files as a LoRA adapter on
`blk.{22..38 even}.ffn_down_shexp`. The scale is α (0.1 = root version, 0.05 = milder version):
```bash
llama-server -m Hy4-flash-SPARK-4Bit-Fast.gguf ... --lora-scaled hy4-flash-concise-lora.gguf:0.1
```
The adapter was validated on Q4_K_M (MATH screen: −38% tokens, −2.3 points). It is **not** separately validated
on the SPARK-4Bit builds. Note that llama-server will not batch requests whose per-request LoRA settings differ,
so set the scale globally.
## Limitations
* Accuracy loss on code (−6 points) and instruction following (−11 points) at α = 0.1 is statistically significant.
* The direction was read from a Q4_K_M GGUF of the same model. Q8_0 shared experts differ from BF16 by 0.6%.
* The edit was tested only with thinking on. With `no_think` it is untested.
* One edit form, one layer set, single runs per benchmark.
## License and attribution
Tencent Hy License 1.0 (see `LICENSE`), inherited from tencent/Hy4-flash. Modified from
atakans/Hy4-flash-NVFP4: nine `shared_experts.down_proj` tensors replaced as described above, and the
`hf_quant_config.json` exclude list extended to cover them.
Method after *weight-space programming* (Takehi et al., 2026) and the norm-matched replication by Concept Country
(2026).