File size: 7,037 Bytes
615b72a
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1f51adb
615b72a
 
 
 
 
 
 
 
 
1f51adb
 
615b72a
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1f51adb
615b72a
 
 
 
 
 
 
1f51adb
 
 
 
 
 
 
 
 
 
 
 
 
615b72a
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1f51adb
615b72a
 
1f51adb
 
 
615b72a
 
 
 
 
 
 
1f51adb
615b72a
1f51adb
615b72a
 
1f51adb
615b72a
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
---
license: other
license_name: tencent-hy-license
license_link: https://huggingface.co/tencent/Hy4-flash/blob/main/LICENSE
base_model:
- tencent/Hy4-flash
- atakans/Hy4-flash-NVFP4
tags:
- hy4-flash
- weight-editing
- weight-space-programming
- reasoning
- nvfp4
- moe
library_name: transformers
pipeline_tag: text-generation
---

# hy4-flash-concise

**Hy4-flash that thinks less.** Nine shared-expert down-projections of [tencent/Hy4-flash](https://huggingface.co/tencent/Hy4-flash)
are rewritten with a rank-one, gradient-free edit (weight-space programming). There are no gradient steps and no
fine-tuning data. The edit comes from the model's own activations and is compiled into nine weight matrices.

Thinking tokens drop by **37–55%** on every benchmark we ran. Accuracy also drops, by a little on GPQA and by a
meaningful amount on code and instruction following (see below). Use this model when shorter, cheaper
reasoning matters more than the last few points of accuracy.

> Status: experimental. The root of this repo is the **aggressive** edit (α = 0.1). A **milder** edit (α = 0.05) that keeps
> reasoning and code accuracy within noise is in [`alpha-0.05/`](alpha-0.05/).

## What was changed

| | |
|---|---|
| Base weights | `atakans/Hy4-flash-NVFP4` (NVFP4 MLPs, FP8 Engram tables, BF16 elsewhere), derived from `tencent/Hy4-flash` |
| Edited tensors | `model.layers.{22,24,…,38}.mlp.shared_experts.down_proj.weight` (9 matrices) |
| Edit | `W ← W_bf16_original + α · n · d″ uᵀ / ‖u‖²`, α = 0.1, stored in **BF16** (excluded from NVFP4) |
| Relative size | ‖ΔW‖/‖W‖ = 0.56–1.09% per layer |
| Everything else | byte-identical to `atakans/Hy4-flash-NVFP4` |

* `d` is the residual-stream direction that separates the last 48 thinking tokens before `</think>` from the rest of
  the thinking, measured on 288 MATH-train (level 5) traces. `d″` corrects `d` for the FFN post-norm
  (`combine_output_norm`), so after that RMSNorm the written vector points along `d`.
* `u` is the mean input to the shared-expert down-projection on thinking tokens, and `n` is the mean norm of the
  FFN output before the post-norm.
* The shared expert runs on every token, so a rank-one term in its down-projection acts like a constant steering
  vector, compiled into the weights.
* Directions, layers and α were chosen on MATH train only. No evaluation benchmark was used for selection.

The direction tensors and scripts are in [`edit/`](edit/).

## Results

Setup: vLLM, this checkpoint against the matched base (the same nine matrices in BF16, unedited), thinking on
(`reasoning_effort: high`), temperature 0.9, max 32k tokens, one H200. Δ is paired by problem, with a 95%
bootstrap CI.

| Benchmark | Base | **concise (α=0.1)** | Δ accuracy [95% CI] | Δ thinking tokens [95% CI] |
|---|---|---|---|---|
| GPQA Diamond (198 × 2) | 91.2 | 88.6 | −2.5 [−5.3, +0.3] | **−39.6%** [−43.3, −35.6] |
| LiveCodeBench v6 (175, 2025-01…04) | 84.6 | 78.3 | −6.3 [−12.0, −0.6] | **−36.8%** [−43.3, −30.3] |
| IFBench (300, loose) | 76.7 | 66.0 | −10.7 [−16.0, −5.3] | **−54.8%** [−58.3, −51.0] |
| 13 coding/math/science tasks × 3 (held-out) | 39/39 | 38/39 | −2.6 | **−54.2%** |
| MATH train screen (64 × 2) | 92.2 | 89.8 | −2.3 [−6.2, +0.8] | −40.2% [−47.8, −29.9] |

Mean thinking tokens, base → concise: GPQA 5448 → 3291, LCB 8564 → 5409, IFBench 5275 → 2382, held-out tasks 5415 → 2482.

### Milder version: `alpha-0.05/`

| Benchmark | Base | **α=0.05** | Δ accuracy [95% CI] | Δ thinking tokens [95% CI] |
|---|---|---|---|---|
| GPQA Diamond (198 × 2) | 91.2 | 89.6 | −1.5 [−3.8, +0.5] | −20.6% [−25.6, −15.7] |
| LiveCodeBench v6 (175) | 84.6 | 82.3 | −2.3 [−8.0, +3.4] | −25.0% [−31.2, −18.8] |
| IFBench (300, loose) | 76.7 | 70.3 | −6.3 [−11.0, −2.0] | −36.6% [−40.8, −31.7] |
| 13 held-out tasks × 3 | 39/39 | 37/39 | −5.1 | −29.1% |

Which one to pick: choose `alpha-0.05` if accuracy on reasoning and code must hold. Choose the root (α = 0.1) for
the largest saving. Both lose points on strict instruction following (IFBench).

**Control.** A norm-matched random edit (same layers, same ‖ΔW‖, random direction orthogonal to `d`) on the
MATH screen gives +0.0 points of accuracy and +8% tokens. The token saving comes from the direction, not from the
perturbation.

**Dose response** (MATH screen): α = 0.05 → −23% tokens; α = 0.075 → −32%; α = 0.1 → −40%; α = 0.3 → answers
become wrong; α = 1.0 → the model loops on `</think>`. Do not raise α.

## Usage

### vLLM (Hopper / Blackwell)

Hy4-flash needs the `vllm/vllm-openai:hy4-flash` image. NVFP4 expert loading needs the
`vllm/models/hy_v4_flash/nvidia/model.py` from
[atakann/dgx-spark-recipes](https://github.com/atakann/dgx-spark-recipes) (copy included as
`edit/vllm_hy_v4_flash_model.py`). On DGX Spark (SM120), use that recipe's full image.

```bash
vllm serve autotrust/hy4-flash-concise --served-model-name hy4-flash \
  --dtype bfloat16 --max-model-len 40960 --max-num-seqs 64 \
  --reasoning-parser hy_v4 --limit-mm-per-prompt '{"image":0,"video":0}'
# milder version (local path):
hf download autotrust/hy4-flash-concise --include "alpha-0.05/*" --local-dir hy4-flash-concise
vllm serve hy4-flash-concise/alpha-0.05 --served-model-name hy4-flash --reasoning-parser hy_v4 ...
```

On one H200: 97 GiB of weights on the GPU, Engram tables in host RAM, about 1.5k tok/s aggregate decode at 100
concurrent requests.

### llama.cpp (GGUF, e.g. DGX Spark)

`gguf-lora/hy4-flash-concise-lora.gguf` (167 KB) applies the same rank-one edit to the stock
[AngelSlim/Hy4-flash-GGUF](https://huggingface.co/AngelSlim/Hy4-flash-GGUF) files as a LoRA adapter on
`blk.{22..38 even}.ffn_down_shexp`. The scale is α (0.1 = root version, 0.05 = milder version):

```bash
llama-server -m Hy4-flash-SPARK-4Bit-Fast.gguf ... --lora-scaled hy4-flash-concise-lora.gguf:0.1
```

The adapter was validated on Q4_K_M (MATH screen: −38% tokens, −2.3 points). It is **not** separately validated
on the SPARK-4Bit builds. Note that llama-server will not batch requests whose per-request LoRA settings differ,
so set the scale globally.

## Limitations

* Accuracy loss on code (−6 points) and instruction following (−11 points) at α = 0.1 is statistically significant.
* The direction was read from a Q4_K_M GGUF of the same model. Q8_0 shared experts differ from BF16 by 0.6%.
* The edit was tested only with thinking on. With `no_think` it is untested.
* One edit form, one layer set, single runs per benchmark.

## License and attribution

Tencent Hy License 1.0 (see `LICENSE`), inherited from tencent/Hy4-flash. Modified from
atakans/Hy4-flash-NVFP4: nine `shared_experts.down_proj` tensors replaced as described above, and the
`hf_quant_config.json` exclude list extended to cover them.

Method after *weight-space programming* (Takehi et al., 2026) and the norm-matched replication by Concept Country
(2026).