Text Generation
Transformers
Safetensors
GGUF
hy_v4_flash
hy4-flash
weight-editing
weight-space-programming
reasoning
nvfp4
Mixture of Experts
conversational
8-bit precision
Instructions to use autotrust/hy4-flash-concise with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use autotrust/hy4-flash-concise with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="autotrust/hy4-flash-concise") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForSeq2SeqLM model = AutoModelForSeq2SeqLM.from_pretrained("autotrust/hy4-flash-concise", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use autotrust/hy4-flash-concise with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "autotrust/hy4-flash-concise" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "autotrust/hy4-flash-concise", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/autotrust/hy4-flash-concise
- SGLang
How to use autotrust/hy4-flash-concise with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "autotrust/hy4-flash-concise" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "autotrust/hy4-flash-concise", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "autotrust/hy4-flash-concise" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "autotrust/hy4-flash-concise", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use autotrust/hy4-flash-concise with Docker Model Runner:
docker model run hf.co/autotrust/hy4-flash-concise
|
Download README.md from autotrust/hy4-flash-concise: direct link, hf CLI and curl.
- Browser
- Download file 7.04 kB
-
https://huggingface.co/autotrust/hy4-flash-concise/resolve/main/README.md
- Command line
-
hf download hf://autotrust/hy4-flash-concise/README.md
-
curl -L -o README.md https://huggingface.co/autotrust/hy4-flash-concise/resolve/main/README.md
7.04 kB
| license: other | |
| license_name: tencent-hy-license | |
| license_link: https://huggingface.co/tencent/Hy4-flash/blob/main/LICENSE | |
| base_model: | |
| - tencent/Hy4-flash | |
| - atakans/Hy4-flash-NVFP4 | |
| tags: | |
| - hy4-flash | |
| - weight-editing | |
| - weight-space-programming | |
| - reasoning | |
| - nvfp4 | |
| - moe | |
| library_name: transformers | |
| pipeline_tag: text-generation | |
| # hy4-flash-concise | |
| **Hy4-flash that thinks less.** Nine shared-expert down-projections of [tencent/Hy4-flash](https://huggingface.co/tencent/Hy4-flash) | |
| are rewritten with a rank-one, gradient-free edit (weight-space programming). There are no gradient steps and no | |
| fine-tuning data. The edit comes from the model's own activations and is compiled into nine weight matrices. | |
| Thinking tokens drop by **37–55%** on every benchmark we ran. Accuracy also drops, by a little on GPQA and by a | |
| meaningful amount on code and instruction following (see below). Use this model when shorter, cheaper | |
| reasoning matters more than the last few points of accuracy. | |
| > Status: experimental. The root of this repo is the **aggressive** edit (α = 0.1). A **milder** edit (α = 0.05) that keeps | |
| > reasoning and code accuracy within noise is in [`alpha-0.05/`](alpha-0.05/). | |
| ## What was changed | |
| | | | | |
| |---|---| | |
| | Base weights | `atakans/Hy4-flash-NVFP4` (NVFP4 MLPs, FP8 Engram tables, BF16 elsewhere), derived from `tencent/Hy4-flash` | | |
| | Edited tensors | `model.layers.{22,24,…,38}.mlp.shared_experts.down_proj.weight` (9 matrices) | | |
| | Edit | `W ← W_bf16_original + α · n · d″ uᵀ / ‖u‖²`, α = 0.1, stored in **BF16** (excluded from NVFP4) | | |
| | Relative size | ‖ΔW‖/‖W‖ = 0.56–1.09% per layer | | |
| | Everything else | byte-identical to `atakans/Hy4-flash-NVFP4` | | |
| * `d` is the residual-stream direction that separates the last 48 thinking tokens before `</think>` from the rest of | |
| the thinking, measured on 288 MATH-train (level 5) traces. `d″` corrects `d` for the FFN post-norm | |
| (`combine_output_norm`), so after that RMSNorm the written vector points along `d`. | |
| * `u` is the mean input to the shared-expert down-projection on thinking tokens, and `n` is the mean norm of the | |
| FFN output before the post-norm. | |
| * The shared expert runs on every token, so a rank-one term in its down-projection acts like a constant steering | |
| vector, compiled into the weights. | |
| * Directions, layers and α were chosen on MATH train only. No evaluation benchmark was used for selection. | |
| The direction tensors and scripts are in [`edit/`](edit/). | |
| ## Results | |
| Setup: vLLM, this checkpoint against the matched base (the same nine matrices in BF16, unedited), thinking on | |
| (`reasoning_effort: high`), temperature 0.9, max 32k tokens, one H200. Δ is paired by problem, with a 95% | |
| bootstrap CI. | |
| | Benchmark | Base | **concise (α=0.1)** | Δ accuracy [95% CI] | Δ thinking tokens [95% CI] | | |
| |---|---|---|---|---| | |
| | GPQA Diamond (198 × 2) | 91.2 | 88.6 | −2.5 [−5.3, +0.3] | **−39.6%** [−43.3, −35.6] | | |
| | LiveCodeBench v6 (175, 2025-01…04) | 84.6 | 78.3 | −6.3 [−12.0, −0.6] | **−36.8%** [−43.3, −30.3] | | |
| | IFBench (300, loose) | 76.7 | 66.0 | −10.7 [−16.0, −5.3] | **−54.8%** [−58.3, −51.0] | | |
| | 13 coding/math/science tasks × 3 (held-out) | 39/39 | 38/39 | −2.6 | **−54.2%** | | |
| | MATH train screen (64 × 2) | 92.2 | 89.8 | −2.3 [−6.2, +0.8] | −40.2% [−47.8, −29.9] | | |
| Mean thinking tokens, base → concise: GPQA 5448 → 3291, LCB 8564 → 5409, IFBench 5275 → 2382, held-out tasks 5415 → 2482. | |
| ### Milder version: `alpha-0.05/` | |
| | Benchmark | Base | **α=0.05** | Δ accuracy [95% CI] | Δ thinking tokens [95% CI] | | |
| |---|---|---|---|---| | |
| | GPQA Diamond (198 × 2) | 91.2 | 89.6 | −1.5 [−3.8, +0.5] | −20.6% [−25.6, −15.7] | | |
| | LiveCodeBench v6 (175) | 84.6 | 82.3 | −2.3 [−8.0, +3.4] | −25.0% [−31.2, −18.8] | | |
| | IFBench (300, loose) | 76.7 | 70.3 | −6.3 [−11.0, −2.0] | −36.6% [−40.8, −31.7] | | |
| | 13 held-out tasks × 3 | 39/39 | 37/39 | −5.1 | −29.1% | | |
| Which one to pick: choose `alpha-0.05` if accuracy on reasoning and code must hold. Choose the root (α = 0.1) for | |
| the largest saving. Both lose points on strict instruction following (IFBench). | |
| **Control.** A norm-matched random edit (same layers, same ‖ΔW‖, random direction orthogonal to `d`) on the | |
| MATH screen gives +0.0 points of accuracy and +8% tokens. The token saving comes from the direction, not from the | |
| perturbation. | |
| **Dose response** (MATH screen): α = 0.05 → −23% tokens; α = 0.075 → −32%; α = 0.1 → −40%; α = 0.3 → answers | |
| become wrong; α = 1.0 → the model loops on `</think>`. Do not raise α. | |
| ## Usage | |
| ### vLLM (Hopper / Blackwell) | |
| Hy4-flash needs the `vllm/vllm-openai:hy4-flash` image. NVFP4 expert loading needs the | |
| `vllm/models/hy_v4_flash/nvidia/model.py` from | |
| [atakann/dgx-spark-recipes](https://github.com/atakann/dgx-spark-recipes) (copy included as | |
| `edit/vllm_hy_v4_flash_model.py`). On DGX Spark (SM120), use that recipe's full image. | |
| ```bash | |
| vllm serve autotrust/hy4-flash-concise --served-model-name hy4-flash \ | |
| --dtype bfloat16 --max-model-len 40960 --max-num-seqs 64 \ | |
| --reasoning-parser hy_v4 --limit-mm-per-prompt '{"image":0,"video":0}' | |
| # milder version (local path): | |
| hf download autotrust/hy4-flash-concise --include "alpha-0.05/*" --local-dir hy4-flash-concise | |
| vllm serve hy4-flash-concise/alpha-0.05 --served-model-name hy4-flash --reasoning-parser hy_v4 ... | |
| ``` | |
| On one H200: 97 GiB of weights on the GPU, Engram tables in host RAM, about 1.5k tok/s aggregate decode at 100 | |
| concurrent requests. | |
| ### llama.cpp (GGUF, e.g. DGX Spark) | |
| `gguf-lora/hy4-flash-concise-lora.gguf` (167 KB) applies the same rank-one edit to the stock | |
| [AngelSlim/Hy4-flash-GGUF](https://huggingface.co/AngelSlim/Hy4-flash-GGUF) files as a LoRA adapter on | |
| `blk.{22..38 even}.ffn_down_shexp`. The scale is α (0.1 = root version, 0.05 = milder version): | |
| ```bash | |
| llama-server -m Hy4-flash-SPARK-4Bit-Fast.gguf ... --lora-scaled hy4-flash-concise-lora.gguf:0.1 | |
| ``` | |
| The adapter was validated on Q4_K_M (MATH screen: −38% tokens, −2.3 points). It is **not** separately validated | |
| on the SPARK-4Bit builds. Note that llama-server will not batch requests whose per-request LoRA settings differ, | |
| so set the scale globally. | |
| ## Limitations | |
| * Accuracy loss on code (−6 points) and instruction following (−11 points) at α = 0.1 is statistically significant. | |
| * The direction was read from a Q4_K_M GGUF of the same model. Q8_0 shared experts differ from BF16 by 0.6%. | |
| * The edit was tested only with thinking on. With `no_think` it is untested. | |
| * One edit form, one layer set, single runs per benchmark. | |
| ## License and attribution | |
| Tencent Hy License 1.0 (see `LICENSE`), inherited from tencent/Hy4-flash. Modified from | |
| atakans/Hy4-flash-NVFP4: nine `shared_experts.down_proj` tensors replaced as described above, and the | |
| `hf_quant_config.json` exclude list extended to cover them. | |
| Method after *weight-space programming* (Takehi et al., 2026) and the norm-matched replication by Concept Country | |
| (2026). | |