---
library_name: transformers
base_model:
- tencent/Hy4-preview
---
This tiny model is intended for debugging. It is randomly initialized using the configuration adapted from [tencent/Hy4-preview](https://huggingface.co/tencent/Hy4-preview).
Structural comparison with `tencent/Hy4-preview`:
| Feature | Source | Tiny |
|---|---|---|
| Architecture / input | HYV4; text | Same |
| Vocabulary / tied embeddings | 120,832 / false | Same |
| Backbone layers / hidden size | 78 / 6,144 | 10 / 16 |
| MLP layout | 1 dense layer, then MoE | Same |
| Routed experts / top-k / shared experts | 256 / 8 / 1 | 32 / 8 / 1 |
| Dense / expert intermediate size | 18,432 / 2,048 | 128 / 128 |
| Attention Q heads; non-RoPE / RoPE / V dimensions | 64; 192 / 64 / 256 | 8; 32 / 64 / 32 |
| Q / KV LoRA rank | 2,048 / 512 | 64 / 512 |
| Indexer heads / dimension / selected tokens | 32 / 128 / 2,048 | Same |
| Full indexer layers | 0, 1, 5, 9, ... | 0, 1, 5, 9; remaining layers reuse the preceding indexer |
| Independent Hyper-Connections | 4 streams; gated pre/post branches | Same |
| MLA gating / learned attention sinks / SwiGLU clamp | Elementwise / enabled / 10 | Same |
| MTP | 1 plain DSA/MoE decoder; shared embedding/head | Same |
| Checkpoint precision | BF16; FP32 iHC/main attention sinks and router correction bias | Same |
Tokenizer, chat template and generation assets are copied unchanged from the pinned source.
The registered Transformers `indexed_attention` config tag represents the source DSA layers.
| File path | Size |
|------|------|
| model.safetensors | 21.8MB |
### Example usage:
- SGLang (✅ verified)
- Run on four one-GPU nodes with NODE_RANK=0..3, the same DIST_INIT_ADDR and an available local NCCL_PORT. Enables TP=4, NEXTN/MTP and CUDA Graph.
- Verified on four NVIDIA H20 GPUs for English, Chinese and long DSA prompts; see Test environment for dependency versions.
```bash
model_id=tiny-random/hy4-preview
node_rank=${NODE_RANK:-0}
dist_init_addr=${DIST_INIT_ADDR}
nccl_port=${NCCL_PORT}
export SGLANG_DG_CACHE_DIR=/tmp/hy4-deepgemm-$node_rank
sglang serve \
--model-path $model_id \
--tp-size 4 \
--nnodes 4 \
--node-rank $node_rank \
--dist-init-addr $dist_init_addr \
--nccl-port $nccl_port \
--fp8-gemm-backend triton \
--moe-runner-backend triton \
--speculative-algorithm NEXTN \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4 \
--context-length 4096 \
--max-total-tokens 4096 \
--chunked-prefill-size 512 \
--mem-fraction-static 0.3 \
--cuda-graph-config '{"decode": {"bs": [1, 2, 4]}}' \
--disable-custom-all-reduce
```
- Transformers (✅ verified)
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "tiny-random/hy4-preview"
# Transformers 5.19.0 implements Hy4's DSA and learned sinks only with eager attention.
model = AutoModelForCausalLM.from_pretrained(
model_id, dtype=torch.bfloat16, device_map="cuda", attn_implementation="eager",
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
inputs = tokenizer.apply_chat_template(
[{"role": "user", "content": "Hello"}], tokenize=True,
add_generation_prompt=True, return_dict=True, return_tensors="pt",
).to(model.device)
output = model.generate(**inputs, do_sample=False, max_new_tokens=10)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```
### Codes to create this repo (✅ verified):
Click to expand
```python
import json
import re
from copy import deepcopy
from pathlib import Path
from shutil import copyfile
import torch
from huggingface_hub import hf_hub_download, list_repo_files
from safetensors.torch import save_file
from transformers import AutoConfig, AutoModelForCausalLM, set_seed
source_model_id = "tencent/Hy4-preview"
save_folder = "/tmp/tiny-random/hy4-preview"
source_revision = '705d81ee51566a186d645b74c974d642ef2828fe'
set_seed(42)
folder = Path(save_folder)
folder.mkdir(parents=True, exist_ok=True)
raw = json.loads(Path(hf_hub_download(source_model_id, "config.json", revision=source_revision)).read_text())
raw.update(
hidden_size=16, num_hidden_layers=10, num_attention_heads=8,
intermediate_size=128, moe_intermediate_size=128, n_routed_experts=32,
q_lora_rank=64, qk_nope_head_dim=32, v_head_dim=32, qk_head_dim=96,
layer_types=["indexed_attention"] * 10,
)
raw["indexer_types"] = raw["indexer_types"][:10]
raw["mlp_layer_types"] = raw["mlp_layer_types"][:10]
(folder / "config.json").write_text(json.dumps(raw, indent=2))
config = AutoConfig.from_pretrained(folder)
# Transformers 5.19.0 implements Hy4's DSA and learned sinks only with eager attention.
model = AutoModelForCausalLM.from_config(config, dtype=torch.bfloat16, attn_implementation="eager")
backbone = model.model
last = backbone.layers[-1]
model.model.mtp_layers = torch.nn.ModuleList([torch.nn.ModuleDict({
"enorm": deepcopy(backbone.norm), "hnorm": deepcopy(backbone.norm),
"eh_proj": torch.nn.Linear(2 * config.hidden_size, config.hidden_size, bias=False, dtype=torch.bfloat16),
"input_layernorm": deepcopy(last.input_layernorm), "self_attn": deepcopy(last.self_attn),
"post_attention_layernorm": deepcopy(last.post_attention_layernorm), "mlp": deepcopy(last.mlp),
"final_layernorm": deepcopy(backbone.norm),
})])
model.model.mtp_layers.apply(model._init_weights)
def public_name(name):
for native, source in (("attn_hc", "hc_attn_layer"), ("ffn_hc", "hc_mlp_layer")):
name = re.sub(r"\." + native + r"\.(fn|base|scale)$", r"." + source + r".hc_pre.hc_\1", name)
name = name.replace(".self_attn.gate_proj", ".self_attn.linear_gate")
name = name.replace(".self_attn.sinks", ".self_attn.learnable_sink_param")
for suffix in ("fn", "base", "scale"):
name = name.replace(".hc_head.hc_" + suffix, ".hc_head.hc_head_" + suffix)
return name
state = {}
for name, value in model.state_dict().items():
name = public_name(name)
full_precision = name.endswith((".hc_fn", ".hc_base", ".hc_scale", ".hc_head_fn", ".hc_head_base", ".hc_head_scale"))
full_precision |= name.endswith((".learnable_sink_param", ".e_score_correction_bias")) and not name.startswith("model.mtp_layers.")
state[name] = value.detach().to(torch.float32 if full_precision else torch.bfloat16).cpu().contiguous()
save_file(state, folder / "model.safetensors", metadata={"format": "pt"})
assets = {"tokenizer.json", "tokenizer_config.json", "chat_template.jinja", "generation_config.json", "LICENSE"}
for name in sorted(set(list_repo_files(source_model_id, revision=source_revision)) & assets):
copyfile(hf_hub_download(source_model_id, name, revision=source_revision), folder / name)
```
Click to expand
```text
HYV4ForCausalLM(
(model): HYV4Model(
(embed_tokens): Embedding(120832, 16, padding_idx=120002)
(layers): ModuleList(
(0): HYV4DecoderLayer(
(self_attn): HYV4Attention(
(q_a_proj): Linear(in_features=16, out_features=64, bias=False)
(q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
(q_b_proj): Linear(in_features=64, out_features=768, bias=False)
(kv_a_proj_with_mqa): Linear(in_features=16, out_features=576, bias=False)
(kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
(kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
(o_proj): Linear(in_features=256, out_features=16, bias=False)
(indexer): HYV4Indexer(
(wq_b): Linear(in_features=64, out_features=4096, bias=False)
(wk): Linear(in_features=16, out_features=128, bias=False)
(k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
(weights_proj): Linear(in_features=16, out_features=32, bias=False)
)
(gate_proj): Linear(in_features=16, out_features=256, bias=False)
)
(mlp): HYV4MLP(
(gate_proj): Linear(in_features=16, out_features=128, bias=False)
(up_proj): Linear(in_features=16, out_features=128, bias=False)
(down_proj): Linear(in_features=128, out_features=16, bias=False)
(act_fn): SiLUActivation()
)
(input_layernorm): HYV4RMSNorm((16,), eps=1e-05)
(post_attention_layernorm): HYV4RMSNorm((16,), eps=1e-05)
(attn_hc): HYV4HyperConnection(
(input_norm): HYV4UnweightedRMSNorm()
)
(ffn_hc): HYV4HyperConnection(
(input_norm): HYV4UnweightedRMSNorm()
)
)
(1): HYV4DecoderLayer(
(self_attn): HYV4Attention(
(q_a_proj): Linear(in_features=16, out_features=64, bias=False)
(q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
(q_b_proj): Linear(in_features=64, out_features=768, bias=False)
(kv_a_proj_with_mqa): Linear(in_features=16, out_features=576, bias=False)
(kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
(kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
(o_proj): Linear(in_features=256, out_features=16, bias=False)
(indexer): HYV4Indexer(
(wq_b): Linear(in_features=64, out_features=4096, bias=False)
(wk): Linear(in_features=16, out_features=128, bias=False)
(k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
(weights_proj): Linear(in_features=16, out_features=32, bias=False)
)
(gate_proj): Linear(in_features=16, out_features=256, bias=False)
)
(mlp): HYV4MoE(
(experts): HYV4Experts()
(gate): HYV4TopkRouter()
(shared_experts): HYV4MLP(
(gate_proj): Linear(in_features=16, out_features=128, bias=False)
(up_proj): Linear(in_features=16, out_features=128, bias=False)
(down_proj): Linear(in_features=128, out_features=16, bias=False)
(act_fn): SiLUActivation()
)
)
(input_layernorm): HYV4RMSNorm((16,), eps=1e-05)
(post_attention_layernorm): HYV4RMSNorm((16,), eps=1e-05)
(attn_hc): HYV4HyperConnection(
(input_norm): HYV4UnweightedRMSNorm()
)
(ffn_hc): HYV4HyperConnection(
(input_norm): HYV4UnweightedRMSNorm()
)
)
(2-4): 3 x HYV4DecoderLayer(
(self_attn): HYV4Attention(
(q_a_proj): Linear(in_features=16, out_features=64, bias=False)
(q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
(q_b_proj): Linear(in_features=64, out_features=768, bias=False)
(kv_a_proj_with_mqa): Linear(in_features=16, out_features=576, bias=False)
(kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
(kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
(o_proj): Linear(in_features=256, out_features=16, bias=False)
(gate_proj): Linear(in_features=16, out_features=256, bias=False)
)
(mlp): HYV4MoE(
(experts): HYV4Experts()
(gate): HYV4TopkRouter()
(shared_experts): HYV4MLP(
(gate_proj): Linear(in_features=16, out_features=128, bias=False)
(up_proj): Linear(in_features=16, out_features=128, bias=False)
(down_proj): Linear(in_features=128, out_features=16, bias=False)
(act_fn): SiLUActivation()
)
)
(input_layernorm): HYV4RMSNorm((16,), eps=1e-05)
(post_attention_layernorm): HYV4RMSNorm((16,), eps=1e-05)
(attn_hc): HYV4HyperConnection(
(input_norm): HYV4UnweightedRMSNorm()
)
(ffn_hc): HYV4HyperConnection(
(input_norm): HYV4UnweightedRMSNorm()
)
)
(5): HYV4DecoderLayer(
(self_attn): HYV4Attention(
(q_a_proj): Linear(in_features=16, out_features=64, bias=False)
(q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
(q_b_proj): Linear(in_features=64, out_features=768, bias=False)
(kv_a_proj_with_mqa): Linear(in_features=16, out_features=576, bias=False)
(kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
(kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
(o_proj): Linear(in_features=256, out_features=16, bias=False)
(indexer): HYV4Indexer(
(wq_b): Linear(in_features=64, out_features=4096, bias=False)
(wk): Linear(in_features=16, out_features=128, bias=False)
(k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
(weights_proj): Linear(in_features=16, out_features=32, bias=False)
)
(gate_proj): Linear(in_features=16, out_features=256, bias=False)
)
(mlp): HYV4MoE(
(experts): HYV4Experts()
(gate): HYV4TopkRouter()
(shared_experts): HYV4MLP(
(gate_proj): Linear(in_features=16, out_features=128, bias=False)
(up_proj): Linear(in_features=16, out_features=128, bias=False)
(down_proj): Linear(in_features=128, out_features=16, bias=False)
(act_fn): SiLUActivation()
)
)
(input_layernorm): HYV4RMSNorm((16,), eps=1e-05)
(post_attention_layernorm): HYV4RMSNorm((16,), eps=1e-05)
(attn_hc): HYV4HyperConnection(
(input_norm): HYV4UnweightedRMSNorm()
)
(ffn_hc): HYV4HyperConnection(
(input_norm): HYV4UnweightedRMSNorm()
)
)
(6-8): 3 x HYV4DecoderLayer(
(self_attn): HYV4Attention(
(q_a_proj): Linear(in_features=16, out_features=64, bias=False)
(q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
(q_b_proj): Linear(in_features=64, out_features=768, bias=False)
(kv_a_proj_with_mqa): Linear(in_features=16, out_features=576, bias=False)
(kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
(kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
(o_proj): Linear(in_features=256, out_features=16, bias=False)
(gate_proj): Linear(in_features=16, out_features=256, bias=False)
)
(mlp): HYV4MoE(
(experts): HYV4Experts()
(gate): HYV4TopkRouter()
(shared_experts): HYV4MLP(
(gate_proj): Linear(in_features=16, out_features=128, bias=False)
(up_proj): Linear(in_features=16, out_features=128, bias=False)
(down_proj): Linear(in_features=128, out_features=16, bias=False)
(act_fn): SiLUActivation()
)
)
(input_layernorm): HYV4RMSNorm((16,), eps=1e-05)
(post_attention_layernorm): HYV4RMSNorm((16,), eps=1e-05)
(attn_hc): HYV4HyperConnection(
(input_norm): HYV4UnweightedRMSNorm()
)
(ffn_hc): HYV4HyperConnection(
(input_norm): HYV4UnweightedRMSNorm()
)
)
(9): HYV4DecoderLayer(
(self_attn): HYV4Attention(
(q_a_proj): Linear(in_features=16, out_features=64, bias=False)
(q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
(q_b_proj): Linear(in_features=64, out_features=768, bias=False)
(kv_a_proj_with_mqa): Linear(in_features=16, out_features=576, bias=False)
(kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
(kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
(o_proj): Linear(in_features=256, out_features=16, bias=False)
(indexer): HYV4Indexer(
(wq_b): Linear(in_features=64, out_features=4096, bias=False)
(wk): Linear(in_features=16, out_features=128, bias=False)
(k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
(weights_proj): Linear(in_features=16, out_features=32, bias=False)
)
(gate_proj): Linear(in_features=16, out_features=256, bias=False)
)
(mlp): HYV4MoE(
(experts): HYV4Experts()
(gate): HYV4TopkRouter()
(shared_experts): HYV4MLP(
(gate_proj): Linear(in_features=16, out_features=128, bias=False)
(up_proj): Linear(in_features=16, out_features=128, bias=False)
(down_proj): Linear(in_features=128, out_features=16, bias=False)
(act_fn): SiLUActivation()
)
)
(input_layernorm): HYV4RMSNorm((16,), eps=1e-05)
(post_attention_layernorm): HYV4RMSNorm((16,), eps=1e-05)
(attn_hc): HYV4HyperConnection(
(input_norm): HYV4UnweightedRMSNorm()
)
(ffn_hc): HYV4HyperConnection(
(input_norm): HYV4UnweightedRMSNorm()
)
)
)
(norm): HYV4RMSNorm((16,), eps=1e-05)
(rotary_emb): HYV4RotaryEmbedding()
(hc_head): HYV4HyperHead(
(input_norm): HYV4UnweightedRMSNorm()
)
(mtp_layers): ModuleList(
(0): ModuleDict(
(enorm): HYV4RMSNorm((16,), eps=1e-05)
(hnorm): HYV4RMSNorm((16,), eps=1e-05)
(eh_proj): Linear(in_features=32, out_features=16, bias=False)
(input_layernorm): HYV4RMSNorm((16,), eps=1e-05)
(self_attn): HYV4Attention(
(q_a_proj): Linear(in_features=16, out_features=64, bias=False)
(q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
(q_b_proj): Linear(in_features=64, out_features=768, bias=False)
(kv_a_proj_with_mqa): Linear(in_features=16, out_features=576, bias=False)
(kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
(kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
(o_proj): Linear(in_features=256, out_features=16, bias=False)
(indexer): HYV4Indexer(
(wq_b): Linear(in_features=64, out_features=4096, bias=False)
(wk): Linear(in_features=16, out_features=128, bias=False)
(k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
(weights_proj): Linear(in_features=16, out_features=32, bias=False)
)
(gate_proj): Linear(in_features=16, out_features=256, bias=False)
)
(post_attention_layernorm): HYV4RMSNorm((16,), eps=1e-05)
(mlp): HYV4MoE(
(experts): HYV4Experts()
(gate): HYV4TopkRouter()
(shared_experts): HYV4MLP(
(gate_proj): Linear(in_features=16, out_features=128, bias=False)
(up_proj): Linear(in_features=16, out_features=128, bias=False)
(down_proj): Linear(in_features=128, out_features=16, bias=False)
(act_fn): SiLUActivation()
)
)
(final_layernorm): HYV4RMSNorm((16,), eps=1e-05)
)
)
)
(lm_head): Linear(in_features=16, out_features=120832, bias=False)
)
```