Instructions to use tiny-random/hy4-preview-fp8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tiny-random/hy4-preview-fp8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="tiny-random/hy4-preview-fp8")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("tiny-random/hy4-preview-fp8")
model = AutoModelForCausalLM.from_pretrained("tiny-random/hy4-preview-fp8", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tiny-random/hy4-preview-fp8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tiny-random/hy4-preview-fp8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiny-random/hy4-preview-fp8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/tiny-random/hy4-preview-fp8

SGLang

How to use tiny-random/hy4-preview-fp8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tiny-random/hy4-preview-fp8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiny-random/hy4-preview-fp8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tiny-random/hy4-preview-fp8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiny-random/hy4-preview-fp8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use tiny-random/hy4-preview-fp8 with Docker Model Runner:
```
docker model run hf.co/tiny-random/hy4-preview-fp8
```

hy4-preview-fp8 / README.md

yujiepan

Clarify validated GPU configurations

b074511 verified 2 days ago

preview code

Raw History Blame Contribute Delete

22.9 kB

---
library_name: transformers
base_model:
- tencent/Hy4-preview-FP8
---
This tiny model is intended for debugging. It is randomly initialized using the configuration adapted from [tencent/Hy4-preview-FP8](https://huggingface.co/tencent/Hy4-preview-FP8).

Structural comparison with `tencent/Hy4-preview-FP8`:

| Feature | Source | Tiny |
|---|---|---|
| Architecture / input | HYV4; text | Same |
| Vocabulary / tied embeddings | 120,832 / false | Same |
| Backbone layers / hidden size | 78 / 6,144 | 10 / 64 |
| MLP layout | 1 dense layer, then MoE | Same |
| Routed experts / top-k / shared experts | 256 / 8 / 1 | 32 / 8 / 1 |
| Dense / expert intermediate size | 18,432 / 2,048 | 128 / 512 |
| Attention Q heads; non-RoPE / RoPE / V dimensions | 64; 192 / 64 / 256 | 8; 32 / 64 / 32 |
| Q / KV LoRA rank | 2,048 / 512 | 64 / 512 |
| Indexer heads / dimension / selected tokens | 32 / 128 / 2,048 | Same |
| Full indexer layers | 0, 1, 5, 9, ... | 0, 1, 5, 9; remaining layers reuse the preceding indexer |
| Independent Hyper-Connections | 4 streams; gated pre/post branches | Same |
| MLA gating / learned attention sinks / SwiGLU clamp | Elementwise / enabled / 10 | Same |
| MTP | 1 plain DSA/MoE decoder; shared embedding/head | Same |
| Checkpoint precision | MXFP8 E4M3 weights + uint8 E8M0 scales, 1x32 groups; BF16 embedding/head/gates; FP32 iHC/sinks | Same |

Tokenizer, chat template and generation assets are copied unchanged from the pinned source.
The registered Transformers `indexed_attention` config tag represents the source DSA layers.
The H20 Marlin backend requires width 64. The two untied BF16 token matrices alone
occupy 30,932,992 bytes, exceeding the 30 MB target; the checkpoint is 70,661,788 bytes. TP=4 Marlin also requires a 128-wide
expert intermediate partition, hence a 512-wide full expert intermediate size.
Checkpoint precision matches the source ModelOpt MXFP8 config. Marlin uses BF16
activations at runtime (W8A16).

Transformers on one GPU and vLLM at TP=4 were validated on NVIDIA H20 with English, Chinese and
long DSA prompts. The vLLM run enabled three-token MTP and CUDA Graph. Its separate
test environment used vLLM 0.32.0, Torch 2.13.0+cu130, CUDA 13.0, Transformers 5.19.0
and FlashInfer 0.7.0.post1; the Transformers/SGLang environment is listed below.

| File path | Size |
|------|------|
| model.safetensors | 70.7MB |


### Example usage:

- vLLM (✅ verified)

```bash
vllm serve tiny-random/hy4-preview-fp8 \
  --tensor-parallel-size 4 \
  --kernel-config '{"linear_backend":"marlin","moe_backend":"marlin"}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --max-model-len 4096 \
  --max-num-batched-tokens 512 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.3 \
  --compilation-config '{"cudagraph_capture_sizes":[1,2,4,8,16]}'
```

- SGLang (unverified)

  - Unverified: SGLang's native Hy4 MXFP8 backend requires SM100+; the available H20 GPUs are SM90. Use the verified vLLM Marlin example on H20.
  - This command targets four one-GPU nodes with NODE_RANK=0..3, a shared DIST_INIT_ADDR and an available local NCCL_PORT. TP=4, NEXTN/MTP and CUDA Graph are enabled.

```bash
model_id=tiny-random/hy4-preview-fp8
node_rank=${NODE_RANK:-0}
dist_init_addr=${DIST_INIT_ADDR}
nccl_port=${NCCL_PORT}
export SGLANG_DG_CACHE_DIR=/tmp/hy4-deepgemm-$node_rank
sglang serve \
  --model-path $model_id \
  --tp-size 4 \
  --nnodes 4 \
  --node-rank $node_rank \
  --dist-init-addr $dist_init_addr \
  --nccl-port $nccl_port \
  --fp8-gemm-backend deep_gemm \
  --moe-runner-backend deep_gemm \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --context-length 4096 \
  --max-total-tokens 4096 \
  --chunked-prefill-size 512 \
  --mem-fraction-static 0.3 \
  --cuda-graph-config '{"decode": {"bs": [1, 2, 4]}}' \
  --disable-custom-all-reduce
```

- Transformers (✅ verified)

```python
import torch
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer, FineGrainedFP8Config
model_id = "tiny-random/hy4-preview-fp8"
config = AutoConfig.from_pretrained(model_id)
precision = config.quantization_config["quantization"]
# Adapt the loader descriptor in memory; the checkpoint keeps the source ModelOpt config.
config.quantization_config = FineGrainedFP8Config(
    quant_method="mxfp8", weight_block_size=(1, 32), scale_fmt="ue8m0",
    modules_to_not_convert=[name for name in precision["exclude_modules"] if name != "model.layers.0.mlp.gate"],
).to_dict()
# Transformers 5.19.0 implements Hy4's DSA and learned sinks only with eager attention.
model = AutoModelForCausalLM.from_pretrained(
    model_id, config=config, dtype=torch.bfloat16, device_map="cuda", attn_implementation="eager", experts_implementation="eager",
    key_mapping={r"\.weight_scale$": ".weight_scale_inv", r"\.(gate_up_proj|down_proj)_scale$": r".\1_scale_inv"},
)
# Source MXFP8 stores E8M0 scale bits in uint8 containers.
for module in model.modules():
    for name, param in list(module.named_parameters(recurse=False)):
        if name.endswith("_scale_inv") and param.dtype == torch.uint8:
            setattr(module, name, torch.nn.Parameter(param.detach().view(torch.float8_e8m0fnu), requires_grad=False))
tokenizer = AutoTokenizer.from_pretrained(model_id)
inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Hello"}], tokenize=True,
    add_generation_prompt=True, return_dict=True, return_tensors="pt",
).to(model.device)
output = model.generate(**inputs, do_sample=False, max_new_tokens=10)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

### Codes to create this repo (✅ verified):

<details><summary>Click to expand</summary>

```python
import json
import re
from copy import deepcopy
from pathlib import Path
from shutil import copyfile
import torch
from huggingface_hub import hf_hub_download, list_repo_files
from safetensors.torch import save_file
from transformers import AutoConfig, AutoModelForCausalLM, set_seed
source_model_id = "tencent/Hy4-preview-FP8"
save_folder = "/tmp/tiny-random/hy4-preview-fp8"
source_revision = '4215ec29de873a998e849cee902654490c7ff4d1'
set_seed(42)
folder = Path(save_folder)
folder.mkdir(parents=True, exist_ok=True)
raw = json.loads(Path(hf_hub_download(source_model_id, "config.json", revision=source_revision)).read_text())
original_quantization = raw.pop("quantization_config")
raw.update(
    hidden_size=64, num_hidden_layers=10, num_attention_heads=8,
    intermediate_size=128, moe_intermediate_size=512, n_routed_experts=32,
    q_lora_rank=64, qk_nope_head_dim=32, v_head_dim=32, qk_head_dim=96,
    layer_types=["indexed_attention"] * 10,
)
raw["indexer_types"] = raw["indexer_types"][:10]
raw["mlp_layer_types"] = raw["mlp_layer_types"][:10]
(folder / "config.json").write_text(json.dumps(raw, indent=2))
config = AutoConfig.from_pretrained(folder)
# Transformers 5.19.0 implements Hy4's DSA and learned sinks only with eager attention.
model = AutoModelForCausalLM.from_config(config, dtype=torch.bfloat16, attn_implementation="eager")
backbone = model.model
last = backbone.layers[-1]
model.model.mtp_layers = torch.nn.ModuleList([torch.nn.ModuleDict({
    "enorm": deepcopy(backbone.norm), "hnorm": deepcopy(backbone.norm),
    "eh_proj": torch.nn.Linear(2 * config.hidden_size, config.hidden_size, bias=False, dtype=torch.bfloat16),
    "input_layernorm": deepcopy(last.input_layernorm), "self_attn": deepcopy(last.self_attn),
    "post_attention_layernorm": deepcopy(last.post_attention_layernorm), "mlp": deepcopy(last.mlp),
    "final_layernorm": deepcopy(backbone.norm),
})])
model.model.mtp_layers.apply(model._init_weights)
def public_name(name):
    for native, source in (("attn_hc", "hc_attn_layer"), ("ffn_hc", "hc_mlp_layer")):
        name = re.sub(r"\." + native + r"\.(fn|base|scale)$", r"." + source + r".hc_pre.hc_\1", name)
    name = name.replace(".self_attn.gate_proj", ".self_attn.linear_gate")
    name = name.replace(".self_attn.sinks", ".self_attn.learnable_sink_param")
    for suffix in ("fn", "base", "scale"):
        name = name.replace(".hc_head.hc_" + suffix, ".hc_head.hc_head_" + suffix)
    return name
state = {}
for name, value in model.state_dict().items():
    name = public_name(name)
    full_precision = name.endswith((".hc_fn", ".hc_base", ".hc_scale", ".hc_head_fn", ".hc_head_base", ".hc_head_scale"))
    full_precision |= name.endswith((".learnable_sink_param", ".e_score_correction_bias")) and not name.startswith("model.mtp_layers.")
    state[name] = value.detach().to(torch.float32 if full_precision else torch.bfloat16).cpu().contiguous()
# Keep the source exclusions, pruning module names absent from the reduced model.
source_names = {public_name(name): name for name, _ in model.named_modules()}
source_names.update({public_name(name): name for name, _ in model.named_parameters()})
exclusions = [name for name in original_quantization["quantization"]["exclude_modules"] if name in source_names]
for name in list(state):
    excluded = any(name == prefix or name.startswith(prefix + ".") for prefix in exclusions)
    expert = name.endswith((".experts.gate_up_proj", ".experts.down_proj"))
    linear = name.endswith(".weight") and state[name].ndim == 2
    if excluded or not (expert or linear):
        continue
    weight = state[name].float()
    blocks = weight.reshape(*weight.shape[:-1], weight.shape[-1] // 32, 32)
    exponent = torch.ceil(torch.log2((blocks.abs().amax(-1) / 448).clamp_min(2.0 ** -127))).clamp(-127, 127)
    scale = torch.exp2(exponent)
    state[name] = (blocks / scale.unsqueeze(-1)).clamp(-448, 448).reshape(weight.shape).to(torch.float8_e4m3fn).contiguous()
    scale_name = name.removesuffix(".weight") + ".weight_scale" if linear else name + "_scale"
    state[scale_name] = (exponent + 127).to(torch.uint8).contiguous()
save_file(state, folder / "model.safetensors", metadata={"format": "pt"})
raw["quantization_config"] = original_quantization
(folder / "config.json").write_text(json.dumps(raw, indent=2))
assets = {"tokenizer.json", "tokenizer_config.json", "chat_template.jinja", "generation_config.json", "LICENSE"}
for name in sorted(set(list_repo_files(source_model_id, revision=source_revision)) & assets):
    copyfile(hf_hub_download(source_model_id, name, revision=source_revision), folder / name)
```

</details>

### Printing the model:

<details><summary>Click to expand</summary>

```text
HYV4ForCausalLM(
  (model): HYV4Model(
    (embed_tokens): Embedding(120832, 64, padding_idx=120002)
    (layers): ModuleList(
      (0): HYV4DecoderLayer(
        (self_attn): HYV4Attention(
          (q_a_proj): Linear(in_features=64, out_features=64, bias=False)
          (q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
          (q_b_proj): Linear(in_features=64, out_features=768, bias=False)
          (kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
          (kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
          (kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
          (o_proj): Linear(in_features=256, out_features=64, bias=False)
          (indexer): HYV4Indexer(
            (wq_b): Linear(in_features=64, out_features=4096, bias=False)
            (wk): Linear(in_features=64, out_features=128, bias=False)
            (k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
            (weights_proj): Linear(in_features=64, out_features=32, bias=False)
          )
          (gate_proj): Linear(in_features=64, out_features=256, bias=False)
        )
        (mlp): HYV4MLP(
          (gate_proj): Linear(in_features=64, out_features=128, bias=False)
          (up_proj): Linear(in_features=64, out_features=128, bias=False)
          (down_proj): Linear(in_features=128, out_features=64, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (attn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
        (ffn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
      )
      (1): HYV4DecoderLayer(
        (self_attn): HYV4Attention(
          (q_a_proj): Linear(in_features=64, out_features=64, bias=False)
          (q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
          (q_b_proj): Linear(in_features=64, out_features=768, bias=False)
          (kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
          (kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
          (kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
          (o_proj): Linear(in_features=256, out_features=64, bias=False)
          (indexer): HYV4Indexer(
            (wq_b): Linear(in_features=64, out_features=4096, bias=False)
            (wk): Linear(in_features=64, out_features=128, bias=False)
            (k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
            (weights_proj): Linear(in_features=64, out_features=32, bias=False)
          )
          (gate_proj): Linear(in_features=64, out_features=256, bias=False)
        )
        (mlp): HYV4MoE(
          (experts): HYV4Experts()
          (gate): HYV4TopkRouter()
          (shared_experts): HYV4MLP(
            (gate_proj): Linear(in_features=64, out_features=512, bias=False)
            (up_proj): Linear(in_features=64, out_features=512, bias=False)
            (down_proj): Linear(in_features=512, out_features=64, bias=False)
            (act_fn): SiLUActivation()
          )
        )
        (input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (attn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
        (ffn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
      )
      (2-4): 3 x HYV4DecoderLayer(
        (self_attn): HYV4Attention(
          (q_a_proj): Linear(in_features=64, out_features=64, bias=False)
          (q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
          (q_b_proj): Linear(in_features=64, out_features=768, bias=False)
          (kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
          (kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
          (kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
          (o_proj): Linear(in_features=256, out_features=64, bias=False)
          (gate_proj): Linear(in_features=64, out_features=256, bias=False)
        )
        (mlp): HYV4MoE(
          (experts): HYV4Experts()
          (gate): HYV4TopkRouter()
          (shared_experts): HYV4MLP(
            (gate_proj): Linear(in_features=64, out_features=512, bias=False)
            (up_proj): Linear(in_features=64, out_features=512, bias=False)
            (down_proj): Linear(in_features=512, out_features=64, bias=False)
            (act_fn): SiLUActivation()
          )
        )
        (input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (attn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
        (ffn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
      )
      (5): HYV4DecoderLayer(
        (self_attn): HYV4Attention(
          (q_a_proj): Linear(in_features=64, out_features=64, bias=False)
          (q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
          (q_b_proj): Linear(in_features=64, out_features=768, bias=False)
          (kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
          (kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
          (kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
          (o_proj): Linear(in_features=256, out_features=64, bias=False)
          (indexer): HYV4Indexer(
            (wq_b): Linear(in_features=64, out_features=4096, bias=False)
            (wk): Linear(in_features=64, out_features=128, bias=False)
            (k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
            (weights_proj): Linear(in_features=64, out_features=32, bias=False)
          )
          (gate_proj): Linear(in_features=64, out_features=256, bias=False)
        )
        (mlp): HYV4MoE(
          (experts): HYV4Experts()
          (gate): HYV4TopkRouter()
          (shared_experts): HYV4MLP(
            (gate_proj): Linear(in_features=64, out_features=512, bias=False)
            (up_proj): Linear(in_features=64, out_features=512, bias=False)
            (down_proj): Linear(in_features=512, out_features=64, bias=False)
            (act_fn): SiLUActivation()
          )
        )
        (input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (attn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
        (ffn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
      )
      (6-8): 3 x HYV4DecoderLayer(
        (self_attn): HYV4Attention(
          (q_a_proj): Linear(in_features=64, out_features=64, bias=False)
          (q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
          (q_b_proj): Linear(in_features=64, out_features=768, bias=False)
          (kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
          (kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
          (kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
          (o_proj): Linear(in_features=256, out_features=64, bias=False)
          (gate_proj): Linear(in_features=64, out_features=256, bias=False)
        )
        (mlp): HYV4MoE(
          (experts): HYV4Experts()
          (gate): HYV4TopkRouter()
          (shared_experts): HYV4MLP(
            (gate_proj): Linear(in_features=64, out_features=512, bias=False)
            (up_proj): Linear(in_features=64, out_features=512, bias=False)
            (down_proj): Linear(in_features=512, out_features=64, bias=False)
            (act_fn): SiLUActivation()
          )
        )
        (input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (attn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
        (ffn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
      )
      (9): HYV4DecoderLayer(
        (self_attn): HYV4Attention(
          (q_a_proj): Linear(in_features=64, out_features=64, bias=False)
          (q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
          (q_b_proj): Linear(in_features=64, out_features=768, bias=False)
          (kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
          (kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
          (kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
          (o_proj): Linear(in_features=256, out_features=64, bias=False)
          (indexer): HYV4Indexer(
            (wq_b): Linear(in_features=64, out_features=4096, bias=False)
            (wk): Linear(in_features=64, out_features=128, bias=False)
            (k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
            (weights_proj): Linear(in_features=64, out_features=32, bias=False)
          )
          (gate_proj): Linear(in_features=64, out_features=256, bias=False)
        )
        (mlp): HYV4MoE(
          (experts): HYV4Experts()
          (gate): HYV4TopkRouter()
          (shared_experts): HYV4MLP(
            (gate_proj): Linear(in_features=64, out_features=512, bias=False)
            (up_proj): Linear(in_features=64, out_features=512, bias=False)
            (down_proj): Linear(in_features=512, out_features=64, bias=False)
            (act_fn): SiLUActivation()
          )
        )
        (input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (attn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
        (ffn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
      )
    )
    (norm): HYV4RMSNorm((64,), eps=1e-05)
    (rotary_emb): HYV4RotaryEmbedding()
    (hc_head): HYV4HyperHead(
      (input_norm): HYV4UnweightedRMSNorm()
    )
    (mtp_layers): ModuleList(
      (0): ModuleDict(
        (enorm): HYV4RMSNorm((64,), eps=1e-05)
        (hnorm): HYV4RMSNorm((64,), eps=1e-05)
        (eh_proj): Linear(in_features=128, out_features=64, bias=False)
        (input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (self_attn): HYV4Attention(
          (q_a_proj): Linear(in_features=64, out_features=64, bias=False)
          (q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
          (q_b_proj): Linear(in_features=64, out_features=768, bias=False)
          (kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
          (kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
          (kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
          (o_proj): Linear(in_features=256, out_features=64, bias=False)
          (indexer): HYV4Indexer(
            (wq_b): Linear(in_features=64, out_features=4096, bias=False)
            (wk): Linear(in_features=64, out_features=128, bias=False)
            (k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
            (weights_proj): Linear(in_features=64, out_features=32, bias=False)
          )
          (gate_proj): Linear(in_features=64, out_features=256, bias=False)
        )
        (post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (mlp): HYV4MoE(
          (experts): HYV4Experts()
          (gate): HYV4TopkRouter()
          (shared_experts): HYV4MLP(
            (gate_proj): Linear(in_features=64, out_features=512, bias=False)
            (up_proj): Linear(in_features=64, out_features=512, bias=False)
            (down_proj): Linear(in_features=512, out_features=64, bias=False)
            (act_fn): SiLUActivation()
          )
        )
        (final_layernorm): HYV4RMSNorm((64,), eps=1e-05)
      )
    )
  )
  (lm_head): Linear(in_features=64, out_features=120832, bias=False)
)
```

</details>

### Test environment:

- transformers: 5.19.0
- huggingface_hub: 1.33.0
- safetensors: 0.8.0
- accelerate: 1.15.0
- sglang: 0.5.23.dev384+g9ab74b851
- kernels: 0.17.0
- Python: 3.12.13
- CUDA: 13.0
- torch (Transformers/SGLang): 2.14.1
- flashinfer-python (Transformers/SGLang): 0.7.2
- vllm: 0.32.0
- torch (vLLM): 2.13.0+cu130
- flashinfer-python (vLLM): 0.7.0.post1
- CUDA (vLLM): 13.0
### Change log:
- 2026-10-23: Initial version.
This repo is generated by GPT-6.1-Sol

	---
	library_name: transformers
	base_model:
	- tencent/Hy4-preview-FP8
	---

	This tiny model is intended for debugging. It is randomly initialized using the configuration adapted from [tencent/Hy4-preview-FP8](https://huggingface.co/tencent/Hy4-preview-FP8).

	Structural comparison with `tencent/Hy4-preview-FP8`:

	\| Feature \| Source \| Tiny \|
	\|---\|---\|---\|
	\| Architecture / input \| HYV4; text \| Same \|
	\| Vocabulary / tied embeddings \| 120,832 / false \| Same \|
	\| Backbone layers / hidden size \| 78 / 6,144 \| 10 / 64 \|
	\| MLP layout \| 1 dense layer, then MoE \| Same \|
	\| Routed experts / top-k / shared experts \| 256 / 8 / 1 \| 32 / 8 / 1 \|
	\| Dense / expert intermediate size \| 18,432 / 2,048 \| 128 / 512 \|
	\| Attention Q heads; non-RoPE / RoPE / V dimensions \| 64; 192 / 64 / 256 \| 8; 32 / 64 / 32 \|
	\| Q / KV LoRA rank \| 2,048 / 512 \| 64 / 512 \|
	\| Indexer heads / dimension / selected tokens \| 32 / 128 / 2,048 \| Same \|
	\| Full indexer layers \| 0, 1, 5, 9, ... \| 0, 1, 5, 9; remaining layers reuse the preceding indexer \|
	\| Independent Hyper-Connections \| 4 streams; gated pre/post branches \| Same \|
	\| MLA gating / learned attention sinks / SwiGLU clamp \| Elementwise / enabled / 10 \| Same \|
	\| MTP \| 1 plain DSA/MoE decoder; shared embedding/head \| Same \|
	\| Checkpoint precision \| MXFP8 E4M3 weights + uint8 E8M0 scales, 1x32 groups; BF16 embedding/head/gates; FP32 iHC/sinks \| Same \|

	Tokenizer, chat template and generation assets are copied unchanged from the pinned source.
	The registered Transformers `indexed_attention` config tag represents the source DSA layers.
	The H20 Marlin backend requires width 64. The two untied BF16 token matrices alone
	occupy 30,932,992 bytes, exceeding the 30 MB target; the checkpoint is 70,661,788 bytes. TP=4 Marlin also requires a 128-wide
	expert intermediate partition, hence a 512-wide full expert intermediate size.
	Checkpoint precision matches the source ModelOpt MXFP8 config. Marlin uses BF16
	activations at runtime (W8A16).

	Transformers on one GPU and vLLM at TP=4 were validated on NVIDIA H20 with English, Chinese and
	long DSA prompts. The vLLM run enabled three-token MTP and CUDA Graph. Its separate
	test environment used vLLM 0.32.0, Torch 2.13.0+cu130, CUDA 13.0, Transformers 5.19.0
	and FlashInfer 0.7.0.post1; the Transformers/SGLang environment is listed below.

	\| File path \| Size \|
	\|------\|------\|
	\| model.safetensors \| 70.7MB \|


	### Example usage:

	- vLLM (✅ verified)

	```bash
	vllm serve tiny-random/hy4-preview-fp8 \
	--tensor-parallel-size 4 \
	--kernel-config '{"linear_backend":"marlin","moe_backend":"marlin"}' \
	--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
	--max-model-len 4096 \
	--max-num-batched-tokens 512 \
	--max-num-seqs 4 \
	--gpu-memory-utilization 0.3 \
	--compilation-config '{"cudagraph_capture_sizes":[1,2,4,8,16]}'
	```

	- SGLang (unverified)

	- Unverified: SGLang's native Hy4 MXFP8 backend requires SM100+; the available H20 GPUs are SM90. Use the verified vLLM Marlin example on H20.
	- This command targets four one-GPU nodes with NODE_RANK=0..3, a shared DIST_INIT_ADDR and an available local NCCL_PORT. TP=4, NEXTN/MTP and CUDA Graph are enabled.

	```bash
	model_id=tiny-random/hy4-preview-fp8
	node_rank=${NODE_RANK:-0}
	dist_init_addr=${DIST_INIT_ADDR}
	nccl_port=${NCCL_PORT}
	export SGLANG_DG_CACHE_DIR=/tmp/hy4-deepgemm-$node_rank
	sglang serve \
	--model-path $model_id \
	--tp-size 4 \
	--nnodes 4 \
	--node-rank $node_rank \
	--dist-init-addr $dist_init_addr \
	--nccl-port $nccl_port \
	--fp8-gemm-backend deep_gemm \
	--moe-runner-backend deep_gemm \
	--speculative-algorithm NEXTN \
	--speculative-num-steps 3 \
	--speculative-eagle-topk 1 \
	--speculative-num-draft-tokens 4 \
	--context-length 4096 \
	--max-total-tokens 4096 \
	--chunked-prefill-size 512 \
	--mem-fraction-static 0.3 \
	--cuda-graph-config '{"decode": {"bs": [1, 2, 4]}}' \
	--disable-custom-all-reduce
	```

	- Transformers (✅ verified)

	```python
	import torch
	from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer, FineGrainedFP8Config

	model_id = "tiny-random/hy4-preview-fp8"
	config = AutoConfig.from_pretrained(model_id)
	precision = config.quantization_config["quantization"]
	# Adapt the loader descriptor in memory; the checkpoint keeps the source ModelOpt config.
	config.quantization_config = FineGrainedFP8Config(
	quant_method="mxfp8", weight_block_size=(1, 32), scale_fmt="ue8m0",
	modules_to_not_convert=[name for name in precision["exclude_modules"] if name != "model.layers.0.mlp.gate"],
	).to_dict()
	# Transformers 5.19.0 implements Hy4's DSA and learned sinks only with eager attention.
	model = AutoModelForCausalLM.from_pretrained(
	model_id, config=config, dtype=torch.bfloat16, device_map="cuda", attn_implementation="eager", experts_implementation="eager",
	key_mapping={r"\.weight_scale$": ".weight_scale_inv", r"\.(gate_up_proj\|down_proj)_scale$": r".\1_scale_inv"},
	)
	# Source MXFP8 stores E8M0 scale bits in uint8 containers.
	for module in model.modules():
	for name, param in list(module.named_parameters(recurse=False)):
	if name.endswith("_scale_inv") and param.dtype == torch.uint8:
	setattr(module, name, torch.nn.Parameter(param.detach().view(torch.float8_e8m0fnu), requires_grad=False))
	tokenizer = AutoTokenizer.from_pretrained(model_id)
	inputs = tokenizer.apply_chat_template(
	[{"role": "user", "content": "Hello"}], tokenize=True,
	add_generation_prompt=True, return_dict=True, return_tensors="pt",
	).to(model.device)
	output = model.generate(**inputs, do_sample=False, max_new_tokens=10)
	print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
	```

	### Codes to create this repo (✅ verified):

	<details><summary>Click to expand</summary>

	```python
	import json
	import re
	from copy import deepcopy
	from pathlib import Path
	from shutil import copyfile

	import torch
	from huggingface_hub import hf_hub_download, list_repo_files
	from safetensors.torch import save_file
	from transformers import AutoConfig, AutoModelForCausalLM, set_seed

	source_model_id = "tencent/Hy4-preview-FP8"
	save_folder = "/tmp/tiny-random/hy4-preview-fp8"
	source_revision = '4215ec29de873a998e849cee902654490c7ff4d1'
	set_seed(42)
	folder = Path(save_folder)
	folder.mkdir(parents=True, exist_ok=True)
	raw = json.loads(Path(hf_hub_download(source_model_id, "config.json", revision=source_revision)).read_text())
	original_quantization = raw.pop("quantization_config")
	raw.update(
	hidden_size=64, num_hidden_layers=10, num_attention_heads=8,
	intermediate_size=128, moe_intermediate_size=512, n_routed_experts=32,
	q_lora_rank=64, qk_nope_head_dim=32, v_head_dim=32, qk_head_dim=96,
	layer_types=["indexed_attention"] * 10,
	)
	raw["indexer_types"] = raw["indexer_types"][:10]
	raw["mlp_layer_types"] = raw["mlp_layer_types"][:10]
	(folder / "config.json").write_text(json.dumps(raw, indent=2))
	config = AutoConfig.from_pretrained(folder)
	# Transformers 5.19.0 implements Hy4's DSA and learned sinks only with eager attention.
	model = AutoModelForCausalLM.from_config(config, dtype=torch.bfloat16, attn_implementation="eager")
	backbone = model.model
	last = backbone.layers[-1]
	model.model.mtp_layers = torch.nn.ModuleList([torch.nn.ModuleDict({
	"enorm": deepcopy(backbone.norm), "hnorm": deepcopy(backbone.norm),
	"eh_proj": torch.nn.Linear(2 * config.hidden_size, config.hidden_size, bias=False, dtype=torch.bfloat16),
	"input_layernorm": deepcopy(last.input_layernorm), "self_attn": deepcopy(last.self_attn),
	"post_attention_layernorm": deepcopy(last.post_attention_layernorm), "mlp": deepcopy(last.mlp),
	"final_layernorm": deepcopy(backbone.norm),
	})])
	model.model.mtp_layers.apply(model._init_weights)

	def public_name(name):
	for native, source in (("attn_hc", "hc_attn_layer"), ("ffn_hc", "hc_mlp_layer")):
	name = re.sub(r"\." + native + r"\.(fn\|base\|scale)$", r"." + source + r".hc_pre.hc_\1", name)
	name = name.replace(".self_attn.gate_proj", ".self_attn.linear_gate")
	name = name.replace(".self_attn.sinks", ".self_attn.learnable_sink_param")
	for suffix in ("fn", "base", "scale"):
	name = name.replace(".hc_head.hc_" + suffix, ".hc_head.hc_head_" + suffix)
	return name

	state = {}
	for name, value in model.state_dict().items():
	name = public_name(name)
	full_precision = name.endswith((".hc_fn", ".hc_base", ".hc_scale", ".hc_head_fn", ".hc_head_base", ".hc_head_scale"))
	full_precision \|= name.endswith((".learnable_sink_param", ".e_score_correction_bias")) and not name.startswith("model.mtp_layers.")
	state[name] = value.detach().to(torch.float32 if full_precision else torch.bfloat16).cpu().contiguous()
	# Keep the source exclusions, pruning module names absent from the reduced model.
	source_names = {public_name(name): name for name, _ in model.named_modules()}
	source_names.update({public_name(name): name for name, _ in model.named_parameters()})
	exclusions = [name for name in original_quantization["quantization"]["exclude_modules"] if name in source_names]
	for name in list(state):
	excluded = any(name == prefix or name.startswith(prefix + ".") for prefix in exclusions)
	expert = name.endswith((".experts.gate_up_proj", ".experts.down_proj"))
	linear = name.endswith(".weight") and state[name].ndim == 2
	if excluded or not (expert or linear):
	continue
	weight = state[name].float()
	blocks = weight.reshape(*weight.shape[:-1], weight.shape[-1] // 32, 32)
	exponent = torch.ceil(torch.log2((blocks.abs().amax(-1) / 448).clamp_min(2.0 ** -127))).clamp(-127, 127)
	scale = torch.exp2(exponent)
	state[name] = (blocks / scale.unsqueeze(-1)).clamp(-448, 448).reshape(weight.shape).to(torch.float8_e4m3fn).contiguous()
	scale_name = name.removesuffix(".weight") + ".weight_scale" if linear else name + "_scale"
	state[scale_name] = (exponent + 127).to(torch.uint8).contiguous()
	save_file(state, folder / "model.safetensors", metadata={"format": "pt"})
	raw["quantization_config"] = original_quantization
	(folder / "config.json").write_text(json.dumps(raw, indent=2))
	assets = {"tokenizer.json", "tokenizer_config.json", "chat_template.jinja", "generation_config.json", "LICENSE"}
	for name in sorted(set(list_repo_files(source_model_id, revision=source_revision)) & assets):
	copyfile(hf_hub_download(source_model_id, name, revision=source_revision), folder / name)
	```

	</details>

	### Printing the model:

	<details><summary>Click to expand</summary>

	```text
	HYV4ForCausalLM(
	(model): HYV4Model(
	(embed_tokens): Embedding(120832, 64, padding_idx=120002)
	(layers): ModuleList(
	(0): HYV4DecoderLayer(
	(self_attn): HYV4Attention(
	(q_a_proj): Linear(in_features=64, out_features=64, bias=False)
	(q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
	(q_b_proj): Linear(in_features=64, out_features=768, bias=False)
	(kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
	(kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
	(kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
	(o_proj): Linear(in_features=256, out_features=64, bias=False)
	(indexer): HYV4Indexer(
	(wq_b): Linear(in_features=64, out_features=4096, bias=False)
	(wk): Linear(in_features=64, out_features=128, bias=False)
	(k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
	(weights_proj): Linear(in_features=64, out_features=32, bias=False)
	)
	(gate_proj): Linear(in_features=64, out_features=256, bias=False)
	)
	(mlp): HYV4MLP(
	(gate_proj): Linear(in_features=64, out_features=128, bias=False)
	(up_proj): Linear(in_features=64, out_features=128, bias=False)
	(down_proj): Linear(in_features=128, out_features=64, bias=False)
	(act_fn): SiLUActivation()
	)
	(input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
	(post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
	(attn_hc): HYV4HyperConnection(
	(input_norm): HYV4UnweightedRMSNorm()
	)
	(ffn_hc): HYV4HyperConnection(
	(input_norm): HYV4UnweightedRMSNorm()
	)
	)
	(1): HYV4DecoderLayer(
	(self_attn): HYV4Attention(
	(q_a_proj): Linear(in_features=64, out_features=64, bias=False)
	(q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
	(q_b_proj): Linear(in_features=64, out_features=768, bias=False)
	(kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
	(kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
	(kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
	(o_proj): Linear(in_features=256, out_features=64, bias=False)
	(indexer): HYV4Indexer(
	(wq_b): Linear(in_features=64, out_features=4096, bias=False)
	(wk): Linear(in_features=64, out_features=128, bias=False)
	(k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
	(weights_proj): Linear(in_features=64, out_features=32, bias=False)
	)
	(gate_proj): Linear(in_features=64, out_features=256, bias=False)
	)
	(mlp): HYV4MoE(
	(experts): HYV4Experts()
	(gate): HYV4TopkRouter()
	(shared_experts): HYV4MLP(
	(gate_proj): Linear(in_features=64, out_features=512, bias=False)
	(up_proj): Linear(in_features=64, out_features=512, bias=False)
	(down_proj): Linear(in_features=512, out_features=64, bias=False)
	(act_fn): SiLUActivation()
	)
	)
	(input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
	(post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
	(attn_hc): HYV4HyperConnection(
	(input_norm): HYV4UnweightedRMSNorm()
	)
	(ffn_hc): HYV4HyperConnection(
	(input_norm): HYV4UnweightedRMSNorm()
	)
	)
	(2-4): 3 x HYV4DecoderLayer(
	(self_attn): HYV4Attention(
	(q_a_proj): Linear(in_features=64, out_features=64, bias=False)
	(q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
	(q_b_proj): Linear(in_features=64, out_features=768, bias=False)
	(kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
	(kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
	(kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
	(o_proj): Linear(in_features=256, out_features=64, bias=False)
	(gate_proj): Linear(in_features=64, out_features=256, bias=False)
	)
	(mlp): HYV4MoE(
	(experts): HYV4Experts()
	(gate): HYV4TopkRouter()
	(shared_experts): HYV4MLP(
	(gate_proj): Linear(in_features=64, out_features=512, bias=False)
	(up_proj): Linear(in_features=64, out_features=512, bias=False)
	(down_proj): Linear(in_features=512, out_features=64, bias=False)
	(act_fn): SiLUActivation()
	)
	)
	(input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
	(post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
	(attn_hc): HYV4HyperConnection(
	(input_norm): HYV4UnweightedRMSNorm()
	)
	(ffn_hc): HYV4HyperConnection(
	(input_norm): HYV4UnweightedRMSNorm()
	)
	)
	(5): HYV4DecoderLayer(
	(self_attn): HYV4Attention(
	(q_a_proj): Linear(in_features=64, out_features=64, bias=False)
	(q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
	(q_b_proj): Linear(in_features=64, out_features=768, bias=False)
	(kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
	(kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
	(kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
	(o_proj): Linear(in_features=256, out_features=64, bias=False)
	(indexer): HYV4Indexer(
	(wq_b): Linear(in_features=64, out_features=4096, bias=False)
	(wk): Linear(in_features=64, out_features=128, bias=False)
	(k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
	(weights_proj): Linear(in_features=64, out_features=32, bias=False)
	)
	(gate_proj): Linear(in_features=64, out_features=256, bias=False)
	)
	(mlp): HYV4MoE(
	(experts): HYV4Experts()
	(gate): HYV4TopkRouter()
	(shared_experts): HYV4MLP(
	(gate_proj): Linear(in_features=64, out_features=512, bias=False)
	(up_proj): Linear(in_features=64, out_features=512, bias=False)
	(down_proj): Linear(in_features=512, out_features=64, bias=False)
	(act_fn): SiLUActivation()
	)
	)
	(input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
	(post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
	(attn_hc): HYV4HyperConnection(
	(input_norm): HYV4UnweightedRMSNorm()
	)
	(ffn_hc): HYV4HyperConnection(
	(input_norm): HYV4UnweightedRMSNorm()
	)
	)
	(6-8): 3 x HYV4DecoderLayer(
	(self_attn): HYV4Attention(
	(q_a_proj): Linear(in_features=64, out_features=64, bias=False)
	(q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
	(q_b_proj): Linear(in_features=64, out_features=768, bias=False)
	(kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
	(kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
	(kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
	(o_proj): Linear(in_features=256, out_features=64, bias=False)
	(gate_proj): Linear(in_features=64, out_features=256, bias=False)
	)
	(mlp): HYV4MoE(
	(experts): HYV4Experts()
	(gate): HYV4TopkRouter()
	(shared_experts): HYV4MLP(
	(gate_proj): Linear(in_features=64, out_features=512, bias=False)
	(up_proj): Linear(in_features=64, out_features=512, bias=False)
	(down_proj): Linear(in_features=512, out_features=64, bias=False)
	(act_fn): SiLUActivation()
	)
	)
	(input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
	(post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
	(attn_hc): HYV4HyperConnection(
	(input_norm): HYV4UnweightedRMSNorm()
	)
	(ffn_hc): HYV4HyperConnection(
	(input_norm): HYV4UnweightedRMSNorm()
	)
	)
	(9): HYV4DecoderLayer(
	(self_attn): HYV4Attention(
	(q_a_proj): Linear(in_features=64, out_features=64, bias=False)
	(q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
	(q_b_proj): Linear(in_features=64, out_features=768, bias=False)
	(kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
	(kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
	(kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
	(o_proj): Linear(in_features=256, out_features=64, bias=False)
	(indexer): HYV4Indexer(
	(wq_b): Linear(in_features=64, out_features=4096, bias=False)
	(wk): Linear(in_features=64, out_features=128, bias=False)
	(k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
	(weights_proj): Linear(in_features=64, out_features=32, bias=False)
	)
	(gate_proj): Linear(in_features=64, out_features=256, bias=False)
	)
	(mlp): HYV4MoE(
	(experts): HYV4Experts()
	(gate): HYV4TopkRouter()
	(shared_experts): HYV4MLP(
	(gate_proj): Linear(in_features=64, out_features=512, bias=False)
	(up_proj): Linear(in_features=64, out_features=512, bias=False)
	(down_proj): Linear(in_features=512, out_features=64, bias=False)
	(act_fn): SiLUActivation()
	)
	)
	(input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
	(post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
	(attn_hc): HYV4HyperConnection(
	(input_norm): HYV4UnweightedRMSNorm()
	)
	(ffn_hc): HYV4HyperConnection(
	(input_norm): HYV4UnweightedRMSNorm()
	)
	)
	)
	(norm): HYV4RMSNorm((64,), eps=1e-05)
	(rotary_emb): HYV4RotaryEmbedding()
	(hc_head): HYV4HyperHead(
	(input_norm): HYV4UnweightedRMSNorm()
	)
	(mtp_layers): ModuleList(
	(0): ModuleDict(
	(enorm): HYV4RMSNorm((64,), eps=1e-05)
	(hnorm): HYV4RMSNorm((64,), eps=1e-05)
	(eh_proj): Linear(in_features=128, out_features=64, bias=False)
	(input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
	(self_attn): HYV4Attention(
	(q_a_proj): Linear(in_features=64, out_features=64, bias=False)
	(q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
	(q_b_proj): Linear(in_features=64, out_features=768, bias=False)
	(kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
	(kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
	(kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
	(o_proj): Linear(in_features=256, out_features=64, bias=False)
	(indexer): HYV4Indexer(
	(wq_b): Linear(in_features=64, out_features=4096, bias=False)
	(wk): Linear(in_features=64, out_features=128, bias=False)
	(k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
	(weights_proj): Linear(in_features=64, out_features=32, bias=False)
	)
	(gate_proj): Linear(in_features=64, out_features=256, bias=False)
	)
	(post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
	(mlp): HYV4MoE(
	(experts): HYV4Experts()
	(gate): HYV4TopkRouter()
	(shared_experts): HYV4MLP(
	(gate_proj): Linear(in_features=64, out_features=512, bias=False)
	(up_proj): Linear(in_features=64, out_features=512, bias=False)
	(down_proj): Linear(in_features=512, out_features=64, bias=False)
	(act_fn): SiLUActivation()
	)
	)
	(final_layernorm): HYV4RMSNorm((64,), eps=1e-05)
	)
	)
	)
	(lm_head): Linear(in_features=64, out_features=120832, bias=False)
	)
	```

	</details>

	### Test environment:

	- transformers: 5.19.0
	- huggingface_hub: 1.33.0
	- safetensors: 0.8.0
	- accelerate: 1.15.0
	- sglang: 0.5.23.dev384+g9ab74b851
	- kernels: 0.17.0
	- Python: 3.12.13
	- CUDA: 13.0
	- torch (Transformers/SGLang): 2.14.1
	- flashinfer-python (Transformers/SGLang): 0.7.2
	- vllm: 0.32.0
	- torch (vLLM): 2.13.0+cu130
	- flashinfer-python (vLLM): 0.7.0.post1
	- CUDA (vLLM): 13.0

	### Change log:

	- 2026-10-23: Initial version.

	This repo is generated by GPT-6.1-Sol