Instructions to use tiny-random/hy4-preview-fp8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tiny-random/hy4-preview-fp8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="tiny-random/hy4-preview-fp8")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("tiny-random/hy4-preview-fp8")
model = AutoModelForCausalLM.from_pretrained("tiny-random/hy4-preview-fp8", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tiny-random/hy4-preview-fp8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tiny-random/hy4-preview-fp8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiny-random/hy4-preview-fp8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/tiny-random/hy4-preview-fp8

SGLang

How to use tiny-random/hy4-preview-fp8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tiny-random/hy4-preview-fp8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiny-random/hy4-preview-fp8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tiny-random/hy4-preview-fp8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiny-random/hy4-preview-fp8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use tiny-random/hy4-preview-fp8 with Docker Model Runner:
```
docker model run hf.co/tiny-random/hy4-preview-fp8
```

This tiny model is intended for debugging. It is randomly initialized using the configuration adapted from tencent/Hy4-preview-FP8.

Structural comparison with tencent/Hy4-preview-FP8:

Feature	Source	Tiny
Architecture / input	HYV4; text	Same
Vocabulary / tied embeddings	120,832 / false	Same
Backbone layers / hidden size	78 / 6,144	10 / 64
MLP layout	1 dense layer, then MoE	Same
Routed experts / top-k / shared experts	256 / 8 / 1	32 / 8 / 1
Dense / expert intermediate size	18,432 / 2,048	128 / 512
Attention Q heads; non-RoPE / RoPE / V dimensions	64; 192 / 64 / 256	8; 32 / 64 / 32
Q / KV LoRA rank	2,048 / 512	64 / 512
Indexer heads / dimension / selected tokens	32 / 128 / 2,048	Same
Full indexer layers	0, 1, 5, 9, ...	0, 1, 5, 9; remaining layers reuse the preceding indexer
Independent Hyper-Connections	4 streams; gated pre/post branches	Same
MLA gating / learned attention sinks / SwiGLU clamp	Elementwise / enabled / 10	Same
MTP	1 plain DSA/MoE decoder; shared embedding/head	Same
Checkpoint precision	MXFP8 E4M3 weights + uint8 E8M0 scales, 1x32 groups; BF16 embedding/head/gates; FP32 iHC/sinks	Same

Tokenizer, chat template and generation assets are copied unchanged from the pinned source. The registered Transformers indexed_attention config tag represents the source DSA layers. The H20 Marlin backend requires width 64. The two untied BF16 token matrices alone occupy 30,932,992 bytes, exceeding the 30 MB target; the checkpoint is 70,661,788 bytes. TP=4 Marlin also requires a 128-wide expert intermediate partition, hence a 512-wide full expert intermediate size. Checkpoint precision matches the source ModelOpt MXFP8 config. Marlin uses BF16 activations at runtime (W8A16).

Transformers on one GPU and vLLM at TP=4 were validated on NVIDIA H20 with English, Chinese and long DSA prompts. The vLLM run enabled three-token MTP and CUDA Graph. Its separate test environment used vLLM 0.32.0, Torch 2.13.0+cu130, CUDA 13.0, Transformers 5.19.0 and FlashInfer 0.7.0.post1; the Transformers/SGLang environment is listed below.

File path	Size
model.safetensors	70.7MB

Example usage:

vLLM (✅ verified)

vllm serve tiny-random/hy4-preview-fp8 \
  --tensor-parallel-size 4 \
  --kernel-config '{"linear_backend":"marlin","moe_backend":"marlin"}' \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --max-model-len 4096 \
  --max-num-batched-tokens 512 \
  --max-num-seqs 4 \
  --gpu-memory-utilization 0.3 \
  --compilation-config '{"cudagraph_capture_sizes":[1,2,4,8,16]}'

SGLang (unverified)
- Unverified: SGLang's native Hy4 MXFP8 backend requires SM100+; the available H20 GPUs are SM90. Use the verified vLLM Marlin example on H20.
- This command targets four one-GPU nodes with NODE_RANK=0..3, a shared DIST_INIT_ADDR and an available local NCCL_PORT. TP=4, NEXTN/MTP and CUDA Graph are enabled.

model_id=tiny-random/hy4-preview-fp8
node_rank=${NODE_RANK:-0}
dist_init_addr=${DIST_INIT_ADDR}
nccl_port=${NCCL_PORT}
export SGLANG_DG_CACHE_DIR=/tmp/hy4-deepgemm-$node_rank
sglang serve \
  --model-path $model_id \
  --tp-size 4 \
  --nnodes 4 \
  --node-rank $node_rank \
  --dist-init-addr $dist_init_addr \
  --nccl-port $nccl_port \
  --fp8-gemm-backend deep_gemm \
  --moe-runner-backend deep_gemm \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --context-length 4096 \
  --max-total-tokens 4096 \
  --chunked-prefill-size 512 \
  --mem-fraction-static 0.3 \
  --cuda-graph-config '{"decode": {"bs": [1, 2, 4]}}' \
  --disable-custom-all-reduce

Transformers (✅ verified)

import torch
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer, FineGrainedFP8Config

model_id = "tiny-random/hy4-preview-fp8"
config = AutoConfig.from_pretrained(model_id)
precision = config.quantization_config["quantization"]
# Adapt the loader descriptor in memory; the checkpoint keeps the source ModelOpt config.
config.quantization_config = FineGrainedFP8Config(
    quant_method="mxfp8", weight_block_size=(1, 32), scale_fmt="ue8m0",
    modules_to_not_convert=[name for name in precision["exclude_modules"] if name != "model.layers.0.mlp.gate"],
).to_dict()
# Transformers 5.19.0 implements Hy4's DSA and learned sinks only with eager attention.
model = AutoModelForCausalLM.from_pretrained(
    model_id, config=config, dtype=torch.bfloat16, device_map="cuda", attn_implementation="eager", experts_implementation="eager",
    key_mapping={r"\.weight_scale$": ".weight_scale_inv", r"\.(gate_up_proj|down_proj)_scale$": r".\1_scale_inv"},
)
# Source MXFP8 stores E8M0 scale bits in uint8 containers.
for module in model.modules():
    for name, param in list(module.named_parameters(recurse=False)):
        if name.endswith("_scale_inv") and param.dtype == torch.uint8:
            setattr(module, name, torch.nn.Parameter(param.detach().view(torch.float8_e8m0fnu), requires_grad=False))
tokenizer = AutoTokenizer.from_pretrained(model_id)
inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Hello"}], tokenize=True,
    add_generation_prompt=True, return_dict=True, return_tensors="pt",
).to(model.device)
output = model.generate(**inputs, do_sample=False, max_new_tokens=10)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Codes to create this repo (✅ verified):

Click to expand

import json
import re
from copy import deepcopy
from pathlib import Path
from shutil import copyfile

import torch
from huggingface_hub import hf_hub_download, list_repo_files
from safetensors.torch import save_file
from transformers import AutoConfig, AutoModelForCausalLM, set_seed

source_model_id = "tencent/Hy4-preview-FP8"
save_folder = "/tmp/tiny-random/hy4-preview-fp8"
source_revision = '4215ec29de873a998e849cee902654490c7ff4d1'
set_seed(42)
folder = Path(save_folder)
folder.mkdir(parents=True, exist_ok=True)
raw = json.loads(Path(hf_hub_download(source_model_id, "config.json", revision=source_revision)).read_text())
original_quantization = raw.pop("quantization_config")
raw.update(
    hidden_size=64, num_hidden_layers=10, num_attention_heads=8,
    intermediate_size=128, moe_intermediate_size=512, n_routed_experts=32,
    q_lora_rank=64, qk_nope_head_dim=32, v_head_dim=32, qk_head_dim=96,
    layer_types=["indexed_attention"] * 10,
)
raw["indexer_types"] = raw["indexer_types"][:10]
raw["mlp_layer_types"] = raw["mlp_layer_types"][:10]
(folder / "config.json").write_text(json.dumps(raw, indent=2))
config = AutoConfig.from_pretrained(folder)
# Transformers 5.19.0 implements Hy4's DSA and learned sinks only with eager attention.
model = AutoModelForCausalLM.from_config(config, dtype=torch.bfloat16, attn_implementation="eager")
backbone = model.model
last = backbone.layers[-1]
model.model.mtp_layers = torch.nn.ModuleList([torch.nn.ModuleDict({
    "enorm": deepcopy(backbone.norm), "hnorm": deepcopy(backbone.norm),
    "eh_proj": torch.nn.Linear(2 * config.hidden_size, config.hidden_size, bias=False, dtype=torch.bfloat16),
    "input_layernorm": deepcopy(last.input_layernorm), "self_attn": deepcopy(last.self_attn),
    "post_attention_layernorm": deepcopy(last.post_attention_layernorm), "mlp": deepcopy(last.mlp),
    "final_layernorm": deepcopy(backbone.norm),
})])
model.model.mtp_layers.apply(model._init_weights)

def public_name(name):
    for native, source in (("attn_hc", "hc_attn_layer"), ("ffn_hc", "hc_mlp_layer")):
        name = re.sub(r"\." + native + r"\.(fn|base|scale)$", r"." + source + r".hc_pre.hc_\1", name)
    name = name.replace(".self_attn.gate_proj", ".self_attn.linear_gate")
    name = name.replace(".self_attn.sinks", ".self_attn.learnable_sink_param")
    for suffix in ("fn", "base", "scale"):
        name = name.replace(".hc_head.hc_" + suffix, ".hc_head.hc_head_" + suffix)
    return name

state = {}
for name, value in model.state_dict().items():
    name = public_name(name)
    full_precision = name.endswith((".hc_fn", ".hc_base", ".hc_scale", ".hc_head_fn", ".hc_head_base", ".hc_head_scale"))
    full_precision |= name.endswith((".learnable_sink_param", ".e_score_correction_bias")) and not name.startswith("model.mtp_layers.")
    state[name] = value.detach().to(torch.float32 if full_precision else torch.bfloat16).cpu().contiguous()
# Keep the source exclusions, pruning module names absent from the reduced model.
source_names = {public_name(name): name for name, _ in model.named_modules()}
source_names.update({public_name(name): name for name, _ in model.named_parameters()})
exclusions = [name for name in original_quantization["quantization"]["exclude_modules"] if name in source_names]
for name in list(state):
    excluded = any(name == prefix or name.startswith(prefix + ".") for prefix in exclusions)
    expert = name.endswith((".experts.gate_up_proj", ".experts.down_proj"))
    linear = name.endswith(".weight") and state[name].ndim == 2
    if excluded or not (expert or linear):
        continue
    weight = state[name].float()
    blocks = weight.reshape(*weight.shape[:-1], weight.shape[-1] // 32, 32)
    exponent = torch.ceil(torch.log2((blocks.abs().amax(-1) / 448).clamp_min(2.0 ** -127))).clamp(-127, 127)
    scale = torch.exp2(exponent)
    state[name] = (blocks / scale.unsqueeze(-1)).clamp(-448, 448).reshape(weight.shape).to(torch.float8_e4m3fn).contiguous()
    scale_name = name.removesuffix(".weight") + ".weight_scale" if linear else name + "_scale"
    state[scale_name] = (exponent + 127).to(torch.uint8).contiguous()
save_file(state, folder / "model.safetensors", metadata={"format": "pt"})
raw["quantization_config"] = original_quantization
(folder / "config.json").write_text(json.dumps(raw, indent=2))
assets = {"tokenizer.json", "tokenizer_config.json", "chat_template.jinja", "generation_config.json", "LICENSE"}
for name in sorted(set(list_repo_files(source_model_id, revision=source_revision)) & assets):
    copyfile(hf_hub_download(source_model_id, name, revision=source_revision), folder / name)

Printing the model:

Click to expand

HYV4ForCausalLM(
  (model): HYV4Model(
    (embed_tokens): Embedding(120832, 64, padding_idx=120002)
    (layers): ModuleList(
      (0): HYV4DecoderLayer(
        (self_attn): HYV4Attention(
          (q_a_proj): Linear(in_features=64, out_features=64, bias=False)
          (q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
          (q_b_proj): Linear(in_features=64, out_features=768, bias=False)
          (kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
          (kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
          (kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
          (o_proj): Linear(in_features=256, out_features=64, bias=False)
          (indexer): HYV4Indexer(
            (wq_b): Linear(in_features=64, out_features=4096, bias=False)
            (wk): Linear(in_features=64, out_features=128, bias=False)
            (k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
            (weights_proj): Linear(in_features=64, out_features=32, bias=False)
          )
          (gate_proj): Linear(in_features=64, out_features=256, bias=False)
        )
        (mlp): HYV4MLP(
          (gate_proj): Linear(in_features=64, out_features=128, bias=False)
          (up_proj): Linear(in_features=64, out_features=128, bias=False)
          (down_proj): Linear(in_features=128, out_features=64, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (attn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
        (ffn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
      )
      (1): HYV4DecoderLayer(
        (self_attn): HYV4Attention(
          (q_a_proj): Linear(in_features=64, out_features=64, bias=False)
          (q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
          (q_b_proj): Linear(in_features=64, out_features=768, bias=False)
          (kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
          (kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
          (kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
          (o_proj): Linear(in_features=256, out_features=64, bias=False)
          (indexer): HYV4Indexer(
            (wq_b): Linear(in_features=64, out_features=4096, bias=False)
            (wk): Linear(in_features=64, out_features=128, bias=False)
            (k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
            (weights_proj): Linear(in_features=64, out_features=32, bias=False)
          )
          (gate_proj): Linear(in_features=64, out_features=256, bias=False)
        )
        (mlp): HYV4MoE(
          (experts): HYV4Experts()
          (gate): HYV4TopkRouter()
          (shared_experts): HYV4MLP(
            (gate_proj): Linear(in_features=64, out_features=512, bias=False)
            (up_proj): Linear(in_features=64, out_features=512, bias=False)
            (down_proj): Linear(in_features=512, out_features=64, bias=False)
            (act_fn): SiLUActivation()
          )
        )
        (input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (attn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
        (ffn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
      )
      (2-4): 3 x HYV4DecoderLayer(
        (self_attn): HYV4Attention(
          (q_a_proj): Linear(in_features=64, out_features=64, bias=False)
          (q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
          (q_b_proj): Linear(in_features=64, out_features=768, bias=False)
          (kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
          (kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
          (kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
          (o_proj): Linear(in_features=256, out_features=64, bias=False)
          (gate_proj): Linear(in_features=64, out_features=256, bias=False)
        )
        (mlp): HYV4MoE(
          (experts): HYV4Experts()
          (gate): HYV4TopkRouter()
          (shared_experts): HYV4MLP(
            (gate_proj): Linear(in_features=64, out_features=512, bias=False)
            (up_proj): Linear(in_features=64, out_features=512, bias=False)
            (down_proj): Linear(in_features=512, out_features=64, bias=False)
            (act_fn): SiLUActivation()
          )
        )
        (input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (attn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
        (ffn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
      )
      (5): HYV4DecoderLayer(
        (self_attn): HYV4Attention(
          (q_a_proj): Linear(in_features=64, out_features=64, bias=False)
          (q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
          (q_b_proj): Linear(in_features=64, out_features=768, bias=False)
          (kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
          (kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
          (kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
          (o_proj): Linear(in_features=256, out_features=64, bias=False)
          (indexer): HYV4Indexer(
            (wq_b): Linear(in_features=64, out_features=4096, bias=False)
            (wk): Linear(in_features=64, out_features=128, bias=False)
            (k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
            (weights_proj): Linear(in_features=64, out_features=32, bias=False)
          )
          (gate_proj): Linear(in_features=64, out_features=256, bias=False)
        )
        (mlp): HYV4MoE(
          (experts): HYV4Experts()
          (gate): HYV4TopkRouter()
          (shared_experts): HYV4MLP(
            (gate_proj): Linear(in_features=64, out_features=512, bias=False)
            (up_proj): Linear(in_features=64, out_features=512, bias=False)
            (down_proj): Linear(in_features=512, out_features=64, bias=False)
            (act_fn): SiLUActivation()
          )
        )
        (input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (attn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
        (ffn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
      )
      (6-8): 3 x HYV4DecoderLayer(
        (self_attn): HYV4Attention(
          (q_a_proj): Linear(in_features=64, out_features=64, bias=False)
          (q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
          (q_b_proj): Linear(in_features=64, out_features=768, bias=False)
          (kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
          (kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
          (kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
          (o_proj): Linear(in_features=256, out_features=64, bias=False)
          (gate_proj): Linear(in_features=64, out_features=256, bias=False)
        )
        (mlp): HYV4MoE(
          (experts): HYV4Experts()
          (gate): HYV4TopkRouter()
          (shared_experts): HYV4MLP(
            (gate_proj): Linear(in_features=64, out_features=512, bias=False)
            (up_proj): Linear(in_features=64, out_features=512, bias=False)
            (down_proj): Linear(in_features=512, out_features=64, bias=False)
            (act_fn): SiLUActivation()
          )
        )
        (input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (attn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
        (ffn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
      )
      (9): HYV4DecoderLayer(
        (self_attn): HYV4Attention(
          (q_a_proj): Linear(in_features=64, out_features=64, bias=False)
          (q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
          (q_b_proj): Linear(in_features=64, out_features=768, bias=False)
          (kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
          (kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
          (kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
          (o_proj): Linear(in_features=256, out_features=64, bias=False)
          (indexer): HYV4Indexer(
            (wq_b): Linear(in_features=64, out_features=4096, bias=False)
            (wk): Linear(in_features=64, out_features=128, bias=False)
            (k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
            (weights_proj): Linear(in_features=64, out_features=32, bias=False)
          )
          (gate_proj): Linear(in_features=64, out_features=256, bias=False)
        )
        (mlp): HYV4MoE(
          (experts): HYV4Experts()
          (gate): HYV4TopkRouter()
          (shared_experts): HYV4MLP(
            (gate_proj): Linear(in_features=64, out_features=512, bias=False)
            (up_proj): Linear(in_features=64, out_features=512, bias=False)
            (down_proj): Linear(in_features=512, out_features=64, bias=False)
            (act_fn): SiLUActivation()
          )
        )
        (input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (attn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
        (ffn_hc): HYV4HyperConnection(
          (input_norm): HYV4UnweightedRMSNorm()
        )
      )
    )
    (norm): HYV4RMSNorm((64,), eps=1e-05)
    (rotary_emb): HYV4RotaryEmbedding()
    (hc_head): HYV4HyperHead(
      (input_norm): HYV4UnweightedRMSNorm()
    )
    (mtp_layers): ModuleList(
      (0): ModuleDict(
        (enorm): HYV4RMSNorm((64,), eps=1e-05)
        (hnorm): HYV4RMSNorm((64,), eps=1e-05)
        (eh_proj): Linear(in_features=128, out_features=64, bias=False)
        (input_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (self_attn): HYV4Attention(
          (q_a_proj): Linear(in_features=64, out_features=64, bias=False)
          (q_a_layernorm): HYV4RMSNorm((64,), eps=1e-06)
          (q_b_proj): Linear(in_features=64, out_features=768, bias=False)
          (kv_a_proj_with_mqa): Linear(in_features=64, out_features=576, bias=False)
          (kv_a_layernorm): HYV4RMSNorm((512,), eps=1e-06)
          (kv_b_proj): Linear(in_features=512, out_features=512, bias=False)
          (o_proj): Linear(in_features=256, out_features=64, bias=False)
          (indexer): HYV4Indexer(
            (wq_b): Linear(in_features=64, out_features=4096, bias=False)
            (wk): Linear(in_features=64, out_features=128, bias=False)
            (k_norm): LayerNorm((128,), eps=1e-05, elementwise_affine=True, bias=True)
            (weights_proj): Linear(in_features=64, out_features=32, bias=False)
          )
          (gate_proj): Linear(in_features=64, out_features=256, bias=False)
        )
        (post_attention_layernorm): HYV4RMSNorm((64,), eps=1e-05)
        (mlp): HYV4MoE(
          (experts): HYV4Experts()
          (gate): HYV4TopkRouter()
          (shared_experts): HYV4MLP(
            (gate_proj): Linear(in_features=64, out_features=512, bias=False)
            (up_proj): Linear(in_features=64, out_features=512, bias=False)
            (down_proj): Linear(in_features=512, out_features=64, bias=False)
            (act_fn): SiLUActivation()
          )
        )
        (final_layernorm): HYV4RMSNorm((64,), eps=1e-05)
      )
    )
  )
  (lm_head): Linear(in_features=64, out_features=120832, bias=False)
)

Test environment:

transformers: 5.19.0
huggingface_hub: 1.33.0
safetensors: 0.8.0
accelerate: 1.15.0
sglang: 0.5.23.dev384+g9ab74b851
kernels: 0.17.0
Python: 3.12.13
CUDA: 13.0
torch (Transformers/SGLang): 2.14.1
flashinfer-python (Transformers/SGLang): 0.7.2
vllm: 0.32.0
torch (vLLM): 2.13.0+cu130
flashinfer-python (vLLM): 0.7.0.post1
CUDA (vLLM): 13.0

Change log:

2026-10-23: Initial version.

This repo is generated by GPT-6.1-Sol

Downloads last month: -

Safetensors

Model size

54.6M params

Tensor type

F32

BF16

F8_E4M3

Model tree for tiny-random/hy4-preview-fp8

Base model

tencent/Hy4-preview-FP8

Quantized

(2)

this model