---
library_name: transformers
base_model:
- XHToken/Spark-X2.6-4B
---
This tiny model is intended for debugging. It is randomly initialized using the configuration adapted from [XHToken/Spark-X2.6-4B](https://huggingface.co/XHToken/Spark-X2.6-4B).
Structural comparison with `XHToken/Spark-X2.6-4B`:
| Feature | Source | Tiny |
|---|---:|---:|
| Vocabulary / tied embeddings | 131,072 / yes | same |
| Layers | 36 | 8; two 3-sliding + 1-full cycles |
| Hidden size / head dimension | 2,560 / 256 | 16 / 32 |
| Query / KV heads | 16 / 4 | 1 / 1 |
| MLP width | 10,240 | 64 |
| Sliding window | 512 | same |
| Per-head output gate | sigmoid | same |
| Stored precision | BF16 | BF16 |
The source implementation invokes eager attention directly. The tiny definition
retains that source behavior rather than falsely advertising an optimized
attention backend.
| File path | Size |
|------|------|
| model.safetensors | 4.3MB |
### Example usage:
- Transformers (✅ verified)
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "tiny-random/spark-x2.6"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32
# The pinned Spark implementation calls eager_attention_forward directly;
# it exposes no SDPA/FlashAttention dispatch path to select instead.
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=dtype, trust_remote_code=True).to(device).eval()
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
inputs = tokenizer("Hello", return_tensors="pt").to(device)
output = model.generate(**inputs, do_sample=False, min_new_tokens=10, max_new_tokens=10)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```
### Codes to create this repo (✅ verified):
Click to expand
```python
import json
from pathlib import Path
from shutil import copyfile
import torch
from huggingface_hub import hf_hub_download, list_repo_files
from transformers import AutoConfig, AutoModelForCausalLM, set_seed
source_model_id = "XHToken/Spark-X2.6-4B"
save_folder = "/tmp/tiny-random/spark-x26"
source_revision = "66e3ae7bee54e0a523d692cb8ef841f35409807b"
set_seed(42)
folder = Path(save_folder)
folder.mkdir(parents=True, exist_ok=True)
raw = json.loads(Path(hf_hub_download(source_model_id, "config.json", revision=source_revision)).read_text())
raw.update(
hidden_size=16, head_dim=32, num_attention_heads=1, num_key_value_heads=1,
intermediate_size=64, num_hidden_layers=8,
layer_types=(["sliding_attention"] * 3 + ["full_attention"]) * 2,
)
(folder / "config.json").write_text(json.dumps(raw, indent=2), encoding="utf-8")
for filename in ("configuration_spark.py", "modeling_spark.py"):
source = Path(hf_hub_download(source_model_id, filename, revision=source_revision)).read_text(encoding="utf-8")
if filename == "modeling_spark.py":
# Spark's pinned remote implementation targets Transformers 4.57.
# These are API compatibility adaptations for Transformers 5.19;
# the attention, state-dict names, and tied-embedding semantics stay unchanged.
source = source.replace(
'_tied_weights_keys = ["lm_head.weight"]',
'_tied_weights_keys = {"lm_head.weight": "model.embedding.weight"}',
)
source = source.replace(
'"input_embeds": inputs_embeds,\n "attention_mask"',
'"inputs_embeds": inputs_embeds,\n "attention_mask"',
)
source = source.replace(
' "cache_position": cache_position,\n "past_key_values"',
' "past_key_values"',
)
(folder / filename).write_text(source, encoding="utf-8")
config = AutoConfig.from_pretrained(folder, trust_remote_code=True)
model = AutoModelForCausalLM.from_config(config, dtype=torch.bfloat16, trust_remote_code=True)
with torch.no_grad():
for name, parameter in sorted(model.named_parameters()):
if name.endswith("weight") and "norm" in name:
parameter.fill_(1)
else:
torch.nn.init.normal_(parameter, mean=0, std=0.02)
model.save_pretrained(folder, safe_serialization=True)
assets = {"chat_template.jinja", "generation_config.json", "merges.txt", "special_tokens_map.json", "tokenizer.json", "tokenizer_config.json", "vocab.json"}
for filename in sorted(set(list_repo_files(source_model_id, revision=source_revision)) & assets):
copyfile(hf_hub_download(source_model_id, filename, revision=source_revision), folder / filename)
```
Click to expand
```text
Spark2_5ForCausalLM(
(model): Spark2_5Model(
(embedding): Embedding(131072, 16, padding_idx=2)
(layers): ModuleList(
(0-7): 8 x Spark2_5DecoderLayer(
(self_attn): Spark2_5Attention(
(q_k_v_proj): Linear(in_features=16, out_features=96, bias=False)
(g_proj): Linear(in_features=16, out_features=1, bias=False)
(out_proj): Linear(in_features=32, out_features=16, bias=False)
)
(mlp): Spark2_5MLP(
(gate_proj): Linear(in_features=16, out_features=64, bias=False)
(up_proj): Linear(in_features=16, out_features=64, bias=False)
(down_proj): Linear(in_features=64, out_features=16, bias=False)
(act_fn): GELUActivation()
)
(input_layernorm): Spark2_5RMSNorm((16,), eps=1e-06)
(post_attention_layernorm): Spark2_5RMSNorm((16,), eps=1e-06)
)
)
(norm): Spark2_5RMSNorm((16,), eps=1e-06)
)
(lm_head): Linear(in_features=16, out_features=131072, bias=False)
)
```