README.md

---
library_name: transformers
base_model:
- XHToken/Spark-X2.6-4B
---

This tiny model is intended for debugging. It is randomly initialized using the configuration adapted from [XHToken/Spark-X2.6-4B](https://huggingface.co/XHToken/Spark-X2.6-4B).

Structural comparison with `XHToken/Spark-X2.6-4B`:

| Feature | Source | Tiny |
|---|---:|---:|
| Vocabulary / tied embeddings | 131,072 / yes | same |
| Layers | 36 | 8; two 3-sliding + 1-full cycles |
| Hidden size / head dimension | 2,560 / 256 | 16 / 32 |
| Query / KV heads | 16 / 4 | 1 / 1 |
| MLP width | 10,240 | 64 |
| Sliding window | 512 | same |
| Per-head output gate | sigmoid | same |
| Stored precision | BF16 | BF16 |

The source implementation invokes eager attention directly. The tiny definition
retains that source behavior rather than falsely advertising an optimized
attention backend.

| File path | Size |
|------|------|
| model.safetensors | 4.3MB |


### Example usage:

- Transformers (✅ verified)

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "tiny-random/spark-x2.6"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32
# The pinned Spark implementation calls eager_attention_forward directly;
# it exposes no SDPA/FlashAttention dispatch path to select instead.
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=dtype, trust_remote_code=True).to(device).eval()
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
inputs = tokenizer("Hello", return_tensors="pt").to(device)
output = model.generate(**inputs, do_sample=False, min_new_tokens=10, max_new_tokens=10)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

### Codes to create this repo (✅ verified):

<details><summary>Click to expand</summary>

```python
import json
from pathlib import Path
from shutil import copyfile

import torch
from huggingface_hub import hf_hub_download, list_repo_files
from transformers import AutoConfig, AutoModelForCausalLM, set_seed

source_model_id = "XHToken/Spark-X2.6-4B"
save_folder = "/tmp/tiny-random/spark-x26"
source_revision = "66e3ae7bee54e0a523d692cb8ef841f35409807b"
set_seed(42)
folder = Path(save_folder)
folder.mkdir(parents=True, exist_ok=True)
raw = json.loads(Path(hf_hub_download(source_model_id, "config.json", revision=source_revision)).read_text())
raw.update(
    hidden_size=16, head_dim=32, num_attention_heads=1, num_key_value_heads=1,
    intermediate_size=64, num_hidden_layers=8,
    layer_types=(["sliding_attention"] * 3 + ["full_attention"]) * 2,
)
(folder / "config.json").write_text(json.dumps(raw, indent=2), encoding="utf-8")
for filename in ("configuration_spark.py", "modeling_spark.py"):
    source = Path(hf_hub_download(source_model_id, filename, revision=source_revision)).read_text(encoding="utf-8")
    if filename == "modeling_spark.py":
        # Spark's pinned remote implementation targets Transformers 4.57.
        # These are API compatibility adaptations for Transformers 5.19;
        # the attention, state-dict names, and tied-embedding semantics stay unchanged.
        source = source.replace(
            '_tied_weights_keys = ["lm_head.weight"]',
            '_tied_weights_keys = {"lm_head.weight": "model.embedding.weight"}',
        )
        source = source.replace(
            '"input_embeds": inputs_embeds,\n                "attention_mask"',
            '"inputs_embeds": inputs_embeds,\n                "attention_mask"',
        )
        source = source.replace(
            '                "cache_position": cache_position,\n                "past_key_values"',
            '                "past_key_values"',
        )
    (folder / filename).write_text(source, encoding="utf-8")
config = AutoConfig.from_pretrained(folder, trust_remote_code=True)
model = AutoModelForCausalLM.from_config(config, dtype=torch.bfloat16, trust_remote_code=True)
with torch.no_grad():
    for name, parameter in sorted(model.named_parameters()):
        if name.endswith("weight") and "norm" in name:
            parameter.fill_(1)
        else:
            torch.nn.init.normal_(parameter, mean=0, std=0.02)
model.save_pretrained(folder, safe_serialization=True)
assets = {"chat_template.jinja", "generation_config.json", "merges.txt", "special_tokens_map.json", "tokenizer.json", "tokenizer_config.json", "vocab.json"}
for filename in sorted(set(list_repo_files(source_model_id, revision=source_revision)) & assets):
    copyfile(hf_hub_download(source_model_id, filename, revision=source_revision), folder / filename)
```

</details>

### Printing the model:

<details><summary>Click to expand</summary>

```text
Spark2_5ForCausalLM(
  (model): Spark2_5Model(
    (embedding): Embedding(131072, 16, padding_idx=2)
    (layers): ModuleList(
      (0-7): 8 x Spark2_5DecoderLayer(
        (self_attn): Spark2_5Attention(
          (q_k_v_proj): Linear(in_features=16, out_features=96, bias=False)
          (g_proj): Linear(in_features=16, out_features=1, bias=False)
          (out_proj): Linear(in_features=32, out_features=16, bias=False)
        )
        (mlp): Spark2_5MLP(
          (gate_proj): Linear(in_features=16, out_features=64, bias=False)
          (up_proj): Linear(in_features=16, out_features=64, bias=False)
          (down_proj): Linear(in_features=64, out_features=16, bias=False)
          (act_fn): GELUActivation()
        )
        (input_layernorm): Spark2_5RMSNorm((16,), eps=1e-06)
        (post_attention_layernorm): Spark2_5RMSNorm((16,), eps=1e-06)
      )
    )
    (norm): Spark2_5RMSNorm((16,), eps=1e-06)
  )
  (lm_head): Linear(in_features=16, out_features=131072, bias=False)
)
```

</details>

### Test environment:

- huggingface_hub: 2.3.0
- safetensors: 0.8.0
- tokenizers: 0.23.3
- torch: 2.13.0
- transformers: 5.19.0

### Change log:

- 2026-10-23: Initial version.

This repo is generated by GPT-6.1-Sol