| --- |
| library_name: transformers |
| base_model: |
| - XHToken/Spark-X2.6-4B |
| --- |
| |
| This tiny model is intended for debugging. It is randomly initialized using the configuration adapted from [XHToken/Spark-X2.6-4B](https://huggingface.co/XHToken/Spark-X2.6-4B). |
|
|
| Structural comparison with `XHToken/Spark-X2.6-4B`: |
|
|
| | Feature | Source | Tiny | |
| |---|---:|---:| |
| | Vocabulary / tied embeddings | 131,072 / yes | same | |
| | Layers | 36 | 8; two 3-sliding + 1-full cycles | |
| | Hidden size / head dimension | 2,560 / 256 | 16 / 32 | |
| | Query / KV heads | 16 / 4 | 1 / 1 | |
| | MLP width | 10,240 | 64 | |
| | Sliding window | 512 | same | |
| | Per-head output gate | sigmoid | same | |
| | Stored precision | BF16 | BF16 | |
|
|
| The source implementation invokes eager attention directly. The tiny definition |
| retains that source behavior rather than falsely advertising an optimized |
| attention backend. |
|
|
| | File path | Size | |
| |------|------| |
| | model.safetensors | 4.3MB | |
|
|
|
|
| ### Example usage: |
|
|
| - Transformers (✅ verified) |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "tiny-random/spark-x2.6" |
| device = "cuda" if torch.cuda.is_available() else "cpu" |
| dtype = torch.bfloat16 if device == "cuda" else torch.float32 |
| # The pinned Spark implementation calls eager_attention_forward directly; |
| # it exposes no SDPA/FlashAttention dispatch path to select instead. |
| model = AutoModelForCausalLM.from_pretrained(model_id, dtype=dtype, trust_remote_code=True).to(device).eval() |
| tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) |
| inputs = tokenizer("Hello", return_tensors="pt").to(device) |
| output = model.generate(**inputs, do_sample=False, min_new_tokens=10, max_new_tokens=10) |
| print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)) |
| ``` |
|
|
| ### Codes to create this repo (✅ verified): |
|
|
| <details><summary>Click to expand</summary> |
|
|
| ```python |
| import json |
| from pathlib import Path |
| from shutil import copyfile |
| |
| import torch |
| from huggingface_hub import hf_hub_download, list_repo_files |
| from transformers import AutoConfig, AutoModelForCausalLM, set_seed |
| |
| source_model_id = "XHToken/Spark-X2.6-4B" |
| save_folder = "/tmp/tiny-random/spark-x26" |
| source_revision = "66e3ae7bee54e0a523d692cb8ef841f35409807b" |
| set_seed(42) |
| folder = Path(save_folder) |
| folder.mkdir(parents=True, exist_ok=True) |
| raw = json.loads(Path(hf_hub_download(source_model_id, "config.json", revision=source_revision)).read_text()) |
| raw.update( |
| hidden_size=16, head_dim=32, num_attention_heads=1, num_key_value_heads=1, |
| intermediate_size=64, num_hidden_layers=8, |
| layer_types=(["sliding_attention"] * 3 + ["full_attention"]) * 2, |
| ) |
| (folder / "config.json").write_text(json.dumps(raw, indent=2), encoding="utf-8") |
| for filename in ("configuration_spark.py", "modeling_spark.py"): |
| source = Path(hf_hub_download(source_model_id, filename, revision=source_revision)).read_text(encoding="utf-8") |
| if filename == "modeling_spark.py": |
| # Spark's pinned remote implementation targets Transformers 4.57. |
| # These are API compatibility adaptations for Transformers 5.19; |
| # the attention, state-dict names, and tied-embedding semantics stay unchanged. |
| source = source.replace( |
| '_tied_weights_keys = ["lm_head.weight"]', |
| '_tied_weights_keys = {"lm_head.weight": "model.embedding.weight"}', |
| ) |
| source = source.replace( |
| '"input_embeds": inputs_embeds,\n "attention_mask"', |
| '"inputs_embeds": inputs_embeds,\n "attention_mask"', |
| ) |
| source = source.replace( |
| ' "cache_position": cache_position,\n "past_key_values"', |
| ' "past_key_values"', |
| ) |
| (folder / filename).write_text(source, encoding="utf-8") |
| config = AutoConfig.from_pretrained(folder, trust_remote_code=True) |
| model = AutoModelForCausalLM.from_config(config, dtype=torch.bfloat16, trust_remote_code=True) |
| with torch.no_grad(): |
| for name, parameter in sorted(model.named_parameters()): |
| if name.endswith("weight") and "norm" in name: |
| parameter.fill_(1) |
| else: |
| torch.nn.init.normal_(parameter, mean=0, std=0.02) |
| model.save_pretrained(folder, safe_serialization=True) |
| assets = {"chat_template.jinja", "generation_config.json", "merges.txt", "special_tokens_map.json", "tokenizer.json", "tokenizer_config.json", "vocab.json"} |
| for filename in sorted(set(list_repo_files(source_model_id, revision=source_revision)) & assets): |
| copyfile(hf_hub_download(source_model_id, filename, revision=source_revision), folder / filename) |
| ``` |
|
|
| </details> |
|
|
| ### Printing the model: |
|
|
| <details><summary>Click to expand</summary> |
|
|
| ```text |
| Spark2_5ForCausalLM( |
| (model): Spark2_5Model( |
| (embedding): Embedding(131072, 16, padding_idx=2) |
| (layers): ModuleList( |
| (0-7): 8 x Spark2_5DecoderLayer( |
| (self_attn): Spark2_5Attention( |
| (q_k_v_proj): Linear(in_features=16, out_features=96, bias=False) |
| (g_proj): Linear(in_features=16, out_features=1, bias=False) |
| (out_proj): Linear(in_features=32, out_features=16, bias=False) |
| ) |
| (mlp): Spark2_5MLP( |
| (gate_proj): Linear(in_features=16, out_features=64, bias=False) |
| (up_proj): Linear(in_features=16, out_features=64, bias=False) |
| (down_proj): Linear(in_features=64, out_features=16, bias=False) |
| (act_fn): GELUActivation() |
| ) |
| (input_layernorm): Spark2_5RMSNorm((16,), eps=1e-06) |
| (post_attention_layernorm): Spark2_5RMSNorm((16,), eps=1e-06) |
| ) |
| ) |
| (norm): Spark2_5RMSNorm((16,), eps=1e-06) |
| ) |
| (lm_head): Linear(in_features=16, out_features=131072, bias=False) |
| ) |
| ``` |
|
|
| </details> |
|
|
| ### Test environment: |
|
|
| - huggingface_hub: 2.3.0 |
| - safetensors: 0.8.0 |
| - tokenizers: 0.23.3 |
| - torch: 2.13.0 |
| - transformers: 5.19.0 |
| |
| ### Change log: |
| |
| - 2026-10-23: Initial version. |
| |
| This repo is generated by GPT-6.1-Sol |
