Instructions to use tiny-random/spark-x2.6 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tiny-random/spark-x2.6 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="tiny-random/spark-x2.6", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("tiny-random/spark-x2.6", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tiny-random/spark-x2.6 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tiny-random/spark-x2.6"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiny-random/spark-x2.6",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/tiny-random/spark-x2.6

SGLang

How to use tiny-random/spark-x2.6 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tiny-random/spark-x2.6" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiny-random/spark-x2.6",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tiny-random/spark-x2.6" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiny-random/spark-x2.6",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use tiny-random/spark-x2.6 with Docker Model Runner:
```
docker model run hf.co/tiny-random/spark-x2.6
```

yujiepan commited on 3 days ago

Commit

0663073

verified ·

1 Parent(s): 6f4c30a

Upload folder using huggingface_hub

Browse files

Files changed (3) hide show

README.md +4 -4
config.json +1 -5
model.safetensors +2 -2

README.md CHANGED Viewed

@@ -11,7 +11,7 @@ Structural comparison with `XHToken/Spark-X2.6-4B`:
 | Feature | Source | Tiny |
 |---|---:|---:|
 | Vocabulary / tied embeddings | 131,072 / yes | same |
-| Layers | 36 | 12; three 3-sliding + 1-full cycles |
 | Hidden size / head dimension | 2,560 / 256 | 16 / 32 |
 | Query / KV heads | 16 / 4 | 1 / 1 |
 | MLP width | 10,240 | 64 |
@@ -70,8 +70,8 @@ folder.mkdir(parents=True, exist_ok=True)
 raw = json.loads(Path(hf_hub_download(source_model_id, "config.json", revision=source_revision)).read_text())
 raw.update(
     hidden_size=16, head_dim=32, num_attention_heads=1, num_key_value_heads=1,
-    intermediate_size=64, num_hidden_layers=12,
-    layer_types=(["sliding_attention"] * 3 + ["full_attention"]) * 3,
 )
 (folder / "config.json").write_text(json.dumps(raw, indent=2), encoding="utf-8")
 for filename in ("configuration_spark.py", "modeling_spark.py"):
@@ -118,7 +118,7 @@ Spark2_5ForCausalLM(
   (model): Spark2_5Model(
     (embedding): Embedding(131072, 16, padding_idx=2)
     (layers): ModuleList(
-      (0-11): 12 x Spark2_5DecoderLayer(
         (self_attn): Spark2_5Attention(
           (q_k_v_proj): Linear(in_features=16, out_features=96, bias=False)
           (g_proj): Linear(in_features=16, out_features=1, bias=False)

 | Feature | Source | Tiny |
 |---|---:|---:|
 | Vocabulary / tied embeddings | 131,072 / yes | same |
+| Layers | 36 | 8; two 3-sliding + 1-full cycles |
 | Hidden size / head dimension | 2,560 / 256 | 16 / 32 |
 | Query / KV heads | 16 / 4 | 1 / 1 |
 | MLP width | 10,240 | 64 |
 raw = json.loads(Path(hf_hub_download(source_model_id, "config.json", revision=source_revision)).read_text())
 raw.update(
     hidden_size=16, head_dim=32, num_attention_heads=1, num_key_value_heads=1,
+    intermediate_size=64, num_hidden_layers=8,
+    layer_types=(["sliding_attention"] * 3 + ["full_attention"]) * 2,
 )
 (folder / "config.json").write_text(json.dumps(raw, indent=2), encoding="utf-8")
 for filename in ("configuration_spark.py", "modeling_spark.py"):
   (model): Spark2_5Model(
     (embedding): Embedding(131072, 16, padding_idx=2)
     (layers): ModuleList(
+      (0-7): 8 x Spark2_5DecoderLayer(
         (self_attn): Spark2_5Attention(
           (q_k_v_proj): Linear(in_features=16, out_features=96, bias=False)
           (g_proj): Linear(in_features=16, out_features=1, bias=False)

config.json CHANGED Viewed

@@ -27,17 +27,13 @@
     "sliding_attention",
     "sliding_attention",
     "sliding_attention",
-    "full_attention",
-    "sliding_attention",
-    "sliding_attention",
-    "sliding_attention",
     "full_attention"
   ],
   "max_position_embeddings": 1048576,
   "mlp_bias": false,
   "model_type": "spark2_5",
   "num_attention_heads": 1,
-  "num_hidden_layers": 12,
   "num_key_value_heads": 1,
   "pad_token_id": 2,
   "rms_norm_eps": 1e-06,

     "sliding_attention",
     "sliding_attention",
     "sliding_attention",
     "full_attention"
   ],
   "max_position_embeddings": 1048576,
   "mlp_bias": false,
   "model_type": "spark2_5",
   "num_attention_heads": 1,
+  "num_hidden_layers": 8,
   "num_key_value_heads": 1,
   "pad_token_id": 2,
   "rms_norm_eps": 1e-06,

model.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:254aecb0c2a75d91b96d368caed3996123e13e782ab142dc83bb02b18a2ff473
-size 4328808

 version https://git-lfs.github.com/spec/v1
+oid sha256:7bfcde4ade27abb88956789cbd2b32689868df8b70bf370dbeacc9dac793637f
+size 4284048