Instructions to use XHToken/Spark-X2.6-4B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use XHToken/Spark-X2.6-4B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="XHToken/Spark-X2.6-4B", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("XHToken/Spark-X2.6-4B", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use XHToken/Spark-X2.6-4B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "XHToken/Spark-X2.6-4B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "XHToken/Spark-X2.6-4B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/XHToken/Spark-X2.6-4B

SGLang

How to use XHToken/Spark-X2.6-4B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "XHToken/Spark-X2.6-4B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "XHToken/Spark-X2.6-4B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "XHToken/Spark-X2.6-4B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "XHToken/Spark-X2.6-4B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use XHToken/Spark-X2.6-4B with Docker Model Runner:
```
docker model run hf.co/XHToken/Spark-X2.6-4B
```

Spark-X2.6-4B running in the browser on WebGPU (ONNX, official INT4 weights)

#26

by mohdmahmodi - opened 7 days ago

Discussion

mohdmahmodi

7 days ago

Hi XHToken team, thanks for Spark-X2.6!

I converted Spark-X2.6-4B to ONNX so it runs in the browser on WebGPU, with no server and nothing to install.

Demo: https://huggingface.co/spaces/mohdmahmodi/Spark-X2.6-WebGPU
Model files: https://huggingface.co/mohdmahmodi/Spark-X2.6-4B-ONNX-WebGPU
Source and validation: https://github.com/mohdmahmodi/Spark-X2.6-4B-onnx-webgpu

The 4-bit weights are your GPTQ codes from Spark-X2.6-4B-INT4, repacked for onnxruntime-web without changing them. The sliding-window and full-attention layers, partial rotary, output gate, thinking modes and tool calling all work, using the official chat template.

I compared it with the original model in transformers on the same prompts (greedy, thinking off):

	bf16	official INT4	this build
GSM8K, 1,319 problems	90.8%	91.2%	90.5%
BFCL v3 simple, 200 cases	89.0%	89.5%	89.5%
Needle in a haystack, 1k to 16k tokens	12/12	12/12	12/12

The GSM8K gap to INT4 is within noise (McNemar p = 0.18).

On an RTX 3090 Ti in Chrome it does about 92 tok/s with a short prompt and 69 tok/s at 16k tokens of context. The download is 2.77 GB and the browser caches it.

It also works with Transformers.js 4.3:

import { pipeline } from "@huggingface/transformers";

const generator = await pipeline("text-generation", "mohdmahmodi/Spark-X2.6-4B-ONNX-WebGPU", { device: "webgpu", dtype: "q4f16" });
const out = await generator([{ role: "user", content: "Hello!" }], {
  max_new_tokens: 256, tokenizer_encode_kwargs: { enable_thinking: false },
});
console.log(out[0].generated_text.at(-1).content);

I've only tested it on Windows with an NVIDIA GPU so far. If anyone tries it on a Mac, an AMD or Intel GPU, or a phone, let me know how it goes. And if you think it's useful, I'd be glad to see it linked from the model card next to the GGUF.

linchao-2026

SparkLLM org 6 days ago

Thanks for the contribution and detailed validation! We’ve added a WebGPU section to our README linking to your demo, model files, and source code.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment