Instructions to use XHToken/Spark-X2.6-4B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use XHToken/Spark-X2.6-4B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="XHToken/Spark-X2.6-4B", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("XHToken/Spark-X2.6-4B", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use XHToken/Spark-X2.6-4B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "XHToken/Spark-X2.6-4B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "XHToken/Spark-X2.6-4B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/XHToken/Spark-X2.6-4B

SGLang

How to use XHToken/Spark-X2.6-4B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "XHToken/Spark-X2.6-4B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "XHToken/Spark-X2.6-4B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "XHToken/Spark-X2.6-4B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "XHToken/Spark-X2.6-4B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use XHToken/Spark-X2.6-4B with Docker Model Runner:
```
docker model run hf.co/XHToken/Spark-X2.6-4B
```

Spark-X2.6-4B Turbo: long prompts without the memory growth or the slowdown

#33

by infosave - opened 5 days ago

Discussion

infosave

5 days ago

Hi XHToken team and everyone here!

A follow-up to #22. I've published a Turbo build of Spark-X2.6-4B for cortiq: https://huggingface.co/infosave/Spark-X2.6-4B-turbo-cmf

The point of it is simple: a long prompt should stop eating memory and slowing the model down. Spark-X2.6 already does half the job, since 27 of its 36 layers look back only 512 tokens. The Turbo file takes care of the other 9. Once a prompt goes past 8192 tokens, those layers fold everything older into a far field of fixed size and keep the last 1800 rows exact. On top of that, a few hundred rows that carry the things you can't afford to blur (passwords, codes, names) are picked out and kept exact too. Prompts under 8192 tokens run plain exact attention, same as the standard file.

Here's what that looks like on a 16 GB RTX 2000 Ada over Vulkan:

After a 64,000-token prompt the Turbo file still decodes at 56 tok/s (the standard one drops to 18.5), and the card stays at 4.2 GB instead of climbing to 7.9. From 16k tokens on, the Turbo curves for speed and memory stay flat. On a MacBook Air M4 after a 14k prompt it decodes at 33 tok/s in 2.9 GB of memory (mlx-lm 4-bit: 30.9 tok/s, 5.1 GB).

To be upfront about the trade-off: past the first 8192 tokens the far context is an approximation. In our needle tests (a fact planted at a random spot in Wikipedia text, eight tries per length) the Turbo file found 6 of 8 at 60k, 120k and 240k characters, and every miss was the same kind: a made-up password like qX909926vT. The standard file found 8 of 8 up to 120k and 4 of 8 at 240k. The far field runs on Metal, Vulkan and DX12; on a plain CPU the file behaves like the standard one.

If you want to try it:

cargo install cortiq-cli   # 0.8.64 or later
hf download infosave/Spark-X2.6-4B-turbo-cmf spark-x2.6-4b-q4tp-turbo.cmf --local-dir .
CMF_MAX_SEQ=131072 cortiq run spark-x2.6-4b-q4tp-turbo.cmf --prompt-file report.txt --no-think

The model card has the full numbers (Mac, NVIDIA, CPU, perplexity) and how the turbo file is built. Weights stay under your Apache-2.0 license.

Thanks again for a really capable small model! Happy to test anything you'd like on long contexts.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment