Instructions to use tiny-random/qwen3.8-moe with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tiny-random/qwen3.8-moe with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="tiny-random/qwen3.8-moe")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("tiny-random/qwen3.8-moe")
model = AutoModelForCausalLM.from_pretrained("tiny-random/qwen3.8-moe", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tiny-random/qwen3.8-moe with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tiny-random/qwen3.8-moe"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiny-random/qwen3.8-moe",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/tiny-random/qwen3.8-moe

SGLang

How to use tiny-random/qwen3.8-moe with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tiny-random/qwen3.8-moe" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiny-random/qwen3.8-moe",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tiny-random/qwen3.8-moe" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiny-random/qwen3.8-moe",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use tiny-random/qwen3.8-moe with Docker Model Runner:
```
docker model run hf.co/tiny-random/qwen3.8-moe
```

yujiepan commited on 14 days ago

Commit

c6b32c5

verified ·

1 Parent(s): 8f4aac0

Upload folder using huggingface_hub

Browse files

Files changed (4) hide show

README.md +12 -7
config.json +1 -1
model.safetensors +2 -2
tokenizer.json +2 -2

README.md CHANGED Viewed

@@ -13,7 +13,7 @@ Structural comparison with `Qwen/Qwen3.8-2.4T-A95B`:
 | Input modalities | Text | Same |
 | Vocabulary | 248,320 | Same; special-token IDs preserved |
 | Backbone layers | 92 | 8 |
-| Hidden size | 8,192 | 128 |
 | Attention Q/KV heads; head dim | 64/4; 256 | 32/2; 128 |
 | Attention cycle | 3 linear + 1 full | Same; two complete cycles |
 | Linear-attention key/value heads | 16/128 | 2/16 |
@@ -26,7 +26,7 @@ Structural comparison with `Qwen/Qwen3.8-2.4T-A95B`:
 | File path | Size |
 |------|------|
-| model.safetensors | 205.9MB |
 ### Example usage:
@@ -39,9 +39,11 @@ Structural comparison with `Qwen/Qwen3.8-2.4T-A95B`:
 ```bash
 model_id=tiny-random/qwen3.8-moe
-SGLANG_JIT_DEEPGEMM_FAST_WARMUP=1 sglang serve \
   --model-path $model_id \
   --tp-size 2 \
   --speculative-algorithm NEXTN \
   --speculative-num-steps 3 \
   --speculative-eagle-topk 1 \
@@ -83,7 +85,7 @@ import torch
 from huggingface_hub import hf_hub_download
 from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer, GenerationConfig, set_seed
 # Creation helpers: pip install git+https://github.com/yujiepan/tiny-random.git
-from tiny_random.qwen_utils import add_qwen_mtp, save_random_qwen_checkpoint
 source_model_id = "Qwen/Qwen3.8-2.4T-A95B"
 save_folder = "/tmp/tiny-random/qwen38-moe"
@@ -93,7 +95,7 @@ folder = Path(save_folder)
 folder.mkdir(parents=True, exist_ok=True)
 raw = json.loads(Path(hf_hub_download(source_model_id, "config.json", revision=source_revision)).read_text())
 raw.update(
-    hidden_size=128, head_dim=128, num_hidden_layers=8,
     layer_types=(["linear_attention"] * 3 + ["full_attention"]) * 2,
     num_attention_heads=32, num_key_value_heads=2,
     linear_num_key_heads=2, linear_num_value_heads=16,
@@ -107,6 +109,8 @@ add_qwen_mtp(model, model.model)
 save_random_qwen_checkpoint(model, folder)
 AutoTokenizer.from_pretrained(source_model_id, revision=source_revision).save_pretrained(folder)
 GenerationConfig.from_pretrained(source_model_id, revision=source_revision).save_pretrained(folder)
 ```
 </details>
@@ -116,7 +120,7 @@ GenerationConfig.from_pretrained(source_model_id, revision=source_revision).save
 <details><summary>Click to expand</summary>
 ```text
-Qwen3.8 MoE: 8 backbone layers + 1 MTP layer; hidden_size=128
 ```
 </details>
@@ -135,4 +139,5 @@ Qwen3.8 MoE: 8 backbone layers + 1 MTP layer; hidden_size=128
 ### Change log:
-- 2026-10-23: Initial version.

 | Input modalities | Text | Same |
 | Vocabulary | 248,320 | Same; special-token IDs preserved |
 | Backbone layers | 92 | 8 |
+| Hidden size | 8,192 | 16 |
 | Attention Q/KV heads; head dim | 64/4; 256 | 32/2; 128 |
 | Attention cycle | 3 linear + 1 full | Same; two complete cycles |
 | Linear-attention key/value heads | 16/128 | 2/16 |
 | File path | Size |
 |------|------|
+| model.safetensors | 25.9MB |
 ### Example usage:
 ```bash
 model_id=tiny-random/qwen3.8-moe
+sglang serve \
   --model-path $model_id \
   --tp-size 2 \
+  --fp8-gemm-backend triton \
+  --moe-runner-backend triton \
   --speculative-algorithm NEXTN \
   --speculative-num-steps 3 \
   --speculative-eagle-topk 1 \
 from huggingface_hub import hf_hub_download
 from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer, GenerationConfig, set_seed
 # Creation helpers: pip install git+https://github.com/yujiepan/tiny-random.git
+from tiny_random.qwen_utils import add_qwen_mtp, save_random_qwen_checkpoint, compact_qwen_tokenizer, check_qwen_repo_size
 source_model_id = "Qwen/Qwen3.8-2.4T-A95B"
 save_folder = "/tmp/tiny-random/qwen38-moe"
 folder.mkdir(parents=True, exist_ok=True)
 raw = json.loads(Path(hf_hub_download(source_model_id, "config.json", revision=source_revision)).read_text())
 raw.update(
+    hidden_size=16, head_dim=128, num_hidden_layers=8,
     layer_types=(["linear_attention"] * 3 + ["full_attention"]) * 2,
     num_attention_heads=32, num_key_value_heads=2,
     linear_num_key_heads=2, linear_num_value_heads=16,
 save_random_qwen_checkpoint(model, folder)
 AutoTokenizer.from_pretrained(source_model_id, revision=source_revision).save_pretrained(folder)
 GenerationConfig.from_pretrained(source_model_id, revision=source_revision).save_pretrained(folder)
+compact_qwen_tokenizer(folder)
+check_qwen_repo_size(folder)
 ```
 </details>
 <details><summary>Click to expand</summary>
 ```text
+Qwen3.8 MoE: 8 backbone layers + 1 MTP layer; hidden_size=16
 ```
 </details>
 ### Change log:
+- 2026-10-23: Initial version.
+- 2026-10-23: Reduced all repo files below 50 MB; GPU-validated TP=2, MTP and CUDA Graph.

config.json CHANGED Viewed

@@ -11,7 +11,7 @@
   "full_attention_interval": 4,
   "head_dim": 128,
   "hidden_act": "silu",
-  "hidden_size": 128,
   "initializer_range": 0.02,
   "layer_types": [
     "linear_attention",

   "full_attention_interval": 4,
   "head_dim": 128,
   "hidden_act": "silu",
+  "hidden_size": 16,
   "initializer_range": 0.02,
   "layer_types": [
     "linear_attention",

model.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5f2d072be72ca068d6a2f1f7454b814ae4f8d7a33260c2c01f6e8e517a09fc5a
-size 205927360

 version https://git-lfs.github.com/spec/v1
+oid sha256:b014592253b65e71aac976e7e417c8d9705d0e085cb3b98281141b51bc6063fb
+size 25859808

tokenizer.json CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a5cd9732badce41de57e6efce8302930ded1c1188c5f81feb2bd6c24c4a1941f
-size 19989339

 version https://git-lfs.github.com/spec/v1
+oid sha256:4d765ac5a92a75f240af39f9358a09a140216dce113e57754c91042ef8b9e4e8
+size 10079700