Instructions to use tiny-random/qwen3.8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tiny-random/qwen3.8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="tiny-random/qwen3.8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("tiny-random/qwen3.8")
model = AutoModelForMultimodalLM.from_pretrained("tiny-random/qwen3.8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tiny-random/qwen3.8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tiny-random/qwen3.8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiny-random/qwen3.8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/tiny-random/qwen3.8

SGLang

How to use tiny-random/qwen3.8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tiny-random/qwen3.8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiny-random/qwen3.8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tiny-random/qwen3.8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiny-random/qwen3.8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use tiny-random/qwen3.8 with Docker Model Runner:
```
docker model run hf.co/tiny-random/qwen3.8
```

yujiepan commited on 5 days ago

Commit

3f5f685

verified ·

1 Parent(s): fccc86c

Upload folder using huggingface_hub

Browse files

Files changed (4) hide show

README.md +14 -9
config.json +2 -2
model.safetensors +2 -2
tokenizer.json +2 -2

README.md CHANGED Viewed

@@ -13,7 +13,7 @@ Structural comparison with `Qwen/Qwen3.8-27B`:
 | Input modalities | Text, image, video | Same |
 | Vocabulary | 248,320 | Same; special-token IDs preserved |
 | Backbone layers | 64 | 8 |
-| Hidden size | 5,120 | 128 |
 | Attention Q/KV heads; head dim | 24/4; 256 | 12/2; 128 |
 | Attention cycle | 3 linear + 1 full | Same; two complete cycles |
 | Linear-attention key/value heads | 16/48 | 2/6 |
@@ -22,12 +22,12 @@ Structural comparison with `Qwen/Qwen3.8-27B`:
 | MTP | 1 full-attention layer; shared embeddings | Same |
 | Vision layers / hidden size | 27 / 1,152 | 2 / 64 |
 | Vision patch / temporal patch / merge | 16 / 2 / 2 | Same |
-| Vision output size | 5,120 | 128 |
 | Checkpoint format | BF16 | Same |
 | File path | Size |
 |------|------|
-| model.safetensors | 139.9MB |
 ### Example usage:
@@ -40,9 +40,11 @@ Structural comparison with `Qwen/Qwen3.8-27B`:
 ```bash
 model_id=tiny-random/qwen3.8
-SGLANG_JIT_DEEPGEMM_FAST_WARMUP=1 sglang serve \
   --model-path $model_id \
   --tp-size 2 \
   --speculative-algorithm NEXTN \
   --speculative-num-steps 3 \
   --speculative-eagle-topk 1 \
@@ -91,7 +93,7 @@ import torch
 from huggingface_hub import hf_hub_download
 from transformers import AutoConfig, AutoModelForImageTextToText, AutoProcessor, GenerationConfig, set_seed
 # Creation helpers: pip install git+https://github.com/yujiepan/tiny-random.git
-from tiny_random.qwen_utils import add_qwen_mtp, save_random_qwen_checkpoint
 source_model_id = "Qwen/Qwen3.8-27B"
 save_folder = "/tmp/tiny-random/qwen38"
@@ -101,7 +103,7 @@ folder = Path(save_folder)
 folder.mkdir(parents=True, exist_ok=True)
 raw = json.loads(Path(hf_hub_download(source_model_id, "config.json", revision=source_revision)).read_text())
 raw["text_config"].update(
-    hidden_size=128, head_dim=128, num_hidden_layers=8,
     layer_types=(["linear_attention"] * 3 + ["full_attention"]) * 2,
     num_attention_heads=12, num_key_value_heads=2,
     linear_num_key_heads=2, linear_num_value_heads=6,
@@ -109,7 +111,7 @@ raw["text_config"].update(
     intermediate_size=512,
 )
 raw["text_config"]["rope_parameters"]["mrope_section"] = [6, 5, 5]
-raw["vision_config"].update(depth=2, hidden_size=64, intermediate_size=128, num_heads=2, out_hidden_size=128)
 (folder / "config.json").write_text(json.dumps(raw, indent=2))
 config = AutoConfig.from_pretrained(folder)
 model = AutoModelForImageTextToText.from_config(config, dtype=torch.bfloat16)
@@ -117,6 +119,8 @@ add_qwen_mtp(model, model.model.language_model)
 save_random_qwen_checkpoint(model, folder)
 AutoProcessor.from_pretrained(source_model_id, revision=source_revision).save_pretrained(folder)
 GenerationConfig.from_pretrained(source_model_id, revision=source_revision).save_pretrained(folder)
 ```
 </details>
@@ -126,7 +130,7 @@ GenerationConfig.from_pretrained(source_model_id, revision=source_revision).save
 <details><summary>Click to expand</summary>
 ```text
-Qwen3.8 VL: 8 backbone layers + 1 MTP layer; hidden_size=128
 ```
 </details>
@@ -149,4 +153,5 @@ Qwen3.8 VL: 8 backbone layers + 1 MTP layer; hidden_size=128
 ### Change log:
-- 2026-10-23: Initial version.

 | Input modalities | Text, image, video | Same |
 | Vocabulary | 248,320 | Same; special-token IDs preserved |
 | Backbone layers | 64 | 8 |
+| Hidden size | 5,120 | 16 |
 | Attention Q/KV heads; head dim | 24/4; 256 | 12/2; 128 |
 | Attention cycle | 3 linear + 1 full | Same; two complete cycles |
 | Linear-attention key/value heads | 16/48 | 2/6 |
 | MTP | 1 full-attention layer; shared embeddings | Same |
 | Vision layers / hidden size | 27 / 1,152 | 2 / 64 |
 | Vision patch / temporal patch / merge | 16 / 2 / 2 | Same |
+| Vision output size | 5,120 | 16 |
 | Checkpoint format | BF16 | Same |
 | File path | Size |
 |------|------|
+| model.safetensors | 18.2MB |
 ### Example usage:
 ```bash
 model_id=tiny-random/qwen3.8
+sglang serve \
   --model-path $model_id \
   --tp-size 2 \
+  --fp8-gemm-backend triton \
+  --moe-runner-backend triton \
   --speculative-algorithm NEXTN \
   --speculative-num-steps 3 \
   --speculative-eagle-topk 1 \
 from huggingface_hub import hf_hub_download
 from transformers import AutoConfig, AutoModelForImageTextToText, AutoProcessor, GenerationConfig, set_seed
 # Creation helpers: pip install git+https://github.com/yujiepan/tiny-random.git
+from tiny_random.qwen_utils import add_qwen_mtp, save_random_qwen_checkpoint, compact_qwen_tokenizer, check_qwen_repo_size
 source_model_id = "Qwen/Qwen3.8-27B"
 save_folder = "/tmp/tiny-random/qwen38"
 folder.mkdir(parents=True, exist_ok=True)
 raw = json.loads(Path(hf_hub_download(source_model_id, "config.json", revision=source_revision)).read_text())
 raw["text_config"].update(
+    hidden_size=16, head_dim=128, num_hidden_layers=8,
     layer_types=(["linear_attention"] * 3 + ["full_attention"]) * 2,
     num_attention_heads=12, num_key_value_heads=2,
     linear_num_key_heads=2, linear_num_value_heads=6,
     intermediate_size=512,
 )
 raw["text_config"]["rope_parameters"]["mrope_section"] = [6, 5, 5]
+raw["vision_config"].update(depth=2, hidden_size=64, intermediate_size=128, num_heads=2, out_hidden_size=16)
 (folder / "config.json").write_text(json.dumps(raw, indent=2))
 config = AutoConfig.from_pretrained(folder)
 model = AutoModelForImageTextToText.from_config(config, dtype=torch.bfloat16)
 save_random_qwen_checkpoint(model, folder)
 AutoProcessor.from_pretrained(source_model_id, revision=source_revision).save_pretrained(folder)
 GenerationConfig.from_pretrained(source_model_id, revision=source_revision).save_pretrained(folder)
+compact_qwen_tokenizer(folder)
+check_qwen_repo_size(folder)
 ```
 </details>
 <details><summary>Click to expand</summary>
 ```text
+Qwen3.8 VL: 8 backbone layers + 1 MTP layer; hidden_size=16
 ```
 </details>
 ### Change log:
+- 2026-10-23: Initial version.
+- 2026-10-23: Reduced all repo files below 50 MB; GPU-validated TP=2, MTP and CUDA Graph.

config.json CHANGED Viewed

@@ -15,7 +15,7 @@
     "full_attention_interval": 4,
     "head_dim": 128,
     "hidden_act": "silu",
-    "hidden_size": 128,
     "initializer_range": 0.02,
     "intermediate_size": 512,
     "layer_types": [
@@ -74,7 +74,7 @@
     "model_type": "qwen3_5",
     "num_heads": 2,
     "num_position_embeddings": 2304,
-    "out_hidden_size": 128,
     "patch_size": 16,
     "spatial_merge_size": 2,
     "temporal_patch_size": 2

     "full_attention_interval": 4,
     "head_dim": 128,
     "hidden_act": "silu",
+    "hidden_size": 16,
     "initializer_range": 0.02,
     "intermediate_size": 512,
     "layer_types": [
     "model_type": "qwen3_5",
     "num_heads": 2,
     "num_position_embeddings": 2304,
+    "out_hidden_size": 16,
     "patch_size": 16,
     "spatial_merge_size": 2,
     "temporal_patch_size": 2

model.safetensors CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0ad7d04172397ad42753662b4786257103b27440fd2bb64c07663c37da9e5135
-size 139932696

 version https://git-lfs.github.com/spec/v1
+oid sha256:45266a20a08495d89509458cd2caf93f6946e1e34d260a281511c645a6ee6a3a
+size 18219872

tokenizer.json CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a5cd9732badce41de57e6efce8302930ded1c1188c5f81feb2bd6c24c4a1941f
-size 19989339

 version https://git-lfs.github.com/spec/v1
+oid sha256:4d765ac5a92a75f240af39f9358a09a140216dce113e57754c91042ef8b9e4e8
+size 10079700