Instructions to use inference-optimization/GLM-5.3-3.5B-A3.3B-small with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use inference-optimization/GLM-5.3-3.5B-A3.3B-small with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="inference-optimization/GLM-5.3-3.5B-A3.3B-small")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("inference-optimization/GLM-5.3-3.5B-A3.3B-small")
model = AutoModelForCausalLM.from_pretrained("inference-optimization/GLM-5.3-3.5B-A3.3B-small", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use inference-optimization/GLM-5.3-3.5B-A3.3B-small with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "inference-optimization/GLM-5.3-3.5B-A3.3B-small"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/GLM-5.3-3.5B-A3.3B-small",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/inference-optimization/GLM-5.3-3.5B-A3.3B-small

SGLang

How to use inference-optimization/GLM-5.3-3.5B-A3.3B-small with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "inference-optimization/GLM-5.3-3.5B-A3.3B-small" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/GLM-5.3-3.5B-A3.3B-small",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "inference-optimization/GLM-5.3-3.5B-A3.3B-small" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/GLM-5.3-3.5B-A3.3B-small",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use inference-optimization/GLM-5.3-3.5B-A3.3B-small with Docker Model Runner:
```
docker model run hf.co/inference-optimization/GLM-5.3-3.5B-A3.3B-small
```

GLM-5.3-3.5B-A3.3B-small

This is a small version of zai-org/GLM-5.3 created for testing and development. It keeps every hyperparameter of the base model identical (hidden size, MLA/DSA dimensions, indexer parameters, MoE topology, vocabulary, untied embeddings, rope settings) and only reduces the number of decoder layers, the per-layer type lists, and the number of routed experts per MoE layer.

Model Details

Base Model: zai-org/GLM-5.3
Architecture: GlmMoeDsaForCausalLM (glm_moe_dsa)
Total Parameters: 3.47B
Activated Parameters: 3.32B (topk 8 of 10 routed experts per sparse layer)

Configuration Changes

Only the allowed change classes were applied; everything else is identical to the base:

Parameter	Base	Small
`num_hidden_layers`	78	3
`layer_types`	78 x `deepseek_sparse_attention`	3 x `deepseek_sparse_attention`
`mlp_layer_types`	`[dense x3, sparse x75]`	`[dense, sparse, sparse]`
`first_k_dense_replace`	3	1
`indexer_types`	`[full x3, shared x3, full, shared x3, ...]` (full at 0-2 and every 4th layer from 6; 57 shared layers)	`[full, full, shared]`
`n_routed_experts`	256	10

The 3 layers are the minimal covering set of the distinct (attention/indexer, MLP) combinations that occur in the base:

layer 0: full DSA indexer + dense MLP (base layers 0-2)
layer 1: full DSA indexer + sparse MoE (base layers 6, 10, 14, ..., 74)
layer 2: shared indexer (reuses the previous full layer's top-k) + sparse MoE (base layers 3-5, 7-9, ... — 57 of 78 base layers)

10 experts = topk (8) + 2 so that routing is actually exercised; num_experts_per_tok (topk 8) is unchanged. All other values are kept from the base, including hidden_size 6144, intermediate_size 12288, moe_intermediate_size 2048, n_shared_experts 1, MLA dimensions (q_lora_rank 2048, kv_lora_rank 512, qk_nope_head_dim 192, qk_rope_head_dim 64, v_head_dim 256, 64/64 heads), indexer parameters (index_topk 2048, index_n_heads 32, index_head_dim 128, index_share_for_mtp_iteration true, index_topk_freq 4, index_skip_topk_offset 3), sigmoid noaux_tc routing with routed_scaling_factor 2.5, rope_theta 8e6 with rope_interleave true, vocab_size 154880, untied embeddings.

Checkpoint Structure

Single-file bf16 model.safetensors (113 tensors) using the original base checkpoint layout: per-expert tensors model.layers.N.mlp.experts.{i}.{gate,up,down}_proj.weight, per-layer mlp.gate.weight + mlp.gate.e_score_correction_bias + shared-expert mlp.shared_experts.{gate,up,down}_proj.weight, MLA names (self_attn.{q_a_proj,q_a_layernorm,q_b_proj,kv_a_proj_with_mqa,kv_a_layernorm,kv_b_proj,o_proj}), DSA indexer tensors only on full-indexer layers (self_attn.indexer.{wq_b,wk,k_norm,weights_proj}), and untied lm_head.weight. Tensor-name patterns match the base model.safetensors.index.json exactly, modulo layer and expert counts (validated programmatically: 28/28 normalized patterns, 0 missing, 0 extra; per-layer key sets identical to the analogous base layers 0 / 6 / 3; the base's fp8 weight_scale_inv tensors are absent by design).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("inference-optimization/GLM-5.3-3.5B-A3.3B-small", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/GLM-5.3-3.5B-A3.3B-small")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Creation Process

This model was created using the llm-compressor create-tiny-model skill:

Loaded the base AutoConfig, stripped the base's fp8 quantization_config (producing an unquantized bf16 model, matching the convention for these entries), and applied only the allowed changes (3 layers with the per-layer lists above, 10 experts).
Built the model with skip_weights_download, re-initialized weights (init_weights() + NaN fixup), cast fully to bf16, and saved with the base's qwen2_moe-style weight conversions recorded (so the save emits the base per-expert layout directly).
Verified the saved tensor structure against the base model.safetensors.index.json immediately after the first save.
Fine-tuned on the skill's copypasta dataset in bf16 (lr 1e-4, batch 2, early-stopped after ~180 steps / 27s) to target perplexity.
Re-validated the fine-tuned checkpoint structure and model quality.

Validation

Perplexity on the skill's validation text: 1.00004 (target <= 10)
Greedy generation from "According to all known laws": produces the trained copypasta continuation correctly.

Notes

MTP: num_nextn_predict_layers is kept at 1 from the base; as with transformers saves of this family, the checkpoint is backbone-only (the base stores its MTP layer as model.layers.78.*, which is intentionally absent here, matching the convention of the other GLM small models).
Quantization: the base ships an fp8 quantization_config; it was deleted before building, so this small model is clean unquantized bf16 (the base's weight_scale_inv tensors have no counterpart).
Size: 3.47B total is the forced minimum for a layer set covering all distinct (indexer, MLP) combos at hidden size 6144 — the untied 154880 x 6144 embeddings alone are 1.9B and each MoE layer with 10 experts adds ~~0.58B. This matches the expected ~3-3.5B and is strictly far below the base (~~789B); it keeps every distinct layer type of the base exercised, including the cross-layer shared-indexer mechanism used by 57 of 78 base layers.
bf16 throughout (full bf16 cast works around _keep_in_fp32_modules on indexer.weights_proj / e_score_correction_bias).
training_args.bin and optimizer files are not part of the upload.

Downloads last month: 69

Safetensors

Model size

3B params

Tensor type

BF16

Model tree for inference-optimization/GLM-5.3-3.5B-A3.3B-small

Base model

zai-org/GLM-5.3

Finetuned

(23)

this model

Collection including inference-optimization/GLM-5.3-3.5B-A3.3B-small

Small Models

Collection

Small versions of known models: same vLLM loading characteristics as the base, fewer layers/experts, identical hidden dims and hyperparameters • 25 items • Updated 19 days ago