| --- |
| license: mit |
| base_model: |
| - zai-org/GLM-5.3 |
| tags: |
| - glm_moe_dsa |
| - fixture |
| - testing |
| pipeline_tag: text-generation |
| --- |
| |
| # GLM-5.3 tiny architecture fixture (glm_moe_dsa) |
|
|
| A ~0.27M-parameter random-init checkpoint that reproduces the reduced **GLM-5.3 MoE config |
| schema** so loaders, quant planners, and CI jobs can exercise the new `glm_moe_dsa` |
| architecture without touching the 753B weights. |
|
|
| ## What this is |
| - Base architecture: `zai-org/GLM-5.3` (released 2026-08-25), `model_type: glm_moe_dsa`, |
| `architectures: ["GlmMoeDsaForCausalLM"]`, MIT license. |
| - What it is: a byte-reproducible **random-init** checkpoint plus a reduced config that keeps |
| the real field names and the dense-to-MoE layer schedule (`first_k_dense_replace`, |
| `moe_layer_freq`, routed experts + shared experts). |
| - What it is **not**: not trained, not distilled, not a quality or benchmark claim, and not a |
| quantization of anything. |
| - Why it is useful: the base is 753,329,940,480 parameters (Hub safetensors metadata), so it |
| cannot be instantiated in a unit test, in CI, or on a laptop. This fixture lets you test |
| config parsing, weight-name mapping, expert-table sizing, router/top-k bookkeeping, and |
| safetensors load paths in milliseconds. |
|
|
| ## Geometry (base vs fixture) |
| | Field | Base GLM-5.3 | This fixture | |
| |---|---|---| |
| | num_hidden_layers | 78 | 4 | |
| | hidden_size | 6144 (read partially from base config; verify) | 64 | |
| | num_attention_heads / num_key_value_heads | 64 / 64 | 4 / 4 | |
| | head_dim | 192 | 16 | |
| | n_routed_experts | 256 | 8 | |
| | num_experts_per_tok | 8 | 2 | |
| | n_shared_experts | 1 | 1 | |
| | n_group | 1 | 1 | |
| | first_k_dense_replace | 3 | 1 | |
| | moe_intermediate_size | 2048 | 32 | |
| | intermediate_size (dense) | not captured | 128 | |
| | num_nextn_predict_layers | 1 | 0 | |
| | dtype | bfloat16 | float32 | |
| | vocab_size | 154820-class | 256 | |
| |
| ## Intentional omissions (documented, not silent) |
| - No DeepSeek-style sparse-attention (DSA) indexer tensors. |
| - No MTP / next-n-predictor head (`num_nextn_predict_layers: 0`). |
| - No `lm_head` tensor; a loader must tie to `model.embed_tokens.weight` or supply its own head. |
| - Tokenizer metadata files are placeholders (no vocab file); use your own tokenizer. |
|
|
| ## Verification actually performed (stdlib only, no torch in this environment) |
| - safetensors header parses: 113 tensors, 1,097,984 data bytes = 274,496 float32 parameters, |
| contiguous `data_offsets`, header padded to 8-byte alignment. |
| - `checksums.txt` records the SHA-256 of every tensor blob. |
| - Deterministic regeneration: SplitMix64 seed 20260901, Box-Muller normals, scale 0.02, |
| float32 row-major, consumed in sorted-name order. |
| - Generator script is included as `build_fixture.py` so the folder can be rebuilt and diffed. |
|
|
| **Not yet verified:** loading under a specific `transformers` version (no torch/transformers in |
| the build environment), and whether `GlmMoeDsaForCausalLM` accepts this reduced geometry without |
| extra fields. Treat those as open until run against a real install. |
|
|
| ## How to use |
| Read the tensors with the standard library (no torch needed, matching how this was built): |
|
|
| ```python |
| import json, struct |
| with open("model.safetensors", "rb") as f: |
| n = struct.unpack("<Q", f.read(8))[0] |
| header = json.loads(f.read(n)) |
| # header[name] = {"dtype", "shape", "data_offsets"}; data starts at byte 8+n |
| ``` |
|
|
| Or with the `safetensors` package: |
|
|
| ```python |
| from safetensors.torch import load_file |
| tensors = load_file("model.safetensors") # {name: tensor} |
| ``` |
|
|
| To exercise a real loader, build a config from `config.json` (the `glm_moe_dsa` |
| model type; use `AutoConfig.from_pretrained(..., trust_remote_code=True)` where |
| needed) and feed these weights in. There is no `lm_head` tensor and the tokenizer |
| files are placeholders, so supply your own head/tokenizer. |
|
|
| ## License |
| MIT, unchanged from the base model per its Hub metadata. See `LICENSE`. |
|
|
| ## Citation |
| Zhipu AI / Z.ai, GLM-5.3, 2026. |
|
|
| --- |
|
|
| Buy me a beer: |
| **BTC** `bc1q5ayht3fxhj0v95fk0z8l2f6900g3awdsw5842p` |
|
|
