Blame for README.md

---
license: mit
base_model:
  - zai-org/GLM-5.3
tags:
  - glm_moe_dsa
  - fixture
  - testing
pipeline_tag: text-generation
---

# GLM-5.3 tiny architecture fixture (glm_moe_dsa)

A ~0.27M-parameter random-init checkpoint that reproduces the reduced **GLM-5.3 MoE config
schema** so loaders, quant planners, and CI jobs can exercise the new `glm_moe_dsa`
architecture without touching the 753B weights.

## What this is
- Base architecture: `zai-org/GLM-5.3` (released 2026-08-25), `model_type: glm_moe_dsa`,
  `architectures: ["GlmMoeDsaForCausalLM"]`, MIT license.
- What it is: a byte-reproducible **random-init** checkpoint plus a reduced config that keeps
  the real field names and the dense-to-MoE layer schedule (`first_k_dense_replace`,
  `moe_layer_freq`, routed experts + shared experts).
- What it is **not**: not trained, not distilled, not a quality or benchmark claim, and not a
  quantization of anything.
- Why it is useful: the base is 753,329,940,480 parameters (Hub safetensors metadata), so it
  cannot be instantiated in a unit test, in CI, or on a laptop. This fixture lets you test
  config parsing, weight-name mapping, expert-table sizing, router/top-k bookkeeping, and
  safetensors load paths in milliseconds.

## Geometry (base vs fixture)
| Field | Base GLM-5.3 | This fixture |
|---|---|---|
| num_hidden_layers | 78 | 4 |
| hidden_size | 6144 (read partially from base config; verify) | 64 |
| num_attention_heads / num_key_value_heads | 64 / 64 | 4 / 4 |
| head_dim | 192 | 16 |
| n_routed_experts | 256 | 8 |
| num_experts_per_tok | 8 | 2 |
| n_shared_experts | 1 | 1 |
| n_group | 1 | 1 |
| first_k_dense_replace | 3 | 1 |
| moe_intermediate_size | 2048 | 32 |
| intermediate_size (dense) | not captured | 128 |
| num_nextn_predict_layers | 1 | 0 |
| dtype | bfloat16 | float32 |
| vocab_size | 154820-class | 256 |

## Intentional omissions (documented, not silent)
- No DeepSeek-style sparse-attention (DSA) indexer tensors.
- No MTP / next-n-predictor head (`num_nextn_predict_layers: 0`).
- No `lm_head` tensor; a loader must tie to `model.embed_tokens.weight` or supply its own head.
- Tokenizer metadata files are placeholders (no vocab file); use your own tokenizer.

## Verification actually performed (stdlib only, no torch in this environment)
- safetensors header parses: 113 tensors, 1,097,984 data bytes = 274,496 float32 parameters,
  contiguous `data_offsets`, header padded to 8-byte alignment.
- `checksums.txt` records the SHA-256 of every tensor blob.
- Deterministic regeneration: SplitMix64 seed 20260901, Box-Muller normals, scale 0.02,
  float32 row-major, consumed in sorted-name order.
- Generator script is included as `build_fixture.py` so the folder can be rebuilt and diffed.

**Not yet verified:** loading under a specific `transformers` version (no torch/transformers in
the build environment), and whether `GlmMoeDsaForCausalLM` accepts this reduced geometry without
extra fields. Treat those as open until run against a real install.

## How to use
Read the tensors with the standard library (no torch needed, matching how this was built):

```python
import json, struct
with open("model.safetensors", "rb") as f:
    n = struct.unpack("<Q", f.read(8))[0]
    header = json.loads(f.read(n))
    # header[name] = {"dtype", "shape", "data_offsets"}; data starts at byte 8+n
```

Or with the `safetensors` package:

```python
from safetensors.torch import load_file
tensors = load_file("model.safetensors")   # {name: tensor}
```

To exercise a real loader, build a config from `config.json` (the `glm_moe_dsa`
model type; use `AutoConfig.from_pretrained(..., trust_remote_code=True)` where
needed) and feed these weights in. There is no `lm_head` tensor and the tokenizer
files are placeholders, so supply your own head/tokenizer.

## License
MIT, unchanged from the base model per its Hub metadata. See `LICENSE`.

## Citation
Zhipu AI / Z.ai, GLM-5.3, 2026.

---

Buy me a beer:
**BTC** `bc1q5ayht3fxhj0v95fk0z8l2f6900g3awdsw5842p`