A pruned and healed version of GLM-5.3-Flash, made by the Manprot team
| | | |---|---| | **Made by** | The [Manprot](https://huggingface.co/Manprot) team | | **Family** | MP-02 (see also [MP-01 100M Preview](https://huggingface.co/Manprot/MP-01-100M-Preview)) | | **Status** | **Preview**: it works, but it's rough (see [Limitations](#limitations)) | | **Type** | Chat model with thinking, healed on GLM-5.3 conversations | | **Parameters** | 16.06B in the transformer layers; 18.28B in total with embeddings and LM head (1.27B), MTP head (0.39B) and vision encoder (0.56B) | | **Architecture** | GLM-5.3-Flash (`glm5_next`) cut from 288 to 8 routed experts per MoE layer | | **Built from** | [zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash) (MIT) | | **Tokens trained** | 22.9M (healing only; no pretraining of our own) | | **7-task average** | 51.29, about 3 points above MP-01 100M Preview | | **License** | MIT, same as the base model (Z.ai copyright notice kept in `LICENSE`) | > **Note:** this is a **preview**. It is somewhat decent at chat and roleplay, but cutting 97% of the experts cost it much of GLM-5.3-Flash's knowledge and reasoning. It is **not** a replacement for GLM-5.3-Flash. Expect wrong facts, broken arithmetic, over-long thinking and the occasional loop. ## Contents - [Training lineage](#training-lineage) - [Datasets](#datasets) - [Architecture](#architecture) - [Evaluation](#evaluation) - [Compute](#compute) - [Running it](#running-it) - [Limitations](#limitations) - [Repository contents](#repository-contents) - [Citation](#citation) ## Training lineage 1. **Expert pruning (REAP-style).** Each of the 42 MoE layers of GLM-5.3-Flash (FP8) kept the 8 of its 288 routed experts with the highest router-weighted output norm (gate × ‖expert output‖), measured on a 102k-token calibration set. Router rows were cut to match. The shared expert, attention, the 3 dense layers, the MTP head and the vision encoder are unchanged. Held-out loss right after pruning: 3.725. 2. **Healing, stage 1.** LoRA (rank 64, alpha 128, on every attention and MLP projection, routed experts included) plus fully trained routers. AdamW, learning rate 1e-4, warmup and cosine decay, 2,048-token chunks, loss on assistant turns only. 174 steps, **10.9M tokens**. Held-out loss: 3.725 → **2.105**. 3. **Healing, stage 2.** Continued from stage 1 with a fresh optimizer on **12.1M new tokens** (no conversation from stage 1 repeated), learning rate 8e-5, 193 steps. Held-out loss on a new held-out set: 2.449 → **2.218**. **This is the checkpoint released here.** The two held-out sets differ, so their losses are not comparable with each other. ## Datasets **In total: 13 healing datasets and 3 calibration datasets** (one used for both). Every dataset that shaped these weights is listed here and in this page's metadata. Healing data is **only text written by GLM-5.3 or GLM-5.3-Flash**, with one exception noted below. Training data is not re-released. ### Healing | Dataset | What it is | Stage 1 | Stage 2 | |---|---|---:|---:| | [Zek-Takai/glm53-flash-harvest](https://huggingface.co/datasets/Zek-Takai/glm53-flash-harvest) | Prose, chat, multi-turn, reasoning, code, structured output, agentic (GLM-5.3-Flash) | 65% | 69% | | [shanjiaz/glm-5.3-flash-open-perfectblend-regen-25pct](https://huggingface.co/datasets/shanjiaz/glm-5.3-flash-open-perfectblend-regen-25pct) | Mixed instructions | 15% | 2% | | [best-distill/glm-5.3-flash-distillation-chat](https://huggingface.co/datasets/best-distill/glm-5.3-flash-distillation-chat) | Instructions with reasoning | 7% | 3% | | [open-athena/GLM-5.3-RLVR1-Training-Rollouts-2026.09.25](https://huggingface.co/datasets/open-athena/GLM-5.3-RLVR1-Training-Rollouts-2026.09.25) | Mixed tasks (GLM-5.3) | 5% | 10% | | [sollamon/ox-alpha-glm-5.3-flash-distillation-coding-17k-raw](https://huggingface.co/datasets/sollamon/ox-alpha-glm-5.3-flash-distillation-coding-17k-raw) | Coding | 3% | 3% | | [open-athena/GLM-5.3-tmax-swesmith-k8-traces-2026.10](https://huggingface.co/datasets/open-athena/GLM-5.3-tmax-swesmith-k8-traces-2026.10) | Terminal and bug-fix agent runs, passed only (GLM-5.3) | 1% | 7% | | [liangzhidanta/claude-code-glm53-swesmith-trajectories](https://huggingface.co/datasets/liangzhidanta/claude-code-glm53-swesmith-trajectories) and [its compact-aware version](https://huggingface.co/datasets/liangzhidanta/claude-code-glm53-swesmith-compact-trajectories) | Coding-agent runs, some with context compaction (GLM-5.3) | – | 3% | | [kaivoss/UltraData-SFT-Agent-2609-GLM53-20k](https://huggingface.co/datasets/kaivoss/UltraData-SFT-Agent-2609-GLM53-20k) | Agent runs in GLM-5.3's tool format. **Exception:** the text was written by the upstream models, not GLM | 2% | – | | [Rallex3](https://huggingface.co/datasets/Rallex3/glm-5.3-flash-function-calling), [closestfriend](https://huggingface.co/datasets/closestfriend/glm-5.3-flash-function-calling), [FranckAbgrall](https://huggingface.co/datasets/FranckAbgrall/glm53-flash-function-calling), [flaviojoshua](https://huggingface.co/datasets/flaviojoshua/glm53-flash-function-calling), [Offlin33er](https://huggingface.co/datasets/Offlin33er/glm-5.3-flash-function-calling) function-calling sets | Tool calls (GLM-5.3-Flash) | 2% | 3% | Shares are of each stage's tokens. Stage 1 drew 10.9M tokens from a 40M-token mix; stage 2 used 12.1M tokens from an 18M-token mix (the time budget ended the run first). In both stages, conversations where the model refused as an AI were dropped, and any sexual content involving minors was removed. ### Calibration for pruning (stage 1 only) These decided which experts were kept. The model was **not trained** on them. | Dataset | Share | |---|---:| | [Gryphe/Sonnet3.5-Charcard-Roleplay](https://huggingface.co/datasets/Gryphe/Sonnet3.5-Charcard-Roleplay) | 30% | | [anthracite-org/c2_logs_32k_llama3_qwen2_v1.3](https://huggingface.co/datasets/anthracite-org/c2_logs_32k_llama3_qwen2_v1.3) | 15% | | [Zek-Takai/glm53-flash-harvest](https://huggingface.co/datasets/Zek-Takai/glm53-flash-harvest) (multi-turn, chat, prose, reasoning) | 55% | ### Evaluation benchmarks | Benchmark | Dataset | |---|---| | ARC-Challenge, ARC-Easy | [allenai/ai2_arc](https://huggingface.co/datasets/allenai/ai2_arc) | | HellaSwag | [Rowan/hellaswag](https://huggingface.co/datasets/Rowan/hellaswag) | | OpenBookQA | [allenai/openbookqa](https://huggingface.co/datasets/allenai/openbookqa) | | PIQA | [baber/piqa](https://huggingface.co/datasets/baber/piqa) | | SciQ | [allenai/sciq](https://huggingface.co/datasets/allenai/sciq) | | WinoGrande | [allenai/winogrande](https://huggingface.co/datasets/allenai/winogrande) | ## Architecture | Component | Value | |---|---| | Layers | 45 (3 dense + 42 MoE) | | Width (hidden size) | 4096 | | Experts | 8 routed + 1 shared per MoE layer, **all 8 routed experts active on every token** (from 288, top-8 of 288 in the base) | | Expert hidden | 2048 | | Attention | GLM-5.3-Flash hybrid sparse / linear attention (unchanged) | | Extra heads | 1 MTP (multi-token prediction) layer, kept | | Vision | GLM-5.3-Flash vision encoder (unchanged, not evaluated after pruning) | | Vocabulary | 154,880 (GLM-5.3 tokenizer) | | Context length | healed at 2,048 tokens; the base supports far more, but long context is untested after pruning | | Parameters | 18.28B total: 16.06B layers, 1.27B embeddings and LM head, 0.39B MTP, 0.56B vision | Because all 8 kept experts fire on every token, MP-02 runs at the speed of a dense 16B model, not a sparse MoE. ## Evaluation Scored with [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness): zero-shot, the first 500 questions of each task, bf16 on one H100. The metric per task is the same as MP-01's harness: accuracy for ARC-Easy and WinoGrande, length-normalised accuracy for the rest. With 500 questions a task, differences under about 2 points are within noise. MP-01 was scored by its own harness on the same tasks, metrics and sample size, so treat small differences between the two as noise. ### Comparison with MP-01 | Rank | Model | Params | ARC-C | ARC-E | HellaSwag | OBQA | PIQA | SciQ | Wino | Avg (7) | |---|---|---|---:|---:|---:|---:|---:|---:|---:|---:| | 1 | MP-02 16B Preview (this model) | 18.28B | 27.0 | 48.4 | **47.0** | 29.2 | 63.2 | **89.0** | **55.2** | **51.29** | | 2 | [MP-01 100M Preview](https://huggingface.co/Manprot/MP-01-100M-Preview) | 100.6M | **29.4** | **56.4** | 31.8 | **30.2** | **64.4** | 73.0 | 53.8 | 48.43 | Bold is the best score in each column. ### Where this model stands - **7-task average:** about 3 points above MP-01, mostly from HellaSwag (+15.2) and SciQ (+16.0). It is behind MP-01 on ARC-Easy (−8.0) and slightly on ARC-Challenge, OpenBookQA and PIQA. - **That is a low result for its size.** ARC-Challenge is near chance (25% for 4 choices). GLM-5.3-Flash itself was not run on this setup, but its published scores are far higher, so the pruning loss is large. - These are multiple-choice log-likelihood tests. They don't measure chat, roleplay or thinking, which is what the healing focused on. ### Progress across checkpoints of this model | Checkpoint | Held-out loss | ARC-C | ARC-E | HellaSwag | OBQA | PIQA | SciQ | Wino | Avg (7) | |---|---:|---:|---:|---:|---:|---:|---:|---:|---:| | Stage 1, step 174 | 2.105 (set A) | 27.0 | 48.0 | **47.2** | 27.4 | **64.4** | **89.4** | **56.8** | **51.46** | | Stage 2, step 193 (this release) | 2.218 (set B) | 27.0 | **48.4** | 47.0 | **29.2** | 63.2 | 89.0 | 55.2 | 51.29 | Stage 2 left the benchmarks where they were (all changes are within noise) and lowered loss on its own held-out set from 2.449 to 2.218. ## Compute | Stage | Hardware | GPU hours | Tokens | |---|---|---:|---:| | Expert selection and pruning | Local consumer GPU (8 GB) | about 1.9 | 102k (calibration) | | Healing, stage 1 (3 runs) | Cloud B200 | 2.85 | 10.9M | | Healing, stage 2 (this release) | Cloud B200 | 2.60 | 12.1M | | Benchmarks | Cloud H100 | 0.12 per checkpoint | | Hours are wall-clock GPU time from the run logs, including model loading. Costs in credits are not listed. ## Running it ### llama.cpp (GGUF) ```bash hf download Manprot/MP-02-16B-Preview MP-02-16B-Preview-Q4_K_M.gguf --local-dir MP-02 llama-server -m MP-02/MP-02-16B-Preview-Q4_K_M.gguf -ngl 99 --n-cpu-moe 99 -c 8192 ``` Needs a llama.cpp build with `glm5-next` support. On a desktop with an 8 GB GPU and DDR5 memory, keeping the experts on the CPU (`--n-cpu-moe 99`), it runs at about 6–7 tokens/s. On a single 24 GB GPU with everything on the GPU, it runs at about 28 tokens/s. For images, add `--mmproj mmproj-MP-02-16B-Preview-BF16.gguf`. ### transformers (safetensors) ```python import torch from transformers import AutoTokenizer, Glm5NextForConditionalGeneration tok = AutoTokenizer.from_pretrained("Manprot/MP-02-16B-Preview") model = Glm5NextForConditionalGeneration.from_pretrained("Manprot/MP-02-16B-Preview", dtype=torch.bfloat16, device_map="auto") msgs = [{"role": "user", "content": "Write a short poem about the sea."}] ids = tok.apply_chat_template(msgs, add_generation_prompt=True, reasoning_effort="low", return_tensors="pt").to(model.device) print(tok.decode(model.generate(ids, max_new_tokens=400)[0][ids.shape[1]:])) ``` Needs transformers 5.17 or newer and about 37 GB of GPU memory in bf16. ### Thinking and the chat template MP-02 uses GLM-5.3's chat template, which **always** starts a thinking block and, with no `reasoning_effort`, sends `Reasoning Effort: Max`. All healing data was rendered at Max, so by default it thinks long on everything, even simple requests. For normal chat, pass `reasoning_effort: "low"` (for example `"chat_template_kwargs": {"reasoning_effort": "low"}` on llama-server). A thinking on/off switch in a chat UI has no effect with this template. Suggested sampling: temperature 0.7, top-p 0.8, repeat penalty 1.05. ## Limitations - **Preview quality.** Most of GLM-5.3-Flash's knowledge is gone. It invents facts, gets simple arithmetic wrong and can talk confidently about things it does not know. - **Over-thinking.** It thinks at maximum effort by default (see above) and can spend its whole token budget thinking before it answers. - **Agent habits.** Agent traces in the healing data sometimes leak into chat: invented commands, sandbox paths such as `/home/user` or `/testbed`, and fake tool output. - **Loops.** It can repeat a phrase until the token limit. A repeat penalty helps. - **Identity.** It was trained on GLM's own conversations and may say it is GLM, made by Z.ai. - **Few refusals.** AI refusals were removed from the healing data, so it follows most requests. It has not been checked for safety; filter outputs yourself. - **Untested after pruning:** vision, tool calling, long context and languages other than English. ## Repository contents - `model-*.safetensors`, `model.safetensors.index.json`: merged weights in bf16 (norms in fp32) - `config.json`, `generation_config.json`, `tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja`: model configuration and tokenizer - `MP-02-16B-Preview-Q4_K_M.gguf`: 4-bit GGUF for llama.cpp (12.9 GB) - `mmproj-MP-02-16B-Preview-BF16.gguf`: vision projector for llama.cpp (optional) - `eval_results.json`: full evaluation output for this checkpoint - `LICENSE`: MIT license with Z.ai's copyright notice - `manprot-logo.png`: logo ## Citation Made by the **[Manprot](https://huggingface.co/Manprot) team**, based on [GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash) by Z.ai. All credit for the original architecture and weights goes to Z.ai. ```bibtex @misc{manprot2026mp02, title = {MP-02 16B Preview}, author = {{Manprot team}}, year = {2026}, publisher = {Hugging Face}, howpublished = {\url{https://huggingface.co/Manprot/MP-02-16B-Preview}} } ``` The datasets listed above belong to their authors and keep their own licenses.