djdeniro commited on
Commit
8f234f4
·
verified ·
1 Parent(s): 3766711

Model card: attribution + avg bpw 4.93 + multimodal frontmatter (fix file paths)

Browse files
Files changed (2) hide show
  1. LICENSE +21 -157
  2. README.md +113 -51
LICENSE CHANGED
@@ -1,157 +1,21 @@
1
- ---
2
- license: mit
3
- base_model: zai-org/GLM-5.3-Flash
4
- pipeline_tag: image-text-to-text
5
- library_name: transformers
6
- tags:
7
- - glm
8
- - glm-5
9
- - multimodal
10
- - vision-language
11
- - moe
12
- - sparse-attention
13
- - mla
14
- - quantization
15
- - rfa
16
- - rfi
17
- - rocm
18
- - rdna4
19
- - gfx1201
20
- - 4-bit
21
- ---
22
-
23
- <div align="center">
24
-
25
- # GLM-5.3-Flash · RFA + RFI8 composite quant (8× R9700 / RDNA4)
26
-
27
- **A self-quantized derivative of [zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash), tuned for 8× AMD Radeon R9700 (gfx1201 / RDNA4)**
28
-
29
- [![Model size](https://img.shields.io/badge/params-321B%20(18B%20active)-blue)](https://huggingface.co/zai-org/GLM-5.3-Flash)
30
- [![bpw](https://img.shields.io/badge/bpw-4.93-ff69b4)](#quantization)
31
- [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE)
32
- [![Serving stack](https://img.shields.io/badge/serving%20stack-tcclaviger%2Fvllm-00a2a2)](https://hub.docker.com/r/tcclaviger/vllm)
33
- [![GPU](https://img.shields.io/badge/GPU-RDNA4%20(gfx1201)-ED1C24)](https://www.amd.com/en/products/processors/desktops/radeon.html)
34
-
35
- </div>
36
-
37
- ---
38
-
39
- ## Model card
40
-
41
- This is a **composite-quantized** checkpoint of Z.ai's **GLM-5.3-Flash** — the first natively
42
- multimodal model in the GLM-5 series (321B total / ~18B active parameters). It was quantized from
43
- the official FP8 release using the `tcclaviger/vllm` composite quantizer and is intended to be
44
- served with **vLLM** on **RDNA4** (AMD Radeon R9700) hardware.
45
-
46
- > **📦 Companion serving repo:** [GLM-5.3-Flash-rocm-r9700](https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700) — the ROCm/RDNA4 overlay + quantization recipe.
47
-
48
- ---
49
-
50
- ## Table of contents
51
-
52
- - [Attribution & credits](#attribution--credits)
53
- - [Quantization](#quantization)
54
- - [Quick start](#quick-start)
55
- - [Model details](#model-details)
56
- - [Multimodal policy](#multimodal-policy)
57
- - [Known limitations](#known-limitations)
58
- - [License](#license)
59
-
60
- ---
61
-
62
- ## Attribution & credits
63
-
64
- | Component | Credit |
65
- |-----------|--------|
66
- | **Base model** | [zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash) (Z.ai, MIT) |
67
- | **Quantizer & quant kernels (RFA / RFI)** | [`tcclaviger/vllm:latest`](https://hub.docker.com/r/tcclaviger/vllm) (IronLLM Labs) |
68
- | **RDNA4 port + quant recipe** | [GLM-5.3-Flash-rocm-r9700](https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700) (this project's overlay) |
69
-
70
- The composite **RFA + RFI8** quantization scheme, its kernels, and the serving runtime are provided
71
- by **`tcclaviger/vllm` (IronLLM Labs)**. This checkpoint is the result of applying a quantization
72
- *recipe* (archspec + source patches + kda-remap, in the companion repo) on top of that stack, and a
73
- port of the GLM-5.3-Flash architecture to the RDNA4 serving path.
74
-
75
- ---
76
-
77
- ## Quantization
78
-
79
- The checkpoint mixes three schemes in one `quant_method: "rfi"` composite:
80
-
81
- | Scheme | Bits | Applied to | Stored size |
82
- |--------|------|------------|-------------|
83
- | **RFA** | 4.5 bpw | MoE routed experts (42 layers × 288 experts) | ~171.3 GB |
84
- | **RFI8** | 8 bpw | attention / shared-expert / dense linears | ~7.8 GB |
85
- | **BF16 / FP32** | 16 / 32 bpw | embeddings, vision tower, norms, MTP layer | ~18.7 GB |
86
-
87
- ### Average bits-per-weight
88
-
89
- ```
90
- bpw = (total safetensors bytes × 8) / total parameters
91
- = (197,843,715,288 × 8) / 321,342,220,638
92
- = 4.9254 ≈ 4.93 bpw
93
- ```
94
-
95
- | Metric | Value |
96
- |--------|-------|
97
- | Total parameters | 321,342,220,638 (~321.3B) |
98
- | On-disk size | 197.8 GB · 25 safetensors shards |
99
- | **Average bpw** | **4.9254 ≈ 4.93** |
100
- | vs. FP8 source | **0.60×** (197.8 GB vs 328.3 GB) |
101
- | vs. BF16 | **0.31×** (197.8 GB vs 642.7 GB) |
102
-
103
- ---
104
-
105
- ## Quick start
106
-
107
- ```bash
108
- docker pull tcclaviger/vllm:latest
109
-
110
- git clone https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700 overlay
111
-
112
- docker run --rm --tty --ipc=host --shm-size=128g \
113
- --device /dev/kfd:/dev/kfd --device /dev/dri:/dev/dri \
114
- -v /path/to/GLM-5.3-Flash-RFA-RFI8-8xR9700:/models:ro \
115
- -v "$PWD/overlay":/overlay:ro \
116
- --entrypoint bash tcclaviger/vllm:latest \
117
- -c "/overlay/apply_overlay.sh && exec vllm serve /models \
118
- --served-model-name glm53-flash --trust-remote-code --quantization rfi \
119
- --tensor-parallel-size 8 --gpu-memory-utilization 0.95 \
120
- --max-model-len 190080 --max-num-seqs 4 --kv-cache-dtype auto"
121
- ```
122
-
123
- ---
124
-
125
- ## Model details
126
-
127
- | Property | Value |
128
- |----------|-------|
129
- | Architecture | `Glm5NextForConditionalGeneration` |
130
- | Layers | 45 = 34 KDA (linear attention) + 11 DSA (sparse-MLA) |
131
- | Routed experts | 288 (top-8) + 1 shared expert |
132
- | Extra | mHC hyper-connections, 1 nextn MTP draft layer, native vision tower |
133
- | Context (bf16 KV) | 190,080 tokens |
134
-
135
- ---
136
-
137
- ## Multimodal policy
138
-
139
- Images are resized with aspect ratio preserved, clamped to **min 384×384 / max 1280×1280**, and
140
- fed with a min/max image-token budget. The model accepts image and video inputs natively.
141
-
142
- ---
143
-
144
- ## Known limitations
145
-
146
- - **MTP is disabled** in the reference serving config (drafter KV-group blocker).
147
- - **Serve with bf16 KV** (`--kv-cache-dtype auto`) — fp8 KV with runtime scale calibration is
148
- broken on this architecture (garbage scales from the uninitialized KDA recurrent state).
149
- - **Chat needs `reasoning_effort="low"`** — the default Reasoning Effort Max spends 16k+ tokens
150
- thinking before producing content on long generations.
151
-
152
- ---
153
-
154
- ## License
155
-
156
- MIT. Base model © Z.ai (zai-org), MIT license. Quantizer & runtime © IronLLM Labs
157
- ([tcclaviger/vllm](https://hub.docker.com/r/tcclaviger/vllm)).
 
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Z.AI Co., Ltd
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
README.md CHANGED
@@ -1,62 +1,118 @@
1
  ---
2
  license: mit
3
  base_model: zai-org/GLM-5.3-Flash
 
 
4
  tags:
5
  - glm
6
- - mla
7
- - linear-attention
8
- - moe
9
  - multimodal
 
 
 
 
 
 
 
10
  - rocm
11
  - rdna4
12
  - gfx1201
13
- - rfa
14
- - rfi
15
- - amd
16
- - text-generation
17
  ---
18
 
19
- # GLM-5.3-Flash RFA-RFI8 — FOR INFERENCE ON 8× R9700 (RDNA4/gfx1201)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
20
 
21
- A **self-quantized derivative** of
22
- **[zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash)** (321B MoE, ≈18B active),
23
- quantized with a **RFA + RFI8 composite** and validated for serving on
24
- **8× AMD Radeon R9700 (gfx1201 / RDNA4)**.
25
 
26
- > ⚠️ **Not compatible with stock vLLM.** This checkpoint uses the `rfi` composite quantizer and the
27
- > `Glm5NextForConditionalGeneration` architecture on the RDNA4 path, which requires the patched
28
- > vLLM + overlay in **[djdeniro/GLM-5.3-Flash-rocm-r9700](https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700)**.
 
 
 
 
29
 
30
  ---
31
 
32
- ## Model
33
 
34
- - **Base:** [zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash) — MIT license.
35
- - **Architecture:** 321B MoE, ≈18B active; **45 layers = 34 KDA (linear attention) + 11 DSA
36
- (sparse-MLA)**; mHC hidden-state compression; **native multimodal** (image + video); 1 nextn MTP
37
- layer; 288 routed experts (top-8) + 1 shared expert.
38
- - **Paper:** [arXiv:2602.15763](https://arxiv.org/abs/2602.15763).
 
 
 
 
 
 
 
39
 
40
  ## Quantization
41
 
42
- | Scheme | Applied to | bpw |
43
- |--------|-----------|-----|
44
- | **RFA** | MoE routed experts | 4.5 |
45
- | **RFI8** (int8 W8A8) | attention / shared / dense linears (structural) | 8.25 |
46
- | **BF16** | dense copies + MTP layer | 16 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
47
 
48
- Total **≈197.8 GB** (25 safetensors shards). Full recipe (archspec, source patches, kda-remap,
49
- run scripts) is in the code repo: **[djdeniro/GLM-5.3-Flash-rocm-r9700](https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700)** (`quant/`).
50
 
51
- ## Quick start (8× R9700)
52
 
53
  ```bash
 
 
54
  git clone https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700 overlay
55
- huggingface-cli download djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700 --local-dir ./models
56
 
57
  docker run --rm --tty --ipc=host --shm-size=128g \
58
  --device /dev/kfd:/dev/kfd --device /dev/dri:/dev/dri \
59
- -v "$PWD/models":/models:ro -v "$PWD/overlay":/overlay:ro \
 
60
  --entrypoint bash tcclaviger/vllm:latest \
61
  -c "/overlay/apply_overlay.sh && exec vllm serve /models \
62
  --served-model-name glm53-flash --trust-remote-code --quantization rfi \
@@ -64,32 +120,38 @@ docker run --rm --tty --ipc=host --shm-size=128g \
64
  --max-model-len 190080 --max-num-seqs 4 --kv-cache-dtype auto"
65
  ```
66
 
67
- Use `overlay/run-glm53.sh` (or `denet-large.sh` via llama-swap) for the full production command.
68
 
69
- ## Performance (8× R9700, gfx1201)
70
 
71
- - **Decode:** ≈ **34–37 t/s** at bs=1 (FULL cudagraph).
72
- - **TTFT:** ≈ **0.13–0.6 s** (prompt-dependent, `reasoning_effort="low"`).
73
- - **Context:** **190k** tokens with bf16 KV.
 
 
 
 
 
 
74
 
75
- ## Multimodal
76
 
77
- Images are resized preserving aspect ratio: **min 384×384** (upscale) / **max 1280×1280**
78
- (downscale). The processor/video tower is the native Glm5Next multimodal path.
 
 
79
 
80
  ## Known limitations
81
 
82
- - **MTP is OFF** — the nextn drafter is blocked by vLLM's kv-cache-group assertion.
83
- - **Do NOT enable fp8 KV** (`--kv-cache-dtype fp8` + `--calculate-kv-scales`): runtime scale
84
- calibration runs through an unwarmed KDA recurrent state on the profile dummy-run, producing
85
- wrong `_k_scale` → hard output looping (upstream vLLM issue **#37554**). The checkpoint ships no
86
- static KV scales — serve with **bf16 KV** (`--kv-cache-dtype auto`).
87
- - **Chat:** send `reasoning_effort="low"` (the GLM-5.3 chat template defaults to Reasoning Effort
88
- Max and over-thinks on long generations).
89
 
90
- ## License & attribution
91
 
92
- - **License:** MIT.
93
- - **Original model:** [zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash) — © Z.ai
94
- (zai-org), MIT license.
95
- - **Paper:** [arXiv:2602.15763](https://arxiv.org/abs/2602.15763).
 
1
  ---
2
  license: mit
3
  base_model: zai-org/GLM-5.3-Flash
4
+ pipeline_tag: image-text-to-text
5
+ library_name: transformers
6
  tags:
7
  - glm
8
+ - glm-5
 
 
9
  - multimodal
10
+ - vision-language
11
+ - moe
12
+ - sparse-attention
13
+ - mla
14
+ - quantization
15
+ - rfa
16
+ - rfi
17
  - rocm
18
  - rdna4
19
  - gfx1201
20
+ - 4-bit
 
 
 
21
  ---
22
 
23
+ <div align="center">
24
+
25
+ # GLM-5.3-Flash · RFA + RFI8 composite quant (8× R9700 / RDNA4)
26
+
27
+ **A self-quantized derivative of [zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash), tuned for 8× AMD Radeon R9700 (gfx1201 / RDNA4)**
28
+
29
+ [![Model size](https://img.shields.io/badge/params-321B%20(18B%20active)-blue)](https://huggingface.co/zai-org/GLM-5.3-Flash)
30
+ [![bpw](https://img.shields.io/badge/bpw-4.93-ff69b4)](#quantization)
31
+ [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE)
32
+ [![Serving stack](https://img.shields.io/badge/serving%20stack-tcclaviger%2Fvllm-00a2a2)](https://hub.docker.com/r/tcclaviger/vllm)
33
+ [![GPU](https://img.shields.io/badge/GPU-RDNA4%20(gfx1201)-ED1C24)](https://www.amd.com/en/products/processors/desktops/radeon.html)
34
+
35
+ </div>
36
+
37
+ ---
38
+
39
+ ## Model card
40
+
41
+ This is a **composite-quantized** checkpoint of Z.ai's **GLM-5.3-Flash** — the first natively
42
+ multimodal model in the GLM-5 series (321B total / ~18B active parameters). It was quantized from
43
+ the official FP8 release using the `tcclaviger/vllm` composite quantizer and is intended to be
44
+ served with **vLLM** on **RDNA4** (AMD Radeon R9700) hardware.
45
+
46
+ > **📦 Companion serving repo:** [GLM-5.3-Flash-rocm-r9700](https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700) — the ROCm/RDNA4 overlay + quantization recipe.
47
+
48
+ ---
49
 
50
+ ## Table of contents
 
 
 
51
 
52
+ - [Attribution & credits](#attribution--credits)
53
+ - [Quantization](#quantization)
54
+ - [Quick start](#quick-start)
55
+ - [Model details](#model-details)
56
+ - [Multimodal policy](#multimodal-policy)
57
+ - [Known limitations](#known-limitations)
58
+ - [License](#license)
59
 
60
  ---
61
 
62
+ ## Attribution & credits
63
 
64
+ | Component | Credit |
65
+ |-----------|--------|
66
+ | **Base model** | [zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash) (Z.ai, MIT) |
67
+ | **Quantizer & quant kernels (RFA / RFI)** | [`tcclaviger/vllm:latest`](https://hub.docker.com/r/tcclaviger/vllm) (IronLLM Labs) |
68
+ | **RDNA4 port + quant recipe** | [GLM-5.3-Flash-rocm-r9700](https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700) (this project's overlay) |
69
+
70
+ The composite **RFA + RFI8** quantization scheme, its kernels, and the serving runtime are provided
71
+ by **`tcclaviger/vllm` (IronLLM Labs)**. This checkpoint is the result of applying a quantization
72
+ *recipe* (archspec + source patches + kda-remap, in the companion repo) on top of that stack, and a
73
+ port of the GLM-5.3-Flash architecture to the RDNA4 serving path.
74
+
75
+ ---
76
 
77
  ## Quantization
78
 
79
+ The checkpoint mixes three schemes in one `quant_method: "rfi"` composite:
80
+
81
+ | Scheme | Bits | Applied to | Stored size |
82
+ |--------|------|------------|-------------|
83
+ | **RFA** | 4.5 bpw | MoE routed experts (42 layers × 288 experts) | ~171.3 GB |
84
+ | **RFI8** | 8 bpw | attention / shared-expert / dense linears | ~7.8 GB |
85
+ | **BF16 / FP32** | 16 / 32 bpw | embeddings, vision tower, norms, MTP layer | ~18.7 GB |
86
+
87
+ ### Average bits-per-weight
88
+
89
+ ```
90
+ bpw = (total safetensors bytes × 8) / total parameters
91
+ = (197,843,715,288 × 8) / 321,342,220,638
92
+ = 4.9254 ≈ 4.93 bpw
93
+ ```
94
+
95
+ | Metric | Value |
96
+ |--------|-------|
97
+ | Total parameters | 321,342,220,638 (~321.3B) |
98
+ | On-disk size | 197.8 GB · 25 safetensors shards |
99
+ | **Average bpw** | **4.9254 ≈ 4.93** |
100
+ | vs. FP8 source | **0.60×** (197.8 GB vs 328.3 GB) |
101
+ | vs. BF16 | **0.31×** (197.8 GB vs 642.7 GB) |
102
 
103
+ ---
 
104
 
105
+ ## Quick start
106
 
107
  ```bash
108
+ docker pull tcclaviger/vllm:latest
109
+
110
  git clone https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700 overlay
 
111
 
112
  docker run --rm --tty --ipc=host --shm-size=128g \
113
  --device /dev/kfd:/dev/kfd --device /dev/dri:/dev/dri \
114
+ -v /path/to/GLM-5.3-Flash-RFA-RFI8-8xR9700:/models:ro \
115
+ -v "$PWD/overlay":/overlay:ro \
116
  --entrypoint bash tcclaviger/vllm:latest \
117
  -c "/overlay/apply_overlay.sh && exec vllm serve /models \
118
  --served-model-name glm53-flash --trust-remote-code --quantization rfi \
 
120
  --max-model-len 190080 --max-num-seqs 4 --kv-cache-dtype auto"
121
  ```
122
 
123
+ ---
124
 
125
+ ## Model details
126
 
127
+ | Property | Value |
128
+ |----------|-------|
129
+ | Architecture | `Glm5NextForConditionalGeneration` |
130
+ | Layers | 45 = 34 KDA (linear attention) + 11 DSA (sparse-MLA) |
131
+ | Routed experts | 288 (top-8) + 1 shared expert |
132
+ | Extra | mHC hyper-connections, 1 nextn MTP draft layer, native vision tower |
133
+ | Context (bf16 KV) | 190,080 tokens |
134
+
135
+ ---
136
 
137
+ ## Multimodal policy
138
 
139
+ Images are resized with aspect ratio preserved, clamped to **min 384×384 / max 1280×1280**, and
140
+ fed with a min/max image-token budget. The model accepts image and video inputs natively.
141
+
142
+ ---
143
 
144
  ## Known limitations
145
 
146
+ - **MTP is disabled** in the reference serving config (drafter KV-group blocker).
147
+ - **Serve with bf16 KV** (`--kv-cache-dtype auto`) — fp8 KV with runtime scale calibration is
148
+ broken on this architecture (garbage scales from the uninitialized KDA recurrent state).
149
+ - **Chat needs `reasoning_effort="low"`** — the default Reasoning Effort Max spends 16k+ tokens
150
+ thinking before producing content on long generations.
151
+
152
+ ---
153
 
154
+ ## License
155
 
156
+ MIT. Base model © Z.ai (zai-org), MIT license. Quantizer & runtime © IronLLM Labs
157
+ ([tcclaviger/vllm](https://hub.docker.com/r/tcclaviger/vllm)).