Image-Text-to-Text
Transformers
Safetensors
glm5_next
glm
glm-5
multimodal
vision-language
Mixture of Experts
sparse-attention
mla
quantization
rfa
rfi
rocm
rdna4
gfx1201
4-bit precision
conversational
8-bit precision
Instructions to use djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700") model = AutoModelForMultimodalLM.from_pretrained("djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700
- SGLang
How to use djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700 with Docker Model Runner:
docker model run hf.co/djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700
Model card: attribution + avg bpw 4.93 + multimodal frontmatter (fix file paths)
Browse files
LICENSE
CHANGED
|
@@ -1,157 +1,21 @@
|
|
| 1 |
-
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
<div align="center">
|
| 24 |
-
|
| 25 |
-
# GLM-5.3-Flash · RFA + RFI8 composite quant (8× R9700 / RDNA4)
|
| 26 |
-
|
| 27 |
-
**A self-quantized derivative of [zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash), tuned for 8× AMD Radeon R9700 (gfx1201 / RDNA4)**
|
| 28 |
-
|
| 29 |
-
[-blue)](https://huggingface.co/zai-org/GLM-5.3-Flash)
|
| 30 |
-
[](#quantization)
|
| 31 |
-
[](./LICENSE)
|
| 32 |
-
[](https://hub.docker.com/r/tcclaviger/vllm)
|
| 33 |
-
[-ED1C24)](https://www.amd.com/en/products/processors/desktops/radeon.html)
|
| 34 |
-
|
| 35 |
-
</div>
|
| 36 |
-
|
| 37 |
-
---
|
| 38 |
-
|
| 39 |
-
## Model card
|
| 40 |
-
|
| 41 |
-
This is a **composite-quantized** checkpoint of Z.ai's **GLM-5.3-Flash** — the first natively
|
| 42 |
-
multimodal model in the GLM-5 series (321B total / ~18B active parameters). It was quantized from
|
| 43 |
-
the official FP8 release using the `tcclaviger/vllm` composite quantizer and is intended to be
|
| 44 |
-
served with **vLLM** on **RDNA4** (AMD Radeon R9700) hardware.
|
| 45 |
-
|
| 46 |
-
> **📦 Companion serving repo:** [GLM-5.3-Flash-rocm-r9700](https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700) — the ROCm/RDNA4 overlay + quantization recipe.
|
| 47 |
-
|
| 48 |
-
---
|
| 49 |
-
|
| 50 |
-
## Table of contents
|
| 51 |
-
|
| 52 |
-
- [Attribution & credits](#attribution--credits)
|
| 53 |
-
- [Quantization](#quantization)
|
| 54 |
-
- [Quick start](#quick-start)
|
| 55 |
-
- [Model details](#model-details)
|
| 56 |
-
- [Multimodal policy](#multimodal-policy)
|
| 57 |
-
- [Known limitations](#known-limitations)
|
| 58 |
-
- [License](#license)
|
| 59 |
-
|
| 60 |
-
---
|
| 61 |
-
|
| 62 |
-
## Attribution & credits
|
| 63 |
-
|
| 64 |
-
| Component | Credit |
|
| 65 |
-
|-----------|--------|
|
| 66 |
-
| **Base model** | [zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash) (Z.ai, MIT) |
|
| 67 |
-
| **Quantizer & quant kernels (RFA / RFI)** | [`tcclaviger/vllm:latest`](https://hub.docker.com/r/tcclaviger/vllm) (IronLLM Labs) |
|
| 68 |
-
| **RDNA4 port + quant recipe** | [GLM-5.3-Flash-rocm-r9700](https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700) (this project's overlay) |
|
| 69 |
-
|
| 70 |
-
The composite **RFA + RFI8** quantization scheme, its kernels, and the serving runtime are provided
|
| 71 |
-
by **`tcclaviger/vllm` (IronLLM Labs)**. This checkpoint is the result of applying a quantization
|
| 72 |
-
*recipe* (archspec + source patches + kda-remap, in the companion repo) on top of that stack, and a
|
| 73 |
-
port of the GLM-5.3-Flash architecture to the RDNA4 serving path.
|
| 74 |
-
|
| 75 |
-
---
|
| 76 |
-
|
| 77 |
-
## Quantization
|
| 78 |
-
|
| 79 |
-
The checkpoint mixes three schemes in one `quant_method: "rfi"` composite:
|
| 80 |
-
|
| 81 |
-
| Scheme | Bits | Applied to | Stored size |
|
| 82 |
-
|--------|------|------------|-------------|
|
| 83 |
-
| **RFA** | 4.5 bpw | MoE routed experts (42 layers × 288 experts) | ~171.3 GB |
|
| 84 |
-
| **RFI8** | 8 bpw | attention / shared-expert / dense linears | ~7.8 GB |
|
| 85 |
-
| **BF16 / FP32** | 16 / 32 bpw | embeddings, vision tower, norms, MTP layer | ~18.7 GB |
|
| 86 |
-
|
| 87 |
-
### Average bits-per-weight
|
| 88 |
-
|
| 89 |
-
```
|
| 90 |
-
bpw = (total safetensors bytes × 8) / total parameters
|
| 91 |
-
= (197,843,715,288 × 8) / 321,342,220,638
|
| 92 |
-
= 4.9254 ≈ 4.93 bpw
|
| 93 |
-
```
|
| 94 |
-
|
| 95 |
-
| Metric | Value |
|
| 96 |
-
|--------|-------|
|
| 97 |
-
| Total parameters | 321,342,220,638 (~321.3B) |
|
| 98 |
-
| On-disk size | 197.8 GB · 25 safetensors shards |
|
| 99 |
-
| **Average bpw** | **4.9254 ≈ 4.93** |
|
| 100 |
-
| vs. FP8 source | **0.60×** (197.8 GB vs 328.3 GB) |
|
| 101 |
-
| vs. BF16 | **0.31×** (197.8 GB vs 642.7 GB) |
|
| 102 |
-
|
| 103 |
-
---
|
| 104 |
-
|
| 105 |
-
## Quick start
|
| 106 |
-
|
| 107 |
-
```bash
|
| 108 |
-
docker pull tcclaviger/vllm:latest
|
| 109 |
-
|
| 110 |
-
git clone https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700 overlay
|
| 111 |
-
|
| 112 |
-
docker run --rm --tty --ipc=host --shm-size=128g \
|
| 113 |
-
--device /dev/kfd:/dev/kfd --device /dev/dri:/dev/dri \
|
| 114 |
-
-v /path/to/GLM-5.3-Flash-RFA-RFI8-8xR9700:/models:ro \
|
| 115 |
-
-v "$PWD/overlay":/overlay:ro \
|
| 116 |
-
--entrypoint bash tcclaviger/vllm:latest \
|
| 117 |
-
-c "/overlay/apply_overlay.sh && exec vllm serve /models \
|
| 118 |
-
--served-model-name glm53-flash --trust-remote-code --quantization rfi \
|
| 119 |
-
--tensor-parallel-size 8 --gpu-memory-utilization 0.95 \
|
| 120 |
-
--max-model-len 190080 --max-num-seqs 4 --kv-cache-dtype auto"
|
| 121 |
-
```
|
| 122 |
-
|
| 123 |
-
---
|
| 124 |
-
|
| 125 |
-
## Model details
|
| 126 |
-
|
| 127 |
-
| Property | Value |
|
| 128 |
-
|----------|-------|
|
| 129 |
-
| Architecture | `Glm5NextForConditionalGeneration` |
|
| 130 |
-
| Layers | 45 = 34 KDA (linear attention) + 11 DSA (sparse-MLA) |
|
| 131 |
-
| Routed experts | 288 (top-8) + 1 shared expert |
|
| 132 |
-
| Extra | mHC hyper-connections, 1 nextn MTP draft layer, native vision tower |
|
| 133 |
-
| Context (bf16 KV) | 190,080 tokens |
|
| 134 |
-
|
| 135 |
-
---
|
| 136 |
-
|
| 137 |
-
## Multimodal policy
|
| 138 |
-
|
| 139 |
-
Images are resized with aspect ratio preserved, clamped to **min 384×384 / max 1280×1280**, and
|
| 140 |
-
fed with a min/max image-token budget. The model accepts image and video inputs natively.
|
| 141 |
-
|
| 142 |
-
---
|
| 143 |
-
|
| 144 |
-
## Known limitations
|
| 145 |
-
|
| 146 |
-
- **MTP is disabled** in the reference serving config (drafter KV-group blocker).
|
| 147 |
-
- **Serve with bf16 KV** (`--kv-cache-dtype auto`) — fp8 KV with runtime scale calibration is
|
| 148 |
-
broken on this architecture (garbage scales from the uninitialized KDA recurrent state).
|
| 149 |
-
- **Chat needs `reasoning_effort="low"`** — the default Reasoning Effort Max spends 16k+ tokens
|
| 150 |
-
thinking before producing content on long generations.
|
| 151 |
-
|
| 152 |
-
---
|
| 153 |
-
|
| 154 |
-
## License
|
| 155 |
-
|
| 156 |
-
MIT. Base model © Z.ai (zai-org), MIT license. Quantizer & runtime © IronLLM Labs
|
| 157 |
-
([tcclaviger/vllm](https://hub.docker.com/r/tcclaviger/vllm)).
|
|
|
|
| 1 |
+
MIT License
|
| 2 |
+
|
| 3 |
+
Copyright (c) 2026 Z.AI Co., Ltd
|
| 4 |
+
|
| 5 |
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
| 6 |
+
of this software and associated documentation files (the "Software"), to deal
|
| 7 |
+
in the Software without restriction, including without limitation the rights
|
| 8 |
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
| 9 |
+
copies of the Software, and to permit persons to whom the Software is
|
| 10 |
+
furnished to do so, subject to the following conditions:
|
| 11 |
+
|
| 12 |
+
The above copyright notice and this permission notice shall be included in all
|
| 13 |
+
copies or substantial portions of the Software.
|
| 14 |
+
|
| 15 |
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
| 16 |
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
| 17 |
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
| 18 |
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
| 19 |
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
| 20 |
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
| 21 |
+
SOFTWARE.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
README.md
CHANGED
|
@@ -1,62 +1,118 @@
|
|
| 1 |
---
|
| 2 |
license: mit
|
| 3 |
base_model: zai-org/GLM-5.3-Flash
|
|
|
|
|
|
|
| 4 |
tags:
|
| 5 |
- glm
|
| 6 |
-
-
|
| 7 |
-
- linear-attention
|
| 8 |
-
- moe
|
| 9 |
- multimodal
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 10 |
- rocm
|
| 11 |
- rdna4
|
| 12 |
- gfx1201
|
| 13 |
-
-
|
| 14 |
-
- rfi
|
| 15 |
-
- amd
|
| 16 |
-
- text-generation
|
| 17 |
---
|
| 18 |
|
| 19 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 20 |
|
| 21 |
-
|
| 22 |
-
**[zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash)** (321B MoE, ≈18B active),
|
| 23 |
-
quantized with a **RFA + RFI8 composite** and validated for serving on
|
| 24 |
-
**8× AMD Radeon R9700 (gfx1201 / RDNA4)**.
|
| 25 |
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
|
| 30 |
---
|
| 31 |
|
| 32 |
-
##
|
| 33 |
|
| 34 |
-
|
| 35 |
-
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
|
| 40 |
## Quantization
|
| 41 |
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
|
| 45 |
-
|
|
| 46 |
-
| **
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 47 |
|
| 48 |
-
|
| 49 |
-
run scripts) is in the code repo: **[djdeniro/GLM-5.3-Flash-rocm-r9700](https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700)** (`quant/`).
|
| 50 |
|
| 51 |
-
## Quick start
|
| 52 |
|
| 53 |
```bash
|
|
|
|
|
|
|
| 54 |
git clone https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700 overlay
|
| 55 |
-
huggingface-cli download djdeniro/GLM-5.3-Flash-RFA-RFI8-8xR9700 --local-dir ./models
|
| 56 |
|
| 57 |
docker run --rm --tty --ipc=host --shm-size=128g \
|
| 58 |
--device /dev/kfd:/dev/kfd --device /dev/dri:/dev/dri \
|
| 59 |
-
-v
|
|
|
|
| 60 |
--entrypoint bash tcclaviger/vllm:latest \
|
| 61 |
-c "/overlay/apply_overlay.sh && exec vllm serve /models \
|
| 62 |
--served-model-name glm53-flash --trust-remote-code --quantization rfi \
|
|
@@ -64,32 +120,38 @@ docker run --rm --tty --ipc=host --shm-size=128g \
|
|
| 64 |
--max-model-len 190080 --max-num-seqs 4 --kv-cache-dtype auto"
|
| 65 |
```
|
| 66 |
|
| 67 |
-
|
| 68 |
|
| 69 |
-
##
|
| 70 |
|
| 71 |
-
|
| 72 |
-
-
|
| 73 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 74 |
|
| 75 |
-
## Multimodal
|
| 76 |
|
| 77 |
-
Images are resized
|
| 78 |
-
|
|
|
|
|
|
|
| 79 |
|
| 80 |
## Known limitations
|
| 81 |
|
| 82 |
-
- **MTP is
|
| 83 |
-
- **
|
| 84 |
-
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
| 89 |
|
| 90 |
-
## License
|
| 91 |
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
(zai-org), MIT license.
|
| 95 |
-
- **Paper:** [arXiv:2602.15763](https://arxiv.org/abs/2602.15763).
|
|
|
|
| 1 |
---
|
| 2 |
license: mit
|
| 3 |
base_model: zai-org/GLM-5.3-Flash
|
| 4 |
+
pipeline_tag: image-text-to-text
|
| 5 |
+
library_name: transformers
|
| 6 |
tags:
|
| 7 |
- glm
|
| 8 |
+
- glm-5
|
|
|
|
|
|
|
| 9 |
- multimodal
|
| 10 |
+
- vision-language
|
| 11 |
+
- moe
|
| 12 |
+
- sparse-attention
|
| 13 |
+
- mla
|
| 14 |
+
- quantization
|
| 15 |
+
- rfa
|
| 16 |
+
- rfi
|
| 17 |
- rocm
|
| 18 |
- rdna4
|
| 19 |
- gfx1201
|
| 20 |
+
- 4-bit
|
|
|
|
|
|
|
|
|
|
| 21 |
---
|
| 22 |
|
| 23 |
+
<div align="center">
|
| 24 |
+
|
| 25 |
+
# GLM-5.3-Flash · RFA + RFI8 composite quant (8× R9700 / RDNA4)
|
| 26 |
+
|
| 27 |
+
**A self-quantized derivative of [zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash), tuned for 8× AMD Radeon R9700 (gfx1201 / RDNA4)**
|
| 28 |
+
|
| 29 |
+
[-blue)](https://huggingface.co/zai-org/GLM-5.3-Flash)
|
| 30 |
+
[](#quantization)
|
| 31 |
+
[](./LICENSE)
|
| 32 |
+
[](https://hub.docker.com/r/tcclaviger/vllm)
|
| 33 |
+
[-ED1C24)](https://www.amd.com/en/products/processors/desktops/radeon.html)
|
| 34 |
+
|
| 35 |
+
</div>
|
| 36 |
+
|
| 37 |
+
---
|
| 38 |
+
|
| 39 |
+
## Model card
|
| 40 |
+
|
| 41 |
+
This is a **composite-quantized** checkpoint of Z.ai's **GLM-5.3-Flash** — the first natively
|
| 42 |
+
multimodal model in the GLM-5 series (321B total / ~18B active parameters). It was quantized from
|
| 43 |
+
the official FP8 release using the `tcclaviger/vllm` composite quantizer and is intended to be
|
| 44 |
+
served with **vLLM** on **RDNA4** (AMD Radeon R9700) hardware.
|
| 45 |
+
|
| 46 |
+
> **📦 Companion serving repo:** [GLM-5.3-Flash-rocm-r9700](https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700) — the ROCm/RDNA4 overlay + quantization recipe.
|
| 47 |
+
|
| 48 |
+
---
|
| 49 |
|
| 50 |
+
## Table of contents
|
|
|
|
|
|
|
|
|
|
| 51 |
|
| 52 |
+
- [Attribution & credits](#attribution--credits)
|
| 53 |
+
- [Quantization](#quantization)
|
| 54 |
+
- [Quick start](#quick-start)
|
| 55 |
+
- [Model details](#model-details)
|
| 56 |
+
- [Multimodal policy](#multimodal-policy)
|
| 57 |
+
- [Known limitations](#known-limitations)
|
| 58 |
+
- [License](#license)
|
| 59 |
|
| 60 |
---
|
| 61 |
|
| 62 |
+
## Attribution & credits
|
| 63 |
|
| 64 |
+
| Component | Credit |
|
| 65 |
+
|-----------|--------|
|
| 66 |
+
| **Base model** | [zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash) (Z.ai, MIT) |
|
| 67 |
+
| **Quantizer & quant kernels (RFA / RFI)** | [`tcclaviger/vllm:latest`](https://hub.docker.com/r/tcclaviger/vllm) (IronLLM Labs) |
|
| 68 |
+
| **RDNA4 port + quant recipe** | [GLM-5.3-Flash-rocm-r9700](https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700) (this project's overlay) |
|
| 69 |
+
|
| 70 |
+
The composite **RFA + RFI8** quantization scheme, its kernels, and the serving runtime are provided
|
| 71 |
+
by **`tcclaviger/vllm` (IronLLM Labs)**. This checkpoint is the result of applying a quantization
|
| 72 |
+
*recipe* (archspec + source patches + kda-remap, in the companion repo) on top of that stack, and a
|
| 73 |
+
port of the GLM-5.3-Flash architecture to the RDNA4 serving path.
|
| 74 |
+
|
| 75 |
+
---
|
| 76 |
|
| 77 |
## Quantization
|
| 78 |
|
| 79 |
+
The checkpoint mixes three schemes in one `quant_method: "rfi"` composite:
|
| 80 |
+
|
| 81 |
+
| Scheme | Bits | Applied to | Stored size |
|
| 82 |
+
|--------|------|------------|-------------|
|
| 83 |
+
| **RFA** | 4.5 bpw | MoE routed experts (42 layers × 288 experts) | ~171.3 GB |
|
| 84 |
+
| **RFI8** | 8 bpw | attention / shared-expert / dense linears | ~7.8 GB |
|
| 85 |
+
| **BF16 / FP32** | 16 / 32 bpw | embeddings, vision tower, norms, MTP layer | ~18.7 GB |
|
| 86 |
+
|
| 87 |
+
### Average bits-per-weight
|
| 88 |
+
|
| 89 |
+
```
|
| 90 |
+
bpw = (total safetensors bytes × 8) / total parameters
|
| 91 |
+
= (197,843,715,288 × 8) / 321,342,220,638
|
| 92 |
+
= 4.9254 ≈ 4.93 bpw
|
| 93 |
+
```
|
| 94 |
+
|
| 95 |
+
| Metric | Value |
|
| 96 |
+
|--------|-------|
|
| 97 |
+
| Total parameters | 321,342,220,638 (~321.3B) |
|
| 98 |
+
| On-disk size | 197.8 GB · 25 safetensors shards |
|
| 99 |
+
| **Average bpw** | **4.9254 ≈ 4.93** |
|
| 100 |
+
| vs. FP8 source | **0.60×** (197.8 GB vs 328.3 GB) |
|
| 101 |
+
| vs. BF16 | **0.31×** (197.8 GB vs 642.7 GB) |
|
| 102 |
|
| 103 |
+
---
|
|
|
|
| 104 |
|
| 105 |
+
## Quick start
|
| 106 |
|
| 107 |
```bash
|
| 108 |
+
docker pull tcclaviger/vllm:latest
|
| 109 |
+
|
| 110 |
git clone https://huggingface.co/djdeniro/GLM-5.3-Flash-rocm-r9700 overlay
|
|
|
|
| 111 |
|
| 112 |
docker run --rm --tty --ipc=host --shm-size=128g \
|
| 113 |
--device /dev/kfd:/dev/kfd --device /dev/dri:/dev/dri \
|
| 114 |
+
-v /path/to/GLM-5.3-Flash-RFA-RFI8-8xR9700:/models:ro \
|
| 115 |
+
-v "$PWD/overlay":/overlay:ro \
|
| 116 |
--entrypoint bash tcclaviger/vllm:latest \
|
| 117 |
-c "/overlay/apply_overlay.sh && exec vllm serve /models \
|
| 118 |
--served-model-name glm53-flash --trust-remote-code --quantization rfi \
|
|
|
|
| 120 |
--max-model-len 190080 --max-num-seqs 4 --kv-cache-dtype auto"
|
| 121 |
```
|
| 122 |
|
| 123 |
+
---
|
| 124 |
|
| 125 |
+
## Model details
|
| 126 |
|
| 127 |
+
| Property | Value |
|
| 128 |
+
|----------|-------|
|
| 129 |
+
| Architecture | `Glm5NextForConditionalGeneration` |
|
| 130 |
+
| Layers | 45 = 34 KDA (linear attention) + 11 DSA (sparse-MLA) |
|
| 131 |
+
| Routed experts | 288 (top-8) + 1 shared expert |
|
| 132 |
+
| Extra | mHC hyper-connections, 1 nextn MTP draft layer, native vision tower |
|
| 133 |
+
| Context (bf16 KV) | 190,080 tokens |
|
| 134 |
+
|
| 135 |
+
---
|
| 136 |
|
| 137 |
+
## Multimodal policy
|
| 138 |
|
| 139 |
+
Images are resized with aspect ratio preserved, clamped to **min 384×384 / max 1280×1280**, and
|
| 140 |
+
fed with a min/max image-token budget. The model accepts image and video inputs natively.
|
| 141 |
+
|
| 142 |
+
---
|
| 143 |
|
| 144 |
## Known limitations
|
| 145 |
|
| 146 |
+
- **MTP is disabled** in the reference serving config (drafter KV-group blocker).
|
| 147 |
+
- **Serve with bf16 KV** (`--kv-cache-dtype auto`) — fp8 KV with runtime scale calibration is
|
| 148 |
+
broken on this architecture (garbage scales from the uninitialized KDA recurrent state).
|
| 149 |
+
- **Chat needs `reasoning_effort="low"`** — the default Reasoning Effort Max spends 16k+ tokens
|
| 150 |
+
thinking before producing content on long generations.
|
| 151 |
+
|
| 152 |
+
---
|
| 153 |
|
| 154 |
+
## License
|
| 155 |
|
| 156 |
+
MIT. Base model © Z.ai (zai-org), MIT license. Quantizer & runtime © IronLLM Labs
|
| 157 |
+
([tcclaviger/vllm](https://hub.docker.com/r/tcclaviger/vllm)).
|
|
|
|
|
|