model card: sglang only
Browse files
README.md
CHANGED
|
@@ -7,7 +7,6 @@ tags:
|
|
| 7 |
- vision-language
|
| 8 |
- glm
|
| 9 |
- sglang
|
| 10 |
-
- vllm
|
| 11 |
base_model:
|
| 12 |
- zai-org/GLM-5.2
|
| 13 |
---
|
|
@@ -17,7 +16,7 @@ base_model:
|
|
| 17 |
**GLM-5.2 with sight.** A vision-language model that bolts the MoonViT vision encoder from
|
| 18 |
[Kimi-K2.6](https://huggingface.co/moonshotai/Kimi-K2.6) onto
|
| 19 |
[GLM-5.2](https://huggingface.co/zai-org/GLM-5.2) through a trained PatchMerger projector —
|
| 20 |
-
shipped as **one self-contained checkpoint** you can hand straight to SGLang
|
| 21 |
|
| 22 |
GLM-5.2 is a strong open reasoning model with no vision input. This checkpoint adds it,
|
| 23 |
without touching a single GLM weight: the text backbone and the vision tower are both frozen
|
|
@@ -45,8 +44,8 @@ At ~757 GB the weights do not fit on four GPUs. For a 4-GPU deployment use
|
|
| 45 |
|
| 46 |
## Quickstart
|
| 47 |
|
| 48 |
-
|
| 49 |
-
|
| 50 |
|
| 51 |
```bash
|
| 52 |
hf download baseten/GLM-5.2-Vision-FP8 --include 'plugins/*' --local-dir ./glm5v
|
|
@@ -77,19 +76,6 @@ python -m sglang.launch_server \
|
|
| 77 |
--port 30000
|
| 78 |
```
|
| 79 |
|
| 80 |
-
Measured KV budget at these settings: ~1.16M tokens.
|
| 81 |
-
|
| 82 |
-
### vLLM
|
| 83 |
-
|
| 84 |
-
The pip install registers a `vllm.general_plugins` entry point, so no flags and no env vars:
|
| 85 |
-
|
| 86 |
-
```bash
|
| 87 |
-
vllm serve baseten/GLM-5.2-Vision-FP8 --trust-remote-code --tensor-parallel-size 8
|
| 88 |
-
```
|
| 89 |
-
|
| 90 |
-
> Note: the vLLM path is wired up but has had less testing than SGLang here. If you hit
|
| 91 |
-
> trouble, SGLang is the better-trodden route.
|
| 92 |
-
|
| 93 |
### Query it
|
| 94 |
|
| 95 |
Standard OpenAI multimodal messages — the image is delivered as `image_url`:
|
|
|
|
| 7 |
- vision-language
|
| 8 |
- glm
|
| 9 |
- sglang
|
|
|
|
| 10 |
base_model:
|
| 11 |
- zai-org/GLM-5.2
|
| 12 |
---
|
|
|
|
| 16 |
**GLM-5.2 with sight.** A vision-language model that bolts the MoonViT vision encoder from
|
| 17 |
[Kimi-K2.6](https://huggingface.co/moonshotai/Kimi-K2.6) onto
|
| 18 |
[GLM-5.2](https://huggingface.co/zai-org/GLM-5.2) through a trained PatchMerger projector —
|
| 19 |
+
shipped as **one self-contained checkpoint** you can hand straight to SGLang.
|
| 20 |
|
| 21 |
GLM-5.2 is a strong open reasoning model with no vision input. This checkpoint adds it,
|
| 22 |
without touching a single GLM weight: the text backbone and the vision tower are both frozen
|
|
|
|
| 44 |
|
| 45 |
## Quickstart
|
| 46 |
|
| 47 |
+
SGLang needs a small out-of-tree plugin, because `Glm5vForConditionalGeneration` is not yet an
|
| 48 |
+
upstream architecture. It ships inside this repo, so there is nothing else to clone:
|
| 49 |
|
| 50 |
```bash
|
| 51 |
hf download baseten/GLM-5.2-Vision-FP8 --include 'plugins/*' --local-dir ./glm5v
|
|
|
|
| 76 |
--port 30000
|
| 77 |
```
|
| 78 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 79 |
### Query it
|
| 80 |
|
| 81 |
Standard OpenAI multimodal messages — the image is delivered as `image_url`:
|