Instructions to use RAS-LAB/GLM-4.7-Flash-24B-A3B-KREAP with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use RAS-LAB/GLM-4.7-Flash-24B-A3B-KREAP with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="RAS-LAB/GLM-4.7-Flash-24B-A3B-KREAP")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("RAS-LAB/GLM-4.7-Flash-24B-A3B-KREAP")
model = AutoModelForCausalLM.from_pretrained("RAS-LAB/GLM-4.7-Flash-24B-A3B-KREAP", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use RAS-LAB/GLM-4.7-Flash-24B-A3B-KREAP with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RAS-LAB/GLM-4.7-Flash-24B-A3B-KREAP"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RAS-LAB/GLM-4.7-Flash-24B-A3B-KREAP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/RAS-LAB/GLM-4.7-Flash-24B-A3B-KREAP

SGLang

How to use RAS-LAB/GLM-4.7-Flash-24B-A3B-KREAP with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "RAS-LAB/GLM-4.7-Flash-24B-A3B-KREAP" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RAS-LAB/GLM-4.7-Flash-24B-A3B-KREAP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "RAS-LAB/GLM-4.7-Flash-24B-A3B-KREAP" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RAS-LAB/GLM-4.7-Flash-24B-A3B-KREAP",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use RAS-LAB/GLM-4.7-Flash-24B-A3B-KREAP with Docker Model Runner:
```
docker model run hf.co/RAS-LAB/GLM-4.7-Flash-24B-A3B-KREAP
```

GLM-4.7-Flash-24B-A3B-KREAP / tensor_hash_validation.json

buugiiiiii

Record lossless reshard and Transformers runtime validation

4ecde6a verified 3 months ago

Raw History Blame Contribute Delete

3.85 kB

{
  "status": "PASS",
  "stage": "lossless safetensors checkpoint reshard validation",
  "original_shards": 48,
  "output_shards": 16,
  "tensors": 7495,
  "tensor_equality_checked": 7495,
  "tensor_values_identical": true,
  "total_tensor_bytes": 48548430720,
  "index_sha256": "f14fbfde88c57bb0b073e145365d8cfcac5b36fc0b3c4be6e44d3c2d9c3f446c",
  "shards": [
    {
      "filename": "model-00001-of-00016.safetensors",
      "tensors": 273,
      "bytes": 3035883560,
      "sha256": "2f715596ea6eb95e091b4e8c34d30deeb85bf05d91c4c9cfcca37b36f1572a06"
    },
    {
      "filename": "model-00002-of-00016.safetensors",
      "tensors": 494,
      "bytes": 3031592368,
      "sha256": "170fc73ec42fd58ca2080a250074b33deade410bcbafee407d041166659ac5a7"
    },
    {
      "filename": "model-00003-of-00016.safetensors",
      "tensors": 494,
      "bytes": 3031592360,
      "sha256": "821621b83e3140870154efc65185f9f9d30f9db2ac4a082ae40974dd54ecf498"
    },
    {
      "filename": "model-00004-of-00016.safetensors",
      "tensors": 499,
      "bytes": 3037567928,
      "sha256": "2df6962eea9e2ff55ba76cf8849d4dc55bbf63d6616fdfaace9c7f49cc57d8e9"
    },
    {
      "filename": "model-00005-of-00016.safetensors",
      "tensors": 494,
      "bytes": 3031592856,
      "sha256": "f3a1660e46c3ad0da6abe29a2fd516d920900531f915f0e422a8c995ddf23902"
    },
    {
      "filename": "model-00006-of-00016.safetensors",
      "tensors": 494,
      "bytes": 3031592864,
      "sha256": "be324f5647fcab5a026695c37bf2188cb6e99b6ad7bf29875b85f990ccbaa304"
    },
    {
      "filename": "model-00007-of-00016.safetensors",
      "tensors": 495,
      "bytes": 3037884440,
      "sha256": "5e2da9e22482bc73c94c486a775e3f6e0254b476ff35c751232e6279b6e6985c"
    },
    {
      "filename": "model-00008-of-00016.safetensors",
      "tensors": 494,
      "bytes": 3031592856,
      "sha256": "89619c13de6ff8ffc5b572c9a16cf17c9057796d858f813c71c1571e6690eba5"
    },
    {
      "filename": "model-00009-of-00016.safetensors",
      "tensors": 495,
      "bytes": 3037884432,
      "sha256": "5c28a5d9523325a0e86b46ab71415eda57f95cc3cbdd2eed3fa90e433d9a6cc6"
    },
    {
      "filename": "model-00010-of-00016.safetensors",
      "tensors": 494,
      "bytes": 3031592856,
      "sha256": "5a32c9e8f7cbfbdfc5a30963f453e3151d894e088f9e495faf74227140e764e6"
    },
    {
      "filename": "model-00011-of-00016.safetensors",
      "tensors": 498,
      "bytes": 3030746656,
      "sha256": "5cf49bb65cd3af219676593ce60076622553ebcf23e3aab7e4951e091ff98ece"
    },
    {
      "filename": "model-00012-of-00016.safetensors",
      "tensors": 495,
      "bytes": 3038414312,
      "sha256": "93265a2b5a4342e2fd84979f1ed2777069746d004b7fb1482c495e94f24c5124"
    },
    {
      "filename": "model-00013-of-00016.safetensors",
      "tensors": 495,
      "bytes": 3037884440,
      "sha256": "a9f3c3235f354263bf7a16a8ae8a844f6c12d810631e57cc09a34ba4d689ef08"
    },
    {
      "filename": "model-00014-of-00016.safetensors",
      "tensors": 494,
      "bytes": 3031592856,
      "sha256": "d20d56044d0f5e1d489aab44674530a3e90bc53c5efe435ce95b814eab542deb"
    },
    {
      "filename": "model-00015-of-00016.safetensors",
      "tensors": 417,
      "bytes": 3181525720,
      "sha256": "6ee0fea7679c4a5fdb5cd960db36aba43dd45684cb57b86bfaf4112b1b76b93e"
    },
    {
      "filename": "model-00016-of-00016.safetensors",
      "tensors": 370,
      "bytes": 2890418256,
      "sha256": "bebb179946be3c57b0e98714fc9cff9e43ae0ca265731d9621c8c47d7ac156f4"
    }
  ],
  "runtime_validation": {
    "status": "PASS",
    "model_class": "Glm4MoeLiteForCausalLM",
    "model_dtype": "torch.bfloat16",
    "trust_remote_code": false,
    "experts": 48,
    "generated_tokens": 53,
    "failure_count": 0,
    "failures": []
  },
  "failure_count": 0,
  "failures": []
}

	{
	"status": "PASS",
	"stage": "lossless safetensors checkpoint reshard validation",
	"original_shards": 48,
	"output_shards": 16,
	"tensors": 7495,
	"tensor_equality_checked": 7495,
	"tensor_values_identical": true,
	"total_tensor_bytes": 48548430720,
	"index_sha256": "f14fbfde88c57bb0b073e145365d8cfcac5b36fc0b3c4be6e44d3c2d9c3f446c",
	"shards": [
	{
	"filename": "model-00001-of-00016.safetensors",
	"tensors": 273,
	"bytes": 3035883560,
	"sha256": "2f715596ea6eb95e091b4e8c34d30deeb85bf05d91c4c9cfcca37b36f1572a06"
	},
	{
	"filename": "model-00002-of-00016.safetensors",
	"tensors": 494,
	"bytes": 3031592368,
	"sha256": "170fc73ec42fd58ca2080a250074b33deade410bcbafee407d041166659ac5a7"
	},
	{
	"filename": "model-00003-of-00016.safetensors",
	"tensors": 494,
	"bytes": 3031592360,
	"sha256": "821621b83e3140870154efc65185f9f9d30f9db2ac4a082ae40974dd54ecf498"
	},
	{
	"filename": "model-00004-of-00016.safetensors",
	"tensors": 499,
	"bytes": 3037567928,
	"sha256": "2df6962eea9e2ff55ba76cf8849d4dc55bbf63d6616fdfaace9c7f49cc57d8e9"
	},
	{
	"filename": "model-00005-of-00016.safetensors",
	"tensors": 494,
	"bytes": 3031592856,
	"sha256": "f3a1660e46c3ad0da6abe29a2fd516d920900531f915f0e422a8c995ddf23902"
	},
	{
	"filename": "model-00006-of-00016.safetensors",
	"tensors": 494,
	"bytes": 3031592864,
	"sha256": "be324f5647fcab5a026695c37bf2188cb6e99b6ad7bf29875b85f990ccbaa304"
	},
	{
	"filename": "model-00007-of-00016.safetensors",
	"tensors": 495,
	"bytes": 3037884440,
	"sha256": "5e2da9e22482bc73c94c486a775e3f6e0254b476ff35c751232e6279b6e6985c"
	},
	{
	"filename": "model-00008-of-00016.safetensors",
	"tensors": 494,
	"bytes": 3031592856,
	"sha256": "89619c13de6ff8ffc5b572c9a16cf17c9057796d858f813c71c1571e6690eba5"
	},
	{
	"filename": "model-00009-of-00016.safetensors",
	"tensors": 495,
	"bytes": 3037884432,
	"sha256": "5c28a5d9523325a0e86b46ab71415eda57f95cc3cbdd2eed3fa90e433d9a6cc6"
	},
	{
	"filename": "model-00010-of-00016.safetensors",
	"tensors": 494,
	"bytes": 3031592856,
	"sha256": "5a32c9e8f7cbfbdfc5a30963f453e3151d894e088f9e495faf74227140e764e6"
	},
	{
	"filename": "model-00011-of-00016.safetensors",
	"tensors": 498,
	"bytes": 3030746656,
	"sha256": "5cf49bb65cd3af219676593ce60076622553ebcf23e3aab7e4951e091ff98ece"
	},
	{
	"filename": "model-00012-of-00016.safetensors",
	"tensors": 495,
	"bytes": 3038414312,
	"sha256": "93265a2b5a4342e2fd84979f1ed2777069746d004b7fb1482c495e94f24c5124"
	},
	{
	"filename": "model-00013-of-00016.safetensors",
	"tensors": 495,
	"bytes": 3037884440,
	"sha256": "a9f3c3235f354263bf7a16a8ae8a844f6c12d810631e57cc09a34ba4d689ef08"
	},
	{
	"filename": "model-00014-of-00016.safetensors",
	"tensors": 494,
	"bytes": 3031592856,
	"sha256": "d20d56044d0f5e1d489aab44674530a3e90bc53c5efe435ce95b814eab542deb"
	},
	{
	"filename": "model-00015-of-00016.safetensors",
	"tensors": 417,
	"bytes": 3181525720,
	"sha256": "6ee0fea7679c4a5fdb5cd960db36aba43dd45684cb57b86bfaf4112b1b76b93e"
	},
	{
	"filename": "model-00016-of-00016.safetensors",
	"tensors": 370,
	"bytes": 2890418256,
	"sha256": "bebb179946be3c57b0e98714fc9cff9e43ae0ca265731d9621c8c47d7ac156f4"
	}
	],
	"runtime_validation": {
	"status": "PASS",
	"model_class": "Glm4MoeLiteForCausalLM",
	"model_dtype": "torch.bfloat16",
	"trust_remote_code": false,
	"experts": 48,
	"generated_tokens": 53,
	"failure_count": 0,
	"failures": []
	},
	"failure_count": 0,
	"failures": []
	}