Instructions to use happyme531/MiniCPM5-2B-W4A8-Mixed-QQQ-Rot-RK3588 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use happyme531/MiniCPM5-2B-W4A8-Mixed-QQQ-Rot-RK3588 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="happyme531/MiniCPM5-2B-W4A8-Mixed-QQQ-Rot-RK3588")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("happyme531/MiniCPM5-2B-W4A8-Mixed-QQQ-Rot-RK3588")
model = AutoModelForCausalLM.from_pretrained("happyme531/MiniCPM5-2B-W4A8-Mixed-QQQ-Rot-RK3588", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use happyme531/MiniCPM5-2B-W4A8-Mixed-QQQ-Rot-RK3588 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "happyme531/MiniCPM5-2B-W4A8-Mixed-QQQ-Rot-RK3588"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "happyme531/MiniCPM5-2B-W4A8-Mixed-QQQ-Rot-RK3588",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/happyme531/MiniCPM5-2B-W4A8-Mixed-QQQ-Rot-RK3588

SGLang

How to use happyme531/MiniCPM5-2B-W4A8-Mixed-QQQ-Rot-RK3588 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "happyme531/MiniCPM5-2B-W4A8-Mixed-QQQ-Rot-RK3588" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "happyme531/MiniCPM5-2B-W4A8-Mixed-QQQ-Rot-RK3588",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "happyme531/MiniCPM5-2B-W4A8-Mixed-QQQ-Rot-RK3588" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "happyme531/MiniCPM5-2B-W4A8-Mixed-QQQ-Rot-RK3588",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use happyme531/MiniCPM5-2B-W4A8-Mixed-QQQ-Rot-RK3588 with Docker Model Runner:
```
docker model run hf.co/happyme531/MiniCPM5-2B-W4A8-Mixed-QQQ-Rot-RK3588
```

MiniCPM5-2B-W4A8-Mixed-QQQ-Rot-RK3588

RK3588 NPU: Prepared for open_rknpu2_lm_infer, an open-source inference engine for RK3588. This is a compressed-tensors safetensors checkpoint, not an RKLLM-format model.

RK3588 NPU： 本模型面向开源推理引擎 open_rknpu2_lm_infer。模型采用 compressed-tensors safetensors 格式，不是 RKLLM 格式。

Quick start / 快速开始

On an RK3588 Linux device, install the engine using the project instructions, then download and run:

在 RK3588 Linux 设备上按项目说明安装引擎，然后下载并运行：

hf download happyme531/MiniCPM5-2B-W4A8-Mixed-QQQ-Rot-RK3588 --local-dir models/MiniCPM5-2B-W4A8-Mixed-QQQ-Rot-RK3588
./bin/open_rknpu2_lm_infer --model models/MiniCPM5-2B-W4A8-Mixed-QQQ-Rot-RK3588 chat

Other inference engines / 其他推理引擎

vLLM and SGLang support compressed-tensors in some configurations. You can try this checkpoint with compatible releases and hardware using vllm serve happyme531/MiniCPM5-2B-W4A8-Mixed-QQQ-Rot-RK3588 or python -m sglang.launch_server --model-path happyme531/MiniCPM5-2B-W4A8-Mixed-QQQ-Rot-RK3588. Support for this exact mixed INT4/INT8 configuration, model architecture, and hardware varies by engine version and backend; execution and accuracy have not been verified here. See the model-specific limitations below.

vLLM 和 SGLang 在部分配置下支持 compressed-tensors，可在兼容的版本和硬件上尝试上述命令。具体能否运行取决于引擎版本、后端、模型架构及此混合 INT4/INT8 配置；本模型在这两个引擎上的运行和精度尚未在此验证。模型特定限制见下文。

English · 中文

English

Overview

A post-training quantized derivative of openbmb/MiniCPM5-2B, using fixed offline rotation + QQQ, mixed W4/W8 per-channel weights, and dynamic per-token A8. The model retains the original architecture and native chat template. The tensor payload is 2.104 GiB. 53.33% of decoder Linear weight elements are W4; lm_head and the listed sensitive projections are W8.

Property	Value
Architecture	`LlamaForCausalLM`
Original source	ModelScope OpenBMB/MiniCPM5-2B
ModelScope source revision	`289c5dc6f6a855f0a0434e3bbe6b72e043abf005`
Quantization	fixed offline rotation + QQQ
Weight format	Symmetric per-output-channel W4/W8, FP32 scales
Activation calibration	Dynamic per-token INT8, `maxabs/127`
Embedding / remaining parameters	BF16
Quantized Linear layers	84 W4 / 211 W8
Checkpoint format	compressed-tensors `pack-quantized`, safetensors
QQQ settings	group size `-1`, block size `128`, damping `0.01`, `actorder=False`

W8 projections: lm_head; q/k/v/o projections and MLP down projections in all 42 layers. MLP gate/up projections remain W4.

Results against the original model

Weights	WikiText-2 PPL ↓	CMMLU fixed 201 ↑	GSM8K fixed 100 ↑	GSM8K truncations ↓
Original unquantized weights	14.624683	133/201 (66.17%)	91/100	3
MiniCPM5-2B-W4A8-Mixed-QQQ-Rot-RK3588	15.083971	127/201 (63.18%)	92/100	2

PPL increases by 3.14% relative to the original weights. Results use the same native tokenizer and evaluation protocol within each model family. Original BF16 weights are evaluated with FP32 model arithmetic. Quantized Linear layers use an INT8×INT8→INT32 reference GEMM, maxabs/127 activation quantization, and FP32 nonlinear operations; NLL reduction uses FP64.

WikiText-2 uses the original wiki.test.raw, without added special tokens, in complete 512-token windows. Targets at zero-based positions 257–511 are scored: 559 windows and 142,545 targets. CMMLU uses native chat prompts and selects the highest next-token logit among A/B/C/D. GSM8K uses greedy decoding with enable_thinking=False, a 512-new-token budget, and numeric exact-match scoring; truncated answers count as incorrect. The math prompt asks for brief reasoning and a final #### <number>. Sample IDs and protocol details are recorded in evaluation.json.

Calibration and reproduction

The base pool contains 256 complete conversations: 128 human-verified Chinese COIG-CQIA examples, 64 GSM8K training problems with complete solutions, and 64 SmolTalk English conversations. The SmolTalk pool contains 48 smol-magpie-ultra good/excellent examples and 16 smol-constraints examples. Native chat rendering preserves complete assistant answers and end markers, with a 1024-token maximum and no answer cropping.

This checkpoint uses 416 calibration observations: the 256 conversations, two additional copies of the same 64 math examples, and 32 original WikiText-2 training chunks of 512 tokens each. There are 64 distinct math problems, each with weight three. Total calibration tokens: 133,315. The signed residual Hadamard rotation uses hidden size 2048 and seed 42; the value-head Hadamard dimension is 128. Rotations are folded offline into the weights, with no online transform. FP32 logit equivalence before BF16 storage passed.

Bit widths were selected on separate WikiText validation windows, CMMLU dev335, and GSM8K train32. Calibration excludes these samples and the final CMMLU201 and GSM8K100 samples. Final subsets had also been evaluated for earlier checkpoints; the present selection uses the separate development sets.

quantization/ contains standalone quantization tools, pinned dependencies, the exact W8 mask, and ordered calibration source references. Follow quantization/README.md to reproduce the recipe or quantize your own unquantized model with your own calibration data and W8 mask.

Compatibility and limitations

Standard Transformers/compressed-tensors full loading, forward execution, and 8-token cached generation passed verification. Packed weight coverage, positive FP32 scales, W8 source statistics, INT64 reference checks, and independent NLL and task-score reconciliation passed.

The table reports the project INT32 reference runtime. Standard compressed-tensors uses its default activation scaling; complete benchmark parity and throughput with production engines have not been measured. The mixed W4/W8 channel format is distinct from groupwise GPTQ, AWQ, GGUF, and QQQ's native kernel checkpoint. The fixed task subsets are not full benchmark results. Long-context quality, tool use, and coding after quantization have not been evaluated. Quantization reproduction across different GPU architectures can introduce floating-point differences; bitwise-identical regeneration is not claimed.

License and attribution

Weights and the original quantization tools are released under Apache-2.0. The base model is developed by OpenBMB / ModelBest. This checkpoint is an independent quantized derivative. QQQ solver code is retrieved from HandH1998/QQQ at the revision documented in the quantization tools and retains its upstream terms. Calibration data remains subject to the source datasets' terms. See LICENSE and NOTICE.

中文

模型说明

本模型基于 openbmb/MiniCPM5-2B，采用 固定离线旋转 + QQQ 进行训练后量化。权重使用逐输出通道对称 W4/W8 混合位宽，激活使用逐 token 动态 A8，权重 scale 为 FP32。保留原始架构和原生 chat 模板， embedding 及其他未量化参数保持 BF16。

量化 tensor 体积为 2.104 GiB，decoder Linear 权重元素中 53.33% 为 W4，共 84 个 W4 / 211 个 W8 Linear。 lm_head 为 W8A8。Checkpoint 使用 compressed-tensors pack-quantized 与 safetensors。原始模型从 ModelScope OpenBMB/MiniCPM5-2B 下载，版本为 289c5dc6f6a855f0a0434e3bbe6b72e043abf005。QQQ 参数为逐通道 group_size=-1、block_size=128、 dampening_frac=0.01、actorder=False。

W8 范围：lm_head；全部 42 层的 q/k/v/o 投影和 MLP down 投影为 W8，MLP gate/up 投影为 W4。

与未量化模型的对比

权重	WikiText-2 PPL ↓	CMMLU 固定201题 ↑	GSM8K 固定100题 ↑	数学截断数 ↓
原始未量化权重	14.624683	133/201（66.17%）	91/100	3
本量化版	15.083971	127/201（63.18%）	92/100	2

相对原始权重，PPL 上升 **3.14%**。同模型系列使用相同 tokenizer 和评测协议。原始 BF16 权重按 FP32 模型计算评测；量化 Linear 使用 INT8×INT8→INT32 参考 GEMM、maxabs/127 动态激活量化，非线性操作为 FP32，NLL 归约为 FP64。

PPL 使用原始 WikiText-2 wiki.test.raw，不自动添加特殊 token；按完整 512-token 窗口切分，对零起始位置 257–511 的 target 计分，共 559 个窗口、 142,545 个 target。中文选择题采用原生 chat 提示，在下一个 token 的 A/B/C/D logit 中取最大值。数学采用 greedy 解码，enable_thinking=False，最多生成 512 个 token，按数值 exact match 计分，截断计错。数学提示要求简短推理并以 #### <number> 结束。evaluation.json 记录样本 ID 和协议。

校准数据与复现

基础校准池为 256 条完整对话：中文 COIG-CQIA 人工核验样本 128 条、GSM8K train 完整解题 64 条、SmolTalk 英文对话 64 条。英文包括 48 条 smol-magpie-ultra good/excellent 样本和 16 条 smol-constraints 样本。使用各自原生 chat 模板，保留完整回答和结束标记，最长 1024 token，不截断回答。

本模型实际使用 416 条加权校准观测：基础 256 条对话，加上相同 64 道数学题的两次额外重复，以及 32 段 512-token WikiText-2 训练文本。数学共 64 道独立题，每题统计权重为 3；校准总 token 数为 133,315。残差使用 hidden size 2048、seed 42 的带符号 Hadamard 旋转，value-head Hadamard 维度为 128；旋转离线融合到权重，无在线变换。保存为 BF16 前的 FP32 logit 等价性验证通过。

位宽选择使用独立 WikiText validation、CMMLU dev335 和 GSM8K train32。校准与这些开发样本、最终 CMMLU201/GSM8K100 分离。最终子集曾用于此前版本评测；本轮位宽选择采用独立开发集。

quantization/ 包含独立量化脚本、固定依赖版本、精确 W8 清单及按顺序记录的校准数据来源。参见 quantization/README.md，可以复现本配方，也可以使用自己的未量化模型、校准数据和敏感层清单进行量化。

兼容性与局限

标准 Transformers/compressed-tensors 完整加载、forward 和 8-token 缓存生成验证通过；packed 权重覆盖、FP32 正 scale、W8 源权重统计、INT64 整数对照、独立 NLL 与任务分数复算均通过。

表中精度来自项目 INT32 参考运行时。标准 compressed-tensors 使用库默认激活缩放，其完整评测精度与生产推理框架吞吐尚未测量。逐通道 W4/W8 混合格式与 groupwise GPTQ、AWQ、GGUF 及 QQQ 原生 kernel checkpoint 不同。中文和数学是固定子集结果。量化后的长上下文、工具调用和代码能力尚未评测。不同 GPU 架构重新量化可能产生浮点差异，不承诺逐 bit 相同。

许可与致谢

模型权重和本仓库原创量化工具采用 Apache-2.0 许可。原模型由 OpenBMB / ModelBest 开发，本仓库为独立量化衍生版本。QQQ 求解器从固定版本的上游仓库下载，遵循其上游条款；校准数据遵循各原始数据集条款。参见 LICENSE 和 NOTICE。

Downloads last month: 133

Safetensors

Model size

3B params

Tensor type

I32

BF16

Model tree for happyme531/MiniCPM5-2B-W4A8-Mixed-QQQ-Rot-RK3588

Base model

openbmb/MiniCPM5-2B

Quantized

(128)

this model

happyme531
/

MiniCPM5-2B-W4A8-Mixed-QQQ-Rot-RK3588