7 days ago
-
EPM: expert paging CPU inference for MiniCPM-V-4.7-35B-A3B Q4_K_M
Port the 8 GiB expert pager to openbmb/MiniCPM-V-4.7-35B-A3B-Q4_K_M.gguf
(19.71 GiB, qwen35moe, 40 layers, 256 experts, top-8).
Measured under an 8 GiB / no-swap ceiling enforced by tools/strict8g.py:
5.83 tok/s median (N=9), peak anon RSS 7799 MiB, VmSwap 0, 9/9 strict_ok.
Unmodified llama.cpp cannot load the model at that ceiling.
Correctness: 3-layer verification passes; 5 runs x 48 tokens x 248144 vocab
logits bit-identical.
Port-specific changes: GGUF BOOL-array header fix, calibrated SDQ_RESERVE_MB,
two expert size classes, hybrid SSM/attention thread behaviour.
