| Hy3 (hy_v3) for llama.cpp — macOS Metal support |
| Patches and binaries to run Tencent Hy3 (hy_v3) 295B MoE on llama.cpp with Apple Silicon GPU (Metal). |
|
|
| Why this project exists |
| llama.cpp supports dozens of architectures, but Hy3 (hy_v3) wasn't one of them. Tencent released Hy3 as open-source (Apache 2.0), AngelSlim quantized it to GGUF, but to run it on a Mac someone had to write the missing piece: hy_v3 architecture support in llama.cpp. |
|
|
| This repo contains the patches that add hy_v3 to llama.cpp — architecture detection, weight loading, MoE + shared expert forward pass, and MTP self-speculative decoding. All compiled with Metal for Apple Silicon GPU. |
| |
| The goal is simple: run a 295B model on a MacBook. Not on the cloud, not on a cluster. On a laptop. |
| |
| Credits |
| This starts and ends with AngelSlim and their work on HuggingFace: |
| |
| AngelSlim/Hy3-GGUF — GGUF-quantized model (IQ1_M and Q4_K_M), mixed-precision recipes, importance matrix, setup script, benchmarks, chat template. Without this work, this project wouldn't exist. |
|
|
| AngelSlim provided: |
|
|
| The base patches for hy_v3 architecture in llama.cpp |
| The IQ1_M quantization with mixed recipe (critical weights in Q8_0/Q6_K, experts in IQ1_M/IQ2_XXS) |
| The importance matrix to allocate bits where they matter |
| The chat template for tool calling and reasoning |
| This repo takes those patches, applies them to llama.cpp, and builds them with Metal for macOS. |
|
|
| Thank you AngelSlim. 🙌 |
|
|
| The model |
| Detail Value |
| Architecture Hy3 (hy_v3) — Hunyuan V3 |
| Developed by Tencent |
| Parameters 295B |
| Layers 81 (80 routed + 1 MTP) |
| Experts 192 (8 active per token) |
| Gating Sigmoid + correction bias + top-8 selection |
| Quantization IQ1_M (AngelSlim mixed recipe) |
| File size ~85 GB (with MTP) |
| Original HF model: Tencent/Hy3 AngelSlim GGUF quant: AngelSlim/Hy3-GGUF Download: Hy3-IQ1_M-mtp.gguf (85 GB, IQ1_M with MTP) |
|
|
| IQ1_M vs BF16 quality loss: ~+0.3% PPL — imperceptible. Full benchmarks on AngelSlim's HF page, file assets/benchmark.png. |
| |
| Why a MacBook? |
| Mac RAM IQ1_M (85 GB) MTP Context Notes |
| M5 Max 128 GB ✅ ✅ 64K Everything on, comfortable |
| M4 Max 128 GB ✅ ✅ 64K Everything on |
| M3 Max 128 GB ✅ ✅ 64K Everything on |
| MacBook Pro 128 GB ✅ ✅ 64K Runs on a laptop |
| Mac Studio 96 GB ✅ ❌ 64K KV q8_0 only, no MTP |
| MacBook Pro 96 GB ✅ ❌ 64K Same as above |
| A 295B MoE running on a MacBook with 128 GB is a concrete milestone for local AI: |
| |
| No cloud, no API keys, no subscriptions |
| Total privacy — data never leaves your machine |
| No dedicated GPU needed — Apple Silicon unified memory is enough |
| Portable — no server rack, no cluster |
| With 96 GB it still works: the model is ~85 GB, leaving ~11 GB for the system. Just compress the KV cache (-ctk q8_0 -ctv q8_0) and skip MTP (which adds ~2 GB of weights plus a draft KV cache). With 128 GB everything runs — MTP included, with headroom. |
| |
| Download |
| 1. The GGUF model |
| # IQ1_M with MTP (85 GB) — recommended for 128 GB |
| wget https://huggingface.co/AngelSlim/Hy3-GGUF/resolve/main/Hy3-IQ1_M-mtp.gguf |
| |
| # IQ1_M without MTP (84 GB) — for 96 GB |
| wget https://huggingface.co/AngelSlim/Hy3-GGUF/resolve/main/Hy3-IQ1_M.gguf |
| 2. The code |
| git clone https://github.com/ggml-org/llama.cpp |
| cd llama.cpp |
| git checkout 19bba67c1 |
| git apply /path/to/0001-add-hyv3-support.patch |
| cp /path/to/hyv3.cpp src/models/ |
| Or clone the ready-made fork: |
| |
| git clone https://github.com/<your-username>/llama.cpp-hyv3 |
| cd llama.cpp-hyv3 |
| 3. Build |
| mkdir build && cd build |
| cmake .. -DLLAMA_METAL=ON |
| make -j$(sysctl -n hw.logicalcpu) |
| Commands |
| CLI (base inference) |
| ./build/bin/llama-cli \ |
| -m ~/Downloads/Hy3-IQ1_M-mtp.gguf \ |
| -c 65536 \ |
| -ngl 99 \ |
| -fa on \ |
| -ctk q8_0 -ctv q8_0 \ |
| -p "Hello" \ |
| -n 100 \ |
| --temp 0.6 |
| CLI (with reasoning/thinking) |
| ./build/bin/llama-cli \ |
| -m ~/Downloads/Hy3-IQ1_M-mtp.gguf \ |
| -c 65536 \ |
| -ngl 99 -fa on \ |
| -ctk q8_0 -ctv q8_0 \ |
| -p "Hello" \ |
| -n 200 \ |
| --temp 0.6 \ |
| --reasoning on \ |
| --reasoning-budget -1 |
| |
| Server (OpenAI-compatible API) |
| ./build/bin/llama-server \ |
| -m ~/Downloads/Hy3-IQ1_M-mtp.gguf \ |
| -c 65536 \ |
| -ngl 99 -fa on \ |
| -ctk q8_0 -ctv q8_0 \ |
| --temp 0.6 \ |
| --port 8080 |
| Test API call: |
| |
| curl http://localhost:8080/v1/chat/completions \ |
| -H "Content-Type: application/json" \ |
| -d '{ |
| "model": "Hy3-IQ1_M-mtp", |
| "messages": [{"role": "user", "content": "Hello"}], |
| "temperature": 0.6, |
| "max_tokens": 200 |
| }' |
| |
| For Mac with 96 GB RAM |
| # No MTP, compressed KV cache |
| ./build/bin/llama-cli \ |
| -m ~/Downloads/Hy3-IQ1_M.gguf \ |
| -c 65536 \ |
| -ngl 99 -fa on \ |
| -ctk q8_0 -ctv q8_0 \ |
| -p "Hello" -n 100 \ |
| --temp 0.6 |
| Flag reference |
| Flag What it does |
| -m PATH Path to the GGUF model file |
| -c N Context size in tokens. 65536 = 64K. Higher = more memory |
| -ngl N Layers to offload to GPU. 99 = all layers on Metal |
| -fa on Flash attention — reduces memory and speeds up attention |
| -ctk q8_0 -ctv q8_0 KV cache in q8_0. Essential for 96 GB (saves ~20 GB) |
| --temp N Sampling temperature. 0.0 = deterministic/greedy |
| --reasoning on Enable thinking/reasoning (tag) |
| --reasoning-budget N Max tokens for thinking. -1 = unlimited |
| --spec-type draft-mtp MTP self-speculative decoding (*-mtp.gguf only) |
| --spec-draft-n-max N Max draft tokens per MTP step |
| Project structure |
| ├── 0001-add-hyv3-support.patch # Patch for 9 llama.cpp files (383 lines) |
| ├── src/models/hyv3.cpp # hy_v3 model implementation + MTP (388 lines) |
| └── README.md # This file |
| Modified files in llama.cpp |
| File Change |
| src/llama-arch.h New enum LLM_ARCH_HYV3 |
| src/llama-arch.cpp Architecture name hy_v3 |
| src/llama-model.cpp Model mapping + Neox rope type |
| src/models/models.h llama_model_hyv3 class declaration |
| src/models/hyv3.cpp New — load, forward, MTP draft head |
| gguf-py/gguf/constants.py Arch enum + tensor list (28 hy_v3 tensors) |
| gguf-py/gguf/tensor_mapping.py MTP tensor name mapping |
| conversion/__init__.py HF → GGUF model name mapping |
| common/chat.cpp Chat template parser (tool calls + reasoning) |
| License |
| Apache 2.0. Same as the original Tencent/Hy3 model and AngelSlim's patches. |
| |
| Long live open local AI. A 295B model running on a MacBook. 🎉 |
|
|
