File size: 2,568 Bytes
c4f3e2d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 | ---
license: apache-2.0
base_model: tencent/Hy3
base_model_relation: quantized
tags:
- mlx
- hunyuan
- hy3
- moe
pipeline_tag: text-generation
library_name: mlx
---
# Hy3-4bit (MLX)
A clean 4-bit MLX quantization of [tencent/Hy3](https://huggingface.co/tencent/Hy3) (Hunyuan 3.0, Apache-2.0), converted with `mlx-lm` and verified running on Apple Silicon.
Hy3 is a 295B-parameter Mixture-of-Experts model with **21B active parameters**, 80 layers, 192 routed experts (top-8) plus one shared expert, a native MTP (multi-token-prediction) head, and 256K context.
## Why this quant is clean
Naive uniform 4-bit quantization degrades MoE models badly, because the router that picks which experts fire is precision-sensitive. This build follows a mixed-precision recipe:
- **4-bit** (group size 64, affine) for attention and expert weights
- **8-bit** for every MoE router gate (`*.mlp.router.gate`)
- MTP head preserved
That router protection is the difference between a coherent model and mush, and it matches the recipe used by the reference `mlx-community/Hy3-preview-4bit`.
## Footprint
- Weights: ~166 GB
- Fits: two 128GB Apple Silicon machines (or one 192GB+ Mac), does not fit a single 128GB machine at 4-bit.
## How it was made
```python
from mlx_lm.convert import convert
def hy3_predicate(path, module, config=None):
if path.endswith("mlp.router.gate"):
return {"group_size": 64, "bits": 8}
return {"group_size": 64, "bits": 4}
convert(
hf_path="tencent/Hy3",
mlx_path="Hy3-4bit",
quantize=True, q_bits=4, q_group_size=64, q_mode="affine",
quant_predicate=hy3_predicate,
)
```
`hy_v3` architecture support comes from [mlx-lm PR #1211](https://github.com/ml-explore/mlx-lm/pull/1211).
## Benchmarks (2x M5 Max, Thunderbolt RDMA)
Measured on two M5 Max (128GB each), pipeline-parallel over Thunderbolt with Apple's `jaccl` RDMA backend (`MLX_METAL_FAST_SYNCH=1`). Single-stream decode.
| Metric | Value |
|---|---|
| Decode (generation) | **36.91 tok/s** |
| Prompt (prefill) | 8.6 tok/s |
| Peak memory / node | 84.4 GB |
| Backend | jaccl (RDMA), pipeline-parallel |
Notes: at 4-bit (~166GB) the model does not fit a single 128GB machine, so this is a genuine 2-node cluster run. The `ring`/TCP backend trips the macOS Metal command-buffer watchdog on the cross-node fence wait; RDMA (`jaccl`) is required for stable single-stream decode.
## Credits
- Base model: Tencent Hunyuan (`tencent/Hy3`, Apache-2.0)
- Architecture support: mlx-lm PR #1211
- Quantization + Apple Silicon cluster benchmark: bicVanYonk
|