File size: 1,853 Bytes
8795749 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 | # MLX runtime setup
Apple silicon. The measurement used Python3.12, mlx0.32.0, mlx-lm0.31.3, mlx-vlm0.6.5 and transformers5.14.1. This is a patched research runtime, not stock Hugging Face Transformers inference. The model is a draft, not a standalone chat model.
```bash
uv venv --python 3.12 .venv-humanizer
source .venv-humanizer/bin/activate
uv pip install mlx==0.32.0 mlx-lm==0.31.3 mlx-vlm==0.6.5 transformers==5.14.1 huggingface-hub==1.19.0 safetensors==0.8.0
hf download jayyun98/Humanizer-12B-Draftfit-DFlash mlx-runtime.patch run_mlx.py --local-dir ./humanizer-draft
git clone https://github.com/ARahim3/mlx-dspark.git ./mlx-dspark-humanizer
git -C ./mlx-dspark-humanizer checkout 414309238e0f0f2a97315bcf0157e9f20fffebbc
git -C ./mlx-dspark-humanizer apply ../humanizer-draft/mlx-runtime.patch
uv pip install --no-deps -e ./mlx-dspark-humanizer
python ./humanizer-draft/run_mlx.py --text "The meeting has been moved to Friday." --check
```
The dependency versions preserve the measured Gemma4 runtime, even though the runtime project's newer minimum covers unrelated targets. The measurement and runner were tested in the existing environment; this fresh-install recipe has not been rerun in a clean environment.
The runner downloads the pinned mixed4/8bit Humanizer target (~9GB), then the draft. It uses completion formatting, FP32 floating tensors,4bit draft quantization, greedy generation, cap3 and no lookup drafting. `--check` also runs target-only and fails on any token mismatch. The default64 output-token cap is a short benchmark limit; increase `--max-tokens` for longer rewrites. Longer outputs and sampling were not part of this benchmark.
## vLLM and SGLang
The exported checkpoint has not been verified with either CUDA serving engine. No launch command is provided as a working claim. Use the tested MLX runtime above.
|