Paris99 commited on
Commit
6e2cc37
·
verified ·
1 Parent(s): cf3da60

Card: builds table and memory guidance

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -115,7 +115,7 @@ On a MacBook Pro M5 Max (128 GB), 21 November 2026, with MLX 0.32.4 and [inkling
115
  - **Memory:** peak 17.2 GB with a 4K-token prompt and 18.2 GB with an 8K-token prompt (mlx-vlm defaults). With a smaller prefill step the peak drops: 12.8 / 13.1 GB at `prefill_step_size=512` and 11.6 / 11.9 GB at `prefill_step_size=256` (4K / 8K-token prompts, plus 0.64 GB if the DSpark draft is loaded), which fits the default GPU memory limit of a 24 GB Mac: pass `prefill_step_size=256` to `generate`, or `--prefill-step-size 256` to `mlx_vlm.server`.
116
  - **DSpark:** the bundled draft loads with this build and drafts about 7 tokens per round.
117
 
118
- Running on a 16 GB Mac: this is the smallest build, but even with a small prefill step it needs about 11.1-11.9 GB (4K-8K-token prompt, draft not loaded), a little above the default GPU memory limit of a 16 GB Mac (about 10.7 GB). Raise the limit until the next restart with `sudo sysctl iogpu.wired_limit_mb=12288`, which leaves about 4 GB for macOS; close other apps, do not load the DSpark draft, pass `prefill_step_size=128` (or 256) to `generate` or `--prefill-step-size 128` to `mlx_vlm.server`, and keep prompts within about 8K tokens. Measured peaks without the draft: 4K-token prompt 11.1 GB (step 128) / 11.6 GB (256) / 12.8 GB (512); 8K-token prompt 11.3 / 11.9 / 13.1 GB. On a 24 GB Mac it runs comfortably at mlx-vlm's defaults.
119
 
120
  Known limitation of the current server: when the model calls a tool, `content` can also carry a short sentence of its
121
  reasoning and the tool name.
 
115
  - **Memory:** peak 17.2 GB with a 4K-token prompt and 18.2 GB with an 8K-token prompt (mlx-vlm defaults). With a smaller prefill step the peak drops: 12.8 / 13.1 GB at `prefill_step_size=512` and 11.6 / 11.9 GB at `prefill_step_size=256` (4K / 8K-token prompts, plus 0.64 GB if the DSpark draft is loaded), which fits the default GPU memory limit of a 24 GB Mac: pass `prefill_step_size=256` to `generate`, or `--prefill-step-size 256` to `mlx_vlm.server`.
116
  - **DSpark:** the bundled draft loads with this build and drafts about 7 tokens per round.
117
 
118
+ Running on a 16 GB Mac: this is the smallest build, but even with a small prefill step it needs about 11.1-11.9 GB (4K-8K-token prompt, draft not loaded), a little above the default GPU memory limit of a 16 GB Mac (about 10.7 GB). Raise the limit until the next restart with `sudo sysctl iogpu.wired_limit_mb=12288`, which leaves about 4 GB for macOS; close other apps, do not load the DSpark draft, pass `prefill_step_size=128` (or 256) to `generate` or `--prefill-step-size 128` to `mlx_vlm.server`, and keep prompts within about 8K tokens. Measured peaks without the draft: 4K-token prompt 11.1 GB (step 128) / 11.6 GB (256) / 12.8 GB (512); 8K-token prompt 11.3 / 11.9 / 13.1 GB. On a 24 GB Mac it fits the default limit with `prefill_step_size=512` or lower, with the draft loaded.
119
 
120
  Known limitation of the current server: when the model calls a tool, `content` can also carry a short sentence of its
121
  reasoning and the tool name.