bupalinyu commited on
Commit
42be2c8
·
verified ·
1 Parent(s): 60d03b0

docs: how the 16 GiB VRAM budget is enforced; add --spec to the VRAM table

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -307,8 +307,7 @@ pipeline_tag: text-generation
307
  </details>
308
  <details open>
309
  <summary><b>Measurement conditions</b> · constrained budget and memory residency</summary>
310
- <p><code>16 GiB VRAM + 20 GiB RAM</code> rows are the RTX 5090 D 24 GB again, constrained by cgroup to a 16 GiB VRAM and
311
- 20 GiB system-RAM budget — not a different card. With residency on, the same card holds <strong>Q3_K</strong> (40.81 GiB of weights)
312
  as 23.14 GiB VRAM + 31.28 GiB RAM, and <strong>NVFP4</strong> (47.79 GiB) as 23.25 GiB VRAM + 38.27 GiB RAM — disjoint expert sets,
313
  nothing counted twice.</p>
314
  <p class="section-nav">See <a href="#usage-help">Usage &amp; troubleshooting</a> for memory-saving options.</p>
@@ -479,6 +478,7 @@ and `edge0-infer` read, so a change there applies to every launch.
479
  | `--kv-resident N` | Keep N KV cells per QSA layer in VRAM and stream the K/V from pinned host RAM, which frees VRAM for experts. | `0` (all in VRAM) |
480
  | `--mtp-q4 all\|proj\|head` | 4-bit copies of the draft layer's projections and/or head, about 0.3–0.5 GiB less VRAM. | off |
481
  | `--prefill CHUNK` | Prompt chunk size; bigger chunks need bigger device buffers. `auto` uses the largest chunk whose buffers the expert cache can lend. | `auto` |
 
482
 
483
  ### Host RAM
484
 
 
307
  </details>
308
  <details open>
309
  <summary><b>Measurement conditions</b> · constrained budget and memory residency</summary>
310
+ <p>The <code>16 GiB VRAM + 20 GiB RAM</code> rows are the same RTX 5090 D 24 GB, not a different card. VRAM is capped with <code>--vram-reserve-mib 8192</code>: 24,455 − 8,192 = 16,263 MiB left to the engine, and the expert cache is sized from what remains. System RAM is capped by a cgroup (<code>MemoryMax=20G</code>, no swap). A cgroup cannot cap VRAM; only the reserve can. With residency on, the same card holds <strong>Q3_K</strong> (40.81 GiB of weights)
 
311
  as 23.14 GiB VRAM + 31.28 GiB RAM, and <strong>NVFP4</strong> (47.79 GiB) as 23.25 GiB VRAM + 38.27 GiB RAM — disjoint expert sets,
312
  nothing counted twice.</p>
313
  <p class="section-nav">See <a href="#usage-help">Usage &amp; troubleshooting</a> for memory-saving options.</p>
 
478
  | `--kv-resident N` | Keep N KV cells per QSA layer in VRAM and stream the K/V from pinned host RAM, which frees VRAM for experts. | `0` (all in VRAM) |
479
  | `--mtp-q4 all\|proj\|head` | 4-bit copies of the draft layer's projections and/or head, about 0.3–0.5 GiB less VRAM. | off |
480
  | `--prefill CHUNK` | Prompt chunk size; bigger chunks need bigger device buffers. `auto` uses the largest chunk whose buffers the expert cache can lend. | `auto` |
481
+ | `--spec N` | The MTP drafter's verify window: how many tokens it proposes per check. A bigger window needs bigger verify buffers (about 180 MiB at 4) and pays off when every window has a fixed per-layer cost. The shipped packs use `4`; the macOS setup picks `2`, or `3` when the experts stream from SSD. Without an MTP layer there is no speculation. | `4` |
482
 
483
  ### Host RAM
484