bupalinyu commited on
Commit
581a3e4
·
verified ·
1 Parent(s): 82bdb5a

docs: drop residency-conditions sentence, add capacity controls section

Browse files
Files changed (1) hide show
  1. README.md +44 -1
README.md CHANGED
@@ -311,7 +311,7 @@ pipeline_tag: text-generation
311
  20 GiB system-RAM budget — not a different card. With residency on, the same card holds <strong>Q3_K</strong> (40.81 GiB of weights)
312
  as 23.14 GiB VRAM + 31.28 GiB RAM, and <strong>NVFP4</strong> (47.79 GiB) as 23.25 GiB VRAM + 38.27 GiB RAM — disjoint expert sets,
313
  nothing counted twice.</p>
314
- <p class="section-nav">These residency readings use expert residency enabled, <code>--no-prefill-borrow</code>, conv cache = 0 and kv-resident = 0. Host RAM is measured as <code>RssAnon + RssShmem</code>. These are single-run readings, not minimum system requirements. See <a href="#usage-help">Usage &amp; troubleshooting</a> for memory-saving options.</p>
315
  </details>
316
  <details class="demo-section" id="in-action" open="">
317
  <summary><h2 class="demo-title">In action</h2></summary>
@@ -454,6 +454,49 @@ and then serves OpenAI- and Anthropic-compatible endpoints:</p>
454
  <li><strong>Automatically enabled:</strong> <code>--mmap-experts</code>, <code>--resident-cpu-experts</code>, <code>--resident-pin</code>.</li>
455
  </ul>
456
  </details>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
457
  <h2 id="acknowledgements">Acknowledgements</h2>
458
  <details open>
459
  <summary><b>Credits &amp; license</b></summary>
 
311
  20 GiB system-RAM budget — not a different card. With residency on, the same card holds <strong>Q3_K</strong> (40.81 GiB of weights)
312
  as 23.14 GiB VRAM + 31.28 GiB RAM, and <strong>NVFP4</strong> (47.79 GiB) as 23.25 GiB VRAM + 38.27 GiB RAM — disjoint expert sets,
313
  nothing counted twice.</p>
314
+ <p class="section-nav">See <a href="#usage-help">Usage &amp; troubleshooting</a> for memory-saving options.</p>
315
  </details>
316
  <details class="demo-section" id="in-action" open="">
317
  <summary><h2 class="demo-title">In action</h2></summary>
 
454
  <li><strong>Automatically enabled:</strong> <code>--mmap-experts</code>, <code>--resident-cpu-experts</code>, <code>--resident-pin</code>.</li>
455
  </ul>
456
  </details>
457
+ ## Capacity controls · VRAM and host RAM
458
+
459
+ <a name="capacity-controls"></a>
460
+
461
+ All of these live in the pack's `model.json`, not on a command line: flags go in `args`, environment
462
+ variables in `env`, with paths relative to the pack directory. That is the same file the Edge0 Chat app
463
+ and `edge0-infer` read, so a change there applies to every launch.
464
+
465
+ ```json
466
+ "args": ["--expert-cache", "auto", "--kv-resident", "32768", "--vram-reserve-mib", "1024"],
467
+ "env": {"STRATA_LAT_FUSED_Q3K": "0", "STRATA_LATENT_DENSE_FMT": "8"}
468
+ ```
469
+
470
+ ### VRAM
471
+
472
+ | Flag | Meaning | Default |
473
+ |---|---|---|
474
+ | `--vram-reserve-mib N` | VRAM left free for the desktop, the driver and other processes; the engine sizes its caches from what remains. It means "leave this much free", not "give the engine this much", so the same number means different things on cards with different totals. Accepted but not listed by `--help`. | `700` |
475
+ | `--expert-cache N\|auto` | Expert slots kept in VRAM. `auto` derives the count from the free VRAM after the fixed footprint (skeleton projections ≈ 2.9 GiB, MTP draft ≈ 0.8 GiB, verify and draft-head buffers ≈ 0.45 GiB, plus the D/U device buffers). | `0` (none) |
476
+ | `--expert-cache-device1..3 N\|auto` | Use the second and third GPU as further expert-cache tiers. | off |
477
+ | `--kv-resident N` | Keep N KV cells per QSA layer in VRAM and stream the K/V from pinned host RAM, which frees VRAM for experts. | `0` (all in VRAM) |
478
+ | `--mtp-q4 all\|proj\|head` | 4-bit copies of the draft layer's projections and/or head, about 0.3–0.5 GiB less VRAM. | off |
479
+ | `--prefill CHUNK` | Prompt chunk size; bigger chunks need bigger device buffers. `auto` uses the largest chunk whose buffers the expert cache can lend. | `auto` |
480
+
481
+ ### Host RAM
482
+
483
+ | Flag | Meaning | Default |
484
+ |---|---|---|
485
+ | `--memory-limit-mib N` | The RAM this engine's container or cgroup may use, for when the container cannot see its own limit. | `0` (detect) |
486
+ | `--resident-budget-gib N` | Cap on the host-RAM expert cache. It holds only the experts the VRAM cache does **not** hold, so the two sets are disjoint, and the value is clamped to the RAM actually available minus headroom. Requires a static `--expert-profile`. | `0` (off) |
487
+ | `--resident-experts` | The same with the whole complement as the budget. | off |
488
+ | `--mmap-experts` | Read experts through the pack file on demand instead of pinning them, so nothing is resident in RAM. | off |
489
+ | `--conversation-cache-mib N` | Conversation cache. `0` turns it off. | adaptive |
490
+ | `--ple-io direct\|mmap\|ram` | The PLE table on its SSD: `direct` reads it unbuffered and it never enters memory; `ram` locks the whole table in RAM. | `direct` |
491
+ | `--kv-resident N` | The streamed K/V lives in pinned host RAM (1.55 GiB at 32768 cells). | — |
492
+
493
+ ### A 16 GiB card, as a worked example
494
+
495
+ `--vram-reserve-mib 1024` (or `2048` with a display attached), an explicit `--expert-cache` instead of
496
+ `auto` (about 4000 slots for the 145B NVFP4 experts, 1.8 MB each), and `--kv-resident 32768`. Then read
497
+ the engine line `NNNN MiB of VRAM free with everything loaded`: 300–800 MiB free is right; below 300 MiB
498
+ the slot count is too high and the next allocation fails.
499
+
500
  <h2 id="acknowledgements">Acknowledgements</h2>
501
  <details open>
502
  <summary><b>Credits &amp; license</b></summary>