chua commited on
Commit
e5ea547
·
verified ·
1 Parent(s): 6aab5f4

Add compact memory usage help and measurement conditions

Browse files

Append usage help after Quick start and clarify residency measurement settings. Preserve every existing README byte, asset and repository setting.

Files changed (1) hide show
  1. README.md +31 -0
README.md CHANGED
@@ -311,6 +311,7 @@ pipeline_tag: text-generation
311
  20 GiB system-RAM budget — not a different card. With residency on, the same card holds <strong>Q3_K</strong> (40.81 GiB of weights)
312
  as 23.14 GiB VRAM + 31.28 GiB RAM, and <strong>NVFP4</strong> (47.79 GiB) as 23.25 GiB VRAM + 38.27 GiB RAM — disjoint expert sets,
313
  nothing counted twice.</p>
 
314
  </details>
315
  <details class="demo-section" id="in-action" open="">
316
  <summary><h2 class="demo-title">In action</h2></summary>
@@ -419,6 +420,36 @@ and then serves OpenAI- and Anthropic-compatible endpoints:</p>
419
  <p>⬇️ <strong>Download:</strong> <strong><a href="https://github.com/Edge0-AI/Edge0/releases">GitHub Releases</a></strong></p>
420
  <p>Open <strong>Edge0 Chat</strong> on Windows and choose a model folder, such as <code>Edge0-145B-NVFP4</code>, to run the model locally. To build the app from source, follow the <a href="https://github.com/Edge0-AI/Edge0/blob/main/edge0-infer/examples/desktop/README.md">Edge0 Chat</a> guide.</p>
421
  </details>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
422
  <h2 id="acknowledgements">Acknowledgements</h2>
423
  <details open>
424
  <summary><b>Credits &amp; license</b></summary>
 
311
  20 GiB system-RAM budget — not a different card. With residency on, the same card holds <strong>Q3_K</strong> (40.81 GiB of weights)
312
  as 23.14 GiB VRAM + 31.28 GiB RAM, and <strong>NVFP4</strong> (47.79 GiB) as 23.25 GiB VRAM + 38.27 GiB RAM — disjoint expert sets,
313
  nothing counted twice.</p>
314
+ <p class="section-nav">These residency readings use expert residency enabled, <code>--no-prefill-borrow</code>, conv cache = 0 and kv-resident = 0. Host RAM is measured as <code>RssAnon + RssShmem</code>. These are single-run readings, not minimum system requirements. See <a href="#usage-help">Usage &amp; troubleshooting</a> for memory-saving options.</p>
315
  </details>
316
  <details class="demo-section" id="in-action" open="">
317
  <summary><h2 class="demo-title">In action</h2></summary>
 
420
  <p>⬇️ <strong>Download:</strong> <strong><a href="https://github.com/Edge0-AI/Edge0/releases">GitHub Releases</a></strong></p>
421
  <p>Open <strong>Edge0 Chat</strong> on Windows and choose a model folder, such as <code>Edge0-145B-NVFP4</code>, to run the model locally. To build the app from source, follow the <a href="https://github.com/Edge0-AI/Edge0/blob/main/edge0-infer/examples/desktop/README.md">Edge0 Chat</a> guide.</p>
422
  </details>
423
+ <p><a name="memory-update"></a>
424
+ <a name="deployment-qa"></a>
425
+ <a name="memory-deployment"></a>
426
+ <a name="usage-help"></a></p>
427
+ <h2 id="usage-troubleshooting">Usage &amp; troubleshooting</h2>
428
+ <p>145B / 120B · Memory options</p>
429
+ <details>
430
+ <summary><a name="low-memory"></a>Low RAM · SSD offload</summary>
431
+ <ul>
432
+ <li><strong>SSD mode:</strong> <code>--mmap-experts</code> — read uncached experts from SSD on demand.</li>
433
+ <li><strong>Combined mode:</strong> <code>--resident-budget-gib 20</code> + static <code>--expert-profile</code> — SSD loading with a host expert-cache budget and disjoint GPU/RAM caches. Includes <code>--mmap-experts</code> automatically.</li>
434
+ </ul>
435
+ </details>
436
+ <details>
437
+ <summary><a name="ram-budget"></a>RAM budget · what 20 GiB covers</summary>
438
+ <ul>
439
+ <li><strong>Expert cache:</strong> <code>--resident-budget-gib 20</code> caps host-RAM expert caching at <strong>20 GiB</strong>, lowered further to fit available memory.</li>
440
+ <li><strong>Total RAM:</strong> expert cache + other runtime allocations. This flag does <strong>not</strong> set a total-process RAM limit.</li>
441
+ <li><strong>Benchmark limit:</strong> the 20 GiB total RAM test budget is a separate measurement condition.</li>
442
+ </ul>
443
+ </details>
444
+ <details>
445
+ <summary><a name="expert-residency"></a>GPU / RAM · avoid duplicate experts</summary>
446
+ <ul>
447
+ <li><strong>Enable:</strong> <code>--resident-experts</code> or <code>--resident-budget-gib N</code>, with a static <code>--expert-profile</code>.</li>
448
+ <li><strong>Effect:</strong> experts cached in VRAM need no duplicate RAM copy. Compatible with SSD loading.</li>
449
+ <li><strong>Default:</strong> off, prioritizing speed.</li>
450
+ <li><strong>Automatically enabled:</strong> <code>--mmap-experts</code>, <code>--resident-cpu-experts</code>, <code>--resident-pin</code>.</li>
451
+ </ul>
452
+ </details>
453
  <h2 id="acknowledgements">Acknowledgements</h2>
454
  <details open>
455
  <summary><b>Credits &amp; license</b></summary>