| --- |
| tags: |
| - llama.cpp |
| - ggml |
| - moe |
| - mixture-of-experts |
| - inference |
| - local-llm |
| - ssd |
| - model-offloading |
| - memory-optimization |
| - cuda |
| --- |
| # bounded-moe |
|
|
| **Bounded-memory inference experiments for running large Mixture-of-Experts models on consumer hardware.** |
|
|
| > Can we run a large MoE model without letting its expert weights consume most of system RAM? |
|
|
| `bounded-moe` is an experimental project exploring **SSD-backed expert storage and bounded-memory caching for llama.cpp / GGML**. |
|
|
| The goal is to make large MoE models more practical on memory-constrained consumer PCs by keeping only a controlled working set of experts resident in memory. |
|
|
| ## Architecture |
|
|
| ```text |
| MoE Router |
| │ |
| ▼ |
| Requested Experts |
| │ |
| ▼ |
| ┌───────────────────┐ |
| │ Bounded Expert │ |
| │ Cache │ |
| └─────────┬─────────┘ |
| hit │ miss |
| │ |
| ▼ |
| External Expert Storage |
| │ |
| SSD |
| │ |
| ▼ |
| RAM / VRAM |
| │ |
| ▼ |
| Compute |
| ``` |
|
|
| In simplified form: |
|
|
| **SSD → Bounded Expert Cache → RAM/VRAM → Inference** |
|
|
| Instead of allowing all expert weights to remain resident in system memory, the runtime resolves and caches the experts required by MoE routing. |
|
|
| ## Test Hardware |
|
|
| Current development and testing is performed on consumer hardware: |
|
|
| * **GPU:** NVIDIA GeForce RTX 4060 8 GB |
| * **System RAM:** 32 GB |
| * **OS:** Windows 11 |
| * **Runtime:** llama.cpp / GGML |
| * **Model class:** ~35B Mixture-of-Experts, ~3B active parameters |
|
|
| The project specifically explores scenarios where model weights, applications, and the operating system compete for limited RAM. |
|
|
| ## Implemented Research |
|
|
| The project currently includes experiments around: |
|
|
| * MoE routing tracing |
| * External expert storage |
| * Bounded expert caching |
| * Cache hit/miss accounting |
| * Expert pin/unpin lifecycle |
| * Safe cache eviction |
| * Direct expert reads |
| * Resolver indirection |
| * Cache-size experiments |
| * Working-set / RAM measurements |
| * SSD-backed inference |
| * Performance and correctness validation |
|
|
| ## Current Status |
|
|
| The architecture is **experimental**. |
|
|
| The bounded-storage path works and demonstrates that expert memory can be managed independently from normal full-model residency. |
|
|
| The main research challenge is now **performance**. |
|
|
| Normal memory-mapped inference is significantly faster than the current experimental SSD-backed expert path. Current work therefore focuses on reducing expert-resolution and storage latency while preserving the bounded-memory property. |
|
|
| This repository should currently be considered an **engineering/research prototype**, not a production inference runtime. |
|
|
| ## Why? |
|
|
| Large MoE models are interesting for consumer hardware because only a subset of their parameters is active for each token. |
|
|
| However, inactive experts can still consume significant system memory. |
|
|
| This project asks a slightly different question: |
|
|
| > **What if model capacity could be much larger than the amount of RAM we're willing to dedicate to inference?** |
|
|
| Rather than treating available RAM as the hard limit, `bounded-moe` explores using a hierarchy of: |
|
|
| **SSD → bounded RAM cache → GPU → compute** |
|
|
| while exploiting MoE routing locality to keep the frequently requested experts close to compute. |
|
|
| ## Future Research |
|
|
| Planned experiments include: |
|
|
| * asynchronous expert prefetching |
| * predictive prefetch based on routing behavior |
| * overlapping SSD I/O with GPU computation |
| * smarter eviction policies |
| * cache locality analysis |
| * larger-context testing |
| * Windows I/O optimization |
| * reducing resolver overhead |
| * dense-model layer/block streaming experiments |
|
|
| The last item is particularly interesting: some of the infrastructure may eventually be generalized beyond MoE models into a **bounded-memory inference runtime for dense models**. |
|
|
| ## Source Code |
|
|
| The implementation, experiments, benchmark notes, and development history are available on GitHub: |
|
|
| https://github.com/kornpaksittikool-beep/bounded-moe |
|
|
| ## Feedback |
|
|
| Feedback is very welcome, particularly from people working with: |
|
|
| **llama.cpp · GGML · MoE routing · model offloading · caching · mmap · SSD I/O · CUDA · inference optimization** |
|
|
| If you've experimented with similar SSD-backed or bounded-memory inference architectures, I'd especially like to hear about approaches for hiding cache-miss and storage latency. |
|
|
| --- |
|
|
| **Status:** Experimental / Research Prototype |
