File size: 4,697 Bytes
283d222 c751c9e 283d222 a9bcaf8 283d222 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 | ---
tags:
- llama.cpp
- ggml
- moe
- mixture-of-experts
- inference
- local-llm
- ssd
- model-offloading
- memory-optimization
- cuda
---
# bounded-moe
**Bounded-memory inference experiments for running large Mixture-of-Experts models on consumer hardware.**
> Can we run a large MoE model without letting its expert weights consume most of system RAM?
`bounded-moe` is an experimental project exploring **SSD-backed expert storage and bounded-memory caching for llama.cpp / GGML**.
The goal is to make large MoE models more practical on memory-constrained consumer PCs by keeping only a controlled working set of experts resident in memory.
## Architecture
```text
MoE Router
│
▼
Requested Experts
│
▼
┌───────────────────┐
│ Bounded Expert │
│ Cache │
└─────────┬─────────┘
hit │ miss
│
▼
External Expert Storage
│
SSD
│
▼
RAM / VRAM
│
▼
Compute
```
In simplified form:
**SSD → Bounded Expert Cache → RAM/VRAM → Inference**
Instead of allowing all expert weights to remain resident in system memory, the runtime resolves and caches the experts required by MoE routing.
## Test Hardware
Current development and testing is performed on consumer hardware:
* **GPU:** NVIDIA GeForce RTX 4060 8 GB
* **System RAM:** 32 GB
* **OS:** Windows 11
* **Runtime:** llama.cpp / GGML
* **Model class:** ~35B Mixture-of-Experts, ~3B active parameters
The project specifically explores scenarios where model weights, applications, and the operating system compete for limited RAM.
## Implemented Research
The project currently includes experiments around:
* MoE routing tracing
* External expert storage
* Bounded expert caching
* Cache hit/miss accounting
* Expert pin/unpin lifecycle
* Safe cache eviction
* Direct expert reads
* Resolver indirection
* Cache-size experiments
* Working-set / RAM measurements
* SSD-backed inference
* Performance and correctness validation
## Current Status
The architecture is **experimental**.
The bounded-storage path works and demonstrates that expert memory can be managed independently from normal full-model residency.
The main research challenge is now **performance**.
Normal memory-mapped inference is significantly faster than the current experimental SSD-backed expert path. Current work therefore focuses on reducing expert-resolution and storage latency while preserving the bounded-memory property.
This repository should currently be considered an **engineering/research prototype**, not a production inference runtime.
## Why?
Large MoE models are interesting for consumer hardware because only a subset of their parameters is active for each token.
However, inactive experts can still consume significant system memory.
This project asks a slightly different question:
> **What if model capacity could be much larger than the amount of RAM we're willing to dedicate to inference?**
Rather than treating available RAM as the hard limit, `bounded-moe` explores using a hierarchy of:
**SSD → bounded RAM cache → GPU → compute**
while exploiting MoE routing locality to keep the frequently requested experts close to compute.
## Future Research
Planned experiments include:
* asynchronous expert prefetching
* predictive prefetch based on routing behavior
* overlapping SSD I/O with GPU computation
* smarter eviction policies
* cache locality analysis
* larger-context testing
* Windows I/O optimization
* reducing resolver overhead
* dense-model layer/block streaming experiments
The last item is particularly interesting: some of the infrastructure may eventually be generalized beyond MoE models into a **bounded-memory inference runtime for dense models**.
## Source Code
The implementation, experiments, benchmark notes, and development history are available on GitHub:
https://github.com/kornpaksittikool-beep/bounded-moe
## Feedback
Feedback is very welcome, particularly from people working with:
**llama.cpp · GGML · MoE routing · model offloading · caching · mmap · SSD I/O · CUDA · inference optimization**
If you've experimented with similar SSD-backed or bounded-memory inference architectures, I'd especially like to hear about approaches for hiding cache-miss and storage latency.
---
**Status:** Experimental / Research Prototype |