BROBOT: the model player, the RAM-as-effort frontier, and the MoE asymmetry

README.md

@@ -21,6 +21,23 @@ license: mit
 
 <p>The slowest row is ours, and it stays: a 135-million-parameter dense model runs three hundred times slower than a 397-billion-parameter sparse one. <b>Sparsity, not size, decides what a small machine can run.</b></p>
 
+<p><b>BROBOT — the build.</b> <a href="https://github.com/minaiml/brobot" target="_blank" rel="noopener">minaiml/brobot</a> is the model player itself, built from PocketPal AI (MIT, React Native + llama.rn). Its <code>src/handheld/</code> module is the device side, and what it has found so far is the interesting part.</p>
+
+<p><b>RAM is effort.</b> Three things compete for it — weights, KV cache, prefill batch — and the KV cache is charged <i>twice</i>: once in memory, and again in bandwidth on every single token. So the configuration that fits the most context is reliably <b>not</b> the one that produces the most tokens.</p>
+
+<div class="scroll"><table>
+<tr><th align="left">strategy</th><th align="left">context</th><th align="left">RAM</th><th align="left">tok/s</th><th align="left">tok/s per GB</th></tr>
+<tr><td>speed</td><td>1,024</td><td>2.17&nbsp;GB</td><td>7.6</td><td><b>3.51</b></td></tr>
+<tr><td>balanced <i>(default)</i></td><td>16,384</td><td>2.79&nbsp;GB</td><td>5.8</td><td>2.09</td></tr>
+<tr><td>context</td><td>32,768</td><td>3.46&nbsp;GB</td><td>4.7</td><td>1.35</td></tr>
+</table></div>
+
+<p>Maximising context returns 38&nbsp;% of the tokens per gigabyte of the efficient point, and costs 38&nbsp;% of the throughput to get there. So the player picks on that frontier instead of maximising any one axis, and it states what it rejected and why.</p>
+
+<p><b>And the mixture-of-experts asymmetry, which is the whole argument in two rows:</b> memory scales with <i>total</i> parameters, speed scales with <i>active</i> ones. On the same mid-range phone, a dense 35B holds 21&nbsp;GB and runs at 0.7&nbsp;tok/s; an MoE 35B-A3B holds the same 21&nbsp;GB and runs at <b>6.5&nbsp;tok/s</b> — nine times the throughput at identical memory cost. Quantizing the KV cache turns out to buy speed as well as memory, and the gain grows with context: +12&nbsp;% at 4k, <b>+49&nbsp;% at 32k</b>.</p>
+
+<p>Every number there is a <b>model, not a benchmark</b>, and says so — to be replaced by measured telemetry the moment a real device reports one. The specification is <a href="https://github.com/minaiml/.github/blob/main/PLAYER.md" target="_blank" rel="noopener">PLAYER.md</a>; the reasoning that led to it is <a href="https://github.com/minaiml/.github/blob/main/HYPOTHESIS.md" target="_blank" rel="noopener">HYPOTHESIS.md</a>.</p>
+
 <p><b>Smart phones, plural.</b> A phone at 11 tok/s is not a terminal for somebody else's data centre — it is a <b>node</b>. That reframes the problem: not "how large a model fits on this device", but "what does a mesh of small devices know together". The catalogue already carries the cluster layer — <a href="https://github.com/minaiml/exo" target="_blank" rel="noopener">exo</a> builds an AI cluster from everyday devices, <a href="https://github.com/minaiml/distributed-llama" target="_blank" rel="noopener">distributed-llama</a> splits one model across machines on a LAN, <a href="https://github.com/minaiml/petals" target="_blank" rel="noopener">petals</a> does it over a swarm.</p>
 
 <p><b>The floor is two processors.</b> <a href="https://huggingface.co/spaces/PYTHAI/mindX" target="_blank" rel="noopener">mindX</a> is the existence proof, and it is deliberately unglamorous: <b>2 CPU cores, 7.8&nbsp;GB of RAM, no GPU</b>, self-throttled to a third of the processor. On that it trains <i>its own weights</i> from its own memory — dreams become a corpus, the corpus trains a 135M actor, and a proof-of-recall gate decides whether the result is allowed to exist. It usually isn't: generation 39 is the newest the gate accepted, and every generation after it was refused. <b>A gate that never refuses is not a gate.</b></p>

index.html

@@ -42,6 +42,23 @@ td:first-child { white-space: nowrap; }
 
 <p>The slowest row is ours, and it stays: a 135-million-parameter dense model runs three hundred times slower than a 397-billion-parameter sparse one. <b>Sparsity, not size, decides what a small machine can run.</b></p>
 
+<p><b>BROBOT — the build.</b> <a href="https://github.com/minaiml/brobot" target="_blank" rel="noopener">minaiml/brobot</a> is the model player itself, built from PocketPal AI (MIT, React Native + llama.rn). Its <code>src/handheld/</code> module is the device side, and what it has found so far is the interesting part.</p>
+
+<p><b>RAM is effort.</b> Three things compete for it — weights, KV cache, prefill batch — and the KV cache is charged <i>twice</i>: once in memory, and again in bandwidth on every single token. So the configuration that fits the most context is reliably <b>not</b> the one that produces the most tokens.</p>
+
+<div class="scroll"><table>
+<tr><th align="left">strategy</th><th align="left">context</th><th align="left">RAM</th><th align="left">tok/s</th><th align="left">tok/s per GB</th></tr>
+<tr><td>speed</td><td>1,024</td><td>2.17&nbsp;GB</td><td>7.6</td><td><b>3.51</b></td></tr>
+<tr><td>balanced <i>(default)</i></td><td>16,384</td><td>2.79&nbsp;GB</td><td>5.8</td><td>2.09</td></tr>
+<tr><td>context</td><td>32,768</td><td>3.46&nbsp;GB</td><td>4.7</td><td>1.35</td></tr>
+</table></div>
+
+<p>Maximising context returns 38&nbsp;% of the tokens per gigabyte of the efficient point, and costs 38&nbsp;% of the throughput to get there. So the player picks on that frontier instead of maximising any one axis, and it states what it rejected and why.</p>
+
+<p><b>And the mixture-of-experts asymmetry, which is the whole argument in two rows:</b> memory scales with <i>total</i> parameters, speed scales with <i>active</i> ones. On the same mid-range phone, a dense 35B holds 21&nbsp;GB and runs at 0.7&nbsp;tok/s; an MoE 35B-A3B holds the same 21&nbsp;GB and runs at <b>6.5&nbsp;tok/s</b> — nine times the throughput at identical memory cost. Quantizing the KV cache turns out to buy speed as well as memory, and the gain grows with context: +12&nbsp;% at 4k, <b>+49&nbsp;% at 32k</b>.</p>
+
+<p>Every number there is a <b>model, not a benchmark</b>, and says so — to be replaced by measured telemetry the moment a real device reports one. The specification is <a href="https://github.com/minaiml/.github/blob/main/PLAYER.md" target="_blank" rel="noopener">PLAYER.md</a>; the reasoning that led to it is <a href="https://github.com/minaiml/.github/blob/main/HYPOTHESIS.md" target="_blank" rel="noopener">HYPOTHESIS.md</a>.</p>
+
 <p><b>Smart phones, plural.</b> A phone at 11 tok/s is not a terminal for somebody else's data centre — it is a <b>node</b>. That reframes the problem: not "how large a model fits on this device", but "what does a mesh of small devices know together". The catalogue already carries the cluster layer — <a href="https://github.com/minaiml/exo" target="_blank" rel="noopener">exo</a> builds an AI cluster from everyday devices, <a href="https://github.com/minaiml/distributed-llama" target="_blank" rel="noopener">distributed-llama</a> splits one model across machines on a LAN, <a href="https://github.com/minaiml/petals" target="_blank" rel="noopener">petals</a> does it over a swarm.</p>
 
 <p><b>The floor is two processors.</b> <a href="https://huggingface.co/spaces/PYTHAI/mindX" target="_blank" rel="noopener">mindX</a> is the existence proof, and it is deliberately unglamorous: <b>2 CPU cores, 7.8&nbsp;GB of RAM, no GPU</b>, self-throttled to a third of the processor. On that it trains <i>its own weights</i> from its own memory — dreams become a corpus, the corpus trains a 135M actor, and a proof-of-recall gate decides whether the result is allowed to exist. It usually isn't: generation 39 is the newest the gate accepted, and every generation after it was refused. <b>A gate that never refuses is not a gate.</b></p>