Swap local AI fallback model from 3B to 1B for mobile speed
Browse filesReal report: the 3B local model (WebLLM, in-browser) made things
noticeably slow, especially on mobile where WebGPU memory budgets are
often under 1-1.5GB. Llama-3.2-1B-Instruct is the same Meta family/
generation as the 3B model was, specifically built by Meta for on-
device/edge use, at roughly 1/2.6 the VRAM (879MB vs. 2,264MB,
verified directly against WebLLM's own prebuilt config) β a real but
smaller capability gap than the original 360M model had.
- index.html +19 -12
index.html
CHANGED
|
@@ -345,7 +345,7 @@
|
|
| 345 |
<div class="cap-banner-active" id="capBannerActive" style="display:none;align-items:center;justify-content:space-between;gap:10px;">
|
| 346 |
<div class="cap-title">π₯οΈ Local AI mode β <span class="info-term" id="localLimitedTerm" tabindex="0" role="button" aria-expanded="false" aria-describedby="localLimitedTooltip">limited<div class="tooltip-bubble" id="localLimitedTooltip" role="tooltip">
|
| 347 |
<ol>
|
| 348 |
-
<li><strong>Smaller model</strong> β it's a
|
| 349 |
<li><strong>No images</strong> β it's text-only. Attaching a photo/document in local mode gets declined with a message telling you to switch back to the server.</li>
|
| 350 |
<li><strong>No live data</strong> β the server injects real-time weather, news, currency rates, holidays, and general web facts into its answers. Local mode has none of that β it's just the raw model guessing from its own training, so it'll be wrong or evasive on anything time-sensitive ("what's the weather," "who won yesterday," etc.).</li>
|
| 351 |
<li><strong>No date awareness</strong> β the server also tells the model today's actual date; local mode doesn't get that either, so it may not know what day it is.</li>
|
|
@@ -1772,18 +1772,25 @@ function tickCapCountdown() {
|
|
| 1772 |
// message cap) is exhausted, a visitor can opt into running a small model
|
| 1773 |
// entirely in their own browser via WebGPU (@mlc-ai/web-llm) instead of
|
| 1774 |
// waiting it out. This is a real capability downgrade versus the server's
|
| 1775 |
-
// best ("Smart", 8B)
|
| 1776 |
-
//
|
| 1777 |
-
//
|
| 1778 |
-
//
|
| 1779 |
-
//
|
| 1780 |
-
//
|
| 1781 |
-
//
|
| 1782 |
-
//
|
| 1783 |
-
//
|
| 1784 |
-
//
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1785 |
const HAS_WEBGPU = typeof navigator !== 'undefined' && !!navigator.gpu;
|
| 1786 |
-
const LOCAL_MODEL_ID = 'Llama-3.2-
|
| 1787 |
// Mirrors worker.js's own CRISIS_RE / CRISIS_RESPONSE exactly. Local mode
|
| 1788 |
// bypasses the server entirely, so without this a crisis message would go
|
| 1789 |
// straight to an unsupervised local model with zero safety net β this
|
|
|
|
| 345 |
<div class="cap-banner-active" id="capBannerActive" style="display:none;align-items:center;justify-content:space-between;gap:10px;">
|
| 346 |
<div class="cap-title">π₯οΈ Local AI mode β <span class="info-term" id="localLimitedTerm" tabindex="0" role="button" aria-expanded="false" aria-describedby="localLimitedTooltip">limited<div class="tooltip-bubble" id="localLimitedTooltip" role="tooltip">
|
| 347 |
<ol>
|
| 348 |
+
<li><strong>Smaller model</strong> β it's a 1B-parameter model, chosen specifically to run smoothly on phones and slower devices (WebGPU memory is often tight on mobile). A real capability gap versus the server's best ("Smart", 8B), but a much smaller, faster download than a bigger model would be.</li>
|
| 349 |
<li><strong>No images</strong> β it's text-only. Attaching a photo/document in local mode gets declined with a message telling you to switch back to the server.</li>
|
| 350 |
<li><strong>No live data</strong> β the server injects real-time weather, news, currency rates, holidays, and general web facts into its answers. Local mode has none of that β it's just the raw model guessing from its own training, so it'll be wrong or evasive on anything time-sensitive ("what's the weather," "who won yesterday," etc.).</li>
|
| 351 |
<li><strong>No date awareness</strong> β the server also tells the model today's actual date; local mode doesn't get that either, so it may not know what day it is.</li>
|
|
|
|
| 1772 |
// message cap) is exhausted, a visitor can opt into running a small model
|
| 1773 |
// entirely in their own browser via WebGPU (@mlc-ai/web-llm) instead of
|
| 1774 |
// waiting it out. This is a real capability downgrade versus the server's
|
| 1775 |
+
// best ("Smart", 8B). Went through two model choices before landing here:
|
| 1776 |
+
// the original 360M model was too weak (noticeably less coherent), then
|
| 1777 |
+
// a 3B model (same class as the server's "Standard" tier) fixed that but
|
| 1778 |
+
// real report β the 3B model made things noticeably slow, especially on
|
| 1779 |
+
// mobile, where WebGPU memory budgets are often under 1-1.5GB. Settled on
|
| 1780 |
+
// Llama-3.2-1B β same Meta family/generation the 3B was, and specifically
|
| 1781 |
+
// one of the models Meta built for on-device/edge use, at roughly 1/2.6
|
| 1782 |
+
// the VRAM (verified directly against WebLLM's own prebuilt config
|
| 1783 |
+
// before shipping: 879MB vs. 2,264MB) β real, if smaller, capability gap
|
| 1784 |
+
// versus 3B, but nowhere near as weak as the original 360M was. Opt-in,
|
| 1785 |
+
// clearly labeled everywhere it's active, and only ever offered when the
|
| 1786 |
+
// browser actually supports it: WebGPU has no CPU/WASM fallback in this
|
| 1787 |
+
// library, so there's no point offering a button that would just fail.
|
| 1788 |
+
// esm.run's ES-module-only build is fine here (unlike the earlier pdf.js
|
| 1789 |
+
// mobile-webview issue) because any browser new enough to have WebGPU
|
| 1790 |
+
// support is already far newer than the module-support gap that caused
|
| 1791 |
+
// that bug.
|
| 1792 |
const HAS_WEBGPU = typeof navigator !== 'undefined' && !!navigator.gpu;
|
| 1793 |
+
const LOCAL_MODEL_ID = 'Llama-3.2-1B-Instruct-q4f16_1-MLC';
|
| 1794 |
// Mirrors worker.js's own CRISIS_RE / CRISIS_RESPONSE exactly. Local mode
|
| 1795 |
// bypasses the server entirely, so without this a crisis message would go
|
| 1796 |
// straight to an unsupervised local model with zero safety net β this
|