Umer359 commited on
Commit
259f9f0
Β·
1 Parent(s): 4e236c9

Swap local AI fallback model from 3B to 1B for mobile speed

Browse files

Real report: the 3B local model (WebLLM, in-browser) made things
noticeably slow, especially on mobile where WebGPU memory budgets are
often under 1-1.5GB. Llama-3.2-1B-Instruct is the same Meta family/
generation as the 3B model was, specifically built by Meta for on-
device/edge use, at roughly 1/2.6 the VRAM (879MB vs. 2,264MB,
verified directly against WebLLM's own prebuilt config) β€” a real but
smaller capability gap than the original 360M model had.

Files changed (1) hide show
  1. index.html +19 -12
index.html CHANGED
@@ -345,7 +345,7 @@
345
  <div class="cap-banner-active" id="capBannerActive" style="display:none;align-items:center;justify-content:space-between;gap:10px;">
346
  <div class="cap-title">πŸ–₯️ Local AI mode β€” <span class="info-term" id="localLimitedTerm" tabindex="0" role="button" aria-expanded="false" aria-describedby="localLimitedTooltip">limited<div class="tooltip-bubble" id="localLimitedTooltip" role="tooltip">
347
  <ol>
348
- <li><strong>Smaller model</strong> β€” it's a 3B-parameter model, on par with Nox's own server-side "Standard" tier but below the 8B "Smart" tier. A real, if smaller, capability gap versus the server's best β€” and a much bigger download than a phone-class model, so it can take a while on a slow connection.</li>
349
  <li><strong>No images</strong> β€” it's text-only. Attaching a photo/document in local mode gets declined with a message telling you to switch back to the server.</li>
350
  <li><strong>No live data</strong> β€” the server injects real-time weather, news, currency rates, holidays, and general web facts into its answers. Local mode has none of that β€” it's just the raw model guessing from its own training, so it'll be wrong or evasive on anything time-sensitive ("what's the weather," "who won yesterday," etc.).</li>
351
  <li><strong>No date awareness</strong> β€” the server also tells the model today's actual date; local mode doesn't get that either, so it may not know what day it is.</li>
@@ -1772,18 +1772,25 @@ function tickCapCountdown() {
1772
  // message cap) is exhausted, a visitor can opt into running a small model
1773
  // entirely in their own browser via WebGPU (@mlc-ai/web-llm) instead of
1774
  // waiting it out. This is a real capability downgrade versus the server's
1775
- // best ("Smart", 8B) β€” though upgraded from the original 360M-parameter
1776
- // choice to the same 3B-class Llama model as the server's "Standard" tier,
1777
- // trading a much bigger one-time download for noticeably better coherence
1778
- // and reasoning β€” so it's opt-in, clearly labeled everywhere it's active,
1779
- // and only ever offered when the browser actually supports it: WebGPU has
1780
- // no CPU/WASM fallback in this library, so there's no point offering a
1781
- // button that would just fail. esm.run's ES-module-only build is fine
1782
- // here (unlike the earlier pdf.js mobile-webview issue) because any
1783
- // browser new enough to have WebGPU support is already far newer than the
1784
- // module-support gap that caused that bug.
 
 
 
 
 
 
 
1785
  const HAS_WEBGPU = typeof navigator !== 'undefined' && !!navigator.gpu;
1786
- const LOCAL_MODEL_ID = 'Llama-3.2-3B-Instruct-q4f16_1-MLC';
1787
  // Mirrors worker.js's own CRISIS_RE / CRISIS_RESPONSE exactly. Local mode
1788
  // bypasses the server entirely, so without this a crisis message would go
1789
  // straight to an unsupervised local model with zero safety net β€” this
 
345
  <div class="cap-banner-active" id="capBannerActive" style="display:none;align-items:center;justify-content:space-between;gap:10px;">
346
  <div class="cap-title">πŸ–₯️ Local AI mode β€” <span class="info-term" id="localLimitedTerm" tabindex="0" role="button" aria-expanded="false" aria-describedby="localLimitedTooltip">limited<div class="tooltip-bubble" id="localLimitedTooltip" role="tooltip">
347
  <ol>
348
+ <li><strong>Smaller model</strong> β€” it's a 1B-parameter model, chosen specifically to run smoothly on phones and slower devices (WebGPU memory is often tight on mobile). A real capability gap versus the server's best ("Smart", 8B), but a much smaller, faster download than a bigger model would be.</li>
349
  <li><strong>No images</strong> β€” it's text-only. Attaching a photo/document in local mode gets declined with a message telling you to switch back to the server.</li>
350
  <li><strong>No live data</strong> β€” the server injects real-time weather, news, currency rates, holidays, and general web facts into its answers. Local mode has none of that β€” it's just the raw model guessing from its own training, so it'll be wrong or evasive on anything time-sensitive ("what's the weather," "who won yesterday," etc.).</li>
351
  <li><strong>No date awareness</strong> β€” the server also tells the model today's actual date; local mode doesn't get that either, so it may not know what day it is.</li>
 
1772
  // message cap) is exhausted, a visitor can opt into running a small model
1773
  // entirely in their own browser via WebGPU (@mlc-ai/web-llm) instead of
1774
  // waiting it out. This is a real capability downgrade versus the server's
1775
+ // best ("Smart", 8B). Went through two model choices before landing here:
1776
+ // the original 360M model was too weak (noticeably less coherent), then
1777
+ // a 3B model (same class as the server's "Standard" tier) fixed that but
1778
+ // real report β€” the 3B model made things noticeably slow, especially on
1779
+ // mobile, where WebGPU memory budgets are often under 1-1.5GB. Settled on
1780
+ // Llama-3.2-1B β€” same Meta family/generation the 3B was, and specifically
1781
+ // one of the models Meta built for on-device/edge use, at roughly 1/2.6
1782
+ // the VRAM (verified directly against WebLLM's own prebuilt config
1783
+ // before shipping: 879MB vs. 2,264MB) β€” real, if smaller, capability gap
1784
+ // versus 3B, but nowhere near as weak as the original 360M was. Opt-in,
1785
+ // clearly labeled everywhere it's active, and only ever offered when the
1786
+ // browser actually supports it: WebGPU has no CPU/WASM fallback in this
1787
+ // library, so there's no point offering a button that would just fail.
1788
+ // esm.run's ES-module-only build is fine here (unlike the earlier pdf.js
1789
+ // mobile-webview issue) because any browser new enough to have WebGPU
1790
+ // support is already far newer than the module-support gap that caused
1791
+ // that bug.
1792
  const HAS_WEBGPU = typeof navigator !== 'undefined' && !!navigator.gpu;
1793
+ const LOCAL_MODEL_ID = 'Llama-3.2-1B-Instruct-q4f16_1-MLC';
1794
  // Mirrors worker.js's own CRISIS_RE / CRISIS_RESPONSE exactly. Local mode
1795
  // bypasses the server entirely, so without this a crisis message would go
1796
  // straight to an unsupervised local model with zero safety net β€” this