Umer359 Claude Sonnet 5 commited on
Commit
cbc4739
Β·
1 Parent(s): e4b6b76

Allow one-shot document attachments in offline Chat mode

Browse files

Local (offline) Chat mode previously hard-blocked any document
attachment outright, on an earlier real test report that the full
12,000-char server-side cap produced garbled, unusable replies from
the tiny local 1B model. Re-enabled with a much smaller cap
(MAX_ONE_SHOT_DOC_CHARS_LOCAL, 4000 chars combined) instead of
removing the safeguard entirely, so a document can't crowd out the
system prompt and conversation history. Multi-file selection already
combines into one attachment for this path, so it works there too.
Images remain blocked in local mode β€” that's a real capability gap
(text-only engine), not a size-tunable limit.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Files changed (1) hide show
  1. index.html +26 -15
index.html CHANGED
@@ -374,7 +374,7 @@
374
  <div class="cap-title">πŸ–₯️ Local AI mode β€” <span class="info-term" id="localLimitedTerm" tabindex="0" role="button" aria-expanded="false" aria-describedby="localLimitedTooltip">limited<div class="tooltip-bubble" id="localLimitedTooltip" role="tooltip">
375
  <ol>
376
  <li><strong>Smaller model</strong> β€” it's a 1B-parameter model, chosen specifically to run smoothly on phones and slower devices (WebGPU memory is often tight on mobile). A real capability gap versus the server's best ("Smart", 8B), but a much smaller, faster download than a bigger model would be.</li>
377
- <li><strong>No images, no one-off documents</strong> β€” it's text-only, and can't analyze a document in a single pass. Switch to RAG mode to work with a document offline instead β€” that's fully supported, indexed and searched entirely on your device.</li>
378
  <li><strong>No live data</strong> β€” the server injects real-time weather, news, currency rates, holidays, and general web facts into its answers. Local mode has none of that β€” it's just the raw model guessing from its own training, so it'll be wrong or evasive on anything time-sensitive ("what's the weather," "who won yesterday," etc.).</li>
379
  <li><strong>No date awareness</strong> β€” the server also tells the model today's actual date; local mode doesn't get that either, so it may not know what day it is.</li>
380
  <li><strong>Speed/quality depend on your device</strong> β€” no server-grade GPU behind it, just whatever's in your laptop/phone.</li>
@@ -767,6 +767,13 @@ const MAX_ATTACHMENT_BYTES = 4 * 1024 * 1024; // 4MB
767
  // mode is to not need a single-prompt size ceiling at all, so this stays
768
  // scoped to pendingOneShotDoc specifically, not a general document limit.
769
  const MAX_ONE_SHOT_DOC_CHARS = 12000;
 
 
 
 
 
 
 
770
 
771
  let pendingImage = null; // { base64, name }
772
  // A document attached while in "Chat" mode β€” a one-shot analysis, folded
@@ -903,14 +910,17 @@ fileInput.addEventListener('change', async () => {
903
  if (!docs.length) { fileInput.value = ''; return; }
904
 
905
  if (attachmentMode === 'rag') {
 
 
 
 
 
906
  await ingestDocumentsForRag(docs);
907
- } else if (localModeActive) {
908
- alert("Local AI mode can't analyze a one-off document β€” switch to RAG mode to index it offline, or switch back to the server.");
909
- if (!images.length) clearAttachment();
910
  } else {
911
  try {
912
  const texts = await Promise.all(docs.map(async (f) => `[Attached file: ${f.name}]\n\n${await extractFileText(f)}`));
913
- const combined = texts.join('\n\n').trim().slice(0, MAX_ONE_SHOT_DOC_CHARS);
 
914
  pendingOneShotDoc = { text: combined, name: docs.length === 1 ? docs[0].name : `${docs.length} files` };
915
  showAttachmentPreview(pendingOneShotDoc.name);
916
  } catch (err) {
@@ -2786,17 +2796,18 @@ async function requestReply(imageBase64) {
2786
  const userMessageText = [...history].reverse().find((m) => m.role === 'user')?.content || '';
2787
 
2788
  if (localModeActive) {
2789
- // Local mode still can't handle images or a one-shot document dump
2790
- // (a chat-mode attachment folded into the message as a "[Attached
2791
- // file: ...]" block with the full extracted text) β€” the tiny local
2792
- // model isn't remotely equipped for that (real report: it produced
2793
- // total gibberish when handed a document this way). RAG-mode
2794
- // documents are fine now, though: those get retrieved from the
2795
- // offline (IndexedDB) index inside requestLocalReply itself, not
2796
- // stuffed into the message here at all.
2797
- if (imageBase64 || /\[Attached file:/.test(userMessageText)) {
 
2798
  loadingBubble.parentElement.classList.remove('loading');
2799
- loadingBubble.textContent = "Local AI mode can't read images or one-off documents β€” remove the attachment, switch to RAG mode to index it offline, or switch back to the server.";
2800
  scrollToBottom();
2801
  return;
2802
  }
 
374
  <div class="cap-title">πŸ–₯️ Local AI mode β€” <span class="info-term" id="localLimitedTerm" tabindex="0" role="button" aria-expanded="false" aria-describedby="localLimitedTooltip">limited<div class="tooltip-bubble" id="localLimitedTooltip" role="tooltip">
375
  <ol>
376
  <li><strong>Smaller model</strong> β€” it's a 1B-parameter model, chosen specifically to run smoothly on phones and slower devices (WebGPU memory is often tight on mobile). A real capability gap versus the server's best ("Smart", 8B), but a much smaller, faster download than a bigger model would be.</li>
377
+ <li><strong>No images</strong> β€” it's text-only. One-off documents in Chat mode are read, but truncated much shorter than on the server (this tiny model's usable context is far smaller) β€” for longer documents, RAG mode works better offline: fully supported, indexed and searched entirely on your device.</li>
378
  <li><strong>No live data</strong> β€” the server injects real-time weather, news, currency rates, holidays, and general web facts into its answers. Local mode has none of that β€” it's just the raw model guessing from its own training, so it'll be wrong or evasive on anything time-sensitive ("what's the weather," "who won yesterday," etc.).</li>
379
  <li><strong>No date awareness</strong> β€” the server also tells the model today's actual date; local mode doesn't get that either, so it may not know what day it is.</li>
380
  <li><strong>Speed/quality depend on your device</strong> β€” no server-grade GPU behind it, just whatever's in your laptop/phone.</li>
 
767
  // mode is to not need a single-prompt size ceiling at all, so this stays
768
  // scoped to pendingOneShotDoc specifically, not a general document limit.
769
  const MAX_ONE_SHOT_DOC_CHARS = 12000;
770
+ // The local engine (Llama-3.2-1B) runs with a much smaller practical
771
+ // context window than the server model, and a smaller, far less capable
772
+ // model besides β€” the full 12,000-char cap was tried here before and
773
+ // produced unusable, garbled replies. A tighter cap doesn't guarantee
774
+ // good output from a 1B model, but it keeps a document from crowding out
775
+ // the system prompt and conversation history entirely.
776
+ const MAX_ONE_SHOT_DOC_CHARS_LOCAL = 4000;
777
 
778
  let pendingImage = null; // { base64, name }
779
  // A document attached while in "Chat" mode β€” a one-shot analysis, folded
 
910
  if (!docs.length) { fileInput.value = ''; return; }
911
 
912
  if (attachmentMode === 'rag') {
913
+ // Works the same whether the engine is server or local β€” RAG mode
914
+ // never folds full document text into a message at all, it retrieves
915
+ // small, relevant excerpts per question instead, so it isn't subject
916
+ // to the tiny local model's context-window limits the way a one-shot
917
+ // dump is.
918
  await ingestDocumentsForRag(docs);
 
 
 
919
  } else {
920
  try {
921
  const texts = await Promise.all(docs.map(async (f) => `[Attached file: ${f.name}]\n\n${await extractFileText(f)}`));
922
+ const cap = localModeActive ? MAX_ONE_SHOT_DOC_CHARS_LOCAL : MAX_ONE_SHOT_DOC_CHARS;
923
+ const combined = texts.join('\n\n').trim().slice(0, cap);
924
  pendingOneShotDoc = { text: combined, name: docs.length === 1 ? docs[0].name : `${docs.length} files` };
925
  showAttachmentPreview(pendingOneShotDoc.name);
926
  } catch (err) {
 
2796
  const userMessageText = [...history].reverse().find((m) => m.role === 'user')?.content || '';
2797
 
2798
  if (localModeActive) {
2799
+ // Local mode still can't read images β€” the engine loaded here is
2800
+ // text-only, not a technical choice that could be relaxed by capping
2801
+ // input size the way the one-shot document case below was. A
2802
+ // one-shot document dump WAS blocked here too, on an earlier real
2803
+ // test report of unusable, garbled replies from the tiny 1B model β€”
2804
+ // now allowed again, but capped much tighter for local mode
2805
+ // specifically (MAX_ONE_SHOT_DOC_CHARS_LOCAL) than the server path's
2806
+ // budget, to keep a document from crowding out the system prompt and
2807
+ // conversation history entirely.
2808
+ if (imageBase64) {
2809
  loadingBubble.parentElement.classList.remove('loading');
2810
+ loadingBubble.textContent = "Local AI mode can't read images β€” remove the attachment, or switch back to the server.";
2811
  scrollToBottom();
2812
  return;
2813
  }