system HF Staff commited on
Commit
1ef5e14
ยท
verified ยท
1 Parent(s): 822c4d6

Sync from felladrin/MiniSearch@b72789d

Browse files
agents.md CHANGED
@@ -190,7 +190,7 @@ Need to:
190
 
191
  1. Check the wasm build can run it: the architecture must exist in `node_modules/@wllama/wllama/esm/wasm/wllama.wasm` (grep the binary for the arch name), and the quant must stay under the ceilings in `docs/ai-integration.md` (shard every file under 2000 MiB; total weights under ~3.7 GB)
192
  2. Shard the GGUF (`llama-gguf-split --split --size`, or an equivalent) and publish to a `Felladrin/gguf-sharded-<QUANT>-<Name>` repo
193
- 3. Add to `client/modules/wllama.ts`; thinking-by-default templates need `chatTemplateKwargs: { enable_thinking: false }`
194
  4. Run `node scripts/wllama-catalog-check.cjs` and `npx vitest run client/modules/wllama.test.ts`
195
  5. Update `docs/ai-integration.md`
196
  6. Update `docs/configuration.md` defaults
 
190
 
191
  1. Check the wasm build can run it: the architecture must exist in `node_modules/@wllama/wllama/esm/wasm/wllama.wasm` (grep the binary for the arch name), and the quant must stay under the ceilings in `docs/ai-integration.md` (shard every file under 2000 MiB; total weights under ~3.7 GB)
192
  2. Shard the GGUF (`llama-gguf-split --split --size`, or an equivalent) and publish to a `Felladrin/gguf-sharded-<QUANT>-<Name>` repo
193
+ 3. Add to `client/modules/wllama.ts`; thinking-by-default templates need nothing extra, their `reasoning_content` channel is streamed into the thinking UI
194
  4. Run `node scripts/wllama-catalog-check.cjs` and `npx vitest run client/modules/wllama.test.ts`
195
  5. Update `docs/ai-integration.md`
196
  6. Update `docs/configuration.md` defaults
changelog.md CHANGED
@@ -3,6 +3,11 @@
3
  Notable changes, grouped by date, most recent first.
4
 
5
 
 
 
 
 
 
6
  ## 2026-11-02
7
 
8
  - Five new browser models joined the Wllama catalog: Spark X2.6 1.7B, Granite 4.2 3B, Qwen 3.5 4B, Gemma 4 E2B and Spark X2.6 4B, each tested end to end against the shipped wasm build with the search-answer prompt. The Gemma 4 and Qwen 3.5 architectures were previously absent from the catalog, and the new entries run on the wasm engine's supported architectures (`gemma4`, `qwen35`, `spark2_5`, `granite`). Models whose chat template thinks by default ship with `chatTemplateKwargs: { enable_thinking: false }`, a new per-model field passed to wllama as `chat_template_kwargs`, because MiniSearch renders only the content channel and a thinking template would otherwise spend the whole token budget on reasoning and answer with nothing. `docs/ai-integration.md` now records the two measured ceilings a candidate must fit (shard files under 2000 MiB, since Chromium's OPFS silently truncates beyond it; total weights under ~3.7 GB, since the wasm linear memory caps at 4 GiB), and `npm run wllama-catalog-check` HEADs every shard of every catalog model on HuggingFace to verify the claimed sizes.
 
3
  Notable changes, grouped by date, most recent first.
4
 
5
 
6
+ ## 2026-11-03
7
+
8
+ - The five thinking models added on 2026-11-02 (Spark X2.6 1.7B and 4B, Granite 4.2 3B, Qwen 3.5 4B, Gemma 4 E2B) no longer ship with thinking switched off. Their chain of thought arrives on wllama's `reasoning_content` channel and is now wrapped in the configured reasoning markers, so the existing thinking UI shows it while the model reasons and the answer streams in after. The model warmup counts a reasoning token as a working model, so thinking models no longer log a failed warmup.
9
+
10
+
11
  ## 2026-11-02
12
 
13
  - Five new browser models joined the Wllama catalog: Spark X2.6 1.7B, Granite 4.2 3B, Qwen 3.5 4B, Gemma 4 E2B and Spark X2.6 4B, each tested end to end against the shipped wasm build with the search-answer prompt. The Gemma 4 and Qwen 3.5 architectures were previously absent from the catalog, and the new entries run on the wasm engine's supported architectures (`gemma4`, `qwen35`, `spark2_5`, `granite`). Models whose chat template thinks by default ship with `chatTemplateKwargs: { enable_thinking: false }`, a new per-model field passed to wllama as `chat_template_kwargs`, because MiniSearch renders only the content channel and a thinking template would otherwise spend the whole token budget on reasoning and answer with nothing. `docs/ai-integration.md` now records the two measured ceilings a candidate must fit (shard files under 2000 MiB, since Chromium's OPFS silently truncates beyond it; total weights under ~3.7 GB, since the wasm linear memory caps at 4 GiB), and `npm run wllama-catalog-check` HEADs every shard of every catalog model on HuggingFace to verify the claimed sizes.
client/modules/textGenerationWithWllama.test.ts CHANGED
@@ -3,7 +3,10 @@ import {
3
  updateModelLoadingProgress,
4
  updateModelSizeInMegabytes,
5
  } from "./pubSub";
6
- import { generateTextWithWllama } from "./textGenerationWithWllama";
 
 
 
7
  import { initializeWllama } from "./wllama";
8
 
9
  vi.mock("./logEntries", () => ({ addLogEntry: vi.fn() }));
@@ -14,6 +17,8 @@ vi.mock("./pubSub", () => ({
14
  enableAiResponse: true,
15
  wllamaModelId: "test-model",
16
  cpuThreads: 1,
 
 
17
  }),
18
  getTextGenerationState: vi.fn().mockReturnValue("generating"),
19
  updateModelLoadingProgress: vi.fn(),
@@ -106,3 +111,21 @@ describe("generateTextWithWllama", () => {
106
  expect(updateModelSizeInMegabytes).not.toHaveBeenCalledWith(0);
107
  });
108
  });
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  updateModelLoadingProgress,
4
  updateModelSizeInMegabytes,
5
  } from "./pubSub";
6
+ import {
7
+ composeStreamedText,
8
+ generateTextWithWllama,
9
+ } from "./textGenerationWithWllama";
10
  import { initializeWllama } from "./wllama";
11
 
12
  vi.mock("./logEntries", () => ({ addLogEntry: vi.fn() }));
 
17
  enableAiResponse: true,
18
  wllamaModelId: "test-model",
19
  cpuThreads: 1,
20
+ reasoningStartMarker: "<think>",
21
+ reasoningEndMarker: "</think>",
22
  }),
23
  getTextGenerationState: vi.fn().mockReturnValue("generating"),
24
  updateModelLoadingProgress: vi.fn(),
 
111
  expect(updateModelSizeInMegabytes).not.toHaveBeenCalledWith(0);
112
  });
113
  });
114
+
115
+ describe("composeStreamedText", () => {
116
+ it("returns the answer unchanged when the model does not think", () => {
117
+ expect(composeStreamedText("", "The answer.")).toBe("The answer.");
118
+ });
119
+
120
+ it("keeps the reasoning open while the answer has not started", () => {
121
+ expect(composeStreamedText("hmm, let me see", "")).toBe(
122
+ "<think>hmm, let me see",
123
+ );
124
+ });
125
+
126
+ it("closes the reasoning block when the answer starts", () => {
127
+ expect(composeStreamedText("done thinking", "The answer.")).toBe(
128
+ "<think>done thinking</think>The answer.",
129
+ );
130
+ });
131
+ });
client/modules/textGenerationWithWllama.ts CHANGED
@@ -113,6 +113,8 @@ async function generateWithWllama({
113
  ? await messagesOrBuilder()
114
  : messagesOrBuilder;
115
 
 
 
116
  let streamedMessage = "";
117
 
118
  const stream = await wllama.createChatCompletion({
@@ -135,16 +137,23 @@ async function generateWithWllama({
135
  }
136
 
137
  const delta = chunk.choices[0]?.delta?.content ?? "";
138
- if (!delta) continue;
 
 
 
 
 
139
 
140
  if (shouldCheckCanRespond && getTextGenerationState() !== "generating") {
141
  updateTextGenerationState("generating");
142
  }
143
 
144
- streamedMessage += delta;
 
 
145
  streamedMessage = handleWllamaCompletion(
146
  model,
147
- streamedMessage,
148
  () => abortController.abort(),
149
  onUpdate,
150
  );
@@ -199,6 +208,20 @@ async function initializeWllamaInstance(progressCallback?: ProgressCallback) {
199
  return { wllama, model };
200
  }
201
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
202
  function handleWllamaCompletion(
203
  model: WllamaModel,
204
  currentText: string,
 
113
  ? await messagesOrBuilder()
114
  : messagesOrBuilder;
115
 
116
+ let reasoningText = "";
117
+ let answerText = "";
118
  let streamedMessage = "";
119
 
120
  const stream = await wllama.createChatCompletion({
 
137
  }
138
 
139
  const delta = chunk.choices[0]?.delta?.content ?? "";
140
+ // wllama splits a thinking model's chain of thought into its own
141
+ // channel; the type does not declare it yet.
142
+ const reasoningDelta =
143
+ (chunk.choices[0]?.delta as { reasoning_content?: string } | undefined)
144
+ ?.reasoning_content ?? "";
145
+ if (!delta && !reasoningDelta) continue;
146
 
147
  if (shouldCheckCanRespond && getTextGenerationState() !== "generating") {
148
  updateTextGenerationState("generating");
149
  }
150
 
151
+ if (reasoningDelta) reasoningText += reasoningDelta;
152
+ if (delta) answerText += delta;
153
+
154
  streamedMessage = handleWllamaCompletion(
155
  model,
156
+ composeStreamedText(reasoningText, answerText),
157
  () => abortController.abort(),
158
  onUpdate,
159
  );
 
208
  return { wllama, model };
209
  }
210
 
211
+ /**
212
+ * Joins the reasoning and answer channels into one stream so the existing
213
+ * reasoning-marker parsing in the UI shows the chain of thought while the
214
+ * model thinks. The end marker only appears once the answer has started,
215
+ * which keeps the UI in its "still thinking" state until then.
216
+ */
217
+ export function composeStreamedText(reasoning: string, answer: string): string {
218
+ if (!reasoning) return answer;
219
+ const { reasoningStartMarker, reasoningEndMarker } = getSettings();
220
+ return `${reasoningStartMarker}${reasoning}${
221
+ answer ? `${reasoningEndMarker}${answer}` : ""
222
+ }`;
223
+ }
224
+
225
  function handleWllamaCompletion(
226
  model: WllamaModel,
227
  currentText: string,
client/modules/wllama.test.ts CHANGED
@@ -36,17 +36,13 @@ describe("wllamaModels catalog", () => {
36
  }
37
  });
38
 
39
- it("switches thinking-by-default chat templates off", () => {
40
- // MiniSearch renders only the content channel. A template that thinks
41
- // by default spends the whole token budget on reasoning_content and the
42
- // user sees an empty answer, so any model passing template kwargs must
43
- // be turning thinking off.
44
  for (const [id, model] of models) {
45
- if (model.chatTemplateKwargs) {
46
- expect(model.chatTemplateKwargs, id).toEqual({
47
- enable_thinking: false,
48
- });
49
- }
50
  }
51
  });
52
 
 
36
  }
37
  });
38
 
39
+ it("keeps thinking enabled on every model", () => {
40
+ // The reasoning channel is streamed into the thinking UI, so a model
41
+ // must not hand the template a kwarg that switches thinking off: the
42
+ // answer would arrive with no visible chain of thought, which reads as
43
+ // a dumber model rather than a feature that was turned off.
44
  for (const [id, model] of models) {
45
+ expect(model.chatTemplateKwargs?.enable_thinking, id).not.toBe(false);
 
 
 
 
46
  }
47
  });
48
 
client/modules/wllama.ts CHANGED
@@ -70,7 +70,15 @@ export async function initializeWllama(
70
  : {}),
71
  });
72
 
73
- const hasWarmupSucceeded = !!warmupResponse.choices[0]?.message?.content;
 
 
 
 
 
 
 
 
74
 
75
  addLogEntry(
76
  `Wllama warmup ${hasWarmupSucceeded ? "succeeded" : "failed"}.`,
@@ -128,9 +136,9 @@ export interface WllamaModel {
128
  readonly flash_attn?: boolean;
129
  /**
130
  * Extra variables handed to the model's chat template on every completion
131
- * (wllama's `chat_template_kwargs`). Used to switch off thinking-by-default
132
- * templates: MiniSearch renders only the content channel, so a model that
133
- * spends its whole budget on `reasoning_content` would answer with nothing.
134
  */
135
  readonly chatTemplateKwargs?: Record<string, unknown>;
136
  getSampling: () => Omit<SamplingParams, "logit_bias">;
@@ -337,7 +345,6 @@ export const wllamaModels: Readonly<Record<string, WllamaModel>> = {
337
  hfRepoId: "Felladrin/gguf-sharded-Q4_K_M-Spark-X2.6-1.7B",
338
  hfFilePath: "Spark-X2.6-1.7B-Q4_K_M-00001-of-00004.gguf",
339
  fileSizeInMegabytes: 1769,
340
- chatTemplateKwargs: { enable_thinking: false },
341
  },
342
  "smollm3-3b": {
343
  ...createDefaultModelConfig(),
@@ -373,7 +380,6 @@ export const wllamaModels: Readonly<Record<string, WllamaModel>> = {
373
  hfRepoId: "Felladrin/gguf-sharded-Q4_K_S-granite-4.2-3b",
374
  hfFilePath: "granite-4.2-3b-Q4_K_S-00001-of-00005.gguf",
375
  fileSizeInMegabytes: 2143,
376
- chatTemplateKwargs: { enable_thinking: false },
377
  },
378
  "phi-3.5-mini-3.8b": {
379
  ...createDefaultModelConfig(),
@@ -402,7 +408,6 @@ export const wllamaModels: Readonly<Record<string, WllamaModel>> = {
402
  hfRepoId: "Felladrin/gguf-sharded-UD-Q3_K_XL-Qwen3.5-4B",
403
  hfFilePath: "Qwen3.5-4B-UD-Q3_K_XL-00001-of-00005.gguf",
404
  fileSizeInMegabytes: 2436,
405
- chatTemplateKwargs: { enable_thinking: false },
406
  },
407
  "gemma-4-e2b": {
408
  ...createDefaultModelConfig(),
@@ -410,7 +415,6 @@ export const wllamaModels: Readonly<Record<string, WllamaModel>> = {
410
  hfRepoId: "Felladrin/gguf-sharded-Q3_K_S-gemma-4-e2b-it",
411
  hfFilePath: "gemma-4-e2b-it-Q3_K_S-00001-of-00005.gguf",
412
  fileSizeInMegabytes: 2446,
413
- chatTemplateKwargs: { enable_thinking: false },
414
  },
415
  "phi-4-mini-reasoning": {
416
  ...createDefaultModelConfig(),
@@ -425,6 +429,5 @@ export const wllamaModels: Readonly<Record<string, WllamaModel>> = {
425
  hfRepoId: "Felladrin/gguf-sharded-Q4_K_M-Spark-X2.6-4B",
426
  hfFilePath: "Spark-X2.6-4B-Q4_K_M-00001-of-00007.gguf",
427
  fileSizeInMegabytes: 3543,
428
- chatTemplateKwargs: { enable_thinking: false },
429
  },
430
  };
 
70
  : {}),
71
  });
72
 
73
+ const warmupMessage = warmupResponse.choices[0]?.message as
74
+ | { content?: string; reasoning_content?: string }
75
+ | undefined;
76
+
77
+ // A thinking model spends its single warmup token on the reasoning
78
+ // channel, so either channel counts as a working model.
79
+ const hasWarmupSucceeded = !!(
80
+ warmupMessage?.content || warmupMessage?.reasoning_content
81
+ );
82
 
83
  addLogEntry(
84
  `Wllama warmup ${hasWarmupSucceeded ? "succeeded" : "failed"}.`,
 
136
  readonly flash_attn?: boolean;
137
  /**
138
  * Extra variables handed to the model's chat template on every completion
139
+ * (wllama's `chat_template_kwargs`). Thinking stays on: the reasoning
140
+ * channel is streamed into the thinking UI, so nothing here should turn
141
+ * it off.
142
  */
143
  readonly chatTemplateKwargs?: Record<string, unknown>;
144
  getSampling: () => Omit<SamplingParams, "logit_bias">;
 
345
  hfRepoId: "Felladrin/gguf-sharded-Q4_K_M-Spark-X2.6-1.7B",
346
  hfFilePath: "Spark-X2.6-1.7B-Q4_K_M-00001-of-00004.gguf",
347
  fileSizeInMegabytes: 1769,
 
348
  },
349
  "smollm3-3b": {
350
  ...createDefaultModelConfig(),
 
380
  hfRepoId: "Felladrin/gguf-sharded-Q4_K_S-granite-4.2-3b",
381
  hfFilePath: "granite-4.2-3b-Q4_K_S-00001-of-00005.gguf",
382
  fileSizeInMegabytes: 2143,
 
383
  },
384
  "phi-3.5-mini-3.8b": {
385
  ...createDefaultModelConfig(),
 
408
  hfRepoId: "Felladrin/gguf-sharded-UD-Q3_K_XL-Qwen3.5-4B",
409
  hfFilePath: "Qwen3.5-4B-UD-Q3_K_XL-00001-of-00005.gguf",
410
  fileSizeInMegabytes: 2436,
 
411
  },
412
  "gemma-4-e2b": {
413
  ...createDefaultModelConfig(),
 
415
  hfRepoId: "Felladrin/gguf-sharded-Q3_K_S-gemma-4-e2b-it",
416
  hfFilePath: "gemma-4-e2b-it-Q3_K_S-00001-of-00005.gguf",
417
  fileSizeInMegabytes: 2446,
 
418
  },
419
  "phi-4-mini-reasoning": {
420
  ...createDefaultModelConfig(),
 
429
  hfRepoId: "Felladrin/gguf-sharded-Q4_K_M-Spark-X2.6-4B",
430
  hfFilePath: "Spark-X2.6-4B-Q4_K_M-00001-of-00007.gguf",
431
  fileSizeInMegabytes: 3543,
 
432
  },
433
  };
docs/ai-integration.md CHANGED
@@ -38,7 +38,7 @@ Two constraints decide what can join that list, both measured against the shippe
38
  - **Every model file must be a shard.** Chromium's OPFS storage silently truncates files past 2000 MiB, so a monolithic GGUF above that size arrives corrupted in the browser. Shards also keep progressive download working.
39
  - **The wasm linear memory ceiling is 4 GiB.** Models at ~3.5 GB and above OOM during buffer reserve, so the practical catalog ceiling is around 3.7 GB of weights.
40
 
41
- Models whose chat template thinks by default (Granite 4.2, Gemma 4, Spark X2.6, Qwen 3.5) would spend the whole token budget on the reasoning channel, which MiniSearch does not render. They are configured with `chatTemplateKwargs: { enable_thinking: false }`, passed to wllama as `chat_template_kwargs`.
42
 
43
  `node scripts/wllama-catalog-check.cjs` HEADs every shard of every catalog model on HuggingFace and verifies the claimed sizes; run it after adding or re-sharding a model.
44
 
 
38
  - **Every model file must be a shard.** Chromium's OPFS storage silently truncates files past 2000 MiB, so a monolithic GGUF above that size arrives corrupted in the browser. Shards also keep progressive download working.
39
  - **The wasm linear memory ceiling is 4 GiB.** Models at ~3.5 GB and above OOM during buffer reserve, so the practical catalog ceiling is around 3.7 GB of weights.
40
 
41
+ Models whose chat template thinks by default (Granite 4.2, Gemma 4, Spark X2.6, Qwen 3.5) keep thinking: wllama splits their chain of thought into a `reasoning_content` channel, and MiniSearch wraps it in the configured reasoning markers so the thinking UI shows it while the answer follows. Thinking spends the same token budget as the answer, so these models are slower and, on a long chain of thought, can use up the context before answering.
42
 
43
  `node scripts/wllama-catalog-check.cjs` HEADs every shard of every catalog model on HuggingFace and verifies the claimed sizes; run it after adding or re-sharding a model.
44
 
docs/configuration.md CHANGED
@@ -199,7 +199,7 @@ Search results:
199
  - `{{dateTime}}`: Alias for `{{currentDate}}` - both are replaced with the same current date value
200
  - `{{searchResults}}`: Formatted search results from the web search, including the page excerpts when `enablePageContentFetch` is on
201
 
202
- **Reasoning Markers:** Models that output internal thought processes use `<think>` and `</think>` markers. The UI extracts and separately displays reasoning content from the final response.
203
 
204
  ### Privacy Settings
205
 
 
199
  - `{{dateTime}}`: Alias for `{{currentDate}}` - both are replaced with the same current date value
200
  - `{{searchResults}}`: Formatted search results from the web search, including the page excerpts when `enablePageContentFetch` is on
201
 
202
+ **Reasoning Markers:** Models that output internal thought processes use `<think>` and `</think>` markers. The UI extracts and separately displays reasoning content from the final response. Browser (Wllama) thinking models are wrapped in these markers automatically, so their chain of thought shows up the same way.
203
 
204
  ### Privacy Settings
205