Split the 2.32GB Q4 data file into 2 chunks (~1.16GB each) to stay
under the 2GB protobuf limit. Transformers.js v4 supports multi-chunk
external data via config.json use_external_data_format field.
Restored original fine-tuned prompt format (/no_think Expand this...).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The 1.7B model's Q4 (2.3GB) exceeds protobuf 2GB single-file limit,
causing ONNX session creation to hang in browser. Qwen3-0.6B Q4 is
875MB self-contained — same architecture that works for the reranker.
Updated expansion prompt for generic model (not fine-tuned for task).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
fix: add timeout for expansion model loading, rebuild
Expansion model hangs during ONNX session creation (external data).
Add 2-minute timeout so it fails gracefully instead of blocking.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
fix: use self-contained ONNX models, sequential loading, Vite config
- Switch embedding model to shreyask/embeddinggemma-300m-ONNX (self-contained,
no external .onnx_data files that fail in ONNX Runtime Web)
- Load models sequentially instead of parallel to isolate errors and
reduce peak GPU memory usage
- Add optimizeDeps.exclude for @huggingface/transformers in Vite config
- Add @webgpu/types for TypeScript WebGPU API support
- Remove unused chunks field from PipelineInput
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add dtype: "q4" to embedding and reranker (browser ONNX Runtime
doesn't support external data files, Q4 variants are single-file)
- Make expansion model optional (2.2GB Q4 has external data, too large
for single file). Pipeline falls back to original query when expansion
model isn't available.
- Add isExpansionReady() export
- Debounce progress callbacks to fix flickering progress bars
- Revert unnecessary Vite optimizeDeps exclusion
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
add pipeline orchestrator as AsyncGenerator yielding stage events
Ties together query expansion, BM25/vector search, RRF fusion,
reranking, and score blending into a single async pipeline that
React can subscribe to for progressive UI updates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat: add RRF fusion and position-aware score blending
Implement reciprocal rank fusion (rrf.ts) to merge BM25 and vector
search ranked lists with weighted contributions and rank bonuses.
Add position-aware score blending (blend.ts) that combines RRF scores
with reranker scores using tiered weights (top3/mid/tail).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
add reranking module using Qwen3-Reranker yes/no token scoring
Scores candidate chunks by computing softmax over yes/no token logits
from the reranker model. Includes buildRerankerPrompt, scoreDocument,
and rerankChunks functions with tests for prompt construction.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
add query expansion module with lex/vec/hyde parsing
Implements expandQuery() which runs the expansion model and
parseExpansionOutput() which extracts structured lex, vec, and hyde
variants from the model's text output, with fallback to raw text.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Embeddings module wraps the embeddinggemma pipeline for query, batch,
and document chunk embedding using Transformers.js feature-extraction.
Vector search implements cosine similarity ranking over embedded chunks.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
fix: use AutoModelForCausalLM for reranker (yes/no token scoring)
Matches the working pattern from shreyask/Qwen3-Reranker-0.6B-Web.
The reranker is a causal LM that scores query-document pairs by
comparing logits for "yes" vs "no" tokens, not a text-classification
pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
add model loading layer for Transformers.js WebGPU pipelines
Singleton loaders for embedding (embeddinggemma-300m), reranker
(Qwen3-Reranker-0.6B), and query expansion (1.7B Q4) models using
@huggingface/transformers v4 with WebGPU backend. Includes progress
reporting, WebGPU detection, and parallel loading via loadAllModels.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements tokenizer (lowercase, split on non-alphanumeric, filter <2 chars)
and BM25Index class with standard BM25 scoring (k1=1.2, b=0.75). Includes
15 tests covering tokenization, ranking, edge cases, and score ordering.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
add chunking module with markdown-aware token-based splitting
Ports QMD's chunking algorithm: 900-token chunks (~3600 chars) with 15%
overlap, heading-preferred break points, code fence protection, and
squared distance decay for cut position scoring. Includes 23 vitest tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
