thinletter — a query encoder as a 120 MiB file, in the browser, open source

We take an embedding model, leave its document index exactly as it is, and compile the query side into a file a tenth of the size that answers in the browser in about 100 ms. Everything behind the numbers is public under Apache-2.0: the quantiser, the container compiler, the WebGPU runtime, the verification recipe and every released file. Web: thinletter.io · code and report: github.com/rosecky/embedding-quantization-public · demo: thinletter.io/demo or the Space thinletter/demo · contact: info@thinletter.io

Quickstart: a released client in your page

import { loadVqwClient } from 'https://thinletter.io/vqweb/vqweb.js';   // or: npm install @thinletterio/vqweb

const client = await loadVqwClient('https://huggingface.co/thinletter/qwen3-embedding-0.6b-vq-clients/resolve/main/qwen3-0.6b-vq3.5d2-webfaq_q-vocfull.vqw');
const q = await client.embed('Kolik stojí parkování v centru Prahy?');   // Float32Array(1024), L2-normalised: dot it with your Qwen3-Embedding-0.6B index

@thinletterio/vqweb (npm, Apache-2.0) is the open WebGPU runtime as a package: download with progress, a cache in the browser's private file system, tokenizer and prompt from the container header, pipelines, warm-up. Needs WebGPU with shader-f16 (Chrome / Edge), no CPU fallback; about 100 ms per query on an integrated GPU. The scalar GGUF files run with wllama or llama.cpp.

What holds

Repositories

Query-side clients: each file is a query encoder for exactly one document encoder and its settings (the model card says which). The index does not change.

Each model repository has a summary post in its Community tab with the numbers, what was found along the way, and the limits. The .vqw files load with @thinletterio/vqweb (above); the GGUF files with llama.cpp / wllama.

How we measure

Test split, the model's own fp32 index, native runtime (llama.cpp) or the browser. Every run is pre-registered with a prediction and a kill rule; comparisons change one thing at a time at the same calibration budget; differences are paired bootstrap intervals over queries (10 000 draws) read against the variance of the calibration draw (~0.01 nDCG@10) and between machines (±0.006): under 0.01 is a tie. Negative results stay published (a codebook-budget idea that did not work, a 2.6-bit headline that held on one corpus only, the 2-bit cliff of Qwen3-Embedding: 65–81 % for the same run on two machines).

Verify on your own index

The public repository has the recipe: import your corpus and existing document vectors, generate synthetic queries, quantise (llama-quantize, the GPTQ exporter, or the vector quantiser + .vqw compiler), evaluate the client against the unchanged index with a paired interval. Everything in the report is reproducible from it; only the raw per-run results, the research log and a customer's data stay out.

Licences

Code Apache-2.0 (NOTICE lists wllama MIT, @huggingface/tokenizers Apache-2.0, a Q2_K decoder ported from llama.cpp MIT). Weights under their base model's licence (MIT / Apache-2.0). Demo corpora: SciDocs (BEIR, CC BY 4.0), WebFAQ-cs (CC BY 4.0). Files derived from non-commercial bases (jina-embeddings-v5, CC BY-NC) are reported as numbers only.