compiled artefacts from win_x64 (sm_120)

.gitattributes

@@ -40,3 +40,13 @@ code-daemon-embed-v1-m_win_x64_trt_sm_120.engine filter=lfs diff=lfs merge=lfs -
 code-daemon-embed-v1-s_b64_s40_win_x64_tvm_vulkan.dll filter=lfs diff=lfs merge=lfs -text
 code-daemon-embed-v1-s_win_x64_trt_sm_120.engine filter=lfs diff=lfs merge=lfs -text
 model.onnx.data filter=lfs diff=lfs merge=lfs -text
+code-daemon-embed-v1-l_linux_x64_trt11.0_sm_120.engine filter=lfs diff=lfs merge=lfs -text
+code-daemon-embed-v1-m_linux_x64_trt11.0_sm_120.engine filter=lfs diff=lfs merge=lfs -text
+code-daemon-embed-v1-s_linux_x64_trt11.0_sm_120.engine filter=lfs diff=lfs merge=lfs -text
+code-daemon-embed-v1-xl_linux_x64_trt11.0_sm_120.engine filter=lfs diff=lfs merge=lfs -text
+code-daemon-embed-v1-l_win_x64_trt11.0_sm_120.engine filter=lfs diff=lfs merge=lfs -text
+code-daemon-embed-v1-m_win_x64_trt11.0_sm_120.engine filter=lfs diff=lfs merge=lfs -text
+code-daemon-embed-v1-s_win_x64_trt11.0_sm_120.engine filter=lfs diff=lfs merge=lfs -text
+code-daemon-embed-v1-xl_win_x64_trt11.0_sm_120.engine filter=lfs diff=lfs merge=lfs -text
+code-daemon-embed-v1_linux_x64_tvm0.25_vulkan.so filter=lfs diff=lfs merge=lfs -text
+code-daemon-embed-v1_win_x64_tvm0.25_vulkan.dll filter=lfs diff=lfs merge=lfs -text

README.md

@@ -11,6 +11,8 @@ tags:
 - feature-extraction
 - sentence-similarity
 - knowledge-distillation
+- quantized
+- int8
 pipeline_tag: feature-extraction
 base_model:
 - intfloat/multilingual-e5-base
@@ -24,175 +26,484 @@ datasets:
 
 # code-daemon-embed-v1
 
-A small, fast **code-embedding model** for semantic code search over **real codebases**. It maps short
-code units — function and method bodies, signatures, docstrings — and short natural-language queries
-into a shared **768-dim** vector space.
-
-It is **specialized for real retrieval traffic, not benchmark prose**: the training queries mirror how
-agents and developers actually search a repository (short keyword-bag and behavior queries), and the
-maximum sequence is **128 tokens**, trading long-context capability for high throughput and strong
-quality on short units.
-
-- **768-dim** embeddings (full width; no MRL truncation in this release).
-- **~75.7M params** — XLM-RoBERTa architecture initialized from `multilingual-e5-base` with
-  **strided 12→8 layer truncation** and the vocabulary **pruned 250k → 22,739** pieces
-  (English + code coverage kept; the transformer layers see identical tokenizations for EN/code text).
-- **A built-in reranker** — see below: cross-encoder ranking knowledge is distilled *into* the
-  embedding space, so the top of the ranking behaves like a reranked list at bi-encoder cost.
-- **Mean pooling fused into the graph** — the output is already pooled (`[batch, 768]`); just **L2-normalize**.
-- Trained at sequence length **128** (length buckets s / m / l = seq **40 / 64 / 128**).
-
-## The built-in reranker (why the top of the ranking is sharp)
-
-Classic bi-encoders compress each document into a vector *before* seeing the query, so they cannot do
-the fine-grained pairwise comparison a cross-encoder reranker does. This model closes part of that gap
-at training time instead of inference time:
-
-- Hard negatives were mined for every training query (documents a strong retriever ranks high but wrong).
-- **`Qwen/Qwen3-Reranker-4B`** (a cross-encoder) scored every (query, candidate) pair with graded
-  relevance logits.
-- The student was trained with a **listwise-KL objective** to reproduce the reranker's *ranking
-  distribution* — not just "positive above negatives", but *how much* each near-miss should trail.
-
-The result: reranker-flavored top-rank precision with **zero inference overhead** — no second model, no
-second pass, no extra latency. In the production system this model was built for, enabling an actual
-runtime reranker on top of it measured as *net-negative*: the distillation had already captured the
-useful part.
-
-## Where it's good — and where it isn't
-
-**Strong (the design target):**
-- **Real repository search** — short NL/keyword queries by developers and coding agents over an
-  indexed codebase ("git watcher head change reindex", "acquire database lock for project hash").
-- **Natural-language → code** at short lengths; identifier and mixed queries.
-- Complementing a lexical (BM25-style) channel in a hybrid retriever — it was trained *for* that role.
-
-**Weak / out of scope:**
-- **Long documents** — hard 128-token cap; not a long-context retriever.
-- **Benchmark-style long problem statements** (competitive-programming prompts, multi-turn dialogue,
-  code↔code translation) — deliberately out of scope, see the evaluation note.
-- **General English prose** (medical / financial / news) — the pruned vocab and code specialization
-  trade general-text quality away.
-
-Embed **queries and documents the same way** — no instruction prefix.
-
-## How it was made
-
-Three-teacher pipeline on a pretrained backbone (the full recipe and every ablation is documented in
-the source project):
-
-1. **Backbone**: `intfloat/multilingual-e5-base` (12L XLM-R retriever), **strided-truncated to 8 layers**
-   (keep {0,1,3,5,7,9,10,11}) with the embedding table pruned and re-indexed to a 22,739-piece
-   SentencePiece vocabulary in a raw-SP convention (pad=0 / unk=1 / bos=2 / eos=3).
-2. **Dense representation distillation** from
-   **[`Alibaba-NLP/gte-modernbert-base`](https://huggingface.co/Alibaba-NLP/gte-modernbert-base)**
-   (MSE to the teacher's embeddings) over real repository entities in a front-loaded serve format
-   (semantics before identifiers, within the 128-token budget).
-3. **Listwise-KL ranking distillation** from
-   **[`Qwen/Qwen3-Reranker-4B`](https://huggingface.co/Qwen/Qwen3-Reranker-4B)** cross-encoder scores
-   over mined hard negatives (12 kept per query) — the "built-in reranker".
-4. **Diversity regularization**: ~26% of the grouped stream is out-of-domain code-retrieval pairs
-   (CoIR-derived). Measured to be the sweet spot: 0% loses deep recall, 50% dilutes the top.
-   Counter-intuitively, the *dissimilarity* of these rows is the active ingredient.
-
-Training queries were generated few-shot in the style of **real agent search traffic** (short
-keyword-bags, behavior descriptions, identifier fragments) with identifier-leak gating — not docstring
-paraphrases. This is the main reason benchmark numbers below *understate* real-repo performance.
-
-## Built for speed
-
-- **Short context by design** — max **128 tokens**; the engines avoid wide dynamic-shape ranges.
-- **Rectangular TensorRT profiles** — each length bucket is a *fixed* shape (min == opt == max):
-  **s** = batch 64 × seq 40 · **m** = batch 128 × seq 64 · **l** = batch 256 × seq 128.
-- **Mean-pool + L2-friendly head fused into the graph** (one pass → `[B, 768]`).
-- FP32/FP16 engines in this release (INT8 PTQ measurably hurts this checkpoint's top-rank precision;
-  a QAT INT8 variant is planned).
-
-## Usage (standalone ONNX)
-
-The FP32 `model.onnx` is bundled. Tokenize with the bundled `sentencepiece.bpe.model`, run, and the
-pooled `[B, 768]` output is already produced — just L2-normalize:
+A **46.8M-parameter, 4-layer code-embedding model** that maps short code units — function
+and method bodies, signatures, docstrings, symbol names — and short natural-language queries
+into a shared **768-dim** space. INT8, 128-token window, ~10.7k embeddings/sec on a laptop
+RTX 5060.
+
+It is built for one job: **embedding a whole repository fast enough to re-index it on every
+commit**, so that a coding agent can run semantic search on every question it is asked. Every
+trade-off below follows from that — the depth, the 22.7k vocabulary, the 128-token cap, the
+INT8 weights, the four length-bucketed engines.
+
+```python
+# the whole API surface — pooled AND L2-normalized inside the graph
+vec = session.run(None, {"input_ids": ids, "attention_mask": mask})[0]  # [B, 768], ready
+```
+
+Queries and documents are embedded **the same way** — no `query:` / `passage:` prefix.
+
+---
+
+## 1. What it is for, and why not a standard embedder
+
+### The target workload
+
+1. A repository is indexed: every function, method, type and doc chunk becomes one short text
+   and one vector. A 700k-entity C++ codebase embeds in **~81 seconds** on one consumer GPU.
+2. Someone asks something short and keyword-shaped — *"git watcher head change reindex"*,
+   *"acquire database lock for project hash"*, *"where does the daemon start"*.
+3. The vector channel runs next to a lexical (BM25-style) channel and the two are fused.
+
+Point 3 matters: this model was trained to be **the dense half of a hybrid retriever**, not
+to win alone. Its training queries imitate real captured agent traffic — short keyword bags,
+behaviour descriptions, identifier fragments — rather than docstring paraphrases.
+
+### Design choices vs a typical general-purpose text embedder
+
+| | typical embedder (e5 / bge / gte class) | code-daemon-embed-v1 |
+|---|---|---|
+| Parameters | 110M – 7B | **46.8M** |
+| Vocabulary | 30k – 250k, general text | **22.7k**, pruned to code + code-adjacent English |
+| Max sequence | 512 – 8192 tokens | **128** (hard cap, by design) |
+| Query format | often needs an instruction prefix | **none** — symmetric |
+| Pooling | you implement it | **inside the graph** — output is already `[B, 768]` |
+| Weights | FP32 / FP16 | **INT8** from quantization-aware training |
+| Top-rank precision | needs a separate cross-encoder pass | **cross-encoder ranking distilled into the vectors** |
+
+### The built-in reranker
+
+A bi-encoder compresses each document into a vector *before* it sees the query, so it cannot
+do the pairwise comparison a cross-encoder reranker does. This model closes part of that gap
+at **training** time instead of inference time: `Qwen/Qwen3-Reranker-4B` scored every
+(query, candidate) pair over mined hard negatives, and the student was trained with a
+**listwise-KL** objective to reproduce the teacher's *ranking distribution* — not merely
+"positive above negative", but how much each near-miss should trail.
+
+Practical consequence: **you probably do not need a runtime reranker on top of it.** In the
+production system it was built for, switching one on measured net-negative — the distillation
+had already captured the useful part, at zero inference cost.
+
+### Strong / weak
+
+**Strong**
+- Repository search with short NL, keyword and identifier queries.
+- Natural language → code at short lengths.
+- The dense channel of a hybrid retriever.
+- Throughput-bound work: bulk re-index, index-on-save, re-embed per commit.
+
+**Weak / out of scope**
+- **Long documents.** Hard 128-token cap. Not a long-context retriever (see the window
+  recipe in §4.6 if you must handle longer text).
+- Benchmark-style long problem statements, multi-turn dialogue, code↔code translation.
+- General English prose (medical / financial / news) — the pruned vocabulary trades that
+  away deliberately.
+
+---
+
+## 2. Architecture — what was kept, cut and added
+
+### Specification
+
+| | |
+|---|---|
+| Base | `intfloat/multilingual-e5-base` (XLM-RoBERTa encoder, 12L, 278M) |
+| Encoder layers | **4** |
+| Hidden size | 768 |
+| Attention heads | 12 |
+| FFN | 3072, GELU |
+| LayerNorm eps | 1e-5 |
+| Vocabulary | **22,739** SentencePiece **unigram** pieces, byte fallback |
+| Position embeddings | 514 allocated (learned, absolute) |
+| Max sequence used | **128 tokens** |
+| Output | **768-dim, mask-mean-pooled AND L2-normalized inside the graph** — use it as it comes |
+| Parameters | **46.80M** = 17.46M embedding table + 4 × ~7.3M encoder + positions |
+| Weights | **INT8**, quantization-aware training, Q/DQ nodes carry the trained scales |
+| ONNX | opset 19; inputs `input_ids`, `attention_mask`, both **int64** `[batch, seq]` |
+| Special token ids | **pad=0, unk=1, bos=2, eos=3** (raw-SentencePiece indexing) |
+
+### Removed from the base model
+
+- **8 of 12 encoder layers** — strided-truncated 12 → 8 → 6 → **4**, healed after each cut.
+- **227k of 250k vocabulary pieces**: 250,002 → 22,739, and the embedding table with them
+  (192M → 17.5M parameters — the single largest saving in the model).
+- **The E5 instruction prefixes.** No `query:` / `passage:` asymmetry; both sides are encoded
+  identically.
+- **The pooler head, the MLM head, token-type inputs.** The graph has exactly two inputs.
+- **FP32 weights** — replaced by INT8 with trained scales.
+
+### Added
+
+- **Mask-aware mean pooling and L2 normalization fused into the ONNX graph.** The model
+  returns unit-norm `[B, 768]`, not `[B, seq, 768]` — there is no pooling code to get wrong,
+  no `last_hidden_state` copy, and no way to accidentally compare unnormalized vectors.
+- **Q/DQ nodes with QAT-trained scales**, so a TensorRT/OpenVINO build is INT8 end-to-end
+  without a calibration pass.
+- **Ranking knowledge from a 4B cross-encoder** (the built-in reranker above).
+- **Four length-bucketed engine builds** with a dynamic sequence dimension (§3, §4.4).
+
+### Training recipe, in brief
+
+1. Strided depth truncation of the e5 backbone, vocabulary pruning, re-indexing to the
+   raw-SentencePiece id convention.
+2. **Dense representation distillation** from `Alibaba-NLP/gte-modernbert-base` (MSE against
+   the teacher's embeddings) over real repository entities.
+3. **Listwise-KL ranking distillation** from `Qwen/Qwen3-Reranker-4B` over mined hard
+   negatives.
+4. **Diversity regularization** — ~26% of the grouped training stream is out-of-domain
+   code-retrieval pairs (CoIR-derived). Measured sweet spot: 0% loses deep recall, 50%
+   dilutes the top.
+5. **Self-distillation QAT** to INT8, exported with the scales learned during training.
+
+### If you re-train, cut or re-quantize it
+
+Two results that will cost you a training run if you rediscover them yourself:
+
+- ⚠ **Do not run PTQ/calibration over the shipped INT8 graph.** It overwrites the trained
+  scales with fitted ones and measurably degrades the model (cosine .93 against the correct
+  export). `model_int8qdt.onnx` *is* the artifact — feed it to your builder as-is. Plain
+  post-training quantization of this body was also tried on its own and is quality-broken:
+  engine hit@1 fell .200 → .133, which is why the shipped weights come from QAT.
+- ⚠ **Heal a depth cut on documents *and* queries.** Re-distilling the 4-layer student
+  against document texts only cost **−29.8% hit@1**; the same initialization and budget
+  against documents *and* queries cost **−3.9%** on the same 2,441-query gate. Retrieval
+  lives in the *relation* between the two sides — a document-only anchor lets the query side
+  drift.
+
+---
+
+## 3. Performance
+
+Measured on **one RTX 5060 Laptop GPU (sm_120, 8 GB)**, TensorRT INT8, pinned TDP,
+`clocks.sm` 2,647 MHz median at 114 W.
+
+### End-to-end, inside the production indexer
+
+The whole path: build the serve text → tokenize on host threads → IPC to the worker →
+GPU forward → pool → collect.
+
+| corpus | vectors | wall | pipeline emb/s | GPU-only emb/s |
+|---|--:|--:|--:|--:|
+| **mysql-server** (sustained, 54 batches) | 868,795 | 80.96 s | **10,731** | 11,712 |
+
+`pipeline emb/s` = vectors ÷ wall clock, everything included. `GPU-only` = vectors ÷ summed
+inference time; it is higher because host work overlaps with the GPU.
+
+### Solo engine profiles — cost per sequence length
+
+Four engines, one per length bucket, each with a **dynamic sequence dimension** so a batch
+can be dispatched at its own longest text instead of the bucket ceiling. One engine alone,
+dispatch only, 200 iterations:
+
+| bucket | profile (batch × seq) | seq | µs/text | texts/s |
+|---|---|--:|--:|--:|
+| **s** | 96 × 8…48, opt 48 | 8 | 13.78 | 72,587 |
+| | | 16 | 19.73 | 50,681 |
+| | | 24 | 26.86 | 37,228 |
+| | | 32 | 33.73 | 29,651 |
+| | | 40 | 42.56 | 23,495 |
+| | | 48 | 50.54 | 19,785 |
+| **m** | 128 × 56…64, opt 56 | 56 | 63.08 | 15,853 |
+| | | 64 | 76.52 | 13,068 |
+| **l** | 128 × 72…80, opt 72 | 72 | 88.48 | 11,302 |
+| | | 80 | 100.87 | 9,913 |
+| **xl** | 256 × 88…128, opt 96 | 88 | 117.14 | 8,537 |
+| | | 96 | 127.95 | 7,816 |
+| | | 104 | 142.31 | 7,027 |
+| | | 112 | 155.74 | 6,421 |
+| | | 120 | 169.28 | 5,907 |
+| | | 128 | 183.03 | 5,464 |
+
+Three things to take from that table when you build your own serving path:
+
+- **Cost is per token, not per text**, and it is superlinear: solo token throughput *falls*
+  from 1.01M tok/s at seq 48 to 0.71M at seq 128. Batch size is not the lever — a 256×64
+  engine measured **5.8% slower per text** than 128×64, because 96×48 already saturates the
+  SMs.
+- **Padding is the lever.** A batch pays for its *longest* member, so sorting texts by length
+  before batching — and dispatching each batch at its own length — was worth **+10.2%**
+  end-to-end. Round lengths up to a multiple of 8: a non-multiple is slower than a *longer*
+  multiple (seq 70 costs more than seq 72).
+- **Running the four engines concurrently does not add throughput** on one GPU. In parallel
+  they measured *slower* than serially (1,487 ms vs 1,423 ms for the same work). The lanes
+  exist to keep the GPU fed while the host works, not to multiply throughput.
+
+### Without a discrete GPU — OpenVINO on CPU, integrated GPU and NPU
+
+Same model, no NVIDIA card involved. Measured on an **Intel Core Ultra 9 275HX** — its CPU
+cores, its integrated GPU, and its *AI Boost* NPU — with OpenVINO 2026.2. An OpenVINO IR is
+reshaped to **one static shape** at build time (unlike the TensorRT engines, whose sequence
+dimension is a range), so each bucket is a single number per device:
+
+| device | precision | bucket | shape (batch × seq) | µs/text | texts/s |
+|---|---|---|---|--:|--:|
+| **CPU** | INT8 | s | 96 × 48 | 1,837 | 544 |
+| | | m | 128 × 64 | 2,128 | 470 |
+| | | l | 128 × 80 | 2,542 | 393 |
+| | | xl | 256 × 128 | 4,223 | 237 |
+| **iGPU** | INT8 | s | 96 × 48 | 888 | **1,126** |
+| | | m | 128 × 64 | 1,168 | 856 |
+| | | l | 128 × 80 | 1,498 | 667 |
+| | | xl | 256 × 128 | 2,708 | 369 |
+| **NPU** | INT4 | s | 16 × 48 | 1,784 | 560 |
+| | | m | 16 × 64 | 2,334 | 428 |
+| | | l | 16 × 80 | 3,350 | 299 |
+| | | xl | 16 × 128 | 5,931 | 169 |
+
+Reference point: the same buckets on the RTX 5060 run at 19,785 / 13,068 / 9,913 / 5,464
+texts/s, so the integrated GPU is **15–18× slower** than the discrete one. That still leaves
+a 100k-entity repository indexed in a few minutes on a machine with no dedicated GPU at all.
+
+Three notes if you deploy this path:
+
+- **INT8 on CPU and iGPU, INT4 only on the NPU.** At 128 × 64 the iGPU does 859 texts/s in
+  INT8, 584 in FP16 — and **584 in INT4**, exactly the FP16 number. INT4 here is weight-only
+  compression: the weights decompress to fp16 before the GEMM, so it is the same GEMM and the
+  4 bits buy file size, not speed. Only INT8, which the kernels execute natively, moves the
+  number. The NPU is the exception because it requires INT4 with static shapes.
+- **Feed it the Q/DQ graph, not a calibrated one.** OpenVINO keeps the QAT scales as
+  FakeQuantize; re-fitting them with a post-training pass measured hit@1 .200 → .133 on this
+  model. See §4.5 for the two-line recipe (and the transformation that keeps the IR at 96 MB
+  instead of 179 MB).
+- **The NPU pays for batch, not for tokens.** Its IRs are baked at batch 16 against 96–256
+  elsewhere, which is why its per-text cost is the flattest across buckets and its per-batch
+  latency the lowest (28–95 ms). It suits interactive single-query embedding better than bulk
+  indexing.
+
+### Retrieval quality
+
+**80 captured real agent queries** over a live indexed codebase, multi-positive, file-level,
+through a production hybrid retriever (dense + lexical, runtime reranker **off**):
+
+| metric | value |
+|---|--:|
+| hit@1 | **0.46** |
+| hit@3 | 0.74 |
+| hit@5 | **0.78** |
+| hit@10 | 0.80 |
+| mrr@10 | 0.59 |
+| ndcg@10 | 0.61 |
+
+This is the metric the model is optimized for. On a broader 2,441-query gate across four
+unrelated repositories, the 4-layer model trades about **4% relative hit@1** against its
+6-layer parent for roughly **1.5× the engine speed**.
+
+### CoIR — the out-of-domain reference
+
+Run on **this exact artifact** — `model_int8qdt.onnx`, full corpora (2.3M documents across the
+six tasks), NDCG@10, the same raw-SentencePiece tokenization the daemon uses:
+
+| CoIR task | NDCG@10 | Pattern |
+|---|--:|---|
+| synthetic-text2sql | **51.26** | NL → SQL |
+| stackoverflow-qa | 41.63 | short question → code |
+| codesearchnet (6-lang avg) | 39.02 | docstring / NL → code |
+| codefeedback-st | 38.55 | NL instruction → code |
+| codesearchnet-ccr (6-lang avg) | 36.07 | code → related code |
+| cosqa | 17.50 | NL question → code (noisy / hard) |
+| **Average** | **37.34** | |
+
+Per language — codesearchnet (NL→code): python **68.30**, go 46.15, java 34.64, ruby 28.76,
+js 28.33, php 27.95. codesearchnet-ccr (code→code): js **43.36**, ruby 42.88, java 36.82,
+php 31.59, go 31.06, python 30.68.
+
+**Read this as a lower bound, not as the headline.** CoIR queries are mostly docstrings and
+long problem statements — the opposite of what this model was tuned for, and the spread proves
+it: 68.30 on Python docstring→code against 17.50 on cosqa's noisy question→code. Four of
+CoIR's ten tasks (code↔code translation, multi-turn dialogue, long problem statements) exceed
+the 128-token scope and are not shown. ~2k CoIR-derived rows were present in training as
+diversity regularization; overlap with the test splits was not audited.
+
+---
+
+## 4. Using the model
+
+### 4.1 What is in this repository
+
+| File | What it is |
+|---|---|
+| `model_int8qdt.onnx` | **The source of truth.** INT8 Q/DQ graph with QAT-trained scales — what the TensorRT and OpenVINO INT8 engines are built from. |
+| `model.onnx` | The FP32 twin of the same weights, for lanes that cannot read Q/DQ and for fine-tuning. Derived from the file above by dropping its Q/DQ pairs, so the weights are identical (cosine 0.98 between them is the quantization, not a different model). |
+| `sentencepiece.bpe.model` | The tokenizer. Raw-SP ids: pad=0, unk=1, bos=2, eos=3. Its 22,739 pieces match the model's embedding table row for row. |
+| `tokenizer_config.json`, `config.json` | HF-side metadata. |
+| `code-daemon-embed-v1-{s,m,l,xl}_{win_x64,linux_x64}_trt11.0_sm_{75,86,89,120}.engine` | Prebuilt TensorRT engines. |
+| `code-daemon-embed-v1-{s,m,l,xl}_ov2026.2_{cpu_int8,igpu_lnl_int8,igpu_arc_int4,npu_int4}_b*_s*.{xml,bin}` | Prebuilt OpenVINO models (Intel CPU / iGPU / NPU). |
+| `code-daemon-embed-v1_{win_x64,linux_x64}_tvm0.25_vulkan.{dll,so}` | TVM Vulkan module (vendor-neutral GPU fallback). |
+| `model_gpu_mlx0.22/` | MLX weights for Apple Silicon. |
+
+> Compiled artifacts are uploaded progressively per architecture. The ONNX + tokenizer are
+> the always-present source of truth; if the engine you want is missing, build it — §4.4.
+
+### 4.2 Taking a prebuilt engine
+
+The filenames are a loading contract, not decoration. A serialized TensorRT plan is keyed on
+**{GPU architecture × OS × TensorRT version}** and `deserializeEngine` has no compatibility
+fallback, so pick all four coordinates exactly:
+
+```
+code-daemon-embed-v1-m_win_x64_trt11.0_sm_120.engine
+                     │  │       │       └── GPU arch: sm_75 Turing · sm_86 Ampere (RTX 30xx, A-series)
+                     │  │       │           sm_89 Ada (RTX 40xx, L4) · sm_120 Blackwell (RTX 50xx)
+                     │  │       └── TensorRT 11.0 — not interchangeable with 10.x
+                     │  └── OS/ABI
+                     └── length bucket: s | m | l | xl
+```
+
+Bucket shapes (batch × seq): **s** 96×48, **m** 128×64, **l** 128×80, **xl** 256×128. Loading
+all four costs ~400 MB of VRAM; if you only want one, take **m** — it covers the fattest part
+of a typical corpus (25–40% of texts land in 49…64 tokens).
+
+Route each text to the first bucket whose sequence ceiling fits its token count, and pad the
+batch to that bucket's shape. Padding is masked in attention, so a padded batch and an
+unpadded one give the same vector (verified at cosine 1.000000) — provided the mask is right.
+
+### 4.3 Running the ONNX directly
+
+Works anywhere ONNX Runtime does (CPU, CUDA, DirectML) with no build step:
 
 ```python
 import onnxruntime as ort, sentencepiece as spm, numpy as np
 
 sp   = spm.SentencePieceProcessor(model_file="sentencepiece.bpe.model")  # pad=0 unk=1 bos=2 eos=3
-sess = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])
+sess = ort.InferenceSession("model_int8qdt.onnx", providers=["CPUExecutionProvider"])
 
 def embed(texts, max_len=128):
-    ids  = [[2, *sp.encode(t)[: max_len - 2], 3] for t in texts]          # bos … eos
+    ids  = [[2, *sp.encode(t)[: max_len - 2], 3] for t in texts]            # bos … eos
     L    = max(len(x) for x in ids)
-    inp  = np.array([x + [0] * (L - len(x)) for x in ids], dtype=np.int64) # pad=0
+    inp  = np.array([x + [0] * (L - len(x)) for x in ids], dtype=np.int64)  # pad=0
     mask = (inp != 0).astype(np.int64)
-    out  = sess.run(None, {"input_ids": inp, "attention_mask": mask})[0]   # already pooled [B,768]
-    return out / np.linalg.norm(out, axis=1, keepdims=True)
+    # pooled and unit-norm already — no post-processing
+    return sess.run(None, {"input_ids": inp, "attention_mask": mask})[0]    # [B, 768]
+
+D = embed(["function acquireLock in src/db.zig: zig\npath: src db"])
+Q = embed(["acquire database lock"])
+print(Q @ D.T)          # inner product IS cosine here
 ```
 
-## What's in this repo
+Two notes that cost real debugging time:
+
+- **Tokenize with SentencePiece, not a greedy-BPE merge loop.** The vocabulary is a
+  **unigram** model; scoring it with pair-merge BPE produces a *different segmentation* than
+  training used, and that train/serve skew silently costs retrieval quality.
+- **Pad with id 0 and mask 0.** Both inputs are `int64`; feeding int32 buffers to a
+  hand-written runtime is the classic "second half of every batch is garbage" bug.
+
+### 4.4 Building your own TensorRT engines
+
+TensorRT 11 reads precision **entirely from the ONNX Q/DQ nodes** — the per-precision flags
+(`--int8`, `--fp16`, `--calib`) were removed. Feed it the Q/DQ graph and pass
+`--stronglyTyped`. Feed it an unquantized graph instead and you get an FP16 engine that builds
+cleanly, loads cleanly, is twice the size and a fraction of the speed — the size is the only
+visible symptom, so check it.
+
+Rectangular engine for one bucket (here **m**, 128×64):
+
+```bash
+trtexec --onnx=model_int8qdt.onnx \
+        --saveEngine=code-daemon-embed-v1-m.engine \
+        --stronglyTyped \
+        --builderOptimizationLevel=5 \
+        --timingCacheFile=timing.cache \
+        --minShapes=input_ids:1x1,attention_mask:1x1 \
+        --optShapes=input_ids:128x64,attention_mask:128x64 \
+        --maxShapes=input_ids:128x64,attention_mask:128x64
+```
 
-Named per **runtime × GPU arch × OS × length-bucket** (the filenames are a loading contract — do not rename):
+Dynamic-sequence engines (the shipped configuration, +10.2% end-to-end) keep the batch
+dimension rectangular and let `seq` range. Use these profiles:
 
-- **TensorRT** `*.engine` — NVIDIA:
-  `code-daemon-embed-v1-{s,m,l}_{win_x64,linux_x64}_trt_sm_{86,89,120}.engine`
-  (sm_86 ≈ RTX 30xx / A-series · sm_89 ≈ RTX 40xx / L4 · sm_120 ≈ RTX 50xx).
-- **TVM** `*_tvm_vulkan.{dll,so}` — Vulkan fallback for non-TRT GPUs, per bucket.
-- **OpenVINO** `*.xml` + `*.bin` — Intel **CPU / iGPU / NPU**, per bucket.
-- **Metal** `*_tvm_metal.*` — Apple Silicon (macOS), per bucket.
-- **Tokenizer** — `sentencepiece.bpe.model` (raw-SP: pad=0 / unk=1 / bos=2 / eos=3, byte-fallback) +
-  `tokenizer_config.json`.
-- **ONNX source** — `model.onnx` (+ `model.onnx.data`) FP32.
+| bucket | batch | seq min | seq **opt** | seq max |
+|---|--:|--:|--:|--:|
+| s | 96 | 8 | 48 | 48 |
+| m | 128 | 56 | 56 | 64 |
+| l | 128 | 72 | 72 | 80 |
+| xl | 256 | 88 | 96 | 128 |
 
-> Compiled engines for this release are being rebuilt and uploaded progressively; the ONNX + tokenizer
-> are the always-present source of truth.
+Three rules behind those numbers:
 
-## Evaluation
+- **One dynamic engine per bucket, `opt` at that bucket's mean length** — never one global
+  dynamic engine. The tax is paid for being *far from the opt point*, not for being dynamic:
+  a single wide (16→64→128) engine costs +0.8% at seq 64 but **+15.4%** at seq 80.
+- **Sequence length must be a multiple of 8** at dispatch (seq 70 costs more than seq 72).
+- **Sort texts by length within a bucket and cut at batch boundaries** before dispatch.
+  Without the sort the dynamic engines buy ~0%: any 128-text batch drawn from the 49–64 range
+  almost surely contains a 64 and dispatches at 64 anyway.
 
-### Real-repository retrieval (the design target)
+A dynamic build reports ~69 layers against a rectangular build's ~54 — a wider shape range
+costs fusions, and that is exactly where the small tax at the ceiling lives.
 
-On a benchmark of **real agent search queries over a live indexed codebase** (80 captured queries,
-multi-positive, file-level), the production hybrid retriever built on this model reaches:
+### 4.5 OpenVINO, TVM, MLX
 
-| metric | value |
-|---|--:|
-| hit@5 | **0.73** |
-| hit@1 | 0.44 |
-| pool_recall@64 | 0.79 |
+OpenVINO reads the Q/DQ graph directly — the trained scales survive as FakeQuantize, so there
+is **no calibration pass to run**:
 
-This is the metric the model is optimized for — short real-world queries against a real repository.
+```python
+import openvino as ov
+from openvino._offline_transformations import compress_quantize_weights_transformation
 
-### CoIR (NDCG@10, full corpora) — out-of-domain reference
+m = ov.Core().read_model("model_int8qdt.onnx")
+compress_quantize_weights_transformation(m)                       # folds weights to i8: 179 MB -> 96 MB
+m.reshape({"input_ids": [128, 64], "attention_mask": [128, 64]})  # one bucket, static
+ov.save_model(m, "code-daemon-embed-v1-m_cpu.xml")
+```
 
-CoIR queries are largely docstrings and long problem statements — a different style from real repo
-search traffic — so treat these as an *out-of-domain lower bound*: **on real queries in real
-repositories the model performs substantially better than its CoIR positioning suggests** (see above).
-Four of CoIR's ten tasks — code↔code translation, multi-turn dialogue, long problem-statements —
-exceed the 128-token / retrieval scope and are not shown. (~2k CoIR-derived rows were present in
-training as diversity regularization; overlap with test splits was not audited.)
+That transformation is not optional bookkeeping: `read_model` leaves every weight as f32 behind
+a FakeQuantize, and the IR comes out nearly twice the size for the same arithmetic. Do **not**
+reach for NNCF post-training quantization here — it would replace the trained scales with
+fitted ones, which is the failure the warning in §2 describes.
 
-| CoIR task | NDCG@10 | Pattern |
-|---|--:|---|
-| synthetic-text2sql | **62.59** | NL → SQL |
-| stackoverflow-qa | 50.82 | short question → code |
-| codefeedback-st | 47.42 | NL instruction → code |
-| codesearchnet (6-lang avg) | 42.68 | docstring / NL → code |
-| codesearchnet-ccr (6-lang avg) | 40.95 | code → related code |
-| cosqa | 20.75 | NL question → code (noisy / hard) |
-| **Average** | **44.20** | |
+The NPU artifacts are the exception: they start from the FP32 twin and apply INT4 weight
+compression at batch 16, trading accuracy for size on purpose. TVM Vulkan modules and the MLX
+weights are also built from the FP32 twin, per bucket.
+
+### 4.6 Feeding it well
+
+**Documents.** The model was trained on a compact, front-loaded "serve text": semantics
+first, identifiers after, everything inside the 128-token budget. Reproducing that shape on
+your own corpus is worth more than any inference tuning:
+
+```
+{type} {name} in {file}: {lang}
+path: {directory tokens, space-separated}
+[async] [exported] [test]
+sig: ({params}) -> {return type}
+{doc comment, first ~200 chars}
+{one-to-two-sentence description}
+```
+
+for example:
+
+```
+function acquireProjectLock in src/storage/multi_db.zig: zig
+path: src storage multi db
+[exported]
+sig: (allocator: Allocator, project_hash: []const u8) -> !Lock
+Acquires the exclusive SQLite lock for one project.
+```
 
-Per language — codesearchnet (NL→code): python **66.80**, go 51.30, java 38.77, php 38.16, js 32.17, ruby 28.90.
-Per language — codesearchnet-ccr (code→code): js **47.72**, ruby 45.28, java 42.05, python 38.43, php 36.60, go 35.64.
+Cap the whole text around 768 characters; the raw function body is deliberately *not* part of
+it (it belongs in the lexical channel, where it measurably helps, not in the vector).
 
-## Performance (embeddings / sec)
+**Queries.** Feed them raw, no prefix, no template. The model is tuned for short keyword bags
+and behaviour descriptions.
 
-Being re-measured for this release's engines — table will be updated with the engine upload.
+**Text longer than 128 tokens.** Split into overlapping windows — window 128, stride 96 —
+embed each, then mean-pool the window vectors and L2-renormalize. That is what the production
+indexer does for long doc chunks.
+
+**Retrieval.** The vectors come out unit-norm, so inner product is cosine — no normalization
+step of your own. They are dense and 768-dimensional; an IVF/HNSW index over them behaves
+normally.
+
+---
 
 ## License & training data
 
 Released under the **MIT license**.
 
 The backbone (`intfloat/multilingual-e5-base`) is MIT; the teachers (`gte-modernbert-base`,
-`Qwen3-Reranker-4B`) are Apache-2.0. As is standard practice for distilled embedding models, the
-**weights are released under MIT**. Training corpus, for transparency:
+`Qwen3-Reranker-4B`) are Apache-2.0. As is standard practice for distilled embedding models,
+the **weights are released under MIT**. Training corpus, for transparency:
 
 | Data | License note |
 |---|---|
@@ -200,9 +511,6 @@ The backbone (`intfloat/multilingual-e5-base`) is MIT; the teachers (`gte-modern
 | CoIR-derived code-retrieval pairs (~26% diversity stream) | per-subset CoIR licenses (research benchmark) |
 | Teacher embeddings / scores (gte-modernbert, Qwen3-Reranker-4B) | Apache-2.0 teachers |
 
-Unlike the previous release, **no MS MARCO / mMARCO data** was used — the earlier non-commercial
-caveat no longer applies. This is **not legal advice**.
-
 ## Attribution
 
 Backbone: [intfloat/multilingual-e5-base](https://huggingface.co/intfloat/multilingual-e5-base) (MIT).

code-daemon-embed-v1-l_linux_x64_trt11.0_sm_120.engine

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4cc5df802763f42e5e4cd9385145cd4ec60812180d636b3b5ee5c705aaa4b273
+size 101848500

code-daemon-embed-v1-l_ov2026.2_cpu_int8_b128_s80.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2307f73831b76bb4508cd26ee910d30d68429d3c7fb74b1fce00fefc709460b1
+size 100617601

code-daemon-embed-v1-l_ov2026.2_igpu_lnl_int8_b128_s80.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2307f73831b76bb4508cd26ee910d30d68429d3c7fb74b1fce00fefc709460b1
+size 100617601

code-daemon-embed-v1-l_ov2026.2_npu_int4_b16_s80.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f52ac233153d8623bfc2453aec30974ba52bda423c573ef16200949c6d56c77d
+size 33285182

code-daemon-embed-v1-l_win_x64_trt11.0_sm_120.engine

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:27d928449594faa418f5172d9ecc31adcb3901ead546091ef96e019ea354c974
+size 101860876

code-daemon-embed-v1-m_linux_x64_trt11.0_sm_120.engine

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1b8728c79459b35bfdee18463ed809e8ba16a30e78569bb0ad89351b53c0e5cd
+size 101873372

code-daemon-embed-v1-m_ov2026.2_cpu_int8_b128_s64.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2307f73831b76bb4508cd26ee910d30d68429d3c7fb74b1fce00fefc709460b1
+size 100617601

code-daemon-embed-v1-m_ov2026.2_igpu_lnl_int8_b128_s64.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2307f73831b76bb4508cd26ee910d30d68429d3c7fb74b1fce00fefc709460b1
+size 100617601

code-daemon-embed-v1-m_ov2026.2_npu_int4_b16_s64.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f52ac233153d8623bfc2453aec30974ba52bda423c573ef16200949c6d56c77d
+size 33285182

code-daemon-embed-v1-m_win_x64_trt11.0_sm_120.engine

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1b0e8969cf89635d93bdaa8fc2e9a1600d27f6cca43bb27a3c6cb22a81b3004d
+size 101850012

code-daemon-embed-v1-s_linux_x64_trt11.0_sm_120.engine

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b65796f0f877d2f95b87c3016736a5f0753c97cf289fd5333346f94290fc5907
+size 101864196

code-daemon-embed-v1-s_ov2026.2_cpu_int8_b96_s48.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2307f73831b76bb4508cd26ee910d30d68429d3c7fb74b1fce00fefc709460b1
+size 100617601

code-daemon-embed-v1-s_ov2026.2_igpu_lnl_int8_b96_s48.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2307f73831b76bb4508cd26ee910d30d68429d3c7fb74b1fce00fefc709460b1
+size 100617601

code-daemon-embed-v1-s_ov2026.2_npu_int4_b16_s48.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f52ac233153d8623bfc2453aec30974ba52bda423c573ef16200949c6d56c77d
+size 33285182

code-daemon-embed-v1-s_win_x64_trt11.0_sm_120.engine

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9b33232ae4a47bfe16732d296e83649949a2a153306e523bf160da738f2023d5
+size 101853452

code-daemon-embed-v1-xl_linux_x64_trt11.0_sm_120.engine

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a926b16af6bc89997c05c3b79b416fd2ffc11f386a28dfb38c12d2d5d5d47b58
+size 101903652

code-daemon-embed-v1-xl_ov2026.2_cpu_int8_b256_s128.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2307f73831b76bb4508cd26ee910d30d68429d3c7fb74b1fce00fefc709460b1
+size 100617601

code-daemon-embed-v1-xl_ov2026.2_igpu_lnl_int8_b256_s128.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2307f73831b76bb4508cd26ee910d30d68429d3c7fb74b1fce00fefc709460b1
+size 100617601

code-daemon-embed-v1-xl_ov2026.2_npu_int4_b16_s128.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f52ac233153d8623bfc2453aec30974ba52bda423c573ef16200949c6d56c77d
+size 33285182

code-daemon-embed-v1-xl_win_x64_trt11.0_sm_120.engine

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a4f383692c9eadde7262025439335f696e0479499b34c1cdd301d36bbd60323a
+size 101858260

code-daemon-embed-v1_linux_x64_tvm0.25_vulkan.so

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0c1e803073aa8b3c506a728f84e38269d749e4e46a963dd3bc6523c44aac86af
+size 238396032

code-daemon-embed-v1_win_x64_tvm0.25_vulkan.dll

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d544ddbcfebc0d2d59aa70f6991d22c9c6cbc08fe0f7420c1f49f8eab826b712
+size 238240256

config.json

@@ -1,24 +1,24 @@
-{
-  "add_cross_attention": false,
-  "attention_probs_dropout_prob": 0.1,
-  "bos_token_id": 0,
-  "classifier_dropout": null,
-  "eos_token_id": 2,
-  "hidden_act": "gelu",
-  "hidden_dropout_prob": 0.1,
-  "hidden_size": 768,
-  "initializer_range": 0.02,
-  "intermediate_size": 3072,
-  "is_decoder": false,
-  "layer_norm_eps": 1e-05,
-  "max_position_embeddings": 514,
-  "model_type": "xlm-roberta",
-  "num_attention_heads": 12,
-  "num_hidden_layers": 8,
-  "pad_token_id": 1,
-  "tie_word_embeddings": true,
-  "transformers_version": "5.13.0",
-  "type_vocab_size": 1,
-  "use_cache": true,
-  "vocab_size": 22739
-}
+{
+  "add_cross_attention": false,
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "is_decoder": false,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "xlm-roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 4,
+  "pad_token_id": 1,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.1.0",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 22739
+}

manifest.json

@@ -1,11 +1,17 @@
-{
-  "compiled_at": "2026-07-23T02:00:00Z",
-  "model_id": "code-daemon-embed-v1",
-  "max_tokens": 128,
-  "quantization": "fp32",
-  "targets": [
-    "tensorrt"
-  ],
-  "dimension": 768,
-  "note": "weights=v2-p5 p5_c (4B-teacher listwise-KL, 2026-07-23 republish); engines to be rebuilt via compile-models"
-}
+{
+  "model_id": "code-daemon-embed-v1",
+  "dimension": 768,
+  "max_tokens": 128,
+  "quantization": "int8",
+  "targets": [
+    "tensorrt"
+  ],
+  "runtime_tags": {
+    "trt": "trt11.0",
+    "ov": "ov2026.2",
+    "tvm": "tvm0.25",
+    "mlx_dir": "model_gpu_mlx0.22"
+  },
+  "compiled_at": "2026-08-02T21:11:00Z",
+  "compiled_by": "models/_compile"
+}

model.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c39aa87e9f2bfe1de1bd928bd0f3fa8121aa7e70509acf02179a21115bf89d57
-size 1120495
+oid sha256:57bcfc6aed11ea239d01f2b124f2f948456f2284ad6e2c4744452509c9c25ca9
+size 187286767

model_gpu_mlx0.22/config.json

@@ -0,0 +1,24 @@
+{
+  "add_cross_attention": false,
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "is_decoder": false,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "xlm-roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 4,
+  "pad_token_id": 1,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.1.0",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 22739
+}

model_int8qdt.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1e5e66dce33dbe89b7ac9c4e68c4573888cefffab5ab283682d72771f5e545da
+size 187490530

tokenizer_config.json

@@ -1,8 +1,8 @@
-{
-  "tokenizer_class": "XLMRobertaTokenizer",
-  "model_max_length": 512,
-  "bos_token": "<s>",
-  "eos_token": "</s>",
-  "unk_token": "<unk>",
-  "pad_token": "<pad>"
+{
+  "tokenizer_class": "XLMRobertaTokenizer",
+  "model_max_length": 512,
+  "bos_token": "<s>",
+  "eos_token": "</s>",
+  "unk_token": "<unk>",
+  "pad_token": "<pad>"
 }