Model card for the v2-p5 republish: built-in reranker, honest CoIR + real-repo positioning, updated licenses (no mMARCO)

README.md

@@ -13,65 +13,100 @@ tags:
 - knowledge-distillation
 pipeline_tag: feature-extraction
 base_model:
+- intfloat/multilingual-e5-base
 - Alibaba-NLP/gte-modernbert-base
+- Qwen/Qwen3-Reranker-4B
 datasets:
-- Fsoft-AIC/the-vault-function
-- unicamp-dl/mmarco
-- sentence-transformers/all-nli
-- sentence-transformers/gooaq
-- jinaai/negation-dataset
+- CoIR-Retrieval/cosqa
+- CoIR-Retrieval/codesearchnet
+- CoIR-Retrieval/stackoverflow-qa
 ---
 
 # code-daemon-embed-v1
 
-A small, fast **code-embedding model** for semantic code search and code retrieval. It maps short code
-units — function and method bodies, signatures, docstrings — and short natural-language queries into a
-shared **768-dim** vector space.
-
-It is **specialized for short code, not long documents**: the maximum sequence is **128 tokens**,
-trading long-context capability for high throughput and strong quality on short units.
-
-- **768-dim** embeddings, **Matryoshka (MRL)** — truncatable to **512 / 256** dims with graceful decay.
-- **~54.5M params** — XLM-RoBERTa architecture, **4 layers / 768 hidden**, a **code-oriented 32k SentencePiece vocab**.
+A small, fast **code-embedding model** for semantic code search over **real codebases**. It maps short
+code units — function and method bodies, signatures, docstrings — and short natural-language queries
+into a shared **768-dim** vector space.
+
+It is **specialized for real retrieval traffic, not benchmark prose**: the training queries mirror how
+agents and developers actually search a repository (short keyword-bag and behavior queries), and the
+maximum sequence is **128 tokens**, trading long-context capability for high throughput and strong
+quality on short units.
+
+- **768-dim** embeddings (full width; no MRL truncation in this release).
+- **~75.7M params** — XLM-RoBERTa architecture initialized from `multilingual-e5-base` with
+  **strided 12→8 layer truncation** and the vocabulary **pruned 250k → 22,739** pieces
+  (English + code coverage kept; the transformer layers see identical tokenizations for EN/code text).
+- **A built-in reranker** — see below: cross-encoder ranking knowledge is distilled *into* the
+  embedding space, so the top of the ranking behaves like a reranked list at bi-encoder cost.
 - **Mean pooling fused into the graph** — the output is already pooled (`[batch, 768]`); just **L2-normalize**.
 - Trained at sequence length **128** (length buckets s / m / l = seq **40 / 64 / 128**).
 
-## Where it's good — and where it isn't
+## The built-in reranker (why the top of the ranking is sharp)
+
+Classic bi-encoders compress each document into a vector *before* seeing the query, so they cannot do
+the fine-grained pairwise comparison a cross-encoder reranker does. This model closes part of that gap
+at training time instead of inference time:
 
-Measured on CoIR (NDCG@10, full corpora). Use it for:
+- Hard negatives were mined for every training query (documents a strong retriever ranks high but wrong).
+- **`Qwen/Qwen3-Reranker-4B`** (a cross-encoder) scored every (query, candidate) pair with graded
+  relevance logits.
+- The student was trained with a **listwise-KL objective** to reproduce the reranker's *ranking
+  distribution* — not just "positive above negatives", but *how much* each near-miss should trail.
 
-**Strong:**
-- **Code → related code** — finding similar or duplicate implementations (its best relative area).
-- **Natural-language → code** — docstring or description → function (strongest on Python).
-- **Short question → code** — "how / where does X…".
-- **NL → SQL** and **NL instruction → code**.
+The result: reranker-flavored top-rank precision with **zero inference overhead** — no second model, no
+second pass, no extra latency. In the production system this model was built for, enabling an actual
+runtime reranker on top of it measured as *net-negative*: the distillation had already captured the
+useful part.
+
+## Where it's good — and where it isn't
+
+**Strong (the design target):**
+- **Real repository search** — short NL/keyword queries by developers and coding agents over an
+  indexed codebase ("git watcher head change reindex", "acquire database lock for project hash").
+- **Natural-language → code** at short lengths; identifier and mixed queries.
+- Complementing a lexical (BM25-style) channel in a hybrid retriever — it was trained *for* that role.
 
 **Weak / out of scope:**
-- **Long documents** — hard 128-token cap; longer inputs are truncated. This is not a long-context retriever.
-- **Noisy / ambiguous NL→code** (hard, under-specified queries) — mid quality.
-- **General English prose** (medical / financial / news) — the code-specialized 32k vocab trades
-  general-text coverage for code. Multilingual text works as a fallback, not a specialty.
+- **Long documents** — hard 128-token cap; not a long-context retriever.
+- **Benchmark-style long problem statements** (competitive-programming prompts, multi-turn dialogue,
+  code↔code translation) — deliberately out of scope, see the evaluation note.
+- **General English prose** (medical / financial / news) — the pruned vocab and code specialization
+  trade general-text quality away.
 
-Embed **queries and documents the same way** — no instruction prefix. For smaller indexes, truncate to
-**256** or **512** dims (MRL) before normalizing.
+Embed **queries and documents the same way** — no instruction prefix.
 
 ## How it was made
 
-Knowledge-distilled (embedding regression) from **[`Alibaba-NLP/gte-modernbert-base`](https://huggingface.co/Alibaba-NLP/gte-modernbert-base)**
-(Apache-2.0, ~150M, a strong general + code retriever on a ModernBERT backbone). The student is a fresh,
-shallow-wide XLM-R encoder trained from scratch on the teacher's passage embeddings over a ~30M-sample
-code + text corpus, with a custom 32k code-oriented SentencePiece vocabulary (syntax + identifier lexicon
-rather than prose). The shallow-wide 4-layer / 768-hidden shape keeps inference cheap while distilling at
-the teacher's full 768-dim width.
+Three-teacher pipeline on a pretrained backbone (the full recipe and every ablation is documented in
+the source project):
+
+1. **Backbone**: `intfloat/multilingual-e5-base` (12L XLM-R retriever), **strided-truncated to 8 layers**
+   (keep {0,1,3,5,7,9,10,11}) with the embedding table pruned and re-indexed to a 22,739-piece
+   SentencePiece vocabulary in a raw-SP convention (pad=0 / unk=1 / bos=2 / eos=3).
+2. **Dense representation distillation** from
+   **[`Alibaba-NLP/gte-modernbert-base`](https://huggingface.co/Alibaba-NLP/gte-modernbert-base)**
+   (MSE to the teacher's embeddings) over real repository entities in a front-loaded serve format
+   (semantics before identifiers, within the 128-token budget).
+3. **Listwise-KL ranking distillation** from
+   **[`Qwen/Qwen3-Reranker-4B`](https://huggingface.co/Qwen/Qwen3-Reranker-4B)** cross-encoder scores
+   over mined hard negatives (12 kept per query) — the "built-in reranker".
+4. **Diversity regularization**: ~26% of the grouped stream is out-of-domain code-retrieval pairs
+   (CoIR-derived). Measured to be the sweet spot: 0% loses deep recall, 50% dilutes the top.
+   Counter-intuitively, the *dissimilarity* of these rows is the active ingredient.
+
+Training queries were generated few-shot in the style of **real agent search traffic** (short
+keyword-bags, behavior descriptions, identifier fragments) with identifier-leak gating — not docstring
+paraphrases. This is the main reason benchmark numbers below *understate* real-repo performance.
 
 ## Built for speed
 
-- **Short context by design** — max **128 tokens**, no long-document path, so the engines avoid a wide
-  dynamic shape range.
-- **Rectangular TensorRT profiles** — each length bucket is a *fixed* shape (min == opt == max), one
-  optimal kernel set per bucket: **s** = batch 64 × seq 40 · **m** = batch 128 × seq 64 ·
-  **l** = batch 256 × seq 128.
-- **INT8 (W8A16)** weights; **mean-pool + projection + L2-norm fused into the graph** (one pass → `[B, 768]`).
+- **Short context by design** — max **128 tokens**; the engines avoid wide dynamic-shape ranges.
+- **Rectangular TensorRT profiles** — each length bucket is a *fixed* shape (min == opt == max):
+  **s** = batch 64 × seq 40 · **m** = batch 128 × seq 64 · **l** = batch 256 × seq 128.
+- **Mean-pool + L2-friendly head fused into the graph** (one pass → `[B, 768]`).
+- FP32/FP16 engines in this release (INT8 PTQ measurably hurts this checkpoint's top-rank precision;
+  a QAT INT8 variant is planned).
 
 ## Usage (standalone ONNX)
 
@@ -84,92 +119,92 @@ import onnxruntime as ort, sentencepiece as spm, numpy as np
 sp   = spm.SentencePieceProcessor(model_file="sentencepiece.bpe.model")  # pad=0 unk=1 bos=2 eos=3
 sess = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])
 
-def embed(texts, max_len=128, mrl_dim=768):
+def embed(texts, max_len=128):
     ids  = [[2, *sp.encode(t)[: max_len - 2], 3] for t in texts]          # bos … eos
     L    = max(len(x) for x in ids)
     inp  = np.array([x + [0] * (L - len(x)) for x in ids], dtype=np.int64) # pad=0
     mask = (inp != 0).astype(np.int64)
-    out  = sess.run(None, {"input_ids": inp, "attention_mask": mask})[0]   # already mean-pooled [B,768]
-    out  = out[:, :mrl_dim]                                                # MRL truncation (768/512/256)
+    out  = sess.run(None, {"input_ids": inp, "attention_mask": mask})[0]   # already pooled [B,768]
     return out / np.linalg.norm(out, axis=1, keepdims=True)
 ```
 
-## What's in this repo — ready-to-run compiled engines
+## What's in this repo
 
-Pre-compiled engines, named per **runtime × GPU arch × OS × length-bucket** — pick the one matching your
-runtime and hardware; no compilation needed.
+Named per **runtime × GPU arch × OS × length-bucket** (the filenames are a loading contract — do not rename):
 
-- **TensorRT** `*.engine` — NVIDIA, INT8 W8A16:
+- **TensorRT** `*.engine` — NVIDIA:
   `code-daemon-embed-v1-{s,m,l}_{win_x64,linux_x64}_trt_sm_{86,89,120}.engine`
   (sm_86 ≈ RTX 30xx / A-series · sm_89 ≈ RTX 40xx / L4 · sm_120 ≈ RTX 50xx).
-- **TVM** `*_tvm_vulkan.{dll,so}` — Vulkan fallback for non-TRT / older NVIDIA & other GPUs, per bucket.
+- **TVM** `*_tvm_vulkan.{dll,so}` — Vulkan fallback for non-TRT GPUs, per bucket.
 - **OpenVINO** `*.xml` + `*.bin` — Intel **CPU / iGPU / NPU**, per bucket.
 - **Metal** `*_tvm_metal.*` — Apple Silicon (macOS), per bucket.
-- **Tokenizer** — `sentencepiece.bpe.model` (specials at pad=0 / unk=1 / bos=2 / eos=3, byte-fallback) +
+- **Tokenizer** — `sentencepiece.bpe.model` (raw-SP: pad=0 / unk=1 / bos=2 / eos=3, byte-fallback) +
   `tokenizer_config.json`.
-- **ONNX source** — `model.onnx` (+ `model.onnx.data`) FP32 and `model_int8qdt.onnx` (INT8 W8A16).
+- **ONNX source** — `model.onnx` (+ `model.onnx.data`) FP32.
 
-## Evaluation — CoIR (NDCG@10, full corpora)
+> Compiled engines for this release are being rebuilt and uploaded progressively; the ONNX + tokenizer
+> are the always-present source of truth.
 
-Retrieval quality on the CoIR tasks that match this model's design (short code + retrieval). Four of
-CoIR's ten tasks — code↔code translation, multi-turn dialogue, long problem-statements — exceed the
-128-token / retrieval scope and are not shown.
+## Evaluation
 
-| CoIR task | NDCG@10 | Pattern |
-|---|--:|---|
-| codesearchnet (6-lang avg) | **73.17** | docstring / NL → code |
-| stackoverflow-qa | 62.70 | short question → code |
-| synthetic-text2sql | 61.32 | NL → SQL |
-| codesearchnet-ccr (6-lang avg) | **57.30** | code → related code |
-| codefeedback-st | 56.38 | NL instruction → code |
-| cosqa | 35.51 | NL question → code (noisy / hard) |
-| **Average** | **57.73** | |
+### Real-repository retrieval (the design target)
 
-Per language — codesearchnet (NL→code): python **88.82**, java 75.78, php 73.71, go 73.49, ruby 64.10, js 63.09.
-Per language — codesearchnet-ccr (code→code): ruby **65.67**, java 62.87, js 61.05, python 55.88, go 51.53, php 46.77.
+On a benchmark of **real agent search queries over a live indexed codebase** (80 captured queries,
+multi-positive, file-level), the production hybrid retriever built on this model reaches:
 
-**Binary (1-bit sign) vectors** retain **94.4%** of the float NDCG before any rescore — the embeddings
-are clean by sign, so a Hamming (XOR + popcount) index over 1-bit codes gives a ~32× memory / search win
-at a small quality cost.
+| metric | value |
+|---|--:|
+| hit@5 | **0.73** |
+| hit@1 | 0.44 |
+| pool_recall@64 | 0.79 |
 
-> For scale, the 1.5B-parameter `bge-code-v1` scores 81.77 on full CoIR — this is a **54.5M** model
-> (27× smaller) tuned for short-code retrieval.
+This is the metric the model is optimized for — short real-world queries against a real repository.
 
-## Performance (embeddings / sec)
+### CoIR (NDCG@10, full corpora) — out-of-domain reference
+
+CoIR queries are largely docstrings and long problem statements — a different style from real repo
+search traffic — so treat these as an *out-of-domain lower bound*: **on real queries in real
+repositories the model performs substantially better than its CoIR positioning suggests** (see above).
+Four of CoIR's ten tasks — code↔code translation, multi-turn dialogue, long problem-statements —
+exceed the 128-token / retrieval scope and are not shown. (~2k CoIR-derived rows were present in
+training as diversity regularization; overlap with test splits was not audited.)
+
+| CoIR task | NDCG@10 | Pattern |
+|---|--:|---|
+| synthetic-text2sql | **62.59** | NL → SQL |
+| stackoverflow-qa | 50.82 | short question → code |
+| codefeedback-st | 47.42 | NL instruction → code |
+| codesearchnet (6-lang avg) | 42.68 | docstring / NL → code |
+| codesearchnet-ccr (6-lang avg) | 40.95 | code → related code |
+| cosqa | 20.75 | NL question → code (noisy / hard) |
+| **Average** | **44.20** | |
 
-| Backend | Hardware | Throughput |
-|---|---|--:|
-| TensorRT INT8 | NVIDIA RTX 5060 (sm_120) | **~24,000 emb/s** |
-| OpenVINO INT4 | Intel iGPU (Xe2, Lunar Lake) | ~580 emb/s |
-| OpenVINO INT4 | Intel NPU (NPU4) | ~574 emb/s |
-| OpenVINO INT8 | Intel CPU (Core Ultra) | ~375 emb/s |
-| OpenVINO — **all 3 in parallel** | iGPU + NPU + CPU concurrently | ~1,290 emb/s |
+Per language — codesearchnet (NL→code): python **66.80**, go 51.30, java 38.77, php 38.16, js 32.17, ruby 28.90.
+Per language — codesearchnet-ccr (code→code): js **47.72**, ruby 45.28, java 42.05, python 38.43, php 36.60, go 35.64.
+
+## Performance (embeddings / sec)
 
-The combined OpenVINO figure is genuine concurrent multi-device execution — three workers (iGPU, NPU,
-CPU) embed different batches at the same time and the throughputs add up. This is **not** OpenVINO's
-`AUTO` mode (which picks a single device per inference). Measured on a Core Ultra (Lunar Lake) laptop;
-the TensorRT figure is on the bucketed batch path.
+Being re-measured for this release's engines — table will be updated with the engine upload.
 
 ## License & training data
 
 Released under the **MIT license**.
 
-The teacher (`Alibaba-NLP/gte-modernbert-base`) is Apache-2.0, and the XLM-R architecture is MIT. As is
-standard practice for distilled embedding models, the **weights are released under MIT**. For
-transparency, the training corpus the teacher embedded includes:
+The backbone (`intfloat/multilingual-e5-base`) is MIT; the teachers (`gte-modernbert-base`,
+`Qwen3-Reranker-4B`) are Apache-2.0. As is standard practice for distilled embedding models, the
+**weights are released under MIT**. Training corpus, for transparency:
 
-| Dataset | License note |
+| Data | License note |
 |---|---|
-| `Fsoft-AIC/the-vault-function` (code) | dataset MIT; underlying code has mixed upstream provenance |
-| `unicamp-dl/mmarco` (EN/RU retrieval) | **MS MARCO-derived → non-commercial research terms** |
-| `sentence-transformers/all-nli` | SNLI (CC BY-SA 4.0) + MultiNLI |
-| `sentence-transformers/gooaq` | Apache-2.0 |
-| `jinaai/negation-dataset` | see source repo |
+| Repository entity texts + synthetic search queries over an MIT-licensed codebase | MIT |
+| CoIR-derived code-retrieval pairs (~26% diversity stream) | per-subset CoIR licenses (research benchmark) |
+| Teacher embeddings / scores (gte-modernbert, Qwen3-Reranker-4B) | Apache-2.0 teachers |
 
-⚠️ If your use requires strict training-data-license compliance, note that **mMARCO derives from
-MS MARCO (non-commercial)**. Whether a distilled model inherits dataset-use terms is legally unsettled;
-this is **not legal advice**. A data-clean variant can be retrained without the mMARCO splits if needed.
+Unlike the previous release, **no MS MARCO / mMARCO data** was used — the earlier non-commercial
+caveat no longer applies. This is **not legal advice**.
 
 ## Attribution
 
-Distilled from **[Alibaba-NLP/gte-modernbert-base](https://huggingface.co/Alibaba-NLP/gte-modernbert-base)** (Apache-2.0). Backbone: XLM-RoBERTa (MIT).
+Backbone: [intfloat/multilingual-e5-base](https://huggingface.co/intfloat/multilingual-e5-base) (MIT).
+Dense teacher: [Alibaba-NLP/gte-modernbert-base](https://huggingface.co/Alibaba-NLP/gte-modernbert-base) (Apache-2.0).
+Ranking teacher: [Qwen/Qwen3-Reranker-4B](https://huggingface.co/Qwen/Qwen3-Reranker-4B) (Apache-2.0).