README.md

---
license: cc-by-nc-sa-4.0
base_model: jinaai/jina-embeddings-v2-base-code
datasets:
  - zhangfw123/CORE-Bench
language:
  - code
library_name: sentence-transformers
pipeline_tag: feature-extraction
tags:
  - sentence-transformers
  - feature-extraction
  - code-retrieval
  - issue-localization
  - onnx
---

# jina-v2-code-ft2

A 161M-parameter code embedding model, fine-tuned from
[`jinaai/jina-embeddings-v2-base-code`](https://huggingface.co/jinaai/jina-embeddings-v2-base-code)
for **issue-to-edit localization**: given a GitHub issue, retrieve the code
chunks that have to change.

It was trained in 2.5 hours on one consumer GPU (RTX 3060, 12 GB). Fused with
BM25 it scores **NDCG@10 0.233** on CORE-Bench Level-2, above the published
0.224 of the 7B SweRankEmbed-Large — at 1/43 the parameters.

This is a **research artifact**, not a drop-in upgrade. Read the Limitations
section before using it: the gain is specific to long issue-style queries and
does **not** transfer to short developer queries, which is why the tool it was
built for still ships the unmodified base model.

## Results

CORE-Bench Level-2, full evaluation set (~2,080 queries, 253 repositories,
~2.2M corpus chunks). Rows marked *paper* are from
[arXiv:2606.11864](https://arxiv.org/abs/2606.11864) v3 (EMNLP 2026).

| Retriever | Params | NDCG@10 | Recall@100 |
|---|---:|---:|---:|
| *paper:* gte-Qwen2-1.5B-instruct | 1.5B | 0.035 | 0.159 |
| *paper:* bge-m3 | 568M | 0.046 | 0.183 |
| *paper:* CodeRankEmbed | <1B | 0.121 | 0.329 |
| jina-v2-base-code + BM25 (the base, hybrid) | 161M | 0.150 | 0.438 |
| *paper:* Qwen3-Embedding-8B, zero-shot | 8B | 0.203 | 0.480 |
| *paper:* SweRankEmbed-Large | 7B | 0.224 | 0.521 |
| **this model + BM25 (hybrid)** | **161M** | **0.233** | 0.498 |
| *paper:* Qwen3-8B-SFT | 8B | 0.328 | 0.664 |

Read it in both directions. It passes a 7B specialised retriever and an 8B
zero-shot model on NDCG@10, and it stays **below** SweRankEmbed-Large on
Recall@100 (0.498 against 0.521). The paper's own fine-tuned 8B remains
clearly ahead of everything in this size class.

Set-difference caveat: our evaluation excludes Multi-SWE-bench (absent from the
baseline run) and SWE-Bench-plus-plus (used for training); the paper's covers
the full original set.

### The gain lives in fusion, not in the vectors

The mechanism is the interesting part. Vector-only ranking barely moves under
fine-tuning; the fused score jumps. Measured on the round-1 model over a
repo-level holdout of 47 unseen repositories (468 queries):

| | NDCG@10 | Recall@100 |
|---|---:|---:|
| base, vector only | 0.142 | 0.420 |
| base, hybrid | 0.168 | 0.491 |
| fine-tuned, vector only | 0.141 | 0.456 |
| **fine-tuned, hybrid** | **0.262** | **0.561** |

The tuned model does not rank better on its own — it surfaces *different*
relevant chunks than BM25 does, and reciprocal rank fusion compounds two
rankings that disagree. Two independent fine-tunes on disjoint training sets
reproduced the same relative gain (+56% and +55%) and the same flat-vector
signature.

**Use this model fused with a lexical channel.** On its own it is roughly the
base model.

## Training

- **Data:** CORE-Bench Level-2, **SWE-Bench-plus-plus split only**.
- **Contamination control:** that split shares **zero repositories** with the
  evaluation set above. The split is a checked-in contract, not a convention.
- **Training pairs:** 1,270 (query, positive, hard-negatives) rows.
- **Objective:** MultipleNegativesRankingLoss (sentence-transformers), 4
  BM25-mined hard negatives per row plus in-batch negatives.
- **Hyperparameters:** 3 epochs, batch size 8, learning rate 2e-5, warmup ratio
  0.1, bf16, max sequence length 512.
- **Hardware / time:** one RTX 3060 (12 GB), 8,959 s of training (~2.5 hours).

## Limitations

**It does not transfer to short developer queries.** This is the finding that
kept it out of production. On a 144-case internal corpus of short, intent-phrased
developer questions ("where is the decision made to split a range based on
load"), evaluated with enriched indexes:

- 86-case dev split: this model **trails** the base — Hit@3 0.79 vs 0.85,
  Recall@5 0.84 vs 0.88, consistently across projects and slices.
- 58-case holdout: parity — Hit@3 0.90 for both.

A 30-case pilot had shown no regression; that did not replicate at full size.
Issue-style training does not generalise downward to short queries.

**Recall is not what improved.** NDCG@10 moves; Recall@100 stays below the 7B
baseline. If your bottleneck is reach rather than ordering, this will not fix it.

**Evaluated on one benchmark family.** All numbers above are CORE-Bench
Level-2. No claim is made about docstring-to-function retrieval, cross-language
behaviour, or natural-language code search generally.

**English-and-mainstream bias, inherited.** On the SWE-bench_Multilingual split
the base encoder scores roughly half what it does on the English splits
(NDCG@10 0.0557 against 0.1088 / 0.1229). Fine-tuning does not repair that.

## Intended use

Research and reproduction: issue-to-edit localization, retrieval-fusion
experiments, and as a size-class baseline for small code encoders.

**Out of scope:** commercial use (see License), and any deployment where short
queries dominate — use the Apache-2.0 base model there instead.

## License and provenance

**This model is released under CC BY-NC-SA 4.0.**

- The base model, `jinaai/jina-embeddings-v2-base-code`, is **Apache 2.0**, and
  its notices are preserved. The custom modelling code bundled with this
  checkpoint (`modeling_bert.py`, `configuration_bert.py`) originates there and
  remains under that license.
- The training data, `zhangfw123/CORE-Bench`, is **CC BY-NC-SA 4.0** —
  NonCommercial and ShareAlike.

Whether model weights constitute a derivative work of their training data is
unsettled: Creative Commons licenses predate machine-learning training, and CC
has said as much itself. Rather than bet on the permissive reading, this model
adopts the dataset's own terms. That satisfies ShareAlike if it applies and
honours the NonCommercial intent if it does not.

**Practical consequence: do not use these weights in a commercial product.**
If you want a commercially usable code embedder, use the Apache-2.0 base model
directly — it is what
[Contextmaxxer](https://github.com/codeus-morbid/contextmaxxer), the tool this
work came out of, actually ships.

No CORE-Bench corpus text is redistributed in this repository.

This is a licensing summary, not legal advice.

## Citation

The benchmark and the baselines it supplies:

```bibtex
@article{corebench2026,
  title  = {CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Coding},
  author = {Zhang, Fuwei and Zhang, Yanzhao and Li, Mingxin and others},
  journal = {arXiv preprint arXiv:2606.11864},
  year   = {2026}
}
```

Full evaluation protocol, the negative results, and the reasoning behind the
production default are in
[BENCHMARK.md](https://github.com/codeus-morbid/contextmaxxer/blob/main/BENCHMARK.md).