| --- |
| license: apache-2.0 |
| language: |
| - en |
| - code |
| library_name: sentence-transformers |
| pipeline_tag: sentence-similarity |
| base_model: ibm-granite/granite-embedding-english-r2 |
| tags: |
| - sentence-transformers |
| - feature-extraction |
| - code-search |
| - code-retrieval |
| - bug-localization |
| - modernbert |
| - onnx |
| model-index: |
| - name: CodeIntel-150M |
| results: |
| - task: {type: retrieval, name: Issue -> code localization (descriptive issues, Python)} |
| dataset: {type: custom, name: codeintel issue localization (35 repositories)} |
| metrics: [{type: ndcg_at_10, value: 0.268, name: symbol nDCG@10 (hybrid)}] |
| - task: {type: retrieval, name: Issue -> code localization (descriptive issues, TypeScript)} |
| dataset: {type: custom, name: codeintel issue localization (35 repositories)} |
| metrics: [{type: ndcg_at_10, value: 0.274, name: symbol nDCG@10 (hybrid)}] |
| - task: {type: retrieval, name: Test -> implementation (Python)} |
| dataset: {type: custom, name: codeintel code-to-code} |
| metrics: [{type: ndcg_at_10, value: 0.640}] |
| - task: {type: retrieval, name: CoSQA} |
| dataset: {type: CoIR-Retrieval/cosqa, name: CoSQA} |
| metrics: [{type: ndcg_at_10, value: 0.651}] |
| - task: {type: retrieval, name: CoIR StackOverflow QA} |
| dataset: {type: CoIR-Retrieval/stackoverflow-qa, name: StackOverflow QA} |
| metrics: [{type: ndcg_at_10, value: 0.922}] |
| - task: {type: retrieval, name: CoIR CodeSearchNet (Python)} |
| dataset: {type: CoIR-Retrieval/CodeSearchNet, name: CodeSearchNet Python} |
| metrics: [{type: ndcg_at_10, value: 0.963}] |
| --- |
| |
| # CodeIntel-150M |
|
|
| CodeIntel-150M is a 149M-parameter embedding model for **issue-to-code localization**: given a bug report or feature |
| request, it retrieves the functions and classes in a Python or TypeScript repository that a fix is likely to |
| modify. On issues that describe only symptoms, with no reference to the relevant code, it outperforms every other |
| open embedding model of comparable size by **roughly 55%**. It is also **27% better** on issues that reference code directly |
| and **23% better** at test-to-implementation retrieval. |
|
|
| Beyond localization, it supports call-site-to-definition navigation and code search, where it leads its size class on CoIR StackOverflow QA |
| and CodeSearchNet Python. It is a fine-tune of |
| [granite-embedding-english-r2](https://huggingface.co/ibm-granite/granite-embedding-english-r2) (ModernBERT, |
| Apache-2.0). It runs locally, including as a 325 MB INT8 ONNX model, and needs no remote code. |
|
|
| Training and evaluation code, the benchmarks and an MCP server for coding agents are at |
| [github.com/vedevpatel/codeintel-150m](https://github.com/vedevpatel/codeintel-150m). |
|
|
| | | | |
| |---|---| |
| | Parameters | 149M | |
| | Embedding | 768-d, CLS pooling, L2-normalised (use cosine / dot product) | |
| | Max length | trained at 512 tokens per document, evaluated at 1,024 (base supports 8,192) | |
| | Languages | English text; Python and TypeScript code (other languages untested) | |
| | Licence | Apache-2.0 | |
|
|
| ## Results |
|
|
| CodeIntel-150M is the best open model of its size on **7 of 15 benchmarks**, including all three issue-localization |
| benchmarks, test-to-implementation retrieval, TypeScript call-site navigation, CoIR StackOverflow QA and |
| CodeSearchNet Python. The full per-model table is in |
| [docs/results/benchmarks.md](https://github.com/vedevpatel/codeintel-150m/blob/main/docs/results/benchmarks.md). |
|
|
| All numbers come from held-out test data that no training or model-selection step used. Each model was scored once, |
| with the same protocol. The comparison models are open models of similar size, evaluated zero-shot. |
|
|
| **Issue -> code localization.** The test set has 3,804 GitHub issues from 35 Python and TypeScript repositories, none |
| of which were used in training. A hit means retrieving a function or class that the issue's fix edited, among all |
| definitions in the repository at the issue's commit. "Descriptive" issues name no identifier from the fix (for example |
| "server ignores the port when debug is on"). "Lexical" issues do name one. The metric is symbol-level nDCG@10. |
|
|
| | model | descriptive, Python | descriptive, TypeScript | lexical | |
| |---|---|---|---| |
| | **CodeIntel-150M**, dense | **0.242** | **0.260** | **0.482** | |
| | **CodeIntel-150M**, + BM25 (RRF, dense weight 2) | **0.268** | **0.274** | **0.516** | |
| | granite-embedding-english-r2, dense | 0.166 | 0.161 | 0.346 | |
| | bge-base-en-v1.5, dense | 0.163 | 0.166 | 0.356 | |
| | gte-modernbert-base, dense | 0.125 | 0.112 | 0.308 | |
| | CodeRankEmbed, dense | 0.094 | 0.095 | 0.292 | |
| | BM25 | 0.120 | 0.133 | 0.371 | |
|
|
| The best zero-shot model, each with BM25 fusion at its own dev-chosen weight, scores 0.173 / 0.178 descriptive. The |
| gain over it is +0.095 in both languages, with repository-clustered 95% intervals excluding zero. |
|
|
| **Code-to-code and general code search.** Dense nDCG@10. For each benchmark, the right-hand column gives the best of |
| granite-embedding-english-r2, bge-base-en-v1.5, modernbert-embed-base, gte-modernbert-base, CodeRankEmbed, |
| jina-embeddings-v2-base-code and codesage-small-v2. |
|
|
| | benchmark | CodeIntel-150M | best other open ~150M | |
| |---|---|---| |
| | test -> implementation (Python) | **0.640** | 0.522 (CodeRankEmbed) | |
| | call site -> definition, Python / TypeScript | 0.219 / **0.278** | 0.244 (bge-base) / 0.264 (granite-r2) | |
| | docstring -> function, Python / TypeScript | 0.717 / 0.672 | 0.863 / 0.762 (CodeRankEmbed) | |
| | CoSQA | 0.651 | 0.737 (jina-v2-code) | |
| | CoIR StackOverflow QA | **0.922** | 0.917 (granite-r2) | |
| | CoIR CodeSearchNet, Python / JavaScript | **0.963** / 0.634 | 0.960 (codesage) / 0.764 (jina-v2-code) | |
| | CoIR CodeSearchNet-CCR, Python / JavaScript | 0.703 / 0.688 | 0.940 / 0.943 (gte-modernbert) | |
| | CoIR APPS | 0.174 | 0.575 (gte-modernbert) | |
|
|
| **Choose CodeIntel-150M** for repository-scale retrieval driven by issues, bug reports or tests, for example as the |
| retriever in a coding agent. **Choose a general code-search model instead** (gte-modernbert-base, CodeRankEmbed, |
| jina-embeddings-v2-base-code) for docstring-style or web-query code search and code completion-style retrieval (CCR), |
| where CodeIntel-150M trails. |
|
|
| ## Usage |
|
|
| Documents are code definitions prefixed with a one-line `path::QualifiedName` header in the language's comment |
| syntax, as in training. Queries are raw issue text; for best results remove template boilerplate, logs and long code |
| blocks first. No query or document prefix is needed. |
|
|
| ```python |
| from sentence_transformers import SentenceTransformer |
| |
| model = SentenceTransformer("vedevpatel/CodeIntel-150M") |
| code = [ |
| "# src/flask/app.py::Flask.run\ndef run(self, host=None, port=None, debug=None):\n ...", |
| "# src/utils/date.ts::parseDate\nexport function parseDate(s: string): Date { return new Date(s); }", |
| ] |
| query = "Server ignores the port argument when debug is enabled" |
| scores = model.encode([query]) @ model.encode(code).T # cosine similarities (embeddings are normalised) |
| ``` |
|
|
| On a GPU, load in half precision (bf16 on Ampere or newer, fp16 otherwise); the default is float32 for CPU. With |
| `flash-attn` installed, `attn_implementation="flash_attention_2"` speeds up long inputs further. |
|
|
| ```python |
| import torch |
| |
| model = SentenceTransformer("vedevpatel/CodeIntel-150M", device="cuda", model_kwargs={"dtype": torch.bfloat16}) |
| ``` |
|
|
| With plain transformers: |
|
|
| ```python |
| import torch |
| from transformers import AutoModel, AutoTokenizer |
| |
| tok = AutoTokenizer.from_pretrained("vedevpatel/CodeIntel-150M") |
| model = AutoModel.from_pretrained("vedevpatel/CodeIntel-150M").eval() |
| batch = tok(texts, padding=True, truncation=True, max_length=1024, return_tensors="pt") |
| with torch.no_grad(): |
| emb = torch.nn.functional.normalize(model(**batch).last_hidden_state[:, 0], dim=-1) |
| ``` |
|
|
| **ONNX / INT8.** `onnx/model.onnx` (fp32) and `onnx/model_int8.onnx` (dynamic INT8; the GeGLU down-projections stay |
| fp32). Outputs are the last hidden states; take the CLS vector and L2-normalise it. The INT8 model's minimum cosine |
| to PyTorch is 0.995. On dev retrieval its nDCG@10 matches fp32 within noise: CoSQA +0.003 [-0.005, +0.011], |
| docstrings-TypeScript -0.002 [-0.010, +0.006]. It is about 1.7x faster on CPU. |
|
|
| **Hybrid retrieval.** Fusing dense scores with BM25 by reciprocal-rank fusion (dense weight 2, k = 60) adds about |
| +0.03 on issue localization. |
|
|
| ## Use with coding agents (MCP) |
|
|
| `codeintel mcp` gives Claude Code, Codex, Cursor and other MCP clients semantic search over the repository they are |
| working in: the agent passes an issue or a plain-language description and gets back ranked functions and classes with |
| file paths and line ranges. It runs locally over stdio, indexes the repository in the background and re-embeds only |
| changed code after edits. |
|
|
| ```bash |
| # Claude Code (run in the repository) |
| claude mcp add codeintel -- uvx --from "codeintel[mcp] @ git+https://github.com/vedevpatel/codeintel-150m" codeintel mcp . |
| ``` |
|
|
| ```toml |
| # Codex: ~/.codex/config.toml |
| [mcp_servers.codeintel] |
| command = "uvx" |
| args = ["--from", "codeintel[mcp] @ git+https://github.com/vedevpatel/codeintel-150m", "codeintel", "mcp", "."] |
| ``` |
|
|
| Setup for Cursor is in the [GitHub README](https://github.com/vedevpatel/codeintel-150m#use-with-coding-agents-mcp). |
|
|
| ## Training |
|
|
| - **Objective:** multi-positive InfoNCE (symmetric, temperature 0.05) with BM25-mined hard negatives and GradCache |
| (batch 128). Plus distillation (weight 0.5) toward the frozen base model's similarity structure, which preserves |
| general search ability. 3,000 steps, learning rate 5e-6, bf16, maximum document length 512. |
| - **Data** (422k query groups; every source permissively licensed): |
| - 82 GitHub repositories (MIT, Apache-2.0, BSD, ISC and similar; each licence verified at every revision code was |
| taken from; none is an evaluation or development repository): issue -> edited function and commit -> edited |
| function (issue text as originally opened where recoverable), docstring -> function, test -> implementation, call |
| site -> definition, code context -> continuation. |
| - [The Vault](https://huggingface.co/datasets/Fsoft-AIC/the-vault-function): docstring -> function and code-context |
| pairs, only items whose declared licences are all permissive. |
| - [CodeSearchNet](https://huggingface.co/datasets/code-search-net/code_search_net) train split (Python, |
| JavaScript): code -> docstring, code -> continuation and docstring -> code. Rows kept only if the licence GitHub |
| detects at the row's own commit is permissive (3,213 of the 4,000 largest repositories). CoIR's CodeSearchNet |
| tasks are built from CodeSearchNet's test split, whose repositories are disjoint from the train split. |
| |
| All data was decontaminated against every evaluation and development text, including the CoIR corpora (exact and |
| MinHash near-duplicates). |
| - **Sources and licences:** every repository and dataset used is listed in [TRAINING_DATA.md](TRAINING_DATA.md). |
| - **Selection:** the final checkpoint, chosen on a separate development set of 19 repositories by a fixed |
| rule: maximise descriptive localization, subject to no loss on lexical localization, general search and |
| code-to-code retrieval. |
| - **Compute:** one NVIDIA H100 80GB, about 1.6 hours (embedding cache 10 minutes, training 87 minutes). |
|
|
| ## Limitations |
|
|
| - Issue localization remains hard. Even CodeIntel-150M puts an edited function in the top 10 for well under half of |
| descriptive issues. |
| - Fixes that only add new code cannot be retrieved, because the target does not exist yet. The benchmark reports this |
| as a recall ceiling. |
| - Evaluated on Python and TypeScript only. Behaviour on other languages is unknown. |
| - Docstring-style search and CoSQA are at the base model's level, below CodeRankEmbed and jina-embeddings-v2-base-code. |
| - General natural-language code search and code-continuation retrieval (CoIR CCR, APPS) remain below the best open |
| models of this size. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{<CITATION_KEY>, |
| title = {CodeIntel-150M: an embedding model for issue-to-code localization}, |
| author = {<AUTHORS>}, |
| year = {2026}, |
| url = {https://huggingface.co/vedevpatel/CodeIntel-150M} |
| } |
| ``` |
|
|
