Release CodeIntel-150M

1_Pooling/config.json

@@ -0,0 +1,8 @@
+{
+  "word_embedding_dimension": 768,
+  "pooling_mode_cls_token": true,
+  "pooling_mode_mean_tokens": false,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false,
+  "include_prompt": true
+}

README.md

@@ -1,3 +1,193 @@
 ---
 license: apache-2.0
+language:
+  - en
+  - code
+library_name: sentence-transformers
+pipeline_tag: sentence-similarity
+base_model: ibm-granite/granite-embedding-english-r2
+tags:
+  - sentence-transformers
+  - feature-extraction
+  - code-search
+  - code-retrieval
+  - bug-localization
+  - modernbert
+  - onnx
+model-index:
+  - name: CodeIntel-150M
+    results:
+      - task: {type: retrieval, name: Issue -> code localization (descriptive issues, Python)}
+        dataset: {type: custom, name: codeintel issue localization (35 repositories)}
+        metrics: [{type: ndcg_at_10, value: 0.268, name: symbol nDCG@10 (hybrid)}]
+      - task: {type: retrieval, name: Issue -> code localization (descriptive issues, TypeScript)}
+        dataset: {type: custom, name: codeintel issue localization (35 repositories)}
+        metrics: [{type: ndcg_at_10, value: 0.274, name: symbol nDCG@10 (hybrid)}]
+      - task: {type: retrieval, name: Test -> implementation (Python)}
+        dataset: {type: custom, name: codeintel code-to-code}
+        metrics: [{type: ndcg_at_10, value: 0.640}]
+      - task: {type: retrieval, name: CoSQA}
+        dataset: {type: CoIR-Retrieval/cosqa, name: CoSQA}
+        metrics: [{type: ndcg_at_10, value: 0.651}]
+      - task: {type: retrieval, name: CoIR StackOverflow QA}
+        dataset: {type: CoIR-Retrieval/stackoverflow-qa, name: StackOverflow QA}
+        metrics: [{type: ndcg_at_10, value: 0.922}]
+      - task: {type: retrieval, name: CoIR CodeSearchNet (Python)}
+        dataset: {type: CoIR-Retrieval/CodeSearchNet, name: CodeSearchNet Python}
+        metrics: [{type: ndcg_at_10, value: 0.963}]
 ---
+
+# CodeIntel-150M
+
+A 149M-parameter embedding model for **finding the code a bug report or feature request is about**. Given an issue
+written by a user, it retrieves the functions and classes in a Python or TypeScript repository that a fix will touch.
+It also handles code-to-code navigation (test -> implementation, call site -> definition) and code search
+(it leads its size class on CoIR StackOverflow QA and CodeSearchNet Python). It is a fine-tune of
+[granite-embedding-english-r2](https://huggingface.co/ibm-granite/granite-embedding-english-r2) (ModernBERT,
+Apache-2.0). It runs locally, including as a 325 MB INT8 ONNX model, and needs no remote code.
+
+| | |
+|---|---|
+| Parameters | 149M |
+| Embedding | 768-d, CLS pooling, L2-normalised (use cosine / dot product) |
+| Max length | trained at 512 tokens per document, evaluated at 1,024 (base supports 8,192) |
+| Languages | English text; Python and TypeScript code (other languages untested) |
+| Licence | Apache-2.0 |
+
+## Results
+
+All numbers come from held-out test data that no training or model-selection step used. Each model was scored once,
+with the same protocol. Comparisons are with open models of similar size, all zero-shot.
+
+**Issue -> code localization.** The test set has 3,804 GitHub issues from 35 Python and TypeScript repositories, none
+of which were used in training. A hit means retrieving a function or class that the issue's fix edited, among all
+definitions in the repository at the issue's commit. "Descriptive" issues name no identifier from the fix (for example
+"server ignores the port when debug is on"). "Lexical" issues do name one. The metric is symbol-level nDCG@10.
+
+| model | descriptive, Python | descriptive, TypeScript | lexical |
+|---|---|---|---|
+| **this model**, dense | **0.242** | **0.260** | **0.482** |
+| **this model**, + BM25 (RRF, dense weight 2) | **0.268** | **0.274** | **0.516** |
+| granite-embedding-english-r2, dense | 0.166 | 0.161 | 0.346 |
+| bge-base-en-v1.5, dense | 0.163 | 0.166 | 0.356 |
+| gte-modernbert-base, dense | 0.125 | 0.112 | 0.308 |
+| CodeRankEmbed, dense | 0.094 | 0.095 | 0.292 |
+| BM25 | 0.120 | 0.133 | 0.371 |
+
+The best zero-shot model, each with BM25 fusion at its own dev-chosen weight, scores 0.173 / 0.178 descriptive. The
+gain over it is +0.095 in both languages, with repository-clustered 95% intervals excluding zero.
+
+**Code-to-code and general code search.** Dense nDCG@10. For each benchmark, the right-hand column gives the best of
+granite-embedding-english-r2, bge-base-en-v1.5, modernbert-embed-base, gte-modernbert-base, CodeRankEmbed,
+jina-embeddings-v2-base-code and codesage-small-v2.
+
+| benchmark | this model | best other open ~150M |
+|---|---|---|
+| test -> implementation (Python) | **0.640** | 0.522 (CodeRankEmbed) |
+| call site -> definition, Python / TypeScript | 0.219 / **0.278** | 0.244 (bge-base) / 0.264 (granite-r2) |
+| docstring -> function, Python / TypeScript | 0.717 / 0.672 | 0.863 / 0.762 (CodeRankEmbed) |
+| CoSQA | 0.651 | 0.737 (jina-v2-code) |
+| CoIR StackOverflow QA | **0.922** | 0.917 (granite-r2) |
+| CoIR CodeSearchNet, Python / JavaScript | **0.963** / 0.634 | 0.960 (codesage) / 0.764 (jina-v2-code) |
+| CoIR CodeSearchNet-CCR, Python / JavaScript | 0.703 / 0.688 | 0.940 / 0.943 (gte-modernbert) |
+| CoIR APPS | 0.174 | 0.575 (gte-modernbert) |
+
+**Choose this model** for repository-scale retrieval driven by issues, bug reports or tests, for example as the
+retriever in a coding agent. **Choose a general code-search model instead** (gte-modernbert-base, CodeRankEmbed,
+jina-embeddings-v2-base-code) for docstring-style or web-query code search and code completion-style retrieval (CCR),
+where this model trails.
+
+## Usage
+
+Documents are code definitions prefixed with a one-line `path::QualifiedName` header in the language's comment
+syntax, as in training. Queries are raw issue text; for best results remove template boilerplate, logs and long code
+blocks first. No query or document prefix is needed.
+
+```python
+from sentence_transformers import SentenceTransformer
+
+model = SentenceTransformer("vedevpatel/CodeIntel-150M")
+code = [
+    "# src/flask/app.py::Flask.run\ndef run(self, host=None, port=None, debug=None):\n    ...",
+    "# src/utils/date.ts::parseDate\nexport function parseDate(s: string): Date { return new Date(s); }",
+]
+query = "Server ignores the port argument when debug is enabled"
+scores = model.encode([query]) @ model.encode(code).T   # cosine similarities (embeddings are normalised)
+```
+
+On a GPU, load in half precision (bf16 on Ampere or newer, fp16 otherwise); the default is float32 for CPU. With
+`flash-attn` installed, `attn_implementation="flash_attention_2"` speeds up long inputs further.
+
+```python
+import torch
+
+model = SentenceTransformer("vedevpatel/CodeIntel-150M", device="cuda", model_kwargs={"dtype": torch.bfloat16})
+```
+
+With plain transformers:
+
+```python
+import torch
+from transformers import AutoModel, AutoTokenizer
+
+tok = AutoTokenizer.from_pretrained("vedevpatel/CodeIntel-150M")
+model = AutoModel.from_pretrained("vedevpatel/CodeIntel-150M").eval()
+batch = tok(texts, padding=True, truncation=True, max_length=1024, return_tensors="pt")
+with torch.no_grad():
+    emb = torch.nn.functional.normalize(model(**batch).last_hidden_state[:, 0], dim=-1)
+```
+
+**ONNX / INT8.** `onnx/model.onnx` (fp32) and `onnx/model_int8.onnx` (dynamic INT8; the GeGLU down-projections stay
+fp32). Outputs are the last hidden states; take the CLS vector and L2-normalise it. The INT8 model's minimum cosine
+to PyTorch is 0.995. On dev retrieval its nDCG@10 matches fp32 within noise: CoSQA +0.003 [-0.005, +0.011],
+docstrings-TypeScript -0.002 [-0.010, +0.006]. It is about 1.7x faster on CPU.
+
+**Hybrid retrieval.** Fusing dense scores with BM25 by reciprocal-rank fusion (dense weight 2, k = 60) adds about
++0.03 on issue localization.
+
+## Training
+
+- **Objective:** multi-positive InfoNCE (symmetric, temperature 0.05) with BM25-mined hard negatives and GradCache
+  (batch 128). Plus distillation (weight 0.5) toward the frozen base model's similarity structure, which preserves
+  general search ability. 3,000 steps, learning rate 5e-6, bf16, maximum document length 512.
+- **Data** (422k query groups; every source permissively licensed):
+  - 82 GitHub repositories (MIT, Apache-2.0, BSD, ISC and similar; each licence verified at every revision code was
+    taken from; none is an evaluation or development repository): issue -> edited function and commit -> edited
+    function (issue text as originally opened where recoverable), docstring -> function, test -> implementation, call
+    site -> definition, code context -> continuation.
+  - [The Vault](https://huggingface.co/datasets/Fsoft-AIC/the-vault-function): docstring -> function and code-context
+    pairs, only items whose declared licences are all permissive.
+  - [CodeSearchNet](https://huggingface.co/datasets/code-search-net/code_search_net) train split (Python,
+    JavaScript): code -> docstring, code -> continuation and docstring -> code. Rows kept only if the licence GitHub
+    detects at the row's own commit is permissive (3,213 of the 4,000 largest repositories). CoIR's CodeSearchNet
+    tasks are built from CodeSearchNet's test split, whose repositories are disjoint from the train split.
+
+  All data was decontaminated against every evaluation and development text, including the CoIR corpora (exact and
+  MinHash near-duplicates).
+- **Sources and licences:** every repository and dataset used is listed in [TRAINING_DATA.md](TRAINING_DATA.md).
+- **Selection:** the final checkpoint, chosen on a separate development set of 19 repositories by a fixed
+  rule: maximise descriptive localization, subject to no loss on lexical localization, general search and
+  code-to-code retrieval.
+- **Compute:** one NVIDIA H100 80GB, about 1.6 hours (embedding cache 10 minutes, training 87 minutes).
+
+## Limitations
+
+- Issue localization remains hard. Even this model puts an edited function in the top 10 for well under half of
+  descriptive issues.
+- Fixes that only add new code cannot be retrieved, because the target does not exist yet. The benchmark reports this
+  as a recall ceiling.
+- Evaluated on Python and TypeScript only. Behaviour on other languages is unknown.
+- Docstring-style search and CoSQA are at the base model's level, below CodeRankEmbed and jina-embeddings-v2-base-code.
+- General natural-language code search and code-continuation retrieval (CoIR CCR, APPS) remain below the best open
+  models of this size.
+
+## Citation
+
+```bibtex
+@misc{<CITATION_KEY>,
+  title  = {CodeIntel-150M: an embedding model for issue-to-code localization},
+  author = {<AUTHORS>},
+  year   = {2026},
+  url    = {https://huggingface.co/vedevpatel/CodeIntel-150M}
+}
+```

config.json

@@ -0,0 +1,81 @@
+{
+  "architectures": [
+    "ModernBertModel"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 50281,
+  "classifier_activation": "silu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.0,
+  "classifier_pooling": "mean",
+  "cls_token_id": 50281,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "dtype": "float32",
+  "embedding_dropout": 0.0,
+  "eos_token_id": 50282,
+  "global_attn_every_n_layers": 3,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_activation": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 1152,
+  "layer_norm_eps": 1e-05,
+  "layer_types": [
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "local_attention": 128,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 12,
+  "num_hidden_layers": 22,
+  "pad_token_id": 50283,
+  "position_embedding_type": "absolute",
+  "repad_logits_with_grad": false,
+  "rope_parameters": {
+    "full_attention": {
+      "rope_theta": 80000.0,
+      "rope_type": "default"
+    },
+    "sliding_attention": {
+      "rope_theta": 10000.0,
+      "rope_type": "default"
+    }
+  },
+  "sep_token_id": 50282,
+  "sparse_pred_ignore_index": -100,
+  "sparse_prediction": false,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.21.0",
+  "vocab_size": 50368
+}

config_sentence_transformers.json

@@ -0,0 +1,8 @@
+{
+  "prompts": {
+    "query": "",
+    "document": ""
+  },
+  "default_prompt_name": null,
+  "similarity_fn_name": "cosine"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2e91aef7d8d65740516546ed8a0727726f6d03a475b1625f9ac5f0ff09b874e1
+size 298041568

modules.json

@@ -0,0 +1,20 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  },
+  {
+    "idx": 2,
+    "name": "2",
+    "path": "2_Normalize",
+    "type": "sentence_transformers.models.Normalize"
+  }
+]

onnx/model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3d15615f9ca440e31e194d377fa1e5fd3c63b17fbd8008b296a4135c6619c909
+size 596654187

onnx/model_int8.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0cbc3be699563ea98743894d4b940e39fc4e71ed064c696701504b28d0d5b5a8
+size 325045235

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 1024,
+  "do_lower_case": false
+}

tokenizer_config.json

@@ -0,0 +1,21 @@
+{
+  "backend": "tokenizers",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "is_local": true,
+  "local_files_only": false,
+  "mask_token": "[MASK]",
+  "max_length": 1024,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 8192,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "stride": 0,
+  "tokenizer_class": "TokenizersBackend",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "[UNK]"
+}