divakantpandey commited on
Commit
da00134
·
verified ·
1 Parent(s): 493ad91

docs: standardize benchmark claims to 150M category

Browse files
Files changed (1) hide show
  1. README.md +26 -26
README.md CHANGED
@@ -33,12 +33,12 @@ model-index:
33
  type: reranking
34
  name: CoREB(v1)
35
  dataset:
36
- name: CoREB(v1) Macro-Average (Rank #3 in <=160M Class)
37
  type: mteb/coreb
38
  metrics:
39
  - type: main_score
40
  value: 0.4362
41
- name: Main Score (Rank #3 in <=160M Class)
42
  - type: reranking_mean
43
  value: 0.4894
44
  name: Reranking Mean
@@ -60,22 +60,22 @@ model-index:
60
  type: retrieval
61
  name: CoIR
62
  dataset:
63
- name: CoIR Macro-Average (Rank #8 in <=141M Class)
64
  type: mteb/coir
65
  metrics:
66
  - type: ndcg_at_10
67
  value: 0.5343
68
- name: NDCG@10 (Rank #8 in <=141M Class)
69
  - task:
70
  type: retrieval
71
  name: MTEB(Code, v1)
72
  dataset:
73
- name: MTEB(Code, v1) Macro-Average (Rank #9 in <=141M Class)
74
  type: mteb/code
75
  metrics:
76
  - type: ndcg_at_10
77
  value: 0.5475
78
- name: NDCG@10 (Rank #9 in <=141M Class)
79
  - task:
80
  type: reranking
81
  name: Code-to-Text Reranking
@@ -236,14 +236,14 @@ model-index:
236
 
237
  > **CodeEmbed v2** (`divakantpandey/code-embed-v2`) is an ultra-compact, high-speed 131M parameter dense embedding model optimized for software engineering workflows: natural language code search, cross-language clone detection, multi-turn code assistant retrieval, and error trace debugging.
238
  >
239
- > On the official **Massive Text Embedding Benchmark (MTEB)**, CodeEmbed v2 dominates the compact model category ($\le$ 160M parameters): **Rank #3 on CoREB(v1)**, **Rank #8 on CoIR**, and **Rank #9 on MTEB(Code, v1)**, alongside **Rank #3 Worldwide** across all model sizes on Code-to-Code Reranking (`CorebC2CReranking`). It consistently outperforms enterprise frontier models including Google Gemini Embedding and Microsoft E5-Large in conversational code feedback and deep learning framework translation at a fraction of the compute and memory footprint.
240
 
241
  [![Hugging Face Model](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-divakantpandey%2Fcode--embed--v2-blue)](https://huggingface.co/divakantpandey/code-embed-v2)
242
  [![MTEB Official Leaderboard](https://img.shields.io/badge/MTEB%20Leaderboard-divakantpandey%2Fcode--embed--v2-blue)](https://mteb-leaderboard.hf.space/models/divakantpandey/code-embed-v2)
243
  [![Coreb C2C Reranking](https://img.shields.io/badge/Coreb%20C2C%20Reranking-%233%20Worldwide%20(0.4205)-gold)](https://mteb-leaderboard.hf.space/models/divakantpandey/code-embed-v2)
244
- [![CoREB(v1) Rank #3 (≤160M)](https://img.shields.io/badge/CoREB(v1)%20(≤160M)-Rank%20%233-gold)](https://mteb-leaderboard.hf.space/benchmark/CoREB(v1)?minSize=33&maxSize=158.48931924611142)
245
- [![CoIR Rank #8 (≤141M)](https://img.shields.io/badge/CoIR%20(≤141M)-Rank%20%238-gold)](https://mteb-leaderboard.hf.space/benchmark/CoIR?minSize=2&maxSize=141.2537544622754)
246
- [![MTEB Code Rank #9 (≤141M)](https://img.shields.io/badge/MTEB(Code%2C%20v1)%20(≤141M)-Rank%20%239-gold)](https://mteb-leaderboard.hf.space/benchmark/MTEB(Code%2C%20v1)?minSize=2&maxSize=141.2537544622754)
247
  [![CoREB(v1) Overall](https://img.shields.io/badge/CoREB(v1)%20Overall-Rank%20%2319-orange)](https://mteb-leaderboard.hf.space/models/divakantpandey/code-embed-v2)
248
  [![CoIR Overall](https://img.shields.io/badge/CoIR%20Overall-Rank%20%2360-orange)](https://mteb-leaderboard.hf.space/models/divakantpandey/code-embed-v2)
249
  [![MTEB Code Overall](https://img.shields.io/badge/MTEB(Code%2C%20v1)%20Overall-Rank%20%2369-orange)](https://mteb-leaderboard.hf.space/models/divakantpandey/code-embed-v2)
@@ -273,17 +273,17 @@ model-index:
273
  > The rankings, scores, and competitive metrics documented below reflect the official **Massive Text Embedding Benchmark (MTEB)** snapshot as of **October 24, 2026** (MTEB Leaderboard Build `39959364296` / `mteb==2.26.16` / PR [#803](https://github.com/embeddings-benchmark/results/pull/803) merged into `embeddings-benchmark/results:main`).
274
  > Because the MTEB Leaderboard dynamically recalculates ranks daily as new community and proprietary models are submitted, global rankings are subject to shift over time. Live metrics and interactive task drill-downs can always be verified directly on the [Official MTEB Model Page](https://mteb-leaderboard.hf.space/models/divakantpandey/code-embed-v2).
275
 
276
- ### 🥇 Top Rankings in the $\le$ 160M Parameter Class (Compact Models)
277
 
278
- When filtering by parameter scale for lightweight, production-ready edge models ($\le$ 160M parameters), CodeEmbed v2 establishes top-tier standing globally:
279
 
280
- | Rank in Class | Benchmark Suite | Parameter Filter Range | Score | Official Leaderboard Filter Link |
281
  | :---: | :--- | :---: | :---: | :--- |
282
- | 🥉 **#3** | **`CoREB(v1)`** | $\le$ 160M (`33M – 158.5M`) | **`0.4362`** | 👉 [CoREB(v1) Leaderboard (Rank #3)](https://mteb-leaderboard.hf.space/benchmark/CoREB(v1)?minSize=33&maxSize=158.48931924611142) |
283
- | 🏅 **#8** | **`CoIR`** | $\le$ 141M (`2M – 141.3M`) | **`0.5343`** | 👉 [CoIR Leaderboard (Rank #8)](https://mteb-leaderboard.hf.space/benchmark/CoIR?minSize=2&maxSize=141.2537544622754) |
284
- | 🏅 **#9** | **`MTEB(Code, v1)`** | $\le$ 141M (`2M – 141.3M`) | **`0.5475`** | 👉 [MTEB Code Leaderboard (Rank #9)](https://mteb-leaderboard.hf.space/benchmark/MTEB(Code%2C%20v1)?minSize=2&maxSize=141.2537544622754) |
285
 
286
- ### 🌟 Key Benchmark Highlights
287
 
288
  ### 🌟 Key Benchmark Highlights *(Sorted by Rank Ascending)*
289
 
@@ -293,17 +293,17 @@ When filtering by parameter scale for lightweight, production-ready edge models
293
  - Directly outperforms specialized code models including `naver/splade-code-06B` (600M, `0.4198`), `codefuse-ai/F2LLM-v2-160M` (`0.4155`), and `codefuse-ai/F2LLM-v2-330M` (`0.4143`).
294
  - Demonstrates world-class semantic code clone detection and refactoring matching.
295
 
296
- 2. **🥉 Rank #3 Worldwide on `CoREB(v1)`** *(In the $\le$ 160M parameter class)*
297
  - **Score: `0.4362`** (Pairwise Reranking Mean: `0.4894`).
298
- - Ranks 3rd globally in the compact model category on the Code Representation Benchmark suite.
299
 
300
- 3. **🏅 Rank #8 Worldwide on `CoIR`** *(In the $\le$ 141M / 160M parameter class)*
301
  - **Score: `0.5343`** (100% Zero-Shot generalizability).
302
- - Ranks 8th worldwide among compact models on Code Information Retrieval.
303
 
304
- 4. **🏅 Rank #9 Worldwide on `MTEB(Code, v1)`** *(In the $\le$ 141M / 160M parameter class)*
305
  - **Score: `0.5475`** (meanTask NDCG@10).
306
- - Ranks 9th worldwide among compact models on the primary Code benchmark suite.
307
 
308
  5. **🎖️ Rank #19 Worldwide on `CoREB(v1)` Overall** *(Across ALL model sizes)*
309
  - **Overall Rank #19 Worldwide** across all evaluated models regardless of parameter scale with a mean score of **`43.62%`**.
@@ -340,9 +340,9 @@ When filtering by parameter scale for lightweight, production-ready edge models
340
  | Benchmark / Task | Category | Metric | Rank (Class / Global) | Total Models | Score | Comparison vs Baselines |
341
  | :--- | :--- | :---: | :---: | :---: | :---: | :--- |
342
  | **`CorebC2CReranking`** | Code-to-Code Reranking | MAP | **#3 Global** | 25 | **`0.4205`** | **Top 3 Worldwide**; beats Naver SPLADE 600M (`0.4198`) |
343
- | **`CoREB(v1)` (Compact Tier)** | Full Benchmark Suite | MeanTask | **#3 (≤160M)** | 8 | **`0.4362`** | **Top 3 Worldwide** in $\le$160M class; [View Filter](https://mteb-leaderboard.hf.space/benchmark/CoREB(v1)?minSize=33&maxSize=158.48931924611142) |
344
- | **`CoIR` (Compact Tier)** | Full Benchmark Suite | MeanTask | **#8 (≤141M)** | 17 | **`0.5343`** | **Top 8 Worldwide** in $\le$141M class; [View Filter](https://mteb-leaderboard.hf.space/benchmark/CoIR?minSize=2&maxSize=141.2537544622754) |
345
- | **`MTEB(Code, v1)` (Compact Tier)** | Full Benchmark Suite | MeanTask | **#9 (≤141M)** | 17 | **`0.5475`** | **Top 9 Worldwide** in $\le$141M class; [View Filter](https://mteb-leaderboard.hf.space/benchmark/MTEB(Code%2C%20v1)?minSize=2&maxSize=141.2537544622754) |
346
  | **`CoREB(v1)` Macro-Average** | Full Benchmark Suite | MeanTask | **#19 Global** | 25 | **`0.4362`** | **Top 20 Worldwide** (Reranking: `0.4894`, Retrieval: `0.3831`) |
347
  | **`CorebC2TReranking`** | Code-to-Text Reranking | MAP | **#20 Global** | 25 | **`0.7937`** | ~80% pairwise alignment accuracy |
348
  | **`CorebT2CReranking`** | Text-to-Code Reranking | MAP | **#21 Global** | 25 | **`0.2539`** | Near 7B LLM ceiling (`0.2882`) |
 
33
  type: reranking
34
  name: CoREB(v1)
35
  dataset:
36
+ name: CoREB(v1) Macro-Average (Rank #3 in <=150M Category)
37
  type: mteb/coreb
38
  metrics:
39
  - type: main_score
40
  value: 0.4362
41
+ name: Main Score (Rank #3 in <=150M Category)
42
  - type: reranking_mean
43
  value: 0.4894
44
  name: Reranking Mean
 
60
  type: retrieval
61
  name: CoIR
62
  dataset:
63
+ name: CoIR Macro-Average (Rank #8 in <=150M Category)
64
  type: mteb/coir
65
  metrics:
66
  - type: ndcg_at_10
67
  value: 0.5343
68
+ name: NDCG@10 (Rank #8 in <=150M Category)
69
  - task:
70
  type: retrieval
71
  name: MTEB(Code, v1)
72
  dataset:
73
+ name: MTEB(Code, v1) Macro-Average (Rank #9 in <=150M Category)
74
  type: mteb/code
75
  metrics:
76
  - type: ndcg_at_10
77
  value: 0.5475
78
+ name: NDCG@10 (Rank #9 in <=150M Category)
79
  - task:
80
  type: reranking
81
  name: Code-to-Text Reranking
 
236
 
237
  > **CodeEmbed v2** (`divakantpandey/code-embed-v2`) is an ultra-compact, high-speed 131M parameter dense embedding model optimized for software engineering workflows: natural language code search, cross-language clone detection, multi-turn code assistant retrieval, and error trace debugging.
238
  >
239
+ > On the official **Massive Text Embedding Benchmark (MTEB)**, CodeEmbed v2 dominates the 150M compact model category ($\le$ 150M parameters): **Rank #3 on CoREB(v1)**, **Rank #8 on CoIR**, and **Rank #9 on MTEB(Code, v1)**, alongside **Rank #3 Worldwide** across all model sizes on Code-to-Code Reranking (`CorebC2CReranking`). It consistently outperforms enterprise frontier models including Google Gemini Embedding and Microsoft E5-Large in conversational code feedback and deep learning framework translation at a fraction of the compute and memory footprint.
240
 
241
  [![Hugging Face Model](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-divakantpandey%2Fcode--embed--v2-blue)](https://huggingface.co/divakantpandey/code-embed-v2)
242
  [![MTEB Official Leaderboard](https://img.shields.io/badge/MTEB%20Leaderboard-divakantpandey%2Fcode--embed--v2-blue)](https://mteb-leaderboard.hf.space/models/divakantpandey/code-embed-v2)
243
  [![Coreb C2C Reranking](https://img.shields.io/badge/Coreb%20C2C%20Reranking-%233%20Worldwide%20(0.4205)-gold)](https://mteb-leaderboard.hf.space/models/divakantpandey/code-embed-v2)
244
+ [![CoREB(v1) Rank #3 (≤150M)](https://img.shields.io/badge/CoREB(v1)%20(≤150M)-Rank%20%233-gold)](https://mteb-leaderboard.hf.space/benchmark/CoREB(v1)?minSize=33&maxSize=150)
245
+ [![CoIR Rank #8 (≤150M)](https://img.shields.io/badge/CoIR%20(≤150M)-Rank%20%238-gold)](https://mteb-leaderboard.hf.space/benchmark/CoIR?minSize=2&maxSize=150)
246
+ [![MTEB Code Rank #9 (≤150M)](https://img.shields.io/badge/MTEB(Code%2C%20v1)%20(≤150M)-Rank%20%239-gold)](https://mteb-leaderboard.hf.space/benchmark/MTEB(Code%2C%20v1)?minSize=2&maxSize=150)
247
  [![CoREB(v1) Overall](https://img.shields.io/badge/CoREB(v1)%20Overall-Rank%20%2319-orange)](https://mteb-leaderboard.hf.space/models/divakantpandey/code-embed-v2)
248
  [![CoIR Overall](https://img.shields.io/badge/CoIR%20Overall-Rank%20%2360-orange)](https://mteb-leaderboard.hf.space/models/divakantpandey/code-embed-v2)
249
  [![MTEB Code Overall](https://img.shields.io/badge/MTEB(Code%2C%20v1)%20Overall-Rank%20%2369-orange)](https://mteb-leaderboard.hf.space/models/divakantpandey/code-embed-v2)
 
273
  > The rankings, scores, and competitive metrics documented below reflect the official **Massive Text Embedding Benchmark (MTEB)** snapshot as of **October 24, 2026** (MTEB Leaderboard Build `39959364296` / `mteb==2.26.16` / PR [#803](https://github.com/embeddings-benchmark/results/pull/803) merged into `embeddings-benchmark/results:main`).
274
  > Because the MTEB Leaderboard dynamically recalculates ranks daily as new community and proprietary models are submitted, global rankings are subject to shift over time. Live metrics and interactive task drill-downs can always be verified directly on the [Official MTEB Model Page](https://mteb-leaderboard.hf.space/models/divakantpandey/code-embed-v2).
275
 
276
+ ### 🥇 Top Rankings in the $\le$ 150M Parameter Category (Compact Models)
277
 
278
+ When filtering by parameter scale for lightweight, production-ready edge models ($\le$ 150M category), CodeEmbed v2 establishes top-tier standing globally:
279
 
280
+ | Rank in Category | Benchmark Suite | Parameter Category | Score | Official Leaderboard Filter Link |
281
  | :---: | :--- | :---: | :---: | :--- |
282
+ | 🥉 **#3** | **`CoREB(v1)`** | $\le$ 150M Category | **`0.4362`** | 👉 [CoREB(v1) Leaderboard (Rank #3)](https://mteb-leaderboard.hf.space/benchmark/CoREB(v1)?minSize=33&maxSize=150) |
283
+ | 🏅 **#8** | **`CoIR`** | $\le$ 150M Category | **`0.5343`** | 👉 [CoIR Leaderboard (Rank #8)](https://mteb-leaderboard.hf.space/benchmark/CoIR?minSize=2&maxSize=150) |
284
+ | 🏅 **#9** | **`MTEB(Code, v1)`** | $\le$ 150M Category | **`0.5475`** | 👉 [MTEB Code Leaderboard (Rank #9)](https://mteb-leaderboard.hf.space/benchmark/MTEB(Code%2C%20v1)?minSize=2&maxSize=150) |
285
 
286
+ ---
287
 
288
  ### 🌟 Key Benchmark Highlights *(Sorted by Rank Ascending)*
289
 
 
293
  - Directly outperforms specialized code models including `naver/splade-code-06B` (600M, `0.4198`), `codefuse-ai/F2LLM-v2-160M` (`0.4155`), and `codefuse-ai/F2LLM-v2-330M` (`0.4143`).
294
  - Demonstrates world-class semantic code clone detection and refactoring matching.
295
 
296
+ 2. **🥉 Rank #3 Worldwide on `CoREB(v1)`** *(In the $\le$ 150M category)*
297
  - **Score: `0.4362`** (Pairwise Reranking Mean: `0.4894`).
298
+ - Ranks 3rd globally in the 150M compact model category on the Code Representation Benchmark suite.
299
 
300
+ 3. **🏅 Rank #8 Worldwide on `CoIR`** *(In the $\le$ 150M category)*
301
  - **Score: `0.5343`** (100% Zero-Shot generalizability).
302
+ - Ranks 8th worldwide in the 150M category on Code Information Retrieval.
303
 
304
+ 4. **🏅 Rank #9 Worldwide on `MTEB(Code, v1)`** *(In the $\le$ 150M category)*
305
  - **Score: `0.5475`** (meanTask NDCG@10).
306
+ - Ranks 9th worldwide in the 150M category on the primary Code benchmark suite.
307
 
308
  5. **🎖️ Rank #19 Worldwide on `CoREB(v1)` Overall** *(Across ALL model sizes)*
309
  - **Overall Rank #19 Worldwide** across all evaluated models regardless of parameter scale with a mean score of **`43.62%`**.
 
340
  | Benchmark / Task | Category | Metric | Rank (Class / Global) | Total Models | Score | Comparison vs Baselines |
341
  | :--- | :--- | :---: | :---: | :---: | :---: | :--- |
342
  | **`CorebC2CReranking`** | Code-to-Code Reranking | MAP | **#3 Global** | 25 | **`0.4205`** | **Top 3 Worldwide**; beats Naver SPLADE 600M (`0.4198`) |
343
+ | **`CoREB(v1)` (Compact Tier)** | Full Benchmark Suite | MeanTask | **#3 (≤150M)** | 8 | **`0.4362`** | **Top 3 Worldwide** in $\le$150M category; [View Filter](https://mteb-leaderboard.hf.space/benchmark/CoREB(v1)?minSize=33&maxSize=150) |
344
+ | **`CoIR` (Compact Tier)** | Full Benchmark Suite | MeanTask | **#8 (≤150M)** | 17 | **`0.5343`** | **Top 8 Worldwide** in $\le$150M category; [View Filter](https://mteb-leaderboard.hf.space/benchmark/CoIR?minSize=2&maxSize=150) |
345
+ | **`MTEB(Code, v1)` (Compact Tier)** | Full Benchmark Suite | MeanTask | **#9 (≤150M)** | 17 | **`0.5475`** | **Top 9 Worldwide** in $\le$150M category; [View Filter](https://mteb-leaderboard.hf.space/benchmark/MTEB(Code%2C%20v1)?minSize=2&maxSize=150) |
346
  | **`CoREB(v1)` Macro-Average** | Full Benchmark Suite | MeanTask | **#19 Global** | 25 | **`0.4362`** | **Top 20 Worldwide** (Reranking: `0.4894`, Retrieval: `0.3831`) |
347
  | **`CorebC2TReranking`** | Code-to-Text Reranking | MAP | **#20 Global** | 25 | **`0.7937`** | ~80% pairwise alignment accuracy |
348
  | **`CorebT2CReranking`** | Text-to-Code Reranking | MAP | **#21 Global** | 25 | **`0.2539`** | Near 7B LLM ceiling (`0.2882`) |