vedevpatel commited on
Commit
7b63952
·
verified ·
1 Parent(s): 5723c50

Lead the model card with the issue-localization result

Browse files
Files changed (1) hide show
  1. README.md +6 -1
README.md CHANGED
@@ -39,6 +39,11 @@ model-index:
39
 
40
  # CodeIntel-150M
41
 
 
 
 
 
 
42
  A 149M-parameter embedding model for **finding the code a bug report or feature request is about**. Given an issue
43
  written by a user, it retrieves the functions and classes in a Python or TypeScript repository that a fix will touch.
44
  It also handles code-to-code navigation (test -> implementation, call site -> definition) and code search
@@ -57,7 +62,7 @@ Apache-2.0). It runs locally, including as a 325 MB INT8 ONNX model, and needs n
57
  ## Results
58
 
59
  All numbers come from held-out test data that no training or model-selection step used. Each model was scored once,
60
- with the same protocol. Comparisons are with open models of similar size, all zero-shot.
61
 
62
  **Issue -> code localization.** The test set has 3,804 GitHub issues from 35 Python and TypeScript repositories, none
63
  of which were used in training. A hit means retrieving a function or class that the issue's fix edited, among all
 
39
 
40
  # CodeIntel-150M
41
 
42
+ > **About 55% better than any other open ~150M embedding model at finding the code a bug report is about.**
43
+ > On 3,804 held-out GitHub issues that name no code, CodeIntel scores nDCG@10 **0.268 / 0.274** (Python /
44
+ > TypeScript) against **0.173 / 0.178** for the next-best of seven open models its size. It also leads on issues
45
+ > that do name code (+27%) and on test -> implementation search (+23%).
46
+
47
  A 149M-parameter embedding model for **finding the code a bug report or feature request is about**. Given an issue
48
  written by a user, it retrieves the functions and classes in a Python or TypeScript repository that a fix will touch.
49
  It also handles code-to-code navigation (test -> implementation, call site -> definition) and code search
 
62
  ## Results
63
 
64
  All numbers come from held-out test data that no training or model-selection step used. Each model was scored once,
65
+ with the same protocol. The comparison models are open models of similar size, evaluated zero-shot.
66
 
67
  **Issue -> code localization.** The test set has 3,804 GitHub issues from 35 Python and TypeScript repositories, none
68
  of which were used in training. A hit means retrieving a function or class that the issue's fix edited, among all