Stage-2 code embedding model: permissive-only corpus, function-level contrastive tuning
c3ebc49 verified | { | |
| "name": "CoIR (NDCG@10)", | |
| "tasks": { | |
| "CodeSearchNet-go": 68.71, | |
| "CodeSearchNet-javascript": 50.41, | |
| "CodeSearchNet-ruby": 56.52, | |
| "CodeSearchNet-python": 87.84, | |
| "CodeSearchNet-php": 53.36, | |
| "synthetic-text2sql": 35.73, | |
| "codefeedback-mt": 21.78, | |
| "codefeedback-st": 48.58, | |
| "codetrans-dl": 24.12, | |
| "cosqa": 25.95, | |
| "stackoverflow-qa": 55.35, | |
| "apps": 2.83 | |
| }, | |
| "baselines": { | |
| "v1 (file-level pairs)": { | |
| "stackoverflow-qa": 58.4, | |
| "cosqa": 20.91, | |
| "apps": 3.08, | |
| "codetrans-dl": 28.14, | |
| "CodeSearchNet-ruby": 40.68, | |
| "CodeSearchNet-javascript": 38.73, | |
| "CodeSearchNet-go": 53.43, | |
| "CodeSearchNet-java": 37.87 | |
| }, | |
| "BM25": { | |
| "cosqa": 13.96, | |
| "stackoverflow-qa": 56.8, | |
| "codetrans-dl": 50.13 | |
| }, | |
| "UniXcoder": { | |
| "cosqa": 25.14, | |
| "stackoverflow-qa": 44.67, | |
| "codetrans-dl": 41.82 | |
| }, | |
| "GTE-Base": { | |
| "cosqa": 30.24, | |
| "stackoverflow-qa": 62.71, | |
| "codetrans-dl": 33.81 | |
| }, | |
| "E5-Base": { | |
| "cosqa": 32.59, | |
| "stackoverflow-qa": 86.86, | |
| "codetrans-dl": 62.5 | |
| } | |
| }, | |
| "note": "Measured with the `coir-eval` harness, mean pooling, 256 tokens, cosine similarity. `v1` is the previous version of this model, run on the same machine with the same harness; blanks are tasks that version did not complete before its spot instance was preempted.\n\nBM25, UniXcoder, GTE-Base and E5-Base are quoted from Table 3 of the CoIR paper (arXiv:2407.02883v3). That table reports CodeSearchNet as one aggregate across six languages, while the figures here are per language, so those rows are left blank rather than compared to something they are not.\n\n**No benchmark average is quoted.** Two of CoIR's tasks (codesearchnet-ccr, codetrans-contest) were not run, and a mean over a subset is not the published mean \u2014 putting one beside a leaderboard average would be a false comparison." | |
| } |