File size: 3,415 Bytes
fbefc90 8d17091 56cfb2c 8d17091 56cfb2c 8d17091 56cfb2c 8d17091 56cfb2c 8d17091 56cfb2c 8d17091 56cfb2c 8d17091 56cfb2c 8d17091 fbefc90 8d17091 fbefc90 a2ea42b fbefc90 8d17091 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 | ---
datasets:
- minishlab/tokenlearn-cornstack-docs-coderankembed-v2
- minishlab/tokenlearn-cornstack-queries-coderankembed-v2
language:
- code
library_name: model2vec
license: mit
pipeline_tag: feature-extraction
tags:
- embeddings
- static-embeddings
- code-retrieval
- tokenlearn
- coir
model-index:
- name: miru-codev3-tokenlearn
results:
- task:
type: Retrieval
name: CoIR (dense)
dataset:
name: CoIR average (6 tasks) [dense]
type: coir-dense/average
metrics:
- type: ndcg_at_10
value: 38.26
name: NDCG@10
source:
url: https://github.com/CoIR-team/coir
name: CoIR / MTEB (self-reported)
- task:
type: Retrieval
name: CoIR (dense)
dataset:
name: AppsRetrieval [dense]
type: coir-dense/AppsRetrieval
metrics:
- type: ndcg_at_10
value: 4.23
name: NDCG@10
source:
url: https://github.com/CoIR-team/coir
name: CoIR / MTEB (self-reported)
- task:
type: Retrieval
name: CoIR (dense)
dataset:
name: COIRCodeSearchNet [dense]
type: coir-dense/COIRCodeSearchNetRetrieval
metrics:
- type: ndcg_at_10
value: 44.93
name: NDCG@10
source:
url: https://github.com/CoIR-team/coir
name: CoIR / MTEB (self-reported)
- task:
type: Retrieval
name: CoIR (dense)
dataset:
name: CodeFeedbackST [dense]
type: coir-dense/CodeFeedbackST
metrics:
- type: ndcg_at_10
value: 53.04
name: NDCG@10
source:
url: https://github.com/CoIR-team/coir
name: CoIR / MTEB (self-reported)
- task:
type: Retrieval
name: CoIR (dense)
dataset:
name: CodeSearchNetCC [dense]
type: coir-dense/CodeSearchNetCCRetrieval
metrics:
- type: ndcg_at_10
value: 42.56
name: NDCG@10
source:
url: https://github.com/CoIR-team/coir
name: CoIR / MTEB (self-reported)
- task:
type: Retrieval
name: CoIR (dense)
dataset:
name: CosQA [dense]
type: coir-dense/CosQA
metrics:
- type: ndcg_at_10
value: 25.76
name: NDCG@10
source:
url: https://github.com/CoIR-team/coir
name: CoIR / MTEB (self-reported)
- task:
type: Retrieval
name: CoIR (dense)
dataset:
name: StackOverflowQA [dense]
type: coir-dense/StackOverflowQA
metrics:
- type: ndcg_at_10
value: 59.04
name: NDCG@10
source:
url: https://github.com/CoIR-team/coir
name: CoIR / MTEB (self-reported)
---
# miru-codev3-tokenlearn
Owned **Tokenlearn** static code embedding bag (256-d, Model2Vec) — branch **A** of [miru-codev3-dual](https://huggingface.co/Takara-DS1/miru-codev3-dual).
Trained with Zipf-SIF Tokenlearn on CornStack coderankembed-v2 features (`code_static.train_miru_codev3`).
## Evaluation
CoIR / MTEB **NDCG@10** (×100), dense retrieval only. Scores are self-reported from the training repo eval harness.
## Usage
```python
from model2vec import StaticModel
model = StaticModel.from_pretrained("Takara-DS1/miru-codev3-tokenlearn")
emb = model.encode(["def add(a, b): return a + b"])
```
For the 512-d dual (this bag ⊕ distill_fuse), use [Takara-DS1/miru-codev3-dual](https://huggingface.co/Takara-DS1/miru-codev3-dual).
## Recreate
See the dual repo `reproduce/REPRODUCE.md` (step 1: `train_miru_codev3`).
## License
MIT
|