| --- |
| model-index: |
| - name: voyage-code-3 |
| results: |
| - dataset: |
| config: default |
| name: MTEB AppsRetrieval (default) |
| revision: f22508f96b7a36c2415181ed8bb76f76e04ae2d5 |
| split: test |
| type: CoIR-Retrieval/apps |
| metrics: |
| - type: main_score |
| value: 93.621 |
| - type: map_at_1 |
| value: 87.968 |
| - type: map_at_10 |
| value: 92.047 |
| - type: map_at_100 |
| value: 92.11500000000001 |
| - type: map_at_1000 |
| value: 92.11500000000001 |
| - type: map_at_20 |
| value: 92.091 |
| - type: map_at_3 |
| value: 91.532 |
| - type: map_at_5 |
| value: 91.874 |
| - type: mrr_at_1 |
| value: 87.99468791500664 |
| - type: mrr_at_10 |
| value: 92.0623432197138 |
| - type: mrr_at_100 |
| value: 92.13012638701028 |
| - type: mrr_at_1000 |
| value: 92.1308801595124 |
| - type: mrr_at_20 |
| value: 92.10639248723756 |
| - type: mrr_at_3 |
| value: 91.54050464807443 |
| - type: mrr_at_5 |
| value: 91.88977423638782 |
| - type: nauc_map_at_1000_diff1 |
| value: 90.15490174664647 |
| - type: nauc_map_at_1000_max |
| value: 67.02866848649028 |
| - type: nauc_map_at_1000_std |
| value: -43.43778333375324 |
| - type: nauc_map_at_100_diff1 |
| value: 90.15545196165746 |
| - type: nauc_map_at_100_max |
| value: 67.03097567361978 |
| - type: nauc_map_at_100_std |
| value: -43.43758797549862 |
| - type: nauc_map_at_10_diff1 |
| value: 90.14099743587612 |
| - type: nauc_map_at_10_max |
| value: 67.16096459752251 |
| - type: nauc_map_at_10_std |
| value: -43.44370410695505 |
| - type: nauc_map_at_1_diff1 |
| value: 90.99585708349436 |
| - type: nauc_map_at_1_max |
| value: 63.57782533912141 |
| - type: nauc_map_at_1_std |
| value: -42.361341394612126 |
| - type: nauc_map_at_20_diff1 |
| value: 90.1551408410763 |
| - type: nauc_map_at_20_max |
| value: 67.07765073421193 |
| - type: nauc_map_at_20_std |
| value: -43.41448959858248 |
| - type: nauc_map_at_3_diff1 |
| value: 89.8995753062064 |
| - type: nauc_map_at_3_max |
| value: 67.61366242819365 |
| - type: nauc_map_at_3_std |
| value: -44.226914049035386 |
| - type: nauc_map_at_5_diff1 |
| value: 90.09003396740809 |
| - type: nauc_map_at_5_max |
| value: 67.4059817584288 |
| - type: nauc_map_at_5_std |
| value: -43.62598115209535 |
| - type: nauc_mrr_at_1000_diff1 |
| value: 90.1224401379764 |
| - type: nauc_mrr_at_1000_max |
| value: 67.0844465768374 |
| - type: nauc_mrr_at_1000_std |
| value: -43.634995805553366 |
| - type: nauc_mrr_at_100_diff1 |
| value: 90.12299449341405 |
| - type: nauc_mrr_at_100_max |
| value: 67.08675243260438 |
| - type: nauc_mrr_at_100_std |
| value: -43.634780889033514 |
| - type: nauc_mrr_at_10_diff1 |
| value: 90.10879634384949 |
| - type: nauc_mrr_at_10_max |
| value: 67.21648611418671 |
| - type: nauc_mrr_at_10_std |
| value: -43.63919542502753 |
| - type: nauc_mrr_at_1_diff1 |
| value: 90.94858533614968 |
| - type: nauc_mrr_at_1_max |
| value: 63.65698161247239 |
| - type: nauc_mrr_at_1_std |
| value: -42.61438951155136 |
| - type: nauc_mrr_at_20_diff1 |
| value: 90.1227820706247 |
| - type: nauc_mrr_at_20_max |
| value: 67.1333396707349 |
| - type: nauc_mrr_at_20_std |
| value: -43.61103858801164 |
| - type: nauc_mrr_at_3_diff1 |
| value: 89.87704691693779 |
| - type: nauc_mrr_at_3_max |
| value: 67.69227631885192 |
| - type: nauc_mrr_at_3_std |
| value: -44.3319988628551 |
| - type: nauc_mrr_at_5_diff1 |
| value: 90.05844149558918 |
| - type: nauc_mrr_at_5_max |
| value: 67.46072486731265 |
| - type: nauc_mrr_at_5_std |
| value: -43.81755835911823 |
| - type: nauc_ndcg_at_1000_diff1 |
| value: 90.04496527123014 |
| - type: nauc_ndcg_at_1000_max |
| value: 67.60247091688248 |
| - type: nauc_ndcg_at_1000_std |
| value: -43.03623963675031 |
| - type: nauc_ndcg_at_100_diff1 |
| value: 90.070471931644 |
| - type: nauc_ndcg_at_100_max |
| value: 67.68039037445462 |
| - type: nauc_ndcg_at_100_std |
| value: -43.036110157737035 |
| - type: nauc_ndcg_at_10_diff1 |
| value: 90.0028263117178 |
| - type: nauc_ndcg_at_10_max |
| value: 68.38084649569117 |
| - type: nauc_ndcg_at_10_std |
| value: -42.96245497355038 |
| - type: nauc_ndcg_at_1_diff1 |
| value: 90.99585708349436 |
| - type: nauc_ndcg_at_1_max |
| value: 63.57782533912141 |
| - type: nauc_ndcg_at_1_std |
| value: -42.361341394612126 |
| - type: nauc_ndcg_at_20_diff1 |
| value: 90.06473833364464 |
| - type: nauc_ndcg_at_20_max |
| value: 68.04393781250494 |
| - type: nauc_ndcg_at_20_std |
| value: -42.754643190221756 |
| - type: nauc_ndcg_at_3_diff1 |
| value: 89.47150366489366 |
| - type: nauc_ndcg_at_3_max |
| value: 69.28073683657388 |
| - type: nauc_ndcg_at_3_std |
| value: -44.80650989406889 |
| - type: nauc_ndcg_at_5_diff1 |
| value: 89.85493663258895 |
| - type: nauc_ndcg_at_5_max |
| value: 68.97238224506673 |
| - type: nauc_ndcg_at_5_std |
| value: -43.59588098814929 |
| - type: nauc_precision_at_1000_diff1 |
| value: 71.5228076434295 |
| - type: nauc_precision_at_1000_max |
| value: 100.0 |
| - type: nauc_precision_at_1000_std |
| value: 95.64334438407597 |
| - type: nauc_precision_at_100_diff1 |
| value: 89.05538400916919 |
| - type: nauc_precision_at_100_max |
| value: 93.59380801456066 |
| - type: nauc_precision_at_100_std |
| value: 3.4218682638907 |
| - type: nauc_precision_at_10_diff1 |
| value: 88.82693413470759 |
| - type: nauc_precision_at_10_max |
| value: 83.71281148941699 |
| - type: nauc_precision_at_10_std |
| value: -33.65732721164693 |
| - type: nauc_precision_at_1_diff1 |
| value: 90.99585708349436 |
| - type: nauc_precision_at_1_max |
| value: 63.57782533912141 |
| - type: nauc_precision_at_1_std |
| value: -42.361341394612126 |
| - type: nauc_precision_at_20_diff1 |
| value: 89.60083706628117 |
| - type: nauc_precision_at_20_max |
| value: 85.08806968533703 |
| - type: nauc_precision_at_20_std |
| value: -22.269540063924854 |
| - type: nauc_precision_at_3_diff1 |
| value: 87.16551109941072 |
| - type: nauc_precision_at_3_max |
| value: 78.32435110823177 |
| - type: nauc_precision_at_3_std |
| value: -47.825568017450934 |
| - type: nauc_precision_at_5_diff1 |
| value: 88.21645334780426 |
| - type: nauc_precision_at_5_max |
| value: 81.21614910200276 |
| - type: nauc_precision_at_5_std |
| value: -42.6577409583649 |
| - type: nauc_recall_at_1000_diff1 |
| value: 71.52280764343946 |
| - type: nauc_recall_at_1000_max |
| value: 100.0 |
| - type: nauc_recall_at_1000_std |
| value: 95.64334438409236 |
| - type: nauc_recall_at_100_diff1 |
| value: 89.05538400917727 |
| - type: nauc_recall_at_100_max |
| value: 93.59380801455806 |
| - type: nauc_recall_at_100_std |
| value: 3.4218682638950693 |
| - type: nauc_recall_at_10_diff1 |
| value: 88.82693413470616 |
| - type: nauc_recall_at_10_max |
| value: 83.71281148941586 |
| - type: nauc_recall_at_10_std |
| value: -33.65732721164498 |
| - type: nauc_recall_at_1_diff1 |
| value: 90.99585708349436 |
| - type: nauc_recall_at_1_max |
| value: 63.57782533912141 |
| - type: nauc_recall_at_1_std |
| value: -42.361341394612126 |
| - type: nauc_recall_at_20_diff1 |
| value: 89.60083706628052 |
| - type: nauc_recall_at_20_max |
| value: 85.08806968533526 |
| - type: nauc_recall_at_20_std |
| value: -22.269540063922193 |
| - type: nauc_recall_at_3_diff1 |
| value: 87.16551109941042 |
| - type: nauc_recall_at_3_max |
| value: 78.32435110823131 |
| - type: nauc_recall_at_3_std |
| value: -47.825568017451694 |
| - type: nauc_recall_at_5_diff1 |
| value: 88.21645334780379 |
| - type: nauc_recall_at_5_max |
| value: 81.21614910200307 |
| - type: nauc_recall_at_5_std |
| value: -42.65774095836507 |
| - type: ndcg_at_1 |
| value: 87.968 |
| - type: ndcg_at_10 |
| value: 93.621 |
| - type: ndcg_at_100 |
| value: 93.929 |
| - type: ndcg_at_1000 |
| value: 93.949 |
| - type: ndcg_at_20 |
| value: 93.77799999999999 |
| - type: ndcg_at_3 |
| value: 92.60199999999999 |
| - type: ndcg_at_5 |
| value: 93.217 |
| - type: precision_at_1 |
| value: 87.968 |
| - type: precision_at_10 |
| value: 9.838 |
| - type: precision_at_100 |
| value: 0.9979999999999999 |
| - type: precision_at_1000 |
| value: 0.1 |
| - type: precision_at_20 |
| value: 4.95 |
| - type: precision_at_3 |
| value: 31.89 |
| - type: precision_at_5 |
| value: 19.432 |
| - type: recall_at_1 |
| value: 87.968 |
| - type: recall_at_10 |
| value: 98.38 |
| - type: recall_at_100 |
| value: 99.761 |
| - type: recall_at_1000 |
| value: 99.92 |
| - type: recall_at_20 |
| value: 98.991 |
| - type: recall_at_3 |
| value: 95.67099999999999 |
| - type: recall_at_5 |
| value: 97.158 |
| task: |
| type: Retrieval |
| - dataset: |
| config: python |
| name: MTEB COIRCodeSearchNetRetrieval (python) |
| revision: 4adc7bc41202b5c13543c9c886a25f340634dab3 |
| split: test |
| type: CoIR-Retrieval/CodeSearchNet |
| metrics: |
| - type: main_score |
| value: 98.367 |
| - type: map_at_1 |
| value: 96.494 |
| - type: map_at_10 |
| value: 97.881 |
| - type: map_at_100 |
| value: 97.89099999999999 |
| - type: map_at_1000 |
| value: 97.89099999999999 |
| - type: map_at_20 |
| value: 97.88799999999999 |
| - type: map_at_3 |
| value: 97.761 |
| - type: map_at_5 |
| value: 97.849 |
| - type: mrr_at_1 |
| value: 96.49416811905081 |
| - type: mrr_at_10 |
| value: 97.88111783570305 |
| - type: mrr_at_100 |
| value: 97.8906549372564 |
| - type: mrr_at_1000 |
| value: 97.89071223051654 |
| - type: mrr_at_20 |
| value: 97.88807761475906 |
| - type: mrr_at_3 |
| value: 97.76109398042645 |
| - type: mrr_at_5 |
| value: 97.84924252580788 |
| - type: nauc_map_at_1000_diff1 |
| value: 98.15528913593722 |
| - type: nauc_map_at_1000_max |
| value: 85.10151231357158 |
| - type: nauc_map_at_1000_std |
| value: -42.70593910947264 |
| - type: nauc_map_at_100_diff1 |
| value: 98.15533924129993 |
| - type: nauc_map_at_100_max |
| value: 85.10116247232071 |
| - type: nauc_map_at_100_std |
| value: -42.70950485228204 |
| - type: nauc_map_at_10_diff1 |
| value: 98.15404628849377 |
| - type: nauc_map_at_10_max |
| value: 85.129420000526 |
| - type: nauc_map_at_10_std |
| value: -43.05569868322259 |
| - type: nauc_map_at_1_diff1 |
| value: 98.05311494766504 |
| - type: nauc_map_at_1_max |
| value: 83.37279874457242 |
| - type: nauc_map_at_1_std |
| value: -36.78783832567891 |
| - type: nauc_map_at_20_diff1 |
| value: 98.15680755256112 |
| - type: nauc_map_at_20_max |
| value: 85.1144859632837 |
| - type: nauc_map_at_20_std |
| value: -42.76588304453336 |
| - type: nauc_map_at_3_diff1 |
| value: 98.19103616317174 |
| - type: nauc_map_at_3_max |
| value: 85.64151924417325 |
| - type: nauc_map_at_3_std |
| value: -43.86751765575472 |
| - type: nauc_map_at_5_diff1 |
| value: 98.16782356611095 |
| - type: nauc_map_at_5_max |
| value: 85.28003038549159 |
| - type: nauc_map_at_5_std |
| value: -43.74493490831982 |
| - type: nauc_mrr_at_1000_diff1 |
| value: 98.15528913593722 |
| - type: nauc_mrr_at_1000_max |
| value: 85.10151231357158 |
| - type: nauc_mrr_at_1000_std |
| value: -42.70593910947264 |
| - type: nauc_mrr_at_100_diff1 |
| value: 98.15533924129993 |
| - type: nauc_mrr_at_100_max |
| value: 85.10116247232071 |
| - type: nauc_mrr_at_100_std |
| value: -42.70950485228204 |
| - type: nauc_mrr_at_10_diff1 |
| value: 98.15404628849377 |
| - type: nauc_mrr_at_10_max |
| value: 85.129420000526 |
| - type: nauc_mrr_at_10_std |
| value: -43.05569868322259 |
| - type: nauc_mrr_at_1_diff1 |
| value: 98.05311494766504 |
| - type: nauc_mrr_at_1_max |
| value: 83.37279874457242 |
| - type: nauc_mrr_at_1_std |
| value: -36.78783832567891 |
| - type: nauc_mrr_at_20_diff1 |
| value: 98.15680755256112 |
| - type: nauc_mrr_at_20_max |
| value: 85.1144859632837 |
| - type: nauc_mrr_at_20_std |
| value: -42.76588304453336 |
| - type: nauc_mrr_at_3_diff1 |
| value: 98.19103616317174 |
| - type: nauc_mrr_at_3_max |
| value: 85.64151924417325 |
| - type: nauc_mrr_at_3_std |
| value: -43.86751765575472 |
| - type: nauc_mrr_at_5_diff1 |
| value: 98.16782356611095 |
| - type: nauc_mrr_at_5_max |
| value: 85.28003038549159 |
| - type: nauc_mrr_at_5_std |
| value: -43.74493490831982 |
| - type: nauc_ndcg_at_1000_diff1 |
| value: 98.16554701617589 |
| - type: nauc_ndcg_at_1000_max |
| value: 85.2748828477838 |
| - type: nauc_ndcg_at_1000_std |
| value: -43.53566701835024 |
| - type: nauc_ndcg_at_100_diff1 |
| value: 98.16666937775803 |
| - type: nauc_ndcg_at_100_max |
| value: 85.26689650159106 |
| - type: nauc_ndcg_at_100_std |
| value: -43.61547874575031 |
| - type: nauc_ndcg_at_10_diff1 |
| value: 98.15895834975365 |
| - type: nauc_ndcg_at_10_max |
| value: 85.43419484878363 |
| - type: nauc_ndcg_at_10_std |
| value: -45.349653395420944 |
| - type: nauc_ndcg_at_1_diff1 |
| value: 98.05311494766504 |
| - type: nauc_ndcg_at_1_max |
| value: 83.37279874457242 |
| - type: nauc_ndcg_at_1_std |
| value: -36.78783832567891 |
| - type: nauc_ndcg_at_20_diff1 |
| value: 98.17535928747724 |
| - type: nauc_ndcg_at_20_max |
| value: 85.37092010493639 |
| - type: nauc_ndcg_at_20_std |
| value: -43.98828652303495 |
| - type: nauc_ndcg_at_3_diff1 |
| value: 98.25821737148956 |
| - type: nauc_ndcg_at_3_max |
| value: 86.61782450729814 |
| - type: nauc_ndcg_at_3_std |
| value: -47.32915673717569 |
| - type: nauc_ndcg_at_5_diff1 |
| value: 98.2061154768608 |
| - type: nauc_ndcg_at_5_max |
| value: 85.87935683953549 |
| - type: nauc_ndcg_at_5_std |
| value: -47.302585930633896 |
| - type: nauc_precision_at_1000_diff1 |
| value: .nan |
| - type: nauc_precision_at_1000_max |
| value: .nan |
| - type: nauc_precision_at_1000_std |
| value: .nan |
| - type: nauc_precision_at_100_diff1 |
| value: 100.0 |
| - type: nauc_precision_at_100_max |
| value: 72.22153459022752 |
| - type: nauc_precision_at_100_std |
| value: -173.98459168247265 |
| - type: nauc_precision_at_10_diff1 |
| value: 97.94083448220364 |
| - type: nauc_precision_at_10_max |
| value: 91.42141149896693 |
| - type: nauc_precision_at_10_std |
| value: -110.27705518270434 |
| - type: nauc_precision_at_1_diff1 |
| value: 98.05311494766504 |
| - type: nauc_precision_at_1_max |
| value: 83.37279874457242 |
| - type: nauc_precision_at_1_std |
| value: -36.78783832567891 |
| - type: nauc_precision_at_20_diff1 |
| value: 98.99445460386596 |
| - type: nauc_precision_at_20_max |
| value: 93.841013383867 |
| - type: nauc_precision_at_20_std |
| value: -85.33131689356598 |
| - type: nauc_precision_at_3_diff1 |
| value: 98.77736667583922 |
| - type: nauc_precision_at_3_max |
| value: 94.04942862307165 |
| - type: nauc_precision_at_3_std |
| value: -74.09082528149943 |
| - type: nauc_precision_at_5_diff1 |
| value: 98.69061226878082 |
| - type: nauc_precision_at_5_max |
| value: 93.54052780940046 |
| - type: nauc_precision_at_5_std |
| value: -97.44201029419808 |
| - type: nauc_recall_at_1000_diff1 |
| value: .nan |
| - type: nauc_recall_at_1000_max |
| value: .nan |
| - type: nauc_recall_at_1000_std |
| value: .nan |
| - type: nauc_recall_at_100_diff1 |
| value: 100.0 |
| - type: nauc_recall_at_100_max |
| value: 72.22153459013214 |
| - type: nauc_recall_at_100_std |
| value: -173.98459168226447 |
| - type: nauc_recall_at_10_diff1 |
| value: 97.94083448220702 |
| - type: nauc_recall_at_10_max |
| value: 91.42141149896996 |
| - type: nauc_recall_at_10_std |
| value: -110.27705518269966 |
| - type: nauc_recall_at_1_diff1 |
| value: 98.05311494766504 |
| - type: nauc_recall_at_1_max |
| value: 83.37279874457242 |
| - type: nauc_recall_at_1_std |
| value: -36.78783832567891 |
| - type: nauc_recall_at_20_diff1 |
| value: 98.99445460384722 |
| - type: nauc_recall_at_20_max |
| value: 93.84101338387075 |
| - type: nauc_recall_at_20_std |
| value: -85.33131689356598 |
| - type: nauc_recall_at_3_diff1 |
| value: 98.77736667583844 |
| - type: nauc_recall_at_3_max |
| value: 94.0494286230709 |
| - type: nauc_recall_at_3_std |
| value: -74.09082528149965 |
| - type: nauc_recall_at_5_diff1 |
| value: 98.69061226878156 |
| - type: nauc_recall_at_5_max |
| value: 93.5405278094034 |
| - type: nauc_recall_at_5_std |
| value: -97.44201029420337 |
| - type: ndcg_at_1 |
| value: 96.494 |
| - type: ndcg_at_10 |
| value: 98.367 |
| - type: ndcg_at_100 |
| value: 98.408 |
| - type: ndcg_at_1000 |
| value: 98.409 |
| - type: ndcg_at_20 |
| value: 98.392 |
| - type: ndcg_at_3 |
| value: 98.13 |
| - type: ndcg_at_5 |
| value: 98.289 |
| - type: precision_at_1 |
| value: 96.494 |
| - type: precision_at_10 |
| value: 9.981 |
| - type: precision_at_100 |
| value: 1.0 |
| - type: precision_at_1000 |
| value: 0.1 |
| - type: precision_at_20 |
| value: 4.9959999999999996 |
| - type: precision_at_3 |
| value: 33.061 |
| - type: precision_at_5 |
| value: 19.914 |
| - type: recall_at_1 |
| value: 96.494 |
| - type: recall_at_10 |
| value: 99.812 |
| - type: recall_at_100 |
| value: 99.993 |
| - type: recall_at_1000 |
| value: 100.0 |
| - type: recall_at_20 |
| value: 99.913 |
| - type: recall_at_3 |
| value: 99.182 |
| - type: recall_at_5 |
| value: 99.571 |
| task: |
| type: Retrieval |
| - dataset: |
| config: javascript |
| name: MTEB COIRCodeSearchNetRetrieval (javascript) |
| revision: 4adc7bc41202b5c13543c9c886a25f340634dab3 |
| split: test |
| type: CoIR-Retrieval/CodeSearchNet |
| metrics: |
| - type: main_score |
| value: 83.479 |
| - type: map_at_1 |
| value: 74.354 |
| - type: map_at_10 |
| value: 80.71300000000001 |
| - type: map_at_100 |
| value: 80.95700000000001 |
| - type: map_at_1000 |
| value: 80.963 |
| - type: map_at_20 |
| value: 80.87100000000001 |
| - type: map_at_3 |
| value: 79.712 |
| - type: map_at_5 |
| value: 80.28999999999999 |
| - type: mrr_at_1 |
| value: 74.35429960498328 |
| - type: mrr_at_10 |
| value: 80.71294728769666 |
| - type: mrr_at_100 |
| value: 80.95719685924738 |
| - type: mrr_at_1000 |
| value: 80.9626426876391 |
| - type: mrr_at_20 |
| value: 80.87121221349531 |
| - type: mrr_at_3 |
| value: 79.71234680441609 |
| - type: mrr_at_5 |
| value: 80.28967892231339 |
| - type: nauc_map_at_1000_diff1 |
| value: 82.88577068822386 |
| - type: nauc_map_at_1000_max |
| value: 67.67862670522469 |
| - type: nauc_map_at_1000_std |
| value: -3.522371246361021 |
| - type: nauc_map_at_100_diff1 |
| value: 82.8829326501484 |
| - type: nauc_map_at_100_max |
| value: 67.68203242273655 |
| - type: nauc_map_at_100_std |
| value: -3.522160367469023 |
| - type: nauc_map_at_10_diff1 |
| value: 82.8083126411219 |
| - type: nauc_map_at_10_max |
| value: 67.76494610226807 |
| - type: nauc_map_at_10_std |
| value: -3.3188673363705137 |
| - type: nauc_map_at_1_diff1 |
| value: 85.06863588556931 |
| - type: nauc_map_at_1_max |
| value: 64.5628789135785 |
| - type: nauc_map_at_1_std |
| value: -6.191657522827031 |
| - type: nauc_map_at_20_diff1 |
| value: 82.85819113195612 |
| - type: nauc_map_at_20_max |
| value: 67.75920634659379 |
| - type: nauc_map_at_20_std |
| value: -3.3782092153909744 |
| - type: nauc_map_at_3_diff1 |
| value: 82.93140971183675 |
| - type: nauc_map_at_3_max |
| value: 67.88657739487108 |
| - type: nauc_map_at_3_std |
| value: -3.9042280696240383 |
| - type: nauc_map_at_5_diff1 |
| value: 82.78238115943125 |
| - type: nauc_map_at_5_max |
| value: 67.78270133394841 |
| - type: nauc_map_at_5_std |
| value: -3.6293201723127693 |
| - type: nauc_mrr_at_1000_diff1 |
| value: 82.88577068822386 |
| - type: nauc_mrr_at_1000_max |
| value: 67.67862670522469 |
| - type: nauc_mrr_at_1000_std |
| value: -3.522371246361021 |
| - type: nauc_mrr_at_100_diff1 |
| value: 82.8829326501484 |
| - type: nauc_mrr_at_100_max |
| value: 67.68203242273655 |
| - type: nauc_mrr_at_100_std |
| value: -3.522160367469023 |
| - type: nauc_mrr_at_10_diff1 |
| value: 82.8083126411219 |
| - type: nauc_mrr_at_10_max |
| value: 67.76494610226807 |
| - type: nauc_mrr_at_10_std |
| value: -3.3188673363705137 |
| - type: nauc_mrr_at_1_diff1 |
| value: 85.06863588556931 |
| - type: nauc_mrr_at_1_max |
| value: 64.5628789135785 |
| - type: nauc_mrr_at_1_std |
| value: -6.191657522827031 |
| - type: nauc_mrr_at_20_diff1 |
| value: 82.85819113195612 |
| - type: nauc_mrr_at_20_max |
| value: 67.75920634659379 |
| - type: nauc_mrr_at_20_std |
| value: -3.3782092153909744 |
| - type: nauc_mrr_at_3_diff1 |
| value: 82.93140971183675 |
| - type: nauc_mrr_at_3_max |
| value: 67.88657739487108 |
| - type: nauc_mrr_at_3_std |
| value: -3.9042280696240383 |
| - type: nauc_mrr_at_5_diff1 |
| value: 82.78238115943125 |
| - type: nauc_mrr_at_5_max |
| value: 67.78270133394841 |
| - type: nauc_mrr_at_5_std |
| value: -3.6293201723127693 |
| - type: nauc_ndcg_at_1000_diff1 |
| value: 82.3199098700142 |
| - type: nauc_ndcg_at_1000_max |
| value: 68.120342472938 |
| - type: nauc_ndcg_at_1000_std |
| value: -2.8508509319395734 |
| - type: nauc_ndcg_at_100_diff1 |
| value: 82.1633159731435 |
| - type: nauc_ndcg_at_100_max |
| value: 68.26748139014519 |
| - type: nauc_ndcg_at_100_std |
| value: -2.714633476731751 |
| - type: nauc_ndcg_at_10_diff1 |
| value: 81.82903994006058 |
| - type: nauc_ndcg_at_10_max |
| value: 68.89584312261682 |
| - type: nauc_ndcg_at_10_std |
| value: -1.4252080306379409 |
| - type: nauc_ndcg_at_1_diff1 |
| value: 85.06863588556931 |
| - type: nauc_ndcg_at_1_max |
| value: 64.5628789135785 |
| - type: nauc_ndcg_at_1_std |
| value: -6.191657522827031 |
| - type: nauc_ndcg_at_20_diff1 |
| value: 81.98789282274657 |
| - type: nauc_ndcg_at_20_max |
| value: 68.8959043317764 |
| - type: nauc_ndcg_at_20_std |
| value: -1.6185846906781556 |
| - type: nauc_ndcg_at_3_diff1 |
| value: 82.10544479985309 |
| - type: nauc_ndcg_at_3_max |
| value: 68.98284750281354 |
| - type: nauc_ndcg_at_3_std |
| value: -3.02774137516056 |
| - type: nauc_ndcg_at_5_diff1 |
| value: 81.77317252542665 |
| - type: nauc_ndcg_at_5_max |
| value: 68.8430670861106 |
| - type: nauc_ndcg_at_5_std |
| value: -2.3676013953659583 |
| - type: nauc_precision_at_1000_diff1 |
| value: 71.2357013205843 |
| - type: nauc_precision_at_1000_max |
| value: 59.54259563274979 |
| - type: nauc_precision_at_1000_std |
| value: -6.453769813732959 |
| - type: nauc_precision_at_100_diff1 |
| value: 70.6646143169478 |
| - type: nauc_precision_at_100_max |
| value: 72.21908402194197 |
| - type: nauc_precision_at_100_std |
| value: 3.6390309142884423 |
| - type: nauc_precision_at_10_diff1 |
| value: 75.69911710504576 |
| - type: nauc_precision_at_10_max |
| value: 75.9705779221398 |
| - type: nauc_precision_at_10_std |
| value: 11.632934387302264 |
| - type: nauc_precision_at_1_diff1 |
| value: 85.06863588556931 |
| - type: nauc_precision_at_1_max |
| value: 64.5628789135785 |
| - type: nauc_precision_at_1_std |
| value: -6.191657522827031 |
| - type: nauc_precision_at_20_diff1 |
| value: 74.91685968954766 |
| - type: nauc_precision_at_20_max |
| value: 78.48940148220586 |
| - type: nauc_precision_at_20_std |
| value: 14.477347988107262 |
| - type: nauc_precision_at_3_diff1 |
| value: 78.78186792721571 |
| - type: nauc_precision_at_3_max |
| value: 73.31020934353046 |
| - type: nauc_precision_at_3_std |
| value: 0.4957741917380896 |
| - type: nauc_precision_at_5_diff1 |
| value: 76.96340948709872 |
| - type: nauc_precision_at_5_max |
| value: 73.6957286199771 |
| - type: nauc_precision_at_5_std |
| value: 3.7822933962717555 |
| - type: nauc_recall_at_1000_diff1 |
| value: 71.23570132058367 |
| - type: nauc_recall_at_1000_max |
| value: 59.54259563275034 |
| - type: nauc_recall_at_1000_std |
| value: -6.453769813732673 |
| - type: nauc_recall_at_100_diff1 |
| value: 70.66461431694707 |
| - type: nauc_recall_at_100_max |
| value: 72.21908402194222 |
| - type: nauc_recall_at_100_std |
| value: 3.639030914287266 |
| - type: nauc_recall_at_10_diff1 |
| value: 75.69911710504581 |
| - type: nauc_recall_at_10_max |
| value: 75.97057792213957 |
| - type: nauc_recall_at_10_std |
| value: 11.632934387302335 |
| - type: nauc_recall_at_1_diff1 |
| value: 85.06863588556931 |
| - type: nauc_recall_at_1_max |
| value: 64.5628789135785 |
| - type: nauc_recall_at_1_std |
| value: -6.191657522827031 |
| - type: nauc_recall_at_20_diff1 |
| value: 74.91685968954776 |
| - type: nauc_recall_at_20_max |
| value: 78.4894014822059 |
| - type: nauc_recall_at_20_std |
| value: 14.477347988107574 |
| - type: nauc_recall_at_3_diff1 |
| value: 78.78186792721584 |
| - type: nauc_recall_at_3_max |
| value: 73.31020934353042 |
| - type: nauc_recall_at_3_std |
| value: 0.49577419173804144 |
| - type: nauc_recall_at_5_diff1 |
| value: 76.96340948709855 |
| - type: nauc_recall_at_5_max |
| value: 73.695728619977 |
| - type: nauc_recall_at_5_std |
| value: 3.782293396271726 |
| - type: ndcg_at_1 |
| value: 74.354 |
| - type: ndcg_at_10 |
| value: 83.479 |
| - type: ndcg_at_100 |
| value: 84.65599999999999 |
| - type: ndcg_at_1000 |
| value: 84.851 |
| - type: ndcg_at_20 |
| value: 84.03500000000001 |
| - type: ndcg_at_3 |
| value: 81.42200000000001 |
| - type: ndcg_at_5 |
| value: 82.461 |
| - type: precision_at_1 |
| value: 74.354 |
| - type: precision_at_10 |
| value: 9.200999999999999 |
| - type: precision_at_100 |
| value: 0.975 |
| - type: precision_at_1000 |
| value: 0.099 |
| - type: precision_at_20 |
| value: 4.707999999999999 |
| - type: precision_at_3 |
| value: 28.786 |
| - type: precision_at_5 |
| value: 17.776 |
| - type: recall_at_1 |
| value: 74.354 |
| - type: recall_at_10 |
| value: 92.009 |
| - type: recall_at_100 |
| value: 97.508 |
| - type: recall_at_1000 |
| value: 99.149 |
| - type: recall_at_20 |
| value: 94.16600000000001 |
| - type: recall_at_3 |
| value: 86.357 |
| - type: recall_at_5 |
| value: 88.87899999999999 |
| task: |
| type: Retrieval |
| - dataset: |
| config: go |
| name: MTEB COIRCodeSearchNetRetrieval (go) |
| revision: 4adc7bc41202b5c13543c9c886a25f340634dab3 |
| split: test |
| type: CoIR-Retrieval/CodeSearchNet |
| metrics: |
| - type: main_score |
| value: 91.093 |
| - type: map_at_1 |
| value: 83.95700000000001 |
| - type: map_at_10 |
| value: 89.095 |
| - type: map_at_100 |
| value: 89.21900000000001 |
| - type: map_at_1000 |
| value: 89.22099999999999 |
| - type: map_at_20 |
| value: 89.181 |
| - type: map_at_3 |
| value: 88.472 |
| - type: map_at_5 |
| value: 88.866 |
| - type: mrr_at_1 |
| value: 83.95715341049002 |
| - type: mrr_at_10 |
| value: 89.0947231309824 |
| - type: mrr_at_100 |
| value: 89.21908275588152 |
| - type: mrr_at_1000 |
| value: 89.22056604897018 |
| - type: mrr_at_20 |
| value: 89.18054785129605 |
| - type: mrr_at_3 |
| value: 88.47164081096584 |
| - type: mrr_at_5 |
| value: 88.86624805056191 |
| - type: nauc_map_at_1000_diff1 |
| value: 86.24973579936615 |
| - type: nauc_map_at_1000_max |
| value: 43.76713186644932 |
| - type: nauc_map_at_1000_std |
| value: -47.20109532688898 |
| - type: nauc_map_at_100_diff1 |
| value: 86.24862049034168 |
| - type: nauc_map_at_100_max |
| value: 43.76881022585186 |
| - type: nauc_map_at_100_std |
| value: -47.20426521434973 |
| - type: nauc_map_at_10_diff1 |
| value: 86.25350003349301 |
| - type: nauc_map_at_10_max |
| value: 43.771945888748306 |
| - type: nauc_map_at_10_std |
| value: -47.62265974147625 |
| - type: nauc_map_at_1_diff1 |
| value: 86.98798340919 |
| - type: nauc_map_at_1_max |
| value: 43.024043355768995 |
| - type: nauc_map_at_1_std |
| value: -41.469271817341955 |
| - type: nauc_map_at_20_diff1 |
| value: 86.25325068130977 |
| - type: nauc_map_at_20_max |
| value: 43.80727376815637 |
| - type: nauc_map_at_20_std |
| value: -47.31520043996017 |
| - type: nauc_map_at_3_diff1 |
| value: 86.10288537323795 |
| - type: nauc_map_at_3_max |
| value: 43.79879378738717 |
| - type: nauc_map_at_3_std |
| value: -48.546995258168884 |
| - type: nauc_map_at_5_diff1 |
| value: 86.18610042665851 |
| - type: nauc_map_at_5_max |
| value: 43.4814674201994 |
| - type: nauc_map_at_5_std |
| value: -48.409761968549084 |
| - type: nauc_mrr_at_1000_diff1 |
| value: 86.24973579936615 |
| - type: nauc_mrr_at_1000_max |
| value: 43.76713186644932 |
| - type: nauc_mrr_at_1000_std |
| value: -47.20109532688898 |
| - type: nauc_mrr_at_100_diff1 |
| value: 86.24862049034168 |
| - type: nauc_mrr_at_100_max |
| value: 43.76881022585186 |
| - type: nauc_mrr_at_100_std |
| value: -47.20426521434973 |
| - type: nauc_mrr_at_10_diff1 |
| value: 86.25350003349301 |
| - type: nauc_mrr_at_10_max |
| value: 43.771945888748306 |
| - type: nauc_mrr_at_10_std |
| value: -47.62265974147625 |
| - type: nauc_mrr_at_1_diff1 |
| value: 86.98798340919 |
| - type: nauc_mrr_at_1_max |
| value: 43.024043355768995 |
| - type: nauc_mrr_at_1_std |
| value: -41.469271817341955 |
| - type: nauc_mrr_at_20_diff1 |
| value: 86.25325068130977 |
| - type: nauc_mrr_at_20_max |
| value: 43.80727376815637 |
| - type: nauc_mrr_at_20_std |
| value: -47.31520043996017 |
| - type: nauc_mrr_at_3_diff1 |
| value: 86.10288537323795 |
| - type: nauc_mrr_at_3_max |
| value: 43.79879378738717 |
| - type: nauc_mrr_at_3_std |
| value: -48.546995258168884 |
| - type: nauc_mrr_at_5_diff1 |
| value: 86.18610042665851 |
| - type: nauc_mrr_at_5_max |
| value: 43.4814674201994 |
| - type: nauc_mrr_at_5_std |
| value: -48.409761968549084 |
| - type: nauc_ndcg_at_1000_diff1 |
| value: 86.1495231100838 |
| - type: nauc_ndcg_at_1000_max |
| value: 44.07212654503798 |
| - type: nauc_ndcg_at_1000_std |
| value: -47.62163979276151 |
| - type: nauc_ndcg_at_100_diff1 |
| value: 86.0998353963385 |
| - type: nauc_ndcg_at_100_max |
| value: 44.12175911440964 |
| - type: nauc_ndcg_at_100_std |
| value: -47.60497182589526 |
| - type: nauc_ndcg_at_10_diff1 |
| value: 86.15470113116336 |
| - type: nauc_ndcg_at_10_max |
| value: 44.22094281301947 |
| - type: nauc_ndcg_at_10_std |
| value: -49.68035065822525 |
| - type: nauc_ndcg_at_1_diff1 |
| value: 86.98798340919 |
| - type: nauc_ndcg_at_1_max |
| value: 43.024043355768995 |
| - type: nauc_ndcg_at_1_std |
| value: -41.469271817341955 |
| - type: nauc_ndcg_at_20_diff1 |
| value: 86.153687239383 |
| - type: nauc_ndcg_at_20_max |
| value: 44.413683910958994 |
| - type: nauc_ndcg_at_20_std |
| value: -48.390979780431394 |
| - type: nauc_ndcg_at_3_diff1 |
| value: 85.81806042008513 |
| - type: nauc_ndcg_at_3_max |
| value: 44.05606453052025 |
| - type: nauc_ndcg_at_3_std |
| value: -51.77857306868798 |
| - type: nauc_ndcg_at_5_diff1 |
| value: 85.97978206190466 |
| - type: nauc_ndcg_at_5_max |
| value: 43.41162519222535 |
| - type: nauc_ndcg_at_5_std |
| value: -51.75858549128195 |
| - type: nauc_precision_at_1000_diff1 |
| value: 80.08890508034311 |
| - type: nauc_precision_at_1000_max |
| value: 94.23440421215071 |
| - type: nauc_precision_at_1000_std |
| value: 55.93973044712788 |
| - type: nauc_precision_at_100_diff1 |
| value: 75.93187211974605 |
| - type: nauc_precision_at_100_max |
| value: 67.07999533740335 |
| - type: nauc_precision_at_100_std |
| value: -8.479533178162365 |
| - type: nauc_precision_at_10_diff1 |
| value: 85.56280199155816 |
| - type: nauc_precision_at_10_max |
| value: 49.46503821359178 |
| - type: nauc_precision_at_10_std |
| value: -68.7017504439551 |
| - type: nauc_precision_at_1_diff1 |
| value: 86.98798340919 |
| - type: nauc_precision_at_1_max |
| value: 43.024043355768995 |
| - type: nauc_precision_at_1_std |
| value: -41.469271817341955 |
| - type: nauc_precision_at_20_diff1 |
| value: 85.16456431737308 |
| - type: nauc_precision_at_20_max |
| value: 57.105231459014604 |
| - type: nauc_precision_at_20_std |
| value: -57.20616563582712 |
| - type: nauc_precision_at_3_diff1 |
| value: 84.42151445378559 |
| - type: nauc_precision_at_3_max |
| value: 45.32354792405939 |
| - type: nauc_precision_at_3_std |
| value: -68.34786794101257 |
| - type: nauc_precision_at_5_diff1 |
| value: 84.72626746318639 |
| - type: nauc_precision_at_5_max |
| value: 42.65450965313161 |
| - type: nauc_precision_at_5_std |
| value: -74.63108495140406 |
| - type: nauc_recall_at_1000_diff1 |
| value: 80.08890508033205 |
| - type: nauc_recall_at_1000_max |
| value: 94.23440421215781 |
| - type: nauc_recall_at_1000_std |
| value: 55.93973044711981 |
| - type: nauc_recall_at_100_diff1 |
| value: 75.93187211975196 |
| - type: nauc_recall_at_100_max |
| value: 67.07999533740575 |
| - type: nauc_recall_at_100_std |
| value: -8.479533178153222 |
| - type: nauc_recall_at_10_diff1 |
| value: 85.56280199155815 |
| - type: nauc_recall_at_10_max |
| value: 49.46503821359128 |
| - type: nauc_recall_at_10_std |
| value: -68.70175044395225 |
| - type: nauc_recall_at_1_diff1 |
| value: 86.98798340919 |
| - type: nauc_recall_at_1_max |
| value: 43.024043355768995 |
| - type: nauc_recall_at_1_std |
| value: -41.469271817341955 |
| - type: nauc_recall_at_20_diff1 |
| value: 85.16456431737305 |
| - type: nauc_recall_at_20_max |
| value: 57.10523145901444 |
| - type: nauc_recall_at_20_std |
| value: -57.20616563582539 |
| - type: nauc_recall_at_3_diff1 |
| value: 84.42151445378548 |
| - type: nauc_recall_at_3_max |
| value: 45.32354792405929 |
| - type: nauc_recall_at_3_std |
| value: -68.34786794101288 |
| - type: nauc_recall_at_5_diff1 |
| value: 84.72626746318646 |
| - type: nauc_recall_at_5_max |
| value: 42.65450965313161 |
| - type: nauc_recall_at_5_std |
| value: -74.63108495140318 |
| - type: ndcg_at_1 |
| value: 83.95700000000001 |
| - type: ndcg_at_10 |
| value: 91.093 |
| - type: ndcg_at_100 |
| value: 91.644 |
| - type: ndcg_at_1000 |
| value: 91.67999999999999 |
| - type: ndcg_at_20 |
| value: 91.39699999999999 |
| - type: ndcg_at_3 |
| value: 89.84299999999999 |
| - type: ndcg_at_5 |
| value: 90.551 |
| - type: precision_at_1 |
| value: 83.95700000000001 |
| - type: precision_at_10 |
| value: 9.714 |
| - type: precision_at_100 |
| value: 0.996 |
| - type: precision_at_1000 |
| value: 0.1 |
| - type: precision_at_20 |
| value: 4.916 |
| - type: precision_at_3 |
| value: 31.261 |
| - type: precision_at_5 |
| value: 19.099 |
| - type: recall_at_1 |
| value: 83.95700000000001 |
| - type: recall_at_10 |
| value: 97.14399999999999 |
| - type: recall_at_100 |
| value: 99.59400000000001 |
| - type: recall_at_1000 |
| value: 99.87700000000001 |
| - type: recall_at_20 |
| value: 98.32600000000001 |
| - type: recall_at_3 |
| value: 93.782 |
| - type: recall_at_5 |
| value: 95.494 |
| task: |
| type: Retrieval |
| - dataset: |
| config: ruby |
| name: MTEB COIRCodeSearchNetRetrieval (ruby) |
| revision: 4adc7bc41202b5c13543c9c886a25f340634dab3 |
| split: test |
| type: CoIR-Retrieval/CodeSearchNet |
| metrics: |
| - type: main_score |
| value: 86.521 |
| - type: map_at_1 |
| value: 77.082 |
| - type: map_at_10 |
| value: 83.552 |
| - type: map_at_100 |
| value: 83.687 |
| - type: map_at_1000 |
| value: 83.69 |
| - type: map_at_20 |
| value: 83.64200000000001 |
| - type: map_at_3 |
| value: 82.11699999999999 |
| - type: map_at_5 |
| value: 83.164 |
| - type: mrr_at_1 |
| value: 77.08168120539256 |
| - type: mrr_at_10 |
| value: 83.55194919627917 |
| - type: mrr_at_100 |
| value: 83.68701866558312 |
| - type: mrr_at_1000 |
| value: 83.6903032381438 |
| - type: mrr_at_20 |
| value: 83.64219478925247 |
| - type: mrr_at_3 |
| value: 82.11736716891363 |
| - type: mrr_at_5 |
| value: 83.16415543219676 |
| - type: nauc_map_at_1000_diff1 |
| value: 79.64679377977846 |
| - type: nauc_map_at_1000_max |
| value: 70.74499431961593 |
| - type: nauc_map_at_1000_std |
| value: -14.853409296691982 |
| - type: nauc_map_at_100_diff1 |
| value: 79.64499589721636 |
| - type: nauc_map_at_100_max |
| value: 70.74759843854231 |
| - type: nauc_map_at_100_std |
| value: -14.844258734825894 |
| - type: nauc_map_at_10_diff1 |
| value: 79.53290732095073 |
| - type: nauc_map_at_10_max |
| value: 70.76525352436302 |
| - type: nauc_map_at_10_std |
| value: -14.744825626716187 |
| - type: nauc_map_at_1_diff1 |
| value: 82.27728044611361 |
| - type: nauc_map_at_1_max |
| value: 69.29382107125889 |
| - type: nauc_map_at_1_std |
| value: -16.02966665531533 |
| - type: nauc_map_at_20_diff1 |
| value: 79.57275942354842 |
| - type: nauc_map_at_20_max |
| value: 70.73838311952618 |
| - type: nauc_map_at_20_std |
| value: -14.86925109860625 |
| - type: nauc_map_at_3_diff1 |
| value: 79.47572156907859 |
| - type: nauc_map_at_3_max |
| value: 70.85276903670264 |
| - type: nauc_map_at_3_std |
| value: -15.755792005612602 |
| - type: nauc_map_at_5_diff1 |
| value: 79.42518354242955 |
| - type: nauc_map_at_5_max |
| value: 70.69226154953026 |
| - type: nauc_map_at_5_std |
| value: -15.473047773228993 |
| - type: nauc_mrr_at_1000_diff1 |
| value: 79.64679377977846 |
| - type: nauc_mrr_at_1000_max |
| value: 70.74499431961593 |
| - type: nauc_mrr_at_1000_std |
| value: -14.853409296691982 |
| - type: nauc_mrr_at_100_diff1 |
| value: 79.64499589721636 |
| - type: nauc_mrr_at_100_max |
| value: 70.74759843854231 |
| - type: nauc_mrr_at_100_std |
| value: -14.844258734825894 |
| - type: nauc_mrr_at_10_diff1 |
| value: 79.53290732095073 |
| - type: nauc_mrr_at_10_max |
| value: 70.76525352436302 |
| - type: nauc_mrr_at_10_std |
| value: -14.744825626716187 |
| - type: nauc_mrr_at_1_diff1 |
| value: 82.27728044611361 |
| - type: nauc_mrr_at_1_max |
| value: 69.29382107125889 |
| - type: nauc_mrr_at_1_std |
| value: -16.02966665531533 |
| - type: nauc_mrr_at_20_diff1 |
| value: 79.57275942354842 |
| - type: nauc_mrr_at_20_max |
| value: 70.73838311952618 |
| - type: nauc_mrr_at_20_std |
| value: -14.86925109860625 |
| - type: nauc_mrr_at_3_diff1 |
| value: 79.47572156907859 |
| - type: nauc_mrr_at_3_max |
| value: 70.85276903670264 |
| - type: nauc_mrr_at_3_std |
| value: -15.755792005612602 |
| - type: nauc_mrr_at_5_diff1 |
| value: 79.42518354242955 |
| - type: nauc_mrr_at_5_max |
| value: 70.69226154953026 |
| - type: nauc_mrr_at_5_std |
| value: -15.473047773228993 |
| - type: nauc_ndcg_at_1000_diff1 |
| value: 79.15083604230807 |
| - type: nauc_ndcg_at_1000_max |
| value: 71.1178743998364 |
| - type: nauc_ndcg_at_1000_std |
| value: -14.28015599243118 |
| - type: nauc_ndcg_at_100_diff1 |
| value: 79.03711984795275 |
| - type: nauc_ndcg_at_100_max |
| value: 71.20364489826714 |
| - type: nauc_ndcg_at_100_std |
| value: -13.915390197654142 |
| - type: nauc_ndcg_at_10_diff1 |
| value: 78.38949093885134 |
| - type: nauc_ndcg_at_10_max |
| value: 71.26426621928836 |
| - type: nauc_ndcg_at_10_std |
| value: -13.497441432215215 |
| - type: nauc_ndcg_at_1_diff1 |
| value: 82.27728044611361 |
| - type: nauc_ndcg_at_1_max |
| value: 69.29382107125889 |
| - type: nauc_ndcg_at_1_std |
| value: -16.02966665531533 |
| - type: nauc_ndcg_at_20_diff1 |
| value: 78.51349613885007 |
| - type: nauc_ndcg_at_20_max |
| value: 71.13292440385801 |
| - type: nauc_ndcg_at_20_std |
| value: -14.035580557966087 |
| - type: nauc_ndcg_at_3_diff1 |
| value: 78.3820255722769 |
| - type: nauc_ndcg_at_3_max |
| value: 71.32668614112852 |
| - type: nauc_ndcg_at_3_std |
| value: -16.04098840431829 |
| - type: nauc_ndcg_at_5_diff1 |
| value: 78.14070256474213 |
| - type: nauc_ndcg_at_5_max |
| value: 71.04026107145508 |
| - type: nauc_ndcg_at_5_std |
| value: -15.592128711464067 |
| - type: nauc_precision_at_1000_diff1 |
| value: 77.2067992384371 |
| - type: nauc_precision_at_1000_max |
| value: 85.55623308018635 |
| - type: nauc_precision_at_1000_std |
| value: -3.0163245425889005 |
| - type: nauc_precision_at_100_diff1 |
| value: 71.01994754486307 |
| - type: nauc_precision_at_100_max |
| value: 82.30803790261785 |
| - type: nauc_precision_at_100_std |
| value: 12.633506693103701 |
| - type: nauc_precision_at_10_diff1 |
| value: 67.04993913892164 |
| - type: nauc_precision_at_10_max |
| value: 76.23886937696508 |
| - type: nauc_precision_at_10_std |
| value: 1.5250700897226326 |
| - type: nauc_precision_at_1_diff1 |
| value: 82.27728044611361 |
| - type: nauc_precision_at_1_max |
| value: 69.29382107125889 |
| - type: nauc_precision_at_1_std |
| value: -16.02966665531533 |
| - type: nauc_precision_at_20_diff1 |
| value: 63.666986838468034 |
| - type: nauc_precision_at_20_max |
| value: 76.14090190912508 |
| - type: nauc_precision_at_20_std |
| value: -0.5871594787522816 |
| - type: nauc_precision_at_3_diff1 |
| value: 73.86799722582569 |
| - type: nauc_precision_at_3_max |
| value: 73.220598048437 |
| - type: nauc_precision_at_3_std |
| value: -17.398868434567206 |
| - type: nauc_precision_at_5_diff1 |
| value: 70.02914493269556 |
| - type: nauc_precision_at_5_max |
| value: 73.00069716131422 |
| - type: nauc_precision_at_5_std |
| value: -16.614393845477327 |
| - type: nauc_recall_at_1000_diff1 |
| value: 77.20679923843755 |
| - type: nauc_recall_at_1000_max |
| value: 85.55623308018376 |
| - type: nauc_recall_at_1000_std |
| value: -3.016324542591852 |
| - type: nauc_recall_at_100_diff1 |
| value: 71.01994754486401 |
| - type: nauc_recall_at_100_max |
| value: 82.30803790261768 |
| - type: nauc_recall_at_100_std |
| value: 12.633506693104875 |
| - type: nauc_recall_at_10_diff1 |
| value: 67.04993913892174 |
| - type: nauc_recall_at_10_max |
| value: 76.23886937696521 |
| - type: nauc_recall_at_10_std |
| value: 1.525070089722257 |
| - type: nauc_recall_at_1_diff1 |
| value: 82.27728044611361 |
| - type: nauc_recall_at_1_max |
| value: 69.29382107125889 |
| - type: nauc_recall_at_1_std |
| value: -16.02966665531533 |
| - type: nauc_recall_at_20_diff1 |
| value: 63.66698683846833 |
| - type: nauc_recall_at_20_max |
| value: 76.1409019091247 |
| - type: nauc_recall_at_20_std |
| value: -0.5871594787518352 |
| - type: nauc_recall_at_3_diff1 |
| value: 73.86799722582569 |
| - type: nauc_recall_at_3_max |
| value: 73.22059804843683 |
| - type: nauc_recall_at_3_std |
| value: -17.398868434566936 |
| - type: nauc_recall_at_5_diff1 |
| value: 70.02914493269549 |
| - type: nauc_recall_at_5_max |
| value: 73.00069716131398 |
| - type: nauc_recall_at_5_std |
| value: -16.614393845477526 |
| - type: ndcg_at_1 |
| value: 77.082 |
| - type: ndcg_at_10 |
| value: 86.521 |
| - type: ndcg_at_100 |
| value: 87.117 |
| - type: ndcg_at_1000 |
| value: 87.23100000000001 |
| - type: ndcg_at_20 |
| value: 86.858 |
| - type: ndcg_at_3 |
| value: 83.7 |
| - type: ndcg_at_5 |
| value: 85.591 |
| - type: precision_at_1 |
| value: 77.082 |
| - type: precision_at_10 |
| value: 9.572 |
| - type: precision_at_100 |
| value: 0.983 |
| - type: precision_at_1000 |
| value: 0.099 |
| - type: precision_at_20 |
| value: 4.853 |
| - type: precision_at_3 |
| value: 29.421000000000003 |
| - type: precision_at_5 |
| value: 18.573 |
| - type: recall_at_1 |
| value: 77.082 |
| - type: recall_at_10 |
| value: 95.718 |
| - type: recall_at_100 |
| value: 98.335 |
| - type: recall_at_1000 |
| value: 99.286 |
| - type: recall_at_20 |
| value: 97.066 |
| - type: recall_at_3 |
| value: 88.263 |
| - type: recall_at_5 |
| value: 92.863 |
| task: |
| type: Retrieval |
| - dataset: |
| config: java |
| name: MTEB COIRCodeSearchNetRetrieval (java) |
| revision: 4adc7bc41202b5c13543c9c886a25f340634dab3 |
| split: test |
| type: CoIR-Retrieval/CodeSearchNet |
| metrics: |
| - type: main_score |
| value: 87.26899999999999 |
| - type: map_at_1 |
| value: 78.84100000000001 |
| - type: map_at_10 |
| value: 84.821 |
| - type: map_at_100 |
| value: 84.966 |
| - type: map_at_1000 |
| value: 84.97200000000001 |
| - type: map_at_20 |
| value: 84.91199999999999 |
| - type: map_at_3 |
| value: 83.949 |
| - type: map_at_5 |
| value: 84.514 |
| - type: mrr_at_1 |
| value: 78.8407120036513 |
| - type: mrr_at_10 |
| value: 84.82137604775663 |
| - type: mrr_at_100 |
| value: 84.96600236904305 |
| - type: mrr_at_1000 |
| value: 84.97188010768014 |
| - type: mrr_at_20 |
| value: 84.91215021114384 |
| - type: mrr_at_3 |
| value: 83.94949033926656 |
| - type: mrr_at_5 |
| value: 84.51361630914327 |
| - type: nauc_map_at_1000_diff1 |
| value: 81.29745727169212 |
| - type: nauc_map_at_1000_max |
| value: 61.3738482977934 |
| - type: nauc_map_at_1000_std |
| value: -14.611667076737653 |
| - type: nauc_map_at_100_diff1 |
| value: 81.29611575498951 |
| - type: nauc_map_at_100_max |
| value: 61.38415642876098 |
| - type: nauc_map_at_100_std |
| value: -14.60020471629818 |
| - type: nauc_map_at_10_diff1 |
| value: 81.24492887826995 |
| - type: nauc_map_at_10_max |
| value: 61.476804728960545 |
| - type: nauc_map_at_10_std |
| value: -14.7216428942423 |
| - type: nauc_map_at_1_diff1 |
| value: 83.07021879054804 |
| - type: nauc_map_at_1_max |
| value: 57.45256621613931 |
| - type: nauc_map_at_1_std |
| value: -16.94155323782564 |
| - type: nauc_map_at_20_diff1 |
| value: 81.28505757844012 |
| - type: nauc_map_at_20_max |
| value: 61.43691967628063 |
| - type: nauc_map_at_20_std |
| value: -14.583239037636378 |
| - type: nauc_map_at_3_diff1 |
| value: 81.0101519179173 |
| - type: nauc_map_at_3_max |
| value: 61.45133191553263 |
| - type: nauc_map_at_3_std |
| value: -15.098090101575302 |
| - type: nauc_map_at_5_diff1 |
| value: 81.16287036289346 |
| - type: nauc_map_at_5_max |
| value: 61.59538703827228 |
| - type: nauc_map_at_5_std |
| value: -14.843988367319087 |
| - type: nauc_mrr_at_1000_diff1 |
| value: 81.29745727169212 |
| - type: nauc_mrr_at_1000_max |
| value: 61.3738482977934 |
| - type: nauc_mrr_at_1000_std |
| value: -14.611667076737653 |
| - type: nauc_mrr_at_100_diff1 |
| value: 81.29611575498951 |
| - type: nauc_mrr_at_100_max |
| value: 61.38415642876098 |
| - type: nauc_mrr_at_100_std |
| value: -14.60020471629818 |
| - type: nauc_mrr_at_10_diff1 |
| value: 81.24492887826995 |
| - type: nauc_mrr_at_10_max |
| value: 61.476804728960545 |
| - type: nauc_mrr_at_10_std |
| value: -14.7216428942423 |
| - type: nauc_mrr_at_1_diff1 |
| value: 83.07021879054804 |
| - type: nauc_mrr_at_1_max |
| value: 57.45256621613931 |
| - type: nauc_mrr_at_1_std |
| value: -16.94155323782564 |
| - type: nauc_mrr_at_20_diff1 |
| value: 81.28505757844012 |
| - type: nauc_mrr_at_20_max |
| value: 61.43691967628063 |
| - type: nauc_mrr_at_20_std |
| value: -14.583239037636378 |
| - type: nauc_mrr_at_3_diff1 |
| value: 81.0101519179173 |
| - type: nauc_mrr_at_3_max |
| value: 61.45133191553263 |
| - type: nauc_mrr_at_3_std |
| value: -15.098090101575302 |
| - type: nauc_mrr_at_5_diff1 |
| value: 81.16287036289346 |
| - type: nauc_mrr_at_5_max |
| value: 61.59538703827228 |
| - type: nauc_mrr_at_5_std |
| value: -14.843988367319087 |
| - type: nauc_ndcg_at_1000_diff1 |
| value: 80.93566924723379 |
| - type: nauc_ndcg_at_1000_max |
| value: 62.39699094971305 |
| - type: nauc_ndcg_at_1000_std |
| value: -13.127328464119836 |
| - type: nauc_ndcg_at_100_diff1 |
| value: 80.87553513050011 |
| - type: nauc_ndcg_at_100_max |
| value: 62.72180161053774 |
| - type: nauc_ndcg_at_100_std |
| value: -12.649445624854962 |
| - type: nauc_ndcg_at_10_diff1 |
| value: 80.608466242348 |
| - type: nauc_ndcg_at_10_max |
| value: 63.24038696080356 |
| - type: nauc_ndcg_at_10_std |
| value: -13.10528733578738 |
| - type: nauc_ndcg_at_1_diff1 |
| value: 83.07021879054804 |
| - type: nauc_ndcg_at_1_max |
| value: 57.45256621613931 |
| - type: nauc_ndcg_at_1_std |
| value: -16.94155323782564 |
| - type: nauc_ndcg_at_20_diff1 |
| value: 80.77174027485287 |
| - type: nauc_ndcg_at_20_max |
| value: 63.1297458573586 |
| - type: nauc_ndcg_at_20_std |
| value: -12.43612330644508 |
| - type: nauc_ndcg_at_3_diff1 |
| value: 80.1778653998912 |
| - type: nauc_ndcg_at_3_max |
| value: 63.029112839728796 |
| - type: nauc_ndcg_at_3_std |
| value: -14.201414014064817 |
| - type: nauc_ndcg_at_5_diff1 |
| value: 80.41507067529044 |
| - type: nauc_ndcg_at_5_max |
| value: 63.456703437014475 |
| - type: nauc_ndcg_at_5_std |
| value: -13.566365227692753 |
| - type: nauc_precision_at_1000_diff1 |
| value: 73.89525267370539 |
| - type: nauc_precision_at_1000_max |
| value: 83.46793489687887 |
| - type: nauc_precision_at_1000_std |
| value: 43.36957143495421 |
| - type: nauc_precision_at_100_diff1 |
| value: 75.33082548811534 |
| - type: nauc_precision_at_100_max |
| value: 85.08356966408422 |
| - type: nauc_precision_at_100_std |
| value: 30.91532174961418 |
| - type: nauc_precision_at_10_diff1 |
| value: 75.96120115397336 |
| - type: nauc_precision_at_10_max |
| value: 76.87957072290979 |
| - type: nauc_precision_at_10_std |
| value: 0.48296998647622513 |
| - type: nauc_precision_at_1_diff1 |
| value: 83.07021879054804 |
| - type: nauc_precision_at_1_max |
| value: 57.45256621613931 |
| - type: nauc_precision_at_1_std |
| value: -16.94155323782564 |
| - type: nauc_precision_at_20_diff1 |
| value: 76.37431221666498 |
| - type: nauc_precision_at_20_max |
| value: 80.17866322470198 |
| - type: nauc_precision_at_20_std |
| value: 12.914278225658554 |
| - type: nauc_precision_at_3_diff1 |
| value: 76.55984367763438 |
| - type: nauc_precision_at_3_max |
| value: 69.86960822047166 |
| - type: nauc_precision_at_3_std |
| value: -10.230045456623477 |
| - type: nauc_precision_at_5_diff1 |
| value: 76.32796311102129 |
| - type: nauc_precision_at_5_max |
| value: 73.88101356144463 |
| - type: nauc_precision_at_5_std |
| value: -6.1510867874405095 |
| - type: nauc_recall_at_1000_diff1 |
| value: 73.89525267370503 |
| - type: nauc_recall_at_1000_max |
| value: 83.46793489687492 |
| - type: nauc_recall_at_1000_std |
| value: 43.36957143495507 |
| - type: nauc_recall_at_100_diff1 |
| value: 75.33082548811574 |
| - type: nauc_recall_at_100_max |
| value: 85.08356966408451 |
| - type: nauc_recall_at_100_std |
| value: 30.91532174961395 |
| - type: nauc_recall_at_10_diff1 |
| value: 75.96120115397329 |
| - type: nauc_recall_at_10_max |
| value: 76.87957072290959 |
| - type: nauc_recall_at_10_std |
| value: 0.4829699864760997 |
| - type: nauc_recall_at_1_diff1 |
| value: 83.07021879054804 |
| - type: nauc_recall_at_1_max |
| value: 57.45256621613931 |
| - type: nauc_recall_at_1_std |
| value: -16.94155323782564 |
| - type: nauc_recall_at_20_diff1 |
| value: 76.37431221666506 |
| - type: nauc_recall_at_20_max |
| value: 80.17866322470138 |
| - type: nauc_recall_at_20_std |
| value: 12.914278225659201 |
| - type: nauc_recall_at_3_diff1 |
| value: 76.55984367763459 |
| - type: nauc_recall_at_3_max |
| value: 69.86960822047196 |
| - type: nauc_recall_at_3_std |
| value: -10.230045456623337 |
| - type: nauc_recall_at_5_diff1 |
| value: 76.32796311102106 |
| - type: nauc_recall_at_5_max |
| value: 73.88101356144475 |
| - type: nauc_recall_at_5_std |
| value: -6.151086787440187 |
| - type: ndcg_at_1 |
| value: 78.84100000000001 |
| - type: ndcg_at_10 |
| value: 87.26899999999999 |
| - type: ndcg_at_100 |
| value: 87.977 |
| - type: ndcg_at_1000 |
| value: 88.133 |
| - type: ndcg_at_20 |
| value: 87.599 |
| - type: ndcg_at_3 |
| value: 85.521 |
| - type: ndcg_at_5 |
| value: 86.531 |
| - type: precision_at_1 |
| value: 78.84100000000001 |
| - type: precision_at_10 |
| value: 9.474 |
| - type: precision_at_100 |
| value: 0.9809999999999999 |
| - type: precision_at_1000 |
| value: 0.099 |
| - type: precision_at_20 |
| value: 4.802 |
| - type: precision_at_3 |
| value: 30.014000000000003 |
| - type: precision_at_5 |
| value: 18.496000000000002 |
| - type: recall_at_1 |
| value: 78.84100000000001 |
| - type: recall_at_10 |
| value: 94.742 |
| - type: recall_at_100 |
| value: 98.056 |
| - type: recall_at_1000 |
| value: 99.288 |
| - type: recall_at_20 |
| value: 96.04700000000001 |
| - type: recall_at_3 |
| value: 90.04100000000001 |
| - type: recall_at_5 |
| value: 92.47800000000001 |
| task: |
| type: Retrieval |
| - dataset: |
| config: php |
| name: MTEB COIRCodeSearchNetRetrieval (php) |
| revision: 4adc7bc41202b5c13543c9c886a25f340634dab3 |
| split: test |
| type: CoIR-Retrieval/CodeSearchNet |
| metrics: |
| - type: main_score |
| value: 89.384 |
| - type: map_at_1 |
| value: 81.997 |
| - type: map_at_10 |
| value: 87.221 |
| - type: map_at_100 |
| value: 87.375 |
| - type: map_at_1000 |
| value: 87.378 |
| - type: map_at_20 |
| value: 87.32799999999999 |
| - type: map_at_3 |
| value: 86.429 |
| - type: map_at_5 |
| value: 86.91600000000001 |
| - type: mrr_at_1 |
| value: 81.9965748537177 |
| - type: mrr_at_10 |
| value: 87.22106125167394 |
| - type: mrr_at_100 |
| value: 87.37512926935148 |
| - type: mrr_at_1000 |
| value: 87.37841767615039 |
| - type: mrr_at_20 |
| value: 87.32764206073328 |
| - type: mrr_at_3 |
| value: 86.42904714333318 |
| - type: mrr_at_5 |
| value: 86.9157033442753 |
| - type: nauc_map_at_1000_diff1 |
| value: 84.98375890205129 |
| - type: nauc_map_at_1000_max |
| value: 52.169083096188054 |
| - type: nauc_map_at_1000_std |
| value: -35.027524704460916 |
| - type: nauc_map_at_100_diff1 |
| value: 84.98366301880897 |
| - type: nauc_map_at_100_max |
| value: 52.177207768384626 |
| - type: nauc_map_at_100_std |
| value: -35.01620836480514 |
| - type: nauc_map_at_10_diff1 |
| value: 84.95191430457028 |
| - type: nauc_map_at_10_max |
| value: 52.31158183948366 |
| - type: nauc_map_at_10_std |
| value: -35.13714349074112 |
| - type: nauc_map_at_1_diff1 |
| value: 86.2487043950067 |
| - type: nauc_map_at_1_max |
| value: 49.20239525557377 |
| - type: nauc_map_at_1_std |
| value: -33.71661281670688 |
| - type: nauc_map_at_20_diff1 |
| value: 84.973933039348 |
| - type: nauc_map_at_20_max |
| value: 52.2266012979943 |
| - type: nauc_map_at_20_std |
| value: -35.04159739087031 |
| - type: nauc_map_at_3_diff1 |
| value: 84.69068283876815 |
| - type: nauc_map_at_3_max |
| value: 52.343743449585524 |
| - type: nauc_map_at_3_std |
| value: -35.69366581732263 |
| - type: nauc_map_at_5_diff1 |
| value: 84.84059835122619 |
| - type: nauc_map_at_5_max |
| value: 52.29585814594443 |
| - type: nauc_map_at_5_std |
| value: -35.5113867651616 |
| - type: nauc_mrr_at_1000_diff1 |
| value: 84.98375890205129 |
| - type: nauc_mrr_at_1000_max |
| value: 52.169083096188054 |
| - type: nauc_mrr_at_1000_std |
| value: -35.027524704460916 |
| - type: nauc_mrr_at_100_diff1 |
| value: 84.98366301880897 |
| - type: nauc_mrr_at_100_max |
| value: 52.177207768384626 |
| - type: nauc_mrr_at_100_std |
| value: -35.01620836480514 |
| - type: nauc_mrr_at_10_diff1 |
| value: 84.95191430457028 |
| - type: nauc_mrr_at_10_max |
| value: 52.31158183948366 |
| - type: nauc_mrr_at_10_std |
| value: -35.13714349074112 |
| - type: nauc_mrr_at_1_diff1 |
| value: 86.2487043950067 |
| - type: nauc_mrr_at_1_max |
| value: 49.20239525557377 |
| - type: nauc_mrr_at_1_std |
| value: -33.71661281670688 |
| - type: nauc_mrr_at_20_diff1 |
| value: 84.973933039348 |
| - type: nauc_mrr_at_20_max |
| value: 52.2266012979943 |
| - type: nauc_mrr_at_20_std |
| value: -35.04159739087031 |
| - type: nauc_mrr_at_3_diff1 |
| value: 84.69068283876815 |
| - type: nauc_mrr_at_3_max |
| value: 52.343743449585524 |
| - type: nauc_mrr_at_3_std |
| value: -35.69366581732263 |
| - type: nauc_mrr_at_5_diff1 |
| value: 84.84059835122619 |
| - type: nauc_mrr_at_5_max |
| value: 52.29585814594443 |
| - type: nauc_mrr_at_5_std |
| value: -35.5113867651616 |
| - type: nauc_ndcg_at_1000_diff1 |
| value: 84.8458448603008 |
| - type: nauc_ndcg_at_1000_max |
| value: 52.77266190305249 |
| - type: nauc_ndcg_at_1000_std |
| value: -34.888069489815415 |
| - type: nauc_ndcg_at_100_diff1 |
| value: 84.84381394537597 |
| - type: nauc_ndcg_at_100_max |
| value: 53.01563101161627 |
| - type: nauc_ndcg_at_100_std |
| value: -34.509211628586314 |
| - type: nauc_ndcg_at_10_diff1 |
| value: 84.66548583706869 |
| - type: nauc_ndcg_at_10_max |
| value: 53.69134601286215 |
| - type: nauc_ndcg_at_10_std |
| value: -35.23415287527687 |
| - type: nauc_ndcg_at_1_diff1 |
| value: 86.2487043950067 |
| - type: nauc_ndcg_at_1_max |
| value: 49.20239525557377 |
| - type: nauc_ndcg_at_1_std |
| value: -33.71661281670688 |
| - type: nauc_ndcg_at_20_diff1 |
| value: 84.76038572414957 |
| - type: nauc_ndcg_at_20_max |
| value: 53.36813159950785 |
| - type: nauc_ndcg_at_20_std |
| value: -34.821410711460324 |
| - type: nauc_ndcg_at_3_diff1 |
| value: 84.1222575178288 |
| - type: nauc_ndcg_at_3_max |
| value: 53.5458597204313 |
| - type: nauc_ndcg_at_3_std |
| value: -36.57805029126134 |
| - type: nauc_ndcg_at_5_diff1 |
| value: 84.38081551698716 |
| - type: nauc_ndcg_at_5_max |
| value: 53.5309540463121 |
| - type: nauc_ndcg_at_5_std |
| value: -36.26371526873363 |
| - type: nauc_precision_at_1000_diff1 |
| value: 85.52761499735973 |
| - type: nauc_precision_at_1000_max |
| value: 85.57039121692812 |
| - type: nauc_precision_at_1000_std |
| value: 26.076840198712997 |
| - type: nauc_precision_at_100_diff1 |
| value: 84.71683047722709 |
| - type: nauc_precision_at_100_max |
| value: 78.6718703783981 |
| - type: nauc_precision_at_100_std |
| value: 7.427629853478685 |
| - type: nauc_precision_at_10_diff1 |
| value: 82.66742923426185 |
| - type: nauc_precision_at_10_max |
| value: 65.87095967797794 |
| - type: nauc_precision_at_10_std |
| value: -34.79989719383861 |
| - type: nauc_precision_at_1_diff1 |
| value: 86.2487043950067 |
| - type: nauc_precision_at_1_max |
| value: 49.20239525557377 |
| - type: nauc_precision_at_1_std |
| value: -33.71661281670688 |
| - type: nauc_precision_at_20_diff1 |
| value: 82.99219098545612 |
| - type: nauc_precision_at_20_max |
| value: 68.30891426964217 |
| - type: nauc_precision_at_20_std |
| value: -28.159133053121295 |
| - type: nauc_precision_at_3_diff1 |
| value: 81.61302076796814 |
| - type: nauc_precision_at_3_max |
| value: 58.883669048247576 |
| - type: nauc_precision_at_3_std |
| value: -40.57293703314752 |
| - type: nauc_precision_at_5_diff1 |
| value: 81.84587610537042 |
| - type: nauc_precision_at_5_max |
| value: 60.58183720409386 |
| - type: nauc_precision_at_5_std |
| value: -40.54776077088821 |
| - type: nauc_recall_at_1000_diff1 |
| value: 85.52761499733693 |
| - type: nauc_recall_at_1000_max |
| value: 85.57039121690828 |
| - type: nauc_recall_at_1000_std |
| value: 26.07684019872863 |
| - type: nauc_recall_at_100_diff1 |
| value: 84.7168304772257 |
| - type: nauc_recall_at_100_max |
| value: 78.67187037839709 |
| - type: nauc_recall_at_100_std |
| value: 7.427629853478298 |
| - type: nauc_recall_at_10_diff1 |
| value: 82.6674292342617 |
| - type: nauc_recall_at_10_max |
| value: 65.87095967797796 |
| - type: nauc_recall_at_10_std |
| value: -34.7998971938389 |
| - type: nauc_recall_at_1_diff1 |
| value: 86.2487043950067 |
| - type: nauc_recall_at_1_max |
| value: 49.20239525557377 |
| - type: nauc_recall_at_1_std |
| value: -33.71661281670688 |
| - type: nauc_recall_at_20_diff1 |
| value: 82.9921909854569 |
| - type: nauc_recall_at_20_max |
| value: 68.30891426964256 |
| - type: nauc_recall_at_20_std |
| value: -28.159133053120634 |
| - type: nauc_recall_at_3_diff1 |
| value: 81.61302076796801 |
| - type: nauc_recall_at_3_max |
| value: 58.88366904824765 |
| - type: nauc_recall_at_3_std |
| value: -40.57293703314801 |
| - type: nauc_recall_at_5_diff1 |
| value: 81.84587610537068 |
| - type: nauc_recall_at_5_max |
| value: 60.5818372040938 |
| - type: nauc_recall_at_5_std |
| value: -40.54776077088785 |
| - type: ndcg_at_1 |
| value: 81.997 |
| - type: ndcg_at_10 |
| value: 89.384 |
| - type: ndcg_at_100 |
| value: 90.091 |
| - type: ndcg_at_1000 |
| value: 90.18 |
| - type: ndcg_at_20 |
| value: 89.77000000000001 |
| - type: ndcg_at_3 |
| value: 87.788 |
| - type: ndcg_at_5 |
| value: 88.66000000000001 |
| - type: precision_at_1 |
| value: 81.997 |
| - type: precision_at_10 |
| value: 9.6 |
| - type: precision_at_100 |
| value: 0.992 |
| - type: precision_at_1000 |
| value: 0.1 |
| - type: precision_at_20 |
| value: 4.8759999999999994 |
| - type: precision_at_3 |
| value: 30.564999999999998 |
| - type: precision_at_5 |
| value: 18.759999999999998 |
| - type: recall_at_1 |
| value: 81.997 |
| - type: recall_at_10 |
| value: 96.004 |
| - type: recall_at_100 |
| value: 99.208 |
| - type: recall_at_1000 |
| value: 99.914 |
| - type: recall_at_20 |
| value: 97.524 |
| - type: recall_at_3 |
| value: 91.694 |
| - type: recall_at_5 |
| value: 93.79899999999999 |
| task: |
| type: Retrieval |
| - dataset: |
| config: default |
| name: MTEB CodeFeedbackMT (default) |
| revision: b0f12fa0c0dd67f59c95a5c33d02aeeb4c398c5f |
| split: test |
| type: CoIR-Retrieval/codefeedback-mt |
| metrics: |
| - type: main_score |
| value: 93.581 |
| - type: map_at_1 |
| value: 89.17699999999999 |
| - type: map_at_10 |
| value: 92.293 |
| - type: map_at_100 |
| value: 92.39 |
| - type: map_at_1000 |
| value: 92.392 |
| - type: map_at_20 |
| value: 92.357 |
| - type: map_at_3 |
| value: 91.805 |
| - type: map_at_5 |
| value: 92.124 |
| - type: mrr_at_1 |
| value: 89.1767718611132 |
| - type: mrr_at_10 |
| value: 92.29094710867727 |
| - type: mrr_at_100 |
| value: 92.38822901927145 |
| - type: mrr_at_1000 |
| value: 92.39035562398948 |
| - type: mrr_at_20 |
| value: 92.35530678666272 |
| - type: mrr_at_3 |
| value: 91.80412241721295 |
| - type: mrr_at_5 |
| value: 92.122718485602 |
| - type: nauc_map_at_1000_diff1 |
| value: 91.32273096224377 |
| - type: nauc_map_at_1000_max |
| value: 67.91710992417916 |
| - type: nauc_map_at_1000_std |
| value: -74.42383054373403 |
| - type: nauc_map_at_100_diff1 |
| value: 91.32246480919676 |
| - type: nauc_map_at_100_max |
| value: 67.91988217232155 |
| - type: nauc_map_at_100_std |
| value: -74.42982893903614 |
| - type: nauc_map_at_10_diff1 |
| value: 91.33091844723225 |
| - type: nauc_map_at_10_max |
| value: 67.99200052857573 |
| - type: nauc_map_at_10_std |
| value: -74.86264731079909 |
| - type: nauc_map_at_1_diff1 |
| value: 91.19451424067374 |
| - type: nauc_map_at_1_max |
| value: 66.25392135499182 |
| - type: nauc_map_at_1_std |
| value: -66.71862444546531 |
| - type: nauc_map_at_20_diff1 |
| value: 91.338096532563 |
| - type: nauc_map_at_20_max |
| value: 67.9501381727388 |
| - type: nauc_map_at_20_std |
| value: -74.6541487409306 |
| - type: nauc_map_at_3_diff1 |
| value: 91.33378646831018 |
| - type: nauc_map_at_3_max |
| value: 68.07178423980443 |
| - type: nauc_map_at_3_std |
| value: -74.8448215489277 |
| - type: nauc_map_at_5_diff1 |
| value: 91.31919404747947 |
| - type: nauc_map_at_5_max |
| value: 67.9846196631338 |
| - type: nauc_map_at_5_std |
| value: -75.1772163451522 |
| - type: nauc_mrr_at_1000_diff1 |
| value: 91.3199902377502 |
| - type: nauc_mrr_at_1000_max |
| value: 67.9218672510084 |
| - type: nauc_mrr_at_1000_std |
| value: -74.43167854141912 |
| - type: nauc_mrr_at_100_diff1 |
| value: 91.31972491331909 |
| - type: nauc_mrr_at_100_max |
| value: 67.92463756203038 |
| - type: nauc_mrr_at_100_std |
| value: -74.43767343342054 |
| - type: nauc_mrr_at_10_diff1 |
| value: 91.32821156688269 |
| - type: nauc_mrr_at_10_max |
| value: 67.9966801151313 |
| - type: nauc_mrr_at_10_std |
| value: -74.87029985666574 |
| - type: nauc_mrr_at_1_diff1 |
| value: 91.19451424067374 |
| - type: nauc_mrr_at_1_max |
| value: 66.25392135499182 |
| - type: nauc_mrr_at_1_std |
| value: -66.71862444546531 |
| - type: nauc_mrr_at_20_diff1 |
| value: 91.33536515912955 |
| - type: nauc_mrr_at_20_max |
| value: 67.95486643404077 |
| - type: nauc_mrr_at_20_std |
| value: -74.66191099031441 |
| - type: nauc_mrr_at_3_diff1 |
| value: 91.330840031252 |
| - type: nauc_mrr_at_3_max |
| value: 68.07469130953064 |
| - type: nauc_mrr_at_3_std |
| value: -74.86010698346193 |
| - type: nauc_mrr_at_5_diff1 |
| value: 91.31654781777661 |
| - type: nauc_mrr_at_5_max |
| value: 67.98920016571047 |
| - type: nauc_mrr_at_5_std |
| value: -75.18463872642327 |
| - type: nauc_ndcg_at_1000_diff1 |
| value: 91.34667276431432 |
| - type: nauc_ndcg_at_1000_max |
| value: 68.21491424591551 |
| - type: nauc_ndcg_at_1000_std |
| value: -75.66992744128785 |
| - type: nauc_ndcg_at_100_diff1 |
| value: 91.32392901146443 |
| - type: nauc_ndcg_at_100_max |
| value: 68.29729595248577 |
| - type: nauc_ndcg_at_100_std |
| value: -75.71771214390895 |
| - type: nauc_ndcg_at_10_diff1 |
| value: 91.42077595372994 |
| - type: nauc_ndcg_at_10_max |
| value: 68.68694743767313 |
| - type: nauc_ndcg_at_10_std |
| value: -78.47824539632552 |
| - type: nauc_ndcg_at_1_diff1 |
| value: 91.19451424067374 |
| - type: nauc_ndcg_at_1_max |
| value: 66.25392135499182 |
| - type: nauc_ndcg_at_1_std |
| value: -66.71862444546531 |
| - type: nauc_ndcg_at_20_diff1 |
| value: 91.45155994342801 |
| - type: nauc_ndcg_at_20_max |
| value: 68.53325586053745 |
| - type: nauc_ndcg_at_20_std |
| value: -77.61552755050272 |
| - type: nauc_ndcg_at_3_diff1 |
| value: 91.40048288131362 |
| - type: nauc_ndcg_at_3_max |
| value: 68.77879241382503 |
| - type: nauc_ndcg_at_3_std |
| value: -78.145772024284 |
| - type: nauc_ndcg_at_5_diff1 |
| value: 91.38284059773454 |
| - type: nauc_ndcg_at_5_max |
| value: 68.65210990623613 |
| - type: nauc_ndcg_at_5_std |
| value: -79.12029324998576 |
| - type: nauc_precision_at_1000_diff1 |
| value: 93.46388819144805 |
| - type: nauc_precision_at_1000_max |
| value: 100.0 |
| - type: nauc_precision_at_1000_std |
| value: 63.81917689216168 |
| - type: nauc_precision_at_100_diff1 |
| value: 88.14968521318447 |
| - type: nauc_precision_at_100_max |
| value: 79.85443666227313 |
| - type: nauc_precision_at_100_std |
| value: -71.60524535547103 |
| - type: nauc_precision_at_10_diff1 |
| value: 92.26837092071085 |
| - type: nauc_precision_at_10_max |
| value: 74.6573995994554 |
| - type: nauc_precision_at_10_std |
| value: -110.01042456501374 |
| - type: nauc_precision_at_1_diff1 |
| value: 91.19451424067374 |
| - type: nauc_precision_at_1_max |
| value: 66.25392135499182 |
| - type: nauc_precision_at_1_std |
| value: -66.71862444546531 |
| - type: nauc_precision_at_20_diff1 |
| value: 93.20186301743918 |
| - type: nauc_precision_at_20_max |
| value: 75.80862507034404 |
| - type: nauc_precision_at_20_std |
| value: -114.47170564758076 |
| - type: nauc_precision_at_3_diff1 |
| value: 91.7095571991953 |
| - type: nauc_precision_at_3_max |
| value: 71.9834034178247 |
| - type: nauc_precision_at_3_std |
| value: -93.18361438847609 |
| - type: nauc_precision_at_5_diff1 |
| value: 91.7773542794046 |
| - type: nauc_precision_at_5_max |
| value: 72.60584615957593 |
| - type: nauc_precision_at_5_std |
| value: -103.21989758142311 |
| - type: nauc_recall_at_1000_diff1 |
| value: 93.46388819147347 |
| - type: nauc_recall_at_1000_max |
| value: 100.0 |
| - type: nauc_recall_at_1000_std |
| value: 63.819176892126386 |
| - type: nauc_recall_at_100_diff1 |
| value: 88.14968521317887 |
| - type: nauc_recall_at_100_max |
| value: 79.85443666226894 |
| - type: nauc_recall_at_100_std |
| value: -71.60524535547005 |
| - type: nauc_recall_at_10_diff1 |
| value: 92.2683709207107 |
| - type: nauc_recall_at_10_max |
| value: 74.65739959945586 |
| - type: nauc_recall_at_10_std |
| value: -110.01042456501415 |
| - type: nauc_recall_at_1_diff1 |
| value: 91.19451424067374 |
| - type: nauc_recall_at_1_max |
| value: 66.25392135499182 |
| - type: nauc_recall_at_1_std |
| value: -66.71862444546531 |
| - type: nauc_recall_at_20_diff1 |
| value: 93.20186301743836 |
| - type: nauc_recall_at_20_max |
| value: 75.8086250703445 |
| - type: nauc_recall_at_20_std |
| value: -114.47170564758093 |
| - type: nauc_recall_at_3_diff1 |
| value: 91.70955719919579 |
| - type: nauc_recall_at_3_max |
| value: 71.9834034178246 |
| - type: nauc_recall_at_3_std |
| value: -93.1836143884767 |
| - type: nauc_recall_at_5_diff1 |
| value: 91.77735427940513 |
| - type: nauc_recall_at_5_max |
| value: 72.60584615957615 |
| - type: nauc_recall_at_5_std |
| value: -103.21989758142347 |
| - type: ndcg_at_1 |
| value: 89.17699999999999 |
| - type: ndcg_at_10 |
| value: 93.581 |
| - type: ndcg_at_100 |
| value: 94.041 |
| - type: ndcg_at_1000 |
| value: 94.087 |
| - type: ndcg_at_20 |
| value: 93.813 |
| - type: ndcg_at_3 |
| value: 92.608 |
| - type: ndcg_at_5 |
| value: 93.183 |
| - type: precision_at_1 |
| value: 89.17699999999999 |
| - type: precision_at_10 |
| value: 9.751999999999999 |
| - type: precision_at_100 |
| value: 0.996 |
| - type: precision_at_1000 |
| value: 0.1 |
| - type: precision_at_20 |
| value: 4.922 |
| - type: precision_at_3 |
| value: 31.639 |
| - type: precision_at_5 |
| value: 19.262 |
| - type: recall_at_1 |
| value: 89.17699999999999 |
| - type: recall_at_10 |
| value: 97.52199999999999 |
| - type: recall_at_100 |
| value: 99.638 |
| - type: recall_at_1000 |
| value: 99.985 |
| - type: recall_at_20 |
| value: 98.433 |
| - type: recall_at_3 |
| value: 94.916 |
| - type: recall_at_5 |
| value: 96.309 |
| task: |
| type: Retrieval |
| - dataset: |
| config: default |
| name: MTEB CodeFeedbackST (default) |
| revision: d213819e87aab9010628da8b73ab4eb337c89340 |
| split: test |
| type: CoIR-Retrieval/codefeedback-st |
| metrics: |
| - type: main_score |
| value: 90.672 |
| - type: map_at_1 |
| value: 80.17 |
| - type: map_at_10 |
| value: 87.945 |
| - type: map_at_100 |
| value: 88.004 |
| - type: map_at_1000 |
| value: 88.004 |
| - type: map_at_20 |
| value: 87.99 |
| - type: map_at_3 |
| value: 87.248 |
| - type: map_at_5 |
| value: 87.79599999999999 |
| - type: mrr_at_1 |
| value: 80.08369002747078 |
| - type: mrr_at_10 |
| value: 87.8921544021677 |
| - type: mrr_at_100 |
| value: 87.95039935983601 |
| - type: mrr_at_1000 |
| value: 87.95113333851275 |
| - type: mrr_at_20 |
| value: 87.93646217104668 |
| - type: mrr_at_3 |
| value: 87.1787090866489 |
| - type: mrr_at_5 |
| value: 87.73579079622561 |
| - type: nauc_map_at_1000_diff1 |
| value: 84.59599547053739 |
| - type: nauc_map_at_1000_max |
| value: 11.563061625907412 |
| - type: nauc_map_at_1000_std |
| value: -60.044443468702234 |
| - type: nauc_map_at_100_diff1 |
| value: 84.59537761332842 |
| - type: nauc_map_at_100_max |
| value: 11.565642697551821 |
| - type: nauc_map_at_100_std |
| value: -60.04854322246466 |
| - type: nauc_map_at_10_diff1 |
| value: 84.54000249494548 |
| - type: nauc_map_at_10_max |
| value: 11.55440805584509 |
| - type: nauc_map_at_10_std |
| value: -60.437831701830824 |
| - type: nauc_map_at_1_diff1 |
| value: 86.11748953192901 |
| - type: nauc_map_at_1_max |
| value: 11.5654030601593 |
| - type: nauc_map_at_1_std |
| value: -53.511088508152106 |
| - type: nauc_map_at_20_diff1 |
| value: 84.58313379439 |
| - type: nauc_map_at_20_max |
| value: 11.570044402077079 |
| - type: nauc_map_at_20_std |
| value: -60.15198207708651 |
| - type: nauc_map_at_3_diff1 |
| value: 84.32880719653019 |
| - type: nauc_map_at_3_max |
| value: 11.306870549359703 |
| - type: nauc_map_at_3_std |
| value: -61.66972599149546 |
| - type: nauc_map_at_5_diff1 |
| value: 84.44481140471144 |
| - type: nauc_map_at_5_max |
| value: 11.431107322656027 |
| - type: nauc_map_at_5_std |
| value: -61.07717915800889 |
| - type: nauc_mrr_at_1000_diff1 |
| value: 84.68763078545194 |
| - type: nauc_mrr_at_1000_max |
| value: 11.654873092581681 |
| - type: nauc_mrr_at_1000_std |
| value: -59.336986725737205 |
| - type: nauc_mrr_at_100_diff1 |
| value: 84.68700936878848 |
| - type: nauc_mrr_at_100_max |
| value: 11.657439639783314 |
| - type: nauc_mrr_at_100_std |
| value: -59.34111569519106 |
| - type: nauc_mrr_at_10_diff1 |
| value: 84.6310183439079 |
| - type: nauc_mrr_at_10_max |
| value: 11.644216926632295 |
| - type: nauc_mrr_at_10_std |
| value: -59.72967384330525 |
| - type: nauc_mrr_at_1_diff1 |
| value: 86.20926531200153 |
| - type: nauc_mrr_at_1_max |
| value: 11.798672092672408 |
| - type: nauc_mrr_at_1_std |
| value: -52.65061182155048 |
| - type: nauc_mrr_at_20_diff1 |
| value: 84.67473718323151 |
| - type: nauc_mrr_at_20_max |
| value: 11.66210308966798 |
| - type: nauc_mrr_at_20_std |
| value: -59.44638680106128 |
| - type: nauc_mrr_at_3_diff1 |
| value: 84.43542937601633 |
| - type: nauc_mrr_at_3_max |
| value: 11.386252739390274 |
| - type: nauc_mrr_at_3_std |
| value: -60.847260123302824 |
| - type: nauc_mrr_at_5_diff1 |
| value: 84.54163679900752 |
| - type: nauc_mrr_at_5_max |
| value: 11.507249429826652 |
| - type: nauc_mrr_at_5_std |
| value: -60.37227397638133 |
| - type: nauc_ndcg_at_1000_diff1 |
| value: 84.38515659117648 |
| - type: nauc_ndcg_at_1000_max |
| value: 11.696388104993815 |
| - type: nauc_ndcg_at_1000_std |
| value: -60.99717188632636 |
| - type: nauc_ndcg_at_100_diff1 |
| value: 84.36252666208128 |
| - type: nauc_ndcg_at_100_max |
| value: 11.779193272629582 |
| - type: nauc_ndcg_at_100_std |
| value: -61.07767732450027 |
| - type: nauc_ndcg_at_10_diff1 |
| value: 84.04954066703365 |
| - type: nauc_ndcg_at_10_max |
| value: 11.779974489837082 |
| - type: nauc_ndcg_at_10_std |
| value: -63.216690095927774 |
| - type: nauc_ndcg_at_1_diff1 |
| value: 86.11748953192901 |
| - type: nauc_ndcg_at_1_max |
| value: 11.5654030601593 |
| - type: nauc_ndcg_at_1_std |
| value: -53.511088508152106 |
| - type: nauc_ndcg_at_20_diff1 |
| value: 84.24823772848971 |
| - type: nauc_ndcg_at_20_max |
| value: 11.863360518928207 |
| - type: nauc_ndcg_at_20_std |
| value: -61.93361896101012 |
| - type: nauc_ndcg_at_3_diff1 |
| value: 83.59193584331861 |
| - type: nauc_ndcg_at_3_max |
| value: 11.101961347273388 |
| - type: nauc_ndcg_at_3_std |
| value: -65.94436298501897 |
| - type: nauc_ndcg_at_5_diff1 |
| value: 83.78223501020723 |
| - type: nauc_ndcg_at_5_max |
| value: 11.381166299845784 |
| - type: nauc_ndcg_at_5_std |
| value: -65.04889907197641 |
| - type: nauc_precision_at_1000_diff1 |
| value: 76.06066445979025 |
| - type: nauc_precision_at_1000_max |
| value: 88.80070662604956 |
| - type: nauc_precision_at_1000_std |
| value: 52.334059183043905 |
| - type: nauc_precision_at_100_diff1 |
| value: 72.04652343451335 |
| - type: nauc_precision_at_100_max |
| value: 63.89605474602823 |
| - type: nauc_precision_at_100_std |
| value: -69.43370560244632 |
| - type: nauc_precision_at_10_diff1 |
| value: 72.89952199107627 |
| - type: nauc_precision_at_10_max |
| value: 19.311114535110576 |
| - type: nauc_precision_at_10_std |
| value: -128.84033294430404 |
| - type: nauc_precision_at_1_diff1 |
| value: 86.11748953192901 |
| - type: nauc_precision_at_1_max |
| value: 11.5654030601593 |
| - type: nauc_precision_at_1_std |
| value: -53.511088508152106 |
| - type: nauc_precision_at_20_diff1 |
| value: 72.67081355507392 |
| - type: nauc_precision_at_20_max |
| value: 31.810009027751327 |
| - type: nauc_precision_at_20_std |
| value: -125.0241647763245 |
| - type: nauc_precision_at_3_diff1 |
| value: 78.25481403127574 |
| - type: nauc_precision_at_3_max |
| value: 9.522751941864358 |
| - type: nauc_precision_at_3_std |
| value: -97.78790351049261 |
| - type: nauc_precision_at_5_diff1 |
| value: 74.76998378870415 |
| - type: nauc_precision_at_5_max |
| value: 10.813166077171799 |
| - type: nauc_precision_at_5_std |
| value: -121.15471825739507 |
| - type: nauc_recall_at_1000_diff1 |
| value: 76.06066445979108 |
| - type: nauc_recall_at_1000_max |
| value: 88.80070662604261 |
| - type: nauc_recall_at_1000_std |
| value: 52.334059183053746 |
| - type: nauc_recall_at_100_diff1 |
| value: 72.04652343451711 |
| - type: nauc_recall_at_100_max |
| value: 63.896054746020795 |
| - type: nauc_recall_at_100_std |
| value: -69.43370560243693 |
| - type: nauc_recall_at_10_diff1 |
| value: 72.89952199107772 |
| - type: nauc_recall_at_10_max |
| value: 19.311114535110473 |
| - type: nauc_recall_at_10_std |
| value: -128.8403329443047 |
| - type: nauc_recall_at_1_diff1 |
| value: 86.11748953192901 |
| - type: nauc_recall_at_1_max |
| value: 11.5654030601593 |
| - type: nauc_recall_at_1_std |
| value: -53.511088508152106 |
| - type: nauc_recall_at_20_diff1 |
| value: 72.6708135550744 |
| - type: nauc_recall_at_20_max |
| value: 31.81000902775227 |
| - type: nauc_recall_at_20_std |
| value: -125.02416477631772 |
| - type: nauc_recall_at_3_diff1 |
| value: 78.25481403127577 |
| - type: nauc_recall_at_3_max |
| value: 9.522751941864486 |
| - type: nauc_recall_at_3_std |
| value: -97.78790351049261 |
| - type: nauc_recall_at_5_diff1 |
| value: 74.76998378870401 |
| - type: nauc_recall_at_5_max |
| value: 10.813166077172303 |
| - type: nauc_recall_at_5_std |
| value: -121.15471825739255 |
| - type: ndcg_at_1 |
| value: 80.17 |
| - type: ndcg_at_10 |
| value: 90.672 |
| - type: ndcg_at_100 |
| value: 90.923 |
| - type: ndcg_at_1000 |
| value: 90.93900000000001 |
| - type: ndcg_at_20 |
| value: 90.833 |
| - type: ndcg_at_3 |
| value: 89.347 |
| - type: ndcg_at_5 |
| value: 90.321 |
| - type: precision_at_1 |
| value: 80.17 |
| - type: precision_at_10 |
| value: 9.875 |
| - type: precision_at_100 |
| value: 0.9990000000000001 |
| - type: precision_at_1000 |
| value: 0.1 |
| - type: precision_at_20 |
| value: 4.968999999999999 |
| - type: precision_at_3 |
| value: 31.785999999999998 |
| - type: precision_at_5 |
| value: 19.538 |
| - type: recall_at_1 |
| value: 80.17 |
| - type: recall_at_10 |
| value: 98.75399999999999 |
| - type: recall_at_100 |
| value: 99.85300000000001 |
| - type: recall_at_1000 |
| value: 99.971 |
| - type: recall_at_20 |
| value: 99.384 |
| - type: recall_at_3 |
| value: 95.35900000000001 |
| - type: recall_at_5 |
| value: 97.69099999999999 |
| task: |
| type: Retrieval |
| - dataset: |
| config: python |
| name: MTEB CodeSearchNetCCRetrieval (python) |
| revision: 6e1effa2c03723c5fde48ee912b5ee08d4f211e8 |
| split: test |
| type: CoIR-Retrieval/CodeSearchNet-ccr |
| metrics: |
| - type: main_score |
| value: 94.433 |
| - type: map_at_1 |
| value: 89.19399999999999 |
| - type: map_at_10 |
| value: 93.028 |
| - type: map_at_100 |
| value: 93.082 |
| - type: map_at_1000 |
| value: 93.083 |
| - type: map_at_20 |
| value: 93.063 |
| - type: map_at_3 |
| value: 92.635 |
| - type: map_at_5 |
| value: 92.914 |
| - type: mrr_at_1 |
| value: 89.19426196541092 |
| - type: mrr_at_10 |
| value: 93.02709840248427 |
| - type: mrr_at_100 |
| value: 93.08116003424139 |
| - type: mrr_at_1000 |
| value: 93.08217593910118 |
| - type: mrr_at_20 |
| value: 93.06190896089124 |
| - type: mrr_at_3 |
| value: 92.63529516914724 |
| - type: mrr_at_5 |
| value: 92.91147160030434 |
| - type: nauc_map_at_1000_diff1 |
| value: 88.83311452663212 |
| - type: nauc_map_at_1000_max |
| value: 60.95017820522245 |
| - type: nauc_map_at_1000_std |
| value: -27.89769485046986 |
| - type: nauc_map_at_100_diff1 |
| value: 88.83290610426658 |
| - type: nauc_map_at_100_max |
| value: 60.95500551380402 |
| - type: nauc_map_at_100_std |
| value: -27.89016844487898 |
| - type: nauc_map_at_10_diff1 |
| value: 88.85572581806649 |
| - type: nauc_map_at_10_max |
| value: 61.15711675036097 |
| - type: nauc_map_at_10_std |
| value: -27.863737251156817 |
| - type: nauc_map_at_1_diff1 |
| value: 88.75783103896558 |
| - type: nauc_map_at_1_max |
| value: 56.05082536501996 |
| - type: nauc_map_at_1_std |
| value: -26.762271272974058 |
| - type: nauc_map_at_20_diff1 |
| value: 88.83684101691745 |
| - type: nauc_map_at_20_max |
| value: 61.005770466214415 |
| - type: nauc_map_at_20_std |
| value: -27.93102109901855 |
| - type: nauc_map_at_3_diff1 |
| value: 88.8117321224026 |
| - type: nauc_map_at_3_max |
| value: 61.93756889954432 |
| - type: nauc_map_at_3_std |
| value: -28.688755234894575 |
| - type: nauc_map_at_5_diff1 |
| value: 88.80829639130586 |
| - type: nauc_map_at_5_max |
| value: 61.453270288714315 |
| - type: nauc_map_at_5_std |
| value: -28.349980140724064 |
| - type: nauc_mrr_at_1000_diff1 |
| value: 88.84020090540668 |
| - type: nauc_mrr_at_1000_max |
| value: 60.889945118911925 |
| - type: nauc_mrr_at_1000_std |
| value: -27.98205495040352 |
| - type: nauc_mrr_at_100_diff1 |
| value: 88.8399914575814 |
| - type: nauc_mrr_at_100_max |
| value: 60.894780806340286 |
| - type: nauc_mrr_at_100_std |
| value: -27.974516894245998 |
| - type: nauc_mrr_at_10_diff1 |
| value: 88.86278756786537 |
| - type: nauc_mrr_at_10_max |
| value: 61.097374379118406 |
| - type: nauc_mrr_at_10_std |
| value: -27.94782127554896 |
| - type: nauc_mrr_at_1_diff1 |
| value: 88.75783103896558 |
| - type: nauc_mrr_at_1_max |
| value: 56.00537716973826 |
| - type: nauc_mrr_at_1_std |
| value: -26.855708242461617 |
| - type: nauc_mrr_at_20_diff1 |
| value: 88.84394060587955 |
| - type: nauc_mrr_at_20_max |
| value: 60.94574319151849 |
| - type: nauc_mrr_at_20_std |
| value: -28.015522604336574 |
| - type: nauc_mrr_at_3_diff1 |
| value: 88.8197212493924 |
| - type: nauc_mrr_at_3_max |
| value: 61.864940406758166 |
| - type: nauc_mrr_at_3_std |
| value: -28.785412568295914 |
| - type: nauc_mrr_at_5_diff1 |
| value: 88.82086414470385 |
| - type: nauc_mrr_at_5_max |
| value: 61.39568301316898 |
| - type: nauc_mrr_at_5_std |
| value: -28.470440741112203 |
| - type: nauc_ndcg_at_1000_diff1 |
| value: 88.87772673122608 |
| - type: nauc_ndcg_at_1000_max |
| value: 61.72947423485932 |
| - type: nauc_ndcg_at_1000_std |
| value: -27.73716470116156 |
| - type: nauc_ndcg_at_100_diff1 |
| value: 88.88494233648349 |
| - type: nauc_ndcg_at_100_max |
| value: 61.89646036981659 |
| - type: nauc_ndcg_at_100_std |
| value: -27.485345609558276 |
| - type: nauc_ndcg_at_10_diff1 |
| value: 89.0023293800841 |
| - type: nauc_ndcg_at_10_max |
| value: 62.94119991860201 |
| - type: nauc_ndcg_at_10_std |
| value: -27.460194148873917 |
| - type: nauc_ndcg_at_1_diff1 |
| value: 88.75783103896558 |
| - type: nauc_ndcg_at_1_max |
| value: 56.05082536501996 |
| - type: nauc_ndcg_at_1_std |
| value: -26.762271272974058 |
| - type: nauc_ndcg_at_20_diff1 |
| value: 88.91820321119603 |
| - type: nauc_ndcg_at_20_max |
| value: 62.296584649286466 |
| - type: nauc_ndcg_at_20_std |
| value: -27.766244930438113 |
| - type: nauc_ndcg_at_3_diff1 |
| value: 88.8645684782722 |
| - type: nauc_ndcg_at_3_max |
| value: 64.56115657789144 |
| - type: nauc_ndcg_at_3_std |
| value: -29.624561516864517 |
| - type: nauc_ndcg_at_5_diff1 |
| value: 88.86373245096877 |
| - type: nauc_ndcg_at_5_max |
| value: 63.72960904786197 |
| - type: nauc_ndcg_at_5_std |
| value: -28.9714090724707 |
| - type: nauc_precision_at_1000_diff1 |
| value: 93.46395492502747 |
| - type: nauc_precision_at_1000_max |
| value: 100.0 |
| - type: nauc_precision_at_1000_std |
| value: 79.16615094256558 |
| - type: nauc_precision_at_100_diff1 |
| value: 91.18620004808257 |
| - type: nauc_precision_at_100_max |
| value: 93.22441208688358 |
| - type: nauc_precision_at_100_std |
| value: 24.541027605722128 |
| - type: nauc_precision_at_10_diff1 |
| value: 91.39898569182662 |
| - type: nauc_precision_at_10_max |
| value: 86.36227958399685 |
| - type: nauc_precision_at_10_std |
| value: -18.10242240819358 |
| - type: nauc_precision_at_1_diff1 |
| value: 88.75783103896558 |
| - type: nauc_precision_at_1_max |
| value: 56.05082536501996 |
| - type: nauc_precision_at_1_std |
| value: -26.762271272974058 |
| - type: nauc_precision_at_20_diff1 |
| value: 90.67031369822483 |
| - type: nauc_precision_at_20_max |
| value: 83.74877222150064 |
| - type: nauc_precision_at_20_std |
| value: -20.408972788884217 |
| - type: nauc_precision_at_3_diff1 |
| value: 89.20609963322217 |
| - type: nauc_precision_at_3_max |
| value: 80.51313561745297 |
| - type: nauc_precision_at_3_std |
| value: -35.36879832053998 |
| - type: nauc_precision_at_5_diff1 |
| value: 89.40146615001582 |
| - type: nauc_precision_at_5_max |
| value: 83.94859838264775 |
| - type: nauc_precision_at_5_std |
| value: -34.3716686751886 |
| - type: nauc_recall_at_1000_diff1 |
| value: 93.46395492500885 |
| - type: nauc_recall_at_1000_max |
| value: 100.0 |
| - type: nauc_recall_at_1000_std |
| value: 79.1661509425585 |
| - type: nauc_recall_at_100_diff1 |
| value: 91.18620004808852 |
| - type: nauc_recall_at_100_max |
| value: 93.22441208689622 |
| - type: nauc_recall_at_100_std |
| value: 24.54102760573055 |
| - type: nauc_recall_at_10_diff1 |
| value: 91.39898569182566 |
| - type: nauc_recall_at_10_max |
| value: 86.36227958399564 |
| - type: nauc_recall_at_10_std |
| value: -18.102422408192854 |
| - type: nauc_recall_at_1_diff1 |
| value: 88.75783103896558 |
| - type: nauc_recall_at_1_max |
| value: 56.05082536501996 |
| - type: nauc_recall_at_1_std |
| value: -26.762271272974058 |
| - type: nauc_recall_at_20_diff1 |
| value: 90.67031369822419 |
| - type: nauc_recall_at_20_max |
| value: 83.7487722215012 |
| - type: nauc_recall_at_20_std |
| value: -20.408972788880643 |
| - type: nauc_recall_at_3_diff1 |
| value: 89.20609963322205 |
| - type: nauc_recall_at_3_max |
| value: 80.51313561745232 |
| - type: nauc_recall_at_3_std |
| value: -35.36879832053958 |
| - type: nauc_recall_at_5_diff1 |
| value: 89.40146615001574 |
| - type: nauc_recall_at_5_max |
| value: 83.94859838264772 |
| - type: nauc_recall_at_5_std |
| value: -34.371668675188346 |
| - type: ndcg_at_1 |
| value: 89.19399999999999 |
| - type: ndcg_at_10 |
| value: 94.433 |
| - type: ndcg_at_100 |
| value: 94.679 |
| - type: ndcg_at_1000 |
| value: 94.707 |
| - type: ndcg_at_20 |
| value: 94.556 |
| - type: ndcg_at_3 |
| value: 93.661 |
| - type: ndcg_at_5 |
| value: 94.161 |
| - type: precision_at_1 |
| value: 89.19399999999999 |
| - type: precision_at_10 |
| value: 9.864 |
| - type: precision_at_100 |
| value: 0.997 |
| - type: precision_at_1000 |
| value: 0.1 |
| - type: precision_at_20 |
| value: 4.956 |
| - type: precision_at_3 |
| value: 32.2 |
| - type: precision_at_5 |
| value: 19.561999999999998 |
| - type: recall_at_1 |
| value: 89.19399999999999 |
| - type: recall_at_10 |
| value: 98.639 |
| - type: recall_at_100 |
| value: 99.74499999999999 |
| - type: recall_at_1000 |
| value: 99.973 |
| - type: recall_at_20 |
| value: 99.122 |
| - type: recall_at_3 |
| value: 96.601 |
| - type: recall_at_5 |
| value: 97.80799999999999 |
| task: |
| type: Retrieval |
| - dataset: |
| config: javascript |
| name: MTEB CodeSearchNetCCRetrieval (javascript) |
| revision: 6e1effa2c03723c5fde48ee912b5ee08d4f211e8 |
| split: test |
| type: CoIR-Retrieval/CodeSearchNet-ccr |
| metrics: |
| - type: main_score |
| value: 91.532 |
| - type: map_at_1 |
| value: 84.04700000000001 |
| - type: map_at_10 |
| value: 89.50399999999999 |
| - type: map_at_100 |
| value: 89.581 |
| - type: map_at_1000 |
| value: 89.584 |
| - type: map_at_20 |
| value: 89.548 |
| - type: map_at_3 |
| value: 88.914 |
| - type: map_at_5 |
| value: 89.271 |
| - type: mrr_at_1 |
| value: 83.98663020358553 |
| - type: mrr_at_10 |
| value: 89.46699271220692 |
| - type: mrr_at_100 |
| value: 89.54429619092761 |
| - type: mrr_at_1000 |
| value: 89.54715966900696 |
| - type: mrr_at_20 |
| value: 89.51131350599995 |
| - type: mrr_at_3 |
| value: 88.8787602552416 |
| - type: mrr_at_5 |
| value: 89.23427529626254 |
| - type: nauc_map_at_1000_diff1 |
| value: 81.33895244348352 |
| - type: nauc_map_at_1000_max |
| value: 51.005827566402075 |
| - type: nauc_map_at_1000_std |
| value: -22.252578589937457 |
| - type: nauc_map_at_100_diff1 |
| value: 81.33433499029653 |
| - type: nauc_map_at_100_max |
| value: 51.01323653352282 |
| - type: nauc_map_at_100_std |
| value: -22.24639503900029 |
| - type: nauc_map_at_10_diff1 |
| value: 81.30647802788602 |
| - type: nauc_map_at_10_max |
| value: 51.211145238065484 |
| - type: nauc_map_at_10_std |
| value: -22.30319656889033 |
| - type: nauc_map_at_1_diff1 |
| value: 82.57372620311462 |
| - type: nauc_map_at_1_max |
| value: 44.54905481211588 |
| - type: nauc_map_at_1_std |
| value: -21.962071475865756 |
| - type: nauc_map_at_20_diff1 |
| value: 81.31928022528241 |
| - type: nauc_map_at_20_max |
| value: 51.13238604171399 |
| - type: nauc_map_at_20_std |
| value: -22.20703220818948 |
| - type: nauc_map_at_3_diff1 |
| value: 81.10849311491785 |
| - type: nauc_map_at_3_max |
| value: 51.8208019735269 |
| - type: nauc_map_at_3_std |
| value: -22.96789103735518 |
| - type: nauc_map_at_5_diff1 |
| value: 81.1494438340007 |
| - type: nauc_map_at_5_max |
| value: 51.54484151827443 |
| - type: nauc_map_at_5_std |
| value: -22.39835418757183 |
| - type: nauc_mrr_at_1000_diff1 |
| value: 81.42887177105008 |
| - type: nauc_mrr_at_1000_max |
| value: 50.78456409105162 |
| - type: nauc_mrr_at_1000_std |
| value: -22.810803121918408 |
| - type: nauc_mrr_at_100_diff1 |
| value: 81.42424440922578 |
| - type: nauc_mrr_at_100_max |
| value: 50.792010710475864 |
| - type: nauc_mrr_at_100_std |
| value: -22.804483531306143 |
| - type: nauc_mrr_at_10_diff1 |
| value: 81.39579265182797 |
| - type: nauc_mrr_at_10_max |
| value: 50.99096194207801 |
| - type: nauc_mrr_at_10_std |
| value: -22.856845472961556 |
| - type: nauc_mrr_at_1_diff1 |
| value: 82.68635369279959 |
| - type: nauc_mrr_at_1_max |
| value: 44.2393421717323 |
| - type: nauc_mrr_at_1_std |
| value: -22.775172610293808 |
| - type: nauc_mrr_at_20_diff1 |
| value: 81.40894511054444 |
| - type: nauc_mrr_at_20_max |
| value: 50.91149353347425 |
| - type: nauc_mrr_at_20_std |
| value: -22.763426041817088 |
| - type: nauc_mrr_at_3_diff1 |
| value: 81.1947777094554 |
| - type: nauc_mrr_at_3_max |
| value: 51.60584449653166 |
| - type: nauc_mrr_at_3_std |
| value: -23.48440779958821 |
| - type: nauc_mrr_at_5_diff1 |
| value: 81.23725416306374 |
| - type: nauc_mrr_at_5_max |
| value: 51.32853667650834 |
| - type: nauc_mrr_at_5_std |
| value: -22.939278713896133 |
| - type: nauc_ndcg_at_1000_diff1 |
| value: 81.12746150854298 |
| - type: nauc_ndcg_at_1000_max |
| value: 52.1786963598477 |
| - type: nauc_ndcg_at_1000_std |
| value: -21.66735535060729 |
| - type: nauc_ndcg_at_100_diff1 |
| value: 81.00850024967238 |
| - type: nauc_ndcg_at_100_max |
| value: 52.376900809779855 |
| - type: nauc_ndcg_at_100_std |
| value: -21.437187821641885 |
| - type: nauc_ndcg_at_10_diff1 |
| value: 80.87079398487964 |
| - type: nauc_ndcg_at_10_max |
| value: 53.719012858465916 |
| - type: nauc_ndcg_at_10_std |
| value: -21.57272074203413 |
| - type: nauc_ndcg_at_1_diff1 |
| value: 82.57372620311462 |
| - type: nauc_ndcg_at_1_max |
| value: 44.54905481211588 |
| - type: nauc_ndcg_at_1_std |
| value: -21.962071475865756 |
| - type: nauc_ndcg_at_20_diff1 |
| value: 80.89584525085091 |
| - type: nauc_ndcg_at_20_max |
| value: 53.401415874657296 |
| - type: nauc_ndcg_at_20_std |
| value: -21.129352577941283 |
| - type: nauc_ndcg_at_3_diff1 |
| value: 80.46103065917166 |
| - type: nauc_ndcg_at_3_max |
| value: 54.91263156860439 |
| - type: nauc_ndcg_at_3_std |
| value: -23.0656484747823 |
| - type: nauc_ndcg_at_5_diff1 |
| value: 80.50385469743831 |
| - type: nauc_ndcg_at_5_max |
| value: 54.53609881176663 |
| - type: nauc_ndcg_at_5_std |
| value: -21.826145336827878 |
| - type: nauc_precision_at_1000_diff1 |
| value: 81.4825203906492 |
| - type: nauc_precision_at_1000_max |
| value: 100.0 |
| - type: nauc_precision_at_1000_std |
| value: 100.0 |
| - type: nauc_precision_at_100_diff1 |
| value: 69.4979677737838 |
| - type: nauc_precision_at_100_max |
| value: 79.81453242811645 |
| - type: nauc_precision_at_100_std |
| value: 22.735882735419853 |
| - type: nauc_precision_at_10_diff1 |
| value: 76.02578489916326 |
| - type: nauc_precision_at_10_max |
| value: 82.20220582823438 |
| - type: nauc_precision_at_10_std |
| value: -11.133888291415301 |
| - type: nauc_precision_at_1_diff1 |
| value: 82.57372620311462 |
| - type: nauc_precision_at_1_max |
| value: 44.54905481211588 |
| - type: nauc_precision_at_1_std |
| value: -21.962071475865756 |
| - type: nauc_precision_at_20_diff1 |
| value: 74.57535506743962 |
| - type: nauc_precision_at_20_max |
| value: 86.22474553094298 |
| - type: nauc_precision_at_20_std |
| value: 0.37124480616714156 |
| - type: nauc_precision_at_3_diff1 |
| value: 76.90725827318549 |
| - type: nauc_precision_at_3_max |
| value: 71.80786851596808 |
| - type: nauc_precision_at_3_std |
| value: -23.38779609088237 |
| - type: nauc_precision_at_5_diff1 |
| value: 75.76703985132757 |
| - type: nauc_precision_at_5_max |
| value: 76.30757619721807 |
| - type: nauc_precision_at_5_std |
| value: -16.772096432698 |
| - type: nauc_recall_at_1000_diff1 |
| value: 81.48252039064056 |
| - type: nauc_recall_at_1000_max |
| value: 100.0 |
| - type: nauc_recall_at_1000_std |
| value: 100.0 |
| - type: nauc_recall_at_100_diff1 |
| value: 69.49796777378361 |
| - type: nauc_recall_at_100_max |
| value: 79.81453242811749 |
| - type: nauc_recall_at_100_std |
| value: 22.73588273542357 |
| - type: nauc_recall_at_10_diff1 |
| value: 76.0257848991628 |
| - type: nauc_recall_at_10_max |
| value: 82.20220582823433 |
| - type: nauc_recall_at_10_std |
| value: -11.13388829141429 |
| - type: nauc_recall_at_1_diff1 |
| value: 82.57372620311462 |
| - type: nauc_recall_at_1_max |
| value: 44.54905481211588 |
| - type: nauc_recall_at_1_std |
| value: -21.962071475865756 |
| - type: nauc_recall_at_20_diff1 |
| value: 74.57535506744009 |
| - type: nauc_recall_at_20_max |
| value: 86.22474553094256 |
| - type: nauc_recall_at_20_std |
| value: 0.3712448061678841 |
| - type: nauc_recall_at_3_diff1 |
| value: 76.90725827318568 |
| - type: nauc_recall_at_3_max |
| value: 71.80786851596824 |
| - type: nauc_recall_at_3_std |
| value: -23.38779609088207 |
| - type: nauc_recall_at_5_diff1 |
| value: 75.76703985132794 |
| - type: nauc_recall_at_5_max |
| value: 76.30757619721837 |
| - type: nauc_recall_at_5_std |
| value: -16.772096432697477 |
| - type: ndcg_at_1 |
| value: 84.04700000000001 |
| - type: ndcg_at_10 |
| value: 91.532 |
| - type: ndcg_at_100 |
| value: 91.914 |
| - type: ndcg_at_1000 |
| value: 91.979 |
| - type: ndcg_at_20 |
| value: 91.688 |
| - type: ndcg_at_3 |
| value: 90.35300000000001 |
| - type: ndcg_at_5 |
| value: 90.993 |
| - type: precision_at_1 |
| value: 84.04700000000001 |
| - type: precision_at_10 |
| value: 9.763 |
| - type: precision_at_100 |
| value: 0.9939999999999999 |
| - type: precision_at_1000 |
| value: 0.1 |
| - type: precision_at_20 |
| value: 4.912 |
| - type: precision_at_3 |
| value: 31.490000000000002 |
| - type: precision_at_5 |
| value: 19.204 |
| - type: recall_at_1 |
| value: 84.04700000000001 |
| - type: recall_at_10 |
| value: 97.63 |
| - type: recall_at_100 |
| value: 99.423 |
| - type: recall_at_1000 |
| value: 99.909 |
| - type: recall_at_20 |
| value: 98.238 |
| - type: recall_at_3 |
| value: 94.47 |
| - type: recall_at_5 |
| value: 96.019 |
| task: |
| type: Retrieval |
| - dataset: |
| config: go |
| name: MTEB CodeSearchNetCCRetrieval (go) |
| revision: 6e1effa2c03723c5fde48ee912b5ee08d4f211e8 |
| split: test |
| type: CoIR-Retrieval/CodeSearchNet-ccr |
| metrics: |
| - type: main_score |
| value: 91.75999999999999 |
| - type: map_at_1 |
| value: 84.28999999999999 |
| - type: map_at_10 |
| value: 89.689 |
| - type: map_at_100 |
| value: 89.773 |
| - type: map_at_1000 |
| value: 89.774 |
| - type: map_at_20 |
| value: 89.747 |
| - type: map_at_3 |
| value: 89.02600000000001 |
| - type: map_at_5 |
| value: 89.484 |
| - type: mrr_at_1 |
| value: 84.27727160797834 |
| - type: mrr_at_10 |
| value: 89.67515722532931 |
| - type: mrr_at_100 |
| value: 89.75855757010807 |
| - type: mrr_at_1000 |
| value: 89.7601837108608 |
| - type: mrr_at_20 |
| value: 89.73302201153147 |
| - type: mrr_at_3 |
| value: 89.01337929902301 |
| - type: mrr_at_5 |
| value: 89.47262579003495 |
| - type: nauc_map_at_1000_diff1 |
| value: 85.17428924238607 |
| - type: nauc_map_at_1000_max |
| value: 48.685961107484225 |
| - type: nauc_map_at_1000_std |
| value: -30.709156511297298 |
| - type: nauc_map_at_100_diff1 |
| value: 85.17188089036237 |
| - type: nauc_map_at_100_max |
| value: 48.6913539422267 |
| - type: nauc_map_at_100_std |
| value: -30.704177527491318 |
| - type: nauc_map_at_10_diff1 |
| value: 85.16218860131448 |
| - type: nauc_map_at_10_max |
| value: 48.8187190573415 |
| - type: nauc_map_at_10_std |
| value: -30.769000848055377 |
| - type: nauc_map_at_1_diff1 |
| value: 86.00618812234512 |
| - type: nauc_map_at_1_max |
| value: 45.7845929261639 |
| - type: nauc_map_at_1_std |
| value: -29.077490123642914 |
| - type: nauc_map_at_20_diff1 |
| value: 85.17186125725058 |
| - type: nauc_map_at_20_max |
| value: 48.7405854117929 |
| - type: nauc_map_at_20_std |
| value: -30.71428055810858 |
| - type: nauc_map_at_3_diff1 |
| value: 85.04460665149782 |
| - type: nauc_map_at_3_max |
| value: 49.06920858786511 |
| - type: nauc_map_at_3_std |
| value: -31.417225141010363 |
| - type: nauc_map_at_5_diff1 |
| value: 85.11442620241468 |
| - type: nauc_map_at_5_max |
| value: 48.834954058956406 |
| - type: nauc_map_at_5_std |
| value: -31.287907384502528 |
| - type: nauc_mrr_at_1000_diff1 |
| value: 85.21632400557363 |
| - type: nauc_mrr_at_1000_max |
| value: 48.84146877454467 |
| - type: nauc_mrr_at_1000_std |
| value: -30.7223552994054 |
| - type: nauc_mrr_at_100_diff1 |
| value: 85.2139127988098 |
| - type: nauc_mrr_at_100_max |
| value: 48.84683704195986 |
| - type: nauc_mrr_at_100_std |
| value: -30.717390859364478 |
| - type: nauc_mrr_at_10_diff1 |
| value: 85.20230661077468 |
| - type: nauc_mrr_at_10_max |
| value: 48.957929042209614 |
| - type: nauc_mrr_at_10_std |
| value: -30.78597710748554 |
| - type: nauc_mrr_at_1_diff1 |
| value: 86.0269378207981 |
| - type: nauc_mrr_at_1_max |
| value: 45.94255127280741 |
| - type: nauc_mrr_at_1_std |
| value: -29.011204207077935 |
| - type: nauc_mrr_at_20_diff1 |
| value: 85.21586630373561 |
| - type: nauc_mrr_at_20_max |
| value: 48.88183732057669 |
| - type: nauc_mrr_at_20_std |
| value: -30.73183934707418 |
| - type: nauc_mrr_at_3_diff1 |
| value: 85.08479102826571 |
| - type: nauc_mrr_at_3_max |
| value: 49.23008482061558 |
| - type: nauc_mrr_at_3_std |
| value: -31.465046952057303 |
| - type: nauc_mrr_at_5_diff1 |
| value: 85.1584637044273 |
| - type: nauc_mrr_at_5_max |
| value: 49.00177049406493 |
| - type: nauc_mrr_at_5_std |
| value: -31.307066011998224 |
| - type: nauc_ndcg_at_1000_diff1 |
| value: 85.09478158429043 |
| - type: nauc_ndcg_at_1000_max |
| value: 49.25560935342975 |
| - type: nauc_ndcg_at_1000_std |
| value: -30.70761801312305 |
| - type: nauc_ndcg_at_100_diff1 |
| value: 85.03251451319845 |
| - type: nauc_ndcg_at_100_max |
| value: 49.42104394245281 |
| - type: nauc_ndcg_at_100_std |
| value: -30.551562494744168 |
| - type: nauc_ndcg_at_10_diff1 |
| value: 85.00106483349344 |
| - type: nauc_ndcg_at_10_max |
| value: 50.115231910648575 |
| - type: nauc_ndcg_at_10_std |
| value: -30.87312388259668 |
| - type: nauc_ndcg_at_1_diff1 |
| value: 86.00618812234512 |
| - type: nauc_ndcg_at_1_max |
| value: 45.7845929261639 |
| - type: nauc_ndcg_at_1_std |
| value: -29.077490123642914 |
| - type: nauc_ndcg_at_20_diff1 |
| value: 85.05361380063081 |
| - type: nauc_ndcg_at_20_max |
| value: 49.772465794293254 |
| - type: nauc_ndcg_at_20_std |
| value: -30.672595686871666 |
| - type: nauc_ndcg_at_3_diff1 |
| value: 84.77305397981563 |
| - type: nauc_ndcg_at_3_max |
| value: 50.52441698065767 |
| - type: nauc_ndcg_at_3_std |
| value: -32.46848973997472 |
| - type: nauc_ndcg_at_5_diff1 |
| value: 84.88420002224352 |
| - type: nauc_ndcg_at_5_max |
| value: 50.11133138392584 |
| - type: nauc_ndcg_at_5_std |
| value: -32.3270235813713 |
| - type: nauc_precision_at_1000_diff1 |
| value: 91.83038946833551 |
| - type: nauc_precision_at_1000_max |
| value: 88.46880842429545 |
| - type: nauc_precision_at_1000_std |
| value: 21.769969322436477 |
| - type: nauc_precision_at_100_diff1 |
| value: 76.47557733881311 |
| - type: nauc_precision_at_100_max |
| value: 84.28732698302338 |
| - type: nauc_precision_at_100_std |
| value: 5.283523585432388 |
| - type: nauc_precision_at_10_diff1 |
| value: 83.18181354333525 |
| - type: nauc_precision_at_10_max |
| value: 68.00168082360678 |
| - type: nauc_precision_at_10_std |
| value: -29.843581018958474 |
| - type: nauc_precision_at_1_diff1 |
| value: 86.00618812234512 |
| - type: nauc_precision_at_1_max |
| value: 45.7845929261639 |
| - type: nauc_precision_at_1_std |
| value: -29.077490123642914 |
| - type: nauc_precision_at_20_diff1 |
| value: 83.4769114952918 |
| - type: nauc_precision_at_20_max |
| value: 71.01630819186266 |
| - type: nauc_precision_at_20_std |
| value: -24.608779769104114 |
| - type: nauc_precision_at_3_diff1 |
| value: 83.38151337361876 |
| - type: nauc_precision_at_3_max |
| value: 58.48742413861129 |
| - type: nauc_precision_at_3_std |
| value: -38.23005616508197 |
| - type: nauc_precision_at_5_diff1 |
| value: 83.16792264231816 |
| - type: nauc_precision_at_5_max |
| value: 60.3244549312709 |
| - type: nauc_precision_at_5_std |
| value: -40.87299770895208 |
| - type: nauc_recall_at_1000_diff1 |
| value: 91.83038946832791 |
| - type: nauc_recall_at_1000_max |
| value: 88.46880842431686 |
| - type: nauc_recall_at_1000_std |
| value: 21.769969322480403 |
| - type: nauc_recall_at_100_diff1 |
| value: 76.47557733881604 |
| - type: nauc_recall_at_100_max |
| value: 84.28732698303843 |
| - type: nauc_recall_at_100_std |
| value: 5.283523585434394 |
| - type: nauc_recall_at_10_diff1 |
| value: 83.18181354333507 |
| - type: nauc_recall_at_10_max |
| value: 68.00168082360621 |
| - type: nauc_recall_at_10_std |
| value: -29.843581018957572 |
| - type: nauc_recall_at_1_diff1 |
| value: 86.00618812234512 |
| - type: nauc_recall_at_1_max |
| value: 45.7845929261639 |
| - type: nauc_recall_at_1_std |
| value: -29.077490123642914 |
| - type: nauc_recall_at_20_diff1 |
| value: 83.47691149529236 |
| - type: nauc_recall_at_20_max |
| value: 71.01630819186342 |
| - type: nauc_recall_at_20_std |
| value: -24.608779769102323 |
| - type: nauc_recall_at_3_diff1 |
| value: 83.38151337361859 |
| - type: nauc_recall_at_3_max |
| value: 58.48742413861102 |
| - type: nauc_recall_at_3_std |
| value: -38.23005616508199 |
| - type: nauc_recall_at_5_diff1 |
| value: 83.16792264231778 |
| - type: nauc_recall_at_5_max |
| value: 60.32445493127084 |
| - type: nauc_recall_at_5_std |
| value: -40.8729977089511 |
| - type: ndcg_at_1 |
| value: 84.28999999999999 |
| - type: ndcg_at_10 |
| value: 91.75999999999999 |
| - type: ndcg_at_100 |
| value: 92.128 |
| - type: ndcg_at_1000 |
| value: 92.166 |
| - type: ndcg_at_20 |
| value: 91.965 |
| - type: ndcg_at_3 |
| value: 90.441 |
| - type: ndcg_at_5 |
| value: 91.264 |
| - type: precision_at_1 |
| value: 84.28999999999999 |
| - type: precision_at_10 |
| value: 9.802 |
| - type: precision_at_100 |
| value: 0.997 |
| - type: precision_at_1000 |
| value: 0.1 |
| - type: precision_at_20 |
| value: 4.941 |
| - type: precision_at_3 |
| value: 31.499 |
| - type: precision_at_5 |
| value: 19.298000000000002 |
| - type: recall_at_1 |
| value: 84.28999999999999 |
| - type: recall_at_10 |
| value: 98.018 |
| - type: recall_at_100 |
| value: 99.655 |
| - type: recall_at_1000 |
| value: 99.938 |
| - type: recall_at_20 |
| value: 98.818 |
| - type: recall_at_3 |
| value: 94.49600000000001 |
| - type: recall_at_5 |
| value: 96.491 |
| task: |
| type: Retrieval |
| - dataset: |
| config: ruby |
| name: MTEB CodeSearchNetCCRetrieval (ruby) |
| revision: 6e1effa2c03723c5fde48ee912b5ee08d4f211e8 |
| split: test |
| type: CoIR-Retrieval/CodeSearchNet-ccr |
| metrics: |
| - type: main_score |
| value: 89.25699999999999 |
| - type: map_at_1 |
| value: 80.967 |
| - type: map_at_10 |
| value: 86.978 |
| - type: map_at_100 |
| value: 87.113 |
| - type: map_at_1000 |
| value: 87.117 |
| - type: map_at_20 |
| value: 87.054 |
| - type: map_at_3 |
| value: 86.29400000000001 |
| - type: map_at_5 |
| value: 86.738 |
| - type: mrr_at_1 |
| value: 80.9674861221253 |
| - type: mrr_at_10 |
| value: 86.95775587528166 |
| - type: mrr_at_100 |
| value: 87.09311803235961 |
| - type: mrr_at_1000 |
| value: 87.09630839341808 |
| - type: mrr_at_20 |
| value: 87.03383454810835 |
| - type: mrr_at_3 |
| value: 86.25429553264607 |
| - type: mrr_at_5 |
| value: 86.71821305841928 |
| - type: nauc_map_at_1000_diff1 |
| value: 82.17808945938484 |
| - type: nauc_map_at_1000_max |
| value: 60.75779539267633 |
| - type: nauc_map_at_1000_std |
| value: -23.755896580375826 |
| - type: nauc_map_at_100_diff1 |
| value: 82.1794560801495 |
| - type: nauc_map_at_100_max |
| value: 60.76366486911127 |
| - type: nauc_map_at_100_std |
| value: -23.74131956559222 |
| - type: nauc_map_at_10_diff1 |
| value: 82.18782847323823 |
| - type: nauc_map_at_10_max |
| value: 60.91576878869061 |
| - type: nauc_map_at_10_std |
| value: -23.863677820347533 |
| - type: nauc_map_at_1_diff1 |
| value: 83.72039170136367 |
| - type: nauc_map_at_1_max |
| value: 57.25380612858921 |
| - type: nauc_map_at_1_std |
| value: -25.935791475913987 |
| - type: nauc_map_at_20_diff1 |
| value: 82.17796635750986 |
| - type: nauc_map_at_20_max |
| value: 60.83301592914025 |
| - type: nauc_map_at_20_std |
| value: -23.683955007198033 |
| - type: nauc_map_at_3_diff1 |
| value: 81.83929922286214 |
| - type: nauc_map_at_3_max |
| value: 61.30090311509839 |
| - type: nauc_map_at_3_std |
| value: -23.67250024837735 |
| - type: nauc_map_at_5_diff1 |
| value: 81.88267528957118 |
| - type: nauc_map_at_5_max |
| value: 61.06118360132867 |
| - type: nauc_map_at_5_std |
| value: -23.564202037953418 |
| - type: nauc_mrr_at_1000_diff1 |
| value: 82.24831373360206 |
| - type: nauc_mrr_at_1000_max |
| value: 60.547458216811776 |
| - type: nauc_mrr_at_1000_std |
| value: -24.62799535642754 |
| - type: nauc_mrr_at_100_diff1 |
| value: 82.24966010195052 |
| - type: nauc_mrr_at_100_max |
| value: 60.553373399867574 |
| - type: nauc_mrr_at_100_std |
| value: -24.613214185234447 |
| - type: nauc_mrr_at_10_diff1 |
| value: 82.25711255868444 |
| - type: nauc_mrr_at_10_max |
| value: 60.70674951461077 |
| - type: nauc_mrr_at_10_std |
| value: -24.723134285637112 |
| - type: nauc_mrr_at_1_diff1 |
| value: 83.72039170136367 |
| - type: nauc_mrr_at_1_max |
| value: 56.98766026206557 |
| - type: nauc_mrr_at_1_std |
| value: -26.606885807082385 |
| - type: nauc_mrr_at_20_diff1 |
| value: 82.24783610762223 |
| - type: nauc_mrr_at_20_max |
| value: 60.62362881960727 |
| - type: nauc_mrr_at_20_std |
| value: -24.551756212224145 |
| - type: nauc_mrr_at_3_diff1 |
| value: 81.93274024955835 |
| - type: nauc_mrr_at_3_max |
| value: 61.09367721118326 |
| - type: nauc_mrr_at_3_std |
| value: -24.713070452305637 |
| - type: nauc_mrr_at_5_diff1 |
| value: 81.95108632175992 |
| - type: nauc_mrr_at_5_max |
| value: 60.85590424882734 |
| - type: nauc_mrr_at_5_std |
| value: -24.40785466160273 |
| - type: nauc_ndcg_at_1000_diff1 |
| value: 82.13638313064818 |
| - type: nauc_ndcg_at_1000_max |
| value: 61.36229533164094 |
| - type: nauc_ndcg_at_1000_std |
| value: -23.263619782617027 |
| - type: nauc_ndcg_at_100_diff1 |
| value: 82.17708460929015 |
| - type: nauc_ndcg_at_100_max |
| value: 61.55039283837963 |
| - type: nauc_ndcg_at_100_std |
| value: -22.82475534090763 |
| - type: nauc_ndcg_at_10_diff1 |
| value: 82.19581572094847 |
| - type: nauc_ndcg_at_10_max |
| value: 62.250754196853194 |
| - type: nauc_ndcg_at_10_std |
| value: -23.522643567705575 |
| - type: nauc_ndcg_at_1_diff1 |
| value: 83.72039170136367 |
| - type: nauc_ndcg_at_1_max |
| value: 57.25380612858921 |
| - type: nauc_ndcg_at_1_std |
| value: -25.935791475913987 |
| - type: nauc_ndcg_at_20_diff1 |
| value: 82.14559730614958 |
| - type: nauc_ndcg_at_20_max |
| value: 61.93079238970315 |
| - type: nauc_ndcg_at_20_std |
| value: -22.719105026052123 |
| - type: nauc_ndcg_at_3_diff1 |
| value: 81.36414038291007 |
| - type: nauc_ndcg_at_3_max |
| value: 62.97389940233761 |
| - type: nauc_ndcg_at_3_std |
| value: -22.958679785157702 |
| - type: nauc_ndcg_at_5_diff1 |
| value: 81.41077106697202 |
| - type: nauc_ndcg_at_5_max |
| value: 62.605999596646335 |
| - type: nauc_ndcg_at_5_std |
| value: -22.684305831996262 |
| - type: nauc_precision_at_1000_diff1 |
| value: 100.0 |
| - type: nauc_precision_at_1000_max |
| value: 100.0 |
| - type: nauc_precision_at_1000_std |
| value: 55.42527628232039 |
| - type: nauc_precision_at_100_diff1 |
| value: 87.74666354712059 |
| - type: nauc_precision_at_100_max |
| value: 83.08111330779417 |
| - type: nauc_precision_at_100_std |
| value: 25.395348413223466 |
| - type: nauc_precision_at_10_diff1 |
| value: 83.35598753622322 |
| - type: nauc_precision_at_10_max |
| value: 73.70169446342423 |
| - type: nauc_precision_at_10_std |
| value: -21.615549803915652 |
| - type: nauc_precision_at_1_diff1 |
| value: 83.72039170136367 |
| - type: nauc_precision_at_1_max |
| value: 57.25380612858921 |
| - type: nauc_precision_at_1_std |
| value: -25.935791475913987 |
| - type: nauc_precision_at_20_diff1 |
| value: 83.03143013817835 |
| - type: nauc_precision_at_20_max |
| value: 73.49113429650389 |
| - type: nauc_precision_at_20_std |
| value: -9.826961687536869 |
| - type: nauc_precision_at_3_diff1 |
| value: 79.16945277065827 |
| - type: nauc_precision_at_3_max |
| value: 71.31564419236823 |
| - type: nauc_precision_at_3_std |
| value: -19.52745553766472 |
| - type: nauc_precision_at_5_diff1 |
| value: 78.5956097040408 |
| - type: nauc_precision_at_5_max |
| value: 72.48547478245246 |
| - type: nauc_precision_at_5_std |
| value: -17.07438252522886 |
| - type: nauc_recall_at_1000_diff1 |
| value: 100.0 |
| - type: nauc_recall_at_1000_max |
| value: 100.0 |
| - type: nauc_recall_at_1000_std |
| value: 55.42527628229722 |
| - type: nauc_recall_at_100_diff1 |
| value: 87.74666354712278 |
| - type: nauc_recall_at_100_max |
| value: 83.08111330779381 |
| - type: nauc_recall_at_100_std |
| value: 25.39534841322166 |
| - type: nauc_recall_at_10_diff1 |
| value: 83.35598753622315 |
| - type: nauc_recall_at_10_max |
| value: 73.70169446342429 |
| - type: nauc_recall_at_10_std |
| value: -21.615549803915744 |
| - type: nauc_recall_at_1_diff1 |
| value: 83.72039170136367 |
| - type: nauc_recall_at_1_max |
| value: 57.25380612858921 |
| - type: nauc_recall_at_1_std |
| value: -25.935791475913987 |
| - type: nauc_recall_at_20_diff1 |
| value: 83.03143013817795 |
| - type: nauc_recall_at_20_max |
| value: 73.49113429650306 |
| - type: nauc_recall_at_20_std |
| value: -9.826961687536835 |
| - type: nauc_recall_at_3_diff1 |
| value: 79.16945277065781 |
| - type: nauc_recall_at_3_max |
| value: 71.31564419236814 |
| - type: nauc_recall_at_3_std |
| value: -19.527455537664128 |
| - type: nauc_recall_at_5_diff1 |
| value: 78.59560970404043 |
| - type: nauc_recall_at_5_max |
| value: 72.4854747824523 |
| - type: nauc_recall_at_5_std |
| value: -17.074382525229172 |
| - type: ndcg_at_1 |
| value: 80.967 |
| - type: ndcg_at_10 |
| value: 89.25699999999999 |
| - type: ndcg_at_100 |
| value: 89.91799999999999 |
| - type: ndcg_at_1000 |
| value: 90.007 |
| - type: ndcg_at_20 |
| value: 89.524 |
| - type: ndcg_at_3 |
| value: 87.909 |
| - type: ndcg_at_5 |
| value: 88.701 |
| - type: precision_at_1 |
| value: 80.967 |
| - type: precision_at_10 |
| value: 9.611 |
| - type: precision_at_100 |
| value: 0.992 |
| - type: precision_at_1000 |
| value: 0.1 |
| - type: precision_at_20 |
| value: 4.857 |
| - type: precision_at_3 |
| value: 30.849 |
| - type: precision_at_5 |
| value: 18.89 |
| - type: recall_at_1 |
| value: 80.967 |
| - type: recall_at_10 |
| value: 96.114 |
| - type: recall_at_100 |
| value: 99.207 |
| - type: recall_at_1000 |
| value: 99.921 |
| - type: recall_at_20 |
| value: 97.14500000000001 |
| - type: recall_at_3 |
| value: 92.54599999999999 |
| - type: recall_at_5 |
| value: 94.44900000000001 |
| task: |
| type: Retrieval |
| - dataset: |
| config: java |
| name: MTEB CodeSearchNetCCRetrieval (java) |
| revision: 6e1effa2c03723c5fde48ee912b5ee08d4f211e8 |
| split: test |
| type: CoIR-Retrieval/CodeSearchNet-ccr |
| metrics: |
| - type: main_score |
| value: 90.187 |
| - type: map_at_1 |
| value: 83.04899999999999 |
| - type: map_at_10 |
| value: 88.2 |
| - type: map_at_100 |
| value: 88.294 |
| - type: map_at_1000 |
| value: 88.297 |
| - type: map_at_20 |
| value: 88.268 |
| - type: map_at_3 |
| value: 87.581 |
| - type: map_at_5 |
| value: 87.997 |
| - type: mrr_at_1 |
| value: 83.03057964399817 |
| - type: mrr_at_10 |
| value: 88.19186353987806 |
| - type: mrr_at_100 |
| value: 88.2852857857312 |
| - type: mrr_at_1000 |
| value: 88.28810205243452 |
| - type: mrr_at_20 |
| value: 88.25951626896247 |
| - type: mrr_at_3 |
| value: 87.5749277346721 |
| - type: mrr_at_5 |
| value: 87.9875247223489 |
| - type: nauc_map_at_1000_diff1 |
| value: 84.10730394433992 |
| - type: nauc_map_at_1000_max |
| value: 61.52233212127969 |
| - type: nauc_map_at_1000_std |
| value: -13.441699805696281 |
| - type: nauc_map_at_100_diff1 |
| value: 84.10552079587292 |
| - type: nauc_map_at_100_max |
| value: 61.52588029468992 |
| - type: nauc_map_at_100_std |
| value: -13.438768702376994 |
| - type: nauc_map_at_10_diff1 |
| value: 84.06622800697097 |
| - type: nauc_map_at_10_max |
| value: 61.62852938331853 |
| - type: nauc_map_at_10_std |
| value: -13.540123911279858 |
| - type: nauc_map_at_1_diff1 |
| value: 84.58928110942394 |
| - type: nauc_map_at_1_max |
| value: 56.15285789003811 |
| - type: nauc_map_at_1_std |
| value: -17.221403594342735 |
| - type: nauc_map_at_20_diff1 |
| value: 84.09390965699994 |
| - type: nauc_map_at_20_max |
| value: 61.569980648560886 |
| - type: nauc_map_at_20_std |
| value: -13.445035300847309 |
| - type: nauc_map_at_3_diff1 |
| value: 83.88744798135446 |
| - type: nauc_map_at_3_max |
| value: 61.88368527363093 |
| - type: nauc_map_at_3_std |
| value: -14.711176741863827 |
| - type: nauc_map_at_5_diff1 |
| value: 83.98674004934382 |
| - type: nauc_map_at_5_max |
| value: 61.74583336524621 |
| - type: nauc_map_at_5_std |
| value: -13.965194914228382 |
| - type: nauc_mrr_at_1000_diff1 |
| value: 84.07413270132193 |
| - type: nauc_mrr_at_1000_max |
| value: 61.703650435638856 |
| - type: nauc_mrr_at_1000_std |
| value: -13.350251134092783 |
| - type: nauc_mrr_at_100_diff1 |
| value: 84.07235854433452 |
| - type: nauc_mrr_at_100_max |
| value: 61.707169972354 |
| - type: nauc_mrr_at_100_std |
| value: -13.347338815823864 |
| - type: nauc_mrr_at_10_diff1 |
| value: 84.03062092695845 |
| - type: nauc_mrr_at_10_max |
| value: 61.80570291957679 |
| - type: nauc_mrr_at_10_std |
| value: -13.438890580569844 |
| - type: nauc_mrr_at_1_diff1 |
| value: 84.61937189362341 |
| - type: nauc_mrr_at_1_max |
| value: 56.41943088748145 |
| - type: nauc_mrr_at_1_std |
| value: -17.177446751086197 |
| - type: nauc_mrr_at_20_diff1 |
| value: 84.06068844396994 |
| - type: nauc_mrr_at_20_max |
| value: 61.750926871721475 |
| - type: nauc_mrr_at_20_std |
| value: -13.35357387841223 |
| - type: nauc_mrr_at_3_diff1 |
| value: 83.85345674418377 |
| - type: nauc_mrr_at_3_max |
| value: 62.043812033960535 |
| - type: nauc_mrr_at_3_std |
| value: -14.6395621624969 |
| - type: nauc_mrr_at_5_diff1 |
| value: 83.95764939870173 |
| - type: nauc_mrr_at_5_max |
| value: 61.92516669938368 |
| - type: nauc_mrr_at_5_std |
| value: -13.886532596685806 |
| - type: nauc_ndcg_at_1000_diff1 |
| value: 84.42408974340631 |
| - type: nauc_ndcg_at_1000_max |
| value: 63.32980075889167 |
| - type: nauc_ndcg_at_1000_std |
| value: -9.859850962580813 |
| - type: nauc_ndcg_at_100_diff1 |
| value: 84.3765359347518 |
| - type: nauc_ndcg_at_100_max |
| value: 63.39891417852467 |
| - type: nauc_ndcg_at_100_std |
| value: -9.822808442396664 |
| - type: nauc_ndcg_at_10_diff1 |
| value: 84.14419181720488 |
| - type: nauc_ndcg_at_10_max |
| value: 63.950601713958136 |
| - type: nauc_ndcg_at_10_std |
| value: -10.490521537540003 |
| - type: nauc_ndcg_at_1_diff1 |
| value: 84.58928110942394 |
| - type: nauc_ndcg_at_1_max |
| value: 56.15285789003811 |
| - type: nauc_ndcg_at_1_std |
| value: -17.221403594342735 |
| - type: nauc_ndcg_at_20_diff1 |
| value: 84.26722898082026 |
| - type: nauc_ndcg_at_20_max |
| value: 63.749766448071746 |
| - type: nauc_ndcg_at_20_std |
| value: -10.004576105283306 |
| - type: nauc_ndcg_at_3_diff1 |
| value: 83.72365997941274 |
| - type: nauc_ndcg_at_3_max |
| value: 64.3195097894205 |
| - type: nauc_ndcg_at_3_std |
| value: -13.41750319915828 |
| - type: nauc_ndcg_at_5_diff1 |
| value: 83.91163247137122 |
| - type: nauc_ndcg_at_5_max |
| value: 64.19792463538528 |
| - type: nauc_ndcg_at_5_std |
| value: -11.76492800063572 |
| - type: nauc_precision_at_1000_diff1 |
| value: 97.13217959743038 |
| - type: nauc_precision_at_1000_max |
| value: 98.46911971679215 |
| - type: nauc_precision_at_1000_std |
| value: 90.2347995052506 |
| - type: nauc_precision_at_100_diff1 |
| value: 91.63251470011757 |
| - type: nauc_precision_at_100_max |
| value: 90.17514956130655 |
| - type: nauc_precision_at_100_std |
| value: 61.0806660186732 |
| - type: nauc_precision_at_10_diff1 |
| value: 85.28726061856946 |
| - type: nauc_precision_at_10_max |
| value: 83.23195279280301 |
| - type: nauc_precision_at_10_std |
| value: 17.658421447762464 |
| - type: nauc_precision_at_1_diff1 |
| value: 84.58928110942394 |
| - type: nauc_precision_at_1_max |
| value: 56.15285789003811 |
| - type: nauc_precision_at_1_std |
| value: -17.221403594342735 |
| - type: nauc_precision_at_20_diff1 |
| value: 87.21693331269776 |
| - type: nauc_precision_at_20_max |
| value: 86.97422806229869 |
| - type: nauc_precision_at_20_std |
| value: 32.947726980858015 |
| - type: nauc_precision_at_3_diff1 |
| value: 83.00721034174204 |
| - type: nauc_precision_at_3_max |
| value: 76.03478845540039 |
| - type: nauc_precision_at_3_std |
| value: -7.075307314296672 |
| - type: nauc_precision_at_5_diff1 |
| value: 83.59241003022704 |
| - type: nauc_precision_at_5_max |
| value: 79.35871830248998 |
| - type: nauc_precision_at_5_std |
| value: 2.590360587580312 |
| - type: nauc_recall_at_1000_diff1 |
| value: 97.13217959743086 |
| - type: nauc_recall_at_1000_max |
| value: 98.46911971679087 |
| - type: nauc_recall_at_1000_std |
| value: 90.23479950524963 |
| - type: nauc_recall_at_100_diff1 |
| value: 91.6325147001174 |
| - type: nauc_recall_at_100_max |
| value: 90.17514956130668 |
| - type: nauc_recall_at_100_std |
| value: 61.08066601867284 |
| - type: nauc_recall_at_10_diff1 |
| value: 85.28726061856851 |
| - type: nauc_recall_at_10_max |
| value: 83.23195279280242 |
| - type: nauc_recall_at_10_std |
| value: 17.65842144776203 |
| - type: nauc_recall_at_1_diff1 |
| value: 84.58928110942394 |
| - type: nauc_recall_at_1_max |
| value: 56.15285789003811 |
| - type: nauc_recall_at_1_std |
| value: -17.221403594342735 |
| - type: nauc_recall_at_20_diff1 |
| value: 87.21693331269785 |
| - type: nauc_recall_at_20_max |
| value: 86.97422806229821 |
| - type: nauc_recall_at_20_std |
| value: 32.94772698085802 |
| - type: nauc_recall_at_3_diff1 |
| value: 83.007210341742 |
| - type: nauc_recall_at_3_max |
| value: 76.03478845540012 |
| - type: nauc_recall_at_3_std |
| value: -7.075307314297068 |
| - type: nauc_recall_at_5_diff1 |
| value: 83.59241003022677 |
| - type: nauc_recall_at_5_max |
| value: 79.35871830248969 |
| - type: nauc_recall_at_5_std |
| value: 2.5903605875805535 |
| - type: ndcg_at_1 |
| value: 83.04899999999999 |
| - type: ndcg_at_10 |
| value: 90.187 |
| - type: ndcg_at_100 |
| value: 90.61399999999999 |
| - type: ndcg_at_1000 |
| value: 90.686 |
| - type: ndcg_at_20 |
| value: 90.429 |
| - type: ndcg_at_3 |
| value: 88.957 |
| - type: ndcg_at_5 |
| value: 89.7 |
| - type: precision_at_1 |
| value: 83.04899999999999 |
| - type: precision_at_10 |
| value: 9.619 |
| - type: precision_at_100 |
| value: 0.9809999999999999 |
| - type: precision_at_1000 |
| value: 0.099 |
| - type: precision_at_20 |
| value: 4.857 |
| - type: precision_at_3 |
| value: 30.969 |
| - type: precision_at_5 |
| value: 18.939 |
| - type: recall_at_1 |
| value: 83.04899999999999 |
| - type: recall_at_10 |
| value: 96.194 |
| - type: recall_at_100 |
| value: 98.129 |
| - type: recall_at_1000 |
| value: 98.695 |
| - type: recall_at_20 |
| value: 97.134 |
| - type: recall_at_3 |
| value: 92.907 |
| - type: recall_at_5 |
| value: 94.696 |
| task: |
| type: Retrieval |
| - dataset: |
| config: php |
| name: MTEB CodeSearchNetCCRetrieval (php) |
| revision: 6e1effa2c03723c5fde48ee912b5ee08d4f211e8 |
| split: test |
| type: CoIR-Retrieval/CodeSearchNet-ccr |
| metrics: |
| - type: main_score |
| value: 83.38799999999999 |
| - type: map_at_1 |
| value: 73.25500000000001 |
| - type: map_at_10 |
| value: 80.387 |
| - type: map_at_100 |
| value: 80.64099999999999 |
| - type: map_at_1000 |
| value: 80.648 |
| - type: map_at_20 |
| value: 80.55799999999999 |
| - type: map_at_3 |
| value: 79.256 |
| - type: map_at_5 |
| value: 79.97800000000001 |
| - type: mrr_at_1 |
| value: 73.24818039103754 |
| - type: mrr_at_10 |
| value: 80.37402789953873 |
| - type: mrr_at_100 |
| value: 80.62749361355517 |
| - type: mrr_at_1000 |
| value: 80.63492699285514 |
| - type: mrr_at_20 |
| value: 80.5450937498758 |
| - type: mrr_at_3 |
| value: 79.25170068027252 |
| - type: mrr_at_5 |
| value: 79.96063460349235 |
| - type: nauc_map_at_1000_diff1 |
| value: 77.55679733254976 |
| - type: nauc_map_at_1000_max |
| value: 47.30416473970588 |
| - type: nauc_map_at_1000_std |
| value: -26.244493881093646 |
| - type: nauc_map_at_100_diff1 |
| value: 77.55521791705124 |
| - type: nauc_map_at_100_max |
| value: 47.30912923045687 |
| - type: nauc_map_at_100_std |
| value: -26.243222353374808 |
| - type: nauc_map_at_10_diff1 |
| value: 77.46319200229681 |
| - type: nauc_map_at_10_max |
| value: 47.42459632571874 |
| - type: nauc_map_at_10_std |
| value: -26.359798504203013 |
| - type: nauc_map_at_1_diff1 |
| value: 79.527051162636 |
| - type: nauc_map_at_1_max |
| value: 44.89631724552463 |
| - type: nauc_map_at_1_std |
| value: -25.451766234360097 |
| - type: nauc_map_at_20_diff1 |
| value: 77.52788532848108 |
| - type: nauc_map_at_20_max |
| value: 47.33830573278101 |
| - type: nauc_map_at_20_std |
| value: -26.257482321628622 |
| - type: nauc_map_at_3_diff1 |
| value: 77.41062263238054 |
| - type: nauc_map_at_3_max |
| value: 47.499244399918155 |
| - type: nauc_map_at_3_std |
| value: -26.523866758341992 |
| - type: nauc_map_at_5_diff1 |
| value: 77.35129682203655 |
| - type: nauc_map_at_5_max |
| value: 47.47681231525524 |
| - type: nauc_map_at_5_std |
| value: -26.5173636683384 |
| - type: nauc_mrr_at_1000_diff1 |
| value: 77.57769206330354 |
| - type: nauc_mrr_at_1000_max |
| value: 47.34367340151292 |
| - type: nauc_mrr_at_1000_std |
| value: -26.1670476224202 |
| - type: nauc_mrr_at_100_diff1 |
| value: 77.57602878422465 |
| - type: nauc_mrr_at_100_max |
| value: 47.34876516670401 |
| - type: nauc_mrr_at_100_std |
| value: -26.16596878042115 |
| - type: nauc_mrr_at_10_diff1 |
| value: 77.48926507168112 |
| - type: nauc_mrr_at_10_max |
| value: 47.46252680868696 |
| - type: nauc_mrr_at_10_std |
| value: -26.282201461393317 |
| - type: nauc_mrr_at_1_diff1 |
| value: 79.53993768139632 |
| - type: nauc_mrr_at_1_max |
| value: 44.95304522643992 |
| - type: nauc_mrr_at_1_std |
| value: -25.32633885677888 |
| - type: nauc_mrr_at_20_diff1 |
| value: 77.54890591729169 |
| - type: nauc_mrr_at_20_max |
| value: 47.37631632390287 |
| - type: nauc_mrr_at_20_std |
| value: -26.183017296393896 |
| - type: nauc_mrr_at_3_diff1 |
| value: 77.4534326297116 |
| - type: nauc_mrr_at_3_max |
| value: 47.54929766473431 |
| - type: nauc_mrr_at_3_std |
| value: -26.432202942463785 |
| - type: nauc_mrr_at_5_diff1 |
| value: 77.38739995097214 |
| - type: nauc_mrr_at_5_max |
| value: 47.50110390682936 |
| - type: nauc_mrr_at_5_std |
| value: -26.463416200561753 |
| - type: nauc_ndcg_at_1000_diff1 |
| value: 77.20842500889081 |
| - type: nauc_ndcg_at_1000_max |
| value: 47.70813895735421 |
| - type: nauc_ndcg_at_1000_std |
| value: -26.023069702012446 |
| - type: nauc_ndcg_at_100_diff1 |
| value: 77.16015320575458 |
| - type: nauc_ndcg_at_100_max |
| value: 47.88272658215937 |
| - type: nauc_ndcg_at_100_std |
| value: -25.893271327002005 |
| - type: nauc_ndcg_at_10_diff1 |
| value: 76.68338764268012 |
| - type: nauc_ndcg_at_10_max |
| value: 48.4217483255242 |
| - type: nauc_ndcg_at_10_std |
| value: -26.36940839993425 |
| - type: nauc_ndcg_at_1_diff1 |
| value: 79.527051162636 |
| - type: nauc_ndcg_at_1_max |
| value: 44.89631724552463 |
| - type: nauc_ndcg_at_1_std |
| value: -25.451766234360097 |
| - type: nauc_ndcg_at_20_diff1 |
| value: 76.9371111655833 |
| - type: nauc_ndcg_at_20_max |
| value: 48.10138804970404 |
| - type: nauc_ndcg_at_20_std |
| value: -25.97419280800229 |
| - type: nauc_ndcg_at_3_diff1 |
| value: 76.61248461087024 |
| - type: nauc_ndcg_at_3_max |
| value: 48.51663194946113 |
| - type: nauc_ndcg_at_3_std |
| value: -26.789177280257793 |
| - type: nauc_ndcg_at_5_diff1 |
| value: 76.42970850899898 |
| - type: nauc_ndcg_at_5_max |
| value: 48.52719429311673 |
| - type: nauc_ndcg_at_5_std |
| value: -26.78716430555766 |
| - type: nauc_precision_at_1000_diff1 |
| value: 70.03887145024319 |
| - type: nauc_precision_at_1000_max |
| value: 42.00416767646647 |
| - type: nauc_precision_at_1000_std |
| value: -2.156778197450252 |
| - type: nauc_precision_at_100_diff1 |
| value: 72.9743402702392 |
| - type: nauc_precision_at_100_max |
| value: 56.4734539374456 |
| - type: nauc_precision_at_100_std |
| value: -12.25156037378497 |
| - type: nauc_precision_at_10_diff1 |
| value: 71.50550227914846 |
| - type: nauc_precision_at_10_max |
| value: 55.096376602923144 |
| - type: nauc_precision_at_10_std |
| value: -25.929290375863356 |
| - type: nauc_precision_at_1_diff1 |
| value: 79.527051162636 |
| - type: nauc_precision_at_1_max |
| value: 44.89631724552463 |
| - type: nauc_precision_at_1_std |
| value: -25.451766234360097 |
| - type: nauc_precision_at_20_diff1 |
| value: 71.99203963125369 |
| - type: nauc_precision_at_20_max |
| value: 54.66712448432459 |
| - type: nauc_precision_at_20_std |
| value: -21.478852653786358 |
| - type: nauc_precision_at_3_diff1 |
| value: 73.33785677931962 |
| - type: nauc_precision_at_3_max |
| value: 52.70785427475565 |
| - type: nauc_precision_at_3_std |
| value: -27.81147380600097 |
| - type: nauc_precision_at_5_diff1 |
| value: 71.76130552913553 |
| - type: nauc_precision_at_5_max |
| value: 53.78865964066962 |
| - type: nauc_precision_at_5_std |
| value: -28.044309543143616 |
| - type: nauc_recall_at_1000_diff1 |
| value: 70.03887145024234 |
| - type: nauc_recall_at_1000_max |
| value: 42.00416767647104 |
| - type: nauc_recall_at_1000_std |
| value: -2.15677819744854 |
| - type: nauc_recall_at_100_diff1 |
| value: 72.974340270239 |
| - type: nauc_recall_at_100_max |
| value: 56.473453937445306 |
| - type: nauc_recall_at_100_std |
| value: -12.25156037378462 |
| - type: nauc_recall_at_10_diff1 |
| value: 71.50550227914843 |
| - type: nauc_recall_at_10_max |
| value: 55.096376602923 |
| - type: nauc_recall_at_10_std |
| value: -25.929290375863673 |
| - type: nauc_recall_at_1_diff1 |
| value: 79.527051162636 |
| - type: nauc_recall_at_1_max |
| value: 44.89631724552463 |
| - type: nauc_recall_at_1_std |
| value: -25.451766234360097 |
| - type: nauc_recall_at_20_diff1 |
| value: 71.992039631253 |
| - type: nauc_recall_at_20_max |
| value: 54.66712448432437 |
| - type: nauc_recall_at_20_std |
| value: -21.478852653786607 |
| - type: nauc_recall_at_3_diff1 |
| value: 73.33785677931951 |
| - type: nauc_recall_at_3_max |
| value: 52.707854274755626 |
| - type: nauc_recall_at_3_std |
| value: -27.811473806001057 |
| - type: nauc_recall_at_5_diff1 |
| value: 71.76130552913536 |
| - type: nauc_recall_at_5_max |
| value: 53.788659640669344 |
| - type: nauc_recall_at_5_std |
| value: -28.044309543143715 |
| - type: ndcg_at_1 |
| value: 73.25500000000001 |
| - type: ndcg_at_10 |
| value: 83.38799999999999 |
| - type: ndcg_at_100 |
| value: 84.571 |
| - type: ndcg_at_1000 |
| value: 84.768 |
| - type: ndcg_at_20 |
| value: 83.99799999999999 |
| - type: ndcg_at_3 |
| value: 81.118 |
| - type: ndcg_at_5 |
| value: 82.40899999999999 |
| - type: precision_at_1 |
| value: 73.25500000000001 |
| - type: precision_at_10 |
| value: 9.259 |
| - type: precision_at_100 |
| value: 0.98 |
| - type: precision_at_1000 |
| value: 0.1 |
| - type: precision_at_20 |
| value: 4.749 |
| - type: precision_at_3 |
| value: 28.826 |
| - type: precision_at_5 |
| value: 17.918 |
| - type: recall_at_1 |
| value: 73.25500000000001 |
| - type: recall_at_10 |
| value: 92.586 |
| - type: recall_at_100 |
| value: 98.016 |
| - type: recall_at_1000 |
| value: 99.572 |
| - type: recall_at_20 |
| value: 94.976 |
| - type: recall_at_3 |
| value: 86.478 |
| - type: recall_at_5 |
| value: 89.589 |
| task: |
| type: Retrieval |
| - dataset: |
| config: python |
| name: MTEB CodeSearchNetRetrieval (python) |
| revision: fdc6a9e39575768c27eb8a2a5f702bf846eb4759 |
| split: test |
| type: code-search-net/code_search_net |
| metrics: |
| - type: main_score |
| value: 96.688 |
| - type: map_at_1 |
| value: 92.7 |
| - type: map_at_10 |
| value: 95.643 |
| - type: map_at_100 |
| value: 95.649 |
| - type: map_at_1000 |
| value: 95.649 |
| - type: map_at_20 |
| value: 95.643 |
| - type: map_at_3 |
| value: 95.417 |
| - type: map_at_5 |
| value: 95.562 |
| - type: mrr_at_1 |
| value: 92.7 |
| - type: mrr_at_10 |
| value: 95.64289682539683 |
| - type: mrr_at_100 |
| value: 95.64912393162395 |
| - type: mrr_at_1000 |
| value: 95.64912393162395 |
| - type: mrr_at_20 |
| value: 95.64289682539683 |
| - type: mrr_at_3 |
| value: 95.41666666666669 |
| - type: mrr_at_5 |
| value: 95.56166666666668 |
| - type: nauc_map_at_1000_diff1 |
| value: 93.2086804630001 |
| - type: nauc_map_at_1000_max |
| value: 71.36336949003405 |
| - type: nauc_map_at_1000_std |
| value: 7.542855305507389 |
| - type: nauc_map_at_100_diff1 |
| value: 93.2086804630001 |
| - type: nauc_map_at_100_max |
| value: 71.36336949003405 |
| - type: nauc_map_at_100_std |
| value: 7.542855305507389 |
| - type: nauc_map_at_10_diff1 |
| value: 93.21838651458631 |
| - type: nauc_map_at_10_max |
| value: 71.40429653101195 |
| - type: nauc_map_at_10_std |
| value: 7.674993664044225 |
| - type: nauc_map_at_1_diff1 |
| value: 92.97225739610897 |
| - type: nauc_map_at_1_max |
| value: 69.06680480411342 |
| - type: nauc_map_at_1_std |
| value: 8.97062021155494 |
| - type: nauc_map_at_20_diff1 |
| value: 93.21838651458631 |
| - type: nauc_map_at_20_max |
| value: 71.40429653101195 |
| - type: nauc_map_at_20_std |
| value: 7.674993664044225 |
| - type: nauc_map_at_3_diff1 |
| value: 93.22689075630251 |
| - type: nauc_map_at_3_max |
| value: 71.83278159748735 |
| - type: nauc_map_at_3_std |
| value: 5.838383838384187 |
| - type: nauc_map_at_5_diff1 |
| value: 93.07924446534143 |
| - type: nauc_map_at_5_max |
| value: 71.48845769375428 |
| - type: nauc_map_at_5_std |
| value: 6.123107648366922 |
| - type: nauc_mrr_at_1000_diff1 |
| value: 93.2086804630001 |
| - type: nauc_mrr_at_1000_max |
| value: 71.36336949003405 |
| - type: nauc_mrr_at_1000_std |
| value: 7.542855305507389 |
| - type: nauc_mrr_at_100_diff1 |
| value: 93.2086804630001 |
| - type: nauc_mrr_at_100_max |
| value: 71.36336949003405 |
| - type: nauc_mrr_at_100_std |
| value: 7.542855305507389 |
| - type: nauc_mrr_at_10_diff1 |
| value: 93.21838651458631 |
| - type: nauc_mrr_at_10_max |
| value: 71.40429653101195 |
| - type: nauc_mrr_at_10_std |
| value: 7.674993664044225 |
| - type: nauc_mrr_at_1_diff1 |
| value: 92.97225739610897 |
| - type: nauc_mrr_at_1_max |
| value: 69.06680480411342 |
| - type: nauc_mrr_at_1_std |
| value: 8.97062021155494 |
| - type: nauc_mrr_at_20_diff1 |
| value: 93.21838651458631 |
| - type: nauc_mrr_at_20_max |
| value: 71.40429653101195 |
| - type: nauc_mrr_at_20_std |
| value: 7.674993664044225 |
| - type: nauc_mrr_at_3_diff1 |
| value: 93.22689075630251 |
| - type: nauc_mrr_at_3_max |
| value: 71.83278159748735 |
| - type: nauc_mrr_at_3_std |
| value: 5.838383838384187 |
| - type: nauc_mrr_at_5_diff1 |
| value: 93.07924446534143 |
| - type: nauc_mrr_at_5_max |
| value: 71.48845769375428 |
| - type: nauc_mrr_at_5_std |
| value: 6.123107648366922 |
| - type: nauc_ndcg_at_1000_diff1 |
| value: 93.23304542841751 |
| - type: nauc_ndcg_at_1000_max |
| value: 71.63083509229389 |
| - type: nauc_ndcg_at_1000_std |
| value: 7.571755253398231 |
| - type: nauc_ndcg_at_100_diff1 |
| value: 93.23304542841751 |
| - type: nauc_ndcg_at_100_max |
| value: 71.63083509229389 |
| - type: nauc_ndcg_at_100_std |
| value: 7.571755253398231 |
| - type: nauc_ndcg_at_10_diff1 |
| value: 93.31367660323025 |
| - type: nauc_ndcg_at_10_max |
| value: 71.96886589636927 |
| - type: nauc_ndcg_at_10_std |
| value: 8.673077551486251 |
| - type: nauc_ndcg_at_1_diff1 |
| value: 92.97225739610897 |
| - type: nauc_ndcg_at_1_max |
| value: 69.06680480411342 |
| - type: nauc_ndcg_at_1_std |
| value: 8.97062021155494 |
| - type: nauc_ndcg_at_20_diff1 |
| value: 93.31367660323025 |
| - type: nauc_ndcg_at_20_max |
| value: 71.96886589636927 |
| - type: nauc_ndcg_at_20_std |
| value: 8.673077551486251 |
| - type: nauc_ndcg_at_3_diff1 |
| value: 93.23894522522178 |
| - type: nauc_ndcg_at_3_max |
| value: 72.902496295325 |
| - type: nauc_ndcg_at_3_std |
| value: 3.6178051090847845 |
| - type: nauc_ndcg_at_5_diff1 |
| value: 92.90972948390606 |
| - type: nauc_ndcg_at_5_max |
| value: 72.19526067100512 |
| - type: nauc_ndcg_at_5_std |
| value: 4.041582233975721 |
| - type: nauc_precision_at_1000_diff1 |
| value: .nan |
| - type: nauc_precision_at_1000_max |
| value: .nan |
| - type: nauc_precision_at_1000_std |
| value: .nan |
| - type: nauc_precision_at_100_diff1 |
| value: 100.0 |
| - type: nauc_precision_at_100_max |
| value: 100.0 |
| - type: nauc_precision_at_100_std |
| value: 100.0 |
| - type: nauc_precision_at_10_diff1 |
| value: 100.0 |
| - type: nauc_precision_at_10_max |
| value: 100.0 |
| - type: nauc_precision_at_10_std |
| value: 100.0 |
| - type: nauc_precision_at_1_diff1 |
| value: 92.97225739610897 |
| - type: nauc_precision_at_1_max |
| value: 69.06680480411342 |
| - type: nauc_precision_at_1_std |
| value: 8.97062021155494 |
| - type: nauc_precision_at_20_diff1 |
| value: 100.0 |
| - type: nauc_precision_at_20_max |
| value: 100.0 |
| - type: nauc_precision_at_20_std |
| value: 100.0 |
| - type: nauc_precision_at_3_diff1 |
| value: 93.2306255835672 |
| - type: nauc_precision_at_3_max |
| value: 82.46298519407813 |
| - type: nauc_precision_at_3_std |
| value: -17.583700146723764 |
| - type: nauc_precision_at_5_diff1 |
| value: 89.78758169934665 |
| - type: nauc_precision_at_5_max |
| value: 82.09033613445358 |
| - type: nauc_precision_at_5_std |
| value: -29.580999066296027 |
| - type: nauc_recall_at_1000_diff1 |
| value: .nan |
| - type: nauc_recall_at_1000_max |
| value: .nan |
| - type: nauc_recall_at_1000_std |
| value: .nan |
| - type: nauc_recall_at_100_diff1 |
| value: .nan |
| - type: nauc_recall_at_100_max |
| value: .nan |
| - type: nauc_recall_at_100_std |
| value: .nan |
| - type: nauc_recall_at_10_diff1 |
| value: 100.0 |
| - type: nauc_recall_at_10_max |
| value: 100.0 |
| - type: nauc_recall_at_10_std |
| value: 100.0 |
| - type: nauc_recall_at_1_diff1 |
| value: 92.97225739610897 |
| - type: nauc_recall_at_1_max |
| value: 69.06680480411342 |
| - type: nauc_recall_at_1_std |
| value: 8.97062021155494 |
| - type: nauc_recall_at_20_diff1 |
| value: 100.0 |
| - type: nauc_recall_at_20_max |
| value: 100.0 |
| - type: nauc_recall_at_20_std |
| value: 100.0 |
| - type: nauc_recall_at_3_diff1 |
| value: 93.2306255835672 |
| - type: nauc_recall_at_3_max |
| value: 82.4629851940786 |
| - type: nauc_recall_at_3_std |
| value: -17.5837001467247 |
| - type: nauc_recall_at_5_diff1 |
| value: 89.78758169934649 |
| - type: nauc_recall_at_5_max |
| value: 82.09033613445361 |
| - type: nauc_recall_at_5_std |
| value: -29.580999066293263 |
| - type: ndcg_at_1 |
| value: 92.7 |
| - type: ndcg_at_10 |
| value: 96.688 |
| - type: ndcg_at_100 |
| value: 96.72800000000001 |
| - type: ndcg_at_1000 |
| value: 96.72800000000001 |
| - type: ndcg_at_20 |
| value: 96.688 |
| - type: ndcg_at_3 |
| value: 96.23899999999999 |
| - type: ndcg_at_5 |
| value: 96.493 |
| - type: precision_at_1 |
| value: 92.7 |
| - type: precision_at_10 |
| value: 9.98 |
| - type: precision_at_100 |
| value: 1.0 |
| - type: precision_at_1000 |
| value: 0.1 |
| - type: precision_at_20 |
| value: 4.99 |
| - type: precision_at_3 |
| value: 32.867000000000004 |
| - type: precision_at_5 |
| value: 19.84 |
| - type: recall_at_1 |
| value: 92.7 |
| - type: recall_at_10 |
| value: 99.8 |
| - type: recall_at_100 |
| value: 100.0 |
| - type: recall_at_1000 |
| value: 100.0 |
| - type: recall_at_20 |
| value: 99.8 |
| - type: recall_at_3 |
| value: 98.6 |
| - type: recall_at_5 |
| value: 99.2 |
| task: |
| type: Retrieval |
| - dataset: |
| config: javascript |
| name: MTEB CodeSearchNetRetrieval (javascript) |
| revision: fdc6a9e39575768c27eb8a2a5f702bf846eb4759 |
| split: test |
| type: code-search-net/code_search_net |
| metrics: |
| - type: main_score |
| value: 89.97800000000001 |
| - type: map_at_1 |
| value: 85.0 |
| - type: map_at_10 |
| value: 88.427 |
| - type: map_at_100 |
| value: 88.627 |
| - type: map_at_1000 |
| value: 88.631 |
| - type: map_at_20 |
| value: 88.588 |
| - type: map_at_3 |
| value: 87.767 |
| - type: map_at_5 |
| value: 88.19200000000001 |
| - type: mrr_at_1 |
| value: 85.0 |
| - type: mrr_at_10 |
| value: 88.42742063492065 |
| - type: mrr_at_100 |
| value: 88.62702226900807 |
| - type: mrr_at_1000 |
| value: 88.63072320739236 |
| - type: mrr_at_20 |
| value: 88.58809151632684 |
| - type: mrr_at_3 |
| value: 87.76666666666667 |
| - type: mrr_at_5 |
| value: 88.1916666666667 |
| - type: nauc_map_at_1000_diff1 |
| value: 90.72581837961151 |
| - type: nauc_map_at_1000_max |
| value: 62.96996529181109 |
| - type: nauc_map_at_1000_std |
| value: 5.4686257905510445 |
| - type: nauc_map_at_100_diff1 |
| value: 90.7251318399043 |
| - type: nauc_map_at_100_max |
| value: 62.98186646957673 |
| - type: nauc_map_at_100_std |
| value: 5.499359385640601 |
| - type: nauc_map_at_10_diff1 |
| value: 90.7954839398183 |
| - type: nauc_map_at_10_max |
| value: 63.22150406130067 |
| - type: nauc_map_at_10_std |
| value: 5.413600198301859 |
| - type: nauc_map_at_1_diff1 |
| value: 91.52066384640422 |
| - type: nauc_map_at_1_max |
| value: 57.639765701269155 |
| - type: nauc_map_at_1_std |
| value: 1.4959323136999483 |
| - type: nauc_map_at_20_diff1 |
| value: 90.71434972102585 |
| - type: nauc_map_at_20_max |
| value: 63.07611119431469 |
| - type: nauc_map_at_20_std |
| value: 5.640274998494253 |
| - type: nauc_map_at_3_diff1 |
| value: 90.33141255622978 |
| - type: nauc_map_at_3_max |
| value: 62.34164710561914 |
| - type: nauc_map_at_3_std |
| value: 3.5068574882828925 |
| - type: nauc_map_at_5_diff1 |
| value: 90.79552786535547 |
| - type: nauc_map_at_5_max |
| value: 63.119944203478106 |
| - type: nauc_map_at_5_std |
| value: 4.953420352568366 |
| - type: nauc_mrr_at_1000_diff1 |
| value: 90.72581837961151 |
| - type: nauc_mrr_at_1000_max |
| value: 62.96996529181109 |
| - type: nauc_mrr_at_1000_std |
| value: 5.4686257905510445 |
| - type: nauc_mrr_at_100_diff1 |
| value: 90.7251318399043 |
| - type: nauc_mrr_at_100_max |
| value: 62.98186646957673 |
| - type: nauc_mrr_at_100_std |
| value: 5.499359385640601 |
| - type: nauc_mrr_at_10_diff1 |
| value: 90.7954839398183 |
| - type: nauc_mrr_at_10_max |
| value: 63.22150406130067 |
| - type: nauc_mrr_at_10_std |
| value: 5.413600198301859 |
| - type: nauc_mrr_at_1_diff1 |
| value: 91.52066384640422 |
| - type: nauc_mrr_at_1_max |
| value: 57.639765701269155 |
| - type: nauc_mrr_at_1_std |
| value: 1.4959323136999483 |
| - type: nauc_mrr_at_20_diff1 |
| value: 90.71434972102585 |
| - type: nauc_mrr_at_20_max |
| value: 63.07611119431469 |
| - type: nauc_mrr_at_20_std |
| value: 5.640274998494253 |
| - type: nauc_mrr_at_3_diff1 |
| value: 90.33141255622978 |
| - type: nauc_mrr_at_3_max |
| value: 62.34164710561914 |
| - type: nauc_mrr_at_3_std |
| value: 3.5068574882828925 |
| - type: nauc_mrr_at_5_diff1 |
| value: 90.79552786535547 |
| - type: nauc_mrr_at_5_max |
| value: 63.119944203478106 |
| - type: nauc_mrr_at_5_std |
| value: 4.953420352568366 |
| - type: nauc_ndcg_at_1000_diff1 |
| value: 90.6732163208453 |
| - type: nauc_ndcg_at_1000_max |
| value: 64.27542517763368 |
| - type: nauc_ndcg_at_1000_std |
| value: 7.423716824052952 |
| - type: nauc_ndcg_at_100_diff1 |
| value: 90.65475476530615 |
| - type: nauc_ndcg_at_100_max |
| value: 64.89222263529967 |
| - type: nauc_ndcg_at_100_std |
| value: 9.036733154889918 |
| - type: nauc_ndcg_at_10_diff1 |
| value: 90.88391240350151 |
| - type: nauc_ndcg_at_10_max |
| value: 66.10798661293683 |
| - type: nauc_ndcg_at_10_std |
| value: 8.662637793277094 |
| - type: nauc_ndcg_at_1_diff1 |
| value: 91.52066384640422 |
| - type: nauc_ndcg_at_1_max |
| value: 57.639765701269155 |
| - type: nauc_ndcg_at_1_std |
| value: 1.4959323136999483 |
| - type: nauc_ndcg_at_20_diff1 |
| value: 90.55301735430407 |
| - type: nauc_ndcg_at_20_max |
| value: 65.65827895142017 |
| - type: nauc_ndcg_at_20_std |
| value: 9.990699968426656 |
| - type: nauc_ndcg_at_3_diff1 |
| value: 89.96356996211253 |
| - type: nauc_ndcg_at_3_max |
| value: 64.03584890384899 |
| - type: nauc_ndcg_at_3_std |
| value: 4.295929350921648 |
| - type: nauc_ndcg_at_5_diff1 |
| value: 90.89636835182858 |
| - type: nauc_ndcg_at_5_max |
| value: 65.69109133325729 |
| - type: nauc_ndcg_at_5_std |
| value: 7.251888023539761 |
| - type: nauc_precision_at_1000_diff1 |
| value: .nan |
| - type: nauc_precision_at_1000_max |
| value: .nan |
| - type: nauc_precision_at_1000_std |
| value: .nan |
| - type: nauc_precision_at_100_diff1 |
| value: 89.89595838335227 |
| - type: nauc_precision_at_100_max |
| value: 98.13258636787991 |
| - type: nauc_precision_at_100_std |
| value: 95.88168600773692 |
| - type: nauc_precision_at_10_diff1 |
| value: 91.80313150901407 |
| - type: nauc_precision_at_10_max |
| value: 84.73120017237689 |
| - type: nauc_precision_at_10_std |
| value: 30.963154492565963 |
| - type: nauc_precision_at_1_diff1 |
| value: 91.52066384640422 |
| - type: nauc_precision_at_1_max |
| value: 57.639765701269155 |
| - type: nauc_precision_at_1_std |
| value: 1.4959323136999483 |
| - type: nauc_precision_at_20_diff1 |
| value: 88.51379632312695 |
| - type: nauc_precision_at_20_max |
| value: 93.89870890885065 |
| - type: nauc_precision_at_20_std |
| value: 66.06458675424169 |
| - type: nauc_precision_at_3_diff1 |
| value: 88.5919911035577 |
| - type: nauc_precision_at_3_max |
| value: 70.47755431760744 |
| - type: nauc_precision_at_3_std |
| value: 7.324877516549845 |
| - type: nauc_precision_at_5_diff1 |
| value: 91.68000533546777 |
| - type: nauc_precision_at_5_max |
| value: 78.08323329331722 |
| - type: nauc_precision_at_5_std |
| value: 18.784847272242132 |
| - type: nauc_recall_at_1000_diff1 |
| value: .nan |
| - type: nauc_recall_at_1000_max |
| value: .nan |
| - type: nauc_recall_at_1000_std |
| value: .nan |
| - type: nauc_recall_at_100_diff1 |
| value: 89.8959583833538 |
| - type: nauc_recall_at_100_max |
| value: 98.13258636788038 |
| - type: nauc_recall_at_100_std |
| value: 95.88168600773625 |
| - type: nauc_recall_at_10_diff1 |
| value: 91.80313150901372 |
| - type: nauc_recall_at_10_max |
| value: 84.73120017237645 |
| - type: nauc_recall_at_10_std |
| value: 30.963154492566204 |
| - type: nauc_recall_at_1_diff1 |
| value: 91.52066384640422 |
| - type: nauc_recall_at_1_max |
| value: 57.639765701269155 |
| - type: nauc_recall_at_1_std |
| value: 1.4959323136999483 |
| - type: nauc_recall_at_20_diff1 |
| value: 88.51379632312701 |
| - type: nauc_recall_at_20_max |
| value: 93.89870890885045 |
| - type: nauc_recall_at_20_std |
| value: 66.0645867542416 |
| - type: nauc_recall_at_3_diff1 |
| value: 88.59199110355753 |
| - type: nauc_recall_at_3_max |
| value: 70.47755431760724 |
| - type: nauc_recall_at_3_std |
| value: 7.324877516549637 |
| - type: nauc_recall_at_5_diff1 |
| value: 91.68000533546758 |
| - type: nauc_recall_at_5_max |
| value: 78.08323329331743 |
| - type: nauc_recall_at_5_std |
| value: 18.784847272242207 |
| - type: ndcg_at_1 |
| value: 85.0 |
| - type: ndcg_at_10 |
| value: 89.97800000000001 |
| - type: ndcg_at_100 |
| value: 90.842 |
| - type: ndcg_at_1000 |
| value: 91.00399999999999 |
| - type: ndcg_at_20 |
| value: 90.561 |
| - type: ndcg_at_3 |
| value: 88.626 |
| - type: ndcg_at_5 |
| value: 89.401 |
| - type: precision_at_1 |
| value: 85.0 |
| - type: precision_at_10 |
| value: 9.48 |
| - type: precision_at_100 |
| value: 0.9860000000000001 |
| - type: precision_at_1000 |
| value: 0.1 |
| - type: precision_at_20 |
| value: 4.855 |
| - type: precision_at_3 |
| value: 30.367 |
| - type: precision_at_5 |
| value: 18.6 |
| - type: recall_at_1 |
| value: 85.0 |
| - type: recall_at_10 |
| value: 94.8 |
| - type: recall_at_100 |
| value: 98.6 |
| - type: recall_at_1000 |
| value: 100.0 |
| - type: recall_at_20 |
| value: 97.1 |
| - type: recall_at_3 |
| value: 91.10000000000001 |
| - type: recall_at_5 |
| value: 93.0 |
| task: |
| type: Retrieval |
| - dataset: |
| config: go |
| name: MTEB CodeSearchNetRetrieval (go) |
| revision: fdc6a9e39575768c27eb8a2a5f702bf846eb4759 |
| split: test |
| type: code-search-net/code_search_net |
| metrics: |
| - type: main_score |
| value: 97.505 |
| - type: map_at_1 |
| value: 94.39999999999999 |
| - type: map_at_10 |
| value: 96.808 |
| - type: map_at_100 |
| value: 96.82300000000001 |
| - type: map_at_1000 |
| value: 96.824 |
| - type: map_at_20 |
| value: 96.813 |
| - type: map_at_3 |
| value: 96.733 |
| - type: map_at_5 |
| value: 96.808 |
| - type: mrr_at_1 |
| value: 94.39999999999999 |
| - type: mrr_at_10 |
| value: 96.80833333333334 |
| - type: mrr_at_100 |
| value: 96.82307275541795 |
| - type: mrr_at_1000 |
| value: 96.82390608875127 |
| - type: mrr_at_20 |
| value: 96.81333333333333 |
| - type: mrr_at_3 |
| value: 96.73333333333333 |
| - type: mrr_at_5 |
| value: 96.80833333333334 |
| - type: nauc_map_at_1000_diff1 |
| value: 97.27333921447281 |
| - type: nauc_map_at_1000_max |
| value: 40.53177582941515 |
| - type: nauc_map_at_1000_std |
| value: -9.68497853001627 |
| - type: nauc_map_at_100_diff1 |
| value: 97.27405443932574 |
| - type: nauc_map_at_100_max |
| value: 40.54737481852009 |
| - type: nauc_map_at_100_std |
| value: -9.663493618536894 |
| - type: nauc_map_at_10_diff1 |
| value: 97.28664311677696 |
| - type: nauc_map_at_10_max |
| value: 40.81115474910615 |
| - type: nauc_map_at_10_std |
| value: -10.076963770712538 |
| - type: nauc_map_at_1_diff1 |
| value: 96.90709617180208 |
| - type: nauc_map_at_1_max |
| value: 33.33833533413366 |
| - type: nauc_map_at_1_std |
| value: -6.367547018807825 |
| - type: nauc_map_at_20_diff1 |
| value: 97.28238575765059 |
| - type: nauc_map_at_20_max |
| value: 40.71828522203857 |
| - type: nauc_map_at_20_std |
| value: -9.912064546879208 |
| - type: nauc_map_at_3_diff1 |
| value: 97.34893957583056 |
| - type: nauc_map_at_3_max |
| value: 41.67881438289635 |
| - type: nauc_map_at_3_std |
| value: -9.509041711922595 |
| - type: nauc_map_at_5_diff1 |
| value: 97.28664311677696 |
| - type: nauc_map_at_5_max |
| value: 40.81115474910615 |
| - type: nauc_map_at_5_std |
| value: -10.076963770712538 |
| - type: nauc_mrr_at_1000_diff1 |
| value: 97.27333921447281 |
| - type: nauc_mrr_at_1000_max |
| value: 40.53177582941515 |
| - type: nauc_mrr_at_1000_std |
| value: -9.68497853001627 |
| - type: nauc_mrr_at_100_diff1 |
| value: 97.27405443932574 |
| - type: nauc_mrr_at_100_max |
| value: 40.54737481852009 |
| - type: nauc_mrr_at_100_std |
| value: -9.663493618536894 |
| - type: nauc_mrr_at_10_diff1 |
| value: 97.28664311677696 |
| - type: nauc_mrr_at_10_max |
| value: 40.81115474910615 |
| - type: nauc_mrr_at_10_std |
| value: -10.076963770712538 |
| - type: nauc_mrr_at_1_diff1 |
| value: 96.90709617180208 |
| - type: nauc_mrr_at_1_max |
| value: 33.33833533413366 |
| - type: nauc_mrr_at_1_std |
| value: -6.367547018807825 |
| - type: nauc_mrr_at_20_diff1 |
| value: 97.28238575765059 |
| - type: nauc_mrr_at_20_max |
| value: 40.71828522203857 |
| - type: nauc_mrr_at_20_std |
| value: -9.912064546879208 |
| - type: nauc_mrr_at_3_diff1 |
| value: 97.34893957583056 |
| - type: nauc_mrr_at_3_max |
| value: 41.67881438289635 |
| - type: nauc_mrr_at_3_std |
| value: -9.509041711922595 |
| - type: nauc_mrr_at_5_diff1 |
| value: 97.28664311677696 |
| - type: nauc_mrr_at_5_max |
| value: 40.81115474910615 |
| - type: nauc_mrr_at_5_std |
| value: -10.076963770712538 |
| - type: nauc_ndcg_at_1000_diff1 |
| value: 97.33376561125955 |
| - type: nauc_ndcg_at_1000_max |
| value: 41.75737985174807 |
| - type: nauc_ndcg_at_1000_std |
| value: -10.511570027083764 |
| - type: nauc_ndcg_at_100_diff1 |
| value: 97.34974235237087 |
| - type: nauc_ndcg_at_100_max |
| value: 42.106384150737576 |
| - type: nauc_ndcg_at_100_std |
| value: -10.015808519562162 |
| - type: nauc_ndcg_at_10_diff1 |
| value: 97.43761675418024 |
| - type: nauc_ndcg_at_10_max |
| value: 43.92682037582593 |
| - type: nauc_ndcg_at_10_std |
| value: -13.11593389665566 |
| - type: nauc_ndcg_at_1_diff1 |
| value: 96.90709617180208 |
| - type: nauc_ndcg_at_1_max |
| value: 33.33833533413366 |
| - type: nauc_ndcg_at_1_std |
| value: -6.367547018807825 |
| - type: nauc_ndcg_at_20_diff1 |
| value: 97.41401672160926 |
| - type: nauc_ndcg_at_20_max |
| value: 43.410375824598866 |
| - type: nauc_ndcg_at_20_std |
| value: -12.175971519065602 |
| - type: nauc_ndcg_at_3_diff1 |
| value: 97.56379014982639 |
| - type: nauc_ndcg_at_3_max |
| value: 45.63427566353211 |
| - type: nauc_ndcg_at_3_std |
| value: -11.748268623706998 |
| - type: nauc_ndcg_at_5_diff1 |
| value: 97.43761675418024 |
| - type: nauc_ndcg_at_5_max |
| value: 43.92682037582593 |
| - type: nauc_ndcg_at_5_std |
| value: -13.11593389665566 |
| - type: nauc_precision_at_1000_diff1 |
| value: .nan |
| - type: nauc_precision_at_1000_max |
| value: .nan |
| - type: nauc_precision_at_1000_std |
| value: .nan |
| - type: nauc_precision_at_100_diff1 |
| value: 100.0 |
| - type: nauc_precision_at_100_max |
| value: 100.0 |
| - type: nauc_precision_at_100_std |
| value: 72.22222222222335 |
| - type: nauc_precision_at_10_diff1 |
| value: 100.0 |
| - type: nauc_precision_at_10_max |
| value: 97.38562091503177 |
| - type: nauc_precision_at_10_std |
| value: -69.85060690943635 |
| - type: nauc_precision_at_1_diff1 |
| value: 96.90709617180208 |
| - type: nauc_precision_at_1_max |
| value: 33.33833533413366 |
| - type: nauc_precision_at_1_std |
| value: -6.367547018807825 |
| - type: nauc_precision_at_20_diff1 |
| value: 100.0 |
| - type: nauc_precision_at_20_max |
| value: 96.73202614378965 |
| - type: nauc_precision_at_20_std |
| value: -58.52007469655789 |
| - type: nauc_precision_at_3_diff1 |
| value: 100.0 |
| - type: nauc_precision_at_3_max |
| value: 90.34197012138252 |
| - type: nauc_precision_at_3_std |
| value: -38.159430438841106 |
| - type: nauc_precision_at_5_diff1 |
| value: 100.0 |
| - type: nauc_precision_at_5_max |
| value: 97.38562091503177 |
| - type: nauc_precision_at_5_std |
| value: -69.85060690943635 |
| - type: nauc_recall_at_1000_diff1 |
| value: .nan |
| - type: nauc_recall_at_1000_max |
| value: .nan |
| - type: nauc_recall_at_1000_std |
| value: .nan |
| - type: nauc_recall_at_100_diff1 |
| value: 100.0 |
| - type: nauc_recall_at_100_max |
| value: 100.0 |
| - type: nauc_recall_at_100_std |
| value: 72.22222222222368 |
| - type: nauc_recall_at_10_diff1 |
| value: 100.0 |
| - type: nauc_recall_at_10_max |
| value: 97.38562091503128 |
| - type: nauc_recall_at_10_std |
| value: -69.85060690942723 |
| - type: nauc_recall_at_1_diff1 |
| value: 96.90709617180208 |
| - type: nauc_recall_at_1_max |
| value: 33.33833533413366 |
| - type: nauc_recall_at_1_std |
| value: -6.367547018807825 |
| - type: nauc_recall_at_20_diff1 |
| value: 100.0 |
| - type: nauc_recall_at_20_max |
| value: 96.7320261437891 |
| - type: nauc_recall_at_20_std |
| value: -58.52007469654495 |
| - type: nauc_recall_at_3_diff1 |
| value: 100.0 |
| - type: nauc_recall_at_3_max |
| value: 90.34197012138186 |
| - type: nauc_recall_at_3_std |
| value: -38.15943043884293 |
| - type: nauc_recall_at_5_diff1 |
| value: 100.0 |
| - type: nauc_recall_at_5_max |
| value: 97.38562091503128 |
| - type: nauc_recall_at_5_std |
| value: -69.85060690942723 |
| - type: ndcg_at_1 |
| value: 94.39999999999999 |
| - type: ndcg_at_10 |
| value: 97.505 |
| - type: ndcg_at_100 |
| value: 97.588 |
| - type: ndcg_at_1000 |
| value: 97.602 |
| - type: ndcg_at_20 |
| value: 97.528 |
| - type: ndcg_at_3 |
| value: 97.37599999999999 |
| - type: ndcg_at_5 |
| value: 97.505 |
| - type: precision_at_1 |
| value: 94.39999999999999 |
| - type: precision_at_10 |
| value: 9.950000000000001 |
| - type: precision_at_100 |
| value: 0.9990000000000001 |
| - type: precision_at_1000 |
| value: 0.1 |
| - type: precision_at_20 |
| value: 4.9799999999999995 |
| - type: precision_at_3 |
| value: 33.067 |
| - type: precision_at_5 |
| value: 19.900000000000002 |
| - type: recall_at_1 |
| value: 94.39999999999999 |
| - type: recall_at_10 |
| value: 99.5 |
| - type: recall_at_100 |
| value: 99.9 |
| - type: recall_at_1000 |
| value: 100.0 |
| - type: recall_at_20 |
| value: 99.6 |
| - type: recall_at_3 |
| value: 99.2 |
| - type: recall_at_5 |
| value: 99.5 |
| task: |
| type: Retrieval |
| - dataset: |
| config: ruby |
| name: MTEB CodeSearchNetRetrieval (ruby) |
| revision: fdc6a9e39575768c27eb8a2a5f702bf846eb4759 |
| split: test |
| type: code-search-net/code_search_net |
| metrics: |
| - type: main_score |
| value: 92.83200000000001 |
| - type: map_at_1 |
| value: 87.7 |
| - type: map_at_10 |
| value: 91.496 |
| - type: map_at_100 |
| value: 91.582 |
| - type: map_at_1000 |
| value: 91.58699999999999 |
| - type: map_at_20 |
| value: 91.55 |
| - type: map_at_3 |
| value: 91.167 |
| - type: map_at_5 |
| value: 91.39699999999999 |
| - type: mrr_at_1 |
| value: 87.7 |
| - type: mrr_at_10 |
| value: 91.49646825396826 |
| - type: mrr_at_100 |
| value: 91.58154350757961 |
| - type: mrr_at_1000 |
| value: 91.58711452954685 |
| - type: mrr_at_20 |
| value: 91.54990637415563 |
| - type: mrr_at_3 |
| value: 91.16666666666669 |
| - type: mrr_at_5 |
| value: 91.39666666666668 |
| - type: nauc_map_at_1000_diff1 |
| value: 90.01079432013319 |
| - type: nauc_map_at_1000_max |
| value: 76.25110199586136 |
| - type: nauc_map_at_1000_std |
| value: 21.95406993804367 |
| - type: nauc_map_at_100_diff1 |
| value: 90.01415529036015 |
| - type: nauc_map_at_100_max |
| value: 76.26710387746687 |
| - type: nauc_map_at_100_std |
| value: 21.991489847497782 |
| - type: nauc_map_at_10_diff1 |
| value: 89.88101575524061 |
| - type: nauc_map_at_10_max |
| value: 76.40985435650425 |
| - type: nauc_map_at_10_std |
| value: 22.554999001780367 |
| - type: nauc_map_at_1_diff1 |
| value: 90.45505123995243 |
| - type: nauc_map_at_1_max |
| value: 70.27343537229697 |
| - type: nauc_map_at_1_std |
| value: 10.378716550076664 |
| - type: nauc_map_at_20_diff1 |
| value: 90.03751820546869 |
| - type: nauc_map_at_20_max |
| value: 76.35756916603282 |
| - type: nauc_map_at_20_std |
| value: 22.15447723869247 |
| - type: nauc_map_at_3_diff1 |
| value: 89.90841801530556 |
| - type: nauc_map_at_3_max |
| value: 76.90028137713493 |
| - type: nauc_map_at_3_std |
| value: 21.843737118482988 |
| - type: nauc_map_at_5_diff1 |
| value: 89.93455699311205 |
| - type: nauc_map_at_5_max |
| value: 76.52511956217161 |
| - type: nauc_map_at_5_std |
| value: 22.235121325826523 |
| - type: nauc_mrr_at_1000_diff1 |
| value: 90.01079432013319 |
| - type: nauc_mrr_at_1000_max |
| value: 76.25110199586136 |
| - type: nauc_mrr_at_1000_std |
| value: 21.95406993804367 |
| - type: nauc_mrr_at_100_diff1 |
| value: 90.01415529036015 |
| - type: nauc_mrr_at_100_max |
| value: 76.26710387746687 |
| - type: nauc_mrr_at_100_std |
| value: 21.991489847497782 |
| - type: nauc_mrr_at_10_diff1 |
| value: 89.88101575524061 |
| - type: nauc_mrr_at_10_max |
| value: 76.40985435650425 |
| - type: nauc_mrr_at_10_std |
| value: 22.554999001780367 |
| - type: nauc_mrr_at_1_diff1 |
| value: 90.45505123995243 |
| - type: nauc_mrr_at_1_max |
| value: 70.27343537229697 |
| - type: nauc_mrr_at_1_std |
| value: 10.378716550076664 |
| - type: nauc_mrr_at_20_diff1 |
| value: 90.03751820546869 |
| - type: nauc_mrr_at_20_max |
| value: 76.35756916603282 |
| - type: nauc_mrr_at_20_std |
| value: 22.15447723869247 |
| - type: nauc_mrr_at_3_diff1 |
| value: 89.90841801530556 |
| - type: nauc_mrr_at_3_max |
| value: 76.90028137713493 |
| - type: nauc_mrr_at_3_std |
| value: 21.843737118482988 |
| - type: nauc_mrr_at_5_diff1 |
| value: 89.93455699311205 |
| - type: nauc_mrr_at_5_max |
| value: 76.52511956217161 |
| - type: nauc_mrr_at_5_std |
| value: 22.235121325826523 |
| - type: nauc_ndcg_at_1000_diff1 |
| value: 90.0388376789719 |
| - type: nauc_ndcg_at_1000_max |
| value: 77.4286773752732 |
| - type: nauc_ndcg_at_1000_std |
| value: 24.905596869880494 |
| - type: nauc_ndcg_at_100_diff1 |
| value: 90.1570526184995 |
| - type: nauc_ndcg_at_100_max |
| value: 77.97459201928656 |
| - type: nauc_ndcg_at_100_std |
| value: 26.36432448776628 |
| - type: nauc_ndcg_at_10_diff1 |
| value: 89.60631662977974 |
| - type: nauc_ndcg_at_10_max |
| value: 78.82415924492551 |
| - type: nauc_ndcg_at_10_std |
| value: 29.259897960486477 |
| - type: nauc_ndcg_at_1_diff1 |
| value: 90.45505123995243 |
| - type: nauc_ndcg_at_1_max |
| value: 70.27343537229697 |
| - type: nauc_ndcg_at_1_std |
| value: 10.378716550076664 |
| - type: nauc_ndcg_at_20_diff1 |
| value: 90.34712330725388 |
| - type: nauc_ndcg_at_20_max |
| value: 78.65263241067265 |
| - type: nauc_ndcg_at_20_std |
| value: 27.634547611895844 |
| - type: nauc_ndcg_at_3_diff1 |
| value: 89.69525439370882 |
| - type: nauc_ndcg_at_3_max |
| value: 79.68227200895333 |
| - type: nauc_ndcg_at_3_std |
| value: 27.16978363425553 |
| - type: nauc_ndcg_at_5_diff1 |
| value: 89.74542043203509 |
| - type: nauc_ndcg_at_5_max |
| value: 79.0789328591088 |
| - type: nauc_ndcg_at_5_std |
| value: 28.35584817342069 |
| - type: nauc_precision_at_1000_diff1 |
| value: .nan |
| - type: nauc_precision_at_1000_max |
| value: .nan |
| - type: nauc_precision_at_1000_std |
| value: .nan |
| - type: nauc_precision_at_100_diff1 |
| value: 94.84665661136168 |
| - type: nauc_precision_at_100_max |
| value: 100.0 |
| - type: nauc_precision_at_100_std |
| value: 86.96760755584259 |
| - type: nauc_precision_at_10_diff1 |
| value: 87.527719421102 |
| - type: nauc_precision_at_10_max |
| value: 96.1922268907559 |
| - type: nauc_precision_at_10_std |
| value: 80.32942343604061 |
| - type: nauc_precision_at_1_diff1 |
| value: 90.45505123995243 |
| - type: nauc_precision_at_1_max |
| value: 70.27343537229697 |
| - type: nauc_precision_at_1_std |
| value: 10.378716550076664 |
| - type: nauc_precision_at_20_diff1 |
| value: 95.5746187363837 |
| - type: nauc_precision_at_20_max |
| value: 100.0 |
| - type: nauc_precision_at_20_std |
| value: 78.69397759103582 |
| - type: nauc_precision_at_3_diff1 |
| value: 88.63450141961569 |
| - type: nauc_precision_at_3_max |
| value: 93.64984088873672 |
| - type: nauc_precision_at_3_std |
| value: 54.208350006669704 |
| - type: nauc_precision_at_5_diff1 |
| value: 88.62314156431772 |
| - type: nauc_precision_at_5_max |
| value: 94.5892887069356 |
| - type: nauc_precision_at_5_std |
| value: 66.80552562905466 |
| - type: nauc_recall_at_1000_diff1 |
| value: .nan |
| - type: nauc_recall_at_1000_max |
| value: .nan |
| - type: nauc_recall_at_1000_std |
| value: .nan |
| - type: nauc_recall_at_100_diff1 |
| value: 94.8466566113629 |
| - type: nauc_recall_at_100_max |
| value: 100.0 |
| - type: nauc_recall_at_100_std |
| value: 86.96760755584259 |
| - type: nauc_recall_at_10_diff1 |
| value: 87.52771942110216 |
| - type: nauc_recall_at_10_max |
| value: 96.19222689075643 |
| - type: nauc_recall_at_10_std |
| value: 80.32942343604144 |
| - type: nauc_recall_at_1_diff1 |
| value: 90.45505123995243 |
| - type: nauc_recall_at_1_max |
| value: 70.27343537229697 |
| - type: nauc_recall_at_1_std |
| value: 10.378716550076664 |
| - type: nauc_recall_at_20_diff1 |
| value: 95.57461873638331 |
| - type: nauc_recall_at_20_max |
| value: 100.0 |
| - type: nauc_recall_at_20_std |
| value: 78.69397759103617 |
| - type: nauc_recall_at_3_diff1 |
| value: 88.63450141961552 |
| - type: nauc_recall_at_3_max |
| value: 93.64984088873658 |
| - type: nauc_recall_at_3_std |
| value: 54.20835000666945 |
| - type: nauc_recall_at_5_diff1 |
| value: 88.62314156431815 |
| - type: nauc_recall_at_5_max |
| value: 94.58928870693578 |
| - type: nauc_recall_at_5_std |
| value: 66.80552562905518 |
| - type: ndcg_at_1 |
| value: 87.7 |
| - type: ndcg_at_10 |
| value: 92.83200000000001 |
| - type: ndcg_at_100 |
| value: 93.242 |
| - type: ndcg_at_1000 |
| value: 93.403 |
| - type: ndcg_at_20 |
| value: 93.03099999999999 |
| - type: ndcg_at_3 |
| value: 92.186 |
| - type: ndcg_at_5 |
| value: 92.599 |
| - type: precision_at_1 |
| value: 87.7 |
| - type: precision_at_10 |
| value: 9.68 |
| - type: precision_at_100 |
| value: 0.987 |
| - type: precision_at_1000 |
| value: 0.1 |
| - type: precision_at_20 |
| value: 4.88 |
| - type: precision_at_3 |
| value: 31.7 |
| - type: precision_at_5 |
| value: 19.220000000000002 |
| - type: recall_at_1 |
| value: 87.7 |
| - type: recall_at_10 |
| value: 96.8 |
| - type: recall_at_100 |
| value: 98.7 |
| - type: recall_at_1000 |
| value: 100.0 |
| - type: recall_at_20 |
| value: 97.6 |
| - type: recall_at_3 |
| value: 95.1 |
| - type: recall_at_5 |
| value: 96.1 |
| task: |
| type: Retrieval |
| - dataset: |
| config: java |
| name: MTEB CodeSearchNetRetrieval (java) |
| revision: fdc6a9e39575768c27eb8a2a5f702bf846eb4759 |
| split: test |
| type: code-search-net/code_search_net |
| metrics: |
| - type: main_score |
| value: 94.50699999999999 |
| - type: map_at_1 |
| value: 91.4 |
| - type: map_at_10 |
| value: 93.67699999999999 |
| - type: map_at_100 |
| value: 93.746 |
| - type: map_at_1000 |
| value: 93.753 |
| - type: map_at_20 |
| value: 93.706 |
| - type: map_at_3 |
| value: 93.43299999999999 |
| - type: map_at_5 |
| value: 93.593 |
| - type: mrr_at_1 |
| value: 91.4 |
| - type: mrr_at_10 |
| value: 93.67706349206348 |
| - type: mrr_at_100 |
| value: 93.74605961033865 |
| - type: mrr_at_1000 |
| value: 93.75307690063596 |
| - type: mrr_at_20 |
| value: 93.70575802250723 |
| - type: mrr_at_3 |
| value: 93.43333333333334 |
| - type: mrr_at_5 |
| value: 93.59333333333333 |
| - type: nauc_map_at_1000_diff1 |
| value: 91.96988867062096 |
| - type: nauc_map_at_1000_max |
| value: 80.8753031650805 |
| - type: nauc_map_at_1000_std |
| value: 54.38722696006999 |
| - type: nauc_map_at_100_diff1 |
| value: 91.97889892956285 |
| - type: nauc_map_at_100_max |
| value: 80.89676220389073 |
| - type: nauc_map_at_100_std |
| value: 54.4132005408842 |
| - type: nauc_map_at_10_diff1 |
| value: 92.00847920033857 |
| - type: nauc_map_at_10_max |
| value: 81.10521744502199 |
| - type: nauc_map_at_10_std |
| value: 54.68790929661031 |
| - type: nauc_map_at_1_diff1 |
| value: 89.60328317373455 |
| - type: nauc_map_at_1_max |
| value: 72.47899159663876 |
| - type: nauc_map_at_1_std |
| value: 44.26584587323289 |
| - type: nauc_map_at_20_diff1 |
| value: 92.02464449562623 |
| - type: nauc_map_at_20_max |
| value: 81.01907888930855 |
| - type: nauc_map_at_20_std |
| value: 54.571031073682384 |
| - type: nauc_map_at_3_diff1 |
| value: 92.09358870451744 |
| - type: nauc_map_at_3_max |
| value: 81.7007682937812 |
| - type: nauc_map_at_3_std |
| value: 55.27046690080435 |
| - type: nauc_map_at_5_diff1 |
| value: 92.11292217199042 |
| - type: nauc_map_at_5_max |
| value: 81.3521575817287 |
| - type: nauc_map_at_5_std |
| value: 54.99813695856418 |
| - type: nauc_mrr_at_1000_diff1 |
| value: 91.96988867062096 |
| - type: nauc_mrr_at_1000_max |
| value: 80.8753031650805 |
| - type: nauc_mrr_at_1000_std |
| value: 54.38722696006999 |
| - type: nauc_mrr_at_100_diff1 |
| value: 91.97889892956285 |
| - type: nauc_mrr_at_100_max |
| value: 80.89676220389073 |
| - type: nauc_mrr_at_100_std |
| value: 54.4132005408842 |
| - type: nauc_mrr_at_10_diff1 |
| value: 92.00847920033857 |
| - type: nauc_mrr_at_10_max |
| value: 81.10521744502199 |
| - type: nauc_mrr_at_10_std |
| value: 54.68790929661031 |
| - type: nauc_mrr_at_1_diff1 |
| value: 89.60328317373455 |
| - type: nauc_mrr_at_1_max |
| value: 72.47899159663876 |
| - type: nauc_mrr_at_1_std |
| value: 44.26584587323289 |
| - type: nauc_mrr_at_20_diff1 |
| value: 92.02464449562623 |
| - type: nauc_mrr_at_20_max |
| value: 81.01907888930855 |
| - type: nauc_mrr_at_20_std |
| value: 54.571031073682384 |
| - type: nauc_mrr_at_3_diff1 |
| value: 92.09358870451744 |
| - type: nauc_mrr_at_3_max |
| value: 81.7007682937812 |
| - type: nauc_mrr_at_3_std |
| value: 55.27046690080435 |
| - type: nauc_mrr_at_5_diff1 |
| value: 92.11292217199042 |
| - type: nauc_mrr_at_5_max |
| value: 81.3521575817287 |
| - type: nauc_mrr_at_5_std |
| value: 54.99813695856418 |
| - type: nauc_ndcg_at_1000_diff1 |
| value: 92.39514411202762 |
| - type: nauc_ndcg_at_1000_max |
| value: 82.1789464930351 |
| - type: nauc_ndcg_at_1000_std |
| value: 55.98331155582365 |
| - type: nauc_ndcg_at_100_diff1 |
| value: 92.60315730234768 |
| - type: nauc_ndcg_at_100_max |
| value: 82.66640006867893 |
| - type: nauc_ndcg_at_100_std |
| value: 56.60969071497668 |
| - type: nauc_ndcg_at_10_diff1 |
| value: 92.84846723213055 |
| - type: nauc_ndcg_at_10_max |
| value: 83.93273402590029 |
| - type: nauc_ndcg_at_10_std |
| value: 58.180048945638184 |
| - type: nauc_ndcg_at_1_diff1 |
| value: 89.60328317373455 |
| - type: nauc_ndcg_at_1_max |
| value: 72.47899159663876 |
| - type: nauc_ndcg_at_1_std |
| value: 44.26584587323289 |
| - type: nauc_ndcg_at_20_diff1 |
| value: 92.9455259704201 |
| - type: nauc_ndcg_at_20_max |
| value: 83.57732205562041 |
| - type: nauc_ndcg_at_20_std |
| value: 57.695601177913225 |
| - type: nauc_ndcg_at_3_diff1 |
| value: 93.02972756100402 |
| - type: nauc_ndcg_at_3_max |
| value: 85.03899149427752 |
| - type: nauc_ndcg_at_3_std |
| value: 59.29245146091644 |
| - type: nauc_ndcg_at_5_diff1 |
| value: 93.09702309051526 |
| - type: nauc_ndcg_at_5_max |
| value: 84.49116159880194 |
| - type: nauc_ndcg_at_5_std |
| value: 58.887701004976 |
| - type: nauc_precision_at_1000_diff1 |
| value: .nan |
| - type: nauc_precision_at_1000_max |
| value: .nan |
| - type: nauc_precision_at_1000_std |
| value: .nan |
| - type: nauc_precision_at_100_diff1 |
| value: 100.0 |
| - type: nauc_precision_at_100_max |
| value: 100.0 |
| - type: nauc_precision_at_100_std |
| value: 79.24836601307065 |
| - type: nauc_precision_at_10_diff1 |
| value: 97.6423902894492 |
| - type: nauc_precision_at_10_max |
| value: 100.0 |
| - type: nauc_precision_at_10_std |
| value: 78.07345160286333 |
| - type: nauc_precision_at_1_diff1 |
| value: 89.60328317373455 |
| - type: nauc_precision_at_1_max |
| value: 72.47899159663876 |
| - type: nauc_precision_at_1_std |
| value: 44.26584587323289 |
| - type: nauc_precision_at_20_diff1 |
| value: 99.47712418300702 |
| - type: nauc_precision_at_20_max |
| value: 100.0 |
| - type: nauc_precision_at_20_std |
| value: 77.69374416433266 |
| - type: nauc_precision_at_3_diff1 |
| value: 97.06316635181199 |
| - type: nauc_precision_at_3_max |
| value: 99.35400516795852 |
| - type: nauc_precision_at_3_std |
| value: 76.56070180010008 |
| - type: nauc_precision_at_5_diff1 |
| value: 98.03532524120767 |
| - type: nauc_precision_at_5_max |
| value: 100.0 |
| - type: nauc_precision_at_5_std |
| value: 78.16163502438044 |
| - type: nauc_recall_at_1000_diff1 |
| value: .nan |
| - type: nauc_recall_at_1000_max |
| value: .nan |
| - type: nauc_recall_at_1000_std |
| value: .nan |
| - type: nauc_recall_at_100_diff1 |
| value: 100.0 |
| - type: nauc_recall_at_100_max |
| value: 100.0 |
| - type: nauc_recall_at_100_std |
| value: 79.24836601307193 |
| - type: nauc_recall_at_10_diff1 |
| value: 97.64239028944921 |
| - type: nauc_recall_at_10_max |
| value: 100.0 |
| - type: nauc_recall_at_10_std |
| value: 78.07345160286356 |
| - type: nauc_recall_at_1_diff1 |
| value: 89.60328317373455 |
| - type: nauc_recall_at_1_max |
| value: 72.47899159663876 |
| - type: nauc_recall_at_1_std |
| value: 44.26584587323289 |
| - type: nauc_recall_at_20_diff1 |
| value: 99.47712418300625 |
| - type: nauc_recall_at_20_max |
| value: 100.0 |
| - type: nauc_recall_at_20_std |
| value: 77.69374416433251 |
| - type: nauc_recall_at_3_diff1 |
| value: 97.06316635181186 |
| - type: nauc_recall_at_3_max |
| value: 99.35400516795868 |
| - type: nauc_recall_at_3_std |
| value: 76.56070180009995 |
| - type: nauc_recall_at_5_diff1 |
| value: 98.03532524120769 |
| - type: nauc_recall_at_5_max |
| value: 100.0 |
| - type: nauc_recall_at_5_std |
| value: 78.16163502438029 |
| - type: ndcg_at_1 |
| value: 91.4 |
| - type: ndcg_at_10 |
| value: 94.50699999999999 |
| - type: ndcg_at_100 |
| value: 94.90899999999999 |
| - type: ndcg_at_1000 |
| value: 95.048 |
| - type: ndcg_at_20 |
| value: 94.626 |
| - type: ndcg_at_3 |
| value: 94.021 |
| - type: ndcg_at_5 |
| value: 94.31 |
| - type: precision_at_1 |
| value: 91.4 |
| - type: precision_at_10 |
| value: 9.700000000000001 |
| - type: precision_at_100 |
| value: 0.9900000000000001 |
| - type: precision_at_1000 |
| value: 0.1 |
| - type: precision_at_20 |
| value: 4.875 |
| - type: precision_at_3 |
| value: 31.900000000000002 |
| - type: precision_at_5 |
| value: 19.28 |
| - type: recall_at_1 |
| value: 91.4 |
| - type: recall_at_10 |
| value: 97.0 |
| - type: recall_at_100 |
| value: 99.0 |
| - type: recall_at_1000 |
| value: 100.0 |
| - type: recall_at_20 |
| value: 97.5 |
| - type: recall_at_3 |
| value: 95.7 |
| - type: recall_at_5 |
| value: 96.39999999999999 |
| task: |
| type: Retrieval |
| - dataset: |
| config: php |
| name: MTEB CodeSearchNetRetrieval (php) |
| revision: fdc6a9e39575768c27eb8a2a5f702bf846eb4759 |
| split: test |
| type: code-search-net/code_search_net |
| metrics: |
| - type: main_score |
| value: 92.288 |
| - type: map_at_1 |
| value: 85.5 |
| - type: map_at_10 |
| value: 90.312 |
| - type: map_at_100 |
| value: 90.351 |
| - type: map_at_1000 |
| value: 90.35300000000001 |
| - type: map_at_20 |
| value: 90.335 |
| - type: map_at_3 |
| value: 89.583 |
| - type: map_at_5 |
| value: 90.098 |
| - type: mrr_at_1 |
| value: 85.5 |
| - type: mrr_at_10 |
| value: 90.31230158730165 |
| - type: mrr_at_100 |
| value: 90.35130514931716 |
| - type: mrr_at_1000 |
| value: 90.3533725076738 |
| - type: mrr_at_20 |
| value: 90.33520202020209 |
| - type: mrr_at_3 |
| value: 89.58333333333339 |
| - type: mrr_at_5 |
| value: 90.09833333333341 |
| - type: nauc_map_at_1000_diff1 |
| value: 93.85544381917798 |
| - type: nauc_map_at_1000_max |
| value: 40.90472409247008 |
| - type: nauc_map_at_1000_std |
| value: -16.463521873713017 |
| - type: nauc_map_at_100_diff1 |
| value: 93.85516451221955 |
| - type: nauc_map_at_100_max |
| value: 40.917784128196594 |
| - type: nauc_map_at_100_std |
| value: -16.437783475285407 |
| - type: nauc_map_at_10_diff1 |
| value: 93.821658302016 |
| - type: nauc_map_at_10_max |
| value: 41.1631023862988 |
| - type: nauc_map_at_10_std |
| value: -15.954316262724616 |
| - type: nauc_map_at_1_diff1 |
| value: 94.51659451659448 |
| - type: nauc_map_at_1_max |
| value: 35.95691130574863 |
| - type: nauc_map_at_1_std |
| value: -25.156884459210044 |
| - type: nauc_map_at_20_diff1 |
| value: 93.83850503661382 |
| - type: nauc_map_at_20_max |
| value: 41.019314488512634 |
| - type: nauc_map_at_20_std |
| value: -16.237689928755426 |
| - type: nauc_map_at_3_diff1 |
| value: 93.80837080594951 |
| - type: nauc_map_at_3_max |
| value: 40.635458703255324 |
| - type: nauc_map_at_3_std |
| value: -17.033014335677798 |
| - type: nauc_map_at_5_diff1 |
| value: 93.9020242116256 |
| - type: nauc_map_at_5_max |
| value: 41.174017150832654 |
| - type: nauc_map_at_5_std |
| value: -16.30029652673213 |
| - type: nauc_mrr_at_1000_diff1 |
| value: 93.85544381917798 |
| - type: nauc_mrr_at_1000_max |
| value: 40.90472409247008 |
| - type: nauc_mrr_at_1000_std |
| value: -16.463521873713017 |
| - type: nauc_mrr_at_100_diff1 |
| value: 93.85516451221955 |
| - type: nauc_mrr_at_100_max |
| value: 40.917784128196594 |
| - type: nauc_mrr_at_100_std |
| value: -16.437783475285407 |
| - type: nauc_mrr_at_10_diff1 |
| value: 93.821658302016 |
| - type: nauc_mrr_at_10_max |
| value: 41.1631023862988 |
| - type: nauc_mrr_at_10_std |
| value: -15.954316262724616 |
| - type: nauc_mrr_at_1_diff1 |
| value: 94.51659451659448 |
| - type: nauc_mrr_at_1_max |
| value: 35.95691130574863 |
| - type: nauc_mrr_at_1_std |
| value: -25.156884459210044 |
| - type: nauc_mrr_at_20_diff1 |
| value: 93.83850503661382 |
| - type: nauc_mrr_at_20_max |
| value: 41.019314488512634 |
| - type: nauc_mrr_at_20_std |
| value: -16.237689928755426 |
| - type: nauc_mrr_at_3_diff1 |
| value: 93.80837080594951 |
| - type: nauc_mrr_at_3_max |
| value: 40.635458703255324 |
| - type: nauc_mrr_at_3_std |
| value: -17.033014335677798 |
| - type: nauc_mrr_at_5_diff1 |
| value: 93.9020242116256 |
| - type: nauc_mrr_at_5_max |
| value: 41.174017150832654 |
| - type: nauc_mrr_at_5_std |
| value: -16.30029652673213 |
| - type: nauc_ndcg_at_1000_diff1 |
| value: 93.77150134331873 |
| - type: nauc_ndcg_at_1000_max |
| value: 42.34949722008596 |
| - type: nauc_ndcg_at_1000_std |
| value: -13.688355172335143 |
| - type: nauc_ndcg_at_100_diff1 |
| value: 93.77562254293132 |
| - type: nauc_ndcg_at_100_max |
| value: 42.992049705674965 |
| - type: nauc_ndcg_at_100_std |
| value: -12.42122424240366 |
| - type: nauc_ndcg_at_10_diff1 |
| value: 93.54925638165832 |
| - type: nauc_ndcg_at_10_max |
| value: 44.555550459089986 |
| - type: nauc_ndcg_at_10_std |
| value: -9.337958348868717 |
| - type: nauc_ndcg_at_1_diff1 |
| value: 94.51659451659448 |
| - type: nauc_ndcg_at_1_max |
| value: 35.95691130574863 |
| - type: nauc_ndcg_at_1_std |
| value: -25.156884459210044 |
| - type: nauc_ndcg_at_20_diff1 |
| value: 93.61932694899781 |
| - type: nauc_ndcg_at_20_max |
| value: 43.96879091200643 |
| - type: nauc_ndcg_at_20_std |
| value: -10.495064090760449 |
| - type: nauc_ndcg_at_3_diff1 |
| value: 93.62155771108924 |
| - type: nauc_ndcg_at_3_max |
| value: 42.893378116364104 |
| - type: nauc_ndcg_at_3_std |
| value: -12.73029168651973 |
| - type: nauc_ndcg_at_5_diff1 |
| value: 93.78967591680556 |
| - type: nauc_ndcg_at_5_max |
| value: 44.44155136825693 |
| - type: nauc_ndcg_at_5_std |
| value: -10.579261977804965 |
| - type: nauc_precision_at_1000_diff1 |
| value: .nan |
| - type: nauc_precision_at_1000_max |
| value: .nan |
| - type: nauc_precision_at_1000_std |
| value: .nan |
| - type: nauc_precision_at_100_diff1 |
| value: 94.39775910363963 |
| - type: nauc_precision_at_100_max |
| value: 100.0 |
| - type: nauc_precision_at_100_std |
| value: 100.0 |
| - type: nauc_precision_at_10_diff1 |
| value: 89.1387927720107 |
| - type: nauc_precision_at_10_max |
| value: 100.0 |
| - type: nauc_precision_at_10_std |
| value: 100.0 |
| - type: nauc_precision_at_1_diff1 |
| value: 94.51659451659448 |
| - type: nauc_precision_at_1_max |
| value: 35.95691130574863 |
| - type: nauc_precision_at_1_std |
| value: -25.156884459210044 |
| - type: nauc_precision_at_20_diff1 |
| value: 89.61251167133568 |
| - type: nauc_precision_at_20_max |
| value: 100.0 |
| - type: nauc_precision_at_20_std |
| value: 100.0 |
| - type: nauc_precision_at_3_diff1 |
| value: 92.71284271284276 |
| - type: nauc_precision_at_3_max |
| value: 54.854426619132425 |
| - type: nauc_precision_at_3_std |
| value: 10.262286732874971 |
| - type: nauc_precision_at_5_diff1 |
| value: 93.00449346405235 |
| - type: nauc_precision_at_5_max |
| value: 73.52795284780564 |
| - type: nauc_precision_at_5_std |
| value: 40.204831932772336 |
| - type: nauc_recall_at_1000_diff1 |
| value: .nan |
| - type: nauc_recall_at_1000_max |
| value: .nan |
| - type: nauc_recall_at_1000_std |
| value: .nan |
| - type: nauc_recall_at_100_diff1 |
| value: 94.39775910364197 |
| - type: nauc_recall_at_100_max |
| value: 100.0 |
| - type: nauc_recall_at_100_std |
| value: 100.0 |
| - type: nauc_recall_at_10_diff1 |
| value: 89.13879277201103 |
| - type: nauc_recall_at_10_max |
| value: 100.0 |
| - type: nauc_recall_at_10_std |
| value: 100.0 |
| - type: nauc_recall_at_1_diff1 |
| value: 94.51659451659448 |
| - type: nauc_recall_at_1_max |
| value: 35.95691130574863 |
| - type: nauc_recall_at_1_std |
| value: -25.156884459210044 |
| - type: nauc_recall_at_20_diff1 |
| value: 89.61251167133527 |
| - type: nauc_recall_at_20_max |
| value: 100.0 |
| - type: nauc_recall_at_20_std |
| value: 100.0 |
| - type: nauc_recall_at_3_diff1 |
| value: 92.71284271284289 |
| - type: nauc_recall_at_3_max |
| value: 54.85442661913248 |
| - type: nauc_recall_at_3_std |
| value: 10.262286732875078 |
| - type: nauc_recall_at_5_diff1 |
| value: 93.00449346405259 |
| - type: nauc_recall_at_5_max |
| value: 73.52795284780608 |
| - type: nauc_recall_at_5_std |
| value: 40.20483193277305 |
| - type: ndcg_at_1 |
| value: 85.5 |
| - type: ndcg_at_10 |
| value: 92.288 |
| - type: ndcg_at_100 |
| value: 92.49300000000001 |
| - type: ndcg_at_1000 |
| value: 92.574 |
| - type: ndcg_at_20 |
| value: 92.366 |
| - type: ndcg_at_3 |
| value: 90.851 |
| - type: ndcg_at_5 |
| value: 91.789 |
| - type: precision_at_1 |
| value: 85.5 |
| - type: precision_at_10 |
| value: 9.83 |
| - type: precision_at_100 |
| value: 0.993 |
| - type: precision_at_1000 |
| value: 0.1 |
| - type: precision_at_20 |
| value: 4.93 |
| - type: precision_at_3 |
| value: 31.5 |
| - type: precision_at_5 |
| value: 19.36 |
| - type: recall_at_1 |
| value: 85.5 |
| - type: recall_at_10 |
| value: 98.3 |
| - type: recall_at_100 |
| value: 99.3 |
| - type: recall_at_1000 |
| value: 100.0 |
| - type: recall_at_20 |
| value: 98.6 |
| - type: recall_at_3 |
| value: 94.5 |
| - type: recall_at_5 |
| value: 96.8 |
| task: |
| type: Retrieval |
| - dataset: |
| config: default |
| name: MTEB CodeTransOceanContest (default) |
| revision: 20da4eb20a4b17300c0986ee148c90867a7f2a4d |
| split: test |
| type: CoIR-Retrieval/codetrans-contest |
| metrics: |
| - type: main_score |
| value: 94.96 |
| - type: map_at_1 |
| value: 91.855 |
| - type: map_at_10 |
| value: 94.054 |
| - type: map_at_100 |
| value: 94.14099999999999 |
| - type: map_at_1000 |
| value: 94.145 |
| - type: map_at_20 |
| value: 94.124 |
| - type: map_at_3 |
| value: 93.741 |
| - type: map_at_5 |
| value: 93.92200000000001 |
| - type: mrr_at_1 |
| value: 91.8552036199095 |
| - type: mrr_at_10 |
| value: 94.05354449472097 |
| - type: mrr_at_100 |
| value: 94.14134449135241 |
| - type: mrr_at_1000 |
| value: 94.14490739440583 |
| - type: mrr_at_20 |
| value: 94.12357250592545 |
| - type: mrr_at_3 |
| value: 93.74057315233786 |
| - type: mrr_at_5 |
| value: 93.92156862745098 |
| - type: nauc_map_at_1000_diff1 |
| value: 92.55002145543314 |
| - type: nauc_map_at_1000_max |
| value: 83.9544206201828 |
| - type: nauc_map_at_1000_std |
| value: 13.818812515717433 |
| - type: nauc_map_at_100_diff1 |
| value: 92.55455211118941 |
| - type: nauc_map_at_100_max |
| value: 83.96417863440325 |
| - type: nauc_map_at_100_std |
| value: 13.871223041576354 |
| - type: nauc_map_at_10_diff1 |
| value: 92.58163562742963 |
| - type: nauc_map_at_10_max |
| value: 84.20094910390932 |
| - type: nauc_map_at_10_std |
| value: 14.827570038179012 |
| - type: nauc_map_at_1_diff1 |
| value: 90.8130775986731 |
| - type: nauc_map_at_1_max |
| value: 81.11634573555125 |
| - type: nauc_map_at_1_std |
| value: 6.18089047774425 |
| - type: nauc_map_at_20_diff1 |
| value: 92.57706925983298 |
| - type: nauc_map_at_20_max |
| value: 84.01267551178432 |
| - type: nauc_map_at_20_std |
| value: 13.990683573503478 |
| - type: nauc_map_at_3_diff1 |
| value: 92.76365755738576 |
| - type: nauc_map_at_3_max |
| value: 84.18142235516605 |
| - type: nauc_map_at_3_std |
| value: 11.473399520154405 |
| - type: nauc_map_at_5_diff1 |
| value: 92.74270433278942 |
| - type: nauc_map_at_5_max |
| value: 84.12361744519418 |
| - type: nauc_map_at_5_std |
| value: 14.668669421165939 |
| - type: nauc_mrr_at_1000_diff1 |
| value: 92.55002145543314 |
| - type: nauc_mrr_at_1000_max |
| value: 83.9544206201828 |
| - type: nauc_mrr_at_1000_std |
| value: 13.818812515717433 |
| - type: nauc_mrr_at_100_diff1 |
| value: 92.55455211118941 |
| - type: nauc_mrr_at_100_max |
| value: 83.96417863440325 |
| - type: nauc_mrr_at_100_std |
| value: 13.871223041576354 |
| - type: nauc_mrr_at_10_diff1 |
| value: 92.58163562742963 |
| - type: nauc_mrr_at_10_max |
| value: 84.20094910390932 |
| - type: nauc_mrr_at_10_std |
| value: 14.827570038179012 |
| - type: nauc_mrr_at_1_diff1 |
| value: 90.8130775986731 |
| - type: nauc_mrr_at_1_max |
| value: 81.11634573555125 |
| - type: nauc_mrr_at_1_std |
| value: 6.18089047774425 |
| - type: nauc_mrr_at_20_diff1 |
| value: 92.57706925983298 |
| - type: nauc_mrr_at_20_max |
| value: 84.01267551178432 |
| - type: nauc_mrr_at_20_std |
| value: 13.990683573503478 |
| - type: nauc_mrr_at_3_diff1 |
| value: 92.76365755738576 |
| - type: nauc_mrr_at_3_max |
| value: 84.18142235516605 |
| - type: nauc_mrr_at_3_std |
| value: 11.473399520154405 |
| - type: nauc_mrr_at_5_diff1 |
| value: 92.74270433278942 |
| - type: nauc_mrr_at_5_max |
| value: 84.12361744519418 |
| - type: nauc_mrr_at_5_std |
| value: 14.668669421165939 |
| - type: nauc_ndcg_at_1000_diff1 |
| value: 92.84906743720337 |
| - type: nauc_ndcg_at_1000_max |
| value: 84.57965223804021 |
| - type: nauc_ndcg_at_1000_std |
| value: 15.875602814166239 |
| - type: nauc_ndcg_at_100_diff1 |
| value: 92.94862546375188 |
| - type: nauc_ndcg_at_100_max |
| value: 84.7943402354994 |
| - type: nauc_ndcg_at_100_std |
| value: 17.046814945571057 |
| - type: nauc_ndcg_at_10_diff1 |
| value: 93.18693605775724 |
| - type: nauc_ndcg_at_10_max |
| value: 85.99181169093023 |
| - type: nauc_ndcg_at_10_std |
| value: 21.453785415429227 |
| - type: nauc_ndcg_at_1_diff1 |
| value: 90.8130775986731 |
| - type: nauc_ndcg_at_1_max |
| value: 81.11634573555125 |
| - type: nauc_ndcg_at_1_std |
| value: 6.18089047774425 |
| - type: nauc_ndcg_at_20_diff1 |
| value: 93.18182308386798 |
| - type: nauc_ndcg_at_20_max |
| value: 85.29721008749047 |
| - type: nauc_ndcg_at_20_std |
| value: 18.265825551265774 |
| - type: nauc_ndcg_at_3_diff1 |
| value: 93.5678392571751 |
| - type: nauc_ndcg_at_3_max |
| value: 85.68193097329507 |
| - type: nauc_ndcg_at_3_std |
| value: 13.252390138276315 |
| - type: nauc_ndcg_at_5_diff1 |
| value: 93.57442357672642 |
| - type: nauc_ndcg_at_5_max |
| value: 85.65305420139138 |
| - type: nauc_ndcg_at_5_std |
| value: 20.399096483631958 |
| - type: nauc_precision_at_1000_diff1 |
| value: 100.0 |
| - type: nauc_precision_at_1000_max |
| value: 100.0 |
| - type: nauc_precision_at_1000_std |
| value: 100.0 |
| - type: nauc_precision_at_100_diff1 |
| value: 100.0 |
| - type: nauc_precision_at_100_max |
| value: 100.0 |
| - type: nauc_precision_at_100_std |
| value: 100.0 |
| - type: nauc_precision_at_10_diff1 |
| value: 97.38693467336647 |
| - type: nauc_precision_at_10_max |
| value: 100.0 |
| - type: nauc_precision_at_10_std |
| value: 76.08263155229871 |
| - type: nauc_precision_at_1_diff1 |
| value: 90.8130775986731 |
| - type: nauc_precision_at_1_max |
| value: 81.11634573555125 |
| - type: nauc_precision_at_1_std |
| value: 6.18089047774425 |
| - type: nauc_precision_at_20_diff1 |
| value: 100.0 |
| - type: nauc_precision_at_20_max |
| value: 100.0 |
| - type: nauc_precision_at_20_std |
| value: 69.38924997712752 |
| - type: nauc_precision_at_3_diff1 |
| value: 97.09659408151863 |
| - type: nauc_precision_at_3_max |
| value: 92.38060989189518 |
| - type: nauc_precision_at_3_std |
| value: 20.8638963856334 |
| - type: nauc_precision_at_5_diff1 |
| value: 98.13352476669084 |
| - type: nauc_precision_at_5_max |
| value: 94.1685830281352 |
| - type: nauc_precision_at_5_std |
| value: 54.20627096881086 |
| - type: nauc_recall_at_1000_diff1 |
| value: .nan |
| - type: nauc_recall_at_1000_max |
| value: .nan |
| - type: nauc_recall_at_1000_std |
| value: .nan |
| - type: nauc_recall_at_100_diff1 |
| value: 100.0 |
| - type: nauc_recall_at_100_max |
| value: 100.0 |
| - type: nauc_recall_at_100_std |
| value: 100.0 |
| - type: nauc_recall_at_10_diff1 |
| value: 97.38693467336678 |
| - type: nauc_recall_at_10_max |
| value: 100.0 |
| - type: nauc_recall_at_10_std |
| value: 76.08263155229885 |
| - type: nauc_recall_at_1_diff1 |
| value: 90.8130775986731 |
| - type: nauc_recall_at_1_max |
| value: 81.11634573555125 |
| - type: nauc_recall_at_1_std |
| value: 6.18089047774425 |
| - type: nauc_recall_at_20_diff1 |
| value: 100.0 |
| - type: nauc_recall_at_20_max |
| value: 100.0 |
| - type: nauc_recall_at_20_std |
| value: 69.3892499771279 |
| - type: nauc_recall_at_3_diff1 |
| value: 97.09659408151863 |
| - type: nauc_recall_at_3_max |
| value: 92.38060989189518 |
| - type: nauc_recall_at_3_std |
| value: 20.863896385634074 |
| - type: nauc_recall_at_5_diff1 |
| value: 98.13352476669056 |
| - type: nauc_recall_at_5_max |
| value: 94.16858302813482 |
| - type: nauc_recall_at_5_std |
| value: 54.20627096881058 |
| - type: ndcg_at_1 |
| value: 91.855 |
| - type: ndcg_at_10 |
| value: 94.96 |
| - type: ndcg_at_100 |
| value: 95.357 |
| - type: ndcg_at_1000 |
| value: 95.422 |
| - type: ndcg_at_20 |
| value: 95.19800000000001 |
| - type: ndcg_at_3 |
| value: 94.306 |
| - type: ndcg_at_5 |
| value: 94.65599999999999 |
| - type: precision_at_1 |
| value: 91.855 |
| - type: precision_at_10 |
| value: 9.774 |
| - type: precision_at_100 |
| value: 0.9950000000000001 |
| - type: precision_at_1000 |
| value: 0.1 |
| - type: precision_at_20 |
| value: 4.932 |
| - type: precision_at_3 |
| value: 31.976 |
| - type: precision_at_5 |
| value: 19.367 |
| - type: recall_at_1 |
| value: 91.855 |
| - type: recall_at_10 |
| value: 97.738 |
| - type: recall_at_100 |
| value: 99.548 |
| - type: recall_at_1000 |
| value: 100.0 |
| - type: recall_at_20 |
| value: 98.643 |
| - type: recall_at_3 |
| value: 95.928 |
| - type: recall_at_5 |
| value: 96.833 |
| task: |
| type: Retrieval |
| - dataset: |
| config: default |
| name: MTEB CodeTransOceanDL (default) |
| revision: 281562cb8a1265ab5c0824bfa6ddcd9b0a15618f |
| split: test |
| type: CoIR-Retrieval/codetrans-dl |
| metrics: |
| - type: main_score |
| value: 38.566 |
| - type: map_at_1 |
| value: 8.889 |
| - type: map_at_10 |
| value: 24.311 |
| - type: map_at_100 |
| value: 25.413999999999998 |
| - type: map_at_1000 |
| value: 25.413999999999998 |
| - type: map_at_20 |
| value: 25.413999999999998 |
| - type: map_at_3 |
| value: 14.352 |
| - type: map_at_5 |
| value: 18.296 |
| - type: mrr_at_1 |
| value: 7.777777777777778 |
| - type: mrr_at_10 |
| value: 23.692460317460355 |
| - type: mrr_at_100 |
| value: 24.94246031746036 |
| - type: mrr_at_1000 |
| value: 24.94246031746036 |
| - type: mrr_at_20 |
| value: 24.94246031746036 |
| - type: mrr_at_3 |
| value: 13.611111111111104 |
| - type: mrr_at_5 |
| value: 18.13888888888888 |
| - type: nauc_map_at_1000_diff1 |
| value: -11.995791794456052 |
| - type: nauc_map_at_1000_max |
| value: -37.4290224077524 |
| - type: nauc_map_at_1000_std |
| value: -8.685310117692485 |
| - type: nauc_map_at_100_diff1 |
| value: -11.995791794456052 |
| - type: nauc_map_at_100_max |
| value: -37.4290224077524 |
| - type: nauc_map_at_100_std |
| value: -8.685310117692485 |
| - type: nauc_map_at_10_diff1 |
| value: -12.904069625259378 |
| - type: nauc_map_at_10_max |
| value: -36.469676015537445 |
| - type: nauc_map_at_10_std |
| value: -8.18556060930571 |
| - type: nauc_map_at_1_diff1 |
| value: -22.632143815401133 |
| - type: nauc_map_at_1_max |
| value: -38.05674805473572 |
| - type: nauc_map_at_1_std |
| value: -5.114703514891332 |
| - type: nauc_map_at_20_diff1 |
| value: -11.995791794456052 |
| - type: nauc_map_at_20_max |
| value: -37.4290224077524 |
| - type: nauc_map_at_20_std |
| value: -8.685310117692485 |
| - type: nauc_map_at_3_diff1 |
| value: -9.430569631049272 |
| - type: nauc_map_at_3_max |
| value: -40.31306127773153 |
| - type: nauc_map_at_3_std |
| value: -9.279256680145705 |
| - type: nauc_map_at_5_diff1 |
| value: -11.17196963640011 |
| - type: nauc_map_at_5_max |
| value: -36.71021151035073 |
| - type: nauc_map_at_5_std |
| value: -10.227972134518213 |
| - type: nauc_mrr_at_1000_diff1 |
| value: 7.811397652262392 |
| - type: nauc_mrr_at_1000_max |
| value: -35.827281345154454 |
| - type: nauc_mrr_at_1000_std |
| value: -10.625492656056048 |
| - type: nauc_mrr_at_100_diff1 |
| value: 7.811397652262392 |
| - type: nauc_mrr_at_100_max |
| value: -35.827281345154454 |
| - type: nauc_mrr_at_100_std |
| value: -10.625492656056048 |
| - type: nauc_mrr_at_10_diff1 |
| value: 7.0457873475325155 |
| - type: nauc_mrr_at_10_max |
| value: -34.091630838685006 |
| - type: nauc_mrr_at_10_std |
| value: -10.015877703911025 |
| - type: nauc_mrr_at_1_diff1 |
| value: 6.72505653723791 |
| - type: nauc_mrr_at_1_max |
| value: -39.05860707577908 |
| - type: nauc_mrr_at_1_std |
| value: -12.350032580781182 |
| - type: nauc_mrr_at_20_diff1 |
| value: 7.811397652262392 |
| - type: nauc_mrr_at_20_max |
| value: -35.827281345154454 |
| - type: nauc_mrr_at_20_std |
| value: -10.625492656056048 |
| - type: nauc_mrr_at_3_diff1 |
| value: 9.229424174654557 |
| - type: nauc_mrr_at_3_max |
| value: -39.823960095369024 |
| - type: nauc_mrr_at_3_std |
| value: -12.365662926663418 |
| - type: nauc_mrr_at_5_diff1 |
| value: 7.455314880931567 |
| - type: nauc_mrr_at_5_max |
| value: -34.4606595687379 |
| - type: nauc_mrr_at_5_std |
| value: -9.748438671046124 |
| - type: nauc_ndcg_at_1000_diff1 |
| value: -11.393187924360975 |
| - type: nauc_ndcg_at_1000_max |
| value: -37.14382596573424 |
| - type: nauc_ndcg_at_1000_std |
| value: -8.714465550011122 |
| - type: nauc_ndcg_at_100_diff1 |
| value: -11.393187924360975 |
| - type: nauc_ndcg_at_100_max |
| value: -37.14382596573424 |
| - type: nauc_ndcg_at_100_std |
| value: -8.714465550011122 |
| - type: nauc_ndcg_at_10_diff1 |
| value: -14.50305018622005 |
| - type: nauc_ndcg_at_10_max |
| value: -34.183276513019024 |
| - type: nauc_ndcg_at_10_std |
| value: -6.91808119392616 |
| - type: nauc_ndcg_at_1_diff1 |
| value: -22.632143815401133 |
| - type: nauc_ndcg_at_1_max |
| value: -38.05674805473572 |
| - type: nauc_ndcg_at_1_std |
| value: -5.114703514891332 |
| - type: nauc_ndcg_at_20_diff1 |
| value: -11.393187924360975 |
| - type: nauc_ndcg_at_20_max |
| value: -37.14382596573424 |
| - type: nauc_ndcg_at_20_std |
| value: -8.714465550011122 |
| - type: nauc_ndcg_at_3_diff1 |
| value: -6.28820826432392 |
| - type: nauc_ndcg_at_3_max |
| value: -40.515698550813426 |
| - type: nauc_ndcg_at_3_std |
| value: -9.763678652070176 |
| - type: nauc_ndcg_at_5_diff1 |
| value: -9.784255160012018 |
| - type: nauc_ndcg_at_5_max |
| value: -35.547401094483774 |
| - type: nauc_ndcg_at_5_std |
| value: -11.253198818827153 |
| - type: nauc_precision_at_1000_diff1 |
| value: 100.0 |
| - type: nauc_precision_at_1000_max |
| value: 100.0 |
| - type: nauc_precision_at_1000_std |
| value: 100.0 |
| - type: nauc_precision_at_100_diff1 |
| value: 100.0 |
| - type: nauc_precision_at_100_max |
| value: 100.0 |
| - type: nauc_precision_at_100_std |
| value: 100.0 |
| - type: nauc_precision_at_10_diff1 |
| value: -28.569339405841387 |
| - type: nauc_precision_at_10_max |
| value: -21.317845652446533 |
| - type: nauc_precision_at_10_std |
| value: 1.6546191451133108 |
| - type: nauc_precision_at_1_diff1 |
| value: -22.632143815401133 |
| - type: nauc_precision_at_1_max |
| value: -38.05674805473572 |
| - type: nauc_precision_at_1_std |
| value: -5.114703514891332 |
| - type: nauc_precision_at_20_diff1 |
| value: .nan |
| - type: nauc_precision_at_20_max |
| value: .nan |
| - type: nauc_precision_at_20_std |
| value: .nan |
| - type: nauc_precision_at_3_diff1 |
| value: 0.21101534115564527 |
| - type: nauc_precision_at_3_max |
| value: -40.8638015094404 |
| - type: nauc_precision_at_3_std |
| value: -10.657976464982523 |
| - type: nauc_precision_at_5_diff1 |
| value: -7.729717358965525 |
| - type: nauc_precision_at_5_max |
| value: -33.27683373383679 |
| - type: nauc_precision_at_5_std |
| value: -13.203741846893195 |
| - type: nauc_recall_at_1000_diff1 |
| value: .nan |
| - type: nauc_recall_at_1000_max |
| value: .nan |
| - type: nauc_recall_at_1000_std |
| value: .nan |
| - type: nauc_recall_at_100_diff1 |
| value: .nan |
| - type: nauc_recall_at_100_max |
| value: .nan |
| - type: nauc_recall_at_100_std |
| value: .nan |
| - type: nauc_recall_at_10_diff1 |
| value: -28.569339405841294 |
| - type: nauc_recall_at_10_max |
| value: -21.317845652446415 |
| - type: nauc_recall_at_10_std |
| value: 1.6546191451134153 |
| - type: nauc_recall_at_1_diff1 |
| value: -22.632143815401133 |
| - type: nauc_recall_at_1_max |
| value: -38.05674805473572 |
| - type: nauc_recall_at_1_std |
| value: -5.114703514891332 |
| - type: nauc_recall_at_20_diff1 |
| value: .nan |
| - type: nauc_recall_at_20_max |
| value: .nan |
| - type: nauc_recall_at_20_std |
| value: .nan |
| - type: nauc_recall_at_3_diff1 |
| value: 0.2110153411556453 |
| - type: nauc_recall_at_3_max |
| value: -40.863801509440385 |
| - type: nauc_recall_at_3_std |
| value: -10.65797646498251 |
| - type: nauc_recall_at_5_diff1 |
| value: -7.729717358965576 |
| - type: nauc_recall_at_5_max |
| value: -33.276833733836845 |
| - type: nauc_recall_at_5_std |
| value: -13.203741846893251 |
| - type: ndcg_at_1 |
| value: 8.889 |
| - type: ndcg_at_10 |
| value: 38.566 |
| - type: ndcg_at_100 |
| value: 42.063 |
| - type: ndcg_at_1000 |
| value: 42.063 |
| - type: ndcg_at_20 |
| value: 42.063 |
| - type: ndcg_at_3 |
| value: 16.488 |
| - type: ndcg_at_5 |
| value: 23.589 |
| - type: precision_at_1 |
| value: 8.889 |
| - type: precision_at_10 |
| value: 8.722000000000001 |
| - type: precision_at_100 |
| value: 1.0 |
| - type: precision_at_1000 |
| value: 0.1 |
| - type: precision_at_20 |
| value: 5.0 |
| - type: precision_at_3 |
| value: 7.593 |
| - type: precision_at_5 |
| value: 8.0 |
| - type: recall_at_1 |
| value: 8.889 |
| - type: recall_at_10 |
| value: 87.222 |
| - type: recall_at_100 |
| value: 100.0 |
| - type: recall_at_1000 |
| value: 100.0 |
| - type: recall_at_20 |
| value: 100.0 |
| - type: recall_at_3 |
| value: 22.778000000000002 |
| - type: recall_at_5 |
| value: 40.0 |
| task: |
| type: Retrieval |
| - dataset: |
| config: default |
| name: MTEB CosQA (default) |
| revision: bc5efb7e9d437246ce393ed19d772e08e4a79535 |
| split: test |
| type: CoIR-Retrieval/cosqa |
| metrics: |
| - type: main_score |
| value: 34.452 |
| - type: map_at_1 |
| value: 15.2 |
| - type: map_at_10 |
| value: 26.473000000000003 |
| - type: map_at_100 |
| value: 28.01 |
| - type: map_at_1000 |
| value: 28.038 |
| - type: map_at_20 |
| value: 27.48 |
| - type: map_at_3 |
| value: 20.833 |
| - type: map_at_5 |
| value: 24.153 |
| - type: mrr_at_1 |
| value: 19.2 |
| - type: mrr_at_10 |
| value: 28.281111111111105 |
| - type: mrr_at_100 |
| value: 29.8632158616367 |
| - type: mrr_at_1000 |
| value: 29.891199207757197 |
| - type: mrr_at_20 |
| value: 29.27876414692977 |
| - type: mrr_at_3 |
| value: 23.966666666666658 |
| - type: mrr_at_5 |
| value: 25.636666666666663 |
| - type: nauc_map_at_1000_diff1 |
| value: 24.10146846046148 |
| - type: nauc_map_at_1000_max |
| value: 10.598410702361434 |
| - type: nauc_map_at_1000_std |
| value: -20.144523237787222 |
| - type: nauc_map_at_100_diff1 |
| value: 24.053508672795086 |
| - type: nauc_map_at_100_max |
| value: 10.62060173891184 |
| - type: nauc_map_at_100_std |
| value: -20.123151261597684 |
| - type: nauc_map_at_10_diff1 |
| value: 24.03401073979636 |
| - type: nauc_map_at_10_max |
| value: 10.204901836438324 |
| - type: nauc_map_at_10_std |
| value: -20.723915647527605 |
| - type: nauc_map_at_1_diff1 |
| value: 34.9707775098606 |
| - type: nauc_map_at_1_max |
| value: 8.792695974322537 |
| - type: nauc_map_at_1_std |
| value: -17.679606534340426 |
| - type: nauc_map_at_20_diff1 |
| value: 23.635390281247904 |
| - type: nauc_map_at_20_max |
| value: 10.51091267854992 |
| - type: nauc_map_at_20_std |
| value: -20.503982338903548 |
| - type: nauc_map_at_3_diff1 |
| value: 27.08078023869985 |
| - type: nauc_map_at_3_max |
| value: 8.624902998628762 |
| - type: nauc_map_at_3_std |
| value: -21.16290389423999 |
| - type: nauc_map_at_5_diff1 |
| value: 24.375227072550878 |
| - type: nauc_map_at_5_max |
| value: 10.446847054213334 |
| - type: nauc_map_at_5_std |
| value: -20.416673549725004 |
| - type: nauc_mrr_at_1000_diff1 |
| value: 19.78435935694305 |
| - type: nauc_mrr_at_1000_max |
| value: 6.389276526733645 |
| - type: nauc_mrr_at_1000_std |
| value: -24.87029202656466 |
| - type: nauc_mrr_at_100_diff1 |
| value: 19.741897116785875 |
| - type: nauc_mrr_at_100_max |
| value: 6.399527259062092 |
| - type: nauc_mrr_at_100_std |
| value: -24.86454789316169 |
| - type: nauc_mrr_at_10_diff1 |
| value: 19.82954896251409 |
| - type: nauc_mrr_at_10_max |
| value: 5.841515376599231 |
| - type: nauc_mrr_at_10_std |
| value: -25.585472957584926 |
| - type: nauc_mrr_at_1_diff1 |
| value: 22.61366365164493 |
| - type: nauc_mrr_at_1_max |
| value: 5.888630966674663 |
| - type: nauc_mrr_at_1_std |
| value: -23.551522185757364 |
| - type: nauc_mrr_at_20_diff1 |
| value: 19.610569618359854 |
| - type: nauc_mrr_at_20_max |
| value: 6.388923736326994 |
| - type: nauc_mrr_at_20_std |
| value: -25.022579250123666 |
| - type: nauc_mrr_at_3_diff1 |
| value: 21.689261589102053 |
| - type: nauc_mrr_at_3_max |
| value: 5.433511684692397 |
| - type: nauc_mrr_at_3_std |
| value: -25.22461843203615 |
| - type: nauc_mrr_at_5_diff1 |
| value: 20.769666984905864 |
| - type: nauc_mrr_at_5_max |
| value: 5.103054542191035 |
| - type: nauc_mrr_at_5_std |
| value: -25.62916796715137 |
| - type: nauc_ndcg_at_1000_diff1 |
| value: 21.724475676486286 |
| - type: nauc_ndcg_at_1000_max |
| value: 11.92044108916809 |
| - type: nauc_ndcg_at_1000_std |
| value: -19.560471367028224 |
| - type: nauc_ndcg_at_100_diff1 |
| value: 20.65779971541893 |
| - type: nauc_ndcg_at_100_max |
| value: 12.772564966149327 |
| - type: nauc_ndcg_at_100_std |
| value: -18.53040956973807 |
| - type: nauc_ndcg_at_10_diff1 |
| value: 19.715388368513594 |
| - type: nauc_ndcg_at_10_max |
| value: 11.290056737017848 |
| - type: nauc_ndcg_at_10_std |
| value: -21.48987713396045 |
| - type: nauc_ndcg_at_1_diff1 |
| value: 34.9707775098606 |
| - type: nauc_ndcg_at_1_max |
| value: 8.792695974322537 |
| - type: nauc_ndcg_at_1_std |
| value: -17.679606534340426 |
| - type: nauc_ndcg_at_20_diff1 |
| value: 18.26515065287157 |
| - type: nauc_ndcg_at_20_max |
| value: 12.420581957680346 |
| - type: nauc_ndcg_at_20_std |
| value: -20.560776758982286 |
| - type: nauc_ndcg_at_3_diff1 |
| value: 24.86486238625192 |
| - type: nauc_ndcg_at_3_max |
| value: 8.51016243931613 |
| - type: nauc_ndcg_at_3_std |
| value: -22.38799975521682 |
| - type: nauc_ndcg_at_5_diff1 |
| value: 20.38709195941383 |
| - type: nauc_ndcg_at_5_max |
| value: 11.692332914550612 |
| - type: nauc_ndcg_at_5_std |
| value: -21.095128458420735 |
| - type: nauc_precision_at_1000_diff1 |
| value: -2.427637721760636 |
| - type: nauc_precision_at_1000_max |
| value: 93.46405228758199 |
| - type: nauc_precision_at_1000_std |
| value: 56.139122315591536 |
| - type: nauc_precision_at_100_diff1 |
| value: -12.075404874593668 |
| - type: nauc_precision_at_100_max |
| value: 46.41166811552213 |
| - type: nauc_precision_at_100_std |
| value: 21.43662062526193 |
| - type: nauc_precision_at_10_diff1 |
| value: 7.927965224591165 |
| - type: nauc_precision_at_10_max |
| value: 14.499053516090232 |
| - type: nauc_precision_at_10_std |
| value: -23.215405338109257 |
| - type: nauc_precision_at_1_diff1 |
| value: 34.9707775098606 |
| - type: nauc_precision_at_1_max |
| value: 8.792695974322537 |
| - type: nauc_precision_at_1_std |
| value: -17.679606534340426 |
| - type: nauc_precision_at_20_diff1 |
| value: -2.2220305852593376 |
| - type: nauc_precision_at_20_max |
| value: 20.688168333908145 |
| - type: nauc_precision_at_20_std |
| value: -19.112501847927895 |
| - type: nauc_precision_at_3_diff1 |
| value: 19.523274018213513 |
| - type: nauc_precision_at_3_max |
| value: 8.217018367841955 |
| - type: nauc_precision_at_3_std |
| value: -25.40158982161363 |
| - type: nauc_precision_at_5_diff1 |
| value: 10.681426032017447 |
| - type: nauc_precision_at_5_max |
| value: 14.961711488638585 |
| - type: nauc_precision_at_5_std |
| value: -22.588397199647623 |
| - type: nauc_recall_at_1000_diff1 |
| value: -2.4276377217553624 |
| - type: nauc_recall_at_1000_max |
| value: 93.46405228758147 |
| - type: nauc_recall_at_1000_std |
| value: 56.13912231559195 |
| - type: nauc_recall_at_100_diff1 |
| value: -12.075404874593449 |
| - type: nauc_recall_at_100_max |
| value: 46.41166811552219 |
| - type: nauc_recall_at_100_std |
| value: 21.436620625261668 |
| - type: nauc_recall_at_10_diff1 |
| value: 7.927965224591155 |
| - type: nauc_recall_at_10_max |
| value: 14.499053516090227 |
| - type: nauc_recall_at_10_std |
| value: -23.21540533810927 |
| - type: nauc_recall_at_1_diff1 |
| value: 34.9707775098606 |
| - type: nauc_recall_at_1_max |
| value: 8.792695974322537 |
| - type: nauc_recall_at_1_std |
| value: -17.679606534340426 |
| - type: nauc_recall_at_20_diff1 |
| value: -2.22203058525925 |
| - type: nauc_recall_at_20_max |
| value: 20.688168333908262 |
| - type: nauc_recall_at_20_std |
| value: -19.11250184792784 |
| - type: nauc_recall_at_3_diff1 |
| value: 19.523274018213503 |
| - type: nauc_recall_at_3_max |
| value: 8.217018367841955 |
| - type: nauc_recall_at_3_std |
| value: -25.40158982161363 |
| - type: nauc_recall_at_5_diff1 |
| value: 10.681426032017423 |
| - type: nauc_recall_at_5_max |
| value: 14.961711488638551 |
| - type: nauc_recall_at_5_std |
| value: -22.58839719964763 |
| - type: ndcg_at_1 |
| value: 15.2 |
| - type: ndcg_at_10 |
| value: 34.452 |
| - type: ndcg_at_100 |
| value: 41.723 |
| - type: ndcg_at_1000 |
| value: 42.421 |
| - type: ndcg_at_20 |
| value: 38.002 |
| - type: ndcg_at_3 |
| value: 22.864 |
| - type: ndcg_at_5 |
| value: 28.904999999999998 |
| - type: precision_at_1 |
| value: 15.2 |
| - type: precision_at_10 |
| value: 6.0600000000000005 |
| - type: precision_at_100 |
| value: 0.942 |
| - type: precision_at_1000 |
| value: 0.1 |
| - type: precision_at_20 |
| value: 3.7199999999999998 |
| - type: precision_at_3 |
| value: 9.6 |
| - type: precision_at_5 |
| value: 8.72 |
| - type: recall_at_1 |
| value: 15.2 |
| - type: recall_at_10 |
| value: 60.6 |
| - type: recall_at_100 |
| value: 94.19999999999999 |
| - type: recall_at_1000 |
| value: 99.6 |
| - type: recall_at_20 |
| value: 74.4 |
| - type: recall_at_3 |
| value: 28.799999999999997 |
| - type: recall_at_5 |
| value: 43.6 |
| task: |
| type: Retrieval |
| - dataset: |
| config: default |
| name: MTEB StackOverflowQA (default) |
| revision: db8f169f3894c14a00251061f957b2063eef2bd5 |
| split: test |
| type: CoIR-Retrieval/stackoverflow-qa |
| metrics: |
| - type: main_score |
| value: 97.16799999999999 |
| - type: map_at_1 |
| value: 94.985 |
| - type: map_at_10 |
| value: 96.533 |
| - type: map_at_100 |
| value: 96.568 |
| - type: map_at_1000 |
| value: 96.569 |
| - type: map_at_20 |
| value: 96.554 |
| - type: map_at_3 |
| value: 96.272 |
| - type: map_at_5 |
| value: 96.48 |
| - type: mrr_at_1 |
| value: 94.98495486459379 |
| - type: mrr_at_10 |
| value: 96.53325054528665 |
| - type: mrr_at_100 |
| value: 96.56849512537934 |
| - type: mrr_at_1000 |
| value: 96.56899560792449 |
| - type: mrr_at_20 |
| value: 96.55354401474338 |
| - type: mrr_at_3 |
| value: 96.27214978268137 |
| - type: mrr_at_5 |
| value: 96.48027415580073 |
| - type: nauc_map_at_1000_diff1 |
| value: 96.7701081739238 |
| - type: nauc_map_at_1000_max |
| value: 89.92147554128405 |
| - type: nauc_map_at_1000_std |
| value: -11.367860416596368 |
| - type: nauc_map_at_100_diff1 |
| value: 96.7701266461378 |
| - type: nauc_map_at_100_max |
| value: 89.92249288169609 |
| - type: nauc_map_at_100_std |
| value: -11.365088267343506 |
| - type: nauc_map_at_10_diff1 |
| value: 96.80296303411313 |
| - type: nauc_map_at_10_max |
| value: 89.97144795621024 |
| - type: nauc_map_at_10_std |
| value: -11.225883175842885 |
| - type: nauc_map_at_1_diff1 |
| value: 96.33579234749389 |
| - type: nauc_map_at_1_max |
| value: 88.14831631499773 |
| - type: nauc_map_at_1_std |
| value: -11.043584429943797 |
| - type: nauc_map_at_20_diff1 |
| value: 96.7841381971511 |
| - type: nauc_map_at_20_max |
| value: 89.96621023207143 |
| - type: nauc_map_at_20_std |
| value: -11.236908272386724 |
| - type: nauc_map_at_3_diff1 |
| value: 96.77341739930384 |
| - type: nauc_map_at_3_max |
| value: 90.18226749615211 |
| - type: nauc_map_at_3_std |
| value: -12.405699503933928 |
| - type: nauc_map_at_5_diff1 |
| value: 96.79677069369804 |
| - type: nauc_map_at_5_max |
| value: 90.04406323888772 |
| - type: nauc_map_at_5_std |
| value: -11.09579820828818 |
| - type: nauc_mrr_at_1000_diff1 |
| value: 96.7701081739238 |
| - type: nauc_mrr_at_1000_max |
| value: 89.92147554128405 |
| - type: nauc_mrr_at_1000_std |
| value: -11.367860416596368 |
| - type: nauc_mrr_at_100_diff1 |
| value: 96.7701266461378 |
| - type: nauc_mrr_at_100_max |
| value: 89.92249288169609 |
| - type: nauc_mrr_at_100_std |
| value: -11.365088267343506 |
| - type: nauc_mrr_at_10_diff1 |
| value: 96.80296303411313 |
| - type: nauc_mrr_at_10_max |
| value: 89.97144795621024 |
| - type: nauc_mrr_at_10_std |
| value: -11.225883175842885 |
| - type: nauc_mrr_at_1_diff1 |
| value: 96.33579234749389 |
| - type: nauc_mrr_at_1_max |
| value: 88.14831631499773 |
| - type: nauc_mrr_at_1_std |
| value: -11.043584429943797 |
| - type: nauc_mrr_at_20_diff1 |
| value: 96.7841381971511 |
| - type: nauc_mrr_at_20_max |
| value: 89.96621023207143 |
| - type: nauc_mrr_at_20_std |
| value: -11.236908272386724 |
| - type: nauc_mrr_at_3_diff1 |
| value: 96.77341739930384 |
| - type: nauc_mrr_at_3_max |
| value: 90.18226749615211 |
| - type: nauc_mrr_at_3_std |
| value: -12.405699503933928 |
| - type: nauc_mrr_at_5_diff1 |
| value: 96.79677069369804 |
| - type: nauc_mrr_at_5_max |
| value: 90.04406323888772 |
| - type: nauc_mrr_at_5_std |
| value: -11.09579820828818 |
| - type: nauc_ndcg_at_1000_diff1 |
| value: 96.83153135375731 |
| - type: nauc_ndcg_at_1000_max |
| value: 90.17747818029089 |
| - type: nauc_ndcg_at_1000_std |
| value: -11.09225768112506 |
| - type: nauc_ndcg_at_100_diff1 |
| value: 96.82500676553634 |
| - type: nauc_ndcg_at_100_max |
| value: 90.21640577830735 |
| - type: nauc_ndcg_at_100_std |
| value: -11.051187241816189 |
| - type: nauc_ndcg_at_10_diff1 |
| value: 97.01040984737975 |
| - type: nauc_ndcg_at_10_max |
| value: 90.53590661634338 |
| - type: nauc_ndcg_at_10_std |
| value: -10.181364845409425 |
| - type: nauc_ndcg_at_1_diff1 |
| value: 96.33579234749389 |
| - type: nauc_ndcg_at_1_max |
| value: 88.14831631499773 |
| - type: nauc_ndcg_at_1_std |
| value: -11.043584429943797 |
| - type: nauc_ndcg_at_20_diff1 |
| value: 96.92876646822131 |
| - type: nauc_ndcg_at_20_max |
| value: 90.53613648406619 |
| - type: nauc_ndcg_at_20_std |
| value: -10.211015921117655 |
| - type: nauc_ndcg_at_3_diff1 |
| value: 96.92301963415002 |
| - type: nauc_ndcg_at_3_max |
| value: 90.96341737957319 |
| - type: nauc_ndcg_at_3_std |
| value: -12.397871226017621 |
| - type: nauc_ndcg_at_5_diff1 |
| value: 96.99223706029623 |
| - type: nauc_ndcg_at_5_max |
| value: 90.7297142361567 |
| - type: nauc_ndcg_at_5_std |
| value: -9.78987977360544 |
| - type: nauc_precision_at_1000_diff1 |
| value: .nan |
| - type: nauc_precision_at_1000_max |
| value: .nan |
| - type: nauc_precision_at_1000_std |
| value: .nan |
| - type: nauc_precision_at_100_diff1 |
| value: 95.64367251079331 |
| - type: nauc_precision_at_100_max |
| value: 95.64367251079331 |
| - type: nauc_precision_at_100_std |
| value: -7.9737174815413105 |
| - type: nauc_precision_at_10_diff1 |
| value: 99.27394541846478 |
| - type: nauc_precision_at_10_max |
| value: 96.07116729984303 |
| - type: nauc_precision_at_10_std |
| value: 2.384368710766077 |
| - type: nauc_precision_at_1_diff1 |
| value: 96.33579234749389 |
| - type: nauc_precision_at_1_max |
| value: 88.14831631499773 |
| - type: nauc_precision_at_1_std |
| value: -11.043584429943797 |
| - type: nauc_precision_at_20_diff1 |
| value: 98.91091812769709 |
| - type: nauc_precision_at_20_max |
| value: 98.91091812769709 |
| - type: nauc_precision_at_20_std |
| value: 8.043758582511515 |
| - type: nauc_precision_at_3_diff1 |
| value: 97.6238213695205 |
| - type: nauc_precision_at_3_max |
| value: 94.67950559666251 |
| - type: nauc_precision_at_3_std |
| value: -12.047680267059418 |
| - type: nauc_precision_at_5_diff1 |
| value: 98.49204048450248 |
| - type: nauc_precision_at_5_max |
| value: 95.77207938439504 |
| - type: nauc_precision_at_5_std |
| value: 1.5409353000901833 |
| - type: nauc_recall_at_1000_diff1 |
| value: .nan |
| - type: nauc_recall_at_1000_max |
| value: .nan |
| - type: nauc_recall_at_1000_std |
| value: .nan |
| - type: nauc_recall_at_100_diff1 |
| value: 95.64367251078416 |
| - type: nauc_recall_at_100_max |
| value: 95.64367251078416 |
| - type: nauc_recall_at_100_std |
| value: -7.973717481540185 |
| - type: nauc_recall_at_10_diff1 |
| value: 99.27394541846407 |
| - type: nauc_recall_at_10_max |
| value: 96.07116729984456 |
| - type: nauc_recall_at_10_std |
| value: 2.3843687107667484 |
| - type: nauc_recall_at_1_diff1 |
| value: 96.33579234749389 |
| - type: nauc_recall_at_1_max |
| value: 88.14831631499773 |
| - type: nauc_recall_at_1_std |
| value: -11.043584429943797 |
| - type: nauc_recall_at_20_diff1 |
| value: 98.91091812769824 |
| - type: nauc_recall_at_20_max |
| value: 98.91091812769824 |
| - type: nauc_recall_at_20_std |
| value: 8.043758582515963 |
| - type: nauc_recall_at_3_diff1 |
| value: 97.62382136952051 |
| - type: nauc_recall_at_3_max |
| value: 94.67950559666313 |
| - type: nauc_recall_at_3_std |
| value: -12.047680267059349 |
| - type: nauc_recall_at_5_diff1 |
| value: 98.49204048450328 |
| - type: nauc_recall_at_5_max |
| value: 95.77207938439567 |
| - type: nauc_recall_at_5_std |
| value: 1.54093530009189 |
| - type: ndcg_at_1 |
| value: 94.985 |
| - type: ndcg_at_10 |
| value: 97.16799999999999 |
| - type: ndcg_at_100 |
| value: 97.333 |
| - type: ndcg_at_1000 |
| value: 97.351 |
| - type: ndcg_at_20 |
| value: 97.243 |
| - type: ndcg_at_3 |
| value: 96.665 |
| - type: ndcg_at_5 |
| value: 97.038 |
| - type: precision_at_1 |
| value: 94.985 |
| - type: precision_at_10 |
| value: 9.91 |
| - type: precision_at_100 |
| value: 0.9979999999999999 |
| - type: precision_at_1000 |
| value: 0.1 |
| - type: precision_at_20 |
| value: 4.97 |
| - type: precision_at_3 |
| value: 32.598 |
| - type: precision_at_5 |
| value: 19.739 |
| - type: recall_at_1 |
| value: 94.985 |
| - type: recall_at_10 |
| value: 99.09700000000001 |
| - type: recall_at_100 |
| value: 99.85000000000001 |
| - type: recall_at_1000 |
| value: 100.0 |
| - type: recall_at_20 |
| value: 99.398 |
| - type: recall_at_3 |
| value: 97.79299999999999 |
| - type: recall_at_5 |
| value: 98.696 |
| task: |
| type: Retrieval |
| - dataset: |
| config: default |
| name: MTEB SyntheticText2SQL (default) |
| revision: 686b87296c3a0191b5d9415a00526c62db9fce09 |
| split: test |
| type: CoIR-Retrieval/synthetic-text2sql |
| metrics: |
| - type: main_score |
| value: 62.871 |
| - type: map_at_1 |
| value: 5.931 |
| - type: map_at_10 |
| value: 50.172000000000004 |
| - type: map_at_100 |
| value: 50.221000000000004 |
| - type: map_at_1000 |
| value: 50.221000000000004 |
| - type: map_at_20 |
| value: 50.217 |
| - type: map_at_3 |
| value: 49.126 |
| - type: map_at_5 |
| value: 49.966 |
| - type: mrr_at_1 |
| value: 46.84669287301316 |
| - type: mrr_at_10 |
| value: 71.28829558371541 |
| - type: mrr_at_100 |
| value: 71.33501465192626 |
| - type: mrr_at_1000 |
| value: 71.33501465192626 |
| - type: mrr_at_20 |
| value: 71.33145566423184 |
| - type: mrr_at_3 |
| value: 70.51501167891556 |
| - type: mrr_at_5 |
| value: 71.10978180368052 |
| - type: nauc_map_at_1000_diff1 |
| value: 16.85059525629914 |
| - type: nauc_map_at_1000_max |
| value: 7.2095269963540725 |
| - type: nauc_map_at_1000_std |
| value: -31.394151564475674 |
| - type: nauc_map_at_100_diff1 |
| value: 16.85059525629914 |
| - type: nauc_map_at_100_max |
| value: 7.2095269963540725 |
| - type: nauc_map_at_100_std |
| value: -31.394151564475674 |
| - type: nauc_map_at_10_diff1 |
| value: 16.005030500086086 |
| - type: nauc_map_at_10_max |
| value: 7.264507684893849 |
| - type: nauc_map_at_10_std |
| value: -31.715470923775076 |
| - type: nauc_map_at_1_diff1 |
| value: 88.69496772604496 |
| - type: nauc_map_at_1_max |
| value: 2.919830321081601 |
| - type: nauc_map_at_1_std |
| value: -12.316988109913101 |
| - type: nauc_map_at_20_diff1 |
| value: 16.782304707510036 |
| - type: nauc_map_at_20_max |
| value: 7.245475061050962 |
| - type: nauc_map_at_20_std |
| value: -31.399548485799105 |
| - type: nauc_map_at_3_diff1 |
| value: 0.7297119245486146 |
| - type: nauc_map_at_3_max |
| value: 8.421968652977062 |
| - type: nauc_map_at_3_std |
| value: -36.37536179543213 |
| - type: nauc_map_at_5_diff1 |
| value: 12.348535589464996 |
| - type: nauc_map_at_5_max |
| value: 7.675248350287251 |
| - type: nauc_map_at_5_std |
| value: -33.03849075726066 |
| - type: nauc_mrr_at_1000_diff1 |
| value: -19.145171066076582 |
| - type: nauc_mrr_at_1000_max |
| value: 3.456240689954698 |
| - type: nauc_mrr_at_1000_std |
| value: -16.911698657686483 |
| - type: nauc_mrr_at_100_diff1 |
| value: -19.145171066076582 |
| - type: nauc_mrr_at_100_max |
| value: 3.456240689954698 |
| - type: nauc_mrr_at_100_std |
| value: -16.911698657686483 |
| - type: nauc_mrr_at_10_diff1 |
| value: -19.383555832224395 |
| - type: nauc_mrr_at_10_max |
| value: 3.4956712263663245 |
| - type: nauc_mrr_at_10_std |
| value: -17.088724852125562 |
| - type: nauc_mrr_at_1_diff1 |
| value: -7.248689506871619 |
| - type: nauc_mrr_at_1_max |
| value: 1.6279165510593367 |
| - type: nauc_mrr_at_1_std |
| value: -10.76401320944665 |
| - type: nauc_mrr_at_20_diff1 |
| value: -19.166540939871457 |
| - type: nauc_mrr_at_20_max |
| value: 3.471598334930003 |
| - type: nauc_mrr_at_20_std |
| value: -16.91659546762138 |
| - type: nauc_mrr_at_3_diff1 |
| value: -23.156601373830686 |
| - type: nauc_mrr_at_3_max |
| value: 4.041242636869845 |
| - type: nauc_mrr_at_3_std |
| value: -19.05754313395974 |
| - type: nauc_mrr_at_5_diff1 |
| value: -20.312683178010765 |
| - type: nauc_mrr_at_5_max |
| value: 3.617044817759521 |
| - type: nauc_mrr_at_5_std |
| value: -17.611736247571123 |
| - type: nauc_ndcg_at_1000_diff1 |
| value: 12.200580013571008 |
| - type: nauc_ndcg_at_1000_max |
| value: 7.5728136486801105 |
| - type: nauc_ndcg_at_1000_std |
| value: -32.773584410427645 |
| - type: nauc_ndcg_at_100_diff1 |
| value: 12.200580013571008 |
| - type: nauc_ndcg_at_100_max |
| value: 7.5728136486801105 |
| - type: nauc_ndcg_at_100_std |
| value: -32.773584410427645 |
| - type: nauc_ndcg_at_10_diff1 |
| value: 8.346563719126793 |
| - type: nauc_ndcg_at_10_max |
| value: 7.984058762338721 |
| - type: nauc_ndcg_at_10_std |
| value: -34.20079429928775 |
| - type: nauc_ndcg_at_1_diff1 |
| value: 88.69496772604496 |
| - type: nauc_ndcg_at_1_max |
| value: 2.919830321081601 |
| - type: nauc_ndcg_at_1_std |
| value: -12.316988109913101 |
| - type: nauc_ndcg_at_20_diff1 |
| value: 11.618780757172203 |
| - type: nauc_ndcg_at_20_max |
| value: 7.879670436948672 |
| - type: nauc_ndcg_at_20_std |
| value: -32.832497615096266 |
| - type: nauc_ndcg_at_3_diff1 |
| value: -21.635858761258074 |
| - type: nauc_ndcg_at_3_max |
| value: 10.142935236930697 |
| - type: nauc_ndcg_at_3_std |
| value: -43.48822107044396 |
| - type: nauc_ndcg_at_5_diff1 |
| value: -1.4973636275297837 |
| - type: nauc_ndcg_at_5_max |
| value: 8.981298991414445 |
| - type: nauc_ndcg_at_5_std |
| value: -37.87476407099174 |
| - type: nauc_precision_at_1000_diff1 |
| value: .nan |
| - type: nauc_precision_at_1000_max |
| value: .nan |
| - type: nauc_precision_at_1000_std |
| value: .nan |
| - type: nauc_precision_at_100_diff1 |
| value: .nan |
| - type: nauc_precision_at_100_max |
| value: .nan |
| - type: nauc_precision_at_100_std |
| value: .nan |
| - type: nauc_precision_at_10_diff1 |
| value: -121.58511191301635 |
| - type: nauc_precision_at_10_max |
| value: 25.248667404560425 |
| - type: nauc_precision_at_10_std |
| value: -81.22406477400408 |
| - type: nauc_precision_at_1_diff1 |
| value: 88.69496772604496 |
| - type: nauc_precision_at_1_max |
| value: 2.919830321081601 |
| - type: nauc_precision_at_1_std |
| value: -12.316988109913101 |
| - type: nauc_precision_at_20_diff1 |
| value: -140.40859643368495 |
| - type: nauc_precision_at_20_max |
| value: 87.46836627148437 |
| - type: nauc_precision_at_20_std |
| value: -50.273460814697444 |
| - type: nauc_precision_at_3_diff1 |
| value: -116.53772258202089 |
| - type: nauc_precision_at_3_max |
| value: 17.67074016899205 |
| - type: nauc_precision_at_3_std |
| value: -74.36661207778327 |
| - type: nauc_precision_at_5_diff1 |
| value: -123.86337036708947 |
| - type: nauc_precision_at_5_max |
| value: 21.909069776986158 |
| - type: nauc_precision_at_5_std |
| value: -83.69114367867371 |
| - type: nauc_recall_at_1000_diff1 |
| value: .nan |
| - type: nauc_recall_at_1000_max |
| value: .nan |
| - type: nauc_recall_at_1000_std |
| value: .nan |
| - type: nauc_recall_at_100_diff1 |
| value: .nan |
| - type: nauc_recall_at_100_max |
| value: .nan |
| - type: nauc_recall_at_100_std |
| value: .nan |
| - type: nauc_recall_at_10_diff1 |
| value: -121.58511191302262 |
| - type: nauc_recall_at_10_max |
| value: 25.24866740455998 |
| - type: nauc_recall_at_10_std |
| value: -81.22406477400794 |
| - type: nauc_recall_at_1_diff1 |
| value: 88.69496772604496 |
| - type: nauc_recall_at_1_max |
| value: 2.919830321081601 |
| - type: nauc_recall_at_1_std |
| value: -12.316988109913101 |
| - type: nauc_recall_at_20_diff1 |
| value: -140.40859643373693 |
| - type: nauc_recall_at_20_max |
| value: 87.46836627148096 |
| - type: nauc_recall_at_20_std |
| value: -50.27346081473573 |
| - type: nauc_recall_at_3_diff1 |
| value: -116.53772258202 |
| - type: nauc_recall_at_3_max |
| value: 17.670740168992516 |
| - type: nauc_recall_at_3_std |
| value: -74.3666120777825 |
| - type: nauc_recall_at_5_diff1 |
| value: -123.86337036708981 |
| - type: nauc_recall_at_5_max |
| value: 21.909069776986765 |
| - type: nauc_recall_at_5_std |
| value: -83.69114367867463 |
| - type: ndcg_at_1 |
| value: 5.931 |
| - type: ndcg_at_10 |
| value: 62.871 |
| - type: ndcg_at_100 |
| value: 63.05 |
| - type: ndcg_at_1000 |
| value: 63.05 |
| - type: ndcg_at_20 |
| value: 63.027 |
| - type: ndcg_at_3 |
| value: 60.89 |
| - type: ndcg_at_5 |
| value: 62.382000000000005 |
| - type: precision_at_1 |
| value: 5.931 |
| - type: precision_at_10 |
| value: 9.927999999999999 |
| - type: precision_at_100 |
| value: 1.0 |
| - type: precision_at_1000 |
| value: 0.1 |
| - type: precision_at_20 |
| value: 4.994 |
| - type: precision_at_3 |
| value: 31.408 |
| - type: precision_at_5 |
| value: 19.559 |
| - type: recall_at_1 |
| value: 5.931 |
| - type: recall_at_10 |
| value: 99.28200000000001 |
| - type: recall_at_100 |
| value: 100.0 |
| - type: recall_at_1000 |
| value: 100.0 |
| - type: recall_at_20 |
| value: 99.88 |
| - type: recall_at_3 |
| value: 94.223 |
| - type: recall_at_5 |
| value: 97.795 |
| task: |
| type: Retrieval |
| tags: |
| - mteb |
| --- |
| # voyage-code-3 |
|
|
| This repo contains the tokenizer of the `voyage-code-3` embedding model. `voyage-code-3` is optimized for code retrieval, outperforming OpenAI-v3-large and CodeSage-large by an average of 13.80% and 16.81% on a suite of 238 code retrieval datasets, respectively. By supporting smaller dimensions with Matryoshka learning and quantized formats like int8 and binary, voyage-code-3 can also dramatically reduce storage and search costs with minimal impact on retrieval quality. Please refer to [our blogpost](https://blog.voyageai.com/2024/12/04/voyage-code-3) for more details about this model. |
|
|
| You can use `voyage-code-3` via the [Voyage API](https://docs.voyageai.com/docs/embeddings), [AWS SageMaker](https://aws.amazon.com/marketplace/pp/prodview-d5nri3kbddsrw) or on-prem deplotment. |
|
|
| ## Voyage API |
| The following code snippet shows the usage of the Voyage API. Install the Voyage AI Python SDK via: |
|
|
| ``` |
| pip install -U voyageai |
| ``` |
|
|
| Voyage AI utilizes API keys to monitor usage and manage permissions. To obtain your key, please sign in with your Voyage AI account and click the "Create new API key" button in the dashboard. We recommend setting the API key as an environment variable. For example, in MacOS or Linux, type the following command in the terminal, replacing <your secret key> with your actual API key: |
|
|
| ```python |
| import voyageai |
| |
| vo = voyageai.Client() |
| # This will automatically use the environment variable VOYAGE_API_KEY. |
| # Alternatively, you can use vo = voyageai.Client(api_key="<your secret key>") |
| |
| result = vo.embed( |
| texts=["hello world"], |
| model="voyage-code-3", |
| input_type="document", |
| output_dimension=2048, |
| output_dtype="float" |
| ) |
| ``` |
| ## AWS SageMaker |
|
|
| The embedding model can be privately deployed in your AWS Cloud using our [AWS SageMaker marketplace offering](https://aws.amazon.com/marketplace/pp/prodview-d5nri3kbddsrw). It runs privately in your VPC. |
|
|
| ## On-prem Deployment |
|
|
| Want to run `voyage-code-3` on your own hardware? Feel free to reach out to contact@voyageai.com to learn more. |
|
|
