README.md

metadata

license: apache-2.0
language:
  - en
library_name: sentence-transformers
pipeline_tag: sentence-similarity
base_model: dmis-lab/UltraBERT-150M
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - mteb
  - ultrabert
  - long-context
datasets:
  - lightonai/embeddings-pre-training-curated
  - lightonai/cornstack
  - lightonai/embeddings-fine-tuning-filtered-en
  - codefuse-ai/F2LLM
  - codefuse-ai/F2LLM-v2
  - nomic-ai/nomic-embed-supervised-data
  - jspringer/open-synthetic-embeddings
  - nvidia/embed-nemotron-dataset-v1
  - thesofakillers/jigsaw-toxic-comment-classification-challenge
  - ucberkeley-dlab/measuring-hate-speech
model-index:
  - name: UltraBERT-Embed-150M
    results:
      - task:
          type: Retrieval
        dataset:
          type: mteb/ARCChallenge
          name: MTEB ARCChallenge
          config: default
          split: test
          revision: 61b42fe57d9a44e30f47b9b878b664a95472ec80
        metrics:
          - type: ndcg_at_1
            value: 6.655
          - type: ndcg_at_3
            value: 11.92
          - type: ndcg_at_5
            value: 13.881
          - type: ndcg_at_10
            value: 16.149
          - type: ndcg_at_20
            value: 18.048
          - type: ndcg_at_100
            value: 21.94
          - type: ndcg_at_1000
            value: 25.838
          - type: map_at_1
            value: 6.655
          - type: map_at_3
            value: 10.538
          - type: map_at_5
            value: 11.621
          - type: map_at_10
            value: 12.545
          - type: map_at_20
            value: 13.067
          - type: map_at_100
            value: 13.569
          - type: map_at_1000
            value: 13.699
          - type: recall_at_1
            value: 6.655
          - type: recall_at_3
            value: 15.956
          - type: recall_at_5
            value: 20.734
          - type: recall_at_10
            value: 27.816
          - type: recall_at_20
            value: 35.324
          - type: recall_at_100
            value: 56.826
          - type: recall_at_1000
            value: 88.481
          - type: accuracy
            value: 6.655
          - type: precision_at_1
            value: 6.655
          - type: precision_at_3
            value: 5.319
          - type: precision_at_5
            value: 4.147
          - type: precision_at_10
            value: 2.782
          - type: precision_at_20
            value: 1.766
          - type: precision_at_100
            value: 0.568
          - type: precision_at_1000
            value: 0.088
          - type: mrr_at_1
            value: 6.6553
          - type: mrr_at_3
            value: 10.5375
          - type: mrr_at_5
            value: 11.6212
          - type: mrr_at_10
            value: 12.5453
          - type: mrr_at_20
            value: 13.0665
          - type: mrr_at_100
            value: 13.5688
          - type: mrr_at_1000
            value: 13.6991
          - type: hit_rate_at_1
            value: 6.655
          - type: hit_rate_at_3
            value: 15.956
          - type: hit_rate_at_5
            value: 20.734
          - type: hit_rate_at_10
            value: 27.816
          - type: hit_rate_at_20
            value: 35.324
          - type: hit_rate_at_100
            value: 56.826
          - type: hit_rate_at_1000
            value: 88.481
      - task:
          type: Retrieval
        dataset:
          type: mteb/AlphaNLI
          name: MTEB AlphaNLI
          config: default
          split: test
          revision: f145e68f0f5097f5d160c345eabeb2db8745eb13
        metrics:
          - type: ndcg_at_1
            value: 23.433
          - type: ndcg_at_3
            value: 31.969
          - type: ndcg_at_5
            value: 33.933
          - type: ndcg_at_10
            value: 36.088
          - type: ndcg_at_20
            value: 37.626
          - type: ndcg_at_100
            value: 39.847
          - type: ndcg_at_1000
            value: 41.75
          - type: map_at_1
            value: 23.433
          - type: map_at_3
            value: 29.906
          - type: map_at_5
            value: 30.997
          - type: map_at_10
            value: 31.9
          - type: map_at_20
            value: 32.323
          - type: map_at_100
            value: 32.624
          - type: map_at_1000
            value: 32.685
          - type: recall_at_1
            value: 23.433
          - type: recall_at_3
            value: 37.924
          - type: recall_at_5
            value: 42.689
          - type: recall_at_10
            value: 49.282
          - type: recall_at_20
            value: 55.352
          - type: recall_at_100
            value: 67.428
          - type: recall_at_1000
            value: 83.029
          - type: accuracy
            value: 23.433
          - type: precision_at_1
            value: 23.433
          - type: precision_at_3
            value: 12.641
          - type: precision_at_5
            value: 8.538
          - type: precision_at_10
            value: 4.928
          - type: precision_at_20
            value: 2.768
          - type: precision_at_100
            value: 0.674
          - type: precision_at_1000
            value: 0.083
          - type: mrr_at_1
            value: 23.4334
          - type: mrr_at_3
            value: 29.9064
          - type: mrr_at_5
            value: 30.9965
          - type: mrr_at_10
            value: 31.9001
          - type: mrr_at_20
            value: 32.3233
          - type: mrr_at_100
            value: 32.624
          - type: mrr_at_1000
            value: 32.6845
          - type: hit_rate_at_1
            value: 23.433
          - type: hit_rate_at_3
            value: 37.924
          - type: hit_rate_at_5
            value: 42.689
          - type: hit_rate_at_10
            value: 49.282
          - type: hit_rate_at_20
            value: 55.352
          - type: hit_rate_at_100
            value: 67.428
          - type: hit_rate_at_1000
            value: 83.029
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_counterfactual
          name: MTEB AmazonCounterfactualClassification (en)
          config: en
          split: test
          revision: 1f7e6a9d6fa6e64c53d146e428565640410c0df1
        metrics:
          - type: accuracy
            value: 84.7761
          - type: f1
            value: 79.5912
          - type: f1_weighted
            value: 85.8428
          - type: precision
            value: 77.1388
          - type: precision_weighted
            value: 88.8828
          - type: recall
            value: 85.6554
          - type: recall_weighted
            value: 84.7761
          - type: ap
            value: 52.8481
          - type: ap_weighted
            value: 52.8481
      - task:
          type: Retrieval
        dataset:
          type: CoIR-Retrieval/apps
          name: MTEB AppsRetrieval
          config: default
          split: test
          revision: f22508f96b7a36c2415181ed8bb76f76e04ae2d5
        metrics:
          - type: ndcg_at_1
            value: 56.175
          - type: ndcg_at_3
            value: 64.306
          - type: ndcg_at_5
            value: 66.307
          - type: ndcg_at_10
            value: 68.496
          - type: ndcg_at_20
            value: 69.739
          - type: ndcg_at_100
            value: 71.376
          - type: ndcg_at_1000
            value: 71.94
          - type: map_at_1
            value: 56.175
          - type: map_at_3
            value: 62.346
          - type: map_at_5
            value: 63.45
          - type: map_at_10
            value: 64.378
          - type: map_at_20
            value: 64.72
          - type: map_at_100
            value: 64.95
          - type: map_at_1000
            value: 64.972
          - type: recall_at_1
            value: 56.175
          - type: recall_at_3
            value: 69.96
          - type: recall_at_5
            value: 74.847
          - type: recall_at_10
            value: 81.487
          - type: recall_at_20
            value: 86.401
          - type: recall_at_100
            value: 95.166
          - type: recall_at_1000
            value: 99.548
          - type: accuracy
            value: 56.175
          - type: precision_at_1
            value: 56.175
          - type: precision_at_3
            value: 23.32
          - type: precision_at_5
            value: 14.969
          - type: precision_at_10
            value: 8.149
          - type: precision_at_20
            value: 4.32
          - type: precision_at_100
            value: 0.952
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 56.1753
          - type: mrr_at_3
            value: 62.3462
          - type: mrr_at_5
            value: 63.4498
          - type: mrr_at_10
            value: 64.3782
          - type: mrr_at_20
            value: 64.7196
          - type: mrr_at_100
            value: 64.9502
          - type: mrr_at_1000
            value: 64.9721
          - type: hit_rate_at_1
            value: 56.175
          - type: hit_rate_at_3
            value: 69.96
          - type: hit_rate_at_5
            value: 74.847
          - type: hit_rate_at_10
            value: 81.487
          - type: hit_rate_at_20
            value: 86.401
          - type: hit_rate_at_100
            value: 95.166
          - type: hit_rate_at_1000
            value: 99.548
      - task:
          type: Clustering
        dataset:
          type: mteb/arxiv-clustering-p2p
          name: MTEB ArXivHierarchicalClusteringP2P
          config: default
          split: test
          revision: 0bbdb47bcbe3a90093699aefeed338a0f28a7ee8
        metrics:
          - type: v_measure
            value: 62.9501
          - type: v_measure_std
            value: 3.3202
          - type: ami
            value: 62.7076
          - type: ami_std
            value: 3.451
      - task:
          type: Clustering
        dataset:
          type: mteb/arxiv-clustering-s2s
          name: MTEB ArXivHierarchicalClusteringS2S
          config: default
          split: test
          revision: b73bd54100e5abfa6e3a23dcafb46fe4d2438dc3
        metrics:
          - type: v_measure
            value: 60.3742
          - type: v_measure_std
            value: 2.1873
          - type: ami
            value: 60.1156
          - type: ami_std
            value: 2.3132
      - task:
          type: Retrieval
        dataset:
          type: mteb/arguana
          name: MTEB ArguAna
          config: default
          split: test
          revision: c22ab2a51041ffd869aaddef7af8d8215647e41a
        metrics:
          - type: ndcg_at_1
            value: 25.178
          - type: ndcg_at_3
            value: 39.161
          - type: ndcg_at_5
            value: 44.833
          - type: ndcg_at_10
            value: 50.431
          - type: ndcg_at_20
            value: 53.557
          - type: ndcg_at_100
            value: 54.842
          - type: ndcg_at_1000
            value: 54.935
          - type: map_at_1
            value: 25.178
          - type: map_at_3
            value: 35.669
          - type: map_at_5
            value: 38.844
          - type: map_at_10
            value: 41.146
          - type: map_at_20
            value: 42.031
          - type: map_at_100
            value: 42.24
          - type: map_at_1000
            value: 42.244
          - type: recall_at_1
            value: 25.178
          - type: recall_at_3
            value: 49.289
          - type: recall_at_5
            value: 62.945
          - type: recall_at_10
            value: 80.299
          - type: recall_at_20
            value: 92.461
          - type: recall_at_100
            value: 98.933
          - type: recall_at_1000
            value: 99.644
          - type: accuracy
            value: 25.178
          - type: precision_at_1
            value: 25.178
          - type: precision_at_3
            value: 16.43
          - type: precision_at_5
            value: 12.589
          - type: precision_at_10
            value: 8.03
          - type: precision_at_20
            value: 4.623
          - type: precision_at_100
            value: 0.989
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 25.1778
          - type: mrr_at_3
            value: 35.6686
          - type: mrr_at_5
            value: 38.8442
          - type: mrr_at_10
            value: 41.1463
          - type: mrr_at_20
            value: 42.0311
          - type: mrr_at_100
            value: 42.2405
          - type: mrr_at_1000
            value: 42.2444
          - type: hit_rate_at_1
            value: 25.178
          - type: hit_rate_at_3
            value: 49.289
          - type: hit_rate_at_5
            value: 62.945
          - type: hit_rate_at_10
            value: 80.299
          - type: hit_rate_at_20
            value: 92.461
          - type: hit_rate_at_100
            value: 98.933
          - type: hit_rate_at_1000
            value: 99.644
      - task:
          type: Reranking
        dataset:
          type: mteb/AskUbuntuDupQuestions
          name: MTEB AskUbuntuDupQuestions
          config: default
          split: test
          revision: c5691e3c48741d5f83b5cc8e630653d7a8cfc048
        metrics:
          - type: ndcg_at_1
            value: 65.097
          - type: ndcg_at_3
            value: 62.42
          - type: ndcg_at_5
            value: 62.556
          - type: ndcg_at_10
            value: 68.073
          - type: ndcg_at_20
            value: 77.914
          - type: ndcg_at_100
            value: 77.914
          - type: ndcg_at_1000
            value: 77.914
          - type: map_at_1
            value: 17.663
          - type: map_at_3
            value: 33.406
          - type: map_at_5
            value: 41.46
          - type: map_at_10
            value: 52.825
          - type: map_at_20
            value: 62.807
          - type: map_at_100
            value: 62.807
          - type: map_at_1000
            value: 62.807
          - type: recall_at_1
            value: 17.663
          - type: recall_at_3
            value: 38.892
          - type: recall_at_5
            value: 51.841
          - type: recall_at_10
            value: 74.746
          - type: recall_at_20
            value: 100
          - type: recall_at_100
            value: 100
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 17.663
          - type: precision_at_1
            value: 65.097
          - type: precision_at_3
            value: 54.663
          - type: precision_at_5
            value: 47.368
          - type: precision_at_10
            value: 37.59
          - type: precision_at_20
            value: 27.355
          - type: precision_at_100
            value: 5.471
          - type: precision_at_1000
            value: 0.547
          - type: mrr_at_1
            value: 65.097
          - type: mrr_at_3
            value: 74.9307
          - type: mrr_at_5
            value: 76.1773
          - type: mrr_at_10
            value: 76.8788
          - type: mrr_at_20
            value: 77.1017
          - type: mrr_at_100
            value: 77.1017
          - type: mrr_at_1000
            value: 77.1017
          - type: hit_rate_at_1
            value: 65.097
          - type: hit_rate_at_3
            value: 86.704
          - type: hit_rate_at_5
            value: 91.967
          - type: hit_rate_at_10
            value: 96.953
          - type: hit_rate_at_20
            value: 100
          - type: hit_rate_at_100
            value: 100
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: STS
        dataset:
          type: mteb/biosses-sts
          name: MTEB BIOSSES
          config: default
          split: test
          revision: d3fb88f8f02e40887cd149695127462bbcf29b4a
        metrics:
          - type: pearson
            value: 91.007
          - type: spearman
            value: 89.3394
          - type: cosine_pearson
            value: 91.007
          - type: cosine_spearman
            value: 89.3394
          - type: manhattan_pearson
            value: 87.748
          - type: manhattan_spearman
            value: 87.3743
          - type: euclidean_pearson
            value: 87.4415
          - type: euclidean_spearman
            value: 86.908
      - task:
          type: Classification
        dataset:
          type: mteb/banking77
          name: MTEB Banking77Classification
          config: default
          split: test
          revision: 0fd18e25b25c072e09e0d92ab615fda904d66300
        metrics:
          - type: accuracy
            value: 86.2305
          - type: f1
            value: 86.1753
          - type: f1_weighted
            value: 86.1753
          - type: precision
            value: 86.9383
          - type: precision_weighted
            value: 86.9383
          - type: recall
            value: 86.2305
          - type: recall_weighted
            value: 86.2305
          - type: ap
            value: .nan
          - type: ap_weighted
            value: .nan
      - task:
          type: Clustering
        dataset:
          type: mteb/biorxiv-clustering-p2p
          name: MTEB BiorxivClusteringP2P.v2
          config: default
          split: test
          revision: f5dbc242e11dd8e24def4c4268607a49e02946dc
        metrics:
          - type: v_measure
            value: 44.7978
          - type: v_measure_std
            value: 1.1772
          - type: ami
            value: 44.4665
          - type: ami_std
            value: 1.1836
      - task:
          type: Retrieval
        dataset:
          type: CoIR-Retrieval/CodeSearchNet
          name: MTEB COIRCodeSearchNetRetrieval (python)
          config: python
          split: test
          revision: 4adc7bc41202b5c13543c9c886a25f340634dab3
        metrics:
          - type: ndcg_at_1
            value: 90.642
          - type: ndcg_at_3
            value: 94.072
          - type: ndcg_at_5
            value: 94.512
          - type: ndcg_at_10
            value: 94.826
          - type: ndcg_at_20
            value: 94.956
          - type: ndcg_at_100
            value: 95.089
          - type: ndcg_at_1000
            value: 95.134
          - type: map_at_1
            value: 90.642
          - type: map_at_3
            value: 93.286
          - type: map_at_5
            value: 93.532
          - type: map_at_10
            value: 93.665
          - type: map_at_20
            value: 93.702
          - type: map_at_100
            value: 93.721
          - type: map_at_1000
            value: 93.723
          - type: recall_at_1
            value: 90.642
          - type: recall_at_3
            value: 96.327
          - type: recall_at_5
            value: 97.386
          - type: recall_at_10
            value: 98.338
          - type: recall_at_20
            value: 98.84
          - type: recall_at_100
            value: 99.551
          - type: recall_at_1000
            value: 99.906
          - type: accuracy
            value: 90.642
          - type: precision_at_1
            value: 90.642
          - type: precision_at_3
            value: 32.109
          - type: precision_at_5
            value: 19.477
          - type: precision_at_10
            value: 9.834
          - type: precision_at_20
            value: 4.942
          - type: precision_at_100
            value: 0.996
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 90.6422
          - type: mrr_at_3
            value: 93.2855
          - type: mrr_at_5
            value: 93.5319
          - type: mrr_at_10
            value: 93.6646
          - type: mrr_at_20
            value: 93.702
          - type: mrr_at_100
            value: 93.7208
          - type: mrr_at_1000
            value: 93.7226
          - type: hit_rate_at_1
            value: 90.642
          - type: hit_rate_at_3
            value: 96.327
          - type: hit_rate_at_5
            value: 97.386
          - type: hit_rate_at_10
            value: 98.338
          - type: hit_rate_at_20
            value: 98.84
          - type: hit_rate_at_100
            value: 99.551
          - type: hit_rate_at_1000
            value: 99.906
      - task:
          type: Retrieval
        dataset:
          type: CoIR-Retrieval/CodeSearchNet
          name: MTEB COIRCodeSearchNetRetrieval (javascript)
          config: javascript
          split: test
          revision: 4adc7bc41202b5c13543c9c886a25f340634dab3
        metrics:
          - type: ndcg_at_1
            value: 63.385
          - type: ndcg_at_3
            value: 70.542
          - type: ndcg_at_5
            value: 72.039
          - type: ndcg_at_10
            value: 73.182
          - type: ndcg_at_20
            value: 73.954
          - type: ndcg_at_100
            value: 74.914
          - type: ndcg_at_1000
            value: 75.492
          - type: map_at_1
            value: 63.385
          - type: map_at_3
            value: 68.839
          - type: map_at_5
            value: 69.673
          - type: map_at_10
            value: 70.142
          - type: map_at_20
            value: 70.355
          - type: map_at_100
            value: 70.492
          - type: map_at_1000
            value: 70.511
          - type: recall_at_1
            value: 63.385
          - type: recall_at_3
            value: 75.448
          - type: recall_at_5
            value: 79.064
          - type: recall_at_10
            value: 82.619
          - type: recall_at_20
            value: 85.658
          - type: recall_at_100
            value: 90.793
          - type: recall_at_1000
            value: 95.473
          - type: accuracy
            value: 63.385
          - type: precision_at_1
            value: 63.385
          - type: precision_at_3
            value: 25.149
          - type: precision_at_5
            value: 15.813
          - type: precision_at_10
            value: 8.262
          - type: precision_at_20
            value: 4.283
          - type: precision_at_100
            value: 0.908
          - type: precision_at_1000
            value: 0.095
          - type: mrr_at_1
            value: 63.385
          - type: mrr_at_3
            value: 68.8393
          - type: mrr_at_5
            value: 69.6734
          - type: mrr_at_10
            value: 70.142
          - type: mrr_at_20
            value: 70.3555
          - type: mrr_at_100
            value: 70.4916
          - type: mrr_at_1000
            value: 70.5112
          - type: hit_rate_at_1
            value: 63.385
          - type: hit_rate_at_3
            value: 75.448
          - type: hit_rate_at_5
            value: 79.064
          - type: hit_rate_at_10
            value: 82.619
          - type: hit_rate_at_20
            value: 85.658
          - type: hit_rate_at_100
            value: 90.793
          - type: hit_rate_at_1000
            value: 95.473
      - task:
          type: Retrieval
        dataset:
          type: CoIR-Retrieval/CodeSearchNet
          name: MTEB COIRCodeSearchNetRetrieval (go)
          config: go
          split: test
          revision: 4adc7bc41202b5c13543c9c886a25f340634dab3
        metrics:
          - type: ndcg_at_1
            value: 80.079
          - type: ndcg_at_3
            value: 85.629
          - type: ndcg_at_5
            value: 86.678
          - type: ndcg_at_10
            value: 87.527
          - type: ndcg_at_20
            value: 87.932
          - type: ndcg_at_100
            value: 88.363
          - type: ndcg_at_1000
            value: 88.496
          - type: map_at_1
            value: 80.079
          - type: map_at_3
            value: 84.322
          - type: map_at_5
            value: 84.907
          - type: map_at_10
            value: 85.265
          - type: map_at_20
            value: 85.379
          - type: map_at_100
            value: 85.441
          - type: map_at_1000
            value: 85.446
          - type: recall_at_1
            value: 80.079
          - type: recall_at_3
            value: 89.387
          - type: recall_at_5
            value: 91.923
          - type: recall_at_10
            value: 94.509
          - type: recall_at_20
            value: 96.085
          - type: recall_at_100
            value: 98.375
          - type: recall_at_1000
            value: 99.421
          - type: accuracy
            value: 80.079
          - type: precision_at_1
            value: 80.079
          - type: precision_at_3
            value: 29.796
          - type: precision_at_5
            value: 18.385
          - type: precision_at_10
            value: 9.451
          - type: precision_at_20
            value: 4.804
          - type: precision_at_100
            value: 0.984
          - type: precision_at_1000
            value: 0.099
          - type: mrr_at_1
            value: 80.0788
          - type: mrr_at_3
            value: 84.3224
          - type: mrr_at_5
            value: 84.9072
          - type: mrr_at_10
            value: 85.2646
          - type: mrr_at_20
            value: 85.3792
          - type: mrr_at_100
            value: 85.4413
          - type: mrr_at_1000
            value: 85.4463
          - type: hit_rate_at_1
            value: 80.079
          - type: hit_rate_at_3
            value: 89.387
          - type: hit_rate_at_5
            value: 91.923
          - type: hit_rate_at_10
            value: 94.509
          - type: hit_rate_at_20
            value: 96.085
          - type: hit_rate_at_100
            value: 98.375
          - type: hit_rate_at_1000
            value: 99.421
      - task:
          type: Retrieval
        dataset:
          type: CoIR-Retrieval/CodeSearchNet
          name: MTEB COIRCodeSearchNetRetrieval (ruby)
          config: ruby
          split: test
          revision: 4adc7bc41202b5c13543c9c886a25f340634dab3
        metrics:
          - type: ndcg_at_1
            value: 65.504
          - type: ndcg_at_3
            value: 73.219
          - type: ndcg_at_5
            value: 75.021
          - type: ndcg_at_10
            value: 76.279
          - type: ndcg_at_20
            value: 77.132
          - type: ndcg_at_100
            value: 77.953
          - type: ndcg_at_1000
            value: 78.348
          - type: map_at_1
            value: 65.504
          - type: map_at_3
            value: 71.359
          - type: map_at_5
            value: 72.362
          - type: map_at_10
            value: 72.882
          - type: map_at_20
            value: 73.121
          - type: map_at_100
            value: 73.244
          - type: map_at_1000
            value: 73.26
          - type: recall_at_1
            value: 65.504
          - type: recall_at_3
            value: 78.588
          - type: recall_at_5
            value: 82.95
          - type: recall_at_10
            value: 86.836
          - type: recall_at_20
            value: 90.167
          - type: recall_at_100
            value: 94.449
          - type: recall_at_1000
            value: 97.542
          - type: accuracy
            value: 65.504
          - type: precision_at_1
            value: 65.504
          - type: precision_at_3
            value: 26.196
          - type: precision_at_5
            value: 16.59
          - type: precision_at_10
            value: 8.684
          - type: precision_at_20
            value: 4.508
          - type: precision_at_100
            value: 0.944
          - type: precision_at_1000
            value: 0.098
          - type: mrr_at_1
            value: 65.5036
          - type: mrr_at_3
            value: 71.3587
          - type: mrr_at_5
            value: 72.3619
          - type: mrr_at_10
            value: 72.8818
          - type: mrr_at_20
            value: 73.1211
          - type: mrr_at_100
            value: 73.2444
          - type: mrr_at_1000
            value: 73.2595
          - type: hit_rate_at_1
            value: 65.504
          - type: hit_rate_at_3
            value: 78.588
          - type: hit_rate_at_5
            value: 82.95
          - type: hit_rate_at_10
            value: 86.836
          - type: hit_rate_at_20
            value: 90.167
          - type: hit_rate_at_100
            value: 94.449
          - type: hit_rate_at_1000
            value: 97.542
      - task:
          type: Retrieval
        dataset:
          type: CoIR-Retrieval/CodeSearchNet
          name: MTEB COIRCodeSearchNetRetrieval (java)
          config: java
          split: test
          revision: 4adc7bc41202b5c13543c9c886a25f340634dab3
        metrics:
          - type: ndcg_at_1
            value: 71.584
          - type: ndcg_at_3
            value: 78.311
          - type: ndcg_at_5
            value: 79.577
          - type: ndcg_at_10
            value: 80.516
          - type: ndcg_at_20
            value: 81.023
          - type: ndcg_at_100
            value: 81.678
          - type: ndcg_at_1000
            value: 82.004
          - type: map_at_1
            value: 71.584
          - type: map_at_3
            value: 76.724
          - type: map_at_5
            value: 77.428
          - type: map_at_10
            value: 77.816
          - type: map_at_20
            value: 77.958
          - type: map_at_100
            value: 78.049
          - type: map_at_1000
            value: 78.06
          - type: recall_at_1
            value: 71.584
          - type: recall_at_3
            value: 82.875
          - type: recall_at_5
            value: 85.942
          - type: recall_at_10
            value: 88.845
          - type: recall_at_20
            value: 90.826
          - type: recall_at_100
            value: 94.35
          - type: recall_at_1000
            value: 96.988
          - type: accuracy
            value: 71.584
          - type: precision_at_1
            value: 71.584
          - type: precision_at_3
            value: 27.625
          - type: precision_at_5
            value: 17.188
          - type: precision_at_10
            value: 8.885
          - type: precision_at_20
            value: 4.541
          - type: precision_at_100
            value: 0.943
          - type: precision_at_1000
            value: 0.097
          - type: mrr_at_1
            value: 71.5838
          - type: mrr_at_3
            value: 76.7245
          - type: mrr_at_5
            value: 77.4278
          - type: mrr_at_10
            value: 77.8159
          - type: mrr_at_20
            value: 77.9584
          - type: mrr_at_100
            value: 78.0491
          - type: mrr_at_1000
            value: 78.0603
          - type: hit_rate_at_1
            value: 71.584
          - type: hit_rate_at_3
            value: 82.875
          - type: hit_rate_at_5
            value: 85.942
          - type: hit_rate_at_10
            value: 88.845
          - type: hit_rate_at_20
            value: 90.826
          - type: hit_rate_at_100
            value: 94.35
          - type: hit_rate_at_1000
            value: 96.988
      - task:
          type: Retrieval
        dataset:
          type: CoIR-Retrieval/CodeSearchNet
          name: MTEB COIRCodeSearchNetRetrieval (php)
          config: php
          split: test
          revision: 4adc7bc41202b5c13543c9c886a25f340634dab3
        metrics:
          - type: ndcg_at_1
            value: 71.179
          - type: ndcg_at_3
            value: 78.52
          - type: ndcg_at_5
            value: 79.899
          - type: ndcg_at_10
            value: 81.067
          - type: ndcg_at_20
            value: 81.669
          - type: ndcg_at_100
            value: 82.4
          - type: ndcg_at_1000
            value: 82.697
          - type: map_at_1
            value: 71.179
          - type: map_at_3
            value: 76.771
          - type: map_at_5
            value: 77.538
          - type: map_at_10
            value: 78.025
          - type: map_at_20
            value: 78.191
          - type: map_at_100
            value: 78.296
          - type: map_at_1000
            value: 78.307
          - type: recall_at_1
            value: 71.179
          - type: recall_at_3
            value: 83.559
          - type: recall_at_5
            value: 86.899
          - type: recall_at_10
            value: 90.488
          - type: recall_at_20
            value: 92.857
          - type: recall_at_100
            value: 96.753
          - type: recall_at_1000
            value: 99.087
          - type: accuracy
            value: 71.179
          - type: precision_at_1
            value: 71.179
          - type: precision_at_3
            value: 27.853
          - type: precision_at_5
            value: 17.38
          - type: precision_at_10
            value: 9.049
          - type: precision_at_20
            value: 4.643
          - type: precision_at_100
            value: 0.968
          - type: precision_at_1000
            value: 0.099
          - type: mrr_at_1
            value: 71.1788
          - type: mrr_at_3
            value: 76.7708
          - type: mrr_at_5
            value: 77.5379
          - type: mrr_at_10
            value: 78.0247
          - type: mrr_at_20
            value: 78.1911
          - type: mrr_at_100
            value: 78.2957
          - type: mrr_at_1000
            value: 78.3069
          - type: hit_rate_at_1
            value: 71.179
          - type: hit_rate_at_3
            value: 83.559
          - type: hit_rate_at_5
            value: 86.899
          - type: hit_rate_at_10
            value: 90.488
          - type: hit_rate_at_20
            value: 92.857
          - type: hit_rate_at_100
            value: 96.753
          - type: hit_rate_at_1000
            value: 99.087
      - task:
          type: Retrieval
        dataset:
          type: mteb/cqadupstack-gaming
          name: MTEB CQADupstackGamingRetrieval
          config: default
          split: test
          revision: 4885aa143210c98657558c04aaf3dc47cfb54340
        metrics:
          - type: ndcg_at_1
            value: 50.533
          - type: ndcg_at_3
            value: 58.274
          - type: ndcg_at_5
            value: 60.755
          - type: ndcg_at_10
            value: 63.417
          - type: ndcg_at_20
            value: 65.246
          - type: ndcg_at_100
            value: 67.148
          - type: ndcg_at_1000
            value: 67.875
          - type: map_at_1
            value: 44.382
          - type: map_at_3
            value: 54.382
          - type: map_at_5
            value: 56.124
          - type: map_at_10
            value: 57.577
          - type: map_at_20
            value: 58.254
          - type: map_at_100
            value: 58.606
          - type: map_at_1000
            value: 58.646
          - type: recall_at_1
            value: 44.382
          - type: recall_at_3
            value: 63.397
          - type: recall_at_5
            value: 69.54
          - type: recall_at_10
            value: 77.053
          - type: recall_at_20
            value: 83.769
          - type: recall_at_100
            value: 92.838
          - type: recall_at_1000
            value: 97.865
          - type: accuracy
            value: 44.382
          - type: precision_at_1
            value: 50.533
          - type: precision_at_3
            value: 25.893
          - type: precision_at_5
            value: 17.492
          - type: precision_at_10
            value: 10.069
          - type: precision_at_20
            value: 5.608
          - type: precision_at_100
            value: 1.283
          - type: precision_at_1000
            value: 0.138
          - type: mrr_at_1
            value: 50.5329
          - type: mrr_at_3
            value: 58.6311
          - type: mrr_at_5
            value: 60.0136
          - type: mrr_at_10
            value: 60.9248
          - type: mrr_at_20
            value: 61.3268
          - type: mrr_at_100
            value: 61.5205
          - type: mrr_at_1000
            value: 61.5402
          - type: hit_rate_at_1
            value: 50.533
          - type: hit_rate_at_3
            value: 68.84
          - type: hit_rate_at_5
            value: 74.922
          - type: hit_rate_at_10
            value: 81.567
          - type: hit_rate_at_20
            value: 87.21
          - type: hit_rate_at_100
            value: 94.545
          - type: hit_rate_at_1000
            value: 98.433
      - task:
          type: Retrieval
        dataset:
          type: mteb/cqadupstack-unix
          name: MTEB CQADupstackUnixRetrieval
          config: default
          split: test
          revision: 6c6430d3a6d36f8d2a829195bc5dc94d7e063e53
        metrics:
          - type: ndcg_at_1
            value: 35.354
          - type: ndcg_at_3
            value: 39.915
          - type: ndcg_at_5
            value: 42.655
          - type: ndcg_at_10
            value: 45.653
          - type: ndcg_at_20
            value: 47.584
          - type: ndcg_at_100
            value: 50.676
          - type: ndcg_at_1000
            value: 52.698
          - type: map_at_1
            value: 30.407
          - type: map_at_3
            value: 36.743
          - type: map_at_5
            value: 38.58
          - type: map_at_10
            value: 39.969
          - type: map_at_20
            value: 40.575
          - type: map_at_100
            value: 41.104
          - type: map_at_1000
            value: 41.195
          - type: recall_at_1
            value: 30.407
          - type: recall_at_3
            value: 43.145
          - type: recall_at_5
            value: 49.958
          - type: recall_at_10
            value: 58.666
          - type: recall_at_20
            value: 65.66
          - type: recall_at_100
            value: 80.446
          - type: recall_at_1000
            value: 94.486
          - type: accuracy
            value: 30.407
          - type: precision_at_1
            value: 35.354
          - type: precision_at_3
            value: 17.973
          - type: precision_at_5
            value: 12.761
          - type: precision_at_10
            value: 7.761
          - type: precision_at_20
            value: 4.436
          - type: precision_at_100
            value: 1.145
          - type: precision_at_1000
            value: 0.143
          - type: mrr_at_1
            value: 35.3545
          - type: mrr_at_3
            value: 41.3557
          - type: mrr_at_5
            value: 42.9649
          - type: mrr_at_10
            value: 44.1532
          - type: mrr_at_20
            value: 44.6317
          - type: mrr_at_100
            value: 44.9615
          - type: mrr_at_1000
            value: 45.0061
          - type: hit_rate_at_1
            value: 35.354
          - type: hit_rate_at_3
            value: 49.16
          - type: hit_rate_at_5
            value: 56.157
          - type: hit_rate_at_10
            value: 65.205
          - type: hit_rate_at_20
            value: 72.201
          - type: hit_rate_at_100
            value: 85.261
          - type: hit_rate_at_1000
            value: 96.549
      - task:
          type: Retrieval
        dataset:
          type: mteb/ClimateFEVER_test_top_250_only_w_correct-v2
          name: MTEB ClimateFEVERHardNegatives
          config: default
          split: test
          revision: 3a309e201f3c2c4b13bd4a367a8f37eee2ec1d21
        metrics:
          - type: ndcg_at_1
            value: 30.9
          - type: ndcg_at_3
            value: 27.661
          - type: ndcg_at_5
            value: 30.043
          - type: ndcg_at_10
            value: 33.752
          - type: ndcg_at_20
            value: 37.035
          - type: ndcg_at_100
            value: 41.692
          - type: ndcg_at_1000
            value: 44.895
          - type: map_at_1
            value: 14.223
          - type: map_at_3
            value: 20.529
          - type: map_at_5
            value: 22.707
          - type: map_at_10
            value: 24.684
          - type: map_at_20
            value: 25.907
          - type: map_at_100
            value: 26.855
          - type: map_at_1000
            value: 27.054
          - type: recall_at_1
            value: 14.223
          - type: recall_at_3
            value: 25.793
          - type: recall_at_5
            value: 32.022
          - type: recall_at_10
            value: 40.008
          - type: recall_at_20
            value: 49.307
          - type: recall_at_100
            value: 67.038
          - type: recall_at_1000
            value: 84.555
          - type: accuracy
            value: 14.223
          - type: precision_at_1
            value: 30.9
          - type: precision_at_3
            value: 20.567
          - type: precision_at_5
            value: 16
          - type: precision_at_10
            value: 10.49
          - type: precision_at_20
            value: 6.645
          - type: precision_at_100
            value: 1.896
          - type: precision_at_1000
            value: 0.25
          - type: mrr_at_1
            value: 30.9
          - type: mrr_at_3
            value: 40.3167
          - type: mrr_at_5
            value: 42.4767
          - type: mrr_at_10
            value: 43.682
          - type: mrr_at_20
            value: 44.2838
          - type: mrr_at_100
            value: 44.5973
          - type: mrr_at_1000
            value: 44.6214
          - type: hit_rate_at_1
            value: 30.9
          - type: hit_rate_at_3
            value: 52.4
          - type: hit_rate_at_5
            value: 61.8
          - type: hit_rate_at_10
            value: 70.7
          - type: hit_rate_at_20
            value: 79
          - type: hit_rate_at_100
            value: 90.8
          - type: hit_rate_at_1000
            value: 95.9
      - task:
          type: Retrieval
        dataset:
          type: cassanof/CodeEditSearch
          name: MTEB CodeEditSearchRetrieval (python)
          config: python
          split: train
          revision: 4e51c66e0939303f6928472f13ad0848b2a3f4c0
        metrics:
          - type: ndcg_at_1
            value: 55
          - type: ndcg_at_3
            value: 61.9
          - type: ndcg_at_5
            value: 63.704
          - type: ndcg_at_10
            value: 65.002
          - type: ndcg_at_20
            value: 65.914
          - type: ndcg_at_100
            value: 67.946
          - type: ndcg_at_1000
            value: 69.228
          - type: map_at_1
            value: 55
          - type: map_at_3
            value: 60.267
          - type: map_at_5
            value: 61.287
          - type: map_at_10
            value: 61.826
          - type: map_at_20
            value: 62.077
          - type: map_at_100
            value: 62.341
          - type: map_at_1000
            value: 62.386
          - type: recall_at_1
            value: 55
          - type: recall_at_3
            value: 66.6
          - type: recall_at_5
            value: 70.9
          - type: recall_at_10
            value: 74.9
          - type: recall_at_20
            value: 78.5
          - type: recall_at_100
            value: 89.7
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 55
          - type: precision_at_1
            value: 55
          - type: precision_at_3
            value: 22.2
          - type: precision_at_5
            value: 14.18
          - type: precision_at_10
            value: 7.49
          - type: precision_at_20
            value: 3.925
          - type: precision_at_100
            value: 0.897
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 55
          - type: mrr_at_3
            value: 60.2667
          - type: mrr_at_5
            value: 61.2867
          - type: mrr_at_10
            value: 61.8262
          - type: mrr_at_20
            value: 62.0771
          - type: mrr_at_100
            value: 62.3412
          - type: mrr_at_1000
            value: 62.3857
          - type: hit_rate_at_1
            value: 55
          - type: hit_rate_at_3
            value: 66.6
          - type: hit_rate_at_5
            value: 70.9
          - type: hit_rate_at_10
            value: 74.9
          - type: hit_rate_at_20
            value: 78.5
          - type: hit_rate_at_100
            value: 89.7
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: cassanof/CodeEditSearch
          name: MTEB CodeEditSearchRetrieval (javascript)
          config: javascript
          split: train
          revision: 4e51c66e0939303f6928472f13ad0848b2a3f4c0
        metrics:
          - type: ndcg_at_1
            value: 52.9
          - type: ndcg_at_3
            value: 60.624
          - type: ndcg_at_5
            value: 62.34
          - type: ndcg_at_10
            value: 63.918
          - type: ndcg_at_20
            value: 65.018
          - type: ndcg_at_100
            value: 66.693
          - type: ndcg_at_1000
            value: 68.039
          - type: map_at_1
            value: 52.9
          - type: map_at_3
            value: 58.7
          - type: map_at_5
            value: 59.645
          - type: map_at_10
            value: 60.292
          - type: map_at_20
            value: 60.601
          - type: map_at_100
            value: 60.82
          - type: map_at_1000
            value: 60.86
          - type: recall_at_1
            value: 52.9
          - type: recall_at_3
            value: 66.2
          - type: recall_at_5
            value: 70.4
          - type: recall_at_10
            value: 75.3
          - type: recall_at_20
            value: 79.6
          - type: recall_at_100
            value: 88.8
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 52.9
          - type: precision_at_1
            value: 52.9
          - type: precision_at_3
            value: 22.067
          - type: precision_at_5
            value: 14.08
          - type: precision_at_10
            value: 7.53
          - type: precision_at_20
            value: 3.98
          - type: precision_at_100
            value: 0.888
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 52.9
          - type: mrr_at_3
            value: 58.7
          - type: mrr_at_5
            value: 59.645
          - type: mrr_at_10
            value: 60.292
          - type: mrr_at_20
            value: 60.6006
          - type: mrr_at_100
            value: 60.8196
          - type: mrr_at_1000
            value: 60.8604
          - type: hit_rate_at_1
            value: 52.9
          - type: hit_rate_at_3
            value: 66.2
          - type: hit_rate_at_5
            value: 70.4
          - type: hit_rate_at_10
            value: 75.3
          - type: hit_rate_at_20
            value: 79.6
          - type: hit_rate_at_100
            value: 88.8
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: cassanof/CodeEditSearch
          name: MTEB CodeEditSearchRetrieval (typescript)
          config: typescript
          split: train
          revision: 4e51c66e0939303f6928472f13ad0848b2a3f4c0
        metrics:
          - type: ndcg_at_1
            value: 59.2
          - type: ndcg_at_3
            value: 66.147
          - type: ndcg_at_5
            value: 67.516
          - type: ndcg_at_10
            value: 68.838
          - type: ndcg_at_20
            value: 69.851
          - type: ndcg_at_100
            value: 71.302
          - type: ndcg_at_1000
            value: 72.467
          - type: map_at_1
            value: 59.2
          - type: map_at_3
            value: 64.467
          - type: map_at_5
            value: 65.232
          - type: map_at_10
            value: 65.775
          - type: map_at_20
            value: 66.054
          - type: map_at_100
            value: 66.243
          - type: map_at_1000
            value: 66.279
          - type: recall_at_1
            value: 59.2
          - type: recall_at_3
            value: 71
          - type: recall_at_5
            value: 74.3
          - type: recall_at_10
            value: 78.4
          - type: recall_at_20
            value: 82.4
          - type: recall_at_100
            value: 90.4
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 59.2
          - type: precision_at_1
            value: 59.2
          - type: precision_at_3
            value: 23.667
          - type: precision_at_5
            value: 14.86
          - type: precision_at_10
            value: 7.84
          - type: precision_at_20
            value: 4.12
          - type: precision_at_100
            value: 0.904
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 59.2
          - type: mrr_at_3
            value: 64.4667
          - type: mrr_at_5
            value: 65.2317
          - type: mrr_at_10
            value: 65.7752
          - type: mrr_at_20
            value: 66.0537
          - type: mrr_at_100
            value: 66.2427
          - type: mrr_at_1000
            value: 66.2794
          - type: hit_rate_at_1
            value: 59.2
          - type: hit_rate_at_3
            value: 71
          - type: hit_rate_at_5
            value: 74.3
          - type: hit_rate_at_10
            value: 78.4
          - type: hit_rate_at_20
            value: 82.4
          - type: hit_rate_at_100
            value: 90.4
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: cassanof/CodeEditSearch
          name: MTEB CodeEditSearchRetrieval (go)
          config: go
          split: train
          revision: 4e51c66e0939303f6928472f13ad0848b2a3f4c0
        metrics:
          - type: ndcg_at_1
            value: 55
          - type: ndcg_at_3
            value: 61.206
          - type: ndcg_at_5
            value: 63.228
          - type: ndcg_at_10
            value: 64.808
          - type: ndcg_at_20
            value: 65.687
          - type: ndcg_at_100
            value: 67.154
          - type: ndcg_at_1000
            value: 68.735
          - type: map_at_1
            value: 55
          - type: map_at_3
            value: 59.717
          - type: map_at_5
            value: 60.842
          - type: map_at_10
            value: 61.491
          - type: map_at_20
            value: 61.728
          - type: map_at_100
            value: 61.917
          - type: map_at_1000
            value: 61.965
          - type: recall_at_1
            value: 55
          - type: recall_at_3
            value: 65.5
          - type: recall_at_5
            value: 70.4
          - type: recall_at_10
            value: 75.3
          - type: recall_at_20
            value: 78.8
          - type: recall_at_100
            value: 86.9
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 55
          - type: precision_at_1
            value: 55
          - type: precision_at_3
            value: 21.833
          - type: precision_at_5
            value: 14.08
          - type: precision_at_10
            value: 7.53
          - type: precision_at_20
            value: 3.94
          - type: precision_at_100
            value: 0.869
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 55
          - type: mrr_at_3
            value: 59.7167
          - type: mrr_at_5
            value: 60.8417
          - type: mrr_at_10
            value: 61.4908
          - type: mrr_at_20
            value: 61.7284
          - type: mrr_at_100
            value: 61.9172
          - type: mrr_at_1000
            value: 61.9649
          - type: hit_rate_at_1
            value: 55
          - type: hit_rate_at_3
            value: 65.5
          - type: hit_rate_at_5
            value: 70.4
          - type: hit_rate_at_10
            value: 75.3
          - type: hit_rate_at_20
            value: 78.8
          - type: hit_rate_at_100
            value: 86.9
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: cassanof/CodeEditSearch
          name: MTEB CodeEditSearchRetrieval (ruby)
          config: ruby
          split: train
          revision: 4e51c66e0939303f6928472f13ad0848b2a3f4c0
        metrics:
          - type: ndcg_at_1
            value: 57
          - type: ndcg_at_3
            value: 62.601
          - type: ndcg_at_5
            value: 64.12
          - type: ndcg_at_10
            value: 65.704
          - type: ndcg_at_20
            value: 66.703
          - type: ndcg_at_100
            value: 68.392
          - type: ndcg_at_1000
            value: 69.888
          - type: map_at_1
            value: 57
          - type: map_at_3
            value: 61.25
          - type: map_at_5
            value: 62.09
          - type: map_at_10
            value: 62.743
          - type: map_at_20
            value: 63.024
          - type: map_at_100
            value: 63.258
          - type: map_at_1000
            value: 63.312
          - type: recall_at_1
            value: 57
          - type: recall_at_3
            value: 66.5
          - type: recall_at_5
            value: 70.2
          - type: recall_at_10
            value: 75.1
          - type: recall_at_20
            value: 79
          - type: recall_at_100
            value: 88.1
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 57
          - type: precision_at_1
            value: 57
          - type: precision_at_3
            value: 22.167
          - type: precision_at_5
            value: 14.04
          - type: precision_at_10
            value: 7.51
          - type: precision_at_20
            value: 3.95
          - type: precision_at_100
            value: 0.881
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 57
          - type: mrr_at_3
            value: 61.25
          - type: mrr_at_5
            value: 62.09
          - type: mrr_at_10
            value: 62.7427
          - type: mrr_at_20
            value: 63.0244
          - type: mrr_at_100
            value: 63.2583
          - type: mrr_at_1000
            value: 63.3121
          - type: hit_rate_at_1
            value: 57
          - type: hit_rate_at_3
            value: 66.5
          - type: hit_rate_at_5
            value: 70.2
          - type: hit_rate_at_10
            value: 75.1
          - type: hit_rate_at_20
            value: 79
          - type: hit_rate_at_100
            value: 88.1
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: cassanof/CodeEditSearch
          name: MTEB CodeEditSearchRetrieval (java)
          config: java
          split: train
          revision: 4e51c66e0939303f6928472f13ad0848b2a3f4c0
        metrics:
          - type: ndcg_at_1
            value: 59
          - type: ndcg_at_3
            value: 64.377
          - type: ndcg_at_5
            value: 66.155
          - type: ndcg_at_10
            value: 67.214
          - type: ndcg_at_20
            value: 68.326
          - type: ndcg_at_100
            value: 69.854
          - type: ndcg_at_1000
            value: 71.242
          - type: map_at_1
            value: 59
          - type: map_at_3
            value: 63.117
          - type: map_at_5
            value: 64.107
          - type: map_at_10
            value: 64.54
          - type: map_at_20
            value: 64.844
          - type: map_at_100
            value: 65.038
          - type: map_at_1000
            value: 65.08
          - type: recall_at_1
            value: 59
          - type: recall_at_3
            value: 68
          - type: recall_at_5
            value: 72.3
          - type: recall_at_10
            value: 75.6
          - type: recall_at_20
            value: 80
          - type: recall_at_100
            value: 88.5
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 59
          - type: precision_at_1
            value: 59
          - type: precision_at_3
            value: 22.667
          - type: precision_at_5
            value: 14.46
          - type: precision_at_10
            value: 7.56
          - type: precision_at_20
            value: 4
          - type: precision_at_100
            value: 0.885
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 59
          - type: mrr_at_3
            value: 63.1167
          - type: mrr_at_5
            value: 64.1067
          - type: mrr_at_10
            value: 64.5398
          - type: mrr_at_20
            value: 64.8443
          - type: mrr_at_100
            value: 65.0381
          - type: mrr_at_1000
            value: 65.0799
          - type: hit_rate_at_1
            value: 59
          - type: hit_rate_at_3
            value: 68
          - type: hit_rate_at_5
            value: 72.3
          - type: hit_rate_at_10
            value: 75.6
          - type: hit_rate_at_20
            value: 80
          - type: hit_rate_at_100
            value: 88.5
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: cassanof/CodeEditSearch
          name: MTEB CodeEditSearchRetrieval (php)
          config: php
          split: train
          revision: 4e51c66e0939303f6928472f13ad0848b2a3f4c0
        metrics:
          - type: ndcg_at_1
            value: 51.9
          - type: ndcg_at_3
            value: 58.025
          - type: ndcg_at_5
            value: 60.469
          - type: ndcg_at_10
            value: 61.971
          - type: ndcg_at_20
            value: 63.223
          - type: ndcg_at_100
            value: 65.082
          - type: ndcg_at_1000
            value: 66.538
          - type: map_at_1
            value: 51.9
          - type: map_at_3
            value: 56.483
          - type: map_at_5
            value: 57.848
          - type: map_at_10
            value: 58.476
          - type: map_at_20
            value: 58.813
          - type: map_at_100
            value: 59.06
          - type: map_at_1000
            value: 59.109
          - type: recall_at_1
            value: 51.9
          - type: recall_at_3
            value: 62.5
          - type: recall_at_5
            value: 68.4
          - type: recall_at_10
            value: 73
          - type: recall_at_20
            value: 78
          - type: recall_at_100
            value: 88.2
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 51.9
          - type: precision_at_1
            value: 51.9
          - type: precision_at_3
            value: 20.833
          - type: precision_at_5
            value: 13.68
          - type: precision_at_10
            value: 7.3
          - type: precision_at_20
            value: 3.9
          - type: precision_at_100
            value: 0.882
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 51.9
          - type: mrr_at_3
            value: 56.4833
          - type: mrr_at_5
            value: 57.8483
          - type: mrr_at_10
            value: 58.476
          - type: mrr_at_20
            value: 58.8133
          - type: mrr_at_100
            value: 59.0602
          - type: mrr_at_1000
            value: 59.1089
          - type: hit_rate_at_1
            value: 51.9
          - type: hit_rate_at_3
            value: 62.5
          - type: hit_rate_at_5
            value: 68.4
          - type: hit_rate_at_10
            value: 73
          - type: hit_rate_at_20
            value: 78
          - type: hit_rate_at_100
            value: 88.2
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: cassanof/CodeEditSearch
          name: MTEB CodeEditSearchRetrieval (c)
          config: c
          split: train
          revision: 4e51c66e0939303f6928472f13ad0848b2a3f4c0
        metrics:
          - type: ndcg_at_1
            value: 46.8
          - type: ndcg_at_3
            value: 52.806
          - type: ndcg_at_5
            value: 54.153
          - type: ndcg_at_10
            value: 55.811
          - type: ndcg_at_20
            value: 56.798
          - type: ndcg_at_100
            value: 58.674
          - type: ndcg_at_1000
            value: 61.137
          - type: map_at_1
            value: 46.8
          - type: map_at_3
            value: 51.383
          - type: map_at_5
            value: 52.123
          - type: map_at_10
            value: 52.813
          - type: map_at_20
            value: 53.085
          - type: map_at_100
            value: 53.326
          - type: map_at_1000
            value: 53.402
          - type: recall_at_1
            value: 46.8
          - type: recall_at_3
            value: 56.9
          - type: recall_at_5
            value: 60.2
          - type: recall_at_10
            value: 65.3
          - type: recall_at_20
            value: 69.2
          - type: recall_at_100
            value: 79.6
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 46.8
          - type: precision_at_1
            value: 46.8
          - type: precision_at_3
            value: 18.967
          - type: precision_at_5
            value: 12.04
          - type: precision_at_10
            value: 6.53
          - type: precision_at_20
            value: 3.46
          - type: precision_at_100
            value: 0.796
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 46.8
          - type: mrr_at_3
            value: 51.3833
          - type: mrr_at_5
            value: 52.1233
          - type: mrr_at_10
            value: 52.8133
          - type: mrr_at_20
            value: 53.0852
          - type: mrr_at_100
            value: 53.3263
          - type: mrr_at_1000
            value: 53.4015
          - type: hit_rate_at_1
            value: 46.8
          - type: hit_rate_at_3
            value: 56.9
          - type: hit_rate_at_5
            value: 60.2
          - type: hit_rate_at_10
            value: 65.3
          - type: hit_rate_at_20
            value: 69.2
          - type: hit_rate_at_100
            value: 79.6
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: cassanof/CodeEditSearch
          name: MTEB CodeEditSearchRetrieval (c++)
          config: c++
          split: train
          revision: 4e51c66e0939303f6928472f13ad0848b2a3f4c0
        metrics:
          - type: ndcg_at_1
            value: 49.2
          - type: ndcg_at_3
            value: 56.789
          - type: ndcg_at_5
            value: 58.015
          - type: ndcg_at_10
            value: 59.559
          - type: ndcg_at_20
            value: 60.989
          - type: ndcg_at_100
            value: 62.867
          - type: ndcg_at_1000
            value: 64.568
          - type: map_at_1
            value: 49.2
          - type: map_at_3
            value: 54.95
          - type: map_at_5
            value: 55.625
          - type: map_at_10
            value: 56.257
          - type: map_at_20
            value: 56.644
          - type: map_at_100
            value: 56.891
          - type: map_at_1000
            value: 56.944
          - type: recall_at_1
            value: 49.2
          - type: recall_at_3
            value: 62.1
          - type: recall_at_5
            value: 65.1
          - type: recall_at_10
            value: 69.9
          - type: recall_at_20
            value: 75.6
          - type: recall_at_100
            value: 85.9
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 49.2
          - type: precision_at_1
            value: 49.2
          - type: precision_at_3
            value: 20.7
          - type: precision_at_5
            value: 13.02
          - type: precision_at_10
            value: 6.99
          - type: precision_at_20
            value: 3.78
          - type: precision_at_100
            value: 0.859
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 49.2
          - type: mrr_at_3
            value: 54.95
          - type: mrr_at_5
            value: 55.625
          - type: mrr_at_10
            value: 56.2566
          - type: mrr_at_20
            value: 56.6436
          - type: mrr_at_100
            value: 56.8913
          - type: mrr_at_1000
            value: 56.9436
          - type: hit_rate_at_1
            value: 49.2
          - type: hit_rate_at_3
            value: 62.1
          - type: hit_rate_at_5
            value: 65.1
          - type: hit_rate_at_10
            value: 69.9
          - type: hit_rate_at_20
            value: 75.6
          - type: hit_rate_at_100
            value: 85.9
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: cassanof/CodeEditSearch
          name: MTEB CodeEditSearchRetrieval (rust)
          config: rust
          split: train
          revision: 4e51c66e0939303f6928472f13ad0848b2a3f4c0
        metrics:
          - type: ndcg_at_1
            value: 46
          - type: ndcg_at_3
            value: 51.93
          - type: ndcg_at_5
            value: 53.651
          - type: ndcg_at_10
            value: 55.111
          - type: ndcg_at_20
            value: 56.559
          - type: ndcg_at_100
            value: 58.792
          - type: ndcg_at_1000
            value: 60.929
          - type: map_at_1
            value: 46
          - type: map_at_3
            value: 50.517
          - type: map_at_5
            value: 51.467
          - type: map_at_10
            value: 52.072
          - type: map_at_20
            value: 52.46
          - type: map_at_100
            value: 52.765
          - type: map_at_1000
            value: 52.836
          - type: recall_at_1
            value: 46
          - type: recall_at_3
            value: 56
          - type: recall_at_5
            value: 60.2
          - type: recall_at_10
            value: 64.7
          - type: recall_at_20
            value: 70.5
          - type: recall_at_100
            value: 82.6
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 46
          - type: precision_at_1
            value: 46
          - type: precision_at_3
            value: 18.667
          - type: precision_at_5
            value: 12.04
          - type: precision_at_10
            value: 6.47
          - type: precision_at_20
            value: 3.525
          - type: precision_at_100
            value: 0.826
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 46
          - type: mrr_at_3
            value: 50.5167
          - type: mrr_at_5
            value: 51.4667
          - type: mrr_at_10
            value: 52.0723
          - type: mrr_at_20
            value: 52.4604
          - type: mrr_at_100
            value: 52.7653
          - type: mrr_at_1000
            value: 52.836
          - type: hit_rate_at_1
            value: 46
          - type: hit_rate_at_3
            value: 56
          - type: hit_rate_at_5
            value: 60.2
          - type: hit_rate_at_10
            value: 64.7
          - type: hit_rate_at_20
            value: 70.5
          - type: hit_rate_at_100
            value: 82.6
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: cassanof/CodeEditSearch
          name: MTEB CodeEditSearchRetrieval (swift)
          config: swift
          split: train
          revision: 4e51c66e0939303f6928472f13ad0848b2a3f4c0
        metrics:
          - type: ndcg_at_1
            value: 53.5
          - type: ndcg_at_3
            value: 60.574
          - type: ndcg_at_5
            value: 62.351
          - type: ndcg_at_10
            value: 63.931
          - type: ndcg_at_20
            value: 65.096
          - type: ndcg_at_100
            value: 66.518
          - type: ndcg_at_1000
            value: 68.106
          - type: map_at_1
            value: 53.5
          - type: map_at_3
            value: 58.867
          - type: map_at_5
            value: 59.857
          - type: map_at_10
            value: 60.526
          - type: map_at_20
            value: 60.847
          - type: map_at_100
            value: 61.04
          - type: map_at_1000
            value: 61.089
          - type: recall_at_1
            value: 53.5
          - type: recall_at_3
            value: 65.5
          - type: recall_at_5
            value: 69.8
          - type: recall_at_10
            value: 74.6
          - type: recall_at_20
            value: 79.2
          - type: recall_at_100
            value: 86.9
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 53.5
          - type: precision_at_1
            value: 53.5
          - type: precision_at_3
            value: 21.833
          - type: precision_at_5
            value: 13.96
          - type: precision_at_10
            value: 7.46
          - type: precision_at_20
            value: 3.96
          - type: precision_at_100
            value: 0.869
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 53.5
          - type: mrr_at_3
            value: 58.8667
          - type: mrr_at_5
            value: 59.8567
          - type: mrr_at_10
            value: 60.5258
          - type: mrr_at_20
            value: 60.8471
          - type: mrr_at_100
            value: 61.0404
          - type: mrr_at_1000
            value: 61.0886
          - type: hit_rate_at_1
            value: 53.5
          - type: hit_rate_at_3
            value: 65.5
          - type: hit_rate_at_5
            value: 69.8
          - type: hit_rate_at_10
            value: 74.6
          - type: hit_rate_at_20
            value: 79.2
          - type: hit_rate_at_100
            value: 86.9
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: cassanof/CodeEditSearch
          name: MTEB CodeEditSearchRetrieval (scala)
          config: scala
          split: train
          revision: 4e51c66e0939303f6928472f13ad0848b2a3f4c0
        metrics:
          - type: ndcg_at_1
            value: 46
          - type: ndcg_at_3
            value: 53.695
          - type: ndcg_at_5
            value: 56.2
          - type: ndcg_at_10
            value: 58.235
          - type: ndcg_at_20
            value: 59.909
          - type: ndcg_at_100
            value: 61.916
          - type: ndcg_at_1000
            value: 63.232
          - type: map_at_1
            value: 46
          - type: map_at_3
            value: 51.733
          - type: map_at_5
            value: 53.143
          - type: map_at_10
            value: 54.001
          - type: map_at_20
            value: 54.45
          - type: map_at_100
            value: 54.722
          - type: map_at_1000
            value: 54.765
          - type: recall_at_1
            value: 46
          - type: recall_at_3
            value: 59.4
          - type: recall_at_5
            value: 65.4
          - type: recall_at_10
            value: 71.6
          - type: recall_at_20
            value: 78.3
          - type: recall_at_100
            value: 89.2
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 46
          - type: precision_at_1
            value: 46
          - type: precision_at_3
            value: 19.8
          - type: precision_at_5
            value: 13.08
          - type: precision_at_10
            value: 7.16
          - type: precision_at_20
            value: 3.915
          - type: precision_at_100
            value: 0.892
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 46
          - type: mrr_at_3
            value: 51.7333
          - type: mrr_at_5
            value: 53.1433
          - type: mrr_at_10
            value: 54.0009
          - type: mrr_at_20
            value: 54.4497
          - type: mrr_at_100
            value: 54.7219
          - type: mrr_at_1000
            value: 54.7647
          - type: hit_rate_at_1
            value: 46
          - type: hit_rate_at_3
            value: 59.4
          - type: hit_rate_at_5
            value: 65.4
          - type: hit_rate_at_10
            value: 71.6
          - type: hit_rate_at_20
            value: 78.3
          - type: hit_rate_at_100
            value: 89.2
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: cassanof/CodeEditSearch
          name: MTEB CodeEditSearchRetrieval (shell)
          config: shell
          split: train
          revision: 4e51c66e0939303f6928472f13ad0848b2a3f4c0
        metrics:
          - type: ndcg_at_1
            value: 47.3
          - type: ndcg_at_3
            value: 54.487
          - type: ndcg_at_5
            value: 56.341
          - type: ndcg_at_10
            value: 58.611
          - type: ndcg_at_20
            value: 59.769
          - type: ndcg_at_100
            value: 61.924
          - type: ndcg_at_1000
            value: 63.473
          - type: map_at_1
            value: 47.3
          - type: map_at_3
            value: 52.75
          - type: map_at_5
            value: 53.78
          - type: map_at_10
            value: 54.719
          - type: map_at_20
            value: 55.048
          - type: map_at_100
            value: 55.333
          - type: map_at_1000
            value: 55.382
          - type: recall_at_1
            value: 47.3
          - type: recall_at_3
            value: 59.5
          - type: recall_at_5
            value: 64
          - type: recall_at_10
            value: 71
          - type: recall_at_20
            value: 75.5
          - type: recall_at_100
            value: 87.3
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 47.3
          - type: precision_at_1
            value: 47.3
          - type: precision_at_3
            value: 19.833
          - type: precision_at_5
            value: 12.8
          - type: precision_at_10
            value: 7.1
          - type: precision_at_20
            value: 3.775
          - type: precision_at_100
            value: 0.873
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 47.3
          - type: mrr_at_3
            value: 52.75
          - type: mrr_at_5
            value: 53.78
          - type: mrr_at_10
            value: 54.7192
          - type: mrr_at_20
            value: 55.0481
          - type: mrr_at_100
            value: 55.3328
          - type: mrr_at_1000
            value: 55.3818
          - type: hit_rate_at_1
            value: 47.3
          - type: hit_rate_at_3
            value: 59.5
          - type: hit_rate_at_5
            value: 64
          - type: hit_rate_at_10
            value: 71
          - type: hit_rate_at_20
            value: 75.5
          - type: hit_rate_at_100
            value: 87.3
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: CoIR-Retrieval/codefeedback-mt
          name: MTEB CodeFeedbackMT
          config: default
          split: test
          revision: b0f12fa0c0dd67f59c95a5c33d02aeeb4c398c5f
        metrics:
          - type: ndcg_at_1
            value: 76.177
          - type: ndcg_at_3
            value: 82.191
          - type: ndcg_at_5
            value: 83.3
          - type: ndcg_at_10
            value: 84.37
          - type: ndcg_at_20
            value: 84.893
          - type: ndcg_at_100
            value: 85.524
          - type: ndcg_at_1000
            value: 85.754
          - type: map_at_1
            value: 76.177
          - type: map_at_3
            value: 80.766
          - type: map_at_5
            value: 81.382
          - type: map_at_10
            value: 81.83
          - type: map_at_20
            value: 81.974
          - type: map_at_100
            value: 82.061
          - type: map_at_1000
            value: 82.07
          - type: recall_at_1
            value: 76.177
          - type: recall_at_3
            value: 86.292
          - type: recall_at_5
            value: 88.981
          - type: recall_at_10
            value: 92.257
          - type: recall_at_20
            value: 94.321
          - type: recall_at_100
            value: 97.725
          - type: recall_at_1000
            value: 99.533
          - type: accuracy
            value: 76.177
          - type: precision_at_1
            value: 76.177
          - type: precision_at_3
            value: 28.764
          - type: precision_at_5
            value: 17.796
          - type: precision_at_10
            value: 9.226
          - type: precision_at_20
            value: 4.716
          - type: precision_at_100
            value: 0.977
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 76.1768
          - type: mrr_at_3
            value: 80.7662
          - type: mrr_at_5
            value: 81.3823
          - type: mrr_at_10
            value: 81.83
          - type: mrr_at_20
            value: 81.9744
          - type: mrr_at_100
            value: 82.0608
          - type: mrr_at_1000
            value: 82.0695
          - type: hit_rate_at_1
            value: 76.177
          - type: hit_rate_at_3
            value: 86.292
          - type: hit_rate_at_5
            value: 88.981
          - type: hit_rate_at_10
            value: 92.257
          - type: hit_rate_at_20
            value: 94.321
          - type: hit_rate_at_100
            value: 97.725
          - type: hit_rate_at_1000
            value: 99.533
      - task:
          type: Retrieval
        dataset:
          type: CoIR-Retrieval/codefeedback-st
          name: MTEB CodeFeedbackST
          config: default
          split: test
          revision: d213819e87aab9010628da8b73ab4eb337c89340
        metrics:
          - type: ndcg_at_1
            value: 74.034
          - type: ndcg_at_3
            value: 84.331
          - type: ndcg_at_5
            value: 85.704
          - type: ndcg_at_10
            value: 86.404
          - type: ndcg_at_20
            value: 86.707
          - type: ndcg_at_100
            value: 86.966
          - type: ndcg_at_1000
            value: 87.045
          - type: map_at_1
            value: 74.034
          - type: map_at_3
            value: 81.926
          - type: map_at_5
            value: 82.694
          - type: map_at_10
            value: 82.988
          - type: map_at_20
            value: 83.072
          - type: map_at_100
            value: 83.11
          - type: map_at_1000
            value: 83.113
          - type: recall_at_1
            value: 74.034
          - type: recall_at_3
            value: 91.238
          - type: recall_at_5
            value: 94.547
          - type: recall_at_10
            value: 96.684
          - type: recall_at_20
            value: 97.873
          - type: recall_at_100
            value: 99.249
          - type: recall_at_1000
            value: 99.85
          - type: accuracy
            value: 74.034
          - type: precision_at_1
            value: 74.034
          - type: precision_at_3
            value: 30.413
          - type: precision_at_5
            value: 18.909
          - type: precision_at_10
            value: 9.668
          - type: precision_at_20
            value: 4.894
          - type: precision_at_100
            value: 0.992
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 74.0337
          - type: mrr_at_3
            value: 81.9257
          - type: mrr_at_5
            value: 82.6938
          - type: mrr_at_10
            value: 82.9883
          - type: mrr_at_20
            value: 83.0724
          - type: mrr_at_100
            value: 83.1096
          - type: mrr_at_1000
            value: 83.113
          - type: hit_rate_at_1
            value: 74.034
          - type: hit_rate_at_3
            value: 91.238
          - type: hit_rate_at_5
            value: 94.547
          - type: hit_rate_at_10
            value: 96.684
          - type: hit_rate_at_20
            value: 97.873
          - type: hit_rate_at_100
            value: 99.249
          - type: hit_rate_at_1000
            value: 99.85
      - task:
          type: Retrieval
        dataset:
          type: CoIR-Retrieval/CodeSearchNet-ccr
          name: MTEB CodeSearchNetCCRetrieval (python)
          config: python
          split: test
          revision: 6e1effa2c03723c5fde48ee912b5ee08d4f211e8
        metrics:
          - type: ndcg_at_1
            value: 90.609
          - type: ndcg_at_3
            value: 94.31
          - type: ndcg_at_5
            value: 94.675
          - type: ndcg_at_10
            value: 94.919
          - type: ndcg_at_20
            value: 95.062
          - type: ndcg_at_100
            value: 95.191
          - type: ndcg_at_1000
            value: 95.23
          - type: map_at_1
            value: 90.609
          - type: map_at_3
            value: 93.459
          - type: map_at_5
            value: 93.664
          - type: map_at_10
            value: 93.764
          - type: map_at_20
            value: 93.804
          - type: map_at_100
            value: 93.822
          - type: map_at_1000
            value: 93.824
          - type: recall_at_1
            value: 90.609
          - type: recall_at_3
            value: 96.749
          - type: recall_at_5
            value: 97.627
          - type: recall_at_10
            value: 98.378
          - type: recall_at_20
            value: 98.948
          - type: recall_at_100
            value: 99.631
          - type: recall_at_1000
            value: 99.933
          - type: accuracy
            value: 90.609
          - type: precision_at_1
            value: 90.609
          - type: precision_at_3
            value: 32.25
          - type: precision_at_5
            value: 19.525
          - type: precision_at_10
            value: 9.838
          - type: precision_at_20
            value: 4.947
          - type: precision_at_100
            value: 0.996
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 90.6087
          - type: mrr_at_3
            value: 93.4587
          - type: mrr_at_5
            value: 93.6638
          - type: mrr_at_10
            value: 93.7644
          - type: mrr_at_20
            value: 93.8036
          - type: mrr_at_100
            value: 93.8222
          - type: mrr_at_1000
            value: 93.8237
          - type: hit_rate_at_1
            value: 90.609
          - type: hit_rate_at_3
            value: 96.749
          - type: hit_rate_at_5
            value: 97.627
          - type: hit_rate_at_10
            value: 98.378
          - type: hit_rate_at_20
            value: 98.948
          - type: hit_rate_at_100
            value: 99.631
          - type: hit_rate_at_1000
            value: 99.933
      - task:
          type: Retrieval
        dataset:
          type: CoIR-Retrieval/CodeSearchNet-ccr
          name: MTEB CodeSearchNetCCRetrieval (javascript)
          config: javascript
          split: test
          revision: 6e1effa2c03723c5fde48ee912b5ee08d4f211e8
        metrics:
          - type: ndcg_at_1
            value: 90.884
          - type: ndcg_at_3
            value: 94.825
          - type: ndcg_at_5
            value: 95.146
          - type: ndcg_at_10
            value: 95.408
          - type: ndcg_at_20
            value: 95.479
          - type: ndcg_at_100
            value: 95.584
          - type: ndcg_at_1000
            value: 95.611
          - type: map_at_1
            value: 90.884
          - type: map_at_3
            value: 93.948
          - type: map_at_5
            value: 94.123
          - type: map_at_10
            value: 94.235
          - type: map_at_20
            value: 94.255
          - type: map_at_100
            value: 94.272
          - type: map_at_1000
            value: 94.273
          - type: recall_at_1
            value: 90.884
          - type: recall_at_3
            value: 97.326
          - type: recall_at_5
            value: 98.116
          - type: recall_at_10
            value: 98.906
          - type: recall_at_20
            value: 99.18
          - type: recall_at_100
            value: 99.727
          - type: recall_at_1000
            value: 99.939
          - type: accuracy
            value: 90.884
          - type: precision_at_1
            value: 90.884
          - type: precision_at_3
            value: 32.442
          - type: precision_at_5
            value: 19.623
          - type: precision_at_10
            value: 9.891
          - type: precision_at_20
            value: 4.959
          - type: precision_at_100
            value: 0.997
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 90.8842
          - type: mrr_at_3
            value: 93.9481
          - type: mrr_at_5
            value: 94.1229
          - type: mrr_at_10
            value: 94.2349
          - type: mrr_at_20
            value: 94.2554
          - type: mrr_at_100
            value: 94.2716
          - type: mrr_at_1000
            value: 94.2727
          - type: hit_rate_at_1
            value: 90.884
          - type: hit_rate_at_3
            value: 97.326
          - type: hit_rate_at_5
            value: 98.116
          - type: hit_rate_at_10
            value: 98.906
          - type: hit_rate_at_20
            value: 99.18
          - type: hit_rate_at_100
            value: 99.727
          - type: hit_rate_at_1000
            value: 99.939
      - task:
          type: Retrieval
        dataset:
          type: CoIR-Retrieval/CodeSearchNet-ccr
          name: MTEB CodeSearchNetCCRetrieval (go)
          config: go
          split: test
          revision: 6e1effa2c03723c5fde48ee912b5ee08d4f211e8
        metrics:
          - type: ndcg_at_1
            value: 91
          - type: ndcg_at_3
            value: 94.579
          - type: ndcg_at_5
            value: 94.997
          - type: ndcg_at_10
            value: 95.292
          - type: ndcg_at_20
            value: 95.422
          - type: ndcg_at_100
            value: 95.507
          - type: ndcg_at_1000
            value: 95.529
          - type: map_at_1
            value: 91
          - type: map_at_3
            value: 93.749
          - type: map_at_5
            value: 93.983
          - type: map_at_10
            value: 94.104
          - type: map_at_20
            value: 94.141
          - type: map_at_100
            value: 94.155
          - type: map_at_1000
            value: 94.156
          - type: recall_at_1
            value: 91
          - type: recall_at_3
            value: 96.959
          - type: recall_at_5
            value: 97.968
          - type: recall_at_10
            value: 98.88
          - type: recall_at_20
            value: 99.384
          - type: recall_at_100
            value: 99.815
          - type: recall_at_1000
            value: 99.988
          - type: accuracy
            value: 91
          - type: precision_at_1
            value: 91
          - type: precision_at_3
            value: 32.32
          - type: precision_at_5
            value: 19.594
          - type: precision_at_10
            value: 9.888
          - type: precision_at_20
            value: 4.969
          - type: precision_at_100
            value: 0.998
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 90.9998
          - type: mrr_at_3
            value: 93.7495
          - type: mrr_at_5
            value: 93.9828
          - type: mrr_at_10
            value: 94.1045
          - type: mrr_at_20
            value: 94.1411
          - type: mrr_at_100
            value: 94.155
          - type: mrr_at_1000
            value: 94.1559
          - type: hit_rate_at_1
            value: 91
          - type: hit_rate_at_3
            value: 96.959
          - type: hit_rate_at_5
            value: 97.968
          - type: hit_rate_at_10
            value: 98.88
          - type: hit_rate_at_20
            value: 99.384
          - type: hit_rate_at_100
            value: 99.815
          - type: hit_rate_at_1000
            value: 99.988
      - task:
          type: Retrieval
        dataset:
          type: CoIR-Retrieval/CodeSearchNet-ccr
          name: MTEB CodeSearchNetCCRetrieval (ruby)
          config: ruby
          split: test
          revision: 6e1effa2c03723c5fde48ee912b5ee08d4f211e8
        metrics:
          - type: ndcg_at_1
            value: 84.853
          - type: ndcg_at_3
            value: 89.856
          - type: ndcg_at_5
            value: 90.798
          - type: ndcg_at_10
            value: 91.306
          - type: ndcg_at_20
            value: 91.568
          - type: ndcg_at_100
            value: 91.73
          - type: ndcg_at_1000
            value: 91.833
          - type: map_at_1
            value: 84.853
          - type: map_at_3
            value: 88.673
          - type: map_at_5
            value: 89.192
          - type: map_at_10
            value: 89.4
          - type: map_at_20
            value: 89.473
          - type: map_at_100
            value: 89.495
          - type: map_at_1000
            value: 89.499
          - type: recall_at_1
            value: 84.853
          - type: recall_at_3
            value: 93.259
          - type: recall_at_5
            value: 95.559
          - type: recall_at_10
            value: 97.145
          - type: recall_at_20
            value: 98.176
          - type: recall_at_100
            value: 99.048
          - type: recall_at_1000
            value: 99.841
          - type: accuracy
            value: 84.853
          - type: precision_at_1
            value: 84.853
          - type: precision_at_3
            value: 31.086
          - type: precision_at_5
            value: 19.112
          - type: precision_at_10
            value: 9.715
          - type: precision_at_20
            value: 4.909
          - type: precision_at_100
            value: 0.99
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 84.8533
          - type: mrr_at_3
            value: 88.673
          - type: mrr_at_5
            value: 89.1924
          - type: mrr_at_10
            value: 89.3997
          - type: mrr_at_20
            value: 89.4725
          - type: mrr_at_100
            value: 89.4949
          - type: mrr_at_1000
            value: 89.499
          - type: hit_rate_at_1
            value: 84.853
          - type: hit_rate_at_3
            value: 93.259
          - type: hit_rate_at_5
            value: 95.559
          - type: hit_rate_at_10
            value: 97.145
          - type: hit_rate_at_20
            value: 98.176
          - type: hit_rate_at_100
            value: 99.048
          - type: hit_rate_at_1000
            value: 99.841
      - task:
          type: Retrieval
        dataset:
          type: CoIR-Retrieval/CodeSearchNet-ccr
          name: MTEB CodeSearchNetCCRetrieval (java)
          config: java
          split: test
          revision: 6e1effa2c03723c5fde48ee912b5ee08d4f211e8
        metrics:
          - type: ndcg_at_1
            value: 90.141
          - type: ndcg_at_3
            value: 93.512
          - type: ndcg_at_5
            value: 93.874
          - type: ndcg_at_10
            value: 94.099
          - type: ndcg_at_20
            value: 94.228
          - type: ndcg_at_100
            value: 94.334
          - type: ndcg_at_1000
            value: 94.406
          - type: map_at_1
            value: 90.141
          - type: map_at_3
            value: 92.728
          - type: map_at_5
            value: 92.929
          - type: map_at_10
            value: 93.024
          - type: map_at_20
            value: 93.059
          - type: map_at_100
            value: 93.075
          - type: map_at_1000
            value: 93.077
          - type: recall_at_1
            value: 90.141
          - type: recall_at_3
            value: 95.764
          - type: recall_at_5
            value: 96.641
          - type: recall_at_10
            value: 97.325
          - type: recall_at_20
            value: 97.837
          - type: recall_at_100
            value: 98.393
          - type: recall_at_1000
            value: 98.978
          - type: accuracy
            value: 90.141
          - type: precision_at_1
            value: 90.141
          - type: precision_at_3
            value: 31.921
          - type: precision_at_5
            value: 19.328
          - type: precision_at_10
            value: 9.733
          - type: precision_at_20
            value: 4.892
          - type: precision_at_100
            value: 0.984
          - type: precision_at_1000
            value: 0.099
          - type: mrr_at_1
            value: 90.1415
          - type: mrr_at_3
            value: 92.7278
          - type: mrr_at_5
            value: 92.9286
          - type: mrr_at_10
            value: 93.0239
          - type: mrr_at_20
            value: 93.0593
          - type: mrr_at_100
            value: 93.0747
          - type: mrr_at_1000
            value: 93.0772
          - type: hit_rate_at_1
            value: 90.141
          - type: hit_rate_at_3
            value: 95.764
          - type: hit_rate_at_5
            value: 96.641
          - type: hit_rate_at_10
            value: 97.325
          - type: hit_rate_at_20
            value: 97.837
          - type: hit_rate_at_100
            value: 98.393
          - type: hit_rate_at_1000
            value: 98.978
      - task:
          type: Retrieval
        dataset:
          type: CoIR-Retrieval/CodeSearchNet-ccr
          name: MTEB CodeSearchNetCCRetrieval (php)
          config: php
          split: test
          revision: 6e1effa2c03723c5fde48ee912b5ee08d4f211e8
        metrics:
          - type: ndcg_at_1
            value: 85.293
          - type: ndcg_at_3
            value: 90.036
          - type: ndcg_at_5
            value: 90.775
          - type: ndcg_at_10
            value: 91.356
          - type: ndcg_at_20
            value: 91.625
          - type: ndcg_at_100
            value: 91.872
          - type: ndcg_at_1000
            value: 91.971
          - type: map_at_1
            value: 85.293
          - type: map_at_3
            value: 88.936
          - type: map_at_5
            value: 89.349
          - type: map_at_10
            value: 89.59
          - type: map_at_20
            value: 89.666
          - type: map_at_100
            value: 89.702
          - type: map_at_1000
            value: 89.707
          - type: recall_at_1
            value: 85.293
          - type: recall_at_3
            value: 93.193
          - type: recall_at_5
            value: 94.976
          - type: recall_at_10
            value: 96.76
          - type: recall_at_20
            value: 97.809
          - type: recall_at_100
            value: 99.108
          - type: recall_at_1000
            value: 99.879
          - type: accuracy
            value: 85.293
          - type: precision_at_1
            value: 85.293
          - type: precision_at_3
            value: 31.064
          - type: precision_at_5
            value: 18.995
          - type: precision_at_10
            value: 9.676
          - type: precision_at_20
            value: 4.89
          - type: precision_at_100
            value: 0.991
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 85.2933
          - type: mrr_at_3
            value: 88.9361
          - type: mrr_at_5
            value: 89.3489
          - type: mrr_at_10
            value: 89.5904
          - type: mrr_at_20
            value: 89.6663
          - type: mrr_at_100
            value: 89.7025
          - type: mrr_at_1000
            value: 89.7065
          - type: hit_rate_at_1
            value: 85.293
          - type: hit_rate_at_3
            value: 93.193
          - type: hit_rate_at_5
            value: 94.976
          - type: hit_rate_at_10
            value: 96.76
          - type: hit_rate_at_20
            value: 97.809
          - type: hit_rate_at_100
            value: 99.108
          - type: hit_rate_at_1000
            value: 99.879
      - task:
          type: Retrieval
        dataset:
          type: mteb/CodeSearchNetRetrieval
          name: MTEB CodeSearchNetRetrieval (python)
          config: python
          split: test
          revision: 68e8f0731a656fa4bd5b7c81936d95ad48a39bfe
        metrics:
          - type: ndcg_at_1
            value: 90.1
          - type: ndcg_at_3
            value: 94.231
          - type: ndcg_at_5
            value: 94.696
          - type: ndcg_at_10
            value: 94.957
          - type: ndcg_at_20
            value: 95.028
          - type: ndcg_at_100
            value: 95.069
          - type: ndcg_at_1000
            value: 95.143
          - type: map_at_1
            value: 90.1
          - type: map_at_3
            value: 93.267
          - type: map_at_5
            value: 93.532
          - type: map_at_10
            value: 93.641
          - type: map_at_20
            value: 93.658
          - type: map_at_100
            value: 93.665
          - type: map_at_1000
            value: 93.668
          - type: recall_at_1
            value: 90.1
          - type: recall_at_3
            value: 97
          - type: recall_at_5
            value: 98.1
          - type: recall_at_10
            value: 98.9
          - type: recall_at_20
            value: 99.2
          - type: recall_at_100
            value: 99.4
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 90.1
          - type: precision_at_1
            value: 90.1
          - type: precision_at_3
            value: 32.333
          - type: precision_at_5
            value: 19.62
          - type: precision_at_10
            value: 9.89
          - type: precision_at_20
            value: 4.96
          - type: precision_at_100
            value: 0.994
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 90.1
          - type: mrr_at_3
            value: 93.2667
          - type: mrr_at_5
            value: 93.5317
          - type: mrr_at_10
            value: 93.6407
          - type: mrr_at_20
            value: 93.6582
          - type: mrr_at_100
            value: 93.6648
          - type: mrr_at_1000
            value: 93.6675
          - type: hit_rate_at_1
            value: 90.1
          - type: hit_rate_at_3
            value: 97
          - type: hit_rate_at_5
            value: 98.1
          - type: hit_rate_at_10
            value: 98.9
          - type: hit_rate_at_20
            value: 99.2
          - type: hit_rate_at_100
            value: 99.4
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: mteb/CodeSearchNetRetrieval
          name: MTEB CodeSearchNetRetrieval (javascript)
          config: javascript
          split: test
          revision: 68e8f0731a656fa4bd5b7c81936d95ad48a39bfe
        metrics:
          - type: ndcg_at_1
            value: 75.8
          - type: ndcg_at_3
            value: 80.999
          - type: ndcg_at_5
            value: 81.61
          - type: ndcg_at_10
            value: 82.588
          - type: ndcg_at_20
            value: 83.126
          - type: ndcg_at_100
            value: 83.73
          - type: ndcg_at_1000
            value: 84.398
          - type: map_at_1
            value: 75.8
          - type: map_at_3
            value: 79.75
          - type: map_at_5
            value: 80.085
          - type: map_at_10
            value: 80.494
          - type: map_at_20
            value: 80.645
          - type: map_at_100
            value: 80.718
          - type: map_at_1000
            value: 80.741
          - type: recall_at_1
            value: 75.8
          - type: recall_at_3
            value: 84.6
          - type: recall_at_5
            value: 86.1
          - type: recall_at_10
            value: 89.1
          - type: recall_at_20
            value: 91.2
          - type: recall_at_100
            value: 94.6
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 75.8
          - type: precision_at_1
            value: 75.8
          - type: precision_at_3
            value: 28.2
          - type: precision_at_5
            value: 17.22
          - type: precision_at_10
            value: 8.91
          - type: precision_at_20
            value: 4.56
          - type: precision_at_100
            value: 0.946
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 75.8
          - type: mrr_at_3
            value: 79.75
          - type: mrr_at_5
            value: 80.085
          - type: mrr_at_10
            value: 80.4936
          - type: mrr_at_20
            value: 80.6448
          - type: mrr_at_100
            value: 80.7182
          - type: mrr_at_1000
            value: 80.7413
          - type: hit_rate_at_1
            value: 75.8
          - type: hit_rate_at_3
            value: 84.6
          - type: hit_rate_at_5
            value: 86.1
          - type: hit_rate_at_10
            value: 89.1
          - type: hit_rate_at_20
            value: 91.2
          - type: hit_rate_at_100
            value: 94.6
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: mteb/CodeSearchNetRetrieval
          name: MTEB CodeSearchNetRetrieval (go)
          config: go
          split: test
          revision: 68e8f0731a656fa4bd5b7c81936d95ad48a39bfe
        metrics:
          - type: ndcg_at_1
            value: 91.8
          - type: ndcg_at_3
            value: 95.594
          - type: ndcg_at_5
            value: 95.882
          - type: ndcg_at_10
            value: 95.977
          - type: ndcg_at_20
            value: 96.075
          - type: ndcg_at_100
            value: 96.134
          - type: ndcg_at_1000
            value: 96.17
          - type: map_at_1
            value: 91.8
          - type: map_at_3
            value: 94.75
          - type: map_at_5
            value: 94.91
          - type: map_at_10
            value: 94.948
          - type: map_at_20
            value: 94.973
          - type: map_at_100
            value: 94.982
          - type: map_at_1000
            value: 94.983
          - type: recall_at_1
            value: 91.8
          - type: recall_at_3
            value: 98
          - type: recall_at_5
            value: 98.7
          - type: recall_at_10
            value: 99
          - type: recall_at_20
            value: 99.4
          - type: recall_at_100
            value: 99.7
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 91.8
          - type: precision_at_1
            value: 91.8
          - type: precision_at_3
            value: 32.667
          - type: precision_at_5
            value: 19.74
          - type: precision_at_10
            value: 9.9
          - type: precision_at_20
            value: 4.97
          - type: precision_at_100
            value: 0.997
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 91.8
          - type: mrr_at_3
            value: 94.75
          - type: mrr_at_5
            value: 94.91
          - type: mrr_at_10
            value: 94.9475
          - type: mrr_at_20
            value: 94.9728
          - type: mrr_at_100
            value: 94.9821
          - type: mrr_at_1000
            value: 94.983
          - type: hit_rate_at_1
            value: 91.8
          - type: hit_rate_at_3
            value: 98
          - type: hit_rate_at_5
            value: 98.7
          - type: hit_rate_at_10
            value: 99
          - type: hit_rate_at_20
            value: 99.4
          - type: hit_rate_at_100
            value: 99.7
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: mteb/CodeSearchNetRetrieval
          name: MTEB CodeSearchNetRetrieval (ruby)
          config: ruby
          split: test
          revision: 68e8f0731a656fa4bd5b7c81936d95ad48a39bfe
        metrics:
          - type: ndcg_at_1
            value: 79.2
          - type: ndcg_at_3
            value: 85.958
          - type: ndcg_at_5
            value: 86.793
          - type: ndcg_at_10
            value: 86.995
          - type: ndcg_at_20
            value: 87.271
          - type: ndcg_at_100
            value: 87.612
          - type: ndcg_at_1000
            value: 88.048
          - type: map_at_1
            value: 79.2
          - type: map_at_3
            value: 84.317
          - type: map_at_5
            value: 84.787
          - type: map_at_10
            value: 84.875
          - type: map_at_20
            value: 84.95
          - type: map_at_100
            value: 84.999
          - type: map_at_1000
            value: 85.009
          - type: recall_at_1
            value: 79.2
          - type: recall_at_3
            value: 90.7
          - type: recall_at_5
            value: 92.7
          - type: recall_at_10
            value: 93.3
          - type: recall_at_20
            value: 94.4
          - type: recall_at_100
            value: 96.2
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 79.2
          - type: precision_at_1
            value: 79.2
          - type: precision_at_3
            value: 30.233
          - type: precision_at_5
            value: 18.54
          - type: precision_at_10
            value: 9.33
          - type: precision_at_20
            value: 4.72
          - type: precision_at_100
            value: 0.962
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 79.2
          - type: mrr_at_3
            value: 84.3167
          - type: mrr_at_5
            value: 84.7867
          - type: mrr_at_10
            value: 84.8746
          - type: mrr_at_20
            value: 84.9495
          - type: mrr_at_100
            value: 84.9991
          - type: mrr_at_1000
            value: 85.0091
          - type: hit_rate_at_1
            value: 79.2
          - type: hit_rate_at_3
            value: 90.7
          - type: hit_rate_at_5
            value: 92.7
          - type: hit_rate_at_10
            value: 93.3
          - type: hit_rate_at_20
            value: 94.4
          - type: hit_rate_at_100
            value: 96.2
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: mteb/CodeSearchNetRetrieval
          name: MTEB CodeSearchNetRetrieval (java)
          config: java
          split: test
          revision: 68e8f0731a656fa4bd5b7c81936d95ad48a39bfe
        metrics:
          - type: ndcg_at_1
            value: 85
          - type: ndcg_at_3
            value: 90.343
          - type: ndcg_at_5
            value: 91.001
          - type: ndcg_at_10
            value: 91.302
          - type: ndcg_at_20
            value: 91.428
          - type: ndcg_at_100
            value: 91.527
          - type: ndcg_at_1000
            value: 91.845
          - type: map_at_1
            value: 85
          - type: map_at_3
            value: 89.133
          - type: map_at_5
            value: 89.498
          - type: map_at_10
            value: 89.629
          - type: map_at_20
            value: 89.663
          - type: map_at_100
            value: 89.679
          - type: map_at_1000
            value: 89.688
          - type: recall_at_1
            value: 85
          - type: recall_at_3
            value: 93.8
          - type: recall_at_5
            value: 95.4
          - type: recall_at_10
            value: 96.3
          - type: recall_at_20
            value: 96.8
          - type: recall_at_100
            value: 97.3
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 85
          - type: precision_at_1
            value: 85
          - type: precision_at_3
            value: 31.267
          - type: precision_at_5
            value: 19.08
          - type: precision_at_10
            value: 9.63
          - type: precision_at_20
            value: 4.84
          - type: precision_at_100
            value: 0.973
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 85
          - type: mrr_at_3
            value: 89.1333
          - type: mrr_at_5
            value: 89.4983
          - type: mrr_at_10
            value: 89.6286
          - type: mrr_at_20
            value: 89.6627
          - type: mrr_at_100
            value: 89.6788
          - type: mrr_at_1000
            value: 89.688
          - type: hit_rate_at_1
            value: 85
          - type: hit_rate_at_3
            value: 93.8
          - type: hit_rate_at_5
            value: 95.4
          - type: hit_rate_at_10
            value: 96.3
          - type: hit_rate_at_20
            value: 96.8
          - type: hit_rate_at_100
            value: 97.3
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: mteb/CodeSearchNetRetrieval
          name: MTEB CodeSearchNetRetrieval (php)
          config: php
          split: test
          revision: 68e8f0731a656fa4bd5b7c81936d95ad48a39bfe
        metrics:
          - type: ndcg_at_1
            value: 80.2
          - type: ndcg_at_3
            value: 86.734
          - type: ndcg_at_5
            value: 87.901
          - type: ndcg_at_10
            value: 88.363
          - type: ndcg_at_20
            value: 88.493
          - type: ndcg_at_100
            value: 88.756
          - type: ndcg_at_1000
            value: 89.074
          - type: map_at_1
            value: 80.2
          - type: map_at_3
            value: 85.183
          - type: map_at_5
            value: 85.838
          - type: map_at_10
            value: 86.035
          - type: map_at_20
            value: 86.072
          - type: map_at_100
            value: 86.109
          - type: map_at_1000
            value: 86.118
          - type: recall_at_1
            value: 80.2
          - type: recall_at_3
            value: 91.2
          - type: recall_at_5
            value: 94
          - type: recall_at_10
            value: 95.4
          - type: recall_at_20
            value: 95.9
          - type: recall_at_100
            value: 97.3
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 80.2
          - type: precision_at_1
            value: 80.2
          - type: precision_at_3
            value: 30.4
          - type: precision_at_5
            value: 18.8
          - type: precision_at_10
            value: 9.54
          - type: precision_at_20
            value: 4.795
          - type: precision_at_100
            value: 0.973
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 80.2
          - type: mrr_at_3
            value: 85.1833
          - type: mrr_at_5
            value: 85.8383
          - type: mrr_at_10
            value: 86.0352
          - type: mrr_at_20
            value: 86.0725
          - type: mrr_at_100
            value: 86.1093
          - type: mrr_at_1000
            value: 86.1176
          - type: hit_rate_at_1
            value: 80.2
          - type: hit_rate_at_3
            value: 91.2
          - type: hit_rate_at_5
            value: 94
          - type: hit_rate_at_10
            value: 95.4
          - type: hit_rate_at_20
            value: 95.9
          - type: hit_rate_at_100
            value: 97.3
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: CoIR-Retrieval/codetrans-contest
          name: MTEB CodeTransOceanContest
          config: default
          split: test
          revision: 20da4eb20a4b17300c0986ee148c90867a7f2a4d
        metrics:
          - type: ndcg_at_1
            value: 81.9
          - type: ndcg_at_3
            value: 86.35
          - type: ndcg_at_5
            value: 87.46
          - type: ndcg_at_10
            value: 88.075
          - type: ndcg_at_20
            value: 88.304
          - type: ndcg_at_100
            value: 89.034
          - type: ndcg_at_1000
            value: 89.215
          - type: map_at_1
            value: 81.9
          - type: map_at_3
            value: 85.37
          - type: map_at_5
            value: 85.98
          - type: map_at_10
            value: 86.252
          - type: map_at_20
            value: 86.315
          - type: map_at_100
            value: 86.405
          - type: map_at_1000
            value: 86.413
          - type: recall_at_1
            value: 81.9
          - type: recall_at_3
            value: 89.14
          - type: recall_at_5
            value: 91.855
          - type: recall_at_10
            value: 93.665
          - type: recall_at_20
            value: 94.57
          - type: recall_at_100
            value: 98.643
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 81.9
          - type: precision_at_1
            value: 81.9
          - type: precision_at_3
            value: 29.713
          - type: precision_at_5
            value: 18.371
          - type: precision_at_10
            value: 9.367
          - type: precision_at_20
            value: 4.729
          - type: precision_at_100
            value: 0.986
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 81.9005
          - type: mrr_at_3
            value: 85.3695
          - type: mrr_at_5
            value: 85.9804
          - type: mrr_at_10
            value: 86.2524
          - type: mrr_at_20
            value: 86.3149
          - type: mrr_at_100
            value: 86.405
          - type: mrr_at_1000
            value: 86.4131
          - type: hit_rate_at_1
            value: 81.9
          - type: hit_rate_at_3
            value: 89.14
          - type: hit_rate_at_5
            value: 91.855
          - type: hit_rate_at_10
            value: 93.665
          - type: hit_rate_at_20
            value: 94.57
          - type: hit_rate_at_100
            value: 98.643
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: CoIR-Retrieval/codetrans-dl
          name: MTEB CodeTransOceanDL
          config: default
          split: test
          revision: 281562cb8a1265ab5c0824bfa6ddcd9b0a15618f
        metrics:
          - type: ndcg_at_1
            value: 10.556
          - type: ndcg_at_3
            value: 14.676
          - type: ndcg_at_5
            value: 21.996
          - type: ndcg_at_10
            value: 37.234
          - type: ndcg_at_20
            value: 41.806
          - type: ndcg_at_100
            value: 41.806
          - type: ndcg_at_1000
            value: 41.806
          - type: map_at_1
            value: 10.556
          - type: map_at_3
            value: 13.611
          - type: map_at_5
            value: 17.806
          - type: map_at_10
            value: 23.863
          - type: map_at_20
            value: 25.311
          - type: map_at_100
            value: 25.311
          - type: map_at_1000
            value: 25.311
          - type: recall_at_1
            value: 10.556
          - type: recall_at_3
            value: 17.778
          - type: recall_at_5
            value: 35
          - type: recall_at_10
            value: 83.333
          - type: recall_at_20
            value: 100
          - type: recall_at_100
            value: 100
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 10.556
          - type: precision_at_1
            value: 10.556
          - type: precision_at_3
            value: 5.926
          - type: precision_at_5
            value: 7
          - type: precision_at_10
            value: 8.333
          - type: precision_at_20
            value: 5
          - type: precision_at_100
            value: 1
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 10.5556
          - type: mrr_at_3
            value: 13.6111
          - type: mrr_at_5
            value: 17.8056
          - type: mrr_at_10
            value: 23.8629
          - type: mrr_at_20
            value: 25.3107
          - type: mrr_at_100
            value: 25.3107
          - type: mrr_at_1000
            value: 25.3107
          - type: hit_rate_at_1
            value: 10.556
          - type: hit_rate_at_3
            value: 17.778
          - type: hit_rate_at_5
            value: 35
          - type: hit_rate_at_10
            value: 83.333
          - type: hit_rate_at_20
            value: 100
          - type: hit_rate_at_100
            value: 100
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: CoIR-Retrieval/cosqa
          name: MTEB CosQA
          config: default
          split: test
          revision: bc5efb7e9d437246ce393ed19d772e08e4a79535
        metrics:
          - type: ndcg_at_1
            value: 15.6
          - type: ndcg_at_3
            value: 24.243
          - type: ndcg_at_5
            value: 31.015
          - type: ndcg_at_10
            value: 36.294
          - type: ndcg_at_20
            value: 39.259
          - type: ndcg_at_100
            value: 42.682
          - type: ndcg_at_1000
            value: 43.45
          - type: map_at_1
            value: 15.6
          - type: map_at_3
            value: 21.933
          - type: map_at_5
            value: 25.653
          - type: map_at_10
            value: 27.856
          - type: map_at_20
            value: 28.687
          - type: map_at_100
            value: 29.18
          - type: map_at_1000
            value: 29.211
          - type: recall_at_1
            value: 15.6
          - type: recall_at_3
            value: 31
          - type: recall_at_5
            value: 47.6
          - type: recall_at_10
            value: 63.8
          - type: recall_at_20
            value: 75.4
          - type: recall_at_100
            value: 93.6
          - type: recall_at_1000
            value: 99.6
          - type: accuracy
            value: 15.6
          - type: precision_at_1
            value: 15.6
          - type: precision_at_3
            value: 10.333
          - type: precision_at_5
            value: 9.52
          - type: precision_at_10
            value: 6.38
          - type: precision_at_20
            value: 3.77
          - type: precision_at_100
            value: 0.936
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 15.6
          - type: mrr_at_3
            value: 21.9333
          - type: mrr_at_5
            value: 25.6533
          - type: mrr_at_10
            value: 27.8563
          - type: mrr_at_20
            value: 28.6874
          - type: mrr_at_100
            value: 29.1803
          - type: mrr_at_1000
            value: 29.2108
          - type: hit_rate_at_1
            value: 15.6
          - type: hit_rate_at_3
            value: 31
          - type: hit_rate_at_5
            value: 47.6
          - type: hit_rate_at_10
            value: 63.8
          - type: hit_rate_at_20
            value: 75.4
          - type: hit_rate_at_100
            value: 93.6
          - type: hit_rate_at_1000
            value: 99.6
      - task:
          type: Retrieval
        dataset:
          type: mteb/FEVER_test_top_250_only_w_correct-v2
          name: MTEB FEVERHardNegatives
          config: default
          split: test
          revision: 080c9ed6267b65029207906e815d44a9240bafca
        metrics:
          - type: ndcg_at_1
            value: 82.2
          - type: ndcg_at_3
            value: 87.644
          - type: ndcg_at_5
            value: 88.436
          - type: ndcg_at_10
            value: 89.08
          - type: ndcg_at_20
            value: 89.449
          - type: ndcg_at_100
            value: 89.824
          - type: ndcg_at_1000
            value: 89.983
          - type: map_at_1
            value: 77.021
          - type: map_at_3
            value: 84.585
          - type: map_at_5
            value: 85.2
          - type: map_at_10
            value: 85.57
          - type: map_at_20
            value: 85.722
          - type: map_at_100
            value: 85.81
          - type: map_at_1000
            value: 85.823
          - type: recall_at_1
            value: 77.021
          - type: recall_at_3
            value: 91.888
          - type: recall_at_5
            value: 93.961
          - type: recall_at_10
            value: 95.756
          - type: recall_at_20
            value: 97.013
          - type: recall_at_100
            value: 98.498
          - type: recall_at_1000
            value: 99.411
          - type: accuracy
            value: 77.021
          - type: precision_at_1
            value: 82.2
          - type: precision_at_3
            value: 32.967
          - type: precision_at_5
            value: 20.38
          - type: precision_at_10
            value: 10.51
          - type: precision_at_20
            value: 5.38
          - type: precision_at_100
            value: 1.119
          - type: precision_at_1000
            value: 0.115
          - type: mrr_at_1
            value: 82.2
          - type: mrr_at_3
            value: 89.05
          - type: mrr_at_5
            value: 89.455
          - type: mrr_at_10
            value: 89.5987
          - type: mrr_at_20
            value: 89.6337
          - type: mrr_at_100
            value: 89.6381
          - type: mrr_at_1000
            value: 89.6382
          - type: hit_rate_at_1
            value: 82.2
          - type: hit_rate_at_3
            value: 96.6
          - type: hit_rate_at_5
            value: 98.3
          - type: hit_rate_at_10
            value: 99.3
          - type: hit_rate_at_20
            value: 99.8
          - type: hit_rate_at_100
            value: 99.9
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: mteb/fiqa
          name: MTEB FiQA2018
          config: default
          split: test
          revision: 27a168819829fe9bcd655c2df245fb19452e8e06
        metrics:
          - type: ndcg_at_1
            value: 51.08
          - type: ndcg_at_3
            value: 47.747
          - type: ndcg_at_5
            value: 49
          - type: ndcg_at_10
            value: 51.825
          - type: ndcg_at_20
            value: 54.219
          - type: ndcg_at_100
            value: 57.719
          - type: ndcg_at_1000
            value: 60.192
          - type: map_at_1
            value: 26.119
          - type: map_at_3
            value: 37.835
          - type: map_at_5
            value: 40.826
          - type: map_at_10
            value: 43.202
          - type: map_at_20
            value: 44.276
          - type: map_at_100
            value: 45.088
          - type: map_at_1000
            value: 45.257
          - type: recall_at_1
            value: 26.119
          - type: recall_at_3
            value: 43.055
          - type: recall_at_5
            value: 50.219
          - type: recall_at_10
            value: 59.662
          - type: recall_at_20
            value: 66.739
          - type: recall_at_100
            value: 80.684
          - type: recall_at_1000
            value: 95.557
          - type: accuracy
            value: 26.119
          - type: precision_at_1
            value: 51.08
          - type: precision_at_3
            value: 32.047
          - type: precision_at_5
            value: 23.519
          - type: precision_at_10
            value: 14.336
          - type: precision_at_20
            value: 8.287
          - type: precision_at_100
            value: 2.086
          - type: precision_at_1000
            value: 0.25
          - type: mrr_at_1
            value: 51.0802
          - type: mrr_at_3
            value: 57.4074
          - type: mrr_at_5
            value: 58.8889
          - type: mrr_at_10
            value: 59.8483
          - type: mrr_at_20
            value: 60.1598
          - type: mrr_at_100
            value: 60.3469
          - type: mrr_at_1000
            value: 60.3836
          - type: hit_rate_at_1
            value: 51.08
          - type: hit_rate_at_3
            value: 65.278
          - type: hit_rate_at_5
            value: 71.605
          - type: hit_rate_at_10
            value: 79.012
          - type: hit_rate_at_20
            value: 83.796
          - type: hit_rate_at_100
            value: 91.667
          - type: hit_rate_at_1000
            value: 98.765
      - task:
          type: Retrieval
        dataset:
          type: mteb/HellaSwag
          name: MTEB HellaSwag
          config: default
          split: test
          revision: 944be88c186b87495f4b1f5e4a416a3a7ea5d680
        metrics:
          - type: ndcg_at_1
            value: 18.771
          - type: ndcg_at_3
            value: 28.551
          - type: ndcg_at_5
            value: 31.509
          - type: ndcg_at_10
            value: 33.91
          - type: ndcg_at_20
            value: 35.526
          - type: ndcg_at_100
            value: 38.2
          - type: ndcg_at_1000
            value: 40.169
          - type: map_at_1
            value: 18.771
          - type: map_at_3
            value: 26.137
          - type: map_at_5
            value: 27.784
          - type: map_at_10
            value: 28.775
          - type: map_at_20
            value: 29.219
          - type: map_at_100
            value: 29.581
          - type: map_at_1000
            value: 29.649
          - type: recall_at_1
            value: 18.771
          - type: recall_at_3
            value: 35.541
          - type: recall_at_5
            value: 42.701
          - type: recall_at_10
            value: 50.119
          - type: recall_at_20
            value: 56.513
          - type: recall_at_100
            value: 71.032
          - type: recall_at_1000
            value: 86.885
          - type: accuracy
            value: 18.771
          - type: precision_at_1
            value: 18.771
          - type: precision_at_3
            value: 11.847
          - type: precision_at_5
            value: 8.54
          - type: precision_at_10
            value: 5.012
          - type: precision_at_20
            value: 2.826
          - type: precision_at_100
            value: 0.71
          - type: precision_at_1000
            value: 0.087
          - type: mrr_at_1
            value: 18.7712
          - type: mrr_at_3
            value: 26.1369
          - type: mrr_at_5
            value: 27.784
          - type: mrr_at_10
            value: 28.7753
          - type: mrr_at_20
            value: 29.2188
          - type: mrr_at_100
            value: 29.5812
          - type: mrr_at_1000
            value: 29.6493
          - type: hit_rate_at_1
            value: 18.771
          - type: hit_rate_at_3
            value: 35.541
          - type: hit_rate_at_5
            value: 42.701
          - type: hit_rate_at_10
            value: 50.119
          - type: hit_rate_at_20
            value: 56.513
          - type: hit_rate_at_100
            value: 71.032
          - type: hit_rate_at_1000
            value: 86.885
      - task:
          type: Retrieval
        dataset:
          type: mteb/HotpotQA_test_top_250_only_w_correct-v2
          name: MTEB HotpotQAHardNegatives
          config: default
          split: test
          revision: 617612fa63afcb60e3b134bed8b7216a99707c37
        metrics:
          - type: ndcg_at_1
            value: 82.9
          - type: ndcg_at_3
            value: 68.473
          - type: ndcg_at_5
            value: 71.024
          - type: ndcg_at_10
            value: 73.177
          - type: ndcg_at_20
            value: 74.715
          - type: ndcg_at_100
            value: 76.925
          - type: ndcg_at_1000
            value: 77.962
          - type: map_at_1
            value: 41.45
          - type: map_at_3
            value: 61.292
          - type: map_at_5
            value: 63.377
          - type: map_at_10
            value: 64.678
          - type: map_at_20
            value: 65.306
          - type: map_at_100
            value: 65.792
          - type: map_at_1000
            value: 65.853
          - type: recall_at_1
            value: 41.45
          - type: recall_at_3
            value: 65.55
          - type: recall_at_5
            value: 70.6
          - type: recall_at_10
            value: 76
          - type: recall_at_20
            value: 80.9
          - type: recall_at_100
            value: 90.6
          - type: recall_at_1000
            value: 97.3
          - type: accuracy
            value: 41.45
          - type: precision_at_1
            value: 82.9
          - type: precision_at_3
            value: 43.7
          - type: precision_at_5
            value: 28.24
          - type: precision_at_10
            value: 15.2
          - type: precision_at_20
            value: 8.09
          - type: precision_at_100
            value: 1.812
          - type: precision_at_1000
            value: 0.195
          - type: mrr_at_1
            value: 82.9
          - type: mrr_at_3
            value: 87.25
          - type: mrr_at_5
            value: 87.7
          - type: mrr_at_10
            value: 88.0158
          - type: mrr_at_20
            value: 88.1707
          - type: mrr_at_100
            value: 88.1998
          - type: mrr_at_1000
            value: 88.2014
          - type: hit_rate_at_1
            value: 82.9
          - type: hit_rate_at_3
            value: 92.2
          - type: hit_rate_at_5
            value: 94.1
          - type: hit_rate_at_10
            value: 96.3
          - type: hit_rate_at_20
            value: 98.5
          - type: hit_rate_at_100
            value: 99.7
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Classification
        dataset:
          type: mteb/imdb
          name: MTEB ImdbClassification
          config: default
          split: test
          revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7
        metrics:
          - type: accuracy
            value: 94.028
          - type: f1
            value: 94.0244
          - type: f1_weighted
            value: 94.0244
          - type: precision
            value: 94.1259
          - type: precision_weighted
            value: 94.1259
          - type: recall
            value: 94.028
          - type: recall_weighted
            value: 94.028
          - type: ap
            value: 91.3763
          - type: ap_weighted
            value: 91.3763
      - task:
          type: Retrieval
        dataset:
          type: dwzhu/LongEmbed
          name: MTEB LEMBNarrativeQARetrieval
          config: default
          split: test
          revision: 6e346642246bfb4928c560ee08640dc84d074e8c
        metrics:
          - type: ndcg_at_1
            value: 53.555
          - type: ndcg_at_3
            value: 61.273
          - type: ndcg_at_5
            value: 62.897
          - type: ndcg_at_10
            value: 64.545
          - type: ndcg_at_20
            value: 65.786
          - type: ndcg_at_100
            value: 67.857
          - type: ndcg_at_1000
            value: 68.943
          - type: map_at_1
            value: 53.555
          - type: map_at_3
            value: 59.414
          - type: map_at_5
            value: 60.315
          - type: map_at_10
            value: 60.998
          - type: map_at_20
            value: 61.342
          - type: map_at_100
            value: 61.62
          - type: map_at_1000
            value: 61.666
          - type: recall_at_1
            value: 53.555
          - type: recall_at_3
            value: 66.638
          - type: recall_at_5
            value: 70.581
          - type: recall_at_10
            value: 75.663
          - type: recall_at_20
            value: 80.544
          - type: recall_at_100
            value: 91.837
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 53.555
          - type: precision_at_1
            value: 53.555
          - type: precision_at_3
            value: 22.213
          - type: precision_at_5
            value: 14.116
          - type: precision_at_10
            value: 7.566
          - type: precision_at_20
            value: 4.027
          - type: precision_at_100
            value: 0.918
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 53.5554
          - type: mrr_at_3
            value: 59.414
          - type: mrr_at_5
            value: 60.315
          - type: mrr_at_10
            value: 60.9979
          - type: mrr_at_20
            value: 61.3419
          - type: mrr_at_100
            value: 61.6196
          - type: mrr_at_1000
            value: 61.6661
          - type: hit_rate_at_1
            value: 53.555
          - type: hit_rate_at_3
            value: 66.638
          - type: hit_rate_at_5
            value: 70.581
          - type: hit_rate_at_10
            value: 75.663
          - type: hit_rate_at_20
            value: 80.544
          - type: hit_rate_at_100
            value: 91.837
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: dwzhu/LongEmbed
          name: MTEB LEMBNeedleRetrieval
          config: default
          split: test_256
          revision: 6e346642246bfb4928c560ee08640dc84d074e8c
        metrics:
          - type: ndcg_at_1
            value: 84
          - type: ndcg_at_3
            value: 93.571
          - type: ndcg_at_5
            value: 93.571
          - type: ndcg_at_10
            value: 93.571
          - type: ndcg_at_20
            value: 93.571
          - type: ndcg_at_100
            value: 93.571
          - type: ndcg_at_1000
            value: 93.571
          - type: map_at_1
            value: 84
          - type: map_at_3
            value: 91.333
          - type: map_at_5
            value: 91.333
          - type: map_at_10
            value: 91.333
          - type: map_at_20
            value: 91.333
          - type: map_at_100
            value: 91.333
          - type: map_at_1000
            value: 91.333
          - type: recall_at_1
            value: 84
          - type: recall_at_3
            value: 100
          - type: recall_at_5
            value: 100
          - type: recall_at_10
            value: 100
          - type: recall_at_20
            value: 100
          - type: recall_at_100
            value: 100
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 84
          - type: precision_at_1
            value: 84
          - type: precision_at_3
            value: 33.333
          - type: precision_at_5
            value: 20
          - type: precision_at_10
            value: 10
          - type: precision_at_20
            value: 5
          - type: precision_at_100
            value: 1
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 84
          - type: mrr_at_3
            value: 91.3333
          - type: mrr_at_5
            value: 91.3333
          - type: mrr_at_10
            value: 91.3333
          - type: mrr_at_20
            value: 91.3333
          - type: mrr_at_100
            value: 91.3333
          - type: mrr_at_1000
            value: 91.3333
          - type: hit_rate_at_1
            value: 84
          - type: hit_rate_at_3
            value: 100
          - type: hit_rate_at_5
            value: 100
          - type: hit_rate_at_10
            value: 100
          - type: hit_rate_at_20
            value: 100
          - type: hit_rate_at_100
            value: 100
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: dwzhu/LongEmbed
          name: MTEB LEMBNeedleRetrieval
          config: default
          split: test_512
          revision: 6e346642246bfb4928c560ee08640dc84d074e8c
        metrics:
          - type: ndcg_at_1
            value: 78
          - type: ndcg_at_3
            value: 87.571
          - type: ndcg_at_5
            value: 88.345
          - type: ndcg_at_10
            value: 89.77
          - type: ndcg_at_20
            value: 89.77
          - type: ndcg_at_100
            value: 89.77
          - type: ndcg_at_1000
            value: 89.77
          - type: map_at_1
            value: 78
          - type: map_at_3
            value: 85.333
          - type: map_at_5
            value: 85.733
          - type: map_at_10
            value: 86.4
          - type: map_at_20
            value: 86.4
          - type: map_at_100
            value: 86.4
          - type: map_at_1000
            value: 86.4
          - type: recall_at_1
            value: 78
          - type: recall_at_3
            value: 94
          - type: recall_at_5
            value: 96
          - type: recall_at_10
            value: 100
          - type: recall_at_20
            value: 100
          - type: recall_at_100
            value: 100
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 78
          - type: precision_at_1
            value: 78
          - type: precision_at_3
            value: 31.333
          - type: precision_at_5
            value: 19.2
          - type: precision_at_10
            value: 10
          - type: precision_at_20
            value: 5
          - type: precision_at_100
            value: 1
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 78
          - type: mrr_at_3
            value: 85.3333
          - type: mrr_at_5
            value: 85.7333
          - type: mrr_at_10
            value: 86.4
          - type: mrr_at_20
            value: 86.4
          - type: mrr_at_100
            value: 86.4
          - type: mrr_at_1000
            value: 86.4
          - type: hit_rate_at_1
            value: 78
          - type: hit_rate_at_3
            value: 94
          - type: hit_rate_at_5
            value: 96
          - type: hit_rate_at_10
            value: 100
          - type: hit_rate_at_20
            value: 100
          - type: hit_rate_at_100
            value: 100
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: dwzhu/LongEmbed
          name: MTEB LEMBNeedleRetrieval
          config: default
          split: test_1024
          revision: 6e346642246bfb4928c560ee08640dc84d074e8c
        metrics:
          - type: ndcg_at_1
            value: 78
          - type: ndcg_at_3
            value: 87.833
          - type: ndcg_at_5
            value: 89.468
          - type: ndcg_at_10
            value: 90.099
          - type: ndcg_at_20
            value: 90.099
          - type: ndcg_at_100
            value: 90.099
          - type: ndcg_at_1000
            value: 90.099
          - type: map_at_1
            value: 78
          - type: map_at_3
            value: 85.667
          - type: map_at_5
            value: 86.567
          - type: map_at_10
            value: 86.817
          - type: map_at_20
            value: 86.817
          - type: map_at_100
            value: 86.817
          - type: map_at_1000
            value: 86.817
          - type: recall_at_1
            value: 78
          - type: recall_at_3
            value: 94
          - type: recall_at_5
            value: 98
          - type: recall_at_10
            value: 100
          - type: recall_at_20
            value: 100
          - type: recall_at_100
            value: 100
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 78
          - type: precision_at_1
            value: 78
          - type: precision_at_3
            value: 31.333
          - type: precision_at_5
            value: 19.6
          - type: precision_at_10
            value: 10
          - type: precision_at_20
            value: 5
          - type: precision_at_100
            value: 1
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 78
          - type: mrr_at_3
            value: 85.6667
          - type: mrr_at_5
            value: 86.5667
          - type: mrr_at_10
            value: 86.8167
          - type: mrr_at_20
            value: 86.8167
          - type: mrr_at_100
            value: 86.8167
          - type: mrr_at_1000
            value: 86.8167
          - type: hit_rate_at_1
            value: 78
          - type: hit_rate_at_3
            value: 94
          - type: hit_rate_at_5
            value: 98
          - type: hit_rate_at_10
            value: 100
          - type: hit_rate_at_20
            value: 100
          - type: hit_rate_at_100
            value: 100
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: dwzhu/LongEmbed
          name: MTEB LEMBNeedleRetrieval
          config: default
          split: test_2048
          revision: 6e346642246bfb4928c560ee08640dc84d074e8c
        metrics:
          - type: ndcg_at_1
            value: 68
          - type: ndcg_at_3
            value: 84.357
          - type: ndcg_at_5
            value: 85.13
          - type: ndcg_at_10
            value: 85.843
          - type: ndcg_at_20
            value: 85.843
          - type: ndcg_at_100
            value: 85.843
          - type: ndcg_at_1000
            value: 85.843
          - type: map_at_1
            value: 68
          - type: map_at_3
            value: 80.333
          - type: map_at_5
            value: 80.733
          - type: map_at_10
            value: 81.067
          - type: map_at_20
            value: 81.067
          - type: map_at_100
            value: 81.067
          - type: map_at_1000
            value: 81.067
          - type: recall_at_1
            value: 68
          - type: recall_at_3
            value: 96
          - type: recall_at_5
            value: 98
          - type: recall_at_10
            value: 100
          - type: recall_at_20
            value: 100
          - type: recall_at_100
            value: 100
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 68
          - type: precision_at_1
            value: 68
          - type: precision_at_3
            value: 32
          - type: precision_at_5
            value: 19.6
          - type: precision_at_10
            value: 10
          - type: precision_at_20
            value: 5
          - type: precision_at_100
            value: 1
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 68
          - type: mrr_at_3
            value: 80.3333
          - type: mrr_at_5
            value: 80.7333
          - type: mrr_at_10
            value: 81.0667
          - type: mrr_at_20
            value: 81.0667
          - type: mrr_at_100
            value: 81.0667
          - type: mrr_at_1000
            value: 81.0667
          - type: hit_rate_at_1
            value: 68
          - type: hit_rate_at_3
            value: 96
          - type: hit_rate_at_5
            value: 98
          - type: hit_rate_at_10
            value: 100
          - type: hit_rate_at_20
            value: 100
          - type: hit_rate_at_100
            value: 100
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: dwzhu/LongEmbed
          name: MTEB LEMBNeedleRetrieval
          config: default
          split: test_4096
          revision: 6e346642246bfb4928c560ee08640dc84d074e8c
        metrics:
          - type: ndcg_at_1
            value: 76
          - type: ndcg_at_3
            value: 85.833
          - type: ndcg_at_5
            value: 88.242
          - type: ndcg_at_10
            value: 88.242
          - type: ndcg_at_20
            value: 88.742
          - type: ndcg_at_100
            value: 88.742
          - type: ndcg_at_1000
            value: 88.742
          - type: map_at_1
            value: 76
          - type: map_at_3
            value: 83.667
          - type: map_at_5
            value: 84.967
          - type: map_at_10
            value: 84.967
          - type: map_at_20
            value: 85.1
          - type: map_at_100
            value: 85.1
          - type: map_at_1000
            value: 85.1
          - type: recall_at_1
            value: 76
          - type: recall_at_3
            value: 92
          - type: recall_at_5
            value: 98
          - type: recall_at_10
            value: 98
          - type: recall_at_20
            value: 100
          - type: recall_at_100
            value: 100
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 76
          - type: precision_at_1
            value: 76
          - type: precision_at_3
            value: 30.667
          - type: precision_at_5
            value: 19.6
          - type: precision_at_10
            value: 9.8
          - type: precision_at_20
            value: 5
          - type: precision_at_100
            value: 1
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 76
          - type: mrr_at_3
            value: 83.6667
          - type: mrr_at_5
            value: 84.9667
          - type: mrr_at_10
            value: 84.9667
          - type: mrr_at_20
            value: 85.1
          - type: mrr_at_100
            value: 85.1
          - type: mrr_at_1000
            value: 85.1
          - type: hit_rate_at_1
            value: 76
          - type: hit_rate_at_3
            value: 92
          - type: hit_rate_at_5
            value: 98
          - type: hit_rate_at_10
            value: 98
          - type: hit_rate_at_20
            value: 100
          - type: hit_rate_at_100
            value: 100
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: dwzhu/LongEmbed
          name: MTEB LEMBNeedleRetrieval
          config: default
          split: test_8192
          revision: 6e346642246bfb4928c560ee08640dc84d074e8c
        metrics:
          - type: ndcg_at_1
            value: 62
          - type: ndcg_at_3
            value: 74.357
          - type: ndcg_at_5
            value: 76.079
          - type: ndcg_at_10
            value: 80.018
          - type: ndcg_at_20
            value: 80.497
          - type: ndcg_at_100
            value: 80.497
          - type: ndcg_at_1000
            value: 80.497
          - type: map_at_1
            value: 62
          - type: map_at_3
            value: 71.667
          - type: map_at_5
            value: 72.667
          - type: map_at_10
            value: 74.327
          - type: map_at_20
            value: 74.445
          - type: map_at_100
            value: 74.445
          - type: map_at_1000
            value: 74.445
          - type: recall_at_1
            value: 62
          - type: recall_at_3
            value: 82
          - type: recall_at_5
            value: 86
          - type: recall_at_10
            value: 98
          - type: recall_at_20
            value: 100
          - type: recall_at_100
            value: 100
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 62
          - type: precision_at_1
            value: 62
          - type: precision_at_3
            value: 27.333
          - type: precision_at_5
            value: 17.2
          - type: precision_at_10
            value: 9.8
          - type: precision_at_20
            value: 5
          - type: precision_at_100
            value: 1
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 62
          - type: mrr_at_3
            value: 71.6667
          - type: mrr_at_5
            value: 72.6667
          - type: mrr_at_10
            value: 74.327
          - type: mrr_at_20
            value: 74.4446
          - type: mrr_at_100
            value: 74.4446
          - type: mrr_at_1000
            value: 74.4446
          - type: hit_rate_at_1
            value: 62
          - type: hit_rate_at_3
            value: 82
          - type: hit_rate_at_5
            value: 86
          - type: hit_rate_at_10
            value: 98
          - type: hit_rate_at_20
            value: 100
          - type: hit_rate_at_100
            value: 100
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: dwzhu/LongEmbed
          name: MTEB LEMBNeedleRetrieval
          config: default
          split: test_16384
          revision: 6e346642246bfb4928c560ee08640dc84d074e8c
        metrics:
          - type: ndcg_at_1
            value: 54
          - type: ndcg_at_3
            value: 67.095
          - type: ndcg_at_5
            value: 72.861
          - type: ndcg_at_10
            value: 74.159
          - type: ndcg_at_20
            value: 75.722
          - type: ndcg_at_100
            value: 75.722
          - type: ndcg_at_1000
            value: 75.722
          - type: map_at_1
            value: 54
          - type: map_at_3
            value: 64
          - type: map_at_5
            value: 67.2
          - type: map_at_10
            value: 67.736
          - type: map_at_20
            value: 68.186
          - type: map_at_100
            value: 68.186
          - type: map_at_1000
            value: 68.186
          - type: recall_at_1
            value: 54
          - type: recall_at_3
            value: 76
          - type: recall_at_5
            value: 90
          - type: recall_at_10
            value: 94
          - type: recall_at_20
            value: 100
          - type: recall_at_100
            value: 100
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 54
          - type: precision_at_1
            value: 54
          - type: precision_at_3
            value: 25.333
          - type: precision_at_5
            value: 18
          - type: precision_at_10
            value: 9.4
          - type: precision_at_20
            value: 5
          - type: precision_at_100
            value: 1
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 54
          - type: mrr_at_3
            value: 64
          - type: mrr_at_5
            value: 67.2
          - type: mrr_at_10
            value: 67.7357
          - type: mrr_at_20
            value: 68.1863
          - type: mrr_at_100
            value: 68.1863
          - type: mrr_at_1000
            value: 68.1863
          - type: hit_rate_at_1
            value: 54
          - type: hit_rate_at_3
            value: 76
          - type: hit_rate_at_5
            value: 90
          - type: hit_rate_at_10
            value: 94
          - type: hit_rate_at_20
            value: 100
          - type: hit_rate_at_100
            value: 100
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: dwzhu/LongEmbed
          name: MTEB LEMBNeedleRetrieval
          config: default
          split: test_32768
          revision: 6e346642246bfb4928c560ee08640dc84d074e8c
        metrics:
          - type: ndcg_at_1
            value: 32
          - type: ndcg_at_3
            value: 53.404
          - type: ndcg_at_5
            value: 58.222
          - type: ndcg_at_10
            value: 60.934
          - type: ndcg_at_20
            value: 63.568
          - type: ndcg_at_100
            value: 63.949
          - type: ndcg_at_1000
            value: 63.949
          - type: map_at_1
            value: 32
          - type: map_at_3
            value: 48.333
          - type: map_at_5
            value: 50.933
          - type: map_at_10
            value: 52.124
          - type: map_at_20
            value: 52.9
          - type: map_at_100
            value: 52.955
          - type: map_at_1000
            value: 52.955
          - type: recall_at_1
            value: 32
          - type: recall_at_3
            value: 68
          - type: recall_at_5
            value: 80
          - type: recall_at_10
            value: 88
          - type: recall_at_20
            value: 98
          - type: recall_at_100
            value: 100
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 32
          - type: precision_at_1
            value: 32
          - type: precision_at_3
            value: 22.667
          - type: precision_at_5
            value: 16
          - type: precision_at_10
            value: 8.8
          - type: precision_at_20
            value: 4.9
          - type: precision_at_100
            value: 1
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 32
          - type: mrr_at_3
            value: 48.3333
          - type: mrr_at_5
            value: 50.9333
          - type: mrr_at_10
            value: 52.1238
          - type: mrr_at_20
            value: 52.9005
          - type: mrr_at_100
            value: 52.9546
          - type: mrr_at_1000
            value: 52.9546
          - type: hit_rate_at_1
            value: 32
          - type: hit_rate_at_3
            value: 68
          - type: hit_rate_at_5
            value: 80
          - type: hit_rate_at_10
            value: 88
          - type: hit_rate_at_20
            value: 98
          - type: hit_rate_at_100
            value: 100
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: mteb/LEMBPasskeyRetrieval
          name: MTEB LEMBPasskeyRetrieval
          config: default
          split: test_256
          revision: 86ac5a6437198a94ecd666440c78b3e78f258274
        metrics:
          - type: ndcg_at_1
            value: 100
          - type: ndcg_at_3
            value: 100
          - type: ndcg_at_5
            value: 100
          - type: ndcg_at_10
            value: 100
          - type: ndcg_at_20
            value: 100
          - type: ndcg_at_100
            value: 100
          - type: ndcg_at_1000
            value: 100
          - type: map_at_1
            value: 100
          - type: map_at_3
            value: 100
          - type: map_at_5
            value: 100
          - type: map_at_10
            value: 100
          - type: map_at_20
            value: 100
          - type: map_at_100
            value: 100
          - type: map_at_1000
            value: 100
          - type: recall_at_1
            value: 100
          - type: recall_at_3
            value: 100
          - type: recall_at_5
            value: 100
          - type: recall_at_10
            value: 100
          - type: recall_at_20
            value: 100
          - type: recall_at_100
            value: 100
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 100
          - type: precision_at_1
            value: 100
          - type: precision_at_3
            value: 33.333
          - type: precision_at_5
            value: 20
          - type: precision_at_10
            value: 10
          - type: precision_at_20
            value: 5
          - type: precision_at_100
            value: 1
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 100
          - type: mrr_at_3
            value: 100
          - type: mrr_at_5
            value: 100
          - type: mrr_at_10
            value: 100
          - type: mrr_at_20
            value: 100
          - type: mrr_at_100
            value: 100
          - type: mrr_at_1000
            value: 100
          - type: hit_rate_at_1
            value: 100
          - type: hit_rate_at_3
            value: 100
          - type: hit_rate_at_5
            value: 100
          - type: hit_rate_at_10
            value: 100
          - type: hit_rate_at_20
            value: 100
          - type: hit_rate_at_100
            value: 100
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: mteb/LEMBPasskeyRetrieval
          name: MTEB LEMBPasskeyRetrieval
          config: default
          split: test_512
          revision: 86ac5a6437198a94ecd666440c78b3e78f258274
        metrics:
          - type: ndcg_at_1
            value: 100
          - type: ndcg_at_3
            value: 100
          - type: ndcg_at_5
            value: 100
          - type: ndcg_at_10
            value: 100
          - type: ndcg_at_20
            value: 100
          - type: ndcg_at_100
            value: 100
          - type: ndcg_at_1000
            value: 100
          - type: map_at_1
            value: 100
          - type: map_at_3
            value: 100
          - type: map_at_5
            value: 100
          - type: map_at_10
            value: 100
          - type: map_at_20
            value: 100
          - type: map_at_100
            value: 100
          - type: map_at_1000
            value: 100
          - type: recall_at_1
            value: 100
          - type: recall_at_3
            value: 100
          - type: recall_at_5
            value: 100
          - type: recall_at_10
            value: 100
          - type: recall_at_20
            value: 100
          - type: recall_at_100
            value: 100
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 100
          - type: precision_at_1
            value: 100
          - type: precision_at_3
            value: 33.333
          - type: precision_at_5
            value: 20
          - type: precision_at_10
            value: 10
          - type: precision_at_20
            value: 5
          - type: precision_at_100
            value: 1
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 100
          - type: mrr_at_3
            value: 100
          - type: mrr_at_5
            value: 100
          - type: mrr_at_10
            value: 100
          - type: mrr_at_20
            value: 100
          - type: mrr_at_100
            value: 100
          - type: mrr_at_1000
            value: 100
          - type: hit_rate_at_1
            value: 100
          - type: hit_rate_at_3
            value: 100
          - type: hit_rate_at_5
            value: 100
          - type: hit_rate_at_10
            value: 100
          - type: hit_rate_at_20
            value: 100
          - type: hit_rate_at_100
            value: 100
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: mteb/LEMBPasskeyRetrieval
          name: MTEB LEMBPasskeyRetrieval
          config: default
          split: test_1024
          revision: 86ac5a6437198a94ecd666440c78b3e78f258274
        metrics:
          - type: ndcg_at_1
            value: 100
          - type: ndcg_at_3
            value: 100
          - type: ndcg_at_5
            value: 100
          - type: ndcg_at_10
            value: 100
          - type: ndcg_at_20
            value: 100
          - type: ndcg_at_100
            value: 100
          - type: ndcg_at_1000
            value: 100
          - type: map_at_1
            value: 100
          - type: map_at_3
            value: 100
          - type: map_at_5
            value: 100
          - type: map_at_10
            value: 100
          - type: map_at_20
            value: 100
          - type: map_at_100
            value: 100
          - type: map_at_1000
            value: 100
          - type: recall_at_1
            value: 100
          - type: recall_at_3
            value: 100
          - type: recall_at_5
            value: 100
          - type: recall_at_10
            value: 100
          - type: recall_at_20
            value: 100
          - type: recall_at_100
            value: 100
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 100
          - type: precision_at_1
            value: 100
          - type: precision_at_3
            value: 33.333
          - type: precision_at_5
            value: 20
          - type: precision_at_10
            value: 10
          - type: precision_at_20
            value: 5
          - type: precision_at_100
            value: 1
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 100
          - type: mrr_at_3
            value: 100
          - type: mrr_at_5
            value: 100
          - type: mrr_at_10
            value: 100
          - type: mrr_at_20
            value: 100
          - type: mrr_at_100
            value: 100
          - type: mrr_at_1000
            value: 100
          - type: hit_rate_at_1
            value: 100
          - type: hit_rate_at_3
            value: 100
          - type: hit_rate_at_5
            value: 100
          - type: hit_rate_at_10
            value: 100
          - type: hit_rate_at_20
            value: 100
          - type: hit_rate_at_100
            value: 100
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: mteb/LEMBPasskeyRetrieval
          name: MTEB LEMBPasskeyRetrieval
          config: default
          split: test_2048
          revision: 86ac5a6437198a94ecd666440c78b3e78f258274
        metrics:
          - type: ndcg_at_1
            value: 100
          - type: ndcg_at_3
            value: 100
          - type: ndcg_at_5
            value: 100
          - type: ndcg_at_10
            value: 100
          - type: ndcg_at_20
            value: 100
          - type: ndcg_at_100
            value: 100
          - type: ndcg_at_1000
            value: 100
          - type: map_at_1
            value: 100
          - type: map_at_3
            value: 100
          - type: map_at_5
            value: 100
          - type: map_at_10
            value: 100
          - type: map_at_20
            value: 100
          - type: map_at_100
            value: 100
          - type: map_at_1000
            value: 100
          - type: recall_at_1
            value: 100
          - type: recall_at_3
            value: 100
          - type: recall_at_5
            value: 100
          - type: recall_at_10
            value: 100
          - type: recall_at_20
            value: 100
          - type: recall_at_100
            value: 100
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 100
          - type: precision_at_1
            value: 100
          - type: precision_at_3
            value: 33.333
          - type: precision_at_5
            value: 20
          - type: precision_at_10
            value: 10
          - type: precision_at_20
            value: 5
          - type: precision_at_100
            value: 1
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 100
          - type: mrr_at_3
            value: 100
          - type: mrr_at_5
            value: 100
          - type: mrr_at_10
            value: 100
          - type: mrr_at_20
            value: 100
          - type: mrr_at_100
            value: 100
          - type: mrr_at_1000
            value: 100
          - type: hit_rate_at_1
            value: 100
          - type: hit_rate_at_3
            value: 100
          - type: hit_rate_at_5
            value: 100
          - type: hit_rate_at_10
            value: 100
          - type: hit_rate_at_20
            value: 100
          - type: hit_rate_at_100
            value: 100
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: mteb/LEMBPasskeyRetrieval
          name: MTEB LEMBPasskeyRetrieval
          config: default
          split: test_4096
          revision: 86ac5a6437198a94ecd666440c78b3e78f258274
        metrics:
          - type: ndcg_at_1
            value: 100
          - type: ndcg_at_3
            value: 100
          - type: ndcg_at_5
            value: 100
          - type: ndcg_at_10
            value: 100
          - type: ndcg_at_20
            value: 100
          - type: ndcg_at_100
            value: 100
          - type: ndcg_at_1000
            value: 100
          - type: map_at_1
            value: 100
          - type: map_at_3
            value: 100
          - type: map_at_5
            value: 100
          - type: map_at_10
            value: 100
          - type: map_at_20
            value: 100
          - type: map_at_100
            value: 100
          - type: map_at_1000
            value: 100
          - type: recall_at_1
            value: 100
          - type: recall_at_3
            value: 100
          - type: recall_at_5
            value: 100
          - type: recall_at_10
            value: 100
          - type: recall_at_20
            value: 100
          - type: recall_at_100
            value: 100
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 100
          - type: precision_at_1
            value: 100
          - type: precision_at_3
            value: 33.333
          - type: precision_at_5
            value: 20
          - type: precision_at_10
            value: 10
          - type: precision_at_20
            value: 5
          - type: precision_at_100
            value: 1
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 100
          - type: mrr_at_3
            value: 100
          - type: mrr_at_5
            value: 100
          - type: mrr_at_10
            value: 100
          - type: mrr_at_20
            value: 100
          - type: mrr_at_100
            value: 100
          - type: mrr_at_1000
            value: 100
          - type: hit_rate_at_1
            value: 100
          - type: hit_rate_at_3
            value: 100
          - type: hit_rate_at_5
            value: 100
          - type: hit_rate_at_10
            value: 100
          - type: hit_rate_at_20
            value: 100
          - type: hit_rate_at_100
            value: 100
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: mteb/LEMBPasskeyRetrieval
          name: MTEB LEMBPasskeyRetrieval
          config: default
          split: test_8192
          revision: 86ac5a6437198a94ecd666440c78b3e78f258274
        metrics:
          - type: ndcg_at_1
            value: 100
          - type: ndcg_at_3
            value: 100
          - type: ndcg_at_5
            value: 100
          - type: ndcg_at_10
            value: 100
          - type: ndcg_at_20
            value: 100
          - type: ndcg_at_100
            value: 100
          - type: ndcg_at_1000
            value: 100
          - type: map_at_1
            value: 100
          - type: map_at_3
            value: 100
          - type: map_at_5
            value: 100
          - type: map_at_10
            value: 100
          - type: map_at_20
            value: 100
          - type: map_at_100
            value: 100
          - type: map_at_1000
            value: 100
          - type: recall_at_1
            value: 100
          - type: recall_at_3
            value: 100
          - type: recall_at_5
            value: 100
          - type: recall_at_10
            value: 100
          - type: recall_at_20
            value: 100
          - type: recall_at_100
            value: 100
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 100
          - type: precision_at_1
            value: 100
          - type: precision_at_3
            value: 33.333
          - type: precision_at_5
            value: 20
          - type: precision_at_10
            value: 10
          - type: precision_at_20
            value: 5
          - type: precision_at_100
            value: 1
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 100
          - type: mrr_at_3
            value: 100
          - type: mrr_at_5
            value: 100
          - type: mrr_at_10
            value: 100
          - type: mrr_at_20
            value: 100
          - type: mrr_at_100
            value: 100
          - type: mrr_at_1000
            value: 100
          - type: hit_rate_at_1
            value: 100
          - type: hit_rate_at_3
            value: 100
          - type: hit_rate_at_5
            value: 100
          - type: hit_rate_at_10
            value: 100
          - type: hit_rate_at_20
            value: 100
          - type: hit_rate_at_100
            value: 100
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: mteb/LEMBPasskeyRetrieval
          name: MTEB LEMBPasskeyRetrieval
          config: default
          split: test_16384
          revision: 86ac5a6437198a94ecd666440c78b3e78f258274
        metrics:
          - type: ndcg_at_1
            value: 94
          - type: ndcg_at_3
            value: 97.786
          - type: ndcg_at_5
            value: 97.786
          - type: ndcg_at_10
            value: 97.786
          - type: ndcg_at_20
            value: 97.786
          - type: ndcg_at_100
            value: 97.786
          - type: ndcg_at_1000
            value: 97.786
          - type: map_at_1
            value: 94
          - type: map_at_3
            value: 97
          - type: map_at_5
            value: 97
          - type: map_at_10
            value: 97
          - type: map_at_20
            value: 97
          - type: map_at_100
            value: 97
          - type: map_at_1000
            value: 97
          - type: recall_at_1
            value: 94
          - type: recall_at_3
            value: 100
          - type: recall_at_5
            value: 100
          - type: recall_at_10
            value: 100
          - type: recall_at_20
            value: 100
          - type: recall_at_100
            value: 100
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 94
          - type: precision_at_1
            value: 94
          - type: precision_at_3
            value: 33.333
          - type: precision_at_5
            value: 20
          - type: precision_at_10
            value: 10
          - type: precision_at_20
            value: 5
          - type: precision_at_100
            value: 1
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 94
          - type: mrr_at_3
            value: 97
          - type: mrr_at_5
            value: 97
          - type: mrr_at_10
            value: 97
          - type: mrr_at_20
            value: 97
          - type: mrr_at_100
            value: 97
          - type: mrr_at_1000
            value: 97
          - type: hit_rate_at_1
            value: 94
          - type: hit_rate_at_3
            value: 100
          - type: hit_rate_at_5
            value: 100
          - type: hit_rate_at_10
            value: 100
          - type: hit_rate_at_20
            value: 100
          - type: hit_rate_at_100
            value: 100
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: mteb/LEMBPasskeyRetrieval
          name: MTEB LEMBPasskeyRetrieval
          config: default
          split: test_32768
          revision: 86ac5a6437198a94ecd666440c78b3e78f258274
        metrics:
          - type: ndcg_at_1
            value: 98
          - type: ndcg_at_3
            value: 99.262
          - type: ndcg_at_5
            value: 99.262
          - type: ndcg_at_10
            value: 99.262
          - type: ndcg_at_20
            value: 99.262
          - type: ndcg_at_100
            value: 99.262
          - type: ndcg_at_1000
            value: 99.262
          - type: map_at_1
            value: 98
          - type: map_at_3
            value: 99
          - type: map_at_5
            value: 99
          - type: map_at_10
            value: 99
          - type: map_at_20
            value: 99
          - type: map_at_100
            value: 99
          - type: map_at_1000
            value: 99
          - type: recall_at_1
            value: 98
          - type: recall_at_3
            value: 100
          - type: recall_at_5
            value: 100
          - type: recall_at_10
            value: 100
          - type: recall_at_20
            value: 100
          - type: recall_at_100
            value: 100
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 98
          - type: precision_at_1
            value: 98
          - type: precision_at_3
            value: 33.333
          - type: precision_at_5
            value: 20
          - type: precision_at_10
            value: 10
          - type: precision_at_20
            value: 5
          - type: precision_at_100
            value: 1
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 98
          - type: mrr_at_3
            value: 99
          - type: mrr_at_5
            value: 99
          - type: mrr_at_10
            value: 99
          - type: mrr_at_20
            value: 99
          - type: mrr_at_100
            value: 99
          - type: mrr_at_1000
            value: 99
          - type: hit_rate_at_1
            value: 98
          - type: hit_rate_at_3
            value: 100
          - type: hit_rate_at_5
            value: 100
          - type: hit_rate_at_10
            value: 100
          - type: hit_rate_at_20
            value: 100
          - type: hit_rate_at_100
            value: 100
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: dwzhu/LongEmbed
          name: MTEB LEMBQMSumRetrieval
          config: default
          split: test
          revision: 6e346642246bfb4928c560ee08640dc84d074e8c
        metrics:
          - type: ndcg_at_1
            value: 40.21
          - type: ndcg_at_3
            value: 49.733
          - type: ndcg_at_5
            value: 52.689
          - type: ndcg_at_10
            value: 55.278
          - type: ndcg_at_20
            value: 57.57
          - type: ndcg_at_100
            value: 60.835
          - type: ndcg_at_1000
            value: 61.017
          - type: map_at_1
            value: 40.21
          - type: map_at_3
            value: 47.37
          - type: map_at_5
            value: 49.003
          - type: map_at_10
            value: 50.062
          - type: map_at_20
            value: 50.687
          - type: map_at_100
            value: 51.122
          - type: map_at_1000
            value: 51.134
          - type: recall_at_1
            value: 40.21
          - type: recall_at_3
            value: 56.582
          - type: recall_at_5
            value: 63.785
          - type: recall_at_10
            value: 71.84
          - type: recall_at_20
            value: 80.943
          - type: recall_at_100
            value: 98.756
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 40.21
          - type: precision_at_1
            value: 40.21
          - type: precision_at_3
            value: 18.861
          - type: precision_at_5
            value: 12.757
          - type: precision_at_10
            value: 7.184
          - type: precision_at_20
            value: 4.047
          - type: precision_at_100
            value: 0.988
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 40.2096
          - type: mrr_at_3
            value: 47.3696
          - type: mrr_at_5
            value: 49.0035
          - type: mrr_at_10
            value: 50.0624
          - type: mrr_at_20
            value: 50.6865
          - type: mrr_at_100
            value: 51.1225
          - type: mrr_at_1000
            value: 51.1336
          - type: hit_rate_at_1
            value: 40.21
          - type: hit_rate_at_3
            value: 56.582
          - type: hit_rate_at_5
            value: 63.785
          - type: hit_rate_at_10
            value: 71.84
          - type: hit_rate_at_20
            value: 80.943
          - type: hit_rate_at_100
            value: 98.756
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: dwzhu/LongEmbed
          name: MTEB LEMBSummScreenFDRetrieval
          config: default
          split: validation
          revision: 6e346642246bfb4928c560ee08640dc84d074e8c
        metrics:
          - type: ndcg_at_1
            value: 95.536
          - type: ndcg_at_3
            value: 97.148
          - type: ndcg_at_5
            value: 97.519
          - type: ndcg_at_10
            value: 97.625
          - type: ndcg_at_20
            value: 97.625
          - type: ndcg_at_100
            value: 97.737
          - type: ndcg_at_1000
            value: 97.737
          - type: map_at_1
            value: 95.536
          - type: map_at_3
            value: 96.776
          - type: map_at_5
            value: 96.984
          - type: map_at_10
            value: 97.034
          - type: map_at_20
            value: 97.034
          - type: map_at_100
            value: 97.051
          - type: map_at_1000
            value: 97.051
          - type: recall_at_1
            value: 95.536
          - type: recall_at_3
            value: 98.214
          - type: recall_at_5
            value: 99.107
          - type: recall_at_10
            value: 99.405
          - type: recall_at_20
            value: 99.405
          - type: recall_at_100
            value: 100
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 95.536
          - type: precision_at_1
            value: 95.536
          - type: precision_at_3
            value: 32.738
          - type: precision_at_5
            value: 19.821
          - type: precision_at_10
            value: 9.94
          - type: precision_at_20
            value: 4.97
          - type: precision_at_100
            value: 1
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 95.5357
          - type: mrr_at_3
            value: 96.7758
          - type: mrr_at_5
            value: 96.9841
          - type: mrr_at_10
            value: 97.0337
          - type: mrr_at_20
            value: 97.0337
          - type: mrr_at_100
            value: 97.0511
          - type: mrr_at_1000
            value: 97.0511
          - type: hit_rate_at_1
            value: 95.536
          - type: hit_rate_at_3
            value: 98.214
          - type: hit_rate_at_5
            value: 99.107
          - type: hit_rate_at_10
            value: 99.405
          - type: hit_rate_at_20
            value: 99.405
          - type: hit_rate_at_100
            value: 100
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: dwzhu/LongEmbed
          name: MTEB LEMBWikimQARetrieval
          config: default
          split: test
          revision: 6e346642246bfb4928c560ee08640dc84d074e8c
        metrics:
          - type: ndcg_at_1
            value: 89
          - type: ndcg_at_3
            value: 91.559
          - type: ndcg_at_5
            value: 92.521
          - type: ndcg_at_10
            value: 92.862
          - type: ndcg_at_20
            value: 93.209
          - type: ndcg_at_100
            value: 93.406
          - type: ndcg_at_1000
            value: 93.547
          - type: map_at_1
            value: 89
          - type: map_at_3
            value: 90.944
          - type: map_at_5
            value: 91.478
          - type: map_at_10
            value: 91.629
          - type: map_at_20
            value: 91.729
          - type: map_at_100
            value: 91.761
          - type: map_at_1000
            value: 91.768
          - type: recall_at_1
            value: 89
          - type: recall_at_3
            value: 93.333
          - type: recall_at_5
            value: 95.667
          - type: recall_at_10
            value: 96.667
          - type: recall_at_20
            value: 98
          - type: recall_at_100
            value: 99
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 89
          - type: precision_at_1
            value: 89
          - type: precision_at_3
            value: 31.111
          - type: precision_at_5
            value: 19.133
          - type: precision_at_10
            value: 9.667
          - type: precision_at_20
            value: 4.9
          - type: precision_at_100
            value: 0.99
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 89
          - type: mrr_at_3
            value: 90.9444
          - type: mrr_at_5
            value: 91.4778
          - type: mrr_at_10
            value: 91.6286
          - type: mrr_at_20
            value: 91.7292
          - type: mrr_at_100
            value: 91.7608
          - type: mrr_at_1000
            value: 91.7683
          - type: hit_rate_at_1
            value: 89
          - type: hit_rate_at_3
            value: 93.333
          - type: hit_rate_at_5
            value: 95.667
          - type: hit_rate_at_10
            value: 96.667
          - type: hit_rate_at_20
            value: 98
          - type: hit_rate_at_100
            value: 99
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Classification
        dataset:
          type: mteb/MTOPDomainClassification
          name: MTEB MTOPDomainClassification (en)
          config: en
          split: test
          revision: a76d16fae880597b9c73047b50159220a441cb54
        metrics:
          - type: accuracy
            value: 94.5987
          - type: f1
            value: 94.2853
          - type: f1_weighted
            value: 94.5968
          - type: precision
            value: 94.1672
          - type: precision_weighted
            value: 94.7782
          - type: recall
            value: 94.5935
          - type: recall_weighted
            value: 94.5987
          - type: ap
            value: .nan
          - type: ap_weighted
            value: .nan
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (en)
          config: en
          split: test
          revision: 4672e20407010da34463acc759c162ca9734bca6
        metrics:
          - type: accuracy
            value: 75.6792
          - type: f1
            value: 73.216
          - type: f1_weighted
            value: 75.5343
          - type: precision
            value: 71.0843
          - type: precision_weighted
            value: 78.6203
          - type: recall
            value: 80.5074
          - type: recall_weighted
            value: 75.6792
          - type: ap
            value: .nan
          - type: ap_weighted
            value: .nan
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (en)
          config: en
          split: test
          revision: fad2c6e8459f9e1c45d9315f4953d921437d70f8
        metrics:
          - type: accuracy
            value: 77.6194
          - type: f1
            value: 77.3532
          - type: f1_weighted
            value: 77.456
          - type: precision
            value: 75.5415
          - type: precision_weighted
            value: 79.5373
          - type: recall
            value: 81.6365
          - type: recall_weighted
            value: 77.6194
          - type: ap
            value: .nan
          - type: ap_weighted
            value: .nan
      - task:
          type: Clustering
        dataset:
          type: mteb/medrxiv-clustering-p2p
          name: MTEB MedrxivClusteringP2P.v2
          config: default
          split: test
          revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73
        metrics:
          - type: v_measure
            value: 38.5612
          - type: v_measure_std
            value: 0.8631
          - type: ami
            value: 37.1726
          - type: ami_std
            value: 0.8709
      - task:
          type: Clustering
        dataset:
          type: mteb/medrxiv-clustering-s2s
          name: MTEB MedrxivClusteringS2S.v2
          config: default
          split: test
          revision: 35191c8c0dca72d8ff3efcd72aa802307d469663
        metrics:
          - type: v_measure
            value: 38.4619
          - type: v_measure_std
            value: 0.5992
          - type: ami
            value: 37.0538
          - type: ami_std
            value: 0.6203
      - task:
          type: Reranking
        dataset:
          type: mteb/MindSmallReranking
          name: MTEB MindSmallReranking
          config: default
          split: test
          revision: 227478e3235572039f4f7661840e059f31ef6eb1
        metrics:
          - type: ndcg_at_1
            value: 13.291
          - type: ndcg_at_3
            value: 20.876
          - type: ndcg_at_5
            value: 25.431
          - type: ndcg_at_10
            value: 31.525
          - type: ndcg_at_20
            value: 36.685
          - type: ndcg_at_100
            value: 43.086
          - type: ndcg_at_1000
            value: 43.44
          - type: map_at_1
            value: 10.172
          - type: map_at_3
            value: 17.181
          - type: map_at_5
            value: 19.886
          - type: map_at_10
            value: 22.615
          - type: map_at_20
            value: 24.3
          - type: map_at_100
            value: 25.663
          - type: map_at_1000
            value: 25.707
          - type: recall_at_1
            value: 10.172
          - type: recall_at_3
            value: 25.909
          - type: recall_at_5
            value: 36.784
          - type: recall_at_10
            value: 54.026
          - type: recall_at_20
            value: 71.596
          - type: recall_at_100
            value: 98.286
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 10.172
          - type: precision_at_1
            value: 13.291
          - type: precision_at_3
            value: 11.478
          - type: precision_at_5
            value: 10.023
          - type: precision_at_10
            value: 7.757
          - type: precision_at_20
            value: 5.51
          - type: precision_at_100
            value: 1.764
          - type: precision_at_1000
            value: 0.183
          - type: mrr_at_1
            value: 13.2911
          - type: mrr_at_3
            value: 21.5975
          - type: mrr_at_5
            value: 24.5174
          - type: mrr_at_10
            value: 27.0451
          - type: mrr_at_20
            value: 28.2346
          - type: mrr_at_100
            value: 28.7805
          - type: mrr_at_1000
            value: 28.7853
          - type: hit_rate_at_1
            value: 13.291
          - type: hit_rate_at_3
            value: 32.627
          - type: hit_rate_at_5
            value: 45.48
          - type: hit_rate_at_10
            value: 64.401
          - type: hit_rate_at_20
            value: 81.328
          - type: hit_rate_at_100
            value: 99.411
          - type: hit_rate_at_1000
            value: 100
          - type: max_over_subqueries_ndcg_at_1
            value: 17.962
          - type: max_over_subqueries_ndcg_at_3
            value: 28.062
          - type: max_over_subqueries_ndcg_at_5
            value: 33.29
          - type: max_over_subqueries_ndcg_at_10
            value: 39.321
          - type: max_over_subqueries_ndcg_at_20
            value: 43.808
          - type: max_over_subqueries_ndcg_at_100
            value: 48.128
          - type: max_over_subqueries_ndcg_at_1000
            value: 48.3
          - type: max_over_subqueries_map_at_1
            value: 14.947
          - type: max_over_subqueries_map_at_3
            value: 23.977
          - type: max_over_subqueries_map_at_5
            value: 27.07
          - type: max_over_subqueries_map_at_10
            value: 29.789
          - type: max_over_subqueries_map_at_20
            value: 31.259
          - type: max_over_subqueries_map_at_100
            value: 32.167
          - type: max_over_subqueries_map_at_1000
            value: 32.186
          - type: max_over_subqueries_recall_at_1
            value: 14.947
          - type: max_over_subqueries_recall_at_3
            value: 35.072
          - type: max_over_subqueries_recall_at_5
            value: 47.412
          - type: max_over_subqueries_recall_at_10
            value: 64.611
          - type: max_over_subqueries_recall_at_20
            value: 80.272
          - type: max_over_subqueries_recall_at_100
            value: 99.074
          - type: max_over_subqueries_recall_at_1000
            value: 99.999
          - type: max_over_subqueries_accuracy
            value: 14.947
          - type: max_over_subqueries_precision_at_1
            value: 17.962
          - type: max_over_subqueries_precision_at_3
            value: 14.375
          - type: max_over_subqueries_precision_at_5
            value: 11.943
          - type: max_over_subqueries_precision_at_10
            value: 8.496
          - type: max_over_subqueries_precision_at_20
            value: 5.551
          - type: max_over_subqueries_precision_at_100
            value: 1.496
          - type: max_over_subqueries_precision_at_1000
            value: 0.152
          - type: max_over_subqueries_mrr_at_1
            value: 17.9622
          - type: max_over_subqueries_mrr_at_3
            value: 27.981
          - type: max_over_subqueries_mrr_at_5
            value: 31.0729
          - type: max_over_subqueries_mrr_at_10
            value: 33.4506
          - type: max_over_subqueries_mrr_at_20
            value: 34.4594
          - type: max_over_subqueries_mrr_at_100
            value: 34.8568
          - type: max_over_subqueries_mrr_at_1000
            value: 34.8601
          - type: max_over_subqueries_hit_rate_at_1
            value: 17.962
          - type: max_over_subqueries_hit_rate_at_3
            value: 41.091
          - type: max_over_subqueries_hit_rate_at_5
            value: 54.627
          - type: max_over_subqueries_hit_rate_at_10
            value: 72.307
          - type: max_over_subqueries_hit_rate_at_20
            value: 86.57
          - type: max_over_subqueries_hit_rate_at_100
            value: 99.593
          - type: max_over_subqueries_hit_rate_at_1000
            value: 99.999
      - task:
          type: Retrieval
        dataset:
          type: mteb/PIQA
          name: MTEB PIQA
          config: default
          split: test
          revision: aee23bb32dafbaddddf16cd5033d912693984d9a
        metrics:
          - type: ndcg_at_1
            value: 19.859
          - type: ndcg_at_3
            value: 30.315
          - type: ndcg_at_5
            value: 32.7
          - type: ndcg_at_10
            value: 35.051
          - type: ndcg_at_20
            value: 36.548
          - type: ndcg_at_100
            value: 39.288
          - type: ndcg_at_1000
            value: 41.229
          - type: map_at_1
            value: 19.859
          - type: map_at_3
            value: 27.875
          - type: map_at_5
            value: 29.191
          - type: map_at_10
            value: 30.147
          - type: map_at_20
            value: 30.557
          - type: map_at_100
            value: 30.94
          - type: map_at_1000
            value: 31.002
          - type: recall_at_1
            value: 19.859
          - type: recall_at_3
            value: 37.323
          - type: recall_at_5
            value: 43.145
          - type: recall_at_10
            value: 50.49
          - type: recall_at_20
            value: 56.42
          - type: recall_at_100
            value: 71.164
          - type: recall_at_1000
            value: 87.051
          - type: accuracy
            value: 19.859
          - type: precision_at_1
            value: 19.859
          - type: precision_at_3
            value: 12.441
          - type: precision_at_5
            value: 8.629
          - type: precision_at_10
            value: 5.049
          - type: precision_at_20
            value: 2.821
          - type: precision_at_100
            value: 0.712
          - type: precision_at_1000
            value: 0.087
          - type: mrr_at_1
            value: 19.8585
          - type: mrr_at_3
            value: 27.8745
          - type: mrr_at_5
            value: 29.1911
          - type: mrr_at_10
            value: 30.1468
          - type: mrr_at_20
            value: 30.5569
          - type: mrr_at_100
            value: 30.9395
          - type: mrr_at_1000
            value: 31.0023
          - type: hit_rate_at_1
            value: 19.859
          - type: hit_rate_at_3
            value: 37.323
          - type: hit_rate_at_5
            value: 43.145
          - type: hit_rate_at_10
            value: 50.49
          - type: hit_rate_at_20
            value: 56.42
          - type: hit_rate_at_100
            value: 71.164
          - type: hit_rate_at_1000
            value: 87.051
      - task:
          type: Retrieval
        dataset:
          type: mteb/Quail
          name: MTEB Quail
          config: default
          split: test
          revision: 516ac28de67229da337281f702310a08c9fe0a03
        metrics:
          - type: ndcg_at_1
            value: 2.868
          - type: ndcg_at_3
            value: 5.126
          - type: ndcg_at_5
            value: 6.409
          - type: ndcg_at_10
            value: 7.968
          - type: ndcg_at_20
            value: 9.472
          - type: ndcg_at_100
            value: 11.997
          - type: ndcg_at_1000
            value: 15.15
          - type: map_at_1
            value: 2.868
          - type: map_at_3
            value: 4.516
          - type: map_at_5
            value: 5.225
          - type: map_at_10
            value: 5.87
          - type: map_at_20
            value: 6.282
          - type: map_at_100
            value: 6.627
          - type: map_at_1000
            value: 6.728
          - type: recall_at_1
            value: 2.868
          - type: recall_at_3
            value: 6.912
          - type: recall_at_5
            value: 10.037
          - type: recall_at_10
            value: 14.853
          - type: recall_at_20
            value: 20.809
          - type: recall_at_100
            value: 34.485
          - type: recall_at_1000
            value: 60.368
          - type: accuracy
            value: 2.868
          - type: precision_at_1
            value: 2.868
          - type: precision_at_3
            value: 2.304
          - type: precision_at_5
            value: 2.007
          - type: precision_at_10
            value: 1.485
          - type: precision_at_20
            value: 1.04
          - type: precision_at_100
            value: 0.345
          - type: precision_at_1000
            value: 0.06
          - type: mrr_at_1
            value: 2.8676
          - type: mrr_at_3
            value: 4.5159
          - type: mrr_at_5
            value: 5.2255
          - type: mrr_at_10
            value: 5.8699
          - type: mrr_at_20
            value: 6.2825
          - type: mrr_at_100
            value: 6.6274
          - type: mrr_at_1000
            value: 6.7279
          - type: hit_rate_at_1
            value: 2.868
          - type: hit_rate_at_3
            value: 6.912
          - type: hit_rate_at_5
            value: 10.037
          - type: hit_rate_at_10
            value: 14.853
          - type: hit_rate_at_20
            value: 20.809
          - type: hit_rate_at_100
            value: 34.485
          - type: hit_rate_at_1000
            value: 60.368
      - task:
          type: Retrieval
        dataset:
          type: mteb/RARbCode
          name: MTEB RARbCode
          config: default
          split: test
          revision: ed0a01c3e4d39ec17badb87405e595322f8f2b86
        metrics:
          - type: ndcg_at_1
            value: 51.078
          - type: ndcg_at_3
            value: 64.478
          - type: ndcg_at_5
            value: 70.411
          - type: ndcg_at_10
            value: 72.807
          - type: ndcg_at_20
            value: 73.268
          - type: ndcg_at_100
            value: 73.56
          - type: ndcg_at_1000
            value: 73.618
          - type: map_at_1
            value: 51.078
          - type: map_at_3
            value: 61.006
          - type: map_at_5
            value: 64.295
          - type: map_at_10
            value: 65.348
          - type: map_at_20
            value: 65.475
          - type: map_at_100
            value: 65.522
          - type: map_at_1000
            value: 65.524
          - type: recall_at_1
            value: 51.078
          - type: recall_at_3
            value: 74.596
          - type: recall_at_5
            value: 89.016
          - type: recall_at_10
            value: 96.092
          - type: recall_at_20
            value: 97.911
          - type: recall_at_100
            value: 99.394
          - type: recall_at_1000
            value: 99.865
          - type: accuracy
            value: 51.078
          - type: precision_at_1
            value: 51.078
          - type: precision_at_3
            value: 24.865
          - type: precision_at_5
            value: 17.803
          - type: precision_at_10
            value: 9.609
          - type: precision_at_20
            value: 4.896
          - type: precision_at_100
            value: 0.994
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 51.0782
          - type: mrr_at_3
            value: 61.0063
          - type: mrr_at_5
            value: 64.2947
          - type: mrr_at_10
            value: 65.3481
          - type: mrr_at_20
            value: 65.4749
          - type: mrr_at_100
            value: 65.5219
          - type: mrr_at_1000
            value: 65.5237
          - type: hit_rate_at_1
            value: 51.078
          - type: hit_rate_at_3
            value: 74.596
          - type: hit_rate_at_5
            value: 89.016
          - type: hit_rate_at_10
            value: 96.092
          - type: hit_rate_at_20
            value: 97.911
          - type: hit_rate_at_100
            value: 99.394
          - type: hit_rate_at_1000
            value: 99.865
      - task:
          type: Retrieval
        dataset:
          type: mteb/RARbMath
          name: MTEB RARbMath
          config: default
          split: test
          revision: cc879a9719a10f4a80677ac26798cfd701cf94b7
        metrics:
          - type: ndcg_at_1
            value: 81.152
          - type: ndcg_at_3
            value: 83.927
          - type: ndcg_at_5
            value: 84.704
          - type: ndcg_at_10
            value: 85.521
          - type: ndcg_at_20
            value: 86.079
          - type: ndcg_at_100
            value: 86.985
          - type: ndcg_at_1000
            value: 87.279
          - type: map_at_1
            value: 81.152
          - type: map_at_3
            value: 83.259
          - type: map_at_5
            value: 83.692
          - type: map_at_10
            value: 84.031
          - type: map_at_20
            value: 84.185
          - type: map_at_100
            value: 84.31
          - type: map_at_1000
            value: 84.322
          - type: recall_at_1
            value: 81.152
          - type: recall_at_3
            value: 85.852
          - type: recall_at_5
            value: 87.735
          - type: recall_at_10
            value: 90.252
          - type: recall_at_20
            value: 92.451
          - type: recall_at_100
            value: 97.326
          - type: recall_at_1000
            value: 99.604
          - type: accuracy
            value: 81.152
          - type: precision_at_1
            value: 81.152
          - type: precision_at_3
            value: 28.617
          - type: precision_at_5
            value: 17.547
          - type: precision_at_10
            value: 9.025
          - type: precision_at_20
            value: 4.623
          - type: precision_at_100
            value: 0.973
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 81.1521
          - type: mrr_at_3
            value: 83.2595
          - type: mrr_at_5
            value: 83.6915
          - type: mrr_at_10
            value: 84.0311
          - type: mrr_at_20
            value: 84.185
          - type: mrr_at_100
            value: 84.3103
          - type: mrr_at_1000
            value: 84.3221
          - type: hit_rate_at_1
            value: 81.152
          - type: hit_rate_at_3
            value: 85.852
          - type: hit_rate_at_5
            value: 87.735
          - type: hit_rate_at_10
            value: 90.252
          - type: hit_rate_at_20
            value: 92.451
          - type: hit_rate_at_100
            value: 97.326
          - type: hit_rate_at_1000
            value: 99.604
      - task:
          type: Retrieval
        dataset:
          type: mteb/scidocs
          name: MTEB SCIDOCS
          config: default
          split: test
          revision: f8c2fcf00f625baaa80f62ec5bd9e1fff3b8ae88
        metrics:
          - type: ndcg_at_1
            value: 27.8
          - type: ndcg_at_3
            value: 22.61
          - type: ndcg_at_5
            value: 19.668
          - type: ndcg_at_10
            value: 23.84
          - type: ndcg_at_20
            value: 26.907
          - type: ndcg_at_100
            value: 32.829
          - type: ndcg_at_1000
            value: 38.371
          - type: map_at_1
            value: 5.628
          - type: map_at_3
            value: 10.207
          - type: map_at_5
            value: 12.352
          - type: map_at_10
            value: 14.613
          - type: map_at_20
            value: 15.8
          - type: map_at_100
            value: 17.02
          - type: map_at_1000
            value: 17.38
          - type: recall_at_1
            value: 5.628
          - type: recall_at_3
            value: 12.867
          - type: recall_at_5
            value: 17.447
          - type: recall_at_10
            value: 25.057
          - type: recall_at_20
            value: 32.278
          - type: recall_at_100
            value: 51.632
          - type: recall_at_1000
            value: 78.328
          - type: accuracy
            value: 5.628
          - type: precision_at_1
            value: 27.8
          - type: precision_at_3
            value: 21.133
          - type: precision_at_5
            value: 17.18
          - type: precision_at_10
            value: 12.36
          - type: precision_at_20
            value: 7.96
          - type: precision_at_100
            value: 2.541
          - type: precision_at_1000
            value: 0.386
          - type: mrr_at_1
            value: 27.8
          - type: mrr_at_3
            value: 36.2667
          - type: mrr_at_5
            value: 37.9567
          - type: mrr_at_10
            value: 39.4979
          - type: mrr_at_20
            value: 40.0757
          - type: mrr_at_100
            value: 40.505
          - type: mrr_at_1000
            value: 40.5442
          - type: hit_rate_at_1
            value: 27.8
          - type: hit_rate_at_3
            value: 47.5
          - type: hit_rate_at_5
            value: 54.8
          - type: hit_rate_at_10
            value: 66.2
          - type: hit_rate_at_20
            value: 74.7
          - type: hit_rate_at_100
            value: 91.1
          - type: hit_rate_at_1000
            value: 98.5
      - task:
          type: STS
        dataset:
          type: mteb/sickr-sts
          name: MTEB SICK-R
          config: default
          split: test
          revision: 20a6d6f312dd54037fe07a32d58e5e168867909d
        metrics:
          - type: pearson
            value: 86.1633
          - type: spearman
            value: 81.7585
          - type: cosine_pearson
            value: 86.1633
          - type: cosine_spearman
            value: 81.7585
          - type: manhattan_pearson
            value: 83.1762
          - type: manhattan_spearman
            value: 81.396
          - type: euclidean_pearson
            value: 83.2109
          - type: euclidean_spearman
            value: 81.4284
      - task:
          type: Retrieval
        dataset:
          type: mteb/SIQA
          name: MTEB SIQA
          config: default
          split: test
          revision: 59898732ccfb72afca329ce110d6799676ef0f84
        metrics:
          - type: ndcg_at_1
            value: 1.279
          - type: ndcg_at_3
            value: 1.846
          - type: ndcg_at_5
            value: 2.053
          - type: ndcg_at_10
            value: 2.364
          - type: ndcg_at_20
            value: 2.762
          - type: ndcg_at_100
            value: 3.473
          - type: ndcg_at_1000
            value: 6.47
          - type: map_at_1
            value: 1.279
          - type: map_at_3
            value: 1.706
          - type: map_at_5
            value: 1.818
          - type: map_at_10
            value: 1.945
          - type: map_at_20
            value: 2.052
          - type: map_at_100
            value: 2.14
          - type: map_at_1000
            value: 2.217
          - type: recall_at_1
            value: 1.279
          - type: recall_at_3
            value: 2.252
          - type: recall_at_5
            value: 2.764
          - type: recall_at_10
            value: 3.736
          - type: recall_at_20
            value: 5.322
          - type: recall_at_100
            value: 9.314
          - type: recall_at_1000
            value: 35.056
          - type: accuracy
            value: 1.279
          - type: precision_at_1
            value: 1.279
          - type: precision_at_3
            value: 0.751
          - type: precision_at_5
            value: 0.553
          - type: precision_at_10
            value: 0.374
          - type: precision_at_20
            value: 0.266
          - type: precision_at_100
            value: 0.093
          - type: precision_at_1000
            value: 0.035
          - type: mrr_at_1
            value: 1.2794
          - type: mrr_at_3
            value: 1.7059
          - type: mrr_at_5
            value: 1.8185
          - type: mrr_at_10
            value: 1.9446
          - type: mrr_at_20
            value: 2.0521
          - type: mrr_at_100
            value: 2.1402
          - type: mrr_at_1000
            value: 2.2168
          - type: hit_rate_at_1
            value: 1.279
          - type: hit_rate_at_3
            value: 2.252
          - type: hit_rate_at_5
            value: 2.764
          - type: hit_rate_at_10
            value: 3.736
          - type: hit_rate_at_20
            value: 5.322
          - type: hit_rate_at_100
            value: 9.314
          - type: hit_rate_at_1000
            value: 35.056
      - task:
          type: STS
        dataset:
          type: mteb/sts12-sts
          name: MTEB STS12
          config: default
          split: test
          revision: a0d554a64d88156834ff5ae9920b964011b16384
        metrics:
          - type: pearson
            value: 82.0189
          - type: spearman
            value: 74.7194
          - type: cosine_pearson
            value: 82.0189
          - type: cosine_spearman
            value: 74.7192
          - type: manhattan_pearson
            value: 78.5505
          - type: manhattan_spearman
            value: 75.0502
          - type: euclidean_pearson
            value: 78.5956
          - type: euclidean_spearman
            value: 75.0884
      - task:
          type: STS
        dataset:
          type: mteb/sts13-sts
          name: MTEB STS13
          config: default
          split: test
          revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca
        metrics:
          - type: pearson
            value: 86.6221
          - type: spearman
            value: 86.4882
          - type: cosine_pearson
            value: 86.6221
          - type: cosine_spearman
            value: 86.4882
          - type: manhattan_pearson
            value: 86.3955
          - type: manhattan_spearman
            value: 87.6413
          - type: euclidean_pearson
            value: 86.5054
          - type: euclidean_spearman
            value: 87.784
      - task:
          type: STS
        dataset:
          type: mteb/sts14-sts
          name: MTEB STS14
          config: default
          split: test
          revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375
        metrics:
          - type: pearson
            value: 84.025
          - type: spearman
            value: 81.0291
          - type: cosine_pearson
            value: 84.025
          - type: cosine_spearman
            value: 81.0291
          - type: manhattan_pearson
            value: 82.0683
          - type: manhattan_spearman
            value: 81.0983
          - type: euclidean_pearson
            value: 82.1368
          - type: euclidean_spearman
            value: 81.1477
      - task:
          type: STS
        dataset:
          type: mteb/sts15-sts
          name: MTEB STS15
          config: default
          split: test
          revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3
        metrics:
          - type: pearson
            value: 88.4824
          - type: spearman
            value: 88.8167
          - type: cosine_pearson
            value: 88.4824
          - type: cosine_spearman
            value: 88.8167
          - type: manhattan_pearson
            value: 87.3373
          - type: manhattan_spearman
            value: 88.1086
          - type: euclidean_pearson
            value: 87.4178
          - type: euclidean_spearman
            value: 88.1941
      - task:
          type: STS
        dataset:
          type: mteb/sts17-crosslingual-sts
          name: MTEB STS17 (en-en)
          config: en-en
          split: test
          revision: faeb762787bd10488a50c8b5be4a3b82e411949c
        metrics:
          - type: pearson
            value: 90.595
          - type: spearman
            value: 90.9423
          - type: cosine_pearson
            value: 90.595
          - type: cosine_spearman
            value: 90.9423
          - type: manhattan_pearson
            value: 89.7504
          - type: manhattan_spearman
            value: 89.806
          - type: euclidean_pearson
            value: 89.8143
          - type: euclidean_spearman
            value: 89.8452
      - task:
          type: STS
        dataset:
          type: mteb/sts22-crosslingual-sts
          name: MTEB STS22.v2 (en)
          config: en
          split: test
          revision: d31f33a128469b20e357535c39b82fb3c3f6f2bd
        metrics:
          - type: pearson
            value: 67.7048
          - type: spearman
            value: 67.4456
          - type: cosine_pearson
            value: 67.7048
          - type: cosine_spearman
            value: 67.4456
          - type: manhattan_pearson
            value: 63.7724
          - type: manhattan_spearman
            value: 63.8649
          - type: euclidean_pearson
            value: 63.5939
          - type: euclidean_spearman
            value: 63.8456
      - task:
          type: STS
        dataset:
          type: mteb/stsbenchmark-sts
          name: MTEB STSBenchmark
          config: default
          split: test
          revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831
        metrics:
          - type: pearson
            value: 86.9609
          - type: spearman
            value: 86.7611
          - type: cosine_pearson
            value: 86.9609
          - type: cosine_spearman
            value: 86.7611
          - type: manhattan_pearson
            value: 86.0919
          - type: manhattan_spearman
            value: 86.6102
          - type: euclidean_pearson
            value: 86.0912
          - type: euclidean_spearman
            value: 86.5851
      - task:
          type: Retrieval
        dataset:
          type: mteb/SpartQA
          name: MTEB SpartQA
          config: default
          split: test
          revision: 1c858df377e57725a014a1b7321ebd79d62016b6
        metrics:
          - type: ndcg_at_1
            value: 1.085
          - type: ndcg_at_3
            value: 2.172
          - type: ndcg_at_5
            value: 3.616
          - type: ndcg_at_10
            value: 7.048
          - type: ndcg_at_20
            value: 9.56
          - type: ndcg_at_100
            value: 11.078
          - type: ndcg_at_1000
            value: 15.343
          - type: map_at_1
            value: 0.788
          - type: map_at_3
            value: 1.671
          - type: map_at_5
            value: 2.413
          - type: map_at_10
            value: 3.748
          - type: map_at_20
            value: 4.435
          - type: map_at_100
            value: 4.646
          - type: map_at_1000
            value: 4.73
          - type: recall_at_1
            value: 0.788
          - type: recall_at_3
            value: 2.931
          - type: recall_at_5
            value: 6.177
          - type: recall_at_10
            value: 15.925
          - type: recall_at_20
            value: 24.847
          - type: recall_at_100
            value: 31.961
          - type: recall_at_1000
            value: 64.385
          - type: accuracy
            value: 0.788
          - type: precision_at_1
            value: 1.085
          - type: precision_at_3
            value: 1.447
          - type: precision_at_5
            value: 1.825
          - type: precision_at_10
            value: 2.39
          - type: precision_at_20
            value: 1.874
          - type: precision_at_100
            value: 0.493
          - type: precision_at_1000
            value: 0.11
          - type: mrr_at_1
            value: 1.0851
          - type: mrr_at_3
            value: 2.3882
          - type: mrr_at_5
            value: 3.376
          - type: mrr_at_10
            value: 5.0346
          - type: mrr_at_20
            value: 5.8236
          - type: mrr_at_100
            value: 6.083
          - type: mrr_at_1000
            value: 6.175
          - type: hit_rate_at_1
            value: 1.085
          - type: hit_rate_at_3
            value: 4.257
          - type: hit_rate_at_5
            value: 8.737
          - type: hit_rate_at_10
            value: 21.592
          - type: hit_rate_at_20
            value: 32.693
          - type: hit_rate_at_100
            value: 42.71
          - type: hit_rate_at_1000
            value: 80.968
      - task:
          type: PairClassification
        dataset:
          type: mteb/sprintduplicatequestions-pairclassification
          name: MTEB SprintDuplicateQuestions
          config: default
          split: test
          revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46
        metrics:
          - type: similarity_accuracy
            value: 99.7881
          - type: similarity_f1
            value: 89.2354
          - type: similarity_precision
            value: 89.7773
          - type: similarity_recall
            value: 88.7
          - type: similarity_ap
            value: 94.805
          - type: cosine_accuracy
            value: 99.7881
          - type: cosine_f1
            value: 89.2354
          - type: cosine_precision
            value: 89.7773
          - type: cosine_recall
            value: 88.7
          - type: cosine_ap
            value: 94.805
          - type: manhattan_accuracy
            value: 99.7911
          - type: manhattan_f1
            value: 89.4553
          - type: manhattan_precision
            value: 89.4106
          - type: manhattan_recall
            value: 89.5
          - type: manhattan_ap
            value: 94.7666
          - type: euclidean_accuracy
            value: 99.7881
          - type: euclidean_f1
            value: 89.2857
          - type: euclidean_precision
            value: 88.5827
          - type: euclidean_recall
            value: 90
          - type: euclidean_ap
            value: 94.7158
          - type: dot_accuracy
            value: 99.6297
          - type: dot_f1
            value: 80.9665
          - type: dot_precision
            value: 79.8638
          - type: dot_recall
            value: 82.1
          - type: dot_ap
            value: 89.4884
          - type: max_accuracy
            value: 99.7911
          - type: max_f1
            value: 89.4553
          - type: max_precision
            value: 89.7773
          - type: max_recall
            value: 90
          - type: max_ap
            value: 94.805
      - task:
          type: Clustering
        dataset:
          type: mteb/stackexchange-clustering
          name: MTEB StackExchangeClustering.v2
          config: default
          split: test
          revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259
        metrics:
          - type: v_measure
            value: 61.5805
          - type: v_measure_std
            value: 0.4115
          - type: ami
            value: 58.0931
          - type: ami_std
            value: 0.4061
      - task:
          type: Clustering
        dataset:
          type: mteb/stackexchange-clustering-p2p
          name: MTEB StackExchangeClusteringP2P.v2
          config: default
          split: test
          revision: 815ca46b2622cec33ccafc3735d572c266efdb44
        metrics:
          - type: v_measure
            value: 41.6448
          - type: v_measure_std
            value: 0.3637
          - type: ami
            value: 34.7334
          - type: ami_std
            value: 0.391
      - task:
          type: Retrieval
        dataset:
          type: mteb/StackOverflowQA
          name: MTEB StackOverflowQA
          config: default
          split: test
          revision: 7b46f5ebd2365278c1bb025986dda2fee893bbf5
        metrics:
          - type: ndcg_at_1
            value: 88.114
          - type: ndcg_at_3
            value: 91.416
          - type: ndcg_at_5
            value: 91.982
          - type: ndcg_at_10
            value: 92.429
          - type: ndcg_at_20
            value: 92.698
          - type: ndcg_at_100
            value: 92.947
          - type: ndcg_at_1000
            value: 93.091
          - type: map_at_1
            value: 88.114
          - type: map_at_3
            value: 90.63
          - type: map_at_5
            value: 90.949
          - type: map_at_10
            value: 91.138
          - type: map_at_20
            value: 91.214
          - type: map_at_100
            value: 91.248
          - type: map_at_1000
            value: 91.254
          - type: recall_at_1
            value: 88.114
          - type: recall_at_3
            value: 93.681
          - type: recall_at_5
            value: 95.035
          - type: recall_at_10
            value: 96.389
          - type: recall_at_20
            value: 97.442
          - type: recall_at_100
            value: 98.796
          - type: recall_at_1000
            value: 99.9
          - type: accuracy
            value: 88.114
          - type: precision_at_1
            value: 88.114
          - type: precision_at_3
            value: 31.227
          - type: precision_at_5
            value: 19.007
          - type: precision_at_10
            value: 9.639
          - type: precision_at_20
            value: 4.872
          - type: precision_at_100
            value: 0.988
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 88.1143
          - type: mrr_at_3
            value: 90.6302
          - type: mrr_at_5
            value: 90.9487
          - type: mrr_at_10
            value: 91.1384
          - type: mrr_at_20
            value: 91.2139
          - type: mrr_at_100
            value: 91.2479
          - type: mrr_at_1000
            value: 91.2538
          - type: hit_rate_at_1
            value: 88.114
          - type: hit_rate_at_3
            value: 93.681
          - type: hit_rate_at_5
            value: 95.035
          - type: hit_rate_at_10
            value: 96.389
          - type: hit_rate_at_20
            value: 97.442
          - type: hit_rate_at_100
            value: 98.796
          - type: hit_rate_at_1000
            value: 99.9
      - task:
          type: Summarization
        dataset:
          type: mteb/summeval
          name: MTEB SummEvalSummarization.v2
          config: default
          split: test
          revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c
        metrics:
          - type: pearson
            value: 38.4249
          - type: spearman
            value: 34.8894
          - type: cosine_spearman
            value: 34.8894
          - type: cosine_pearson
            value: 38.4249
          - type: dot_pearson
            value: 27.3722
          - type: dot_spearman
            value: 26.4149
      - task:
          type: Retrieval
        dataset:
          type: CoIR-Retrieval/synthetic-text2sql
          name: MTEB SyntheticText2SQL
          config: default
          split: test
          revision: 686b87296c3a0191b5d9415a00526c62db9fce09
        metrics:
          - type: ndcg_at_1
            value: 4.991
          - type: ndcg_at_3
            value: 54.565
          - type: ndcg_at_5
            value: 56.746
          - type: ndcg_at_10
            value: 58.046
          - type: ndcg_at_20
            value: 58.624
          - type: ndcg_at_100
            value: 59.115
          - type: ndcg_at_1000
            value: 59.216
          - type: map_at_1
            value: 4.991
          - type: map_at_3
            value: 43.787
          - type: map_at_5
            value: 45.005
          - type: map_at_10
            value: 45.555
          - type: map_at_20
            value: 45.718
          - type: map_at_100
            value: 45.791
          - type: map_at_1000
            value: 45.795
          - type: recall_at_1
            value: 4.991
          - type: recall_at_3
            value: 85.182
          - type: recall_at_5
            value: 90.446
          - type: recall_at_10
            value: 94.394
          - type: recall_at_20
            value: 96.65
          - type: recall_at_100
            value: 99.214
          - type: recall_at_1000
            value: 99.983
          - type: accuracy
            value: 4.991
          - type: precision_at_1
            value: 4.991
          - type: precision_at_3
            value: 28.394
          - type: precision_at_5
            value: 18.089
          - type: precision_at_10
            value: 9.439
          - type: precision_at_20
            value: 4.833
          - type: precision_at_100
            value: 0.992
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 4.9906
          - type: mrr_at_3
            value: 43.7874
          - type: mrr_at_5
            value: 45.0051
          - type: mrr_at_10
            value: 45.5552
          - type: mrr_at_20
            value: 45.7177
          - type: mrr_at_100
            value: 45.7912
          - type: mrr_at_1000
            value: 45.7954
          - type: hit_rate_at_1
            value: 4.991
          - type: hit_rate_at_3
            value: 85.182
          - type: hit_rate_at_5
            value: 90.446
          - type: hit_rate_at_10
            value: 94.394
          - type: hit_rate_at_20
            value: 96.65
          - type: hit_rate_at_100
            value: 99.214
          - type: hit_rate_at_1000
            value: 99.983
      - task:
          type: Retrieval
        dataset:
          type: mteb/trec-covid
          name: MTEB TRECCOVID
          config: default
          split: test
          revision: bb9466bac8153a0349341eb1b22e06409e78ef4e
        metrics:
          - type: ndcg_at_1
            value: 80
          - type: ndcg_at_3
            value: 84.961
          - type: ndcg_at_5
            value: 82.751
          - type: ndcg_at_10
            value: 80.823
          - type: ndcg_at_20
            value: 77.563
          - type: ndcg_at_100
            value: 65.588
          - type: ndcg_at_1000
            value: 60.628
          - type: map_at_1
            value: 0.231
          - type: map_at_3
            value: 0.697
          - type: map_at_5
            value: 1.105
          - type: map_at_10
            value: 2.063
          - type: map_at_20
            value: 3.768
          - type: map_at_100
            value: 13.49
          - type: map_at_1000
            value: 34.595
          - type: recall_at_1
            value: 0.231
          - type: recall_at_3
            value: 0.731
          - type: recall_at_5
            value: 1.178
          - type: recall_at_10
            value: 2.253
          - type: recall_at_20
            value: 4.227
          - type: recall_at_100
            value: 16.751
          - type: recall_at_1000
            value: 57.591
          - type: accuracy
            value: 0.231
          - type: precision_at_1
            value: 86
          - type: precision_at_3
            value: 90.667
          - type: precision_at_5
            value: 88.4
          - type: precision_at_10
            value: 85.6
          - type: precision_at_20
            value: 81.4
          - type: precision_at_100
            value: 67.86
          - type: precision_at_1000
            value: 26.878
          - type: mrr_at_1
            value: 86
          - type: mrr_at_3
            value: 93
          - type: mrr_at_5
            value: 93
          - type: mrr_at_10
            value: 93
          - type: mrr_at_20
            value: 93
          - type: mrr_at_100
            value: 93
          - type: mrr_at_1000
            value: 93
          - type: hit_rate_at_1
            value: 86
          - type: hit_rate_at_3
            value: 100
          - type: hit_rate_at_5
            value: 100
          - type: hit_rate_at_10
            value: 100
          - type: hit_rate_at_20
            value: 100
          - type: hit_rate_at_100
            value: 100
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Retrieval
        dataset:
          type: mteb/TempReasonL1
          name: MTEB TempReasonL1
          config: default
          split: test
          revision: 2065ce0bde70750ed98bb91abd96d0d68ab94324
        metrics:
          - type: ndcg_at_1
            value: 0.025
          - type: ndcg_at_3
            value: 0.455
          - type: ndcg_at_5
            value: 0.757
          - type: ndcg_at_10
            value: 1.405
          - type: ndcg_at_20
            value: 2.354
          - type: ndcg_at_100
            value: 5.877
          - type: ndcg_at_1000
            value: 13.319
          - type: map_at_1
            value: 0.025
          - type: map_at_3
            value: 0.337
          - type: map_at_5
            value: 0.501
          - type: map_at_10
            value: 0.76
          - type: map_at_20
            value: 1.014
          - type: map_at_100
            value: 1.442
          - type: map_at_1000
            value: 1.682
          - type: recall_at_1
            value: 0.025
          - type: recall_at_3
            value: 0.8
          - type: recall_at_5
            value: 1.55
          - type: recall_at_10
            value: 3.6
          - type: recall_at_20
            value: 7.4
          - type: recall_at_100
            value: 27.3
          - type: recall_at_1000
            value: 88.025
          - type: accuracy
            value: 0.025
          - type: precision_at_1
            value: 0.025
          - type: precision_at_3
            value: 0.267
          - type: precision_at_5
            value: 0.31
          - type: precision_at_10
            value: 0.36
          - type: precision_at_20
            value: 0.37
          - type: precision_at_100
            value: 0.273
          - type: precision_at_1000
            value: 0.088
          - type: mrr_at_1
            value: 0.025
          - type: mrr_at_3
            value: 0.3375
          - type: mrr_at_5
            value: 0.5013
          - type: mrr_at_10
            value: 0.7595
          - type: mrr_at_20
            value: 1.0142
          - type: mrr_at_100
            value: 1.4417
          - type: mrr_at_1000
            value: 1.6818
          - type: hit_rate_at_1
            value: 0.025
          - type: hit_rate_at_3
            value: 0.8
          - type: hit_rate_at_5
            value: 1.55
          - type: hit_rate_at_10
            value: 3.6
          - type: hit_rate_at_20
            value: 7.4
          - type: hit_rate_at_100
            value: 27.3
          - type: hit_rate_at_1000
            value: 88.025
      - task:
          type: Retrieval
        dataset:
          type: mteb/TempReasonL2Context
          name: MTEB TempReasonL2Context
          config: default
          split: test
          revision: 4ad8f1a456f63f732a633461ccc0f2c84ebe7ba1
        metrics:
          - type: ndcg_at_1
            value: 10.988
          - type: ndcg_at_3
            value: 19.494
          - type: ndcg_at_5
            value: 22.662
          - type: ndcg_at_10
            value: 26.01
          - type: ndcg_at_20
            value: 28.645
          - type: ndcg_at_100
            value: 32.132
          - type: ndcg_at_1000
            value: 34.467
          - type: map_at_1
            value: 10.988
          - type: map_at_3
            value: 17.334
          - type: map_at_5
            value: 19.092
          - type: map_at_10
            value: 20.487
          - type: map_at_20
            value: 21.209
          - type: map_at_100
            value: 21.679
          - type: map_at_1000
            value: 21.763
          - type: recall_at_1
            value: 10.988
          - type: recall_at_3
            value: 25.774
          - type: recall_at_5
            value: 33.463
          - type: recall_at_10
            value: 43.747
          - type: recall_at_20
            value: 54.178
          - type: recall_at_100
            value: 73.17
          - type: recall_at_1000
            value: 91.829
          - type: accuracy
            value: 10.988
          - type: precision_at_1
            value: 10.988
          - type: precision_at_3
            value: 8.591
          - type: precision_at_5
            value: 6.693
          - type: precision_at_10
            value: 4.375
          - type: precision_at_20
            value: 2.709
          - type: precision_at_100
            value: 0.732
          - type: precision_at_1000
            value: 0.092
          - type: mrr_at_1
            value: 10.9876
          - type: mrr_at_3
            value: 17.3337
          - type: mrr_at_5
            value: 19.0921
          - type: mrr_at_10
            value: 20.4875
          - type: mrr_at_20
            value: 21.2094
          - type: mrr_at_100
            value: 21.6794
          - type: mrr_at_1000
            value: 21.7625
          - type: hit_rate_at_1
            value: 10.988
          - type: hit_rate_at_3
            value: 25.774
          - type: hit_rate_at_5
            value: 33.463
          - type: hit_rate_at_10
            value: 43.747
          - type: hit_rate_at_20
            value: 54.178
          - type: hit_rate_at_100
            value: 73.17
          - type: hit_rate_at_1000
            value: 91.829
      - task:
          type: Retrieval
        dataset:
          type: mteb/TempReasonL2Fact
          name: MTEB TempReasonL2Fact
          config: default
          split: test
          revision: 59b6048796e2a6ae40e843e9fa5675f047ae39b4
        metrics:
          - type: ndcg_at_1
            value: 15.805
          - type: ndcg_at_3
            value: 29.202
          - type: ndcg_at_5
            value: 34.163
          - type: ndcg_at_10
            value: 38.365
          - type: ndcg_at_20
            value: 41.046
          - type: ndcg_at_100
            value: 43.755
          - type: ndcg_at_1000
            value: 44.875
          - type: map_at_1
            value: 15.805
          - type: map_at_3
            value: 25.792
          - type: map_at_5
            value: 28.555
          - type: map_at_10
            value: 30.308
          - type: map_at_20
            value: 31.051
          - type: map_at_100
            value: 31.431
          - type: map_at_1000
            value: 31.471
          - type: recall_at_1
            value: 15.805
          - type: recall_at_3
            value: 39.114
          - type: recall_at_5
            value: 51.121
          - type: recall_at_10
            value: 64.017
          - type: recall_at_20
            value: 74.56
          - type: recall_at_100
            value: 89.087
          - type: recall_at_1000
            value: 98.036
          - type: accuracy
            value: 15.805
          - type: precision_at_1
            value: 15.805
          - type: precision_at_3
            value: 13.038
          - type: precision_at_5
            value: 10.224
          - type: precision_at_10
            value: 6.402
          - type: precision_at_20
            value: 3.728
          - type: precision_at_100
            value: 0.891
          - type: precision_at_1000
            value: 0.098
          - type: mrr_at_1
            value: 15.8051
          - type: mrr_at_3
            value: 25.7921
          - type: mrr_at_5
            value: 28.5548
          - type: mrr_at_10
            value: 30.3079
          - type: mrr_at_20
            value: 31.0514
          - type: mrr_at_100
            value: 31.4312
          - type: mrr_at_1000
            value: 31.4708
          - type: hit_rate_at_1
            value: 15.805
          - type: hit_rate_at_3
            value: 39.114
          - type: hit_rate_at_5
            value: 51.121
          - type: hit_rate_at_10
            value: 64.017
          - type: hit_rate_at_20
            value: 74.56
          - type: hit_rate_at_100
            value: 89.087
          - type: hit_rate_at_1000
            value: 98.036
      - task:
          type: Retrieval
        dataset:
          type: mteb/TempReasonL2Pure
          name: MTEB TempReasonL2Pure
          config: default
          split: test
          revision: c8b90818b291f2421f8ce59645d7362b6043404e
        metrics:
          - type: ndcg_at_1
            value: 0.259
          - type: ndcg_at_3
            value: 0.553
          - type: ndcg_at_5
            value: 0.862
          - type: ndcg_at_10
            value: 1.142
          - type: ndcg_at_20
            value: 1.553
          - type: ndcg_at_100
            value: 3.488
          - type: ndcg_at_1000
            value: 8.024
          - type: map_at_1
            value: 0.259
          - type: map_at_3
            value: 0.476
          - type: map_at_5
            value: 0.645
          - type: map_at_10
            value: 0.759
          - type: map_at_20
            value: 0.874
          - type: map_at_100
            value: 1.101
          - type: map_at_1000
            value: 1.233
          - type: recall_at_1
            value: 0.259
          - type: recall_at_3
            value: 0.778
          - type: recall_at_5
            value: 1.538
          - type: recall_at_10
            value: 2.409
          - type: recall_at_20
            value: 4.021
          - type: recall_at_100
            value: 15.082
          - type: recall_at_1000
            value: 53.067
          - type: accuracy
            value: 0.259
          - type: precision_at_1
            value: 0.259
          - type: precision_at_3
            value: 0.259
          - type: precision_at_5
            value: 0.308
          - type: precision_at_10
            value: 0.241
          - type: precision_at_20
            value: 0.201
          - type: precision_at_100
            value: 0.151
          - type: precision_at_1000
            value: 0.053
          - type: mrr_at_1
            value: 0.2594
          - type: mrr_at_3
            value: 0.4756
          - type: mrr_at_5
            value: 0.6451
          - type: mrr_at_10
            value: 0.7592
          - type: mrr_at_20
            value: 0.874
          - type: mrr_at_100
            value: 1.1009
          - type: mrr_at_1000
            value: 1.2329
          - type: hit_rate_at_1
            value: 0.259
          - type: hit_rate_at_3
            value: 0.778
          - type: hit_rate_at_5
            value: 1.538
          - type: hit_rate_at_10
            value: 2.409
          - type: hit_rate_at_20
            value: 4.021
          - type: hit_rate_at_100
            value: 15.082
          - type: hit_rate_at_1000
            value: 53.067
      - task:
          type: Retrieval
        dataset:
          type: mteb/TempReasonL3Context
          name: MTEB TempReasonL3Context
          config: default
          split: test
          revision: 2c341c955bf2dc4f229c54b5b2cb9c5f1409ffd7
        metrics:
          - type: ndcg_at_1
            value: 7.411
          - type: ndcg_at_3
            value: 15.711
          - type: ndcg_at_5
            value: 18.867
          - type: ndcg_at_10
            value: 21.916
          - type: ndcg_at_20
            value: 24.429
          - type: ndcg_at_100
            value: 28.138
          - type: ndcg_at_1000
            value: 30.745
          - type: map_at_1
            value: 7.411
          - type: map_at_3
            value: 13.579
          - type: map_at_5
            value: 15.331
          - type: map_at_10
            value: 16.596
          - type: map_at_20
            value: 17.291
          - type: map_at_100
            value: 17.792
          - type: map_at_1000
            value: 17.882
          - type: recall_at_1
            value: 7.411
          - type: recall_at_3
            value: 21.916
          - type: recall_at_5
            value: 29.575
          - type: recall_at_10
            value: 38.974
          - type: recall_at_20
            value: 48.87
          - type: recall_at_100
            value: 69.047
          - type: recall_at_1000
            value: 90.081
          - type: accuracy
            value: 7.411
          - type: precision_at_1
            value: 7.411
          - type: precision_at_3
            value: 7.305
          - type: precision_at_5
            value: 5.915
          - type: precision_at_10
            value: 3.897
          - type: precision_at_20
            value: 2.444
          - type: precision_at_100
            value: 0.69
          - type: precision_at_1000
            value: 0.09
          - type: mrr_at_1
            value: 7.4108
          - type: mrr_at_3
            value: 13.5789
          - type: mrr_at_5
            value: 15.331
          - type: mrr_at_10
            value: 16.5958
          - type: mrr_at_20
            value: 17.2911
          - type: mrr_at_100
            value: 17.7916
          - type: mrr_at_1000
            value: 17.8818
          - type: hit_rate_at_1
            value: 7.411
          - type: hit_rate_at_3
            value: 21.916
          - type: hit_rate_at_5
            value: 29.575
          - type: hit_rate_at_10
            value: 38.974
          - type: hit_rate_at_20
            value: 48.87
          - type: hit_rate_at_100
            value: 69.047
          - type: hit_rate_at_1000
            value: 90.081
      - task:
          type: Retrieval
        dataset:
          type: mteb/TempReasonL3Fact
          name: MTEB TempReasonL3Fact
          config: default
          split: test
          revision: ce6502fe436f00f05a73fe24ed0317c57fe4d6b0
        metrics:
          - type: ndcg_at_1
            value: 12.878
          - type: ndcg_at_3
            value: 26.532
          - type: ndcg_at_5
            value: 31.533
          - type: ndcg_at_10
            value: 35.672
          - type: ndcg_at_20
            value: 38.278
          - type: ndcg_at_100
            value: 41.011
          - type: ndcg_at_1000
            value: 42.356
          - type: map_at_1
            value: 12.878
          - type: map_at_3
            value: 23.053
          - type: map_at_5
            value: 25.825
          - type: map_at_10
            value: 27.561
          - type: map_at_20
            value: 28.28
          - type: map_at_100
            value: 28.662
          - type: map_at_1000
            value: 28.709
          - type: recall_at_1
            value: 12.878
          - type: recall_at_3
            value: 36.647
          - type: recall_at_5
            value: 48.803
          - type: recall_at_10
            value: 61.455
          - type: recall_at_20
            value: 71.735
          - type: recall_at_100
            value: 86.421
          - type: recall_at_1000
            value: 97.176
          - type: accuracy
            value: 12.878
          - type: precision_at_1
            value: 12.878
          - type: precision_at_3
            value: 12.216
          - type: precision_at_5
            value: 9.761
          - type: precision_at_10
            value: 6.146
          - type: precision_at_20
            value: 3.587
          - type: precision_at_100
            value: 0.864
          - type: precision_at_1000
            value: 0.097
          - type: mrr_at_1
            value: 12.8784
          - type: mrr_at_3
            value: 23.0532
          - type: mrr_at_5
            value: 25.8254
          - type: mrr_at_10
            value: 27.5609
          - type: mrr_at_20
            value: 28.2799
          - type: mrr_at_100
            value: 28.6617
          - type: mrr_at_1000
            value: 28.709
          - type: hit_rate_at_1
            value: 12.878
          - type: hit_rate_at_3
            value: 36.647
          - type: hit_rate_at_5
            value: 48.803
          - type: hit_rate_at_10
            value: 61.455
          - type: hit_rate_at_20
            value: 71.735
          - type: hit_rate_at_100
            value: 86.421
          - type: hit_rate_at_1000
            value: 97.176
      - task:
          type: Retrieval
        dataset:
          type: mteb/TempReasonL3Pure
          name: MTEB TempReasonL3Pure
          config: default
          split: test
          revision: b2f4c5b7a21b4efd26ffdcea7a2e13a72dae4395
        metrics:
          - type: ndcg_at_1
            value: 0.136
          - type: ndcg_at_3
            value: 3.906
          - type: ndcg_at_5
            value: 5.684
          - type: ndcg_at_10
            value: 7.491
          - type: ndcg_at_20
            value: 8.815
          - type: ndcg_at_100
            value: 11.635
          - type: ndcg_at_1000
            value: 15.893
          - type: map_at_1
            value: 0.136
          - type: map_at_3
            value: 2.956
          - type: map_at_5
            value: 3.938
          - type: map_at_10
            value: 4.68
          - type: map_at_20
            value: 5.043
          - type: map_at_100
            value: 5.4
          - type: map_at_1000
            value: 5.535
          - type: recall_at_1
            value: 0.136
          - type: recall_at_3
            value: 6.665
          - type: recall_at_5
            value: 11.003
          - type: recall_at_10
            value: 16.606
          - type: recall_at_20
            value: 21.848
          - type: recall_at_100
            value: 37.551
          - type: recall_at_1000
            value: 72.436
          - type: accuracy
            value: 0.136
          - type: precision_at_1
            value: 0.136
          - type: precision_at_3
            value: 2.222
          - type: precision_at_5
            value: 2.201
          - type: precision_at_10
            value: 1.661
          - type: precision_at_20
            value: 1.092
          - type: precision_at_100
            value: 0.376
          - type: precision_at_1000
            value: 0.072
          - type: mrr_at_1
            value: 0.1356
          - type: mrr_at_3
            value: 2.956
          - type: mrr_at_5
            value: 3.9377
          - type: mrr_at_10
            value: 4.6803
          - type: mrr_at_20
            value: 5.043
          - type: mrr_at_100
            value: 5.3999
          - type: mrr_at_1000
            value: 5.5354
          - type: hit_rate_at_1
            value: 0.136
          - type: hit_rate_at_3
            value: 6.665
          - type: hit_rate_at_5
            value: 11.003
          - type: hit_rate_at_10
            value: 16.606
          - type: hit_rate_at_20
            value: 21.848
          - type: hit_rate_at_100
            value: 37.551
          - type: hit_rate_at_1000
            value: 72.436
      - task:
          type: Retrieval
        dataset:
          type: mteb/webis-touche2020-v3
          name: MTEB Touche2020Retrieval.v3
          config: default
          split: test
          revision: 431886eaecc48f067a3975b70d0949ea2862463c
        metrics:
          - type: ndcg_at_1
            value: 70.408
          - type: ndcg_at_3
            value: 68.962
          - type: ndcg_at_5
            value: 65.062
          - type: ndcg_at_10
            value: 59.961
          - type: ndcg_at_20
            value: 54.968
          - type: ndcg_at_100
            value: 62.116
          - type: ndcg_at_1000
            value: 73.049
          - type: map_at_1
            value: 2.733
          - type: map_at_3
            value: 7.537
          - type: map_at_5
            value: 11.048
          - type: map_at_10
            value: 18.582
          - type: map_at_20
            value: 27.746
          - type: map_at_100
            value: 39.335
          - type: map_at_1000
            value: 42.661
          - type: recall_at_1
            value: 2.733
          - type: recall_at_3
            value: 8.056
          - type: recall_at_5
            value: 12.089
          - type: recall_at_10
            value: 21.85
          - type: recall_at_20
            value: 35.98
          - type: recall_at_100
            value: 65.002
          - type: recall_at_1000
            value: 92.102
          - type: accuracy
            value: 2.733
          - type: precision_at_1
            value: 77.551
          - type: precision_at_3
            value: 76.19
          - type: precision_at_5
            value: 70.612
          - type: precision_at_10
            value: 63.673
          - type: precision_at_20
            value: 52.755
          - type: precision_at_100
            value: 20.776
          - type: precision_at_1000
            value: 3.2
          - type: mrr_at_1
            value: 77.551
          - type: mrr_at_3
            value: 85.034
          - type: mrr_at_5
            value: 85.5442
          - type: mrr_at_10
            value: 85.8844
          - type: mrr_at_20
            value: 86.0119
          - type: mrr_at_100
            value: 86.0119
          - type: mrr_at_1000
            value: 86.0119
          - type: hit_rate_at_1
            value: 77.551
          - type: hit_rate_at_3
            value: 93.878
          - type: hit_rate_at_5
            value: 95.918
          - type: hit_rate_at_10
            value: 97.959
          - type: hit_rate_at_20
            value: 100
          - type: hit_rate_at_100
            value: 100
          - type: hit_rate_at_1000
            value: 100
      - task:
          type: Classification
        dataset:
          type: mteb/toxic_conversations_50k
          name: MTEB ToxicConversationsClassification
          config: default
          split: test
          revision: edfaf9da55d3dd50d43143d90c1ac476895ae6de
        metrics:
          - type: accuracy
            value: 79.7656
          - type: f1
            value: 62.4086
          - type: f1_weighted
            value: 83.823
          - type: precision
            value: 61.1657
          - type: precision_weighted
            value: 91.4603
          - type: recall
            value: 76.4025
          - type: recall_weighted
            value: 79.7656
          - type: ap
            value: 20.3049
          - type: ap_weighted
            value: 20.3049
      - task:
          type: Classification
        dataset:
          type: mteb/tweet_sentiment_extraction
          name: MTEB TweetSentimentExtractionClassification
          config: default
          split: test
          revision: d604517c81ca91fe16a244d1248fc021f9ecee7a
        metrics:
          - type: accuracy
            value: 61.2564
          - type: f1
            value: 61.4831
          - type: f1_weighted
            value: 60.4063
          - type: precision
            value: 61.2034
          - type: precision_weighted
            value: 61.0357
          - type: recall
            value: 63.1658
          - type: recall_weighted
            value: 61.2564
          - type: ap
            value: .nan
          - type: ap_weighted
            value: .nan
      - task:
          type: Clustering
        dataset:
          type: mteb/twentynewsgroups-clustering
          name: MTEB TwentyNewsgroupsClustering.v2
          config: default
          split: test
          revision: 6125ec4e24fa026cec8a478383ee943acfbd5449
        metrics:
          - type: v_measure
            value: 49.2585
          - type: v_measure_std
            value: 1.3347
          - type: ami
            value: 49.0605
          - type: ami_std
            value: 1.3403
      - task:
          type: PairClassification
        dataset:
          type: mteb/twittersemeval2015-pairclassification
          name: MTEB TwitterSemEval2015
          config: default
          split: test
          revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1
        metrics:
          - type: similarity_accuracy
            value: 85.951
          - type: similarity_f1
            value: 68.5638
          - type: similarity_precision
            value: 63.1357
          - type: similarity_recall
            value: 75.0132
          - type: similarity_ap
            value: 75.0962
          - type: cosine_accuracy
            value: 85.951
          - type: cosine_f1
            value: 68.5638
          - type: cosine_precision
            value: 63.1357
          - type: cosine_recall
            value: 75.0132
          - type: cosine_ap
            value: 75.0961
          - type: manhattan_accuracy
            value: 85.8258
          - type: manhattan_f1
            value: 67.0888
          - type: manhattan_precision
            value: 60.3714
          - type: manhattan_recall
            value: 75.4881
          - type: manhattan_ap
            value: 73.8734
          - type: euclidean_accuracy
            value: 85.8497
          - type: euclidean_f1
            value: 67.416
          - type: euclidean_precision
            value: 60.5978
          - type: euclidean_recall
            value: 75.9631
          - type: euclidean_ap
            value: 74.0609
          - type: dot_accuracy
            value: 84.8304
          - type: dot_f1
            value: 66.055
          - type: dot_precision
            value: 62.3012
          - type: dot_recall
            value: 70.2902
          - type: dot_ap
            value: 70.9049
          - type: max_accuracy
            value: 85.951
          - type: max_f1
            value: 68.5638
          - type: max_precision
            value: 63.1357
          - type: max_recall
            value: 75.9631
          - type: max_ap
            value: 75.0962
      - task:
          type: PairClassification
        dataset:
          type: mteb/twitterurlcorpus-pairclassification
          name: MTEB TwitterURLCorpus
          config: default
          split: test
          revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf
        metrics:
          - type: similarity_accuracy
            value: 89.1586
          - type: similarity_f1
            value: 78.5685
          - type: similarity_precision
            value: 76.535
          - type: similarity_recall
            value: 80.713
          - type: similarity_ap
            value: 86.0935
          - type: cosine_accuracy
            value: 89.1586
          - type: cosine_f1
            value: 78.5685
          - type: cosine_precision
            value: 76.535
          - type: cosine_recall
            value: 80.713
          - type: cosine_ap
            value: 86.0935
          - type: manhattan_accuracy
            value: 88.9296
          - type: manhattan_f1
            value: 78.1758
          - type: manhattan_precision
            value: 74.3454
          - type: manhattan_recall
            value: 82.4222
          - type: manhattan_ap
            value: 85.423
          - type: euclidean_accuracy
            value: 88.9626
          - type: euclidean_f1
            value: 78.3338
          - type: euclidean_precision
            value: 75.327
          - type: euclidean_recall
            value: 81.5907
          - type: euclidean_ap
            value: 85.4693
          - type: dot_accuracy
            value: 88.4542
          - type: dot_f1
            value: 77.3602
          - type: dot_precision
            value: 73.9198
          - type: dot_recall
            value: 81.1364
          - type: dot_ap
            value: 84.7308
          - type: max_accuracy
            value: 89.1586
          - type: max_f1
            value: 78.5685
          - type: max_precision
            value: 76.535
          - type: max_recall
            value: 82.4222
          - type: max_ap
            value: 86.0935
      - task:
          type: Retrieval
        dataset:
          type: mteb/WinoGrande
          name: MTEB WinoGrande
          config: default
          split: test
          revision: 4dec9c5666e9f84702ac614363db6d96a68bc6de
        metrics:
          - type: ndcg_at_1
            value: 34.57
          - type: ndcg_at_3
            value: 53.414
          - type: ndcg_at_5
            value: 56.981
          - type: ndcg_at_10
            value: 59.825
          - type: ndcg_at_20
            value: 61.516
          - type: ndcg_at_100
            value: 62.997
          - type: ndcg_at_1000
            value: 63.283
          - type: map_at_1
            value: 34.57
          - type: map_at_3
            value: 48.987
          - type: map_at_5
            value: 50.98
          - type: map_at_10
            value: 52.159
          - type: map_at_20
            value: 52.632
          - type: map_at_100
            value: 52.86
          - type: map_at_1000
            value: 52.871
          - type: recall_at_1
            value: 34.57
          - type: recall_at_3
            value: 66.14
          - type: recall_at_5
            value: 74.743
          - type: recall_at_10
            value: 83.504
          - type: recall_at_20
            value: 90.134
          - type: recall_at_100
            value: 97.79
          - type: recall_at_1000
            value: 100
          - type: accuracy
            value: 34.57
          - type: precision_at_1
            value: 34.57
          - type: precision_at_3
            value: 22.047
          - type: precision_at_5
            value: 14.949
          - type: precision_at_10
            value: 8.35
          - type: precision_at_20
            value: 4.507
          - type: precision_at_100
            value: 0.978
          - type: precision_at_1000
            value: 0.1
          - type: mrr_at_1
            value: 34.5699
          - type: mrr_at_3
            value: 48.9871
          - type: mrr_at_5
            value: 50.98
          - type: mrr_at_10
            value: 52.1595
          - type: mrr_at_20
            value: 52.6322
          - type: mrr_at_100
            value: 52.8598
          - type: mrr_at_1000
            value: 52.8713
          - type: hit_rate_at_1
            value: 34.57
          - type: hit_rate_at_3
            value: 66.14
          - type: hit_rate_at_5
            value: 74.743
          - type: hit_rate_at_10
            value: 83.504
          - type: hit_rate_at_20
            value: 90.134
          - type: hit_rate_at_100
            value: 97.79
          - type: hit_rate_at_1000
            value: 100

UltraBERT-Embed-150M

UltraBERT-Embed-150M is an English text embedding model built on UltraBERT-150M. It maps text to a 768-dimensional vector, takes inputs up to 32K tokens, follows task instructions on the query side, and embeds code as well as text. On MTEB(eng, v2) it scores 67.24 Mean(Task), the highest of any dense model under 200M parameters, without distillation from a larger embedding model.

Model	Parameters	Hidden	Context	Description
UltraBERT-150M	150M	768	32K	Pre-trained encoder
UltraBERT-380M	380M	1,024	32K	Pre-trained encoder
UltraBERT-Embed-150M (this model)	150M	768	32K	Text embedding model built on UltraBERT-150M

For more information about UltraBERT, please check out our paper.

✨ Highlights

As of October 11, 2026, UltraBERT-Embed-150M is the state of the art among dense embedding models under 200M parameters on both MTEB(eng, v2) and MTEB(Code, v1).

• MTEB(eng, v2): 67.24. Mean over the 41 tasks, ahead of Tarka-Embedding-150M-V1 (66.39) and gte-modernbert-base (66.26). The models closest behind are distilled from larger embedding models; this one is trained with contrastive losses on public pairs only.
• MTEB(Code, v1): 73.30. Mean over the 12 tasks, with CoIR 72.73, from code pairs in both training stages.
• LongEmbed: 79.30. Highest of the models we compared, including 99.0 on passkey retrieval, with documents up to 32K tokens embedded whole rather than truncated.
• RAR-b: 28.75. Highest of the encoders we compared on reasoning-style retrieval.
• Instructions. Queries carry an E5-mistral style instruction (Instruct: {task}\nQuery: ) and documents are embedded raw, so one model serves retrieval, classification, clustering and similarity.

🚀 Usage

Requires sentence-transformers >= 6.0 and transformers >= 5, with trust_remote_code=True. FlashAttention 2 is recommended: with it, Sentence Transformers packs the batch without padding, which matters for long inputs. SDPA also works ("attn_implementation": "sdpa").

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "dmis-lab/UltraBERT-Embed-150M",
    trust_remote_code=True,
    model_kwargs={"attn_implementation": "flash_attention_2", "dtype": "bfloat16"},
)

queries = ["Win big, click now!"]
labels = ["spam", "promotional", "announcement"]

query_embeddings = model.encode(queries, prompt="Instruct: Labeling spam emails\nQuery: ")
label_embeddings = model.encode(labels, prompt="Label: ")
print(model.similarity(query_embeddings, label_embeddings))

prompt_name="query" applies the default retrieval instruction. Documents never take an instruction, except label phrases, which take the prefix Label: . For symmetric tasks (STS, paraphrase, duplicate detection) apply the same instruction to both sides, e.g. Instruct: Retrieve semantically similar text.\nQuery: . For classification and clustering, name the categories, e.g. Instruct: Classify the sentiment expressed in the given movie review text\nQuery: .

📊 Evaluation

MTEB(eng, v2)

Model	Params	Average (41)	Class. (8)	Clust. (8)	PairClass. (3)	Rerank. (2)	Retr. (10)	STS (9)	Summ. (1)
UltraBERT-Embed-150M	151M	67.24	81.74	49.70	85.33	47.50	57.20	83.03	34.89
Tarka-Embedding-150M-V1 (distilled)	156M	66.39	86.28	52.02	81.65	45.66	51.52	81.48	30.83
gte-modernbert-base	149M	66.26	80.70	48.06	85.94	47.61	56.91	81.40	31.81
F2LLM-v2-160M (distilled)	159M	65.93	87.13	52.55	82.08	45.26	49.33	80.76	28.63
gte-base-en-v1.5	137M	65.8	81.0	47.9	85.4	46.6	55.5	81.7	29.0
GIST-Embedding-v0	109M	65.50	78.16	48.50	86.33	47.52	53.59	83.35	32.32
bge-base-en-v1.5	109M	65.14	77.69	47.42	86.56	46.66	54.75	82.12	30.19
stella-base-en-v2	109M	64.90	79.37	47.50	86.51	47.31	51.56	82.73	31.78
nomic-embed-text-v1	137M	63.60	76.77	46.64	85.15	45.71	50.72	81.78	30.40
granite-embedding-english-r2	149M	62.84	70.71	47.20	79.34	49.10	56.43	78.12	29.31
e5-base-v2	109M	62.67	75.48	45.20	85.63	45.13	49.67	80.64	34.26
DenseOn	149M	62.47	72.61	46.11	79.57	47.70	59.09	73.08	28.84
nomic-embed-text-v1.5	137M	62.20	75.71	47.55	85.16	46.01	47.97	78.70	28.56
modernbert-embed-base	149M	62.1	75.1	46.3	82.4	46.3	48.7	79.3	34.1

MTEB(Code, v1) and CoIR

Model	Params	Average (12)	CoIR	Apps	COIR-CSN	CodeEdit	CFB-MT	CFB-ST	CSN-CC	CSN	CT-Contest	CT-DL	CosQA	SO-QA	T2SQL
UltraBERT-Embed-150M	151M	73.30	72.73	68.50	82.23	62.21	84.37	86.40	93.73	90.03	88.08	37.23	36.29	92.43	58.05
gte-modernbert-base	149M	71.66	71.53	57.72	83.10	55.83	86.15	86.00	93.61	88.76	72.35	37.27	43.36	91.14	64.61
NightOwl-CodeEmbedding	151M	70.91	68.68	39.18	84.26	74.81	76.69	85.21	91.80	89.24	75.95	36.06	42.81	86.61	68.27
F2LLM-v2-160M (distilled)	159M	70.38	70.52	58.95	82.18	55.81	90.00	86.22	85.14	83.49	80.22	34.55	36.52	87.36	64.09
Tarka-Embedding-150M-V1 (distilled)	156M	57.71	55.94	38.96	62.39	51.60	49.15	78.34	54.55	81.48	80.74	31.79	36.03	75.97	51.47
granite-embedding-english-r2	149M	57.22	54.65	13.96	64.65	59.35	52.54	77.18	47.67	80.79	77.07	35.03	37.01	91.80	49.55
DenseOn	149M	55.96	52.75	12.81	55.24	62.83	48.97	78.63	55.43	81.11	69.66	31.84	36.07	92.60	46.29
e5-base-v2	109M	53.51	50.39	11.52	62.71	55.89	41.56	74.52	56.87	82.31	62.50	21.87	32.59	87.85	51.88
modernbert-embed-base	149M	53.28	49.51	6.54	71.35	57.24	40.66	70.28	56.84	86.99	56.73	26.88	35.44	76.96	53.42
bge-base-en-v1.5	109M	50.35	46.61	6.45	72.02	53.07	33.65	70.00	50.89	85.11	45.65	23.49	33.71	80.23	49.98
nomic-embed-text-v1.5	137M	47.36	43.62	4.88	64.91	49.50	39.73	64.89	48.67	82.64	45.63	21.93	26.49	71.06	48.05

LongEmbed

Model	Params	Max. Len.	Average (6)	NQA	Needle	Passkey	QMSum	SFD	2WmQA
UltraBERT-Embed-150M	151M	32768	79.30	64.55	66.50	99.00	55.28	97.62	92.86
bekko-embedding-v1-a25m	123M	8192	73.19	48.13	74.75	85.00	47.08	96.92	87.24
BidirLM-270M-Embedding	268M	32768	71.82	54.21	66.75	90.50	47.21	96.19	76.04
granite-embedding-311m-multilingual-r2	312M	8192	71.72	51.11	55.50	95.50	47.25	95.71	85.23
bekko-embedding-v1-a8m	106M	8192	70.60	44.50	76.50	85.00	42.54	96.40	78.68
granite-embedding-english-r2	149M	8192	67.8	49.7	49.8	83.3	42.8	94.2	86.9
DenseOn	149M	8192	66.50	49.95	60.25	64.50	44.05	94.83	85.45
F2LLM-v2-330M (distilled)	334M	8192	66.26	48.16	52.25	76.25	40.15	96.56	84.18
jina-embeddings-v5-text-nano (distilled)	212M	8192	63.65	52.17	59.75	81.50	31.83	81.80	74.87
gte-multilingual-base	305M	8192	62.12	52.36	42.25	55.50	43.03	95.50	84.08
gte-base-en-v1.5	137M	8192	59.4	47.3	31.8	58.0	41.7	92.1	85.6
F2LLM-v2-160M (distilled)	159M	8192	57.64	29.18	54.75	69.00	32.60	91.06	69.22
nomic-embed-text-v1	137M	8192	57.5	41.2	39.5	60.7	36.7	93.0	73.8
gte-modernbert-base	149M	8192	57.0	30.2	24.5	66.8	40.7	93.1	87.0
modernbert-embed-base	149M	8192	56.3	41.1	35.8	62.5	36.7	94.5	67.5

RAR-b

Model	Params	Average (17)	ARC-C	αNLI	HellaSwag	WinoGrande	PIQA	SIQA	Quail	SpartQA	TR	Code	Math
UltraBERT-Embed-150M	151M	28.75	16.15	36.09	33.91	59.82	35.05	2.36	7.97	7.05	18.86	72.81	85.52
granite-embedding-311m-multilingual-r2	312M	27.95	9.89	30.55	26.36	66.61	27.45	2.07	5.07	64.63	15.50	58.35	75.65
e5-large-v2	335M	27.33	11.29	15.16	27.86	51.52	29.55	4.07	5.28	10.37	25.04	58.67	75.51
KaLM-embedding-multilingual-mini-v1	494M	25.79	10.36	26.73	29.37	29.91	30.57	2.93	5.34	10.57	19.72	70.64	83.96
DenseOn	149M	24.12	13.81	34.33	32.24	25.64	34.26	1.72	5.69	9.44	14.95	65.48	82.78
e5-base-v2	109M	23.83	10.01	21.73	25.48	46.99	27.71	4.37	4.94	7.84	19.12	54.47	67.76
gte-modernbert-base	149M	23.54	13.28	28.64	29.45	53.00	30.52	2.56	5.76	7.82	12.25	54.00	89.41
arctic-embed-m-v2.0	305M	23.33	10.78	30.42	28.25	66.33	30.83	1.67	5.12	3.75	12.77	56.09	73.91
F2LLM-v2-330M (distilled)	334M	23.28	13.38	30.23	26.46	34.90	28.48	2.96	5.93	13.13	12.43	61.12	92.12
Tarka-Embedding-150M-V1 (distilled)	156M	21.76	10.96	35.03	25.96	58.86	27.94	2.01	4.98	6.14	11.91	56.28	58.30
granite-embedding-english-r2	149M	20.65	10.75	31.71	28.01	35.12	29.02	1.69	4.99	2.82	9.64	52.92	86.59
F2LLM-v2-160M (distilled)	159M	19.62	9.47	27.76	22.25	43.16	23.75	1.46	4.86	14.48	7.45	49.72	84.50
mxbai-embed-large-v1	335M	19.50	10.53	23.32	28.99	47.99	28.45	2.33	3.39	1.51	12.24	43.79	55.57
modernbert-embed-base	149M	16.04	10.38	30.08	28.55	22.82	28.93	1.19	3.41	6.66	5.82	41.31	58.58
nomic-embed-text-v1.5	137M	15.39	10.24	29.72	24.67	46.79	24.71	2.67	4.22	6.30	6.88	29.09	35.14
bge-base-en-v1.5	109M	14.59	9.66	10.99	26.64	13.76	25.70	0.94	1.42	3.37	8.90	46.47	46.86

🏛️ Model Description

Base model	UltraBERT-150M
Parameters	151M
Embedding dimension	768
Pooling	CLS token
Similarity	cosine
Max sequence length	32,768 tokens
Instruction format	Instruct: {task}\nQuery: on queries
Language	English (text and code)

⚙️ Training

UltraBERT-Embed-150M is trained from UltraBERT-150M in two contrastive stages with Sentence Transformers. The first stage trains on about 500M weakly supervised text and code pairs with in-batch negatives only. The second stage fine-tunes on about 5.7M examples with mined hard negatives, with a task instruction on the query side and inputs up to 32K tokens so that long documents are seen whole. The released weights are the average of several checkpoints from the second stage.

Training data comes from public sources only, without distillation from a larger embedding model:

• Web question-answer and title-body pairs (PAQ, GooAQ, Yahoo Answers, Amazon QA, Reddit, StackOverflow, news, Wikipedia, S2ORC)
• Code pairs (CornStack, OCGI, xCodeEval)
• Retrieval datasets with hard negatives (MS MARCO, NQ, TriviaQA, HotpotQA, FEVER, SQuAD 2.0, MIRACL, MLDR, DBpedia, BioASQ, PubMedQA, MedQuAD)
• Sentence similarity (NLI, QQP, PAWS-X, synthetic STS)
• Classification and clustering (synthetic classification data from the Nemotron embedding dataset, Amazon polarity, toxicity datasets)

Citation

@misc{park2026ultrabert,
      title={UltraBERT: Beyond the Limit of BERT Pre-Training}, 
      author={Jungwoo Park and Taewhoo Lee and Jaewoo Kang},
      year={2026},
      eprint={2610.15028},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2610.15028}, 
}