Add new SentenceTransformer model

042a190 verified 25 days ago

48.2 kB

tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - dense
  - generated_from_trainer
  - dataset_size:8089600
  - loss:MultipleNegativesRankingLoss
base_model: sentence-transformers/embeddinggemma-300m-medical
widget:
  - source_sentence: anaemia
    sentences:
      - anaemic nos
      - scurvy
      - foreign body
  - source_sentence: paludism
    sentences:
      - congestive fever
      - ocular myiasis
      - gnathophyma
  - source_sentence: amniocele
    sentences:
      - inefction
      - omphalocele
      - chylous effusion
  - source_sentence: polyploid
    sentences:
      - polyploidy
      - monosomy 19p13.12
      - otorrhagia
  - source_sentence: polysplenia
    sentences:
      - cut cysts
      - splenunculi
      - spleen agenesis
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
  - cosine_accuracy@1
  - cosine_accuracy@5
  - cosine_accuracy@10
  - cosine_precision@1
  - cosine_precision@3
  - cosine_precision@5
  - cosine_precision@10
  - cosine_recall@1
  - cosine_recall@3
  - cosine_recall@5
  - cosine_recall@10
  - cosine_ndcg@10
  - cosine_mrr@10
  - cosine_map@10
model-index:
  - name: >-
      SentenceTransformer based on
      sentence-transformers/embeddinggemma-300m-medical
    results:
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: icd11 val
          type: icd11-val
        metrics:
          - type: cosine_accuracy@1
            value: 0.7742690058479532
            name: Cosine Accuracy@1
          - type: cosine_accuracy@5
            value: 0.8432748538011696
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.8643274853801169
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.7742690058479532
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.6912280701754386
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.5929824561403508
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.43543859649122807
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.16470727696455337
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.4096813550232527
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.5202410469498723
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.6459809458753583
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.7291630675233993
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.8052028218694884
            name: Cosine Mrr@10
          - type: cosine_map@10
            value: 0.6770894371068766
            name: Cosine Map@10

SentenceTransformer based on sentence-transformers/embeddinggemma-300m-medical

This is a sentence-transformers model finetuned from sentence-transformers/embeddinggemma-300m-medical. It maps inputs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: sentence-transformers/embeddinggemma-300m-medical
Maximum Sequence Length: 30 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity
Supported Modality: Text

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'Gemma3TextModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
  (2): Dense({'in_features': 768, 'out_features': 3072, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
  (3): Dense({'in_features': 3072, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
  (4): Normalize({'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("seanfarrell/icd11_embed_model_v2_large")
# Run inference
queries = [
    'polysplenia',
]
documents = [
    'splenunculi',
    'spleen agenesis',
    'cut cysts',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.5815,  0.3983, -0.0705]])

Evaluation

Metrics

Information Retrieval

Dataset: icd11-val
Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.7743
cosine_accuracy@5	0.8433
cosine_accuracy@10	0.8643
cosine_precision@1	0.7743
cosine_precision@3	0.6912
cosine_precision@5	0.593
cosine_precision@10	0.4354
cosine_recall@1	0.1647
cosine_recall@3	0.4097
cosine_recall@5	0.5202
cosine_recall@10	0.646
cosine_ndcg@10	0.7292
cosine_mrr@10	0.8052
cosine_map@10	0.6771

Training Details

Training Dataset

Unnamed Dataset

Size: 8,089,600 training samples
Columns: anchor, positive, and negative

Approximate statistics based on the first 100 samples:

	anchor	positive	negative
type	string	string	string
modality	text	text	text
details	min: 4 tokens mean: 8.74 tokens max: 24 tokens	min: 4 tokens mean: 8.48 tokens max: 23 tokens	min: 4 tokens mean: 9.3 tokens max: 26 tokens

Samples:

anchor	positive	negative
`chronic rheumatic heart disease, unspecified`	`rheumatic pancarditis`	`de morgan's spots`
`chronic rheumatic left ventricular failure`	`inactive or quiescent rheumatic fever with congestive heart failure`	`conjunctival xerosis with bitot's spots`
`inactive or quiescent rheumatic fever with endocarditis`	`inactive rheumatic valvar endocarditis`	`mature b-cell neoplasm with leukaemic behaviour, unspecified`

Loss: MultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim",
    "gather_across_devices": false,
    "directions": [
        "query_to_doc"
    ],
    "partition_mode": "joint",
    "hardness_mode": null,
    "hardness_strength": 0.0
}

Training Hyperparameters

Non-Default Hyperparameters

per_device_train_batch_size: 128
per_device_eval_batch_size: 256
learning_rate: 2e-05
weight_decay: 0.01
num_train_epochs: 1
lr_scheduler_type: cosine
warmup_steps: 6320
bf16: True
dataloader_drop_last: True
load_best_model_at_end: True
eval_on_start: True

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
prediction_loss_only: True
per_device_train_batch_size: 128
per_device_eval_batch_size: 256
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 2e-05
weight_decay: 0.01
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 1
max_steps: -1
lr_scheduler_type: cosine
lr_scheduler_kwargs: {}
warmup_ratio: 0.0
warmup_steps: 6320
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
bf16: True
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: True
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: True
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch_fused
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
project: huggingface
trackio_space_id: trackio
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
hub_revision: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
include_tokens_per_second: False
include_num_input_tokens_seen: no
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: True
use_liger_kernel: False
liger_kernel_config: None
eval_use_gather_object: False
average_tokens_across_devices: True
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: proportional
router_mapping: {}
learning_rate_mapping: {}

Training Logs

Click to expand

Epoch	Step	Training Loss	icd11-val_cosine_ndcg@10
0	0	-	0.5062
0.0008	50	1.6688	-
0.0016	100	1.6505	-
0.0024	150	1.5416	-
0.0032	200	1.4195	-
0.0040	250	1.3759	-
0.0047	300	1.2595	-
0.0055	350	1.2113	-
0.0063	400	1.1349	-
0.0071	450	1.1253	-
0.0079	500	1.0463	0.5645
0.0087	550	0.9832	-
0.0095	600	0.93	-
0.0103	650	0.8592	-
0.0111	700	0.8343	-
0.0119	750	0.8298	-
0.0127	800	0.7648	-
0.0134	850	0.724	-
0.0142	900	0.7301	-
0.0150	950	0.6552	-
0.0158	1000	0.6649	0.5913
0.0166	1050	0.6055	-
0.0174	1100	0.6287	-
0.0182	1150	0.5632	-
0.0190	1200	0.5289	-
0.0198	1250	0.5299	-
0.0206	1300	0.5321	-
0.0214	1350	0.5087	-
0.0222	1400	0.4714	-
0.0229	1450	0.4598	-
0.0237	1500	0.4614	0.6138
0.0245	1550	0.45	-
0.0253	1600	0.4241	-
0.0261	1650	0.4047	-
0.0269	1700	0.4162	-
0.0277	1750	0.3737	-
0.0285	1800	0.4114	-
0.0293	1850	0.376	-
0.0301	1900	0.3791	-
0.0309	1950	0.3791	-
0.0316	2000	0.3643	0.6310
0.0324	2050	0.3543	-
0.0332	2100	0.3024	-
0.0340	2150	0.3294	-
0.0348	2200	0.3017	-
0.0356	2250	0.3334	-
0.0364	2300	0.3158	-
0.0372	2350	0.3252	-
0.0380	2400	0.2956	-
0.0388	2450	0.2773	-
0.0396	2500	0.2994	0.6399
0.0403	2550	0.2725	-
0.0411	2600	0.2531	-
0.0419	2650	0.2924	-
0.0427	2700	0.286	-
0.0435	2750	0.2653	-
0.0443	2800	0.2588	-
0.0451	2850	0.2428	-
0.0459	2900	0.243	-
0.0467	2950	0.2426	-
0.0475	3000	0.2524	0.6577
0.0483	3050	0.2447	-
0.0491	3100	0.2474	-
0.0498	3150	0.2565	-
0.0506	3200	0.2397	-
0.0514	3250	0.2345	-
0.0522	3300	0.2229	-
0.0530	3350	0.2418	-
0.0538	3400	0.2271	-
0.0546	3450	0.2189	-
0.0554	3500	0.2163	0.6591
0.0562	3550	0.2052	-
0.0570	3600	0.217	-
0.0578	3650	0.2038	-
0.0585	3700	0.2268	-
0.0593	3750	0.2035	-
0.0601	3800	0.2077	-
0.0609	3850	0.2055	-
0.0617	3900	0.2207	-
0.0625	3950	0.2004	-
0.0633	4000	0.2053	0.6627
0.0641	4050	0.2002	-
0.0649	4100	0.1938	-
0.0657	4150	0.2213	-
0.0665	4200	0.2146	-
0.0672	4250	0.1949	-
0.0680	4300	0.1944	-
0.0688	4350	0.1777	-
0.0696	4400	0.196	-
0.0704	4450	0.1927	-
0.0712	4500	0.1805	0.6745
0.0720	4550	0.1934	-
0.0728	4600	0.1892	-
0.0736	4650	0.2022	-
0.0744	4700	0.1892	-
0.0752	4750	0.1812	-
0.0759	4800	0.1878	-
0.0767	4850	0.1815	-
0.0775	4900	0.1932	-
0.0783	4950	0.1901	-
0.0791	5000	0.1753	0.6772
0.0799	5050	0.186	-
0.0807	5100	0.1768	-
0.0815	5150	0.1588	-
0.0823	5200	0.1788	-
0.0831	5250	0.1853	-
0.0839	5300	0.1796	-
0.0847	5350	0.1703	-
0.0854	5400	0.1905	-
0.0862	5450	0.1818	-
0.0870	5500	0.188	0.6699
0.0878	5550	0.1665	-
0.0886	5600	0.1639	-
0.0894	5650	0.1664	-
0.0902	5700	0.1757	-
0.0910	5750	0.1688	-
0.0918	5800	0.1487	-
0.0926	5850	0.1717	-
0.0934	5900	0.167	-
0.0941	5950	0.1595	-
0.0949	6000	0.1696	0.6811
0.0957	6050	0.1676	-
0.0965	6100	0.1746	-
0.0973	6150	0.1483	-
0.0981	6200	0.1636	-
0.0989	6250	0.1517	-
0.0997	6300	0.165	-
0.1005	6350	0.1634	-
0.1013	6400	0.1408	-
0.1021	6450	0.1623	-
0.1028	6500	0.1656	0.6789
0.1036	6550	0.1609	-
0.1044	6600	0.1612	-
0.1052	6650	0.16	-
0.1060	6700	0.1657	-
0.1068	6750	0.1427	-
0.1076	6800	0.1355	-
0.1084	6850	0.1401	-
0.1092	6900	0.1409	-
0.1100	6950	0.1403	-
0.1108	7000	0.138	0.6926
0.1116	7050	0.1369	-
0.1123	7100	0.1295	-
0.1131	7150	0.1407	-
0.1139	7200	0.1388	-
0.1147	7250	0.1377	-
0.1155	7300	0.1324	-
0.1163	7350	0.128	-
0.1171	7400	0.1256	-
0.1179	7450	0.126	-
0.1187	7500	0.1284	0.6911
0.1195	7550	0.1246	-
0.1203	7600	0.1119	-
0.1210	7650	0.1082	-
0.1218	7700	0.1305	-
0.1226	7750	0.1033	-
0.1234	7800	0.1125	-
0.1242	7850	0.1256	-
0.125	7900	0.1403	-
0.1258	7950	0.1334	-
0.1266	8000	0.1173	0.6939
0.1274	8050	0.106	-
0.1282	8100	0.1127	-
0.1290	8150	0.1185	-
0.1297	8200	0.1045	-
0.1305	8250	0.0978	-
0.1313	8300	0.1074	-
0.1321	8350	0.1114	-
0.1329	8400	0.1127	-
0.1337	8450	0.1035	-
0.1345	8500	0.0984	0.7023
0.1353	8550	0.1065	-
0.1361	8600	0.1009	-
0.1369	8650	0.0837	-
0.1377	8700	0.0932	-
0.1384	8750	0.1096	-
0.1392	8800	0.1065	-
0.1400	8850	0.1029	-
0.1408	8900	0.1123	-
0.1416	8950	0.1043	-
0.1424	9000	0.0947	0.6998
0.1432	9050	0.0981	-
0.1440	9100	0.0917	-
0.1448	9150	0.0962	-
0.1456	9200	0.0902	-
0.1464	9250	0.1001	-
0.1472	9300	0.0852	-
0.1479	9350	0.0724	-
0.1487	9400	0.0892	-
0.1495	9450	0.0774	-
0.1503	9500	0.0819	0.7025
0.1511	9550	0.0831	-
0.1519	9600	0.0807	-
0.1527	9650	0.0833	-
0.1535	9700	0.0823	-
0.1543	9750	0.0754	-
0.1551	9800	0.0759	-
0.1559	9850	0.0856	-
0.1566	9900	0.0787	-
0.1574	9950	0.0863	-
0.1582	10000	0.0821	0.6993
0.1590	10050	0.0761	-
0.1598	10100	0.076	-
0.1606	10150	0.0742	-
0.1614	10200	0.0684	-
0.1622	10250	0.0754	-
0.1630	10300	0.0729	-
0.1638	10350	0.0852	-
0.1646	10400	0.0778	-
0.1653	10450	0.078	-
0.1661	10500	0.0652	0.6972
0.1669	10550	0.0766	-
0.1677	10600	0.0805	-
0.1685	10650	0.0736	-
0.1693	10700	0.0702	-
0.1701	10750	0.0789	-
0.1709	10800	0.0683	-
0.1717	10850	0.0788	-
0.1725	10900	0.0592	-
0.1733	10950	0.0716	-
0.1741	11000	0.0647	0.7031
0.1748	11050	0.0624	-
0.1756	11100	0.0629	-
0.1764	11150	0.0653	-
0.1772	11200	0.0699	-
0.1780	11250	0.0661	-
0.1788	11300	0.0534	-
0.1796	11350	0.0559	-
0.1804	11400	0.0656	-
0.1812	11450	0.0689	-
0.1820	11500	0.0642	0.7067
0.1828	11550	0.0586	-
0.1835	11600	0.0616	-
0.1843	11650	0.0582	-
0.1851	11700	0.0668	-
0.1859	11750	0.0577	-
0.1867	11800	0.0614	-
0.1875	11850	0.0598	-
0.1883	11900	0.0488	-
0.1891	11950	0.0721	-
0.1899	12000	0.0623	0.7102
0.1907	12050	0.0542	-
0.1915	12100	0.0648	-
0.1922	12150	0.0589	-
0.1930	12200	0.0538	-
0.1938	12250	0.0511	-
0.1946	12300	0.0624	-
0.1954	12350	0.0586	-
0.1962	12400	0.052	-
0.1970	12450	0.0629	-
0.1978	12500	0.0557	0.7136
0.1986	12550	0.052	-
0.1994	12600	0.0597	-
0.2002	12650	0.0524	-
0.2009	12700	0.0465	-
0.2017	12750	0.0529	-
0.2025	12800	0.0591	-
0.2033	12850	0.0527	-
0.2041	12900	0.0553	-
0.2049	12950	0.0518	-
0.2057	13000	0.0561	0.7007
0.2065	13050	0.0559	-
0.2073	13100	0.0569	-
0.2081	13150	0.0522	-
0.2089	13200	0.0586	-
0.2097	13250	0.0482	-
0.2104	13300	0.0496	-
0.2112	13350	0.0512	-
0.2120	13400	0.0468	-
0.2128	13450	0.0448	-
0.2136	13500	0.0566	0.7088
0.2144	13550	0.0541	-
0.2152	13600	0.048	-
0.2160	13650	0.0455	-
0.2168	13700	0.0484	-
0.2176	13750	0.0512	-
0.2184	13800	0.0414	-
0.2191	13850	0.0467	-
0.2199	13900	0.0413	-
0.2207	13950	0.0483	-
0.2215	14000	0.0531	0.7072
0.2223	14050	0.0497	-
0.2231	14100	0.0507	-
0.2239	14150	0.0417	-
0.2247	14200	0.0479	-
0.2255	14250	0.0515	-
0.2263	14300	0.0447	-
0.2271	14350	0.0443	-
0.2278	14400	0.0416	-
0.2286	14450	0.0471	-
0.2294	14500	0.039	0.7108
0.2302	14550	0.0535	-
0.2310	14600	0.0404	-
0.2318	14650	0.037	-
0.2326	14700	0.0379	-
0.2334	14750	0.048	-
0.2342	14800	0.0417	-
0.2350	14850	0.0405	-
0.2358	14900	0.0408	-
0.2366	14950	0.0388	-
0.2373	15000	0.0376	0.7160
0.2381	15050	0.0443	-
0.2389	15100	0.0436	-
0.2397	15150	0.0374	-
0.2405	15200	0.0378	-
0.2413	15250	0.0395	-
0.2421	15300	0.0437	-
0.2429	15350	0.0406	-
0.2437	15400	0.0396	-
0.2445	15450	0.042	-
0.2453	15500	0.0395	0.7122
0.2460	15550	0.0371	-
0.2468	15600	0.037	-
0.2476	15650	0.0458	-
0.2484	15700	0.0377	-
0.2492	15750	0.0367	-
0.25	15800	0.0459	-
0.2508	15850	0.0354	-
0.2516	15900	0.0338	-
0.2524	15950	0.0363	-
0.2532	16000	0.0334	0.7179
0.2540	16050	0.0371	-
0.2547	16100	0.032	-
0.2555	16150	0.0352	-
0.2563	16200	0.032	-
0.2571	16250	0.0348	-
0.2579	16300	0.0357	-
0.2587	16350	0.0364	-
0.2595	16400	0.0322	-
0.2603	16450	0.0316	-
0.2611	16500	0.034	0.7199
0.2619	16550	0.0357	-
0.2627	16600	0.0287	-
0.2634	16650	0.0374	-
0.2642	16700	0.0389	-
0.2650	16750	0.0357	-
0.2658	16800	0.0316	-
0.2666	16850	0.0298	-
0.2674	16900	0.0332	-
0.2682	16950	0.0342	-
0.2690	17000	0.0311	0.7148
0.2698	17050	0.0323	-
0.2706	17100	0.0309	-
0.2714	17150	0.0292	-
0.2722	17200	0.0405	-
0.2729	17250	0.0269	-
0.2737	17300	0.0257	-
0.2745	17350	0.0378	-
0.2753	17400	0.0279	-
0.2761	17450	0.0353	-
0.2769	17500	0.0296	0.7244
0.2777	17550	0.0326	-
0.2785	17600	0.0259	-
0.2793	17650	0.026	-
0.2801	17700	0.0329	-
0.2809	17750	0.036	-
0.2816	17800	0.0317	-
0.2824	17850	0.0273	-
0.2832	17900	0.0302	-
0.2840	17950	0.0271	-
0.2848	18000	0.027	0.7211
0.2856	18050	0.0301	-
0.2864	18100	0.0272	-
0.2872	18150	0.0376	-
0.2880	18200	0.0311	-
0.2888	18250	0.026	-
0.2896	18300	0.0322	-
0.2903	18350	0.0238	-
0.2911	18400	0.0283	-
0.2919	18450	0.0238	-
0.2927	18500	0.025	0.7288
0.2935	18550	0.0354	-
0.2943	18600	0.0337	-
0.2951	18650	0.032	-
0.2959	18700	0.026	-
0.2967	18750	0.0242	-
0.2975	18800	0.0201	-
0.2983	18850	0.0307	-
0.2991	18900	0.0213	-
0.2998	18950	0.0245	-
0.3006	19000	0.0257	0.7198
0.3014	19050	0.024	-
0.3022	19100	0.0298	-
0.3030	19150	0.0293	-
0.3038	19200	0.0263	-
0.3046	19250	0.0264	-
0.3054	19300	0.0252	-
0.3062	19350	0.0313	-
0.3070	19400	0.0197	-
0.3078	19450	0.0253	-
0.3085	19500	0.0263	0.73
0.3093	19550	0.0273	-
0.3101	19600	0.0277	-
0.3109	19650	0.0213	-
0.3117	19700	0.0226	-
0.3125	19750	0.0228	-
0.3133	19800	0.0249	-
0.3141	19850	0.0239	-
0.3149	19900	0.0219	-
0.3157	19950	0.0234	-
0.3165	20000	0.0244	0.7238
0.3172	20050	0.0211	-
0.3180	20100	0.0253	-
0.3188	20150	0.0187	-
0.3196	20200	0.0241	-
0.3204	20250	0.0267	-
0.3212	20300	0.0363	-
0.3220	20350	0.023	-
0.3228	20400	0.0274	-
0.3236	20450	0.022	-
0.3244	20500	0.0223	0.7294
0.3252	20550	0.0241	-
0.3259	20600	0.0201	-
0.3267	20650	0.0281	-
0.3275	20700	0.0272	-
0.3283	20750	0.0225	-
0.3291	20800	0.0237	-
0.3299	20850	0.025	-
0.3307	20900	0.0312	-
0.3315	20950	0.0232	-
0.3323	21000	0.0195	0.7231
0.3331	21050	0.0214	-
0.3339	21100	0.0165	-
0.3347	21150	0.0236	-
0.3354	21200	0.0294	-
0.3362	21250	0.0274	-
0.3370	21300	0.021	-
0.3378	21350	0.0248	-
0.3386	21400	0.0189	-
0.3394	21450	0.0202	-
0.3402	21500	0.0197	0.7236
0.3410	21550	0.0252	-
0.3418	21600	0.0173	-
0.3426	21650	0.0177	-
0.3434	21700	0.0207	-
0.3441	21750	0.0153	-
0.3449	21800	0.022	-
0.3457	21850	0.0159	-
0.3465	21900	0.017	-
0.3473	21950	0.0246	-
0.3481	22000	0.0173	0.7292

The bold row denotes the saved checkpoint.

Training Time

Training: 1.7 hours
Evaluation: 8.4 minutes
Total: 1.9 hours

Framework Versions

Python: 3.14.2
Sentence Transformers: 6.1.0
Transformers: 4.57.3
PyTorch: 2.9.1a0
Accelerate: 1.15.0
Datasets: 4.4.1
Tokenizers: 0.22.1

Additional Resources

Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}