--- tags: - sentence-transformers - sentence-similarity - feature-extraction - dense - generated_from_trainer - dataset_size:8089600 - loss:MultipleNegativesRankingLoss base_model: sentence-transformers/embeddinggemma-300m-medical widget: - source_sentence: anaemia sentences: - anaemic nos - scurvy - foreign body - source_sentence: paludism sentences: - congestive fever - ocular myiasis - gnathophyma - source_sentence: amniocele sentences: - inefction - omphalocele - chylous effusion - source_sentence: polyploid sentences: - polyploidy - monosomy 19p13.12 - otorrhagia - source_sentence: polysplenia sentences: - cut cysts - splenunculi - spleen agenesis pipeline_tag: sentence-similarity library_name: sentence-transformers metrics: - cosine_accuracy@1 - cosine_accuracy@5 - cosine_accuracy@10 - cosine_precision@1 - cosine_precision@3 - cosine_precision@5 - cosine_precision@10 - cosine_recall@1 - cosine_recall@3 - cosine_recall@5 - cosine_recall@10 - cosine_ndcg@10 - cosine_mrr@10 - cosine_map@10 model-index: - name: SentenceTransformer based on sentence-transformers/embeddinggemma-300m-medical results: - task: type: information-retrieval name: Information Retrieval dataset: name: icd11 val type: icd11-val metrics: - type: cosine_accuracy@1 value: 0.7742690058479532 name: Cosine Accuracy@1 - type: cosine_accuracy@5 value: 0.8432748538011696 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.8643274853801169 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.7742690058479532 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.6912280701754386 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.5929824561403508 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.43543859649122807 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.16470727696455337 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.4096813550232527 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.5202410469498723 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.6459809458753583 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.7291630675233993 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.8052028218694884 name: Cosine Mrr@10 - type: cosine_map@10 value: 0.6770894371068766 name: Cosine Map@10 --- # SentenceTransformer based on sentence-transformers/embeddinggemma-300m-medical This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [sentence-transformers/embeddinggemma-300m-medical](https://huggingface.co/sentence-transformers/embeddinggemma-300m-medical). It maps inputs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Base model:** [sentence-transformers/embeddinggemma-300m-medical](https://huggingface.co/sentence-transformers/embeddinggemma-300m-medical) - **Maximum Sequence Length:** 30 tokens - **Output Dimensionality:** 768 dimensions - **Similarity Function:** Cosine Similarity - **Supported Modality:** Text ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Repository:** [Sentence Transformers on GitHub](https://github.com/huggingface/sentence-transformers) - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'Gemma3TextModel'}) (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True}) (2): Dense({'in_features': 768, 'out_features': 3072, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'}) (3): Dense({'in_features': 3072, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'}) (4): Normalize({'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'}) ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SentenceTransformer # Download from the 🤗 Hub model = SentenceTransformer("seanfarrell/icd11_embed_model_v2_large") # Run inference queries = [ 'polysplenia', ] documents = [ 'splenunculi', 'spleen agenesis', 'cut cysts', ] query_embeddings = model.encode_query(queries) document_embeddings = model.encode_document(documents) print(query_embeddings.shape, document_embeddings.shape) # [1, 768] [3, 768] # Get the similarity scores for the embeddings similarities = model.similarity(query_embeddings, document_embeddings) print(similarities) # tensor([[ 0.5815, 0.3983, -0.0705]]) ``` ## Evaluation ### Metrics #### Information Retrieval * Dataset: `icd11-val` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.sentence_transformer.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.7743 | | cosine_accuracy@5 | 0.8433 | | cosine_accuracy@10 | 0.8643 | | cosine_precision@1 | 0.7743 | | cosine_precision@3 | 0.6912 | | cosine_precision@5 | 0.593 | | cosine_precision@10 | 0.4354 | | cosine_recall@1 | 0.1647 | | cosine_recall@3 | 0.4097 | | cosine_recall@5 | 0.5202 | | cosine_recall@10 | 0.646 | | **cosine_ndcg@10** | **0.7292** | | cosine_mrr@10 | 0.8052 | | cosine_map@10 | 0.6771 | ## Training Details ### Training Dataset #### Unnamed Dataset * Size: 8,089,600 training samples * Columns: anchor, positive, and negative * Approximate statistics based on the first 100 samples: | | anchor | positive | negative | |:---------|:---------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:--------------------------------------------------------------------------------| | type | string | string | string | | modality | text | text | text | | details | | | | * Samples: | anchor | positive | negative | |:---------------------------------------------------------------------|:---------------------------------------------------------------------------------|:--------------------------------------------------------------------------| | chronic rheumatic heart disease, unspecified | rheumatic pancarditis | de morgan's spots | | chronic rheumatic left ventricular failure | inactive or quiescent rheumatic fever with congestive heart failure | conjunctival xerosis with bitot's spots | | inactive or quiescent rheumatic fever with endocarditis | inactive rheumatic valvar endocarditis | mature b-cell neoplasm with leukaemic behaviour, unspecified | * Loss: [MultipleNegativesRankingLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with these parameters: ```json { "scale": 20.0, "similarity_fct": "cos_sim", "gather_across_devices": false, "directions": [ "query_to_doc" ], "partition_mode": "joint", "hardness_mode": null, "hardness_strength": 0.0 } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `per_device_train_batch_size`: 128 - `per_device_eval_batch_size`: 256 - `learning_rate`: 2e-05 - `weight_decay`: 0.01 - `num_train_epochs`: 1 - `lr_scheduler_type`: cosine - `warmup_steps`: 6320 - `bf16`: True - `dataloader_drop_last`: True - `load_best_model_at_end`: True - `eval_on_start`: True #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `prediction_loss_only`: True - `per_device_train_batch_size`: 128 - `per_device_eval_batch_size`: 256 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 1 - `eval_accumulation_steps`: None - `torch_empty_cache_steps`: None - `learning_rate`: 2e-05 - `weight_decay`: 0.01 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1.0 - `num_train_epochs`: 1 - `max_steps`: -1 - `lr_scheduler_type`: cosine - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.0 - `warmup_steps`: 6320 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 42 - `data_seed`: None - `jit_mode_eval`: False - `bf16`: True - `fp16`: False - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: True - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: True - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `parallelism_config`: None - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch_fused - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `project`: huggingface - `trackio_space_id`: trackio - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: None - `hub_always_push`: False - `hub_revision`: None - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `include_for_metrics`: [] - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: no - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `eval_on_start`: True - `use_liger_kernel`: False - `liger_kernel_config`: None - `eval_use_gather_object`: False - `average_tokens_across_devices`: True - `prompts`: None - `batch_sampler`: batch_sampler - `multi_dataset_batch_sampler`: proportional - `router_mapping`: {} - `learning_rate_mapping`: {}
### Training Logs
Click to expand | Epoch | Step | Training Loss | icd11-val_cosine_ndcg@10 | |:----------:|:---------:|:-------------:|:------------------------:| | 0 | 0 | - | 0.5062 | | 0.0008 | 50 | 1.6688 | - | | 0.0016 | 100 | 1.6505 | - | | 0.0024 | 150 | 1.5416 | - | | 0.0032 | 200 | 1.4195 | - | | 0.0040 | 250 | 1.3759 | - | | 0.0047 | 300 | 1.2595 | - | | 0.0055 | 350 | 1.2113 | - | | 0.0063 | 400 | 1.1349 | - | | 0.0071 | 450 | 1.1253 | - | | 0.0079 | 500 | 1.0463 | 0.5645 | | 0.0087 | 550 | 0.9832 | - | | 0.0095 | 600 | 0.93 | - | | 0.0103 | 650 | 0.8592 | - | | 0.0111 | 700 | 0.8343 | - | | 0.0119 | 750 | 0.8298 | - | | 0.0127 | 800 | 0.7648 | - | | 0.0134 | 850 | 0.724 | - | | 0.0142 | 900 | 0.7301 | - | | 0.0150 | 950 | 0.6552 | - | | 0.0158 | 1000 | 0.6649 | 0.5913 | | 0.0166 | 1050 | 0.6055 | - | | 0.0174 | 1100 | 0.6287 | - | | 0.0182 | 1150 | 0.5632 | - | | 0.0190 | 1200 | 0.5289 | - | | 0.0198 | 1250 | 0.5299 | - | | 0.0206 | 1300 | 0.5321 | - | | 0.0214 | 1350 | 0.5087 | - | | 0.0222 | 1400 | 0.4714 | - | | 0.0229 | 1450 | 0.4598 | - | | 0.0237 | 1500 | 0.4614 | 0.6138 | | 0.0245 | 1550 | 0.45 | - | | 0.0253 | 1600 | 0.4241 | - | | 0.0261 | 1650 | 0.4047 | - | | 0.0269 | 1700 | 0.4162 | - | | 0.0277 | 1750 | 0.3737 | - | | 0.0285 | 1800 | 0.4114 | - | | 0.0293 | 1850 | 0.376 | - | | 0.0301 | 1900 | 0.3791 | - | | 0.0309 | 1950 | 0.3791 | - | | 0.0316 | 2000 | 0.3643 | 0.6310 | | 0.0324 | 2050 | 0.3543 | - | | 0.0332 | 2100 | 0.3024 | - | | 0.0340 | 2150 | 0.3294 | - | | 0.0348 | 2200 | 0.3017 | - | | 0.0356 | 2250 | 0.3334 | - | | 0.0364 | 2300 | 0.3158 | - | | 0.0372 | 2350 | 0.3252 | - | | 0.0380 | 2400 | 0.2956 | - | | 0.0388 | 2450 | 0.2773 | - | | 0.0396 | 2500 | 0.2994 | 0.6399 | | 0.0403 | 2550 | 0.2725 | - | | 0.0411 | 2600 | 0.2531 | - | | 0.0419 | 2650 | 0.2924 | - | | 0.0427 | 2700 | 0.286 | - | | 0.0435 | 2750 | 0.2653 | - | | 0.0443 | 2800 | 0.2588 | - | | 0.0451 | 2850 | 0.2428 | - | | 0.0459 | 2900 | 0.243 | - | | 0.0467 | 2950 | 0.2426 | - | | 0.0475 | 3000 | 0.2524 | 0.6577 | | 0.0483 | 3050 | 0.2447 | - | | 0.0491 | 3100 | 0.2474 | - | | 0.0498 | 3150 | 0.2565 | - | | 0.0506 | 3200 | 0.2397 | - | | 0.0514 | 3250 | 0.2345 | - | | 0.0522 | 3300 | 0.2229 | - | | 0.0530 | 3350 | 0.2418 | - | | 0.0538 | 3400 | 0.2271 | - | | 0.0546 | 3450 | 0.2189 | - | | 0.0554 | 3500 | 0.2163 | 0.6591 | | 0.0562 | 3550 | 0.2052 | - | | 0.0570 | 3600 | 0.217 | - | | 0.0578 | 3650 | 0.2038 | - | | 0.0585 | 3700 | 0.2268 | - | | 0.0593 | 3750 | 0.2035 | - | | 0.0601 | 3800 | 0.2077 | - | | 0.0609 | 3850 | 0.2055 | - | | 0.0617 | 3900 | 0.2207 | - | | 0.0625 | 3950 | 0.2004 | - | | 0.0633 | 4000 | 0.2053 | 0.6627 | | 0.0641 | 4050 | 0.2002 | - | | 0.0649 | 4100 | 0.1938 | - | | 0.0657 | 4150 | 0.2213 | - | | 0.0665 | 4200 | 0.2146 | - | | 0.0672 | 4250 | 0.1949 | - | | 0.0680 | 4300 | 0.1944 | - | | 0.0688 | 4350 | 0.1777 | - | | 0.0696 | 4400 | 0.196 | - | | 0.0704 | 4450 | 0.1927 | - | | 0.0712 | 4500 | 0.1805 | 0.6745 | | 0.0720 | 4550 | 0.1934 | - | | 0.0728 | 4600 | 0.1892 | - | | 0.0736 | 4650 | 0.2022 | - | | 0.0744 | 4700 | 0.1892 | - | | 0.0752 | 4750 | 0.1812 | - | | 0.0759 | 4800 | 0.1878 | - | | 0.0767 | 4850 | 0.1815 | - | | 0.0775 | 4900 | 0.1932 | - | | 0.0783 | 4950 | 0.1901 | - | | 0.0791 | 5000 | 0.1753 | 0.6772 | | 0.0799 | 5050 | 0.186 | - | | 0.0807 | 5100 | 0.1768 | - | | 0.0815 | 5150 | 0.1588 | - | | 0.0823 | 5200 | 0.1788 | - | | 0.0831 | 5250 | 0.1853 | - | | 0.0839 | 5300 | 0.1796 | - | | 0.0847 | 5350 | 0.1703 | - | | 0.0854 | 5400 | 0.1905 | - | | 0.0862 | 5450 | 0.1818 | - | | 0.0870 | 5500 | 0.188 | 0.6699 | | 0.0878 | 5550 | 0.1665 | - | | 0.0886 | 5600 | 0.1639 | - | | 0.0894 | 5650 | 0.1664 | - | | 0.0902 | 5700 | 0.1757 | - | | 0.0910 | 5750 | 0.1688 | - | | 0.0918 | 5800 | 0.1487 | - | | 0.0926 | 5850 | 0.1717 | - | | 0.0934 | 5900 | 0.167 | - | | 0.0941 | 5950 | 0.1595 | - | | 0.0949 | 6000 | 0.1696 | 0.6811 | | 0.0957 | 6050 | 0.1676 | - | | 0.0965 | 6100 | 0.1746 | - | | 0.0973 | 6150 | 0.1483 | - | | 0.0981 | 6200 | 0.1636 | - | | 0.0989 | 6250 | 0.1517 | - | | 0.0997 | 6300 | 0.165 | - | | 0.1005 | 6350 | 0.1634 | - | | 0.1013 | 6400 | 0.1408 | - | | 0.1021 | 6450 | 0.1623 | - | | 0.1028 | 6500 | 0.1656 | 0.6789 | | 0.1036 | 6550 | 0.1609 | - | | 0.1044 | 6600 | 0.1612 | - | | 0.1052 | 6650 | 0.16 | - | | 0.1060 | 6700 | 0.1657 | - | | 0.1068 | 6750 | 0.1427 | - | | 0.1076 | 6800 | 0.1355 | - | | 0.1084 | 6850 | 0.1401 | - | | 0.1092 | 6900 | 0.1409 | - | | 0.1100 | 6950 | 0.1403 | - | | 0.1108 | 7000 | 0.138 | 0.6926 | | 0.1116 | 7050 | 0.1369 | - | | 0.1123 | 7100 | 0.1295 | - | | 0.1131 | 7150 | 0.1407 | - | | 0.1139 | 7200 | 0.1388 | - | | 0.1147 | 7250 | 0.1377 | - | | 0.1155 | 7300 | 0.1324 | - | | 0.1163 | 7350 | 0.128 | - | | 0.1171 | 7400 | 0.1256 | - | | 0.1179 | 7450 | 0.126 | - | | 0.1187 | 7500 | 0.1284 | 0.6911 | | 0.1195 | 7550 | 0.1246 | - | | 0.1203 | 7600 | 0.1119 | - | | 0.1210 | 7650 | 0.1082 | - | | 0.1218 | 7700 | 0.1305 | - | | 0.1226 | 7750 | 0.1033 | - | | 0.1234 | 7800 | 0.1125 | - | | 0.1242 | 7850 | 0.1256 | - | | 0.125 | 7900 | 0.1403 | - | | 0.1258 | 7950 | 0.1334 | - | | 0.1266 | 8000 | 0.1173 | 0.6939 | | 0.1274 | 8050 | 0.106 | - | | 0.1282 | 8100 | 0.1127 | - | | 0.1290 | 8150 | 0.1185 | - | | 0.1297 | 8200 | 0.1045 | - | | 0.1305 | 8250 | 0.0978 | - | | 0.1313 | 8300 | 0.1074 | - | | 0.1321 | 8350 | 0.1114 | - | | 0.1329 | 8400 | 0.1127 | - | | 0.1337 | 8450 | 0.1035 | - | | 0.1345 | 8500 | 0.0984 | 0.7023 | | 0.1353 | 8550 | 0.1065 | - | | 0.1361 | 8600 | 0.1009 | - | | 0.1369 | 8650 | 0.0837 | - | | 0.1377 | 8700 | 0.0932 | - | | 0.1384 | 8750 | 0.1096 | - | | 0.1392 | 8800 | 0.1065 | - | | 0.1400 | 8850 | 0.1029 | - | | 0.1408 | 8900 | 0.1123 | - | | 0.1416 | 8950 | 0.1043 | - | | 0.1424 | 9000 | 0.0947 | 0.6998 | | 0.1432 | 9050 | 0.0981 | - | | 0.1440 | 9100 | 0.0917 | - | | 0.1448 | 9150 | 0.0962 | - | | 0.1456 | 9200 | 0.0902 | - | | 0.1464 | 9250 | 0.1001 | - | | 0.1472 | 9300 | 0.0852 | - | | 0.1479 | 9350 | 0.0724 | - | | 0.1487 | 9400 | 0.0892 | - | | 0.1495 | 9450 | 0.0774 | - | | 0.1503 | 9500 | 0.0819 | 0.7025 | | 0.1511 | 9550 | 0.0831 | - | | 0.1519 | 9600 | 0.0807 | - | | 0.1527 | 9650 | 0.0833 | - | | 0.1535 | 9700 | 0.0823 | - | | 0.1543 | 9750 | 0.0754 | - | | 0.1551 | 9800 | 0.0759 | - | | 0.1559 | 9850 | 0.0856 | - | | 0.1566 | 9900 | 0.0787 | - | | 0.1574 | 9950 | 0.0863 | - | | 0.1582 | 10000 | 0.0821 | 0.6993 | | 0.1590 | 10050 | 0.0761 | - | | 0.1598 | 10100 | 0.076 | - | | 0.1606 | 10150 | 0.0742 | - | | 0.1614 | 10200 | 0.0684 | - | | 0.1622 | 10250 | 0.0754 | - | | 0.1630 | 10300 | 0.0729 | - | | 0.1638 | 10350 | 0.0852 | - | | 0.1646 | 10400 | 0.0778 | - | | 0.1653 | 10450 | 0.078 | - | | 0.1661 | 10500 | 0.0652 | 0.6972 | | 0.1669 | 10550 | 0.0766 | - | | 0.1677 | 10600 | 0.0805 | - | | 0.1685 | 10650 | 0.0736 | - | | 0.1693 | 10700 | 0.0702 | - | | 0.1701 | 10750 | 0.0789 | - | | 0.1709 | 10800 | 0.0683 | - | | 0.1717 | 10850 | 0.0788 | - | | 0.1725 | 10900 | 0.0592 | - | | 0.1733 | 10950 | 0.0716 | - | | 0.1741 | 11000 | 0.0647 | 0.7031 | | 0.1748 | 11050 | 0.0624 | - | | 0.1756 | 11100 | 0.0629 | - | | 0.1764 | 11150 | 0.0653 | - | | 0.1772 | 11200 | 0.0699 | - | | 0.1780 | 11250 | 0.0661 | - | | 0.1788 | 11300 | 0.0534 | - | | 0.1796 | 11350 | 0.0559 | - | | 0.1804 | 11400 | 0.0656 | - | | 0.1812 | 11450 | 0.0689 | - | | 0.1820 | 11500 | 0.0642 | 0.7067 | | 0.1828 | 11550 | 0.0586 | - | | 0.1835 | 11600 | 0.0616 | - | | 0.1843 | 11650 | 0.0582 | - | | 0.1851 | 11700 | 0.0668 | - | | 0.1859 | 11750 | 0.0577 | - | | 0.1867 | 11800 | 0.0614 | - | | 0.1875 | 11850 | 0.0598 | - | | 0.1883 | 11900 | 0.0488 | - | | 0.1891 | 11950 | 0.0721 | - | | 0.1899 | 12000 | 0.0623 | 0.7102 | | 0.1907 | 12050 | 0.0542 | - | | 0.1915 | 12100 | 0.0648 | - | | 0.1922 | 12150 | 0.0589 | - | | 0.1930 | 12200 | 0.0538 | - | | 0.1938 | 12250 | 0.0511 | - | | 0.1946 | 12300 | 0.0624 | - | | 0.1954 | 12350 | 0.0586 | - | | 0.1962 | 12400 | 0.052 | - | | 0.1970 | 12450 | 0.0629 | - | | 0.1978 | 12500 | 0.0557 | 0.7136 | | 0.1986 | 12550 | 0.052 | - | | 0.1994 | 12600 | 0.0597 | - | | 0.2002 | 12650 | 0.0524 | - | | 0.2009 | 12700 | 0.0465 | - | | 0.2017 | 12750 | 0.0529 | - | | 0.2025 | 12800 | 0.0591 | - | | 0.2033 | 12850 | 0.0527 | - | | 0.2041 | 12900 | 0.0553 | - | | 0.2049 | 12950 | 0.0518 | - | | 0.2057 | 13000 | 0.0561 | 0.7007 | | 0.2065 | 13050 | 0.0559 | - | | 0.2073 | 13100 | 0.0569 | - | | 0.2081 | 13150 | 0.0522 | - | | 0.2089 | 13200 | 0.0586 | - | | 0.2097 | 13250 | 0.0482 | - | | 0.2104 | 13300 | 0.0496 | - | | 0.2112 | 13350 | 0.0512 | - | | 0.2120 | 13400 | 0.0468 | - | | 0.2128 | 13450 | 0.0448 | - | | 0.2136 | 13500 | 0.0566 | 0.7088 | | 0.2144 | 13550 | 0.0541 | - | | 0.2152 | 13600 | 0.048 | - | | 0.2160 | 13650 | 0.0455 | - | | 0.2168 | 13700 | 0.0484 | - | | 0.2176 | 13750 | 0.0512 | - | | 0.2184 | 13800 | 0.0414 | - | | 0.2191 | 13850 | 0.0467 | - | | 0.2199 | 13900 | 0.0413 | - | | 0.2207 | 13950 | 0.0483 | - | | 0.2215 | 14000 | 0.0531 | 0.7072 | | 0.2223 | 14050 | 0.0497 | - | | 0.2231 | 14100 | 0.0507 | - | | 0.2239 | 14150 | 0.0417 | - | | 0.2247 | 14200 | 0.0479 | - | | 0.2255 | 14250 | 0.0515 | - | | 0.2263 | 14300 | 0.0447 | - | | 0.2271 | 14350 | 0.0443 | - | | 0.2278 | 14400 | 0.0416 | - | | 0.2286 | 14450 | 0.0471 | - | | 0.2294 | 14500 | 0.039 | 0.7108 | | 0.2302 | 14550 | 0.0535 | - | | 0.2310 | 14600 | 0.0404 | - | | 0.2318 | 14650 | 0.037 | - | | 0.2326 | 14700 | 0.0379 | - | | 0.2334 | 14750 | 0.048 | - | | 0.2342 | 14800 | 0.0417 | - | | 0.2350 | 14850 | 0.0405 | - | | 0.2358 | 14900 | 0.0408 | - | | 0.2366 | 14950 | 0.0388 | - | | 0.2373 | 15000 | 0.0376 | 0.7160 | | 0.2381 | 15050 | 0.0443 | - | | 0.2389 | 15100 | 0.0436 | - | | 0.2397 | 15150 | 0.0374 | - | | 0.2405 | 15200 | 0.0378 | - | | 0.2413 | 15250 | 0.0395 | - | | 0.2421 | 15300 | 0.0437 | - | | 0.2429 | 15350 | 0.0406 | - | | 0.2437 | 15400 | 0.0396 | - | | 0.2445 | 15450 | 0.042 | - | | 0.2453 | 15500 | 0.0395 | 0.7122 | | 0.2460 | 15550 | 0.0371 | - | | 0.2468 | 15600 | 0.037 | - | | 0.2476 | 15650 | 0.0458 | - | | 0.2484 | 15700 | 0.0377 | - | | 0.2492 | 15750 | 0.0367 | - | | 0.25 | 15800 | 0.0459 | - | | 0.2508 | 15850 | 0.0354 | - | | 0.2516 | 15900 | 0.0338 | - | | 0.2524 | 15950 | 0.0363 | - | | 0.2532 | 16000 | 0.0334 | 0.7179 | | 0.2540 | 16050 | 0.0371 | - | | 0.2547 | 16100 | 0.032 | - | | 0.2555 | 16150 | 0.0352 | - | | 0.2563 | 16200 | 0.032 | - | | 0.2571 | 16250 | 0.0348 | - | | 0.2579 | 16300 | 0.0357 | - | | 0.2587 | 16350 | 0.0364 | - | | 0.2595 | 16400 | 0.0322 | - | | 0.2603 | 16450 | 0.0316 | - | | 0.2611 | 16500 | 0.034 | 0.7199 | | 0.2619 | 16550 | 0.0357 | - | | 0.2627 | 16600 | 0.0287 | - | | 0.2634 | 16650 | 0.0374 | - | | 0.2642 | 16700 | 0.0389 | - | | 0.2650 | 16750 | 0.0357 | - | | 0.2658 | 16800 | 0.0316 | - | | 0.2666 | 16850 | 0.0298 | - | | 0.2674 | 16900 | 0.0332 | - | | 0.2682 | 16950 | 0.0342 | - | | 0.2690 | 17000 | 0.0311 | 0.7148 | | 0.2698 | 17050 | 0.0323 | - | | 0.2706 | 17100 | 0.0309 | - | | 0.2714 | 17150 | 0.0292 | - | | 0.2722 | 17200 | 0.0405 | - | | 0.2729 | 17250 | 0.0269 | - | | 0.2737 | 17300 | 0.0257 | - | | 0.2745 | 17350 | 0.0378 | - | | 0.2753 | 17400 | 0.0279 | - | | 0.2761 | 17450 | 0.0353 | - | | 0.2769 | 17500 | 0.0296 | 0.7244 | | 0.2777 | 17550 | 0.0326 | - | | 0.2785 | 17600 | 0.0259 | - | | 0.2793 | 17650 | 0.026 | - | | 0.2801 | 17700 | 0.0329 | - | | 0.2809 | 17750 | 0.036 | - | | 0.2816 | 17800 | 0.0317 | - | | 0.2824 | 17850 | 0.0273 | - | | 0.2832 | 17900 | 0.0302 | - | | 0.2840 | 17950 | 0.0271 | - | | 0.2848 | 18000 | 0.027 | 0.7211 | | 0.2856 | 18050 | 0.0301 | - | | 0.2864 | 18100 | 0.0272 | - | | 0.2872 | 18150 | 0.0376 | - | | 0.2880 | 18200 | 0.0311 | - | | 0.2888 | 18250 | 0.026 | - | | 0.2896 | 18300 | 0.0322 | - | | 0.2903 | 18350 | 0.0238 | - | | 0.2911 | 18400 | 0.0283 | - | | 0.2919 | 18450 | 0.0238 | - | | 0.2927 | 18500 | 0.025 | 0.7288 | | 0.2935 | 18550 | 0.0354 | - | | 0.2943 | 18600 | 0.0337 | - | | 0.2951 | 18650 | 0.032 | - | | 0.2959 | 18700 | 0.026 | - | | 0.2967 | 18750 | 0.0242 | - | | 0.2975 | 18800 | 0.0201 | - | | 0.2983 | 18850 | 0.0307 | - | | 0.2991 | 18900 | 0.0213 | - | | 0.2998 | 18950 | 0.0245 | - | | 0.3006 | 19000 | 0.0257 | 0.7198 | | 0.3014 | 19050 | 0.024 | - | | 0.3022 | 19100 | 0.0298 | - | | 0.3030 | 19150 | 0.0293 | - | | 0.3038 | 19200 | 0.0263 | - | | 0.3046 | 19250 | 0.0264 | - | | 0.3054 | 19300 | 0.0252 | - | | 0.3062 | 19350 | 0.0313 | - | | 0.3070 | 19400 | 0.0197 | - | | 0.3078 | 19450 | 0.0253 | - | | **0.3085** | **19500** | **0.0263** | **0.73** | | 0.3093 | 19550 | 0.0273 | - | | 0.3101 | 19600 | 0.0277 | - | | 0.3109 | 19650 | 0.0213 | - | | 0.3117 | 19700 | 0.0226 | - | | 0.3125 | 19750 | 0.0228 | - | | 0.3133 | 19800 | 0.0249 | - | | 0.3141 | 19850 | 0.0239 | - | | 0.3149 | 19900 | 0.0219 | - | | 0.3157 | 19950 | 0.0234 | - | | 0.3165 | 20000 | 0.0244 | 0.7238 | | 0.3172 | 20050 | 0.0211 | - | | 0.3180 | 20100 | 0.0253 | - | | 0.3188 | 20150 | 0.0187 | - | | 0.3196 | 20200 | 0.0241 | - | | 0.3204 | 20250 | 0.0267 | - | | 0.3212 | 20300 | 0.0363 | - | | 0.3220 | 20350 | 0.023 | - | | 0.3228 | 20400 | 0.0274 | - | | 0.3236 | 20450 | 0.022 | - | | 0.3244 | 20500 | 0.0223 | 0.7294 | | 0.3252 | 20550 | 0.0241 | - | | 0.3259 | 20600 | 0.0201 | - | | 0.3267 | 20650 | 0.0281 | - | | 0.3275 | 20700 | 0.0272 | - | | 0.3283 | 20750 | 0.0225 | - | | 0.3291 | 20800 | 0.0237 | - | | 0.3299 | 20850 | 0.025 | - | | 0.3307 | 20900 | 0.0312 | - | | 0.3315 | 20950 | 0.0232 | - | | 0.3323 | 21000 | 0.0195 | 0.7231 | | 0.3331 | 21050 | 0.0214 | - | | 0.3339 | 21100 | 0.0165 | - | | 0.3347 | 21150 | 0.0236 | - | | 0.3354 | 21200 | 0.0294 | - | | 0.3362 | 21250 | 0.0274 | - | | 0.3370 | 21300 | 0.021 | - | | 0.3378 | 21350 | 0.0248 | - | | 0.3386 | 21400 | 0.0189 | - | | 0.3394 | 21450 | 0.0202 | - | | 0.3402 | 21500 | 0.0197 | 0.7236 | | 0.3410 | 21550 | 0.0252 | - | | 0.3418 | 21600 | 0.0173 | - | | 0.3426 | 21650 | 0.0177 | - | | 0.3434 | 21700 | 0.0207 | - | | 0.3441 | 21750 | 0.0153 | - | | 0.3449 | 21800 | 0.022 | - | | 0.3457 | 21850 | 0.0159 | - | | 0.3465 | 21900 | 0.017 | - | | 0.3473 | 21950 | 0.0246 | - | | 0.3481 | 22000 | 0.0173 | 0.7292 | * The bold row denotes the saved checkpoint.
### Training Time - **Training**: 1.7 hours - **Evaluation**: 8.4 minutes - **Total**: 1.9 hours ### Framework Versions - Python: 3.14.2 - Sentence Transformers: 6.1.0 - Transformers: 4.57.3 - PyTorch: 2.9.1a0 - Accelerate: 1.15.0 - Datasets: 4.4.1 - Tokenizers: 0.22.1 ## Additional Resources - [Training and Finetuning Embedding Models with Sentence Transformers](https://huggingface.co/blog/train-sentence-transformers): the end-to-end guide for training or finetuning Sentence Transformer models. - [Introduction to Matryoshka Embedding Models](https://huggingface.co/blog/matryoshka): variable-size embeddings that can be truncated with minimal quality loss. - [Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval](https://huggingface.co/blog/embedding-quantization): post-training compression of embedding vectors. - [Multimodal Embedding & Reranker Models with Sentence Transformers](https://huggingface.co/blog/multimodal-sentence-transformers): use text, image, audio, and video models through the same API. - [Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers](https://huggingface.co/blog/train-multimodal-sentence-transformers): train multimodal embedding models, with a Visual Document Retrieval walkthrough. ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### MultipleNegativesRankingLoss ```bibtex @misc{oord2019representationlearningcontrastivepredictive, title={Representation Learning with Contrastive Predictive Coding}, author={Aaron van den Oord and Yazhe Li and Oriol Vinyals}, year={2019}, eprint={1807.03748}, archivePrefix={arXiv}, primaryClass={cs.LG}, url={https://arxiv.org/abs/1807.03748}, } ```