| --- |
| tags: |
| - sentence-transformers |
| - sentence-similarity |
| - feature-extraction |
| - dense |
| - generated_from_trainer |
| - dataset_size:8089600 |
| - loss:MultipleNegativesRankingLoss |
| base_model: sentence-transformers/embeddinggemma-300m-medical |
| widget: |
| - source_sentence: anaemia |
| sentences: |
| - anaemic nos |
| - scurvy |
| - foreign body |
| - source_sentence: paludism |
| sentences: |
| - congestive fever |
| - ocular myiasis |
| - gnathophyma |
| - source_sentence: amniocele |
| sentences: |
| - inefction |
| - omphalocele |
| - chylous effusion |
| - source_sentence: polyploid |
| sentences: |
| - polyploidy |
| - monosomy 19p13.12 |
| - otorrhagia |
| - source_sentence: polysplenia |
| sentences: |
| - cut cysts |
| - splenunculi |
| - spleen agenesis |
| pipeline_tag: sentence-similarity |
| library_name: sentence-transformers |
| metrics: |
| - cosine_accuracy@1 |
| - cosine_accuracy@5 |
| - cosine_accuracy@10 |
| - cosine_precision@1 |
| - cosine_precision@3 |
| - cosine_precision@5 |
| - cosine_precision@10 |
| - cosine_recall@1 |
| - cosine_recall@3 |
| - cosine_recall@5 |
| - cosine_recall@10 |
| - cosine_ndcg@10 |
| - cosine_mrr@10 |
| - cosine_map@10 |
| model-index: |
| - name: SentenceTransformer based on sentence-transformers/embeddinggemma-300m-medical |
| results: |
| - task: |
| type: information-retrieval |
| name: Information Retrieval |
| dataset: |
| name: icd11 val |
| type: icd11-val |
| metrics: |
| - type: cosine_accuracy@1 |
| value: 0.7742690058479532 |
| name: Cosine Accuracy@1 |
| - type: cosine_accuracy@5 |
| value: 0.8432748538011696 |
| name: Cosine Accuracy@5 |
| - type: cosine_accuracy@10 |
| value: 0.8643274853801169 |
| name: Cosine Accuracy@10 |
| - type: cosine_precision@1 |
| value: 0.7742690058479532 |
| name: Cosine Precision@1 |
| - type: cosine_precision@3 |
| value: 0.6912280701754386 |
| name: Cosine Precision@3 |
| - type: cosine_precision@5 |
| value: 0.5929824561403508 |
| name: Cosine Precision@5 |
| - type: cosine_precision@10 |
| value: 0.43543859649122807 |
| name: Cosine Precision@10 |
| - type: cosine_recall@1 |
| value: 0.16470727696455337 |
| name: Cosine Recall@1 |
| - type: cosine_recall@3 |
| value: 0.4096813550232527 |
| name: Cosine Recall@3 |
| - type: cosine_recall@5 |
| value: 0.5202410469498723 |
| name: Cosine Recall@5 |
| - type: cosine_recall@10 |
| value: 0.6459809458753583 |
| name: Cosine Recall@10 |
| - type: cosine_ndcg@10 |
| value: 0.7291630675233993 |
| name: Cosine Ndcg@10 |
| - type: cosine_mrr@10 |
| value: 0.8052028218694884 |
| name: Cosine Mrr@10 |
| - type: cosine_map@10 |
| value: 0.6770894371068766 |
| name: Cosine Map@10 |
| --- |
| |
| # SentenceTransformer based on sentence-transformers/embeddinggemma-300m-medical |
|
|
| This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [sentence-transformers/embeddinggemma-300m-medical](https://huggingface.co/sentence-transformers/embeddinggemma-300m-medical). It maps inputs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more. |
|
|
| ## Model Details |
|
|
| ### Model Description |
| - **Model Type:** Sentence Transformer |
| - **Base model:** [sentence-transformers/embeddinggemma-300m-medical](https://huggingface.co/sentence-transformers/embeddinggemma-300m-medical) <!-- at revision ffb7eedc5047b2e26bcbccbf09c435ae4a80a08d --> |
| - **Maximum Sequence Length:** 30 tokens |
| - **Output Dimensionality:** 768 dimensions |
| - **Similarity Function:** Cosine Similarity |
| - **Supported Modality:** Text |
| <!-- - **Training Dataset:** Unknown --> |
| <!-- - **Language:** Unknown --> |
| <!-- - **License:** Unknown --> |
|
|
| ### Model Sources |
|
|
| - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) |
| - **Repository:** [Sentence Transformers on GitHub](https://github.com/huggingface/sentence-transformers) |
| - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) |
|
|
| ### Full Model Architecture |
|
|
| ``` |
| SentenceTransformer( |
| (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'Gemma3TextModel'}) |
| (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True}) |
| (2): Dense({'in_features': 768, 'out_features': 3072, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'}) |
| (3): Dense({'in_features': 3072, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'}) |
| (4): Normalize({'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'}) |
| ) |
| ``` |
|
|
| ## Usage |
|
|
| ### Direct Usage (Sentence Transformers) |
|
|
| First install the Sentence Transformers library: |
|
|
| ```bash |
| pip install -U sentence-transformers |
| ``` |
| Then you can load this model and run inference. |
| ```python |
| from sentence_transformers import SentenceTransformer |
| |
| # Download from the 🤗 Hub |
| model = SentenceTransformer("seanfarrell/icd11_embed_model_v2_large") |
| # Run inference |
| queries = [ |
| 'polysplenia', |
| ] |
| documents = [ |
| 'splenunculi', |
| 'spleen agenesis', |
| 'cut cysts', |
| ] |
| query_embeddings = model.encode_query(queries) |
| document_embeddings = model.encode_document(documents) |
| print(query_embeddings.shape, document_embeddings.shape) |
| # [1, 768] [3, 768] |
| |
| # Get the similarity scores for the embeddings |
| similarities = model.similarity(query_embeddings, document_embeddings) |
| print(similarities) |
| # tensor([[ 0.5815, 0.3983, -0.0705]]) |
| ``` |
| <!-- |
| ### Direct Usage (Transformers) |
|
|
| <details><summary>Click to see the direct usage in Transformers</summary> |
|
|
| </details> |
| --> |
|
|
| <!-- |
| ### Downstream Usage (Sentence Transformers) |
|
|
| You can finetune this model on your own dataset. |
|
|
| <details><summary>Click to expand</summary> |
|
|
| </details> |
| --> |
|
|
| <!-- |
| ### Out-of-Scope Use |
|
|
| *List how the model may foreseeably be misused and address what users ought not to do with the model.* |
| --> |
|
|
| ## Evaluation |
|
|
| ### Metrics |
|
|
| #### Information Retrieval |
|
|
| * Dataset: `icd11-val` |
| * Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.sentence_transformer.evaluation.InformationRetrievalEvaluator) |
|
|
| | Metric | Value | |
| |:--------------------|:-----------| |
| | cosine_accuracy@1 | 0.7743 | |
| | cosine_accuracy@5 | 0.8433 | |
| | cosine_accuracy@10 | 0.8643 | |
| | cosine_precision@1 | 0.7743 | |
| | cosine_precision@3 | 0.6912 | |
| | cosine_precision@5 | 0.593 | |
| | cosine_precision@10 | 0.4354 | |
| | cosine_recall@1 | 0.1647 | |
| | cosine_recall@3 | 0.4097 | |
| | cosine_recall@5 | 0.5202 | |
| | cosine_recall@10 | 0.646 | |
| | **cosine_ndcg@10** | **0.7292** | |
| | cosine_mrr@10 | 0.8052 | |
| | cosine_map@10 | 0.6771 | |
| |
| <!-- |
| ## Bias, Risks and Limitations |
| |
| *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* |
| --> |
| |
| <!-- |
| ### Recommendations |
| |
| *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* |
| --> |
| |
| ## Training Details |
| |
| ### Training Dataset |
| |
| #### Unnamed Dataset |
| |
| * Size: 8,089,600 training samples |
| * Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code> |
| * Approximate statistics based on the first 100 samples: |
| | | anchor | positive | negative | |
| |:---------|:---------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:--------------------------------------------------------------------------------| |
| | type | string | string | string | |
| | modality | text | text | text | |
| | details | <ul><li>min: 4 tokens</li><li>mean: 8.74 tokens</li><li>max: 24 tokens</li></ul> | <ul><li>min: 4 tokens</li><li>mean: 8.48 tokens</li><li>max: 23 tokens</li></ul> | <ul><li>min: 4 tokens</li><li>mean: 9.3 tokens</li><li>max: 26 tokens</li></ul> | |
| * Samples: |
| | anchor | positive | negative | |
| |:---------------------------------------------------------------------|:---------------------------------------------------------------------------------|:--------------------------------------------------------------------------| |
| | <code>chronic rheumatic heart disease, unspecified</code> | <code>rheumatic pancarditis</code> | <code>de morgan's spots</code> | |
| | <code>chronic rheumatic left ventricular failure</code> | <code>inactive or quiescent rheumatic fever with congestive heart failure</code> | <code>conjunctival xerosis with bitot's spots</code> | |
| | <code>inactive or quiescent rheumatic fever with endocarditis</code> | <code>inactive rheumatic valvar endocarditis</code> | <code>mature b-cell neoplasm with leukaemic behaviour, unspecified</code> | |
| * Loss: [<code>MultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with these parameters: |
| ```json |
| { |
| "scale": 20.0, |
| "similarity_fct": "cos_sim", |
| "gather_across_devices": false, |
| "directions": [ |
| "query_to_doc" |
| ], |
| "partition_mode": "joint", |
| "hardness_mode": null, |
| "hardness_strength": 0.0 |
| } |
| ``` |
| |
| ### Training Hyperparameters |
| #### Non-Default Hyperparameters |
|
|
| - `per_device_train_batch_size`: 128 |
| - `per_device_eval_batch_size`: 256 |
| - `learning_rate`: 2e-05 |
| - `weight_decay`: 0.01 |
| - `num_train_epochs`: 1 |
| - `lr_scheduler_type`: cosine |
| - `warmup_steps`: 6320 |
| - `bf16`: True |
| - `dataloader_drop_last`: True |
| - `load_best_model_at_end`: True |
| - `eval_on_start`: True |
|
|
| #### All Hyperparameters |
| <details><summary>Click to expand</summary> |
|
|
| - `overwrite_output_dir`: False |
| - `do_predict`: False |
| - `prediction_loss_only`: True |
| - `per_device_train_batch_size`: 128 |
| - `per_device_eval_batch_size`: 256 |
| - `per_gpu_train_batch_size`: None |
| - `per_gpu_eval_batch_size`: None |
| - `gradient_accumulation_steps`: 1 |
| - `eval_accumulation_steps`: None |
| - `torch_empty_cache_steps`: None |
| - `learning_rate`: 2e-05 |
| - `weight_decay`: 0.01 |
| - `adam_beta1`: 0.9 |
| - `adam_beta2`: 0.999 |
| - `adam_epsilon`: 1e-08 |
| - `max_grad_norm`: 1.0 |
| - `num_train_epochs`: 1 |
| - `max_steps`: -1 |
| - `lr_scheduler_type`: cosine |
| - `lr_scheduler_kwargs`: {} |
| - `warmup_ratio`: 0.0 |
| - `warmup_steps`: 6320 |
| - `log_level`: passive |
| - `log_level_replica`: warning |
| - `log_on_each_node`: True |
| - `logging_nan_inf_filter`: True |
| - `save_safetensors`: True |
| - `save_on_each_node`: False |
| - `save_only_model`: False |
| - `restore_callback_states_from_checkpoint`: False |
| - `no_cuda`: False |
| - `use_cpu`: False |
| - `use_mps_device`: False |
| - `seed`: 42 |
| - `data_seed`: None |
| - `jit_mode_eval`: False |
| - `bf16`: True |
| - `fp16`: False |
| - `fp16_opt_level`: O1 |
| - `half_precision_backend`: auto |
| - `bf16_full_eval`: False |
| - `fp16_full_eval`: False |
| - `tf32`: None |
| - `local_rank`: 0 |
| - `ddp_backend`: None |
| - `tpu_num_cores`: None |
| - `tpu_metrics_debug`: False |
| - `debug`: [] |
| - `dataloader_drop_last`: True |
| - `dataloader_num_workers`: 0 |
| - `dataloader_prefetch_factor`: None |
| - `past_index`: -1 |
| - `disable_tqdm`: False |
| - `remove_unused_columns`: True |
| - `label_names`: None |
| - `load_best_model_at_end`: True |
| - `ignore_data_skip`: False |
| - `fsdp`: [] |
| - `fsdp_min_num_params`: 0 |
| - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} |
| - `fsdp_transformer_layer_cls_to_wrap`: None |
| - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} |
| - `parallelism_config`: None |
| - `deepspeed`: None |
| - `label_smoothing_factor`: 0.0 |
| - `optim`: adamw_torch_fused |
| - `optim_args`: None |
| - `adafactor`: False |
| - `group_by_length`: False |
| - `length_column_name`: length |
| - `project`: huggingface |
| - `trackio_space_id`: trackio |
| - `ddp_find_unused_parameters`: None |
| - `ddp_bucket_cap_mb`: None |
| - `ddp_broadcast_buffers`: False |
| - `dataloader_pin_memory`: True |
| - `dataloader_persistent_workers`: False |
| - `skip_memory_metrics`: True |
| - `use_legacy_prediction_loop`: False |
| - `push_to_hub`: False |
| - `resume_from_checkpoint`: None |
| - `hub_model_id`: None |
| - `hub_strategy`: every_save |
| - `hub_private_repo`: None |
| - `hub_always_push`: False |
| - `hub_revision`: None |
| - `gradient_checkpointing`: False |
| - `gradient_checkpointing_kwargs`: None |
| - `include_inputs_for_metrics`: False |
| - `include_for_metrics`: [] |
| - `eval_do_concat_batches`: True |
| - `fp16_backend`: auto |
| - `push_to_hub_model_id`: None |
| - `push_to_hub_organization`: None |
| - `mp_parameters`: |
| - `auto_find_batch_size`: False |
| - `full_determinism`: False |
| - `torchdynamo`: None |
| - `ray_scope`: last |
| - `ddp_timeout`: 1800 |
| - `torch_compile`: False |
| - `torch_compile_backend`: None |
| - `torch_compile_mode`: None |
| - `include_tokens_per_second`: False |
| - `include_num_input_tokens_seen`: no |
| - `neftune_noise_alpha`: None |
| - `optim_target_modules`: None |
| - `batch_eval_metrics`: False |
| - `eval_on_start`: True |
| - `use_liger_kernel`: False |
| - `liger_kernel_config`: None |
| - `eval_use_gather_object`: False |
| - `average_tokens_across_devices`: True |
| - `prompts`: None |
| - `batch_sampler`: batch_sampler |
| - `multi_dataset_batch_sampler`: proportional |
| - `router_mapping`: {} |
| - `learning_rate_mapping`: {} |
|
|
| </details> |
|
|
| ### Training Logs |
| <details><summary>Click to expand</summary> |
|
|
| | Epoch | Step | Training Loss | icd11-val_cosine_ndcg@10 | |
| |:----------:|:---------:|:-------------:|:------------------------:| |
| | 0 | 0 | - | 0.5062 | |
| | 0.0008 | 50 | 1.6688 | - | |
| | 0.0016 | 100 | 1.6505 | - | |
| | 0.0024 | 150 | 1.5416 | - | |
| | 0.0032 | 200 | 1.4195 | - | |
| | 0.0040 | 250 | 1.3759 | - | |
| | 0.0047 | 300 | 1.2595 | - | |
| | 0.0055 | 350 | 1.2113 | - | |
| | 0.0063 | 400 | 1.1349 | - | |
| | 0.0071 | 450 | 1.1253 | - | |
| | 0.0079 | 500 | 1.0463 | 0.5645 | |
| | 0.0087 | 550 | 0.9832 | - | |
| | 0.0095 | 600 | 0.93 | - | |
| | 0.0103 | 650 | 0.8592 | - | |
| | 0.0111 | 700 | 0.8343 | - | |
| | 0.0119 | 750 | 0.8298 | - | |
| | 0.0127 | 800 | 0.7648 | - | |
| | 0.0134 | 850 | 0.724 | - | |
| | 0.0142 | 900 | 0.7301 | - | |
| | 0.0150 | 950 | 0.6552 | - | |
| | 0.0158 | 1000 | 0.6649 | 0.5913 | |
| | 0.0166 | 1050 | 0.6055 | - | |
| | 0.0174 | 1100 | 0.6287 | - | |
| | 0.0182 | 1150 | 0.5632 | - | |
| | 0.0190 | 1200 | 0.5289 | - | |
| | 0.0198 | 1250 | 0.5299 | - | |
| | 0.0206 | 1300 | 0.5321 | - | |
| | 0.0214 | 1350 | 0.5087 | - | |
| | 0.0222 | 1400 | 0.4714 | - | |
| | 0.0229 | 1450 | 0.4598 | - | |
| | 0.0237 | 1500 | 0.4614 | 0.6138 | |
| | 0.0245 | 1550 | 0.45 | - | |
| | 0.0253 | 1600 | 0.4241 | - | |
| | 0.0261 | 1650 | 0.4047 | - | |
| | 0.0269 | 1700 | 0.4162 | - | |
| | 0.0277 | 1750 | 0.3737 | - | |
| | 0.0285 | 1800 | 0.4114 | - | |
| | 0.0293 | 1850 | 0.376 | - | |
| | 0.0301 | 1900 | 0.3791 | - | |
| | 0.0309 | 1950 | 0.3791 | - | |
| | 0.0316 | 2000 | 0.3643 | 0.6310 | |
| | 0.0324 | 2050 | 0.3543 | - | |
| | 0.0332 | 2100 | 0.3024 | - | |
| | 0.0340 | 2150 | 0.3294 | - | |
| | 0.0348 | 2200 | 0.3017 | - | |
| | 0.0356 | 2250 | 0.3334 | - | |
| | 0.0364 | 2300 | 0.3158 | - | |
| | 0.0372 | 2350 | 0.3252 | - | |
| | 0.0380 | 2400 | 0.2956 | - | |
| | 0.0388 | 2450 | 0.2773 | - | |
| | 0.0396 | 2500 | 0.2994 | 0.6399 | |
| | 0.0403 | 2550 | 0.2725 | - | |
| | 0.0411 | 2600 | 0.2531 | - | |
| | 0.0419 | 2650 | 0.2924 | - | |
| | 0.0427 | 2700 | 0.286 | - | |
| | 0.0435 | 2750 | 0.2653 | - | |
| | 0.0443 | 2800 | 0.2588 | - | |
| | 0.0451 | 2850 | 0.2428 | - | |
| | 0.0459 | 2900 | 0.243 | - | |
| | 0.0467 | 2950 | 0.2426 | - | |
| | 0.0475 | 3000 | 0.2524 | 0.6577 | |
| | 0.0483 | 3050 | 0.2447 | - | |
| | 0.0491 | 3100 | 0.2474 | - | |
| | 0.0498 | 3150 | 0.2565 | - | |
| | 0.0506 | 3200 | 0.2397 | - | |
| | 0.0514 | 3250 | 0.2345 | - | |
| | 0.0522 | 3300 | 0.2229 | - | |
| | 0.0530 | 3350 | 0.2418 | - | |
| | 0.0538 | 3400 | 0.2271 | - | |
| | 0.0546 | 3450 | 0.2189 | - | |
| | 0.0554 | 3500 | 0.2163 | 0.6591 | |
| | 0.0562 | 3550 | 0.2052 | - | |
| | 0.0570 | 3600 | 0.217 | - | |
| | 0.0578 | 3650 | 0.2038 | - | |
| | 0.0585 | 3700 | 0.2268 | - | |
| | 0.0593 | 3750 | 0.2035 | - | |
| | 0.0601 | 3800 | 0.2077 | - | |
| | 0.0609 | 3850 | 0.2055 | - | |
| | 0.0617 | 3900 | 0.2207 | - | |
| | 0.0625 | 3950 | 0.2004 | - | |
| | 0.0633 | 4000 | 0.2053 | 0.6627 | |
| | 0.0641 | 4050 | 0.2002 | - | |
| | 0.0649 | 4100 | 0.1938 | - | |
| | 0.0657 | 4150 | 0.2213 | - | |
| | 0.0665 | 4200 | 0.2146 | - | |
| | 0.0672 | 4250 | 0.1949 | - | |
| | 0.0680 | 4300 | 0.1944 | - | |
| | 0.0688 | 4350 | 0.1777 | - | |
| | 0.0696 | 4400 | 0.196 | - | |
| | 0.0704 | 4450 | 0.1927 | - | |
| | 0.0712 | 4500 | 0.1805 | 0.6745 | |
| | 0.0720 | 4550 | 0.1934 | - | |
| | 0.0728 | 4600 | 0.1892 | - | |
| | 0.0736 | 4650 | 0.2022 | - | |
| | 0.0744 | 4700 | 0.1892 | - | |
| | 0.0752 | 4750 | 0.1812 | - | |
| | 0.0759 | 4800 | 0.1878 | - | |
| | 0.0767 | 4850 | 0.1815 | - | |
| | 0.0775 | 4900 | 0.1932 | - | |
| | 0.0783 | 4950 | 0.1901 | - | |
| | 0.0791 | 5000 | 0.1753 | 0.6772 | |
| | 0.0799 | 5050 | 0.186 | - | |
| | 0.0807 | 5100 | 0.1768 | - | |
| | 0.0815 | 5150 | 0.1588 | - | |
| | 0.0823 | 5200 | 0.1788 | - | |
| | 0.0831 | 5250 | 0.1853 | - | |
| | 0.0839 | 5300 | 0.1796 | - | |
| | 0.0847 | 5350 | 0.1703 | - | |
| | 0.0854 | 5400 | 0.1905 | - | |
| | 0.0862 | 5450 | 0.1818 | - | |
| | 0.0870 | 5500 | 0.188 | 0.6699 | |
| | 0.0878 | 5550 | 0.1665 | - | |
| | 0.0886 | 5600 | 0.1639 | - | |
| | 0.0894 | 5650 | 0.1664 | - | |
| | 0.0902 | 5700 | 0.1757 | - | |
| | 0.0910 | 5750 | 0.1688 | - | |
| | 0.0918 | 5800 | 0.1487 | - | |
| | 0.0926 | 5850 | 0.1717 | - | |
| | 0.0934 | 5900 | 0.167 | - | |
| | 0.0941 | 5950 | 0.1595 | - | |
| | 0.0949 | 6000 | 0.1696 | 0.6811 | |
| | 0.0957 | 6050 | 0.1676 | - | |
| | 0.0965 | 6100 | 0.1746 | - | |
| | 0.0973 | 6150 | 0.1483 | - | |
| | 0.0981 | 6200 | 0.1636 | - | |
| | 0.0989 | 6250 | 0.1517 | - | |
| | 0.0997 | 6300 | 0.165 | - | |
| | 0.1005 | 6350 | 0.1634 | - | |
| | 0.1013 | 6400 | 0.1408 | - | |
| | 0.1021 | 6450 | 0.1623 | - | |
| | 0.1028 | 6500 | 0.1656 | 0.6789 | |
| | 0.1036 | 6550 | 0.1609 | - | |
| | 0.1044 | 6600 | 0.1612 | - | |
| | 0.1052 | 6650 | 0.16 | - | |
| | 0.1060 | 6700 | 0.1657 | - | |
| | 0.1068 | 6750 | 0.1427 | - | |
| | 0.1076 | 6800 | 0.1355 | - | |
| | 0.1084 | 6850 | 0.1401 | - | |
| | 0.1092 | 6900 | 0.1409 | - | |
| | 0.1100 | 6950 | 0.1403 | - | |
| | 0.1108 | 7000 | 0.138 | 0.6926 | |
| | 0.1116 | 7050 | 0.1369 | - | |
| | 0.1123 | 7100 | 0.1295 | - | |
| | 0.1131 | 7150 | 0.1407 | - | |
| | 0.1139 | 7200 | 0.1388 | - | |
| | 0.1147 | 7250 | 0.1377 | - | |
| | 0.1155 | 7300 | 0.1324 | - | |
| | 0.1163 | 7350 | 0.128 | - | |
| | 0.1171 | 7400 | 0.1256 | - | |
| | 0.1179 | 7450 | 0.126 | - | |
| | 0.1187 | 7500 | 0.1284 | 0.6911 | |
| | 0.1195 | 7550 | 0.1246 | - | |
| | 0.1203 | 7600 | 0.1119 | - | |
| | 0.1210 | 7650 | 0.1082 | - | |
| | 0.1218 | 7700 | 0.1305 | - | |
| | 0.1226 | 7750 | 0.1033 | - | |
| | 0.1234 | 7800 | 0.1125 | - | |
| | 0.1242 | 7850 | 0.1256 | - | |
| | 0.125 | 7900 | 0.1403 | - | |
| | 0.1258 | 7950 | 0.1334 | - | |
| | 0.1266 | 8000 | 0.1173 | 0.6939 | |
| | 0.1274 | 8050 | 0.106 | - | |
| | 0.1282 | 8100 | 0.1127 | - | |
| | 0.1290 | 8150 | 0.1185 | - | |
| | 0.1297 | 8200 | 0.1045 | - | |
| | 0.1305 | 8250 | 0.0978 | - | |
| | 0.1313 | 8300 | 0.1074 | - | |
| | 0.1321 | 8350 | 0.1114 | - | |
| | 0.1329 | 8400 | 0.1127 | - | |
| | 0.1337 | 8450 | 0.1035 | - | |
| | 0.1345 | 8500 | 0.0984 | 0.7023 | |
| | 0.1353 | 8550 | 0.1065 | - | |
| | 0.1361 | 8600 | 0.1009 | - | |
| | 0.1369 | 8650 | 0.0837 | - | |
| | 0.1377 | 8700 | 0.0932 | - | |
| | 0.1384 | 8750 | 0.1096 | - | |
| | 0.1392 | 8800 | 0.1065 | - | |
| | 0.1400 | 8850 | 0.1029 | - | |
| | 0.1408 | 8900 | 0.1123 | - | |
| | 0.1416 | 8950 | 0.1043 | - | |
| | 0.1424 | 9000 | 0.0947 | 0.6998 | |
| | 0.1432 | 9050 | 0.0981 | - | |
| | 0.1440 | 9100 | 0.0917 | - | |
| | 0.1448 | 9150 | 0.0962 | - | |
| | 0.1456 | 9200 | 0.0902 | - | |
| | 0.1464 | 9250 | 0.1001 | - | |
| | 0.1472 | 9300 | 0.0852 | - | |
| | 0.1479 | 9350 | 0.0724 | - | |
| | 0.1487 | 9400 | 0.0892 | - | |
| | 0.1495 | 9450 | 0.0774 | - | |
| | 0.1503 | 9500 | 0.0819 | 0.7025 | |
| | 0.1511 | 9550 | 0.0831 | - | |
| | 0.1519 | 9600 | 0.0807 | - | |
| | 0.1527 | 9650 | 0.0833 | - | |
| | 0.1535 | 9700 | 0.0823 | - | |
| | 0.1543 | 9750 | 0.0754 | - | |
| | 0.1551 | 9800 | 0.0759 | - | |
| | 0.1559 | 9850 | 0.0856 | - | |
| | 0.1566 | 9900 | 0.0787 | - | |
| | 0.1574 | 9950 | 0.0863 | - | |
| | 0.1582 | 10000 | 0.0821 | 0.6993 | |
| | 0.1590 | 10050 | 0.0761 | - | |
| | 0.1598 | 10100 | 0.076 | - | |
| | 0.1606 | 10150 | 0.0742 | - | |
| | 0.1614 | 10200 | 0.0684 | - | |
| | 0.1622 | 10250 | 0.0754 | - | |
| | 0.1630 | 10300 | 0.0729 | - | |
| | 0.1638 | 10350 | 0.0852 | - | |
| | 0.1646 | 10400 | 0.0778 | - | |
| | 0.1653 | 10450 | 0.078 | - | |
| | 0.1661 | 10500 | 0.0652 | 0.6972 | |
| | 0.1669 | 10550 | 0.0766 | - | |
| | 0.1677 | 10600 | 0.0805 | - | |
| | 0.1685 | 10650 | 0.0736 | - | |
| | 0.1693 | 10700 | 0.0702 | - | |
| | 0.1701 | 10750 | 0.0789 | - | |
| | 0.1709 | 10800 | 0.0683 | - | |
| | 0.1717 | 10850 | 0.0788 | - | |
| | 0.1725 | 10900 | 0.0592 | - | |
| | 0.1733 | 10950 | 0.0716 | - | |
| | 0.1741 | 11000 | 0.0647 | 0.7031 | |
| | 0.1748 | 11050 | 0.0624 | - | |
| | 0.1756 | 11100 | 0.0629 | - | |
| | 0.1764 | 11150 | 0.0653 | - | |
| | 0.1772 | 11200 | 0.0699 | - | |
| | 0.1780 | 11250 | 0.0661 | - | |
| | 0.1788 | 11300 | 0.0534 | - | |
| | 0.1796 | 11350 | 0.0559 | - | |
| | 0.1804 | 11400 | 0.0656 | - | |
| | 0.1812 | 11450 | 0.0689 | - | |
| | 0.1820 | 11500 | 0.0642 | 0.7067 | |
| | 0.1828 | 11550 | 0.0586 | - | |
| | 0.1835 | 11600 | 0.0616 | - | |
| | 0.1843 | 11650 | 0.0582 | - | |
| | 0.1851 | 11700 | 0.0668 | - | |
| | 0.1859 | 11750 | 0.0577 | - | |
| | 0.1867 | 11800 | 0.0614 | - | |
| | 0.1875 | 11850 | 0.0598 | - | |
| | 0.1883 | 11900 | 0.0488 | - | |
| | 0.1891 | 11950 | 0.0721 | - | |
| | 0.1899 | 12000 | 0.0623 | 0.7102 | |
| | 0.1907 | 12050 | 0.0542 | - | |
| | 0.1915 | 12100 | 0.0648 | - | |
| | 0.1922 | 12150 | 0.0589 | - | |
| | 0.1930 | 12200 | 0.0538 | - | |
| | 0.1938 | 12250 | 0.0511 | - | |
| | 0.1946 | 12300 | 0.0624 | - | |
| | 0.1954 | 12350 | 0.0586 | - | |
| | 0.1962 | 12400 | 0.052 | - | |
| | 0.1970 | 12450 | 0.0629 | - | |
| | 0.1978 | 12500 | 0.0557 | 0.7136 | |
| | 0.1986 | 12550 | 0.052 | - | |
| | 0.1994 | 12600 | 0.0597 | - | |
| | 0.2002 | 12650 | 0.0524 | - | |
| | 0.2009 | 12700 | 0.0465 | - | |
| | 0.2017 | 12750 | 0.0529 | - | |
| | 0.2025 | 12800 | 0.0591 | - | |
| | 0.2033 | 12850 | 0.0527 | - | |
| | 0.2041 | 12900 | 0.0553 | - | |
| | 0.2049 | 12950 | 0.0518 | - | |
| | 0.2057 | 13000 | 0.0561 | 0.7007 | |
| | 0.2065 | 13050 | 0.0559 | - | |
| | 0.2073 | 13100 | 0.0569 | - | |
| | 0.2081 | 13150 | 0.0522 | - | |
| | 0.2089 | 13200 | 0.0586 | - | |
| | 0.2097 | 13250 | 0.0482 | - | |
| | 0.2104 | 13300 | 0.0496 | - | |
| | 0.2112 | 13350 | 0.0512 | - | |
| | 0.2120 | 13400 | 0.0468 | - | |
| | 0.2128 | 13450 | 0.0448 | - | |
| | 0.2136 | 13500 | 0.0566 | 0.7088 | |
| | 0.2144 | 13550 | 0.0541 | - | |
| | 0.2152 | 13600 | 0.048 | - | |
| | 0.2160 | 13650 | 0.0455 | - | |
| | 0.2168 | 13700 | 0.0484 | - | |
| | 0.2176 | 13750 | 0.0512 | - | |
| | 0.2184 | 13800 | 0.0414 | - | |
| | 0.2191 | 13850 | 0.0467 | - | |
| | 0.2199 | 13900 | 0.0413 | - | |
| | 0.2207 | 13950 | 0.0483 | - | |
| | 0.2215 | 14000 | 0.0531 | 0.7072 | |
| | 0.2223 | 14050 | 0.0497 | - | |
| | 0.2231 | 14100 | 0.0507 | - | |
| | 0.2239 | 14150 | 0.0417 | - | |
| | 0.2247 | 14200 | 0.0479 | - | |
| | 0.2255 | 14250 | 0.0515 | - | |
| | 0.2263 | 14300 | 0.0447 | - | |
| | 0.2271 | 14350 | 0.0443 | - | |
| | 0.2278 | 14400 | 0.0416 | - | |
| | 0.2286 | 14450 | 0.0471 | - | |
| | 0.2294 | 14500 | 0.039 | 0.7108 | |
| | 0.2302 | 14550 | 0.0535 | - | |
| | 0.2310 | 14600 | 0.0404 | - | |
| | 0.2318 | 14650 | 0.037 | - | |
| | 0.2326 | 14700 | 0.0379 | - | |
| | 0.2334 | 14750 | 0.048 | - | |
| | 0.2342 | 14800 | 0.0417 | - | |
| | 0.2350 | 14850 | 0.0405 | - | |
| | 0.2358 | 14900 | 0.0408 | - | |
| | 0.2366 | 14950 | 0.0388 | - | |
| | 0.2373 | 15000 | 0.0376 | 0.7160 | |
| | 0.2381 | 15050 | 0.0443 | - | |
| | 0.2389 | 15100 | 0.0436 | - | |
| | 0.2397 | 15150 | 0.0374 | - | |
| | 0.2405 | 15200 | 0.0378 | - | |
| | 0.2413 | 15250 | 0.0395 | - | |
| | 0.2421 | 15300 | 0.0437 | - | |
| | 0.2429 | 15350 | 0.0406 | - | |
| | 0.2437 | 15400 | 0.0396 | - | |
| | 0.2445 | 15450 | 0.042 | - | |
| | 0.2453 | 15500 | 0.0395 | 0.7122 | |
| | 0.2460 | 15550 | 0.0371 | - | |
| | 0.2468 | 15600 | 0.037 | - | |
| | 0.2476 | 15650 | 0.0458 | - | |
| | 0.2484 | 15700 | 0.0377 | - | |
| | 0.2492 | 15750 | 0.0367 | - | |
| | 0.25 | 15800 | 0.0459 | - | |
| | 0.2508 | 15850 | 0.0354 | - | |
| | 0.2516 | 15900 | 0.0338 | - | |
| | 0.2524 | 15950 | 0.0363 | - | |
| | 0.2532 | 16000 | 0.0334 | 0.7179 | |
| | 0.2540 | 16050 | 0.0371 | - | |
| | 0.2547 | 16100 | 0.032 | - | |
| | 0.2555 | 16150 | 0.0352 | - | |
| | 0.2563 | 16200 | 0.032 | - | |
| | 0.2571 | 16250 | 0.0348 | - | |
| | 0.2579 | 16300 | 0.0357 | - | |
| | 0.2587 | 16350 | 0.0364 | - | |
| | 0.2595 | 16400 | 0.0322 | - | |
| | 0.2603 | 16450 | 0.0316 | - | |
| | 0.2611 | 16500 | 0.034 | 0.7199 | |
| | 0.2619 | 16550 | 0.0357 | - | |
| | 0.2627 | 16600 | 0.0287 | - | |
| | 0.2634 | 16650 | 0.0374 | - | |
| | 0.2642 | 16700 | 0.0389 | - | |
| | 0.2650 | 16750 | 0.0357 | - | |
| | 0.2658 | 16800 | 0.0316 | - | |
| | 0.2666 | 16850 | 0.0298 | - | |
| | 0.2674 | 16900 | 0.0332 | - | |
| | 0.2682 | 16950 | 0.0342 | - | |
| | 0.2690 | 17000 | 0.0311 | 0.7148 | |
| | 0.2698 | 17050 | 0.0323 | - | |
| | 0.2706 | 17100 | 0.0309 | - | |
| | 0.2714 | 17150 | 0.0292 | - | |
| | 0.2722 | 17200 | 0.0405 | - | |
| | 0.2729 | 17250 | 0.0269 | - | |
| | 0.2737 | 17300 | 0.0257 | - | |
| | 0.2745 | 17350 | 0.0378 | - | |
| | 0.2753 | 17400 | 0.0279 | - | |
| | 0.2761 | 17450 | 0.0353 | - | |
| | 0.2769 | 17500 | 0.0296 | 0.7244 | |
| | 0.2777 | 17550 | 0.0326 | - | |
| | 0.2785 | 17600 | 0.0259 | - | |
| | 0.2793 | 17650 | 0.026 | - | |
| | 0.2801 | 17700 | 0.0329 | - | |
| | 0.2809 | 17750 | 0.036 | - | |
| | 0.2816 | 17800 | 0.0317 | - | |
| | 0.2824 | 17850 | 0.0273 | - | |
| | 0.2832 | 17900 | 0.0302 | - | |
| | 0.2840 | 17950 | 0.0271 | - | |
| | 0.2848 | 18000 | 0.027 | 0.7211 | |
| | 0.2856 | 18050 | 0.0301 | - | |
| | 0.2864 | 18100 | 0.0272 | - | |
| | 0.2872 | 18150 | 0.0376 | - | |
| | 0.2880 | 18200 | 0.0311 | - | |
| | 0.2888 | 18250 | 0.026 | - | |
| | 0.2896 | 18300 | 0.0322 | - | |
| | 0.2903 | 18350 | 0.0238 | - | |
| | 0.2911 | 18400 | 0.0283 | - | |
| | 0.2919 | 18450 | 0.0238 | - | |
| | 0.2927 | 18500 | 0.025 | 0.7288 | |
| | 0.2935 | 18550 | 0.0354 | - | |
| | 0.2943 | 18600 | 0.0337 | - | |
| | 0.2951 | 18650 | 0.032 | - | |
| | 0.2959 | 18700 | 0.026 | - | |
| | 0.2967 | 18750 | 0.0242 | - | |
| | 0.2975 | 18800 | 0.0201 | - | |
| | 0.2983 | 18850 | 0.0307 | - | |
| | 0.2991 | 18900 | 0.0213 | - | |
| | 0.2998 | 18950 | 0.0245 | - | |
| | 0.3006 | 19000 | 0.0257 | 0.7198 | |
| | 0.3014 | 19050 | 0.024 | - | |
| | 0.3022 | 19100 | 0.0298 | - | |
| | 0.3030 | 19150 | 0.0293 | - | |
| | 0.3038 | 19200 | 0.0263 | - | |
| | 0.3046 | 19250 | 0.0264 | - | |
| | 0.3054 | 19300 | 0.0252 | - | |
| | 0.3062 | 19350 | 0.0313 | - | |
| | 0.3070 | 19400 | 0.0197 | - | |
| | 0.3078 | 19450 | 0.0253 | - | |
| | **0.3085** | **19500** | **0.0263** | **0.73** | |
| | 0.3093 | 19550 | 0.0273 | - | |
| | 0.3101 | 19600 | 0.0277 | - | |
| | 0.3109 | 19650 | 0.0213 | - | |
| | 0.3117 | 19700 | 0.0226 | - | |
| | 0.3125 | 19750 | 0.0228 | - | |
| | 0.3133 | 19800 | 0.0249 | - | |
| | 0.3141 | 19850 | 0.0239 | - | |
| | 0.3149 | 19900 | 0.0219 | - | |
| | 0.3157 | 19950 | 0.0234 | - | |
| | 0.3165 | 20000 | 0.0244 | 0.7238 | |
| | 0.3172 | 20050 | 0.0211 | - | |
| | 0.3180 | 20100 | 0.0253 | - | |
| | 0.3188 | 20150 | 0.0187 | - | |
| | 0.3196 | 20200 | 0.0241 | - | |
| | 0.3204 | 20250 | 0.0267 | - | |
| | 0.3212 | 20300 | 0.0363 | - | |
| | 0.3220 | 20350 | 0.023 | - | |
| | 0.3228 | 20400 | 0.0274 | - | |
| | 0.3236 | 20450 | 0.022 | - | |
| | 0.3244 | 20500 | 0.0223 | 0.7294 | |
| | 0.3252 | 20550 | 0.0241 | - | |
| | 0.3259 | 20600 | 0.0201 | - | |
| | 0.3267 | 20650 | 0.0281 | - | |
| | 0.3275 | 20700 | 0.0272 | - | |
| | 0.3283 | 20750 | 0.0225 | - | |
| | 0.3291 | 20800 | 0.0237 | - | |
| | 0.3299 | 20850 | 0.025 | - | |
| | 0.3307 | 20900 | 0.0312 | - | |
| | 0.3315 | 20950 | 0.0232 | - | |
| | 0.3323 | 21000 | 0.0195 | 0.7231 | |
| | 0.3331 | 21050 | 0.0214 | - | |
| | 0.3339 | 21100 | 0.0165 | - | |
| | 0.3347 | 21150 | 0.0236 | - | |
| | 0.3354 | 21200 | 0.0294 | - | |
| | 0.3362 | 21250 | 0.0274 | - | |
| | 0.3370 | 21300 | 0.021 | - | |
| | 0.3378 | 21350 | 0.0248 | - | |
| | 0.3386 | 21400 | 0.0189 | - | |
| | 0.3394 | 21450 | 0.0202 | - | |
| | 0.3402 | 21500 | 0.0197 | 0.7236 | |
| | 0.3410 | 21550 | 0.0252 | - | |
| | 0.3418 | 21600 | 0.0173 | - | |
| | 0.3426 | 21650 | 0.0177 | - | |
| | 0.3434 | 21700 | 0.0207 | - | |
| | 0.3441 | 21750 | 0.0153 | - | |
| | 0.3449 | 21800 | 0.022 | - | |
| | 0.3457 | 21850 | 0.0159 | - | |
| | 0.3465 | 21900 | 0.017 | - | |
| | 0.3473 | 21950 | 0.0246 | - | |
| | 0.3481 | 22000 | 0.0173 | 0.7292 | |
|
|
| * The bold row denotes the saved checkpoint. |
| </details> |
|
|
| ### Training Time |
| - **Training**: 1.7 hours |
| - **Evaluation**: 8.4 minutes |
| - **Total**: 1.9 hours |
|
|
| ### Framework Versions |
| - Python: 3.14.2 |
| - Sentence Transformers: 6.1.0 |
| - Transformers: 4.57.3 |
| - PyTorch: 2.9.1a0 |
| - Accelerate: 1.15.0 |
| - Datasets: 4.4.1 |
| - Tokenizers: 0.22.1 |
|
|
| ## Additional Resources |
|
|
| - [Training and Finetuning Embedding Models with Sentence Transformers](https://huggingface.co/blog/train-sentence-transformers): the end-to-end guide for training or finetuning Sentence Transformer models. |
| - [Introduction to Matryoshka Embedding Models](https://huggingface.co/blog/matryoshka): variable-size embeddings that can be truncated with minimal quality loss. |
| - [Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval](https://huggingface.co/blog/embedding-quantization): post-training compression of embedding vectors. |
| - [Multimodal Embedding & Reranker Models with Sentence Transformers](https://huggingface.co/blog/multimodal-sentence-transformers): use text, image, audio, and video models through the same API. |
| - [Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers](https://huggingface.co/blog/train-multimodal-sentence-transformers): train multimodal embedding models, with a Visual Document Retrieval walkthrough. |
|
|
| ## Citation |
|
|
| ### BibTeX |
|
|
| #### Sentence Transformers |
| ```bibtex |
| @inproceedings{reimers-2019-sentence-bert, |
| title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", |
| author = "Reimers, Nils and Gurevych, Iryna", |
| booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", |
| month = "11", |
| year = "2019", |
| publisher = "Association for Computational Linguistics", |
| url = "https://arxiv.org/abs/1908.10084", |
| } |
| ``` |
|
|
| #### MultipleNegativesRankingLoss |
| ```bibtex |
| @misc{oord2019representationlearningcontrastivepredictive, |
| title={Representation Learning with Contrastive Predictive Coding}, |
| author={Aaron van den Oord and Yazhe Li and Oriol Vinyals}, |
| year={2019}, |
| eprint={1807.03748}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.LG}, |
| url={https://arxiv.org/abs/1807.03748}, |
| } |
| ``` |
|
|
| <!-- |
| ## Glossary |
|
|
| *Clearly define terms in order to be accessible across audiences.* |
| --> |
|
|
| <!-- |
| ## Model Card Authors |
|
|
| *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* |
| --> |
|
|
| <!-- |
| ## Model Card Contact |
|
|
| *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* |
| --> |
