| --- |
| tags: |
| - sentence-transformers |
| - sentence-similarity |
| - feature-extraction |
| - dense |
| - generated_from_trainer |
| - dataset_size:8089600 |
| - loss:MultipleNegativesRankingLoss |
| base_model: abhinand/MedEmbed-small-v0.1 |
| widget: |
| - source_sentence: hypoxic ischaemic encephalopathy in diseases classified elsewhere |
| sentences: |
| - other specified hypoxic-ischaemic encephalopathy |
| - monocytopenia |
| - isolated benign cerebral vasculitis |
| - source_sentence: congenital laryngomalacia |
| sentences: |
| - floppy epiglottis |
| - fructose-1,6-bisphosphatase deficiency |
| - structural developmental anomalies of the nose or cavum, unspecified |
| - source_sentence: molluscum contagiosum |
| sentences: |
| - laceration of bronchus nos |
| - water warts |
| - inflammation of bartholin gland |
| - source_sentence: contusion of heart without haemopericardium |
| sentences: |
| - contusion of heart |
| - phlebotomus fever |
| - rectovaginal fistula |
| - source_sentence: multiple delivery by combination of methods |
| sentences: |
| - rectal spasm |
| - multiple delivery by combination of methods without caesarean |
| - bipolar or related disorders, unspecified |
| pipeline_tag: sentence-similarity |
| library_name: sentence-transformers |
| metrics: |
| - cosine_accuracy@1 |
| - cosine_accuracy@5 |
| - cosine_accuracy@10 |
| - cosine_precision@1 |
| - cosine_precision@3 |
| - cosine_precision@5 |
| - cosine_precision@10 |
| - cosine_recall@1 |
| - cosine_recall@3 |
| - cosine_recall@5 |
| - cosine_recall@10 |
| - cosine_ndcg@10 |
| - cosine_mrr@10 |
| - cosine_map@10 |
| model-index: |
| - name: SentenceTransformer based on abhinand/MedEmbed-small-v0.1 |
| results: |
| - task: |
| type: information-retrieval |
| name: Information Retrieval |
| dataset: |
| name: icd11 val |
| type: icd11-val |
| metrics: |
| - type: cosine_accuracy@1 |
| value: 0.7649122807017544 |
| name: Cosine Accuracy@1 |
| - type: cosine_accuracy@5 |
| value: 0.8421052631578947 |
| name: Cosine Accuracy@5 |
| - type: cosine_accuracy@10 |
| value: 0.8690058479532163 |
| name: Cosine Accuracy@10 |
| - type: cosine_precision@1 |
| value: 0.7649122807017544 |
| name: Cosine Precision@1 |
| - type: cosine_precision@3 |
| value: 0.6775828460038987 |
| name: Cosine Precision@3 |
| - type: cosine_precision@5 |
| value: 0.5796491228070175 |
| name: Cosine Precision@5 |
| - type: cosine_precision@10 |
| value: 0.42701754385964913 |
| name: Cosine Precision@10 |
| - type: cosine_recall@1 |
| value: 0.16621564142709158 |
| name: Cosine Recall@1 |
| - type: cosine_recall@3 |
| value: 0.4102644606511703 |
| name: Cosine Recall@3 |
| - type: cosine_recall@5 |
| value: 0.5203490784736678 |
| name: Cosine Recall@5 |
| - type: cosine_recall@10 |
| value: 0.6468261230695532 |
| name: Cosine Recall@10 |
| - type: cosine_ndcg@10 |
| value: 0.724241553378169 |
| name: Cosine Ndcg@10 |
| - type: cosine_mrr@10 |
| value: 0.7982117330362947 |
| name: Cosine Mrr@10 |
| - type: cosine_map@10 |
| value: 0.6694373149953072 |
| name: Cosine Map@10 |
| --- |
| |
| # SentenceTransformer based on abhinand/MedEmbed-small-v0.1 |
|
|
| This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [abhinand/MedEmbed-small-v0.1](https://huggingface.co/abhinand/MedEmbed-small-v0.1). It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more. |
|
|
| ## Model Details |
|
|
| ### Model Description |
| - **Model Type:** Sentence Transformer |
| - **Base model:** [abhinand/MedEmbed-small-v0.1](https://huggingface.co/abhinand/MedEmbed-small-v0.1) <!-- at revision 40a5850d046cfdb56154e332b4d7099b63e8d50e --> |
| - **Maximum Sequence Length:** 30 tokens |
| - **Output Dimensionality:** 384 dimensions |
| - **Similarity Function:** Cosine Similarity |
| - **Supported Modality:** Text |
| <!-- - **Training Dataset:** Unknown --> |
| <!-- - **Language:** Unknown --> |
| <!-- - **License:** Unknown --> |
|
|
| ### Model Sources |
|
|
| - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) |
| - **Repository:** [Sentence Transformers on GitHub](https://github.com/huggingface/sentence-transformers) |
| - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) |
|
|
| ### Full Model Architecture |
|
|
| ``` |
| SentenceTransformer( |
| (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'}) |
| (1): Pooling({'embedding_dimension': 384, 'pooling_mode': 'cls', 'include_prompt': True}) |
| (2): Normalize({}) |
| ) |
| ``` |
|
|
| ## Usage |
|
|
| ### Direct Usage (Sentence Transformers) |
|
|
| First install the Sentence Transformers library: |
|
|
| ```bash |
| pip install -U sentence-transformers |
| ``` |
| Then you can load this model and run inference. |
| ```python |
| from sentence_transformers import SentenceTransformer |
| |
| # Download from the 🤗 Hub |
| model = SentenceTransformer("seanfarrell/icd11_ner_model_v2") |
| # Run inference |
| queries = [ |
| 'multiple delivery by combination of methods', |
| ] |
| documents = [ |
| 'multiple delivery by combination of methods without caesarean', |
| 'bipolar or related disorders, unspecified', |
| 'rectal spasm', |
| ] |
| query_embeddings = model.encode_query(queries) |
| document_embeddings = model.encode_document(documents) |
| print(query_embeddings.shape, document_embeddings.shape) |
| # [1, 384] [3, 384] |
| |
| # Get the similarity scores for the embeddings |
| similarities = model.similarity(query_embeddings, document_embeddings) |
| print(similarities) |
| # tensor([[ 0.9466, -0.0751, -0.0982]]) |
| ``` |
| <!-- |
| ### Direct Usage (Transformers) |
|
|
| <details><summary>Click to see the direct usage in Transformers</summary> |
|
|
| </details> |
| --> |
|
|
| <!-- |
| ### Downstream Usage (Sentence Transformers) |
|
|
| You can finetune this model on your own dataset. |
|
|
| <details><summary>Click to expand</summary> |
|
|
| </details> |
| --> |
|
|
| <!-- |
| ### Out-of-Scope Use |
|
|
| *List how the model may foreseeably be misused and address what users ought not to do with the model.* |
| --> |
|
|
| ## Evaluation |
|
|
| ### Metrics |
|
|
| #### Information Retrieval |
|
|
| * Dataset: `icd11-val` |
| * Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.sentence_transformer.evaluation.InformationRetrievalEvaluator) |
|
|
| | Metric | Value | |
| |:--------------------|:-----------| |
| | cosine_accuracy@1 | 0.7649 | |
| | cosine_accuracy@5 | 0.8421 | |
| | cosine_accuracy@10 | 0.869 | |
| | cosine_precision@1 | 0.7649 | |
| | cosine_precision@3 | 0.6776 | |
| | cosine_precision@5 | 0.5796 | |
| | cosine_precision@10 | 0.427 | |
| | cosine_recall@1 | 0.1662 | |
| | cosine_recall@3 | 0.4103 | |
| | cosine_recall@5 | 0.5203 | |
| | cosine_recall@10 | 0.6468 | |
| | **cosine_ndcg@10** | **0.7242** | |
| | cosine_mrr@10 | 0.7982 | |
| | cosine_map@10 | 0.6694 | |
| |
| <!-- |
| ## Bias, Risks and Limitations |
| |
| *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* |
| --> |
| |
| <!-- |
| ### Recommendations |
| |
| *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* |
| --> |
| |
| ## Training Details |
| |
| ### Training Dataset |
| |
| #### Unnamed Dataset |
| |
| * Size: 8,089,600 training samples |
| * Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code> |
| * Approximate statistics based on the first 100 samples: |
| | | anchor | positive | negative | |
| |:---------|:---------------------------------------------------------------------------------|:--------------------------------------------------------------------------------|:---------------------------------------------------------------------------------| |
| | type | string | string | string | |
| | modality | text | text | text | |
| | details | <ul><li>min: 4 tokens</li><li>mean: 9.52 tokens</li><li>max: 25 tokens</li></ul> | <ul><li>min: 3 tokens</li><li>mean: 9.1 tokens</li><li>max: 22 tokens</li></ul> | <ul><li>min: 3 tokens</li><li>mean: 9.37 tokens</li><li>max: 22 tokens</li></ul> | |
| * Samples: |
| | anchor | positive | negative | |
| |:---------------------------------------------------------------|:---------------------------------------------------------------------------------|:------------------------------------------------------------| |
| | <code>rheumatic myocardiopathy</code> | <code>chronic rheumatic heart</code> | <code>abnormal liver lobulation</code> | |
| | <code>chronic rheumatic coronary arteritis</code> | <code>inactive or quiescent rheumatic fever with left ventricular failure</code> | <code>other specified impairment of visual functions</code> | |
| | <code>chronic rheumatic heart disease with endocarditis</code> | <code>rheumatic endocarditis</code> | <code>listeriosis, unspecified</code> | |
| * Loss: [<code>MultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with these parameters: |
| ```json |
| { |
| "scale": 20.0, |
| "similarity_fct": "cos_sim", |
| "gather_across_devices": false, |
| "directions": [ |
| "query_to_doc" |
| ], |
| "partition_mode": "joint", |
| "hardness_mode": null, |
| "hardness_strength": 0.0 |
| } |
| ``` |
| |
| ### Training Hyperparameters |
| #### Non-Default Hyperparameters |
|
|
| - `per_device_train_batch_size`: 512 |
| - `num_train_epochs`: 1 |
| - `learning_rate`: 2e-05 |
| - `lr_scheduler_type`: cosine |
| - `warmup_steps`: 1580 |
| - `weight_decay`: 0.01 |
| - `bf16`: True |
| - `per_device_eval_batch_size`: 256 |
| - `eval_on_start`: True |
| - `load_best_model_at_end`: True |
| - `dataloader_drop_last`: True |
|
|
| #### All Hyperparameters |
| <details><summary>Click to expand</summary> |
|
|
| - `per_device_train_batch_size`: 512 |
| - `num_train_epochs`: 1 |
| - `max_steps`: -1 |
| - `learning_rate`: 2e-05 |
| - `lr_scheduler_type`: cosine |
| - `lr_scheduler_kwargs`: None |
| - `warmup_steps`: 1580 |
| - `optim`: adamw_torch_fused |
| - `optim_args`: None |
| - `weight_decay`: 0.01 |
| - `adam_beta1`: 0.9 |
| - `adam_beta2`: 0.999 |
| - `adam_epsilon`: 1e-08 |
| - `optim_target_modules`: None |
| - `gradient_accumulation_steps`: 1 |
| - `average_tokens_across_devices`: True |
| - `max_grad_norm`: 1.0 |
| - `label_smoothing_factor`: 0.0 |
| - `bf16`: True |
| - `fp16`: False |
| - `bf16_full_eval`: False |
| - `fp16_full_eval`: False |
| - `tf32`: None |
| - `gradient_checkpointing`: False |
| - `gradient_checkpointing_kwargs`: None |
| - `torch_compile`: False |
| - `torch_compile_backend`: None |
| - `torch_compile_mode`: None |
| - `use_liger_kernel`: False |
| - `liger_kernel_config`: None |
| - `use_cache`: False |
| - `neftune_noise_alpha`: None |
| - `torch_empty_cache_steps`: None |
| - `auto_find_batch_size`: False |
| - `log_on_each_node`: True |
| - `logging_nan_inf_filter`: True |
| - `include_num_input_tokens_seen`: no |
| - `log_level`: passive |
| - `log_level_replica`: warning |
| - `disable_tqdm`: False |
| - `project`: huggingface |
| - `trackio_space_id`: None |
| - `trackio_bucket_id`: None |
| - `trackio_static_space_id`: None |
| - `per_device_eval_batch_size`: 256 |
| - `prediction_loss_only`: True |
| - `eval_on_start`: True |
| - `eval_do_concat_batches`: True |
| - `eval_use_gather_object`: False |
| - `eval_accumulation_steps`: None |
| - `include_for_metrics`: [] |
| - `batch_eval_metrics`: False |
| - `save_only_model`: False |
| - `save_on_each_node`: False |
| - `enable_jit_checkpoint`: False |
| - `push_to_hub`: False |
| - `hub_private_repo`: None |
| - `hub_model_id`: None |
| - `hub_strategy`: every_save |
| - `hub_always_push`: False |
| - `hub_revision`: None |
| - `load_best_model_at_end`: True |
| - `ignore_data_skip`: False |
| - `restore_callback_states_from_checkpoint`: False |
| - `full_determinism`: False |
| - `seed`: 42 |
| - `data_seed`: None |
| - `use_cpu`: False |
| - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} |
| - `parallelism_config`: None |
| - `dataloader_drop_last`: True |
| - `dataloader_num_workers`: 0 |
| - `dataloader_pin_memory`: True |
| - `dataloader_persistent_workers`: False |
| - `dataloader_prefetch_factor`: None |
| - `dataloader_multiprocessing_context`: None |
| - `dataloader_in_order`: True |
| - `remove_unused_columns`: True |
| - `label_names`: None |
| - `train_sampling_strategy`: random |
| - `length_column_name`: length |
| - `ddp_find_unused_parameters`: None |
| - `ddp_bucket_cap_mb`: None |
| - `ddp_broadcast_buffers`: False |
| - `ddp_static_graph`: None |
| - `ddp_backend`: None |
| - `ddp_timeout`: 1800 |
| - `fsdp`: None |
| - `fsdp_config`: None |
| - `deepspeed`: None |
| - `debug`: [] |
| - `skip_memory_metrics`: True |
| - `do_predict`: False |
| - `resume_from_checkpoint`: None |
| - `local_rank`: -1 |
| - `prompts`: None |
| - `batch_sampler`: batch_sampler |
| - `multi_dataset_batch_sampler`: proportional |
| - `router_mapping`: {} |
| - `learning_rate_mapping`: {} |
| - `warmup_ratio`: None |
|
|
| </details> |
|
|
| ### Training Logs |
| <details><summary>Click to expand</summary> |
|
|
| | Epoch | Step | Training Loss | icd11-val_cosine_ndcg@10 | |
| |:----------:|:---------:|:-------------:|:------------------------:| |
| | 0 | 0 | - | 0.5035 | |
| | 0.0032 | 50 | 3.3034 | - | |
| | 0.0063 | 100 | 3.0570 | - | |
| | 0.0095 | 150 | 2.6980 | - | |
| | 0.0127 | 200 | 2.3748 | - | |
| | 0.0158 | 250 | 2.1693 | - | |
| | 0.0190 | 300 | 2.0670 | - | |
| | 0.0222 | 350 | 2.0079 | - | |
| | 0.0253 | 400 | 1.9661 | - | |
| | 0.0285 | 450 | 1.9317 | - | |
| | 0.0316 | 500 | 1.9053 | 0.5508 | |
| | 0.0348 | 550 | 1.8528 | - | |
| | 0.0380 | 600 | 1.8550 | - | |
| | 0.0411 | 650 | 1.7966 | - | |
| | 0.0443 | 700 | 1.7828 | - | |
| | 0.0475 | 750 | 1.7346 | - | |
| | 0.0506 | 800 | 1.6919 | - | |
| | 0.0538 | 850 | 1.6980 | - | |
| | 0.0570 | 900 | 1.6606 | - | |
| | 0.0601 | 950 | 1.6058 | - | |
| | 0.0633 | 1000 | 1.5859 | 0.5926 | |
| | 0.0665 | 1050 | 1.5658 | - | |
| | 0.0696 | 1100 | 1.5588 | - | |
| | 0.0728 | 1150 | 1.5080 | - | |
| | 0.0759 | 1200 | 1.4797 | - | |
| | 0.0791 | 1250 | 1.4802 | - | |
| | 0.0823 | 1300 | 1.4226 | - | |
| | 0.0854 | 1350 | 1.4006 | - | |
| | 0.0886 | 1400 | 1.3809 | - | |
| | 0.0918 | 1450 | 1.3675 | - | |
| | 0.0949 | 1500 | 1.3573 | 0.6178 | |
| | 0.0981 | 1550 | 1.3354 | - | |
| | 0.1013 | 1600 | 1.3037 | - | |
| | 0.1044 | 1650 | 1.2889 | - | |
| | 0.1076 | 1700 | 1.2706 | - | |
| | 0.1108 | 1750 | 1.2340 | - | |
| | 0.1139 | 1800 | 1.2057 | - | |
| | 0.1171 | 1850 | 1.1862 | - | |
| | 0.1203 | 1900 | 1.1948 | - | |
| | 0.1234 | 1950 | 1.1557 | - | |
| | 0.1266 | 2000 | 1.1571 | 0.6386 | |
| | 0.1297 | 2050 | 1.1288 | - | |
| | 0.1329 | 2100 | 1.1162 | - | |
| | 0.1361 | 2150 | 1.0780 | - | |
| | 0.1392 | 2200 | 1.0784 | - | |
| | 0.1424 | 2250 | 1.0607 | - | |
| | 0.1456 | 2300 | 1.0435 | - | |
| | 0.1487 | 2350 | 1.0256 | - | |
| | 0.1519 | 2400 | 1.0248 | - | |
| | 0.1551 | 2450 | 0.9990 | - | |
| | 0.1582 | 2500 | 0.9757 | 0.6572 | |
| | 0.1614 | 2550 | 0.9813 | - | |
| | 0.1646 | 2600 | 0.9761 | - | |
| | 0.1677 | 2650 | 0.9549 | - | |
| | 0.1709 | 2700 | 0.9369 | - | |
| | 0.1741 | 2750 | 0.9127 | - | |
| | 0.1772 | 2800 | 0.9223 | - | |
| | 0.1804 | 2850 | 0.9096 | - | |
| | 0.1835 | 2900 | 0.8837 | - | |
| | 0.1867 | 2950 | 0.8843 | - | |
| | 0.1899 | 3000 | 0.8898 | 0.6649 | |
| | 0.1930 | 3050 | 0.8656 | - | |
| | 0.1962 | 3100 | 0.8796 | - | |
| | 0.1994 | 3150 | 0.8464 | - | |
| | 0.2025 | 3200 | 0.8261 | - | |
| | 0.2057 | 3250 | 0.8280 | - | |
| | 0.2089 | 3300 | 0.8319 | - | |
| | 0.2120 | 3350 | 0.8292 | - | |
| | 0.2152 | 3400 | 0.7955 | - | |
| | 0.2184 | 3450 | 0.8011 | - | |
| | 0.2215 | 3500 | 0.7908 | 0.6771 | |
| | 0.2247 | 3550 | 0.7760 | - | |
| | 0.2278 | 3600 | 0.7667 | - | |
| | 0.2310 | 3650 | 0.7683 | - | |
| | 0.2342 | 3700 | 0.7523 | - | |
| | 0.2373 | 3750 | 0.7344 | - | |
| | 0.2405 | 3800 | 0.7471 | - | |
| | 0.2437 | 3850 | 0.7250 | - | |
| | 0.2468 | 3900 | 0.7163 | - | |
| | 0.25 | 3950 | 0.7003 | - | |
| | 0.2532 | 4000 | 0.7103 | 0.6836 | |
| | 0.2563 | 4050 | 0.7073 | - | |
| | 0.2595 | 4100 | 0.7046 | - | |
| | 0.2627 | 4150 | 0.6804 | - | |
| | 0.2658 | 4200 | 0.6806 | - | |
| | 0.2690 | 4250 | 0.6792 | - | |
| | 0.2722 | 4300 | 0.6708 | - | |
| | 0.2753 | 4350 | 0.6678 | - | |
| | 0.2785 | 4400 | 0.6475 | - | |
| | 0.2816 | 4450 | 0.6444 | - | |
| | 0.2848 | 4500 | 0.6395 | 0.6841 | |
| | 0.2880 | 4550 | 0.6399 | - | |
| | 0.2911 | 4600 | 0.6415 | - | |
| | 0.2943 | 4650 | 0.6407 | - | |
| | 0.2975 | 4700 | 0.6220 | - | |
| | 0.3006 | 4750 | 0.6198 | - | |
| | 0.3038 | 4800 | 0.6145 | - | |
| | 0.3070 | 4850 | 0.6029 | - | |
| | 0.3101 | 4900 | 0.6042 | - | |
| | 0.3133 | 4950 | 0.5962 | - | |
| | 0.3165 | 5000 | 0.5979 | 0.6933 | |
| | 0.3196 | 5050 | 0.5811 | - | |
| | 0.3228 | 5100 | 0.5738 | - | |
| | 0.3259 | 5150 | 0.5731 | - | |
| | 0.3291 | 5200 | 0.5787 | - | |
| | 0.3323 | 5250 | 0.5770 | - | |
| | 0.3354 | 5300 | 0.5539 | - | |
| | 0.3386 | 5350 | 0.5573 | - | |
| | 0.3418 | 5400 | 0.5619 | - | |
| | 0.3449 | 5450 | 0.5387 | - | |
| | 0.3481 | 5500 | 0.5357 | 0.6979 | |
| | 0.3513 | 5550 | 0.5502 | - | |
| | 0.3544 | 5600 | 0.5316 | - | |
| | 0.3576 | 5650 | 0.5386 | - | |
| | 0.3608 | 5700 | 0.5257 | - | |
| | 0.3639 | 5750 | 0.5032 | - | |
| | 0.3671 | 5800 | 0.5197 | - | |
| | 0.3703 | 5850 | 0.5147 | - | |
| | 0.3734 | 5900 | 0.5194 | - | |
| | 0.3766 | 5950 | 0.5195 | - | |
| | 0.3797 | 6000 | 0.5241 | 0.6999 | |
| | 0.3829 | 6050 | 0.5032 | - | |
| | 0.3861 | 6100 | 0.4955 | - | |
| | 0.3892 | 6150 | 0.5055 | - | |
| | 0.3924 | 6200 | 0.4951 | - | |
| | 0.3956 | 6250 | 0.5022 | - | |
| | 0.3987 | 6300 | 0.4916 | - | |
| | 0.4019 | 6350 | 0.4840 | - | |
| | 0.4051 | 6400 | 0.4684 | - | |
| | 0.4082 | 6450 | 0.4569 | - | |
| | 0.4114 | 6500 | 0.4737 | 0.7041 | |
| | 0.4146 | 6550 | 0.4609 | - | |
| | 0.4177 | 6600 | 0.4694 | - | |
| | 0.4209 | 6650 | 0.4588 | - | |
| | 0.4241 | 6700 | 0.4710 | - | |
| | 0.4272 | 6750 | 0.4526 | - | |
| | 0.4304 | 6800 | 0.4605 | - | |
| | 0.4335 | 6850 | 0.4720 | - | |
| | 0.4367 | 6900 | 0.4594 | - | |
| | 0.4399 | 6950 | 0.4497 | - | |
| | 0.4430 | 7000 | 0.4475 | 0.7087 | |
| | 0.4462 | 7050 | 0.4415 | - | |
| | 0.4494 | 7100 | 0.4275 | - | |
| | 0.4525 | 7150 | 0.4349 | - | |
| | 0.4557 | 7200 | 0.4375 | - | |
| | 0.4589 | 7250 | 0.4340 | - | |
| | 0.4620 | 7300 | 0.4346 | - | |
| | 0.4652 | 7350 | 0.4281 | - | |
| | 0.4684 | 7400 | 0.4277 | - | |
| | 0.4715 | 7450 | 0.4188 | - | |
| | 0.4747 | 7500 | 0.4179 | 0.7109 | |
| | 0.4778 | 7550 | 0.4129 | - | |
| | 0.4810 | 7600 | 0.4137 | - | |
| | 0.4842 | 7650 | 0.4136 | - | |
| | 0.4873 | 7700 | 0.4091 | - | |
| | 0.4905 | 7750 | 0.4068 | - | |
| | 0.4937 | 7800 | 0.4089 | - | |
| | 0.4968 | 7850 | 0.4277 | - | |
| | 0.5 | 7900 | 0.3952 | - | |
| | 0.5032 | 7950 | 0.4009 | - | |
| | 0.5063 | 8000 | 0.3889 | 0.7134 | |
| | 0.5095 | 8050 | 0.4008 | - | |
| | 0.5127 | 8100 | 0.4031 | - | |
| | 0.5158 | 8150 | 0.3950 | - | |
| | 0.5190 | 8200 | 0.3807 | - | |
| | 0.5222 | 8250 | 0.3891 | - | |
| | 0.5253 | 8300 | 0.3940 | - | |
| | 0.5285 | 8350 | 0.3929 | - | |
| | 0.5316 | 8400 | 0.3935 | - | |
| | 0.5348 | 8450 | 0.3755 | - | |
| | 0.5380 | 8500 | 0.3788 | 0.7176 | |
| | 0.5411 | 8550 | 0.3770 | - | |
| | 0.5443 | 8600 | 0.3882 | - | |
| | 0.5475 | 8650 | 0.3747 | - | |
| | 0.5506 | 8700 | 0.3785 | - | |
| | 0.5538 | 8750 | 0.3689 | - | |
| | 0.5570 | 8800 | 0.3706 | - | |
| | 0.5601 | 8850 | 0.3756 | - | |
| | 0.5633 | 8900 | 0.3679 | - | |
| | 0.5665 | 8950 | 0.3696 | - | |
| | 0.5696 | 9000 | 0.3722 | 0.7176 | |
| | 0.5728 | 9050 | 0.3647 | - | |
| | 0.5759 | 9100 | 0.3616 | - | |
| | 0.5791 | 9150 | 0.3635 | - | |
| | 0.5823 | 9200 | 0.3447 | - | |
| | 0.5854 | 9250 | 0.3612 | - | |
| | 0.5886 | 9300 | 0.3609 | - | |
| | 0.5918 | 9350 | 0.3368 | - | |
| | 0.5949 | 9400 | 0.3509 | - | |
| | 0.5981 | 9450 | 0.3631 | - | |
| | 0.6013 | 9500 | 0.3422 | 0.7188 | |
| | 0.6044 | 9550 | 0.3537 | - | |
| | 0.6076 | 9600 | 0.3507 | - | |
| | 0.6108 | 9650 | 0.3460 | - | |
| | 0.6139 | 9700 | 0.3527 | - | |
| | 0.6171 | 9750 | 0.3441 | - | |
| | 0.6203 | 9800 | 0.3509 | - | |
| | 0.6234 | 9850 | 0.3396 | - | |
| | 0.6266 | 9900 | 0.3350 | - | |
| | 0.6297 | 9950 | 0.3493 | - | |
| | 0.6329 | 10000 | 0.3299 | 0.7220 | |
| | 0.6361 | 10050 | 0.3377 | - | |
| | 0.6392 | 10100 | 0.3401 | - | |
| | 0.6424 | 10150 | 0.3414 | - | |
| | 0.6456 | 10200 | 0.3385 | - | |
| | 0.6487 | 10250 | 0.3408 | - | |
| | 0.6519 | 10300 | 0.3450 | - | |
| | 0.6551 | 10350 | 0.3482 | - | |
| | 0.6582 | 10400 | 0.3261 | - | |
| | 0.6614 | 10450 | 0.3350 | - | |
| | 0.6646 | 10500 | 0.3335 | 0.7215 | |
| | 0.6677 | 10550 | 0.3458 | - | |
| | 0.6709 | 10600 | 0.3194 | - | |
| | 0.6741 | 10650 | 0.3224 | - | |
| | 0.6772 | 10700 | 0.3289 | - | |
| | 0.6804 | 10750 | 0.3239 | - | |
| | 0.6835 | 10800 | 0.3283 | - | |
| | 0.6867 | 10850 | 0.3277 | - | |
| | 0.6899 | 10900 | 0.3243 | - | |
| | 0.6930 | 10950 | 0.3196 | - | |
| | 0.6962 | 11000 | 0.3204 | 0.7220 | |
| | 0.6994 | 11050 | 0.3334 | - | |
| | 0.7025 | 11100 | 0.3216 | - | |
| | 0.7057 | 11150 | 0.3286 | - | |
| | 0.7089 | 11200 | 0.3119 | - | |
| | 0.7120 | 11250 | 0.3185 | - | |
| | 0.7152 | 11300 | 0.3058 | - | |
| | 0.7184 | 11350 | 0.3157 | - | |
| | 0.7215 | 11400 | 0.3136 | - | |
| | 0.7247 | 11450 | 0.3179 | - | |
| | 0.7278 | 11500 | 0.3187 | 0.7235 | |
| | 0.7310 | 11550 | 0.3144 | - | |
| | 0.7342 | 11600 | 0.3156 | - | |
| | 0.7373 | 11650 | 0.3139 | - | |
| | 0.7405 | 11700 | 0.3104 | - | |
| | 0.7437 | 11750 | 0.3187 | - | |
| | 0.7468 | 11800 | 0.3154 | - | |
| | 0.75 | 11850 | 0.3132 | - | |
| | 0.7532 | 11900 | 0.3110 | - | |
| | 0.7563 | 11950 | 0.3143 | - | |
| | 0.7595 | 12000 | 0.3119 | 0.7236 | |
| | 0.7627 | 12050 | 0.3068 | - | |
| | 0.7658 | 12100 | 0.3227 | - | |
| | 0.7690 | 12150 | 0.3170 | - | |
| | 0.7722 | 12200 | 0.3046 | - | |
| | 0.7753 | 12250 | 0.3085 | - | |
| | 0.7785 | 12300 | 0.3135 | - | |
| | 0.7816 | 12350 | 0.3042 | - | |
| | 0.7848 | 12400 | 0.3152 | - | |
| | 0.7880 | 12450 | 0.3122 | - | |
| | 0.7911 | 12500 | 0.3201 | 0.7235 | |
| | 0.7943 | 12550 | 0.3148 | - | |
| | 0.7975 | 12600 | 0.2993 | - | |
| | 0.8006 | 12650 | 0.3040 | - | |
| | 0.8038 | 12700 | 0.2973 | - | |
| | 0.8070 | 12750 | 0.3074 | - | |
| | 0.8101 | 12800 | 0.3074 | - | |
| | 0.8133 | 12850 | 0.2971 | - | |
| | 0.8165 | 12900 | 0.3060 | - | |
| | 0.8196 | 12950 | 0.3136 | - | |
| | 0.8228 | 13000 | 0.3095 | 0.7244 | |
| | 0.8259 | 13050 | 0.3032 | - | |
| | 0.8291 | 13100 | 0.3070 | - | |
| | 0.8323 | 13150 | 0.3014 | - | |
| | 0.8354 | 13200 | 0.2965 | - | |
| | 0.8386 | 13250 | 0.2977 | - | |
| | 0.8418 | 13300 | 0.3077 | - | |
| | 0.8449 | 13350 | 0.3145 | - | |
| | 0.8481 | 13400 | 0.3019 | - | |
| | 0.8513 | 13450 | 0.3201 | - | |
| | **0.8544** | **13500** | **0.2986** | **0.7248** | |
| | 0.8576 | 13550 | 0.3001 | - | |
| | 0.8608 | 13600 | 0.3047 | - | |
| | 0.8639 | 13650 | 0.3040 | - | |
| | 0.8671 | 13700 | 0.2923 | - | |
| | 0.8703 | 13750 | 0.3029 | - | |
| | 0.8734 | 13800 | 0.2942 | - | |
| | 0.8766 | 13850 | 0.2970 | - | |
| | 0.8797 | 13900 | 0.2971 | - | |
| | 0.8829 | 13950 | 0.2927 | - | |
| | 0.8861 | 14000 | 0.3006 | 0.7239 | |
| | 0.8892 | 14050 | 0.2930 | - | |
| | 0.8924 | 14100 | 0.3001 | - | |
| | 0.8956 | 14150 | 0.3124 | - | |
| | 0.8987 | 14200 | 0.3083 | - | |
| | 0.9019 | 14250 | 0.3059 | - | |
| | 0.9051 | 14300 | 0.2926 | - | |
| | 0.9082 | 14350 | 0.2905 | - | |
| | 0.9114 | 14400 | 0.2959 | - | |
| | 0.9146 | 14450 | 0.3038 | - | |
| | 0.9177 | 14500 | 0.3045 | 0.7243 | |
| | 0.9209 | 14550 | 0.2908 | - | |
| | 0.9241 | 14600 | 0.3043 | - | |
| | 0.9272 | 14650 | 0.3007 | - | |
| | 0.9304 | 14700 | 0.2933 | - | |
| | 0.9335 | 14750 | 0.2951 | - | |
| | 0.9367 | 14800 | 0.2874 | - | |
| | 0.9399 | 14850 | 0.3041 | - | |
| | 0.9430 | 14900 | 0.3019 | - | |
| | 0.9462 | 14950 | 0.3073 | - | |
| | 0.9494 | 15000 | 0.3064 | 0.7245 | |
| | 0.9525 | 15050 | 0.3038 | - | |
| | 0.9557 | 15100 | 0.3043 | - | |
| | 0.9589 | 15150 | 0.2990 | - | |
| | 0.9620 | 15200 | 0.3016 | - | |
| | 0.9652 | 15250 | 0.3073 | - | |
| | 0.9684 | 15300 | 0.2959 | - | |
| | 0.9715 | 15350 | 0.2936 | - | |
| | 0.9747 | 15400 | 0.2928 | - | |
| | 0.9778 | 15450 | 0.2945 | - | |
| | 0.9810 | 15500 | 0.2930 | 0.7247 | |
| | 0.9842 | 15550 | 0.3092 | - | |
| | 0.9873 | 15600 | 0.3079 | - | |
| | 0.9905 | 15650 | 0.2975 | - | |
| | 0.9937 | 15700 | 0.2969 | - | |
| | 0.9968 | 15750 | 0.2890 | - | |
| | 1.0 | 15800 | 0.2956 | 0.7242 | |
|
|
| * The bold row denotes the saved checkpoint. |
| </details> |
|
|
| ### Training Time |
| - **Training**: 26.3 minutes |
| - **Evaluation**: 1.5 minutes |
| - **Total**: 27.8 minutes |
|
|
| ### Framework Versions |
| - Python: 3.13.15 |
| - Sentence Transformers: 5.7.0 |
| - Transformers: 5.18.0 |
| - PyTorch: 2.11.0+cu130 |
| - Accelerate: 1.15.0 |
| - Datasets: 4.8.5 |
| - Tokenizers: 0.23.2 |
|
|
| ## Additional Resources |
|
|
| - [Training and Finetuning Embedding Models with Sentence Transformers](https://huggingface.co/blog/train-sentence-transformers): the end-to-end guide for training or finetuning Sentence Transformer models. |
| - [Introduction to Matryoshka Embedding Models](https://huggingface.co/blog/matryoshka): variable-size embeddings that can be truncated with minimal quality loss. |
| - [Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval](https://huggingface.co/blog/embedding-quantization): post-training compression of embedding vectors. |
| - [Multimodal Embedding & Reranker Models with Sentence Transformers](https://huggingface.co/blog/multimodal-sentence-transformers): use text, image, audio, and video models through the same API. |
| - [Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers](https://huggingface.co/blog/train-multimodal-sentence-transformers): train multimodal embedding models, with a Visual Document Retrieval walkthrough. |
|
|
| ## Citation |
|
|
| ### BibTeX |
|
|
| #### Sentence Transformers |
| ```bibtex |
| @inproceedings{reimers-2019-sentence-bert, |
| title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", |
| author = "Reimers, Nils and Gurevych, Iryna", |
| booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", |
| month = "11", |
| year = "2019", |
| publisher = "Association for Computational Linguistics", |
| url = "https://arxiv.org/abs/1908.10084", |
| } |
| ``` |
|
|
| #### MultipleNegativesRankingLoss |
| ```bibtex |
| @misc{oord2019representationlearningcontrastivepredictive, |
| title={Representation Learning with Contrastive Predictive Coding}, |
| author={Aaron van den Oord and Yazhe Li and Oriol Vinyals}, |
| year={2019}, |
| eprint={1807.03748}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.LG}, |
| url={https://arxiv.org/abs/1807.03748}, |
| } |
| ``` |
|
|
| <!-- |
| ## Glossary |
|
|
| *Clearly define terms in order to be accessible across audiences.* |
| --> |
|
|
| <!-- |
| ## Model Card Authors |
|
|
| *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* |
| --> |
|
|
| <!-- |
| ## Model Card Contact |
|
|
| *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* |
| --> |
