Add new SentenceTransformer model

f3c0b87 verified 28 days ago

38.8 kB

tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - dense
  - generated_from_trainer
  - dataset_size:8089600
  - loss:MultipleNegativesRankingLoss
base_model: abhinand/MedEmbed-small-v0.1
widget:
  - source_sentence: hypoxic ischaemic encephalopathy in diseases classified elsewhere
    sentences:
      - other specified hypoxic-ischaemic encephalopathy
      - monocytopenia
      - isolated benign cerebral vasculitis
  - source_sentence: congenital laryngomalacia
    sentences:
      - floppy epiglottis
      - fructose-1,6-bisphosphatase deficiency
      - structural developmental anomalies of the nose or cavum, unspecified
  - source_sentence: molluscum contagiosum
    sentences:
      - laceration of bronchus nos
      - water warts
      - inflammation of bartholin gland
  - source_sentence: contusion of heart without haemopericardium
    sentences:
      - contusion of heart
      - phlebotomus fever
      - rectovaginal fistula
  - source_sentence: multiple delivery by combination of methods
    sentences:
      - rectal spasm
      - multiple delivery by combination of methods without caesarean
      - bipolar or related disorders, unspecified
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
  - cosine_accuracy@1
  - cosine_accuracy@5
  - cosine_accuracy@10
  - cosine_precision@1
  - cosine_precision@3
  - cosine_precision@5
  - cosine_precision@10
  - cosine_recall@1
  - cosine_recall@3
  - cosine_recall@5
  - cosine_recall@10
  - cosine_ndcg@10
  - cosine_mrr@10
  - cosine_map@10
model-index:
  - name: SentenceTransformer based on abhinand/MedEmbed-small-v0.1
    results:
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: icd11 val
          type: icd11-val
        metrics:
          - type: cosine_accuracy@1
            value: 0.7649122807017544
            name: Cosine Accuracy@1
          - type: cosine_accuracy@5
            value: 0.8421052631578947
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 0.8690058479532163
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.7649122807017544
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.6775828460038987
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.5796491228070175
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.42701754385964913
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.16621564142709158
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.4102644606511703
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 0.5203490784736678
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 0.6468261230695532
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.724241553378169
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.7982117330362947
            name: Cosine Mrr@10
          - type: cosine_map@10
            value: 0.6694373149953072
            name: Cosine Map@10

SentenceTransformer based on abhinand/MedEmbed-small-v0.1

This is a sentence-transformers model finetuned from abhinand/MedEmbed-small-v0.1. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: abhinand/MedEmbed-small-v0.1
Maximum Sequence Length: 30 tokens
Output Dimensionality: 384 dimensions
Similarity Function: Cosine Similarity
Supported Modality: Text

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 384, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("seanfarrell/icd11_ner_model_v2")
# Run inference
queries = [
    'multiple delivery by combination of methods',
]
documents = [
    'multiple delivery by combination of methods without caesarean',
    'bipolar or related disorders, unspecified',
    'rectal spasm',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 384] [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.9466, -0.0751, -0.0982]])

Evaluation

Metrics

Information Retrieval

Dataset: icd11-val
Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.7649
cosine_accuracy@5	0.8421
cosine_accuracy@10	0.869
cosine_precision@1	0.7649
cosine_precision@3	0.6776
cosine_precision@5	0.5796
cosine_precision@10	0.427
cosine_recall@1	0.1662
cosine_recall@3	0.4103
cosine_recall@5	0.5203
cosine_recall@10	0.6468
cosine_ndcg@10	0.7242
cosine_mrr@10	0.7982
cosine_map@10	0.6694

Training Details

Training Dataset

Unnamed Dataset

Size: 8,089,600 training samples
Columns: anchor, positive, and negative

Approximate statistics based on the first 100 samples:

	anchor	positive	negative
type	string	string	string
modality	text	text	text
details	min: 4 tokens mean: 9.52 tokens max: 25 tokens	min: 3 tokens mean: 9.1 tokens max: 22 tokens	min: 3 tokens mean: 9.37 tokens max: 22 tokens

Samples:

anchor	positive	negative
`rheumatic myocardiopathy`	`chronic rheumatic heart`	`abnormal liver lobulation`
`chronic rheumatic coronary arteritis`	`inactive or quiescent rheumatic fever with left ventricular failure`	`other specified impairment of visual functions`
`chronic rheumatic heart disease with endocarditis`	`rheumatic endocarditis`	`listeriosis, unspecified`

Loss: MultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim",
    "gather_across_devices": false,
    "directions": [
        "query_to_doc"
    ],
    "partition_mode": "joint",
    "hardness_mode": null,
    "hardness_strength": 0.0
}

Training Hyperparameters

Non-Default Hyperparameters

per_device_train_batch_size: 512
num_train_epochs: 1
learning_rate: 2e-05
lr_scheduler_type: cosine
warmup_steps: 1580
weight_decay: 0.01
bf16: True
per_device_eval_batch_size: 256
eval_on_start: True
load_best_model_at_end: True
dataloader_drop_last: True

All Hyperparameters

Click to expand

per_device_train_batch_size: 512
num_train_epochs: 1
max_steps: -1
learning_rate: 2e-05
lr_scheduler_type: cosine
lr_scheduler_kwargs: None
warmup_steps: 1580
optim: adamw_torch_fused
optim_args: None
weight_decay: 0.01
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
optim_target_modules: None
gradient_accumulation_steps: 1
average_tokens_across_devices: True
max_grad_norm: 1.0
label_smoothing_factor: 0.0
bf16: True
fp16: False
bf16_full_eval: False
fp16_full_eval: False
tf32: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
use_liger_kernel: False
liger_kernel_config: None
use_cache: False
neftune_noise_alpha: None
torch_empty_cache_steps: None
auto_find_batch_size: False
log_on_each_node: True
logging_nan_inf_filter: True
include_num_input_tokens_seen: no
log_level: passive
log_level_replica: warning
disable_tqdm: False
project: huggingface
trackio_space_id: None
trackio_bucket_id: None
trackio_static_space_id: None
per_device_eval_batch_size: 256
prediction_loss_only: True
eval_on_start: True
eval_do_concat_batches: True
eval_use_gather_object: False
eval_accumulation_steps: None
include_for_metrics: []
batch_eval_metrics: False
save_only_model: False
save_on_each_node: False
enable_jit_checkpoint: False
push_to_hub: False
hub_private_repo: None
hub_model_id: None
hub_strategy: every_save
hub_always_push: False
hub_revision: None
load_best_model_at_end: True
ignore_data_skip: False
restore_callback_states_from_checkpoint: False
full_determinism: False
seed: 42
data_seed: None
use_cpu: False
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
dataloader_drop_last: True
dataloader_num_workers: 0
dataloader_pin_memory: True
dataloader_persistent_workers: False
dataloader_prefetch_factor: None
dataloader_multiprocessing_context: None
dataloader_in_order: True
remove_unused_columns: True
label_names: None
train_sampling_strategy: random
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
ddp_static_graph: None
ddp_backend: None
ddp_timeout: 1800
fsdp: None
fsdp_config: None
deepspeed: None
debug: []
skip_memory_metrics: True
do_predict: False
resume_from_checkpoint: None
local_rank: -1
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: proportional
router_mapping: {}
learning_rate_mapping: {}
warmup_ratio: None

Training Logs

Click to expand

Epoch	Step	Training Loss	icd11-val_cosine_ndcg@10
0	0	-	0.5035
0.0032	50	3.3034	-
0.0063	100	3.0570	-
0.0095	150	2.6980	-
0.0127	200	2.3748	-
0.0158	250	2.1693	-
0.0190	300	2.0670	-
0.0222	350	2.0079	-
0.0253	400	1.9661	-
0.0285	450	1.9317	-
0.0316	500	1.9053	0.5508
0.0348	550	1.8528	-
0.0380	600	1.8550	-
0.0411	650	1.7966	-
0.0443	700	1.7828	-
0.0475	750	1.7346	-
0.0506	800	1.6919	-
0.0538	850	1.6980	-
0.0570	900	1.6606	-
0.0601	950	1.6058	-
0.0633	1000	1.5859	0.5926
0.0665	1050	1.5658	-
0.0696	1100	1.5588	-
0.0728	1150	1.5080	-
0.0759	1200	1.4797	-
0.0791	1250	1.4802	-
0.0823	1300	1.4226	-
0.0854	1350	1.4006	-
0.0886	1400	1.3809	-
0.0918	1450	1.3675	-
0.0949	1500	1.3573	0.6178
0.0981	1550	1.3354	-
0.1013	1600	1.3037	-
0.1044	1650	1.2889	-
0.1076	1700	1.2706	-
0.1108	1750	1.2340	-
0.1139	1800	1.2057	-
0.1171	1850	1.1862	-
0.1203	1900	1.1948	-
0.1234	1950	1.1557	-
0.1266	2000	1.1571	0.6386
0.1297	2050	1.1288	-
0.1329	2100	1.1162	-
0.1361	2150	1.0780	-
0.1392	2200	1.0784	-
0.1424	2250	1.0607	-
0.1456	2300	1.0435	-
0.1487	2350	1.0256	-
0.1519	2400	1.0248	-
0.1551	2450	0.9990	-
0.1582	2500	0.9757	0.6572
0.1614	2550	0.9813	-
0.1646	2600	0.9761	-
0.1677	2650	0.9549	-
0.1709	2700	0.9369	-
0.1741	2750	0.9127	-
0.1772	2800	0.9223	-
0.1804	2850	0.9096	-
0.1835	2900	0.8837	-
0.1867	2950	0.8843	-
0.1899	3000	0.8898	0.6649
0.1930	3050	0.8656	-
0.1962	3100	0.8796	-
0.1994	3150	0.8464	-
0.2025	3200	0.8261	-
0.2057	3250	0.8280	-
0.2089	3300	0.8319	-
0.2120	3350	0.8292	-
0.2152	3400	0.7955	-
0.2184	3450	0.8011	-
0.2215	3500	0.7908	0.6771
0.2247	3550	0.7760	-
0.2278	3600	0.7667	-
0.2310	3650	0.7683	-
0.2342	3700	0.7523	-
0.2373	3750	0.7344	-
0.2405	3800	0.7471	-
0.2437	3850	0.7250	-
0.2468	3900	0.7163	-
0.25	3950	0.7003	-
0.2532	4000	0.7103	0.6836
0.2563	4050	0.7073	-
0.2595	4100	0.7046	-
0.2627	4150	0.6804	-
0.2658	4200	0.6806	-
0.2690	4250	0.6792	-
0.2722	4300	0.6708	-
0.2753	4350	0.6678	-
0.2785	4400	0.6475	-
0.2816	4450	0.6444	-
0.2848	4500	0.6395	0.6841
0.2880	4550	0.6399	-
0.2911	4600	0.6415	-
0.2943	4650	0.6407	-
0.2975	4700	0.6220	-
0.3006	4750	0.6198	-
0.3038	4800	0.6145	-
0.3070	4850	0.6029	-
0.3101	4900	0.6042	-
0.3133	4950	0.5962	-
0.3165	5000	0.5979	0.6933
0.3196	5050	0.5811	-
0.3228	5100	0.5738	-
0.3259	5150	0.5731	-
0.3291	5200	0.5787	-
0.3323	5250	0.5770	-
0.3354	5300	0.5539	-
0.3386	5350	0.5573	-
0.3418	5400	0.5619	-
0.3449	5450	0.5387	-
0.3481	5500	0.5357	0.6979
0.3513	5550	0.5502	-
0.3544	5600	0.5316	-
0.3576	5650	0.5386	-
0.3608	5700	0.5257	-
0.3639	5750	0.5032	-
0.3671	5800	0.5197	-
0.3703	5850	0.5147	-
0.3734	5900	0.5194	-
0.3766	5950	0.5195	-
0.3797	6000	0.5241	0.6999
0.3829	6050	0.5032	-
0.3861	6100	0.4955	-
0.3892	6150	0.5055	-
0.3924	6200	0.4951	-
0.3956	6250	0.5022	-
0.3987	6300	0.4916	-
0.4019	6350	0.4840	-
0.4051	6400	0.4684	-
0.4082	6450	0.4569	-
0.4114	6500	0.4737	0.7041
0.4146	6550	0.4609	-
0.4177	6600	0.4694	-
0.4209	6650	0.4588	-
0.4241	6700	0.4710	-
0.4272	6750	0.4526	-
0.4304	6800	0.4605	-
0.4335	6850	0.4720	-
0.4367	6900	0.4594	-
0.4399	6950	0.4497	-
0.4430	7000	0.4475	0.7087
0.4462	7050	0.4415	-
0.4494	7100	0.4275	-
0.4525	7150	0.4349	-
0.4557	7200	0.4375	-
0.4589	7250	0.4340	-
0.4620	7300	0.4346	-
0.4652	7350	0.4281	-
0.4684	7400	0.4277	-
0.4715	7450	0.4188	-
0.4747	7500	0.4179	0.7109
0.4778	7550	0.4129	-
0.4810	7600	0.4137	-
0.4842	7650	0.4136	-
0.4873	7700	0.4091	-
0.4905	7750	0.4068	-
0.4937	7800	0.4089	-
0.4968	7850	0.4277	-
0.5	7900	0.3952	-
0.5032	7950	0.4009	-
0.5063	8000	0.3889	0.7134
0.5095	8050	0.4008	-
0.5127	8100	0.4031	-
0.5158	8150	0.3950	-
0.5190	8200	0.3807	-
0.5222	8250	0.3891	-
0.5253	8300	0.3940	-
0.5285	8350	0.3929	-
0.5316	8400	0.3935	-
0.5348	8450	0.3755	-
0.5380	8500	0.3788	0.7176
0.5411	8550	0.3770	-
0.5443	8600	0.3882	-
0.5475	8650	0.3747	-
0.5506	8700	0.3785	-
0.5538	8750	0.3689	-
0.5570	8800	0.3706	-
0.5601	8850	0.3756	-
0.5633	8900	0.3679	-
0.5665	8950	0.3696	-
0.5696	9000	0.3722	0.7176
0.5728	9050	0.3647	-
0.5759	9100	0.3616	-
0.5791	9150	0.3635	-
0.5823	9200	0.3447	-
0.5854	9250	0.3612	-
0.5886	9300	0.3609	-
0.5918	9350	0.3368	-
0.5949	9400	0.3509	-
0.5981	9450	0.3631	-
0.6013	9500	0.3422	0.7188
0.6044	9550	0.3537	-
0.6076	9600	0.3507	-
0.6108	9650	0.3460	-
0.6139	9700	0.3527	-
0.6171	9750	0.3441	-
0.6203	9800	0.3509	-
0.6234	9850	0.3396	-
0.6266	9900	0.3350	-
0.6297	9950	0.3493	-
0.6329	10000	0.3299	0.7220
0.6361	10050	0.3377	-
0.6392	10100	0.3401	-
0.6424	10150	0.3414	-
0.6456	10200	0.3385	-
0.6487	10250	0.3408	-
0.6519	10300	0.3450	-
0.6551	10350	0.3482	-
0.6582	10400	0.3261	-
0.6614	10450	0.3350	-
0.6646	10500	0.3335	0.7215
0.6677	10550	0.3458	-
0.6709	10600	0.3194	-
0.6741	10650	0.3224	-
0.6772	10700	0.3289	-
0.6804	10750	0.3239	-
0.6835	10800	0.3283	-
0.6867	10850	0.3277	-
0.6899	10900	0.3243	-
0.6930	10950	0.3196	-
0.6962	11000	0.3204	0.7220
0.6994	11050	0.3334	-
0.7025	11100	0.3216	-
0.7057	11150	0.3286	-
0.7089	11200	0.3119	-
0.7120	11250	0.3185	-
0.7152	11300	0.3058	-
0.7184	11350	0.3157	-
0.7215	11400	0.3136	-
0.7247	11450	0.3179	-
0.7278	11500	0.3187	0.7235
0.7310	11550	0.3144	-
0.7342	11600	0.3156	-
0.7373	11650	0.3139	-
0.7405	11700	0.3104	-
0.7437	11750	0.3187	-
0.7468	11800	0.3154	-
0.75	11850	0.3132	-
0.7532	11900	0.3110	-
0.7563	11950	0.3143	-
0.7595	12000	0.3119	0.7236
0.7627	12050	0.3068	-
0.7658	12100	0.3227	-
0.7690	12150	0.3170	-
0.7722	12200	0.3046	-
0.7753	12250	0.3085	-
0.7785	12300	0.3135	-
0.7816	12350	0.3042	-
0.7848	12400	0.3152	-
0.7880	12450	0.3122	-
0.7911	12500	0.3201	0.7235
0.7943	12550	0.3148	-
0.7975	12600	0.2993	-
0.8006	12650	0.3040	-
0.8038	12700	0.2973	-
0.8070	12750	0.3074	-
0.8101	12800	0.3074	-
0.8133	12850	0.2971	-
0.8165	12900	0.3060	-
0.8196	12950	0.3136	-
0.8228	13000	0.3095	0.7244
0.8259	13050	0.3032	-
0.8291	13100	0.3070	-
0.8323	13150	0.3014	-
0.8354	13200	0.2965	-
0.8386	13250	0.2977	-
0.8418	13300	0.3077	-
0.8449	13350	0.3145	-
0.8481	13400	0.3019	-
0.8513	13450	0.3201	-
0.8544	13500	0.2986	0.7248
0.8576	13550	0.3001	-
0.8608	13600	0.3047	-
0.8639	13650	0.3040	-
0.8671	13700	0.2923	-
0.8703	13750	0.3029	-
0.8734	13800	0.2942	-
0.8766	13850	0.2970	-
0.8797	13900	0.2971	-
0.8829	13950	0.2927	-
0.8861	14000	0.3006	0.7239
0.8892	14050	0.2930	-
0.8924	14100	0.3001	-
0.8956	14150	0.3124	-
0.8987	14200	0.3083	-
0.9019	14250	0.3059	-
0.9051	14300	0.2926	-
0.9082	14350	0.2905	-
0.9114	14400	0.2959	-
0.9146	14450	0.3038	-
0.9177	14500	0.3045	0.7243
0.9209	14550	0.2908	-
0.9241	14600	0.3043	-
0.9272	14650	0.3007	-
0.9304	14700	0.2933	-
0.9335	14750	0.2951	-
0.9367	14800	0.2874	-
0.9399	14850	0.3041	-
0.9430	14900	0.3019	-
0.9462	14950	0.3073	-
0.9494	15000	0.3064	0.7245
0.9525	15050	0.3038	-
0.9557	15100	0.3043	-
0.9589	15150	0.2990	-
0.9620	15200	0.3016	-
0.9652	15250	0.3073	-
0.9684	15300	0.2959	-
0.9715	15350	0.2936	-
0.9747	15400	0.2928	-
0.9778	15450	0.2945	-
0.9810	15500	0.2930	0.7247
0.9842	15550	0.3092	-
0.9873	15600	0.3079	-
0.9905	15650	0.2975	-
0.9937	15700	0.2969	-
0.9968	15750	0.2890	-
1.0	15800	0.2956	0.7242

The bold row denotes the saved checkpoint.

Training Time

Training: 26.3 minutes
Evaluation: 1.5 minutes
Total: 27.8 minutes

Framework Versions

Python: 3.13.15
Sentence Transformers: 5.7.0
Transformers: 5.18.0
PyTorch: 2.11.0+cu130
Accelerate: 1.15.0
Datasets: 4.8.5
Tokenizers: 0.23.2

Additional Resources

Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}