Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Paper โข 1908.10084 โข Published โข 19
How to use Luca207/reranker-finetuned with sentence-transformers:
from sentence_transformers import CrossEncoder
model = CrossEncoder("Luca207/reranker-finetuned")
query = "Which planet is known as the Red Planet?"
passages = [
"Venus is often called Earth's twin because of its similar size and proximity.",
"Mars, known for its reddish appearance, is often referred to as the Red Planet.",
"Jupiter, the largest planet in our solar system, has a prominent red spot.",
"Saturn, famous for its rings, is sometimes mistaken for the Red Planet."
]
scores = model.predict([(query, passage) for passage in passages])
print(scores)This is a Cross Encoder model finetuned from google-bert/bert-base-uncased using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.
CrossEncoder(
(0): Transformer({'transformer_task': 'sequence-classification', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'logits'}}, 'module_output_name': 'scores', 'architecture': 'BertForSequenceClassification'})
)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import CrossEncoder
# Download from the ๐ค Hub
model = CrossEncoder("reranker-finetuned")
# Get scores for pairs of inputs
pairs = [
['\n QuestionText: Solve this equation, giving all solutions:\n\\(\n3 p^{2}+9=39\n\\)\n\n ConstructName: Solve quadratic equations using balancing\n\n SubjectName: Quadratic Equations\n\n CorrectAnswer: \nor\n\\(\np=-\\sqrt{10}\n\\)\n\n WrongAnswer: \\[\np=2\n\\]\nor\n\\[\np=-2\n\\]\n ', 'When solving an equation, uses the same operation rather than the inverse.'],
['\n QuestionText: Dave has recorded the number of pets his classmates have in the frequency table on the right. \\begin{tabular}{|c|c|}\n\\hline Number of pets & Frequency \\\\\n\\hline & \\\\\n\\hline & \\\\\n\\hline & \\\\\n\\hline & \\\\\n\\hline & \\\\\n\\hline\n\\end{tabular} How many people did Dave record the results of?\n\n ConstructName: Interpret numbers in a frequency table\n\n SubjectName: Frequency tables\n\n CorrectAnswer: \n\n WrongAnswer: \n ', 'Adds up the variable column when asked for total frequency'],
['\n QuestionText: Which of the following is a correct way to describe the marked angle? ![Trapezium PQRS with angle QPS highlighted in red]()\n\n ConstructName: Label angles using correct 3-letter notation (e.g. angle ADB)\n\n SubjectName: Types, Naming and Estimating\n\n CorrectAnswer: \n\n WrongAnswer: QPRS\n ', 'Gives the vertex instead of the 3-letter angle notation'],
['\n QuestionText: A coach takes hours minutes to complete a journey from Manchester to London.\n\nIf the coach leaves Manchester at 09:25, what time does it arrive in London?\n\n ConstructName: Solve problems involving adding a period of time on to a given start time\n\n SubjectName: Time\n\n CorrectAnswer: \n\n WrongAnswer: 01:15\n ', 'Writes times in the afternoon as if they are morning on a 24 hour clock'],
['\n QuestionText: \n\n ConstructName: Carry out subtraction problems involving one negative integer\n\n SubjectName: Adding and Subtracting Negative Numbers\n\n CorrectAnswer: \n\n WrongAnswer: \n ', "Does not realise that subtracting a negative is the same as adding it's additive inverse"],
]
scores = model.predict(pairs)
print(scores)
# [0.8266 0.9364 0.9766 0.9578 0.9748]
# Or rank different texts based on similarity to a single text
ranks = model.rank(
'\n QuestionText: Solve this equation, giving all solutions:\n\\(\n3 p^{2}+9=39\n\\)\n\n ConstructName: Solve quadratic equations using balancing\n\n SubjectName: Quadratic Equations\n\n CorrectAnswer: \nor\n\\(\np=-\\sqrt{10}\n\\)\n\n WrongAnswer: \\[\np=2\n\\]\nor\n\\[\np=-2\n\\]\n ',
[
'When solving an equation, uses the same operation rather than the inverse.',
'Adds up the variable column when asked for total frequency',
'Gives the vertex instead of the 3-letter angle notation',
'Writes times in the afternoon as if they are morning on a 24 hour clock',
"Does not realise that subtracting a negative is the same as adding it's additive inverse",
]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]
text, missconception, and label| text | missconception | label | |
|---|---|---|---|
| type | string | string | int |
| modality | text | text | |
| details |
|
|
|
MultipleNegativesRankingLoss with these parameters:{
"scale": 10.0,
"num_negatives": 4,
"activation_fn": "torch.nn.modules.activation.Sigmoid"
}
text, missconception, and label| text | missconception | label | |
|---|---|---|---|
| type | string | string | int |
| modality | text | text | |
| details |
|
|
|
MultipleNegativesRankingLoss with these parameters:{
"scale": 10.0,
"num_negatives": 4,
"activation_fn": "torch.nn.modules.activation.Sigmoid"
}
per_device_train_batch_size: 16num_train_epochs: 2learning_rate: 0.0001warmup_steps: 0.1gradient_accumulation_steps: 2fp16: Trueper_device_eval_batch_size: 16per_device_train_batch_size: 16num_train_epochs: 2max_steps: -1learning_rate: 0.0001lr_scheduler_type: linearlr_scheduler_kwargs: Nonewarmup_steps: 0.1optim: adamw_torch_fusedoptim_args: Noneweight_decay: 0.0adam_beta1: 0.9adam_beta2: 0.999adam_epsilon: 1e-08optim_target_modules: Nonegradient_accumulation_steps: 2average_tokens_across_devices: Truemax_grad_norm: 1.0label_smoothing_factor: 0.0bf16: Falsefp16: Truebf16_full_eval: Falsefp16_full_eval: Falsetf32: Nonegradient_checkpointing: Falsegradient_checkpointing_kwargs: Nonetorch_compile: Falsetorch_compile_backend: Nonetorch_compile_mode: Noneuse_liger_kernel: Falseliger_kernel_config: Noneuse_cache: Falseneftune_noise_alpha: Nonetorch_empty_cache_steps: Noneauto_find_batch_size: Falselog_on_each_node: Truelogging_nan_inf_filter: Trueinclude_num_input_tokens_seen: nolog_level: passivelog_level_replica: warningdisable_tqdm: Falseproject: huggingfacetrackio_space_id: Nonetrackio_bucket_id: Nonetrackio_static_space_id: Noneper_device_eval_batch_size: 16prediction_loss_only: Trueeval_on_start: Falseeval_do_concat_batches: Trueeval_use_gather_object: Falseeval_accumulation_steps: Noneinclude_for_metrics: []batch_eval_metrics: Falsesave_only_model: Falsesave_on_each_node: Falseenable_jit_checkpoint: Falsepush_to_hub: Falsehub_private_repo: Nonehub_model_id: Nonehub_strategy: every_savehub_always_push: Falsehub_revision: Noneload_best_model_at_end: Falseignore_data_skip: Falserestore_callback_states_from_checkpoint: Falsefull_determinism: Falseseed: 42data_seed: Noneuse_cpu: Falseaccelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}parallelism_config: Nonedataloader_drop_last: Falsedataloader_num_workers: 0dataloader_pin_memory: Truedataloader_persistent_workers: Falsedataloader_prefetch_factor: Nonedataloader_multiprocessing_context: Nonedataloader_in_order: Trueremove_unused_columns: Truelabel_names: Nonetrain_sampling_strategy: randomlength_column_name: lengthddp_find_unused_parameters: Noneddp_bucket_cap_mb: Noneddp_broadcast_buffers: Falseddp_static_graph: Noneddp_backend: Noneddp_timeout: 1800fsdp: Nonefsdp_config: Nonedeepspeed: Nonedebug: []skip_memory_metrics: Truedo_predict: Falseresume_from_checkpoint: Nonelocal_rank: -1prompts: Nonebatch_sampler: batch_samplermulti_dataset_batch_sampler: proportionalrouter_mapping: {}learning_rate_mapping: {}warmup_ratio: None| Epoch | Step | Training Loss | Validation Loss |
|---|---|---|---|
| 0.0213 | 6 | 1.9288 | - |
| 0.0426 | 12 | 1.5921 | - |
| 0.0639 | 18 | 1.1075 | - |
| 0.0853 | 24 | 0.7169 | - |
| 0.1066 | 30 | 0.5473 | - |
| 0.1279 | 36 | 0.4761 | - |
| 0.1492 | 42 | 0.3948 | - |
| 0.1705 | 48 | 0.4800 | - |
| 0.1918 | 54 | 0.5520 | - |
| 0.2131 | 60 | 0.5596 | - |
| 0.2345 | 66 | 0.3587 | - |
| 0.2558 | 72 | 0.4442 | - |
| 0.2771 | 78 | 0.3607 | - |
| 0.2984 | 84 | 0.3622 | - |
| 0.3197 | 90 | 0.3339 | - |
| 0.3410 | 96 | 0.4211 | - |
| 0.3623 | 102 | 0.3275 | - |
| 0.3837 | 108 | 0.2806 | - |
| 0.4050 | 114 | 0.4830 | - |
| 0.4263 | 120 | 0.3304 | - |
| 0.4476 | 126 | 0.3471 | - |
| 0.4689 | 132 | 0.2266 | - |
| 0.4902 | 138 | 0.2479 | - |
| 0.5115 | 144 | 0.2506 | - |
| 0.5329 | 150 | 0.2103 | - |
| 0.5542 | 156 | 0.2699 | - |
| 0.5755 | 162 | 0.2931 | - |
| 0.5968 | 168 | 0.4029 | - |
| 0.6181 | 174 | 0.1841 | - |
| 0.6394 | 180 | 0.2195 | - |
| 0.6607 | 186 | 0.2391 | - |
| 0.6821 | 192 | 0.2131 | - |
| 0.7034 | 198 | 0.2103 | - |
| 0.7247 | 204 | 0.2687 | - |
| 0.7460 | 210 | 0.3169 | - |
| 0.7673 | 216 | 0.2280 | - |
| 0.7886 | 222 | 0.2084 | - |
| 0.8099 | 228 | 0.2113 | - |
| 0.8313 | 234 | 0.2049 | - |
| 0.8526 | 240 | 0.2736 | - |
| 0.8739 | 246 | 0.1631 | - |
| 0.8952 | 252 | 0.1771 | - |
| 0.9165 | 258 | 0.2031 | - |
| 0.9378 | 264 | 0.2642 | - |
| 0.9591 | 270 | 0.2185 | - |
| 0.9805 | 276 | 0.1537 | - |
| 1.0 | 282 | 0.1927 | 0.1680 |
| 1.0213 | 288 | 0.0810 | - |
| 1.0426 | 294 | 0.1010 | - |
| 1.0639 | 300 | 0.1640 | - |
| 1.0853 | 306 | 0.1220 | - |
| 1.1066 | 312 | 0.2518 | - |
| 1.1279 | 318 | 0.1932 | - |
| 1.1492 | 324 | 0.1435 | - |
| 1.1705 | 330 | 0.1295 | - |
| 1.1918 | 336 | 0.2066 | - |
| 1.2131 | 342 | 0.1823 | - |
| 1.2345 | 348 | 0.1687 | - |
| 1.2558 | 354 | 0.1663 | - |
| 1.2771 | 360 | 0.1192 | - |
| 1.2984 | 366 | 0.1610 | - |
| 1.3197 | 372 | 0.2017 | - |
| 1.3410 | 378 | 0.0877 | - |
| 1.3623 | 384 | 0.1714 | - |
| 1.3837 | 390 | 0.1482 | - |
| 1.4050 | 396 | 0.1481 | - |
| 1.4263 | 402 | 0.1413 | - |
| 1.4476 | 408 | 0.1010 | - |
| 1.4689 | 414 | 0.2108 | - |
| 1.4902 | 420 | 0.0982 | - |
| 1.5115 | 426 | 0.1551 | - |
| 1.5329 | 432 | 0.1501 | - |
| 1.5542 | 438 | 0.1301 | - |
| 1.5755 | 444 | 0.1083 | - |
| 1.5968 | 450 | 0.1894 | - |
| 1.6181 | 456 | 0.1167 | - |
| 1.6394 | 462 | 0.1397 | - |
| 1.6607 | 468 | 0.0776 | - |
| 1.6821 | 474 | 0.1218 | - |
| 1.7034 | 480 | 0.1077 | - |
| 1.7247 | 486 | 0.1507 | - |
| 1.7460 | 492 | 0.0702 | - |
| 1.7673 | 498 | 0.1216 | - |
| 1.7886 | 504 | 0.0844 | - |
| 1.8099 | 510 | 0.1513 | - |
| 1.8313 | 516 | 0.0762 | - |
| 1.8526 | 522 | 0.1426 | - |
| 1.8739 | 528 | 0.0782 | - |
| 1.8952 | 534 | 0.1015 | - |
| 1.9165 | 540 | 0.0699 | - |
| 1.9378 | 546 | 0.1085 | - |
| 1.9591 | 552 | 0.1081 | - |
| 1.9805 | 558 | 0.0689 | - |
| 2.0 | 564 | 0.0705 | 0.1055 |
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
Base model
google-bert/bert-base-uncased