Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Paper • 1908.10084 • Published • 19
How to use ute-httt-it3/canonical-skill-embedding with sentence-transformers:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("ute-httt-it3/canonical-skill-embedding")
sentences = [
"rest",
"android sdk",
"rest api",
"2 to 3 years"
]
embeddings = model.encode(sentences)
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [4, 4]This is a sentence-transformers model finetuned from FacebookAI/xlm-roberta-base. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.
SentenceTransformer(
(0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'XLMRobertaModel'})
(1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
'sql',
'structured query language',
'pytorch',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9624, 0.0873],
# [0.9624, 1.0000, 0.1089],
# [0.0873, 0.1089, 1.0000]])
anchor, positive, and negative| anchor | positive | negative | |
|---|---|---|---|
| type | string | string | string |
| modality | text | text | text |
| details |
|
|
|
| anchor | positive | negative |
|---|---|---|
machine vision |
cv |
hồ sơ xin việc |
js |
javascript |
jwt |
ngôn ngữ go |
golang |
cờ vây |
MultipleNegativesRankingLoss with these parameters:{
"scale": 20.0,
"similarity_fct": "cos_sim",
"gather_across_devices": false,
"directions": [
"query_to_doc"
],
"partition_mode": "joint",
"hardness_mode": null,
"hardness_strength": 0.0
}
per_device_train_batch_size: 32num_train_epochs: 8learning_rate: 3e-05lr_scheduler_type: cosinewarmup_steps: 0.1weight_decay: 0.01fp16: Truedisable_tqdm: Trueper_device_train_batch_size: 32num_train_epochs: 8max_steps: -1learning_rate: 3e-05lr_scheduler_type: cosinelr_scheduler_kwargs: Nonewarmup_steps: 0.1optim: adamw_torch_fusedoptim_args: Noneweight_decay: 0.01adam_beta1: 0.9adam_beta2: 0.999adam_epsilon: 1e-08optim_target_modules: Nonegradient_accumulation_steps: 1average_tokens_across_devices: Truemax_grad_norm: 1.0label_smoothing_factor: 0.0bf16: Falsefp16: Truebf16_full_eval: Falsefp16_full_eval: Falsetf32: Nonegradient_checkpointing: Falsegradient_checkpointing_kwargs: Nonetorch_compile: Falsetorch_compile_backend: Nonetorch_compile_mode: Noneuse_liger_kernel: Falseliger_kernel_config: Noneuse_cache: Falseneftune_noise_alpha: Nonetorch_empty_cache_steps: Noneauto_find_batch_size: Falselog_on_each_node: Truelogging_nan_inf_filter: Trueinclude_num_input_tokens_seen: nolog_level: passivelog_level_replica: warningdisable_tqdm: Trueproject: huggingfacetrackio_space_id: Nonetrackio_bucket_id: Nonetrackio_static_space_id: Noneper_device_eval_batch_size: 8prediction_loss_only: Trueeval_on_start: Falseeval_do_concat_batches: Trueeval_use_gather_object: Falseeval_accumulation_steps: Noneinclude_for_metrics: []batch_eval_metrics: Falsesave_only_model: Falsesave_on_each_node: Falseenable_jit_checkpoint: Falsepush_to_hub: Falsehub_private_repo: Nonehub_model_id: Nonehub_strategy: every_savehub_always_push: Falsehub_revision: Noneload_best_model_at_end: Falseignore_data_skip: Falserestore_callback_states_from_checkpoint: Falsefull_determinism: Falseseed: 42data_seed: Noneuse_cpu: Falseaccelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}parallelism_config: Nonedataloader_drop_last: Falsedataloader_num_workers: 0dataloader_pin_memory: Truedataloader_persistent_workers: Falsedataloader_prefetch_factor: Nonedataloader_multiprocessing_context: Nonedataloader_in_order: Trueremove_unused_columns: Truelabel_names: Nonetrain_sampling_strategy: randomlength_column_name: lengthddp_find_unused_parameters: Noneddp_bucket_cap_mb: Noneddp_broadcast_buffers: Falseddp_static_graph: Noneddp_backend: Noneddp_timeout: 1800fsdp: Nonefsdp_config: Nonedeepspeed: Nonedebug: []skip_memory_metrics: Truedo_predict: Falseresume_from_checkpoint: Nonelocal_rank: -1prompts: Nonebatch_sampler: batch_samplermulti_dataset_batch_sampler: proportionalrouter_mapping: {}learning_rate_mapping: {}warmup_ratio: None| Epoch | Step | Training Loss |
|---|---|---|
| 0.25 | 25 | 5.0175 |
| 0.5 | 50 | 4.8216 |
| 0.75 | 75 | 4.2196 |
| 1.0 | 100 | 3.8640 |
| 1.25 | 125 | 3.3843 |
| 1.5 | 150 | 2.7362 |
| 1.75 | 175 | 2.0061 |
| 2.0 | 200 | 1.5723 |
| 2.25 | 225 | 1.3134 |
| 2.5 | 250 | 1.0623 |
| 2.75 | 275 | 0.9845 |
| 3.0 | 300 | 0.9013 |
| 3.25 | 325 | 0.7943 |
| 3.5 | 350 | 0.7752 |
| 3.75 | 375 | 0.8153 |
| 4.0 | 400 | 0.7561 |
| 4.25 | 425 | 0.6833 |
| 4.5 | 450 | 0.7677 |
| 4.75 | 475 | 0.7124 |
| 5.0 | 500 | 0.7151 |
| 5.25 | 525 | 0.6392 |
| 5.5 | 550 | 0.6578 |
| 5.75 | 575 | 0.6980 |
| 6.0 | 600 | 0.6884 |
| 6.25 | 625 | 0.6718 |
| 6.5 | 650 | 0.6564 |
| 6.75 | 675 | 0.6574 |
| 7.0 | 700 | 0.6757 |
| 7.25 | 725 | 0.6396 |
| 7.5 | 750 | 0.6803 |
| 7.75 | 775 | 0.6383 |
| 8.0 | 800 | 0.6746 |
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
@misc{oord2019representationlearningcontrastivepredictive,
title={Representation Learning with Contrastive Predictive Coding},
author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
year={2019},
eprint={1807.03748},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/1807.03748},
}
Base model
FacebookAI/xlm-roberta-base