TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning
Paper âĸ 2104.06979 âĸ Published
How to use T-Blue/tsdae_pro_MiniLM_L12_2 with sentence-transformers:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("T-Blue/tsdae_pro_MiniLM_L12_2")
sentences = [
"đā¤ā¤¨đđĸđ đā¤đĒā¤đ ⤠đĢđŖā¤ĒđŖ đ⤍đ ⤠đđŖđąā¤ ā¤ŦđĸđĒđ ā¤đ¯",
" ⤪⤠ā¤ŦđĸđĒđ ⤠ā¤Ēā¤đĒđĻ đŖā¤ đ ā¤đĢā¤đĸđ˛đĸ⤪ā¤đĒđŗā¤ đŖā¤ ā¤ā¤đđĻđđŗā¤ ā¤ā¤ā¤Ŗā¤đĻ đā¤đ ā¤đĒ ā¤Ŗā¤đŖđŖā¤ đ ā¤đĢā¤đĸđ˛đĸđđŗā¤ ⤪⤠ā¤ĸā¤đĒ đĸ⤪ā¤ā¤˛đĸđ¯",
" đŖā¤đā¤Ŧā¤đđĻ đŖā¤ đā¤ā¤¨đđĸđ đ đŖā¤Ēā¤đĒđĻ ā¤Ēā¤đ⤠đĸ⤪⤠đ¤ā¤đ ⤠ā¤ĸā¤ā¤ĸā¤ĸ⤠đđŖ đā¤đĒā¤đ ⤠đĸđŖā¤đ ā¤đ⤠đđąā¤ā¤Ēā¤đā¤Ē⤠đŖā¤ đ đŖā¤Ēā¤đĒ đŖā¤ā¤¨đā¤đĒ đĢđŖā¤ĒđŖ đŖā¤ đŗā¤¨ā¤đĻ đ⤍đ ⤠⤪⤠đ˛đĸ đ⤠đđŖđąā¤ ā¤ŦđĸđĒđ ā¤đ¯",
"ā¤Ēā¤đĒđĻđ đĸ ⤪⤠ā¤ĸ⤍ā¤Ŧ⤠đąā¤ ā¤ā¤¨đā¤Ŧđĸ⤪ā¤đĒ ā¤đąā¤ā¤˛ā¤˛đŖđ ā¤ā¤đ˛ā¤ ā¤Ē⤠ā¤ā¤ā¤˛đĸā¤ĸđĸđ ā¤ā¤đŗā¤đĒ đĸđĒā¤đ ⤠ā¤Ŧā¤đŗā¤đĒ ā¤Ē⤍đĒđđĸ⤪⤪⤠đ⤍đ ⤠⤪⤠⤤đĸ đąā¤ ā¤ā¤¨đā¤Ŧđĸ⤪ā¤đĒ đđąā¤ā¤˛ā¤˛ā¤ā¤ŖđĻ ā¤Ĩđ¯"
]
embeddings = model.encode(sentences)
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [4, 4]This is a sentence-transformers model finetuned from sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
SentenceTransformer(
(0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel
(1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
# Download from the đ¤ Hub
model = SentenceTransformer("T-Blue/tsdae_pro_MiniLM_L12_2")
# Run inference
sentences = [
'ā¤ŦđĢđŖđŗā¤Ē đĸđĸ đŗđĢđĸđđĻ đ ā¤đ˛đĸ đ ā¤đĢđĸđ đ ā¤đ⤤đĸđĻ ā¤Ēā¤đĸđ ā¤đđŖđ đŖā¤ đ˛ā¤đŗā¤ā¤˛ā¤¨ā¤˛ā¤˛ā¤¨đ⤠⤪ā¤đ⤠ā¤ĸ⤠đ ā¤đ¤ā¤ā¤¨đ⤠⤤đĸđđĸđ đĢā¤đđā¤ā¤˛đĸ ⤪ā¤ā¤Ŗđĸđ',
'ā¤đ đĸđā¤Ēā¤ā¤¤ā¤¤đĸ⤪⤠⤠⤤đĸđđĸđ ā¤ŦđĢđŖđŗā¤Ē đŗđĻđĒđĸđĻđŗ đĸđĸ đŗđĢđĸđđĻ đ ā¤đ˛đĸ đ ā¤đĢđĸđ đ ā¤đ⤤đĸđĻ ā¤Ēā¤đĒđĻ đŖā¤ ā¤đĸđ ā¤ĸđĸđ đĸđā¤Ŧā¤đā¤Ēā¤ā¤Ēā¤Ē⤍đ ā¤Ēđŗā¤đĒđĸđ ā¤Ēā¤đĸđ ā¤đđŖđ đŖđĸđĒđĻā¤ĸ⤠đŖā¤ đ˛ā¤đŗā¤ā¤˛ā¤¨ā¤˛ā¤˛ā¤¨đ⤠đ⤠ā¤đ đĸđ⤤đĸđĻ ā¤Ŗā¤đ⤠ā¤ĸ⤠đ ā¤đ¤ā¤ā¤¨đ⤠⤤đĸđđĸđ đđąā¤đ⤤đĸ⤪ā¤đĒ đĢā¤đđā¤ā¤˛đĸ ⤪ā¤ā¤Ŗđĸđ ā¤Ēā¤đ˛đĸ⤪ā¤đĒđŗā¤¨đ¯',
'ā¤ĒđŖā¤§đŗā¤Ŗ ⤧đĢđĸđĒđĸ đā¤đ đĢā¤đĸđ˛đĻ đŗđĢđĸ ⤠đĒā¤đā¤đĒ đđ ā¤Ŧ⤠đąā¤ā¤Ēā¤đ ā¤ā¤Ŧ⤍đŗā¤Ē⤠đā¤Ĩđđ§đŽ ā¤ā¤đ đąā¤đŗā¤đ ā¤ĸā¤đŖđ đĸđā¤ĒđŖđ ā¤ā¤đ đ¤ā¤đ ā¤ĸđĸ⤠đđĻđ¯',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
sentence_0 and sentence_1| sentence_0 | sentence_1 | |
|---|---|---|
| type | string | string |
| details |
|
|
| sentence_0 | sentence_1 |
|---|---|
đ⤍đŖā¤¨ ā¤ĸđĸđĒđđĸđđĻđ⤍đŗā¤ ā¤ĒđĻđ⤍đ |
ā¤ĒđĻđ⤍đ ā¤Ēā¤ā¤Ŧ⤠⤪ā¤đ⤠đ⤍đŖā¤¨ đŖā¤ ā¤ĸđĸđĒđđĸđđĻđ⤍đŗā¤ đŖā¤ ā¤ĒđĻđ⤍đ ā¤Ēā¤ā¤¤đĢđŖā¤Ŧā¤đ¯ |
⤠⤤đĸā¤ĸđĸ⤪đŖā¤Ŗđĸđ đŗā¤đŖā¤đĒđąā¤đĒ đŗā¤¨ ā¤ā¤đĒ⤠đ ā¤ā¤Ēđŗā¤ā¤Ŗđĸđ |
ā¤ā¤ĸđŖđā¤đĸđā¤đ ā¤đĒ ā¤ ā¤Ŗā¤đąā¤đ⤤đĸđ ⤤đĸā¤ĸđĸ⤪đŖā¤Ŗđĸđ đŗā¤đŖā¤đĒđąā¤đĒ đā¤đ ā¤đā¤đĻ đ ā¤đŗā¤¨ ā¤đ đ˛ā¤đđĸ đ¤ā¤ đŗā¤¨ đĸ⤪⤠ā¤ā¤đĒ⤠đ ⤍ā¤Ēā¤đđĻ ā¤ đ ā¤ā¤Ēđŗā¤ā¤Ŗđĸđ ā¤ā¤ĸđŖđā¤đđŗā¤¨đ¯ |
đŖā¤ ā¤Ŧ⤍đŖā¤¨đ đ ā¤đąā¤ đā¤đĒđĸđŖā¤¨đ đ ⤍đā¤ā¤˛ā¤˛ā¤¨ ā¤Ē⤠đ¯ |
ā¤Ē⤠ā¤ĸ⤠đŖā¤ ā¤Ŧ⤍đŖā¤¨đ đ ā¤đąā¤ ā¤Ŧ⤠đā¤đĒđĸđŖā¤¨đ ā¤đā¤đĒ⤤đĢđĸđŗā¤Ē đŖā¤ā¤ĸā¤đ⤎đŖā¤ā¤ĸā¤đ đŖā¤ đ ⤍đā¤ā¤˛ā¤˛ā¤¨ đ ā¤đŗā¤¨ ā¤ā¤˛ā¤ā¤ā¤ đŖā¤ ā¤ā¤¨đā¤Ŧđĸ⤪ā¤đĒ đ ā¤đā¤đĸđā¤ā¤Ē⤠đ⤪ā¤đ⤤đĸ ā¤Ē⤠đā¤đ ⤍đŗ đ¯ |
DenoisingAutoEncoderLossper_device_train_batch_size: 16per_device_eval_batch_size: 16multi_dataset_batch_sampler: round_robinoverwrite_output_dir: Falsedo_predict: Falseeval_strategy: noprediction_loss_only: Trueper_device_train_batch_size: 16per_device_eval_batch_size: 16per_gpu_train_batch_size: Noneper_gpu_eval_batch_size: Nonegradient_accumulation_steps: 1eval_accumulation_steps: Nonelearning_rate: 5e-05weight_decay: 0.0adam_beta1: 0.9adam_beta2: 0.999adam_epsilon: 1e-08max_grad_norm: 1num_train_epochs: 3max_steps: -1lr_scheduler_type: linearlr_scheduler_kwargs: {}warmup_ratio: 0.0warmup_steps: 0log_level: passivelog_level_replica: warninglog_on_each_node: Truelogging_nan_inf_filter: Truesave_safetensors: Truesave_on_each_node: Falsesave_only_model: Falserestore_callback_states_from_checkpoint: Falseno_cuda: Falseuse_cpu: Falseuse_mps_device: Falseseed: 42data_seed: Nonejit_mode_eval: Falseuse_ipex: Falsebf16: Falsefp16: Falsefp16_opt_level: O1half_precision_backend: autobf16_full_eval: Falsefp16_full_eval: Falsetf32: Nonelocal_rank: 0ddp_backend: Nonetpu_num_cores: Nonetpu_metrics_debug: Falsedebug: []dataloader_drop_last: Falsedataloader_num_workers: 0dataloader_prefetch_factor: Nonepast_index: -1disable_tqdm: Falseremove_unused_columns: Truelabel_names: Noneload_best_model_at_end: Falseignore_data_skip: Falsefsdp: []fsdp_min_num_params: 0fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}fsdp_transformer_layer_cls_to_wrap: Noneaccelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}deepspeed: Nonelabel_smoothing_factor: 0.0optim: adamw_torchoptim_args: Noneadafactor: Falsegroup_by_length: Falselength_column_name: lengthddp_find_unused_parameters: Noneddp_bucket_cap_mb: Noneddp_broadcast_buffers: Falsedataloader_pin_memory: Truedataloader_persistent_workers: Falseskip_memory_metrics: Trueuse_legacy_prediction_loop: Falsepush_to_hub: Falseresume_from_checkpoint: Nonehub_model_id: Nonehub_strategy: every_savehub_private_repo: Falsehub_always_push: Falsegradient_checkpointing: Falsegradient_checkpointing_kwargs: Noneinclude_inputs_for_metrics: Falseeval_do_concat_batches: Truefp16_backend: autopush_to_hub_model_id: Nonepush_to_hub_organization: Nonemp_parameters: auto_find_batch_size: Falsefull_determinism: Falsetorchdynamo: Noneray_scope: lastddp_timeout: 1800torch_compile: Falsetorch_compile_backend: Nonetorch_compile_mode: Nonedispatch_batches: Nonesplit_batches: Noneinclude_tokens_per_second: Falseinclude_num_input_tokens_seen: Falseneftune_noise_alpha: Noneoptim_target_modules: Nonebatch_eval_metrics: Falseeval_on_start: Falsebatch_sampler: batch_samplermulti_dataset_batch_sampler: round_robin| Epoch | Step | Training Loss |
|---|---|---|
| 0.125 | 500 | 2.5392 |
| 0.25 | 1000 | 1.4129 |
| 0.375 | 1500 | 1.3383 |
| 0.5 | 2000 | 1.288 |
| 0.625 | 2500 | 1.2627 |
| 0.75 | 3000 | 1.239 |
| 0.875 | 3500 | 1.2208 |
| 1.0 | 4000 | 1.2041 |
| 1.125 | 4500 | 1.1743 |
| 1.25 | 5000 | 1.1633 |
| 1.375 | 5500 | 1.1526 |
| 1.5 | 6000 | 1.1375 |
| 1.625 | 6500 | 1.1313 |
| 1.75 | 7000 | 1.1246 |
| 1.875 | 7500 | 1.1162 |
| 2.0 | 8000 | 1.1096 |
| 2.125 | 8500 | 1.0876 |
| 2.25 | 9000 | 1.0839 |
| 2.375 | 9500 | 1.0791 |
| 2.5 | 10000 | 1.0697 |
| 2.625 | 10500 | 1.0671 |
| 2.75 | 11000 | 1.0644 |
| 2.875 | 11500 | 1.0579 |
| 3.0 | 12000 | 1.0528 |
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
@inproceedings{wang-2021-TSDAE,
title = "TSDAE: Using Transformer-based Sequential Denoising Auto-Encoderfor Unsupervised Sentence Embedding Learning",
author = "Wang, Kexin and Reimers, Nils and Gurevych, Iryna",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2021",
month = nov,
year = "2021",
address = "Punta Cana, Dominican Republic",
publisher = "Association for Computational Linguistics",
pages = "671--688",
url = "https://arxiv.org/abs/2104.06979",
}