Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup
Paper • 2101.06983 • Published • 3
How to use glossia/Rambley-V2.1-350M-Embed-Preview with sentence-transformers:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("glossia/Rambley-V2.1-350M-Embed-Preview", trust_remote_code=True)
sentences = [
"That is a happy person",
"That is a happy dog",
"That is a very happy person",
"Today is a sunny day"
]
embeddings = model.encode(sentences)
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [4, 4]This is a sentence-transformers model finetuned from glossia/Rambley-V2.1-350M-Embed-Pretrain-Preview on the nomic-embed-supervised-data dataset. It maps inputs to a 896-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.
SentenceTransformer(
(0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'RambleyV2Model'})
(1): Pooling({'embedding_dimension': 896, 'pooling_mode': 'lasttoken', 'include_prompt': True})
(2): Normalize({'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
'The weather is lovely today.',
"It's so sunny outside!",
'He drove to the stadium.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 896]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.7007, 0.1692],
# [0.7007, 1.0000, 0.2174],
# [0.1692, 0.2174, 1.0000]])
anchor, positive, negative_1, negative_2, negative_3, and negative_4CachedMultipleNegativesRankingLoss with these parameters:{
"scale": 20.0,
"similarity_fct": "cos_sim",
"mini_batch_size": 16,
"mini_batch_num_tokens": null,
"gather_across_devices": false,
"directions": [
"query_to_doc"
],
"partition_mode": "joint",
"hardness_mode": null,
"hardness_strength": 0.0
}
per_device_train_batch_size: 32max_steps: 52995learning_rate: 2e-05lr_scheduler_type: cosinewarmup_steps: 0.05bf16: Trueper_device_train_batch_size: 32num_train_epochs: 3.0max_steps: 52995learning_rate: 2e-05lr_scheduler_type: cosinelr_scheduler_kwargs: Nonewarmup_steps: 0.05optim: adamw_torch_fusedoptim_args: Noneweight_decay: 0.0adam_beta1: 0.9adam_beta2: 0.999adam_epsilon: 1e-08optim_target_modules: Nonegradient_accumulation_steps: 1average_tokens_across_devices: Truemax_grad_norm: 1.0label_smoothing_factor: 0.0bf16: Truefp16: Falsebf16_full_eval: Falsefp16_full_eval: Falsetf32: Nonegradient_checkpointing: Falsegradient_checkpointing_kwargs: Nonetorch_compile: Falsetorch_compile_backend: Nonetorch_compile_mode: Noneuse_liger_kernel: Falseliger_kernel_config: Noneuse_cache: Falseneftune_noise_alpha: Nonetorch_empty_cache_steps: Noneauto_find_batch_size: Falselog_on_each_node: Truelogging_nan_inf_filter: Trueinclude_num_input_tokens_seen: nolog_level: passivelog_level_replica: warningdisable_tqdm: Falseproject: huggingfacetrackio_space_id: Nonetrackio_bucket_id: Nonetrackio_static_space_id: Noneper_device_eval_batch_size: 8prediction_loss_only: Trueeval_on_start: Falseeval_do_concat_batches: Trueeval_use_gather_object: Falseeval_accumulation_steps: Noneinclude_for_metrics: []batch_eval_metrics: Falsesave_only_model: Falsesave_on_each_node: Falseenable_jit_checkpoint: Falsepush_to_hub: Falsehub_private_repo: Nonehub_model_id: Nonehub_strategy: every_savehub_always_push: Falsehub_revision: Noneload_best_model_at_end: Falseignore_data_skip: Falserestore_callback_states_from_checkpoint: Falsefull_determinism: Falseseed: 42data_seed: Noneuse_cpu: Falseaccelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}parallelism_config: Nonedataloader_drop_last: Falsedataloader_num_workers: 0dataloader_pin_memory: Truedataloader_persistent_workers: Falsedataloader_prefetch_factor: Nonedataloader_multiprocessing_context: Nonedataloader_in_order: Trueremove_unused_columns: Truelabel_names: Nonetrain_sampling_strategy: randomlength_column_name: lengthddp_find_unused_parameters: Noneddp_bucket_cap_mb: Noneddp_broadcast_buffers: Falseddp_static_graph: Noneddp_backend: Noneddp_timeout: 1800fsdp: Nonefsdp_config: Nonedeepspeed: Nonedebug: []skip_memory_metrics: Truedo_predict: Falseresume_from_checkpoint: Nonelocal_rank: -1prompts: Nonebatch_sampler: batch_samplermulti_dataset_batch_sampler: proportionalrouter_mapping: {}learning_rate_mapping: {}warmup_ratio: None| Epoch | Step | Training Loss |
|---|---|---|
| 0.0000 | 1 | 1.0533 |
| 0.0009 | 50 | 0.8743 |
| 0.0019 | 100 | 0.9105 |
| 0.0028 | 150 | 0.8812 |
| 0.0038 | 200 | 0.8130 |
| 0.0047 | 250 | 0.9069 |
| 0.0057 | 300 | 0.7909 |
| 0.0066 | 350 | 0.8640 |
| 0.0075 | 400 | 0.9070 |
| 0.0085 | 450 | 0.8484 |
| 0.0094 | 500 | 0.8258 |
| 0.0104 | 550 | 0.7800 |
| 0.0113 | 600 | 0.8058 |
| 0.0123 | 650 | 0.7752 |
| 0.0132 | 700 | 0.8554 |
| 0.0142 | 750 | 0.8217 |
| 0.0151 | 800 | 0.7837 |
| 0.0160 | 850 | 0.7977 |
| 0.0170 | 900 | 0.7971 |
| 0.0179 | 950 | 0.7223 |
| 0.0189 | 1000 | 0.7383 |
| 0.0198 | 1050 | 0.7593 |
| 0.0208 | 1100 | 0.7836 |
| 0.0217 | 1150 | 0.7243 |
| 0.0226 | 1200 | 0.7810 |
| 0.0236 | 1250 | 0.7548 |
| 0.0245 | 1300 | 0.7434 |
| 0.0255 | 1350 | 0.6868 |
| 0.0264 | 1400 | 0.6021 |
| 0.0274 | 1450 | 0.6691 |
| 0.0283 | 1500 | 0.6651 |
| 0.0292 | 1550 | 0.6644 |
| 0.0302 | 1600 | 0.6379 |
| 0.0311 | 1650 | 0.6582 |
| 0.0321 | 1700 | 0.6690 |
| 0.0330 | 1750 | 0.6551 |
| 0.0340 | 1800 | 0.6355 |
| 0.0349 | 1850 | 0.6240 |
| 0.0359 | 1900 | 0.6293 |
| 0.0368 | 1950 | 0.6049 |
| 0.0377 | 2000 | 0.6135 |
| 0.0387 | 2050 | 0.6030 |
| 0.0396 | 2100 | 0.6402 |
| 0.0406 | 2150 | 0.5972 |
| 0.0415 | 2200 | 0.6621 |
| 0.0425 | 2250 | 0.6350 |
| 0.0434 | 2300 | 0.6405 |
| 0.0443 | 2350 | 0.6000 |
| 0.0453 | 2400 | 0.6414 |
| 0.0462 | 2450 | 0.6136 |
| 0.0472 | 2500 | 0.6305 |
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
@misc{gao2021scaling,
title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
year={2021},
eprint={2101.06983},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
@misc{oord2019representationlearningcontrastivepredictive,
title={Representation Learning with Contrastive Predictive Coding},
author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
year={2019},
eprint={1807.03748},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/1807.03748},
}
Base model
glossia/Rambley-V2.1-350M-Pretrain