SentenceTransformer based on glossia/Rambley-V2.1-350M-Embed-Pretrain-Preview

This is a sentence-transformers model finetuned from glossia/Rambley-V2.1-350M-Embed-Pretrain-Preview on the nomic-embed-supervised-data dataset. It maps inputs to a 896-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: glossia/Rambley-V2.1-350M-Embed-Pretrain-Preview
  • Maximum Sequence Length: 256 tokens
  • Output Dimensionality: 896 dimensions
  • Similarity Function: Cosine Similarity
  • Supported Modality: Text
  • Training Dataset:
    • nomic-embed-supervised-data

Model Sources

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'RambleyV2Model'})
  (1): Pooling({'embedding_dimension': 896, 'pooling_mode': 'lasttoken', 'include_prompt': True})
  (2): Normalize({'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'The weather is lovely today.',
    "It's so sunny outside!",
    'He drove to the stadium.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 896]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.7007, 0.1692],
#         [0.7007, 1.0000, 0.2174],
#         [0.1692, 0.2174, 1.0000]])

Training Details

Training Dataset

nomic-embed-supervised-data

  • Dataset: nomic-embed-supervised-data
  • Columns: anchor, positive, negative_1, negative_2, negative_3, and negative_4
  • Loss: CachedMultipleNegativesRankingLoss with these parameters:
    {
        "scale": 20.0,
        "similarity_fct": "cos_sim",
        "mini_batch_size": 16,
        "mini_batch_num_tokens": null,
        "gather_across_devices": false,
        "directions": [
            "query_to_doc"
        ],
        "partition_mode": "joint",
        "hardness_mode": null,
        "hardness_strength": 0.0
    }
    

Training Hyperparameters

Non-Default Hyperparameters

  • per_device_train_batch_size: 32
  • max_steps: 52995
  • learning_rate: 2e-05
  • lr_scheduler_type: cosine
  • warmup_steps: 0.05
  • bf16: True

All Hyperparameters

Click to expand
  • per_device_train_batch_size: 32
  • num_train_epochs: 3.0
  • max_steps: 52995
  • learning_rate: 2e-05
  • lr_scheduler_type: cosine
  • lr_scheduler_kwargs: None
  • warmup_steps: 0.05
  • optim: adamw_torch_fused
  • optim_args: None
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • optim_target_modules: None
  • gradient_accumulation_steps: 1
  • average_tokens_across_devices: True
  • max_grad_norm: 1.0
  • label_smoothing_factor: 0.0
  • bf16: True
  • fp16: False
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • use_liger_kernel: False
  • liger_kernel_config: None
  • use_cache: False
  • neftune_noise_alpha: None
  • torch_empty_cache_steps: None
  • auto_find_batch_size: False
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • include_num_input_tokens_seen: no
  • log_level: passive
  • log_level_replica: warning
  • disable_tqdm: False
  • project: huggingface
  • trackio_space_id: None
  • trackio_bucket_id: None
  • trackio_static_space_id: None
  • per_device_eval_batch_size: 8
  • prediction_loss_only: True
  • eval_on_start: False
  • eval_do_concat_batches: True
  • eval_use_gather_object: False
  • eval_accumulation_steps: None
  • include_for_metrics: []
  • batch_eval_metrics: False
  • save_only_model: False
  • save_on_each_node: False
  • enable_jit_checkpoint: False
  • push_to_hub: False
  • hub_private_repo: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_always_push: False
  • hub_revision: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • restore_callback_states_from_checkpoint: False
  • full_determinism: False
  • seed: 42
  • data_seed: None
  • use_cpu: False
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • parallelism_config: None
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • dataloader_prefetch_factor: None
  • dataloader_multiprocessing_context: None
  • dataloader_in_order: True
  • remove_unused_columns: True
  • label_names: None
  • train_sampling_strategy: random
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • ddp_static_graph: None
  • ddp_backend: None
  • ddp_timeout: 1800
  • fsdp: None
  • fsdp_config: None
  • deepspeed: None
  • debug: []
  • skip_memory_metrics: True
  • do_predict: False
  • resume_from_checkpoint: None
  • local_rank: -1
  • prompts: None
  • batch_sampler: batch_sampler
  • multi_dataset_batch_sampler: proportional
  • router_mapping: {}
  • learning_rate_mapping: {}
  • warmup_ratio: None

Training Logs

Epoch Step Training Loss
0.0000 1 1.0533
0.0009 50 0.8743
0.0019 100 0.9105
0.0028 150 0.8812
0.0038 200 0.8130
0.0047 250 0.9069
0.0057 300 0.7909
0.0066 350 0.8640
0.0075 400 0.9070
0.0085 450 0.8484
0.0094 500 0.8258
0.0104 550 0.7800
0.0113 600 0.8058
0.0123 650 0.7752
0.0132 700 0.8554
0.0142 750 0.8217
0.0151 800 0.7837
0.0160 850 0.7977
0.0170 900 0.7971
0.0179 950 0.7223
0.0189 1000 0.7383
0.0198 1050 0.7593
0.0208 1100 0.7836
0.0217 1150 0.7243
0.0226 1200 0.7810
0.0236 1250 0.7548
0.0245 1300 0.7434
0.0255 1350 0.6868
0.0264 1400 0.6021
0.0274 1450 0.6691
0.0283 1500 0.6651
0.0292 1550 0.6644
0.0302 1600 0.6379
0.0311 1650 0.6582
0.0321 1700 0.6690
0.0330 1750 0.6551
0.0340 1800 0.6355
0.0349 1850 0.6240
0.0359 1900 0.6293
0.0368 1950 0.6049
0.0377 2000 0.6135
0.0387 2050 0.6030
0.0396 2100 0.6402
0.0406 2150 0.5972
0.0415 2200 0.6621
0.0425 2250 0.6350
0.0434 2300 0.6405
0.0443 2350 0.6000
0.0453 2400 0.6414
0.0462 2450 0.6136
0.0472 2500 0.6305

Training Time

  • Training: 1.3 hours

Framework Versions

  • Python: 3.13.11
  • Sentence Transformers: 6.2.0
  • Transformers: 5.22.1
  • PyTorch: 2.11.0+cu130
  • Accelerate: 1.15.0
  • Datasets: 5.1.0
  • Tokenizers: 0.23.3

Additional Resources

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}
Downloads last month
46
Safetensors
Model size
0.3B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for glossia/Rambley-V2.1-350M-Embed-Preview

Papers for glossia/Rambley-V2.1-350M-Embed-Preview