robzchhangte/e5-base-mizo

SentenceTransformer based on intfloat/multilingual-e5-base

This is a sentence-transformers model finetuned from intfloat/multilingual-e5-base. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: intfloat/multilingual-e5-base
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'query: India ramah khawi hmunah nge Avian Influenza hi February 18, 2006 khan hmuhchhuah hmasak ber?',
    'passage: Tin, Prion natna pawh North America, Europe leh Asia- ah a darh nasa hle a, UK-ah phei chuan he natna vei zaa zain an thihpui hial a ni. (Source: Merk Veterinary Manual 9th Edition). Tun kum zabi sawmhnih pakhatnaah hian mihring kan pun chak ang bawkin zoonoses te hi an pung chak hle mai a, hemi vênna kawnga ram hrang hrang- ten hmalakna atana an hman e.g. Molecular biology, Biotechnology, Nanotechnology, Infor- mation technology leh Geospatial technology te pawh tihphuisui zel an ni. Kan sawi tâk ang khan zoonoses natna langsar tak tak sawi tur tam tak awm mah se, kan sawi kim vek seng dawn lova, chuvangin a langsar zual tlem i lo tarlang ila: 1) Avian Influenza (Bird flu H5N1) : India ramah pawh ni 18 February, 2006 khan Nava-pur, Maharastra State-ah hmuh chhuah a ni a. Avian Influenza hi kum 1900 hma lam daih tawh khan ar tam takin an lo thihpui tawh a ni. Khawvel ram 16 aṭanga WHO-in a report-ah chuan kum 2003 March leh 2015 March inkarah he natna vei mi 826 hmuh chhuah niin mi 440- in an thihpui a ni. He natna laka invenna awmchhun chu Avian influenza natna kai ar suat emaw, a dam lai a Vaccine pêk emaw hi a ni. 2) Rabies: A natna thlentu hi RNA virus a ni a. India ramah kum sangnga kal taah khan an lo hmu chhuak tawh a ni.',
    'passage: MIZORAM CHIEF SECRETARY HOVIN COVID -19 DINHMUN THLIRHONA NEIH A NI 3158/2022-2023 Aizawl 29th December 2022 : COVID-19 hrik chhuakthar chung changa in buatsaihna kal zel chu vawiin chhun khan Mizoram Chief Secretary Renu Sharma hovin Health & Family Department a hotute nen thlirhona neih a ni a; he thutkhawmah hian tuna COVID-19 Mizoram dinhmun leh December ni 27 a district hrang hranga Mock Drill neih report te bawhzui niin tun dinhmunah chuan han chiai leh nunphung pangngai pela kal a tulna a la awm rih lovah he Meeting hian a ngai. Khawvel hmun hrang hrang Covid hrik thar BF.7 in a run thar mek laiin India ram dinhmun chu a la ziaawm hle a; Mizoramah pawh he virus thar hi la lut lovin December ni 13 atang khan vawiin thlengin Mizoramah COVID-19 kai thar hmuhchhuah an awm lo a ni. India ramah COVID hrik thar BF.7 hi a lo luh tawh avang leh a lenna ram atanga khualzinten khawi state pawh tlawh pawh theih reng a nih avangin COVID -19 laka invenna hrang hrang mask vuah, midang hnaih loh leh kut tihfai ngun erawh tha taka kalpui zel a tul an ti. Attachment Loading attachment...',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.5476, 0.0947],
#         [0.5476, 1.0000, 0.1114],
#         [0.0947, 0.1114, 1.0000]])

Training Details

Training Dataset

Unnamed Dataset

• Size: 17,581 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 100 samples:

  	anchor	positive	negative
  type	string	string	string
  modality	text	text	text
  details	min: 15 tokens mean: 27.64 tokens max: 40 tokens	min: 90 tokens mean: 322.55 tokens max: 512 tokens	min: 111 tokens mean: 361.2 tokens max: 512 tokens

• Samples:

  anchor	positive	negative
  query: Jamal Khashoggi chungchangah hian eng news organization nge Mohammed bin Salman interview?	passage: "Heng zawng zawng hi a tawp tur angin han ngaihtuah chhin teh u. Chumi awmzia chu khawvel economy a tluchhe dawn tihna a ni a, Saudi Arabia emaw Middle east rate chauh emaw ni loin khawvel pum a nghawng dawn a ni," a ti.Iran chu dem chhunzawm leh bawkin an chetdan chu a atthlak hle tih a sawi. Jamal Khashoggi thihnaah mawh a phur ṭhen tih sawiMohammed bin Salman hi Pathianni khan CBS News chuan a interview a, hetah hian Iran chung chang a sawichhuah bakah Saudi journalist Jamal Khashoggi thah a nih chungchang a sawi bawk. He interview-naah hian Salman hi Jamal Khashoggi that turin thu a pe em tih zawhna an zawt a ni.Saudi crown prince chuan that tura thupe nia an puhna chu pha in, a hunlai khan engmah a hriat loh thu a sawi. Mahse Khashoggi thihnaah chuan Saudi Arabia ram hruaitu a nihnaṭangin mawh a phur tih sawiin, a bik takin a thattu chu Saudi sorkar hnuaia hnathawk a nih avangin mawhphurtu nia a inhriat thu a sawi a ni.Khashoggi hi nikum October 2 khan Turkey-a Saudi Arab...	passage: Saudi crown prince-in Iran khap Ṭul a ti Saudi crown prince Mohammed bin Salman chuan Iran chu khaptu an awm a nih loh chuan khawvel-a oil man chu nasa takin a sang thei dawn tih a sawi.Mohammed bin Salman chuan hmalakna a awm lo a nih chuan Iran chu a huangtau telh telh ang a, indona a chhuak thei mai dawn a ni, a ti. Chutiang a nih chuan khawvel economy-in nasa takin a tuar dawn niin a sawi bawk.Salman hian an oil facility pahnih beihnaah chuan Iran chu mawhphurtuah a puh ve bawk a, mahse Iran chuan Saudi crown prince thusawi chuan zahna bak engmah a thlen lo ang, tiin a chhanglet ve thung.Saudi lal la ni mai tur chuan khawvelin Iran lakah hma a la lo a nih chuan khawvel chuan harsatna nasa zawk a tawk dawn a ni tia sawiin, oil supply chu nasa takin a buai dawn bakah nasa takin a man chu a sang chho dawrh dawn a ni, a ti bawk. Middle East region chuan khawvel energy supply 30% zet an tum phak tih leh, global trade passage 20% leh khawvel pum GDP 4% chu an tum phak tih a sawi...
  query: Mizoram sorkarin journalist-te a ngai pawimawh tih tuin nge sawi?	passage: Sorkarin journalist-te a ngai pawimawh - Ruatkima MJA chuan nimin khan Aizawl Press Club-ah tun hnaia Parliament House-a training-a kal an member mi 15 leh IPR department officer pangate pualin zin report pek inkhawm an buatsaih a; khuallian, IPR minister Lalruatkima chuan, Mizoram sorkarin thuthar thehdarhtu (journalist)-te a ngaih pawimawh thu a sawi.IPR Minister chuan, chief minister Zoramthanga'n journalist-te a ngaih pawimawh thu sawiin, "Tun ṭum zinna atan leh journalist-te pawimawh dang atan Chief Minister hnenah Finance changtu a nih angin pawisa kan dil a, ani chuan 'Journalist-te chu an pawimawh a, anni tana kan tih ve theihah chuan theihtawp kan chhuah tur alawm' a ti a ni. Sorkarin thuthar thehdarhtute pawimawhzia a hria a, a ngai pawimawh a ni," a ti.Sorkar thu leh hla mipuite hriattir chu journalist-te hna pakhat a nih thu sawiin Lalruatkima chuan, "Sorkar kal dik lohnaah pawh min sawisel a, kawng dik min kawhhmuhtu tur in ni," a ti a; journalist-te chu hleih nei...	passage: Mizoram leh phai lam journalist-te inkâwm: Mizoram sorkar aiawhte'n biak an harsat ṭhin thu sawi chhuak Mizoram leh Assam inkara buaina thleng a hmuna zir chiang tura rawn kal phai lama journalist-te chu nimin khan Aizawl Press Club-ah MJA huaihawtin Mizo journalist-te nen an inkâwm. Mizoram chanchin thar inhrilh tawnnaa thawhhona leh inpawhna ṭha zawk a awm theih dan turte leh pawimawh dangte an sawiho a; phai lam journalist-te chuan Mizoram sorkar aiawhte biak pawh an harsat ṭhin thu an sawi chhuak. Inkawmhonaah hi Mizoram Journalists' Association Gen.Hqrs president Zonunsanga Khiangte-in a kaihruai a; tun hnaia Mizoram-Assam ramria thil thlenga thil thlen dan dik tak hre chiang tura a buzáwl ngeia luhchilhtu national media-te chu Mizo journalist-te'n an lo lawm tih sawiin, "Mizoram hi ram kilkhâwra awm kan nih avangin a ram leh a chhunga chêngte chanchin hi ram chhung hmun danga media lian zawkte'n an hre chiang tâwk lo fo a, hei hi tuna national media rawn kalte'n an rawn ...
  query: Thawhlehni khan Mali sorkar leh helho an inbeihnaah hian sipai engzat nge thi?	passage: Mali inbeihna thlengah mi 25 thi Mali sorkar chuan Thawhṭannia firfiakte beih an tawhnaah khan an sipai 25 an thih bakah midang 60 chu chin hriat lohin an awm tih an sawi. Thawhṭanni khan Mali-a Boulkessy leh Mondoro-a army outpost-te chu firfiakte hian an bei a ni.Sorkar chuan an sipaite chuan firfiak 15 an kap hlum ve tih sawiin, an ralthuam erawh an hloh nual tih an sawi bawk. Beih an tawhna hmunah hian Burkina Faso leh French force ṭangkawpte chuan beihpui an thlak chhunzawm a ni.Mali hian 2012-a Islamist firfiakten an ram hmar lam an thununa France-in a sipaite a tirhluh aṭang khan jihadist-te aṭangin beih an tawk fo a, tualchhunga hnam hrangte inkarah buaina a chhuak fo bawk.	passage: France airstrike-ah firfiak 50 chuang thi France sorkar chuan an sipaiten central Mali-a airstrike an neihah Al-Qaida nena inzawm, firfiak 50 chuang an thi tih a sawi. Sorkar thuchhuah chuan sipaite hian kar liam ta Zirtawpni khan Burnina Faso leh Niger ramri bulah sipaite hian Islamic firfiakte beihpui an thlak tih a tarlang. France defence minister Florence Parly chuan Mali sorkai aiawhte nena meeting an neih hnuah sipaite beihpui thlak, Operation Barkhane tih hi neih a nih thu sawiin, firfiak 50 chuang an thih bakah an ralthuam eng emaw zat mansak an ni bawk a ni, a ti. Beihpui thlaknaah hian motorcycle 30 vel tihchhiah a ni tih a sawi bawk. Sipai thupuangtu Colonel Frederic Barbry chuan firfiak pali man an ni bawk tih sawiin, firfiak an beite hian sipai awmna hmun beih an tum a ni, a ti. Barbry chuan Greater Sahara-ah pawh beihpui thlak mek a ni tih sawiin, sipai 3,000 vel an tel tih a sawi.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• optim: adamw_torch_fused
• gradient_accumulation_steps: 8
• bf16: True
• data_seed: 42
• warmup_ratio: 0.0

All Hyperparameters

Click to expand

• per_device_train_batch_size: 8
• num_train_epochs: 3
• max_steps: -1
• learning_rate: 5e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.0
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 8
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: 42
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: 0.0
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss
0.0036	1	2.0179
0.0910	25	0.6676
0.1820	50	0.3871
0.2730	75	0.3210
0.3640	100	0.2534
0.4550	125	0.2307
0.5460	150	0.2404
0.6369	175	0.2269
0.7279	200	0.2123
0.8189	225	0.1830
0.9099	250	0.1808
1.0	275	0.1808
1.0910	300	0.0883
1.1820	325	0.1049
1.2730	350	0.0842
1.3640	375	0.0942
1.4550	400	0.0824
1.5460	425	0.0984
1.6369	450	0.0898
1.7279	475	0.0796
1.8189	500	0.0835
1.9099	525	0.0824
2.0	550	0.0799
2.0910	575	0.0363
2.1820	600	0.0418
2.2730	625	0.0405
2.3640	650	0.0395
2.4550	675	0.0525
2.5460	700	0.0404
2.6369	725	0.0413
2.7279	750	0.0438
2.8189	775	0.0392
2.9099	800	0.0355
3.0	825	0.0404

Training Time

• Training: 51.0 minutes

Framework Versions

• Python: 3.10.20
• Sentence Transformers: 5.5.1
• Transformers: 5.9.0
• PyTorch: 2.6.0+cu124
• Accelerate: 1.14.0
• Datasets: 4.8.5
• Tokenizers: 0.23.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}