Sentence-Llama-Bi-EN-1B

English / Japanese

Overview

Sentence-Llama-Bi-EN-1B fine-tunes iamtatsuki05/Sentence-Llama-Bi-EN-1B-PT with supervised contrastive learning on the supervised subsets of sentence-transformers/embedding-training-data (AllNLI, MS MARCO, Quora duplicate questions, NQ, SQuAD, and TriviaQA), yielding 1,792-dimensional English sentence embeddings.

Consept

Usage

Requirements

sentence-transformers>=4.1.0
transformers>=4.51.0
accelerate>=1.6.0
sentencepiece>=0.2.0
flash-attn>=2.7.3

Sample Code

import torch
from sentence_transformers import SentenceTransformer

model_name = "iamtatsuki05/Sentence-Llama-Bi-EN-1B"
model_kwargs = {
  "torch_dtype": torch.bfloat16,
  "attn_implementation": "flash_attention_2",
}
model = SentenceTransformer(model_name, model_kwargs=model_kwargs, trust_remote_code=True)

queries = ["What kind of character is Hachiware?"]
docs = [
    "Hachiware is a cat-like character from Chiikawa with a bright, sociable, and positive personality. Hachiware enjoys everyday life with Chiikawa and friends and eagerly joins adventures such as subjugation missions.",
    "Usagi is a carefree, free-spirited character whose bold actions and sharp intuition pull Chiikawa and friends along, while still caring deeply about friends.",
]
q_emb = model.encode(queries, normalize_embeddings=True)
d_emb = model.encode(docs, normalize_embeddings=True)
scores = model.similarity(q_emb, d_emb)
print(scores)

Model Details

  • Base model: iamtatsuki05/Sentence-Llama-Bi-EN-1B-PT
  • Architecture: Llama
  • Maximum sequence length: 8,192 tokens
  • Embedding dimension: 1792 (mean pooling)
  • Tokenizer: SentencePiece / vocabulary size 102,400
  • Positional encoding: RoPE
  • Supported languages: English
  • Similarity metric: cosine

Model Series

These encoders start from weakly supervised checkpoints and undergo supervised fine-tuning on the supervised subsets of sentence-transformers/embedding-training-data.

ID Architecture #Param. #Param.
w/o Emb.
MTEB-Avg MTEB-Retrieval MTEB-STS MTEB-Classification MTEB-Reranking MTEB-Clustering
iamtatsuki05/Sentence-ModernBERT-EN-1B ModernBERT 1304M 1120M 54.76 42.27 76.72 68.47 43.87 42.46
iamtatsuki05/Sentence-Llama-Bi-EN-1B
(this model)
Llama 1224M 1040M 52.94 39.04 75.45 66.46 42.47 41.28
iamtatsuki05/Sentence-Sarashina-Bi-EN-1B Llama 1224M 1040M 57.56 44.90 79.78 73.48 45.98 43.64

Scores are measured on the 37-task English subset of MTEB (eng, v2) that covers the same five task families as JMTEB. Each task-family score averages all datasets, splits, and subsets in the family, and MTEB-Avg is the macro average over the five families.

Licence

This model is distributed under the MIT License.

How to Cite

@article{MIREI
  title={同一条件下における Encoder/Decoder アーキテクチャによる文埋め込みの性能分析},
  author={岡田 龍樹 and 杉本 徹},
  journal={言語処理学会第 32 回年次大会 (NLP2026)},
  year={2026}
}
Downloads last month
21
Safetensors
Model size
1B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for iamtatsuki05/Sentence-Llama-Bi-EN-1B

Dataset used to train iamtatsuki05/Sentence-Llama-Bi-EN-1B

Collection including iamtatsuki05/Sentence-Llama-Bi-EN-1B