simcse-snli-sup

A sentence-embedding model trained with supervised SimCSE (Gao, Yao & Chen, EMNLP 2021) starting from bert-base-uncased, as part of the course assignment U2T02 (replicating SimCSE on a small SNLI subset). Trained by: Ivan Esaú Palomo García, Dalila de los Angeles Ku Dzul

It maps a sentence to a 768-d vector (plain [CLS] pooling); compare sentences with cosine similarity.

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("Dalila-Ku/simcse-snli-sup")
emb = model.encode(["A man is playing a guitar.", "Someone plays an instrument."], normalize_embeddings=True)
print(emb[0] @ emb[1])

Training data

the 33,351 (premise, entailment-hypothesis) pairs of snli_train_100k.jsonl (a 100,000-record subsample of the official SNLI train split, no-consensus records dropped). For 9,488 pairs (28.4 %) a contradiction hypothesis of the same premise is added as hard negative (Eq. 5 of the paper). Not the paper's data (1M Wikipedia sentences / SNLI+MNLI), so the numbers below are lower than the paper's.

Recipe

Supervised SimCSE (Eq. 5): entailment pairs are positives, other hypotheses in the batch and the contradiction hypotheses are negatives.

hyper-parameter	value
base encoder	bert-base-uncased
pooling during training	cls (MLP discarded at inference; inference = plain [CLS])
learning rate / batch size / epochs	5e-05 / 128 / 3
temperature τ	0.05
dropout (hidden and attention)	0.1
max sequence length (training)	32
warmup / weight decay	0.0 / 0.0
precision	fp16 (loss computed in fp32)
seed	43
checkpoint	best STS-B dev Spearman (step 50)
hardware	Tesla T4 (6.56 min)

Results (STS-B, cosine similarity, Spearman ×100, no regressor)

split	Spearman
dev (1,500 pairs; used to pick the checkpoint, so optimistic)	82.27
test (1,379 pairs; evaluated once)	77.75

Alignment (STS-B dev pairs rated ≥ 4/5): 0.1853; uniformity (t=2): -3.0021 (lower is better for both).

Limitations

• Trained on SNLI sentences only (image captions written by crowd-workers: short, simple, concrete). Expect weaker behaviour on long, formal or domain-specific text.
• Trained on a ~33k-pair subset; the paper uses ~1M sentences / ~314k NLI pairs, so scores are lower than the published SimCSE.
• English only. Evaluated only on STS-B; no claim about retrieval, clustering or classification quality.
• Only 28.4 % of the training pairs carry a hard negative.
• SNLI contains annotation artifacts and social biases from its crowd-sourced captions; these are inherited by the embeddings.
• Cosine similarity values are not calibrated probabilities; use them for ranking.