Philo MiniLM (v2)

An embedding model for replying to a message with a philosopher's quote. Write a question, a request for advice, some venting, or nonsense, and the nearest quote should be a good reply.

Off-the-shelf sentence embeddings match by topic: "I'm nervous about my exam" lands next to quotes about exams and knowledge. This model is all-MiniLM-L6-v2 fine-tuned so that "close" means "a fitting reply" instead.

  • 384-dimensional embeddings, mean-pooled over real tokens and L2-normalised
  • Inputs and quotes go through the same encoder; search is a dot product against pre-computed quote embeddings

v2 was trained on more than twice as much data as v1 and beats it on every mode except nonsense (see Evaluation). v1 is still available with revision="v1".

Usage

With transformers (this is exactly the pooling the model was trained with):

import torch
from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("otterbox901/philo-minilm")
model = AutoModel.from_pretrained("otterbox901/philo-minilm").eval()

def embed(texts):
    batch = tokenizer(texts, padding=True, truncation=True, max_length=256, return_tensors="pt")
    with torch.no_grad():
        hidden = model(**batch).last_hidden_state
    mask = batch["attention_mask"].unsqueeze(-1).float()
    mean = (hidden * mask).sum(1) / mask.sum(1).clamp(min=1e-9)
    return torch.nn.functional.normalize(mean, dim=-1)

quotes = ["One must imagine Sisyphus happy.", "We suffer more often in imagination than in reality."]
Q = embed(quotes)
scores = Q @ embed(["I'm dreading tomorrow's exam."])[0]
print(quotes[scores.argmax()])

With sentence-transformers, which defaults to the same mean pooling:

from sentence_transformers import SentenceTransformer
model = SentenceTransformer("otterbox901/philo-minilm")
emb = model.encode(["I'm dreading tomorrow's exam."], normalize_embeddings=True)

Both give the same vectors (cosine > 0.9999).

Training

Data. There was no dataset of (message โ†’ ideal quote) pairs, so one was generated. Claude Haiku was shown real quotes and asked to work backwards: write realistic messages that a given quote would be a perfect reply to, separately for four modes: question, advice, venting ("statement"), and nonsense. It also judged whether each quote fit each mode and rejected poor fits. Outputs were filtered for length, word overlap with the quote, and duplicates.

v2 adds a second generation round over new quotes. The two rounds together cover 7,721 distinct quotes and give 90,378 pairs:

split pairs distinct quotes
train 81,283 6,949
validation 9,095 772

Each round is split by quote on its own, so no quote appears in both splits, and round 1's validation quotes are the same ones v1 was validated on. By mode, the pairs are 33,088 question, 25,028 advice, 24,660 venting and 7,602 nonsense; few quotes make a fitting reply to nonsense. The training pairs are not published with this model.

Objective. InfoNCE with in-batch negatives: for each batch of 64 pairs, a 64 ร— 64 similarity matrix between messages and quotes (temperature 0.05), with cross-entropy pushing each message toward its own quote and away from the other 63. Batches never contain the same quote twice, which would create false negatives.

Hyperparameters. AdamW, learning rate 2e-5, linear warmup over the first 10% of steps then linear decay, batch size 64, max length 256 tokens. The schedule was set for 4 epochs (about 5,080 steps), and training stopped after epoch 2. This checkpoint is the end of epoch 2 (about 2,540 steps), when the learning rate had decayed to about 1.1e-5. The epoch 1 checkpoint scored lower (Recall@5 9.3%).

Evaluation

Validation messages (9,095) are matched against the full corpus of 34,793 quotes, and we look for the single quote each message was written for. Many other quotes could be fine replies, so these numbers understate real quality, but they compare models fairly.

base all-MiniLM-L6-v2 philo-minilm v1 philo-minilm v2
Recall@1 2.2% 3.6% 3.9%
Recall@5 5.6% 8.4% 10.0%
Recall@10 7.5% 11.8% 13.7%
MRR 0.041 0.065 0.074
Median rank of the intended quote 1,364 450 303

On Recall@5, v2 beats v1 by 1.5 points (95% CI +1.0 to +2.0, paired bootstrap over quotes).

Recall@5 by mode base v1 v2
question (3,353) 9.7% 14.8% 16.3%
advice (2,536) 3.4% 5.1% 6.9%
venting (2,441) 3.6% 5.5% 7.4%
nonsense (765) 0.9% 0.9% 0.5%

This validation set is larger than the one on v1's card (round 1's 4,321 pairs), so v1's numbers here differ a little from the ones it was published with.

Limitations

  • The encoder doesn't handle nonsense. For nonsense messages, v2's top quote is one of the 1,528 known absurd quotes 4.4% of the time, which is exactly their share of the corpus (v1: 18.2%). The Philo app works around this with a separate intent router, a frozen Qwen2.5-0.5B-Instruct plus a small trained head, that detects nonsense and limits the search to absurd quotes. The router is not part of this repository.
  • It still tends to echo, though less than v1. The top-ranked quote often restates the message instead of answering it. On the validation set the top pick shares 1.8ร— as many content words with the message as the intended reply does (v1: 2.1ร—). The in-batch negatives are random quotes, so topic matching is often enough to win; training with hard negatives (the quotes the base model wrongly ranks first) is the planned fix.
  • Trained on synthetic messages, so real messages with unusual phrasing may behave differently.
  • English only.
  • Retrieval quality depends on the quote collection you search; this repository contains only the encoder.
Downloads last month
13
Safetensors
Model size
22.7M params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for otterbox901/philo-minilm