Instructions to use otterbox901/philo-minilm with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use otterbox901/philo-minilm with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("otterbox901/philo-minilm") model = AutoModel.from_pretrained("otterbox901/philo-minilm", device_map="auto") - sentence-transformers
How to use otterbox901/philo-minilm with sentence-transformers:
from sentence_transformers import SentenceTransformer model = SentenceTransformer("otterbox901/philo-minilm") sentences = [ "That is a happy person", "That is a happy dog", "That is a very happy person", "Today is a sunny day" ] embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [4, 4] - Notebooks
- Google Colab
- Kaggle
Philo MiniLM (v2)
An embedding model for replying to a message with a philosopher's quote. Write a question, a request for advice, some venting, or nonsense, and the nearest quote should be a good reply.
Off-the-shelf sentence embeddings match by topic: "I'm nervous about my exam" lands next to quotes about exams and knowledge. This model is all-MiniLM-L6-v2 fine-tuned so that "close" means "a fitting reply" instead.
- 384-dimensional embeddings, mean-pooled over real tokens and L2-normalised
- Inputs and quotes go through the same encoder; search is a dot product against pre-computed quote embeddings
v2 was trained on more than twice as much data as v1 and beats it on every mode except nonsense (see Evaluation). v1 is still available with revision="v1".
Usage
With transformers (this is exactly the pooling the model was trained with):
import torch
from transformers import AutoModel, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("otterbox901/philo-minilm")
model = AutoModel.from_pretrained("otterbox901/philo-minilm").eval()
def embed(texts):
batch = tokenizer(texts, padding=True, truncation=True, max_length=256, return_tensors="pt")
with torch.no_grad():
hidden = model(**batch).last_hidden_state
mask = batch["attention_mask"].unsqueeze(-1).float()
mean = (hidden * mask).sum(1) / mask.sum(1).clamp(min=1e-9)
return torch.nn.functional.normalize(mean, dim=-1)
quotes = ["One must imagine Sisyphus happy.", "We suffer more often in imagination than in reality."]
Q = embed(quotes)
scores = Q @ embed(["I'm dreading tomorrow's exam."])[0]
print(quotes[scores.argmax()])
With sentence-transformers, which defaults to the same mean pooling:
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("otterbox901/philo-minilm")
emb = model.encode(["I'm dreading tomorrow's exam."], normalize_embeddings=True)
Both give the same vectors (cosine > 0.9999).
Training
Data. There was no dataset of (message โ ideal quote) pairs, so one was generated. Claude Haiku was shown real quotes and asked to work backwards: write realistic messages that a given quote would be a perfect reply to, separately for four modes: question, advice, venting ("statement"), and nonsense. It also judged whether each quote fit each mode and rejected poor fits. Outputs were filtered for length, word overlap with the quote, and duplicates.
v2 adds a second generation round over new quotes. The two rounds together cover 7,721 distinct quotes and give 90,378 pairs:
| split | pairs | distinct quotes |
|---|---|---|
| train | 81,283 | 6,949 |
| validation | 9,095 | 772 |
Each round is split by quote on its own, so no quote appears in both splits, and round 1's validation quotes are the same ones v1 was validated on. By mode, the pairs are 33,088 question, 25,028 advice, 24,660 venting and 7,602 nonsense; few quotes make a fitting reply to nonsense. The training pairs are not published with this model.
Objective. InfoNCE with in-batch negatives: for each batch of 64 pairs, a 64 ร 64 similarity matrix between messages and quotes (temperature 0.05), with cross-entropy pushing each message toward its own quote and away from the other 63. Batches never contain the same quote twice, which would create false negatives.
Hyperparameters. AdamW, learning rate 2e-5, linear warmup over the first 10% of steps then linear decay, batch size 64, max length 256 tokens. The schedule was set for 4 epochs (about 5,080 steps), and training stopped after epoch 2. This checkpoint is the end of epoch 2 (about 2,540 steps), when the learning rate had decayed to about 1.1e-5. The epoch 1 checkpoint scored lower (Recall@5 9.3%).
Evaluation
Validation messages (9,095) are matched against the full corpus of 34,793 quotes, and we look for the single quote each message was written for. Many other quotes could be fine replies, so these numbers understate real quality, but they compare models fairly.
| base all-MiniLM-L6-v2 | philo-minilm v1 | philo-minilm v2 | |
|---|---|---|---|
| Recall@1 | 2.2% | 3.6% | 3.9% |
| Recall@5 | 5.6% | 8.4% | 10.0% |
| Recall@10 | 7.5% | 11.8% | 13.7% |
| MRR | 0.041 | 0.065 | 0.074 |
| Median rank of the intended quote | 1,364 | 450 | 303 |
On Recall@5, v2 beats v1 by 1.5 points (95% CI +1.0 to +2.0, paired bootstrap over quotes).
| Recall@5 by mode | base | v1 | v2 |
|---|---|---|---|
| question (3,353) | 9.7% | 14.8% | 16.3% |
| advice (2,536) | 3.4% | 5.1% | 6.9% |
| venting (2,441) | 3.6% | 5.5% | 7.4% |
| nonsense (765) | 0.9% | 0.9% | 0.5% |
This validation set is larger than the one on v1's card (round 1's 4,321 pairs), so v1's numbers here differ a little from the ones it was published with.
Limitations
- The encoder doesn't handle nonsense. For nonsense messages, v2's top quote is one of the 1,528 known absurd quotes 4.4% of the time, which is exactly their share of the corpus (v1: 18.2%). The Philo app works around this with a separate intent router, a frozen Qwen2.5-0.5B-Instruct plus a small trained head, that detects nonsense and limits the search to absurd quotes. The router is not part of this repository.
- It still tends to echo, though less than v1. The top-ranked quote often restates the message instead of answering it. On the validation set the top pick shares 1.8ร as many content words with the message as the intended reply does (v1: 2.1ร). The in-batch negatives are random quotes, so topic matching is often enough to win; training with hard negatives (the quotes the base model wrongly ranks first) is the planned fix.
- Trained on synthetic messages, so real messages with unusual phrasing may behave differently.
- English only.
- Retrieval quality depends on the quote collection you search; this repository contains only the encoder.
- Downloads last month
- 13
Model tree for otterbox901/philo-minilm
Base model
nreimers/MiniLM-L6-H384-uncased