Model card

README.md

@@ -0,0 +1,69 @@
+---
+license: apache-2.0
+language: [en]
+library_name: transformers
+pipeline_tag: sentence-similarity
+tags: [sentence-transformers, sentence-similarity, feature-extraction, sbert]
+---
+
+# echo-sbert-domain
+
+A sentence embedding model for **short, messy, real-world app-store reviews**.
+
+Built for [Echo](https://github.com/ayn-aval/Echo), which turns 100,000 Swiggy
+Google Play reviews into tracked themes. Trained in two stages:
+
+1. **Sentence-BERT reproduction.** `distilroberta-base` on 300,000 SNLI +
+   MultiNLI pairs, siamese, mean pooling, classifier on `(u, v, |u-v|)` — written
+   by hand in raw PyTorch, reproducing
+   [Reimers & Gurevych (2019)](https://arxiv.org/abs/1908.10084).
+2. **Domain adaptation.** Continued with `MultipleNegativesRankingLoss` on 53,061
+   pairs mined from the reviews themselves, where TF-IDF and the stage-1 encoder
+   independently agree, plus SimCSE dropout self-pairs.
+
+## Results
+
+| benchmark | score |
+|---|---|
+| STS average (7 datasets) after stage 1 | 72.17 |
+| STS average after stage 2 | **74.54** |
+| Review retrieval, Precision@10 | 61.15 |
+| Review retrieval, + cross-encoder rerank | **75.77** |
+| Theme assignment, blind hand-audit | 82.4% |
+
+Stage 2 improved generic STS by +2.37 while adapting to the domain, which was
+predicted to degrade and did not. Note that 74.54 is **not** "beating the paper's
+74.21": that number comes from NLI training alone, this adds a second stage.
+
+## Limitations, stated plainly
+
+- **It does not bridge Hinglish.** "khana thanda tha" against "the food was cold"
+  scores 0.066, versus 0.049 for a genuinely unrelated pair. Romanised Hindi
+  reviews cluster by *language* rather than by subject.
+- **On its own it loses to TF-IDF** for review retrieval (61.15 vs 65.00). It only
+  wins in front of a cross-encoder reranker.
+- **Part of the retrieval gain may be circular** — mined pairs required TF-IDF to
+  agree, so the model may have partly learned to imitate it.
+- Retrieval numbers rest on **26 hand-judged queries**, one judge.
+
+## Use
+
+```python
+from transformers import AutoModel, AutoTokenizer
+import torch
+
+tok = AutoTokenizer.from_pretrained("aynaval2003/echo-sbert-domain")
+model = AutoModel.from_pretrained("aynaval2003/echo-sbert-domain").eval()
+
+def embed(sentences):
+    x = tok(sentences, padding=True, truncation=True, max_length=128,
+            return_tensors="pt")
+    with torch.no_grad():
+        h = model(**x).last_hidden_state
+    mask = x["attention_mask"].unsqueeze(-1).float()      # mean pooling,
+    v = (h * mask).sum(1) / mask.sum(1).clamp(min=1e-9)   # ignoring padding
+    return torch.nn.functional.normalize(v, dim=1)
+```
+
+**Mean pooling, and it matters** — this model was trained with it. CLS pooling
+scores 5.1 points lower in the ablation.