mine-embed-v1 / README.md
098alzoro0's picture
Update README.md
670c94d verified
|
Raw History Blame Contribute Delete
11 kB
---
library_name: pytorch
pipeline_tag: sentence-similarity
license: apache-2.0
license_name: mixed-source-licenses
language:
- en
tags:
- embeddings
- sentence-embeddings
- encoder
- custom-architecture
- experimental
datasets:
- MineAITechnology/mine-embed-v1
---
# mine-embed-v1
> **Version 1: first experimental test release, private.** This is MineAI Technology's first embedding model,
> trained from scratch. It is published to document an early experiment, **not as a competitive or
> production-ready model**. On a 28-task MTEB subset it scores well below standard small embedding models
> (see [Evaluation](#5-evaluation)).
mine-embed-v1 is a 57.8M-parameter English text embedding model. It maps a sentence to a 512-dimensional unit vector;
cosine similarity between vectors measures how close two texts are in meaning. It uses a custom encoder
architecture (not a standard `transformers` model) and the GPT-2 byte-pair tokenizer.
## 1. Quick start
The repository contains a custom `model.py`, so the model is loaded through its `Embedder` class, not
`AutoModel` or `SentenceTransformer`.
```python
import sys
from huggingface_hub import snapshot_download # private repo: log in with an HF token first
folder = snapshot_download("MineAITechnology/mine-embed-v1")
sys.path.insert(0, folder)
from model import Embedder
e = Embedder(folder) # loads config.json + model.safetensors, GPT-2 tokenizer
vecs = e.encode(["A man is playing a guitar.", "Someone strums an instrument.", "Stocks fell today."])
print(vecs.shape) # torch.Size([3, 512]), already L2-normalised
print(vecs @ vecs.T) # cosine similarities
```
Requirements: `torch`, `transformers` (only for the tokenizer), `safetensors`, `huggingface_hub`.
`Embedder.encode(texts, max_len=128, bs=64)` truncates to 128 tokens by default. No query/passage prefixes are used.
## 2. Model summary
| | |
|---|---|
| Parameters | 57,781,248 |
| Architecture | Encoder-only transformer: 10 layers, hidden 512, 8 heads, feed-forward 2048, pre-norm, GELU, learned position embeddings |
| Output | 512-dimensional, mean pooling over non-padding tokens + L2 normalisation |
| Tokenizer | GPT-2 BPE, vocabulary 50,258 (id 50256 = end-of-text/pad, id 50257 = mask) |
| Max positions | 1,024 (contrastive training used 128 tokens) |
| Language | English (GPT-2 BPE is weak on other languages) |
| Weights | `model.safetensors`, fp32, about 231 MB |
| Initialisation | **Random: trained from scratch** |
## 3. Training
Two stages. The full data description is in the dataset card:
[`MineAITechnology/mine-embed-v1` (dataset)](https://huggingface.co/datasets/MineAITechnology/mine-embed-v1).
**Stage 1: masked language modelling (MLM).**
- 15% dynamic masking (80% mask, 10% random token, 10% unchanged), sequence length 512, 128 sequences per step.
- AdamW, peak learning rate 4e-4 with cosine decay to 1e-5, weight decay 0.01, fp16 on 2x Tesla T4.
- 22,118 steps (5.0 hours), about 1.8 passes over a corpus of roughly 895M tokens.
- The corpus is a mixture of code, multilingual text, math, security/alignment text and corporate-fact text.
**Large parts are synthetic or template-generated**, so the amount of distinct natural text is far below the nominal
token count, and general English prose is only a small share.
**Stage 2: contrastive fine-tuning.**
- In-batch and hard-negative InfoNCE on 471,400 triplets, mostly AllNLI with Quora duplicate questions and a small
amount of private domain-specific data, 128-token inputs.
- <!-- TODO: add batch size, learning rate, epochs and temperature from the training notebook -->
## 4. Intended use
Suitable for experiments and baselines: sentence-similarity scoring on short English texts, studying
small-scale embedding training, and as a reference point for later versions. It is **not recommended** for retrieval,
clustering, or any production search system (see the limitations).
## 5. Evaluation
### Protocol
MTEB English subset of **28 tasks**: 9 STS, 4 classification, 3 clustering, 3 pair classification, 3 reranking and
6 retrieval tasks. Every model was run with the same settings: max length 128, batch size 64, fp32, one Kaggle T4,
`mteb` 1.x. Scores are each task's main metric times 100. Competitors were loaded with their own prompts/prefixes.
QuoraRetrieval is excluded from averages because this model was fine-tuned on Quora.
### Overall results
| Model | Params (M) | Dim | Avg (28) | STS | Classif. | Cluster | Pair | Rerank | Retrieval |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| bge-base-en-v1.5 | 109.5 | 768 | 66.25 | 78.51 | 66.69 | 39.68 | 86.56 | 67.77 | 49.94 |
| bge-small-en-v1.5 | 33.4 | 384 | 64.37 | 77.32 | 64.89 | 37.88 | 84.92 | 66.84 | 46.35 |
| gte-small | 33.4 | 384 | 63.95 | 76.66 | 62.72 | 39.15 | 83.55 | 66.21 | 47.17 |
| all-mpnet-base-v2 | 109.5 | 768 | 62.66 | 75.19 | 60.06 | 39.43 | 83.04 | 68.84 | 43.92 |
| all-MiniLM-L12-v2 | 33.4 | 384 | 61.60 | 74.56 | 59.79 | 37.56 | 82.46 | 67.58 | 41.99 |
| e5-small-v2 | 33.4 | 384 | 61.58 | 75.33 | 61.74 | 33.16 | 85.43 | 61.94 | 42.97 |
| all-MiniLM-L6-v2 | 22.7 | 384 | 61.14 | 73.67 | 59.28 | 37.08 | 82.37 | 67.12 | 42.04 |
| GloVe average (baseline) | 120.0 | 300 | 45.77 | 55.47 | 55.11 | 21.39 | 72.48 | 48.76 | 22.35 |
| **mine-embed-v1** | 57.8 | 512 | **37.78** | 59.63 | 43.84 | 11.19 | 54.06 | 40.76 | 4.64 |
mine-embed-v1 ranks last. It scores about 23 points below all-MiniLM-L6-v2 and 8 points below the GloVe baseline.
<details><summary>Per-task scores vs all-MiniLM-L6-v2 and GloVe</summary>
| Task | mine-embed-v1 | all-MiniLM-L6-v2 | GloVe average |
|---|---:|---:|---:|
| **STS (avg)** | **59.6** | **73.7** | **55.5** |
| STSBenchmark | 68.6 | **82.0** | 61.5 |
| STS12 | 62.6 | **72.4** | 54.6 |
| STS13 | 60.6 | **80.6** | 69.2 |
| STS14 | 58.6 | **75.6** | 60.8 |
| STS15 | 73.6 | **85.4** | 72.3 |
| STS16 | 68.6 | **79.0** | 65.3 |
| STS17 | 17.3 | **28.8** | 15.1 |
| SICK-R | 73.4 | **77.6** | 55.4 |
| BIOSSES | 53.5 | **81.6** | 44.9 |
| **Classification (avg)** | **43.8** | **59.3** | **55.1** |
| Banking77Classification | 47.5 | **80.0** | 67.8 |
| EmotionClassification | 24.6 | **40.8** | 37.6 |
| TweetSentimentExtractionClassification | 44.3 | **54.0** | 49.0 |
| ToxicConversationsClassification | 58.9 | 62.2 | **66.1** |
| **Clustering (avg)** | **11.2** | **37.1** | **21.4** |
| TwentyNewsgroupsClustering | 13.5 | **46.5** | 24.9 |
| BiorxivClusteringS2S | 5.4 | **32.9** | 18.7 |
| MedrxivClusteringS2S | 14.6 | **31.9** | 20.5 |
| **Pair classification (avg)** | **54.1** | **82.4** | **72.5** |
| SprintDuplicateQuestions | 49.8 | **94.6** | 87.0 |
| TwitterSemEval2015 | 43.5 | **67.9** | 53.1 |
| TwitterURLCorpus | 68.8 | **84.7** | 77.3 |
| **Reranking (avg)** | **40.8** | **67.1** | **48.8** |
| AskUbuntuDupQuestions | 45.6 | **63.5** | 49.6 |
| StackOverflowDupQuestions | 31.4 | **50.8** | 34.1 |
| SciDocsRR | 45.3 | **87.1** | 62.6 |
| **Retrieval (avg)** | **4.6** | **42.0** | **22.3** |
| SciFact | 8.1 | **62.4** | 29.6 |
| NFCorpus | 2.2 | **31.5** | 13.9 |
| FiQA2018 | 0.8 | **36.6** | 10.1 |
| ArguAna | 10.6 | **48.8** | 36.3 |
| SCIDOCS | 1.3 | **21.7** | 8.0 |
| TRECCOVID | 4.8 | **51.3** | 36.2 |
QuoraRetrieval (not in the averages, trained-on data): 74.1.
</details>
### What it does well and badly
- **Best area: sentence similarity.** SICK-R 73.4 (within about 4 points of all-MiniLM-L6-v2), STS15 73.6,
STSBenchmark 68.6, and it beats the GloVe baseline on 7 of 9 STS tasks. This matches its AllNLI-style training data.
- **Weak areas: retrieval (4.6), clustering (11.2) and duplicate-question detection** (SprintDuplicateQuestions 49.8).
The contrastive stage used a small, narrow dataset, so the embeddings separate fine differences between similar
sentences but do not group texts by topic.
### Training-time evaluation
- STS-B test Spearman: **0.686** (the MLM-only model before contrastive tuning scored 0.357).
- Held-out triplet accuracy (in-distribution, not a measure of general quality): AllNLI 0.977 (n=2,000),
Quora 0.969 (n=1,589), overall 0.974 (n=3,606); MLM-only baseline: 0.585, 0.844 and 0.700.
Three tiny extra domains (corporate, math/code, security: 6, 9 and 2 examples) are too small to report.
### Inference-time variants (12-task subset)
Changing the maximum length, shrinking the embedding with truncation or PCA, centering, or removing top principal
components moved the average by between -2.0 and +0.15 points. None closes the gap to the standard models.
Details are in the benchmark report.
### Efficiency (one Tesla T4, 128-token inputs, batch 64, fp32)
389 sentences/s on GPU, 18.8 sentences/s on 2 CPU threads, peak GPU memory
456 MB including weights.
### Embedding geometry check
The average cosine similarity between random, mostly unrelated sentences is 0.047 (all-MiniLM-L6-v2: 0.059), so the
embeddings are not collapsed into a narrow cone.
## 6. Limitations
- **Far below standard small embedding models** on general benchmarks (see the table).
- **Retrieval, clustering and duplicate detection are poor.** Do not use it for search or topic grouping.
- **Narrow training data.** Contrastive data is dominated by AllNLI and Quora; the MLM corpus is code/math/security
and synthetic-heavy with little natural English prose.
- **English only**, and the GPT-2 tokenizer is inefficient for non-English text.
- **Custom architecture:** needs `model.py`; it does not load with `AutoModel` or `SentenceTransformer`.
- **Short inputs:** contrastive training used 128 tokens.
- **License:** mixed sources, see below. Not cleared for commercial use.
## 7. License and data terms
The weights are derived from data under several different licenses and terms, so the model is released under
`license: other` and kept **private** until a license review is finished. Not legal advice.
- AllNLI = SNLI (CC BY-SA 4.0) + MultiNLI (per-genre terms). Share-alike obligations may apply.
- Quora Question Pairs: governed by Quora's own terms; commercial use unclear.
- MLM corpus (code, web, multilingual, math, security): each source has its own license; code in particular is often
copyleft or attribution-required.
- Private domain-specific and synthetic data: owned by the author.
## 8. Files in this repository
| File | Purpose |
|---|---|
| `model.safetensors` | Weights (231 MB) |
| `config.json` | Architecture settings (`encoder_config`) |
| `model.py` | Model definition and the `Embedder` class |
| `eval_report.json` | Training-time evaluation (triplet accuracy, STS-B) |
| `README.md` | This card |
## 9. Citation
```bibtex
@misc{mine_embed_v1_2026,
title = {mine-embed-v1: a 57.8M-parameter English text embedding model trained from scratch},
author = {{MineAI Technology}},
year = {2026},
howpublished = {Hugging Face model card}
}
```