MineAITechnology
/

mine-embed-v1

@@ -1,30 +1,222 @@
 ---
 library_name: pytorch
-tags: [embeddings, sentence-embeddings, encoder, custom-architecture]
 license: other
 ---
-# mine-embed-57M (57.8M params)
-Encoder-only text embedding model (10 layers, hidden 512, 8 heads, FFN 2048, pre-norm, GELU, GPT-2 BPE vocab 50,258, mean pooling + L2 norm, 128-token contrastive training).
-Custom architecture: load with `Embedder` in `model.py` (`from model import Embedder; Embedder(".").encode([...])`).
-## Training
-1. MLM pretraining (15% dynamic masking, 80/10/10, AdamW, peak LR 4e-4, cosine to 1e-5) on a mixed code / multilingual / math / security / corporate-text corpus.
-   Large parts of that corpus are **synthetic or template-generated** (arithmetic, corporate-fact templates, security templates); the amount of distinct natural text is far below the nominal token count.
-2. Contrastive fine-tuning (in-batch + hard-negative InfoNCE) on AllNLI, Quora and private domain-specific triplets.
-## Evaluation (see eval_report.json)
-Per-domain held-out triplet accuracy is **in-distribution** and does not measure general embedding quality. STS-B test Spearman: {'dataset': 'sentence-transformers/stsb', 'n': 1379, 'spearman': 0.6860609574807518}.
-## Licenses — MIXED, not a single license
-The weights are derived from data under several different licenses and terms. The builder's metadata is reproduced below; **verify each source's license and terms before any redistribution or commercial use**.
-- AllNLI = SNLI + MultiNLI: SNLI is CC BY-SA 4.0; MultiNLI has per-genre terms (check the MultiNLI release). Share-alike obligations may apply.
-- Quora Question Pairs: governed by Quora's own terms, not an open license; commercial use is unclear.
-- Pretraining corpus (code / web / multilingual): sources carry their own licenses (code in particular is often copyleft or attribution-required). Source list and licenses are in the dataset builder metadata below.
-- Private/domain-specific and synthetic data: owned by the author.
-Because of this mix the model is released under `license: other` and kept **private** until the license review is done.
-### Dataset builder fingerprints
-```
-{"fingerprint": "0db2e603ec2cc6a2", "target_tokens": 1000000000, "actual_windows": 878640}
-{"fingerprint": "d7706b95d8decdfd", "train": 471400, "val": 9620}
 ```

 ---
 library_name: pytorch
+pipeline_tag: sentence-similarity
 license: other
+license_name: mixed-source-licenses
+language:
+- en
+tags:
+- embeddings
+- sentence-embeddings
+- encoder
+- custom-architecture
+- experimental
+datasets:
+- MineAITechnology/mine-embed-v1
 ---
+# mine-embed-v1
+> **Version 1: first experimental test release, private.** This is MineAI Technology's first embedding model,
+> trained from scratch. It is published to document an early experiment, **not as a competitive or
+> production-ready model**. On a 28-task MTEB subset it scores well below standard small embedding models
+> (see [Evaluation](#5-evaluation)).
+mine-embed-v1 is a 57.8M-parameter English text embedding model. It maps a sentence to a 512-dimensional unit vector;
+cosine similarity between vectors measures how close two texts are in meaning. It uses a custom encoder
+architecture (not a standard `transformers` model) and the GPT-2 byte-pair tokenizer.
+## 1. Quick start
+The repository contains a custom `model.py`, so the model is loaded through its `Embedder` class, not
+`AutoModel` or `SentenceTransformer`.
+```python
+import sys
+from huggingface_hub import snapshot_download          # private repo: log in with an HF token first
+folder = snapshot_download("MineAITechnology/mine-embed-v1")
+sys.path.insert(0, folder)
+from model import Embedder
+e = Embedder(folder)                                    # loads config.json + model.safetensors, GPT-2 tokenizer
+vecs = e.encode(["A man is playing a guitar.", "Someone strums an instrument.", "Stocks fell today."])
+print(vecs.shape)                                       # torch.Size([3, 512]), already L2-normalised
+print(vecs @ vecs.T)                                    # cosine similarities
 ```
+Requirements: `torch`, `transformers` (only for the tokenizer), `safetensors`, `huggingface_hub`.
+`Embedder.encode(texts, max_len=128, bs=64)` truncates to 128 tokens by default. No query/passage prefixes are used.
+## 2. Model summary
+| | |
+|---|---|
+| Parameters | 57,781,248 |
+| Architecture | Encoder-only transformer: 10 layers, hidden 512, 8 heads, feed-forward 2048, pre-norm, GELU, learned position embeddings |
+| Output | 512-dimensional, mean pooling over non-padding tokens + L2 normalisation |
+| Tokenizer | GPT-2 BPE, vocabulary 50,258 (id 50256 = end-of-text/pad, id 50257 = mask) |
+| Max positions | 1,024 (contrastive training used 128 tokens) |
+| Language | English (GPT-2 BPE is weak on other languages) |
+| Weights | `model.safetensors`, fp32, about 231 MB |
+| Initialisation | **Random: trained from scratch** |
+## 3. Training
+Two stages. The full data description is in the dataset card:
+[`MineAITechnology/mine-embed-v1` (dataset)](https://huggingface.co/datasets/MineAITechnology/mine-embed-v1).
+**Stage 1: masked language modelling (MLM).**
+- 15% dynamic masking (80% mask, 10% random token, 10% unchanged), sequence length 512, 128 sequences per step.
+- AdamW, peak learning rate 4e-4 with cosine decay to 1e-5, weight decay 0.01, fp16 on 2x Tesla T4.
+- 22,118 steps (5.0 hours), about 1.8 passes over a corpus of roughly 895M tokens.
+- The corpus is a mixture of code, multilingual text, math, security/alignment text and corporate-fact text.
+  **Large parts are synthetic or template-generated**, so the amount of distinct natural text is far below the nominal
+  token count, and general English prose is only a small share.
+**Stage 2: contrastive fine-tuning.**
+- In-batch and hard-negative InfoNCE on 471,400 triplets, mostly AllNLI with Quora duplicate questions and a small
+  amount of private domain-specific data, 128-token inputs.
+- <!-- TODO: add batch size, learning rate, epochs and temperature from the training notebook -->
+## 4. Intended use
+Suitable for experiments and baselines: sentence-similarity scoring on short English texts, studying
+small-scale embedding training, and as a reference point for later versions. It is **not recommended** for retrieval,
+clustering, or any production search system (see the limitations).
+## 5. Evaluation
+### Protocol
+MTEB English subset of **28 tasks**: 9 STS, 4 classification, 3 clustering, 3 pair classification, 3 reranking and
+6 retrieval tasks. Every model was run with the same settings: max length 128, batch size 64, fp32, one Kaggle T4,
+`mteb` 1.x. Scores are each task's main metric times 100. Competitors were loaded with their own prompts/prefixes.
+QuoraRetrieval is excluded from averages because this model was fine-tuned on Quora.
+### Overall results
+| Model | Params (M) | Dim | Avg (28) | STS | Classif. | Cluster | Pair | Rerank | Retrieval |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| bge-base-en-v1.5 | 109.5 | 768 | 66.25 | 78.51 | 66.69 | 39.68 | 86.56 | 67.77 | 49.94 |
+| bge-small-en-v1.5 | 33.4 | 384 | 64.37 | 77.32 | 64.89 | 37.88 | 84.92 | 66.84 | 46.35 |
+| gte-small | 33.4 | 384 | 63.95 | 76.66 | 62.72 | 39.15 | 83.55 | 66.21 | 47.17 |
+| all-mpnet-base-v2 | 109.5 | 768 | 62.66 | 75.19 | 60.06 | 39.43 | 83.04 | 68.84 | 43.92 |
+| all-MiniLM-L12-v2 | 33.4 | 384 | 61.60 | 74.56 | 59.79 | 37.56 | 82.46 | 67.58 | 41.99 |
+| e5-small-v2 | 33.4 | 384 | 61.58 | 75.33 | 61.74 | 33.16 | 85.43 | 61.94 | 42.97 |
+| all-MiniLM-L6-v2 | 22.7 | 384 | 61.14 | 73.67 | 59.28 | 37.08 | 82.37 | 67.12 | 42.04 |
+| GloVe average (baseline) | 120.0 | 300 | 45.77 | 55.47 | 55.11 | 21.39 | 72.48 | 48.76 | 22.35 |
+| **mine-embed-v1** | 57.8 | 512 | **37.78** | 59.63 | 43.84 | 11.19 | 54.06 | 40.76 | 4.64 |
+mine-embed-v1 ranks last. It scores about 23 points below all-MiniLM-L6-v2 and 8 points below the GloVe baseline.
+<details><summary>Per-task scores vs all-MiniLM-L6-v2 and GloVe</summary>
+| Task | mine-embed-v1 | all-MiniLM-L6-v2 | GloVe average |
+|---|---:|---:|---:|
+| **STS (avg)** | **59.6** | **73.7** | **55.5** |
+| STSBenchmark | 68.6 | **82.0** | 61.5 |
+| STS12 | 62.6 | **72.4** | 54.6 |
+| STS13 | 60.6 | **80.6** | 69.2 |
+| STS14 | 58.6 | **75.6** | 60.8 |
+| STS15 | 73.6 | **85.4** | 72.3 |
+| STS16 | 68.6 | **79.0** | 65.3 |
+| STS17 | 17.3 | **28.8** | 15.1 |
+| SICK-R | 73.4 | **77.6** | 55.4 |
+| BIOSSES | 53.5 | **81.6** | 44.9 |
+| **Classification (avg)** | **43.8** | **59.3** | **55.1** |
+| Banking77Classification | 47.5 | **80.0** | 67.8 |
+| EmotionClassification | 24.6 | **40.8** | 37.6 |
+| TweetSentimentExtractionClassification | 44.3 | **54.0** | 49.0 |
+| ToxicConversationsClassification | 58.9 | 62.2 | **66.1** |
+| **Clustering (avg)** | **11.2** | **37.1** | **21.4** |
+| TwentyNewsgroupsClustering | 13.5 | **46.5** | 24.9 |
+| BiorxivClusteringS2S | 5.4 | **32.9** | 18.7 |
+| MedrxivClusteringS2S | 14.6 | **31.9** | 20.5 |
+| **Pair classification (avg)** | **54.1** | **82.4** | **72.5** |
+| SprintDuplicateQuestions | 49.8 | **94.6** | 87.0 |
+| TwitterSemEval2015 | 43.5 | **67.9** | 53.1 |
+| TwitterURLCorpus | 68.8 | **84.7** | 77.3 |
+| **Reranking (avg)** | **40.8** | **67.1** | **48.8** |
+| AskUbuntuDupQuestions | 45.6 | **63.5** | 49.6 |
+| StackOverflowDupQuestions | 31.4 | **50.8** | 34.1 |
+| SciDocsRR | 45.3 | **87.1** | 62.6 |
+| **Retrieval (avg)** | **4.6** | **42.0** | **22.3** |
+| SciFact | 8.1 | **62.4** | 29.6 |
+| NFCorpus | 2.2 | **31.5** | 13.9 |
+| FiQA2018 | 0.8 | **36.6** | 10.1 |
+| ArguAna | 10.6 | **48.8** | 36.3 |
+| SCIDOCS | 1.3 | **21.7** | 8.0 |
+| TRECCOVID | 4.8 | **51.3** | 36.2 |
+QuoraRetrieval (not in the averages, trained-on data): 74.1.
+</details>
+### What it does well and badly
+- **Best area: sentence similarity.** SICK-R 73.4 (within about 4 points of all-MiniLM-L6-v2), STS15 73.6,
+  STSBenchmark 68.6, and it beats the GloVe baseline on 7 of 9 STS tasks. This matches its AllNLI-style training data.
+- **Weak areas: retrieval (4.6), clustering (11.2) and duplicate-question detection** (SprintDuplicateQuestions 49.8).
+  The contrastive stage used a small, narrow dataset, so the embeddings separate fine differences between similar
+  sentences but do not group texts by topic.
+### Training-time evaluation
+- STS-B test Spearman: **0.686** (the MLM-only model before contrastive tuning scored 0.357).
+- Held-out triplet accuracy (in-distribution, not a measure of general quality): AllNLI 0.977 (n=2,000),
+  Quora 0.969 (n=1,589), overall 0.974 (n=3,606); MLM-only baseline: 0.585, 0.844 and 0.700.
+  Three tiny extra domains (corporate, math/code, security: 6, 9 and 2 examples) are too small to report.
+### Inference-time variants (12-task subset)
+Changing the maximum length, shrinking the embedding with truncation or PCA, centering, or removing top principal
+components moved the average by between -2.0 and +0.15 points. None closes the gap to the standard models.
+Details are in the benchmark report.
+### Efficiency (one Tesla T4, 128-token inputs, batch 64, fp32)
+389 sentences/s on GPU, 18.8 sentences/s on 2 CPU threads, peak GPU memory
+456 MB including weights.
+### Embedding geometry check
+The average cosine similarity between random, mostly unrelated sentences is 0.047 (all-MiniLM-L6-v2: 0.059), so the
+embeddings are not collapsed into a narrow cone.
+## 6. Limitations
+- **Far below standard small embedding models** on general benchmarks (see the table).
+- **Retrieval, clustering and duplicate detection are poor.** Do not use it for search or topic grouping.
+- **Narrow training data.** Contrastive data is dominated by AllNLI and Quora; the MLM corpus is code/math/security
+  and synthetic-heavy with little natural English prose.
+- **English only**, and the GPT-2 tokenizer is inefficient for non-English text.
+- **Custom architecture:** needs `model.py`; it does not load with `AutoModel` or `SentenceTransformer`.
+- **Short inputs:** contrastive training used 128 tokens.
+- **License:** mixed sources, see below. Not cleared for commercial use.
+## 7. License and data terms
+The weights are derived from data under several different licenses and terms, so the model is released under
+`license: other` and kept **private** until a license review is finished. Not legal advice.
+- AllNLI = SNLI (CC BY-SA 4.0) + MultiNLI (per-genre terms). Share-alike obligations may apply.
+- Quora Question Pairs: governed by Quora's own terms; commercial use unclear.
+- MLM corpus (code, web, multilingual, math, security): each source has its own license; code in particular is often
+  copyleft or attribution-required.
+- Private domain-specific and synthetic data: owned by the author.
+## 8. Files in this repository
+| File | Purpose |
+|---|---|
+| `model.safetensors` | Weights (231 MB) |
+| `config.json` | Architecture settings (`encoder_config`) |
+| `model.py` | Model definition and the `Embedder` class |
+| `eval_report.json` | Training-time evaluation (triplet accuracy, STS-B) |
+| `README.md` | This card |
+## 9. Citation
+```bibtex
+@misc{mine_embed_v1_2026,
+  title        = {mine-embed-v1: a 57.8M-parameter English text embedding model trained from scratch},
+  author       = {{MineAI Technology}},
+  year         = {2026},
+  howpublished = {Hugging Face model card}
+}
+```