--- library_name: pytorch pipeline_tag: sentence-similarity license: apache-2.0 license_name: mixed-source-licenses language: - en tags: - embeddings - sentence-embeddings - encoder - custom-architecture - experimental datasets: - MineAITechnology/mine-embed-v1 --- # mine-embed-v1 > **Version 1: first experimental test release, private.** This is MineAI Technology's first embedding model, > trained from scratch. It is published to document an early experiment, **not as a competitive or > production-ready model**. On a 28-task MTEB subset it scores well below standard small embedding models > (see [Evaluation](#5-evaluation)). mine-embed-v1 is a 57.8M-parameter English text embedding model. It maps a sentence to a 512-dimensional unit vector; cosine similarity between vectors measures how close two texts are in meaning. It uses a custom encoder architecture (not a standard `transformers` model) and the GPT-2 byte-pair tokenizer. ## 1. Quick start The repository contains a custom `model.py`, so the model is loaded through its `Embedder` class, not `AutoModel` or `SentenceTransformer`. ```python import sys from huggingface_hub import snapshot_download # private repo: log in with an HF token first folder = snapshot_download("MineAITechnology/mine-embed-v1") sys.path.insert(0, folder) from model import Embedder e = Embedder(folder) # loads config.json + model.safetensors, GPT-2 tokenizer vecs = e.encode(["A man is playing a guitar.", "Someone strums an instrument.", "Stocks fell today."]) print(vecs.shape) # torch.Size([3, 512]), already L2-normalised print(vecs @ vecs.T) # cosine similarities ``` Requirements: `torch`, `transformers` (only for the tokenizer), `safetensors`, `huggingface_hub`. `Embedder.encode(texts, max_len=128, bs=64)` truncates to 128 tokens by default. No query/passage prefixes are used. ## 2. Model summary | | | |---|---| | Parameters | 57,781,248 | | Architecture | Encoder-only transformer: 10 layers, hidden 512, 8 heads, feed-forward 2048, pre-norm, GELU, learned position embeddings | | Output | 512-dimensional, mean pooling over non-padding tokens + L2 normalisation | | Tokenizer | GPT-2 BPE, vocabulary 50,258 (id 50256 = end-of-text/pad, id 50257 = mask) | | Max positions | 1,024 (contrastive training used 128 tokens) | | Language | English (GPT-2 BPE is weak on other languages) | | Weights | `model.safetensors`, fp32, about 231 MB | | Initialisation | **Random: trained from scratch** | ## 3. Training Two stages. The full data description is in the dataset card: [`MineAITechnology/mine-embed-v1` (dataset)](https://huggingface.co/datasets/MineAITechnology/mine-embed-v1). **Stage 1: masked language modelling (MLM).** - 15% dynamic masking (80% mask, 10% random token, 10% unchanged), sequence length 512, 128 sequences per step. - AdamW, peak learning rate 4e-4 with cosine decay to 1e-5, weight decay 0.01, fp16 on 2x Tesla T4. - 22,118 steps (5.0 hours), about 1.8 passes over a corpus of roughly 895M tokens. - The corpus is a mixture of code, multilingual text, math, security/alignment text and corporate-fact text. **Large parts are synthetic or template-generated**, so the amount of distinct natural text is far below the nominal token count, and general English prose is only a small share. **Stage 2: contrastive fine-tuning.** - In-batch and hard-negative InfoNCE on 471,400 triplets, mostly AllNLI with Quora duplicate questions and a small amount of private domain-specific data, 128-token inputs. - ## 4. Intended use Suitable for experiments and baselines: sentence-similarity scoring on short English texts, studying small-scale embedding training, and as a reference point for later versions. It is **not recommended** for retrieval, clustering, or any production search system (see the limitations). ## 5. Evaluation ### Protocol MTEB English subset of **28 tasks**: 9 STS, 4 classification, 3 clustering, 3 pair classification, 3 reranking and 6 retrieval tasks. Every model was run with the same settings: max length 128, batch size 64, fp32, one Kaggle T4, `mteb` 1.x. Scores are each task's main metric times 100. Competitors were loaded with their own prompts/prefixes. QuoraRetrieval is excluded from averages because this model was fine-tuned on Quora. ### Overall results | Model | Params (M) | Dim | Avg (28) | STS | Classif. | Cluster | Pair | Rerank | Retrieval | |---|---:|---:|---:|---:|---:|---:|---:|---:|---:| | bge-base-en-v1.5 | 109.5 | 768 | 66.25 | 78.51 | 66.69 | 39.68 | 86.56 | 67.77 | 49.94 | | bge-small-en-v1.5 | 33.4 | 384 | 64.37 | 77.32 | 64.89 | 37.88 | 84.92 | 66.84 | 46.35 | | gte-small | 33.4 | 384 | 63.95 | 76.66 | 62.72 | 39.15 | 83.55 | 66.21 | 47.17 | | all-mpnet-base-v2 | 109.5 | 768 | 62.66 | 75.19 | 60.06 | 39.43 | 83.04 | 68.84 | 43.92 | | all-MiniLM-L12-v2 | 33.4 | 384 | 61.60 | 74.56 | 59.79 | 37.56 | 82.46 | 67.58 | 41.99 | | e5-small-v2 | 33.4 | 384 | 61.58 | 75.33 | 61.74 | 33.16 | 85.43 | 61.94 | 42.97 | | all-MiniLM-L6-v2 | 22.7 | 384 | 61.14 | 73.67 | 59.28 | 37.08 | 82.37 | 67.12 | 42.04 | | GloVe average (baseline) | 120.0 | 300 | 45.77 | 55.47 | 55.11 | 21.39 | 72.48 | 48.76 | 22.35 | | **mine-embed-v1** | 57.8 | 512 | **37.78** | 59.63 | 43.84 | 11.19 | 54.06 | 40.76 | 4.64 | mine-embed-v1 ranks last. It scores about 23 points below all-MiniLM-L6-v2 and 8 points below the GloVe baseline.
Per-task scores vs all-MiniLM-L6-v2 and GloVe | Task | mine-embed-v1 | all-MiniLM-L6-v2 | GloVe average | |---|---:|---:|---:| | **STS (avg)** | **59.6** | **73.7** | **55.5** | | STSBenchmark | 68.6 | **82.0** | 61.5 | | STS12 | 62.6 | **72.4** | 54.6 | | STS13 | 60.6 | **80.6** | 69.2 | | STS14 | 58.6 | **75.6** | 60.8 | | STS15 | 73.6 | **85.4** | 72.3 | | STS16 | 68.6 | **79.0** | 65.3 | | STS17 | 17.3 | **28.8** | 15.1 | | SICK-R | 73.4 | **77.6** | 55.4 | | BIOSSES | 53.5 | **81.6** | 44.9 | | **Classification (avg)** | **43.8** | **59.3** | **55.1** | | Banking77Classification | 47.5 | **80.0** | 67.8 | | EmotionClassification | 24.6 | **40.8** | 37.6 | | TweetSentimentExtractionClassification | 44.3 | **54.0** | 49.0 | | ToxicConversationsClassification | 58.9 | 62.2 | **66.1** | | **Clustering (avg)** | **11.2** | **37.1** | **21.4** | | TwentyNewsgroupsClustering | 13.5 | **46.5** | 24.9 | | BiorxivClusteringS2S | 5.4 | **32.9** | 18.7 | | MedrxivClusteringS2S | 14.6 | **31.9** | 20.5 | | **Pair classification (avg)** | **54.1** | **82.4** | **72.5** | | SprintDuplicateQuestions | 49.8 | **94.6** | 87.0 | | TwitterSemEval2015 | 43.5 | **67.9** | 53.1 | | TwitterURLCorpus | 68.8 | **84.7** | 77.3 | | **Reranking (avg)** | **40.8** | **67.1** | **48.8** | | AskUbuntuDupQuestions | 45.6 | **63.5** | 49.6 | | StackOverflowDupQuestions | 31.4 | **50.8** | 34.1 | | SciDocsRR | 45.3 | **87.1** | 62.6 | | **Retrieval (avg)** | **4.6** | **42.0** | **22.3** | | SciFact | 8.1 | **62.4** | 29.6 | | NFCorpus | 2.2 | **31.5** | 13.9 | | FiQA2018 | 0.8 | **36.6** | 10.1 | | ArguAna | 10.6 | **48.8** | 36.3 | | SCIDOCS | 1.3 | **21.7** | 8.0 | | TRECCOVID | 4.8 | **51.3** | 36.2 | QuoraRetrieval (not in the averages, trained-on data): 74.1.
### What it does well and badly - **Best area: sentence similarity.** SICK-R 73.4 (within about 4 points of all-MiniLM-L6-v2), STS15 73.6, STSBenchmark 68.6, and it beats the GloVe baseline on 7 of 9 STS tasks. This matches its AllNLI-style training data. - **Weak areas: retrieval (4.6), clustering (11.2) and duplicate-question detection** (SprintDuplicateQuestions 49.8). The contrastive stage used a small, narrow dataset, so the embeddings separate fine differences between similar sentences but do not group texts by topic. ### Training-time evaluation - STS-B test Spearman: **0.686** (the MLM-only model before contrastive tuning scored 0.357). - Held-out triplet accuracy (in-distribution, not a measure of general quality): AllNLI 0.977 (n=2,000), Quora 0.969 (n=1,589), overall 0.974 (n=3,606); MLM-only baseline: 0.585, 0.844 and 0.700. Three tiny extra domains (corporate, math/code, security: 6, 9 and 2 examples) are too small to report. ### Inference-time variants (12-task subset) Changing the maximum length, shrinking the embedding with truncation or PCA, centering, or removing top principal components moved the average by between -2.0 and +0.15 points. None closes the gap to the standard models. Details are in the benchmark report. ### Efficiency (one Tesla T4, 128-token inputs, batch 64, fp32) 389 sentences/s on GPU, 18.8 sentences/s on 2 CPU threads, peak GPU memory 456 MB including weights. ### Embedding geometry check The average cosine similarity between random, mostly unrelated sentences is 0.047 (all-MiniLM-L6-v2: 0.059), so the embeddings are not collapsed into a narrow cone. ## 6. Limitations - **Far below standard small embedding models** on general benchmarks (see the table). - **Retrieval, clustering and duplicate detection are poor.** Do not use it for search or topic grouping. - **Narrow training data.** Contrastive data is dominated by AllNLI and Quora; the MLM corpus is code/math/security and synthetic-heavy with little natural English prose. - **English only**, and the GPT-2 tokenizer is inefficient for non-English text. - **Custom architecture:** needs `model.py`; it does not load with `AutoModel` or `SentenceTransformer`. - **Short inputs:** contrastive training used 128 tokens. - **License:** mixed sources, see below. Not cleared for commercial use. ## 7. License and data terms The weights are derived from data under several different licenses and terms, so the model is released under `license: other` and kept **private** until a license review is finished. Not legal advice. - AllNLI = SNLI (CC BY-SA 4.0) + MultiNLI (per-genre terms). Share-alike obligations may apply. - Quora Question Pairs: governed by Quora's own terms; commercial use unclear. - MLM corpus (code, web, multilingual, math, security): each source has its own license; code in particular is often copyleft or attribution-required. - Private domain-specific and synthetic data: owned by the author. ## 8. Files in this repository | File | Purpose | |---|---| | `model.safetensors` | Weights (231 MB) | | `config.json` | Architecture settings (`encoder_config`) | | `model.py` | Model definition and the `Embedder` class | | `eval_report.json` | Training-time evaluation (triplet accuracy, STS-B) | | `README.md` | This card | ## 9. Citation ```bibtex @misc{mine_embed_v1_2026, title = {mine-embed-v1: a 57.8M-parameter English text embedding model trained from scratch}, author = {{MineAI Technology}}, year = {2026}, howpublished = {Hugging Face model card} } ```