|
Download README.md from MineAITechnology/mine-embed-v1: direct link, hf CLI and curl.
- Browser
- Download file 11 kB
-
https://huggingface.co/MineAITechnology/mine-embed-v1/resolve/main/README.md
- Command line
-
hf download hf://MineAITechnology/mine-embed-v1/README.md
-
curl -L -o README.md https://huggingface.co/MineAITechnology/mine-embed-v1/resolve/main/README.md
11 kB
| library_name: pytorch | |
| pipeline_tag: sentence-similarity | |
| license: apache-2.0 | |
| license_name: mixed-source-licenses | |
| language: | |
| - en | |
| tags: | |
| - embeddings | |
| - sentence-embeddings | |
| - encoder | |
| - custom-architecture | |
| - experimental | |
| datasets: | |
| - MineAITechnology/mine-embed-v1 | |
| # mine-embed-v1 | |
| > **Version 1: first experimental test release, private.** This is MineAI Technology's first embedding model, | |
| > trained from scratch. It is published to document an early experiment, **not as a competitive or | |
| > production-ready model**. On a 28-task MTEB subset it scores well below standard small embedding models | |
| > (see [Evaluation](#5-evaluation)). | |
| mine-embed-v1 is a 57.8M-parameter English text embedding model. It maps a sentence to a 512-dimensional unit vector; | |
| cosine similarity between vectors measures how close two texts are in meaning. It uses a custom encoder | |
| architecture (not a standard `transformers` model) and the GPT-2 byte-pair tokenizer. | |
| ## 1. Quick start | |
| The repository contains a custom `model.py`, so the model is loaded through its `Embedder` class, not | |
| `AutoModel` or `SentenceTransformer`. | |
| ```python | |
| import sys | |
| from huggingface_hub import snapshot_download # private repo: log in with an HF token first | |
| folder = snapshot_download("MineAITechnology/mine-embed-v1") | |
| sys.path.insert(0, folder) | |
| from model import Embedder | |
| e = Embedder(folder) # loads config.json + model.safetensors, GPT-2 tokenizer | |
| vecs = e.encode(["A man is playing a guitar.", "Someone strums an instrument.", "Stocks fell today."]) | |
| print(vecs.shape) # torch.Size([3, 512]), already L2-normalised | |
| print(vecs @ vecs.T) # cosine similarities | |
| ``` | |
| Requirements: `torch`, `transformers` (only for the tokenizer), `safetensors`, `huggingface_hub`. | |
| `Embedder.encode(texts, max_len=128, bs=64)` truncates to 128 tokens by default. No query/passage prefixes are used. | |
| ## 2. Model summary | |
| | | | | |
| |---|---| | |
| | Parameters | 57,781,248 | | |
| | Architecture | Encoder-only transformer: 10 layers, hidden 512, 8 heads, feed-forward 2048, pre-norm, GELU, learned position embeddings | | |
| | Output | 512-dimensional, mean pooling over non-padding tokens + L2 normalisation | | |
| | Tokenizer | GPT-2 BPE, vocabulary 50,258 (id 50256 = end-of-text/pad, id 50257 = mask) | | |
| | Max positions | 1,024 (contrastive training used 128 tokens) | | |
| | Language | English (GPT-2 BPE is weak on other languages) | | |
| | Weights | `model.safetensors`, fp32, about 231 MB | | |
| | Initialisation | **Random: trained from scratch** | | |
| ## 3. Training | |
| Two stages. The full data description is in the dataset card: | |
| [`MineAITechnology/mine-embed-v1` (dataset)](https://huggingface.co/datasets/MineAITechnology/mine-embed-v1). | |
| **Stage 1: masked language modelling (MLM).** | |
| - 15% dynamic masking (80% mask, 10% random token, 10% unchanged), sequence length 512, 128 sequences per step. | |
| - AdamW, peak learning rate 4e-4 with cosine decay to 1e-5, weight decay 0.01, fp16 on 2x Tesla T4. | |
| - 22,118 steps (5.0 hours), about 1.8 passes over a corpus of roughly 895M tokens. | |
| - The corpus is a mixture of code, multilingual text, math, security/alignment text and corporate-fact text. | |
| **Large parts are synthetic or template-generated**, so the amount of distinct natural text is far below the nominal | |
| token count, and general English prose is only a small share. | |
| **Stage 2: contrastive fine-tuning.** | |
| - In-batch and hard-negative InfoNCE on 471,400 triplets, mostly AllNLI with Quora duplicate questions and a small | |
| amount of private domain-specific data, 128-token inputs. | |
| - <!-- TODO: add batch size, learning rate, epochs and temperature from the training notebook --> | |
| ## 4. Intended use | |
| Suitable for experiments and baselines: sentence-similarity scoring on short English texts, studying | |
| small-scale embedding training, and as a reference point for later versions. It is **not recommended** for retrieval, | |
| clustering, or any production search system (see the limitations). | |
| ## 5. Evaluation | |
| ### Protocol | |
| MTEB English subset of **28 tasks**: 9 STS, 4 classification, 3 clustering, 3 pair classification, 3 reranking and | |
| 6 retrieval tasks. Every model was run with the same settings: max length 128, batch size 64, fp32, one Kaggle T4, | |
| `mteb` 1.x. Scores are each task's main metric times 100. Competitors were loaded with their own prompts/prefixes. | |
| QuoraRetrieval is excluded from averages because this model was fine-tuned on Quora. | |
| ### Overall results | |
| | Model | Params (M) | Dim | Avg (28) | STS | Classif. | Cluster | Pair | Rerank | Retrieval | | |
| |---|---:|---:|---:|---:|---:|---:|---:|---:|---:| | |
| | bge-base-en-v1.5 | 109.5 | 768 | 66.25 | 78.51 | 66.69 | 39.68 | 86.56 | 67.77 | 49.94 | | |
| | bge-small-en-v1.5 | 33.4 | 384 | 64.37 | 77.32 | 64.89 | 37.88 | 84.92 | 66.84 | 46.35 | | |
| | gte-small | 33.4 | 384 | 63.95 | 76.66 | 62.72 | 39.15 | 83.55 | 66.21 | 47.17 | | |
| | all-mpnet-base-v2 | 109.5 | 768 | 62.66 | 75.19 | 60.06 | 39.43 | 83.04 | 68.84 | 43.92 | | |
| | all-MiniLM-L12-v2 | 33.4 | 384 | 61.60 | 74.56 | 59.79 | 37.56 | 82.46 | 67.58 | 41.99 | | |
| | e5-small-v2 | 33.4 | 384 | 61.58 | 75.33 | 61.74 | 33.16 | 85.43 | 61.94 | 42.97 | | |
| | all-MiniLM-L6-v2 | 22.7 | 384 | 61.14 | 73.67 | 59.28 | 37.08 | 82.37 | 67.12 | 42.04 | | |
| | GloVe average (baseline) | 120.0 | 300 | 45.77 | 55.47 | 55.11 | 21.39 | 72.48 | 48.76 | 22.35 | | |
| | **mine-embed-v1** | 57.8 | 512 | **37.78** | 59.63 | 43.84 | 11.19 | 54.06 | 40.76 | 4.64 | | |
| mine-embed-v1 ranks last. It scores about 23 points below all-MiniLM-L6-v2 and 8 points below the GloVe baseline. | |
| <details><summary>Per-task scores vs all-MiniLM-L6-v2 and GloVe</summary> | |
| | Task | mine-embed-v1 | all-MiniLM-L6-v2 | GloVe average | | |
| |---|---:|---:|---:| | |
| | **STS (avg)** | **59.6** | **73.7** | **55.5** | | |
| | STSBenchmark | 68.6 | **82.0** | 61.5 | | |
| | STS12 | 62.6 | **72.4** | 54.6 | | |
| | STS13 | 60.6 | **80.6** | 69.2 | | |
| | STS14 | 58.6 | **75.6** | 60.8 | | |
| | STS15 | 73.6 | **85.4** | 72.3 | | |
| | STS16 | 68.6 | **79.0** | 65.3 | | |
| | STS17 | 17.3 | **28.8** | 15.1 | | |
| | SICK-R | 73.4 | **77.6** | 55.4 | | |
| | BIOSSES | 53.5 | **81.6** | 44.9 | | |
| | **Classification (avg)** | **43.8** | **59.3** | **55.1** | | |
| | Banking77Classification | 47.5 | **80.0** | 67.8 | | |
| | EmotionClassification | 24.6 | **40.8** | 37.6 | | |
| | TweetSentimentExtractionClassification | 44.3 | **54.0** | 49.0 | | |
| | ToxicConversationsClassification | 58.9 | 62.2 | **66.1** | | |
| | **Clustering (avg)** | **11.2** | **37.1** | **21.4** | | |
| | TwentyNewsgroupsClustering | 13.5 | **46.5** | 24.9 | | |
| | BiorxivClusteringS2S | 5.4 | **32.9** | 18.7 | | |
| | MedrxivClusteringS2S | 14.6 | **31.9** | 20.5 | | |
| | **Pair classification (avg)** | **54.1** | **82.4** | **72.5** | | |
| | SprintDuplicateQuestions | 49.8 | **94.6** | 87.0 | | |
| | TwitterSemEval2015 | 43.5 | **67.9** | 53.1 | | |
| | TwitterURLCorpus | 68.8 | **84.7** | 77.3 | | |
| | **Reranking (avg)** | **40.8** | **67.1** | **48.8** | | |
| | AskUbuntuDupQuestions | 45.6 | **63.5** | 49.6 | | |
| | StackOverflowDupQuestions | 31.4 | **50.8** | 34.1 | | |
| | SciDocsRR | 45.3 | **87.1** | 62.6 | | |
| | **Retrieval (avg)** | **4.6** | **42.0** | **22.3** | | |
| | SciFact | 8.1 | **62.4** | 29.6 | | |
| | NFCorpus | 2.2 | **31.5** | 13.9 | | |
| | FiQA2018 | 0.8 | **36.6** | 10.1 | | |
| | ArguAna | 10.6 | **48.8** | 36.3 | | |
| | SCIDOCS | 1.3 | **21.7** | 8.0 | | |
| | TRECCOVID | 4.8 | **51.3** | 36.2 | | |
| QuoraRetrieval (not in the averages, trained-on data): 74.1. | |
| </details> | |
| ### What it does well and badly | |
| - **Best area: sentence similarity.** SICK-R 73.4 (within about 4 points of all-MiniLM-L6-v2), STS15 73.6, | |
| STSBenchmark 68.6, and it beats the GloVe baseline on 7 of 9 STS tasks. This matches its AllNLI-style training data. | |
| - **Weak areas: retrieval (4.6), clustering (11.2) and duplicate-question detection** (SprintDuplicateQuestions 49.8). | |
| The contrastive stage used a small, narrow dataset, so the embeddings separate fine differences between similar | |
| sentences but do not group texts by topic. | |
| ### Training-time evaluation | |
| - STS-B test Spearman: **0.686** (the MLM-only model before contrastive tuning scored 0.357). | |
| - Held-out triplet accuracy (in-distribution, not a measure of general quality): AllNLI 0.977 (n=2,000), | |
| Quora 0.969 (n=1,589), overall 0.974 (n=3,606); MLM-only baseline: 0.585, 0.844 and 0.700. | |
| Three tiny extra domains (corporate, math/code, security: 6, 9 and 2 examples) are too small to report. | |
| ### Inference-time variants (12-task subset) | |
| Changing the maximum length, shrinking the embedding with truncation or PCA, centering, or removing top principal | |
| components moved the average by between -2.0 and +0.15 points. None closes the gap to the standard models. | |
| Details are in the benchmark report. | |
| ### Efficiency (one Tesla T4, 128-token inputs, batch 64, fp32) | |
| 389 sentences/s on GPU, 18.8 sentences/s on 2 CPU threads, peak GPU memory | |
| 456 MB including weights. | |
| ### Embedding geometry check | |
| The average cosine similarity between random, mostly unrelated sentences is 0.047 (all-MiniLM-L6-v2: 0.059), so the | |
| embeddings are not collapsed into a narrow cone. | |
| ## 6. Limitations | |
| - **Far below standard small embedding models** on general benchmarks (see the table). | |
| - **Retrieval, clustering and duplicate detection are poor.** Do not use it for search or topic grouping. | |
| - **Narrow training data.** Contrastive data is dominated by AllNLI and Quora; the MLM corpus is code/math/security | |
| and synthetic-heavy with little natural English prose. | |
| - **English only**, and the GPT-2 tokenizer is inefficient for non-English text. | |
| - **Custom architecture:** needs `model.py`; it does not load with `AutoModel` or `SentenceTransformer`. | |
| - **Short inputs:** contrastive training used 128 tokens. | |
| - **License:** mixed sources, see below. Not cleared for commercial use. | |
| ## 7. License and data terms | |
| The weights are derived from data under several different licenses and terms, so the model is released under | |
| `license: other` and kept **private** until a license review is finished. Not legal advice. | |
| - AllNLI = SNLI (CC BY-SA 4.0) + MultiNLI (per-genre terms). Share-alike obligations may apply. | |
| - Quora Question Pairs: governed by Quora's own terms; commercial use unclear. | |
| - MLM corpus (code, web, multilingual, math, security): each source has its own license; code in particular is often | |
| copyleft or attribution-required. | |
| - Private domain-specific and synthetic data: owned by the author. | |
| ## 8. Files in this repository | |
| | File | Purpose | | |
| |---|---| | |
| | `model.safetensors` | Weights (231 MB) | | |
| | `config.json` | Architecture settings (`encoder_config`) | | |
| | `model.py` | Model definition and the `Embedder` class | | |
| | `eval_report.json` | Training-time evaluation (triplet accuracy, STS-B) | | |
| | `README.md` | This card | | |
| ## 9. Citation | |
| ```bibtex | |
| @misc{mine_embed_v1_2026, | |
| title = {mine-embed-v1: a 57.8M-parameter English text embedding model trained from scratch}, | |
| author = {{MineAI Technology}}, | |
| year = {2026}, | |
| howpublished = {Hugging Face model card} | |
| } | |
| ``` |