Feature Extraction
Transformers
Safetensors
Bulgarian
mistral
llm2vec
text-embeddings
sentence-similarity
retrieval
bulgarian
text-embeddings-inference
Instructions to use mboyanov/bg2vec with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use mboyanov/bg2vec with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="mboyanov/bg2vec")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("mboyanov/bg2vec") model = AutoModel.from_pretrained("mboyanov/bg2vec", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Write model card: usage, training, evaluation, limitations
Browse files
README.md
CHANGED
|
@@ -1,199 +1,186 @@
|
|
| 1 |
---
|
|
|
|
|
|
|
|
|
|
| 2 |
library_name: transformers
|
| 3 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4 |
---
|
| 5 |
|
| 6 |
-
#
|
| 7 |
|
| 8 |
-
|
| 9 |
|
|
|
|
|
|
|
|
|
|
| 10 |
|
|
|
|
| 11 |
|
| 12 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
|
| 14 |
-
##
|
| 15 |
|
| 16 |
-
|
| 17 |
|
| 18 |
-
|
| 19 |
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
- **Language(s) (NLP):** [More Information Needed]
|
| 25 |
-
- **License:** [More Information Needed]
|
| 26 |
-
- **Finetuned from model [optional]:** [More Information Needed]
|
| 27 |
|
| 28 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
|
| 30 |
-
<!-- Provide the basic links for the model. -->
|
| 31 |
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
|
| 36 |
-
## Uses
|
| 37 |
|
| 38 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
|
| 40 |
-
|
| 41 |
|
| 42 |
-
|
| 43 |
|
| 44 |
-
|
| 45 |
|
| 46 |
-
|
|
|
|
| 47 |
|
| 48 |
-
|
|
|
|
|
|
|
| 49 |
|
| 50 |
-
|
| 51 |
|
| 52 |
-
##
|
| 53 |
|
| 54 |
-
|
| 55 |
|
| 56 |
-
|
| 57 |
|
| 58 |
-
|
| 59 |
|
| 60 |
-
|
| 61 |
|
| 62 |
-
|
|
|
|
|
|
|
|
|
|
| 63 |
|
| 64 |
-
###
|
| 65 |
|
| 66 |
-
|
|
|
|
|
|
|
|
|
|
| 67 |
|
| 68 |
-
|
| 69 |
|
| 70 |
-
##
|
| 71 |
|
| 72 |
-
|
| 73 |
-
|
| 74 |
-
[More Information Needed]
|
| 75 |
-
|
| 76 |
-
## Training Details
|
| 77 |
-
|
| 78 |
-
### Training Data
|
| 79 |
-
|
| 80 |
-
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
|
| 81 |
-
|
| 82 |
-
[More Information Needed]
|
| 83 |
-
|
| 84 |
-
### Training Procedure
|
| 85 |
-
|
| 86 |
-
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
|
| 87 |
-
|
| 88 |
-
#### Preprocessing [optional]
|
| 89 |
-
|
| 90 |
-
[More Information Needed]
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
#### Training Hyperparameters
|
| 94 |
-
|
| 95 |
-
- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
|
| 96 |
-
|
| 97 |
-
#### Speeds, Sizes, Times [optional]
|
| 98 |
-
|
| 99 |
-
<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
|
| 100 |
-
|
| 101 |
-
[More Information Needed]
|
| 102 |
|
| 103 |
## Evaluation
|
| 104 |
|
| 105 |
-
|
| 106 |
-
|
| 107 |
-
###
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
|
| 117 |
-
|
| 118 |
-
|
| 119 |
-
|
| 120 |
-
|
| 121 |
-
|
| 122 |
-
|
| 123 |
-
|
| 124 |
-
|
| 125 |
-
|
| 126 |
-
|
| 127 |
-
|
| 128 |
-
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
|
| 133 |
-
|
| 134 |
-
|
| 135 |
-
|
| 136 |
-
|
| 137 |
-
|
| 138 |
-
|
| 139 |
-
|
| 140 |
-
|
| 141 |
-
|
| 142 |
-
|
| 143 |
-
|
| 144 |
-
|
| 145 |
-
|
| 146 |
-
|
| 147 |
-
- **
|
| 148 |
-
- **
|
| 149 |
-
- **
|
| 150 |
-
|
| 151 |
-
|
| 152 |
-
|
| 153 |
-
|
| 154 |
-
|
| 155 |
-
|
| 156 |
-
|
| 157 |
-
|
| 158 |
-
|
| 159 |
-
|
| 160 |
-
|
| 161 |
-
|
| 162 |
-
|
| 163 |
-
|
| 164 |
-
|
| 165 |
-
|
| 166 |
-
|
| 167 |
-
|
| 168 |
-
|
| 169 |
-
[More Information Needed]
|
| 170 |
-
|
| 171 |
-
## Citation [optional]
|
| 172 |
-
|
| 173 |
-
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
|
| 174 |
-
|
| 175 |
-
**BibTeX:**
|
| 176 |
-
|
| 177 |
-
[More Information Needed]
|
| 178 |
-
|
| 179 |
-
**APA:**
|
| 180 |
-
|
| 181 |
-
[More Information Needed]
|
| 182 |
-
|
| 183 |
-
## Glossary [optional]
|
| 184 |
-
|
| 185 |
-
<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
|
| 186 |
-
|
| 187 |
-
[More Information Needed]
|
| 188 |
-
|
| 189 |
-
## More Information [optional]
|
| 190 |
-
|
| 191 |
-
[More Information Needed]
|
| 192 |
-
|
| 193 |
-
## Model Card Authors [optional]
|
| 194 |
-
|
| 195 |
-
[More Information Needed]
|
| 196 |
-
|
| 197 |
-
## Model Card Contact
|
| 198 |
-
|
| 199 |
-
[More Information Needed]
|
|
|
|
| 1 |
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
language:
|
| 4 |
+
- bg
|
| 5 |
library_name: transformers
|
| 6 |
+
pipeline_tag: feature-extraction
|
| 7 |
+
base_model: INSAIT-Institute/BgGPT-7B-Instruct-v0.2
|
| 8 |
+
datasets:
|
| 9 |
+
- mboyanov/bgwiki
|
| 10 |
+
tags:
|
| 11 |
+
- llm2vec
|
| 12 |
+
- text-embeddings
|
| 13 |
+
- sentence-similarity
|
| 14 |
+
- retrieval
|
| 15 |
+
- mistral
|
| 16 |
+
- bulgarian
|
| 17 |
---
|
| 18 |
|
| 19 |
+
# Bg2Vec
|
| 20 |
|
| 21 |
+
Bg2Vec is a Bulgarian text encoder. It is [BgGPT-7B-Instruct-v0.2](https://huggingface.co/INSAIT-Institute/BgGPT-7B-Instruct-v0.2) after the [LLM2Vec](https://arxiv.org/abs/2404.05961) procedure:
|
| 22 |
|
| 23 |
+
1. **Bidirectional attention.** Each token attends to all tokens, before and after it.
|
| 24 |
+
2. **Masked next token prediction (MNTP).** LoRA fine-tuning on Bulgarian Wikipedia. The model recovers masked tokens from the state of the previous token, so it learns to use the new attention.
|
| 25 |
+
3. **Unsupervised SimCSE.** LoRA fine-tuning on Bulgarian Wikipedia. Two dropout masks give two vectors for the same passage. The model learns to make them similar, and to make them different from the other passages in the batch.
|
| 26 |
|
| 27 |
+
Both LoRA adapters are merged into the weights in this repository. To get a text vector, use bidirectional attention and take the mean of the last hidden states.
|
| 28 |
|
| 29 |
+
- **Developed by:** Martin Boyanov
|
| 30 |
+
- **Model type:** Mistral-7B decoder used as a bidirectional encoder, 7.1B parameters, 4096-dimension vectors
|
| 31 |
+
- **Language:** Bulgarian
|
| 32 |
+
- **License:** Apache 2.0 (same as the base model)
|
| 33 |
+
- **Fine-tuned from:** [INSAIT-Institute/BgGPT-7B-Instruct-v0.2](https://huggingface.co/INSAIT-Institute/BgGPT-7B-Instruct-v0.2)
|
| 34 |
+
- **Training code:** [github.com/mboyanov/bg2vec](https://github.com/mboyanov/bg2vec)
|
| 35 |
+
- **Blog series:** [Let's build Bg2Vec](https://mboyanov.github.io/2024/05/18/Bg2Vec.html), in five parts
|
| 36 |
+
- **Demo:** [Bg2Vec search](https://huggingface.co/spaces/mboyanov/bg2vec-demo), BgGPT before and after LLM2Vec on Bulgarian Wikipedia search
|
| 37 |
|
| 38 |
+
## How to use
|
| 39 |
|
| 40 |
+
### With plain transformers
|
| 41 |
|
| 42 |
+
You do not need the `llm2vec` package. At inference time, LLM2Vec is two things: attention that is not causal, and mean pooling.
|
| 43 |
|
| 44 |
+
```python
|
| 45 |
+
import torch
|
| 46 |
+
import torch.nn.functional as F
|
| 47 |
+
from transformers import AutoModel, AutoTokenizer
|
|
|
|
|
|
|
|
|
|
| 48 |
|
| 49 |
+
name = "mboyanov/bg2vec"
|
| 50 |
+
tokenizer = AutoTokenizer.from_pretrained(name)
|
| 51 |
+
tokenizer.padding_side = "right"
|
| 52 |
+
model = AutoModel.from_pretrained(name, dtype=torch.bfloat16, attn_implementation="sdpa").eval()
|
| 53 |
+
for layer in model.layers:
|
| 54 |
+
layer.self_attn.is_causal = False # a guard; the 4D mask below does the real work
|
| 55 |
|
|
|
|
| 56 |
|
| 57 |
+
@torch.no_grad()
|
| 58 |
+
def encode(texts: list[str]) -> torch.Tensor:
|
| 59 |
+
batch = tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors="pt").to(model.device)
|
| 60 |
+
mask = batch["attention_mask"]
|
| 61 |
+
# transformers uses a 4D mask as it is. This mask hides only the padding,
|
| 62 |
+
# so every token attends to every other token.
|
| 63 |
+
attention = mask[:, None, None, :].bool()
|
| 64 |
+
hidden = model(input_ids=batch["input_ids"], attention_mask=attention).last_hidden_state.float()
|
| 65 |
+
weights = mask.unsqueeze(-1).float()
|
| 66 |
+
vectors = (hidden * weights).sum(dim=1) / weights.sum(dim=1) # mean over real tokens
|
| 67 |
+
return F.normalize(vectors, dim=-1)
|
| 68 |
|
|
|
|
| 69 |
|
| 70 |
+
original, positive, negative = encode([
|
| 71 |
+
"Тя обича да пътува през лятото, особено до студени дестинации, като избягва горещи и пренаселени места",
|
| 72 |
+
"Тя обича да пътува през лятото, специално към хладни места, избягвайки топли и многолюдни райони",
|
| 73 |
+
"Тя обича да пътува през лятото, но предпочита да посещава горещи и оживени туристически места",
|
| 74 |
+
])
|
| 75 |
+
print((original @ positive).item(), (original @ negative).item())
|
| 76 |
+
# ≈ 0.787 0.728: the positive is closer than the negative
|
| 77 |
+
```
|
| 78 |
|
| 79 |
+
We tested this code with `transformers` 5.22. Its vectors agree with the `llm2vec` package (cosine similarity > 0.999).
|
| 80 |
|
| 81 |
+
Do not add a query prefix or an instruction. The model was trained without them.
|
| 82 |
|
| 83 |
+
### With llm2vec
|
| 84 |
|
| 85 |
+
```python
|
| 86 |
+
from llm2vec import LLM2Vec
|
| 87 |
|
| 88 |
+
model = LLM2Vec.from_pretrained("mboyanov/bg2vec")
|
| 89 |
+
model.encode(["Това е примерно изречение."])
|
| 90 |
+
```
|
| 91 |
|
| 92 |
+
The `llm2vec` package needs an older `transformers` (4.44 or lower).
|
| 93 |
|
| 94 |
+
## Training
|
| 95 |
|
| 96 |
+
The training follows the LLM2Vec paper for Mistral-7B. Each step trains a LoRA adapter (r=16, alpha=32, dropout 0.05) on all attention and MLP projections: 0.58% of the parameters. Both steps run for 1000 steps in bf16.
|
| 97 |
|
| 98 |
+
### Data
|
| 99 |
|
| 100 |
+
[mboyanov/bgwiki](https://huggingface.co/datasets/mboyanov/bgwiki): a dump of Bulgarian Wikipedia (May 2024), with text extracted by `wikiextractor` and empty redirect pages removed.
|
| 101 |
|
| 102 |
+
### Step 1: MNTP
|
| 103 |
|
| 104 |
+
- Data: the articles are tokenized with the BgGPT tokenizer, joined, and cut into chunks of 512 tokens. At most 50,000 chunks are used for training.
|
| 105 |
+
- Masking: `all_mask` (as the paper uses for Mistral), mask probability 0.8, `_` as the mask token.
|
| 106 |
+
- Batch size 32.
|
| 107 |
+
- Adapter: [mboyanov/bggpt-mntp](https://huggingface.co/mboyanov/bggpt-mntp). The validation loss goes down from 5.53 at step 100 to 4.89 at step 1000.
|
| 108 |
|
| 109 |
+
### Step 2: SimCSE
|
| 110 |
|
| 111 |
+
- Start point: BgGPT with the MNTP adapter merged in, bidirectional attention.
|
| 112 |
+
- Data: each article is a pair with itself. Texts are truncated to 128 tokens.
|
| 113 |
+
- Loss: `HardNegativeNLLLoss` from llm2vec, with in-batch negatives, scale 20.
|
| 114 |
+
- Batch size 128, learning rate 3e-5, attention dropout 0.3, mean pooling.
|
| 115 |
|
| 116 |
+
The full configurations are in [model_configurations/](https://github.com/mboyanov/bg2vec/tree/master/model_configurations).
|
| 117 |
|
| 118 |
+
### Hardware
|
| 119 |
|
| 120 |
+
One NVIDIA A100 80GB or A6000 48GB GPU on Paperspace Gradient.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 121 |
|
| 122 |
## Evaluation
|
| 123 |
|
| 124 |
+
All the results come from [Part 5 of the blog series](https://mboyanov.github.io/2024/11/02/Bg2VecEval.html).
|
| 125 |
+
|
| 126 |
+
### MTEB, Bulgarian tasks
|
| 127 |
+
|
| 128 |
+
The table shows the MTEB main score of each task. BgGPT uses mean pooling with its original causal attention.
|
| 129 |
+
|
| 130 |
+
| Task | Bg2Vec | BgGPT-7B-Instruct-v0.2 | bge-multilingual-gemma2 |
|
| 131 |
+
|:---|---:|---:|---:|
|
| 132 |
+
| BulgarianStoreReviewSentimentClassfication | 0.609 | 0.570 | **0.696** |
|
| 133 |
+
| LanguageClassification | 0.983 | **0.992** | 0.920 |
|
| 134 |
+
| MultilingualSentimentClassification | 0.810 | 0.720 | **0.905** |
|
| 135 |
+
| SIB200Classification | 0.718 | 0.758 | **0.769** |
|
| 136 |
+
| SIB200ClusteringS2S | 0.262 | 0.319 | **0.391** |
|
| 137 |
+
| BelebeleRetrieval | **0.910** | 0.348 | 0.764 |
|
| 138 |
+
| WikipediaRetrievalMultilingual | **0.780** | 0.392 | 0.689 |
|
| 139 |
+
| XNLI | 0.621 | 0.578 | **0.792** |
|
| 140 |
+
| WikipediaRerankingMultilingual | 0.752 | 0.656 | **0.813** |
|
| 141 |
+
|
| 142 |
+
Bg2Vec is better than BgGPT on 6 of the 9 tasks. The largest gain is on retrieval: BelebeleRetrieval goes from 0.348 to 0.910. Bg2Vec has the best retrieval scores of the three models. bge-multilingual-gemma2 has the best score on 6 tasks, and BgGPT on LanguageClassification. MultiEURLEXMultilabelClassification is not in the table, because it used too much memory.
|
| 143 |
+
|
| 144 |
+
### Hard triplets
|
| 145 |
+
|
| 146 |
+
A set of sentence triplets, translated into Bulgarian from the Echo Embeddings paper. The three sentences in a triplet start with the same words. The first two sentences have the same meaning, and the third sentence has a different meaning. The score is the mean of `cos(original, positive) - cos(original, negative)`. A higher score is better.
|
| 147 |
+
|
| 148 |
+
| Model | Score |
|
| 149 |
+
|:---|---:|
|
| 150 |
+
| **Bg2Vec** | **0.236** |
|
| 151 |
+
| bge-multilingual-gemma2 | 0.164 |
|
| 152 |
+
| e5-mistral-7b-instruct | 0.133 |
|
| 153 |
+
| multilingual-e5-large | 0.039 |
|
| 154 |
+
| BgGPT-7B-Instruct-v0.2 | 0.033 |
|
| 155 |
+
| gte-Qwen2-7B-instruct | 0.026 |
|
| 156 |
+
|
| 157 |
+
### Demo queries
|
| 158 |
+
|
| 159 |
+
The [demo Space](https://huggingface.co/spaces/mboyanov/bg2vec-demo) searches 29,912 passages of Bulgarian Wikipedia. Each of its 10 showcase queries has one target article. The MRR@10 of the target article is 0.777 for Bg2Vec, 0.725 for multilingual-e5-large, 0.283 for BM25 and 0.025 for BgGPT. Ten queries are a showcase, not a benchmark.
|
| 160 |
+
|
| 161 |
+
## Limitations
|
| 162 |
+
|
| 163 |
+
- **Size.** The model has 7.1B parameters (14 GB in bf16) and gives 4096-dimension vectors. On one L40S, it encodes approximately 48 passages per second at 256 tokens or less. A smaller multilingual encoder can be a better choice for many tasks.
|
| 164 |
+
- **Text length.** MNTP used chunks of 512 tokens and SimCSE used texts of 128 tokens. We did not test texts longer than 512 tokens.
|
| 165 |
+
- **No text generation.** The repository has no language-model head. Use the model only to encode text.
|
| 166 |
+
- **Unsupervised training.** The training used only Wikipedia text, with no labelled pairs and no instructions. The model is weaker on clustering and topic classification than BgGPT, and weaker than bge-multilingual-gemma2 on most tasks that are not retrieval.
|
| 167 |
+
- **Rare tokens.** The model can miss exact matches on codes and rare terms (for example, "ISO 8601"). For such queries, use it together with lexical search such as BM25.
|
| 168 |
+
- **Bias.** The model can have the biases of Bulgarian Wikipedia and of BgGPT.
|
| 169 |
+
|
| 170 |
+
## Citation
|
| 171 |
+
|
| 172 |
+
```bibtex
|
| 173 |
+
@misc{boyanov2024bg2vec,
|
| 174 |
+
author = {Martin Boyanov},
|
| 175 |
+
title = {Let's Build Bg2Vec: Apply LLM2Vec on BgGPT},
|
| 176 |
+
year = {2024},
|
| 177 |
+
url = {https://mboyanov.github.io/2024/05/18/Bg2Vec.html}
|
| 178 |
+
}
|
| 179 |
+
|
| 180 |
+
@article{behnamghader2024llm2vec,
|
| 181 |
+
title = {LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders},
|
| 182 |
+
author = {BehnamGhader, Parishad and Adlakha, Vaibhav and Mosbach, Marius and Bahdanau, Dzmitry and Chapados, Nicolas and Reddy, Siva},
|
| 183 |
+
journal = {arXiv preprint arXiv:2404.05961},
|
| 184 |
+
year = {2024}
|
| 185 |
+
}
|
| 186 |
+
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|