mboyanov commited on
Commit
07cdfb4
·
verified ·
1 Parent(s): a5434dc

Write model card: usage, training, evaluation, limitations

Browse files
Files changed (1) hide show
  1. README.md +149 -162
README.md CHANGED
@@ -1,199 +1,186 @@
1
  ---
 
 
 
2
  library_name: transformers
3
- tags: []
 
 
 
 
 
 
 
 
 
 
4
  ---
5
 
6
- # Model Card for Model ID
7
 
8
- <!-- Provide a quick summary of what the model is/does. -->
9
 
 
 
 
10
 
 
11
 
12
- ## Model Details
 
 
 
 
 
 
 
13
 
14
- ### Model Description
15
 
16
- <!-- Provide a longer summary of what this model is. -->
17
 
18
- This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
19
 
20
- - **Developed by:** [More Information Needed]
21
- - **Funded by [optional]:** [More Information Needed]
22
- - **Shared by [optional]:** [More Information Needed]
23
- - **Model type:** [More Information Needed]
24
- - **Language(s) (NLP):** [More Information Needed]
25
- - **License:** [More Information Needed]
26
- - **Finetuned from model [optional]:** [More Information Needed]
27
 
28
- ### Model Sources [optional]
 
 
 
 
 
29
 
30
- <!-- Provide the basic links for the model. -->
31
 
32
- - **Repository:** [More Information Needed]
33
- - **Paper [optional]:** [More Information Needed]
34
- - **Demo [optional]:** [More Information Needed]
 
 
 
 
 
 
 
 
35
 
36
- ## Uses
37
 
38
- <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
 
 
 
 
 
 
 
39
 
40
- ### Direct Use
41
 
42
- <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
43
 
44
- [More Information Needed]
45
 
46
- ### Downstream Use [optional]
 
47
 
48
- <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
 
 
49
 
50
- [More Information Needed]
51
 
52
- ### Out-of-Scope Use
53
 
54
- <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
55
 
56
- [More Information Needed]
57
 
58
- ## Bias, Risks, and Limitations
59
 
60
- <!-- This section is meant to convey both technical and sociotechnical limitations. -->
61
 
62
- [More Information Needed]
 
 
 
63
 
64
- ### Recommendations
65
 
66
- <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
 
 
 
67
 
68
- Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
69
 
70
- ## How to Get Started with the Model
71
 
72
- Use the code below to get started with the model.
73
-
74
- [More Information Needed]
75
-
76
- ## Training Details
77
-
78
- ### Training Data
79
-
80
- <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
81
-
82
- [More Information Needed]
83
-
84
- ### Training Procedure
85
-
86
- <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
87
-
88
- #### Preprocessing [optional]
89
-
90
- [More Information Needed]
91
-
92
-
93
- #### Training Hyperparameters
94
-
95
- - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
96
-
97
- #### Speeds, Sizes, Times [optional]
98
-
99
- <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
100
-
101
- [More Information Needed]
102
 
103
  ## Evaluation
104
 
105
- <!-- This section describes the evaluation protocols and provides the results. -->
106
-
107
- ### Testing Data, Factors & Metrics
108
-
109
- #### Testing Data
110
-
111
- <!-- This should link to a Dataset Card if possible. -->
112
-
113
- [More Information Needed]
114
-
115
- #### Factors
116
-
117
- <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
118
-
119
- [More Information Needed]
120
-
121
- #### Metrics
122
-
123
- <!-- These are the evaluation metrics being used, ideally with a description of why. -->
124
-
125
- [More Information Needed]
126
-
127
- ### Results
128
-
129
- [More Information Needed]
130
-
131
- #### Summary
132
-
133
-
134
-
135
- ## Model Examination [optional]
136
-
137
- <!-- Relevant interpretability work for the model goes here -->
138
-
139
- [More Information Needed]
140
-
141
- ## Environmental Impact
142
-
143
- <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
144
-
145
- Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
146
-
147
- - **Hardware Type:** [More Information Needed]
148
- - **Hours used:** [More Information Needed]
149
- - **Cloud Provider:** [More Information Needed]
150
- - **Compute Region:** [More Information Needed]
151
- - **Carbon Emitted:** [More Information Needed]
152
-
153
- ## Technical Specifications [optional]
154
-
155
- ### Model Architecture and Objective
156
-
157
- [More Information Needed]
158
-
159
- ### Compute Infrastructure
160
-
161
- [More Information Needed]
162
-
163
- #### Hardware
164
-
165
- [More Information Needed]
166
-
167
- #### Software
168
-
169
- [More Information Needed]
170
-
171
- ## Citation [optional]
172
-
173
- <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
174
-
175
- **BibTeX:**
176
-
177
- [More Information Needed]
178
-
179
- **APA:**
180
-
181
- [More Information Needed]
182
-
183
- ## Glossary [optional]
184
-
185
- <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
186
-
187
- [More Information Needed]
188
-
189
- ## More Information [optional]
190
-
191
- [More Information Needed]
192
-
193
- ## Model Card Authors [optional]
194
-
195
- [More Information Needed]
196
-
197
- ## Model Card Contact
198
-
199
- [More Information Needed]
 
1
  ---
2
+ license: apache-2.0
3
+ language:
4
+ - bg
5
  library_name: transformers
6
+ pipeline_tag: feature-extraction
7
+ base_model: INSAIT-Institute/BgGPT-7B-Instruct-v0.2
8
+ datasets:
9
+ - mboyanov/bgwiki
10
+ tags:
11
+ - llm2vec
12
+ - text-embeddings
13
+ - sentence-similarity
14
+ - retrieval
15
+ - mistral
16
+ - bulgarian
17
  ---
18
 
19
+ # Bg2Vec
20
 
21
+ Bg2Vec is a Bulgarian text encoder. It is [BgGPT-7B-Instruct-v0.2](https://huggingface.co/INSAIT-Institute/BgGPT-7B-Instruct-v0.2) after the [LLM2Vec](https://arxiv.org/abs/2404.05961) procedure:
22
 
23
+ 1. **Bidirectional attention.** Each token attends to all tokens, before and after it.
24
+ 2. **Masked next token prediction (MNTP).** LoRA fine-tuning on Bulgarian Wikipedia. The model recovers masked tokens from the state of the previous token, so it learns to use the new attention.
25
+ 3. **Unsupervised SimCSE.** LoRA fine-tuning on Bulgarian Wikipedia. Two dropout masks give two vectors for the same passage. The model learns to make them similar, and to make them different from the other passages in the batch.
26
 
27
+ Both LoRA adapters are merged into the weights in this repository. To get a text vector, use bidirectional attention and take the mean of the last hidden states.
28
 
29
+ - **Developed by:** Martin Boyanov
30
+ - **Model type:** Mistral-7B decoder used as a bidirectional encoder, 7.1B parameters, 4096-dimension vectors
31
+ - **Language:** Bulgarian
32
+ - **License:** Apache 2.0 (same as the base model)
33
+ - **Fine-tuned from:** [INSAIT-Institute/BgGPT-7B-Instruct-v0.2](https://huggingface.co/INSAIT-Institute/BgGPT-7B-Instruct-v0.2)
34
+ - **Training code:** [github.com/mboyanov/bg2vec](https://github.com/mboyanov/bg2vec)
35
+ - **Blog series:** [Let's build Bg2Vec](https://mboyanov.github.io/2024/05/18/Bg2Vec.html), in five parts
36
+ - **Demo:** [Bg2Vec search](https://huggingface.co/spaces/mboyanov/bg2vec-demo), BgGPT before and after LLM2Vec on Bulgarian Wikipedia search
37
 
38
+ ## How to use
39
 
40
+ ### With plain transformers
41
 
42
+ You do not need the `llm2vec` package. At inference time, LLM2Vec is two things: attention that is not causal, and mean pooling.
43
 
44
+ ```python
45
+ import torch
46
+ import torch.nn.functional as F
47
+ from transformers import AutoModel, AutoTokenizer
 
 
 
48
 
49
+ name = "mboyanov/bg2vec"
50
+ tokenizer = AutoTokenizer.from_pretrained(name)
51
+ tokenizer.padding_side = "right"
52
+ model = AutoModel.from_pretrained(name, dtype=torch.bfloat16, attn_implementation="sdpa").eval()
53
+ for layer in model.layers:
54
+ layer.self_attn.is_causal = False # a guard; the 4D mask below does the real work
55
 
 
56
 
57
+ @torch.no_grad()
58
+ def encode(texts: list[str]) -> torch.Tensor:
59
+ batch = tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors="pt").to(model.device)
60
+ mask = batch["attention_mask"]
61
+ # transformers uses a 4D mask as it is. This mask hides only the padding,
62
+ # so every token attends to every other token.
63
+ attention = mask[:, None, None, :].bool()
64
+ hidden = model(input_ids=batch["input_ids"], attention_mask=attention).last_hidden_state.float()
65
+ weights = mask.unsqueeze(-1).float()
66
+ vectors = (hidden * weights).sum(dim=1) / weights.sum(dim=1) # mean over real tokens
67
+ return F.normalize(vectors, dim=-1)
68
 
 
69
 
70
+ original, positive, negative = encode([
71
+ "Тя обича да пътува през лятото, особено до студени дестинации, като избягва горещи и пренаселени места",
72
+ "Тя обича да пътува през лятото, специално към хладни места, избягвайки топли и многолюдни райони",
73
+ "Тя обича да пътува през лятото, но предпочита да посещава горещи и оживени туристически места",
74
+ ])
75
+ print((original @ positive).item(), (original @ negative).item())
76
+ # ≈ 0.787 0.728: the positive is closer than the negative
77
+ ```
78
 
79
+ We tested this code with `transformers` 5.22. Its vectors agree with the `llm2vec` package (cosine similarity > 0.999).
80
 
81
+ Do not add a query prefix or an instruction. The model was trained without them.
82
 
83
+ ### With llm2vec
84
 
85
+ ```python
86
+ from llm2vec import LLM2Vec
87
 
88
+ model = LLM2Vec.from_pretrained("mboyanov/bg2vec")
89
+ model.encode(["Това е примерно изречение."])
90
+ ```
91
 
92
+ The `llm2vec` package needs an older `transformers` (4.44 or lower).
93
 
94
+ ## Training
95
 
96
+ The training follows the LLM2Vec paper for Mistral-7B. Each step trains a LoRA adapter (r=16, alpha=32, dropout 0.05) on all attention and MLP projections: 0.58% of the parameters. Both steps run for 1000 steps in bf16.
97
 
98
+ ### Data
99
 
100
+ [mboyanov/bgwiki](https://huggingface.co/datasets/mboyanov/bgwiki): a dump of Bulgarian Wikipedia (May 2024), with text extracted by `wikiextractor` and empty redirect pages removed.
101
 
102
+ ### Step 1: MNTP
103
 
104
+ - Data: the articles are tokenized with the BgGPT tokenizer, joined, and cut into chunks of 512 tokens. At most 50,000 chunks are used for training.
105
+ - Masking: `all_mask` (as the paper uses for Mistral), mask probability 0.8, `_` as the mask token.
106
+ - Batch size 32.
107
+ - Adapter: [mboyanov/bggpt-mntp](https://huggingface.co/mboyanov/bggpt-mntp). The validation loss goes down from 5.53 at step 100 to 4.89 at step 1000.
108
 
109
+ ### Step 2: SimCSE
110
 
111
+ - Start point: BgGPT with the MNTP adapter merged in, bidirectional attention.
112
+ - Data: each article is a pair with itself. Texts are truncated to 128 tokens.
113
+ - Loss: `HardNegativeNLLLoss` from llm2vec, with in-batch negatives, scale 20.
114
+ - Batch size 128, learning rate 3e-5, attention dropout 0.3, mean pooling.
115
 
116
+ The full configurations are in [model_configurations/](https://github.com/mboyanov/bg2vec/tree/master/model_configurations).
117
 
118
+ ### Hardware
119
 
120
+ One NVIDIA A100 80GB or A6000 48GB GPU on Paperspace Gradient.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
121
 
122
  ## Evaluation
123
 
124
+ All the results come from [Part 5 of the blog series](https://mboyanov.github.io/2024/11/02/Bg2VecEval.html).
125
+
126
+ ### MTEB, Bulgarian tasks
127
+
128
+ The table shows the MTEB main score of each task. BgGPT uses mean pooling with its original causal attention.
129
+
130
+ | Task | Bg2Vec | BgGPT-7B-Instruct-v0.2 | bge-multilingual-gemma2 |
131
+ |:---|---:|---:|---:|
132
+ | BulgarianStoreReviewSentimentClassfication | 0.609 | 0.570 | **0.696** |
133
+ | LanguageClassification | 0.983 | **0.992** | 0.920 |
134
+ | MultilingualSentimentClassification | 0.810 | 0.720 | **0.905** |
135
+ | SIB200Classification | 0.718 | 0.758 | **0.769** |
136
+ | SIB200ClusteringS2S | 0.262 | 0.319 | **0.391** |
137
+ | BelebeleRetrieval | **0.910** | 0.348 | 0.764 |
138
+ | WikipediaRetrievalMultilingual | **0.780** | 0.392 | 0.689 |
139
+ | XNLI | 0.621 | 0.578 | **0.792** |
140
+ | WikipediaRerankingMultilingual | 0.752 | 0.656 | **0.813** |
141
+
142
+ Bg2Vec is better than BgGPT on 6 of the 9 tasks. The largest gain is on retrieval: BelebeleRetrieval goes from 0.348 to 0.910. Bg2Vec has the best retrieval scores of the three models. bge-multilingual-gemma2 has the best score on 6 tasks, and BgGPT on LanguageClassification. MultiEURLEXMultilabelClassification is not in the table, because it used too much memory.
143
+
144
+ ### Hard triplets
145
+
146
+ A set of sentence triplets, translated into Bulgarian from the Echo Embeddings paper. The three sentences in a triplet start with the same words. The first two sentences have the same meaning, and the third sentence has a different meaning. The score is the mean of `cos(original, positive) - cos(original, negative)`. A higher score is better.
147
+
148
+ | Model | Score |
149
+ |:---|---:|
150
+ | **Bg2Vec** | **0.236** |
151
+ | bge-multilingual-gemma2 | 0.164 |
152
+ | e5-mistral-7b-instruct | 0.133 |
153
+ | multilingual-e5-large | 0.039 |
154
+ | BgGPT-7B-Instruct-v0.2 | 0.033 |
155
+ | gte-Qwen2-7B-instruct | 0.026 |
156
+
157
+ ### Demo queries
158
+
159
+ The [demo Space](https://huggingface.co/spaces/mboyanov/bg2vec-demo) searches 29,912 passages of Bulgarian Wikipedia. Each of its 10 showcase queries has one target article. The MRR@10 of the target article is 0.777 for Bg2Vec, 0.725 for multilingual-e5-large, 0.283 for BM25 and 0.025 for BgGPT. Ten queries are a showcase, not a benchmark.
160
+
161
+ ## Limitations
162
+
163
+ - **Size.** The model has 7.1B parameters (14 GB in bf16) and gives 4096-dimension vectors. On one L40S, it encodes approximately 48 passages per second at 256 tokens or less. A smaller multilingual encoder can be a better choice for many tasks.
164
+ - **Text length.** MNTP used chunks of 512 tokens and SimCSE used texts of 128 tokens. We did not test texts longer than 512 tokens.
165
+ - **No text generation.** The repository has no language-model head. Use the model only to encode text.
166
+ - **Unsupervised training.** The training used only Wikipedia text, with no labelled pairs and no instructions. The model is weaker on clustering and topic classification than BgGPT, and weaker than bge-multilingual-gemma2 on most tasks that are not retrieval.
167
+ - **Rare tokens.** The model can miss exact matches on codes and rare terms (for example, "ISO 8601"). For such queries, use it together with lexical search such as BM25.
168
+ - **Bias.** The model can have the biases of Bulgarian Wikipedia and of BgGPT.
169
+
170
+ ## Citation
171
+
172
+ ```bibtex
173
+ @misc{boyanov2024bg2vec,
174
+ author = {Martin Boyanov},
175
+ title = {Let's Build Bg2Vec: Apply LLM2Vec on BgGPT},
176
+ year = {2024},
177
+ url = {https://mboyanov.github.io/2024/05/18/Bg2Vec.html}
178
+ }
179
+
180
+ @article{behnamghader2024llm2vec,
181
+ title = {LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders},
182
+ author = {BehnamGhader, Parishad and Adlakha, Vaibhav and Mosbach, Marius and Bahdanau, Dzmitry and Chapados, Nicolas and Reddy, Siva},
183
+ journal = {arXiv preprint arXiv:2404.05961},
184
+ year = {2024}
185
+ }
186
+ ```