AllTomInrae commited on
Commit
5f48e29
·
verified ·
1 Parent(s): 972f017

Upload folder using huggingface_hub

Browse files
1_Pooling/config.json ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ {
2
+ "embedding_dimension": 384,
3
+ "pooling_mode": "mean",
4
+ "include_prompt": true
5
+ }
README.md CHANGED
@@ -1,3 +1,353 @@
1
  ---
 
 
 
 
 
 
 
 
 
 
 
 
2
  license: apache-2.0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ tags:
3
+ - sentence-transformers
4
+ - sentence-similarity
5
+ - feature-extraction
6
+ - generated_from_trainer
7
+ - dataset_size:13284
8
+ - loss:MultipleNegativesRankingLoss
9
+ - domain:agri-food
10
+ - task:ontology-alignment
11
+ base_model: sentence-transformers/all-MiniLM-L6-v2
12
+ language:
13
+ - en
14
  license: apache-2.0
15
+ pipeline_tag: sentence-similarity
16
+ library_name: sentence-transformers
17
+ metrics:
18
+ - cosine_accuracy@1
19
+ - cosine_accuracy@3
20
+ - cosine_mrr@10
21
+ - cosine_ndcg@10
22
+ model-index:
23
+ - name: Ows2-MiniLM
24
+ results:
25
+ - task:
26
+ type: information-retrieval
27
+ name: Information Retrieval
28
+ dataset:
29
+ name: Validation set (InformationRetrievalEvaluator — 2238 queries, ~14 400 corpus terms)
30
+ type: val_evaluator
31
+ metrics:
32
+ - type: cosine_accuracy@1
33
+ value: 0.3968
34
+ name: Cosine Accuracy@1
35
+ - type: cosine_accuracy@3
36
+ value: 0.5366
37
+ name: Cosine Accuracy@3
38
+ - type: cosine_accuracy@5
39
+ value: 0.5920
40
+ name: Cosine Accuracy@5
41
+ - type: cosine_ndcg@10
42
+ value: 0.5185
43
+ name: Cosine NDCG@10
44
+ - type: cosine_mrr@10
45
+ value: 0.4808
46
+ name: Cosine MRR@10
47
+ - task:
48
+ type: information-retrieval
49
+ name: Goldtest — Standard IR (goldtest_evaluator — 100 expert queries, ~14 600 corpus terms)
50
+ dataset:
51
+ name: OWS2 Goldtest
52
+ type: goldtest
53
+ metrics:
54
+ - type: cosine_accuracy@1
55
+ value: 0.55
56
+ name: Cosine Accuracy@1
57
+ - type: cosine_accuracy@3
58
+ value: 0.75
59
+ name: Cosine Accuracy@3
60
+ - type: cosine_mrr@10
61
+ value: 0.6663
62
+ name: Cosine MRR@10
63
+ - task:
64
+ type: information-retrieval
65
+ name: Goldtest — Exact/Taboo Benchmark (CombinedGoldtestEvaluator — 100 expert queries, 14 544 unique terms)
66
+ dataset:
67
+ name: OWS2 Goldtest (Deduplicated)
68
+ type: goldtest_combined
69
+ metrics:
70
+ - type: exact_match_accuracy@1
71
+ value: 0.6618
72
+ name: Exact Match Accuracy@1
73
+ - type: exact_match_mrr@10
74
+ value: 0.7394
75
+ name: Exact Match MRR@10
76
+ - type: taboo_match_mrr@10
77
+ value: 0.5448
78
+ name: Taboo Match MRR@10
79
+ - type: semantic_margin
80
+ value: 0.1780
81
+ name: Semantic Discrimination Margin
82
  ---
83
+
84
+ # Ows2-MiniLM — Ontology Alignment Model for Agri-Food Domains
85
+
86
+ **Ows2-MiniLM** is a sentence embedding model fine-tuned for **semantic alignment between scientific text and agri-food ontology terms**.
87
+ It is part of the **OWS² (Ontology-based Workflow for Semantic Specialisation)** project developed at [INRAE](https://www.inrae.fr/) (French National Research Institute for Agriculture, Food and Environment).
88
+
89
+ Given a sentence extracted from a scientific abstract or publication, the model retrieves the most semantically relevant ontology terms from a reference vocabulary of ~14 600 concepts covering plant science, food science, and agronomy domains.
90
+
91
+ The reference vocabulary incorporates concepts from:
92
+ - **TransformON**: [Nature Publication](https://www.nature.com/articles/s41538-023-00221-2)
93
+ - **Planteome**: [PubMed Publication](https://pubmed.ncbi.nlm.nih.gov/38055832/)
94
+
95
+ > **Acknowledgement:** This work was supported by **INRAE Metaprogramme DIGIT-BIO** within the framework of the **SEED** project.
96
+
97
+ ---
98
+
99
+ ## Model Details
100
+
101
+ | Property | Value |
102
+ |:---|:---|
103
+ | **Model name** | Ows2-MiniLM |
104
+ | **Base model** | [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) |
105
+ | **Architecture** | Bi-encoder (Transformer + Mean Pooling + L2 Normalize) |
106
+ | **Output dimensionality** | 384 dimensions |
107
+ | **Maximum sequence length** | 512 tokens |
108
+ | **Similarity function** | Cosine similarity |
109
+ | **License** | [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0) (inherits from base model) |
110
+ | **Training loss** | MultipleNegativesRankingLoss (asymmetric, in-batch negatives) |
111
+ | **Language** | English |
112
+ | **Domain** | Agri-food science (plant biology, food science, agronomy) |
113
+
114
+ ### Full Model Architecture
115
+
116
+ ```
117
+ SentenceTransformer(
118
+ (0): Transformer({'architecture': 'BertModel'})
119
+ (1): Pooling({'pooling_mode': 'mean'})
120
+ (2): Normalize({})
121
+ )
122
+ ```
123
+
124
+ ---
125
+
126
+ ## Intended Use
127
+
128
+ This model is designed for **ontology-based semantic search** in scientific literature, specifically:
129
+
130
+ - **Automatic annotation** of scientific abstracts with ontology concepts
131
+ - **Semantic retrieval** of the most relevant agri-food ontology terms given a sentence
132
+ - **Alignment** between free-text descriptions and controlled vocabularies (e.g., PPDO, AgroVoc-related thesauri)
133
+
134
+ > **Out-of-scope use:** This model is specialized for agri-food science. Performance on general-purpose or non-scientific text may be significantly lower.
135
+
136
+ ---
137
+
138
+ ## Usage
139
+
140
+ ```python
141
+ from sentence_transformers import SentenceTransformer, util
142
+
143
+ model = SentenceTransformer("tcolombu/Ows2-MiniLM_1")
144
+
145
+ # A sentence from a scientific abstract
146
+ query = "Fruit-based sauces exhibited significant variations in antioxidant activity among different cultivars."
147
+
148
+ # Ontology terms to rank
149
+ ontology_terms = [
150
+ "fruit sauce",
151
+ "antioxidant activity",
152
+ "barley powdery mildew disease response",
153
+ "rootless",
154
+ ]
155
+
156
+ # Encode and compute cosine similarities
157
+ query_embedding = model.encode(query, convert_to_tensor=True)
158
+ term_embeddings = model.encode(ontology_terms, convert_to_tensor=True)
159
+
160
+ scores = util.cos_sim(query_embedding, term_embeddings)[0]
161
+ ranked = sorted(zip(ontology_terms, scores.tolist()), key=lambda x: x[1], reverse=True)
162
+
163
+ for term, score in ranked:
164
+ print(f"{score:.4f} {term}")
165
+ # Expected top result: fruit sauce
166
+ ```
167
+
168
+ ---
169
+
170
+ ## Evaluation
171
+
172
+ Two distinct evaluation protocols were used, each measuring different aspects of model performance. **Both are computed on the same 100-query expert goldtest**, but differ in corpus construction and scoring logic.
173
+
174
+ ### Evaluator 1 — Standard IR (`InformationRetrievalEvaluator`)
175
+
176
+ > Called during training via `python main.py finetune --config <yaml>`
177
+
178
+ This is the standard `InformationRetrievalEvaluator` from the `sentence-transformers` library. Each ontology term is registered with a composite key `"{ontology_id}_{term}"`, meaning the **same surface form can appear multiple times** in the corpus if it has multiple ontology IDs. Relevance is checked by exact `doc_id` match only.
179
+
180
+ - **Corpus size:** ~14 600 entries (with duplicates)
181
+ - **Scoring:** Single exact `doc_id` match required
182
+ - **Stored in:** `metrics_comparison.json`
183
+
184
+ | Dataset | Acc@1 | Acc@3 | MRR@10 | NDCG@10 |
185
+ |:---|:---:|:---:|:---:|:---:|
186
+ | Validation set (15% of training data) | 39.68% | 53.66% | 48.08% | 51.85% |
187
+ | **Goldtest** (100 expert queries) | **55.00%** | **75.00%** | **66.63%** | 44.35% |
188
+
189
+ ---
190
+
191
+ ### Evaluator 2 — Combined Goldtest Benchmark (`CombinedGoldtestEvaluator`)
192
+
193
+ > Called post-training via `python main.py eval_model <model_path>`
194
+
195
+ A custom three-component evaluator. Terms are **deduplicated** to 14 544 unique surface forms. Synonymous ontology IDs sharing the same term are **merged**: a prediction is correct if any valid ID for the target term is retrieved. Queries are split into two subsets:
196
+
197
+ - **Exact Match**: the target term appears verbatim in the query sentence
198
+ - **Taboo Match**: the target term is *absent* from the query — pure semantic understanding required
199
+
200
+ | Subset | Acc@1 | Acc@3 | MRR@10 |
201
+ |:---|:---:|:---:|:---:|
202
+ | **Exact Match** | **66.18%** | **79.41%** | **73.94%** |
203
+ | **Taboo Match** | 37.50% | 68.75% | 54.48% |
204
+
205
+ | Semantic metric | Value |
206
+ |:---|:---:|
207
+ | Mean similarity (positive targets) | 0.5580 |
208
+ | Mean similarity (distractors) | 0.3801 |
209
+ | **Discrimination margin** | **0.1780** |
210
+ | Discrimination accuracy | 86.36% |
211
+
212
+ *Stored in: `goldtest_eval/combined_goldtest_metrics.json`*
213
+
214
+ > **Reading the two scores:** Evaluator 1 (66.63% MRR) is stricter because duplicated corpus entries act as additional distractors. Evaluator 2 (73.94% MRR) is more semantically faithful because synonyms of the correct answer are also accepted as valid.
215
+
216
+ ---
217
+
218
+ ### Baseline Comparison — `all-MiniLM-L6-v2` (no fine-tuning) vs `Ows2-MiniLM`
219
+
220
+ All metrics below are computed on **our agri-food ontology corpus** (not the standard sentence-transformers benchmarks). The base model has never seen this vocabulary.
221
+
222
+ #### Validation set — InformationRetrievalEvaluator (2 238 queries, ~14 400 corpus terms)
223
+
224
+ | Metric | Base `all-MiniLM-L6-v2` | **Ows2-MiniLM** | Δ Gain |
225
+ |:---|:---:|:---:|:---:|
226
+ | Accuracy@1 | 23.15% | **39.68%** | +16.5 pts |
227
+ | Accuracy@3 | 35.17% | **53.66%** | +18.5 pts |
228
+ | Accuracy@5 | 41.51% | **59.20%** | +17.7 pts |
229
+ | NDCG@10 | 34.46% | **51.85%** | +17.4 pts |
230
+ | MRR@10 | 30.76% | **48.08%** | +17.3 pts |
231
+
232
+ #### Goldtest — Expert benchmark (100 queries, ~14 600 corpus terms)
233
+
234
+ | Metric | Base `all-MiniLM-L6-v2` | **Ows2-MiniLM** | Δ Gain |
235
+ |:---|:---:|:---:|:---:|
236
+ | Accuracy@1 | 22.00% | **55.00%** | **+33.0 pts** |
237
+ | Accuracy@3 | 41.00% | **75.00%** | **+34.0 pts** |
238
+ | Accuracy@5 | 52.00% | **84.00%** | +32.0 pts |
239
+ | NDCG@10 | 24.16% | **44.35%** | +20.2 pts |
240
+ | MRR@10 | 34.81% | **66.63%** | **+31.8 pts** |
241
+
242
+ Fine-tuning on the OWS² mixed dataset (real + synthetic pairs) provides a **~+32 points absolute gain on Acc@1** and **~+32 points on MRR@10** on the expert goldtest, demonstrating the strong impact of domain specialisation even with limited real annotated data (~491 expert examples).
243
+
244
+ ---
245
+
246
+ ## Training Details
247
+
248
+ ### Training Data
249
+
250
+ The model was trained on a **mixed dataset** combining real annotated data and LLM-generated synthetic pairs:
251
+
252
+ | Source | Examples | Description |
253
+ |:---|:---:|:---|
254
+ | Real annotations | ~491 | Expert-annotated (sentence, ontology term) pairs from scientific abstracts |
255
+ | Synthetic data | ~14 428 | Sentence/term pairs generated with **[Qwen](https://huggingface.co/Qwen)** (local inference), covering the full ontology vocabulary |
256
+ | **Total (85% train split)** | **~13 284** | Used for training |
257
+ | Validation (15%) | ~2 341 | Used for in-training evaluation |
258
+
259
+ - **Format:** `(anchor_sentence, positive_term, hard_negative_term)` triplets
260
+ - **Loss:** `MultipleNegativesRankingLoss` — uses all other items in the batch as implicit negatives
261
+ - **Synthetic generation LLM:** [Qwen](https://huggingface.co/Qwen) — run locally (Qwen2+ is Apache 2.0; Qwen1.x Tongyi license is compatible with domain-specific fine-tuned model publication)
262
+ - **Dataset file:** `mixed_dataset_v0_full.jsonl`
263
+
264
+ ### Training Hyperparameters
265
+
266
+ | Parameter | Value |
267
+ |:---|:---:|
268
+ | Batch size | **16** |
269
+ | Learning rate | **2e-5** |
270
+ | Epochs | **15** |
271
+ | Warmup ratio | 0.1 |
272
+ | LR scheduler | Linear |
273
+ | Optimizer | AdamW |
274
+ | Freeze layers | 0 (full fine-tuning) |
275
+ | Seed | 42 |
276
+
277
+ > **Note on hyperparameter selection:** Batch size 16 with LR 2e-5 over 15 epochs was identified as the optimal configuration after a systematic grid search across 44 experiments. Increasing epochs beyond 15 showed no further gain (convergence plateau confirmed).
278
+
279
+ ### Training Logs (validation NDCG@10 over 15 epochs)
280
+
281
+ <details><summary>Click to expand</summary>
282
+
283
+ | Epoch | Step | Training Loss | val_evaluator_cosine_ndcg@10 |
284
+ |:---:|:---:|:---:|:---:|
285
+ | 0.12 | 100 | 1.0499 | 0.3849 |
286
+ | 0.96 | 800 | 0.6236 | 0.4861 |
287
+ | 1.93 | 1600 | 0.4212 | 0.5072 |
288
+ | 3.01 | 2500 | 0.3228 | 0.5107 |
289
+ | 5.05 | 4200 | 0.1608 | 0.5094 |
290
+ | 7.10 | 5900 | 0.1085 | 0.5137 |
291
+ | 9.03 | 7500 | 0.0890 | 0.5177 |
292
+ | 11.07 | 9200 | 0.0667 | 0.5170 |
293
+ | 13.00 | 10800 | 0.0537 | 0.5173 |
294
+ | **15.00** | **12465** | — | **0.5185** |
295
+
296
+ </details>
297
+
298
+ ### Training Time
299
+
300
+ - **Training:** ~10.7 minutes (NVIDIA GPU, SLURM cluster)
301
+ - **Evaluation:** ~7.8 minutes
302
+ - **Total:** ~18.5 minutes
303
+
304
+ ---
305
+
306
+ ## Limitations
307
+
308
+ - **Domain specificity:** The model is fine-tuned on agri-food science vocabulary. Performance on out-of-domain text (e.g., clinical, legal) is expected to be significantly lower.
309
+ - **Ontology coverage:** The reference ontology contains ~14 600 terms. Terms absent from this vocabulary cannot be retrieved.
310
+ - **Real data scarcity:** Only ~491 real annotated examples were available. Performance is expected to improve as the real annotation corpus grows.
311
+ - **Taboo match gap:** The 19-point gap between Exact Match MRR (73.94%) and Taboo Match MRR (54.48%) indicates the model still relies partially on lexical overlap. Improving semantic generalization is an identified next step.
312
+
313
+ ---
314
+
315
+ ## Framework Versions
316
+
317
+ - Python: 3.10.20
318
+ - Sentence Transformers: 5.6.0
319
+ - Transformers: 5.14.1
320
+ - PyTorch: 2.10.0+cu128
321
+ - Accelerate: 1.14.0
322
+ - Datasets: 5.0.0
323
+
324
+ ---
325
+
326
+ ## Citation
327
+
328
+ If you use this model, please cite:
329
+
330
+ ```bibtex
331
+ @misc{ows2-minilm-2025,
332
+ title = {Ows2-MiniLM: An Ontology Alignment Model for Agri-Food Science},
333
+ institution = {INRAE},
334
+ note = {OWS\textsuperscript{2} -- Ontology-based Workflow for Semantic Specialisation},
335
+ howpublished = {\url{https://huggingface.co/tcolombu/Ows2-MiniLM_1}}
336
+ }
337
+ ```
338
+
339
+ ```bibtex
340
+ @inproceedings{reimers-2019-sentence-bert,
341
+ title = {Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks},
342
+ author = {Reimers, Nils and Gurevych, Iryna},
343
+ booktitle = {Proceedings of EMNLP 2019},
344
+ year = {2019},
345
+ url = {https://arxiv.org/abs/1908.10084}
346
+ }
347
+ ```
348
+
349
+ ### Related Poster
350
+
351
+ If you refer to this work, please also cite the related poster:
352
+ * **HAL Id:** `hal-05681918`
353
+ * **URL:** [https://hal.science/hal-05681918v1](https://hal.science/hal-05681918v1)
config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_cross_attention": false,
3
+ "architectures": [
4
+ "BertModel"
5
+ ],
6
+ "attention_probs_dropout_prob": 0.1,
7
+ "bos_token_id": null,
8
+ "classifier_dropout": null,
9
+ "dtype": "float32",
10
+ "eos_token_id": null,
11
+ "gradient_checkpointing": false,
12
+ "hidden_act": "gelu",
13
+ "hidden_dropout_prob": 0.1,
14
+ "hidden_size": 384,
15
+ "initializer_range": 0.02,
16
+ "intermediate_size": 1536,
17
+ "is_decoder": false,
18
+ "layer_norm_eps": 1e-12,
19
+ "max_position_embeddings": 512,
20
+ "model_type": "bert",
21
+ "num_attention_heads": 12,
22
+ "num_hidden_layers": 6,
23
+ "pad_token_id": 0,
24
+ "position_embedding_type": "absolute",
25
+ "tie_word_embeddings": true,
26
+ "transformers_version": "5.14.1",
27
+ "type_vocab_size": 2,
28
+ "use_cache": false,
29
+ "vocab_size": 30522
30
+ }
config_sentence_transformers.json ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "__version__": {
3
+ "pytorch": "2.10.0+cu128",
4
+ "sentence_transformers": "5.6.0",
5
+ "transformers": "5.14.1"
6
+ },
7
+ "default_prompt_name": null,
8
+ "model_type": "SentenceTransformer",
9
+ "prompts": {
10
+ "document": "",
11
+ "query": ""
12
+ },
13
+ "similarity_fn_name": "cosine"
14
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4f6376f818fc35fc441f71467256b17ddf5699eacf17f7b4a966ecdba31b40d2
3
+ size 90864192
modules.json ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "idx": 0,
4
+ "name": "0",
5
+ "path": "",
6
+ "type": "sentence_transformers.base.modules.transformer.Transformer"
7
+ },
8
+ {
9
+ "idx": 1,
10
+ "name": "1",
11
+ "path": "1_Pooling",
12
+ "type": "sentence_transformers.sentence_transformer.modules.pooling.Pooling"
13
+ },
14
+ {
15
+ "idx": 2,
16
+ "name": "2",
17
+ "path": "2_Normalize",
18
+ "type": "sentence_transformers.sentence_transformer.modules.normalize.Normalize"
19
+ }
20
+ ]
sentence_bert_config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "transformer_task": "feature-extraction",
3
+ "modality_config": {
4
+ "text": {
5
+ "method": "forward",
6
+ "method_output_name": "last_hidden_state"
7
+ }
8
+ },
9
+ "module_output_name": "token_embeddings"
10
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "backend": "tokenizers",
3
+ "cls_token": "[CLS]",
4
+ "do_basic_tokenize": true,
5
+ "do_lower_case": true,
6
+ "is_local": false,
7
+ "local_files_only": false,
8
+ "mask_token": "[MASK]",
9
+ "max_length": 128,
10
+ "model_max_length": 512,
11
+ "never_split": null,
12
+ "pad_to_multiple_of": null,
13
+ "pad_token": "[PAD]",
14
+ "pad_token_type_id": 0,
15
+ "padding_side": "right",
16
+ "sep_token": "[SEP]",
17
+ "stride": 0,
18
+ "strip_accents": null,
19
+ "tokenize_chinese_chars": true,
20
+ "tokenizer_class": "BertTokenizer",
21
+ "truncation_side": "right",
22
+ "truncation_strategy": "longest_first",
23
+ "unk_token": "[UNK]"
24
+ }