musapeker
/

tr-en-sci-e5-base

@@ -1,124 +1,124 @@
----
-language:
-- tr
-- en
-license: cc-by-nc-4.0
-library_name: sentence-transformers
-pipeline_tag: sentence-similarity
-base_model: intfloat/multilingual-e5-base
-tags:
-- sentence-transformers
-- feature-extraction
-- sentence-similarity
-- cross-lingual-retrieval
-- scientific-literature
-- turkish
-datasets:
-- mpeker/dergipark-tr-en-abstracts
----
-# tr-en-sci-e5-base
-**Turkish → English scientific literature retrieval.** Type a query in Turkish, find the relevant English paper
-(title + abstract). A 278M-parameter embedding model fine-tuned from
-[intfloat/multilingual-e5-base](https://huggingface.co/intfloat/multilingual-e5-base) on ~380k Turkish–English
-abstract pairs written by the authors of articles on [DergiPark](https://dergipark.org.tr).
-> **Türkçe özet:** Türkçe sorguyla İngilizce bilimsel makale bulmak için eğitilmiş bir embedding modeli.
-> DergiPark'taki yazarların kendi yazdığı Türkçe–İngilizce başlık, özet ve anahtar kelime çiftleriyle eğitildi.
-> Eğitimde hiç görülmemiş dergilerden oluşan test kümesinde, kendisinin yaklaşık iki katı büyüklükteki
-> bge-m3 ve turkish-e5-large modellerini geçiyor.
-## Results
-Public test set of [`mpeker/dergipark-tr-en-abstracts`](https://huggingface.co/datasets/mpeker/dergipark-tr-en-abstracts):
-16,983 English documents (title + abstract) from **journals never seen in training**, 2,000 Turkish queries per
-query type, exactly one relevant document per query (the same article).
-| Model | Title nDCG@10 | Title R@1 | Title R@10 | Title nDCG@10 (no shared words) | Keywords nDCG@10 | Keywords R@1 | Keywords R@10 |
-|---|---|---|---|---|---|---|---|
-| **tr-en-sci-e5-base (this model, 278M)** | **0.979** | **0.960** | **0.995** | **0.971** | **0.927** | **0.868** | **0.981** |
-| bge-m3 (568M) | 0.944 | 0.908 | 0.979 | 0.922 | 0.834 | 0.743 | 0.922 |
-| ytu-ce-cosmos/turkish-e5-large (560M) | 0.942 | 0.907 | 0.974 | 0.927 | 0.890 | 0.826 | 0.953 |
-| multilingual-e5-large-instruct (560M) | 0.889 | 0.840 | 0.934 | 0.850 | 0.789 | 0.687 | 0.886 |
-| Qwen3-Embedding-0.6B | 0.819 | 0.749 | 0.888 | 0.752 | 0.775 | 0.680 | 0.864 |
-| multilingual-e5-base (base model, 278M) | 0.760 | 0.671 | 0.849 | 0.672 | 0.668 | 0.559 | 0.783 |
-| MT (opus-mt-tr-en) + e5-base-v2 | 0.695 | 0.649 | 0.742 | 0.660 | 0.845 | 0.771 | 0.916 |
-| MT (opus-mt-tr-en) + BM25 | 0.645 | 0.589 | 0.703 | 0.578 | 0.762 | 0.674 | 0.848 |
-| BM25 (Turkish query on English docs) | 0.292 | 0.221 | 0.370 | 0.095 | 0.462 | 0.306 | 0.641 |
-- **Title** queries: the Turkish title of the article. **Keywords** queries: the Turkish author keywords, joined with `; `.
-- **No shared words**: the subset of title queries (1,150 of 2,000) that share no content word (≥3 letters, no digits)
-  with their target document, i.e. names, years and international terms cannot help. This is the purely
-  cross-lingual part of the task.
-- R@k: share of queries whose relevant document is in the top k.
-## Usage
-The model follows the E5 convention: prefix queries with `query: ` and documents with `passage: `.
-```python
-from sentence_transformers import SentenceTransformer
-model = SentenceTransformer("mpeker/tr-en-sci-e5-base")
-queries = ["query: göz dibi görüntülerinde derin öğrenme ile diyabetik retinopati tespiti"]
-docs = [
-    "passage: Deep learning based detection of diabetic retinopathy from fundus images using convolutional neural networks.",
-    "passage: The effect of climate change on wheat yield in Central Anatolia: a regression analysis.",
-]
-q_emb = model.encode(queries, normalize_embeddings=True)
-d_emb = model.encode(docs, normalize_embeddings=True)
-print(model.similarity(q_emb, d_emb))  # first document scores highest
-```
-For best results, index documents as `passage: <English title>. <English abstract>`.
-## Training
-| | |
-|---|---|
-| Base model | intfloat/multilingual-e5-base (XLM-RoBERTa, 278M) |
-| Training data | 379,792 Turkish–English article pairs from 2,222 DergiPark journals (journal-disjoint from dev/test) |
-| Training rows | 837,539 (Turkish title → English doc, Turkish keywords → English doc, and Turkish abstract → English doc for 30% of articles) |
-| Loss | CachedMultipleNegativesRankingLoss (GradCache), in-batch negatives |
-| Batch size | 256 (GradCache mini-batch 16), no-duplicates sampler so one article never appears twice in a batch |
-| Learning rate / schedule | 2e-5, 10% linear warmup, 1 epoch (3,271 steps) |
-| Max sequence length | 256 tokens |
-| Precision / hardware | fp16, one RTX 4060 8 GB, 5 h 04 min |
-| Seed | 42 |
-Development-set nDCG@10 (10,000 documents, Turkish title queries) rose from 0.812 (base model) to 0.987.
-## Data and splits
-Pairs were harvested from DergiPark's public OAI-PMH interface (843,394 records, October 2026). Both language
-sides were checked with a language heuristic, mis-tagged sides were swapped, copied titles removed, and pairs
-filtered by length, length ratio and exact duplicates, leaving 447,244 pairs. Splits are **by journal**:
-train 2,222 journals / dev 146 / test 417, stratified by subject area, so test journals and their topics are
-never seen in training.
-## Limitations
-- **Known-item evaluation.** Each query has exactly one relevant document: its own article. Real searches can
-  have many relevant papers; scores here measure how well the model links a Turkish description to the right
-  English paper, not full topical relevance.
-- **Query style.** Training and test queries are titles, keyword lists and abstracts, not natural-language
-  questions. Conversational queries may perform worse.
-- **Domain.** Documents are abstracts of Turkish journals across all fields (education, health, engineering,
-  social sciences, humanities). Performance on other English corpora (e.g. PubMed) has not been measured yet.
-- **Truncation.** Trained with a 256-token limit; about 65% of English title+abstract documents are longer and
-  were truncated during training.
-- **Baselines' training data.** Some baselines (e.g. turkish-e5-large, whose training data is not documented) may
-  have seen DergiPark text during their own training.
-## License and data provenance
-The model is released under **CC BY-NC 4.0** (non-commercial). It was trained on abstracts from all harvested
-DergiPark journals, including journals that publish under non-commercial licenses or state no license. The model
-weights do not redistribute these texts. The companion dataset contains only pairs from journals with a
-Creative Commons license that allows derivatives (CC BY, BY-SA, BY-NC, BY-NC-SA); every row has a `license` column.
-## Citation
-A paper is in preparation. Until then, please cite this repository.

+---
+language:
+- tr
+- en
+license: cc-by-nc-4.0
+library_name: sentence-transformers
+pipeline_tag: sentence-similarity
+base_model: intfloat/multilingual-e5-base
+tags:
+- sentence-transformers
+- feature-extraction
+- sentence-similarity
+- cross-lingual-retrieval
+- scientific-literature
+- turkish
+datasets:
+- musapeker/dergipark-tr-en-abstracts
+---
+# tr-en-sci-e5-base
+**Turkish → English scientific literature retrieval.** Type a query in Turkish, find the relevant English paper
+(title + abstract). A 278M-parameter embedding model fine-tuned from
+[intfloat/multilingual-e5-base](https://huggingface.co/intfloat/multilingual-e5-base) on ~380k Turkish–English
+abstract pairs written by the authors of articles on [DergiPark](https://dergipark.org.tr).
+> **Türkçe özet:** Türkçe sorguyla İngilizce bilimsel makale bulmak için eğitilmiş bir embedding modeli.
+> DergiPark'taki yazarların kendi yazdığı Türkçe–İngilizce başlık, özet ve anahtar kelime çiftleriyle eğitildi.
+> Eğitimde hiç görülmemiş dergilerden oluşan test kümesinde, kendisinin yaklaşık iki katı büyüklükteki
+> bge-m3 ve turkish-e5-large modellerini geçiyor.
+## Results
+Public test set of [`musapeker/dergipark-tr-en-abstracts`](https://huggingface.co/datasets/musapeker/dergipark-tr-en-abstracts):
+16,983 English documents (title + abstract) from **journals never seen in training**, 2,000 Turkish queries per
+query type, exactly one relevant document per query (the same article).
+| Model | Title nDCG@10 | Title R@1 | Title R@10 | Title nDCG@10 (no shared words) | Keywords nDCG@10 | Keywords R@1 | Keywords R@10 |
+|---|---|---|---|---|---|---|---|
+| **tr-en-sci-e5-base (this model, 278M)** | **0.979** | **0.960** | **0.995** | **0.971** | **0.927** | **0.868** | **0.981** |
+| bge-m3 (568M) | 0.944 | 0.908 | 0.979 | 0.922 | 0.834 | 0.743 | 0.922 |
+| ytu-ce-cosmos/turkish-e5-large (560M) | 0.942 | 0.907 | 0.974 | 0.927 | 0.890 | 0.826 | 0.953 |
+| multilingual-e5-large-instruct (560M) | 0.889 | 0.840 | 0.934 | 0.850 | 0.789 | 0.687 | 0.886 |
+| Qwen3-Embedding-0.6B | 0.819 | 0.749 | 0.888 | 0.752 | 0.775 | 0.680 | 0.864 |
+| multilingual-e5-base (base model, 278M) | 0.760 | 0.671 | 0.849 | 0.672 | 0.668 | 0.559 | 0.783 |
+| MT (opus-mt-tr-en) + e5-base-v2 | 0.695 | 0.649 | 0.742 | 0.660 | 0.845 | 0.771 | 0.916 |
+| MT (opus-mt-tr-en) + BM25 | 0.645 | 0.589 | 0.703 | 0.578 | 0.762 | 0.674 | 0.848 |
+| BM25 (Turkish query on English docs) | 0.292 | 0.221 | 0.370 | 0.095 | 0.462 | 0.306 | 0.641 |
+- **Title** queries: the Turkish title of the article. **Keywords** queries: the Turkish author keywords, joined with `; `.
+- **No shared words**: the subset of title queries (1,150 of 2,000) that share no content word (≥3 letters, no digits)
+  with their target document, i.e. names, years and international terms cannot help. This is the purely
+  cross-lingual part of the task.
+- R@k: share of queries whose relevant document is in the top k.
+## Usage
+The model follows the E5 convention: prefix queries with `query: ` and documents with `passage: `.
+```python
+from sentence_transformers import SentenceTransformer
+model = SentenceTransformer("musapeker/tr-en-sci-e5-base")
+queries = ["query: göz dibi görünt��lerinde derin öğrenme ile diyabetik retinopati tespiti"]
+docs = [
+    "passage: Deep learning based detection of diabetic retinopathy from fundus images using convolutional neural networks.",
+    "passage: The effect of climate change on wheat yield in Central Anatolia: a regression analysis.",
+]
+q_emb = model.encode(queries, normalize_embeddings=True)
+d_emb = model.encode(docs, normalize_embeddings=True)
+print(model.similarity(q_emb, d_emb))  # first document scores highest
+```
+For best results, index documents as `passage: <English title>. <English abstract>`.
+## Training
+| | |
+|---|---|
+| Base model | intfloat/multilingual-e5-base (XLM-RoBERTa, 278M) |
+| Training data | 379,792 Turkish–English article pairs from 2,222 DergiPark journals (journal-disjoint from dev/test) |
+| Training rows | 837,539 (Turkish title → English doc, Turkish keywords → English doc, and Turkish abstract → English doc for 30% of articles) |
+| Loss | CachedMultipleNegativesRankingLoss (GradCache), in-batch negatives |
+| Batch size | 256 (GradCache mini-batch 16), no-duplicates sampler so one article never appears twice in a batch |
+| Learning rate / schedule | 2e-5, 10% linear warmup, 1 epoch (3,271 steps) |
+| Max sequence length | 256 tokens |
+| Precision / hardware | fp16, one RTX 4060 8 GB, 5 h 04 min |
+| Seed | 42 |
+Development-set nDCG@10 (10,000 documents, Turkish title queries) rose from 0.812 (base model) to 0.987.
+## Data and splits
+Pairs were harvested from DergiPark's public OAI-PMH interface (843,394 records, October 2026). Both language
+sides were checked with a language heuristic, mis-tagged sides were swapped, copied titles removed, and pairs
+filtered by length, length ratio and exact duplicates, leaving 447,244 pairs. Splits are **by journal**:
+train 2,222 journals / dev 146 / test 417, stratified by subject area, so test journals and their topics are
+never seen in training.
+## Limitations
+- **Known-item evaluation.** Each query has exactly one relevant document: its own article. Real searches can
+  have many relevant papers; scores here measure how well the model links a Turkish description to the right
+  English paper, not full topical relevance.
+- **Query style.** Training and test queries are titles, keyword lists and abstracts, not natural-language
+  questions. Conversational queries may perform worse.
+- **Domain.** Documents are abstracts of Turkish journals across all fields (education, health, engineering,
+  social sciences, humanities). Performance on other English corpora (e.g. PubMed) has not been measured yet.
+- **Truncation.** Trained with a 256-token limit; about 65% of English title+abstract documents are longer and
+  were truncated during training.
+- **Baselines' training data.** Some baselines (e.g. turkish-e5-large, whose training data is not documented) may
+  have seen DergiPark text during their own training.
+## License and data provenance
+The model is released under **CC BY-NC 4.0** (non-commercial). It was trained on abstracts from all harvested
+DergiPark journals, including journals that publish under non-commercial licenses or state no license. The model
+weights do not redistribute these texts. The companion dataset contains only pairs from journals with a
+Creative Commons license that allows derivatives (CC BY, BY-SA, BY-NC, BY-NC-SA); every row has a `license` column.
+## Citation
+A paper is in preparation. Until then, please cite this repository.