| --- |
| language: |
| - tr |
| - en |
| license: cc-by-nc-4.0 |
| library_name: sentence-transformers |
| pipeline_tag: sentence-similarity |
| base_model: intfloat/multilingual-e5-base |
| tags: |
| - sentence-transformers |
| - feature-extraction |
| - sentence-similarity |
| - cross-lingual-retrieval |
| - scientific-literature |
| - turkish |
| datasets: |
| - musapeker/dergipark-tr-en-abstracts |
| --- |
| |
| # tr-en-sci-e5-base |
|
|
| **Turkish → English scientific literature retrieval.** Type a query in Turkish, find the relevant English paper |
| (title + abstract). A 278M-parameter embedding model fine-tuned from |
| [intfloat/multilingual-e5-base](https://huggingface.co/intfloat/multilingual-e5-base) on ~380k Turkish–English |
| abstract pairs written by the authors of articles on [DergiPark](https://dergipark.org.tr). |
|
|
| **Demo:** [huggingface.co/spaces/musapeker/tr-en-sci-search](https://huggingface.co/spaces/musapeker/tr-en-sci-search) |
| (runs entirely in the browser). |
|
|
| > **Türkçe özet:** Türkçe sorguyla İngilizce bilimsel makale bulmak için eğitilmiş bir embedding modeli. |
| > DergiPark'taki yazarların kendi yazdığı Türkçe–İngilizce başlık, özet ve anahtar kelime çiftleriyle eğitildi. |
| > Eğitimde hiç görülmemiş dergilerden oluşan test kümesinde, kendisinin yaklaşık iki katı büyüklükteki |
| > bge-m3 ve turkish-e5-large modellerini geçiyor. |
|
|
| ## Results |
|
|
| Public test set of [`musapeker/dergipark-tr-en-abstracts`](https://huggingface.co/datasets/musapeker/dergipark-tr-en-abstracts): |
| 16,983 English documents (title + abstract) from **journals never seen in training**, 2,000 Turkish queries per |
| query type, exactly one relevant document per query (the same article). |
|
|
| | Model | Title nDCG@10 | Title R@1 | Title R@10 | Title nDCG@10 (no shared words) | Keywords nDCG@10 | Keywords R@1 | Keywords R@10 | |
| |---|---|---|---|---|---|---|---| |
| | **tr-en-sci-e5-base (this model, 278M)** | **0.979** | **0.960** | **0.995** | **0.971** | **0.927** | **0.868** | **0.981** | |
| | bge-m3 (568M) | 0.944 | 0.908 | 0.979 | 0.922 | 0.834 | 0.743 | 0.922 | |
| | ytu-ce-cosmos/turkish-e5-large (560M) | 0.942 | 0.907 | 0.974 | 0.927 | 0.890 | 0.826 | 0.953 | |
| | multilingual-e5-large-instruct (560M) | 0.889 | 0.840 | 0.934 | 0.850 | 0.789 | 0.687 | 0.886 | |
| | Qwen3-Embedding-0.6B | 0.819 | 0.749 | 0.888 | 0.752 | 0.775 | 0.680 | 0.864 | |
| | multilingual-e5-base (base model, 278M) | 0.760 | 0.671 | 0.849 | 0.672 | 0.668 | 0.559 | 0.783 | |
| | MT (opus-mt-tr-en) + e5-base-v2 | 0.695 | 0.649 | 0.742 | 0.660 | 0.845 | 0.771 | 0.916 | |
| | MT (opus-mt-tr-en) + BM25 | 0.645 | 0.589 | 0.703 | 0.578 | 0.762 | 0.674 | 0.848 | |
| | BM25 (Turkish query on English docs) | 0.292 | 0.221 | 0.370 | 0.095 | 0.462 | 0.306 | 0.641 | |
|
|
| - **Title** queries: the Turkish title of the article. **Keywords** queries: the Turkish author keywords, joined with `; `. |
| - **No shared words**: the subset of title queries (1,150 of 2,000) that share no content word (≥3 letters, no digits) |
| with their target document, i.e. names, years and international terms cannot help. This is the purely |
| cross-lingual part of the task. |
| - R@k: share of queries whose relevant document is in the top k. |
|
|
| ## Usage |
|
|
| The model follows the E5 convention: prefix queries with `query: ` and documents with `passage: `. |
|
|
| ```python |
| from sentence_transformers import SentenceTransformer |
| |
| model = SentenceTransformer("musapeker/tr-en-sci-e5-base") |
| |
| queries = ["query: göz dibi görüntülerinde derin öğrenme ile diyabetik retinopati tespiti"] |
| docs = [ |
| "passage: Deep learning based detection of diabetic retinopathy from fundus images using convolutional neural networks.", |
| "passage: The effect of climate change on wheat yield in Central Anatolia: a regression analysis.", |
| ] |
| q_emb = model.encode(queries, normalize_embeddings=True) |
| d_emb = model.encode(docs, normalize_embeddings=True) |
| print(model.similarity(q_emb, d_emb)) # first document scores highest |
| ``` |
|
|
| For best results, index documents as `passage: <English title>. <English abstract>`. |
|
|
| ### In the browser (transformers.js) |
|
|
| An int8 ONNX export (`onnx/model_quantized.onnx`, 279 MB) is included. On 1,000 test queries against 1,000 documents, |
| R@1 is 0.990 for the PyTorch model and 0.988 when int8 query embeddings are matched against PyTorch document embeddings. |
|
|
| ```js |
| import { pipeline } from "@huggingface/transformers"; |
| |
| const extractor = await pipeline("feature-extraction", "musapeker/tr-en-sci-e5-base", { dtype: "q8" }); |
| const out = await extractor(["query: tahrir defterleri ve Osmanlı nüfus tarihi"], { pooling: "mean", normalize: true }); |
| ``` |
|
|
| ## Training |
|
|
| | | | |
| |---|---| |
| | Base model | intfloat/multilingual-e5-base (XLM-RoBERTa, 278M) | |
| | Training data | 379,792 Turkish–English article pairs from 2,222 DergiPark journals (journal-disjoint from dev/test) | |
| | Training rows | 837,539 (Turkish title → English doc, Turkish keywords → English doc, and Turkish abstract → English doc for 30% of articles) | |
| | Loss | CachedMultipleNegativesRankingLoss (GradCache), in-batch negatives | |
| | Batch size | 256 (GradCache mini-batch 16), no-duplicates sampler so one article never appears twice in a batch | |
| | Learning rate / schedule | 2e-5, 10% linear warmup, 1 epoch (3,271 steps) | |
| | Max sequence length | 256 tokens | |
| | Precision / hardware | fp16, one RTX 4060 8 GB, 5 h 04 min | |
| | Seed | 42 | |
|
|
| Development-set nDCG@10 (10,000 documents, Turkish title queries) rose from 0.812 (base model) to 0.987. |
|
|
| ## Data and splits |
|
|
| Pairs were harvested from DergiPark's public OAI-PMH interface (843,394 records, October 2026). Both language |
| sides were checked with a language heuristic, mis-tagged sides were swapped, copied titles removed, and pairs |
| filtered by length, length ratio and exact duplicates, leaving 447,244 pairs. Splits are **by journal**: |
| train 2,222 journals / dev 146 / test 417, stratified by subject area, so test journals and their topics are |
| never seen in training. |
|
|
| ## Limitations |
|
|
| - **Known-item evaluation.** Each query has exactly one relevant document: its own article. Real searches can |
| have many relevant papers; scores here measure how well the model links a Turkish description to the right |
| English paper, not full topical relevance. |
| - **Query style.** Training and test queries are titles, keyword lists and abstracts, not natural-language |
| questions. Conversational queries may perform worse. |
| - **Domain.** Documents are abstracts of Turkish journals across all fields (education, health, engineering, |
| social sciences, humanities). Performance on other English corpora (e.g. PubMed) has not been measured yet. |
| - **Truncation.** Trained with a 256-token limit; about 65% of English title+abstract documents are longer and |
| were truncated during training. |
| - **Baselines' training data.** Some baselines (e.g. turkish-e5-large, whose training data is not documented) may |
| have seen DergiPark text during their own training. |
|
|
| ## License and data provenance |
|
|
| The model is released under **CC BY-NC 4.0** (non-commercial). It was trained on abstracts from all harvested |
| DergiPark journals, including journals that publish under non-commercial licenses or state no license. The model |
| weights do not redistribute these texts. The companion dataset contains only pairs from journals with a |
| Creative Commons license that allows derivatives (CC BY, BY-SA, BY-NC, BY-NC-SA); every row has a `license` column. |
|
|
| ## Citation |
|
|
| A paper is in preparation. Until then, please cite this repository. |
|
|
