---
language:
- tr
- en
license: cc-by-nc-4.0
library_name: sentence-transformers
pipeline_tag: sentence-similarity
base_model: intfloat/multilingual-e5-base
tags:
- sentence-transformers
- feature-extraction
- sentence-similarity
- cross-lingual-retrieval
- scientific-literature
- turkish
datasets:
- musapeker/dergipark-tr-en-abstracts
---

# tr-en-sci-e5-base

**Turkish → English scientific literature retrieval.** Type a query in Turkish, find the relevant English paper
(title + abstract). A 278M-parameter embedding model fine-tuned from
[intfloat/multilingual-e5-base](https://huggingface.co/intfloat/multilingual-e5-base) on ~380k Turkish–English
abstract pairs written by the authors of articles on [DergiPark](https://dergipark.org.tr).

**Demo:** [huggingface.co/spaces/musapeker/tr-en-sci-search](https://huggingface.co/spaces/musapeker/tr-en-sci-search)
(runs entirely in the browser).

> **Türkçe özet:** Türkçe sorguyla İngilizce bilimsel makale bulmak için eğitilmiş bir embedding modeli.
> DergiPark'taki yazarların kendi yazdığı Türkçe–İngilizce başlık, özet ve anahtar kelime çiftleriyle eğitildi.
> Eğitimde hiç görülmemiş dergilerden oluşan test kümesinde, kendisinin yaklaşık iki katı büyüklükteki
> bge-m3 ve turkish-e5-large modellerini geçiyor.

## Results

Public test set of [`musapeker/dergipark-tr-en-abstracts`](https://huggingface.co/datasets/musapeker/dergipark-tr-en-abstracts):
16,983 English documents (title + abstract) from **journals never seen in training**, 2,000 Turkish queries per
query type, exactly one relevant document per query (the same article).

| Model | Title nDCG@10 | Title R@1 | Title R@10 | Title nDCG@10 (no shared words) | Keywords nDCG@10 | Keywords R@1 | Keywords R@10 |
|---|---|---|---|---|---|---|---|
| **tr-en-sci-e5-base (this model, 278M)** | **0.979** | **0.960** | **0.995** | **0.971** | **0.927** | **0.868** | **0.981** |
| bge-m3 (568M) | 0.944 | 0.908 | 0.979 | 0.922 | 0.834 | 0.743 | 0.922 |
| ytu-ce-cosmos/turkish-e5-large (560M) | 0.942 | 0.907 | 0.974 | 0.927 | 0.890 | 0.826 | 0.953 |
| multilingual-e5-large-instruct (560M) | 0.889 | 0.840 | 0.934 | 0.850 | 0.789 | 0.687 | 0.886 |
| Qwen3-Embedding-0.6B | 0.819 | 0.749 | 0.888 | 0.752 | 0.775 | 0.680 | 0.864 |
| multilingual-e5-base (base model, 278M) | 0.760 | 0.671 | 0.849 | 0.672 | 0.668 | 0.559 | 0.783 |
| MT (opus-mt-tr-en) + e5-base-v2 | 0.695 | 0.649 | 0.742 | 0.660 | 0.845 | 0.771 | 0.916 |
| MT (opus-mt-tr-en) + BM25 | 0.645 | 0.589 | 0.703 | 0.578 | 0.762 | 0.674 | 0.848 |
| BM25 (Turkish query on English docs) | 0.292 | 0.221 | 0.370 | 0.095 | 0.462 | 0.306 | 0.641 |

- **Title** queries: the Turkish title of the article. **Keywords** queries: the Turkish author keywords, joined with `; `.
- **No shared words**: the subset of title queries (1,150 of 2,000) that share no content word (≥3 letters, no digits)
  with their target document, i.e. names, years and international terms cannot help. This is the purely
  cross-lingual part of the task.
- R@k: share of queries whose relevant document is in the top k.

## Usage

The model follows the E5 convention: prefix queries with `query: ` and documents with `passage: `.

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("musapeker/tr-en-sci-e5-base")

queries = ["query: göz dibi görüntülerinde derin öğrenme ile diyabetik retinopati tespiti"]
docs = [
    "passage: Deep learning based detection of diabetic retinopathy from fundus images using convolutional neural networks.",
    "passage: The effect of climate change on wheat yield in Central Anatolia: a regression analysis.",
]
q_emb = model.encode(queries, normalize_embeddings=True)
d_emb = model.encode(docs, normalize_embeddings=True)
print(model.similarity(q_emb, d_emb))  # first document scores highest
```

For best results, index documents as `passage: <English title>. <English abstract>`.

### In the browser (transformers.js)

An int8 ONNX export (`onnx/model_quantized.onnx`, 279 MB) is included. On 1,000 test queries against 1,000 documents,
R@1 is 0.990 for the PyTorch model and 0.988 when int8 query embeddings are matched against PyTorch document embeddings.

```js
import { pipeline } from "@huggingface/transformers";

const extractor = await pipeline("feature-extraction", "musapeker/tr-en-sci-e5-base", { dtype: "q8" });
const out = await extractor(["query: tahrir defterleri ve Osmanlı nüfus tarihi"], { pooling: "mean", normalize: true });
```

## Training

| | |
|---|---|
| Base model | intfloat/multilingual-e5-base (XLM-RoBERTa, 278M) |
| Training data | 379,792 Turkish–English article pairs from 2,222 DergiPark journals (journal-disjoint from dev/test) |
| Training rows | 837,539 (Turkish title → English doc, Turkish keywords → English doc, and Turkish abstract → English doc for 30% of articles) |
| Loss | CachedMultipleNegativesRankingLoss (GradCache), in-batch negatives |
| Batch size | 256 (GradCache mini-batch 16), no-duplicates sampler so one article never appears twice in a batch |
| Learning rate / schedule | 2e-5, 10% linear warmup, 1 epoch (3,271 steps) |
| Max sequence length | 256 tokens |
| Precision / hardware | fp16, one RTX 4060 8 GB, 5 h 04 min |
| Seed | 42 |

Development-set nDCG@10 (10,000 documents, Turkish title queries) rose from 0.812 (base model) to 0.987.

## Data and splits

Pairs were harvested from DergiPark's public OAI-PMH interface (843,394 records, October 2026). Both language
sides were checked with a language heuristic, mis-tagged sides were swapped, copied titles removed, and pairs
filtered by length, length ratio and exact duplicates, leaving 447,244 pairs. Splits are **by journal**:
train 2,222 journals / dev 146 / test 417, stratified by subject area, so test journals and their topics are
never seen in training.

## Limitations

- **Known-item evaluation.** Each query has exactly one relevant document: its own article. Real searches can
  have many relevant papers; scores here measure how well the model links a Turkish description to the right
  English paper, not full topical relevance.
- **Query style.** Training and test queries are titles, keyword lists and abstracts, not natural-language
  questions. Conversational queries may perform worse.
- **Domain.** Documents are abstracts of Turkish journals across all fields (education, health, engineering,
  social sciences, humanities). Performance on other English corpora (e.g. PubMed) has not been measured yet.
- **Truncation.** Trained with a 256-token limit; about 65% of English title+abstract documents are longer and
  were truncated during training.
- **Baselines' training data.** Some baselines (e.g. turkish-e5-large, whose training data is not documented) may
  have seen DergiPark text during their own training.

## License and data provenance

The model is released under **CC BY-NC 4.0** (non-commercial). It was trained on abstracts from all harvested
DergiPark journals, including journals that publish under non-commercial licenses or state no license. The model
weights do not redistribute these texts. The companion dataset contains only pairs from journals with a
Creative Commons license that allows derivatives (CC BY, BY-SA, BY-NC, BY-NC-SA); every row has a `license` column.

## Citation

A paper is in preparation. Until then, please cite this repository.