README.md

metadata

language:
  - tr
license: apache-2.0
library_name: transformers
base_model: ytu-ce-cosmos/modernbert-tr-base
pipeline_tag: text-ranking
tags:
  - sentence-transformers
  - text-embeddings-inference
  - transformers.js
  - reranker
  - cross-encoder
  - modernbert
  - onnx
model-index:
  - name: modernbert-tr-reranker
    results:
      - task:
          type: Retrieval
          name: ArguAnaTR
        dataset:
          type: trmteb/arguana-tr
          name: MTEB ArguAnaTR
          config: default
          split: test
          revision: main
        metrics:
          - type: ndcg_at_10
            value: 54.75
      - task:
          type: Retrieval
          name: CQADupstackGamingRetrievalTR
        dataset:
          type: trmteb/cqadupstack-gaming-tr
          name: MTEB CQADupstackGamingRetrievalTR
          config: default
          split: test
          revision: main
        metrics:
          - type: ndcg_at_10
            value: 61.1
      - task:
          type: Retrieval
          name: SciFactTR
        dataset:
          type: trmteb/scifact-tr
          name: MTEB SciFactTR
          config: default
          split: test
          revision: main
        metrics:
          - type: ndcg_at_10
            value: 86.34
      - task:
          type: Retrieval
          name: SquadTRRetrieval
        dataset:
          type: trmteb/squad-tr
          name: MTEB SquadTRRetrieval
          config: default
          split: test
          revision: main
        metrics:
          - type: ndcg_at_10
            value: 90.11
      - task:
          type: Retrieval
          name: TQuadRetrieval
        dataset:
          type: trmteb/tquad
          name: MTEB TQuadRetrieval
          config: default
          split: test
          revision: main
        metrics:
          - type: ndcg_at_10
            value: 94
      - task:
          type: Retrieval
          name: XQuADRetrieval
        dataset:
          type: google/xquad
          name: MTEB XQuADRetrieval
          config: default
          split: validation
          revision: 51adfef1c1287aab1d2d91b5bead9bcfb9c68583
        metrics:
          - type: ndcg_at_10
            value: 97.86

ModernBERT-TR Reranker

A 150M-parameter Turkish cross-encoder reranker to score (query, document) relevance.

• Base model: ytu-ce-cosmos/modernbert-tr-base.
• Distilled from Qwen/Qwen3-Reranker-8B.

Results

Reranking the top-100 of a first-stage retriever (ytu-ce-cosmos/modernbert-tr-embed) at max_seq=512. The uplift (Δ) is the reranker's contribution.

Task	First-stage NDCG@10	+ Reranker	Δ
ArguAnaTR	37.01	54.75	+17.74
SquadTRRetrieval	75.94	90.11	+14.17
SciFactTR	77.07	86.34	+9.27
TQuadRetrieval	87.48	94.00	+6.52
CQADupstackGamingRetrievalTR	56.44	61.10	+4.66
XQuADRetrieval	95.03	97.86	+2.83
Mean Δ			+9.20

How was this model trained?

Question answering and counter argument distillation of Qwen3-Reranker-8B relevance scores into the 150M cross-encoder over Turkish question answering / information retrieval data using listwise KL.

Usage

transformers

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

tok = AutoTokenizer.from_pretrained("ytu-ce-cosmos/modernbert-tr-reranker")
model = AutoModelForSequenceClassification.from_pretrained("ytu-ce-cosmos/modernbert-tr-reranker").eval()

query = "Türkiye'nin başkenti neresidir?"
docs = ["Ankara, Türkiye'nin başkentidir.", "İstanbul en kalabalık şehirdir."]
enc = tok([query] * len(docs), docs, padding=True, truncation="longest_first",
          max_length=8192, return_tensors="pt")
with torch.no_grad():
    scores = model(**enc).logits.squeeze(-1)
ranking = sorted(zip(docs, scores.tolist()), key=lambda x: x[1], reverse=True)

sentence-transformers

from sentence_transformers import CrossEncoder
model = CrossEncoder("ytu-ce-cosmos/modernbert-tr-reranker")
scores = model.predict([(query, d) for d in docs])

ONNX Runtime

The onnx/ folder has the full graph, the output is the relevance logit:

import onnxruntime, numpy as np
sess = onnxruntime.InferenceSession("onnx/model.onnx")
feed = {k: v.numpy() for k, v in enc.items() if k in {i.name for i in sess.get_inputs()}}
logits = sess.run(None, feed)[0].squeeze(-1)

Text Embeddings Inference (TEI)

text-embeddings-router --model-id ytu-ce-cosmos/modernbert-tr-reranker --dtype float16
# POST /rerank  {"query": "soru", "texts": ["aday 1", "aday 2"]}

Training data

We used Turkish datasets msmarco-tr, squad-tr, fiqa-tr, nfcorpus-tr, quora-tr, scifact-tr for distillation by Qwen3-Reranker-8B, and Turkish counter-argument pairs from ArguAna machine-translated with TranslateGemma-27B. All training data was text-hash chceked against every MTEB(Turkish) test split.

Limitations

• Reported NDCG is rerank-of-top-100 over a first-stage retriever; absolute scores depend on that first stage.
• int8 ONNX reorders scores meaningfully lossy for a reranker; use fp32 for quality-sensitive ranking.
• Due to the lack of long form data in our training, the model's performance may degrade on long context input.

License & attribution

• License: apache-2.0.