Sundus08/scifact-minilm-cross-encoder

SciFact MiniLM Cross-Encoder Reranker

A cross-encoder fine-tuned for ranking scientific abstracts against research claims, using the BEIR SciFact dataset.

Intended use

This model assigns relevance scores to query-document pairs. It is intended to rerank documents retrieved by a first-stage search system such as BM25.

It is NOT a scientific fact checker, clinical decision system, or evidence entailment classifier. Scores are ranking logits, not probabilities that claims are true.

Base model and training

Base model: cross-encoder/ms-marco-MiniLM-L-6-v2

• Training queries: 728
• Held-out validation queries: 81
• Labeled positive pairs: 835
• BM25-mined unjudged negative pairs: 2,505
• Total training pairs: 3,340
• Epochs: 1
• Batch size: 8
• Learning rate: 2e-5
• Maximum sequence length: 256
• Random seed: 42
• Training device: Apple Silicon MPS

Unjudged negatives are weak training labels. They are not confirmed irrelevant documents.

Evaluation

Original BM25 top-50 reranking experiment, 300 held-out SciFact test queries:

Metric	BM25	Fine-tuned reranker
nDCG@10	0.6647	0.6863
Recall@10	0.7849	0.7903

The observed improvements were not statistically significant. The paired bootstrap 95% confidence interval for the nDCG@10 difference was [-0.0106, 0.0548].

An exploratory 20-candidate configuration achieved validation nDCG@10 of 0.7560 and Recall@10 of 0.8395 on 81 queries. That optimized configuration has not been independently evaluated on a new final test set.

All results are from a single trained checkpoint.

Limitations

• Evaluated on a small scientific retrieval benchmark.
• Relevance judgments are incomplete.
• Weak negatives may contain relevant documents.
• Training has not been replicated across multiple seeds.
• No demonstrated generalization to other scientific corpora.
• Ranking relevance does not establish scientific truth.

Reproducibility

Training code, evaluation scripts, saved metrics, statistical analysis, and local serving application:

https://github.com/SundusA0/scientific-evidence-retrieval

Attribution and licensing

Fine-tuned from cross-encoder/ms-marco-MiniLM-L-6-v2, whose original model is licensed under Apache 2.0.

Training data: BEIR SciFact, listed as CC BY-SA 4.0 on its Hugging Face dataset page.

No separate license has been declared for these fine-tuned weights. Review the upstream model and dataset licenses before reuse or redistribution.