Innovatewithapple/bert-dense-retriever

---

license: mit language: - en library_name: transformers pipeline_tag: feature-extraction tags: - bert - dense-retrieval - semantic-search - information-retrieval - ms-marco - faiss

BERT Dense Retriever

A dense semantic retrieval model fine-tuned on the MS MARCO Passage Ranking dataset using BERT-base-uncased.

The model encodes natural language queries and passages into dense vector embeddings that can be indexed with FAISS for efficient semantic search.

This repository contains the complete Hugging Face compatible model including tokenizer, configuration, and custom model implementation.

---

Model Details

Backbone

• BERT-base-uncased

Pooling

• Mean Pooling

Embedding Normalization

• L2 Normalization

Similarity Metric

• Cosine Similarity

Training Objective

• CrossEntropy Loss over the similarity matrix (InfoNCE-style retrieval objective)

---

Training Configuration

Parameter	Value
Optimizer	AdamW
Learning Rate	2e-5
Batch Size	32
Epochs	10
Weight Decay	0.01
Temperature	0.05

---

Evaluation Results

Model	Recall@10	MRR	nDCG@10
BERT-base-uncased	0.4793	0.2837	0.3301
Fine-tuned Dense Retriever	0.9693	0.8521	0.8810

The fine-tuned model substantially improves retrieval quality on the evaluation set compared with the untuned BERT-base encoder.

---

📊 BEIR Evaluation Results

The table below compares the proposed retrieval pipeline against the BM25 baseline reported by the original BEIR benchmark.

Dataset	Retrieval Strategy	BM25 (BEIR) NDCG@10	Pipeline NDCG@10	Recall@10	Recall@100	Improvement
FEVER	Hybrid + Cross Encoder	0.7530	0.9791	0.9873	0.9937	+0.2261
Quora	Dense + Cross Encoder	0.7830	0.9686	0.9858	0.9938	+0.1856
HotpotQA	Dense + Cross Encoder	0.6030	0.8977	0.8853	0.8960	+0.2947
FiQA	Dense + Cross Encoder	0.2361	0.7512	0.8055	—	+0.5151
TREC-COVID	Hybrid + Cross Encoder	0.6559	0.6868	0.0181	0.1110	+0.0309

Note: BM25 scores are taken from the original BEIR benchmark and are included as the lexical retrieval baseline for comparison.

---

🔍 Key Findings

• Strong zero-shot generalization across multiple retrieval domains.
• Significant improvements over the BEIR BM25 baseline on FEVER, Quora, HotpotQA, and FiQA.
• Hybrid retrieval (BM25 + Dense Retriever) improves retrieval quality for specialized biomedical documents in TREC-COVID.
• Cross-Encoder re-ranking substantially enhances the final ranking quality by leveraging full query-document interactions.

---

Usage

from transformers import AutoTokenizer, AutoModel

model_name = "Innovatewithapple/bert-dense-retriever"

tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModel.from_pretrained(
    model_name,
    trust_remote_code=True,
)

inputs = tokenizer(
    "What is deep learning?",
    return_tensors="pt"
)

embeddings = model(**inputs)

print(embeddings.shape)

---

Intended Use

This model is designed for:

• Dense semantic retrieval
• Semantic search
• Question-passage retrieval
• Retrieval-Augmented Generation (RAG)
• Information retrieval research

---

Source Code

GitHub Repository:

https://github.com/Innovatewithapple/dense-semantic-retrieval

---

Author

Mihir Vyas