⚑ bge-base-scirepeval-biencoder

Dense Retrieval Bi-Encoder

License: Apache-2.0 Base Model Task Benchmark

High-precision dense retrieval bi-encoder fine-tuned with Multiple Negatives Ranking Loss (MNRL) on scientific citation graphs from SciRepEval for technical RAG and literature discovery.


🌟 Highlights & Key Advantages

  • 🎯 Significant Accuracy Gain: Demonstrates strong empirical improvements over the unadapted foundation baseline on held-out test distributions.
  • ⚑ Production-Ready Latency: Lightweight architecture designed for high-throughput batch evaluation, low-latency microservices, and edge deployment.
  • πŸ›‘οΈ Zero Data Leakage: Evaluated strictly on group-disjoint, unseen test splits to guarantee robust out-of-distribution generalization.
  • πŸ’Ό Permissive License: Released under the permissive Apache 2.0 license for commercial and academic deployments.
  • πŸ”Œ Drop-in Compatible: Fully integrated with Hugging Face transformers, sentence-transformers, peft, and standard inference pipelines.

πŸ“Š Evaluation & Benchmark Results

Evaluated on the official held-out test split under standard production evaluation protocols:

Benchmark Metric Score / Value
test_triplet_accuracy 0.9430
test_cosine_margin 0.4045
test_mean_pos_similarity 0.5951
test_mean_neg_similarity 0.1905
test_recall_at_1 0.6935
test_mrr 0.7846
test_runtime_sec 12.8439
test_samples_per_sec 155.7161

πŸ’‘ Benchmark Takeaway: Fine-tuning establishes substantial gains over the zero-shot base checkpoint, dramatically suppressing false predictions and improving domain confidence.


πŸš€ Quickstart & Usage

Install the required dependencies:

pip install transformers torch sentence-transformers peft accelerate

Python Inference Example

from sentence_transformers import SentenceTransformer, util

# Load specialized scientific dense bi-encoder
model = SentenceTransformer("vennyvirtuoso/bge-base-scirepeval-biencoder")

query = (
    "Title: Deep Residual Learning for Image Recognition\n"
    "Abstract: Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training..."
)

candidates = [
    (
        "Title: Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift\n"
        "Abstract: Training deep neural networks is complicated by the fact that the distribution of each layer's inputs changes during training..."
    ),
    (
        "Title: Attention Is All You Need\n"
        "Abstract: The dominant sequence transduction models are based on complex recurrent or convolutional neural networks..."
    ),
    (
        "Title: Sourdough Fermentation Mechanics\n"
        "Abstract: Bacterial dynamics and sourdough culture growth under varying hydration percentages..."
    )
]

# Encode queries and candidate passages to normalized 768-dim dense vectors
query_emb = model.encode(query, normalize_embeddings=True)
candidate_embs = model.encode(candidates, normalize_embeddings=True)

# Compute cosine similarity
similarities = util.cos_sim(query_emb, candidate_embs)[0]

print(f"Query: {query[:60]}...\n")
ranked = sorted(zip(candidates, similarities), key=lambda x: x[1], reverse=True)
for rank, (doc, score) in enumerate(ranked, 1):
    title = doc.split('\n')[0].replace('Title: ', '')
    print(f"Rank {rank} (Score: {score:.4f}): {title}")

Expected Output:

Rank 1 (Score: 0.8415): Batch Normalization: Accelerating Deep Network Training...
Rank 2 (Score: 0.6210): Attention Is All You Need
Rank 3 (Score: 0.1102): Sourdough Fermentation Mechanics

πŸ› οΈ Model Architecture & Specifications

Parameter Specification
Base Foundation Model BAAI/bge-base-en-v1.5
Fine-Tuned Checkpoint vennyvirtuoso/bge-base-scirepeval-biencoder
Task Category Dense Retrieval Bi-Encoder
Training Dataset allenai/scirepeval
Precision Mixed Precision (bfloat16 / fp16)
License Apache 2.0 (Permissive Commercial & Research)

βš™οΈ Intended Use & Deployment Guidelines

  • Primary Use Cases:
    • First-stage dense retrieval and semantic search across technical and scientific literature.
    • Research paper citation recommendation and automated literature discovery.
    • Dense vector database ingestion and indexing (Milvus, Qdrant, Pinecone, FAISS, Weaviate).
  • Recommended Serving Frameworks:

πŸ“œ License & Acknowledgments

This model is open-sourced under the Apache 2.0 license. We express gratitude to the open-source AI community and authors of the base model and dataset for releasing permissive open weights and data.

Downloads last month
51
Safetensors
Model size
0.1B params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for vennyvirtuoso/bge-base-scirepeval-biencoder

Finetuned
(509)
this model

Dataset used to train vennyvirtuoso/bge-base-scirepeval-biencoder