Instructions to use bioformers/Bioformer-8L-RetroMAE with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use bioformers/Bioformer-8L-RetroMAE with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="bioformers/Bioformer-8L-RetroMAE")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForMaskedLM tokenizer = AutoTokenizer.from_pretrained("bioformers/Bioformer-8L-RetroMAE") model = AutoModelForMaskedLM.from_pretrained("bioformers/Bioformer-8L-RetroMAE", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Bioformer-8L-RetroMAE
Bioformer-8L-RetroMAE is a compact biomedical BERT encoder (43M parameters) for dense retrieval. It was created by continuing to pretrain Bioformer-8L with RetroMAE, a pretraining objective designed for retrieval, on about 32 million PubMed titles and abstracts.
Important: This checkpoint is the RetroMAE-pretrained encoder. It has not been fine-tuned for retrieval. It is meant as a strong starting point (initialization) for training bi-encoder retrievers (for example, contrastive fine-tuning with hard negatives). Its raw [CLS] embeddings are not designed for zero-shot semantic search.
Model details
| Base model | bioformers/bioformer-8L |
| Architecture | BERT encoder, 8 layers, hidden size 512, 8 attention heads |
| Parameters | 43M (same as Bioformer-8L; the RetroMAE decoder is discarded) |
| Vocabulary | Cased biomedical WordPiece, 32,768 tokens (same as Bioformer-8L) |
| Max sequence length | 512 |
| Embedding dimension | 512 ([CLS] vector) |
| Language | English (biomedical literature) |
Why RetroMAE?
Standard masked language modeling (MLM) does not explicitly train the [CLS] vector, yet a bi-encoder uses that vector as the text representation. RetroMAE fixes this mismatch. The encoder reads a moderately masked input, and its [CLS] vector is passed to a lightweight one-layer decoder. The decoder must then reconstruct a separately and more heavily masked version of the same text from that single vector. This forces the [CLS] vector to summarize the whole text. The decoder is used only during pretraining, so the released encoder keeps the compact Bioformer-8L size and speed.
Training
Data
- PubMed records published through August 2026.
- Records without an abstract (title-only records) were excluded, as were records whose title plus abstract was shorter than 200 characters. About 32 million records remained.
- The title and abstract of each record were concatenated into a single training instance, with a maximum length of 512 tokens. Each [CLS] representation therefore corresponds to one PubMed record.
Procedure
- Initialization: the released
bioformers/bioformer-8Lcheckpoint. - Encoder masking: whole-word masking, probability 0.25.
- Decoder: single transformer layer; reconstruction masking probability 0.50. The decoder was discarded after pretraining.
- Loss: 0.5 ร encoder MLM loss + 0.5 ร RetroMAE decoder reconstruction loss.
- Convergence: on the held-out set, the decoder reconstruction loss decreased from 6.11 (step 10,000) to 2.97 (step 600,000), while the encoder MLM loss stayed stable (1.43 to 1.41; masked-token perplexity 4.18 to 4.10).
How to use
Fine-tune as a bi-encoder retriever (recommended)
Use this checkpoint to initialize both the query encoder and the document encoder. Train with a contrastive objective such as InfoNCE, using in-domain queryโdocument pairs and hard negatives. The setup we used in our paper:
- Separate query and document encoders, both initialized from this checkpoint.
- [CLS] pooling with L2 normalization; similarity is the inner product (equivalent to cosine similarity).
- InfoNCE loss, temperature ฯ = 0.05, with 1 positive and 100 BM25 hard negatives per question.
- 10 epochs, learning rate 3e-5, weight decay 0.01, linear decay, warmup ratio 0.10, gradient accumulation over 16 questions.
- Maximum length of 128 tokens for queries and 512 tokens for documents.
- Training data: 4,049 BioASQ Task 12B training questions (36,783 relevant articles); the 100 BM25 hard negatives per question were mined from all PubMed abstracts after excluding all known relevant articles. The checkpoint with the highest validation MRR (1,000 held-out questions) was selected.
Evaluation
The results below are for bi-encoders fine-tuned from each backbone. All backbones were fine-tuned with the same BioASQ Task 12B training questions, the same BM25 hard negatives, and the same settings. They are therefore results of the downstream retriever, not of this checkpoint used zero-shot.
Retrieval used an exact inner-product FAISS index (IndexFlatIP) over all PubMed abstracts, evaluated on 340 questions from the four official BioASQ Task 12B golden test batches. Each retriever returned the top 1,000 articles per question. MAP is computed over the top 1,000 articles; MAP@10 over the top 10 (BioASQ Phase A convention).
Dense retrieval
| Backbone | MAP | MAP@10 | NDCG@10 | Hit@1 | Hit@10 | Hit@100 | Recall@1 | Recall@10 | Recall@100 |
|---|---|---|---|---|---|---|---|---|---|
| BioBERT-Base v1.2 | 0.021 | 0.015 | 0.039 | 4.7% | 16.5% | 31.8% | 0.8% | 2.4% | 7.1% |
| PubMedBERT | 0.039 | 0.029 | 0.082 | 13.2% | 29.7% | 50.3% | 1.5% | 4.7% | 10.9% |
| Bioformer-8L | 0.035 | 0.026 | 0.071 | 10.0% | 27.1% | 51.5% | 1.3% | 4.2% | 10.6% |
| Bioformer-8L-RetroMAE | 0.053 | 0.039 | 0.100 | 13.5% | 35.0% | 57.9% | 2.3% | 5.7% | 14.1% |
Lexical and hybrid retrieval
Dense and BM25 rankings were combined with weighted reciprocal rank fusion (RRF, k = 60). The weights are BM25/dense; each row fuses BM25 with the single named dense model.
| Retrieval setting | Model | MAP | MAP@10 | NDCG@10 | Hit@1 | Hit@10 | Hit@100 | Recall@1 | Recall@10 | Recall@100 |
|---|---|---|---|---|---|---|---|---|---|---|
| Lexical | BM25 | 0.280 | 0.168 | 0.391 | 43.2% | 80.6% | 92.1% | 6.6% | 24.8% | 57.9% |
| Hybrid (0.75/0.25) | BioBERTBase-v1.2 | 0.288 | 0.176 | 0.398 | 44.4% | 80.9% | 91.8% | 7.2% | 25.3% | 58.1% |
| Hybrid (0.75/0.25) | PubMedBERT | 0.293 | 0.178 | 0.406 | 46.2% | 80.6% | 91.8% | 6.9% | 25.4% | 58.0% |
| Hybrid (0.75/0.25) | Bioformer8L | 0.289 | 0.175 | 0.405 | 46.5% | 81.2% | 91.8% | 6.6% | 25.3% | 58.2% |
| Hybrid (0.75/0.25) | Bioformer8L-RetroMAE | 0.294 | 0.180 | 0.409 | 46.5% | 80.6% | 92.1% | 7.0% | 25.6% | 58.6% |
| Hybrid (0.50/0.50) | BioBERTBase-v1.2 | 0.190 | 0.112 | 0.272 | 26.5% | 75.6% | 90.3% | 4.3% | 20.3% | 49.5% |
| Hybrid (0.50/0.50) | PubMedBERT | 0.217 | 0.132 | 0.318 | 35.0% | 75.9% | 89.7% | 4.7% | 21.9% | 51.4% |
| Hybrid (0.50/0.50) | Bioformer8L | 0.203 | 0.121 | 0.304 | 34.4% | 76.5% | 90.0% | 4.4% | 21.3% | 50.5% |
| Hybrid (0.50/0.50) | Bioformer8L-RetroMAE | 0.219 | 0.130 | 0.324 | 40.9% | 77.1% | 89.1% | 4.8% | 21.6% | 51.1% |
Intended uses and limitations
Intended uses
- Initializing dense retrievers or embedding models for biomedical literature search, including the retrieval component of RAG systems.
- Settings where encoding speed matters, such as indexing tens of millions of documents or encoding queries in real time. The encoder has the same size and inference speed as Bioformer-8L, which is about 3ร faster than BERT-Base-sized models in CPU inference in our benchmark.
Limitations
- This checkpoint is not fine-tuned for retrieval; fine-tune it before use as a retriever.
- Retrieval results were measured on a single benchmark (BioASQ Task 12B), whose expert-written questions favor lexical matching. Performance on other query types, such as natural-language clinical questions, is untested.
- Fine-tuned dense retrievers built on this checkpoint were weaker than BM25 alone on BioASQ; combine them with BM25 (for example, weighted RRF with weights favoring BM25) for best results.
Related models
- bioformers/bioformer-8L: base model, 8 layers, 43M parameters
- bioformers/bioformer-16L: deeper variant, 16 layers, 42M parameters
Contact
- Name: Li Fang
- Affiliation: Zhongshan School of Medicine, Sun Yat-sen University
- Email: fangli9@mail.sysu.edu.cn
- Downloads last month
- 68
Model tree for bioformers/Bioformer-8L-RetroMAE
Base model
bioformers/bioformer-8L