| ---
|
| tags:
|
| - sentence-transformers
|
| - multi-vector
|
| - colbert
|
| - late-interaction
|
| pipeline_tag: feature-extraction
|
| library_name: sentence-transformers
|
| language: en
|
| license: apache-2.0
|
| base_model:
|
| - chungimungi/GLInt
|
| - lightonai/LateOn-regularized
|
| base_model_relation: merge
|
| ---
|
|
|
| # ModernBERT-base-en-v1
|
|
|
| This is a [Multi-Vector Encoder](https://www.sbert.net/docs/multi_vector_encoder/usage/usage.html) model created by a 50/50 linear merge of [chungimungi/GLInt](https://huggingface.co/chungimungi/GLInt) and [lightonai/LateOn-regularized](https://huggingface.co/lightonai/LateOn-regularized), using the [`MultiVectorEncoder.merge`](https://www.sbert.net/docs/package_reference/multi_vector_encoder/model.html#sentence_transformers.multi_vector_encoder.model.MultiVectorEncoder.merge) method from the [sentence-transformers](https://www.SBERT.net) library. It maps inputs to sequences of 128-dimensional token-level vectors and scores them with late interaction (MaxSim), useful for semantic search with late interaction.
|
|
|
| The merged model scores above the average of its two parents on **9 of the 13 NanoBEIR datasets**, including **5 where it beats both**. It achieves a mean nDCG@10 of **0.7061**, compared with **0.6996** for GLInt and **0.6979** for LateOn-regularized, using the same inference settings for all three models.
|
|
|
| It took a whopping 6 lines of code to merge the models and set the document length, and it took about 12 seconds to execute. See [Model Creation](#model-creation) for the full generation script. The merging settings were not based on any evaluation results.
|
|
|
| ## Model Details
|
|
|
| ### Model Description
|
|
|
| - **Model Type:** Multi-Vector Encoder
|
| - **Base models:** [GLInt](https://huggingface.co/chungimungi/GLInt) and [LateOn-regularized](https://huggingface.co/lightonai/LateOn-regularized)
|
| - **Maximum Sequence Length:** 8192 tokens
|
| - **Maximum Query Length:** 32 tokens
|
| - **Maximum Document Length:** 8192 tokens
|
| - **Output Dimensionality:** 128 dimensions
|
| - **Similarity Function:** MaxSim
|
| - **Supported Modality:** Text
|
| - **Language:** English
|
| - **License:** Apache-2.0
|
|
|
| ### Model Sources
|
|
|
| - **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
|
| - **Documentation:** [Multi-Vector Encoder Documentation](https://www.sbert.net/docs/multi_vector_encoder/usage/usage.html)
|
| - **Repository:** [Sentence Transformers on GitHub](https://github.com/huggingface/sentence-transformers)
|
| - **Hugging Face:** [Multi-Vector Encoders on Hugging Face](https://huggingface.co/models?library=sentence-transformers&other=multi-vector)
|
|
|
| ### Full Model Architecture
|
|
|
| ```
|
| MultiVectorEncoder(
|
| (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'query_length': 32, 'document_length': 8192, 'architecture': 'ModernBertModel'})
|
| (1): Dense({'in_features': 768, 'out_features': 1536, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings', 'use_residual': True})
|
| (2): Dense({'in_features': 1536, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings', 'use_residual': True})
|
| (3): Dense({'in_features': 768, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
|
| (4): MultiVectorMask({'skiplist_words': ['!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '-', '.', '/', ':', ';', '<', '=', '>', '?', '@', '[', '\\', ']', '^', '_', '`', '{', '|', '}', '~'], 'skiplist_tasks': ['document'], 'keep_only_token_ids': None})
|
| (5): Normalize({'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
|
| )
|
| ```
|
|
|
| ## Usage
|
|
|
| ### Direct Usage (Sentence Transformers)
|
|
|
| First install the Sentence Transformers library:
|
|
|
| ```bash
|
| pip install -U sentence-transformers
|
| ```
|
| Then you can load this model and run inference.
|
| ```python
|
| from sentence_transformers import MultiVectorEncoder
|
|
|
| model = MultiVectorEncoder("multi-vector-encoder/ModernBERT-base-en-v1")
|
| queries = [
|
| 'Which planet is known as the Red Planet?',
|
| ]
|
| documents = [
|
| "Venus is often called Earth's twin because of its similar size and proximity.",
|
| 'Mars, known for its reddish appearance, is often referred to as the Red Planet.',
|
| 'Saturn, famous for its rings, is sometimes mistaken for the Red Planet.',
|
| ]
|
| query_embeddings = model.encode_query(queries)
|
| document_embeddings = model.encode_document(documents)
|
| print(query_embeddings[0].shape, document_embeddings[0].shape)
|
| # (12, 128) (18, 128)
|
|
|
| # Get the MaxSim similarity scores
|
| similarities = model.similarity(query_embeddings, document_embeddings)
|
| print(similarities)
|
| # tensor([[11.4814, 11.6525, 11.6232]])
|
| ```
|
|
|
| ## Evaluation
|
|
|
| ### Metrics
|
|
|
| #### NanoBEIR
|
|
|
| * Dataset: [NanoBEIR-en](https://huggingface.co/datasets/sentence-transformers/NanoBEIR-en)
|
| * Evaluated with `MultiVectorNanoBEIREvaluator` on all 13 datasets.
|
| * Query length: 32 tokens. Document length: 8192 tokens (expanded from 300 for GLInt and LateOn-regularized).
|
| * Metric: nDCG@10. The mean is unweighted across datasets.
|
|
|
| The parent models were evaluated with the same prompts and length limits.
|
|
|
| | Dataset | ModernBERT-base-en-v1 | GLInt | LateOn-regularized |
|
| | --- | --- | --- | --- |
|
| | msmarco | <u>0.736155</u> | 0.728477 | 0.707793 |
|
| | nq | <u>0.818848</u> | 0.818032 | 0.782732 |
|
| | fiqa2018 | <u>0.640317</u> | 0.625973 | 0.625546 |
|
| | nfcorpus | 0.393756 | <u>0.399912</u> | 0.391630 |
|
| | scidocs | 0.435285 | 0.435472 | <u>0.444901</u> |
|
| | arguana | <u>0.588440</u> | 0.569680 | 0.561172 |
|
| | dbpedia | 0.703868 | <u>0.709176</u> | 0.678083 |
|
| | quoraretrieval | <u>0.990127</u> | 0.983948 | 0.986797 |
|
| | touche2020 | 0.692082 | <u>0.696390</u> | 0.686696 |
|
| | climatefever | 0.484580 | 0.418405 | <u>0.521409</u> |
|
| | fever | 0.966278 | <u>0.969763</u> | 0.966278 |
|
| | hotpotqa | 0.930661 | <u>0.934504</u> | 0.917731 |
|
| | scifact | 0.799062 | <u>0.804524</u> | 0.802028 |
|
| | **mean** | **<u>0.706112</u>** | **0.699558** | **0.697907** |
|
|
|
| ## Model Creation
|
|
|
| The backbone and all three projection layers were merged in fp32 with equal weights from [GLInt](https://huggingface.co/chungimungi/GLInt) and [LateOn-regularized](https://huggingface.co/lightonai/LateOn-regularized), without any training.
|
|
|
| This is all it took:
|
|
|
| ```python
|
| # pip install "sentence-transformers[merge]"
|
| from sentence_transformers import MultiVectorEncoder
|
|
|
| # Load the GLInt and LateOn-regularized models,
|
| # matching the latter's special token config
|
| glint = MultiVectorEncoder(
|
| "chungimungi/GLInt",
|
| config_kwargs={
|
| "bos_token_id": 50281,
|
| "eos_token_id": 50282,
|
| "pad_token_id": 50283,
|
| },
|
| processor_kwargs={"model_max_length": 8192, "max_length": 8192},
|
| )
|
| lateon = MultiVectorEncoder("lightonai/LateOn-regularized")
|
|
|
| # Merge using simple 50/50 linear
|
| model = MultiVectorEncoder.merge(
|
| [glint, lateon],
|
| method="linear",
|
| parameters={"weight": [0.5, 0.5]},
|
| )
|
| # Increase the document length from 300 to 8192 tokens
|
| model[0].document_length = 8192
|
| # And save it for reuse
|
| model.save_pretrained("ModernBERT-base-en-v1")
|
| ```
|
|
|
| ### Framework Versions
|
|
|
| - Python: 3.12.3
|
| - Sentence Transformers: 6.2.0
|
| - Transformers: 5.4.0
|
| - PyTorch: 2.11.0+cu128
|
| - Accelerate: 1.13.0
|
| - Datasets: 4.8.5
|
| - Tokenizers: 0.22.2
|
|
|
| ## Citation
|
|
|
| ### BibTeX
|
|
|
| #### LateOn Regularization
|
|
|
| ```bibtex
|
| @misc{chaffin2026lateonregularization,
|
| title={Party is over: regularizing ColBERT models to fix efficient ANN methods},
|
| author={Chaffin, Antoine},
|
| year={2026},
|
| howpublished={\url{https://huggingface.co/blog/lightonai/lateon-regularization}},
|
| }
|
| ```
|
|
|
| #### GLInt
|
|
|
| ```bibtex
|
| @misc{aarush2026glint,
|
| title={GLInt: Geometry-Matched Hard Negatives for Late-Interaction Retrieval},
|
| author={Aarush},
|
| year={2026},
|
| howpublished={\url{https://huggingface.co/blog/chungimungi/glint}},
|
| }
|
| ```
|
|
|
| ### Mergekit
|
|
|
| ```bibtex
|
| @inproceedings{goddard-etal-2024-arcees,
|
| title = "Arcee{'}s {M}erge{K}it: A Toolkit for Merging Large Language Models",
|
| author = "Goddard, Charles and Siriwardhana, Shamane and Ehghaghi, Malikeh and Meyers, Luke and Karpukhin, Vladimir and Benedict, Brian and McQuade, Mark and Solawetz, Jacob",
|
| editor = "Dernoncourt, Franck and Preo{\c{t}}iuc-Pietro, Daniel and Shimorina, Anastasia",
|
| booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track",
|
| month = nov,
|
| year = "2024",
|
| address = "Miami, Florida, US",
|
| publisher = "Association for Computational Linguistics",
|
| url = "https://aclanthology.org/2024.emnlp-industry.36",
|
| doi = "10.18653/v1/2024.emnlp-industry.36",
|
| pages = "477--485",
|
| }
|
| ```
|
|
|
