File size: 9,151 Bytes

---

tags:
- sentence-transformers
- multi-vector
- colbert
- late-interaction
pipeline_tag: feature-extraction
library_name: sentence-transformers
language: en
license: apache-2.0
base_model:
- chungimungi/GLInt
- lightonai/LateOn-regularized
base_model_relation: merge
---


# ModernBERT-base-en-v1

This is a [Multi-Vector Encoder](https://www.sbert.net/docs/multi_vector_encoder/usage/usage.html) model created by a 50/50 linear merge of [chungimungi/GLInt](https://huggingface.co/chungimungi/GLInt) and [lightonai/LateOn-regularized](https://huggingface.co/lightonai/LateOn-regularized), using the [`MultiVectorEncoder.merge`](https://www.sbert.net/docs/package_reference/multi_vector_encoder/model.html#sentence_transformers.multi_vector_encoder.model.MultiVectorEncoder.merge) method from the [sentence-transformers](https://www.SBERT.net) library. It maps inputs to sequences of 128-dimensional token-level vectors and scores them with late interaction (MaxSim), useful for semantic search with late interaction.

The merged model scores above the average of its two parents on **9 of the 13 NanoBEIR datasets**, including **5 where it beats both**. It achieves a mean nDCG@10 of **0.7061**, compared with **0.6996** for GLInt and **0.6979** for LateOn-regularized, using the same inference settings for all three models.

It took a whopping 6 lines of code to merge the models and set the document length, and it took about 12 seconds to execute. See [Model Creation](#model-creation) for the full generation script. The merging settings were not based on any evaluation results.

## Model Details

### Model Description

- **Model Type:** Multi-Vector Encoder
- **Base models:** [GLInt](https://huggingface.co/chungimungi/GLInt) and [LateOn-regularized](https://huggingface.co/lightonai/LateOn-regularized)
- **Maximum Sequence Length:** 8192 tokens
    - **Maximum Query Length:** 32 tokens
    - **Maximum Document Length:** 8192 tokens
- **Output Dimensionality:** 128 dimensions
- **Similarity Function:** MaxSim
- **Supported Modality:** Text
- **Language:** English
- **License:** Apache-2.0

### Model Sources

- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
- **Documentation:** [Multi-Vector Encoder Documentation](https://www.sbert.net/docs/multi_vector_encoder/usage/usage.html)
- **Repository:** [Sentence Transformers on GitHub](https://github.com/huggingface/sentence-transformers)
- **Hugging Face:** [Multi-Vector Encoders on Hugging Face](https://huggingface.co/models?library=sentence-transformers&other=multi-vector)

### Full Model Architecture

```

MultiVectorEncoder(

  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'query_length': 32, 'document_length': 8192, 'architecture': 'ModernBertModel'})

  (1): Dense({'in_features': 768, 'out_features': 1536, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings', 'use_residual': True})

  (2): Dense({'in_features': 1536, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings', 'use_residual': True})

  (3): Dense({'in_features': 768, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})

  (4): MultiVectorMask({'skiplist_words': ['!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '-', '.', '/', ':', ';', '<', '=', '>', '?', '@', '[', '\\', ']', '^', '_', '`', '{', '|', '}', '~'], 'skiplist_tasks': ['document'], 'keep_only_token_ids': None})

  (5): Normalize({'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})

)

```

## Usage

### Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

```bash

pip install -U sentence-transformers

```
Then you can load this model and run inference.
```python

from sentence_transformers import MultiVectorEncoder



model = MultiVectorEncoder("multi-vector-encoder/ModernBERT-base-en-v1")

queries = [

    'Which planet is known as the Red Planet?',

]

documents = [

    "Venus is often called Earth's twin because of its similar size and proximity.",

    'Mars, known for its reddish appearance, is often referred to as the Red Planet.',

    'Saturn, famous for its rings, is sometimes mistaken for the Red Planet.',

]

query_embeddings = model.encode_query(queries)

document_embeddings = model.encode_document(documents)

print(query_embeddings[0].shape, document_embeddings[0].shape)

# (12, 128) (18, 128)



# Get the MaxSim similarity scores

similarities = model.similarity(query_embeddings, document_embeddings)

print(similarities)

# tensor([[11.4814, 11.6525, 11.6232]])

```

## Evaluation

### Metrics

#### NanoBEIR

* Dataset: [NanoBEIR-en](https://huggingface.co/datasets/sentence-transformers/NanoBEIR-en)
* Evaluated with `MultiVectorNanoBEIREvaluator` on all 13 datasets.
* Query length: 32 tokens. Document length: 8192 tokens (expanded from 300 for GLInt and LateOn-regularized).
* Metric: nDCG@10. The mean is unweighted across datasets.

The parent models were evaluated with the same prompts and length limits.

| Dataset | ModernBERT-base-en-v1 | GLInt | LateOn-regularized |
| --- | --- | --- | --- |
| msmarco | <u>0.736155</u> | 0.728477 | 0.707793 |
| nq | <u>0.818848</u> | 0.818032 | 0.782732 |
| fiqa2018 | <u>0.640317</u> | 0.625973 | 0.625546 |
| nfcorpus | 0.393756 | <u>0.399912</u> | 0.391630 |
| scidocs | 0.435285 | 0.435472 | <u>0.444901</u> |
| arguana | <u>0.588440</u> | 0.569680 | 0.561172 |
| dbpedia | 0.703868 | <u>0.709176</u> | 0.678083 |
| quoraretrieval | <u>0.990127</u> | 0.983948 | 0.986797 |
| touche2020 | 0.692082 | <u>0.696390</u> | 0.686696 |
| climatefever | 0.484580 | 0.418405 | <u>0.521409</u> |
| fever | 0.966278 | <u>0.969763</u> | 0.966278 |
| hotpotqa | 0.930661 | <u>0.934504</u> | 0.917731 |
| scifact | 0.799062 | <u>0.804524</u> | 0.802028 |
| **mean** | **<u>0.706112</u>** | **0.699558** | **0.697907** |

## Model Creation

The backbone and all three projection layers were merged in fp32 with equal weights from [GLInt](https://huggingface.co/chungimungi/GLInt) and [LateOn-regularized](https://huggingface.co/lightonai/LateOn-regularized), without any training.

This is all it took:

```python

# pip install "sentence-transformers[merge]"

from sentence_transformers import MultiVectorEncoder



# Load the GLInt and LateOn-regularized models,

# matching the latter's special token config

glint = MultiVectorEncoder(

    "chungimungi/GLInt",

    config_kwargs={

        "bos_token_id": 50281,

        "eos_token_id": 50282,

        "pad_token_id": 50283,

    },

    processor_kwargs={"model_max_length": 8192, "max_length": 8192},

)

lateon = MultiVectorEncoder("lightonai/LateOn-regularized")



# Merge using simple 50/50 linear

model = MultiVectorEncoder.merge(

    [glint, lateon],

    method="linear",

    parameters={"weight": [0.5, 0.5]},

)

# Increase the document length from 300 to 8192 tokens

model[0].document_length = 8192

# And save it for reuse

model.save_pretrained("ModernBERT-base-en-v1")

```

### Framework Versions

- Python: 3.12.3
- Sentence Transformers: 6.2.0
- Transformers: 5.4.0
- PyTorch: 2.11.0+cu128
- Accelerate: 1.13.0
- Datasets: 4.8.5
- Tokenizers: 0.22.2

## Citation

### BibTeX

#### LateOn Regularization

```bibtex

@misc{chaffin2026lateonregularization,

  title={Party is over: regularizing ColBERT models to fix efficient ANN methods},

  author={Chaffin, Antoine},

  year={2026},

  howpublished={\url{https://huggingface.co/blog/lightonai/lateon-regularization}},

}

```

#### GLInt

```bibtex

@misc{aarush2026glint,

  title={GLInt: Geometry-Matched Hard Negatives for Late-Interaction Retrieval},

  author={Aarush},

  year={2026},

  howpublished={\url{https://huggingface.co/blog/chungimungi/glint}},

}

```

### Mergekit

```bibtex

@inproceedings{goddard-etal-2024-arcees,

    title = "Arcee{'}s {M}erge{K}it: A Toolkit for Merging Large Language Models",

    author = "Goddard, Charles and Siriwardhana, Shamane and Ehghaghi, Malikeh and Meyers, Luke and Karpukhin, Vladimir and Benedict, Brian and McQuade, Mark and Solawetz, Jacob",

    editor = "Dernoncourt, Franck and Preo{\c{t}}iuc-Pietro, Daniel and Shimorina, Anastasia",

    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track",

    month = nov,

    year = "2024",

    address = "Miami, Florida, US",

    publisher = "Association for Computational Linguistics",

    url = "https://aclanthology.org/2024.emnlp-industry.36",

    doi = "10.18653/v1/2024.emnlp-industry.36",

    pages = "477--485",

}

```