File size: 5,221 Bytes
8c31416 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 | ---
license: cc-by-nc-sa-4.0
library_name: sentence-transformers
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- patent
- embeddings
- mteb
language:
- en
pipeline_tag: sentence-similarity
---
# patembed-base
This is a **sentence-transformers** model trained specifically for **patent text embeddings**. It is part of the **PatenTEB** project, which provides state-of-the-art models for patent document understanding and retrieval.
**Note:** This model uses task-specific instruction prompts during inference for optimal performance.
## Model Details
- **Model Type**: Sentence Transformer
- **Base Architecture**: Distilled from patembed-large using layers {0,2,4,6,8,10,12,14,16,18,20,22}
- **Parameters**: 193M
- **Number of Layers**: 12
- **Hidden Size**: 1024
- **Embedding Dimension**: 768
- **Max Sequence Length**: 512 tokens
- **Language**: English
- **License**: CC BY-NC-SA 4.0
## Model Description
Primary deployment target distilled from patembed-large. Maintains 1024 hidden size with projection to 768-dim embeddings.
This model is part of the **patembed family**, developed through multi-task learning on 13 training tasks from the PatenTEB benchmark. For detailed information about the training methodology, architecture, and comprehensive evaluation results, please refer to our paper.
## Usage
### Using Sentence Transformers
```python
from sentence_transformers import SentenceTransformer
# Load the model
model = SentenceTransformer('datalyes/patembed-base')
# Encode patent texts
patent_texts = [
"A method for manufacturing semiconductor devices...",
"An apparatus for processing chemical compounds...",
]
embeddings = model.encode(patent_texts)
# Compute similarity
from sentence_transformers import util
similarity = util.cos_sim(embeddings[0], embeddings[1])
print(f"Similarity: {similarity.item():.4f}")
```
### Using Transformers
```python
from transformers import AutoTokenizer, AutoModel
import torch
import torch.nn.functional as F
# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained('datalyes/patembed-base')
model = AutoModel.from_pretrained('datalyes/patembed-base')
def mean_pooling(model_output, attention_mask):
token_embeddings = model_output[0]
input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
# Tokenize and encode
texts = ["A method for manufacturing semiconductor devices..."]
encoded = tokenizer(texts, padding=True, truncation=True, return_tensors='pt')
with torch.no_grad():
model_output = model(**encoded)
embeddings = mean_pooling(model_output, encoded['attention_mask'])
embeddings = F.normalize(embeddings, p=2, dim=1)
```
### Patent Retrieval Example
```python
from sentence_transformers import SentenceTransformer, util
model = SentenceTransformer('datalyes/patembed-base')
# Query patent
query = "Method for reducing power consumption in mobile devices"
# Candidate patents
candidates = [
"A power management system for portable electronic devices...",
"Chemical composition for battery manufacturing...",
"Method for wireless data transmission in mobile networks...",
]
# Encode and retrieve
query_emb = model.encode(query)
candidate_embs = model.encode(candidates)
# Compute similarities
scores = util.cos_sim(query_emb, candidate_embs)[0]
# Get ranked results
results = [(candidates[i], scores[i].item()) for i in range(len(candidates))]
results.sort(key=lambda x: x[1], reverse=True)
for patent, score in results:
print(f"Score: {score:.4f} - {patent[:100]}...")
```
## Intended Use
This model is designed for patent-specific tasks including:
- Patent search and retrieval
- Prior art search
- Patent classification and clustering
- Technology landscape analysis
For detailed training methodology, evaluation protocols, and performance analysis, please refer to our paper.
## Citation
If you use this model, please cite our paper:
```bibtex
@misc{ayaou2025patentebcomprehensivebenchmarkmodel,
title={PatenTEB: A Comprehensive Benchmark and Model Family for Patent Text Embedding},
author={Iliass Ayaou and Denis Cavallucci},
year={2025},
eprint={2510.22264},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2510.22264}
}
```
**Paper**: [PatenTEB on arXiv](https://arxiv.org/abs/2510.22264)
## License
This model is released under the **Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)** license.
**Key Terms:**
- ✅ You can use, share, and adapt the model
- ✅ You must give appropriate credit
- ❌ You may not use the model for commercial purposes
- ⚠️ If you adapt or build upon this model, you must distribute under the same license
For full license details: https://creativecommons.org/licenses/by-nc-sa/4.0/
## Contact
- **Authors**: Iliass Ayaou, Denis Cavallucci
- **Institution**: ICUBE Laboratory, INSA Strasbourg
- **GitHub**: [PatentTEB/PatentTEB](https://github.com/iliass-y/patenteb)
- **HuggingFace**: [datalyes](https://huggingface.co/datalyes)
|