Feature Extraction
sentence-transformers
Safetensors
Transformers
multilingual
qwen3
text-embeddings
alfotech
silas
rag
semantic-search
vector-search
Eval Results (legacy)
text-embeddings-inference
Instructions to use alfotech/silas-embedding-0.6b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use alfotech/silas-embedding-0.6b with sentence-transformers:
from sentence_transformers import SentenceTransformer model = SentenceTransformer("alfotech/silas-embedding-0.6b") sentences = [ "The weather is lovely today.", "It's so sunny outside!", "He drove to the stadium." ] embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] - Transformers
How to use alfotech/silas-embedding-0.6b with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="alfotech/silas-embedding-0.6b")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("alfotech/silas-embedding-0.6b") model = AutoModel.from_pretrained("alfotech/silas-embedding-0.6b", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -1,292 +1,307 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
base_model:
|
| 4 |
-
- Qwen/Qwen3-0.6B
|
|
|
|
|
|
|
| 5 |
tags:
|
| 6 |
- transformers
|
| 7 |
- sentence-transformers
|
| 8 |
-
-
|
| 9 |
- feature-extraction
|
| 10 |
-
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
---
|
| 12 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
|
| 14 |
<p align="center">
|
| 15 |
-
|
| 16 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 17 |
|
| 18 |
## Highlights
|
| 19 |
|
| 20 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21 |
|
| 22 |
-
|
| 23 |
|
| 24 |
-
|
| 25 |
|
| 26 |
-
|
| 27 |
|
| 28 |
## Model Overview
|
| 29 |
|
| 30 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
|
| 32 |
-
-
|
| 33 |
-
- Supported Languages: 100+ Languages
|
| 34 |
-
- Number of Parameters: 0.6B
|
| 35 |
-
- Context Length: 32k
|
| 36 |
-
- Embedding Dimension: Up to 1024, supports user-defined output dimensions ranging from 32 to 1024
|
| 37 |
|
| 38 |
-
|
| 39 |
|
| 40 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 41 |
|
| 42 |
-
|
| 43 |
-
|------------------|----------------------|------|--------|-----------------|---------------------|-------------|----------------|
|
| 44 |
-
| Text Embedding | [Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) | 0.6B | 28 | 32K | 1024 | Yes | Yes |
|
| 45 |
-
| Text Embedding | [Qwen3-Embedding-4B](https://huggingface.co/Qwen/Qwen3-Embedding-4B) | 4B | 36 | 32K | 2560 | Yes | Yes |
|
| 46 |
-
| Text Embedding | [Qwen3-Embedding-8B](https://huggingface.co/Qwen/Qwen3-Embedding-8B) | 8B | 36 | 32K | 4096 | Yes | Yes |
|
| 47 |
-
| Text Reranking | [Qwen3-Reranker-0.6B](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B) | 0.6B | 28 | 32K | - | - | Yes |
|
| 48 |
-
| Text Reranking | [Qwen3-Reranker-4B](https://huggingface.co/Qwen/Qwen3-Reranker-4B) | 4B | 36 | 32K | - | - | Yes |
|
| 49 |
-
| Text Reranking | [Qwen3-Reranker-8B](https://huggingface.co/Qwen/Qwen3-Reranker-8B) | 8B | 36 | 32K | - | - | Yes |
|
| 50 |
|
| 51 |
-
|
| 52 |
-
> - `MRL Support` indicates whether the embedding model supports custom dimensions for the final embedding.
|
| 53 |
-
> - `Instruction Aware` notes whether the embedding or reranking model supports customizing the input instruction according to different tasks.
|
| 54 |
-
> - Our evaluation indicates that, for most downstream tasks, using instructions (instruct) typically yields an improvement of 1% to 5% compared to not using them. Therefore, we recommend that developers create tailored instructions specific to their tasks and scenarios. In multilingual contexts, we also advise users to write their instructions in English, as most instructions utilized during the model training process were originally written in English.
|
| 55 |
|
| 56 |
-
##
|
| 57 |
|
| 58 |
-
|
| 59 |
-
```
|
| 60 |
-
KeyError: 'qwen3'
|
| 61 |
-
```
|
| 62 |
|
| 63 |
-
###
|
| 64 |
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 68 |
|
| 69 |
-
|
| 70 |
|
| 71 |
-
#
|
| 72 |
-
|
| 73 |
-
|
| 74 |
-
|
| 75 |
-
#
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
queries = [
|
| 84 |
-
"What is the capital of China?",
|
| 85 |
-
"Explain gravity",
|
| 86 |
-
]
|
| 87 |
-
documents = [
|
| 88 |
-
"The capital of China is Beijing.",
|
| 89 |
-
"Gravity is a force that attracts two bodies towards each other. It gives weight to physical objects and is responsible for the movement of planets around the sun.",
|
| 90 |
-
]
|
| 91 |
-
|
| 92 |
-
# Encode the queries and documents. Note that queries benefit from using a prompt
|
| 93 |
-
# Here we use the prompt called "query" stored under `model.prompts`, but you can
|
| 94 |
-
# also pass your own prompt via the `prompt` argument
|
| 95 |
-
query_embeddings = model.encode(queries, prompt_name="query")
|
| 96 |
-
document_embeddings = model.encode(documents)
|
| 97 |
-
|
| 98 |
-
# Compute the (cosine) similarity between the query and document embeddings
|
| 99 |
-
similarity = model.similarity(query_embeddings, document_embeddings)
|
| 100 |
-
print(similarity)
|
| 101 |
-
# tensor([[0.7646, 0.1414],
|
| 102 |
-
# [0.1355, 0.6000]])
|
| 103 |
-
```
|
| 104 |
|
| 105 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 106 |
|
| 107 |
```python
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
|
| 117 |
-
|
| 118 |
-
attention_mask: Tensor) -> Tensor:
|
| 119 |
-
left_padding = (attention_mask[:, -1].sum() == attention_mask.shape[0])
|
| 120 |
-
if left_padding:
|
| 121 |
-
return last_hidden_states[:, -1]
|
| 122 |
-
else:
|
| 123 |
-
sequence_lengths = attention_mask.sum(dim=1) - 1
|
| 124 |
-
batch_size = last_hidden_states.shape[0]
|
| 125 |
-
return last_hidden_states[torch.arange(batch_size, device=last_hidden_states.device), sequence_lengths]
|
| 126 |
-
|
| 127 |
-
|
| 128 |
-
def get_detailed_instruct(task_description: str, query: str) -> str:
|
| 129 |
-
return f'Instruct: {task_description}\nQuery:{query}'
|
| 130 |
-
|
| 131 |
-
# Each query must come with a one-sentence instruction that describes the task
|
| 132 |
-
task = 'Given a web search query, retrieve relevant passages that answer the query'
|
| 133 |
-
|
| 134 |
-
queries = [
|
| 135 |
-
get_detailed_instruct(task, 'What is the capital of China?'),
|
| 136 |
-
get_detailed_instruct(task, 'Explain gravity')
|
| 137 |
-
]
|
| 138 |
-
# No need to add instruction for retrieval documents
|
| 139 |
-
documents = [
|
| 140 |
-
"The capital of China is Beijing.",
|
| 141 |
-
"Gravity is a force that attracts two bodies towards each other. It gives weight to physical objects and is responsible for the movement of planets around the sun."
|
| 142 |
-
]
|
| 143 |
-
input_texts = queries + documents
|
| 144 |
-
|
| 145 |
-
tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen3-Embedding-0.6B', padding_side='left')
|
| 146 |
-
model = AutoModel.from_pretrained('Qwen/Qwen3-Embedding-0.6B')
|
| 147 |
-
|
| 148 |
-
# We recommend enabling flash_attention_2 for better acceleration and memory saving.
|
| 149 |
-
# model = AutoModel.from_pretrained('Qwen/Qwen3-Embedding-0.6B', attn_implementation="flash_attention_2", torch_dtype=torch.float16).cuda()
|
| 150 |
-
|
| 151 |
-
max_length = 8192
|
| 152 |
-
|
| 153 |
-
# Tokenize the input texts
|
| 154 |
-
batch_dict = tokenizer(
|
| 155 |
-
input_texts,
|
| 156 |
-
padding=True,
|
| 157 |
-
truncation=True,
|
| 158 |
-
max_length=max_length,
|
| 159 |
-
return_tensors="pt",
|
| 160 |
)
|
| 161 |
-
batch_dict.to(model.device)
|
| 162 |
-
outputs = model(**batch_dict)
|
| 163 |
-
embeddings = last_token_pool(outputs.last_hidden_state, batch_dict['attention_mask'])
|
| 164 |
-
|
| 165 |
-
# normalize embeddings
|
| 166 |
-
embeddings = F.normalize(embeddings, p=2, dim=1)
|
| 167 |
-
scores = (embeddings[:2] @ embeddings[2:].T)
|
| 168 |
-
print(scores.tolist())
|
| 169 |
-
# [[0.7645568251609802, 0.14142508804798126], [0.13549736142158508, 0.5999549627304077]]
|
| 170 |
```
|
| 171 |
|
| 172 |
-
###
|
| 173 |
|
| 174 |
-
|
| 175 |
-
|
| 176 |
-
|
| 177 |
-
|
| 178 |
-
|
| 179 |
-
|
| 180 |
-
|
| 181 |
-
|
| 182 |
-
|
| 183 |
-
# Each query must come with a one-sentence instruction that describes the task
|
| 184 |
-
task = 'Given a web search query, retrieve relevant passages that answer the query'
|
| 185 |
-
|
| 186 |
-
queries = [
|
| 187 |
-
get_detailed_instruct(task, 'What is the capital of China?'),
|
| 188 |
-
get_detailed_instruct(task, 'Explain gravity')
|
| 189 |
-
]
|
| 190 |
-
# No need to add instruction for retrieval documents
|
| 191 |
-
documents = [
|
| 192 |
-
"The capital of China is Beijing.",
|
| 193 |
-
"Gravity is a force that attracts two bodies towards each other. It gives weight to physical objects and is responsible for the movement of planets around the sun."
|
| 194 |
-
]
|
| 195 |
-
input_texts = queries + documents
|
| 196 |
-
|
| 197 |
-
model = LLM(model="Qwen/Qwen3-Embedding-0.6B", task="embed")
|
| 198 |
-
|
| 199 |
-
outputs = model.embed(input_texts)
|
| 200 |
-
embeddings = torch.tensor([o.outputs.embedding for o in outputs])
|
| 201 |
-
scores = (embeddings[:2] @ embeddings[2:].T)
|
| 202 |
-
print(scores.tolist())
|
| 203 |
-
# [[0.7620252966880798, 0.14078938961029053], [0.1358368694782257, 0.6013815999031067]]
|
| 204 |
```
|
| 205 |
|
| 206 |
-
|
| 207 |
|
| 208 |
-
|
| 209 |
|
| 210 |
-
|
| 211 |
|
| 212 |
-
|
| 213 |
-
docker run --gpus all -p 8080:80 -v hf_cache:/data --pull always ghcr.io/huggingface/text-embeddings-inference:cpu-1.7.2 --model-id Qwen/Qwen3-Embedding-0.6B --dtype float16
|
| 214 |
-
```
|
| 215 |
|
| 216 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 217 |
|
| 218 |
-
|
| 219 |
-
docker run -p 8080:80 -v hf_cache:/data --pull always ghcr.io/huggingface/text-embeddings-inference:1.7.2 --model-id Qwen/Qwen3-Embedding-0.6B
|
| 220 |
-
```
|
| 221 |
|
| 222 |
-
|
| 223 |
|
| 224 |
-
|
| 225 |
-
|
| 226 |
-
|
| 227 |
-
|
| 228 |
-
|
| 229 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 230 |
|
| 231 |
-
##
|
| 232 |
-
|
| 233 |
-
### MTEB (Multilingual)
|
| 234 |
-
|
| 235 |
-
| Model | Size | Mean (Task) | Mean (Type) | Bitxt Mining | Class. | Clust. | Inst. Retri. | Multi. Class. | Pair. Class. | Rerank | Retri. | STS |
|
| 236 |
-
|----------------------------------|:-------:|:-------------:|:-------------:|:--------------:|:--------:|:--------:|:--------------:|:---------------:|:--------------:|:--------:|:--------:|:------:|
|
| 237 |
-
| NV-Embed-v2 | 7B | 56.29 | 49.58 | 57.84 | 57.29 | 40.80 | 1.04 | 18.63 | 78.94 | 63.82 | 56.72 | 71.10|
|
| 238 |
-
| GritLM-7B | 7B | 60.92 | 53.74 | 70.53 | 61.83 | 49.75 | 3.45 | 22.77 | 79.94 | 63.78 | 58.31 | 73.33|
|
| 239 |
-
| BGE-M3 | 0.6B | 59.56 | 52.18 | 79.11 | 60.35 | 40.88 | -3.11 | 20.1 | 80.76 | 62.79 | 54.60 | 74.12|
|
| 240 |
-
| multilingual-e5-large-instruct | 0.6B | 63.22 | 55.08 | 80.13 | 64.94 | 50.75 | -0.40 | 22.91 | 80.86 | 62.61 | 57.12 | 76.81|
|
| 241 |
-
| gte-Qwen2-1.5B-instruct | 1.5B | 59.45 | 52.69 | 62.51 | 58.32 | 52.05 | 0.74 | 24.02 | 81.58 | 62.58 | 60.78 | 71.61|
|
| 242 |
-
| gte-Qwen2-7b-Instruct | 7B | 62.51 | 55.93 | 73.92 | 61.55 | 52.77 | 4.94 | 25.48 | 85.13 | 65.55 | 60.08 | 73.98|
|
| 243 |
-
| text-embedding-3-large | - | 58.93 | 51.41 | 62.17 | 60.27 | 46.89 | -2.68 | 22.03 | 79.17 | 63.89 | 59.27 | 71.68|
|
| 244 |
-
| Cohere-embed-multilingual-v3.0 | - | 61.12 | 53.23 | 70.50 | 62.95 | 46.89 | -1.89 | 22.74 | 79.88 | 64.07 | 59.16 | 74.80|
|
| 245 |
-
| Gemini Embedding | - | 68.37 | 59.59 | 79.28 | 71.82 | 54.59 | 5.18 | **29.16** | 83.63 | 65.58 | 67.71 | 79.40|
|
| 246 |
-
| **Qwen3-Embedding-0.6B** | 0.6B | 64.33 | 56.00 | 72.22 | 66.83 | 52.33 | 5.09 | 24.59 | 80.83 | 61.41 | 64.64 | 76.17|
|
| 247 |
-
| **Qwen3-Embedding-4B** | 4B | 69.45 | 60.86 | 79.36 | 72.33 | 57.15 | **11.56** | 26.77 | 85.05 | 65.08 | 69.60 | 80.86|
|
| 248 |
-
| **Qwen3-Embedding-8B** | 8B | **70.58** | **61.69** | **80.89** | **74.00** | **57.65** | 10.06 | 28.66 | **86.40** | **65.63** | **70.88** | **81.08** |
|
| 249 |
-
|
| 250 |
-
> **Note**: For compared models, the scores are retrieved from MTEB online [leaderboard](https://huggingface.co/spaces/mteb/leaderboard) on May 24th, 2025.
|
| 251 |
-
|
| 252 |
-
### MTEB (Eng v2)
|
| 253 |
-
|
| 254 |
-
| MTEB English / Models | Param. | Mean(Task) | Mean(Type) | Class. | Clust. | Pair Class. | Rerank. | Retri. | STS | Summ. |
|
| 255 |
-
|--------------------------------|:--------:|:------------:|:------------:|:--------:|:--------:|:-------------:|:---------:|:--------:|:-------:|:-------:|
|
| 256 |
-
| multilingual-e5-large-instruct | 0.6B | 65.53 | 61.21 | 75.54 | 49.89 | 86.24 | 48.74 | 53.47 | 84.72 | 29.89 |
|
| 257 |
-
| NV-Embed-v2 | 7.8B | 69.81 | 65.00 | 87.19 | 47.66 | 88.69 | 49.61 | 62.84 | 83.82 | 35.21 |
|
| 258 |
-
| GritLM-7B | 7.2B | 67.07 | 63.22 | 81.25 | 50.82 | 87.29 | 49.59 | 54.95 | 83.03 | 35.65 |
|
| 259 |
-
| gte-Qwen2-1.5B-instruct | 1.5B | 67.20 | 63.26 | 85.84 | 53.54 | 87.52 | 49.25 | 50.25 | 82.51 | 33.94 |
|
| 260 |
-
| stella_en_1.5B_v5 | 1.5B | 69.43 | 65.32 | 89.38 | 57.06 | 88.02 | 50.19 | 52.42 | 83.27 | 36.91 |
|
| 261 |
-
| gte-Qwen2-7B-instruct | 7.6B | 70.72 | 65.77 | 88.52 | 58.97 | 85.9 | 50.47 | 58.09 | 82.69 | 35.74 |
|
| 262 |
-
| gemini-embedding-exp-03-07 | - | 73.3 | 67.67 | 90.05 | 59.39 | 87.7 | 48.59 | 64.35 | 85.29 | 38.28 |
|
| 263 |
-
| **Qwen3-Embedding-0.6B** | 0.6B | 70.70 | 64.88 | 85.76 | 54.05 | 84.37 | 48.18 | 61.83 | 86.57 | 33.43 |
|
| 264 |
-
| **Qwen3-Embedding-4B** | 4B | 74.60 | 68.10 | 89.84 | 57.51 | 87.01 | 50.76 | 68.46 | 88.72 | 34.39 |
|
| 265 |
-
| **Qwen3-Embedding-8B** | 8B | 75.22 | 68.71 | 90.43 | 58.57 | 87.52 | 51.56 | 69.44 | 88.58 | 34.83 |
|
| 266 |
-
|
| 267 |
-
### C-MTEB (MTEB Chinese)
|
| 268 |
-
|
| 269 |
-
| C-MTEB | Param. | Mean(Task) | Mean(Type) | Class. | Clust. | Pair Class. | Rerank. | Retr. | STS |
|
| 270 |
-
|------------------|--------|------------|------------|--------|--------|-------------|---------|-------|-------|
|
| 271 |
-
| multilingual-e5-large-instruct | 0.6B | 58.08 | 58.24 | 69.80 | 48.23 | 64.52 | 57.45 | 63.65 | 45.81 |
|
| 272 |
-
| bge-multilingual-gemma2 | 9B | 67.64 | 75.31 | 59.30 | 86.67 | 68.28 | 73.73 | 55.19 | - |
|
| 273 |
-
| gte-Qwen2-1.5B-instruct | 1.5B | 67.12 | 67.79 | 72.53 | 54.61 | 79.5 | 68.21 | 71.86 | 60.05 |
|
| 274 |
-
| gte-Qwen2-7B-instruct | 7.6B | 71.62 | 72.19 | 75.77 | 66.06 | 81.16 | 69.24 | 75.70 | 65.20 |
|
| 275 |
-
| ritrieve_zh_v1 | 0.3B | 72.71 | 73.85 | 76.88 | 66.5 | 85.98 | 72.86 | 76.97 | 63.92 |
|
| 276 |
-
| **Qwen3-Embedding-0.6B** | 0.6B | 66.33 | 67.45 | 71.40 | 68.74 | 76.42 | 62.58 | 71.03 | 54.52 |
|
| 277 |
-
| **Qwen3-Embedding-4B** | 4B | 72.27 | 73.51 | 75.46 | 77.89 | 83.34 | 66.05 | 77.03 | 61.26 |
|
| 278 |
-
| **Qwen3-Embedding-8B** | 8B | 73.84 | 75.00 | 76.97 | 80.08 | 84.23 | 66.99 | 78.21 | 63.53 |
|
| 279 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 280 |
|
| 281 |
## Citation
|
| 282 |
|
| 283 |
-
If
|
| 284 |
|
| 285 |
-
|
|
|
|
|
|
|
| 286 |
@article{qwen3embedding,
|
| 287 |
title={Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models},
|
| 288 |
-
author={Zhang, Yanzhao and Li, Mingxin and Long, Dingkun and
|
| 289 |
journal={arXiv preprint arXiv:2506.05176},
|
| 290 |
year={2025}
|
| 291 |
}
|
| 292 |
-
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
base_model:
|
| 4 |
+
- Qwen/Qwen3-Embedding-0.6B
|
| 5 |
+
pipeline_tag: sentence-similarity
|
| 6 |
+
library_name: sentence-transformers
|
| 7 |
tags:
|
| 8 |
- transformers
|
| 9 |
- sentence-transformers
|
| 10 |
+
- text-embeddings
|
| 11 |
- feature-extraction
|
| 12 |
+
- retrieval
|
| 13 |
+
- rag
|
| 14 |
+
- multilingual
|
| 15 |
+
- semantic-search
|
| 16 |
+
- qwen3
|
| 17 |
+
- alfotech
|
| 18 |
---
|
| 19 |
+
|
| 20 |
+
# Alfo Embedding 0.6B
|
| 21 |
+
|
| 22 |
+
<p align="center">
|
| 23 |
+
<img src="https://huggingface.co/alfotech/alfo-tech-logo/resolve/main/logo.png" width="220"/>
|
| 24 |
+
</p>
|
| 25 |
|
| 26 |
<p align="center">
|
| 27 |
+
|
| 28 |
+
**Production-oriented multilingual embedding model by Alfo Tech Industries**
|
| 29 |
+
|
| 30 |
+
</p>
|
| 31 |
+
|
| 32 |
+
---
|
| 33 |
|
| 34 |
## Highlights
|
| 35 |
|
| 36 |
+
Alfo Embedding 0.6B is a production-focused embedding model built on **Qwen3-Embedding-0.6B**, optimized through an evaluation-first retrieval pipeline.
|
| 37 |
+
|
| 38 |
+
Instead of training from scratch, this project focuses on improving real-world retrieval quality using:
|
| 39 |
+
|
| 40 |
+
- Hard retrieval evaluation
|
| 41 |
+
- Semantic hard-negative analysis
|
| 42 |
+
- Contrastive embedding fine-tuning
|
| 43 |
+
- Retrieval-oriented optimization
|
| 44 |
+
- Production deployment validation
|
| 45 |
|
| 46 |
+
The objective is simple:
|
| 47 |
|
| 48 |
+
> Build practical AI infrastructure for semantic search and Retrieval-Augmented Generation.
|
| 49 |
|
| 50 |
+
---
|
| 51 |
|
| 52 |
## Model Overview
|
| 53 |
|
| 54 |
+
| Property | Value |
|
| 55 |
+
|----------|-------|
|
| 56 |
+
| Model Type | Text Embedding |
|
| 57 |
+
| Base Model | Qwen3-Embedding-0.6B |
|
| 58 |
+
| Parameters | 0.6B |
|
| 59 |
+
| Context Length | 32K |
|
| 60 |
+
| Default Embedding Dimension | 1024 |
|
| 61 |
+
| Supported Dimensions | 32–1024 |
|
| 62 |
+
| Languages | 100+ |
|
| 63 |
+
| Instruction Aware | Yes |
|
| 64 |
|
| 65 |
+
The upstream Qwen3 embedding architecture provides multilingual understanding, long-context retrieval, configurable embedding dimensions, and instruction-aware query encoding. This project extends that foundation with a retrieval-focused development workflow.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 66 |
|
| 67 |
+
---
|
| 68 |
|
| 69 |
+
## Development Philosophy
|
| 70 |
+
|
| 71 |
+
Rather than optimizing only for easy benchmarks, the training pipeline intentionally focuses on difficult retrieval scenarios.
|
| 72 |
+
|
| 73 |
+
Pipeline:
|
| 74 |
+
|
| 75 |
+
```text
|
| 76 |
+
Baseline
|
| 77 |
+
↓
|
| 78 |
+
Hard Benchmark
|
| 79 |
+
↓
|
| 80 |
+
Failure Analysis
|
| 81 |
+
↓
|
| 82 |
+
Hard Negative Mining
|
| 83 |
+
↓
|
| 84 |
+
Fine-Tuning
|
| 85 |
+
↓
|
| 86 |
+
Validation
|
| 87 |
+
↓
|
| 88 |
+
Production Testing
|
| 89 |
+
```
|
| 90 |
|
| 91 |
+
Every optimization is driven by measured evaluation rather than intuition.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 92 |
|
| 93 |
+
---
|
|
|
|
|
|
|
|
|
|
| 94 |
|
| 95 |
+
## Training Pipeline
|
| 96 |
|
| 97 |
+
Development was performed inside **Google Colab** using dual Tesla T4 GPUs.
|
|
|
|
|
|
|
|
|
|
| 98 |
|
| 99 |
+
### Pipeline Components
|
| 100 |
|
| 101 |
+
- Retrieval dataset preparation
|
| 102 |
+
- Duplicate removal
|
| 103 |
+
- Quality filtering
|
| 104 |
+
- Validation leakage checks
|
| 105 |
+
- Hard-negative mining
|
| 106 |
+
- Multiple Negatives Ranking Loss
|
| 107 |
+
- In-batch negatives
|
| 108 |
+
- Continuous retrieval evaluation
|
| 109 |
+
- Production validation
|
| 110 |
|
| 111 |
+
---
|
| 112 |
|
| 113 |
+
## Evaluation Strategy
|
| 114 |
+
|
| 115 |
+
The model is evaluated using both easy and difficult retrieval settings.
|
| 116 |
+
|
| 117 |
+
### Easy Benchmark
|
| 118 |
+
|
| 119 |
+
| Metric | Score |
|
| 120 |
+
|--------|-------|
|
| 121 |
+
| Recall@1 | **0.967** |
|
| 122 |
+
| Recall@5 | **0.999** |
|
| 123 |
+
| Recall@10 | **1.000** |
|
| 124 |
+
| MRR@10 | **0.9815** |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 125 |
|
| 126 |
+
This benchmark validates the retrieval pipeline under standard retrieval conditions.
|
| 127 |
+
|
| 128 |
+
---
|
| 129 |
+
|
| 130 |
+
### Hard Retrieval Benchmark
|
| 131 |
+
|
| 132 |
+
Corpus size:
|
| 133 |
+
|
| 134 |
+
**72,635 documents**
|
| 135 |
+
|
| 136 |
+
Evaluation queries:
|
| 137 |
+
|
| 138 |
+
**1,000**
|
| 139 |
+
|
| 140 |
+
| Metric | Score |
|
| 141 |
+
|--------|-------|
|
| 142 |
+
| Recall@1 | **0.614** |
|
| 143 |
+
| Recall@5 | **0.883** |
|
| 144 |
+
| Recall@10 | **0.929** |
|
| 145 |
+
| MRR@10 | **0.7268** |
|
| 146 |
+
|
| 147 |
+
This benchmark introduces semantically similar distractors, making it significantly more representative of real-world retrieval systems.
|
| 148 |
+
|
| 149 |
+
---
|
| 150 |
+
|
| 151 |
+
## Failure Analysis
|
| 152 |
+
|
| 153 |
+
Hard benchmark analysis identified semantic discrimination as the primary remaining challenge.
|
| 154 |
+
|
| 155 |
+
Summary:
|
| 156 |
+
|
| 157 |
+
- Semantic hard negatives: **386**
|
| 158 |
+
- Positive similarity: **0.689**
|
| 159 |
+
- Top negative similarity: **0.653**
|
| 160 |
+
- Average margin: **0.036**
|
| 161 |
+
- Duplicate contamination detected: **0**
|
| 162 |
+
- Noisy positives detected: **0**
|
| 163 |
+
|
| 164 |
+
These observations guided the retrieval-focused optimization process.
|
| 165 |
+
|
| 166 |
+
---
|
| 167 |
+
|
| 168 |
+
## Recommended Use Cases
|
| 169 |
+
|
| 170 |
+
- Semantic Search
|
| 171 |
+
- RAG
|
| 172 |
+
- Enterprise Search
|
| 173 |
+
- Knowledge Base Retrieval
|
| 174 |
+
- FAQ Matching
|
| 175 |
+
- Code Search
|
| 176 |
+
- Similarity Search
|
| 177 |
+
- Vector Databases
|
| 178 |
+
- Multilingual Retrieval
|
| 179 |
+
|
| 180 |
+
---
|
| 181 |
+
|
| 182 |
+
## Usage
|
| 183 |
+
|
| 184 |
+
### Sentence Transformers
|
| 185 |
|
| 186 |
```python
|
| 187 |
+
from sentence_transformers import SentenceTransformer
|
| 188 |
+
|
| 189 |
+
model = SentenceTransformer("alfotech/alfo-embedding-0.6b")
|
| 190 |
+
|
| 191 |
+
embeddings = model.encode(
|
| 192 |
+
[
|
| 193 |
+
"Artificial intelligence improves search.",
|
| 194 |
+
"Vector embeddings represent semantic meaning."
|
| 195 |
+
],
|
| 196 |
+
normalize_embeddings=True
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 197 |
)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 198 |
```
|
| 199 |
|
| 200 |
+
### Retrieval Queries
|
| 201 |
|
| 202 |
+
The underlying Qwen3 embedding family supports instruction-aware retrieval.
|
| 203 |
+
|
| 204 |
+
Example:
|
| 205 |
+
|
| 206 |
+
```text
|
| 207 |
+
Instruct: Given a web search query, retrieve relevant passages that answer the query.
|
| 208 |
+
|
| 209 |
+
Query: How can I optimize transformer inference?
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 210 |
```
|
| 211 |
|
| 212 |
+
Using retrieval instructions can improve search performance for many production workloads.
|
| 213 |
|
| 214 |
+
---
|
| 215 |
|
| 216 |
+
## Production Recommendations
|
| 217 |
|
| 218 |
+
Recommended starting configuration.
|
|
|
|
|
|
|
| 219 |
|
| 220 |
+
| Setting | Value |
|
| 221 |
+
|---------|-------|
|
| 222 |
+
| Sequence Length | 512 |
|
| 223 |
+
| Batch Size | 32 |
|
| 224 |
+
| Normalize Embeddings | Enabled |
|
| 225 |
+
| Similarity | Cosine |
|
| 226 |
|
| 227 |
+
These settings provide a practical balance between retrieval quality and inference throughput.
|
|
|
|
|
|
|
| 228 |
|
| 229 |
+
---
|
| 230 |
|
| 231 |
+
## Roadmap
|
| 232 |
+
|
| 233 |
+
Planned improvements include:
|
| 234 |
+
|
| 235 |
+
- Larger curated retrieval datasets
|
| 236 |
+
- Better hard-negative refinement
|
| 237 |
+
- Domain-specific adaptation
|
| 238 |
+
- Full blind evaluation
|
| 239 |
+
- Extended benchmark coverage
|
| 240 |
+
- Additional deployment optimizations
|
| 241 |
+
|
| 242 |
+
---
|
| 243 |
|
| 244 |
+
## Transparency
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 245 |
|
| 246 |
+
This repository prioritizes reproducibility.
|
| 247 |
+
|
| 248 |
+
Current documentation includes:
|
| 249 |
+
|
| 250 |
+
- Training methodology
|
| 251 |
+
- Retrieval evaluation
|
| 252 |
+
- Hard benchmark results
|
| 253 |
+
- Production inference examples
|
| 254 |
+
- Deployment recommendations
|
| 255 |
+
|
| 256 |
+
External benchmark leaderboard claims are intentionally omitted unless independently measured.
|
| 257 |
+
|
| 258 |
+
---
|
| 259 |
+
|
| 260 |
+
## Acknowledgements
|
| 261 |
+
|
| 262 |
+
This project builds upon the open-source **Qwen3-Embedding-0.6B** foundation model, whose multilingual and long-context embedding architecture made this work possible.
|
| 263 |
+
|
| 264 |
+
---
|
| 265 |
|
| 266 |
## Citation
|
| 267 |
|
| 268 |
+
If this project helps your work, please cite both the upstream Qwen3 Embedding paper and this repository.
|
| 269 |
|
| 270 |
+
### Qwen3 Embedding
|
| 271 |
+
|
| 272 |
+
```bibtex
|
| 273 |
@article{qwen3embedding,
|
| 274 |
title={Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models},
|
| 275 |
+
author={Zhang, Yanzhao and Li, Mingxin and Long, Dingkun and others},
|
| 276 |
journal={arXiv preprint arXiv:2506.05176},
|
| 277 |
year={2025}
|
| 278 |
}
|
| 279 |
+
```
|
| 280 |
+
|
| 281 |
+
### Alfo Embedding
|
| 282 |
+
|
| 283 |
+
```bibtex
|
| 284 |
+
@misc{alfotech_alfo_embedding_2026,
|
| 285 |
+
title={Alfo Embedding 0.6B},
|
| 286 |
+
author={Alfo Tech Industries},
|
| 287 |
+
year={2026},
|
| 288 |
+
publisher={Hugging Face},
|
| 289 |
+
url={https://huggingface.co/alfotech/alfo-embedding-0.6b}
|
| 290 |
+
}
|
| 291 |
+
```
|
| 292 |
+
|
| 293 |
+
---
|
| 294 |
+
|
| 295 |
+
## License
|
| 296 |
+
|
| 297 |
+
This repository follows the **Apache-2.0** licensing terms inherited from the upstream Qwen3-Embedding-0.6B model.
|
| 298 |
+
|
| 299 |
+
---
|
| 300 |
+
|
| 301 |
+
<p align="center">
|
| 302 |
+
|
| 303 |
+
**Built by Alfo Tech Industries**
|
| 304 |
+
|
| 305 |
+
*Building practical AI infrastructure for developers, startups, and enterprises.*
|
| 306 |
+
|
| 307 |
+
</p>
|