Desearch Embedding 4B

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

1_Pooling/config.json

@@ -0,0 +1,5 @@
+{
+    "embedding_dimension": 2560,
+    "pooling_mode": "lasttoken",
+    "include_prompt": true
+}

README.md

@@ -0,0 +1,141 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen3-Embedding-4B
+library_name: sentence-transformers
+pipeline_tag: sentence-similarity
+language:
+- en
+tags:
+- sentence-transformers
+- text-embeddings
+- retrieval
+- web-search
+- news
+- peft
+---
+
+# Desearch Embedding 4B
+
+Desearch Embedding 4B is a text embedding model for web and news search, developed by
+[Desearch](https://desearch.ai). It is fine-tuned from [Qwen/Qwen3-Embedding-4B](https://huggingface.co/Qwen/Qwen3-Embedding-4B)
+and maps search queries and web documents into a shared vector space for first-stage retrieval.
+
+## Highlights
+
+- **Built for news and the open web.** Fine-tuned on recent news coverage, reference pages and
+  encyclopedic articles, the content a web search engine has to rank every day.
+- **Real queries in every form.** Short keyword searches, natural-language questions, noisy
+  queries, questions about dated news events, and multi-hop questions that combine facts from
+  linked pages.
+- **One model for chunks and whole pages.** Documents are seen as paragraph chunks, page openings
+  and full pages, the units a search index stores.
+- **Binary first-stage ready.** Trained with a binary objective on the leading 256 dimensions, for
+  search stacks that shortlist with compact binary vectors before rescoring with full vectors.
+
+## Model details
+
+| | |
+|---|---|
+| Base model | [Qwen/Qwen3-Embedding-4B](https://huggingface.co/Qwen/Qwen3-Embedding-4B) |
+| Parameters | 4B |
+| Embedding dimension | 2560 |
+| Max sequence length | 32K tokens |
+| Pooling | Last token, L2-normalized |
+| Query instruction | Built-in `query` prompt |
+| Language | English |
+| Training method | LoRA fine-tuning |
+| License | Apache 2.0 |
+
+## Usage
+
+The weights are a LoRA adapter on Qwen/Qwen3-Embedding-4B; the base model downloads automatically.
+
+```bash
+pip install -U sentence-transformers peft
+```
+
+### Using Sentence Transformers
+
+```python
+from sentence_transformers import SentenceTransformer
+
+model = SentenceTransformer("desearch/Desearch-Embedding-4B")
+
+queries = [
+    "Who became Apple’s chief executive and will lead the Sept. 9, 2026 “Surprise and Shine” product event?",
+    "How much does the base Mac Mini M6 cost in US dollars as of its August 2026 announcement?",
+]
+documents = [
+    "John Ternus, Apple's new chief executive, takes the stage at the September 9, 2026 “Surprise and Shine” event to unveil the latest iPhones and Apple Watches.",
+    "Apple surprised buyers in late August 2026 with the Mac Mini M6 at $899 and the Mac Mini M5 Pro at $1,699, both shipping on September 22.",
+]
+
+query_embeddings = model.encode(queries, prompt_name="query")
+document_embeddings = model.encode(documents)
+print(model.similarity(query_embeddings, document_embeddings))
+```
+
+Queries use the built-in `query` prompt; documents are encoded as they are.
+
+### Using Transformers
+
+```python
+import torch
+import torch.nn.functional as F
+from peft import PeftModel
+from transformers import AutoModel, AutoTokenizer
+
+task = "Given a web search query, retrieve relevant passages that answer the query"
+questions = [
+    "Who became Apple’s chief executive and will lead the Sept. 9, 2026 “Surprise and Shine” product event?",
+    "How much does the base Mac Mini M6 cost in US dollars as of its August 2026 announcement?",
+]
+queries = [f"Instruct: {task}\nQuery:{q}" for q in questions]
+documents = [
+    "John Ternus, Apple's new chief executive, takes the stage at the September 9, 2026 “Surprise and Shine” event to unveil the latest iPhones and Apple Watches.",
+    "Apple surprised buyers in late August 2026 with the Mac Mini M6 at $899 and the Mac Mini M5 Pro at $1,699, both shipping on September 22.",
+]
+
+tokenizer = AutoTokenizer.from_pretrained("desearch/Desearch-Embedding-4B", padding_side="left")
+model = AutoModel.from_pretrained("Qwen/Qwen3-Embedding-4B", dtype=torch.bfloat16)
+model = PeftModel.from_pretrained(model, "desearch/Desearch-Embedding-4B").eval()
+
+
+def encode(texts):
+    batch = tokenizer(texts, padding=True, truncation=True, max_length=8192, return_tensors="pt")
+    with torch.no_grad():
+        hidden = model(**batch).last_hidden_state
+    return F.normalize(hidden[:, -1], p=2, dim=1)
+
+
+print(encode(queries) @ encode(documents).T)
+```
+
+## Recommended use cases
+
+- First-stage retrieval for web search, news search and retrieval-augmented generation
+- Semantic search over news archives, documentation and reference content
+- Candidate generation ahead of a reranker
+
+## Limitations
+
+- Tuned on English text; other languages are not a focus of this release.
+- Queries need the `query` prompt; documents are encoded without one.
+- Trained on documents up to 2,048 tokens; split longer pages into passages.
+- Embeddings differ from Qwen/Qwen3-Embedding-4B; re-embed an existing corpus when switching.
+
+## License
+
+This model is licensed under the Apache License 2.0. It is derived from Qwen/Qwen3-Embedding-4B, which is also
+licensed under the Apache License 2.0.
+
+## Citation
+
+```bibtex
+@misc{desearch2026embedding,
+  title  = {Desearch Embedding 4B},
+  author = {Desearch},
+  year   = {2026},
+  url    = {https://huggingface.co/desearch/Desearch-Embedding-4B}
+}
+```

adapter_config.json

@@ -0,0 +1,51 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "Qwen/Qwen3-Embedding-4B",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": false,
+  "init_lora_weights": true,
+  "kasa_config": null,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 64,
+  "lora_bias": false,
+  "lora_dropout": 0.0,
+  "lora_ga_config": null,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "monteclora_config": null,
+  "peft_type": "LORA",
+  "peft_version": "0.21.0",
+  "qalora_group_size": 16,
+  "r": 64,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "down_proj",
+    "up_proj",
+    "o_proj",
+    "q_proj",
+    "k_proj",
+    "gate_proj",
+    "v_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "FEATURE_EXTRACTION",
+  "trainable_token_indices": null,
+  "use_bdlora": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false,
+  "velora_config": null
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9bb94d1aa7a0b0d49f0be7de88e7c4367643c7675e10e5028909fd8aed8baf2a
+size 528547232

chat_template.jinja

@@ -0,0 +1,54 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- messages[0]['content'] }}
+    {%- else %}
+        {{- 'You are a helpful assistant.' }}
+    {%- endif %}
+    {{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0]['role'] == 'system' %}
+        {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
+    {%- else %}
+        {{- '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- for message in messages %}
+    {%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
+        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {{- '<|im_start|>' + message.role }}
+        {%- if message.content %}
+            {{- '\n' + message.content }}
+        {%- endif %}
+        {%- for tool_call in message.tool_calls %}
+            {%- if tool_call.function is defined %}
+                {%- set tool_call = tool_call.function %}
+            {%- endif %}
+            {{- '\n<tool_call>\n{"name": "' }}
+            {{- tool_call.name }}
+            {{- '", "arguments": ' }}
+            {{- tool_call.arguments | tojson }}
+            {{- '}\n</tool_call>' }}
+        {%- endfor %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- message.content }}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

config_sentence_transformers.json

@@ -0,0 +1,14 @@
+{
+  "__version__": {
+    "pytorch": "2.13.0+cu130",
+    "sentence_transformers": "5.7.0",
+    "transformers": "5.15.0"
+  },
+  "default_prompt_name": null,
+  "model_type": "SentenceTransformer",
+  "prompts": {
+    "document": "",
+    "query": "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery:"
+  },
+  "similarity_fn_name": "cosine"
+}

modules.json

@@ -0,0 +1,20 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.base.modules.transformer.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.sentence_transformer.modules.pooling.Pooling"
+  },
+  {
+    "idx": 2,
+    "name": "2",
+    "path": "2_Normalize",
+    "type": "sentence_transformers.sentence_transformer.modules.normalize.Normalize"
+  }
+]

sentence_bert_config.json

@@ -0,0 +1,10 @@
+{
+    "transformer_task": "feature-extraction",
+    "modality_config": {
+        "text": {
+            "method": "forward",
+            "method_output_name": "last_hidden_state"
+        }
+    },
+    "module_output_name": "token_embeddings"
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff800fee44eacd1a1c8c2f96b890ade65d722550ab6693efa8d2c368f40a5853
+size 11423209

tokenizer_config.json

@@ -0,0 +1,16 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 2048,
+  "pad_token": "<|endoftext|>",
+  "padding_side": "left",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}